Development of a Remote Examination of Deglutition Based on Consensus Surveys of Clinicians (Part I): Selection of Examination Items.

In order to ensure appropriateness and feasibility of examination items for remote evaluation for deglutition disorders, a questionnaire based on the Delphi method was administered to 122 speech-language-hearing therapists (STs), and a set of examination items was selected. The participants were instructed to view a video recording of a remote assessment situation and answer a 30-item questionnaire. Of 19 items ensuring the appropriateness for deglutition disorders detection, 13 items ultimately met the consensus criteria for remote feasibility. Factor analysis extracted three factors: 'oral observation,' 'overall evaluation,' and 'perceptual voice judgment.' In free-text responses, "quality and stability of the voice that may be heard through the device" were the most common concerns, followed by "the need to correct of the camera angle, magnification, and targets that should be projected," "concerns about the technical aspects of the assistants and their role in relation with the examiner/ST," and "the need for palpation as well as visual confirmation." The proposed 13-item examination is considered to capture the characteristics of deglutition disorders, while items that appeared difficult to implement remotely were excluded. The fact that some items could be influenced by the video calling experience when judging the feasibility of remote implementation, the acceptability of such items, is likely to increase in the future.

Introduction

Although more than 30% of elderly people requiring home care have some form of dysphagia [1], many of them were untreated or provided with support without proper evaluation in Japan. This is probably attributed to their transportation-related difficulties to go to hospitals as well as shortage of medical staff who can visit patients’ home. In addition, providing quick and impartial dysphagia evaluation is challenging, possibly due to lack of skilled professionals, increase of the aging population, and geographical reasons [2]. Therefore, to smoothly allow patients to have access to the necessary treatment and support in the future, tools that can efficiently provide deglutition evaluation to a large number of elderly patients are needed, particularly in Japan.

In response to these challenges, remote services have been improving access to dysphagia assessment. While only preliminary studies have been conducted in Japan [3, 4], several studies overseas have investigated the efficacy of remote dysphagia assessment that are consistent with face-to-face assessment [2, 5–12] with high patient satisfaction [13, 14].

Recently, in response to the spread of COVID-19, the Centers for Medicare & Medicaid Services allowed audiologists and speech–language pathologists (SLPs) to provide select telehealth services to Medicare Part B (outpatient) beneficiaries for the duration of the public health emergency in the USA. This includes swallowing assessment and training [15]. This is due to the fact that when SLP tasks are classified in terms of aerosol infection risk, many tasks, including clinical swallow examinations, fall into the high-risk category [16].

There are a variety of assessments used to evaluate dysphagia remotely. For example, Ward et al. [6, 7] evaluated oral function, swallowing function, and communication by a structured clinician script. Some other studies have used clinical swallow examinations (CSE) remotely [2, 5, 8, 14]. In addition, Morrell et al. [10] evaluated the level of dietary recommendations at a distance.

It should be noted that some of the articles recommended the use of specifically validated clinical evaluation tools, screenings, or patient-reported outcome measures via telehealth [17, 18]. Studies using standardized tests remotely include those by Kantarcigil et al. [19] and Borders et al. [20]. Kantarcigil et al. [19] administered the Dysphagia Disorder Survey to children with cerebral palsy in face-to-face and remote conditions. The results showed that there was no substantive agreement on two items: "oral transport" and "oral pharyngeal swallow." Borders et al. [20] conducted two tests, the Timed Water Swallow Test (TWST) and the Test of Masticating and Swallowing Solids (TOMASS), and found high intraclass correlation coefficients in TOMASS except for "the number of bites" and "the number of swallows.” In other words, when standardized tests are directly applied remotely, there may be a small number of items that may reveal in consistent results between face-to-face and remote conditions. Furthermore, even if a standardized test is used, as long as there are items with questionable consistency between face-to-face and remote conditions, the total score and severity of dysphagia obtained from the test may lose its meaning.

Although all previous studies validated agreement under face-to-face conditions, there are no reports that examined what items would be suitable remotely to evaluate deglutition disorders. One of the factors which prevents broader use of remote swallowing evaluation is that it has more restrictions than face-to-face condition. As a result, clinicians who have not had experience with remote evaluation tend to be skeptical about it. It would be important to develop assessment items that would be easily accepted by clinicians who do not have enough previous experiences with remote assessment, in order to enhance the use of the test.

In the studies by Ward et al. [2, 8, 14], a white tape was placed to the larynx to visualize laryngeal movements during swallowing. Nevertheless, it is doubtful that they were able to detect all laryngeal elevations. Furthermore, certain risks such as aspiration of the test material and choking may be involved in remote dysphagia assessment.

Malandraki et al. [21] stated that in remote training, the viscosity and amount of food to be tried needs to be carefully selected. Since assessments are performed on patients of various severities and diseases, it is necessary to obtain consensus among experts who are candidates for telemedicine on the amount and viscosity of food and water when conducting assessments remotely.

Also, in studies using standardized tests remotely, there is no mention of specific methods of instruction. We would like to create a clinical swallow examination that includes items that are considered feasible to perform remotely. Then, we would like to verify its reliability and validity under face-to-face conditions and check whether the results (scores) obtained from face-to-face and remote conditions are consistent. If the agreement between the face-to-face condition and the remote condition is high for all items, the obtained score can be compared directly to each other. In this study, as a first step, we conducted questionnaire surveys based on the Delphi method to obtain a consensus among experts on the items that can be administered remotely, as well as on the instructional methods and the remote environment.

Methods

Remote Conditions

In this study, remote conditions were assumed as follows: the ST and the examinee were in different remote locations, and the assistant was present next to the examinee operating the equipment and preparing the food and water for oral intake. Since assistants have been present in many of the previous studies, we assumed that they would assist in this study as well. The candidates for the assistants were health-care managers, nurses, and caregivers who have many opportunities to interact with the elderly at home. Since health-care managers and caregivers have different skills in dealing with choking and other problems, we chose nurses as assistants. When nurses are used as assistants, one might think that nurses should perform the test. However, we positioned this examination not only to detect deglutition but also to obtain information that can be used for deglutition training and support. For this reason, we envisioned a situation in which the ST would perform this examination remotely. The assistant was given about an hour of training beforehand to understand his/her roles in the remote environment and the script of this examination. Zoom®, a type of Web conference system, was used for remote implementation. A laptop computer was set up at both examinee and ST sides. On the examinee’s side, a device in which a speakerphone and a camera are combined (Group: Logitech, Tokyo, Japan) was connected to the laptop. This device allowed the assistant to magnify the image and adjust the directions (up, down, left, and right) of the camera. On the ST’s side, we used a laptop computer with a built-in camera and a headset. A wired network was used for communication with a bandwidth of 1100–2676 kbit/s and a resolution of 640 × 360 pixels. An intraoral camera (BONIDA DUAL Alpha: Max Dental, Gyeonggi) was brought up to a distance of about 15 cm from the lips when observing tongue movement. When checking the movement of the soft palate and the elevation of the back tongue, the camera was moved closer to a distance of about 5 cm from the lips. In addition, when confirming oral residuals, the camera was moved closer to about 5 cm from the lips and the angle of the camera was adjusted upper, lower, left, and right. A pharyngeal microphone (SH-12jK: Nanzu, Simoda) was used to detect the sound of swallowing auditorily. The aforementioned camera (Group) was used to visually observe the laryngeal movements by shooting from slightly below using the zoom-in function (Fig. 1).

Fig. 1

Set-up of a remote examination used in this study. A Equipment arrangement on the examinee’s side. B Position of a pharyngeal microphone (the video image monitored on ST’s laptop). C An assistant projecting the examinee’s oral cavity with an intraoral camera. D Intraoral view of the examinee (the video image monitored on ST’s laptop)

Study Design

A three-step Delphi method, one of the consensus methods, was adopted as a qualitative research study. It is a research method that consolidates and converges the opinions, findings, and experiences of experts from many fields, using the “survey–analysis–feedback–survey” technique in order to establish a consensus by repeatedly informing participants about the results of previous stages and encouraging them to deliberate [22]. Consolidating opinions is expected to result in highly accurate evaluations because it can minimize the personal interests among participants [23].

Participants

A total of 122 STs with an experience in dysphagia rehabilitation were included in the study. We used snowball sampling to recruit university faculty, graduate students, and their acquaintances. They belonged to different organizations, including hospitals, geriatric health-care facilities, home nursing stations, and educational institutes.

Survey Method

An overview from the first to third surveys is shown in Fig. 2.

Fig. 2

Overview from the first to third surveys

First Survey

Five specialists selected assessments for oral function and deglutition disorders that are currently available and frequently used in Japan. Specifically, the following items were included: 1 item from the Repetitive Saliva Swallowing Test [24, 25], 1 from the Modified Water Swallowing Test, 1 from the Food Test, 5 from the Toronto Bedside Swallowing Screening Test [26], 21 from the Assessment of Motor Function for Dysphagia [27], 24 from the Mann Assessment of Swallowing Ability [28], 10 from the assessment velopharyngeal function (Epipharynx Closure Functional Inspection) [29], and 2 from the measures obtained with the Tongue Pressure Measurement Device (TPM-02, JMS, Tokyo). A total number of 65 items were included. We were aware that, theoretically, we should have made videos of all 65 items and asked the participants to judge them. However, in order to reduce the time burden on the participants, we excluded items that five experienced specialists (three otorhinolaryngologists and two STs) judged to be obviously difficult to perform remotely or items that were duplicates. Specifically, the five specialists were asked to answer on a Likert scale (1: strongly disagree to 5: strongly agree) whether the items could be performed remotely or not, and 22 items for which three or more of them answered "1: strongly disagree" or "2: disagree" were excluded. In addition, similar items were discussed by five specialists, and 13 items were further excluded. Of the five specialists, one otolaryngologist was a certified swallowing physician of the Society of Swallowing and Dysphagia of Japan. Both otorhinolaryngologists had clinical experience in dysphagia; one ST was certified speech pathologist (in the field of dysphagia) with 12 years of clinical experience and belonged to the Japanese Telemedicine and Telecare Association.

For the remaining 30 items, a video recording of the remote examination scene was made, including an ST, an assistant, and an examinee. The examinee/actor was the same person in all studies, a healthy adult male in his 30 s, and was asked to demonstrate normal competence. This was because we believed that if any item could not be remotely assessed in a healthy adult, it would not be applicable to patients with dysphagia. Therefore, no training was given to the examinee to play the role of the patient. The role of the assistant, the position of the camera, and the equipment to be used were decided in consultation with one of the co-authors from the beginning of the research. This co-author was a member of the psychiatry subcommittee of the Japanese Telemedicine and Telecare Association and had conducted a verification of the reliability and validity of cognitive function tests remotely. In regard to 30 items, it was explained to the participants which test/assessment each item was originally included in.

For all items, we provided our own criteria: “abnormal: 0 point” and “normal: 1 point.” This criterion was presented to the participants so that they know how to look at the elements. However, the participants were never asked to judge whether the target was normal or abnormal.

The 30 items were classified into six categories: overall impression (4 items); lip and jaw (4 items), tongue (5 items); soft plate (4 items); breathing, larynx, and voice (5 items); and swallowing ability (8 items).

Participants were asked to view the video and fill out a questionnaire. At the beginning of the video, they were briefed on the equipment, conditions used in the remote scene. The assessment scene was recorded on a DVD, and it was sent to each participant with a questionnaire enclosed. After completing the questionnaire, the participants were asked to return the entire shipped materials to the investigator with their answers within 1 month.

The questionnaire asked participants to respond to each item on a Likert scale (1: strongly disagree to 5: strongly agree) with respect to (1) the appropriateness in deglutition disorders detection and (2) the remote feasibility. The participants responded whether each evaluation items is particularly useful and effective in identifying key elements of deglutition as well as detecting disorders, regardless of whether the evaluation is performed under face-to-face or remote condition. They were also asked to write what need to be modified in free-text descriptions, as well as their years of clinical experience as STs.

Second Survey

After tabulating the results of the first survey, five specialists (three otolaryngologists and two STs) discussed the results. They combined/deleted test items and modified the methods. The response trends of the first survey were provided to the participants shown in graphs as a feedback. Based on the results of the first survey, examination items and methods were modified, and specific items were selected for the second survey. The remote examination scene was newly recorded, and participants were asked to view videos and answer a questionnaire, as was in the first survey. For each item, the questionnaire asked the participants to answer about the remote feasibility on a 5-point Likert scale. Additionally, they were asked to indicate what should be modified in free-text descriptions.

Third Survey

The response trends of the second survey were provided to the participants shown in graphs as the feedback. Based on the results of the second survey, 2 items were changed. The remote examination scene for these 2 items was recorded, and participants were instructed to view the video and answer the questionnaire. The questionnaire was the same as in the second survey, but respondents were newly asked to indicate whether they had ever made a video call in their daily life.

Analysis Method

The appropriateness in deglutition disorders detection was judged to be effective if the percentage of respondents who answered “agree” (Likert scale 4) and “strongly agree” (Likert scale 5) reached 80% and higher. Items with less than 80% of agreement were excluded from further analysis.

With respect to remote feasibility, the percentage of the respondents who answered “agree” (Likert scale 4) and “strongly agree” (Likert scale 5) was calculated as the consensus rate. If the final rate was 80% or higher, the consensus criteria were considered to be met. If the agreement rate was low in the first and second surveys, it was changed to reflect the responses obtained in the free-text descriptions, and participants were asked to respond again in the next survey. In order to confirm which items had high rate for remote feasibility, factor analysis was conducted on the final items agreed upon through three surveys.

The Mann–Whitney U test was used to check the statistical difference between the respondents who had experiences with video calling and those who do not, regarding the responses on the remote feasibility for the final items.

Results

The effective recovery rate of the first survey was 95.1% (n = 116), and the mean years of experience of STs was 10.1 ± 6.5 years.

The results for the appropriateness in deglutition disorders detection are shown in Table 1. The percentage of the respondents who answered “4” and “5” on the Likert scale exceeded 80% for 19 items (“alertness,” “speech intelligibility,” “lip closure,” “oral diadochokinesis/ka/,” “swallowing tongue pressure,” “maximum tongue pressure,” “tongue movement,” “soft palate movement,” “tracheotomy,” “maximum phonation time,” “pre-swallow voice,” “voluntary cough,” “saliva,” “repetitive saliva swallowing,” “water intake,” “post-swallow voice quality,” “oral residue,” “pharyngeal response,” and “pudding intake”). The percentage of the remaining 11 items was less than 80%. These 11 items were checked whether any of them had a low "appropriateness in detecting dysphagia" due to remote conditions that needed to be corrected. After confirmation by five specialists, 11 items were excluded from further surveys and analysis.

Table 1

The appropriateness for deglutition disorders detection according to items

Items	Rating (%)
Alertness	83.0
Speech intelligibility	93.2
Auditory comprehension	65.0
Cooperation	68.3
Lip closure	91.5
Jaw opening	71.2
Jaw closing	76.1
Oral diadochokinesis/pa/	79.7
Oral diadochokinesis/ta/	78.0
Oral diadochokinesis/ka/	80.5
Swallowing tongue pressure	91.5
Maximum tongue pressure	86.4
Tongue movement	93.1
Soft palate movement	87.9
Hypernasality	79.2
Consonant distortion due to nasal emission	62.7
Nasal noise	51.3
Tracheotomy	85.5
Maximum exhalation time	66.7
Maximum phonation time	81.4
Pre-swallow voice quality	88.9
Voluntary cough	86.4
Saliva	83.1
Repetitive saliva swallowing	88.1
Water intake	91.4
Post-swallow voice quality	89.0
Oral preparation	76.9
Oral residue	94.9
Pharyngeal response	88.9
Pudding intake	91.2

The numbers in the table indicate the percentage of respondents who answered “4” and “5” on the Likert scale, with items below 80% shown in bold

The results for the remote feasibility are shown in Table 2. Of the 19 items that were found to be effective in detecting deglutition disorders, five items showed the consensus rate of less than 80% for remote feasibility: “swallowing tongue pressure,” “maximum tongue pressure,” “saliva,” “repetitive saliva swallowing,” and “pharyngeal response.” The representative comments stated in the free-text descriptions of these five items and the authors’ correspondences to them are indicated below. In the case of “swallowing tongue pressure” and “maximum tongue pressure,” some participants described, “It is often difficult for a patient to understand the examiner’s instructions even in face-to-face situation and can be easily influenced by technical familiarity and/or problems encountered by the assistants.” In response to this opinion, the written explanations were prepared, so that the assistant and the examinee/patient could read the printed instructions visually. In the “saliva” section, participants requested to modify or add a camera angles, stating as “I want the camera to show the corner as well as the buccal side of the mouth.” Regarding “repetitive saliva swallowing,” some described that it is difficult for the clinician (participant) to visually confirm the distance of the laryngeal elevation without manual palpation, even in person. Since palpation is not possible in remote situations, “repetitive saliva swallowing” was deleted. With respect to “pharyngeal response,” some described that “they want to observe the examinee’s/patient’s face, laryngeal elevation, and the oxygen saturation level (SpO2) simultaneously.” However, since it would require expensive equipment to display all of them simultaneously, we have assigned the roles as follows: the examiner monitors the examinee’s/patient's face and larynx, while the assistant monitors the pulse oximeter values.

Table 2

Trends in remote feasibility

No	Items	First	Second	Third
1	Alertness	93.2%	95.5%
2	Speech intelligibility	89.8%	98.2%
3	Lip closure	94.0%	97.3%
4	Oral diadochokinesis/ka/	86.4%	100%
5	Swallowing tongue pressure	75.4%	85.6%a
6	Maximum tongue pressure	69.5%	79.3%a
6’	Strength of the tongue			91.7%
7	Tongue movement	88.0%	97.3%
8	Soft palate movement	80.1%	95.5%
9	Tracheotomy	92.3%	97.3%
10	Maximum phonation time	93.2%b
11	Pre-swallow voice quality	89.7%b
10 + 11	Sustained phonation and voice quality		97.3%
12	Voluntary cough	87.3%	99.1%
13	Saliva	78.0%	86.8%
14	Repetitive saliva swallowing	78.0%c
15	Water intake	80.2%	90.8%d	97.2%
16	Post-swallow voice quality	86.4%e
17	Oral residue	89.0%f
18	Pharyngeal response	77.8%f
17 + 18	Staple food intake		93.7%
19	Pudding intake	85.3%g	98.2%

The numbers in the table indicate the consensus rate

aItems 5 and 6 were deleted and changed to item 6’ because of expensive device, difficulties in instructing to actual patient, and the exclusion of edentulous patients

bItems 10 and 11 were combined

cItem 14 was deleted because it is difficult for the clinician to visually confirm the distance of the laryngeal elevation without manual palpation

d Evaluation of 10 ml of water was added

eItem 16 was included in 15, 17 + 18, and 19

fItems 17 and 18 were combined

g Item 19 was deleted because its priority was stated as low

The authors categorized the 156 comments indicated in the free-text descriptions on the above-mentioned 19 items in Table 3. The most common comments were “concerns about the quality and stability of the voice that may be heard through the device,” followed by “the need to correct the camera angle, magnification, and targets that should be projected,” “concerns about the technical aspects of the assistants and their role in relation with the examiner/ST,” “the need for palpation as well as visual confirmation,” “concerns about the assistant’s and the actual patient’s difficulty in understanding the examiner’s remote instructions,” and “a high risk of remote implementation.” The point about the "voice quality" was not an answer referring to the video used in the survey, but was mere expression of concern and worry about possible trouble for future Internet connection between the hospital and home.

Table 3

Breakdown of free-text descriptions for remote evaluation

Main contents	Number of confirmed
Concerns about the quality and stability of the voice that may be heard through the device	41
The need to correct the camera angle, magnification, and targets that should be projected	35
Concerns about the technical aspects of the assistants and their role in relation with the examiner/ST	32
The need for palpation as well as visual confirmation	17
Concerns about the assistant’s and the actual patient’s difficulty in understanding the examiner’s remote instructions	14
A high risk of remote implementation	5
Other	10

Of the 19 items that were found to be appropriate for deglutition disorders detection in the first survey, a breakdown of the free-text descriptions for remote assessment is shown in the table

The authors also reviewed the free-text descriptions for the items that met the consensus criteria. Subsequently, “maximum phonation time” and “pre-swallow voice quality” were combined and re-named “sustained phonation and voice quality.” In addition, since staple foods such as rice and porridge were used for “oral residue” and “pharyngeal response,” these were merged into one as “staple food intake.” “post-swallow voice quality” was included in the existing “water intake,” “pudding intake,” and the newly added “staple food intake.”

The effective recovery rate of the second survey was 97.4% (n = 113).

The consensus rate of the remote feasibility exceeded 80% for 14 of 15 items in the second survey, whereas the rate was less than 80% for “maximum tongue pressure.” The consensus rate for all 15 items tended to be higher than that of the first survey.

In the case of “maximum tongue pressure” and “swallowing tongue pressure,” even though their instructions were presented visually in a printed form, respondents still felt “The procedure is complicated and difficult to instruct to the actual patient and have him/her understand,” “It would be difficult to perform the evaluation with edentulous patients.” In addition, there was a comment stating that the Tongue Pressure Measurement Device is too expensive to purchase; thus, few facilities can afford it. In response to these comments, we decided not to use this device. Instead, we changed the item to “strength of the tongue,” which examines whether the examinee/patient has the strength to hold a tongue blade between the apex of the tongue and palate for 5 s.

In the “water intake” section, 3 ml of water was given. However, since there were comments indicating the need to include 10 ml, evaluation of 10 ml of water was added when no signs/symptoms of aspiration, such as cough, wet voice, and/or change in SpO2, were observed with 3 ml.

The item “pudding intake” was deleted because its priority was stated as low if “water intake” or “staple food intake” is performed.”

The effective recovery rate of the third survey was 95.6% (n = 108).

In the third survey, two items, “strength of the tongue” and “water intake,” were surveyed, and the consensus rate for both items was high, exceeding 90%. The final 13 items and their instructions are shown in “Appendix.”

Factor analysis using the principal factor method was conducted on 13 items, which were selected through the second and third surveys. Considering the change in eigenvalues and the interpretability of factors, a three-factor structure was considered to be appropriate (Fig. 3). One item, “tracheotomy,” which did not show sufficient factor loadings, was excluded, and factor analysis using the principal factor method and varimax rotation was conducted again for the remaining 12 items. The final factor patterns after varimax rotation are shown in Table 4. The percentage of three factors before the rotation explaining the variance of 12 items was 52.6%.

Fig. 3

Scree plot of factors. The results of the factor analysis of the final 13 items; for deglutition disorders detection, a three-factor structure was considered appropriate. One item (tracheotomy) that did not exhibit significant loading was excluded

Table 4

Final factor pattern after varimax rotation

The first factor consisted of four items (“soft palate movement,” “saliva,” “tongue movement,” and “staple food intake”), all of which were observed using an intraoral camera, and was therefore named as the ‘oral observation’ factor. The second factor consisted of four items (“water intake,” “lip closure,” “speech intelligibility,” and “strength of the tongue”) and was named the ‘overall evaluation’ factor. The third factor consisted of four items (“voluntary cough,” “sustained phonation time and voice quality,” “alertness,” and “oral diadochokinesis /ka/”), and since there were many items that required auditory judgment, it was named the ‘perceptual voice judgment.’ factor. Although “tracheotomy” did not show sufficient factor loadings, the authors discussed and decided to leave it as a single item because it has a significant impact on swallowing ability.

Regarding the question whether they had previous experience with video calls or not, 86 of 102 (84.3%) participants respond positively. In relation to the remote feasibility for the above-mentioned 13 items that finally reached the consensus criteria, the Mann–Whitney U test was used to statistically assess the difference in the responses between those who had video calling experiences and those who did not. As a result, a significant higher number of STs with video calling experience answered positively toward the remote feasibility for the item “speech intelligibility” than those without (Z = − 2.59, U = 445.5, p = 0.01). No significant differences were found between the two groups of STs for other items.

Discussion

With the prevalence of COVID-19, remote evaluation of dysphagia has become more widespread, and efforts to improve the accuracy of remote evaluation are needed. In this study, we conducted surveys to gather opinions of STs who serve as examiners for remote assessment. The novelty of the study lies in the fact that we obtained consensus not only on items that can be implemented remotely but also on instructional methods and environments for assessment.

In the Delphi method, the setting and selection of experts are important to ensure the reliability of results [30]. In this study, we set the selection criteria for STs as clinicians with experience in dysphagia rehabilitation and obtained cooperation from those working in 18 prefectures as well as remote islands. The level of demand for telemedicine greatly varies by region. This survey was considered to have gathered diverse opinions, as it covered 18 of 47 prefectures in Japan. Furthermore, the percentage of STs who have some experience in video calls was also high, reaching at 84.3%. We believe that we could gather the opinions of STs who have certain understanding of online communication. In addition, we conducted the survey more than three times, which is recommended as an effective numbers [22], and obtained a high recovery rate of more than 90%. These may have increased the content validity of each item.

Of the 19 items for which appropriateness for deglutition disorders detection was ensured, 5 items did not meet the consensus criteria for remote feasibility in the first survey. The fact that the “swallowing tongue pressure” and “maximum tongue pressure” did not meet the consensus criteria was due to the difficulty in instructing to the actual patient and have him/her understand. By visually presenting the method, the second survey showed a slight increase in the consensus rate, but was excluded from the items to be enforced due to the edentulous patients and the high cost of the equipment.

In “repetitive saliva swallowing,” many indicated that it is difficult for the clinician (participant) to visually confirm the distance of the laryngeal elevation without manual palpation, even in person. The detection rate of laryngeal elevation is higher when palpation is used than sole visual examination, or when visual examination and palpation are used together [31]. Conversely, the CSE used remotely by Ward et al. [2, 8, 14] included “no. of swallows” and “laryngeal elevation,” using a strip of white surgical tape positioned over the patient’s thyroid notch to enhance visualization of laryngeal elevation during the swallow. Other studies have not specified detailed detection methods for laryngeal elevation. In our study, although swallowing sounds were used to detect laryngeal elevation in addition to visual inspection, many respondents indicated that the number of swallows and the distance of laryngeal elevation should be confirmed by palpation, and composition of examination items should be considered in light of these limitations. It is difficult for assistants to determine the distance of the laryngeal elevation by palpation, and this is a good example of limitations specific to remote examination.

In the “saliva,” many comments were raised with regard to capturing the targets to be observed by the intraoral camera. In response, we had a staff meeting and revised the points to be captured by the intraoral camera. In the “pharyngeal response,” there were many comments about “difficulty in observing multiple points at the same time” such as the examinee’s/patient’s face, laryngeal elevation, and the SpO2. To solve this issue, the assistant was instructed to solely observe the assigned point. As a result, in the second survey, the consensus criteria were met for both "saliva" and "staple food intake," the items that included "pharyngeal response." In majority of the previous studies, the assistants were trained in advance, and it was necessary to have assistants with excellent operating skills. The free-text responses in this study also demonstrated the importance of advance training for the operating skills of the assistants.

The most common comments obtained in this study were the “concerns about the quality and stability of the voice that may be heard through the device.” Ward et al. [2] found a high degree of agreement between remote and face-to-face comparisons in their evaluation of wet voice. However, weak voice intensity impacted on the ability to hear any subtle changes in voice quality after swallowing, forcing the online speech pathologists to rely more heavily on other signs of potential aspiration risk [9], indicating that remote evaluation may not be the most efficient means for some patients. The quality of voice signal can be affected by various factors when the signal is recorded, stored, and transferred [32]. Therefore, it is necessary to clarify whether the same level of agreement can be obtained between remote and face-to-face conditions envisioned. In addition, further study is needed in the future in regard to the exclusion criteria of the patients for remote evaluation. In terms of speech intelligibility, STs with video call experience rated the feasibility of remote implementation higher than those without video call experience. As the chances of video calls and Web conferencing systems are expected to increase in the future, their acceptability to remote evaluation is likely to improve.

For the 19 items for which the appropriateness for deglutition disorders detection was ensured, revisions were repeated, and finally, 13 items exceeded the consensus criteria of remote feasibility. As a result of the factor analysis of these 13 items, three factors (‘oral observation,’ ‘overall evaluation,’ and ‘perceptual voice judgment’) were identified from the 12 items except for one. These three factors are also important in face-to-face evaluations, and it is possible that the proposed examination consists of items that capture the characteristics of deglutition disorders, even though it excluded items that are difficult to implement remotely.

Previous studies have focused on patient satisfaction after the remote evaluation [14] and on the satisfaction of speech–language pathologists who conducted the evaluation remotely [8, 14]. In addition to these studies, it is important to reflect the opinions of STs or SLPs who have not experienced in remote evaluation in order to improve its generalizability. We believe that this study can contribute to the improvement of such generalizability. Future tasks of this research include verifying the reliability and validity of the proposed examination for detecting dysphagia, criterion-related validity, construct validity, inter-rater reliability, and internal consistency under face-to-face conditions. Subsequent studies include the agreement between face-to-face and remote conditions.

Conclusion

In order to develop a remote examination of deglutition, a questionnaire based on the Delphi method was administered to 122 STs. In the end, 13 items met the consensus criteria for remote feasibility. The result of the factor analysis extracted three factors: ‘oral observation,’ ‘overall evaluation,’ and ‘perceptual voice judgment.’ While excluding the items that appeared difficult to implement remotely, it was still possible to compose a test that could capture the characteristics of deglutition disorders. In the future, we plan to compare face-to-face and remote conditions after examining the reliability and validity of this proposed examination under face-to-face conditions. In the free-text descriptions, “quality and stability of the examinee’s/patient’s voice that may be heard through the device” were the most frequently expressed concerns. However, a significantly higher number of STs who had experiences with video calls gave positive judgment toward the remote implementation of “speech intelligibility” when compared with those who did not. As the use of video calls and Web conferencing systems is expected to increase in the future, their acceptability is also likely to improve.

Appendix

Alertness	□ “Assistants, please correct the position of the camera so that it shows from the neck up.” □ “Hello. Nice to meet you.” □ Conduct a brief free talk
Speech Intelligibility	□ “Assistants, please hand in your prints.” □ “Please read it aloud.” (1) I buy a blue house (2) My body is sluggish and sluggish (3) I am lured by the whispering murmur of the shallows (4) This tatami room was built by my brother and his friends (5) The sun seeth all things and discovereth all things (6) When the fog clears, we can descend from the sky (7) Papa and Mama all threw beans together
Tracheotomy	□ “Let me observe your neck. Can you pull the collar down a little?” □ “Assistant, please zoom in on his/her neck.” □ “Assistant, return the camera to its original angle.”
Voluntary cough	□ “Please cough loudly, as I did.” □ “Please clear your throat loudly, as I did.”
Sustained phonation and voice quality	□ “Take a deep breath and then say 'ah' for as long as you can.” Conduct twice
Lip closure	□ “Now I'm going to watch your mouth movement. Open your mouth.” □ “Assistants, please zoom in on his/her mouth.” □ “Close your mouth.” □ “Pull your mouth to the side, as I did.” □ “Stick your mouth, as I did.” □ “Please repeat as I did, pull your mouth and stick your mouth.” □ “Puff out your cheeks as I did. Hold it like that for five seconds.”
Oral diadochokinesis /ka/	□ “Repeat ‘ka ka ka···’ as quickly as you can until I give you the signal.” Perform twice for 3 s
Tongue movement	□ “Assistants, please turn on and switch on the intraoral camera. We use a camera that gives us a clear view of the inside of the mouth.” □”Open your mouth.” □ “Keep your mouth open and stick your tongue out as far forward as possible.” □ “Keep your mouth open and pull your tongue back.” □ “Touch the edge of your mouth with your tongue in this way.” □ “Next, the other side.” □ “Keep your mouth open and place the tip of your tongue on the back of your front teeth, as I did.” □ “Say ‘ka’ with your mouth open.”
Strength of the tongue	□ “Assistant, please prepare the tongue blade.” □ “We will check the strength of your tongue. With your mouth wide open, support the stick with your tongue, as I did. Please keep it for 5 s.” □ “Assistant, please use the intraoral camera to view the image from the side and downward.” □ “Now put the tongue blade in your mouth.”
Soft palate movement	□ “Please check your soft palate movement. Open your mouth.” □ “Breathe in through your nose and say ‘uh’.”
Saliva	□ “Let me check your mouth is moist. Let me see the right side. Now the left side. Now the upper side, now the lower side.” □ “Assistant, switch to Group's camera and show us his/her neck to his/her mouth.” □ “We will now attach a microphone to confirm the sound of swallowing. Please turn it on.” □ “Please swallow your saliva.” □ “Assistant, please turn off the pharyngeal microphone.”
Water intake	□ “You will be asked to drink 3 ml of water. First, say ‘ah’.” □ “"Put the pulse oximeter on your finger. Assistant, please monitor it and tell me if there are any abnormalities.” □ “Assistants, please fill syringes with water. Please turn on the pharyngeal microphone.” □ “Please open your mouth slightly as we put water in your mouth. Please hold it until I give you the signal.” □ “Then swallow it.” □ * “Assistant, please turn off the pharyngeal microphone.” □ * “Now, please say ‘ah’. Again, please.” □ “Next is 10 ml of water. Please take a glass.” □ “Assistant, please turn on the pharyngeal microphone.” □ “Now, try to drink it as usual.” Repeat *
Staple food intake	□ “Assistant, prepare his/her staple food (in the form he/she currently intake).” □ “Assistant, please turn on the pharyngeal microphone.” □ “Please chew and swallow on your own time.” Repeat * □ “Assistants, please turn on and switch on the intraoral camera.” □ "May I see the inside of your mouth, please? Open your mouth wide, please. Let me see the right side. Now the left side. Now the upper side, now the lower side.”