Development of a Remote Examination of Deglutition Based on Consensus Surveys of Clinicians (Part II): Reliability and Validity in Healthy Elderly Individuals and Oral Cancer Patients.

In our prior published study, we extracted evaluation items suitable for remote administration, and made a relatively simple Remote Examination of Deglutition (RED). This study aimed at verifying the reliability and validity of RED. The participants were 21 healthy elderly individuals and 72 postoperative oral cancer (OC) patients. OC patients underwent videofluoroscopic dysphagia examination, and severity was judged on the dysphagia severity scale (DSS). Reliability and validity of RED were examined in all participants under face-to-face conditions, in comparison with the Mann Assessment of Swallowing Ability (MASA). Reliability and validity of remote administration of RED were examined in 40 participants. ROC curves were used to find cut-off RED scores to predict aspiration and deglutition disorders. The Cronbach's alpha coefficient for the items was 0.882. There was a high correlation between the total score of RED and MASA in the face-to-face condition. When RED score was compared among different severity groups (DSS1-4, DSS5-6, and DSS7), the total and oral preparatory stage scores revealed significant group differences. The area under the curve (AUC) for aspiration based on the ROC curve was 0.913, with a sensitivity/specificity of 0.80/0.98. The AUC for deglutition disorders was 0.819, with a sensitivity/specificity of 0.74/0.67. In both face-to-face and remote conditions, the reliability of RED was good.The reliability and validity of RED were confirmed. RED has shown the potential to assess the likelihood of aspiration and deglutition disorders in OC patients remotely as an initial assessment tool.

Introduction

Since the 1990s, there has been a gradual increase in investigations into the feasibility, validity, and reliability of telemedicine approaches to dysphagia [1]. The majority of these studies have examined dysphagia tele-assessments and have repeatedly demonstrated their safety, validity, and reliability when compared to traditional in-person dysphagia evaluations [1-12]. With the spread of COVID-19, telemedicine progress has accelerated, necessitating improvements in the accuracy of remote dysphagia evaluations.

In 2021, the Centers for Medicare & Medicaid Services allowed audiologists and speech-language pathologists to provide select telehealth services to Medicare Part B (outpatient) beneficiaries for the duration of the public health emergency in the United States. These services include swallowing assessment and training [13], because many tasks, including clinical swallowing examinations (CSE), fall into the high-risk category for aerosol infection [14]. Fritz et al. [15] stated that the risk of exposure can be minimized by incorporating remote approaches into swallowing assessments during the pandemic. Also, according to the review article by Maladraki et al. [1], the most common dysphagia procedure for which telemedicine was recommended/used was CSE.

The leading studies of CSE at a distance are those of Ward et al. [10, 12, 16]. Their studies found good agreement between face-to-face and remote conditions on a number of items but did not describe whether the assessments used were reliable or validated. It should be noted that some of the articles recommended the use of specifically validated clinical evaluation tools, screenings, and/or patient-reported outcome measures via telehealth [15, 17]. Studies using standardized tests remotely include those by Kantarcigil et al. [5] and Borders et al. [18]. Kantarcigil et al. [5] administered the Dysphagia Disorder Survey (DDS) to children with cerebral palsy in face-to-face and remote conditions. The study showed that there was no substantive agreement on two items: "oral transport" and "oral pharyngeal swallow." Borders et al. [18] conducted two tests, the Timed Water Swallow Test (TWST) and the Test of Masticating and Swallowing Solids (TOMASS), and found high intraclass correlation coefficients in TOMASS, with the exception of "number of bites" and "number of swallows." Based on these findings, it is likely that direct remote application of a standardized test developed for face-to-face evaluation can lead to inconsistent results when compared to face-to-face conditions. If the agreement between face-to-face and remote evaluation is questionable for a particular item of the test, the total as well as categorical scores of the tests can become invalid. Thus, remote examination of deglutition must include evaluation items that have been verified for remote administration.

Telemedicine has been deemed feasible in head and neck cancer (HNC) patients and other adult patient populations who need continued support from speech-language pathologists [19-21]. We believe that priority should be given to verifying the reliability and validity of the tests in patient populations, possibly with those who have high affinity in telemedicine. In this sense, HNC patients are good candidates for such a research study.

In Part I of our study [22], we extracted items that were appropriate for detecting deglutition disorders and for remote administration, using questionnaire surveys of Japan’s nationally licensed speech-language-hearing therapists (SLHTs). In addition, we sought to obtain professional consensus on the type and amount of test materials (food), role of an assistant, and components of visualization, etc.

In this study, we constructed a remote examination of deglutition (RED) with the items extracted in Part I. The purpose of the present study is two-hold: (1) to test reliability and validity of RED under face-to-face conditions, in both healthy elderly individuals and postoperative patients with oral cancer (OC), and (2) to test reliability of the RED under remote conditions, along with a questionnaire survey of the patients after remote implementation of RED.

Methods

RED

RED is a test that has been developed for remote evaluation of deglutition. The test is intended to be used in situations where a nurse or other medical personnel visits the patient's home with the necessary equipment, and remotely helps the SLHT in the hospital to evaluate the patient. This system also enables an SLHT in a hospital to be connected to an elderly care facility where there is no SLHT.

In Part I of our study, we conducted questionnaire surveys among 122 SLHTs three times using the Delphi consensus method. A remote evaluation situation was video-recorded for 30 items that are frequently used as deglutition examination in Japan. The participants were instructed to view the video and answer the questionnaires. The questionnaire asked participants to respond to each item on a Likert scale (1: strongly disagree to 5: strongly agree) with respect to (1) the appropriateness in deglutition disorders detection and (2) the remote feasibility. They were also asked to write what need to be modified in free-text descriptions. As a result, the authors repeatedly made modifications to the visualization elements, role sharing with assistants, types and amount of test materials (food), criteria for judgment, etc. Finally, by the third survey, 13 items were extracted that met the criteria of remote feasibility and appropriateness for detecting deglutition disorders (Table 1).

Table 1

15 items to be discussed in Part II and corresponding physiological stages of deglutition

13 items extracted in Part I	15 items verified in Part II	Corresponding stage
Alertness	Alertness※	–
Speech intelligibility	Speech intelligibility	–
Tracheotomy	Tracheotomy	Pharyngeal
Voluntary cough	Voluntary cough	Pharyngeal
Sustained phonation and voice quality	Sustained phonation and voice quality	Pharyngeal
Lip closure	Lip closure	Oral preparatory
Oral diadochokinesis /ka/	Oral diadochokinesis /ka/	Oral preparatory
Tongue movement	Tongue movement	Oral preparatory
Strength of the tongue	Strength of the tongue	Oral preparatory
Soft palate movement	Soft palate movement※	Oral
Saliva	Saliva	Oral
Water intake	aWater intake (3 ml)	Pharyngeal
	aWater intake (10 ml)	Pharyngeal
Staple food intake	bStaple food intake (oral residue)	Oral
	bStaple food intake (pharyngeal response)	Pharyngeal

aThis item was divided into two items because of the different difficulty levels

bThis item was divided into two items because the oral residue observed in this item corresponds to the oral stage and the pharyngeal response corresponds to the pharyngeal stage

※Based on the internal consistency analysis presented later, this item was excluded from the RED

Of these 13 items, "water intake" was divided into "water intake (3 ml)" and "water intake (10 ml)" because of the different difficulty levels. In addition, "staple food intake" was divided into "staple food intake (oral residue)" and "staple food intake (pharyngeal response)" because the oral residue and pharyngeal response correspond to different physiological stages of deglutition. Subsequently, RED consisted of 15 items. The RED items and corresponding physiological stages (oral preparatory stage, oral stage, and pharyngeal stage) are shown in Table 1. Each item was scored as normal (1 point) or abnormal (0 points) during evaluation. The judgment criteria for each item are shown in Appendix 1.

There are two major differences between RED and other existing tests. First, RED consists only of items that were judged to be remotely feasible in our Part I study [22]. In contrast, other standardized tests developed for face-to-face conditions have been directly adapted for remote evaluation. In such cases, some of the evaluation items may not be suitable for remote evaluation, and the test score may lose its meaning. Second, RED allows the calculation of the oral preparatory stage score, oral stage score, and pharyngeal stage score, with a possibility of estimating the elements of disorders in each stage. Although RED does not fall into the category of a CSE that evaluates function and ability in detail, we aimed at making RED as a useful screening tool for deglutition disorders and aspiration, as well as a rough initial assessment to identify major components of deglutition disorders and to determine the need for instrumental evaluation.

STEP 1 (Reliability and Validity Under Face-to-Face Conditions)

Participants

The participants in our study were healthy elderly individuals and OC patients. The healthy volunteers were recruited from the Silver Human Resources Center, OC patients were recruited from those who had undergone surgery at the Oral Surgery Department of our hospital and who had been referred to the Department of Otolaryngology for swallowing function evaluation within 4 weeks of surgery. The participants were given written information on the contents and methods of the study. Exclusion criteria were patients with: (1) contraindications to oral intake, (2) severe cognitive impairment, (3) severe visual/hearing impairment, (4) severe aphasia/language impairment. OC patients were excluded if they had a history of potential causes of dysphagia other than oral cancer. In addition, the healthy elderly individuals were excluded if they had a history of potential causes of dysphagia, or if they scored 3 or higher on the Eating Assessment Tool (EAT-10; [23]). A total of 21 consenting elderly individuals and 78 OC patients were included in the study.

Procedure

Administration of RED and MASA

All participants were administered RED, and the Mann Assessment of Swallowing Ability (MASA; [24]) under face-to-face conditions. RED and MASA were administered by an SLHT who has more than 10 years of experience in swallowing rehabilitation (Examiner A) and by another SLHT with 2 years of experience (Examiner B). MASA was used for comparison with RED because MASA is representative of standardized tests and also it is a test that can estimate disorders in different physiological stages of deglutition.

(2) Videofluoroscopic examination and severity classification.

In the OC patients, videofluoroscopic examination of swallowing (VF) was performed by an otolaryngologist to identify the presence/absence of aspiration and disorders in the oral preparatory, oral, and pharyngeal stages as well as to determine the severity on the dysphagia severity scale (DSS; [25, 26]). In Japan, SLHTs are not qualified to perform VF and it is common for a physician to perform VF. The otolaryngologist, who performed the VF in this study, was certified as a swallowing consultant physician by the Society of Swallowing and Dysphagia of Japan, and had more than 20 years of experience in interpreting VF. DSS is a clinical severity classification of dysphagia with seven categories: 1: saliva aspiration (unstable medical condition due to severe saliva aspiration); 2: food aspiration (food aspiration with no effect from compensatory techniques or food consistency changes); 3: water aspiration (aspiration of thin liquids; change in food consistency is effective); 4: occasional aspiration (possible aspiration or aspiration is suspected due to pharyngeal residue); 5: oral problems (significant symptoms in the pre-oral anticipatory stage or oral stage without aspiration); 6: minimum problems (some symptoms of dysphagia but no need for rehabilitation or exercise); 7: within normal limits (no symptoms of dysphagia). Of the OC patients who were diagnosed not to have deglutition disorders by VF, those who complained of difficulty in swallowing were classified under DSS6. VF was performed within 5 days after RED administration, while the otorhinolaryngologist was blinded to the RED results.

MASA was performed on healthy elderly individuals who had an EAT-10 score of less than 3. All healthy elderly individuals were found to be within the normal range of 179 points or higher, so all healthy elderly individuals were classified into the DSS7 group.

Subjects in both the normal and OC groups were then divided into 3 groups based on their DSS results: DSS1-4, DSS5-6, and DSS7.

Inter-Rater Reliability

In order to examine inter-rater reliability under face-to-face conditions, Examiner B was present during the evaluations conducted by Examiner A for 20 healthy elderly individuals and 18 OC patients. The Examiner B observed and evaluated the participants simultaneously with Examiner A.

The patient's age, gender, diagnoses, and surgical procedure were disclosed to the examiners in advance, as they are often revealed in routine clinical situations.

Setting for Evaluation

Since RED was developed based on commonly used assessment tools for oral and swallowing functions, it does not require special training for the examiner when performing RED in-person. The examiners were given a script that specifically described the instructions (Appendix 2). RED took place in a therapy room in the hospital.

Preparations included a 10 ml syringe, stopwatch, tongue depressor, pulse oximeter, water, staple foods (pureed porridge (Yasashiikondate Namerakagohan, Kewpie, Tokyo), porridge (Yasashiikondate Yawarakagohan, Kewpie, Tokyo), softly cooked rice (Papattoraisu Yanwakagohan Koshihikari, Hagoromo Foods, Shizuoka), regularly cooked rice (Papattoraisu Koshihikari, Hagoromo Foods, Shizuoka)) (Fig. 1).

Fig. 1

Materials used in “staple food intake”. From left to right: pureed porridge, porridge, softly cooked rice, and regularly cooked rice

When each type of food was compared to the International Dysphagia Diet Standardisation Initiative, pureed porridge corresponded to level 4 (pureed), porridge to level 5 (minced & moist), softly cooked rice to level 6 (soft & bite-sized), and regularly cooked rice was equivalent to level 7 (regular). For those who were already eating food in their daily meals, the type of food they were consuming at the time was selected for RED. For patients on tube feedings, we chose a pureed porridge.

Laryngeal elevation was confirmed visually as well as by palpation. Pulse oximeters were worn during "water intake" and "staple food intake", in order to monitor SpO2 change following possible aspiration.

Statistical Analysis

Cronbach's alpha coefficient and Kappa coefficients were calculated, respectively, to examine the internal consistency and inter-rater reliability of RED under face-to-face conditions. To examine the criterion-related validity, Spearman's correlation coefficient between total RED and MASA scores was calculated. In addition, the OC patients were divided into two groups according to the presence or absence of disorders in respective stage of deglutition based on VF. Then, the RED stage scores were compared between the two groups (one with disorders and the other without disorders) using Mann–Whitney's U test. To examine the construct validity of RED under the face-to-face condition, the total and stage RED scores were compared among the three groups (DSS1–4, DSS5–6, and DSS7) using the Kruskal–Wallis test with Dunn-Bonferroni correction for multiple comparisons.

OC patients were defined as having "no deglutition disorders" if their swallowing were not impaired in any of the stages, while as "with deglutition disorders" if an impairment was identified in any of the stages.　The receiver operating characteristics (ROC) curves were plotted to test the sensitivity and specificity of the total RED score in predicting deglutition disorders and aspiration. ROC analysis compared: (1) between OC patients with no deglutition disorders and those with deglutition disorders on VF, and (2) between OC patients with no aspiration and those with aspiration on VF. The sensitivity and specificity with the optimal cut-off value were derived from the analysis as a measure of validity. ROC curves and the corresponding areas under the curve (AUC) were calculated for the total RED score.

Statistical analyses were performed through SPSS Statistics 25 (IBM, New York).

STEP 2 (Reliability of RED Under Remote Conditions)

Among the subjects who participated in STEP 1, 20 healthy elderly people and 20 OC patients were included in STEP 2.

Intra-Rater and Inter-Rater Reliability

First, Examiner A administered the RED to 40 participants remotely, in order to verify intra-rater reliability between face-to-face and remote conditions. The face-to-face data were taken from STEP 1. The maximum evaluation interval was 7 days. Next, in order to examine inter-rater reliability in the remote condition, the recorded remote examination scene was later observed by Examiner B. Examiner A was a member of the Japanese Telemedicine and Telecare Association. If any problems were encountered during the remote implementation, they were noted by the examiners and assistants.

Questionnaire to the Participants

After RED was conducted remotely, a questionnaire was administered to the participants of the evaluation on the spot. The questionnaire was partially modified in reference to Iiboshi et al. [27]. The reason for the modifications was that the different emotions of fear and nervousness were asked as one question in our study. In the questionnaire, the participants were asked to respond to fear, nervousness, and ease of answering the questions on a 5-point Likert scale (range 1: strongly agree to 5: strongly disagree). In addition, a free-text response field was provided for comments from the participants. The participant responded to the questionnaire anonymously and responses were placed in an envelope for each participant. When the entire survey was completed, the authors tabulated the results of the questionnaire responses.

Setting for Remote Conditions

In this study, we created a simulated remote scene by connecting a training room and another room in the hospital via a web conference system [22]. Specifically, the SLHT and the participant were in separate rooms in the facility, with an assistant present on the participant's side. The assistant was there to operate equipment and prepare the food and water for oral intake. The assistant was given about an hour of training beforehand to understand the RED script and his/her role in the remote environment. The Examiners A and B were given a script that specifically described the instructions (Appendix 2). A total of three medical students served as assistants for STEP 2. They were assigned to different participant based on their class schedule.

Besides the items prepared for the face-to-face condition, the following systems and equipment were used. Zoom®, an internet-based video conference system, was used for remote implementation. A laptop computer was set up on both participant and SLHT sides. On the participant’s side, a device in which a speakerphone and a camera are combined (Group: Logitech, Tokyo, Japan), was connected to the laptop. This device allowed the assistant to magnify the image and adjust the directions (up, down, left, and right) of the camera. On the SLHT’s side, we used a laptop computer with a built-in camera and a headset. A wired network was used for communication with a bandwidth of 1,100–2,676 kbit/s and a resolution of 640 × 360 pixels.

An intraoral camera (BONIDA DUAL Alpha: Max Dental, Gyeonggi) was moved by the assistant toward approximately 15 cm from the lips when observing tongue movement. When viewing the movement of the soft palate and the elevation of the posterior tongue, the camera was moved toward about 5 cm from the lips. In addition, when confirming oral residuals, the angle of the camera was adjusted upper, lower, left, and right.

A pharyngeal microphone (SH-12jK: Nanzu, Simoda) was used to detect the sound of deglutition. The sound picked up by the pharyngeal microphone was enhanced by a loudspeaker (NZ-680-A: Nanzu, Simoda).

At the start of the remote examination, the examiner had an assistant operate the Group's camera so that the image was taken from the participant's neck up. This camera was also used to visually observe laryngeal movements by using the zoom in function (Fig. 2). The intraoral camera, which is different from Group’s camera, was used to observe the oral cavity. The details of the settings are described in Appendix 2.

Fig. 2

Set-up of a remote examination used in this study. A Equipment arrangement on the participant’s side. B Position of a pharyngeal microphone (the video image monitored on SLHT’s laptop). C An assistant projecting the participant’s oral cavity with an intraoral camera. D Intraoral view of the participant (the video image monitored on SLHT’s laptop). SLHT: speech-language-hearing therapist

Kappa coefficients were calculated for intra-rater reliability between face-to-face and remote conditions of RED, as well as inter-rater reliability for remote administration of RED.

For participants who performed RED remotely, the time required　to complete assessment was compared between the two conditions (i.e., in-person (STEP 1) and remote (STEP 2)), using a paired t-test with an alpha level of 0.05.

Results

STEP 1 (Reliability and Validity Under Face-to-Face Conditions)

Six OC patients who did not meet the criteria for the postoperative period and time to VF were excluded, and the final analysis included 21 healthy elderly individuals and 72 OC patients. The mean age of the 21 healthy elderly individuals was 69.9 ± 4.5 years (7 males, 14 females); the mean age of the 72 OC patients was 69.4 ± 11.1 years (41 males, 31 females). The diagnoses of OC patients were as follows: tongue cancer (43 patients), lower gingival cancer (10), floor of mouth cancer (7), upper gingival cancer (4), buccal mucosa cancer (3), and other diseases (5). The DSS classification for the healthy elderly individuals revealed DSS7 (within normal limits) for all 21 individuals. In OC patients, 35 patients were judged as DSS1–4 while 37 as DSS5–6.

Internal consistency was verified by excluding the item with zero variance ("alertness") and "soft palate movement" with a corrected item total correlation of 0.236. The Cronbach's alpha coefficient for the remaining 13 items was 0.882, indicating high internal consistency (Table 2). Two items, "alertness" and "soft palate movement," were excluded, and the remaining 13 items were henceforth referred to as RED.

Table 2

Internal consistency

	Corrected item total correlation	α
Speech intelligibility	0.623	0.864
Tracheotomy	0.497	0.871
Voluntary cough	0.510	0.870
Sustained phonation and voice quality	0.498	0.871
Lip closure	0.554	0.868
Oral diadochokinesis /ka/	0.648	0.863
Tongue movement	0.576	0.867
Strength of the tongue	0.495	0.871
Soft palate movement	0.236	0.882
Saliva	0.551	0.868
Water intake (3 ml)	0.630	0.865
Water intake (10 ml)	0.677	0.862
Staple food intake (oral residue)	0.490	0.871
Staple food intake (pharyngeal response)	0.625	0.864

"Alertness" with zero variance was not listed in the Table 2. Two items (alertness, soft palate movement) were excluded from the RED, leaving 13 items. The Cronbach's α coefficient for the 13 items was 0.882

The inter-rater reliability calculated in the face-to-face condition using the kappa coefficient ranged from 0.770 to 1.000. According to Landis and Koch’s method [28], we found substantial agreement for "voluntary cough" and "staple food intake (pharyngeal response)," and almost perfect agreement for the remaining items (Table 3).

Table 3

Inter-rater reliability of RED

	Inter-rater reliability(face-to-face)
	n = 38(18 OC patients, 20 healthy elderly individuals)70.7 ± 6.7 years
Speech intelligibility	0.865
Tracheotomy	1.000
Voluntary cough	0.773
Sustained phonation and voice quality	0.854
Lip closure	0.872
Oral diadochokinesis /ka/	1.000
Tongue movement	0.892
Strength of the tongue	0.943
Saliva	0.803
Water intake (3 ml)	1.000
Water intake (10 ml)	0.803
Staple food intake (oral residue)	0.887
Staple food intake (pharyngeal response)	0.770

In our criterion-related validity verification under the face-to-face conditions, the Speaman’s rank correlation coefficient (rs) between the total scores of RED and MASA was 0.923 (p < 0.001). When the RED score for the oral preparatory stage was compared between the two groups (with and without disorders based on VF), there was a statistically significant difference (Z = -4.506, p < 0.001). Similar results were also found for the oral (Z = − 3.847, p < 0.001) and pharyngeal stage scores (Z = − 4.149, p < 0.001, Table 4).

Table 4

RED category scores of the participants with/without disorders for different physiological stages

		n	Mean age	Median of the corresponding stage score ( inter quartile range)	Statistical significance
Oral preparatory stage disorder	With Without	33 39	70.0 ± 9.3 69.3 ± 12.4	0 ( 0–1) 2 ( 1–3)	***
Oral stage disorder	With Without	49 23	69.6 ± 10.7 69.3 ± 11.9	1 ( 0–1) 2 ( 1–2)	***
Pharyngeal stage disorder	With Without	41 31	72.4 ± 9.2 65.5 ± 12.1	3 ( 1–5) 6 ( 5–6)	***

Mann-Whitney U test: *** p < 0.001

In our verification of construct validity under the face-to-face condition, the total as well as respective stage score of RED were compared in accordance with the DSS. We found significant differences among the three DSS groups for the total and oral preparatory stage. In the oral stage score, there were significant differences between the DSS1-4 and DSS7, and between the DSS5-6 and DSS7, but no significant differences between the DSS1-4 and DSS5-6 groups. In the pharyngeal stage score, there were significant differences between the DSS1–4 and DSS5–6, and between the DSS1–4 and DSS7, but no significant differences between the DSS5–6 and DSS7 groups (Fig. 3).

Fig. 3

Differences in RED-14 total and stage score among the 3 DSS groups. n = 93 (DSS1–4: 35, DSS5–6: 37, DSS1–4: 21), Kruskal–Wallis test, Dunn-Bonferroni. ** p < 0.01, *** p < 0.001, × : median

The cumulative percentage of the group with deglutition disorders are shown in the Table 5. The area under the curve (AUC) for deglutition disorders based on the ROC curve (Fig. 4) was 0.819 (p < 0.001), the sensitivity/specificity was 0.74/0.67, and the negative predictive value was 0.46 (Table 6). The cumulative percentage of the group with aspiration are shown in the Table 5. The AUC for aspiration was 0.913 (p < 0.001), with a sensitivity/specificity of 0.80/0.98 and a negative predictive value of 0.93. The cut-off point was 8/13 points or less for the likelihood of deglutition disorders, while 3/13 points or less for the likelihood of aspiration (Fig. 4, Table 6).

Table 5

Cumulative percentage of deglutition disorders/aspiration

Total score	With deglutition disorders	No deglutition disorders	With aspiration	No aspiration
	n = 5470.2 ± 10.7 years	n = 1867.2 ± 11.9 years	n = 2072.7 ± 9.9 years	n = 5268.2 ± 11.2 years
0 1 2 3 4 5 6 7	7.4 16.6 25.9 31.5 33.3 38.9 44.4 63.0	– – – – – – 5.6 22.2	20.0 45.0 70.0 80.0 80.0 80.0 80.0 95.0	– – – 1.9 3.8 9.6 17.3 36.5
8	74.1	33.3	95.0	51.9
9	92.6	50.0	95.0	76.9
10	98.1	66.7	95.0	88.5
11	100	77.8	100	92.3
12	100	88.9	100	96.2
13	100	100	100	100

Bolded letters indicates the area where the sum of sensitivity and specificity is maximized

The numbers in the table indicate the cumulative percentage (%)

Fig. 4

ROC curves for (A) aspiration and (B) deglutition disorders

Table 6

The ROC-AUC values for the detection of the occurrence of deglutition disorders/aspiration by the VF

	AUC	Standard error	Statistical significance	CI (95%)Lower Upper		Sensitivity	Specificity	Negative predictive value
Deglutition disorders	0.819	0.055	***	0.712	0.926	0.74	0.67	0.46
Aspiration	0.913	0.048	***	0.818	1.000	0.80	0.98	0.93

AUC area under the receiver operator characteristic curve, CI confidence interval

***: p < 0.001

–

–

–

–

–

–

–

–

–

STEP 2 (Reliability of RED Under Remote Conditions and Questionnaire to Participants)

The intra-rater reliability between the face-to-face and remote conditions was high for all items, with Kappa coefficients ranging from 0.654 to 1.000. Substantial agreement was found for "water intake (3 ml)," "water intake (10 ml)," and "staple food intake (oral residue)," while almost perfect agreement was found for the remaining items (Table 7).

Table 7

Intra-rater reliability (face-to-face and remote) and Inter-rater reliability (remote)

	Intra-rater reliability (face-to-face and remote)	Inter-rater reliability(remote)
	n = 40(20 OC patients, 20 healthy elderly individuals)70.0 ± 7.5 years	n = 40(20 OC patients, 20 healthy elderly individuals)70.0 ± 7.5 years
Speech intelligibility	1.000	0.881
Tracheotomy	1.000	1.000
Voluntary cough	0.925	0.688
Sustained phonation and voice quality	0.942	0.772
Lip closure	0.838	0.781
Oral diadochokinesis /ka/	0.925	0.756
Tongue movement	0.899	0.899
Strength of the tongue	0.900	0.950
Saliva	0.908	0.773
Water intake (3 ml)	0.684	0.895
Water intake (10 ml)	0.654	0.827
Staple food intake (oral residue)	0.730	0.721
Staple food intake (pharyngeal response)	0.895	0.660

The inter-rater reliability (kappa coefficient) under the remote condition ranged from 0.660 to 1.000. Substantial agreement was found for "voluntary cough," "sustained phonation and voice quality," "lip closure," "oral diadochokinesis /ka/," "saliva," "staple food intake (oral residue)," and "staple food intake (pharyngeal response)," and almost perfect agreement for the remaining items (Table 7).

The time required for the face-to-face evaluation was 612.1 ± 111.1 s, and for remote evaluation was 877.8 ± 107.4 s, showing a significant difference (t(39) = − 12.364, p < 0.001). Problems that occurred during the remote evaluation included "the voice did not reach the patient from the SLHT side" and "the image paused when the intraoral camera was operating." These problems were reported only once and resolved by restarting the web conference system.

Other two incidences included "incorrect attachment of the pharyngeal microphone" and "noise due to incorrect placement of the speakerphone," for which the examiner pointed out the error on the screen and corrected immediately. Since the deglutition sound was enhanced by the loudspeaker, noise cancellation was activated for some subjects, and the deglutition sound could not be heard. Therefore, it was necessary to adjust the distance between the loudspeaker and the speakerphone and the volume of the loudspeaker accordingly. With four OC patients, the examiners had difficulty in visually confirming laryngeal elevation during "water intake" or "staple food intake." In one case, the patient quickly covered some of the food that had spilled from his mouth with his hand, and in another, there was difficulty determining the position of the thyroid cartilage due to edema from surgery. The remaining two patients swallowed while bending their head and neck forward during deglutition. In all four participants, examiners were able to confirm the timing of swallowing by listening to swallowing sounds using a pharyngeal microphone. There were no problems with equipment not working or having to conduct another session on another day.

In the questionnaire given to the participants after the remote examination, 5.0% of the respondents answered "1: strongly agree" or "2: agree" for the "fear" item. Similarly, 35.0% answered "nervous" and 95.0% answered "easy to answer." Free-text description responses included, "I was nervous," "not so different from face-to-face, no sense of discomfort," and "I found it interesting and fun" (Table 8).

Table 8

Breakdown of participant free-text descriptions

Themes	Number
I was nervous	11
Not much different from face-to-face conditions, no discomfort	8
I found it interesting and fun	6
I feel more confident about the future and find it more convenient	4
Face-to-face conditions are best, but remote conditions are also good as an option	4
Remote is fine if you are acquainted with the examiner	3
I was shy or anxious	2
Easy to understand with a clear view of the examiner's face	2
I felt some discomfort	3

Discussion

STEP 1 (Reliability and Validity in Face-to-Face Conditions)

Difficulties in accessing specialists due to distance, transportation, and physical disabilities, as well as advances in Internet technology and the spread of COVID-19, have increased the need for telemedicine. Many studies have been conducted on deglutition evaluation remotely, but most of them intended to verify the degree of agreement between the face-to-face and remote uses of the examinations originally designed for face-to-face conditions. In Part I of our study, we extracted items that are suitable for remote administration. In addition, we obtained agreement from SLHTs on the viscosity and amount of food to be tried, and the role of the assistant. In the present study, we constructed RED with these items, and conducted reliability and validity verification in face-to-face conditions in a sample of healthy elderly individuals and OC patients.

We found high internal consistency of 13 items of RED under the face-to-face condition, with a Cronbach's alpha coefficient of 0.882. Mann [24] reported that the alpha coefficient of MASA was 0.896, which suggests that RED is comparable with MASA and consists of items with high internal consistency.

The inter-rater reliability of RED under face-to-face conditions was high for all items. The RED was found to be a reliable examination under face-to-face conditions.

We found a strong correlation between the RED total score and the MASA total score in our verification of the criterion-related validity. Furthermore, there was a statistically significant difference in the RED scores for the oral preparation stage, the oral stage, and the pharyngeal stage when the two groups (those with and without VF-based disability) were compared (Table 4). This result indicates a high criterion-related validity.

Next, to examine the construct validity under the face-to-face condition, we compared the RED total score and respective stage score according to the severity of dysphagia based on the DSS. We found significant differences among the three DSS groups in the total score and oral preparatory stage score. On the other hand, there was no significant difference in the oral stage score between the DSS1-4 and DSS5-6 groups. This may be attributed to the fact that the maximum score for the oral stage score is 2 points. The oral stage score should be considered as a rough guide to determine the presence or absence of disability. In addition, there was no significant difference in the pharyngeal stage scores between the DSS5–6 and DSS7 groups. Tanaka et al. [29] reported that hemi-glossectomy is less likely to cause major problems in the pharyngeal stage of swallowing, but causes problems in the oral preparatory, and oral stages. Since the DSS5–6 group represented minimum oral problems, and the DSS7 represented normal limits, we can infer that our results reflect the characteristics of patients without pharyngeal stage disorder. Therefore, our results support the RED construct validity.

The present study showed high AUC, sensitivity, and specificity. Horii et al. [30] found that the combination of the modified water swallowing test and the food test was able to detect dysphagia in patients who have undergone resection for head and neck cancer with higher accuracy than either test alone. Our OC patients and the combination of items that targeted the detection of deglutition disorders may have contributed to the high accuracy of the results.

The cut-off point (8/13 or less for the likelihood of deglutition disorders, 3/13 or less for the likelihood of aspiration) can be used as an indication for further assessment. If the score is below the cut-off point, it may be necessary to proceed to an instrument evaluation.

Both the intra-rater reliability (agreement between face-to-face and remote conditions) and the inter-rater reliability at a distance were high for all RED items. Although some studies have validated the agreement through simultaneous face-to-face and remote assessments [10, 16], in the present study, high agreement was obtained despite the assessments were conducted on different days. This may be attributed to the fact that we extracted items with high potential for remote implementation in Part I of our study. We conclude that RED provides comparable results in both remote and face-to-face settings.

In the questionnaire surveys of SLHTs in Part I of our study, the most common comments were "concerns about the quality and stability of the voice that may be heard through the device. [22]" However, there was also a high level of agreement in the items that included assessment of voice quality: "sustained phonation and voice quality," "water intake (3 ml, 10 ml)," and "staple food intake (pharyngeal response)." Ward et al. [16] found a high degree of agreement between remote and face-to-face comparisons in their evaluation of wet voice. Nevertheless, weak voice intensity impacted the assessors’ ability to hear any subtle changes in voice quality after swallowing, forcing the online speech pathologists to rely more heavily on other signs of potential aspiration risk [11], indicating that remote evaluation may not be the most efficient means for some patients. Although the voice quality was highly consistent under the investigated conditions of this study, it remains unknown whether this level of agreement can be maintained in other patient populations.

The time required for our RED was about 10 min in face-to-face conditions and about 15 min in remote conditions. RED was designed to reduce the evaluation time by focusing on items with key elements of deglutition disorders. It is anticipated that many subjects undergoing the deglutition evaluation will have a combination of decreased endurance and ability to maintain a seated position. These patients may benefit from RED, which can be done in a short time. Numerous authors have recommended the use of telemedicine for the initial deglutition evaluation or subsets of these evaluations during the pandemic [15, 17, 31–35]. The fact that RED can be performed in a shorter time shows its potential as a screening test and/or an initial rough clinical evaluation of deglutition.

Most of the technical problems which occurred during the remote evaluation were resolved by restarting the internet-based video conference system or by on-screen instructions from the SLHT. Ward et al. [12] conducted remote evaluations on 100 participants and found that 6 of them needed to be rescheduled. Although the number of subjects in our study was small and the remote environment was a simulated setting in the same facility, the problem of rescheduling did not occur.

In the four patients for whom the examiners had difficulty determining laryngeal elevation visually, the timing of deglutition could be confirmed in all cases by listening to deglutition sounds using the pharyngeal microphone. It has been reported that the measurement of swallowing frequency by swallowing sounds has a high degree of agreement with actual swallowing [36]. Ward et al. [10] used another device by applying white tape to the larynx to visualize laryngeal movement during deglutition. However, many people with deglutition disorders routinely flex their head and neck as a compensatory manner, which may obscure visualization. Detection of swallowing sounds with a pharyngeal microphone may be a potential adjunct when observation of laryngeal elevation is difficult.

Iiboshi et al. [27] conducted a clock drawing test in a remote environment, during which 69% of the subjects felt fearful or nervous, and 70% reported that the ease of answering the questions was equal to or better than in-person. Similarly, Ward et al. [12] found that 83% of patients who experienced remote swallowing assessments in-person responded that they were equivalent to face-to-face assessments. In our study, only 5.0% of the subjects felt fear, and 95.0% reported that it was easy to answer. These findings along with the large number of positive free-text participant comments on the remote assessment indicated that there was a high degree of acceptability regarding the remote administration of RED, although consideration must be given to participants who experience fear or nervousness. This could be addressed by inviting a supporter to be present, revising the examiner training package to include more information to improve patient understanding and allay fears, as well as explicitly allowing subjects to ask questions beforehand.

Finally, we should mention the limitation of this study. RED is not a comprehensive assessment, although it is useful in detecting the likelihood of aspiration and key elements of deglutition disorders. It should be noted that RED was only performed on OC patients, thus the efficacy data are only applicable to this patient population. Obviously, further studies are needed to apply RED to other patient populations. Furthermore, equipment such as the intraoral camera used in this study may not be available at all sites. Use of simpler equipment needs to be investigated in future studies as well.

Conclusion

In this study of healthy individuals and OC patients, we found high reliability and validity of RED under the face-to-face conditions. High sensitivity, specificity, and negative predictive values were also found. Intra-rater reliability between face-to-face and remote conditions was good. RED can be performed in 10–15 min, thus, it is suitable for screening and/or initial rough evaluation of deglutition. In addition to the visualization of laryngeal movement, auditory signal using a pharyngeal microphone was useful in detecting laryngeal elevation. RED was well tolerated by both OC patients and healthy elderly individuals when performed remotely, although consideration should be given to nervousness reported by some of the participants.

Table 9

Judgment criteria for each item

Items	0 (abnormal)	1 (normal)
Alertnessa	No eye opening, or poor response to questions is observed	Attitude toward answering questions is ensured
Speech intelligibility	Decrease/deterioration in clarity, speed, or resonance of speech is observed	Speech is clear
Tracheotomy	The person is wearing a cannula, or the tracheostoma is not closed	No cannula is used, no tracheostoma is present, or tracheostoma is closed
Voluntary cough	Voluntary cough is weak, or breathy	Voluntary cough is good and strong
Sustained phonation and voice quality	The maximum phonation time is less than 10 s on two trials, or there is an obvious abnormality in the voice quality	The maximum phonation time is 10 s or longer on at least one of the two trials, or no abnormality is found in voice quality
Lip closure	There is insufficient lip closure, asymmetry, or limited range of motion	Lip closure is possible, no asymmetry is observed, and/or the range of motion is unrestricted
Oral diadochokinesis /ka/	It is not possible to repeat /ka/ more than 12 times in 3 s	It is possible to repeat /ka/ more than 12 times in 3 s
Tongue movement	Limited range of motion is observed in any one of the followings: anterior, posterior, or lateral tongue movement, elevation of the tongue tip or the posterior tongue	There is no limitation in the range of motion in any one of the followings: anterior, posterior, or lateral tongue movement, elevation of the tongue tip or the posterior tongue
Strength of the tongue	Unable to hold the tongue depressor at the tongue tip for 5 s	It is possible to hold the tongue depressor at the tongue tip for 5 s
Soft palate movementa	There is an obvious limitation in the range of motion of the soft palate, or there is an obvious asymmetry	There is no obvious limitation or obvious asymmetry in the range of motion of the soft palate
Saliva	If "1 (normal)" is not applicable	If all of the followings are met: 1) no drooling, 2) no obvious saliva retention in the oral cavity, 3) no obvious xerostomia
Water intake (3 ml)	If "1 (normal)" is not applicable	If all of the followings are met: 1) swallowing is possible, 2) there is no throat clearing/cough, 3) there is no wet voice, and 4) SpO2 is decreased by 2% or less
Water intake (10 ml)	If "1 (normal)" is not applicable	If all of the followings are met: 1) swallowing is possible, 2) there is no throat clearing/cough, 3) there is no wet voice, and 4) SpO2 is decreased by 2% or less
Staple food intake (oral residue)	There is obvious oral residue	There is　no obvious oral residue
Staple food intake (pharyngeal response)	If "1 (normal)" is not applicable	If all of the followings are met: 1) swallowing is possible, 2) there is no throat clearing/cough, 3) there is no wet voice, and 4) SpO2 is decreased by 2% or less

aBased on the internal consistency analysis presented later, this item was excluded from the RED

Table 10

Manuscript for the examiner

Alertnessa	□“Assistant, please correct the position of the Group’s camera so that it shows from the neck up.” □ “Hello. Nice to meet you.” □Conduct a brief free talk
Speech Intelligibility	□ “Assistant, please give the reading material to the participant.” □ “Please read the sentences aloud.” (1) I buy a blue house Akai iewo kau (in Japanese) (2) My body is sluggish and sluggish Karadaga darukute darukute shikataga nai (in Japanese) (3) I am lured by the whispering murmur of the shallows Sasayakuyona asaseno seseragini sasowareru (in Japanese) (4) This tatami room was built by my brother and his friends Kono tatamino heyawa otototo tomodachitode tatetamonodesu (in Japanese) (5) The sun seeth all things and discovereth all things Rurimo harimo teraseba hikaru (in Japanese) (6) When the fog clears, we can descend from the sky Kiriga harereba sorakara orirareru (in Japanese) (7) Papa and Mama all threw beans together Papamo mamamo minnade mamemakiwo shita (in Japanese)
Tracheotomy	□ “Let me observe your neck. Can you pull the collar down a little?” □ “Assistant, please zoom in on his/her neck.” □ “Assistant, return the Group’s camera to its original angle.”
Voluntary cough	□ “Please cough loudly, as I do.” □ “Please clear your throat loudly, as I do.”
Sustained phonation and voice quality	□ “Take a deep breath and then say 'ah' as long as you can.” Conduct twice
Lip closure	□ “Now I'm going to watch your mouth movement. Open your mouth.” □ “Assistants, please zoom in on his/her mouth.” (Group’s camera) □ “Close your mouth.” □ “Pull spread your lips to the side, as I do.” □ “Protrude your lips, as I do.” □ “Please repeat as I do, spread your lips to the side and then protrude them.” □ “Puff out your cheeks as I do. Hold it like that for five seconds.”
Oral diadochokinesis /ka/	□ “Repeat ‘ka ka ka ･･･’ as quickly as you can until I tell you to stop.” Perform twice for 3 s each
Tongue movement	□ “Assistant, please switch on the intraoral camera. We use the intraoral camera, as it gives us a clear view of the inside of your mouth.” □ “Open your mouth.” □ “Keep your mouth open and stick your tongue out as far forward as possible.” □ “Keep your mouth open and pull your tongue back.” □ “Touch the corner of your mouth with your tongue in this way.” □ “Next, the other side.” □ “Keep your mouth open and place the tip of your tongue on the back of your front teeth, as I do.” □ “Say ‘ka’ with your mouth open.”
Strength of the tongue	□ “Assistant, please prepare the tongue blade.” □ “We will check the strength of your tongue. With your mouth wide open, hold the tongue blade with your tongue, as I do. Please keep it for 5 s.” □ “Assistant, please use the intraoral camera to view the image from the side and downward.” □ “Now put the tongue blade in your mouth.”
Soft palate movementa	□ “Let me check the back of your mouth. Open your mouth.” □ “Breathe in through your nose and say ‘ah’.”
Saliva	□ “Let me check your mouth is moist. Let me see the right side. Now the left side. Now the upper side, now the lower side.” □ “Assistant, switch to Group's camera and show us starting from his/her neck to his/her mouth.” □ “We will now attach a microphone to confirm the sound of swallowing. Please turn it on.” □ “Please swallow your saliva.” □ “Assistant, please turn off the pharyngeal microphone.”
Water intake	□ “You will be asked to drink 3 ml of water. First, say ‘ah’.” □ “Put the pulse oximeter on your finger. Assistant, please monitor it and tell me if there are any abnormalities.” □ “Assistant, please fill the syringes with water. Please turn on the pharyngeal microphone.” □ “Please open your mouth slightly as we put water in your mouth. Please hold it until I tell you to swallow.” □ “Now, swallow it.” □ * “Assistant, please turn off the pharyngeal microphone.” □ * “Now, please say ‘ah’. Again, please.” □ “Next is 10 ml of water. Please hold the cup.” □ “Assistant, please turn on the pharyngeal microphone.” □ “Now, try to drink it as you usually do.” Repeat *
Staple food intake	□ “Assistant, prepare his/her staple food (in the form of his/her current diet)” □ “Assistant, please turn on the pharyngeal microphone.” □ “Please chew and swallow it whenever you are ready.” Repeat * □ “Assistant, please switch on the intraoral camera.” □ "May I see the inside of your mouth, please? Open your mouth wide, please. Let me see the right side. Now the left side. Now the upper side, now the lower side.”

Instructions in the bold face are omitted during the face-to-face evaluation

aBased on the internal consistency analysis presented later, this item was excluded from the RED