Development of video otoscopy quiz using a smartphone adaptable otoscope.

BACKGROUND: Otoscopy examination can be challenging. Traditional teaching uses still image illustrations. Newer attempts use video samples to simulate the otoscopy exam which is a dynamic process.Aims/Objective: To assess whether recorded otoscopy videos from a smartphone adaptable otoscope can be used to develop a video-based otoscopy quiz which may be used for instructing and familiarizing participants to normal anatomy and pathologic ear conditions. To use this quiz to assess current pediatric residents' competency of common otoscopy diagnosis. METHOD AND MATERIALS: This study was conducted in 2018. Video samples of ear pathology were collected at the Albany Medical Center using a smartphone adaptable otoscope- Cellscope. The videos were used to create a video otoscopy quiz (VOQ) without clinical vignettes. 45 pediatric residents from 3 academic institutions were evaluated with the quiz.
RESULTS: The weighted mean for the VOQ was 66.90% (95%CI 58.89%-68.42%). The breakdown by questions are: myringosclerosis 72.88%, retraction pocket 80.65%, cholesteatoma 42.22%, hemotympanum 75.04%, tympanic membrane perforation 79.62%, cerumen impaction 95.46%, otitis externa 52.54%, otitis media with effusion 63.30%, acute otitis media 75.55%, normal ear 36.39%.
CONCLUSION: We found that videos of otoscopy exams can be obtained with a smartphone adaptable otoscope and validated to develop a video-based quiz, which may be used to supplement otoscopic instruction. Following our testing process, we found pediatric residents are relatively well equipped to identify ear pathology on VOQ.

Introduction

The diagnosis of otolaryngologic pathology can be challenging, and exposure to otolaryngology is somewhat limited during medical school and residency. Many primary care residents report not getting adequate otolaryngologic instruction despite strong interest (O’Brien et al., 2018). This is an important issue given that an estimated 20–50% of visits to primary care providers and pediatricians are for otolaryngology-based disorders (Donnelly et al., 1995; Griffiths, 1979; Hannaford et al., 2005; Hu et al., 2012; Uijen et al., 2011). Of particular importance is the otoscopic exam which is used widely across specialties but takes considerable practice and instruction to master. In an evaluation of otoscopy skills among family medicine, pediatric and otolaryngology residents, Oyewumi et al. illustrated that pretest skills of primary care residents were poor; however, improvement could be seen after just 1-h of teaching (Oyewumi et al., 2016).

Otoscopy may be taught through various modalities including classroom instruction, textbooks, and clinical experiences (Davies et al., 2014; Lee et al., 2015; Morris et al., 2012; Wickens et al., 2015; Wu and Beyea, 2017). More recently, online resources and simulators using software subscriptions or commercially purchased hardware have become popular. Each of these methods show some improvement on otoscopy skills with their use; but, typically rely upon still images or illustrations to demonstrate pathology. Video otoscopy has been proven valuable in determining the presence of middle ear pathology and found to improve diagnostic accuracy of middle ear effusion when paired with pneumatic otoscopy (Al-Khatib et al., 2010; Jones and Kaleida, 2003). Videos of ear pathology can easily be taken using attachments to a smart phone and provide dynamic instruction. Studies have validated this technology as a diagnostic tool and patients generally find the recorded videos helpful in understanding their diagnosis (Moshtaghi et al., 2017; Richards et al., 2015; Sahyouni et al., 2016). As otoscopy is dynamic, video rather than still images may better prepare pediatricians for otoscopic examinations. A study by Jones and Kaleida found that assessment using videotaped otoscopy examinations was able to distinguish the skill levels of pediatric residents, novices, and experts at diagnosing middle ear effusion, validating the use of video otoscopy as a means for assessing the otoscopic interpretive skills of pediatric residents (Jones et al., 2004). This study proposes a method of developing and validating a video library of ear pathology which may be used for both teaching and assessment of otoscopic skills.

Methods

This study was conducted in 2018 and included 3 separate phases consisting of an video acquisition phase, a validation phase, and a testing phase. Fig. 1. Prior to study initiation, protocols were submitted to the institutional review board at all participating institutions for expedited review and exemption approvals were granted.

Fig. 1

Three phases of the study.

Video acquisition phase

The video acquisition phase consisted of identifying and compiling a video bank of common ear pathology. Commonly identified otologic pathology included a normal ear, acute otitis media (AOM), otitis media with effusion (OME), tympanic membrane perforation, cerumen impaction, myringosclerosis, otitis externa, retraction pocket, cholesteatoma, and hemotympanum Videos were obtained by 2nd year otolaryngology residents in the setting of an academic medical center from the emergency department, various hospital units, operating room, and the otolaryngology clinic. Consent was obtained prior to obtaining each video. Videos were collected with use of the commercially available smart phone otoscope attachment CellScope Oto (CellScope Inc., San Francisco, Ca). The smart phone otoscope is paired with a downloadable app developed by the CellScope Oto that processes and calibrates the recorded video. Recorded videos were de-identified prior to use. Our goal was to obtain 5 videos for each of the 9 different pathologies and 5 videos of the normal ear for a total of 50 videos prior to video validation. Each video was roughly 5–10 s long and some were further trimmed to focus on the pertinent portion of the otoscopic examination.

Validation phase

Our goals for the validation phase of the study were to determine whether proper diagnosis of the various pathologies could be performed using videos obtained with the smart phone otoscope, and to filter down the library to the most representative videos for our testing phase. Validation was performed by administering a computerized multiple-choice video otoscopic quiz with the collected videos to an expert panel. The expert panel consisted of six otolaryngologists from three different academic centers who were either fellowship trained in otology or devote a significant amount of their practice to the diagnosis and treatment of otologic disorders. Questions on the expert quiz were without patient history to require a diagnosis based solely on each presented video. Videos on the expert quiz that were correctly identified by at least 80% of the expert panel were then used to develop the final video otoscopy quiz (VOQ).

Testing phase

The testing phase involved testing pediatric residents with our VOQ. The final VOQ consisted of 19 questions using videos validated by the expert panel in multiple choice format. Two videos were included for each of the 10 ear conditions with the exception of AOM. Only one AOM video was included in the VOQ since only one of five AOM video representations reached a concordance of 80% among the experts during the validation phase. Participants included 45 pediatric residents between PGY 1–3 years of training, 20 from Albany Medical Center, 10 from Boston Children’s Hospital, and 15 from Keck Hospital of University of Southern California. Among the total residents participating in our study, 20 were in their PGY1 year, 12 were in their PGY2 year, and 13 were in their PGY3 year. Results were recorded, analyzed, and compared using Microsoft Excel.

Results

Video acquisition for the 9 ear pathologies and normal ear resulted in 50 total videos representing 5 videos from each of the 10 ear conditions. The 50 videos were used to develop a 50 questions expert quiz for validation by the expert panel. The average time taken to complete the expert quiz was 40m 24s. The otoscopic finding with the highest average concordance rate was the normal ear followed by cerumen impaction with an average of 86.6% and 86.2% respectively. The group with the lowest concordance rate was AOM at 49.8%. A total of 27 videos with a concordance rate greater than 80%. The breakdown of the 27 validated videos was acute otitis media (1), cerumen impaction (5), cholesteatoma (2), hemotympanum (2), otitis media with effusion (2), myringosclerosis (3), normal ear (4), otitis externa (4), perforation (2), and retraction pocket (2). Every group except for AOM had at least 2 videos with at least an 80% concordance. Results for expert validation of each group are found in Table 1. Two videos were selected randomly from each ear pathology group with the exception of acute otitis media which only one video was available to be included in the VOQ.

Table 1

Results of expert otoscopic video validation.

Pathology	Questions	N > 80% concordance	All questions average concordance (%)
Acute Otitis Media	5	1	49.8
Cerumen Impaction	6	5	86.2
Cholesteatoma	5	2	56.6
Hemotympanum	4	2	66.8
Otitis Media with Effusion	4	2	70.8
Myringosclerosis	6	3	61.2
Normal	5	4	86.6
Otitis Externa	5	4	73.2
Perforation	5	2	73.4
Retraction Pocket	5	2	73.4

A total of 45 pediatric residents took the VOQ. The weighted mean score of the VOQ of pediatric residents PGY 1–3 was 66.90% (95%CI 58.89%–68.42%). There was no significant differences between the mean scores of PGY1, PGY2, and PGY3 participants, which were 64.7% (95%CI 58.5%–70.1%), 67.9% (95%CI 61.2%–74.7%), and 69.2% (95%CI 61.0%–77.4% respectively. Table 2. Cerumen Impaction had the overall highest correct responses with 95.46%. The most missed question was of the normal ear with only 36.39% correct responses Table 3. The breakdown of the percentage correct by questions are: myringosclerosis 72.88%, retraction pocket 80.65%, cholesteatoma 42.22%, hemotympanum 75.04%, tympanic membrane perforation 79.62%, cerumen impaction 95.46%, otitis externa 52.54%, otitis media with effusion 63.30%, acute otitis media 75.55%, normal ear 36.39%.

Table 2

Pediatric resident video otoscopy quiz score.

Level of Training (N)	Mean Score (out of 19)	Range (out of 19)	95% CI	StandardDeviation
PGY 1–3 (45)	12.71	8–17	11.19–13.0	3.97
PGY 1 (20)	12.3	8–17	11.1–13.5	2.68
PGY 2 (12)	12.91	10–16	11.6–14.2	2.28
PGY 3 (13)	13.15	8–17	11.6–14.7	2.87

Table 3

Pediatric resident percentage correct by question.

Ear Pathology	Percent Correct (%)
Myringosclerosis	72.22
Retraction Pocket	81.11
Cholesteatoma	42.22
Hemotympanum	75.55
Tympanic Membrane Perforation	78.88
Cerumen Impaction	95.55
Otitis Externa	52.22
Otitis Media with Effusion	63.33
Acute Otitis Media	75.55
Normal Ear	36.66

Web address of the Video Otoscopy Quiz (VOQ): https://amc.az1.qualtrics.com/jfe/form/SV_0O51gXOA0ugd4Al.

Discussion

Without the inclusion of clinical vignettes, we found pediatric residents scored relatively well on the VOQ. Accuracy tended to be tied more closely to pathology than to training year. Most residents were able to correctly identify cerumen impaction, but surprisingly, the most common incorrect answer choice was the normal ear, with only 36.69% of residents correctly identifying the exam finding. Residents responded with a wide range of incorrect responses showing a lack of certainty which may also represent an anticipation of challenging questions on the part of the examinee. The next most commonly misidentified pathology was cholesteatoma with 35% of residents incorrectly identifying it as AOM. We believe the low percentage of these two ear conditions was potentially due to the lack patient history since AOM and cholesteatoma present very differently clinically. While most pathology included in the VOQ were relatively easy to identify through otoscopic examination, normal ear and cholesteatoma might represent examples where otoscopy itself is insufficient to yield an accurate diagnosis and additional clinical history must be considered. This can be mitigated in the future by including a short clinical vignette in the question or by tailoring the answer choices such that it contains fewer conflicting answers.

Prior studies have evaluated pediatric resident understanding of otoscopy and broader otolaryngologic knowledge. Jones et al. tested pediatric residents PGY1-3 with a videotaped otoscopy exam and also found a similar trend in score with training progression. In their study, they found an overall mean of 76.23% (Jones et al., 2004). With PGY1-3 mean score of 74%, 77%, and 78%. We saw a trend of increasing percentage correct on the VOQ as pediatric residents progress through their training. However, this trend does not appear to be statistically significant as the confidence intervals are far too broad. Compared to Jones et al., our study showed lower mean scores across training years which could be a result of more challenging set of questions used for our VOQ or other underlying differences in study design.

Otoscopy simulators have come into favor recently and studies have demonstrated their effectiveness. Medical students have typically been the learners receiving simulation-based training and report an increase in their confidence in otoscopy diagnosis after its use (Davies et al., 2014; Lee et al., 2015). One noticeable advantage of a simulator is the tactile feedback while viewing ear pathology. A study by Morris et al. showed that after a training session with medical students on an otoscopy simulator, they were able to apply more appropriate pressure during pneumatic otoscopy and better diagnose the presence of effusion compared to counterparts not trained on the simulator (79.2% vs 57.3%). The drawbacks of simulators are the need for hardware, availability to the students, and cost. While our web-based VOQ does not provide tactile feedback, it is an open source of ear pathology videos accessible at the learner’s convenience. The overall participant response from this study was favorable towards adding this type of learning tool to home study; however, further study is required to determine its effectiveness as a teaching module.

Overall, we found the use of a smart-phone enabled otoscope to be straightforward and see its potential as an important teaching tool. On the other hand, the process of obtaining the necessary videos was somewhat difficult as the presentation of patients with the desired pathology was unpredictable. Collection of videos demonstrating AOM was challenging as these cases tend not typically present to the otolaryngology clinic and are seen by a primary care physician or in the emergency department. Collecting useable videos demonstrating hemotympanum was also technically challenging as the traumas observed typically have blood obstructing the canal from a laceration.

During the validation phase of our study, the validation experts also faced the significant challenge of determining a correct diagnosis without a clinical history. History can be pivotal in determining a diagnosis and studies have shown the importance of a good history in relation to physical exam or other ancillary tests (Shikino et al., 2015; Paley et al., 2011). In our study, we aimed to limit the impact of patient history on the expert panel validation of the videos and so patient vignettes were eliminated. History would have allowed better understand of whether the disease process was acute or chronic, primarily infective in nature or related to cholesteatoma. We would anticipate an increase in concordance if histories were given. Variation in brightness and color of the videos also attributed to difficult validation. We noticed variations largely depended on the size and brand of ear speculum used on the otoscope.

Validation of AOM proved especially difficult. Even though we obtained our goal of 5 videos for expert review, only one video had 80% concordance. Overall, the concordance for AOM was the lowest for any pathology (49.8%). This is not surprising given the lack of clinical history and sometimes difficult characterization of middle ear fluid and acuity of illness. Special attention should be payed to expanding the number of AOM videos in the study library when it is utilized for future study.

Limitations of our study include an inability to achieve 80% concordance in at least two videos for AOM resulting in including only one AOM question instead of two on the VOQ. Additionally, the video bank is not comprehensive to all ear pathology and will need to be updated as further pathologies are collected. While many of the videos were created by a single examiner using similar equipment, further limitations include the inability to completely standardize the videos creating variation in the otoscopic examinations. The validation process did help to eliminate some variation and should be used as the library is expanded.

Future study should focus on the educational potential of a VOQ like ours. Development of a training program and comparing its effectiveness to conventional otoscopic simulators or didactics would help to describe this potential. Future studies may also focus on developing education modules that offer information to help residents differentiate cholesteatoma from AOM which was a particularly difficult distinction in our study.

Conclusion

Videos can be obtained using a smartphone adaptable otoscope and validated to be used to develop a video-based quiz. The VOQ has potential to supplement otoscopic instruction, and help familiarize learners to normal anatomy and pathologic conditions, and this remains an area for future study. Pediatric residents are relatively well equipped to identify ear pathology on VOQ, but further educational opportunities exist. While most ear pathology can be identified through otoscopy exam alone without clinical history, otoscopy examination itself is insufficient to allow residents to properly identify certain pathology, like cholesteatoma.

Author contribution

Garrett Ni: Writing - original draft, PGY-1 Resident, Department of Otolaryngology-Head and Neck Surgery, Temple University Hospital, Philadelphia PA. Drafted the protocol and manuscript, obtained videos, recruited pediatric residents. Stuart Curtis: Writing - original draft, Pediatric Otolaryngology Fellow, Department of Otolaryngology, University of Michigan Medical Center, Ann Arbor MI. Drafted the protocol and manuscript, obtained videos, recruited pediatric residents. Adam Kaplon: PGY-1 Resident, Department of Surgery, Temple University Hospital, Philadelphia, PA. Provided data interpretation and manuscript edits. Neil Gildener-Leapman: Validation, Associate Professor, Department of Otolaryngology, Albany Medical Center, Albany NY. Coordinated validation phase, provided data interpretation and manuscript edits. Jacob Brodsky: The Balance and Vestibular Program. Assistant Professor of Otolaryngology, Harvard Medical School Department of Otolaryngology, Boston Children’s Hospital, Boston MA. Recruited pediatric residents, provided data interpretation and manuscript edits. Ksenia Aaron: Neurotology Fellow, Department of Otolaryngology, Stanford University Medical Center, Stanford CA. Recruited pediatric residents, provided data interpretation and manuscript edits. Jason Mouzakes: Validation, Chief of the Division of Otolaryngology and Section Chief for Pediatric Otolaryngology. Assistant Professor of Surgery and Director of the Residency Program. Albany Medical Center, Albany NY. Coordinated validation phase, recruited pediatric residents, provided data interpretation and manuscript edits.

Declaration of competing interest

None.