Standardized Checklist for Otoscopy Performance Evaluation: A Validation Study of a Tool to Assess Pediatric Otoscopy Skills.

INTRODUCTION: Acute otitis media (AOM) is the most frequently diagnosed pediatric illness in the United States and the most common indication for antibiotic prescription. Skill in pediatric otoscopy is essential to correctly identify children with AOM. However, studies have found diagnostic inconsistency and significant overdiagnosis among practitioners. Given the potential public and individual health consequences, there has been a call for improved education regarding the diagnostic certainty of AOM. Yet educational efforts continue to be limited, particularly in regard to competency assessment. The lack of a validated tool to assess otoscopy skill attainment objectively diminishes the instructor's ability to provide useful feedback and direction to the learner.
METHODS: We have undertaken an educational intervention with the goal of developing a validated Standardized Checklist for Otoscopy Performance Evaluation (SCOPE), building on key principles of the general pediatric ear exam. The SCOPE was developed with the input of process and content experts with attention to specific domains of validity.
RESULTS: Our analysis provides important validity evidence for the SCOPE assessment tool. The instrument was piloted and successfully implemented with medical students and varying levels of residents in pediatrics and emergency medicine over a 5-year period in varied settings: urgent care, large and small pediatric clinics, and the emergency departments at two institutions. It has been used for both instruction and assessment purposes. DISCUSSION: Because the SCOPE can be used in teaching demonstration purposes, in formative and summative assessment settings, and across the continuum of learners, this instrument offers the potential for more educational efforts in the field of assessment in direct patient care. We anticipate that the SCOPE will foster an environment of efficient yet high-yield review and discussion of otoscopy and diagnostic competency.

Educational Objectives

With appropriate use of the Standardized Checklist for Otoscopy Performance Evaluation, learners will be able to:

Demonstrate a competent technique for the general ear exam for a pediatric patient.

Demonstrate a competent technique for cerumen removal for a pediatric patient.

Demonstrate a competent technique for pneumatic otoscopy for a pediatric patient.

Introduction

Acute otitis media (AOM) is the most frequently diagnosed pediatric illness in the United States and the most common indication for antibiotic prescription.[1,2] Skill in pediatric otoscopy is essential to correctly identify children with AOM. However, studies have found diagnostic inconsistency among practitioners and significant overdiagnosis of AOM. This has resulted in an increased incidence of antimicrobial resistance and higher health care costs due to unnecessary antibiotic prescriptions and surgical referrals.[3]

Given the potential public and individual health consequences, there has been a call for improved education regarding the diagnostic certainty of AOM.[4] Revised clinical guidelines from the American Academy of Pediatrics (AAP) specify that “educational and dissemination methods both at the practicing physician level and especially at the resident level need to be examined.”[3] Furthermore, the AAP's recommendations reinforce the importance of ongoing education with instruction beginning early on in medical school and continuing throughout postgraduate training.[3,5] In response to this call for improved competency in the evaluation and diagnosis of AOM, otoscopy curricula are emerging in the literature. Yet educational efforts continue to be limited, particularly in regard to competency assessment.[2,6-18]

The lack of a validated tool to assess otoscopy skill attainment objectively diminishes the instructor's ability to provide useful feedback and direction to the learner. We undertook an educational intervention with the goal of developing a validated Standardized Checklist for Otoscopy Performance Evaluation (SCOPE), building on key principles of the general pediatric ear exam. This assessment tool enables instructors to determine level of competency across a range of clinical and simulated learning environments covering the continuum of trainee levels and specialties.

The checklist was developed within the framework of Glassick and Kirkpatrick's scholarship criteria, and it can be disseminated for other sound scholarship purposes.[19,20] The checklist provides defined and clear goals for learner assessment. It has been tested in and can be used with appropriate scholarship in the setting of at least the first three tiers of Kirkpatrick's levels of evaluation.[20] Regarding its content, curriculum process experts contributed to the content itemization of the instrument checklist. The SCOPE is found in Appendix A. Also provided is a version of the checklist containing facilitator instructions (Appendix B).

Methods

An extensive search of the literature including pediatrics, otolaryngology, simulation, international, and ancillary (nursing) was performed and updated over the course of 2 years to determine sources of expert content for the basis of the checklist instrument. Two key resources containing expert content emerged from the review: Kaleida et al.'s “Mastering Diagnostic Skills: Enhancing Proficiency in Otitis Media, a Model for Diagnostic Skills Training,”[10] and Shaikh, Hoberman, Kaleida, Ploof, and Paradise's “Diagnosing Otitis Media—Otoscopy and Cerumen Removal.”[11] From these resources, as well as from content experts in the field of pediatrics otoscopy, general pediatrics, and otolaryngology, points in the continuum of proficiency for the following three domains were identified and deliberated upon: (1) general approach to the pediatric ear exam, (2) cerumen removal, and (3) pneumatic otoscopy.

The first instrument subsection, on the general examination, was initially developed with a target audience of third-year pediatrics clerkship medical students. This subsection was developed to serve as the essential base to becoming proficient in pediatric otoscopy. The subsection should be used as the base for all levels of learners since proficiency and competency cannot be assumed.

The instrument was piloted for a period of 6 months over four clerkship rotations prior to actual implementation. The instrument was then utilized in various direct patient care settings with third-year medical students for a period of 2 years. For over 4 years, the instrument has been used in pediatric otoscopy student workshop sessions during demonstrations and as students practiced their otoscopy skills on both manikins and each other.

The second and third subsections of the instrument, on cerumen removal and pneumatic otoscopy, respectively, were developed as more advanced skills in the longitudinal learning of pediatric otoscopy. Again, the aforementioned content experts and expert process input were used to develop these two subsections of the checklist. Specifically for pneumatic otoscopy, a validated novel ear simulator was used for development of the checklist and then for teaching, evaluation, and validity purposes.[21]

General pediatricians from community settings and academic centers also contributed to the final phases of checklist development. The instrument was pilot tested over a 1-year period with intern and senior residents in urgent care and emergency department settings and in ambulatory general pediatric settings of various sizes and patient populations.

The intent of the checklist is for integration as the assessment component in the instructor's current otoscopy skills curriculum. There is an assumed level of competency for the assessor. However, if assessors wish to gain additional competence in this area, we favor review of Shaikh et al.'s otoscopy videos[11] and Kaleida et al.'s images[10] as excellent resources for curriculum content. Additional resources may be used for this purpose if needed. Estimation of time to review this material depends on prior experience of the assessor and has ranged between 2 and 8 hours.

The examinees should have undergone training in the pediatric ear exam and otoscopy as per the instructor's/institution's curriculum. The assessment may be performed at any level of the learner continuum. However, we recommend initial evaluation immediately following administration of the curriculum. Repeat assessments may be timed either upon completion of a clinical rotation (i.e., medical student pediatric clerkship) or prior to the subsequent postgraduate year or graduation.

The assessment may be utilized in a variety of settings. We have utilized it to evaluate pediatric ear exam and otoscopy skills on simulated manikins, on OSCE patients, and during actual patient encounters. In all instances, as a direct observation tool, there are no significant additional resources required.

The assessment tool requires approximately 5 to 10 minutes for direct observation of the learner and completion of the checklist. One evaluator is necessary for each trainee. To maintain maximal reliability during formative/summative evaluations, the same evaluator should be utilized for reassessments of a single trainee.

The assessment instrument is completed via a combination of yes/no check boxes and brief fill-in-the-blank questions. The final component of the assessment requires the instructor to perform her/his own examination to confirm the diagnosis.

The components of the assessment include (a) discussion with caregiver, (b) selection of equipment, (c) positioning of the patient, (d) distraction techniques, (e) otoscopy exam technique (including insertion and stabilization techniques, technique of cerumen removal if indicated, and pneumatic otoscopy), and (f) diagnosis.

Review of the assessment results between instructor and trainee may be performed at an appropriately scheduled time. Performance is based on checklist findings and provides concrete feedback on areas of competency and skill deficiency. Deficient areas may be addressed through review of the instructor's curriculum or provision of additional resources, such as Shaikh et al.'s otoscopy videos[11] or other suitable materials.

Examples of use: Videos with pediatric patient ear exams were developed to assess inter-and intrarater reliability for validity purposes (see the Results section). These videos are not needed to teach use of the checklist.

Results

Content Validity

Content experts, pediatric otoscopy experts, and general pediatricians from diverse clinical practices contributed to the development of the checklist, providing for evidence of content validity.[10,11] The checklist focuses on the general examination, pneumatic otoscopy, and cerumen removal. Pertinent well-established and peer-reviewed principles of pediatric otoscopy were gleaned from the expert content sources and incorporated into the checklist's final form.

The checklist was developed with facilitated feedback collected over a 2-year period from stakeholders such as faculty, clinicians, and learners. During this period, the checklist demonstrated that it matched the intended construct and also demonstrated feasibility.

The instrument was demonstrated to be feasible with various groups of learners: medical students (n = 83), pediatric interns at the beginning and end of their intern year across two institutions (n = 40), pediatric senior residents (n = 14), and emergency medicine residents (n = 12) over a 3- to 4-year period at two separate institutions. The instrument was also demonstrated to be feasible in a variety of clinical settings, including general pediatric clinics, pediatric urgent care clinics, and a pediatric emergency medicine department.

Furthermore, the instrument was found to be useful among a variety of general pediatric clinic preceptors across a number of different clinic sizes and settings (e.g., community vs. academic). Preceptors in two separate pediatrics institutions who worked with emergency medicine faculty also found this useful. An institutional review board–approved survey of the general pediatrics preceptors demonstrated feasibility of the instrument with their students. Of surveyed faculty, 100% reported that the instrument did not interfere with patient care or with teaching their students, 88% reported ease of checklist implementation in their teaching clinics, and 75% reported that the instrument improved their direct observation of their students.[22]

Internal Structure

Reproducibility of the checklist

The checklist was successfully implemented in different clinical settings, such as general pediatric clinics, pediatric urgent care clinics, and a pediatric emergency medicine department, over a 2-year period. Clinics varied in number of preceptors on site (ranging between two and eight), setting (community vs. academic), and patient population (socioeconomic status, ethnicity, etc.).

The scores on the general examination subsection of the checklist significantly and appropriately increased with advancing levels of learners (see Table 1 amp; Table 2). The scores on the general examination, pneumatic otoscopy, and cerumen removal subsections also significantly increased between the first and second years for both pediatric and emergency medicine residents (see Table 3).[23]

Table 1.

General Examination Scores by Learner Group as Percentages of the Maximum Score

Learner Groupa	Score M (SD)
Medical student (n = 83)	45.9% (17.8%)
Preinternship year (n = 10)	40.4% (17.8%)
Postinternship year (n = 9)	75.9% (17.4%)
Postpediatric level 2 year (n = 9)	88.9% (11.8%)

Samples include carefully tracked cohorts with similar exposure.

Table 2.

Comparison of General Examination Scores Between Learner Groups

Comparison of Learner Groups	Effect Sizea	pb
Preinternship year vs. postinternship year	1.7	<.001
Postinternship year vs. postpediatric level 2 year	0.9	0.002
Medical student vs. postinternship year	1.7	<.001

Cohen's d test.

Student's t test.

Table 3.

Subsection Scores as Percentages of Total Correct Compared Across First and Second Years of Each Residency

Subsection and Residency Group	Intern Year 1 Score M (SD)	Intern Year 2 Score M (SD)	pa
General exam
Combined	41.2% (23.2%)	94.8% (7.7%)	<.001
Pediatrics	21.7% (6.1%)	98.3% (5.3%)	<.001
Emergency medicine	57.4% (19.2%)	91.7% (8.3%)	<.001
Pneumatic otoscopy
Combined	42.4% (33.9%)	79.4% (15.7%)	<.001
Pediatrics	35.6% (10.7%)	76.7% (14.1%)	<.001
Emergency medicine	48.1% (44.6%)	81.8% (17.4%)	0.025
Cerumen removal
Combined	3.3% (14.1%)	83.6% (15.0%)	<.001
Pediatrics	2.0% (4.1%)	79.0% (14.5%)	<.001
Emergency medicine	4.4% (18.9%)	95.0% (10.0%)	<.001

Student's t test.

Reliability of the checklist

Ten faculty preceptors utilized the checklist to assess standardized ear exam skills with real pediatric patients in videos specifically developed to assess the accuracy of their assessments compared with the correct answers, which were developed a priori. Preceptors were asked to assess the examiner's otoscopy skills according to the checklist. Scores were analyzed for accuracy of assessment. The checklist demonstrated high accuracy, with faculty preceptors achieving 95%-97% correct responses when using the checklist to assess the standardized ear exams.[24]

An intraclass correlation statistic was used to assess the interobserver reliability of the checklist between the five preceptors who evaluated students in the real-time patient setting using the standardized ear exams. Using standard criteria, the intraclass correlation of 0.8 indicated an excellent level of interobserver reliability of the checklist.[23]

Response Process

Faculty and learners can become familiar with the checklist's content by assessor training (see the Methods section, Implementation subsection, above) and by various teaching venues for the learner. The checklist has been successfully demonstrated in teaching settings, such as in student labs and resident seminars at multiple institutions. A think-aloud approach was used during its development process among academic pediatricians from different clinical practices, pediatric otoscopy experts, and learners. Assessors have been shown to be consistent in how they assess their learners (see Reliability of the checklist subsection above).

Relationships to Other Variables

One weakness in the method of development of this instrument is the inability to fully assess its relationship to other variables. The appropriate and significant increase of scores with increasing levels of learners suggests some validity evidence for relationship to other variables. Further work is needed in this domain.

Consequences

The checklist was designed primarily for formative assessment, but it may be adapted for summative assessment. The construct is remediable, and the skills can be learned. The checklist facilitates feedback specific to content items, allowing for more effective feedback. Peer-reviewed resources may be consulted to aid in feedback.[10,11] The checklist has been used as a teaching/demonstration/practice tool in curriculum studies that demonstrated gains in learners’ clinic skills.[25]

Discussion

The need to learn and become proficient at the pediatric ear exam has been critically linked to accurate diagnosis of AOM, judicious use of antibiotics, and control of health care costs, including subspecialty care.[3] The literature is beginning to describe curricula for various learners, but there are no validated assessment instruments for pediatric otoscopy skills.

The SCOPE helps fill this gap regarding standardized assessment of pediatric otoscopy skills including cerumen removal and pneumatic otoscopy. That being said, use of the checklist in a clinical setting is not without certain limitations. Specifically, limitations inherent to checklists include the need to appropriately determine the relevant skills to be observed, potential recorder bias, and the need for clear observation of the skill by the instructor. Furthermore, the instructor must be familiar with the constructs of proper otoscopy skills and appropriately budget additional time to perform, review, and discuss results. Finally, parental agreement and cooperation are needed when the checklist is used in a clinical environment.

We have attempted to address these limitations through the validation of observed skills and diagnosis. To address instructor expertise, we offered standardized resources to improve the instructor's own competence. The checklist was specifically created with time constraints in mind and is not overly detailed, resulting in very little additional time being needed to perform the assessment.

The SCOPE has undergone extensive critique with regard to content and other domains of validity as well as clinical relevance to varying learner levels (medical students, pediatric interns, pediatric senior residents, and emergency medicine residents) and varied clinical settings (small and large general pediatric clinics, a pediatric urgent care clinic, and a pediatric emergency room) in two medical institutions over a 5-year period. Because it can be used in teaching demonstration purposes, in formative and summative assessment settings, and across the continuum of learners, this instrument offers the potential for more educational efforts in the field of assessment in direct patient care. We anticipate that the SCOPE will foster an environment of efficient yet high-yield review and discussion of otoscopy and diagnostic competency.

A. SCOPE Checklist.docx

B. SCOPE Checklist & Instructions.docx

All appendices are peer reviewed as integral parts of the Original Publication.