The Ophthalmology Surgical Competency Assessment Rubric (OSCAR) for Open Globe Surgical Management.

Background/Aims: To produce an internationally developed rubric to assess surgical competency in open globe management.
Methods: An international expert panel of seven ophthalmologist educators developed a standardized competency-based rubric. The steps to perform an open globe repair were outlined. Based on a modified Dreyfus model, the experts agreed on the steps of surgery and certain global indices. They then assigned descriptors for the competency expected of a novice, beginner, advanced beginner, and competent surgeon. The tool was then vetted by another panel of ten ophthalmologists. The main outcome measure was the final version of the tool as agreed upon by the expert review panel.
Results: The steps of open globe repair and key global indices were established. Descriptive wording for each step and global indices for novice, beginner, advanced beginner, and competent surgery were listed. All the expert comments were integrated to establish face and content validity.
Conclusion: This standardized rubric to evaluate resident competency should be used globally in training programs to assess open globe repair surgical skills. Using a modified Dreyfus model, four different levels of training competency were defined allowing a non-biased, objective, numerical and simple assessment.

Introduction

In 1999, the Accreditation Council for Graduate Medical Education (ACGME) established 6 core competencies.1 In response to the initial struggles by ophthalmology residency programs in the early 2000s to implement these competencies, the American Board of Ophthalmologists established the Program Directors’ Task Force on the Competencies which identified surgical proficiency as an additional competence that should be achieved during ophthalmology training.2,3 At that time the group identified surgical case logs as already implemented, with low faculty burden and having a moderate ability to assess surgical competency, while surgical skills assessment had a high faculty burden, but the greatest ability to evaluate surgical competency.2 However, since then there has been little advancement towards implementation of codified surgical skills assessment in ophthalmology resident training. In part this was due to the lack of well-established grading rubrics. The International Council of Ophthalmology recognized this gap and in the early 2010s began to develop various Ophthalmology Surgical Competency Assessment Rubrics (ICO-OSCARs). The ICO-OSCARs facilitate assessment and acquisition of surgical skills and many rubrics have been created ranging from phacoemulsification, strabismus, trabeculectomy, vitrectomy,4–7 and in-office procedures for panretinal photocoagulation and intravitreal injections.8,9 However, no ICO-OSCAR nor another surgical competency rubric for open globe surgical management has been developed.10 The development of OSCARs is now led by the Ophthalmology Foundation, which supports the continuation of this important initiative.

Ocular trauma, including open globe injuries, is a major cause of monocular blindness worldwide.11 In the United States, open globe injuries occur at a frequency of 4.49 per 100,000, with most injuries occurring among those aged 40 or less.12 Early and competent surgical repair to restore structural globe integrity is central in open globe management and the subsequent visual rehabilitation. Further, open globe injuries can have a significant economic impact.13 Currently, the ACGME Review Committee for Ophthalmology deems that graduating residents should achieve at a minimum four primary surgeon globe trauma cases,14 however, the document explicitly states that achievement does not signify competence. Thus, despite these well-established surgical case minimums, there is a critical need for supplemental tools to assist residency program directors in objectively assess the surgical competency of their residents.15–17 The goal of this project was to develop an internationally valid tool to evaluate the progression of a resident ophthalmologist’s surgical competence in performing open globe repair.

Methods

This study was approved by the Institutional Review Board of the Johns Hopkins University School of Medicine. All study activities adhered to the tenets of the Declaration of Helsinki. A group of seven content experts from the United States, Iran, Pakistan, India and Australia worked together online to develop an initial rubric draft for open globe repair. These experts represent multiple different subspecialties in ophthalmology. For this study, a rubric was defined as an explicit set of criteria for assessing procedural skill. Open globe repair was divided into nine steps and there were nine additional global indices. Initially, indications for open globe repair, assessment of open globe injury, and pre-operative and post-operative medical management were included, but were later excluded as the rubric’s goal was to score surgical competency in performing open globe repair.

Using a modified Dreyfus model for skill acquisition (novice, beginner, advanced beginner, and competent), each step and global indices was assigned behavioral narrative anchors for each level of skill.18 As per previous studies,7,8 the expert category was omitted as this level of skill is not achieved in training. Further, each category was assigned a numerical value so that scores can be calculated to track improvement. Finally, descriptors were written to reduce bias in assessment, ensure completion of certain steps and reduce any interpretative errors in the criteria.

The rubric was modified repeatedly until there was full consensus on the rubric’s face and content validity. A second group of content experts from India, United States, Australia, China, Singapore, Hungary, and the United Kingdom were then invited to review the draft for constructive feedback. Aiming for global representation, the reviewers were selected for their expertise, diverse ophthalmic subspecialties and experience in teaching open globe repair. Their suggestions were cataloged, reviewed, and incorporated to produce the final rubric.

Results

The steps of open globe repair were broken down into nine key steps (Figure 1). Further, nine global indices demonstrating the surgical safety, fluidity and critical surgical knowledge of instrumentation were developed. A modified Dreyfus model was then applied to each of the 18 steps and indices. Depending on the competency in each of the 18 components between 2 and 5 points could be scored for a total of 90 possible points.

Continued.

OSCAR:OpenGlobe.

The international panel had general and specific suggestions on the initial draft. All expert comments were considered, and the authors incorporated appropriate suggestions, thus establishing a level of face and content validity. Most comments pertained to the addition of new items. These were mostly related to preoperative evaluation, anesthesia considerations, postoperative complications and/or postoperative care. The content experts considered these comments but believed including them would be out of scope for a surgical skill assessment tool. Additionally, including these other parameters would make the rubric too extensive and burdensome to complete, which could potentially discourage uptake of the tool. Thus, the consensus of the experts was to not add these categories. The final draft was called the Ophthalmology Surgical Competency Assessment Rubric for open globe repair (OSCAR:OpenGlobe).

Discussion

As demonstrated by the OSCAR: OpenGlobe, open globe injuries frequently require complex repair and a diverse range of ophthalmic skills including iris manipulation, muscle disinsertion, and the management of vitreous and lens fragments. These cases can place a practicing ophthalmologist in an unfamiliar position surgically, due to the unpredictability given the various mechanisms and degrees of presentations.19 Hence, having a rubric that allows for accurate grading of surgical management of open globe injuries is paramount in ensuring that residents develop critical sight-saving surgical skills. In the United States, graduating ophthalmology residents are required to complete only a minimum of 4 globe trauma cases as primary surgeon.13 However, these globe trauma cases may include anterior chamber washout and intraocular foreign body removal as well as open globe repair, which means that residents may meet their minimum numbers without necessarily gaining experience in globe repair. From ACGME surgical case logs, the average graduating resident in United States in 2018 performed 7.9 ± 5 open globe repairs as a primary surgeon.20 Further, 3.6% of residents did not meet the requirement, and 9.6% of residents met only the bare minimum of four ocular trauma procedures. Internationally, even more variation exists as many countries do not require a minimum number of surgeries, let alone define standards for competence. Further, simply logging the number of globe repairs performed is insufficient for assessing competence.21 Surgical logs do not demonstrate the quality of the procedure performed nor do they assure competency. Finally, surgical case logs are frequently miscoded making it difficult to assess what procedures were actually performed.22

A recent systematic review of the literature found that there is currently no universally accepted standard competency assessment for open globe repair.10 This assessment tool therefore fills a critical gap and serves three purposes: first, it will decrease subjectivity of the assessment OSCAR:OpenGlobe by defining for the assessor what behavior must be observed for each level of proficiency; second, the rubric clearly communicates to the learner what is expected to attain competence allowing self-directed learning; third, it provides a numerical system that allow for monitoring for improvement by the residency program director. Further, the grading sheet is simple and allows feedback in a timely fashion. Thus, this standardized rubric is a globally validated and standardized platform to allow non-biased objective teaching and evaluation of surgical skills in open globe repair.

In conclusion, the OSCAR:OpenGlobe is a key step towards standardizing training and evaluation in this domain that can be used globally. Its ease of use and the ability to generate assessment for each resident and from multiple evaluators allows it to be a meaningful and reliable tool. Moreover, by including an international panel of experts, we believe the rubric to have content and face validity.