An update on the novel approaches towards skills assessment of ophthalmology residents in the Indian scenario.

An essential part of the teaching-learning paradigm is assessment. It is one of the ways to achieve feedback for the various methods that have been used to impart a particular skill. This is true of ophthalmology training, where various clinical and surgical skills are learned as part of the residency program. In preparation for residents to become proficient ophthalmologists, both formative and summative assessments are of paramount importance. At present, assessment is primarily summative in the form of a university examination, including theory and practical examinations that are conducted at the end of the three years of residency. A formative assessment can make course corrections early on, allowing for an improved understanding of the subject and the acquisition of clinical and surgical skills. Formative assessments also allow us to customize the teaching methodology considering individual residents' learning capabilities. In addition, formative assessments have the advantage of alleviating the stress of a "final" examination, which could sometimes result in a less-than-optimum performance by the residents. The COVID-19 pandemic has forced us to adopt new teaching methods, which has led to the adoption of changes in assessment. In this regard, we discuss the different assessment tools available, their pros and cons, and how best these tools can be made applicable in the setting of an ophthalmology residency program.

Learning is incomplete without the process of assessment. Assessments are a useful tool in giving the teacher an insight into the depth of student learning. Formative assessments help the learner make course corrections during the learning process, while summative assessments aid the teacher in understanding the competency of the student at the end of the defined period within which a student should have acquired certain skills and competencies. Assessing residents during the training program is as important as the assessment done at the end of their period of residency in order to ensure adequate training. Assessments should be efficient in terms of time, cost, and ability to test residents.[1] Ophthalmology training is unique in that it requires the resident to develop both clinical and microsurgical competencies.[23] According to the Accreditation Council for Graduate Medical Education (ACGME) guidelines, residents need to acquire and be assessed on the following parameters: Medical knowledge, patient care, practice-based learning, interpersonal and communication skills, professionalism, and systems-based practice. In addition, the American Board of Ophthalmology mandates the assessment of surgical skills.[4] At present, only the International Council of Ophthalmology has an internationally developed mode of assessment of ophthalmologists in training.[567]

The Scenario of Assessments in Ophthalmology Residency in India

Under the National Medical Council, universities in India have suggested regular assessments in the form of annual examinations at the end of each year, including theory, practical, and viva voce. Nevertheless, there is no uniformity in the practice of the same and no emphasis on documentation of having conducted these examinations. Moreover, the thrust is on the final university examination, including theoretical assessments, practical assessments, and viva-voce. This is true of the National Board of Examinations as well.

In India, residency training is usually based on Halsted’s apprentice model. This model involves a discovery-based learning mode, where a resident attempts a procedure and “discovers” how the procedure is done.[8] However, with the present knowledge of the teaching-learning process and the fact that tolerance toward medical error has decreased, this mode has become less acceptable. Thus, a formal method of formative assessment is necessary.

Formative assessments, at present, are done in the form of journal presentations and clinical case presentations, which are graded. However, it is neither universally done nor does the university mandate it. Maintenance of logbooks to document the various clinical and laboratory procedures done during residency is encouraged. It also includes the case presentations, journal presentations, and seminars that the resident has presented.[9] A logbook only gives information about the resident’s experience and not of his/her expertise. In addition, when practiced, they do not carry any grades or any marks, and the onus finally lies in the final university exams conducted at the end of the residency. There is no standardized model of assessment across colleges/institutes and universities/boards. Therefore, the level of competency of students cannot be assessed and compared objectively.

This assessment strategy has been followed for many years now. However, the COVID-19 pandemic heralded a paradigm shift in our teaching and assessment methodologies. For the first time, the clinical assessment was modified to exclude patients out of the assessment area. Further, residents were assessed based on clinical scenarios. Finally, their clinical knowledge and judgment were assessed rather than based on the demonstration of skills. Thus, newer methods of assessments also need to be looked into and assimilated.

Competency-based medical education has been applied in undergraduate medical education in India since 2019. This method focuses on the development of competencies required to fulfill patients’ needs in a real-life situation. It emphasizes continued training of the student until the competency is achieved. This method assesses each student in an objective, measurable standard and is independent of the performance of other students.[10] This method can also be incorporated into the postgraduate ophthalmology curriculum.[11] In this method, assessments are done repetitively and in a criterion-referenced manner in the likeness of or actual clinical setting.[12] The guidelines for competency-based postgraduate training in ophthalmology only provide broad activities under which the resident has to be assessed but fails to provide the assessment methodology.[11]

Various assessment tools need to be incorporated in our rubric to optimize learning for our ophthalmology residents in the present scenario. This article discusses the tools available for assessment and elaborates the caveats and nuances.

The assessment tools will be discussed under the following subheadings:

Assessment of clinical skills

Assessment of surgical skills

Composite tools

Assessment of clinical skills

Clinical skills are the cornerstone of any medical or surgical specialty. It involves a conglomeration of communication and examination skills in view of history taking and clinical examination, respectively. In addition, organization and collation of information, arriving at a differential diagnosis, and forming a management plan are crucial. Assessment of these skills is essential. In the present scenario, this is being done during the university examinations in the form of a summative assessment. The following are the various tools described in assessing the clinical skills which can be employed easily for formative assessments.

Directly Observed Procedural Skills and Video-Observed Procedural Skills

This tool assesses the trainee’s ability to apply his knowledge and skills in performing a particular procedure and provides an immediate assessment of the skill performed. Sethi et al.[13] conducted a study on the utility of this method in teaching interns. The core areas focused on were visual acuity assessment, torchlight examination of the anterior segment (difficulty level: 1), direct ophthalmoscopy, and ocular movements (difficulty level: 2). It was seen that repeated use of the directly observed procedural skills (DOPS) method during the internship program improved the clinical skills of the stakeholders. This method can also be used to assess surgical skills as adopted and proved as an effective method by Hassanpour et al.[14] in their study on assessment of resident performed trabeculectomy. The Royal College of Ophthalmologists has a standardized DOPS assessment score for many clinical skills, which can be easily adapted to the residents’ program.[15] An example of a clinical rating scale used for IOP evaluation is shown in Supplement 1. There are similar scales for various clinical skills that can be modified to suit Indian clinical scenarios and used. The templates of these scores can be accessed on the website Resources - The Royal College of Ophthalmologists (rcophth.ac.UK). The DOPS method requires a significant amount of time investment, and residents being aware of being observed may affect their performance. A similar tool is video-observed procedural skills (VOPS), wherein instead of direct observation, the procedure done by the resident is videotaped and then assessed by the faculty. In the assessment of surgical skills, it was shown that VOPS is a feasible and valid assessment method and had a good correlation when compared to DOPS grades.[14]

Ophthalmology Clinical Evaluation Exercise

This tool was designed by the International Council of Ophthalmology and has been implemented in several languages. The resident is assessed on 33 parameters during the process of history taking, examination, and clinical case presentation.[16] The residents are graded as below expectations, meets some expectations, meets all expectations, or exceeds expectations. The advantage of this tool is that it has been proven reliable and valid and has the advantages of both clinical evaluation exercise (CEX) in being comprehensive and of a mini CEX in reviewing real-time situations and less time-consuming and providing immediate feedback to the residents. The disadvantage is that it has not been internationally developed. Therefore, cultural differences have not been factored in.

Palis et al.[16] developed a modified version of the ophthalmology CEX (OCEX). A modified 3-point Dreyfus scale was used in this rubric, which included novice, beginner, and competent stages. The aspects that were assessed were interview skills, examination, interpersonal and communication skills, and case presentation. The parameters that were tested are shown in Table 1.

Table 1

The parameters assessed in the modified OCEX[16]

Interview skills	Clinical examination	Interpersonal and communication skills	Case presentation
Introduction	Hand/instrument hygiene	Approach towards patient comfort	Brevity
Presenting complaint	Assessment of visual acuity	Empathy	Organization of facts and findings
History of presenting illness	Pupil examination	Respectfulness	Differential diagnoses
Drug history	Visual field examination	Explanation of findings, diagnosis, and treatment	Plan of care
Family history	Examination of ocular motility	Ability to respond to patient queries	Responses to examiner
Past History	Slit-lamp examination
Pertinent negatives	Examination of retina

An essential aspect of the modified OCEX is the addition of pertinent negative history as negative history can be a valuable model of arriving at the diagnosis. This mini-CEX was also found to be valid and reliable.[16]

Pediatric Examination Assessment Rubric toolkit

Pediatric ophthalmologic examination requires proficiency in many skills. To provide a means to assess the complex set of skills, Langue et al.[17] developed a comprehensive rubric called the pediatric examination assessment rubric (PEAR) toolkit. In this rubric, 12 examination skills pertinent to pediatric ophthalmological examination were assessed using videographic recordings. The clinical encounters included visual acuity examination, anterior segment examination, intraocular pressure measurement, retinoscopy, fundus examination, strabismus evaluation, and measurement of stereoacuity. In addition to the aforementioned parameters, the resident was assessed based on their rapport with the patient and his/her family. This tool was found to have minimal inter-rater variability and fair reliability. Though this tool was designed to assess pediatric ophthalmological examination clinical skills, the rubric can be used to design assessment tools for other subspecialties with some modifications.

Clinical assessment scores can be developed for each clinical skill and residents assessed accordingly. In addition, periodic reviews will help residents hone their skills by establishing a feedback system that will help residents correct their mistakes early in residency and learn skills optimally.

Assessment of surgical skills

Surgical skills are not very rigorously or structurally assessed in the existing assessment modules in the Indian arena of postgraduate ophthalmology training. A standardized toolbox needs to be assimilated into our present system of ophthalmology residency for an objective and unbiased assessment.[18] In addition, attention needs to be paid to changing scenarios such as the COVID-19 pandemic, where innovative methods of assessment need incorporation.

Automated tools have also arrived at the assessment scene in ophthalmology residency, providing the advantage of being repeatable, reliable, and devoid of human bias. Furthermore, with the COVID-19 pandemic making social distancing an imperative, these techniques ensure the safety of the assessors, assessee, and patients.

The following are some of the tools available to assess surgical skills objectively.

Objective assessment of skills in intraocular surgery

The objective assessment of skills in intraocular surgery (OASIS) scoring was developed at the Harvard Medical School to assess residents’ competency in phacoemulsification.[19] It included three aspects: preoperative, intraoperative, and postoperative. The intraoperative aspect was further divided into the following thrust areas: phacoemulsification technique used, total phacoemulsification time, amount of irrigation fluid used, the resident’s surgical time, total time in the operating room, location of the incision, use of limbal relaxing incisions, type of blade, and instruments used. The OASIS database allows for evaluating postoperative astigmatism, rates of complications in individual residents, and the various cohorts of patients that were operated upon by the residents, such as pseudoexfoliation. This assessment tool is purely objective and hence has no scope for inter-rater variability.[20] It is a one-page standardized form that is less time-consuming and has no financial constraints on the residents or clinicians, thus making it an effective and affordable tool.[21]

Global Rating Assessment of Skills in Intraocular Surgery

The Surgical Education Research Group, University of Toronto, developed a more comprehensive tool named global rating assessment of skills in intraocular surgery (GRASIS) that included the objective and subjective aspects of surgical skills training. GRASIS includes the objective parameters of the OASIS tool, and in conjunction with it, has a one-page subjective assessment. The assessed parameters are the manner of treatment of intraocular structures, time, motion, and energy applied on the intraocular structures, eye position and microscope use, instrument handling, and use of the non-dominant hand. Further, the resident is also assessed on knowledge of equipment used for phacoemulsification and vitrectomy, operation flow, and specific procedures. In addition to this, the residents’ interaction with the scrub nurse and handling of unexpected events are assessed. Based on this, an overall score is given. This subjective assessment pays attention to the resident’s surgical knowledge, surgical preparedness, and interpersonal skills.[22]

Objective Structured Assessment of Cataract Surgical Skill

Saleh et al.[23] described a tool named objective structured assessment of cataract surgical skill (OSACSS) that focused on both global and phacoemulsification-specific competencies. Surgical videos that were taped when the residents performed cataract surgery were assessed based on 14 cataract-specific stems and six global indicators. In the study that led up to the defining of OSACSS, it was found that when residents performed 250 or more surgeries, the tool was not able to identify differences in competencies. However, in the group of residents that had performed less than 250 surgeries, the competencies were much better in those who had performed 50 or more surgeries. It is a useful tool during the early days of residency training.

Scoring of the residents’ performance could be done by the faculty and/or the trainees. However, Casswell et al.[24] found that the senior trainees’ self-assessment correlated better with faculty assessment than the junior trainees’ self-assessment.

Imperial College Surgical Assessment Device

The tool, Imperial College surgical assessment device (ICSAD), uses a motion-sensing device to assess a resident’s suturing technique on a model eye by using an operating microscope with standardized instruments. A single passive receiver is attached to the index finger of the resident, and the parameters, namely total path length, time, and the number of individual hand movements, are analyzed. In addition, a video is captured, and two independent observers assess the parameters. This tool correlates with the objective structured assessment of technical skills (OSATS) tool in assessing the suturing competency.[25]

International Council of Ophthalmology- Ophthalmology Surgical Competency Assessment Rubric

The disadvantage of the previously mentioned tools is that they have been developed locally, keeping the relevant culture in mind. An internationally developed tool transcends borders and allows for easy adaptation. In addition, these tools pertain primarily to cataract surgery. Tools specific to other ophthalmic surgeries are essential to assess holistic learning during the ophthalmology resident program. With this in mind, the International Council of Ophthalmology- ophthalmology surgical competency assessment rubric (ICO-OSCAR) tool was developed. The OSCAR rubrics have been developed for various surgeries, such as extracapsular cataract extraction, lateral tarsal strip surgery, pediatric cataract surgery, phacoemulsification, ptosis, small-incision cataract surgery (SICS), strabismus, trabeculectomy, and vitrectomy. There are also tools available for procedures such as panretinal photocoagulation.

The ICO-OSCAR is a standardized, internationally valid tool for the educator (and the resident) to evaluate competence in performing a specific procedure objectively.[24] In this rubric, the surgical procedure is broken down into its individual steps, and the proficiency is graded based on the 4-point Dreyfus scale, viz., novice, beginner, advanced beginner, and competent. Each step is described in the tool, and the preceptor has to circle the observed performance description given. This has to be done immediately after the learner performs the procedure in order to be able to give timely, structured, and specific feedback, thus enhancing the quality of the learning process. At the end of this assessment, an improvement plan has to be made so that the learner improves upon the deficiencies that were seen during the surgical procedure. The tool has been translated into various languages, such as Mandarin Chinese, French, Portuguese, Russian, Spanish, Thai, and Vietnamese, for use in the countries where the above are known. These tools are available online and in the form of an ICO-OSCAR application.[26] Table 2 shows the various rubrics available in the ICO-OSCAR tool.

Table 2

Tools available in the ICO-OSCAR spectrum[26]

Cataract

ICO-OSCAR ECCE

ICO-OSCAR SICS

ICO-OSCAR Phaco

Orbit and Oculoplasty

External Dacryocystorhinostomy

Anterior approach Ptosis surgery-ICO-OSCAR-Ptosis

Lateral tarsal strip surgery-ICO-OSCAR-LTS

Pediatric Ophthalmology and Strabismus

Strabismus surgeries-ICO-OSCAR-Strabismus

Pediatric Cataract-ICO-OSCAR-Pediatric Cataract Surgery

Retina

Vitrectomy

Pan Retinal Photocoagulation

Ocular surface

Pterygium

Self-assessment and peer assessment

Cheon et al.[27] described the use of ICO-OSCAR by residents for peer and self-assessment. In their study, it was found that peer assessment was as efficient as assessment by teachers, while self-assessment was not as consistent. Thus, peer assessment can be an addendum to the armamentarium of assessment tools. This was corroborated by a study done by Srikumaran et al.,[28] where it was found that self-assessment was an inaccurate representation of the trainee’s proficiency.

Assessment scale of corneal rupture suturing

Zhang et al.[29] described this scale to assess residents’ proficiency in performing suturing in eyes with a corneal rupture. Porcine eyes were used, and the residents were required to suture an L-shaped corneal tear under an operating microscope. This process was videotaped and assessed by the faculty. This comprehensive assessment involved the following aspects: preoperative preparation, microscope use, instrument handling, hand-eye coordination, suturing technique, wound closure, and postoperative clean-up. This tool was found to be reliable and repeatable. However, this tool does not ascribe to real-world situations and cannot describe the resident’s judgment.

Eye surgical skills assessment test

The eye surgical skills assessment test (ESSAT) was developed by Fisher et al.[30] to assess students’ proficiency before they enter the operating room. Their skillsets are tested in a wet laboratory mode. There are three stations, which include skin suturing, muscle recession, phacoemulsification/wound construction, and assessment of the suturing technique. The resident’s performance is videotaped, and the faculty assesses the residents’ performance based on a station-specific checklist and a global rating scale of performance. Instead of this, an assessment tool involving the Eyesi® simulator may also be considered, as enunciated by Le et al.[31]

Eyesi® simulator as an assessment tool

This simulated assessment model correlates well with the real-life metrics and can thus work as an effective tool to assess surgical competencies. Eyesi® assessment scores correlated well with real-life cataract surgery assessment scores.[32] However, as the motion tracking rubric can have inter-individual variations, it is wise to use this assessment tool along with the other tools to gain a better picture of the acquisition of competencies by the resident.[33]

These tools, which objectively assess the surgical skills, can be modified to suit the Indian ophthalmology surgical scenario. Most universities provide guidelines regarding the number of surgeries and type of surgeries that each resident has to be proficient in by the end of his/her residency. Along with this recommendation, if tools to assess surgical skills are mandated, then the quality of residents will also become comparable across different universities and colleges. This would help in the standardization of the residency program across India.

Composite tools

These tools fall in line to some extent with the ACGME guidelines and test the resident’s interpersonal skills, communication skills, professionalism, and system-based practice.

ICO-360-degree evaluation

This is a comprehensive evaluation of a resident’s all-round performance in the ophthalmology setup. The assessment is done by peers, coworkers, patients, and faculty. The various parameters that are tested are professionalism, interpersonal and communication skills, and system-based practice.[26]

National curriculum for ophthalmology residency training

Developed by Grover et al.[34] under the aegis of the All India Ophthalmological Society, the curriculum gives guidelines for the assessment of ophthalmology residents as well. Formative and summative assessments form integral parts of the assessment prescript. The proposition is that formative assessments would include assessing personal attributes, clinical skills and performance, academic activities, and practical assessments after each clinical posting viz., the subspecialties such as orbit and oculoplasty, cornea, retina, pediatric ophthalmology and strabismus, and glaucoma.

A summative assessment would comprise theory examinations that are to be conducted at the end of 1 year, 2 years, and 2 years and 9 months. In addition, summative assessments would include the following:

Logbook

Theory examinations divided into four papers for ease of assessment

Practical examinations which comprise

Clinicals- One long case, two short cases, two fundus cases, one refraction case, and one OCEX case

Viva Voce- Instruments, pathology, microbiology specimens, drugs, imaging modalities, visual fields, and other ophthalmic diagnostic charts.

On-call assessment tool

This tool was designed by Golnik et al.,[35] where a retrospective chart audit was done of the residents on call charts. This was assessed with a tool that comprised testing of timeliness of consultation, history, examination, assessment and plan, and urgency rating. The residents’ performance was assessed as satisfactory, borderline, and unsatisfactory.

Tool to assess integrated clinical communications skill

As much as assessing clinical and surgical skills is essential, assessing a student’s communication skills is equally important. It has been said that more than a patient needing to know how much a doctor knows, it is vital that the patient knows how much a doctor cares. In this regard, various tools have been tested but more so in the space of undergraduate medical education. A case in point is the tool devised by Brouwers et al.[36] that was used among undergraduate medical students but can be applied to ophthalmology residents. Students were taught the various aspects of communications based on the biopsychosocial model during their third year in the undergraduate medical course. At the end of the course, an objective structured clinical examination (OSCE) was conducted, including two stations dedicated to communication skills. Various aspects were assessed, including verbal and non-verbal communication.

The National Medical Council has developed a module known as the Attitude, Ethics, and Communication (AETCOM) module for undergraduate students.[37] In this module, the student’s active participation in planned focused group discussions, small group discussions, and skill lab sessions are assessed by a trained evaluator and forms part of the formative assessment. Summative assessment is conducted in the form of theory questions on attitudes, ethics, and communication in the year-end examinations. Modules that pertain to ophthalmology can be formulated, with formative and summative assessments that test the resident’s competency in the above parameters.

A peek into the future

Eye movements and surgical proficiency

Brouwers et al.[36] conducted a study that involved residents performing simulated surgical tasks while their eye movements were recorded. It was seen that eye movement data can be used to ascertain whether the resident had beginner and intermediate proficiency in microsurgical skills. Though this study has not been done specifically in the ophthalmology setting, it provides an innovative approach to assessing surgical skills. Further studies would be required to apply this model in the ophthalmic microsurgical setting.

Wireless sensor glove for surgical skills assessment

This is a unique approach to assess surgical skills. The study was done to assess skills in laparoscopic surgeons and requires modification in the rubric for ophthalmological microsurgeries. A glove was designed that could transfer via a wireless mode the data collected from the sensors to a base station fitted on a computer or laptop. Hand gestures that are used while performing the task were compared between novice and expert surgeons. Exploring this tool for the assessment of ophthalmic surgeons would be very innovative and helpful.[38]

Machine learning and deep learning

With artificial intelligence becoming ubiquitous in its applicability, an ophthalmology skill assessment is no stranger to its possibilities. In a study done by Yu et al.,[39] ten phases of resident and faculty performed cataract surgeries were assessed by videotaping them. Convoluted neural networks (CNNs) and recurrent neural networks (RNNs) were used to assess the parameters, including side port incision, main incision, capsulorhexis, hydrodissection, phacoemulsification, cortical removal, lens insertion, ophthalmic viscosurgical device removal, and wound closure. The steps were noted for the number of attempts made and any failed steps. Various algorithms were tested and compared. It was found that model instrument labels and video images were the best way to assess the various steps. Nonetheless, further research is required in this direction to find and refine such automated testing tools in the setting of ophthalmology residency.

Caveats

Extensive studies have been done on tools that involve assessing cataract surgery, while tools that assess residents’ performance of other ophthalmic surgeries are not well researched. Currently, the focus of research is on surgical skills; clinical skills assessment requires further scrutiny. These scoring tools are considered to be time and cost-intensive. We need to adopt tools that are effective and easy to implement in the Indian scenario, taking into account the surgical skills and the clinical skill assessment. These assessments should be objective, efficient in terms of time and reliability.

Conclusion

Assessment is an important aspect of training as it is one of the tools that give feedback to the learner and helps the teacher modify the training process. Summative assessments aid in understanding the proficiency of the resident at the end of the period of residency, while formative assessment provides us with an opportunity to change the teaching method considering each student’s progress. There are a variety of tools that assess the diverse skills that a resident is expected to acquire during his/her residency. These tools need incorporation into the present system of residency training in India. Both clinical and surgical skills require regular assessment in order to enhance the learning process of the resident. With the COVID-19 pandemic at the fore, novel approaches for skills assessment need to be imbibed into the present system to allow for safer modes of assessment while maintaining objectivity and ease of assessment.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.