International Council of Ophthalmology-Small Incision Cataract Surgery rubric: A roadmap to evaluate cataract surgical skill acquisition during residency training.

PURPOSE: The International Council of Ophthalmology-Small Incision Cataract Surgery (ICO-SICS) rubric is a tool to grade SICS steps from novice to competent. The study aimed to evaluate the progress of residents' surgical skills by using the ICO-SICS rubric and the perceptions of residents and faculties about its use.
METHODS: This prospective educational interventional study, done in the Ophthalmology department between September 2019 and February-2020 included 14 residents and five faculties. Faculties scored residents' SICS by ICO-SICS-rubric (four sessions/resident) and helped them identify three steps as "area of focus" to work upon. Feedback was taken using a semi-structured pretested questionnaire. Focus group discussion was done for residents. Data were entered in MS Excel and analyzed using SPSS. Perception analysis presented as percentage of written responses.
RESULTS: Step-wise rubric score showed improvement for initial SICS steps and wound closure (P < 0.05). Critical surgical steps and scores for three areas of focus steps showed no statistically significant improvement. Three steps as an area of focus changed partly for 11 residents and completely for three residents at the end of 6 months. Perception analysis of faculty and residents showed that the ICO-SICS rubric is a good tool to record surgical performance, identify steps needing improvement and provide structured feedback hence opined to continue it. Residents considered it as an effective learning and assessment tool.
CONCLUSION: ICO-SICS rubric is a good teaching tool and helps to assess the progress of surgical skills. Identification of areas of poor performance and feedback given motivates them to focus on those areas leading to continuous professional development, resulting in competent surgeons performing SICS surgery independently at the end of the residency.

The competency-based medical education (CBME) curriculum for postgraduation in ophthalmology lists various clinical/surgical skills to be acquired during residency training.[1] CBME brought a paradigm shift to move from “assessment of learning” (summative) to “assessment for learning” (formative). However, surgical skill assessment is a critical issue for both trainees and trainers. Several tools have been devised for the evaluation of surgical skills.[234] Ophthalmology Surgical Competency Assessment Rubrics (ICO-OSCAR) was designed by the International Council of Ophthalmology to facilitate the assessment and teaching of various surgical skills.[5] This Dreyfus-inspired model of skill acquisition is a valid and reliable assessment tool.[6]

A resident doctor should be confident in performing small-incision cataract surgery (SICS) independently at the end of residency training. Multiple formative assessments of SICS techniques are needed to ensure achievement of the desired level of competency. Conventionally, we audit surgical cases post-operatively by assessing the overall outcome but not the step-wise level of surgical performance. This can be easily done by ICO-OSCAR: SICS-rubric. Thus, we planned this study to introduce this rubric-based evaluation to assess SICS surgical skills of resident doctors and evaluate the perceptions of residents and faculties about the use of the rubric. The outcome of the study can be used for future modifications to improve the current post-graduate program in Ophthalmology as per the CBME curriculum.

Methods

After approval from the institutional ethical committee, the study was conducted in the Department of Ophthalmology at Medical College of South Gujarat. All post-graduate students in 2nd and 3rd-year residency who consented for participation were included in the study. The duration of the study was 6 months from September 2019 to February 2020.

Fourteen residents and five faculties were sensitized about the ICO-SICS rubric. The students were informed that the performance of any surgery done by them will be scored according to the rubric at various intervals so that they are not apprehensive. They were informed that participation would be voluntary and the score will not be added to their final assessment. All students gave consent for participation.

Before commencing the study, we did a pilot run and scored two surgeries done by senior resident doctors. As per the suggestion of faculty members, rubrics for basic steps such as painting and bridle suture (not in the ICO system) were designed, validated by subject experts and incorporated in the ICO-SICS rubric scoring sheet.

As per the departmental protocol, all surgeries done by residents are under faculty guidance. The faculty, who supervised the surgery, scored the surgical steps according to ICO-OSCAR: SICS-rubric on the same day of surgery or subsequent day during post-operative rounds. Each surgical step is rated either as novice, beginner, advanced beginner, or competent, with the respective scores being two, three, four, and five. The step was scored as zero when done by the preceptor. The faculty gave individual feedback to students and discussed the surgical areas that required improvement. The students were asked to decide three surgical steps as areas of focus to work upon. This was documented in the “comment section” at the end of the ICO-SICS rubric score sheet.

During the training period, the students were motivated to practice in wet labs, view surgical videos by experts and read the surgical manuals with particular focus on pre-decided areas. The faulty techniques and tips to overcome them were discussed with the students.

During the 6 months period, four such rubric scoring sessions were done for each student. At each scoring, trainees were asked whether they wanted to focus upon the same area or wanted to change the area of focus. Their response was documented.

We took feedback from resident doctors and faculties through a prevalidated structured questionnaire with open and closed-ended questions to know their perceptions about the use of the rubric. Residents’ feedback form had 10 close-ended items based on the five-point Likert scale and two open-ended items. The faculty feedback form had nine close-ended items (eight were based on the five-point Likert scale and one multiple choice question {MCQ}) and three open-ended items. Open-ended questions for both students and faculty were regarding suggestions about continuation, frequency, person doing rubric scoring along with reasoning and suggestions for any modifications in the scoring process.

Also, a focus group discussion (FGD) was done with eight resident doctors and recorded digitally with prior consent.

The data were entered in MS Excel version 10 and analyzed using SPSS software. ANOVA test was applied for testing statistical significance. P < 0.05 was considered statistically significant. Post HOC test was used to assess between which sessions statistical significance was found. Perception analysis was presented as the percentage of written responses. Transcripts prepared from a digital recording of FGD were explored using code generation and thematic analysis was done.

Results

In total, 14 residents and five faculties from the Department of Ophthalmology participated in the study. Six second-year residents and eight third-year residents participated in the study. All were female residents except one. In total, four scoring sessions by ICO-SICS rubric were done for each resident; thus, 56 scoring sessions were done altogether. Feedback forms were anonymously filled by all participants. The results will be discussed in quantitative and qualitative aspects separately.

Quantitative analysis

Observations for stepwise scoring

Prior to intervention, at least four out of six second-year residents were in the category of beginner to advanced-beginner for basic steps such as patient preparation (painting and draping), preliminary steps (bridal suture and cauterization) and other steps (e.g., paracentesis, IOL insertion, wound closure and iris protection under global indices rating). All other steps were in the category of novice to beginner.

At the end of 6 months, they were novice to beginner for the surgical step of tunnel construction, entry, capsulotomy, nucleus management, hydro procedures, irrigation aspiration (IA) and steps of global indices rating except for the step of “iris protection.” The scoring for basic steps (painting, draping, bridal suture and cauterization) and a few steps such as paracentesis, IOL insertion, wound closure, and iris protection were in the category of beginner to advanced-beginner. Thus, the areas where the majority (at least four out of six) were at the same level were IOL insertion, wound closure, nucleus management, IA and overall surgical time. All other steps showed improvement by a minimum of one level.

At least five out of eight third-year residents prior to intervention were beginner to advanced beginner for steps of entry, capsulotomy, nucleus extraction, and global indices rating (except iris protection). They were advanced beginner to competent in basic steps (painting, draping, bridal suture, and cauterization), tunnel construction, paracentesis, IOL insertion and iris protection. The scoring was variable for steps of hydro procedures, nucleus prolapse, IA, suturing and tissue handling.

The third-year residents showed improvement by a minimum of one level for all steps except tunnel construction, IOL insertion, wound closure and overall timing of surgery, where the majority of them had the same level of performance. At end of 6 months, they were competent for steps of painting-draping, bridal suture, cauterization; advanced-beginner to competent for steps of tunnel construction, entry and paracentesis, capsulotomy, hydro procedures, nucleus management, IA, IOL insertion, suturing and global indices rating except the timing of surgery where they were in the category of beginner to advance-beginner.

On applying the ANOVA test, the steps showing significant improvement (P < 0.05) were patient preparation (painting and draping), preliminary steps of cataract surgery (bridal suture and cauterization) and wound closure. The critical surgical steps such as wound construction (tunnel, paracentesis, and entry), capsulotomy, hydro procedures, nucleus management, cortex wash, and IOL insertion as well as global indices showed no statistically significant improvement. The post HOC analysis was statistically significant on comparing the first session with subsequent sessions (mostly the fourth session) for steps of patient preparation, preliminary steps, sclerocorneal tunnel, corneal entry, hydro dissection and wound closure [Table 1].

Table 1

Scores of surgical steps for all participants which showed statistically significant improvement (n=14)

	Surgical step	Intervention				ANOVA test		Post Hoc test
		1st	2nd	3rd	4th	F	P
1	Patient Preparation  i. Painting  ii. Draping	120	137	140	140	12.33	0.000	Donnet T3 significant (P<0.05) between session 1 and 2, 3, 4
2	Preliminary Steps  i. Bridle suture  ii. Scleral access & Cauterization	113	136	133	135	8.746	0.000	LSD significant (P<0.05) between session 1 and 2, 3, 4
3	Sclerocorneal Tunnel	50	55	58	60	2.012	0.124	LSD significant (P<0.05) between session 1 and 4
4	Corneal entry	43	53	54	56	2.552	0.066	LSD significant (P<0.05) between session 1 and 3,4
5	Hydro dissection: Visible Fluid Wave and Free prolapse of one pole of nucleus	45	47	45	53	0.736	0.535	LSD 1 and 3 significant (P<0.5)
6	Wound Closure (Including Suturing, Hydration, and Checking Security as Required)	50	67	57	61	5.923	0.001	DunnettT3 significant

Score of each surgical step: - Min 2=Novice & Max 5=Competent. Steps 1, 2, are summation of two related steps as per ICO: SICS rubric. Score of surgical step 1, 2: - Min 4=Novice & Max 10=Competent. Total 14 residents. So, score for each step: Min is 4×14=56 (Novice); Max. is 10×14=140 (Competent). Score of surgical step 3, 4, 5, and 6: - Min 2=Novice & Max 5=Competent. Total 14 residents. So, score for each step: Min is 2×14=28 (Novice); Max is 5×14=70 (Competent)

Observations for three steps selected as areas of focus

The steps selected as areas of focus during the feedback session as a part of rubric scoring were mainly corneal entry, IA and tunnel construction at the first scoring session, whereas at the time of fourth scoring, the steps were nucleus delivery, IOL insertion, and corneal entry. The scores for steps as areas of focus showed some improvement in 11 out of 14 students. The average score improved by 2.84 and 1.75 marks between the 1st and 4th sessions for second-year and third-year, respectively, but it was not statistically significant [Table 2].

Table 2

Scores for areas of focus of individual participants for each surgery

Participant No	Received score for area of interest (Max Score=15, Min total=6)*
	2nd year				3rd year
	1st	2nd	3rd	4th	1st	2nd	3rd	4th
1	7	7	7	6	10	12	7	7
2	6	10	12	12	10	12	11	13
3	4	12	12	12	11	9	11	13
4	7	8	8	9	6	12	7	12
5	10	13	9	11	13	12	8	13
6	7	8	7	8	10	12	13	13
7					10	12	12	12
8					12	15	13	13
Mean±2SD	6.83±3.88	9.67±4.84	9.17±4.63	9.67±4.84	10.25±4.11	12±3.21	10.25±5.10	12±4.14
ANOVA test	F: 2.740 Sig: 0.07				F: 2.434 Sig: 0.086
Post hoc Analysis	Dunnet T3 significant between 1st and 2nd sessions				LSD significant between the 1st and 4th sessions

Max total scores for areas of interest is 5×3=15 (Novice=2 & Competent=5). Min total scores for areas of interest is 2×3=6

The three steps as areas of focus changed partly for 11 residents and completely for three residents at end of the 4th scoring session. Three residents had the same three areas of focus during the second scoring; two residents during 3rd scoring but none had the same steps as areas of focus at the end of 4th scoring.

Qualitative Analysis

Feedback questionnaire analysis

Feedback by residents

Twelve trainees were maintaining records of surgeries performed and surgical time and received feedback from faculty for all surgery. All except one mentioned the need for a structured scoring system to know their level of performance in surgery, and 11 residents agreed to the need for the incorporation of rubric scoring in the logbook as a part of formative assessment. All trainees believed that the ICO rubric is a good tool to record surgical performance in sequential steps and helps them to identify steps where improvement is needed. Structured feedback given at end of each scoring helped them to achieve better performance. Two-third of residents strongly agreed that rubric scoring will help to customize the training of residents according to the level of performance and not cadre. Six residents felt that the knowledge of their surgery being scored made them apprehensive [Fig. 1].

Figure 1

Distribution of Likert scale scores of residents for ICO-SICS rubric feedback questionnaire items (N = 14)

All residents believed that rubric scoring should continue. Half of them felt that the scoring should be done monthly and 70% believed that rubric should be scored by the faculty as they can better judge the mistakes of the resident.

Feedback by faculties

Four faculties believed that it is important to audit the surgical steps of residents to improve their surgical skills, but three disagreed with the comment that they always grade the surgical steps of residents. Four faculties mentioned the need for a structured scoring system to grade the surgical skills of residents. They perceived the ICO rubric as a good tool to record surgical performance in sequential steps, provide structured feedback to students, help them to identify steps where improvement is needed, and customize training of residents according to the level of performance and not cadre. Three opined to incorporate rubric scoring in the logbook as a part of formative assessment, and four agreed to continue the use of rubric in the department [Fig. 2].

Figure 2

Distribution of Likert scale scores of faculties for ICO-SICS rubric feedback questionnaire items (N = 5)

There was no common consensus among the faculties about the frequency of use of rubric, time required for scoring, and person scoring it. Some felt that the faculty themselves were the better person to score and guide surgeries. Those who felt that the resident himself should do scoring justified that it will result in better self-assessment. One faculty and resident recommended senior resident should do scoring as faculty may not be around due to time constraints.

Suggestions for the rubric-based assessment (faculty and resident)

One faculty and four residents believed that scoring should be done by the same faculty, while two residents suggested that scoring should be done on the same day too. Two faculty and residents were of the opinion to incorporate difficulty level of cataract-related conditions. One faculty opined that rubric cannot be used as an assessment tool for overall surgery but to decide competency for various steps.

Focus group discussion analysis

The major themes after FGD Analysis were “learning tool for cataract surgical steps,” “beneficial for assessing surgical skill,” “useful tool for maintaining a record of steps performed,” “scoring is valid,” and “need for continuation of rubric.” Most trainees suggested that rubric scoring should be implemented for other surgeries and routine clinical procedures too. They also raised concern that the “difficulty grade” of cataract also affects performance and scoring. Thus, it should be taken into consideration and documented in the scoring sheet. The issue of apprehension out of the knowledge that surgery might be scored was discussed and the fact was denied by all except one [Table 3].

Table 3

The major themes generated by thematic analysis with verbatim quotes

Learning tool for cataract surgical steps

“It helped to learn each step individually”

“It helped us to develop insight of every step”

“Grading from novice to expert level”

“Before this, we did a surgery and that was it… It (rubric) helped us to retrospectively see and analyze each step critically and what improvements need to be done ”

Beneficial for assessing surgical skill

“It gives steps that we need to improve to achieve next level”

“Each and every cataract surgery we came to know have we improved or not”

“It was very helpful me to know how much improvement after each surgery, whether we are achieving progressive improvement or not”

“It also helped to grade surgical steps chronologically. As our number of surgery increased whether our major surgical steps improved”

Useful tool for maintaining surgical record of steps performed

“Documentation of each step which showed last time how we had performed”

“Excellent tool, grading of each step, even basic step like using commercial drape and drapes available in hospital”

Rubric scoring is valid

“Score was justified”

“It was not biased”

Need for continuation of rubric

“It should be continued as it is in benefit of resident”

“it should be done often”

“it can be used for pterygium surgery and for procedures like refraction with checker points used for scoring”

“Such type of system can be used for all procedures and other surgery”

“It would be better to include the level of difficulty in surgery allotted to the resident”

Discussion

The inadequacy of the current residency program for ophthalmology and the need to improve it had been urged long back.[7] Moreover, the implementation of CBME in India for post-graduate training has set the notion that every assessment should be objective for it to be of any value. Rubrics are one such learner-centric assessment tools that not only augment performance while enriching education through experience but also provide a constant review of results achieved in relation to the outcome desired.[8]

The assessment of surgical skills of residents by “rubric” referring to “performance standard” as done in this study, is necessary to know the actual progress of residents for their surgical skills which will help to improve the training program. Perception of the ICO-SICS rubric by residents and faculty to the best of our knowledge is done for the first time.

The stepwise scoring of SICS steps specifically graded the level of performance of students for the steps performed. The conventional “case audit” generally states whether the step “performed” or “not performed” and lacks the “criterion-based performance feedback.” This can be a frustrating learning experience, especially for sensitive learners. The identification of steps where students are “novice” or “stagnant” and not showing improvement helps the student to focus on those steps, especially while practicing in wet labs. It also helps the faculty to assist learning by conducting video-based demonstrations and discussions focused on those steps. Thus, the rubric helps to guide the training in the right direction.

The critical surgical steps (e.g., wound construction, capsulotomy, nucleus management, and hydro procedures) did not show significant improvement in our study. This is likely because acquisition of competency for such skilled steps is time bound and the current study was of short duration. A similar study based on a rubric for phacoemulsification training in a wet lab reported that such type of tool leads to a continuous professional development plan, eventually improving the surgical skills of trainees.[9]

Though the area of focus-related score did not show significant improvement, the three steps selected as areas of focus changed partly or completely for all the participants at the end of 6 months. This reflects the fact that improvement was perceived by residents in predefined areas as they now wanted to focus on a different step. Helping the learner to identify some steps as areas of focus through rubric-based assessment and motivating him to focus on a few crucial steps at a time shall drive him to achieve the short-term goals. Such workplace-based assessment (WPBA) and targeted learning approach for complex skills such as SICS make the learning experience joyous rather than intimidating. In a slightly different study setting of assessment by Direct Observation of Procedural Skills (DOPS) in the general surgery branch, the residents reflected that WPBA helped in improving their surgical skills and to be proficient in basic procedures.[10]

The overall perception of most residents and faculty about the ICO-SICS rubric being a good tool in terms of recording surgical performance and identifying areas of improvement suggests that such tools make training more focused and outcome-oriented. Students perceived this assessment tool as an opportunity for student–teacher interactions and getting structured feedback on a one-to-one basis. Studies have reported feedback as the most valuable feature of WPBA.[10] Declarative sentences such as “very good,” “well performed,” or “you messed up” do not suffice to promote learning. Studies have established that narrative, descriptive, and complete feedback is more appreciated by learners.[11]

Residents and faculty agreed that rubric scoring will help to customize the training of residents according to the level of performance and not cadre. This reflects their belief that it would be unfair to allot surgery to a first or second-year resident having better surgical proficiency, at par with his peers, who are still novice to beginner. Customized training helps residents with good surgical skills to achieve “expertise” (Dreyfus 5th model) during the course of training. It also motivates residents with a slow learning curve to work upon identified areas without affecting their self-confidence or compromising patient’s outcome.

The positive response from residents and faculty in questionnaire-based feedback to continue the use of rubric reflects that it was overall perceived as a useful tool for surgical training. During FGD, students elaborated on the utility of rubric scoring as a learning and assessment tool. The varied response from participants about the frequency and person scoring the rubric requires some policy-making to make it more feasible and effective. Time constraints have been identified as a major factor preventing trainees from achieving the intended benefit.[121314] To overcome this issue on part of the faculty, training senior residents to score the surgical steps is a practical and sensible alternative. In a setup with the facility of video recording, residents can record and self-assess their surgery using the OSCAR rubric and then reflect on what they need to improve.[15] Such reflective learning along with the trainers feedback is profoundly effective and can also help to overcome the issue of time constraints.

Suggestions by some residents for the same faculty to score may be out of logical reasoning that improvement in subsequent surgeries can better be judged if scoring is done by the same faculty. Also, a suggestion about scoring on the same day is valid due to the quality of feedback when inadvertently delayed. A study on simulation-based skill acquisition also reported that early feedback was associated with higher performance scores and is more beneficial to novice learners.[16]

Limitations

The study was of short duration and we could not continue further due to the COVID-19 pandemic. Long-term study with enrolment of more participants is required to understand the timeline in achieving competence for critical surgical steps.

Conclusion

ICO-SICS rubric is a useful tool for formative assessment of surgical skills of our residents. The checklist-based system helps trainees to identify the level of performance for each surgical step and helps trainers to facilitate learning in the identified area. Such a focused approach helps residents to be competent and confident surgeons independently performing SICS, an important skill to be acquired at the end of the training program. Implementing rubric-based periodic assessment of surgical and clinical skills is a need of the day for better evaluation of post-graduate training programs.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.