Assessment principles and tools.

The goal of ophthalmology residency training is to produce competent ophthalmologists. Competence can only be determined by appropriately assessing resident performance. There are accepted guiding principles that should be applied to competence assessment methods. These principles are enumerated herein and ophthalmology-specific assessment tools that are available are described.

INTRODUCTION

Assessing resident performance is an essential part of the training process and integral to the development of a competent ophthalmologist. Without good assessments in place, one cannot be sure if the resident has learned what is expected. Appropriate tools are needed for assessment, which should be based on accepted principles. This article will review relevant principles of assessment and ophthalmology-specific assessment tools that have already been published and are available for use. Many variations of the principles of assessment have been described. Williams previously described ten specific guidelines for good assessment and I describe those I believe to be most relevant to medical training assessment.1

ASSESSMENT PRINCIPLES

Principle

“Time is valuable and limited”. Ophthalmologist teachers are typically very busy and are paid little or nothing for their teaching activities. Thus, any assessment should be as time efficient as possible but still provide the needed information. The need for brevity must be balanced and at the same time gather adequate valid and reliable information. Too brief an assessment may not provide what is needed and if it is too long, the assessor may not complete the assessment. The most important performance aspects only should be assessed and one should be ready to act on the assessment results to maximize their effect.

“Test application of knowledge/skills rather than just facts”. Although there is clearly a need to assess basic memorized ophthalmic knowledge, assessment of surgical and clinical skill is also essential. A multiple-choice test may be adequate for measuring medical knowledge but it is not a good measure of procedural skill. This means that multiple assessment tools must be used to assess important aspects of the resident's performance. Ideally, multiple assessors will also be involved.

“Performance is situation specific”. Good performance in one situation (e.g. medical knowledge multiple-choice test) does not mean there will be good performances in other situations (e.g. surgical skill or examination techniques). Nevertheless, sometimes it is assumed that the “smart” resident is also professional or a good communicator. Clearly this is not always the case and thus assessment tools must be used to evaluate multiple aspects of the resident's performance.

“Define assessments clearly in advance”. Assessments should not be kept a secret. Let the residents know exactly how they will be assessed and be sure the assessments are aligned with your objectives and performance expectations. Doing this will minimize unnecessary assessments and assure relevant assessments are performed.

“Make every assessment a learning experience”. This may not always be possible if the assessment is an examination that leads to passing or failing a course, rotation or residency. This type of assessment is known as summative feedback. On the contrary, formative feedback is assessment that is designed to improve performance and thus must be shared with the resident. The vast majority of assessments should be formative. When the assessment is shared with the resident on a frequent basis and is designed to improve performance it is called “teaching” Good formative feedback is timely and specific and ideally given every day. It should be used to reinforce good behavior (make a good resident even better) and used to change undesirable behavior.

Good assessment methods and tools have become increasingly important as the spectrum of desired competencies and abilities are more clearly defined and emphasized. Both the Accreditation Council for Graduate Medical Education (ACGME)2 in the United States and the Royal College of Physicians and Surgeons in Canada3 have developed and emphasized competencies thought essential to become a “good” physician. These competencies are similar to certain extent and are listed in Tables 1 and 2. Assessment tools must be available to assure the resident that he/she has achieved these competencies. Fortunately, variety of ophthalmology-specific assessment tools has already been developed to assess these competencies.

Table 1

The ACGME core competencies

Table 2

The CanMEDS competencies

ASSESSMENT TOOLS

Many ophthalmology programs have unpublished and very subjective methods of resident assessment. Typically, these will be evaluations carried out by the teaching faculty where residents are rated on a Likert scale (e.g. 1-5 or 1-10). This form of assessment is probably better than not having a rating scale but it does not meet many of the assessment principles described above. Additionally, medical knowledge is usually assessed by multiple-choice tests at a program or national level. The International Council of Ophthalmology (ICO) offers the only internationally developed test of ophthalmic medical knowledge. The ICO examinations include both basic and advanced levels.4

However, other forms of assessment tools are necessary to assess procedures and skill. These tools usually consist of rubrics ideally with behavioral descriptors. A rubric is a tool that can help one give timely, specific, structured feedback and is defined as an explicit set of criteria used to assess a particular skill. Rubrics can be used to assess virtually any skill including playing a musical instrument, lecturing, or performing a surgical procedure. Good rubrics consist of three parts: (1) Dimensions (e.g. steps of a surgical procedure); (2) Levels (e.g. score of 1-5 or novice, beginner, advanced beginner, competent, expert); and (3) Behavioral Descriptors (what it means to perform at a certain level for any of the dimensions). For example, an assessment rubric for cataract surgery might include dimensions of prepping and draping the patient; and levels from 1-5; and descriptions of exactly what behavior is necessary to score 1-5. Specific examples of procedural rubrics will be described below and are available at (www.icoph.org). Rubrics should be given to the resident in advance so they will serve both, as a tool for teaching and assessing. The resident can read the rubric to learn what is required to be competent in various stages of the procedure. The rubrics should be completed by the teacher immediately after the observed performance and then reviewed with the resident to provide structured, timely and specific formative feedback as described above. Additionally, the rubric allows more objective, structured assessment of resident performance thus allowing areas of deficiency to be identified and remediated.

The first of these ophthalmology-specific assessment rubrics was developed to assess patient care and called the Ophthalmic Clinical Evaluation Exercise (OCEX).5 The specified tool is completed by a teaching physician as they observe the resident performing a patient history, examination and then listens to the case presentation. The teaching physician completes scoring in 33 categories that rate the residents’ ability to communicate effectively, perform a history and examination, and synthesize the information into a differential diagnosis and plan. Importantly, a rubric that describes the behavior necessary to achieve each grade on the OCEX was developed. The OCEX has been shown to have content validity and inter-rater reliability.56 It is available on the ICO's website in multiple languages (www.icoph.org). It was not developed by an international panel and thus may need to be modified to reflect cultural differences. The OCEX is a valid and reliable tool for assessing the competencies of patient care, medical knowledge and communication skills.

Several methods of surgical skill assessment have been devised. Cremers and associates developed the “Objective Assessment of Skills in Intraocular Surgery” (OASIS), a one-page objective evaluation form to assess residents’ skills in cataract surgery.7 The form is completed by an evaluator who directly observes the surgical procedure and includes objective data such as wound placement and size, phacoemulsification time, and total surgical time, etc., They showed that the OASIS had both face and content validity. To complement this objective assessment the same group developed a subjective rating of surgical skills named “Global Rating Assessment of Skills in Intraocular Surgery” (GRASIS).8 This one-page form allows the evaluator to assign scores from 1-5 based on a behaviorally anchored rubric to domains such as preoperative knowledge, microscope use, instrument handling, and tissue treatment in addition to seven other areas. Thus the use of the combination of the OASIS and GRACIS provides objective and subjective evaluation of surgical skills. Feldman and Geist described the Subjective Phacoemulsification Skills Assessment as an evaluative instrument designed specifically for intraoperative assessment of resident phacoemulsification cataract extraction (PCE) surgery.9 This form delineates PCE into overall performance and specific steps of the procedure (e.g. capsulorrhexis, hydrodelineation, IOL implantation, etc). The performance was graded with a rubric defining a good outcome at each step and asking the evaluator to rate on a 1-5 spectrumfrom ‘strongly agree’ to ‘strongly disagree’. They were able to show a degree of inter-rater reliability.

Saleh and colleagues described an assessment tool called the “Objective Structured Assessment of Cataract Surgical Skill” (OSACSS).10 This tool breaks down the phacoemulsification procedure into 20 steps that are scored on a 5-point Likert scale. The scale anchors are: 1= “poorly or inadequately performed”, 3= “performed with some errors or hesitation” and 5= “performed well with no prompting or hesitation”. There are no scale anchors for scores of 2 or 4. Further, there is no behavioral or skill-based rubric for the evaluators to use when assessing the resident's competence. An international panel of author's modified the OSACSS by producing a globally-applicable rubric with levels based on the Dreyfus model of skill acquisition (novice, beginner, advanced beginner, competent, and expert) and with behavioral anchors for each level in each step of the surgical procedure was created.11 Once drafted, the content and face validity were achieved by having an international panel of 15 experts review the draft instrument and provide feedback. The international panel was composed of representatives from North America, Latin America, Asia, the Middle East, Europe and Africa. After incorporating suggestions from the international panel, a final document, the ICO-Ophthalmology Surgical Competency Assessment Rubric (OSCAR) – phacoemulsification was produced.11 Furthermore, it was subsequently shown to have inter-rater reliability.12 In a similar fashion internationally applicable assessment tools for extracapsular cataract surgery (ICO-OSCAR: ECCE),11 small incision cataract surgery (ICO-OSCAR: SICS),13 lateral tarsal strip surgery (ICO-OSCAR: LTS),14 and strabismus surgery (ICO-OSCAR: Strabismus)15 were developed. Additional ICO-OSCARS are being developed for panretinal photocoagulation, corneal transplant, vitrectomy, trabeculectomy and epiretinal membrane peeling. Recently, another cataract surgery assessment tool was developed in Canada and was shown to have a degree of validity and reliability.16 It was not created by an international panel nor does it has a specific behaviorally grounded rubric.

The ICO-OSCAR assessment tools serve a variety of purposes: (1) They are internationally applicable, as comments from an international panel of experts were used to adapt it and make it flexible to any setting; (2) they will decrease subjectivity of the assessment by clearly defining for the assessor what behavior must be observed for each level of proficiency; (3) the rubric will clearly communicate to the learner what is expected to attain competence, and thus this tool can be used for both assessment and teaching.

Other authors have investigated surgical skills outside of actual human surgery. Fisher and associates developed the Eye Surgical Skills Assessment Test (ESSAT); a 3-station (skin suturing, muscle recession, phacoemulsification/wound construction and suturing technique) wet laboratory surgical skills obstacle course for ophthalmology residents.17 In contrast to other surgical assessments, the ESSAT is designed to evaluate residents’ basic skills before entering the operating room. Lee and associates developed an ophthalmology wet laboratory curriculum for teaching and assessing cataract surgical competency.18 The specified curriculum includes pre- and post-tests of cognitive skills in addition to a structured wet lab curriculum with observed ratings of surgical skill. The same group from Iowa has shown that changes in their surgical curriculum have decreased resident complications during cataract surgery.19 Finally, surgical simulation may afford non-patient methods of surgical skill assessment. Le and associates have shown that the anti-tremor and anterior forceps modules of the Eyesi simulator's have construct validity and may be valuable assessment (and teaching) tool in the future.20

Medical educators have also been devising methods to teach and assess the competencies other than patient care and medical knowledge. Lee and associates described an assessment tool involving a structured assessment of journal club that leads to assessment of practice-based learning and improvement.2122 Golnik and associates developed and showed validity of a tool to assess resident on-call performance, the “On Call Assessment Tool” (OCAT).23 The OCAT is a one page checklist to be used retrospectively during random chart review of on-call consultations. Finally, a “360 degree” evaluation tool can be very helpful in evaluating professionalism and communication skills. The intention is to get feedback of the residents from all the groups they interact. Thus, faculty evaluations can be considered 90 degrees and the patients’ surveys, ancillary staff and peer evaluations constitute the other 270 degrees. Self-evaluations may be utilized in addition. Of course, patients and staff are not asked about the resident's medical knowledge but rather about professionalism and communication skills. Jagadeesan and associates have shown that their patients’ satisfaction surveys can discriminate levels of resident communication skill and thus may be useful to assess this competency.24 Internationally, cultural differences may produce difficulties with this type of tool. The information must be anonymous and the evaluators must believe this as the case. Emphasizing to the assessors that this is an attempt to improve the young doctors’ performance can facilitate acceptance of this method. To my knowledge, no internationally valid 360-degree assessment tool exists and thus the ICO is currently developing one in a manner similar to the ICO-OSCARs.

CONCLUSION

There are accepted principles of assessment that should guide our assessment of resident performance. A number of ophthalmology-specific resident assessment tools has been developed. Ideally, every resident training program would utilize a minimum test of medical knowledge, rubrics to assess procedural and clinical skill and a 360-degree evaluation. Assessment of the resident's performance is essential in both teaching and demonstrating the graduating residents, capable to function as competent ophthalmologists.