Clinical procedural skills assessment during internship in ophthalmology.

INTRODUCTION: Directly observed procedural skills (DOPS) is a unique method for assessment since it tests the trainee's ability to apply his knowledge and skills in performing a particular procedure and provides an assessment of the practical work performed by the trainee on a 'real' patient under supervision of an experienced faculty. The study aims to make use of DOPS rating for assessment and further improvement in procedural skills in interns in Ophthalmology rotational posting.
METHODS: A prospective study was planned and 15 interns on 2 weeks' rotational posting in the department of Ophthalmology were included by purposive sampling over a duration of 2 months. Four clinical procedural skills were identified and in the second week of posting, the interns were assessed by three DOPS encounters at an interval of 2 days for each clinical skill. The DOPS ratings were analyzed quantitatively using R-statistical software by repeated measure ANOVA and Banfuroni test.
RESULTS: A total of 180 DOPS were undertaken for 15 interns in 4 core areas of ophthalmic examination. The mean overall DOPS rating for DOPS-1 was 3.70±0.82, DOPS-2, 3.83±1.82 and DOPS-3, 4.93±1.65; the difference in DOPS rating between the first and second encounter was not statistically significant (p=0.497), between the second and the third and between the first and the third were statistically significant (p=0.000 in both cases) using Banfuroni test. The overall difference was also statistically significant (p=0.000) using repeated measure ANOVA. Both the assessor and intern satisfaction increased significantly from the first to the third DOPS, but not statistically significant between the first and the second and between the second and the third DOPS.
CONCLUSIONS: We found significant improvement in interns' clinical skills through repeated DOPS and the method was well accepted by both the students and the faculty. Internship period can be well utilized for improving clinical skills and novel performance assessment methods like DOPS might prove to be highly beneficial in ensuring adequacy of learning during internship and also to assess their readiness for accepting professional responsibilities in future.

Introduction

Authentic assessment is an essential and integral part of medical education. In the modern era of competency-based curriculum, the concept of assessment of clinical competency has received increased attention and the methods of assessment associated with higher levels of ‘shows how’ and ‘does’ of Miller Pyramid are being highly recommended (1-3).

Day, et al in the United States documented that most of the first year trainees in medicine had not been observed more than once by a faculty member in a patient encounter while taking a history or doing a physical examination (4). Another observation showed that almost 50% of students were not observed while performing clinical examination (5). Still another study suggested that less than one third of clinical encounters are observed during training (6). Without this direct observation, there is lack of opportunity for assessment of clinical skills and there is increasing evidence that lack of such skills can result in problems in future in actual clinical practice (7). To overcome these limitations and for better predicting the actual performance of a clinician in future in real life problems, the concept of workplace-based assessment (WPBA) started gaining acceptance (8). An advantage of workplace-based assessments is that they fulfil the three basic requirements for assessment techniques that facilitate learning: alignment of the content of the training program, the competency expected as outcome and the assessment practice. Furthermore, the assessment and the training feedback provided during/after it can be used strategically to facilitate learning towards the desired outcomes.

Directly Observed Procedural Skills (DOPS), formally introduced in 2005 and designed by the United Kingdom foundation programme for assessing clinical skills, is considered one of the best known models of WPBA (9,10). It allows assessment of a student in the workplace-based setting where the examiner observes the trainee on a real patient while doing a routine procedure. It can be instrumental in provision of feedback to trainees to improve their performance and facilitate their learning towards desired outcomes.

In the Indian medical education system, the medical students, before graduating, have to undergo one year of compulsory rotational internship in various clinical departments according to a fixed schedule, after which the students can be certified primary care practitioners. Internship is thus a critical period to practice the required clinical skills; this is very important since a large part of our Indian population resides in rural areas and primary health practitioners need to be efficient in diagnosing common disorders. Despite this, the internship posting is limited to filling and signing of log-books and there are no means for formally assessing clinical skills. This study was conducted to introduce DOPS to assess clinical skills in interns posted in the department in performing common ophthalmic procedures and to make use of repeated DOPS for further improvement in procedural skills in interns in Ophthalmology rotational posting.

Methods

Medical undergraduates, after passing the final MBBS exam as part of one year compulsory rotational posting, undergo a two-week posting in the department of Ophthalmology. A prospective study was planned and a total of 15 interns were included by purposive sampling over a duration of 2 months after obtaining approval from Institutional Ethical Committee.

Four clinical procedural skills identified by Royal College of Ophthalmologist as core clinical skills (11) were included after consensus from faculty members of the department. These included visual acuity assessment, torch light examination of anterior segment (difficulty level-1); direct ophthalmoscopy and ocular movements (difficulty level-2).

Though there are many methods of WPBA, DOPS was chosen in our study as the method to assess clinical procedural skills. Results of the studies in literature indicate that DOPS is a high quality instrument and has an appropriate degree of validity and an acceptable reliability; other advantages being positive impact on learning and high satisfaction of students. Despite its acceptance, there are very few studies done exclusively on interns and the effect of repeated DOPS on the improvement of procedural skills during internship has not been well reported in literature. In this study, the clinical procedural skills were assessed repeatedly using DOPS structured checklist on three different encounters at an interval of 2 days for each core clinical skill.

An initial sensitization of assessors was done towards DOPS; in the first week of the internship posting, the interns were provided the list of the four commonly performed procedures they are expected to perform and the first week of their posting was utilized to familiarize them with working in the department. From the second week onwards, the interns were assessed by different faculty members on different occasions over a one-week period with a gap of two days between repeated procedures. For each of the 10 items in the form, the assessor rated the intern on a nine-point scale; in addition an overall grade was awarded. These are standard versions of DOPS grading based on Anghoff standard setting. The grades are based on how likely minimally acceptable or competent candidates are to perform each item correctly (5). There was an additional option for ‘not applicable’ category to imply inability to comment as behavior was not observed or were not applicable to the procedure being observed. After the rating scale, there were two open ended questions for feedback regarding effectiveness of DOPS and the identification of shortcomings. Lastly, the assessor and the intern had to rate their self-perceived satisfaction level on a ten-point scale from 1-9 with 1 as the lowest and 9 as the highest level of satisfaction in using DOPS as a method of assessment. The whole assessment procedure was generally expected to take 15 minutes for observation and 5 minutes for feedback.

DOPS ratings were analyzed quantitatively using R-statistical software by repeated measure ANOVA and Banfuroni test; p-value of ≤0.5 was considered significant. The feedback response to the two-open ended questions was analyzed qualitatively.

Results

A total of 180 DOPS were undertaken for 15 interns. This included four core areas and three DOPS were undertaken per procedure per intern by five assessors.

Overall DOPS rating: Table 1 and Figure 1 summarize comparative overall DOPS scores in all the procedures combined. Mean overall DOPS rating for DOPS-1 was 3.70± 0.82, DOPS-2, 3.83±1.82 and DOPS-3, 4.93±1.65; the difference in value between DOPS 1 and 2 was not statistically significant (p=0.497), between DOPS 2 and 3 and between 1 and 3 were statistically significant (p=0.000 in both cases) using Banfuroni test. Making use of repeated measure ANOVA, the overall difference was also statistically significant (p=0.000), indicating that repeated DOPS encounter does result in improvement of procedural skills of trainees.

Table 1

Number of participants (%) as per their performance after the individual DOPS sessions in the four core areas combined

Performance	DOPS-1	DOPS-2	DOPS-3
Unsatisfactory	47 (78.3%)	32 (53.4%)	4 (6.6%)
Satisfactory	13 (21.7%)	24 (40%)	52 (86.8%)
Superior	0 (0%)	4 (6.6%)	4 (6.6%)
Total (180)	60 (100%)	60 (100%)	60 (100%)

DOPS- Directly observed procedural skills

Figure1

Number of participants (%) as per their performance after the individual DOPS sessions

DOPS ratings for the four core areas: Mean DOPS rating for the four core areas (procedural skills) and the significance values are summarized in Table 2. The overall mean DOPS score increased from DOPS-1 to 3; the difference between DOPS 1 and 3 was statistically significant in all four core areas; values were not statistically significant between 2 and 3 and between 1 and 2. Thus irrespective of difficulty level of the procedure, the DOPS ratings improved significantly between the first and the third assessment in all four procedures, using Banfuroni test as well as repeated measure ANOVA.

Table 2

DPS scoring [Mean±SD] for the four core areas (visual acuity, Torch light examination, Ophthalmoscopy and Ocular movements) at the end of each DOPS session

Core area	DOPS-1	DOPS-2	DOPS-3	DOPS-1	DOPS-1	DOPS-2	Repeated Measure ANOVA
	Mean±SD	Mean±SD	Mean±SD	vs 2 (p)	vs 3 (p)	vs 3 (p)
Visual acuity	3.53±0.74	3.93±1.22	4.60±0.98	0.084	0.001	0.058	0.020
Torch light examination	3.00±0.38	3.66±1.67	4.47±0.99	0.437	0.001	0.142	0.004
Ophthalmoscopy	2.93±0.46	3.87±1.50	4.40±1.12	0.115	0.001	0.493	0.003
Ocular movements	3.33±0.61	4.06±1.38	4.53±1.06	0.332	0.008	0.435	0.013

DOPS- Directly observed procedural skills

Assessor and intern satisfaction ratings: Mean ratings for self-perceived intern satisfaction and assessor satisfaction along with significance values are summarized in Table 3. Both the assessor and intern satisfaction increased from DOPS-1 to 3; the difference was statistically significant between DOPS 1 and 3 but not between DOPS 2 and 3 and DOPS 1 and 2. The findings again indicate that repeated DOPS encounter does result in increase in satisfaction level for both trainees and trainers.

Table 3

Self-perceived intern satisfaction and assessor satisfaction values [Mean±SD] as scored on a 1-9 point scale

Parameter	DOPS-1	DOPS-2	DOPS-3	DOPS-1	DOPS-1	DOPS-2	Repeated measure ANOVA
	Mean±SD	Mean±SD	Mean±SD	vs 2 (p)	vs 3 (p)	vs 3 (p)
Assessor satisfaction	3.66±0.83	3.82±1.82	4.93±1.65	1.00	0.000	0.000	0.001
Intern satisfaction	3.37±1.06	3.75±1.55	4.33±1.43	0.130	0.000	0.000	0.001

Feedback: As of response to the open-ended feedback questions, both faculty and interns found it an efficient tool for assessment. A few limitations were mentioned by the faculty members including time limitation and monotony because of repetition of the procedures.

Discussion

DOPS, as a workplace-based assessment, was specifically designed to evaluate clinical skills and provide feedback. Since the method requires direct observation of trainees while doing a procedure in real life situations, it is particularly useful in evaluating the practical skills of the trainee and also gives opportunity to receive constructive feedback to improve the performance. An extensive literature review of DOPS done by Naeem, N found it to be a high quality instrument with good reliability and acceptability (9). The author analyzed 13 articles on DOPS published from Jan 2000 to Jan 2012 and in the reviewed literature DOPS was found to be a useful tool for assessment of procedural skills. Another systematic review of evidence of DOPS evaluation method by Erfani Khanghahi, M., et al. revealed that DOPS can be used as an effective and efficient evaluation method to assess medical students because of appropriate validity and reliability, positive impact on learning and the students’ high level of satisfaction (12,13).

The present study made use of repeated DOPS for assessing and improving clinical performance of interns for four commonly used ophthalmic procedures. From the first to the third DOPS, the mean overall rating increased significantly, indicating not only the effectiveness of DOPS for assessing clinical skills but also the improvement of students’ performance by repeated DOPS. Similar results were reported by studies showing DOPS as an effective method of assessment in emergency medical students and concluded that students had appropriately good performance and evaluation by DOPS had a significant effect on students’ learning (12). Arezou Farajpour, et al. made use of modified DOPS to assess undergraduate medical students and concluded that WPBA should be taken seriously even during medical school and this should not just be unique to postgraduates (13).

It was an interesting observation that students’ performance, as reflected by DOPS rating, improved significantly on third DOPS rather than the second, in all four core areas, irrespective of the difficulty level. Acceptance of the fact that repeated DOPS results in better performance now raises another question on the appropriate number of DOPS that should be used to make it effective. Naina Kumar, et al. did a prospective study on postgraduate students of Obstetrics and Gynecology department and assessed their competence using DOPS structured checklist on six sessions. The authors found that repeated DOPS resulted in improvement in skills and competence of students in all steps efficiently, irrespective of the teaching methodology used for providing knowledge (14). Ammi, A. (2015) while designing DOPS for selective skills of Orthopedic residents observed that the second testing was useful for residents with good performance but repetition of DOPS for the third time was only useful for residents with weak performance (15). The authors suggested DOPS to be repeated for the third time only for residents with weak performance in the second time. In another study students of restorative dentistry found that 86% of the students believed that two stages of DOPS in each period were sufficient (16). More than half of them were not satisfied with the number of times the test was held (17). It appears that more repetition of exams could appear to be a time wasting exercise to students and could also make evaluators tired, which could discourage them from participating. However, the number of tests should be appropriated by the content of the course; more tests can be used in areas with complicated and widespread content.

Most of the studies in literature have been done on residents and a few on undergraduates; ours is one of the few studies done on interns. Kapoor, H., et al. introduced DOPS and Mini CEX as part of competency-based curriculum for interns and emphasized the importance of structuring the internship program for enhanced learning in the context of Indian Medical education (18). Results of a study by Farajpour, A., et al. showed that the final ward exam mean scores was significantly more than the mean score of DOPS scores; as a result, students with good grades in final exam may not have acquired adequate clinical skills (13). Internship is an appropriate time for enhancement of clinical skills; improvement of clinical skills of interns as evidenced by repeated DOPS ratings in our pilot study and its acceptance both by faculty and by students supports our hypothesis. This might help in future in designing structured internship program for our students with pre-defined expected competencies in alignment with appropriate assessment method so as to reliably predict and improve clinical performance of our students.

It was another interesting observation that both assessor and intern satisfaction level increased from DOPS 1 to 3; here also the scores were statistically significant between the first and the third DOPS and between the second and the third DOPS. From this it can be concluded that satisfaction level also reached a significant level at the third encounter and not at the second. Thus from our study findings, to make the test effective, we recommend a minimum of three DOPS encounters for each clinical skill for interns.

The study had certain limitations. A drawback of DOPS is that it evaluates a specific encounter, which might not be representative of the trainees’ overall performance, rather than rating based on assessment over a longer period of time and that specific encounter. Also, it might be that trainees’ behavior might be influenced when they know that they are being observed. There remains an issue of deciding how many procedures should be observed to achieve adequate reliability and also of determining appropriate checklists and rating scales of different procedures. Another practical issue is that of feasibility of DOPS which can be limited by availability of a ‘real’ patient for a particular procedure and availability of an assessor at the same time. This often becomes difficult in a busy outpatient set-up along with operative commitments to find an assessor with enough time to allocate to this mode of assessment.

Conclusion

Direct observation and evaluation of competence in clinical procedures is not routinely undertaken. It has been accepted that controlled situations during conventional assessment in examinations cannot reliably predict the expected performance of physicians in future. DOPS is unique as it tests the DOES level of Miller’s pyramid and thus tests the ability of the trainee to apply his knowledge and skills to perform a particular procedure in real life situation. Techniques like this, with inherent provision of feedback, provide an opportunity to make use of assessments for teaching in both postgraduate and undergraduate programs. We found significant improvement in clinical skills of interns by using repeated DOPS and the method was well accepted by the students as well as the faculty. Internship period can thus be well utilized for improving clinical skills and novel performance assessment methods like DOPS might prove to be highly beneficial in ensuring adequacy of learning during internship and also to assess the students’ readiness for accepting professional responsibilities in future. The number of tests should be appropriated by the content of the course; a minimum of three encounters appears appropriate for interns, more tests can be used in areas with complicated and widespread content.