Effect of formative evaluation using direct observation of procedural skills in assessment of postgraduate students of obstetrics and gynecology: Prospective study.

INTRODUCTION: Direct Observation of Procedural Skills (DOPS) is a way of evaluating procedural skills through observation in the workplace. The purpose of this study was to assess the role of DOPS in teaching and assessment of postgraduate students and to know the effect of repeated DOPS on improvement of the skills and confidence of the students.
METHODS: In both phases, significant difference was observed between the two groups on first DOPS comparison (1st phase: p=0.000; 2nd phase: p=0.002), with simulation group performing better. Comparison of sixth DOPS in the two groups revealed no difference in both phases, but significant difference on first and sixth DOPS comparison in each group (p=0.000).
RESULTS: Repeated DOPS results in improved skills and confidence of students in managing real life obstetric emergencies irrespective of the teaching modality.
CONCLUSION: Repeated DOPS results in improved skills and confidence of students in managing real life obstetric emergencies irrespective of the teaching modality.

Introduction

Acquisition of clinical and procedural skills is an important part of postgraduate training. The new millennium has started a period of major changes in postgraduate medical education, notably from knowledge based towards competency-based programs aimed at achieving pre-defined training outcomes (1,2).

Assessment is an important input required for improving medical education quality. This is especially important for helping students acquire good clinical skills during their learning phase as postgraduates (3). This function of assessment is sparingly used in India because of obsession with objectivity (4,5) and almost 50% of students are not observed while performing clinical examination (6). Other reports also suggest that during training less than one-third of clinical encounters are observed (7). Also even at postgraduate level, 80% of time, the trainees do not receive any productive feedback (8). Here, comes the role of Work Place Based Assessment (WPBA) which is defined as ‘assessment of working practices based on what doctors actually do in clinical setting, predominantly carried out in workplace itself (9). It helps in collecting quantitative and qualitative data about trainee’s performance and using it for providing feedback at the same time, thus enabling learning. Direct observation of Procedural Skills (DOPS) is a form of WPBA which allows direct observation of the trainee in real-life setting while performing actual tasks and enables the trainer to give feedback to the trainee based on the observations (10). The main purpose of DOPS is to provide formative assessment and feedback about performance of the trainee (11). Also it helps in assessing a number of soft skills such as communication, ethical and professional skills, and team work among health workers required while treating a patient and these skills are best assessed in real-life scenario (12). There is increasing evidence suggesting that lack of such skills can result in things go wrong in actual clinical practice (13). Despite its known merits in promoting learning, use of feedback based on observation of performance is often not well utilized in medical education (14,15). urthermore, in India very few studies are available that have evaluated the role of WPBA in assessing medical students, but still all of these reports encourage results in terms of acceptance by the faculty and students (16-19).

Hence, the present study was conducted with the aim of evaluating the role of DOPS in teaching as well as assessment of postgraduate students of Obstetrics and Gynecology department in dealing with obstetric emergencies. The aim of the study was to assess the role of DOPS as a teaching and assessment tool amongst postgraduate students of the department of Obstetrics and Gynecology and to know the effect of repeated DOPS encounter on overall improvement of skills and confidence of the students in dealing with obstetric emergencies in department of Obstetrics and Gynecology of a rural tertiary care centre.

Methods

The present study was a Prospective educational research study with all the postgraduate students of department of Obstetrics and Gynecology of a rural tertiary care centre of Northern India as study subjects. The sample size was 20 postgraduate students and the study was carried out for duration of 7 months.

The present prospective educational research was conducted in the department of Obstetrics and Gynecology of a rural tertiary care centre in Northern India after written informed consent and proper Institutional ethical approval. The study was conducted in two phases over a period of seven months. All twenty postgraduate students (including first, second and third year) of Obstetrics and Gynecology department were randomly divided into two groups by lottery system; ten students in each group. Both groups contained equal number of first, second and third year students; this was done by randomly asking each year resident to pick the chit with either group 1 or group 2 written over it. In the first phase of study the students in group one received simulation-based teaching using demonstration with hands-on exposure on mannequins for 4 encounters and Group 2 students were taught by didactic lecture on Active Management of Third Stage of Labor (AMTSL) over a period of one month. After teaching by either modality, all the students in two groups were asked to perform steps of AMTSL on live delivering woman in labor room and their different levels of competency involved in this core skill were assessed using DOPS structured checklist on 6 encounters of 15-30 minutes each at regular intervals over a period of two months. In the second phase flipping of the groups was done and the topic of teaching changed to management of PPH, that is Group 1 students received didactic lecture and Group 2 students received simulation-based teaching using demonstration with hands-on exposure on mannequins for 4 encounters on management of PPH over a period of one month. The main purpose was to remove bias and to expose all students to both modalities of teaching used in the study. This was followed by assessment of the performance of all students on live patients, the steps of management of PPH using DOPS structured checklist on 6 encounters of 15-30 minutes at regular intervals over a period of two months. The final analysis of quantitative data in the form of marks obtained in DOPS checklist was done by Wilcoxon rank-sum (Mann-Whitney) test using SPSS software, version 20.

Results

A total of twenty postgraduate students of the department of Obstetrics and Gynecology were enrolled in this study with ten students in each group. In first phase, comparison of first DOPS structured checklist encounter of 10 marks revealed a significant difference between the two groups (p=0.000), with Group 1 taught by simulation-based teaching (Mean±SD=5.20±0.79) performing better than Group 2 taught by didactic lecture (Mean±SD=3.20±0.63) (Table 1). Repeated encounter of DOPS resulted in overall improvement of all students in both groups, with results of sixth DOPS encounter showing significant difference in comparison to first DOPS encounter in each group (p=0.000), but no significant difference was noted on comparing results of sixth DOPS encounter between the two groups (Table 1), indicating that repeated DOPS encounter results in improvement of skills and confidence of the students in both groups in performing all the steps efficiently, irrespective of teaching modality used for providing knowledge. Similarly, in the second phase of study after flipping of groups, comparison of results of first DOPS encounter revealed a statistical difference between the two groups’ performance (p=0.002) with Group 2 (Mean±SD=5.10±1.10) performing better than Group 1 (Mean±SD=3.40±0.84) (Table 2). Again with repeated DOPS encounter the performance of all students in both groups went on improving with results of sixth encounter of DOPS showing statistically significant difference in comparison to first DOPS encounter (p=0.000) in both groups, with no significant difference on comparison of results of sixth DOPS encounter in Group 1 and 2 (Table 2). This reflects that when students were made to perform the same task repeatedly under observation on live patients, their skills, knowledge and confidence increased irrespective of the method of teaching used.

Table 1

Comparison of students’ performance in first and sixth encounter of DOPS on AMTSL in the both groups

AMSTL
Mean±SD	DOPS 1	DOPS 6
Simulation (Mean±SD)	5.20±0.79	9.70±0.48	z=3.844
			p=0.0001
Didactic lecture (Mean±SD)	3.20±0.63	9.70±0.48	z=3.863
			p=0.0001

Table 2

Comparison of student performance in first and sixth encounter of DOPS on PPH management in both groups

PPH
Didactic	DOPS 1	DOPS 6
Didactic lecture (Mean±SD)	3.40±0.84	8.90±0.74	z=3.824
			p=0.0001
Simulation (Mean±SD)	5.10±1.10	9.60±0.52	z=3.822
			p=0.0001

Discussion

Postgraduate teaching is directed not only at attainment of knowledge, attitude and skills but also towards responsiveness and proper functioning in real life situations. It is known that controlled situations in conventional assessment during examination settings usually fail to measure these outcomes. Hence, there is always a need to observe and assess postgraduate trainees in real life scenarios for necessary corrections and WPBA is one such tool that is being increasingly used for assessment of students by direct observation and help in learning (20). Despite clear evidence in favor, and power of observation of actual clinical work, feedback remains grossly underutilized in medical education. While much data is not available from India, studies from western countries suggest that very few clinical encounters are actually observed during training (7,21). Also experience related to DOPS in India is very limited, but still studies suggest that students enjoy receiving constructive feedback on their performance (16). Few studies are conducted so far to assess acceptance of this procedure in medical education (22).

The present study revealed that DOPS is an effective tool for teaching as well as assessment of postgraduate students of Obstetrics and Gynecology. Similar results were reported by studies which revealed that using DOPS is known to augment medical students’ skills in neurology and gynecology departments (22,23). A study from University of Toronto, Canada reported that direct observation and assessment of competence in clinical tasks is not routinely done by educational supervisors (24) and suggested that this void in training evaluation can be filled using surgical DOPS as an assessment instrument.

In our study it was observed that repeated DOPS encounter for a particular procedure resulted in overall improvement of skills and confidence of students irrespective of teaching modality. In the first phase of study, on comparison of first DOPS encounter with results of sixth DOPS encounter revealed statistically significant difference in both the groups (p=0.0001). Similarly in the second phase after crossover significant improvement was noted in both groups on comparison of first and sixth DOPS results (0.000), indicating that repeated DOPS has a positive effect on improvement of students’ performance, irrespective of the teaching modality. Similar results were observed in a study, reporting that DOPS was an effective tool in facilitating students’ learning and skills and the fact that repeated performance by students resulted in better performance (performance increased from 50.6% to 59.4%). Also the students believed that DOPS tests decreased their stress in final examination (25). Similar results were observed in another study suggesting that DOPS played an effective role in improving students’ learning and skills. They also observed that test repetition for second time is sufficient and useful for evaluating students’ clinical and instrumental skills (26). Dabhadkar, et al. reported that all participating students and faculty members found DOPS as 100% relevant to curriculum and feasible to accept as regular formative assessment and a learning tool for postgraduate students in the department of Obstetrics and Gynecology. Also higher patient satisfaction observed during the second DOPS session suggested that such an intervention can result in better patient care in future (27). In a similar study on the advantages of DOPS in students of restorative dentistry, it was found that 86% of the students believed two stages of DOPS tests in each period were good for skills development. There was an increasing trend in students’ performance from the first stage to the second one (28).

Other studies on use of DOPS in assessment and teaching of students revealed that DOPS results in greater level of student satisfaction and helps in promoting students’ learning and clinical skills (22,29-32).

Another study investigating the effect of evaluation on emergency medical students’ clinical skills taught through DOPS showed that students performed well and evaluation by DOPS had a significant effect on students’ learning (33). Also a study conducted to evaluate feasibility and acceptability of DOPS in medical education for teaching and assessment of postgraduates of Pediatrics department reported that all 15 trainees enrolled in the study reported that direct observation enhanced their procedural skills and wanted that the use of DOPS should be extended to all procedures (34).

Hence, DOPS is an important tool for teaching as well as assessment of postgraduate students of department of Obstetrics and Gynecology for all obstetrical procedures.

Conclusion

Apart from challenges and time constraint for implementation of DOPS, we report a high level of satisfaction and improvement in performance of skills among postgraduate trainees. Perhaps the strongest advantage of DOPS is provision of rapid feedback in the form of both marks and constructive comments in health care contexts. Hence, DOPS can be successfully implemented into postgraduate teaching curriculum. More research is needed in future to further investigate the importance and suitability of DOPS in medical education, especially individual desciplines.

Lessons for Practice

DOPS is an effective teaching and assessment tool for training postgraduate students of the department of Obstetrics and Gynecology, especially for obstetric emergencies. Apart from its few limitations like time constraint, prior sensitization and training of faculty members, it can be successfully implemented in postgraduate teaching curriculum in future for better training of the students.