Changing trends in assessment: Effectiveness of Direct observation of procedural skills (DOPS) as an assessment tool in anesthesiology postgraduate students.

Background and Aims: National Medical commission of India (NMC) has introduced Competency based Medical Education (CBME) following the international trend. Competency based assessment differs from traditional assessment and we need to adopt to newer work place-based assessments (WPBA). Direct Observation of Procedural Skills (DOPS) is one such assessment tool, which assesses procedural skills of a student. Hence this study was conducted to evaluate the effectiveness of DOPS among Anesthesiology postgraduate students. Material and
Methods: A prospective interventional study was conducted, where in 55 postgraduate students underwent DOPS assessment while performing three routine core skills of Anesthesiology. Observations were documented (Pre-DOPS-Score) and immediate verbal feedback was given. Students were educated through demonstration of all three procedures on actual patients by three faculty members using the DOPS checklist. The students were finally assessed again with proforma and DOPS scores were documented (Post-DOPS Score). Mean Pre-DOPS and Post-DOPS assessment Scores were compared. Student and faculty Perceptions were taken regarding the DOPS assessment.
Results: There was significant improvement in the procedural skill performance after DOPS with a statistically significant difference. Average time taken for DOPS test was 11 min. Average time taken for giving feedback was 13 min. Students and faculty members gave positive feedback about DOPS.
Conclusion: DOPS is very feasible and effective assessment tool, improves procedural skills of Anaesthesiology postgraduate students, helps to develop good clinical skills which finally brings good clinical care. Copyright:

Introduction

The aim of the postgraduate courses is to prepare competent doctors capable of independent specialist practice with ability to perform procedures safely. But the traditional assessment methods mainly assess learning by knowledge, i.e., cognitive domain. Now with international trend toward competence based medical curriculum, work place-based assessments (WPBA) are introduced to assess psychomotor domain of procedural skills.

There has been an increase in the rate of iatrogenic postprocedural complications in the field of anesthesiology.[1] Current practices of teaching and assessment of postgraduate students in developing countries are not up to the mark. Students are seldom observed (<50%), assessed, and given feedback (20%).[2] There is little evidence-based research on assessment of procedural skills in anesthesiology. Direct Observation of Procedural Skills (DOPS) is one such WPBA tool where in examiner observes the student directly while performing a routine procedure on a real patient in real situation and gives immediate feedback.[34] Hence this project was taken among postgraduate students of anesthesiology to assess the effectiveness of Direct Observation of Procedural Skills (DOPS), a work place-based assessment (WPBA) tool. DOPS promotes self-directed learning, helps to develop good clinical skills and brings good clinical care.

DOPS was formally introduced by Royal college of Physicians in UK in the year 2005, piloted by the United Kingdom Foundation Programme.[5] DOPS falls on highest levels of Millers Pyramid of clinical competence, i.e., “Shows How” and “Does” where in maximum learning takes place[6] [Figure 1]. Other commonly used workplace-based assessment tools are Mini-Clinical Evaluation Exercise (mini-CEX) and Case-based discussion (CBD).[78]

Figure 1

Millers Pyramid

Material and Methods

A one-year prospective interventional study was conducted in the department of anesthesiology, where in 55 postgraduate students and 21 faculty from department of anesthesiology participated in the study. Seminar was held in the department of Anesthesiology to orient the faculty regarding DOPS assessment and method of giving feedback to students using PowerPoint presentation which included videos. Students were also oriented toward DOPS assessment. Students willing to participate were enrolled for the study after taking informed consent. Students who went on rotational postings were excluded from the study.

Three commonly performed core skills of anesthesiology, i.e., Spinal Anesthesia (first year postgraduate students), Epidural Anesthesia (second year students), and Laryngoscopy and Endotracheal intubation (third year students), were the selected procedural skills for the assessment. Assessment was done using a prevalidated structured proforma wherein eleven parameters were assessed (DOPS Checklist form). Each parameter was rated on five-point scale of 1 to 5. Every student was directly observed while performing the procedures by faculty members in the operation theatre. Observations were documented on the DOPS checklist and final score was noted down as Pre-DOPS-Score.

Immediate verbal feedback was given to students once the procedure was over. Sandwich method of feedback was followed where in students were made to identify their strengths, weaknesses and areas for improvement. Minimum of 60% was considered passing marks for all the three procedures. During the first DOPS assessment constructive feedback was given, which encourages reflective practice. No further training was done for next one month. Second DOPS assessment was done after one month with proforma to assess the improvement in procedural skills and to check the retention of knowledge, which was noted as Post-DOPS Score. Student and faculty Perceptions were taken regarding the conduct of DOPS, perception of DOPS as an assessment tool which were rated on Likert scale of 1 to 5.

Quantitative data from questionnaires was analyzed using the statistical program IBM SPSS (Statistical package for social sciences) version 24 (IBM Corp., Armonk, NY, USA) for MS windows. Quantitative variables, including scores obtained, were presented as means and standard deviation. Discrete data were presented as counts and percentages. Pearson Chi-square test was used to analyze the frequency of pass percentage in pre-DOPS and post-DOPS phase. Mean Pre-DOPS and Post-DOPS assessment Scores were compared. P value of <0.05 was considered as statistically significant.

Prior to the commencement of the study, approval from the Ethical committee was obtained. Approval Chairperson Name Dr Roopa M. Bellad, Jawaharlal Nehru Medical college, KLE academy of higher education and research, Belagavi. Date-10-6-2019, Ethical approval number – MDC/ DOME/131.

Results

Fifty-five students participated in the study. Twenty-six (45%) students were in their first year, fourteen (27%) were in second year, while fifteen (28%) students were in third year of postgraduation. Average time taken for DOPS test was 11 minutes. Average time taken for giving feedback was 13 minutes. Various strengths of postgraduate students identified were their knowledge regarding anatomy and physiology related to various procedures (4.1 ± 0.71), pre-procedural preparation (3.9 ± 0.78), and technical ability (4.4 ± 0.75). Various weaknesses of students, identified in different parameters for procedural skills assessment were lack of professionalism and consideration for the patient, lack of communication skills, imperfect universal precautions, inappropriate aseptic precautions, not willing to seek help when required, and inadequate post-procedural management [Table 1]. Pre-DOPS assessment revealed that majority of students failed to satisfy their consultants by scoring less than 60% of desired marks. Comparison of mean Pre-DOPS and Post-DOPS assessment scores was done [Table 2]. The mean overall performance score in Pre-DOPS phase was 2.8, 2.1, and 2.5 for Spinal, Epidural, and Intubation, respectively [Figure 2]. In post-DOPS phase, there was significant improvement in the procedural skill performance after DOPS feedback with a statistically significant difference with P value of 0.04.

Table 1

Weaknesses of students in different parameters of DOPS for spinal anesthesia

Parameters	Mean	Standard deviation	Scale
Consideration of patient/professionalism	1.8	0.9	1-5
Communication skills	1.9	1.1	1-5
Universal precautions and aseptic precautions	2.2	1	1.5
Painting and draping	1.9	1	1-5
Seeking help when appropriate	2.3	1	1-5
Post-procedure management	2.3	1	1-5

Table 2

Comparative results of Pre-DOPS and Post-DOPS

n	Procedure	Pre-DOPS		Post-DOPS		Pearson Chi-square test
		Pass	Fail	Pass	Fail
55	Spinal Anestdesia	19 (35%)	36 (65%)	46 (85%)	9 (15%)	P 0.04
55	Epidural Anesthesia	9 (15%)	46 (85%)	38 (70%)	17 (30%)
55	Laryngoscopy endotracheal Intubation	17 (30%)	38 (70%)	38 (70%)	17 (30%)

Figure 2

Comparison of mean pre-DOPS and post-DOPS scores

Students and faculty members gave positive feedback about DOPS [Table 3]. Majority of students valued immediate feedback given by faculty [Table 4]. Various negative comments given on DOPS were: it is difficult to integrate DOPS into normal working day; there is insufficiency of supervisors to evaluate and provide feedback; and it is time consuming and tiring for both students and faculty to conduct DOPS assessment and give feedback to students.

Table 3

Anesthesiology student’s perceptions regarding DOPS

No	Survey questions	Average
1	DOPS is fair in testing skills	4.31
2	DOPS assessment scores reflect my performance	4.25
3	DOPS is beneficial for training	4.6
4	DOPS helps in preparing for examinations	4.4
5	This method was more engaging and interesting in comparison to traditional methods	4.31
6	DOPS helps in rectifying misconceptions andgreatly enhance (d) my learning.	4.31
7	DOPS helps to be proficient in basic anaesthesiology skills and helped my understanding of the key concepts	4.54
8	DOPS provides specific and timely feedback, which is helpful to student development	4.42
9	DOPS alleviates fear of examination	4.2
10	DOPS should be continued in anesthesiology as a method of assessment	4.37
11	DOPS can be integrated into the fabric of the working day or normal routine	4.34
12	DOPS provides a chance for better clinical practice after passing exams	4.4

Table 4

Faculty Perceptions regarding DOPS

No	Items	Average
1	Is DOPS an accurate way of determining students’ practical skills	100%
2	Questions included in DOPS were relevant	100%
3	DOPS assessed all components of practical exams	69%
4	DOPS assessed all the students uniformly	77%
5	DOPS measured practical skills better than traditional assessment	92%
6	DOPS eliminates examiner bias	85%
7	DOPS should be continued in anesthesiology as a method of assessment	100%
8	Is DOPS assessment easy	100%
9	Is DOPS assessment cost effective	100%

Discussion

There has been increased emphasis on patient safety and greater accountability to the public and government and rise in the anesthesiology related litigations.[9] If the postgraduates are not well trained and assessed, suboptimal skills will lead to rise in anesthesia related complications including death.[10] Hence there is a global paradigm shift in medical education toward CBME and evaluation of procedural skill.[11] The traditional assessment methods in anesthesiology mainly test knowledge, i.e., cognitive domain.[12] It is not reasonable to assess trainees’ knowledge, judgment, and communication skills by written, oral, and objective structured clinical examinations [OSCE].[13] Skills assessment in traditional teaching is highly subjective and is mainly by evaluating log book entry of procedures.[12] This is because there is no universally accepted and comprehensive assessment tool for anesthesiology procedural skills.[14] Today is the era of Competence based medicine (CBME).[15] With implementation of CBME in India and many other countries, decisions about professional competence are based on work-place based assessment tools (WPBA) in undergraduates as well as postgraduate students.[16]

DOPS assessment brings higher levels of learning as it touches higher levels of learning on Bloom’s taxonomy, i.e., applying, analyzing, and evaluating [Figure 3].[17] Taxonomy of educational objectives is a framework for categorizing educational goals popularly known as Blooms Taxonomy, published by Benjamin Bloom with his colleagues in 1956.[18] Blooms Taxonomy is a hierarchal ordering of cognitive skills.

Figure 3

Bloom’s Taxnomy

The stages of learning of any procedural skills include three steps: cognition, integration, and automation. First step is understanding the task, cognition. Second step is integration of knowledge into the motor skills. Finally, with practice the skill gets imbibed by the learner and becomes a natural part subconsciously.[19] Recent studies prove that feedback given after the completion of a procedure brings out better learning of skills than during the performance of the procedure.[20] and DOPS includes feedback after the procedure.

Summative assessments are high stake examinations and are described as “Assessment of learning” whereas formative assessments are “Assessment for learning” and are usually low stake examinations. DOPS is a very good formative assessment tool in medical education[14] and hence can be included in anesthesiology postgraduate formative assessment.

In our study there was a significant improvement in the post-DOPS scores indicating the improvement in the procedural skills. A study done by McLeod et al.[21] concluded that, DOPS provides an overall insightful perspective of a students’ procedural skills assessment in undergraduate medical students in UK. Morris et al.[7] concluded that DOPS possesses significant positive feedback, results in improving competence-based learning among interns. The advantages of DOPS noted were, short time commitment for the procedural observation and practice opportunities for the commonly performed procedures.

Kumar et al.[22] proved significant improvement in pre- and post-DOPS analyses for different gynecological procedures, concluded that DOPS offers high level of satisfaction and improvement in surgical skills for OBG postgraduate trainees. The advantages of DOPS noted were, provision of rapid and constructive feedback in the form of both marks and comments. Hence, DOPS can be successfully implemented into postgraduate teaching curriculum.

John Roger Barton et al.[23] evaluated DOPS assessment on practitioners in endoscopy colonoscopy. They concluded that DOPS is highly reliable and valid and can be applied for a high-stakes assessment. Authors suggested that use of DOPS can be expanded for relicensing or recredentialing and should be considered for assessment of competence in all clinical areas.

Salman Yousuf conducted search on the databases of Cochrane Library, EMBASE, MEDLINE, and CINAHL, and suggested, though WPBA, cannot replace traditional methods of assessment but adds on to training and formative assessment. These assessment tools not only have educational impact on learning but also provide valuable insight to trainee and assessor.[24]

Liaqat Ali et al.[25] did a comparative study among urology postgraduates while performing various urology procedures of TURP, Cystoscopy, URS, etc., and demonstrated that DOPS is effective assessment tool and improves the skills of urology postgraduate residents. Hill and colleagues[26] used DOPS assessment tool during ultrasound-guided central line insertion in anaesthesia department. They developed a DOPS tool focusing on key components of safe practice with emphasis on the ultrasound guided intravascular needle and guidewire insertion components of central line insertion. They demonstrated that a structured standardised training programme comprising the key components of ultrasound guided central line insertion can result in key competencies being achieved in most participants. Profanter and Perathoner[27] in their benchmark study of prospective randomized trial in small groups of undergraduates, concluded that DOPS is an efficient tool in teaching clinical skills as compared to OSCE.

DOPS requires initial commitment toward development of validated structured checklist for the given procedure and short period of procedural observation.[28] It also requires coordination between students and assessor regarding convenient time for both. Selecting patients is also important, as one cannot select emergency cases, extremely difficult or rare cases for assessment. Assessors need initial training regarding uniform assessment of students using checklist and giving constructive feedback,[2930] which was done with help of a PowerPoint presentation and video demonstration in our study. But with the implementation of competence based medical education it is mandatory that we include WPBA for formative assessments.[3132] Competency based assessment aids in the process of learning.[33] Effective feedback helps the students to improvise their skills. Competence based assessment is an ongoing process and helps to detect lacunae in learning and rectify them early by providing formative feedback.[33]

DOPS assessment became an integral part of the revised training program of Australian and New Zealand College of Anesthetists (ANZCA).[34] The ANZCA DOPS demonstrates construct validity in the assessment of ultrasound-guided regional anesthesia and is potentially feasible in daily practice.[35]

Strength of this study lies in being one of the initial study in India with more objective assessment tools for DOPS in the field of anesthesiology. Limitations of our study were; single center study with lesser number of students as the study was among postgraduate students and it cannot be generalized in other medical disciplines.

Conclusions

DOPS is an effective work place-based assessment tool in improving core procedural skills in Anaesthesia postgraduate students in real time operation theatre scenarios. It also provides immediate and effective constructive feedback to the students.

DOPS should be implemented as a formative work place-based assessment tool in anesthesia postgraduate curriculum.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.