The Effect of Repeated Direct Observation of Procedural Skills (R-DOPS) Assessment Method on the Clinical Skills of Anesthesiology Residents.

BACKGROUND: The ultimate result of patient care is one of the most important outcomes in medical education. Several methods, including the direct observation of procedural skills (DOPS), have been proposed to assess professional competencies in clinical practice.
OBJECTIVES: This study aimed to assess the effects of the Repeated DOPS (R-DOPS) method on the performance of procedural skills in anesthesiology residents.
METHODS: The procedural skill performance of anesthesiology residents was assessed using a standard DOPS protocol from May to October 2019. Their scores were then objectively recorded, and the satisfaction rates regarding the 2 DOPS exams were assessed.
RESULTS: We found a considerable improvement in anesthesiology residents' procedural skill performance, especially in the anesthesiology residency curriculum's basic items. Besides, anesthesiology residents' satisfaction was significantly improved after the 2nd DOPS.
CONCLUSIONS: R-DOPS leads to improved training outcomes, including assessing the procedural skills, time to feedback to trainees, and trainee satisfaction.

1. Background

Excellence in procedural skills could significantly improve patient safety and reduce complications, while inadequate experience and skills increase the chance of complications and errors. Assessment of procedural skills is an integral part of thriving clinical anesthesiology residents (1, 2).

It is a significant turning point for clinical anesthesiology residents’ proficiency to conduct procedures independently (2-4). Several methods have been proposed for assessment of these professional competencies (5); Mini Clinical Evaluation Exercise (Mini-CEX) introduced in 1995 and the Direct Observation of Procedural Skills (DOPS) introduced in 2003 are two commonly used workplace-based assessments (6-9).

However, many influencing factors could affect the outcome of Mini-CEX/DOPS (7, 10). Meanwhile, much more research is needed regarding the application of Mini-CEX/DOPS in clinical practice (6, 8, 11). There is a necessity of research on “the trainees’ perception of Mini-CEX/DOPS” and Mini-CEX/DOPS’s impact on the outcome of clinical education (7, 8, 12).

2. Objectives

Would there be any change in DOPS outcome when repeated after a while for anesthesiology residents? Is there any role in improving the trainee’s competency if he or she passes the same DOPS after a while? This study was performed to assess the effects of the Repeated DOPS (R-DOPS) method on the performance of procedural skills in anesthesiology residents of the Department of Anesthesiology and Critical Care (DACC) in Shahid Beheshti University of Medical Sciences (SBMU).

3. Methods

This was a descriptive-analytical medical education research. The evaluators were faculty members, DACC, SBMU.

The target population was all anesthesiology residents who were in their training course and included 78 ones. All the residents were in general anesthesia rotation. Among the latter target population, 18 residents were selected using a “Convenience Sampling” method (13). Data collection was mainly performed using checklists that were filled out by the faculty members.

This study aimed to assess the effects of the Repeated DOPS (R-DOPS) method on the performance of procedural skills in anesthesiology residents, DACC, SBMU in 6 months (May 2019-October 2019). To do so, the DOPS protocol was prepared based on the previous standard models for DOPS and the national curriculum for anesthesiology residents at the specialty level (7, 8, 10, 14); then, the checklists were finalized for the examiners; including the following steps:

1- The DOPS exam was performed upon request of the trainee (each anesthesiology resident)

2- The R-DOPS was carried out by the same assessor due to the potential role of supervisor knowledge on the final results of the assessment

3- The examiner observed and monitored the anesthesiology residents “directly” when performing the procedure.

4- Immediately after completion of the procedure, the examiner gave direct feedback to each anesthesiology resident.

5- The examiner signed and confirmed he DOPS score sheet and, per request of the anesthesiology residents, gave a copy of each DOPS results exclusively to him/her

The skills of the anesthesiology residents were ordered by their level; the detailed list is available here.

Level 1 anesthesiology residents, the 1st half of anesthesiology residency: mask ventilation, laryngoscopy, oral intubation, laryngeal mask placement

Level 1 anesthesiology residents, the 2nd half of anesthesiology residency: Nasal intubation, laryngeal mask placement, spinal anesthesia in sitting position

Level 2 anesthesiology residents: Nasal intubation, spinal anesthesia in the lateral and sitting position, cannulation of the arterial line, epidural anesthesia, central vein cannulation

Level 3 anesthesiology residents: Spinal anesthesia in sitting and lateral position, cannulation of the arterial line, central vein cannulation, epidural anesthesia, endotracheal intubation using fiberoptic bronchoscope, peripheral nerve blocks

Level 4 anesthesiology residents: Central vein cannulation, peripheral neural blocks, epidural anesthesia, endotracheal intubation using a fiberoptic bronchoscope

Besides, the DOPS checklist included the following items:

1- The first part contained the information of each anesthesiology resident and the examiner.

2- The second part included the main steps of procedures’ implementation, while the examiners evaluated the procedural skills of each anesthesiology resident at each stage accordingly: “lower than expected, borderline acceptable limit, within the expected level, and higher than expected level.”

3- The third part consisted of two stages: the final anesthesiology residents’ evaluation and ultimate feedback

4- A separate checklist was used for each procedure; since for each clinical procedure, a standard DOPS checklist is required; however, for the sake of briefness, we did not add the individual checklists to the text

In the final resident evaluation stage, the abilities of anesthesiology residents were defined based on the level of need for faculty supervision at four levels.

Level 1 anesthesiology residents were unable to perform the process without supervision

Level 2 anesthesiology residents were able to perform the procedure with full supervision

Level 3 anesthesiology residents were able to perform the procedure with minimal supervision

Level 4 anesthesiology residents were able to perform the procedure without the need for any supervision

Upon completion of the test, anesthesiology residents were given constructive and practical feedback, including both strengths and weaknesses. The time which was taken to complete each DOPS assessment and the time taken to give the final feedback to the residents was recorded. Besides, attitudes, opinions, potential suggestions, and potential criticisms of anesthesiology residents regarding the test were asked, and their satisfaction with the test was asked using a numerical scale of 0 to 10 (0 = no satisfaction; 10 the greatest ever satisfaction). Three DOPS were performed for level 1 anesthesiology residents in each 6-month rotation period, while for level 2 to level 4 anesthesiology residents, two tests were performed in each 2-months rotation period.

The study protocol was approved by the Research Ethics Committee, SBMU, Tehran, Iran; coded IR.SBMU.RETECH.REC.1399.692.

3.1. Data Analysis

the data retrieved from the checklists were presented and analyzed as a cumulative mean ± standard deviation for each checklist item. SPSS (version 11.5; SPSS Inc, Chicago, IL, USA) was used for data entry and analysis.

4. Results

During six months, 18 anesthesiology residents participated in the study (including eight female and ten male participants). The procedures performed by each level are summarized in Table 1; however, their level composition was accordingly:

Table 1.

The Procedures Performed in R-DOPS

Procedure Title	Frequency of Procedures
Airway management (basic level)	2
Arterial line cannulation	3
Laryngeal mask airway insertion	1
Fiberoptic bronchoscopy for endotracheal intubation	3
Spinal anesthesia	2
Peripheral Nerve Blocks
Inter-scalene block	5
Supraclavicular block	9
Axillary block	9
Femoral block	6
Popliteal block	5
Total	45

level 1 anesthesiology residents: 4 people

level 2 anesthesiology residents: 5 people

level 3 anesthesiology residents: 5 people

level 4 anesthesiology residents: 4 people

The time spent observing the exam was significantly shorter in the second DOPS compared with the first DOPS (13.28 ± 4.03 vs. 9.88 ± 3.46; P value = 0.0000). Also, the time spent giving feedback to the residents was significantly shorter in the second DOPS compared with the first DOPS (4.88 ± 1.5 vs. 2.91 ± 1.16; P value = 0.0000). Besides, Repeated DOPS (2nd exam) led to improved satisfaction of the anesthesiology residents (8.28 ± 0.78 in the first DOPS compared with 9.46 ± 0.58 in the 2nd DOPS; P value = 0.0000; all scores from a maximum of 10; 0 = completely dissatisfied; 10 = completely satisfied).

The practical skills of the anesthesiology residents in the second DOPS improved significantly compared to the first DOPS. Besides, the anesthesiology residents’ level of ability increased with the 2nd test (Tables 2 and 3). However, this improvement was much more significant in procedures other than peripheral nerve blocks (Table 2 vs. Table 3). While all DOPS items improved significantly in the 2nd DOPS, the following items did not have significant improvements in the 2nd DOPS in peripheral nerve blocks: Indications of block; Alternatives for the block; Complications of the block; Preparation for the block; Safe and appropriate sedation; Managing potential complications; Calling for help if indicated; Patient management after performing the procedure; Evaluation procedure success or failure.

Table 2.

Cumulative Results of DOPS in Peripheral Nerve Blocks (Including Inter-Scalene Block, Supraclavicular Block, Axillary Block, Femoral Block, And Popliteal Block) [a]

Assessment Index	Performance of Anesthesiology Residents in the First DOPS [b]				Performance of Anesthesiology Residents in R-DOPS [b]				P-Value [c]
	Level 1	Level 2	Level 3	Level 4	Level 1	Level 2	Level 3	Level 4
1. Anatomy of block	0	20	14	0	0	0	32	2	0.008
2. Indications of block	1	3	30	0	0	0	34	0	0.1
3. Block technique	1	13	20	0	0	0	25	9	0.03
4. Alternatives for the block	0	0	34	0	0	0	34	0	0.22
5. Appropriate knowledge about block complications	0	1	33	0	0	0	34	0	0.12
6. Informed written consent	0	9	25	0	0	0	28	6	0.002
7. Preparation for the block	0	2	32	0	0	0	34	0	0.004
8. Safe and appropriate sedation	0	0	34	0	0	0	33	1	0.12
9. Local anesthesia at the site of needle entry	0	7	27	0	0	0	34	0	0.03
10. Coordination with aiding personnel	0	0	34	0	0	0	24	10	0.02
11. Constructive rapport with the patient	1	7	26	0	0	1	23	10	0.01
12. Technical ability for performing block	1	12	21	0	0	0	19	15	0.001
13. Managing potential complications	0	0	34	0	0	0	34	0	0.12
14. Calling for help if indicated	0	0	31	3	0	0	31	3	0.13
15. Patient management after performing the procedure	0	1	33	0	0	0	34	0	0.16
16. Evaluation procedure success or failure	0	5	29	0	0	0	33	1	0.09
17. Professionalism	0	0	34	0	0	0	24	10	0.01
18. Overall procedure performance	1	11	22	0	0	0	20	14	0.005

a Level 1: anesthesiology resident was unable to perform the process without supervision; Level 2: anesthesiology resident was able to perform the procedure with full supervision; Level 3: anesthesiology resident was able to perform the procedure with minimal supervision; Level 4: anesthesiology resident was able to perform the procedure without the need for any supervision.

bIn Term of Anesthesiology Residents’ Number

cThe Chi 2 Test

Table 3.

Cumulative Results of DOPS in Other Procedures (Except For Peripheral Nerve Blocks) [a]

Assessment index	Performance of Anesthesiology Residents in the First DOPS [b]				Performance of Anesthesiology Residents in R-DOPS [b]				P-Value [c]
	Level 1	Level 2	Level 3	Level 4	Level 1	Level 2	Level 3	Level 4
1. Anatomy of the site	1	6	4	0	0	1	8	2	0.005
2. Indications of the procedure	0	5	6	0	0	0	8	3	0.004
3. Appropriate Preparation for the Procedure	0	2	9	0	0	0	5	6	0.005
4. Familiarity with the Technique of the Procedure	0	0	11	9	0	0	4	7	0.005
5. Appropriate knowledge about block complications	2	6	3	0	0	1	6	4	0.005
6. Safe and appropriate sedation	1	3	7	0	0	1	8	2	0.003
7. Patient management after performing the procedure	1	3	7	0	0	0	7	4	0.004
8. Managing potential complications	4	4	3	0	0	1	9	1	0.005
9. Calling for help if indicated	1	2	7	1	0	0	9	2	0.005
10. Coordination with aiding personnel	2	7	2	0	0	0	5	6	0.005
11. Constructive rapport with the patient	1	5	4	1	0	0	6	5	0.007
12. Evaluation Procedure success or failure	0	4	6	1	0	0	8	3	0.006
13. Professionalism	0	2	8	1	0	2	1	8	0.008
14. Technical Ability for Performing the Procedure	0	5	6	0	0	0	8	6	0.001
15. Informed written consent	1	6	4	0	0	1	8	2	0.005

a Level 1: anesthesiology resident was unable to perform the process without supervision; Level 2: anesthesiology resident was able to perform the procedure with full supervision; Level 3: anesthesiology resident was able to perform the procedure with minimal supervision; Level 4: anesthesiology resident was able to perform the procedure without the need for any supervision.

bIn Term of Anesthesiology Residents Number

c The Chi 2 Test

5. Discussion

This study demonstrated that the R-DOPS exam improves the clinical performance of anesthesiology residents, especially in areas other than nerve blocks; however, even in peripheral nerve blocks, the R-DOPS exam improves some aspects of clinical performance.

Mini-CEX/DOPS are formative assessment methods (15-17); this could be a possible explanation for the majority of our findings. In other words, repeated DOPS leads to improved performance of anesthesiology residents; which is an example of the “learning by doing supported by feedback” approach; especially if “workplace assessment” is going to be evaluated, which is the exact aim of Mini-CEX/DOPS (9, 17-19).

Though our R-DOPS results were often much better than the first DOPS, some may argue the effect of the time interval between the two DOPS exams, which could be a potential factor, improving the R-DOPS results. Though this could not be ruled out, the DOPS per se is a formative assessment method (20-23); so, the R-DOPS leads to improved anesthesiology residents’ results.

Meanwhile, some items in the “regional anesthesia” fields were not improved with 2nd DOPS. The most probable explanations for this finding could be:

1- The difference between the p-value of peripheral blocks in comparison to other skills can be due to fewer blocks being done by residents in their residency course; possibly, regional anesthesia should be addressed in more detail in DOPS

2- DOPS for regional anesthesia needs a separate setting, in such a way, the assessment would be more focused and with much more attention

3- When areas like airway management have assessed the level of care is different from “regional anesthesia” care

The clinical course of medical education in anesthesiology residency includes several “must need” items (14, 24, 25).; similar to many other medical specialties (26, 27), the current study included many of these basic items. In other words, we did not include a limited list of topics; instead, we assessed a relatively comprehensive list. So, the results could be used as extrapolation for R-DOPS in other anesthesiology residency programs.

Feedback to trainees is one of the main areas that should be focused on DOPS (8, 28, 29). Our results demonstrated that Re-DOPS decreased the time needed for feedback to the residents, which could be considered an advantage of Re-DOPS and a marker for improved learning by the trainee.

Based on the studies by Lörwald and colleagues (7, 8, 10, 30), Mini-CEX/DOPS are affected by four main groups of factors and their interactions: Context, Users, Implementation and Outcome.

Based on the latter classification, the results of our study could exactly cover 3 of 4 areas from the above list (7, 8, 10, 30), including: Context (i.e., time to feedback to anesthesiology residents); Users (i.e., trainees’ satisfaction); Outcome (i.e., the time needed for the 1st DOPS compared with the 2nd DOPS).

Our study demonstrated that Re-DOPS could improve DOPS in the above areas, a finding supported by the previous studies, including studies by Lörwald and colleagues and other studies (1, 7, 8, 10, 30).

5.1. Study Limitations

There are several limitations to our study: 1- DOPS is a clinical evaluation that aims to identify weaknesses in individuals’ clinical skills. Identifying weaknesses and giving feedback to the trainee is only a part of the assessment process, and the other important part is designing the necessary strategy to eliminate the weaknesses. Therefore, a second evaluation is necessary to evaluate the effectiveness of our strategy in eliminating weaknesses. We did not assess our educational and clinical strategies to strengthen our weaknesses. The quality of strategies may be affected due to considering the quantity evaluation; 2- Our study demonstrated that appropriate feedback to the trainee could play an important role in improving their skills and the outcome of the education; however, we could not prove whether it was just the repetition of DOPS that improved the skill or any other potential factor(s) has been involved; 3- During the six months, there was no specific focus on training of the items of DOPS; this time lag though inevitable, could be a potential limitation of the study; 4- The convenience sampling method used in our study is always associated with some degrees of bias; however, the sample size could be a potential drawback of the study; 5- The amount of the residents’ knowledge about DOPS before the study is essential in the first DOPS score; in such a way that improvement in R-DOPS can be related to the residents’ ability to perform the test and what is essential for the investigators; this might be one of the uncontrolled confounders in our study; 6- Some other factors may affect the performance improvement, including the residents’ knowledge level, how they trained before the test and during this “6 months” period; this might be one of the study limitations.

5.2. Conclusion

The results of the current study demonstrated that Repeated DOPS (R-DOPS) could lead to improved training outcomes, including the results of the assessed items in the checklist, time to feedback to the trainees, and the results of resident satisfaction.