Assessment of procedural skills in residents working in a research and training institute: An effort to ensure patient safety and quality control.

BACKGROUND: To ensure patient safety, it is important to regularly assess the knowledge and practical skills of anesthesia trainees. This study was conducted to evaluate the competency of the residents and the impact of various corrective measures in the form of didactic lectures and clinical skill demonstrations on the conduct of various procedural skills by the residents.
MATERIALS AND METHODS: Ninety-five junior residents were enrolled in this study. Assessment of competency of 1st, 2nd, and 3rd year residents in performing various procedure skills of anesthesia was done in two stages using procedure specific checklist (PSC) and Global Rating Scales (GRSs). Preliminary results of the first assessment (Score 1) were discussed with the residents; deficiencies were identified and corrective measures suggested by didactic lectures and clinical skill demonstrations which were followed by a subsequent assessment after 3 months (Score 2).
RESULTS: There was a statistically significant improvement in the PSC and GRS scores after corrective measures for all the procedural interventions studied. Percentage increase in scores was maximum in 1st year (42.98 ± 6.62) followed by 2nd year (34.62 ± 5.49) and minimum in 3rd year residents (18.06 ± 3.69). The percentage increase of scores was almost similar for all subset of procedural skills; low, intermediate, and high skill anesthetic procedures.
CONCLUSION: For assessment of procedural skills of residents, use of PSC and GRS scores should be incorporated and the same should be used to monitor the impact of various corrective measures (didactic lectures and clinical skill demonstrations) on the conduct of various procedural skills by the resident.

Introduction

The practice of anesthesia is challenging and dynamic. The anesthesiologist is often faced with time sensitive situations which demand critical decision-making and high-quality surgical interventions. It is thus important to regularly assess the knowledge and practical skills of the trainees to ensure patient safety.[123]

Our hospital is a tertiary care referral hospital. The Department of Anesthesia and Intensive Care is one of the largest with 90–100 junior residents (students enrolled in MD Anesthesia for a period of 3 years). The teaching faculty consists of approximately thirty consultants and forty senior residents (anesthesiologists recruited for a period of 3 years after completion of MD). They are responsible for the training of these residents and monitoring of their clinical and academic activities. The aim of this prospective observational study was to incorporate a combination of previously validated checklists and Global Rating Scales (GRSs)[4] for the evaluation of the competency of the residents during the performance of various procedural skills. This was to ensure better quality of health-care delivery by identifying deficiencies and initiating corrective measures in the form of didactic lectures and clinical skill demonstrations. A reevaluation of the residents done after a period of 3 months was used to assess the impact of the corrective measures.

Materials and Methods

The study was conducted after informed consent from the junior residents and approval by the institutional ethics committee. Assessment of a particular procedural skill was done only if the log book of the resident showed more than twenty independent, successful performances of the skill in the past. Any resident with less than this number was excluded from the study. Three subsets of residents were identified based on number of years of admission in the residency program; 1st year (1st and 2nd semester), 2nd year (3rd and 4th semester), and 3rd year (5th and 6th semester; each semester comprises 6 months duration).

The trainees were assessed for competency in six different procedures which were grouped in three subsets based on the level of skill needed for successful completion of the same.

Low skill procedures

Endotracheal intubation

Subarachnoid block.

Intermediate skill procedure

Lumbar epidural block

Arterial line cannulation (radial artery).

High skill procedure

Central line (internal jugular vein) cannulation

Nerve stimulator-guided supraclavicular brachial plexus block.

Assessment of each procedural skill was done by the consultant anesthetist (second and third investigator) using two evaluation tools:

Procedure-specific checklist (PSC) (Appendix A): A resident's score was calculated at the completion of the procedure based on the proportion of items done correctly in the checklist

GRS (Appendix B): It consists of seven dimensions, each related to some aspect of procedure performance. Each dimension was graded on a 5-point scale with the middle and extreme points anchored by explicit descriptors. Each 5-point item was scored from 0 (poor performance) to 4 (good performance). A resident's score for a given station was determined by summing the marks on the seven dimensions and dividing by 28 to obtain a percentage score. Items on the GRS were developed in accordance with the procedure planned/evaluated.

Data collection

Assessment of competency was done in two stages. Preliminary results of the first assessment (Score 1) were discussed with the residents and corrective measures suggested by didactic lectures and clinical skill demonstrations which was followed by a subsequent assessment after 3 months (Score 2).

Paired t-test was used to compare the score before and after didactic lectures and clinical skill demonstrations. Percentage change in score for different subset of residents (1st, 2nd, and 3rd year) was compared within the group and between the group using one-way ANOVA and post hoc analysis with Bonferroni corrections, respectively. All tests were considered two tailed with 95% confidence interval and statistical significant P < 0.05.

Results

We enrolled 95 junior residents in this study. The distribution of residents based on years of experience is summarized in Figure 1.

Figure 1

Distribution of residents based on years of experience

Comparison of scores (procedure-specific checklist and Global Rating Scale) of residents for various procedural skills

There was a statistically significant improvement in the PSC and GRS scores after corrective measures for all the procedural interventions studied [Table 1].

Table 1

Results of evaluation of the residents during performance of procedural skills

Comparison of percentage increase in scores (procedure-specific checklist and Global Rating Scale) of residents based on complexity of the procedure

The percentage increase of scores was almost similar for all subset of procedural skills; low, intermediate, and high skill [Table 2].

Table 2

Percentage increase of scores based on complexity of procedures performed

Comparison of scores (procedure-specific checklist) of residents based on years of training

The PSC scores of residents in the three subsets based on the duration of residency are summarized in Table 3. Percentage increase in performance of PSC was maximum in 1st year (42.98 ± 6.62) followed by 2nd year (34.62 ± 5.49) and minimum in 3rd year (18.06 ± 3.69). The difference was statistically significant (P = 0.00) for all comparisons between 1st, 2nd, and 3rd year.

Table 3

Percentage increase of scores in residents in 1st, 2nd, and 3rd year of training

Comparison of percentage improvement in scores for all procedures performed by residents

Detailed intergroup comparison was done for all procedures performed by residents of either of the six semesters [Figure 2a and b]. Improvement of PSC score of 1st semester residents was significantly more compared to improvement of 5th and 6th semester residents in all anesthetic procedures. Compared to 4th semester residents, 1st and 2nd semester residents showed significantly more improvement in performing subarachnoid block, and for other procedures, the percentage improvement was not statistically significant (P > 0.05). When improvement was compared among 1st, 2nd, and 3rd semester residents, for central line insertion and supraclavicular brachial block, 1st semester residents showed significantly more improvement compared to 2nd and 3rd semester, and for all other procedures, no statistically significant difference in percentage improvement was seen among them. Improvement of 3rd semester residents was significantly more compared to 4th semester in arterial and central line insertion. Improvement of 4th semester residents was significantly more compared to 6th semester for all procedures. Compared to 5th semester residents, 4th semester residents showed significantly more improvement in performing subarachnoid and epidural block and arterial line insertion. Improvement of 5th and 6th semester residents was comparable for all procedures.

Figure 2

(a) Percentage improvement in procedure-specific checklist score. (b) Percentage improvement in Global Rating Scale Score

Discussion

In our study, regular assessment of procedural skills of residents and corrective measures in the form of didactic lectures and clinical skill demonstrations led to a statistically significant improvement in the performance of the procedural skills which were evaluated using PSC and GRS scores. The values of the percentage increase in the cumulative scores however did not differ significantly based on the complexity of the procedure which we graded as low, intermediate, and high in our study. The percentage increase in score of PSC and GRS for all procedures was maximum in 1st year residents and minimum in 3rd year residents, and the differences were statistically significant. This reemphasizes the fact that residents are more susceptible to change during early years of training.

The cumulative value of the GRS scores for all residents [Table 1] decreases as the complexity of the procedure increases. The previous study has also reported that complex procedures such as peripheral nerve blocks, central line insertion, and epidural block are more difficult to learn than basic manual skills (e.g., endotracheal intubation).[5] This point needs to be considered when formulating the contents of the residency program. Early exposure to complex procedural interventions may lead to failures and protocol violations, but introduction of residents at a very late stage makes them less susceptible to changes and improvements. Thus, the 2nd year of residency may be considered as the “golden period” and should be utilized for improvement of complex procedural skills.

We used two different assessment tools because each has its merits and demerits. An advantage of checklists is that they have intrinsic content validity.[67] The use of checklists has been shown to be inappropriate in higher levels of experience and more complex skills.[8] Another problem with checklists is that as all steps of procedure are weighted equally regardless of clinical importance, a trainee might obtain a high score, despite omitting important steps of procedure. Advantage of GRS is that they are not confined to one procedure but can be used for different procedural skills. As the GRS has a gradation of response in each category, it is less objective than a checklist, although this allows the assessment to be more qualitative.[9] Potential pitfalls with GRS include the “halo effect,” when good or bad performance in one domain unduly influences the grading of performance in other domains.[10]

Basic science and clinical knowledge are examined on a routine basis using written and oral examinations, but assessment of procedural skill is often neglected. Lack of a uniform objective method could be one of the factor.[23] Task-specific checklists and GRS when used for assessment of residents performing an interscalene brachial plexus block and axillary block have reliably discriminated between different levels of training and are thus valid measures of performance.[611] Cumulative sum analysis has been found to be an effective tool for measuring the competence of anesthetic trainees for practical procedures.[1213] All above-mentioned studies have emphasized on assessment of procedural skills, but in our study, we have also shown the improvement in performance after training using the above-mentioned assessment tools.

Cumulative sum analysis is an alternative tool to assess an individual's performance during the conduct of various procedural skills.[121314] Problems with using cumulative sum analysis (CUSUM) analysis is that it is a statistical method that looks at the outcome rather than at the process of performing procedural skills and there are no nationally agreed definitions for success or failure for a given procedure, and those used in the literature vary greatly.[10]

We recruited residents after the performance of minimum twenty procedures because Konrad et al.[5] have demonstrated that the learning curves reveal a marked improvement of skill after twenty attempts. Learning manual skills is a multimodal function depending on many variables[15] and varies from individual to individual. Attaining a prescribed number of procedures lacks validity and may not guarantee competence as trainees might have learnt incorrect technique and could continually perform techniques incorrectly.[16] It is clear, though, that there is a wide spectrum of learning curves, and consequently, the only way to guarantee competency is to tailor training to the individual rather than to focus on minimum numbers.

One of the limitations of our study was that the data collection was done from only one center; the learning situation can vary greatly from institute to institute and thus the components of checklist for various procedures may not be the same. We assessed the residents only one time both before and after training; ideally, residents should have been assessed three times and final score would have been mean of the three scores.

Conclusion

Use of PSC and GRS scores should be incorporated for the assessment of procedural skills of residents. The same should be used to monitor the impact of various corrective measures on the conduct of a procedural skill by the resident. Intensive supervision and quality control are least effective for residents in the last years of residency. Periodic review of the curriculum of the residency program and necessary changes can be done based on the results of assessment using PSC and GRS.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.