Supervising the supervisors--procedural training and supervision in internal medicine residency.

BACKGROUND: At teaching hospitals, bedside procedures (paracentesis, thoracentesis, lumbar puncture, arthrocentesis and central venous catheter insertion) are performed by junior residents and supervised by senior peers. Residents' perceptions about supervision or how often peer supervision produces unsafe clinical situations are unknown.
OBJECTIVE: To examine the experience and practice patterns of residents performing bedside procedures. DESIGN AND PARTICIPANTS: Cross-sectional e-mail survey of 653 internal medicine (IM) residents at seven California teaching hospitals. MEASUREMENTS: Surveys asked questions in three areas: (1) resident experience performing procedures: numbers of procedures performed and whether they received other (e.g., simulator) training; (2) resident comfort performing and supervising procedures; (3) resident reports of their current level of supervision doing procedures, experience with complications as well as perceptions of factors that may have contributed to complications.
RESULTS: Three hundred sixty-seven (56%) of the residents responded. Most PGY1 residents had performed fewer than five of any of the procedures, but most PGY-3 residents had performed at least ten by the end of their training. Resident comfort for each procedure increased with the number of procedures performed (p < 0.001). Although residents reported that peer supervision happened often, they also reported high rates of supervising a procedure before feeling comfortable with proper technique. The majority of residents (64%) reported at least one complication and did not feel supervision would have prevented complications, even though many reported complications represented technique- or preparation-related problems.
CONCLUSIONS: Residents report low levels of comfort and experience with procedures, and frequently report supervising prior to feeling comfortable. Our findings suggest a need to examine best practices for procedural supervision of trainees.

INTRODUCTION

Given the variability in the types and numbers of procedures performed by internists in practice, the American Board of Internal Medicine (ABIM) rescinded numeric procedural requirements in July 2007 in favor of residents being able to “recognize indications, contraindications and manage complications” of selected bedside procedures1. ABIM’s change in policy was aimed to limit the practice of “learning by doing,” and encourage the the competency-based education and assessment now recommended by the ACGME2,3.

However, at many of the nation’s academic hospitals, bedside invasive procedures, such as paracentesis, thoracentesis, lumbar puncture and central venous catheter (CVC) insertion, continue to be performed by trainees with widely variable experience and skill4. Evidence suggests more procedural experience is associated with reduced complications5–8, perhaps leading graduating Internal Medicine residents to report feeling comfortable performing inpatient bedside procedures9–11. However, few articles have examined residents’ experiences and perceptions around supervision during the years of their training, or how often peer-to-peer supervision may produce unsafe (or at least uncomfortable) clinical situations.

To better understand residents’ experiences with learning bedside procedures, we carried out a cross-sectional survey of medicine residents in a network of training hospitals in California. The aim of this study was to examine trainees’ perceptions of procedural training during their residency, comfort performing procedures, their perceptions of supervision as well as potential contributing factors to reported complications.

METHODS

Sites and subjects We surveyed trainees at seven teaching institutions participating in the Medical Education Research Network (MERN) of California. MERN is comprised of nine internal medicine residency programs from SGIM’s California and Hawaii Region, who meet annually to collaborate on educational research initiatives. MERN institutions include one community-based (Kaiser Permanente Medical Center-Oakland) and six university-based teaching institutions [Stanford, University of California (UC) Irvine, UC Los Angeles, UC San Diego, UC San Francisco and University of Southern California]. Participating MERN sites had ABIM board pass rates slightly above national averages, with MERN institutions averaging a 96% pass rate (91–100%) compared to a national average of 92%12. At the time of our survey, no MERN site had a didactic procedure service, and only three programs provided systems to track procedure completion and supervision. However, use of these systems was not required, and participation was low as a result. For example, at one site only 32% of all IM residents used the system in the year of our study. This study was reviewed and approved by the IRB and Committee on Human Research in each participating institution. Using rosters provided by the sites, we identified 653 first, second or third year Internal Medicine residents. We excluded preliminary residents and medicine-pediatrics residents.

Survey development The survey used in this study was a refined version of a survey we piloted at UCSF in 2007. Our 2007 survey was developed by the study authors including the associate program director responsible for meeting residency core competencies. The survey asked basic demographic information in addition to questions regarding procedure frequency, resident reported comfort and experience with complications for paracentesis, thoracentesis, lumbar puncture and CVC insertion. These procedures were selected from the procedures listed in the ABIM’s Policies and Procedures for Certification1 based on their frequency and risk for complications2,13–16. We enhanced our 2007 survey by adding questions focused on the following areas: (1) experience with performance of these procedures on a simulator; (2) comfort performing and supervising these procedures; (3) whether the procedures were supervised and how supervision was perceived; (4) complications, if any, and whether the resident felt there were any obvious contributing causes. Complications were defined by those most commonly described in the literature14–17.

All survey questions underwent a formal review for content validity by the UCSF Department of Medical Education. After the formal review, clarity and comprehension of questions were finalized by pretesting among a group of attending physicians and residents as well as the investigative team. (See online Appendix.)

Survey process Our survey was distributed via e-mail invitation at the end of the 2008 academic year, and responses between May 2008 and July 2008 were included. Each survey e-mail was accompanied by a note describing the purposes of the study and that results would remain anonymous and be used for research purposes only. Up to three reminder e-mails were sent to non-respondents, with an option to decline to receive additional automated reminders with each e-mail.

Participants were specifically asked to reflect on procedures performed in non-emergent situations and to estimate the total number of common procedures [paracenteses, thoracenteses, lumbar punctures, femoral central venous catheter insertions and internal jugular (IJ)/subclavian CVC insertions] they had performed during training.

Data Analyses

We first characterized survey respondents using simple univariate and bivariate statistics, which we used to determine frequencies of responses to questions in our major domains. In order to replicate previous studies, we determined a ‘comfort threshold,’ defined as the reported case volumes where three-fourths of residents agreed to being comfortable performing a procedure independently2,18,19.

We used multivariate logistic regression methods to determine factors associated with reaching the ‘comfort threshold.’ Potential factors chosen from our demographic survey and entered into models manually included: trainee site, gender, career aspiration in an invasive subspecialty, residency year, report of a complication, experience on a simulator and increasing numbers of procedures performed. Factors were retained in the model if they remained associated with the outcome of interest at p < 0.05 or lower. All statistical tests were performed using the SAS statistical application program (release 9.2).

RESULTS

Respondent Characteristics Our survey response rate was 56% (367/653) and ranged from 31–72% across sites. Respondents were evenly distributed across residency year; differences in gender and career aspirations were similar to national residency profiles20. Responders and non-responders did not differ significantly in gender or residency year, but response rate by site was a significant source of responder variation (Table 1).

Table 1

Respondent Characteristics (n = 367)

	Value (%)
Resident year, n (%)
PGY-1	126 (34%)
PGY-2	119 (32%)
PGY-3	122 (33%)
Training site, n (%)
Kaiser Permanente Medical Center, Oakland	20 (5%)
Stanford	53 (14%)
UC Irvine	23 (6%)
UC Los Angeles	59 (16%)
UC San Diego	39 (11%)
UC San Francisco	99 (27%)
University Southern California	74 (20%)
Response rate, n (%)
Kaiser Permanente Medical Center, Oakland	20/29 (69%)
Stanford	53/74 (72%)
UC Irvine	23/74 (31%)
UC Los Angeles	59/90 (66%)
UC San Diego	39/68 (57%)
UC San Francisco	99/156 (63%)
University Southern California	74/162 (46%)
Male, n (%)	212 (58%)
Career aspirations, n (%)
Non-clinical	6 (2%)
General practice	51 (14%)
Hospitalist	77 (21%)
Non-invasive subspecialty	85 (23%)
Invasive subspecialty	148 (40%)
Satisfied with supervision for paracentesis/thoracentesis/lumbar puncture, n (%)	331 (90%)
Satisfied with supervision for central venous catheter insertion, n (%)	328 (90%)

Procedure Volume The median number of procedures reported by each resident increased with each successive postgraduate year. Most residents had performed fewer than five of each procedure at the end of the PGY-1 year, but had performed at least ten for nearly every procedure by the end of their training. Thoracentesis was consistently the least often performed procedure. The range of experience for each procedure was wide, demonstrating a variable experience of residents even within the same program. Residents reported little experience with procedure simulation, with only 4% of residents having experience with simulation for paracentesis, thoracentesis and lumbar puncture, and 21% having experience with CVC insertion simulation (Table 2).

Table 2

Median Procedure Volume by Resident Year

Procedure type	PGY1 median (range) n = 126	PGY2 median (range) n = 119	PGY3 median (range) n = 122	P value
Paracentesis	4 (0–>10)	7 (1–>10)	10 (3–>10)	<0.001
Thoracentesis	1 (0–8)	4 (0–>10)	7 (0–>10)	<0.001
Lumbar puncture	3 (0–>10)	6 (0–>10)	10 (2–>10)	<0.001
Femoral CVC	1 (0–>10)	5 (0–>10)	10 (0–>10)	<0.001
IJ/subclavian CVC	2 (0–>10)	7 (0–>10)	10 (0–>10)	<0.001

Comfort Performing Procedures Resident comfort increased with the number of procedures performed (p < 0.001, Fig. 1a). More than three-fourths of residents reported being comfortable (our defined “comfort threshold”) with paracentesis, lumbar puncture and femoral CVCs after having performed three to four, thoracentesis after having performed five to six and IJ and subclavian CVCs after seven to nine. Nearly all residents were comfortable after having performed ten procedures. The residency year at which trainees surpass the comfort threshold is seen in Figure 1b. In multivariable models adjusting for PGY year, gender, training site and whether the trainee was planning a procedural specialty, the number of procedures remained strongly independently associated with meeting the comfort threshold. The adjusted odds ratio for each incremental procedure yielding comfort was 2.48 (95% CI: 1.81, 3.40) for paracentesis, 2.31 (95% CI: 1.85, 2.87) for thoracentesis, 2.34 (95% CI: 1.79, 3.05) for lumbar puncture, 2.31 (95% CI: 1.80, 2.96) for femoral CVCs and 1.90 (95% CI: 1.65, 2.20) for IJ or subclavian CVCs, all significant at the p < 0.05 level. Neither pursuit of invasive subspecialty career, nor use of a simulator, nor report of a complication was a significant predictor of comfort. Resident year had only a very limited impact, being only associated with increased comfort for PGY-3s performing femoral CVCs (OR 3.55, CI 1.01–12.53). Interestingly, male gender was associated with a two- to three-fold statistically significant increase in comfort for all procedures except paracentesis (OR 2.60–2.78). Site-specific variations in comfort were also noted, though these differences were based on small numbers of procedures reported within each subgroup at each site.

Figure 1

Resident comfort with procedure performance. (a) Percentage of residents reporting being “comfortable” or “very comfortable” performing each procedure based on numbers of procedures performed. (b) Percentage of residents reporting being “comfortable” or “very comfortable” performing each procedure by resident year.

Resident comfort with procedure supervision. (a) Percentage of residents reporting being “comfortable” or “very comfortable” supervising each procedure based on numbers of procedures performed. (b) Percentage of residents reporting being “comfortable” or “very comfortable” supervising each procedure by resident year.

Procedure Supervision Ninety-eight percent of residents reported peer supervision for or independent performance of paracentesis, thoracentesis and lumbar puncture, while 70% reported peer supervision or independent performance of CVC insertion. By the end of PGY-2 year, more than 60% of residents had supervised thoracencenteses and CVC insertion, and more than 80% had supervised paracentesis and lumbar puncture. By the end of PGY-3 year, more than 80% of residents had supervised all procedures. Satisfaction with the levels of supervision was high for all procedures and did not vary significantly by reported supervisor or by resident year. Despite the large number of residents reporting that their responsibilities include supervision of procedures, comfort supervising was only achieved by the end of the PGY-2 year for lumbar puncture and femoral CVCs, and by the end of the PGY-3 year for thoracentesis and IJ/subclavian CVCs, the latter procedures being those with the highest complication rates14. A sizable proportion of residents reported supervising a procedure prior to feeling comfortable with the procedure themselves (Fig. 2, Table 3).

Table 3

Supervision Practices of Residents

	Number of residents who reported supervising before feeling comfortable with procedure performance
Procedure	PGY-1	PGY-2	PGY-3	All years
	n = 126 (%)	n = 119 (%)	n = 122 (%)	n = 367 (%)
Paracentesis	14 (11%)	50 (42%)	32 (26%)	96 (26%)
Thoracentesis	4 (3%)	29 (24%)	32 (26%)	65 (18%)
Lumbar puncture	9 (7%)	46 (39%)	3 (2%)	58 (16%)
Femoral CVC	4 (3%)	31 (26%)	27 (22%)	62 (17%)
Internal jugular CVC	2 (2%)	27 (23%)	30 (25%)	59 (16%)
Subclavian CVC	1 (1%)	10 (8%)	12 (10%)	23 (6%)

Complications The majority (64%) of residents’ reported a complication with a procedure (defined as arterial puncture, arterial hematoma, cerebrospinal fluid leak requiring intervention, epidural hematoma, guide wire/catheter tip embolism, hemorrhage, hemothorax, inadequate analgesia, organ puncture, pneumothorax, venous hematoma). Of the 439 complications reported, 140 (32%) were instances of inadequate analgesia, 109 (25%) arterial puncture, 69 (16%) venous hematoma, 49 (11%) pneumothorax, 30 (7%) arterial hematoma, 18 (4%) CSF leak requiring intervention, 7 (2%) each of hemothorax and hemorrhage, and ≤4 (≤1%) each of bowel perforation, epidural hematoma, guide wire embolism and death. Eighty-eight percent of these residents felt that the complication could have been avoided.When asked to select factors that would have prevented complications, 89% of residents felt that that quality of supervision had no role in avoiding the complications. The most commonly reported factors that could have avoided complications were better analgesia (20%), better patient positioning (15%), use of ultrasound (11%), better training in proper procedure technique (10%) and better characterization of anatomy (10%). Those who reported a complication had no differences in their comfort performing or supervising a procedure.

DISCUSSION

In this multicenter, cross-sectional survey, residents’ procedural experience and comfort were low in early and mid-level trainees and higher in more senior trainees. However, many residents reported supervising procedures they felt uncomfortable performing, and most reported involvement in at least one procedure with a complication. Residents felt that most complications would not have been prevented with improved supervision, even though many reported complications represented issues stemming from inadequate training or supervision (e.g., positioning, use of ultrasound).

While the vast majority of respondents were satisfied with the level of supervision, a large number of the supervisors in our study admitted to supervising prior to being comfortable with procedure technique. It is possible that difficulties providing criticism of peers and lack of qualified role models may underlie the disconnect between satisfaction with peer supervision and ‘premature’ supervisory roles. All institutions lacked formal competency assessment to ensure residents had mastery of procedure technique before attempting independent performance and supervision; lack of competency-based training would also potentially exacerbate ‘premature’ supervisory roles. Finally, peer-to-peer supervision and lack of qualified supervisors may have also limited residents’ level of insight into the causes of complications. Even though residents reporting experiencing complications amenable to supervision (in which case the supervisor could assist in overcoming the challenge) or training (where knowledge and self-efficacy could be maximized prior to the procedure), they rarely selected inadequate supervision or training as a reason for a complication21. Learning primarily from inexperienced peers, residents may have a poor understanding of the role of supervision not only in stepwise instruction, but also in anticipating and troubleshooting potential complications.

Our findings are particularly important given national trends that suggest Internal Medicine residents are performing fewer bedside procedures22–24. The number and variety of bedside procedures performed by board-certified internists are falling as well13. As a result, Internal Medicine residency programs are facing a lack of qualified trainees and supervisors for invasive bedside procedures. A first step in rectifying this shortage will be to rethink the traditional clinical practice of “learning by doing.” To this end, MERN participating sites have begun employing results from this survey to justify systems and support to provide increased procedural training. All seven institutions have web-based applications to track procedure completion, though three report suboptimal compliance by residents. Four institutions now require that a supervising resident have completed a requisite number of supervised procedures prior to independent performance or supervising others. Five institutions have incorporated simulation in their resident procedures training (both high and low fidelity), and two institutions have instituted a didactic procedure service at one of their hospital sites.

Simulation technology provides potentially useful approaches to ensure that residents become comfortable performing and supervising common procedures before performance in clinical settings25,26. Recent literature suggests that simulation not only aids in the development of the clinical knowledge and skills necessary for safe performance27,28, but also translates into measurable improvements in patient care29,30. At places with less established simulation technology, procedure services can also provide residents with the didactic teaching, opportunities for deliberate practice and supervision during procedures. Use of procedure services for resident education has been shown to increase resident comfort and self-perceived knowledge with a low rate of complications23,31,32.

This study has several limitations. As a cross-sectional survey, our data are a snapshot of a population of trainees who were at the end of a year of training and were therefore at a standard point, but who were asked to recall their experiences. A major limitation of our study is that resident’s procedure experience is determined by self-report alone. Residents who are comfortable with a procedure might overestimate the number of procedures they performed, whereas those uncomfortable may be more likely to minimize their experience. Existing tools available at the time of the study to track procedure completion provided limited and potentially unreliable results. Additionally, because we could not observe residents directly, our survey used reported level of comfort as a surrogate for competency. Competency refers to a resident’s ability to safely prepare for, perform and navigate the complications of a procedure, whereas comfort is likely mitigated by performer characteristics and may not correlate with observed performance33. The finding that women were less comfortable for most procedures might possibly reflect a published gender difference in comfort with technically invasive procedures rather than any difference in ability, and demonstrates the limitations of self-assessment34–36. Our survey cannot account for differences between responders and non-responders in procedural experience and comfort, and is likely subject to non-response bias. Though responders did not differ from non-responders in resident year and gender, plausibly those who were more experienced in procedures or more satisfied with their procedural experience and training may have not responded. Additionally our response rate is highly variable by site, leaving several sites underrepresented. We studied comfort threshold as a measure of a required procedure volume. While our threshold was chosen based on previous studies, in our data the cutoff left one-fourth of participants feeling uncomfortable. Our study was performed primarily at university-based teaching centers and may not represent the experiences at community-based teaching hospitals, or procedures performed by non-trainees. Lastly, as a cross-sectional study, we cannot establish a causal link between procedure volume and reported comfort performing procedures.

CONCLUSION

Even as certification requirements change, residents continue to face situations in which they are expected to perform these procedures supervised by inexperienced, uncomfortable and perhaps under-qualified residents. At the institutions we surveyed, many residents are still performing procedures, supervising procedures prior to being comfortable performing them and report a high frequency of complications. Advances in simulation and the advent of didactic procedure services offer options to residency programs seeking to provide effective procedural education and increase patient safety.

Below is the link to the electronic supplementary material.

(PDF 20 kb)