Training Pulmonary Critical Care Medicine Fellows in Thoracentesis Using a Head-Mounted Video Camera.

BACKGROUND: Determination of competence to perform procedures during pulmonary critical care medicine fellowship training has traditionally been based on subjective faculty opinion and numerical requirements.
OBJECTIVE: To describe an objective means of assessing competence of fellows to perform thoracentesis using a head-mounted video camera with offline scoring of the thoracentesis performed on an actual patient.
METHODS: To test competence in performance of thoracentesis after a multimodality training program, a total of eight first-year fellows performed a thoracentesis on an actual patient while recording the procedure with a lightweight head-mounted video camera in 2017 and 2018. The recordings were scored offline by two faculty members using a 30-point checklist. The percentage agreement between scorers was measured, as was the opinion of the fellows and the scorers on the testing process. If a fellow failed completion of all checklist items, they were provided with further training and retested to assure competence. As part of their training, fellows reviewed the video record of their procedures.
RESULTS: Eight first-year fellows were tested, of whom seven successfully completed key checklist items as determined by the video scorers. One failing fellow passed after further training and testing. The percentage agreement between the scorers was high, and fellows indicated that the video device was useful for training.
CONCLUSION: This study supports the use of video-based testing for assessment of competence and for training in performance of thoracentesis by fellows.

The American College of Graduate Medical Education (ACGME) guidelines for training require that pulmonary critical care medicine (PCCM) fellows achieve competence to perform thoracentesis (1). The ACGME has not defined criteria for establishing competence in this procedure, specifying that fellows “demonstrate technical skill . . . and maximize patient comfort and safety when performing procedures.” (2) Traditionally, fellows have been trained following the apprenticeship model, whereby competence has been based on subjective faculty opinion and numerical requirements. There is an uncertain relationship between numbers of procedures performed and actual competence (3).

The American Board of Internal Medicine recommends that procedural training be conducted using simulation (4). Studies have utilized the concept of mastery learning during simulation for procedural training (5–9). As simulation does not replicate the actual work environment, testing for competence should take place during the procedure on an actual patient. There is growing use of video recordings in the medical field to enhance training, education, and quality assurance (10–14). Several studies report on the use of video recording of surgeries, trauma resuscitation, and airway management for training and quality purposes (14–23). The present study describes video recording as part of a program to train PCCM fellows to perform thoracentesis.

Methods

This study was conducted at Northwell Health by the Division of Pulmonary, Critical Care, and Sleep Medicine of the Donald and Barbara Zucker School of Medicine at Hofstra-Northwell, which has a 3-year ACGME-accredited PCCM fellowship program with 13 fellows.

First-year fellows starting in July 2017 and July 2018 were the subjects of this study. All fellows signed informed consent to participate in the study, which was approved by the Northwell Health Institutional Review Board (Institutional Review Board #17-0185).

Training in thoracentesis occurred during fellowship orientation in the first 2 weeks of July 2017 and July 2018.

Pretraining Assessment

The fellows took a written test consisting of 10 multiple choice questions assessing knowledge of thoracentesis. Fellows were surveyed regarding their confidence to perform thoracentesis. Both knowledge tests and surveys were developed by the program director (PD) and faculty members who were involved with the project.

Training

The fellows attended a didactic lecture on pleural anatomy, sterile technique, and procedural elements of thoracentesis. They reviewed the thoracentesis checklist, which was designed to capture key safety and procedural steps (Figure 1), and watched a video demonstrating thoracentesis (24). Following this, they engaged in deliberate practice of thoracentesis using an ultrasonography capable task trainer. The task trainer (Blue Phantom) was used to familiarize the fellows with the performance of thoracentesis (identification of safe site, preparation of the sterile field, setting up the equipment, and performance of needle insertion with aspiration of fluid under the direct guidance of a faculty member). Deliberate practice with the task trainer occurred within a 2-hour session devoted to thoracentesis training before the performance of the procedure on a patient.

Figure 1.

Thoracentesis checklist. INR = international normalized ratio; PT = prothrombin time; PTT = partial thromboplastin time.

Testing Procedure

The fellows were tested using video recording during the performance of a thoracentesis on a patient. All patients or their appropriate surrogate gave informed consent for the procedure. The standard hospital consent form included specific consent for videotaping for training and/or education purposes, and the videotaping was explicitly discussed with the patient or surrogate prior to the procedure.

First-year fellows underwent video recording of thoracentesis at two separate times.

Time one.

Fellows underwent video recording before achieving self-perceived mastery of the procedure. The purpose of the first video recording was to familiarize the fellows with wearing the video camera and to use the recording for debriefing the fellow after the procedure.

Time two.

After completing at least five thoracenteses under the direct supervision of a faculty member, the fellow was retested when they indicated self-perceived mastery to perform the procedure. Readiness to undergo testing during the second video recording had to be confirmed by a faculty attending.

Video Recording of Procedure

Before starting the procedure, a small video camera (GoPro Hero5) was attached to the forehead of the fellow using a lightweight attachment device with angling of the camera for optimal image acquisition. The camera was turned on at the beginning of the standard time out and turned off after the post-procedure check for pneumothorax using ultrasonography. A supervising attending was present at the bedside throughout the procedure. At time two, the bedside attending was not permitted to guide the fellow unless required to assure patient safety. The procedure was scored by the bedside attending using the checklist (Figure 1).

Post-training Assessment

At the time of final testing, first-year fellows completed a post-training knowledge test, which was identical to the pretraining test, and they were surveyed regarding their confidence to perform thoracentesis. After reviewing the final video with the PD, they were surveyed on the use of video recording in the training process. The video scorers were surveyed regarding the use of video in the training process. All surveys were based on a 5-point Likert scale.

Video Scoring

Two separate faculty members used a checklist that captured 30 visual and/or audible elements of the procedure to review the video recording of the procedure (Figure 1). These faculty members were not the supervising faculty during the actual procedure. The checklist was developed in an iterative fashion by PCCM fellows and faculty and was available to the fellows for review for several months before final testing. The checklist was designed to provide explicit guidance to the fellows to achieve mastery of the performance of thoracentesis. The two video scorers independently reviewed and scored each video recording. Interrater agreement between the faculty score of the video record and the bedside attending score was determined.

Determination of Competence

Fellows were determined to be competent after final testing if all scorable elements of the checklist were performed correctly. There were 23 out of 30 procedural elements that could be scored from the video record. If the video scorers disagreed on a checklist item, the video was reviewed by the fellowship PD who made final determination as to whether the item was completed correctly. Fellows that did not complete all scorable items successfully were provided with further training and retested at a later date.

All video recordings were saved in a double-password–protected hospital-based hard drive that was compliant with Health Insurance Portability and Accountability Act standards. Access to the video files was limited to the PD and the two scorers. Surveys, knowledge tests, and checklists for each procedure were collected and managed using REDCap electronic data capture tools hosted at Northwell Health (25). After scoring, the video record was permanently erased, except for several short deidentified clips kept for training purposes.

Statistics

Composite scores for the confidence assessment, knowledge test, training survey, and reviewer survey were computed by summing, with higher scores indicating greater endorsement on the confidence assessment and survey perceptions and better performance on the knowledge test. Paired sample t tests were used to examine pre- and posttest differences for the fellows. The percentage agreement between video scorers and between video scorers and the bedside attending was computed.

Results

Eight first-year fellows completed training, with four fellows starting in July 2017 and four fellows starting in July 2018. Each first-year fellow wore the video camera during a patient procedure at least twice. One fellow had to repeat testing and wore the camera three times.

Seven of the eight first-year fellows were determined to have successfully completed all scorable items of the checklist (Table 1). For one fellow out of seven, the video scorers disagreed on whether the rib had been palpated during the procedure. For the second fellow, the video scorers disagreed on whether the fellow queried the patient about chest pain. The PD refereed the disagreements and determined that the steps had been successfully completed.

Table 1.

Thoracentesis checklist scores comparing video scorers with bedside attending

PCCM Fellow	Video Scorer 1	Video Scorer 2	Bedside Attending
1	23	22	NA
2	22	23	23
3	21	21	23
4	23	23	22
5	23	23	23
6	23	23	23
7	23	23	23
8	23	23	23

Definition of abbreviations: NA = results not available; PCCM = pulmonary critical care medicine.

Scores are derived from the 23 items that were scorable from the video record.

Based on video scoring, one fellow failed because of incorrect manipulation of the stopcock and failure to check for lung sliding after thoracentesis. The bedside attending indicated that these steps had been successfully completed. The fellow reviewed the video with the PD, which clearly identified the errors, and repeated the procedure. This did not require retraining using the task trainer; proper manipulation of the stopcock from the thoracentesis kit was reviewed with the fellow. After additional testing, the fellow was determined to be competent.

The percentage agreement among all scorers was high. Video scorers 1 and 2 agreed 96.3% of the time when completing the checklist at time 2. Agreement for video scorer 1, video scorer 2, and the bedside attending was 95.6%. Comparing scores before and after training, knowledge increased from 6.75 to 9.13 [t(7) = −2.68, P = 0. 03; Table 2] and confidence increased from 36.88 to 48.13 [t(7) = −7.83, P < 0.001].

Table 2.

Thoracentesis knowledge test scores

PCCM Fellow	Pretraining Knowledge Test	Post-training Knowledge Test
1	7	10
2	9	8
3	3	9
4	8	10
5	8	9
6	9	10
7	2	8
8	8	9
Mean	6.75*	9.13*

Correctly answered questions out of exam total of 10.

P = 0.03.

All but one fellow indicated that wearing the video camera did not interfere with their performance of thoracentesis. All fellows indicated that video review provided valuable feedback and improved their confidence and skill to perform the procedure (Table 3). Both fellows and the video scorers indicated that video review is an objective means of assessing competence to perform thoracentesis. Video scorers found the checklist helpful in scoring the videos and indicated that errors could be readily identified (Table 4). No patient or nonmedical support staff raised any concern about the video recording. Consent for video recording did not require extra time, as the standard consent form for thoracentesis included consent to perform video recording. Faculty time required for review of each video recording was approximately 30 minutes. Attaching the head-mounted video recording device, correctly angling the camera, and initiating the recording took approximately 1–2 minutes. The use of the video recording device did not interfere with the sterility of the procedure or use of personal protective equipment. The cost of the recording device was $420.00 US dollars.

Table 3.

Fellow survey results regarding use of the video camera

Survey Statement	PCCM Fellow Responses								Mean ± SD
	1	2	3	4	5	6	7	8
Wearing a head mounted video camera did not interfere with my performance of thoracentesis.	5	5	2	5	5	4	5	3	4.25 ± 1.16
Reviewing the video of my performance of thoracentesis provided valuable feedback.	5	4	3	5	4	5	3	3	4.00 ± 0.93
Reviewing the video of my performance of thoracentesis improved my confidence in performing the procedure.	5	4	3	4	4	5	5	3	4.13 ± 0.83
Reviewing the video of my performance of thoracentesis improved my skill in performing the procedure.	5	4	3	3	4	5	5	3	4.00 ± 0.93
Video review of thoracentesis is an objective means of assessing competence to perform the procedure.	5	4	4	5	4	5	5	3	4.38 ± 0.74

Likert scale key: 1, strongly disagree; 2, disagree; 3, neutral; 4, agree; 5, strongly agree.

Table 4.

Scorer survey results regarding use of the video camera

Survey Statement	Response Score* (n = 2)
Wearing a head-mounted video camera did not impair fellows’ ability to perform thoracentesis.	5.0
Reviewing the video was helpful when completing the checklist of fellow performance.	5.0
I could readily identify errors when reviewing the videos.	4.5
Review of the video is an objective means of assessing competency to perform the procedure.	5.0
The review of the video allows determination of competence with a high level of confidence.	4.5

Scores represent mean responses and are based on a 5-point Likert scale, with 1 = strongly disagree, 3 = neutral, and 5 = strongly agree.

Discussion

Our results demonstrate that video recording of thoracentesis performed by PCCM fellows on actual patients with offline scoring is a feasible way to determine competence. To our knowledge, this is the first study to assess PCCM fellows’ competence to perform thoracentesis using video recording on actual patients.

There were several benefits to reviewing the video record. Fellows indicated on the postsurvey that the ability to review the video of their performance provided valuable feedback. The availability of a video record of the procedure allowed the PD to personally assess each fellow’s procedure. Review of the video recording allowed for objective assessment of thoracentesis on an actual patient. Another advantage of video scoring is that it reduces reliance on numerical goals that are often used to establish competence.

Review of the video record led to specific improvements in performance of the procedure. In a pilot study, we identified that the sterile drape in the thoracentesis kit was too small. The video recording demonstrated several examples of violations of the sterile field owing to small drape size. This resulted in a requirement that the procedure be performed with a full-body sterile gown and large surgical drape. Our final checklist incorporated this change.

From a practical point of view, the video device was low cost and easy to set up. Only one fellow indicated that the video device hindered performance of thoracentesis. No patient or team member had any objection to the use of the video recording device, and all patients gave informed consent to the recording as part of consent for the thoracentesis.

For fellowship faculty who are interested in video recording for formative development of skill at thoracentesis and summative assessment of competence, there are barriers to its use. Although equipment costs are negligible and video recording does not add much time to the procedure, scoring of the video record is time intensive and requires significant faculty commitment. Data storage and analysis should be Health Insurance Portability and Accountability Act compliant, and we recommend that video scoring be classified as a quality-improvement activity. Our policy is to delete the video record after its review with the fellow and formal scoring.

The fellows had to perform all components of the checklist before they were considered competent to perform thoracentesis, and all components were considered equally important. The fellow could not refer to the checklist at the time of the procedure, as they were expected to have committed it to memory, thus demonstrating full mastery of the multiitem checklist.

This study has several limitations. First, it is a single-center study; methods and results may not be transferable to another institution. Second, the study was not designed to measure the effectiveness of the training program, which included video recording, as compared with traditional experiential training. The number of subjects was too low for a meaningful statistical comparison given the small size of the training group. Third, the determination of competence using video recording occurred at a single point in time and does not necessarily indicate that subsequent procedures would be performed with equal skill. Fourth, the video recording was not used to assess all components of the checklist. Practical constraints made it difficult to record such elements as the consent process, review of laboratory values before the procedure, and handwashing. Fifth, the checklist was developed by consensus opinion of the faculty and fellows. We recognize that other clinical groups may choose to develop their own checklist. Sixth, the scores of the bedside attending and the offline video scorers were similar. The scoring by the bedside attending may have been altered by the knowledge that they were involved in a research study (Hawthorne effect). The advantage of determining competence using video recording lies in the objective nature of the method. In a routine clinical environment, we envision that the supervisory attending might be influenced by a variety of subjective factors in determining the competence of the trainee. Direct visual scoring at the bedside may introduce antecedent personal bias by the attending at the time of scoring. A related limitation is that the attendings and the PD who scored the video recordings may have been biased by antecedent knowledge of the fellow’s performance gained during other aspects of training. Ideally, the video scorer should have had no knowledge of the fellow at the time of the scoring. Seventh, there was no control group that did not receive the training sequence described in this study. Because of the small number of fellows, it was not feasible to have a control group. This is a drawback of a single-center study.

Conclusions

We describe the use of video recording as part of a multimodality program to train fellows in the performance of thoracentesis. We propose that video scoring of procedures on actual patients provides an objective means of assessing skill and can be utilized for training purposes and determination of competence.