Cost of a learner in pediatric ED.

Background: Experiential learning in a cornerstone of medical education, but impacts throughput, utilization of resources and patient outcomes.
Objectives: Our study sought to determine the cost of a resident across various throughput, utilization, and patient outcome measures.
Methods: This retrospective study was conducted in 2016 in the pediatric emergency department of an urban tertiary care hospital. . We compared various throughput, utilization and patient outcome measures between resident-covered and nonresident-covered patients. A subgroup analysis was performed based on complexity as determined by CPT codes.
Results: 33,278 patient encounters occurred between 1 January 2016, and 31 December 2016. Of these, 8,434 (25.42%) were resident-covered patients. Across all encounters, throughput, utilization and patient experience measures were unfavorable for the resident covered group. In subgroup analysis based on complexity of patients, throughput measures were either unfavorable or there was a trend towards unfavorability across all complexities for the resident covered group. Overall utilization and patient outcome measures were unfavorable in low and moderate complexity patients for the resident covered group. In high complexity patients, most of the utilization and patient outcome measures were similar in both groups.
Conclusion: Presence of a resident led to unfavorable increases in many throughput, utilization and outcome measures, a difference which disappeared in most cases with higher complexity patients. Therefore, the cost of a resident may actually decrease with increasing patient complexity.

Introduction

Experiential learning remains a cornerstone of medical education, yet teaching time is challenged by focuses on direct revenue-generating activities and throughput [1]. To balance academic and revenue generating priorities-while aspiring to a just allocation of limited resources-the academic medical community needs to understand the true cost of learners on productivity [1].

Most studies have examined the impact of learners on workflow and throughput, e.g., patients per hour (PPH) or relative-value units (RVUs) generated. [2-4] A smaller subset of studies has linked learner presence to metrics of potential costs of care, including number of laboratory tests or radiographs per patient ordered by learners compared to attending physicians [4-6]. However, studies to date have been limited by focusing on a small number of outcomes, reducing the ability to draw broad-based conclusions.

This study aimed to evaluate broad metrics concurrently to assist pediatric emergency department (ED) directors and health system administrators as they create staffing, compensation, and patient experience strategies.

Methods

This retrospective study compared throughput, utilization, and patient outcome measures between resident-covered and nonresident–covered patients. This study was done in the pediatric ED of an urban tertiary care hospital with 33,000 annual visits. The department utilizes several care delivery models via a split flow process, including: pediatric ED attending working alone, pediatric ED attending with advance practice provider (APP), pediatric ED attending with the resident, and APP alone.

Throughput metrics were ED length of stay (time spent by a patient in the ED from arrival to departure), provider to treat and release disposition time (time from arrival to ED until a discharge disposition was entered in the electronic medical record system); provider to admission time (time from arrival in ED until an admission order was entered in the electronic medical record system); and door to provider time (time from patient arrival to ED until they were seen by a provider). Utilization metrics were labs, plain radiographs, CT scans, ultrasounds, and MRIs ordered per 100 visits, respectively. Outcome metrics were admission rate, Return to ED in 72 hours rate and a combined Left before Treatment Complete (LBTC) rate which included patients left without being seen, left without treatment, and left against medical advice.

Patient encounters from 1 January 2016, to 31 December 2016, were included. The data was extracted via a Crystal Report (SAP SE, Walldorf, Baden-Württemberg, Germany), a SQL (structured query language)-based service, which utilizes data drawn from the electronic health record (Epic, Epic Systems Corporation, Verona, Wisconsin, USA) to acquire clinical material, which is then de-identified prior to analysis. Professional billing codes were used to assign RVUs to the encounter. The patient encounter was attributed to the provider of record at the time of discharge, admission, transfer, LBTC, or expiration in the ED.

Comparisons were performed using throughput, utilization, patient outcome and complexity (professional CPT codes) measures. Continuous variables were compared by t-test with bootstrap and categorical variables by chi-square test. Data analysis was performed by SPSS v25.0.0.0 software (International Business Machines, Armonk, New York, USA). This study was approved by the Institutional Review Board of Rochester General Hospital.

Results

A total of 33,278 patient encounters occurred between 1 January 2016, and 31 December 2016. 110 were excluded due to lack of attributable providers, leaving a total of 33,168 encounters in the analysis. Of these, 8,434 (25.42%) were resident-covered patients and 24,734 (74.58%) were nonresident-covered patients.

All encounters

Mean ED LOS (176.63 min versus 134.15 min, P < 0.05), provider-to-treat-and-release disposition time (136.02 min versus 114.05 min, P < 0.05), provider-to-admission time (242.35 min versus 216.48 min, P < 0.05) and door-to-provider time (19.13 min vs. 18.62 min, P < 0.05) were higher in the resident-covered patients. Resident-covered patients had higher utilization of CT scans (4.8/100 visits versus 2.8/100 visits, P < 0.05), laboratory testing (26.5/100 visits versus 17.7/100 visits, P < 0.05) and ultrasound (9/100 visits versus 5/100 visits, P < 0.05). Resident covered patients had lesser utilization of plain films (24 plain films/100 visits versus 26 plain films/100 visits, P < 0.05). There was no difference in utilization of MRIs between the groups (0.15 MRI/100 visits versus 0.09 MRI/100 visits). Admission rate (5.8% versus 3.6%, P < 0.05), return to ED in 72 hour rate (4.2% vs. 3.6%, P < 0.05) and LBTC rate (0.9% vs. 0.7%, P = 0.038) were higher in resident-covered patients. RVUs generated per patient encounter for resident-covered patients were higher (2.77 vs. 2.50, P < 0.05) (Table 1).

Table 1.

All CPT codes (n = 33,168).

Metric	Mean and SD in minutes With Resident (n = 8434)	Mean and SD in minutes Without Resident (n = 24,734)	P value	95% CI for mean difference
Throughput Measures
ED length of stay	176.63 (120.36) (n = 8434)	134.15 (100.72) (n = 24,734)	0.001	39.81 −44.87
Provider to treat-and-release disposition	136.02 (101.46) (n = 8284)	114.05 (87.46) (n = 23,322)	0.001	19.46–24.48
Provider to admission time	242.35 (146.65) (n = 401)	216.48 (133.73) (n = 712)	0.006	8.89–42.13
Door to provider time	19.13 (15.93) (n = 8430)	18.62 (16.20) (n = 24,724)	0.009	.15 – .90
Utilization Measures
Lab Order/100 visits	26.5% (n = 2233)	17.7% (n = 4385)	<0.001	NA
CT Scans/100 Visits	4.8 (24) (n = 409)	2.8 (19.4) (n = 702)	0.001	1.4–2.5
Plain radiographs/100 Visits	24 (60) (n = 2009)	26 (60) (n = 6370)	0.007	0.5–3.4
MRIs/100 Visits	0.15 (4.4) (n = 13)	0.09 (3.6) (n = 23)	NS	NA
US/100 visits	9 (35) (n = 788)	5 (26) (n = 1276)	0.001	3.3 −5.0
Patient Outcomes Measures
RVU per patient visit	2.77 (1.26) (n = 8434)	2.5 (1.15) (n = 24,734)	0.001	0.24–0.30
Admission Rate including transfer	5.8% (n = 492)	3.6% (n = 896)	<0.001	NA
Combined LBTC rate	0.9% (n = 77)	0.7% (n = 170)	0.038	NA
Returned to ER Within 72 Hours Rate	4.2% (n = 352)	3.4% (n = 839)	0.001	NA

NA = not applicable

Subgroup analyses based on patient complexity

CPT codes were defined as low complexity (99282), moderate complexity (99283) or high complexity (99284, 99285, 99291). CPT code 99281 was excluded from subgroup analysis due to low event rate.

CPT code 99282: ED LOS (90.2 min versus 67.69 min, P < 0.05), provider-to-treat-and-release disposition time (54.62 min versus 35.85 min, P < 0.05) and door-to-provider time (21.97 min versus 18.39 min, P < 0.05) were less favorable for resident-covered patients. Other throughput, utilization or patient outcome measures had very low event rates (Table 2).

Table 2.

Low complexity CPT codes.

CPT 99,282 (n = 874)
Metric	Mean and SD in minutes With Resident (n = 174)	Mean and SD in minutes Without Resident (n = 700)	P value	95% CI for mean difference
Throughput measures
ED Total LOS	90.2 (43.36) (n = 174)	67.69 (39.69) (n = 700)	0.001	15.55–29.44
Provider to Treat-and-Release Disposition	54.62 (33.13) (n = 172)	35.85 (32.63) (n = 677)	0.001	13.39–24.36
Provider to Admission Time	no admissions	no admissions	NA	NA
Door to provider Time	21.97 (16.95) (n = 174)	18.39 (14.68) (n = 700)	0.007	0.98–6.33
Labs order/100 visits	0 (n = 0)	0.1% (n = 1)	NS	NA
Plain films/100 visits	0 (n = 0)	0.02% (n = 2)	NS	NA
Combined LBTC rate	1.1% (n = 2)	1% (n = 7)	NS	NA
Return to Ed in 72 hours rate	1.1% (n = 2)	1.7% (n = 12)	NS	NA

NA = not applicable

CPT code 99283: patients covered by residents had a longer ED LOS (124.94 min vs. 93.57 min P < 0.05), and provider-to-treat-and-release disposition time (89.38 min vs. 61.28 min P < 0.05). There were more labs ordered (8.8/100 visits vs. 4.9/100 visits, P < 0.05) and less plain radiographs ordered (7.37/100 visits vs. 12.60/100 visits, P < 0.05) in resident-covered patients. Patients covered by residents had higher admission rate (0.2/100 visits vs. 0.03/100 visits, P < 0.05), return to ED in 72 hours rate (4.1% vs. 3% P < 0.05) and LBTC rate (0.7% vs. 0.4%, P < 0.05). There was no significant difference in door-to-provider time. Admissions rate, CT scan usage, MRI usage, and ultrasound usage were too infrequent for meaningful analysis (Table 3).

Table 3.

Medium complexity CPT codes.

CPT 99,283 (n = 18,633)
Metric	Mean and SD in minutes With Resident (n = 4084)	Mean and SD in minutes Without Resident (n = 14,549)	P value	95% CI for mean difference
Throughput Measures
ED length of stay	124.94 (63.3) (n = 4084)	93.57 (52.74) (n = 14,549)	0.001	29.30–33.56
Provider to treat-and-release disposition	89.38 (56.68) (n = 4017)	61.28 (49.47) (n = 14,306)	0.001	26.14–30.09
Provider to admission time	287.25 (127.20) (n = 4)	242.5 (163.34) (n = 2)	NS	NA
Door to provider time	19.71 (16.46) (n = 4083)	19.32 (16.80) (n = 14,546)	NS	NA
Utilization Measures
Lab Order/100 visits	8.8% (n = 361)	4.9% (n = 711)	<0.001	NA
CT Scans/100 Visits	.05 (3.1) (n = 2)	.04 (2.3) (n = 6)	NS	NA
Plain radiographs/100 Visits	7.37 (29.55) (n = 301)	12.60 (39.27) (n = 1832)	0.001	4.1 to −6.2
MRIs/100 Visits	none	none		NA
US/100 visits	.02 (1.5 (n = 1)	.02 (1.4) (n = 3)	NS	NA
Patient Outcomes Measures
Admit Rate including transfers	0.2% (n = 7)	0.03% (n = 5)	0.007	NA
Combined LBTC rate	0.7% (n = 30)	0.4% (n = 55)	0.004	NA
Returned to ER Within 72 Hours Rate	4.1% (n = 166)	3.00% (n = 437)	0.001	NA

NA = not applicable

CPT code 99284: resident covered patient had higher ED LOS (194.09 min vs. 170.71 min, P < 0.05), provider-to-treat-and-release disposition time (156.51 min vs. 137.87 min, P < 0.05) and provider-to-admission time (249.38 min vs. 177.74 min, P < 0.05). Resident-covered patients also had higher utilization of CT scans (5.86/100 visits vs. 3.95/100 visits, P < 0.05), laboratory test (32.4/100 visits vs. 26.8/100 visits, P < 0.05) and ultrasound (5.1/100 visits vs. 2.65/100, visit P < 0.05). There w as a lower utilization of plain radiographs (42.54/100 visits vs. 52.47/100, visits) in resident covered patients. There was no difference in MRI utilization, admission rate, and return to ED in 72 hours rate and LBTC rates (Table 4).

Table 4.

High complexity CPT codes.

Metric	Mean and SD in minutes With Resident (n = 2903)	Mean and SD in minutes Without Resident (n = 6941)	P value	95% CI for mean difference
CPT 99,284 (n = 9844)
Throughput Measures
ED length of stay	194.09 (100.87) (n = 2903)	170.71 (93.26) (n = 6941)	0.001	18.93–27.57
Provider to treat-and-release disposition	156.51 (91.26) (n = 2862)	137.87 (87.58) (6839)	0.001	14.74–22.63
Provider to admission time	249.38 (107.64) (n = 26)	177.74 (141.47) (n = 50)	0.01	13.27–122.42
Door to provider time	18.53 (15.03) (n = 2902)	17.98 (15.66) (n = 6941)	NS	NA
Utilization Measures
Lab Order/100 visits	32.4% (n = 942)	26.8% (n = 1863)	<0.001	NA
CT Scans/100 Visits	5.86 (27.41) (n = 170)	3.95 (22.55) (n = 274)	0.003	0.08–3.09
Plain radiographs/100 Visits	42.54 (74.61) (n = 1235)	52.47 (83.01) (n = 3642)	0.001	13.4–65.24
MRIs/100 Visits	0.17 (4.14) (n = 5)	0.12 (3.83) (n = 9)	NS	NA
US/100 visits	5.1 (24.99) (n = 149)	2.65 (16.9) (n = 184)	0.001	1.3–3.5
Patient Outcomes Measures
Admission Rate including transfers	1.1% (n = 33)	0.9% (n = 65)	NS	NA
Combined LBTC rate	0.6% (n = 16)	0.7% (n = 47)	NS	NA
Returned to ER Within 72 Hours Rate	5% (n = 145)	4.4% (n = 302)	NS	NA
CPT 99,285 (n = 2866)
Throughput Measures
ED length of stay	327.32 (177.45) (n = 966)	301.35 (143.37) (n = 1900)	0.001	12.82–39.67
Provider to treat-and-release disposition	258.82 (132.02) (n = 947)	247.27 (124.15) (n = 1854)	0.031	1.14–22.61
Provider to admission time	235.01 (150.49) (n = 298)	216.54 (137.07) (n = 519)	NS (0.086)	-−3.18284 to −38.53
Door to provider time	18.68 (15.90) (n = 965)	16.98 (14.43) (n = 1899)	0.006	.54–2.96
Utilization Measures
Lab Order/100 visits	86.3% (n = 834)	85.2% (n = 1618)	NS	NA
CT Scans/100 Visits	21.22 (44.54) (n = 205)	19.00 (43.56) (n = 361)	NS	NA
Plain radiographs/100 Visits	36.23 (80.23) (n = 350)	36.18 (72.25) (n = 687)	NS	NA
MRIs/100 Visits	0.52 (8.50) (n = 5)	0.68 (10.49) (n = 13)	NS	NA
US/100 visits	63.87 (73.18) (n = 617)	56.68 (68.86) (n = 1077)	0.006	1.6–12.68
Patient Outcomes Measures
Admission Rate including transfers	35.9% (n = 347)	33.4% (n = 635)	NS	NA
Combined LBTC rate	0.9% (n = 9)	0.8% (n = 16)	NS	NA
Returned to ER Within 72 Hours Rate	3.3% (n = 32)	3.7% (n = 70)	NS	NA
CPT 99,291 (n = 787)
Throughput Measures
ED length of stay	299.70 (149.63) (n = 269)	277.87 (129.54) (n = 518)	0.032	1.15–41.07
Provider to treat-and-release disposition	239.02 (114.88) (n = 267)	223.46 (104.96) (n = 505)	NS (0.054)	0.41 −31.74
Provider to admission time	269.13 (142.97) (n = 72)	229.65 (115.70) (n = 141)	0.044	3.42–77.97
Door to provider time	15.17 (13.66) (n = 269)	13.53 (11.03) (n = 518)	NS	NA
Utilization Measures
Lab Order/100 visits	34.6% (n = 93)	36.3% (n = 188)	NS	NA
CT Scans/100 Visits	11.52 (47.09) (n = 31)	11.78 (53.10) (n = 61)	NS	NA
Plain radiographs/100 Visits	45.35 (86.09) (n = 122)	40.54 (76.73) (n = 210)	NS	NA
MRIs/100 Visits	1.12 (13.61) (n = 3)	0.19 (4.3) (n = 1)	NS	NA
US/100 visits	7.06 (30.94) (n = 19)	1.7 (13.07) (n = 9)	0.016	1.7 −9.1
Patient Outcomes Measures
Admission Rate including transfer	39% (n = 105)	37.3% (n = 193)	NS	NA
Combined LBTC rate	0% (n = 0)	0.2% (n = 1)	NS	NA
Returned to ER Within 72 Hours of Disposition Rate	2.6% (n = 7)	3.1% (n = 16)	NS	NA

NA = not applicable

CPT code 99285: ED LOS (327.32 min vs.301.35 min, P < 0.05), provider-to-treat-and-release disposition time (258.82 min vs. 247.27 min, P < 0.05) and door to provider time (18.68 min vs. 16.98 min, P < 0.05) were higher in resident-covered patients. There was no significant difference in provider-to-admission time (235.01 min vs. 216.54 min, P = 0.086). The ultrasound usage was higher in resident-covered patients (63.87/100 visits vs. 56.68/100 visits, P < 0.05). There was no difference between CT scan usage, plain radiograph usage, MRI usage, lab usage, admission rate, LBTC rate and return to ED in 72-hour rate (Table 4).

CPT code 99291: resident-covered patient had a higher ED LOS (299.70 min vs. 277.87 min, P < 0.05) and provider to admission time (269.13 min vs. 229.65, P < 0.05). There was a higher utilization of ultrasound in the resident-covered patient (7.06/100 visits vs. 1.7/100 visit, P < 0.05). There were no differences in provider-to-treat-and-release disposition time, the door-to-provider time, laboratory usage, CT scan usage, plain radiograph usage, MRI utilization, admission rate and Return to ED in72 hour rate (Table 4).

Discussion

To better elucidate the cost of a resident in a pediatric emergency department, we sought to measure various hypothesized or known costs concurrently. Similar to Clinkscale et al, RVUs per visit for resident-covered patients were higher than nonresident-covered patients [7]. It was hypothesized that resident-covered patients could be more complex than nonresident-covered patients. Therefore, a subgroup analysis was performed using CPT codes as measures of complexity. CPT codes are determined after a complete assessment by a provider, as opposed to ESI (Emergency Severity Index), which is done as the initial triage by a nurse. This methodology was not typically used in previous studies (which used ESI), but we believe allows for greater detection of changes across metrics on the basis of complexity.

Similar to James et al, across all encounters, throughput metrics were unfavorable for resident-covered patients [3]. This could be explained by prolongation of evaluation due to resident inexperience and increased attending time taken for teaching and supervision. The door-to-provider time was significantly different between the groups, but by less than one minute, which suggests that the other unfavorable throughput metrics were not due to delays in seeing patients on the part of residents. These unfavorable throughput metrics/trends toward unfavorability in resident-covered patients continued across all CPT codes in the subgroup analysis.

We also found that there was increased utilization of CT scan, lab, and ultrasound for resident-covered patients which could be attributed to inexperienced residents with less confidence in their physical examination, history-taking, and differential diagnosis skills. Similar to Hemani et al, the utilization of plain radiographs was less in resident-covered patients [8]. We suspect that this outcome was driven by CPT code 99283, constituting more than half of the total patients, in which a higher proportion of orthopedic cases were assigned to nonresident- providers through the ED split flow process. The utilization of MRI was very low across both groups limiting our ability to detect a significant difference.

There were higher rates of admission, unscheduled return ER visit in 72 hours, and the LBTC rate for resident-covered patients. While this could be related to a resident’s relative inexperience, this finding may also have been partially driven by the residents seeing a higher proportion of medical (i.e., non-orthopedic) patients whom are more likely to return to the ER than orthopedic patients [9]. Given the findings of 20–25 additional minutes in the ER when under a resident’s care, it is possible that LBTC rate was driven by patient dissatisfaction with delays, but in the absence of attributable patient satisfaction data, this could not be determined.

In the subgroup analysis of utilization metrics, laboratory orders and CT scan order rate by residents were initially unfavorable but were no longer significantly different in the highest complexity codes. This is consistent with the sickest patients receiving high intensity, high service utilization care due to clinical need, regardless of provider type. Plain radiographs were favorable at the lower complexity codes for residents, most likely due to their relative lack of orthopedic cases compared to other providers, but, as expected in the highest complexity patients, the favorability vanished. Ultrasound utilization was higher in resident-covered patients; we suspect this is driven by the higher rates of abdominal pain evaluations that were assigned to resident-providers through the ED split flow process. Another possibility is that residency curricular focus on radiation safety may lead the resident to order an ultrasound over other imaging modalities. MRI utilization was too infrequent to draw meaningful conclusions.

In the subgroup analysis of outcomes metrics, the unfavorability in the general analysis appears to have been driven by CPT 99283, representing greater than 50% of the total patients. In the low complexity codes, there were too few events to evaluate. Across all high complexity codes, outcomes measures were the same between the groups. This could be related to more direct supervision by the attending with these higher complexity patients.

While this study included a large cohort of patients that could be directly compared for the presence of a resident, there are several limitations. This was a retrospective, single-center study, limiting generalizability. The non-resident patients were covered by a heterogeneous mixture of models. Differences in practice patterns between these models may exist which could not be measured. The low event rate for some utilization and outcome metrics across low and medium complexity CPT codes limited our analysis of those metrics. We were unable to measure benchmarked patient experience differences due to lack of attribution to residents. The surrogate, LBTC rate, may not adequately reflect patient experience.

Conclusion

The true cost of a learner in a clinical environment remains elusive. Presence of a resident did lead to unfavorable increases in many throughput measures, utilization and outcome measures, a difference which disappeared in most cases with higher complexity patients. It is reasonable to conclude that the cost of a learner actually decreases with complexity. This may lead to staffing and teaching models in which residents are utilized more in high complexity patients, which may be an educational threat but may better reflect the reality of clinical practice now and in the future. It seems possible that the moderate complexity codes, with less resident experience and less direct oversight, combined with higher volume, therefore, may present the most opportunity fosr improvement in system design.