Productivity, efficiency, and overall performance comparisons between attendings working solo versus attendings working with residents staffing models in an emergency department: A Large-Scale Retrospective Observational Study.

BACKGROUND AND
OBJECTIVE: Attending physician productivity and efficiency can be affected when working simultaneously with Residents. To gain a better understanding of this effect, we aim to compare productivity, efficiency, and overall performance differences among Attendings working solo versus working with Residents in an Emergency Department (ED).
METHODS: Data were extracted from the electronic medical records of all patients seen by ED Attendings and/or Residents during the period July 1, 2014 through June 30, 2017. Attending productivity was measured based on the number of new patients enrolled per hour per provider. Attending efficiency was measured based on the provider-to-disposition time (PDT). Attending overall performance was measured by Attending Performance Index (API). Furthermore, Attending productivity, efficiency, and overall performance metrics were compared between Attendings working solo and Attendings working with Residents. The comparisons were analyzed after adjusting for confounders via propensity score matching.
RESULTS: A total of 15 Attendings and 266 Residents managing 111,145 patient encounters over the study period were analyzed. The mean (standard deviation) of Attending productivity and efficiency were 2.9 (1.6) new patients per hour and 2.7 (1.8) hours per patient for Attendings working solo, in comparison to 3.3 (1.9) and 3.0 (2.0) for Attendings working with Residents. When paired with Residents, the API decreased for those Attendings who had a higher API when working solo (average API dropped from 0.21 to 0.19), whereas API increased for those who had a lower API when working solo (average API increased from 0.13 to 0.16).
CONCLUSION: In comparison to the Attending working solo staffing model, increased productivity with decreased efficiency occurred among Attendings when working with Residents. The overall performance of Attendings when working with Residents varied inversely against their performance when working solo.

Introduction

Emergency Department (ED) provider productivity and efficiency are two important performance measures. Productivity is viewed as the number of patient encounters per provider per unit time. It can also be measured by number of Relative Value Units (RVUs) generated per provider per unit time or encounter. Efficiency refers to the time and resources required to complete an ED patient encounter. ED length of stay (LOS), defined as the total time spent within the ED for a given patient encounter, and provider-to-disposition time (PDT), defined as the interval starting with initial provider involvement and ending with disposition selection for a given patient encounter, are among recognized efficiency metrics [1-4]. However, both are affected by multiple factors including ED crowding, patient acuity, and supervision of Residents. [5-7].

Most of the care delivered in training institutions occurs via the Attending oversight of Resident care model [8]. Mixed results are documented regarding provider productivity and efficiency of the Attending-Resident Team as a function of relative team clinical experience [9-11]. A study focused on productivity used the number of new patients per hour seen by either residents or Advanced Practice Providers (APPs) in a Fast Track area (i.e., low acuity patient care area), they found Resident productivity was less than that of the APPs’ indicating resident in training might affect their productivities [12]. Previous studies revealed prolonged LOS was experienced by patients receiving care in the Attending oversight of Resident model indicating decreased provider efficiency in the training institution setting [9,10]. However, another study determined that ED LOS was not significantly affected by the presence or total number of trainees in the ED [11]. At present, only a few studies compare provider productivity and efficiency between Attendings working solo versus Attendings working with Residents and none of them examine the differences in productivity and efficiency of individual Attendings within these two groups.

Productivity and efficiency are often consistent when used to evaluate provider performance (i.e., efficient providers are also productive or vice versa). However, it is not uncommon to observe some inconsistencies (e.g., high productivity but low efficiency), thus creating a significant challenge to understanding overall provider performance. To overcome this challenge, prior research introduced a composite index, which was calculated by combining productivity and efficiency [13]. However, a major shortcoming of said calculation is the lack of external validation.

To gain a deeper understanding as to whether working with Residents affects Attending performance within an academic environment, we aim to measure productivity, efficiency, and overall performance of Attendings and further compare the differences among Attendings working solo versus Attendings working with Residents at an individual level.

Methods

Study setting and design

This is a single center retrospective observational study. The institutional review board of John Peter Smith Health Network approved this study (IRB No. 010713.004ex) with the approval of the waiver of the written informed consent. The study hospital is a tertiary referral center located in an urban area serving a community of approximately 2 million. It is a publicly funded hospital. It is also a regional Chest Pain Center, Comprehensive Stroke Center, and a Level 1 Trauma Center. The hospital ED sponsors a 3-year EM Residency Program and managed approximately 120,000 annual visits during the study period. The majority of our patients have no commercial insurance coverage. Approximately 15–20% of patients are covered by Medicare and Medicaid. Approximately 25% of patients are Hispanic.

The ED has dedicated Fast Track and main ED areas. APPs staff Fast Track and see mainly low acuity patients (i.e., Emergency Severity Index [ESI] Level 4 and 5 and low risk Level 3, see detail explanation of ESI in S1.1 Appendix) requiring minimal oversight (< 5%) by Attendings. Residents see mainly high acuity patients (i.e., ESI 1–2, and high risk ESI-3) under the supervision of Attendings in the main ED area. The study ED also provides Medical Student rotations year around. However, Medical Students were not allowed to formally document within ED patient medical records and Residents oversee Medical Student patients.

The study ED is continuously staffed with Attendings without any daily gaps in coverage. Residents are not scheduled during the 16 consecutive hours from Wednesday 2300 through Thursday 1500 each week to facilitate didactics. Attendings work solo during this time frame. Residents and Attendings are scheduled to work together during all other times weekly. Attendings are typically scheduled with one senior EM Resident (PGY-2 or PGY-3) and one junior Resident/off-service Resident (EM PGY-1 or non-EM). In general, teams are composed of one Attending and two Residents working within a fixed geographic location within the ED. Senior and junior residents are balanced when Attendings work with Residents. When Residents are not scheduled, these same geographic areas are staffed by solo Attendings.

This study divided patients into two groups. Attending working solo group included patients who were seen only by Attending physicians during the period Wednesday 2300 to Thursday 1500. Attending working with Residents group included patients who were seen by both the Attending and the Resident during all other times. However, if patients were only seen by Attending physicians during the Attending working with Residents time frame, these patients were included in the Attending working with Resident group. This was done because during that time frame, these Attendings have to supervise Residents simultaneously regardless of whether they are able to see patients by themselves. This occurs very rarely during Attending working with Resident shifts. EPIC® (Epic Systems Corporation, Verona, WI) electronic medical record (EMR) system was used for medical documentation.

Participants

The study participants were EM Attendings and Residents. All patient encounters registered at the study ED and seen by participating Attendings and Residents during the period July 1, 2014 through June 30, 2017 were enrolled and analyzed. Patients who returned within 72 hours or were repeatedly seen at the study ED were considered as new patient encounters and treated as new patients for study purposes. As this study mainly focused on ED Attending performance, we excluded: 1) patients seen by other providers (e.g., APPs, non-EM Attendings); 2) part-time Attendings who worked fewer than 2 shifts per month; 3) Attendings who only worked solo or who only worked with Residents; and 4) Attendings who worked unbalanced shifts. An Attending with balanced shifts was defined as one who regularly worked both solo (≥ 1 eight-hours shift per month) and with Residents (≥ 4 eight-hours shifts per month) during the study period. Therefore, study Attendings were enrolled in both groups (Attending working solo versus Attending working with Residents) and we performed cross-over comparisons between these two groups.

Data source

All data were retrieved from the EMR by persons from the hospital’s Information Technology (IT) Department who were blinded to the study’s outcomes. All data were subjected to internal validation assessment. Twenty random samples were selected at six separate phases and assessed manually by searching the EMR to determine the validity of the retrieved data.

Variables

We collected patient general characteristics including age, gender, and ethnicity. Other variables included patient acuity level (ESI) at triage, patient total ED length of stay (LOS), provider-to-disposition time (PDT), the number of new patients per hour seen by a given provider, and ED crowding status [14] upon patient arrival to the ED. Detail variable explanations are addressed in S1.2 Appendix.

Outcome measurements

We used three measurements: 1) the number of new patients per hour seen by a given Attending measured productivity; 2) PDT of each patient measured efficiency; and 3) Attending Performance Index (API) measured overall performance (see formula). API is a composite metric that integrates both productivity and efficiency. It is modified based on an established performance index [13]. A detail explanation of API is addressed in S1.3 Appendix.

Reporting guideline

We followed the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) reporting guideline in this study [15].

Data analysis

We performed provider productivity, efficiency, and performance comparisons between Attendings working solo versus Attendings working with Residents at both the group and individual levels. We used Pearson chi square test for categorical data comparisons (gender, ethnicity, level of acuity, and ED crowding). For continuous data comparisons, we calculated mean with its standard deviation (SD) for the number of new patients per hour seen by a given provider and patient PDT. We also calculated median with its interquartile range (IQR) for age, the number of new patients per hour seen by a given provider, patient PDT, and API. A student t test was used for mean comparisons between groups. A Wilcoxon rank-sum test was used for median comparisons between groups. To avoid potential confounders, 1:1 propensity score matching was performed between individual Attendings working solo and individual Attendings working with Residents after adjusting for patient age, gender, ethnicity, level of acuity, and ED crowding. To assess overall performance changes, the APIs of individual Attendings were calculated and compared between individual Attendings working solo and working with Residents. Scant literature exists regarding overall performance measurement of Attendings and no quantitative benchmark value has been established to date indicating whether calculated API is predictive of Attending performance. Therefore, a threshold value API was determined in this study based on the changes of individual Attending APIs working with or without Residents. Indeed, this threshold value was a cutoff value at which different patterns of Attending overall performance were observed when working solo versus working with Residents. All analyses were performed using STATA® 14.2 (StataCorp LLC, College Station, TX) software with a p-value < 0.05 considered a statistically significant difference.

Results

During the study period, a total of 49 ED Attendings and 271 ED Residents worked at the study ED. We excluded 14 part-time ED Attendings, 4 Attendings that either consistently worked solo or worked only with Residents, and 16 Attendings without balanced schedules. A final group of 15 ED Attendings and 266 Residents that collectively managed a total of 111,145 patient encounters during the study period was enrolled. (S1 Appendix Fig). Study patient general information is shown in Table 1. Patient and clinical characteristics (age, ESI, ED crowding, and ED LOS) in the final analysis differ between Attendings working solo and Attendings working with Residents groups (Table 1, p < 0.001). It was noted that the Attending working solo group saw a slightly more high-acuity (ESI 1) patients. More patients were seen by the Attending working with Residents group than the Attending working solo group during times when the ED was overcrowded (Table 1, p < 0.001).

Table 1

Study patient population general characteristics.

	Attendings Working Solo(N = 7,283)	Attendings Working with Residents(N = 103,871)	Total Patients Managed by Attendings Working Solo and Attendings Working with Residents Combined (N = 111,154)
Age—y (median, IQR)a	48 (33, 58)	47 (32, 57)	47 (32, 57)
Gender—male (n, %)b	3,855 (53)	55,272 (53)	59,127 (53)
Ethnicity—Hispanic (n, %)c	1,802 (25)	26,535 (26)	28,337 (25)
ESI—(n,%)a
ESI-1	294 (4.2)	4,310 (4.0)	4,604 (4.1)
ESI-2	2,391 (33)	38,517 (37)	40,908 (37)
ESI-3	3,953 (54)	54,722 (53)	58,675 (53)
ESI-4	531 (7.3)	5,499 (5.3)	6,030 (5.4)
ESI-5	99 (1.4)	605 (0.6)	704 (0.6)
Unknown	15 (0.2)	218 (0.2)	233 (0.2)
ED Crowding—(n, %)a
Not crowded	3,949 (54)	40,012 (39)	43,961 (40)
Crowded	1,669 (23)	29,855 (29)	31,524 (28)
Over-crowded	1,665 (23)	34,004 (33)	35,669 (32)
ED Crowding—NEDOCS score (median, IQR)a	92 (58, 136)	115 (82, 154)	114 (80, 153)
ED LOS—hours (median, IQR)a	4.1 (2.7, 5.9)	4.5 (3.1, 6.4)	4.5 (3.1, 6.4)

a:p < 0.001

b: p = 0.65

c: p = 0.19.

Abbreviations and definitions: IQR, Interquartile Range (25th, 75th); n, number; y, year; ED, Emergency Department; ESI, Emergency Severity Index; LOS, Length of Stay.

Analysis of productivity reveals more patients per hour were seen by Attendings working with Residents than Attendings working solo (Table 2). Total numbers of patients presenting to the ED from 2300 on a given day to 1500 the following day were calculated from Monday through Sunday revealing that patient volumes were within median range during the Attending working solo timeframe (Wednesday 2300 to Thursday 1500) (S2 Appendix). Shorter PDT (i.e., efficiency) is noted in the Attending working solo group (Table 2). Essentially, Attending productivity increased but efficiency decreased when working with Residents. Overall performance seems to be increased among Attendings working with Residents in comparison to Attendings working solo. However, the benefit of increased Attending provider performance while working with Residents appears to be diminished when propensity score matching is applied (Table 2).

Table 2

Provider productivity, efficiency, and performance measurements comparison between attendings working solo and attendings working with residents.

	ProductivityNumber of New Patients per HourMedian (IQR)Mean (SD)	EfficiencyProvider to Disposition Time (Hours)Median (IQR)Mean (SD)	Performance*Attending Performance IndexMedian (IQR)
	Total Patients Before Propensity Score Matching (N = 111,154)
Attendings Working Solo	3 (2, 4) a	2.4 (1.4, 3.6) c	0.16 (0.08, 0.36) e
	2.9 (1.6) a	2.7(1.8) c
Attendings Working with Residents	3 (2, 4)	2.7 (1.7, 3.9)	0.17 (0.09, 0.34)
	3.3 (1.9)	3.0 (2.0)
	Total Patients After Propensity Score Matching (N = 14,074)
Attendings Working Solo	3 (2, 4) b	2.4 (1.4, 3.6) d	0.16 (0.08, 0.36) f
	2.9 (1.6) b	2.7 (1.8) d
Attendings Working with Residents	3 (2, 4)	3.0 (1.9, 4.1)	0.15 (0.08, 0.27)
	3.1 (1.7)	3.2 (1.9)

a: p < 0.001 (productivity comparison between Attendings Working Solo and Attendings Working with Residents)

b: p < 0.001 (productivity comparison between Attendings Working Solo and Attendings Working with Residents) using propensity score matching.

c: p < 0.001 (efficiency comparison between Attendings Working Solo and Attendings Working with Residents)

d: p < 0.001 (efficiency comparison between Attendings Working Solo and Attendings Working with Residents) using propensity score matching.

e: p = 0.037 (performance index comparison between Attendings Working Solo and Attendings Working with Residents)

f: p < 0.001 (performance index comparison between Attendings Working Solo and Attendings Working with Residents) using propensity score matching.

* Attending Performance Index (API) refers to formula: API = (number of new patients per hour seen by a provider)/((patient acuity level determined by ESI)2 x (provider to disposition time in hours)).

Abbreviations and definitions: IQR, Interquartile Range; Provider to Disposition Time, time interval between initial provider encounter to disposition (i.e., admit, discharge, transfer) in hours

Additionally, when Attending productivity, efficiency, and overall performance are measured at an individual Attending level, the case of increased productivity with decreased efficiency is observed for most individual Attendings when working with Residents compared to working solo (S3 Appendix and S4 Appendix). All 15 Attendings were divided into two groups in terms of their solo performance. Higher Attending performance indexes (APIs) are observed among individual Attendings when working with Residents if their solo API < 0.18. On the contrary, lower APIs were noted among individual Attendings whose solo API ≥ 0.18 when working with Residents (S5 Appendix). This pattern remained when confounding factors were adjusted using the 1:1 propensity score matching model (Table 3). Therefore, a threshold API (high API ≥ 0.18 and low API < 0.18) of individual Attending performance was determined in this study.

Table 3

Attending overall performance comparisons between attendings working solo versus attendings working with residents based on attending solo performance.

	Original Data			Propensity Score Matching Data
	Attendings WorkingSoloMedian (IQR)	Attendings Working with ResidentsMedian (IQR)	p	Attendings WorkingSoloMedian (IQR)	Attendings Working with ResidentsMedian (IQR)	p
Productivity
Attendings withlow baseline API	2 (2, 3)	3 (2, 4)	< 0.001	2 (2, 3)	3 (2, 4)	< 0.001
Attendings withhigh baseline API	3 (2, 4)	3 (2, 5)	< 0.001	3 (2, 4)	3 (2, 4)	0.009
Efficiency
Attendings withlow baseline API	2.8 (1.8, 4.0)	2.8 (1.8, 4.1)	0.018	2.8 (1.8, 4.0)	3.0 (2.0, 4.3)	< 0.001
Attendings withhigh baseline API	2.1 (1.1, 3.2)	2.6 (1.6, 3.8)	< 0.001	2.1 (1.1, 3.2)	2.8 (1.8, 4.0)	< 0.001
Performance index
Attendings withlow baseline API	0.13 (0.07, 0.26)	0.16 (0.08, 0.31)	< 0.001	0.13 (0.07, 0.26)	0.14 (0.07, 0.25)	< 0.001
Attendings withhigh baseline API	0.21 (0.11, 0.48)	0.19 (0.05, 0.38)	< 0.001	0.21 (0.11, 0.48)	0.15 (0.08, 0.29)	< 0.001

Abbreviations and definitions: API, Attending Performance Index; IQR, Interquartile Range; Performance measure = Attending Performance Index [(number of new patients per hour seen by a provider)/((patient acuity level determined by ESI)2 x (provider to disposition time in hours))].

Discussion

Our study found increased productivity with decreased efficiency among Attendings working with Residents. When API was used for performance assessment, we found that individual Attendings with high solo performances rendered an overall decreased performance while working with Residents. On the contrary, working with Residents increased overall performance among Attendings with low solo API. Though the study was performed in an ED, interpretation of study findings might not be limited to the ED setting since they reflect a similar academic teaching model whereby Attending physicians work with Residents in most clinical practice settings. Therefore, our results add more evidence to summative provider performance measurements in an academic institution thereby providing valuable insight to current and future approaches to Resident training and faculty evaluation across different residency specialty programs.

Productivity and efficiency can be affected by differences in patient acuity. The study ED preferentially flows patients such that Attendings working solo and Attendings working with Residents provide high acuity care and the APP staff provide low acuity care. Though the majority of high acuity patients (ESI 1–2) were seen by Attendings regardless of whether they worked solo or with Residents, productivity and efficiency differences occurred between these two groups. The possible reasons might be: 1) the daily variety of high acuity patients present at the study ED (e.g. slightly more ESI-1 patients present during non-resident shifts); and 2) the exclusion of the APPs in this study. During the ED non-resident time (16h/week), an extra APP was scheduled to work. However, this study excluded APPs who usually only saw low-acuity patients. Under this circumstance, it is necessary to use propensity score matching to minimize potential confounder effects in this study. When propensity score matching was applied, the median number of new patients per hour seen by either Attendings working solo or Attendings working with Residents reveals no changes both at the group (Table 2) and individual (Table 3) levels, indicating that patient acuity level has no effect on provider productivity measurement. In addition, different metrics have been used to measure provider productivity in the literature, including RVUs or number of new patients per shift. However, RVUs can be significantly affected based on encounter documentation alone. Our providers usually spend extra time outside of their clinical shift to complete documentation, thus it is extremely difficult and inaccurate to add extra time for documentation when deriving the provider productivity calculation. Moreover, providers working at the study ED typically work different length shifts (e.g., 8h, 9h, 10h, or 12h shifts), serving as a potential confounder that fails to compare provider productivity on a common basis when number of new patients per shift is used. Therefore, to simplify this study, the number of new patients per hour, as opposed to RVUs per hour and/or patients per shift, was used as our Attending productivity measurement [4,16]. On the other hand, we used PDT instead of patient LOS for provider efficiency measurement because LOS is often affected by different system- and patient-associated variables, which could be potential confounders in the context of provider efficiency (e.g., ED crowding, waiting room time, etc.) [17,18]. Unlike LOS, PDT is more directly affected by a given provider during the decision-making process impacting downstream diagnostic and therapeutic resource needs turnaround intervals [4,13]. Therefore, PDT delivers better interpretive quality regarding Attending efficiency. Ideally, a higher number of new patients per hour (high productivity) along with a lower PDT per patient (high efficiency) is considered the most desirable overall provider performance outcome combination. A balanced provider performance index was therefore used in this study [13].

We refrained from including all ED Attending data in this analysis to avoid developing different biases. We believe that inclusion of ED Attendings with non-balanced shift schedules based either on working solo or working with Residents arms will produce bias thereby complicating comparison of performance changes at an individual level. Furthermore, provider productivity and efficiency could vary significantly based on provider proficiency levels, thereby potentially affecting Attending productivity and efficiency when working non-balanced shifts. Working non-balanced shifts will affect Attending proficiency when spending more time performing unusual tasks (e.g., increased documentation time in absence of Resident input to overall encounter workflow). We therefore only enrolled Attendings with balanced shift schedules and analyzed median values to stabilize potential confounders. As shown in Tables 2 and 3, no changes to median values are noted after all potential confounders are adjusted by propensity score matching. Such findings further confirm the stability of using medians (IQR) for productivity and performance measurements.

Many previous studies investigated the impact of trainees on ED Attending performance concluding diverse findings [7,11,19,20]. When patient ED LOS was measured, some studies showed that ED LOS was not affected significantly by the presence of trainees rotating in the ED [11], whereas other studies showed an increase in ED LOS when Attendings worked with Residents [19,20]. In one study, an average of 7 minutes of increased LOS was reported with each additional trainee working with a given Attending [20]. In terms of provider productivity, a previous study also revealed no significant difference in number of patients per hour seen by junior EM, senior EM, or off-service Residents [7]. The reasons that our study findings differ from previous reports include: 1) we balanced our samples of Attendings in both groups, therefore the changes in their operational performance can be compared at an individual Attending level. In contrast, other studies mainly assess the operational performance at a group level; 2) we used PDT instead of LOS for provider efficiency measurement which may help reduce the effects of other confounding factors (e.g., ED crowding). The use of propensity score matching further minimizes potential biases; 3) the study ED had a 1:2 Attending-to-Resident ratio instead of 1:1 ratio staffing. This might affect Attending performance more significantly than the 1:1 teaching model, which could indirectly result in increased Attending productivity (i.e., more providers are available to see patients during the same time interval) with decreased efficiency (i.e., more trainees requiring more supervision per Attending per unit time), which differs from the previous report [7].

Our study has its limitations. First, this is a single center retrospective study which cannot demonstrate causality due to potentially harvesting incorrect information and selection bias. Second, although two key performance metrics were used for provider productivity and efficiency analysis, other metrics (e.g., number of RVUs per hour per provider, number of patients treated per shift) were not collected and compared in this study. Additionally, productivity and efficiency could be affected multi-factorially to include patient disease severity, nursing staffing levels, efficiency levels of nursing staff, etc. Study results might be inaccurate without the measure of individual patient disease severity and nursing staffing and their efficiencies. Third, among the Attending working with Residents group, due to differences in Resident efficiency and performance, Attending productivity, efficiency, and performance might potentially be affected. We also did not analyze and interpret how different levels of resident (EM versus non-EM residents) and medical students could impact attendings’ productivity, efficiency, and performance. However, we consider such effects to be minimal due to 1) blinded scheduling of Attendings and Residents resulting in random Attending-Resident shift combinations thereby avoiding significant heterogeneity; and 2) this is a 3-year study with over 100,000 patients seen by 15 Attendings working with 266 different Residents therefore each Attending had opportunity to work with Residents performing at different efficiency levels producing minimal overall effect on performance. Fourth, under rare conditions, Attendings might see patients by themselves while also supervising Residents during the Attending working with Residents shifts. Though it rarely occurs (< 1% in this study), it could still potentially affect our study results. Last, patients in the Attending working solo group accounted for a relatively small portion of the entire study population due to the current staffing model that employs Residents 90% of the available weekly schedule (152/168 hours). Given the fact of a fixed schedule of Attending only shifts, patient volume differences and potential patient selection bias could possibly occur when compared to patients from other groups. Therefore, a multi-center prospective study is warranted for external validation.

Conclusion

Diverse Attending productivity and efficiency exists. In our study, Attending overall performance, when working with Residents, varied inversely when compared to their performance working solo.

(S1.1 Appendix: Detail Explanation of Emergency Severity Index (ESI); S1.2 Appendix: Detail Variables Explanation; S1.3 Appendix: Detail Explanation of Attending Performance Index (API)).

(DOCX)

Click here for additional data file.

Daily patient volume as measured during time interval 2300 (previous day) through 1500 (next day).

(DOCX)

Click here for additional data file.

Productivity comparisons between attendings working solo versus attendings working with residents.

(DOCX)

Click here for additional data file.

Efficiency (provider to disposition time) comparisons between attendings working solo versus attendings working with residents.

(DOCX)

Click here for additional data file.

Overall performance comparisons between attendings working solo versus attendings working with residents.

(DOCX)

Click here for additional data file.

Study flow diagram.

(PDF)

Click here for additional data file.

12 Nov 2019

PONE-D-19-26595

Productivity, Efficiency, and Overall Performance Comparisons Between Solo Attending Versus Attending with Residents Staffing Models in an Emergency Department

PLOS ONE

Dear Dr. Wang,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Dec 27 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Andrew Carl Miller

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

1. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

2. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: No

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: -Based on the data presented in the results section of the abstract, the conclusion that “increased productivity […] often occurs among attendings working with residents” isn’t supported by the data cited as you cited identical median numbers. If other data shows this, cite that instead. Also I would take out the word “often.”

First paragraph: However, both are affected by multiple factors including ED crowding, patient acuity, (AND) supervision of Residents and Medical Students. [5-7].

Second paragraph: “Whereas, another study determined that ED LOS was not significantly affected by the presence or total number of trainees in the ED [11]” is a sentence fragment. Change to “However, another study determined that ED LOS was not significantly affected by the presence or total number of trainees in the ED [11].”

Second paragraph: “At present, few studies compare provider productivity and efficiency between Attendings working solo versus Attendings working with Residents.” How does this study differ from or add to what we already have in the literature on this?

Third paragraph: “Overall provider performance may be gained (change to “calculated” or “defined”)by combining productivity and efficiency arriving at a composite measure [13].” This paragraph at the end of the introduction does not make sense all by itself. If you want to introduce a new metric, I would do it in the first paragraph where you start defining the metrics used.

The paper switches frequently between passive and active tense; this needs to be more consistent.

In the results section, the conclusion that “Analysis of productivity reveals more patients per hour were seen by Attendings working with Residents than Attendings working solo (Table 2).” Is not supported by the data cited which shows identical median patient numbers for both groups.

In the results section, why was the cut off of 0.18 used for the API for high index or low index categorization? Has this been looked at and defined in the past?

Discussion: First sentence “Our study found increased productivity with decreased efficiency among Attendings working with Residents.” Is not supported by the data presented which cites identical median productivity numbers. How are you getting to this conclusion?

“Though the majority of high acuity patients (ESI 1-2-3) were seen by Attendings regardless of whether they worked solo or with Residents, differences occurred between these two groups.” What differences are these?

Discussion:

“Therefore, to simplify this study, the number of new patients per hour as opposed to RVUs per hour was used as our Attending productivity measurement [4,15]. On the other hand, we used PDT instead of patient LOS for provider efficiency measurements because LOS is often affected by different system and patient associated variables serving as meaningful confounders (e.g., ED crowding, waiting room time, etc.) [16,17]. Unlike LOS, PDT is more directly affected by a given provider during the decision-making process and resultant downstream diagnostic and therapeutic resource needs [4,13]. Therefore, PDT delivers better interpretive quality regarding Attending efficiency.” All of this likely would do better in the methods section as it does not discuss the results of your study.

Reviewer #2: General:

- Inserting line numbers will make it easier for reviewers to provide more focused feedback.

- Please be sure to list the appropriate guideline used and provide citation. For example:

von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. PLoS Med. 2007;4(10):e296. PMID: 17941714

Title:

- Indicate the study’s design with a commonly used term in the title or the abstract

Abstract:

- Please reposition the headings Methods, Results, and Conclusions so that they are not contiguous with the preceding paragraph.

Introduction:

- Pg 4: Resident efficiency has been reported as less than that of APPs. How does this compare with the number of bounce-back or repeat visits? The number of M&M cases? APP practice environments vary greatly; are the APPs in this comparison practicing autonomously (attending consult available but not required), or do they have to present their cases to an attending still? Were they APPs who had done an EM “residency” or practicing straight out of school?

- Same line as above: is there any evidence that the APPs document better, thereby billing at a higher level and generating more RVUs? To aid discussion of these points, one my consider incorporating the reference by McDonnell into the discussion (PMID: 25654675 DOI: 10.1097/PEC.0000000000000349).

Methods:

- The ESI is a poor judge of illness severity. From the EMR one may be able to obtain the necessary info to determine illness severity using a validated tool such as Charlson Comorbidity Index, or 3M-APR-DRG. This would be helpful for inter-group comparison.

- Explain how the study size was arrived at. Provide a calculation to justify sample size & method used.

- STATA 14.2 software. It is convention to list the manufacturer & location in parentheses after the name. Additional ™ or ® should be listed if applicable.

- Do the physicians chart using scribes (see PMID: 30700408; PMID: 27856140), dictation, manual electronic, or paper charting?

- Was the presence of medical students and other learners recorded and factored in?

- Were the same attendings enrolled in both groups depending on whether or not they had a resident, or was there no cross-over?

- Just to confirm, were all residents emergency medicine residents, or were some off-service residents (eg. internal medicine, ob-gyn, etc.)?

Results

- Was time of shift and staffing levels recorded? Did these differ between groups? If so, changes in nursing staffing could also have impacted the results.

- What was the distribution of the year of resident training (1, 2, 3, etc.)? It would be important to know if it was not balanced between senior and junior residents.

Discussion

- Would benefit from a deeper discussion of why they think they found no difference when others have. Compare and contrast with the results of other published studies on the topic including: (PMID: 24578767; PMID: 24672605; PMID: 25972206; PMID: 24238313; PMID: 18973640)

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Marina Boushra

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

16 Dec 2019

AUTHORS RESPONSE TO EDITORS AND REVIEWERS

As requested, we have included the original letter and comments with our point by point response in red colored font.

PONE-D-19-26595

Productivity, Efficiency, and Overall Performance Comparisons Between Solo Attending Versus Attending with Residents Staffing Models in an Emergency Department

PLOS ONE

Dear Dr. Wang,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Dec 27 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

• A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

• An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Andrew Carl Miller

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

1. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

Response: Yes, our data contain identifiable patient information including patient name, age, and admission date/time. This project was approved by the local Institutional Review Board (IRB). We provided information of the contact person to whom data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

2. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

Response: Yes, we revised and included captions for our supplemental figure at the end of our manuscript.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: No

Reviewer #2: Yes

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1:

1. -Based on the data presented in the results section of the abstract, the conclusion that “increased productivity […] often occurs among attendings working with residents” isn’t supported by the data cited as you cited identical median numbers. If other data shows this, cite that instead. Also I would take out the word “often.”

Response: Although no obvious MEDIAN differences were observed between Attendings working solo versus working with Residents by reviewing the median (IQR) of the performance measures, the Wilcoxon-Rank Sum test did show statistically significant differences in their means (p-value <0.001). Wilcoxon-Rank Sum test is a non-parametric statistical test for comparing means of two samples that may not be normally distributed. Sorry for the confusion. In order to better interpret our data, we revised and added the mean (SD) along with the median (IQR) in Table 2 (statistically significant differences between these two groups were found). We used mean (SD) of productivity and efficiency reported in the abstract (Results section). In addition, we deleted “often” under the Conclusions section of the abstract.

2. First paragraph: However, both are affected by multiple factors including ED crowding, patient acuity, (AND) supervision of Residents and Medical Students. [5-7].

Second paragraph: “Whereas, another study determined that ED LOS was not significantly affected by the presence or total number of trainees in the ED [11]” is a sentence fragment. Change to “However, another study determined that ED LOS was not significantly affected by the presence or total number of trainees in the ED [11].”

Response: Yes, we revised and added “and” between “patient acuity,” and “supervision of Residents and Medical Students.”. We also revised, and changed to “However, another study determined that ED LOS was not significantly affected by the presence of total number of trainees in the ED [11].” (see introduction section, the first and second paragraphs).

3. Second paragraph: “At present, few studies compare provider productivity and efficiency between Attendings working solo versus Attendings working with Residents.” How does this study differ from or add to what we already have in the literature on this?

Response: Thanks for reviewer’s valued comment. We revised and added the importance of this study to address how it differs from other studies reported in the literature (see below). It was inserted in the second paragraph of the Introduction section.

“At present, only a few studies compare provider productivity and efficiency between Attendings working solo versus Attendings working with Residents, and none of them examine the differences in productivity and efficiency of individual Attendings within these two groups.”

4. Third paragraph: “Overall provider performance may be gained (change to “calculated” or “defined”)by combining productivity and efficiency arriving at a composite measure [13].” This paragraph at the end of the introduction does not make sense all by itself. If you want to introduce a new metric, I would do it in the first paragraph where you start defining the metrics used.

Response: Yes, we revised and changed “gained” to “calculated”. We realize this was confusing. Sorry about that. The whole paragraph mentioned above is indeed part of the literature review. We intend to report a composite index was developed previously to measure providers’ overall performance; however, this index lacks external validation. In terms of the flow of the paper, we think it would be appropriate to describe it here together with other existing work.

”Productivity and efficiency are often consistent when used to evaluate provider operational performance (i.e., efficient providers are also productive or vice versa). However, it is not uncommon to observe some inconsistencies (e.g., high productivity but low efficiency), thus creating a significant challenge to understanding provider overall operational performance. To overcome this challenge, prior research introduced a composite index, which was calculated by combining productivity and efficiency [13]. However, a major shortcoming of said calculation is the lack of external validation.”

5. The paper switches frequently between passive and active tense; this needs to be more consistent.

Response: Yes, we revised the entire manuscript to be more consistent.

6. In the results section, the conclusion that “Analysis of productivity reveals more patients per hour were seen by Attendings working with Residents than Attendings working solo (Table 2).” Is not supported by the data cited which shows identical median patient numbers for both groups.

Response: Again, sorry for the confusion. We revised it and added both median (IQR) and mean (SD) of productivity and efficiency of attendings solo versus attendings working with residents (see revised Table 2).

7. In the results section, why was the cut off of 0.18 used for the API for high index or low index categorization? Has this been looked at and defined in the past?

Response: No, the cutoff of 0.18 has not been reported in the past. This number was derived by interpreting this study’s data at the individual provider level (see Append Table 5). We revised and added explanations in the Methods and Results sections as follows:

1. In the Methods section (last paragraph of the Methods section: data analysis)

“To assess overall performance changes, the APIs of individual Attendings were calculated and compared between individual Attendings working solo and working with Residents. Scant literature exists regarding overall performance measurement of Attendings and no quantitative benchmark value has been established to date indicating whether calculated API is predictive of Attending performance. Therefore, a threshold value API was determined in this study based on the changes of individual Attending APIs working with or without Residents. Indeed, this threshold value was a cutoff value at which different patterns of Attending overall performance were observed when working solo and working with Residents.”

2. In the Results section (third paragraph of the Results section)

“Additionally, when Attending productivity, efficiency, and overall performance are measured at an individual Attending level, the case of increased productivity with decreased efficiency is observed for most individual Attendings when working with Residents compared to working solo (Appendix-3 and Appendix-4). All 15 Attendings are divided into two groups in terms of their solo performance. Higher Attending performance indexes (APIs) are observed among individual Attendings when working with Residents if their solo API < 0.18. On the contrary, lower APIs were noted among individual Attendings whose solo API ≥ 0.18 when working with Residents (Appendix-5). This pattern remained when confounding factors were adjusted using the 1:1 propensity score matching model (Table 3). Therefore, a threshold API (high API ≥ 0.18 and low API < 0.18) of individual Attending performance was determined in this study.”

8. Discussion: First sentence “Our study found increased productivity with decreased efficiency among Attendings working with Residents.” Is not supported by the data presented which cites identical median productivity numbers. How are you getting to this conclusion?

Response: As we addressed before, we added our mean (SD) of productivity and efficiency to both attendings working solo and attendings working with resident groups. Both showed statistically significant differences. (see response of comment #1 and Table 2)

9. “Though the majority of high acuity patients (ESI 1-2-3) were seen by Attendings regardless of whether they worked solo or with Residents, differences occurred between these two groups.” What differences are these?

Response: Sorry for the confusion. We revised this sentence and added “productivity and efficiency differences”. Please see the revised following:

“Though the majority of high acuity patients (ESI 1-2-3) were seen by Attendings regardless of whether they worked solo or with Residents, productivity and efficiency differences occurred between these two groups.”

10. Discussion:

“Therefore, to simplify this study, the number of new patients per hour as opposed to RVUs per hour was used as our Attending productivity measurement [4,15]. On the other hand, we used PDT instead of patient LOS for provider efficiency measurements because LOS is often affected by different system and patient associated variables serving as meaningful confounders (e.g., ED crowding, waiting room time, etc.) [16,17]. Unlike LOS, PDT is more directly affected by a given provider during the decision-making process and resultant downstream diagnostic and therapeutic resource needs [4,13]. Therefore, PDT delivers better interpretive quality regarding Attending efficiency.” All of this likely would do better in the methods section as it does not discuss the results of your study.

Response: Thanks for reviewer’s valued comment. We considered it necessary to discuss the reason we use the number of new patients per hour as a parameter for provider productivity measurement in this study instead of using others (e.g. RVU, number of new patients per shift). Same as efficiency measurement, we addressed why we use PDT instead of patient LOS as the parameter for efficiency measurement. We believe a discussion on the variety of operational metrics used for productivity/efficiency measurements might be suitably placed in the Discussion section. However, we understand the confusion, we revised our discussion as follows (see the second paragraph under Discussion section).

“Productivity and efficiency can be affected by differences in patient acuity. The study ED preferentially flows patients such that Attendings working solo and Attendings working with Residents provide high acuity care and the APP staff provide low acuity care. Though the majority of high acuity patients (ESI 1-2-3) were seen by Attendings regardless of whether they worked solo or with Residents, productivity and efficiency differences occurred between these two groups. Therefore, it is necessary to use propensity matching to minimize potential confounder effects in this study. When propensity score matching was applied, the median number of new patients per hour seen by either Attendings working solo or Attendings working with Residents reveals no changes both at the group (Table 2) and individual (Table 3) levels, indicating that patient acuity level has no effect on provider productivity measurement. In addition, different metrics have been used to measure provider productivity in the literature, including RVUs or number of new patients per shift. However, RVUs can be significantly affected based on encounter documentation alone. Our providers usually spend extra time outside of their clinical shift to complete documentation, thus it is extremely difficult and inaccurate to add extra time for documentation when deriving the provider productivity calculation. Moreover, providers working at the study ED typically work different length shifts (e.g., 8h, 9h, 10h, or 12h shifts), serving as a potential confounder that fails to compare provider productivity on a common basis when number of new patients per shift is used. Therefore, to simplify this study, the number of new patients per hour, as opposed to RVUs per hour and/or patients per shift, was used as our Attending productivity measurement [4,15]. On the other hand, we used PDT instead of patient LOS for provider efficiency measurement because LOS is often affected by different system- and patient-associated variables, which could be potential confounders in the context of provider efficiency (e.g., ED crowding, waiting room time, etc.) [16,17]. Unlike LOS, PDT is more directly affected by a given provider during the decision-making process impacting downstream diagnostic and therapeutic resource needs turnaround intervals [4,13]. Therefore, PDT delivers better interpretive quality regarding Attending efficiency. Ideally, a higher number of new patients per hour (high productivity) along with a lower PDT per patient (high efficiency) is considered the most desirable overall provider performance outcome combination. A balanced provider performance index was therefore used in this study [13]. ”

Reviewer #2: General:

11. - Inserting line numbers will make it easier for reviewers to provide more focused feedback.

Response: yes, we inserted line numbers to the manuscript.

12. - Please be sure to list the appropriate guideline used and provide citation. For example:

von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. PLoS Med. 2007;4(10):e296. PMID: 17941714

Response: Yes, we revised and added the appropriate guideline under the method section. (see method section, under reporting guideline)

13. Title:

- Indicate the study’s design with a commonly used term in the title or the abstract

Response: Yes, we revised the title as the following:

“Productivity, Efficiency, and Overall Performance Comparisons Between Solo Attending Versus Attending with Residents Staffing Models in an Emergency Department: A Large-scale Retrospective Observational Study”

14. Abstract:

- Please reposition the headings Methods, Results, and Conclusions so that they are not contiguous with the preceding paragraph.

Response: Yes, they are revised.

15. Introduction:

- Pg 4: Resident efficiency has been reported as less than that of APPs. How does this compare with the number of bounce-back or repeat visits? The number of M&M cases? APP practice environments vary greatly; are the APPs in this comparison practicing autonomously (attending consult available but not required), or do they have to present their cases to an attending still? Were they APPs who had done an EM “residency” or practicing straight out of school?

Response: Thanks for reviewer’s valued comment. Patients who had bounce-back or repeated visits will be counted as another patient encounter and treated as a new patient. In this study, we did not specifically address the number of M&M cases. We understand APP practice differs in different environments. Therefore, in this study, we excluded patients seen by APPs. In the study ED, APPs see patients independently and can disposition patients by themselves without the need to notify attending physicians. Under very rare circumstances, an APP might call an attending physician to get a second opinion on their patients since APPs only see low acuity patients. All APPs will need ED experience before they work at the study ED. We realize such uncertainty, we revised and addressed these under the Methods section.

16. - Same line as above: is there any evidence that the APPs document better, thereby billing at a higher level and generating more RVUs? To aid discussion of these points, one my consider incorporating the reference by McDonnell into the discussion (PMID: 25654675 DOI: 10.1097/PEC.0000000000000349).

Response: Since this project mainly discussed attending and resident productivity, efficiency and their performance, we did not include APPs in this study.

17. Methods:

- The ESI is a poor judge of illness severity. From the EMR one may be able to obtain the necessary info to determine illness severity using a validated tool such as Charlson Comorbidity Index, or 3M-APR-DRG. This would be helpful for inter-group comparison.

Response: Thanks for reviewer’s valued comment. We understand that ESI is a poor indicator for illness severity. Unfortunately, due to the nature of this study’s design, we are unable to add CCI or APR-DRG to the final analysis. We revised and addressed these in our limitation section. (see the fifth paragraph under discussion section)

18. - Explain how the study size was arrived at. Provide a calculation to justify sample size & method used.

Response: This is a retrospective study and our intent is to include all patients seen by attendings and residents during the study period. Therefore, we did not perform the sample size estimation in this study.

19. - STATA 14.2 software. It is convention to list the manufacturer & location in parentheses after the name. Additional ™ or ® should be listed if applicable.

Response: Yes, it is revised and added. (see last paragraph under method section)

20. - Do the physicians chart using scribes (see PMID: 30700408; PMID: 27856140), dictation, manual electronic, or paper charting?

Response: EMR (electronic medical record) system was used for provider charting in this study. We revised and added under the method section.

21. - Was the presence of medical students and other learners recorded and factored in?

Response: Yes, our ED provides medical students rotation all year around. However, medical students are not allowed to document under EMR system. Therefore, residents oversee medical students’ patients. We revised and addressed it under the method section.

22. - Were the same attendings enrolled in both groups depending on whether or not they had a resident, or was there no cross-over?

Response: Yes, the same attendings were enrolled in both groups and this is a cross-over comparison study. We revised and added to the method section.

23. - Just to confirm, were all residents emergency medicine residents, or were some off-service residents (eg. internal medicine, ob-gyn, etc.)?

Response: Yes, study included off-service residents (non-EM residents). These non-EM residents were treated as PGY-1 EM residents. We addressed it under the method section.

24. Results

- Was time of shift and staffing levels recorded? Did these differ between groups? If so, changes in nursing staffing could also have impacted the results.

Response: Yes, time of the shift was recorded, however, due to the different length of shifts (e.g. 8h, 9h, 10h, 12h-shifts), this makes it hard to calculate number of patients per shift. Therefore, we use number of new patients per HOUR for productivity measurement. We understand that nursing staff will also affect productivity and efficiency, unfortunately, we did not record nursing staffing levels nor record the experience levels of nursing staff (RN1 versus RN2 versus more advanced RN levels). We revised and addressed it in the limitation section under the discussion. (see fifth paragraph under discussion section)

25. - What was the distribution of the year of resident training (1, 2, 3, etc.)? It would be important to know if it was not balanced between senior and junior residents.

Response: We have balanced distribution of residents with different levels of training. Each month, we have approximately equal numbers of senior EM residents, PGY-1 EM residents, and non-EM residents. We revised under the method section as the following:

“Attendings are typically scheduled with one senior EM Resident (PGY-2 or PGY-3) and one junior Resident (EM PGY-1 or non-EM). In general, teams are composed of one Attending and two Residents working within a fixed geographic location within the ED. Senior and junior residents are balanced when Attendings working with the residents.”

26. Discussion

- Would benefit from a deeper discussion of why they think they found no difference when others have. Compare and contrast with the results of other published studies on the topic including: (PMID: 24578767; PMID: 24672605; PMID: 25972206; PMID: 24238313; PMID: 18973640)

Response: Thanks for reviewer’s valued comments. We reviewed all mentioned references, some studies have been cited in this manuscript (PMID 24578767, PMID25972206), while others have not. We realize that our study required deeper discussion. Therefore, we revised and added a separate paragraph under the Discussion section to further discuss the differences when attending physicians work with trainees. See the following revision (the fourth paragraph under discussion section).

“Many previous studies investigated the impact of trainees on ED Attending performance concluding diverse findings [7,11,18,19]. When patient ED LOS was measured, some studies showed that ED LOS was not affected significantly by the presence of trainees rotating in the ED [11], whereas other studies showed an increase in ED LOS when Attendings worked with Residents [18,19]. An average of 7 minutes of increased LOS was reported with each additional trainee working with a given Attending [19]. In terms of provider productivity, a previous study also revealed no significant difference in number of patients per hour seen by junior EM, senior EM, or off-service Residents [7]. The reasons that our study findings differ from previous reports include: 1) we balanced our samples of Attendings in both groups. Therefore, the changes in their operational performance can be compared at an individual Attending level. In contrast, other studies mainly assess the operational performance at a group level; 2) we used PDT instead of LOS for provider proficiency measurement which may help reduce the effects of other confounding factors (e.g., ED crowding). The use of propensity score matching could further minimize potential biases; 3) the study ED had 1:2 Attending-to-Resident ratio instead of 1:1 ratio staffing. This might affect attending operational performance more significantly than 1:1 teaching model, which could indirectly result in increased Attending productivity (i.e., more providers are available to see patients during the same time interval) with decreased efficiency (i.e., more trainees requiring more supervision per Attending per unit time), which differs from the previous report [7]. “

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Marina Boushra

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: response to editor and reviewers comments (R1).docx

Click here for additional data file.

10 Jan 2020

PONE-D-19-26595R1

Productivity, Efficiency, and Overall Performance Comparisons Between Attendings Working Solo Versus Attendings Working with Residents Staffing Models in an Emergency Department: A Large-Scale Retrospective Observational Study

PLOS ONE

Dear Dr. Wang,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Feb 24 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Andrew Carl Miller

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #2: It would be great if you were able to run additional analysis to address these 2 other questions:

Impact of EM vs. non-EM residents on attending productivity

Impact on attending and resident productivity when students are present

Line 87: students [5-7]., not students. [5-7].

Line 92: This is a loaded statement. The APPs were likely practicing independently, but the residents have to do more steps before disposition: present to attending, attending sees patient, then disposition. It's not really a fair comparison.

Line 164: Define how ED crowding was determined (i.e. NEDOCS score). Also state how overcrowding thresholds were selected, including references.

Line 175: please provide citation for STROBE

Line 204: “more elderly”: The statement is a little misleading. Although statistically significant, its not really clinically significant. The means were 48 vs 47 years of age.

Line 205: Fewer high acuity patients seen by solo attendings: This is a little misleading. Most places consider high acuity to be ESI 1&2 (not 1-3). The solo patients saw MORE ESI 1 (highest acuity) patients.

All high acuity patients are seen by an attending regardless of resident presence. There is a work-flow nuance here. On shifts with residents, there are likely fewer attendings, and on attending only shifts there is likely more attending coverage. The attending overseeing residents is likely to get more High Acuity patients because multiple residents are picking them up and presenting them. He/She can manage more at 1 time. On the attending only shifts, they are likely to divide up these patients, so any one attending will have fewer than if they were alone with residents. Please discuss this.

Table 1: what characterized ED overcrowding? Was it a NEDOCS score ≥ level 4?

For tables, please use superscript letters a for legends rather than symbols.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Marina Boushra

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

20 Jan 2020

AUTHORS RESPONSE TO EDITORS AND REVIEWERS

As requested, we have included the original letter and comments with our point by point response in red colored font.

PONE-D-19-26595R1

Productivity, Efficiency, and Overall Performance Comparisons Between Attendings Working Solo Versus Attendings Working with Residents Staffing Models in an Emergency Department: A Large-Scale Retrospective Observational Study

PLOS ONE

Dear Dr. Wang,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Feb 24 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

• A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

• An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Andrew Carl Miller

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #2: It would be great if you were able to run additional analysis to address these 2 other questions:

Impact of EM vs. non-EM residents on attending productivity

Response: Thanks for reviewer’s valued comments. Our attending physicians are staffed to work with one senior EM resident and one junior EM resident or non-EM resident during the same working hours. In this study, we reported attending productivity using number of new patients per hour and this is a mixture of new patients seen by either one senior EM resident + one junior EM resident, or one senior EM resident + one non-EM resident. Therefore, we are unable to do a separate report to see the impact of EM vs. non-EM residents on attending productivity. However, we did another separate analysis to only determine the resident productivity and we found that non-EM resident saw an average of 1.7 patients/hour, similar to junior EM resident (1.8 patients/hour). Based on this evidence, we assume that attending productivity have no significant differences when attending working with 1) one senior EM resident and one junior EM resident versus 2) one senior EM resident and one non-EM resident. We revised our manuscript and added a discussion to our limitation. (Page 16, Line 344-6)

Impact on attending and resident productivity when students are present

Response: Again, thanks for reviewer’s valued comment. In the study ED, medical students see patients together with our senior EM residents, but they are not allowed to do any documentation in the EMR system during study period. Senior EM residents report patients to the attending and do the medical documentations. This could potentially affect senior EM resident productivity to a certain level. Unfortunately, due to the lack of medical students’ data, we were unable to report this impact on attending productivity when students are present. We revised our manuscript and added to our limitation to address this. (Page 16, Line 344-6)

Line 87: students [5-7]., not students. [5-7].

Response: Yes, it is revised. (see Page 4, Line 87)

Line 92: This is a loaded statement. The APPs were likely practicing independently, but the residents have to do more steps before disposition: present to attending, attending sees patient, then disposition. It's not really a fair comparison.

Response: Thanks for reviewers’ valued comments and sorry for the confusion. This is truly what we want to address: residents in training might be less efficient than other providers not in training statuses (such as APPs). We revised this sentence and make it clearer. (see Page 4, Line 90-93).

Line 164: Define how ED crowding was determined (i.e. NEDOCS score). Also state how overcrowding thresholds were selected, including references.

Response: Yes, we defined ED crowding using NEDOCS in Appendix (Appendix-1.2) and also added the thresholds of not-crowding, crowding, and overcrowding separately with references (see Appendix with separate references). Details are as the followings:

Appendix 1.2: Detail Variables Explanation

ED Length of Stay (LOS) is defined as the time in minutes beginning at the point the patient is initially arrived and registered into the EMR at the ED and ending at the point that the patient physically leaves the ED indicating closure of that specific encounter.

Provider-to-Disposition time (PDT) is defined as the time interval documented in the EMR beginning at the point when the patient is initially seen and evaluated by a provider and ending at the point when the disposition decision is made.

The number of new patients per hour is defined as the number of new patients assigned to individual providers within a one-hour block (e.g., 0200 to 0259).

ED crowding is measured using the NEDOCS score (National Emergency Department Over-Crowding Study) upon each patient’s arrival to the ED (see below). The definitions of these variables are consistent with those previously published (Welch SJ, Asplin BR, Stone-Griffith S, Davidson SJ, Augustine J, Schuur J: Emergency department operational metrics, measures and definitions: results of the Second Performance Measures and Benchmarking Summit. Ann Emerg Med 2011, 58: 33-40.). NEDOCS score equals or less than 100 (NEDOCS≤100) is considered ED not crowding, NEDOCS score less than 140 but greater than 100 (100

Detail Explanation of NEDOCS Score Calculation

Variables Definition

NEDOCS = 85.8T + 600B + 5.64W + 0.93L + 13.4C -20

T The total number of ED patients collected divided by the number of licensed beds at the time a score is calculated

B The number of admitted patients/number of hospital beds at the time a score is calculated

W Longest wait time in hours for patients in the waiting room at the time a score is calculated

L Longest time in hours since registration among boarding patients at the time a score is calculated

C Number of critical care patients at the time a score is calculated. Typically, this is a site-specific variable which usually refers to patients that require one-to-one nursing care. In the study ED, critical care patients are defined as ICU patients and ICU consulted patients including but not limited to patients on mechanical ventilators, receiving tPA, diagnosed with septic shock, critical trauma patients, and patients requiring conscious sedation at the time a score is calculated, etc.

Line 175: please provide citation for STROBE

Response: Yes, it is added. (see Page 8, Line 176)

Line 204: “more elderly”: The statement is a little misleading. Although statistically significant, its not really clinically significant. The means were 48 vs 47 years of age.

Response: Yes, we deleted “more elderly” since there was no clinically significant value for such comparisons. We revised as the following: “It was noted that the attending working solo group saw slightly more high-acuity (ESI-1) patients”. (Page 9, Line 205-206)

Line 205: Fewer high acuity patients seen by solo attendings: This is a little misleading. Most places consider high acuity to be ESI 1&2 (not 1-3). The solo patients saw MORE ESI 1 (highest acuity) patients.

All high acuity patients are seen by an attending regardless of resident presence. There is a work-flow nuance here. On shifts with residents, there are likely fewer attendings, and on attending only shifts there is likely more attending coverage. The attending overseeing residents is likely to get more High Acuity patients because multiple residents are picking them up and presenting them. He/She can manage more at 1 time. On the attending only shifts, they are likely to divide up these patients, so any one attending will have fewer than if they were alone with residents. Please discuss this.

Response: Sorry for the confusion. We revised the result and addressed more accurately as the following: “It was noted that the attending working solo group saw slightly more high-acuity (ESI-1) patients”. This could possibly be due to: 1) the daily variety of high acuity patients present at the study ED (e.g. slightly more ESI-1 patients present during non-resident shifts); and 2) the exclusion of the APPs in this study. During the ED non-resident time (16h/week), an extra APP was scheduled to work. However, this study excluded APPs who usually only saw low-acuity patients. This is the reason that propensity score matching comparison was used in the study to minimize these confounders. We also revised and addressed it in our discussion section. (See Page 13, Line 279-283).

Table 1: what characterized ED overcrowding? Was it a NEDOCS score ≥ level 4?

For tables, please use superscript letters a for legends rather than symbols.

Response: Yes, ED overcrowding was addressed in detail in Appendix. NEDOCS score >140 is considered overcrowding. In addition, we revised and used superscript letters for legends (See revised Table 1 and 2).

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Marina Boushra

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: Response to Reviewers (R2).docx

Click here for additional data file.

23 Jan 2020

Productivity, Efficiency, and Overall Performance Comparisons Between Attendings Working Solo Versus Attendings Working with Residents Staffing Models in an Emergency Department: A Large-Scale Retrospective Observational Study

PONE-D-19-26595R2

Dear Dr. Wang,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Andrew Carl Miller

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

The authors appropriately addressed each of the reviewer comments.

28 Jan 2020

PONE-D-19-26595R2

Productivity, Efficiency, and Overall Performance Comparisons Between Attendings Working Solo Versus Attendings Working with Residents Staffing Models in an Emergency Department: A Large-Scale Retrospective Observational Study

Dear Dr. Wang:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Andrew Carl Miller

Academic Editor

PLOS ONE