Measuring without a ruler: Limited data to characterize the relationship between physician assistant/nurse practitioner staffing and emergency department performance.

OBJECTIVE: Physician assistant (PA) and nurse practitioner (NP) staffing is increasingly common in emergency departments (EDs), with variable physician supervision. We examined the feasibility of using publicly reported metrics as a measure of ED performance by staffing model.
METHODS: We classified a convenience sample of 915 EDs by staffing model using the National Emergency Department Inventory-USA 2016 and a follow-up survey. Staffing models included 24/7 attending coverage with PAs/NPs, 24/7 attending coverage without PAs/NPs, and PAs/NPs without 24/7 attending coverage. We linked EDs with Hospital Compare data to examine availability of metrics and compared metric performance by staffing model. We used regression modeling to examine the independent relationship between staffing model and ED performance after adjusting for ED characteristics.
RESULTS: Of 915 EDs surveyed, 767 (83%) responded and 436 (48%) had complete staffing data and any Hospital Compare data. The 216 EDs without any Hospital Compare data more frequently had no 24/7 attending coverage, were smaller, and were more often rural. Of 5 clinical metrics, 3 had data from < 100 EDs (range: 2%-21%), and 2 had data from 0 EDs. Of the 5 clinical metrics, only median time-to-ECG had enough data for analysis and found no difference among staffing models. Among the 3 process metrics, only time to discharge was significantly faster in EDs with any PA/NP staffing.
CONCLUSION: Many EDs in our national sample lacked sufficient Hospital Compare data to evaluate performance, likely because of lower patient volumes for condition-specific metrics. Alternative strategies to measure quality of care delivery in these settings should be developed.

INTRODUCTION

Background

Emergency departments are increasingly staffed by nurse practitioners (NPs) and physician assistants (PAs), with over 70% of EDs employing PAs/NPs in 2015. In some EDs PAs/NPs are expanding their roles to include more complex procedures and patients.

Importance

Current levels of physician supervision of PAs/NPs in the ED are variable, with approximately 50% of patients being seen by PAs/NPs without physician involvement in 2018. Many states also allow or have ongoing lobbing efforts for independent practice for NPs. Thus, it is increasingly important to understand variation in quality of care delivery and patient outcomes among these varying staffing models. Although there are some studies assessing process metrics, such as time to provider, patient satisfaction, and resource use, data are limited regarding quality of care and patient outcomes. The few small studies available are focused on limited condition‐specific outcomes or limited settings. , Publicly reported metrics, such as Centers for Medicare & Medicaid Services (CMS) Hospital Compare, have been traditionally used as an assessment of clinical care and patient outcomes and may be valuable to address this question.

Goals of this investigation

Our primary objective was to assess the feasibility of publicly reported Hospital Compare data for evaluating care delivery and patient outcomes among EDs with non‐physician staffing models. Secondarily, we aimed to determine whether ED performance on publicly available clinical and process care outcomes reported by Hospital Compare varied by staffing model.

METHODS

Study design

This is a secondary analysis of data collected for a previously‐described study focused on understanding barriers and facilitators of ED telemedicine use. Briefly, the National ED Inventory (NEDI)‐USA is a survey that was sent to all US EDs that were open in 2016 and includes questions on basic ED characteristics including size and resource availability (eg, telemedicine use; full survey available in the Appendix). The survey is sent to all US EDs that are open 24/7 and available to the general public; it includes both hospital‐based and freestanding EDs. Surveys are mailed to ED directors and may be completed on paper, in an online version, or over a telephone call. Of EDs responding to the NEDI‐USA survey, we then sent followup surveys to a sample of 915 EDs (Appendix). This sample was based on the original study design and included 453 rural EDs without telemedicine and another 462 rural and urban EDs with telemedicine for various clinical applications. The follow‐up survey included questions on ED staffing, including number of full‐time equivalent (FTE) ED attending physicians, number of FTE PAs, and number of FTE NPs. The survey also asked “Is at least 1 attending physician (not resident) on duty in the ED 24 hours/day, 7 days/week? (exclude on‐call physicians)” and “If NO, when a physician is unavailable, is any physician available to the ED by 2‐way voice communication 24 hours/day, 7 days/week: (i) From within your hospital? (ii) From outside of your hospital?” We used responses to categorize EDs as those with 24/7 attending coverage plus PAs/NPs, those with 24/7 attending coverage without PAs/NPs, and those with PAs/NPs without 24/7 attending coverage. We excluded 26 EDs without 24/7 coverage.

This study examined outcome differences among emergency departments with or without advanced practice clinicians. After linking data from a national emergency department (ED) survey with Centers for Medicare & Medicaid Services Hospital Compare data, the results showed that a high proportion of missing data in Hospital Compare precluded its use for this purpose. This finding highlights the need to standardize metrics to compare EDs.

Variables of interest

Our outcomes of interest were availability of, and performance on, publicly reported Hospital Compare metrics. After undergoing a linkage process to consolidate NEDI‐USA survey responses to CMS hospital identification numbers, responding EDs were linked with 2017 Hospital Compare measures covering calendar year 2016. We identified 5 clinical metrics of time‐sensitive care and 3 process of care metrics of interest.

Other covariates of interest were annual ED volume, academic status (defined as a primary teaching site for an emergency medicine residency), number of ED beds (based on NEDI‐USA), number of FTE attendings (based on NEDI‐USA), ED‐reported stroke center status based on Joint Commission or alterative certification, transfer rate among all adults requiring admission (based on follow‐up survey), rural location (based on location outside of a core‐based statistical area), presence of physician available for communication within/outside of hospital (based on follow‐up survey), receipt of telemedicine services (based on NEDI‐USA), and proportion of attendings who are board certified/eligible by the American Board of Emergency Medicine/American Osteopathic Board of Emergency Medicine (based on follow‐up survey).

Data analysis

After determining availability of Hospital Compare data, we examined the characteristics of EDs with and without availability. Next, among those with Hospital Compare data, we used standard descriptive statistics to report characteristics of EDs overall and by staffing model and to describe performance on each metric overall and by ED staffing model. We examined differences in characteristics across the 3 staffing models using x and Kruskal‐Wallis tests as appropriate.

Among the 4 metrics for which we had sufficient data, we fit linear or logistic regression models, depending on the metric, to determine associations between the staffing model and ED metric performance. For median time to ECG for acute myocardial infarction (AMI), we compared EDs with and without 24/7 attending coverage in order to focus on the effect of in‐person attending coverage. For the 3 ED process of care metrics, we compared EDs with and without PAs/NPs to isolate the effect of PAs/NPs involvement. Other covariates in the model were specified a priori based on clinical experience, prior literature, and study hypotheses. All analyses were conducted with Stata IC version 15.1 (StataCorp, College Station, TX, USA).

RESULTS

EDs and availability of hospital compare data

Of the 5375 US EDs open in 2016, 4507 EDs responded to the NEDI‐USA survey (84%), and 915 received the second survey. After linkage of the 767 (84%) responding EDs to the CMS Hospital Compare data set 653 reported data on staffing, and 436 of these had data in Hospital Compare (Figure 1). One ED was not found in Hospital Compare; the other 216 EDs without any Hospital Care data more frequently had no 24/7 attending coverage; were smaller with respect to volume, bed size, and physician FTEs; and more often were rural (Table 1).

FIGURE 1

Flow chart of study. Abbreviations: AMI, acute myocardial infarction; CMS, Centers for Medicare & Medicaid Services; CT, computed tomography

TABLE 1

Comparison of EDs with and without available hospital compare data

ED characteristics	Included EDs (n = 436)	Excluded EDs (n = 216)	P
Staffing, n (%)			<0.001
EDs with 24/7 attending and PAs/NPs	174 (40)	37 (17)
EDs with 24/7 attending and no PAs/NPs	161 (37)	75 (35)
EDs with PAs/NPs and no 24/7 attending coverage	101 (23)	104 (48)
Annual ED volume, median (IQR)	8190.5 (3776–18288)	2920 (1097.5–7269)	<0.001
Rural, n (%)	311 (71)	185 (86)	<0.001
Academic ED, n (%)	5 (1)	0	0.11
Number of ED beds, median (IQR)	7 (5–12)	4 (3–7)	<0.001
Number of FTE ED attendings, median (IQR)	4 (2–6)	2 (1–4)	<0.001
Among EDs without 24/7 attending in ED, presence of physician available for communication WITHIN the hospital, n (%)	48 (49)	38 (37)	0.09
Among EDs without 24/7 attending in ED, presence of physician available for communication from OUTSIDE the hospital, n (%)	89 (89)	93 (89)	0.92
Receive telemedicine services, n (%)	231 (53)	96 (44)	0.04
Joint Commission stroke certified, n (%)	94 (22)	33 (15)	0.06
Comprehensive stroke center a , n (%)	12 (13)	3 (9)	0.26
Primary stroke center a , n (%)	34 (37)	7 (21)
Acute stroke ready hospital a , n (%)	30 (33)	16 (48)
Not sure a , n (%)	16 (17)	7 (21)
Alternative stroke certification (not Joint Commission certified) b , n (%)	80 (24)	26 (14)	0.01
Percentage of adult patients requiring admission that were transferred to another facility			0.004
0%–4%, n (%)	68 (16)	26 (12)
5%–19%, n (%)	198 (47)	74 (35)
20%–49%, n (%)	89 (21)	56 (26)
50%–79%, n (%)	43 (10)	37 (17)
80%–100%, n (%)	26 (6)	19 (9)

Abbreviations: ED, emergency department; FTE, full‐time equivalent; IQR, interquartile range; PAs/NPs, physician assistants/nurse practioners.

Among hospitals with Joint Commission stroke certification.

Among hospitals without Joint Commission stroke certification.

Among the 436 EDs with Hospital Compare data, the median number of measures reported was 5 (interquartile range 3–5) and ranged from 1–7 with none of the EDs reporting all 8 measures. Of the 8 measures, the most frequently available was median time to provider (96%) and the least frequently available was median proportion of patients with fibrinolytic within 30 minutes for acute myocardial infarction (2%). Three of the 5 clinical care metrics were available for less than 25% of our sample (Figure 1).

Among the group of EDs with PAs/NPs without 24/7 attending coverage (n = 101), there were 0 EDs reporting data for 2 of the metrics (median time to transfer for AMI and median proportion of patients receiving brain imaging within 45 minutes for stroke), and only 1 ED reporting data for median proportion of patients with fibrinolytic within 30 minutes for AMI (Appendix Table A1).

ED characteristics by staffing model

EDs with PAs/NPs and no 24/7 attending physician coverage tended to be smaller with fewer beds and seeing a lower annual volume and were more often rural. Of those EDs, none were academic hospitals, fewer were stroke centers (Appendix Table A1).

ED performance on available hospital compare metrics

We were able to examine the relationship between staffing model and ED performance on only 4 of the intended 8 metrics (Table 2). Of the clinical care metrics, median time to ECG for AMI was low across all staffing models. Median proportion of patients receiving aspirin within 24 hours for AMI was uniformly high negating any comparison across groups, and the other 3 had too few EDs reporting data to make comparisons feasible. Of the process of care metrics, left without being seen rates were low across all staffing models. Median time to provider and median time to discharge were somewhat lower in EDs with PAs/NPs without 24/7 attending coverage.

TABLE 2

ED characteristics associated with performance on clinical and process care metrics in adjusted analysis

	Median time to ECG for AMI (n = 307)		Left without being seen (n = 423)		Median time to provider (n = 404)		Median time to discharge (n = 403)
	B (95% CI)	P	OR (95% CI)	P	B (95% CI)	P	B (95% CI)	P
ED with 24/7 attending coverage
No	Referent
Yes	−0.70 (−2.59, 1.19)	0.46
ED with PAs/NPs
No			Referent		Referent		Referent
Yes			0.73 (0.48, 1.13)	0.16	−2.02 (−4.64, 0.60)	0.13	−7.57 (−13.52, −1.62)	0.01
Annual ED volume
< 10,000 visits	Referent		Referent		Referent		Referent
10,000−19,999 visits	−1.71 (−3.18, −0.24)	0.02	2.85 (1.72, 4.71)	<0.001	2.05 (−0.94, 5.05)	0.18	11.50 (3.88, 19.11)	0.003
20,000−39,999 visits	−1.46 (−3.49, 0.56)	0.16	9.95 (4.02, 24.59)	<0.001	8.11 (3.97, 12.24)	<0.001	34.55 (24.14, 44.96)	<0.001
≥40,000 visits	−1.73 (−4.74, 1.27)	0.26	7.27 (2.13, 24.87)	0.002	10.56 (3.75, 17.36)	0.002	47.28 (32.04, 62.53)	<0.001
Number of ED beds
< 5 beds	Referent		Referent		Referent		Referent
5–19 beds	−1.04 (−3.00, 0.93)	0.30	5.46 (2.94, 10.13)	<0.001	1.01 (−1.51, 3.53)	0.43	11.47 (5.26, 17.68)	<0.001
≥20 beds	−0.57 (−3.35, 2.21)	0.69	5.59 (1.91, 16.38)	0.002	3.01 (−2.32, 8.35)	0.27	18.26 (6.48, 30.05)	0.002
Transfers
0%–4%	Referent		Referent		Referent		Referent
5%–19%	−1.24 (−2.86, 0.39)	0.14	0.72 (0.41, 1.27)	0.26	−1.41 (−4.62, 1.80)	0.39	−4.52 (−11.65, 2.62)	0.21
20%–49%	−1.37 (−3.10, 0.35)	0.12	0.67 (0.35, 1.29)	0.23	−0.78 (−4.62, 3.07)	0.69	−1.78 (−10.56, 6.99)	0.69
50%–79%	−0.70 (−2.97, 1.56)	0.54	0.55 (0.25, 1.18)	0.13	0.30 (−3.92, 4.51)	0.89	1.60 (−7.54, 10.73)	0.73
80%–100%	−1.63 (−4.21, 0.95)	0.21	0.43 (0.16, 1.13)	0.09	−2.75 (−7.15, 1.65)	0.22	−0.14 (−12.93, 12.64)	0.98
Receive telemedicine services
No	Referent		Referent		Referent		Referent
Yes	−0.57 (−1.85, 0.72)	0.39	0.62 (0.38, 1.01)	0.053	−1.55 (−3.66, 0.56)	0.15	−3.74 (−8.88, 1.40)	0.15
Urbanicity
Urban	Referent		Referent		Referent		Referent
Rural	−0.44 (−2.19, 1.30)	0.62	0.88 (0.43, 1.80)	0.73	2.78 (−0.33, 5.89)	0.08	1.32 (−6.54, 9.19)	0.74

Abbreviations: AMI, acute myocardial infarction; CI, confidence interval; ED, emergency department; OR, odds ratio.

ED characteristics associated with metric performance

Because of the limited data available for 3 of the clinical care metrics (median time to transfer for AMI, median proportion of patients receiving aspirin within 24 hours for AMI, and median proportion of patients receiving brain imaging within 45 minutes for stroke), and the consistently high performance on median proportion of patients receiving aspirin within 24 hours for AMI, the only clinical care metric we examined in bivariate and logistic analyses was median time to ECG, which did not vary by ED staffing model (Table 2).

Among the process of care metrics, after accounting for annual ED visit volume, bed size, transfer rate, telemedicine receipt, and urbanicity, ED staffing models were not associated with performance, with the exception of time to discharge. Median time to discharge was 8 minutes faster among EDs with PAs/NPs relative to EDs without PAs/NPs (Table 2).

Limitations

Potential limitations of this study include the nature of our convenience sample. The sample of EDs were selected for use of telemedicine, including rural EDs that did not use telemedicine, as well as rural and urban ED with telemedicine; therefore, results may not be generalizable. Bias related to survey non‐response was possible, but we believe this is mitigated by the > 80% response rate that we achieved. There is also potential for bias related to presence of data in Hospital Compare, given that lower volume was the most likely reason for absence of data. EDs without data tended to have lower volume and different staffing models than those with data. This further underscores the importance of alternative data sources for evaluating quality and performance of alternative staffing models in lower volume EDs.

DISCUSSION

Our findings underscore the absence of data for most clinical care metrics available in Hospital Compare for this sample of smaller, more rural EDs. Our intended sample was reduced from 652 EDs to the included 436 because 216 EDs had no Hospital Compare data available. Of those EDs with data, only 5 of the intended 8 metrics were widely available, with reporting by hospital varying from 1 to 7 metrics, and none with data for all 8 metrics.

Staffing models among these EDs varied, with 23% staffed with PAs/NPs without 24/7 attending physician coverage. EDs with primary PA/NP coverage and less physician coverage tended to be smaller with fewer beds and lower annual volumes and more likely to be rural consistent with prior research. In addition, we found that they were less likely to have high‐level stroke certification of any type and none were academic.

Hospital Compare data are dependent upon having sufficient case volume so that significant outliers do not introduce noise into the data. A number of possible reasons for missing data are outlined on Hospital Compare, including the number of cases/patients being too few to report; results being based on shorter time period than required; hospitals not participating in the Inpatient Quality Reporting and Outpatient Quality Reporting programs; no cases met the criteria for a measure; and results that are not available for the reporting period. Given the characteristics of the EDs in our sample, with relatively low annual ED volume, it is most likely that the absence of metric data among these EDs was an artifact of smaller EDs with lower volume of cases.

Because of low numbers of critical cases in these smaller hospitals, looking to existing reported clinical metrics to assess quality of care is not a viable option for evaluating the quality of care delivery in these EDs. Because these lower volume EDs are more likely to have non‐physician staffing models it is imperative that an alternative strategy for evaluating quality is found in order to identify potential problems and thereby ensure consistently high‐quality emergency care. With the current policy environment and controversies regarding staffing and independent practice, our findings highlight the importance of finding other data sources or metrics to adequately assess clinical outcomes in these setting. Future work should concentrate on finding alterative metrics that are not subject to the limitations of small sample size and less dependent on high volume of specific conditions. Further complicating the picture is the lack of standardization of how PAs/NPs are integrated into physician workflow in EDs, with some EDs having very little attending physician oversight and others having a great deal. Standardization of oversight would be a good first step in being able to find measurable data. Subsequent steps might then include studies assessing metrics that are not condition specific or other evidence‐based metrics such as computed tomography use and risk assessment for patients undergoing a workup for pulmonary embolism and antibiotic use in patients with bronchitis or sinusitis. Although less critical, these data may be more prevalent and therefore not as dependent on volume.

Although bivariate comparisons were suggestive that some of the process of care metrics may be improved with PA/NP involvement, after accounting for other important ED characteristics, only time to discharge remained faster among EDs with PA/NP involvement and the association between staffing model and performance on the other metrics was no longer significant. Given that ED volumes were quite different between EDs with different staffing models, and ED volume was also strongly associated with metric performance in the multivariable models, it is likely that differences in volume in EDs with different staffing models confounded the bivariate relationship and therefore volume, not staffing may be implicated. Persistently significant differences in time to discharge by ED staffing model in the regression model may reflect residual confounding from ED‐level differences. It is also possible, though we suspect less likely, that this is due to other causes, such as different work styles, as this was an isolated finding among the process metrics.

In conclusion, EDs without 24/7 attending coverage tended to be smaller, rural, lower volume EDs and as a result have low patient volumes for disease‐specific metrics and insufficient data available in CMS Hospital Compare and, therefore, Hospital Compare cannot be used to meaningfully compare these EDs with different staffing models. It will be important to develop alternative strategies to measure quality of care delivery in these settings.

CONFLICT OF INTEREST

The authors declare conflict of interest.

AUTHOR CONTRIBUTIONS

Elizabeth S. Temin, Margaret E. Samuels‐Kalow, and Kori S. Zachrison conceived and designed the study. Krislyn M. Boggs and Carlos A. Camargo acquired the data. Rebecca E. Cash analyzed the data. All authors contributed to interpretation of the results. Elizabeth S. Temin drafted the manuscript and all authors contributed to critical revisions of the manuscript. Elizabeth S. Temin takes responsibility for the study as a whole.

SUPPORTING INFORMATION

Click here for additional data file.