Progression of Emergency Medicine Resident Patient Experience Scores by Level of Training.

BACKGROUND: Patient satisfaction surveys are vital to measuring a patient's experience of care. How scores of patients managed by emergency medicine (EM) residents change as residents progress through training is not known.
OBJECTIVES: To evaluate whether EM residents' patient satisfaction scores improve as residency training progresses, similar to clinical skill improvement.
METHODS: A retrospective cross-sectional study evaluated the correlation of patient satisfaction scores with EM resident year of training from 2015 through 2017. We evaluated for a change in score over time for the 4 "physician questions" and the "overall" score.
RESULTS: We evaluated 1684 Press Ganey surveys linked to 40 EM resident physicians during the study period. The mean top box scores for the 4 physician questions (concern for comfort [P = .72], courtesy [P = .55], informative about treatment [P = .46], and listening [P = .91]) and overall assessment of emergency department care (P = .51) were not significantly improved over the course of resident.
CONCLUSION: We did not observe a difference in EM residents' patient experience scores as their level of training progressed. Comprehensive patient experience training for residents might be needed.

Introduction

In 2007, the Institute for Healthcare Improvement created a framework entitled the “Triple Aim,” that identified 3 dimensions of health care to address for health-care improvement (1). These 3 dimensions included population health, per capita cost, and experience of care. Patient satisfaction surveys are vital components to measuring a patient’s “experience of care” (2,3). Patient satisfaction is affected by operational factors such as wait time, along with nursing and provider interactions, listening and communication skills, among other physician- and facility-specific characteristics (4 –6).

In academic emergency departments (EDs), residents provide a significant amount of the provider–patient interaction, and residents have been shown to influence patient experience scores (7). Patient survey vendors are able to report patient satisfaction data by site and by attending provider. However, they are unable to directly report satisfaction data by resident, challenging a program’s ability to evaluate a resident’s performance.

A recent survey by the Council of Residency Directors–Emergency Medicine revealed that less than one-third of emergency medicine (EM) training programs share any patient satisfaction data with residents and showed that only 27% of residencies have dedicated curricula focused on patient experience (8). Residents improve their clinical capabilities throughout their training, from a skill set where they cannot perform independently to a graduating level where they can be ready for independent practice (9). Clinical knowledge and skills progress throughout residency, but little is known about whether a resident’s patient experience scores progress.

Our center used a centralized datamart to combine patient satisfaction scores with the treating EM resident signed up in the chart. We evaluated data from 2015 through 2017 to see if more advanced residents (those closer to the end of training) had better scores than those early in their training.

Methods

Study Design

This is a retrospective cross-sectional study to evaluate the correlation of patient satisfaction scores by resident year of training. Data were gathered as part of a quality improvement project and was deemed exempt from the institutional review board of our institution. Local EM residency leadership approved this project.

Study Setting and Population

The study was conducted at an academic, tertiary care, level 1 trauma center ED that has an integrated pediatric ED. Our ED manages 75 000 patient visits annually, 82% of whom are adults (age >17 years). Thirty-five percent of adult patients and 16% of pediatric patients are admitted. Our EM resident training program is accredited by the Accreditation Council for Graduate Medical Education and has 8 residents annually. Only EM residents were included in this evaluation; off-service rotating residents were excluded due to their brief time in the ED, lack of ability to look for changes over time, and training program differences. Residents were classified by level of training at the time of each visit, broken down by quarter. For example, Post-Graduate Year (PGY) 1 residents start in the third quarter of the calendar year, so the lowest level of training is Q3 PGY1. They finish after the second quarter of PGY3, so the highest level of training is Q2 PGY3.

Data Collection

Patient satisfaction data were obtained via a previously contracted commercial vendor (Press Ganey Associates, South Bend, Indiana). Data were gathered from July 2015 through June 2017. As is commonplace for commercial patient satisfaction hospital contracts, each survey contained the visit date and the attending physician’s name. The resident physician name is not sent to the commercial vendor, so the surveys do not contain the resident’s name. The vendor sends our institution’s survey data to us weekly, and it is stored within our hospital’s databases. We collected all returned satisfaction survey data and, using unique visit identifiers, linked each survey to the resident who was signed up in the electronic medical record as the treating resident for that patient’s visit. Attending physicians infrequently see patients by themselves, but any qualifying visit that did not have a resident signed up was excluded. In addition, we excluded any qualifying visit that had an off-service (non-EM) resident assigned due to their relative inexperience in the ED setting, as well as visits that included a nurse practitioner or physician assistant as an extender.

Study Protocol

Prior to data collection, we developed a protocol and obtained ED quality committee and residency leadership approval. This study adheres to the STrengthening the Reporting of OBservational studies in Epidemiology (10).

Key Outcome Measures

Our key outcome measure was the “Overall rating of care received during your visit” question, referred to herein as “overall rating.” In addition, we analyzed the 4 questions under the “Physician” category, which include: “courtesy of the doctor,” “degree to which the doctor took the time to listen to you,” “doctor’s concern to keep you informed about your treatment,” and “doctor’s concern for your comfort while treating you.” All questions are rated on a scale from 1 (lowest score) through 5 (highest score). We analyzed how frequently, out of all surveys returned, the “top box” (or a rank of 5 out of 5) was chosen—a measurement known as “percent top box.” These scores were stratified across the resident’s training progression, from the first quarter of the PGY1 academic year to the last quarter of the PGY3 academic year.

Data Analysis

Summary scores were grouped by quarter and year of postgraduate training. Assumption of nonindependence between the observations was used, and changes in score each quarter during the 3-year period of training were calculated with a Cochran-Armitage test for trend for each question.

Results

A total of 4550 patient surveys were returned during the study period (21.6% return rate), and 2866 surveys from patients who were not seen by an EM resident (eg, with an off-service resident, nurse practitioner, physician assistant, or less frequently by the attending primarily) were excluded. Due to incompletely filled out surveys, responses among the questions evaluated ranged from 1652 to 1684.

A total of 40 individual residents were evaluated during the study period. All EM residents were included in the data set, with a range of 14 to 16 residents having surveys returned in each of the academic quarters we evaluated (Figure 1).

Figure 1.

Proportion of top box scores over training period. Patient satisfaction scores associated with residents at different levels of training remained unchanged over the course of residency, with nonsignificant P values (Table 1).

Table 1.

Percent Top Box Scores for Each of the 4 “Physician Questions” by Post-Graduate Year (PGY) Level of Training.

		Comfort	Courtesy	Informative	Listening	Overall
PGY1	Q3	0.659	0.674	0.556	0.622	0.609
	Q4	0.63	0.716	0.598	0.664	0.602
	Q1	0.68	0.739	0.672	0.705	0.654
	Q2	0.754	0.798	0.719	0.767	0.688
PGY2	Q3	0.62	0.653	0.55	0.624	0.6
	Q4	0.712	0.791	0.696	0.777	0.626
	Q1	0.648	0.71	0.621	0.676	0.643
	Q2	0.664	0.715	0.656	0.7	0.646
PGY3	Q3	0.608	0.608	0.538	0.613	0.563
	Q4	0.659	0.724	0.667	0.698	0.623
	Q1	0.708	0.764	0.697	0.743	0.7
	Q2	0.634	0.682	0.642	0.662	0.628
P value		.72	.55	.046	.91	.51

Discussion

A resident’s clinical competencies, including communication skills, have been shown to improve over the course of training (9). Our hypothesis was that patient satisfaction scores would improve over the course of training, similar to these clinical and communication skills. As residents become more comfortable navigating the flow of an ED, we expected they would have the opportunity to interact more in a way that would positively affect the patient’s experience. Our analysis shows that when patient satisfaction surveys are linked to the EM residents who cared for them, there is no improvement in score as training progresses.

Shanafelt and colleagues have shown a relationship between increased personal well-being and greater empathy among EM residents (11). Prior studies have shown that medical students have more empathy than residents, and empathy decreases during residency training, possibly secondary to burnout and other factors (12). It is possible that more empathetic residents are more likely to have increased patient satisfaction, but a decline in empathy over training may offset any operational or communication improvements.

The evaluation of patient experience continues to be a focus on the national health-care scene. Very little is known about resident progression in ability to provide a patient-centered experience. As EM residents progress from trainees to attending physicians, they often have little exposure to patient feedback or information on how to improve the patient experience. Meanwhile, hospital compensation for care has included consideration of patient satisfaction since the initiation of Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) and will continue to do so under the Centers for Medicare and Medicaid (CMS) Hospital Value-Based Purchasing (HVBP) program, and eventually the Medicare Access and CHIP Reauthorization Act of 2015 (13). For fiscal year 2017, 2% of Medicare payments will be linked to HVBP. The position of patient satisfaction has evolved from a focus simply on “Patient Experience of Care” in 2013 to “Patient and Caregiver Centered Experience of Care/Care Coordination” measured via HCAHPS, and this patient experience accounts for 25% of the payment modeling calculations used by CMS (14). As hospital finances continue to be entwined with the measurement of patient experience, the livelihood of physicians is also linked—whether simply by the financial viability of the hospital or via patient survey-based compensation adjustments levied by employers.

Prior publications have evaluated the effect of residents in a variety of settings and found mixed results. Some areas found that participation of residents in patient care had no effect on experience (15), and others found that residents were associated with decreased satisfaction (16,17). Resident effect on patient satisfaction has also been looked at through the lens of duty-hour restrictions, showing no improvement with a reduction in duty hours (18). Residents have been evaluated as to their relative impact on the patient experience compared to faculty and nursing staff and found that they do impact scores, however to a lesser degree than nursing (7). Until now, no publication has looked at how EM residents perform as they progress through training.

One prior study evaluated internal medicine residents utilizing a patient satisfaction evaluation instrument from the American Board of Internal Medicine and found that PGY1 trainees had higher scores than more advanced trainees (19). Conversely, Stewart et al found that internal medicine trainee scores remained stable throughout their years of training, despite predicting a decline in scores (20).

Resident training programs have evaluated efforts to improve patient experience. Google Glass was utilized in one program as a way to passively observe a patient encounter and to provide coaching after video review and assessment (21). Simulation training focusing on communication skills has been deployed in training programs and was associated with improved patient experience evaluations (22). A rigorous program of feedback, educational conferences, recognition, and rewards for top performers was utilized in an internal medicine program and they found they had an improvement in HCAHPS scores (23).

In the ED, evaluation of the impact and performance of residents is even more scarce. The effect of bedside patient presentation by resident to faculty was evaluated to see if this affected patient satisfaction and no significant difference was detected (24). Another study evaluated the effect of providing Spanish-language and cultural competency training to residents and found that this did improve the patient experience of care (25). No prior studies that we have found looked specifically at the performance of EM residents on patient satisfaction over time. A recent study by Pines et al showed that several factors were found to be associated with facility and provider scores, although few factors were under the control of either physicians or facilities. Younger physician age, participating in satisfaction training, increasing relative value units per visit, more commercially insured patients, higher computed tomography or magnetic resonance imaging use, working during less crowded times, and fewer night shifts predicted higher scores. Patient satisfaction scores varied widely month to month and even year to year (6).

As we consider why our top box scores did not improve as residents progressed in their training, some plausible explanations are that, at the same time than a resident becomes more comfortable with the ED environment, he or she might need less time at the bedside to initiate the assessment of the patient, might take on more patients simultaneously, and as result might spend less time directly interacting with patients.

Patient satisfaction is complex and affected by multiple factors. We found that there was no difference in patient satisfaction scores as a resident’s training progresses. Future research needs to determine how factors such as increasing knowledge, greater responsibility, increasing stress, and burnout are related to this lack of improvement. There is an opportunity to improve the patient’s experience by including a focus on patient satisfaction within residency curricula.

Limitations

Our study has several limitations. It is a single-center study, involving 1 patient population and 1 training program. We are uncertain whether different patient populations, and different training programs, would produce similar results. Additionally, surveys are attributed to a single provider and do not account for sign out from one care team to another as shifts change over. It is possible that patient satisfaction scores are affected by provider(s) not primarily signed up in the EM resident. Prior studies have evaluated the problematic nature of finding the variables that truly affect results of patient satisfaction surveys (6). We recognize the difficulties in determining which factors do and do not impact an individual patient’s assessment of the aspects of care provided in the ED. However, we also note that because surveys from commercial vendors are widespread in health care and are the basis for reimbursement in many cases, they have real consequences both to hospital systems and at times individual providers. These surveys are the tool that we have to gain insight into the patient perspectives of care across the American health-care system. Utilization of a project-specific, site-specific survey may provide more robust data but would be less applicable to the real-world problem of addressing patient satisfaction and improving the care we provide.

Conclusions

Contrary to clinical skills and milestones progression, we did not find an escalation in performance when evaluating a resident’s patient satisfaction scores. Graduates of EM training programs will find they are evaluated and held accountable for their patient experience scores and, by and large, are not being provided training in this area.

While prior studies in EM have focused on specific interventions such as Spanish-language and cultural competency or bedside presentation, it may be of further benefit to develop, study, and implement a more comprehensive patient experience training program to positively impact patient experience to better prepare them for faculty positions and the future of health care.