Dimensions of Patient Experience and Overall Satisfaction in Emergency Departments.

OBJECTIVE: To determine the correlation between individual patient experience dimensions and overall patient satisfaction using text-based analysis of subjective comments of patients treated in emergency departments.
METHODS: Open-ended comments from 331 patients who visited the emergency departments of 4 hospitals were used for coding different dimensions of patient experience. Regression coefficients were calculated to assess the relationships between dimensions of patient experiences with overall satisfaction.
RESULTS: Positive and negative experience of nursing, communications, and infrastructure influence the overall satisfaction. Positive experience attributes of overall care quality influence overall satisfaction, whereas negative experience of the same does not have any influence. Further, experiences of interactions with doctors and scheduling do not have any effect on overall satisfaction in emergency departments.
CONCLUSIONS: Emergency departments may get higher overall patient evaluations by focusing on positive aspects of care, nursing, communication, and infrastructure attributes. Doctors and scheduling (emergency) may be considered as expected quality attributes and so not surprising that they did not play a role in overall satisfaction.

Introduction

Patient reviews of their experience and satisfaction with care delivery is emerging as a benchmark for health-care quality in the United States. Centers for Medicare and Medicaid Services measures patient experience using Hospital Consumer Assessment of Healthcare Providers and Systems and now uses it to differentiate payment reimbursements for care delivery. On the other hand, online reviews are becoming a source for patient decision-making, for example, choosing a doctor. A Pew Internet and American Life study found that 37% of American adults who are looking for information about doctors (ie, about 20% of all American Internet users) have reviewed online Web sites to select doctors or other providers (1). A study by Lagu et al (2) reports that there are more than 30 doctor rating Web sites, such as rateMDs.com or avvo.com, and several hospital rating Web sites such as yelp.com or hospitalcompare.com.

Irrespective of the increasing focus by patients on ratings and reviews of health-care providers, there is concern about the adequacy and reliability of such ratings. The primary concern is regarding the reliability of such ratings, as often a patient’s overall rating is not a true indicator of the quality of the medical services provided. One needs a more granular level of ratings related to care effectiveness on several dimensions. Studies suggest that a patients’ subjective assessment of the care provided may not be credible because patients lack the medical training to fully understand various facets of care. Any attempt to optimize care based on patients’ satisfaction may lead to lower quality of care (3,4).

A second criticism is that the overall rating does not reflect what a person is thinking or want to say about his experience, as numerical scores have inherent challenges in capturing one’s subjective experience (5). For example, a patient may feel very happy about his or her treatment, but when it comes to providing a score, he/she may give a rating of 3 of 5 thinking that it is a high enough rating. Or, a slightly disappointed patient may actually give a very low score of 0 thinking that a score of 1 may lead hospital thinking that there is not sufficient disappointment. In other words, objective ratings often may not capture the subjectivity of the patient’s perception of the quality of care. However, interpretation of the subjective reviews expressed in a free-form text is also a challenge, unless a good text mining and analytics engine can capture the review details and interpret them in a meaningful way.

Against this criticism that existing review systems do not truly reflect a patient’s interactions with care provisions, this study uses a mixed analytic approach: using manual coding of qualitative comments (similar to content analysis rather than automated text mining) and statistical analysis to gain insights into what actually contributed to patient satisfaction in emergency departments (EDs). We used a data set of open-ended comments from 331 patients who visited the EDs of 4 hospitals. We found both negative and positive comments on nursing care, infrastructure, and communication using our coding process and analyzed the coded comments statistically to a patient’s overall satisfaction. Furthermore, the findings show that the positive experience with care delivery influences overall satisfaction, whereas negative experience appears to show no influence. However, experiences related to interactions with doctors and scheduling do not appear to have any effect on the overall satisfaction in these EDs.

Method

Data and Variables

Data for this study are from a payer firm who is responsible for providing physicians and other contract services to hospitals and is directly responsible for physicians’ revenues and partially responsible for the success of ED operations. The firm has internal tracking and monitoring system that uses Press Ganey surveys used by hospitals in the United States. The surveys used in this study were administered by 4 hospitals in the NE region of United States between January and March 2013. Using a structured protocol typically followed by hospitals, the surveys were mailed to patients within 2 weeks of their visit to the ED. Due to confidentiality reasons, the firm could not share the complete survey for this study and provided only deidentified patient comments from these surveys for this study. The sample has a total of 331 patients, with 70, 37, 78, and 146 patients from each of the 4 hospitals.

The variables used in this study are coded from the comment texts manually. Three researchers have reviewed the coding process and outcomes thoroughly; any discrepancies were resolved with discussions. Table 1 provides the examples of the words and verbatims used to code the variables. Table 2 shows the description of variables used in this study. Table 3 presents descriptive statistics for the 5 EDs’ pooled data, with pairwise correlations among the variables.

Table 1.

Sentiment Words and Verbatim Examples.

Scorea	Examples of Positive Sentiment Words	Examples of Positive Sentiment Verbatim	Examples of Negative Sentiment Words	Examples of Negative Sentiment Verbatim
1	Good, fine, decent, worthy	Nurse XYZ was Nick was terrific!! Very happy with treatment. We had a good experience and worthy of our time.	Bad, regret, poor, inadequate, awful	This was an AWFUL experience! I have mailed letter to Dr MNP. I was so much in pain without anyone’s care, so I consider it bad experience. Was admitted but D/C from hospital POOR!!!!
2	Great, superb, remarkable, wonderful	The treatment was superb!!! ABC staff was great. She made sure I was comfortable. Every time we’ve been there the nurses have been wonderful!	Worse, inferior, deficient, indigent	Really just wrote me…(back injury). Did not ask range of motion, etc. Worse experience Needed chair to elevate injured ankle—none offered. So deficient infrastructure Doctor had an attitude. I would never see this doctor—indigent. Not sure about his name. He is older guy.
3	Excellent, awesome, astounding, amazing, terrific, splendid	Dr ABC accessed me very quickly! He took great care of me!! Excellent doctor!! Excellent communication skills - made sure we understood all our options.	Worst, poorest, unfortunate, substandard, filthy	We arrived around 10:30 pm with a 1-year-old and a 2-year-old. We waited almost 2 hours and our daughter was never visited by anyone. No doctor. No nurses, nothing. We had the left without treatment and went elsewhere. We had the worst night Will never return or recommend. It is unbelievable how poorly we were treated at a medical facility. Very unacceptable!! Unfortunate…Never!

a+ve score for positive sentiments and −ve score for negative sentiments.

Table 2.

Description of Variables.

SATS	Overall satisfaction of the patient, measured and coded on a scale of −3 to +3
PCCARE	Whether the patient positively commented about emergency care. 1 = yes, 0 = no
NCCARE	Whether the patient negatively commented about emergency care. 1 = yes, 0 = no
PCDOC	Whether the patient positively commented about doctors. 1 = yes, 0 = no
NCDOC	Whether the patient negatively commented about doctors.1 = yes, 0 = no
PCNURS	Whether the patient positively commented about nurses. 1 = yes, 0 = no
NCNURS	Whether the patient negatively commented about nurses. 1 = yes, 0 = no
PCCOM	Whether the patient positively commented about communication. 1 = yes, 0 =no
NCCOM	Whether the patient negatively commented about communication. 1 = yes, 0 = no
PCSCD	Whether the patient positively commented about scheduling. 1 = yes, 0 = no
NCSCD	Whether the patient negatively commented about scheduling. 1 = yes, 0 = no
PCINFR	Whether the patient positively commented about infrastructure. 1 = yes, 0 = no
NCINFR	Whether the patient negatively commented about infrastructure.1 = yes, 0 = no

Table 3.

Descriptive Statistics and Pairwise Correlation Among Variables.a

	Variable	Obs	Mean	SD	Min	Max	1	2	3	4	5	6	7	8	9	10	11	12	13
1	SATS	331	0.15	1.25	−3	3	1.00
2	PCCARE	331	0.13	0.33	0	1	0.32	1.00
3	NCCARE	331	0.09	0.29	0	1	−0.31	−0.12	1.00
4	PCDOC	331	0.03	0.18	0	1	0.10	0.18	−0.06	1.00
5	NCDOC	331	0.07	0.25	0	1	−0.26	−0.10	0.30	−0.05	1.00
6	PCNURS	331	0.04	0.19	0	1	0.21	0.12	−0.06	0.32	−0.05	1.00
7	NCNURS	331	0.04	0.19	0	1	−0.24	−0.07	0.16	−0.04	0.14	−0.04	1.00
8	PCCOM	331	0.24	0.43	0	1	0.21	0.30	−0.18	0.17	−0.15	0.23	−0.11	1.00
9	NCCOM	331	0.15	0.36	0	1	−0.25	−0.16	0.23	−0.08	0.27	−0.08	0.15	−0.24	1.00
10	PCSCD	331	0.03	0.18	0	1	0.08	0.23	−0.06	0.25	−0.05	0.14	−0.04	0.25	−0.08	1.00
11	NCSCD	331	0.07	0.26	0	1	−0.27	−0.11	0.36	−0.05	0.21	−0.05	0.28	−0.16	0.24	−0.05	1.00
12	PCINFR	331	0.18	0.39	0	1	0.30	0.36	−0.15	0.22	−0.13	0.12	−0.09	0.23	−0.20	0.26	−0.13	1.00
13	NCINFR	331	0.19	0.40	0	1	−0.24	−0.19	0.43	−0.09	0.25	−0.10	0.11	−0.28	0.28	−0.09	0.25	−0.23	1.00

Abbreviations: Max, maximum; Min, minimum; NCCARE, negative comments about care; NCCOM, Negative Comment about Communication; NCDOC, Negative Comment about Doctor; NCINFR, Negative Comment about Infrastructure; NCNURS, Negative Comment about Nurses; NCSCD, Negative Comment about Scheduling; PCCARE, positive comments about care; PCCOM, Positive Comment about Communication; PCDOC, Positive Comments about Doctor; PCINFR, Positive Comment about Infrastructure; PCNURS, Positive Comment about Nurses; PCSCD, Positive Comment about Scheduling; SATS, satisfaction; SD, standard deviation.

aAll correlations above 0.1 are significant at P < .01 level.

We used the variable “overall satisfaction” (SATS) on a scale of −3 to +3 reflecting a patient’s satisfaction level based on a rule-based hierarchy of words. For example, neutral or mixed comments were used to code the zero value for SATS. The words similar to “good” or “fine” were used to provide the sentiment of additional point and were coded as overall satisfaction with +1 value. Furthermore, the words “great” or “superb” were used to code +2 value, and words similar to “excellent” or “awesome” were used to code +3 values. Similar hierarchy of negative adjectives and words was used to code the negative values for SATS.

The independent variables were also coded for words and sentiments expressed on dimensions that a patient viewed as serious enough to express a positive or negative sentiment in their comments. For example, if the patient has mentioned care delivery (CARE) or treatment in the ED in a positive manner, then the variable is defined as Positive Comments about Care (PCCARE) and given a value 1. If the patient has mentioned negatively about care delivery, the variable Negative Comments about Care (NCCARE) was given a value of 1. Similar rules were applied to code the variables Positive Comments about Doctor (PCDOC) and Negative Comment about Doctor (NCDOC) when doctors were mentioned; Positive Comment about Nurses (PCNURS) and Negative Comment about Nurses (NCNURS) when nurses were mentioned; Positive Comment about Communication (PCCOM) and Negative Comment about Communication (NCCOM) for communication issues raised; Positive Comment about Scheduling (PCSCD) and NCSCD when comments are made on scheduling issues; and finally Positive Comment about Infrastructure (PCINFR) and NCINFR for patient comments on infrastructure. Each of these CARE, DOC, NURS, COM, SCD, and INFR attributes forms the dimensions on which the patient is evaluating care quality of the ED.

Analyses and Results

The dependent variable in our analysis is the interval scale variable SATS. We used an ordered Probit and ordinary least square models for empirical analysis. The ordered Probit model does not assume equal intervals between levels in the dependent variable SATS and does a nonlinear estimation, whereas the ordinary least square assumes a linear specification. Table 4 presents results of estimation models. Column 1 presents the ordered Probit specification and column 2 the ordinary least square estimation. We obtain broadly similar results for ordinary least square and ordered Probit models. To account for the fact that some rooms in the EDs have issues that may influence patient satisfaction, we report the estimated coefficients with cluster-adjusted standard errors by the rooms (6).

Table 4.

Estimation Models.a

Variables	Ordered Probit	Ordinary Least Square
	(1)	(2)
	SATO	SATO
PCCARE	0.271b (0.152)	0.236b (0.122)
NCCARE	−0.048 (0.247)	−0.016 (0.188)
PCDOC	0.089 (0.246)	0.050 (0.191)
NCDOC	0.348 (0.263)	0.261 (0.214)
PCNURS	0.715c (0.321)	0.582c (0.263)
NCNURS	−1.049d (0.371)	−0.879d (0.309)
PCCOM	1.004d (0.135)	0.781d (0.107)
NCCOM	−0.517c (0.258)	−0.391b (0.198)
PCSCD	0.049 (0.229)	0.040 (0.187)
NCSCD	−0.185 (0.264)	−0.153 (0.211)
PCINFR	0.528d (0.161)	0.416d (0.120)
NCINFR	−1.258d (0.237)	−1.040d (0.162)
LOS	−0.001 (0.008)	−0.002 (0.006)
Age	0.004 (0.003)	0.003 (0.002)
Gender	0.106 (0.130)	0.095 (0.107)
Observations	331	331
R 2		.502
Adj./Pseudo R 2	.225	.472
χ2/F-stat	187.07d	52.61d

Abbreviations: NCCARE, Negative Comments about Care; NCCOM, Negative Comment about Communication; NCDOC, Negative Comment about Doctor; NCINFR, Negative Comment about Infrastructure; NCNURS, Negative Comment about Nurses; NCSCD, Negative Comment about Scheduling; PCCARE, Positive Comments about Care; PCCOM, Positive Comment about Communication; PCDOC, Positive Comments about Doctor; PCINFR, Positive Comment about Infrastructure; PCNURS, Positive Comment about Nurses; PCSCD, Positive Comment about Scheduling; LOS, Length of Stay.

a Standard errors in parentheses. Standard errors are adjusted for 130 clusters accounting for same rooms. Models have appropriate intercepts and include hospital and shift dummies, none of which are significant.

b P < .1.

c P < .05.

d P < .01.

We find that the coefficient of PCCARE is positive and significant (column 1, Table 4, β = 0.271, P < .1), indicating that if a patient mentions care in a positive manner, it leads to increase in overall satisfaction. We also find that PCNURS coefficient is also positive and significant (column 1, Table 4, β = 0.715, P < .05), suggesting that positive words about nurses influence an increase in overall satisfaction. Similarly, NCNURS is significant and negative (column 1, Table 4, β = −1.049, P < .01), indicating that negative comments or words about nurses reduce the overall satisfaction.

Regarding communication attributes, the results show that PCCOM is also positive and significant (column 1, Table 4, β = 1.004, P < .05), suggesting that positive comments about communication influences an increase in overall satisfaction. We find that NCCOM is significant and negative (column 1, Table 4, β = −0.517, P < .01), indicating that negative comments on communication reduces the overall satisfaction.

The coefficient of PCINFR is also significant and positive (column 1, Table 4, β = 0.528, P < .05) and NCINFR is significant and negative (column 1, Table 4, β = −1.258, P < .05). Together, these imply that positive words about the infrastructure during a patient’s ED stay lead to increase in the overall satisfaction and vice versa.

We did not find NCCARE, PCDOC, NCDOC, PCSCD, and NCSCD as significant in the models. In other words, patients’ negative perceptions regarding care, positive or negative comments on doctors, and positive or negative comments regarding the schedules have no influence on the overall satisfaction. We conducted a number of robustness checks for our analysis. We tested for multicollinearity by computing condition indices. The mean variance inflation factor was less than 4 in our models, indicating that multicollinearity is not a serious concern in our analyses. In addition, because the dependent and independent variables came from the same survey instrument, we conducted Harman 1-factor test to assess the sensitivity of our results to common method bias. The principal component analysis for key variables yielded multiple factors, some with eigenvalues exceeding 1. Because no single factor emerged as a dominant factor accounting for most of the variance, common method variance does not seem to be a serious problem.

Discussion

The objective of this study was to assess the relationship between patient’s comments on different dimensions, each representing attributes associated with service in an ED and the overall patient satisfaction. The design of the study used manual coding of the text associated with a patient’s subjective comments on each dimension identified for positive or negative comments that described patient experience, rather than any objective ratings for these positive and negative comments. The empirical analysis of the coded data related patients experience on these dimensions to overall satisfaction.

We found that only positive comments on overall care delivery and both positive and negative comments on nurses, communications, and infrastructure have a direct influence on higher or lower overall satisfaction, respectively. In other words, the perceptions associated with nurses, communication, infrastructure, and only positive care issues in an ED have an influence on patient’s overall satisfaction.

Three managerial implications can be drawn from this study. First, patient care and interaction are highly relevant than treatment by a doctor or how one ended up in an ED (planned or unplanned scheduling). Among the care and interactions, patients do pay attention to positive care provided by the ED, and when they mention it on their reviews positively, it has influence on their overall satisfaction. However, they either do not mention negative comments on care delivery and even if they do mention it on their reviews they seem to have no influence on their overall satisfaction. Part of the reason is possibly because they don’t expect great care delivery in ED because of the complexity of care provided and if any positive care experience is noted and may influence patient satisfaction.

Nurses are highly influential toward overall satisfaction, more so on their negative perceptions than positive reviews. Similarly, positive communication reflects better on overall satisfaction than negative communication. However, negative infrastructure has a higher bearing on satisfaction than positive infrastructure issues, although both appear in patient reviews.

A contribution of this study is to inform the design and development of systems to analyze patients’ text reviews. Indeed, information systems using automated opinion mining and sentiment analysis using natural language processing are now emerging in health care to conduct real-time analysis of the reviews and provide information to take immediate action (7). The rule-based hierarchy mentioned in this study can be incorporated in such systems to make those highly effective.

The results can be explained through multiple theories. For example, a line of research in service management literature suggests that experience with core, peripheral, and support services comprise the overall experience, with differing impacts (8). Consumers often take a core service for granted and expect peripheral or support services to be great if they provide them better experience. For example, travelers may view that an airline’s core service is to transport them in time to their destination. However, they give high importance to catering or baggage handling services (9). Similarly, instead of a hotel’s room comfort or bedding (often a core service), consumers ascribe higher importance in their reviews to the reception staff or valet services (10). From these perspectives, it can be argued that although patients come to EDs, they view that a hospital’s core service is for a doctor to treat them with effectiveness. However, the other aspects such as nursing care, communication on the situation at hand, and hospital infrastructure to support a patient that comes under significant stress become important, leading them to remember their experiences on these and put these in comments and influence their overall satisfaction.

Second explanation of our findings can come from expectation–confirmation theories. These theories suggest that what is expected and may lead to no discernible influence except when it is not observed and can lead to low patient satisfaction. However, when it is not expected and observed, it is viewed positively and can lead to improved satisfaction. In other words, satisfaction is higher when the unexpected get disconfirmed, or expected gets confirmed, through a better experience (11,12). Similar perspectives can also be explained using the Kano model that relate the customer satisfaction in an insightful way to understanding and categorizing 5 types of customer requirements (or potential features) for new products and services and is used to explain the asymmetrical or nonlinear relationships between the levels of customer expectations fulfilled and customer satisfaction (13,14). From these perspectives, doctors are expected to treat them effectively with no other expectation from patients. However, patients do expect a comfortable infrastructure where care is provided and quality nursing care and communication. When some of these lead to unexpected behavior, they can influence patient satisfaction.

The study informs health-care providers to be diligent about excellence in all dimensions of patient care. As much as treating patients effectively to address their condition is important, it is not sufficient if patient satisfaction is to become another metric on which a hospital wants to judge itself or by its peers and regulators. Attributes of high-quality care such as infrastructure used to provide care, quality nurse interactions, and communication about care processes influence the cumulative patient experience and, in turn, their evaluation. Although it is possible that patients with short-term treatments (like in an ED) have different assessment of the care attributes such as doctors and schedules, even these dimensions may play a role in patient satisfaction when patients come to a hospital for elective surgeries or not emergency-related treatments.

This study has some limitations. First, this study is cross-sectional and analysis is based on correlations among independent and dependent variables and hence can’t contribute to establishing any type of causality. Second, although it will be interesting to compare the results across hospitals, relatively small sample sizes limit such comparative analyses. Third, we could not really correlate subjective assessments through their comments with any of their objective ratings, as this could have provided robust set of insights based on triangulated design. Also, the study leverages a large administrative database from a third party in a specific setting such as ED patients with no clinical information on how they got there. So, the statistical significance represented in the analysis does not take into consideration these clinical conditions. In spite of these short comings, the findings do provide some systemic ways care is going to be viewed and responded to by patients, and what hospitals may have to do address patient experiences. For example, ignoring the nursing care dimension (attributed often to overburdening of the nurses with low bed–nurse ratios) can lead to financial impact to providers (ie, reimbursements based on patient satisfaction). Similarly, care-based communication especially with regard to postdischarge care of a patient can influence not only patient satisfaction but also unplanned readmissions, another cost to the hospital.

In summary, care quality has to be complemented with other peripheral and support services as patients start to share their experiences not only through their hospital-administered surveys of today but also through social media-based communication going forward.