Six years of measuring patient experiences in Belgium: Limited improvement and lack of association with improvement strategies.

OBJECTIVE: To examine trends in patient experiences in the period 2014-2019, describe improvement strategies implemented by hospitals in the same period, and study associations between patient experiences and implemented strategies.
DESIGN: Multi-center retrospective region-wide observational design.
SETTING: Flanders, Belgium. PARTICIPANTS: 44 out of 46 Flemish acute-care hospitals publicly reporting patient experiences via the Flemish Patient Survey (FPS). MAIN OUTCOME MEASURE(S): Primary outcomes were the two global FPS ratings: percentage of patients rating the hospital 9 or 10 and percentage of patients definitely recommending the hospital. Secondary outcomes were the average top-box score percentages for each of the 8 remaining dimensions of the FPS.
RESULTS: Between 2014 and 2019, there was a significant improvement in patients scoring the hospital 9 or 10 (56% to 61%) and patients definitely recommending (67% to 70%) the hospital. Significant increases in patient experiences over time were also observed in other dimensions, except for the dimension discharge. Hospital key informants reported various improvement strategies related to patient experiences with care and the FPS. Feedback to nursing wards (n = 44, 100%) and clinicians (n = 39, 89%) were most common. Overall, most improvement strategies were not or only weakly associated with patient experience ratings in 2019 and changes in ratings over time. Still, positive associations were discovered between the strategies 'nursing ward interventions' and 'hospital wide education' and recommendation of the hospital.
CONCLUSIONS: Patient experiences have improved modestly in Flemish acute-care hospitals. Hospitals report to have invested in patient experience improvement strategies but positive associations between such strategies and FPS scores are weak, although there is potential in further exploring nursing ward interventions and hospital wide education. Hospitals should continue their efforts to improve the patient's experience, but with a more targeted approach, taking the lessons learned on the efficacy of strategies into consideration.

Introduction

Hospitals are increasingly integrating patient-centeredness within their policy statements. Its importance as one of the dimensions of healthcare quality [1] is becoming more and more recognized. Patient-centered care is associated with improved clinical outcomes and reduced costs [1-4]. Assessing the patient’s perspective of quality has long been described as a valuable quality indicator [5] and the foundation of patient-centeredness. Many health systems have therefore developed survey instruments aimed at measuring patient experiences, like the Hospital Consumer Assessment of Healthcare Providers and Systems (USA) [6] and the NHS Patient Survey (UK) [7] for acute-care hospitals. In Flanders, the northern part of Belgium, a uniform instrument was developed by the Flemish Patient Platform and validated [8] under the heading of the Flemish Patient Survey (FPS). The stakeholder-initiated Flemish Hospital Indicator Initiative (VIP2) aimed to increase insight into the quality of its hospitals by using clinical process and outcome indicators. Amongst other indicators, patient experiences with care, are voluntarily gathered hospital-wide via FPS by nearly all Flemish hospitals. In order to support quality improvement initiatives, feedback is available to all organizations. Communication of individual results on hospital websites is encouraged. In 2015, a central website (http://www.zorgkwaliteit.be) was developed where findings can be consulted by the public in an aggregated manner. The top-box scores of two global patient experience measures, i.e. patients definitely recommending the hospital and patients rating the hospital 9 or 10, are publicly reported once a year since July 2015.

Merely implementing a patient experience survey does not suffice to improve patients’ experiences [9]. Reporting of patients’ perspectives of hospital care can, however, be an incentive to enhance and reinforce quality improvement, although international evidence remains scant and ambiguous [10] and is often based on case studies and expert opinion [11-13]. A recent systematic review [14] looked into initiatives to improve patient satisfaction and observed potential in strategies concerning communication [15], patient [16] and physician education [17] and increasing pharmacists’ involvement [18]. Making use of online platforms like Yelp or Facebook could be linked with improvements in patient experiences [19, 20]. Aboumatar and colleagues [21] studied high-performing US hospitals of patients’ reports of care and found involvement and responsibility at multiple levels of the organization, from leaders to clinicians, to be a common trait. They found that high-performing hospitals used multiple and similar concurrent interventions to improve patient experiences, like nursing ward interventions or hospital-wide feedback. External incentives like accreditation [22-24] or pay for quality in a Value Based Purchasing program [25] were found to have little impact on the patient’s experience.

How patient experiences have evolved in Flanders since the first public release in July 2015 of 2014 scores, is unclear. Additionally, which quality improvement strategies concerning patient experiences have been introduced in Flemish hospitals remains unexplored. The aim of this study was to describe associations between improvement strategies and patient experiences as assessed via the FPS. We therefore first examined trends in patient experiences from 2014 to 2019. Subsequently, we described which strategies Flemish acute-care hospitals have implemented during the same time period. Finally, associations between patient experiences and improvement strategies were explored.

Materials and methods

Study design

A multi-center retrospective region-wide observational study.

Study sample and recruitment

The FPS is handed out to all eligible patients (i.e. all discharged non-psychiatric patients above 18 years of age) during two periods of the year (6 weeks in March-April and 6 weeks in September-October) and with a yearly minimum of 300 filled out surveys per hospital. Over the study period, on average 78% of hospitals distribute their surveys on paper, 11.6% handed out an electronic version of the FPS and 10.4% combined electronic with paper distributions. Key informants from all Flemish acute-care hospitals (n = 55) who have chosen to publicly report (n = 46) patient experience scores on http://www.zorgkwaliteit.be were contacted for participation in this study, encouraged by the hospital umbrella organization Zorgnet-Icuro. Email and telephone reminders were sent by the research team to non-responsive hospitals.

Data collection

To describe trends in FPS results, the Flemish Institute for the Quality of Care was contacted as the official organization overseeing the development and measurement of quality indicators. Patient-mix adjusted quality indicators, aggregated at hospital-level, were provided from the earliest collections in 2014 to the first semester of 2019 within the ‘patient experiences’ domain of the Flemish Indicator Initiative. This encompasses the percentages of top-box scores on 28 questions concerning nine dimensions of patient experience: hospital stay preparation, information about condition, information about treatment and procedures, dealing with patients and collaboration between healthcare providers, privacy, safe care, pain management, discharge and global experience. The two global patient experience measures, i.e. patients grading the hospital and patients recommending the hospital, are the sole indicators publicly reported online at the time of the study. Patient-mix adjustments include patient age, sex, housing type, health status and level of education.

To outline currently implemented quality improvement strategies, an online survey with personal code was sent out in summer 2019 via Qualtrics© to all quality managers within the study sample. The survey was developed within the research team and contained 16 binary (yes/no) questions about hospital participation in strategies. The inquired strategies were based on international literature of frequently implemented initiatives aimed at improving patient experiences.

Statistical analysis

We first described our sample characteristics. Main outcomes were the two global patient experience measures: the percentage of patients rating the hospital 9 or 10 and the percentage of patients definitely recommending the hospital. Secondary outcomes were the average top-box score percentages for each of the 8 remaining dimensions of the FPS. To describe the trend in patient experiences, our first research objective, we plotted the two global top-box measures from 2014 to 2019 for each participating hospital. Linear changes in top-box percentages over time were modelled using a separate multilevel model for each outcome, accounting for repeated measures through a random intercept for hospital. In a second set of models, year was treated as a categorical variable to allow for non-linear trends. For our second objective concerning implemented strategies, we present the findings from the survey on quality improvement initiatives visually by percentage of participating hospitals and by percentage of implemented strategies. For our final research objective, we studied the effect of improvement strategies as potential predictors of superior patient experience scores on the FPS. Using separate models for each outcome, we tested differences in percentage top-box scores measured in 2019 between hospitals with and without a specific strategy (linear regression), as well as differences in linear trends, i.e. the evolution of percentage top-box scores from 2014 to 2019 (multilevel linear regression). Differences in time trends between hospitals with and without a strategy were assessed using an interaction term between a binary indicator for strategy implementation and a linear variable for year. The strategy “FPS feedback to nursing wards” was not tested as this was implemented by all 44 hospitals. Statistical significance of the regression analyses was determined at an alpha level of 0.05. The critical threshold for the regression analyses concerning associations with implemented strategies was determined at p<0.0033, which is derived from a Bonferroni correction [26] to control for multiple testing, i.e. alpha level of 0.05 divided by 15, the number of strategies tested. The analyses for this paper were generated using SAS© software, Version 9.4 of the SAS System for Windows.

Ethical considerations

The study protocol was approved as part of a larger retrospective observational study concerning the impact of improvement initiatives on patient outcomes by the Ethics Committee of University Hospitals Leuven (S63449).

Results

Sample

Our final sample included 44 (response rate: 96%) acute-care hospitals who agreed to participate. Four included hospitals were university hospitals (9%) and the number of beds ranged from 170 to 1764. Seven (16%) hospitals did not start FPS measurements until 2015. Four hospitals (9%) did not measure patient experiences for one or two study years due to reasons like hospital mergers, external accreditation or moving to another building. The total number of participants filling out their patient experience increased each year from on average 613 per hospital (SD: 360.7) in 2014 to a mean of 741 (SD: 440.4) in 2018. For all participating hospitals, this totals to a sample set of 23 549 patients in 2014 and 32 464 in 2018. For the first semester of 2019, already 16 193 patients (on average 378 per hospital) filled out the FPS, which is in accordance with expectations.

Trend in patient experiences

The overall and hospital-specific trends in global patient experiences are plotted in Fig 1. Overall, the percentage of patients rating the hospital 9 or 10 has steadily increased from 56% in 2014 to 61% in 2019, while the percentage definitely recommending the hospital ranged from 67% in 2014 to 70% in 2019. Some hospitals (e.g. AI, AJ, and AQ) appear to follow an upward trend, while patient experiences seem to deteriorate in e.g. AH, BE and BJ. For each hospital, both global questions appear to follow similar trends, although exceptions exist (e.g. AO, AY, BA).

Fig 1

Hospital trends in patient experience scores for the two global questions.

Each figure represents the percentage top-box scores in one of 44 participating Flemish acute-care hospitals. The upper left figure represents results aggregated for all participating hospitals.

S1 Table displays the yearly top-box percentages and the results of the multilevel regression models across time for the two global FPS questions and the averages for the 8 remaining FPS dimensions. Large variation in average percentage top-box scores exists between the 8 dimensions, ranging from 51% to 89% in 2014 and from 53% to 88% in 2019. Assuming linearity, a significant improvement in patient experiences was observed for the two global questions and for all dimension averages except for the dimension discharge. The estimated yearly increases in the percentage of patients rating the hospital 9 or 10 and the percentage of patients definitely recommending the hospital were 1.10 (95% CI: 0.80; 1.40) and 0.39 (95% CI: 0.15; 0.63) respectively. Results from regression models treating year as a categorical variable indicate that improvements are primarily observed in recent measurement periods: compared with 2014, a significant increase in top-box percentages was observed for 2 out of 10 outcomes in 2017, and for 8 out of 10 outcomes in 2019. The largest improvement in patients’ experience was observed for the dimension safe care, with 52% of patients answering the top-box score in 2014, improving to 64% in 2019 (β = 11.69, 95% CI: 10.03; 13.34). Worsening of patient experiences could be observed in the dimension discharge. However, deteriorations are small and scores remain high (average percentage top-box scores 89% in 2014 and 88% in 2019, β = -0.63, 95% CI: -1.19; -0.08).

Implemented strategies to improve patient experiences

An overview of the surveyed strategies with a description of each strategy is provided in Table 1, which includes examples of strategies employed by participating hospitals. Analysis of the binary survey questions on improvement strategies resulted in the heatmap displayed in Fig 2. FPS feedback to nursing wards is a strategy implemented by all hospitals (100%, n = 44), while direct feedback to clinicians (89%, n = 39) is second most common. In a shared third and fourth place come nursing ward interventions (86%, n = 38) and hospital wide interventions (86%, n = 38). Conversely, hiring external consultants to improve patient experiences is the least explored strategy (7%, n = 3). Discharging the patient with a multidisciplinary team (25%, n = 11) and both rewarding the best FPS performing nursing ward (27%, n = 12) and social media follow-up (27%, n = 12) are relatively infrequent as well. A large variation between the number of strategies a hospital implements can be observed, ranging from 4 to 14 out of 16 surveyed initiatives. The number of strategies is independent of hospital size or teaching status. Among the 5 hospitals employing the most strategies for example, both academic (n = 2) and general (n = 3) hospitals are represented, which are located in 4 of the 5 Flemish provinces and with the number of beds ranging between 271 and 1049.

Table 1

Surveyed strategies and their description.

Surveyed strategy	Description
FPS feedback to nursing wards	Flemish Patient Survey feedback is received by nursing wards on a regular basis. Feedback can occur on internal data collection as well as on the external benchmark reports released twice a year.
FPS feedback to clinicians	Flemish Patient Survey feedback is received by clinicians on a regular basis. Feedback can occur on internal data collection as well as on the external benchmark reports released twice a year.
Nursing ward interventions	Interventions at the level of the nursing ward are implemented to improve patient experiences. Examples include the introduction of a Magic Table© on geriatrics, interventions on pain management, organizing mealtimes between staff and patients where patients can express their concerns, or the introduction of Patient Reported Outcome Measures (PROMs) on specific wards.
Hospital wide interventions	Hospital wide interventions are launched to improve patient experiences. Examples are the implementation of an incident reporting system designed for patients or the organization of consultation hours between hospital staff and management and patients. Additionally, interventions could comprise hospital-wide campaigns aimed at improving the patient’s experience. Examples include participation in the internationally renowned ‘What Matters to You’ campaign, based on Barry and Edgman-Levitan’s perspective [27] or campaigns concerning Mangomoments based on research by Vanhaecht et al. [28].
Board sets strategy	The hospital board sets the strategy to improve patient experiences. The strategy can e.g. be documented in a charter which is then distributed to all staff.
FPS targets	Specific targets concerning Flemish Patient Survey are premised. A hospital can e.g. choose to aim for more than the required 300 yearly surveys, or can aim for a specific percentage gain in one or more patient experience dimensions.
Hospital wide education	Hospital wide education, like workshops or seminars, to improve patient experiences are organized. For example, hospitals could develop a hospital academy, wherein both online and offline courses are organized for both care professionals and patients. Topics for professionals could include ways of introducing yourself to the patient and techniques on informing patients about their treatment.
Discharge info on admission	Discharge information is provided at the time of a patient’s admission.
Nursing rounds	Nursing rounds specifically aiming to improve patient experiences are organized.
HR Policy	Improving patient experiences is an area of concern for human resources management. How an individual care provider scores on his/her patient’s experience, can be a topic of a performance appraisal.
Proactive discharge calls	A selection of patients is called proactively after discharge.
Bedside briefing	Briefing of care providers at shift transfer takes place at the patient’s bedside.
Social media follow-up	Reviews by patients on online platforms like Facebook, Twitter, Google Reviews, etc. (social media) are systematically followed up on.
FPS nursing ward rewards	Nursing wards receive a reward when scoring excellently on Flemish Patient Survey. The reward can be of a financial nature, but can also e.g. entail a teambuilding outing.
Multidisciplinary discharge	A multidisciplinary team of care providers is present at patient’s discharge.
External consultants	A consultancy firm is hired to improve patient experience scores.

Fig 2

Implemented quality improvement strategies to improve patient experiences across hospitals.

Each cell represents a quality improvement strategy in one particular participating hospital (n = 44). A green cell represents the strategy being implemented, whereas a red cell represents an unimplemented strategy.

Associations between patient experiences and improvement strategies

Associations between the strategies reported by the participating hospitals and the two global patient experience questions for the first semester of 2019 are displayed in Table 2. None of the strategies were associated with rating of the hospital, whereas top-box scores for recommendation of the hospital were significantly higher for hospitals having implemented nursing ward interventions and hospital wide education. For both strategies, the difference in percentage definitely recommending the hospital between hospitals with and without the strategy was around 6.6%, but these associations were not significant after Bonferroni correction. At an alpha level of 0.05, significant positive associations were observed for 6 strategy-dimension combinations (S2 Table), including 3 dimensions for the strategy nursing ward interventions and 2 dimensions for the strategy hospital wide intervention. The dimension discharge, however, was negatively associated with the strategies FPS feedback to clinicians and external consultants. The latter was also negatively associated with the dimension preparing for hospital stay. However, after Bonferroni correction, none of these associations remained significant.

Table 2

Associations between quality improvement strategies and top-box scores for global patient experience questions in 2019.

Surveyed quality improvement strategy	Percentage rating the hospital 9 or 10		Percentage definitely recommending the hospital
	β(1)	(95% CI)	β(1)	(95% CI)
FPS feedback to clinicians	-0.64	(-6.61; 5.32)	-2.66	(-9.89; 4.58)
Nursing ward interventions	4.69	(-0.64; 10.01)	6.64	(0.23; 13.05)*
Hospital wide interventions	3.30	(-2.13; 8.72)	5.00	(-1.56; 11.56)
Board sets strategy	-1.06	(-5.98; 3.86)	-0.81	(-6.83; 5.21)
FPS targets	-0.14	(-4.45; 4.16)	1.92	(-3.31; 7.14)
Hospital wide education	2.61	(-1.34; 6.55)	6.69	(2.26; 11.13)**
Discharge info on admission	1.03	(-2.98; 5.05)	3.63	(-1.15; 8.41)
Nursing rounds	2.24	(-1.65; 6.13)	2.45	(-2.31; 7.21)
HR policy	0.08	(-3.87; 4.03)	1.74	(-3.05; 6.53)
Proactive discharge calls	1.60	(-2.36; 5.56)	4.68	(-0.11; 9.48)
Bedside briefing	-0.26	(-4.29; 3.77)	1.74	(-3.15; 6.63)
Social media follow-up	-0.54	(-5.09; 4.02)	0.09	(-5.48; 5.66)
FPS nursing ward rewards	0.39	(-4.03; 4.81)	3.47	(-1.81; 8.76)
Multidisciplinary discharge	0.12	(-4.82; 5.05)	-1.52	(-7.52; 4.49)
External consultants	-6.48	(-13.68; 0.72)	0.21	(-8.94; 9.36)

(1) The difference (with 95% confidence interval) in percentage top-box scores between hospitals with and without the improvement strategy.

* Statistically significant at an alpha level of 0.05.

** Statistically significant at an alpha level of 0.01.

None of the estimates were significant after Bonferroni correction.

Associations between strategies and trends in top-box score percentages over time are presented in Fig 3 (two global questions) and S1 Fig (8 remaining dimensions). Significant differences in time trend slopes were observed for the strategy nursing ward interventions: top-box scores for both global questions increased over time in hospitals with nursing ward interventions, whereas patient experiences remained constant (rating the hospital) or deteriorated (recommending the hospital) in hospitals without nursing ward interventions. For recommendation of the hospital, significant differences in time trends were also observed for the strategies board sets strategy, social media follow-up, and multidisciplinary discharge, with hospitals that implemented these strategies showing more positive slopes than hospitals without the strategy. Hospital rating, however, increased more steeply in hospitals without than in hospitals with bedside briefing, but the latter started with higher scores and both ended with similar scores in 2019. Only the association between nursing ward interventions and recommendation of the hospital remained significant after Bonferroni correction. Bonferroni-corrected significant differences in time trends between hospitals with and without nursing ward interventions were also observed in the dimension dealing with patients and collaboration between healthcare providers, with patient experience scores increasing over time in hospitals with nursing ward interventions, but decreasing in hospitals without nursing ward interventions. Patient experience scores in the dimension safe care increased more steeply over time in hospitals with board setting strategy than in hospitals without this strategy (significant after Bonferroni correction).

Fig 3

Associations between quality improvement strategies and time trends in top-box scores for global patient experience questions (upper panel: Rating the hospital; bottom panel: Recommending the hospital).

The plotted time trends are the predictions from multilevel regression models containing a binary indicator for strategy implementation, a linear variable for year, and an interaction between these variables. The p-value represents the significance of the interaction term and indicates whether time trends are significantly different between hospitals with and without a given strategy.

Discussion

Although individual results of global FPS questions are already publicly reported from 2014 onwards, this paper provides the first overview of the evolution of FPS results in Flanders across time. The overall improvement, strongest in most recent years, is commendable, yet small. The most recent top-box score of 61% of patients rating the hospital 9 or 10 e.g. is still 11 percentage points lower compared to the average of 73% in the US [29]. The percentage of patients recommending the hospital in 2019 in Flanders (70%) is still 4 percentage points removed from the current US average of 74% [29]. While one cannot unambiguously compare patient experiences across cultures and health care systems [30], the evidence seems to suggest that Flemish hospitals should keep striving for better achievements. Moreover, our study brought to light a large variability in patient experience scores across both individual hospitals and FPS dimensions. Reducing this variation has long been known as a valuable tool to improve quality of care [31]. While patient experience scores improved in 8 out of 9 dimensions, especially when concerning the safety of care, further opportunities lie in optimizing the discharge process, which seems to have stagnated over time, as well as focusing on the provision of information about both condition and treatment. The latter remain low-scoring dimensions that have shown little improvement over time. From December 2019 onwards, the website https://www.zorgkwaliteit.be has started to also publicly report specific FPS scores of all domains next to the global measures. What the impact of this public reporting on specific FPS scores will be, needs to be studied further.

As demonstrated by our survey concerning improvement strategies, Flemish hospitals have been investing modestly in improving patient experiences. While considerable variation in strategy implementation can be observed between hospitals, it is worth noting that each hospital has implemented more than one strategy. Many strategies described by Aboumatar and colleagues [21] as implemented in top-scoring US hospitals, like nursing ward interventions and hospital wide education, are also frequently implemented in Flemish hospitals. What’s more, both nursing ward interventions and hospital wide education were found to be associated with better 2019 FPS results. Additionally, nursing ward interventions in particular were positively associated with improved global patient experiences over time. Flemish hospitals who did not employ nursing ward interventions scored on average 7 percentage points lower on recommendation of the hospital and even decreased across time.

To our knowledge, this is the first assessment of associations between quality improvement strategies and patient experience scores. Despite the positive associations between both nursing ward interventions and hospital wide education and 2019 FPS results and the positive relationship between nursing ward interventions and recommendation of the hospital, improvement strategies were overall not or only weakly associated with patient experience ratings. After Bonferroni correction, only the association between nursing ward interventions and improvements in recommendation remained. Additionally, the relationship with 8 specific patient dimensions is non-existent, apart from a coherent positive influence of nursing ward interventions and strategies by the board on the change in dealing with patients and provision of safe care respectively. A thorough revision of the hospitals’ current approach on improving patients’ experiences is therefore recommended. Considering its potential, further research into the benefits of nursing ward interventions or a hospital-wide educational program is advised. By researching the evidence-base on the interventions that have shown most promise, we hope future healthcare policy and practice might be altered towards a more unified care, instead of the wide spectrum of sometimes ineffective interventions currently implemented. The examples provided by some participating hospitals such as e.g. mealtimes between staff and patients or the development of hospital-wide courses, suggest a large variety of ways to execute strategies. We thus encourage hospitals to share and learn from both their positive and negative experiences. By focusing on both nursing ward interventions and hospital wide education, a high visibility for the patient as well as a widespread reach of all healthcare staff can be ensured.

Next to the surveyed internal strategies, the external pay-for-performance (P4P) initiative appears to have limited impact on patient experiences at first glance. Implemented In 2018, the federal P4P initiative [32] comprised an adjusted reimbursement based on high-value quality metrics like patient experiences. No strong overall improvement could be observed between FPS results in 2018 and 2019. Today, P4P solely depends on participation in the FPS and is thus not related to hospital results. Only a small portion of hospital payment is currently at stake, i.e. about 5 million on a total budget of 6.4 billion euros for acute-care hospitals. What the impact of larger payments within the P4P scheme, tied to actual FPS results, will be, needs to be studied further. Impact of external evaluations in the form of international accreditation and governmental inspection will be studied in the near future as part of a larger retrospective study of quality improvement initiatives in Flanders.

A number of considerations that merit further attention and highlight a number of limitations to this study needs to be outlined. Firstly, our study might have suffered from recall bias. Secondly, associations between strategies and FPS results need to be interpreted prudently due to multiple testing. However, using a Bonferroni correction controls for this multiplicity issue. Thirdly, we lacked specific information on the quality improvement strategies employed by participating hospitals, like implementation date and detail on how and on what wards the hospitals chose to implement their strategies. Informal conversations with participants showed this information was not always well recorded at the management level. Often due to high staff turn-over on quality departments, more detail was unavailable for a majority of participating hospitals. Fourthly, no confounding factors like e.g. employment of experience experts or other initiatives were accounted for in this study. The survey sent to every participating hospital left room to fill out additional information in an open-ended question concerning other initiatives taken. Unfortunately, only 50% of participants filled out this question, making it unusable for regression analysis. Lastly, due to the retrospective nature of this research, no causality can be established. Still, with the large representative sample of acute-care Flemish hospitals, we managed to obtain a first overview of current quality improvement strategies and how they have affected patient experience scores.

Conclusion

This study demonstrated how patient experiences across Flemish acute-care hospitals have marginally improved and how hospitals have invested modestly in quality improvement strategies concerning patient experiences. A large variability across hospitals persists, obstructing overall improvement. Beside nursing ward interventions and hospital wide education, which was demonstrated to have potential in further improving patient experiences, no associations between employed strategies and global patient experience scores could be identified. Within the Flemish hospital landscape, the patient’s experience remains an area where progress is required. Future healthcare policy will hopefully take the conclusions from this research into account and thus lead the way towards better patient care.

Trends in patient experience scores across Flemish acute-care hospitals (n = 44).

(DOCX)

Click here for additional data file.

Associations between quality improvement strategies and average top-box scores of the 8 patient experience dimensions in 2019.

(DOCX)

Click here for additional data file.

Associations between quality improvement strategies and time trends in average top-box scores of the 8 patient experience dimensions.

(DOCX)

Click here for additional data file.

11 Jun 2020

PONE-D-20-04886

Six years of measuring patient experiences in Belgium: limited improvement and lack of association with improvement strategies.

PLOS ONE

Dear Dr. Van Wilder,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Reviewers have provided detailed feedback on your manuscript and the analysis of your data. Please revise as per recommendations of reviewers. In addition, I would like to emphasize the limitation of the lack of relationship between the QI strategies and the patient experience surveys. You have mentioned the same, but this needs to be made more clear. Furthermore, I would suggest that you be more specific in your recommendations for future research, policy and practice, particularly when you talk about nursing actions.

Please submit your revised manuscript by August 10. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Nelly Oelke

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The results reported on the paper are based on a national survey of Flemish hospitals, confined to hospitals which had chosen to publicly report their data (46 hospitals, not clear what proportion of the total number of hospitals this was). Data from 44 hospitals were included in the final analysis. There is no detail on how the surveys were administered in this national effort, but a minimum of 300 surveys had to be returned by patients who had been discharged from in-patient stays. Depending on the year, between 15,000 and 32,000 patient responses were available for the authors to include in their analysis.

The authors contacted quality managers in these 46 hospitals to ask what strategies had been implemented to improve the quality of patient experience. The statistical analysis is well described and appears unremarkable. However, I thought it odd that the authors looked for associations between quality improvement strategies and survey scores in 2019 – would it not have been more logical to look for an association between QI strategies and improvement in scores (i.e. from 2014 to 2019)?

Generally speaking, there was little change in patient experience during the period studied. There was considerable variation between hospitals; a few things got a bit better and a few things got worse. The overall trend was of slight improvement.

Likewise there was little relationship between scores in 2019 and QI strategies. In two domains, hiring external consultants was associated with worse scores in 2019. This rather reinforces my view that the authors have done the wrong analysis here. Surely its likely that the hospitals with the worst scores will have been most likely to hire external consultants. Indeed, the authors comment that one of the three hospitals using external consultants was a ‘strong negative outlier’. I’d have been more interested in looking at QI strategies (and their timing) in relation to changes in scores. At the very least the authors should point out this limitation – it would be even better if they redid the analysis.

The lack of improvement is well described in the paper, but hardly surprising. Unfortunately there is little information about what the hospitals actually did, i.e. the intensity of the interventions, as the data on the interventions was based solely on the answers to 16 binary questions about what strategies the hospitals employed (e.g. “Did you feed back the results to clinicians – Yes/No”). The commonest improvement strategy reported was feedback to nursing ward and clinicians and feedback on its own is well known to be a relatively ineffective strategy for quality improvement. As the authors point out, high-performing hospitals use “multiple and similar concurrent interventions to improve patient experiences”.

Reviewer #2: The authors provide results of a study of patient experiences at hospitals in Belgium from 2014-2019. They also assess associations between different strategies employed by the hospital to improve care and patient experiences. Authors find a small increase in patient satisfaction over the study period. The manuscript will be strengthened if the authors consider the following points:

1. In lines 132-136, the authors should clarify if the survey about strategies specified a time period for the implementation of those strategies as this is important in understanding the strategy analyses.

2. lines 143-144: authors should state how the repeated measures within hospitals were accounted for. (In the results, they specify the use of multilevel linear regression, but this should be stated here in the methods for the 1st objective).

3. Authors state they use multilevel linear regression when assessing the association between strategy use and patient experience outcomes, but on line 229, they say they are only analyzing results for the 1st semester of 2019. If it is only the patient experience results from the 1st half of 2019, it is not clear why multilevel linear regression is needed.

4. Did the authors check the underlying assumptions of the models? Particularly when they start analyzing individual questions, I imagine there could be some highly skewed distributions which could pose some issues with the assumptions.

5. To give a better sense of the data, the authors should provide the median and the average number of participants per hospital (and standard deviation) that filled out patient experience across the years (here I don't mean hospital level-data, but rather across the 44 hospitals, what was the mean, median, and sd in number of participants in 2014? in 2015? etc.) This will help understand the percentage level hospital data that are being analyzed.

6. In many of the analyses, the authors use year as a categorical variable. What is the justification for this? If they want to understand a general trend, would it make more sense to use time as a continuous variable (time since 2014, for example)?

7. In Tables 1, 2, Supplemental Table 1, and the text, authors should include 95% confidence intervals for the beta estimates.

8. lines 219-220: the authors state that the number of strategies is independent of hospital size or teaching status, but no data are provided to support this statement.

9. In the analyses of the association between patient experience and improvement strategies, I wonder if more rationale can be provided, especially for the analysis of the individuals questions (rather than the global questions), since it is not always clear to me why certain improvement strategies might be associated with certain questions.

10. Also in the experience/improvement strategies analysis, authors use a Bonferroni correction that accounts for the 16 strategies investigated, but not necessarily across all outcomes. I might be misunderstanding what is done here (see point 3 above), but there are many more than just 16 comparisons being made.

11. In the Discussion (line 286), the authors mention a strong negative outlier in the external consultants analysis - what happens to the findings if this outlier is removed from the analysis (to get a sense of the influence of this one observation)?

Minor points:

1. line 203: in the Figure 1 caption, I believe "on of" should be "one of"

2. lines 215-217: the given percentages do not match Figure 2 (nor calculations out of 44 hospitals), so these should be corrected.

3. lines 276-277: it is not clear which FPS outcome this line refers to. Also, no analyses are presented to demonstrate that use (or not) of a strategy was associated with improvement over time.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Martin Roland

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

9 Aug 2020

Academic Editor’s Comments

1. I would like to emphasize the limitation of the lack of relationship between the QI strategies and the patient experience surveys. You have mentioned the same, but this needs to be made more clear.

Per request of the reviewers (see below), we conducted new statistical analyses on the associations between Flemish Patient Survey (FPS) top-box score percentages and surveyed quality improvement (QI) strategies. Now, we not only look at associations for the first semester of 2019, but we also study the changes in ratings over time. Additionally, for our secondary outcomes, we no longer study each individual FPS question, but we analyze the average top-box score percentages for each of the 8 remaining dimensions of the FPS.

The new analyses could not find any (Bonferroni-corrected) associations between QI strategies and global patient experience ratings (primary outcome) in 2019, nor with changes in global ratings over time. Only the implementation of nursing ward interventions was significantly associated with an increase in the percentage of patients recommending the hospital. Our initial message narrating a lack of relationship between QI strategies and patient experience surveys is therefore now further strengthened.

We emphasize this in the manuscript in both discussion (lines 317-322) and conclusion (lines 355-357).

2. Furthermore, I would suggest that you be more specific in your recommendations for future research, policy and practice, particularly when you talk about nursing actions.

We added a more specific research recommendation concerning nursing ward interventions in the discussion (lines 322-326). By researching which nursing ward interventions in particular are most promising and collecting a more in-depth evidence-base, we hope that future healthcare policy and practice might be altered to provide a more unified care that can further enhance the patient’s experience.

3. Please ensure that your manuscript meets PLOS ONE’s style requirements, including those for file naming.

Throughout the manuscript we made formatting changes to ensure we meet PLOS ONE’s style requirements. We added e.g. indentations at the beginning of each paragraph, used correct symbols on the author page, altered some references to comply with Vancouver guidelines and named tables and figures appropriately, both within the manuscript as in the file names.

Reviewers' comments:

Reviewer's Responses to Questions

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Party

After reading the reviewers’ comments, we came to the realization that our manuscript would benefit from additional analyses concerning the association with change in patient experience scores between 2014 and 2019. Additionally, as mentioned by reviewer #2, our initial analyses investigating associations between individual patient survey questions and employed strategies included some irrelevant strategy-question combinations for which no association would be expected. In the new analyses, we grouped the individual questions into 9 dimensions of patient care and we analyzed dimension-specific average scores, thereby reducing the number of statistical tests and overcoming the problem of irrelevant strategy-question combinations, resulting in more sound conclusions.

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

We agree with the remarks given by Reviewer #2 concerning our original statistical analysis. Shortcomings have been solved (see section on statistical analysis, lines 142-166) and will be further elucidated down below. We also noticed a small error in our original manuscript, which stated that a Bonferroni correction was applied on the 16 tested QI strategies. However, 1 strategy was employed by all hospitals, meaning it was never included in the testing of strategies (see lines 159-160). We therefore corrected our mistake in lines 163-164.

3. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #1: Yes

Reviewer #2: Yes

Like we did in our original manuscript, we have made all data available in either the revised document or in supplemental files.

4. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #1: Yes

Reviewer #2: Yes

The revised manuscript was proofread in detail by both AVW, BC and LB to ensure a correct usage of standard English.

Reviewers’ Comments

REVIEWER #1

1. The results reported on the paper are based on a national survey of Flemish hospitals, confined to hospitals which had chosen to publicly report their data (46 hospitals, not clear what proportion of the total number of hospitals this was). Data from 44 hospitals were included in the final analysis.

The region of Flanders has 55 acute-care hospitals, of which 46 hospitals publicly reported their patient experiences at the time of the study. The section ‘study sample and recruitment’ was amended in line 117 to make clear what proportion of the total number of hospitals (83.6%) were reached out to for the purpose of this study.

2. There is no detail on how the surveys were administered in this national effort, but a minimum of 300 surveys had to be returned by patients who had been discharged from in-patient stays. Depending on the year, between 15,000 and 32,000 patient responses were available for the authors to include in their analysis.

We provided further detail on how the surveys were administered in lines 114-116 of the section ‘study sample and recruitment’. As displayed in the table below, the majority of hospitals publicly reporting their patient experiences distributed the Flemish Patient Survey to their patients on paper. As the percentages of distributions fluctuate across time (no clear increase or decrease could be observed), we opted to report on the average distribution percentage across the study years.

Year Distribution on paper Electronic distribution Mixed distribution

2014 78% 9,8% 12,2%

2015 79,2% 8,3% 12,5%

2016 83,3% 12,5% 4,2%

2017 76,5% 17,6% 5,9%

2018 80% 10% 10%

2019 71,2% 11,5% 17,3%

Additionally, we provided more information on the study sample, as was also requested by Reviewer #2 (see below for more detail). The additional information on number of patient responses per hospital, is added in the ‘Sample’ section of the results in lines 178-182.

3. The authors contacted quality managers in these 46 hospitals to ask what strategies had been implemented to improve the quality of patient experience. The statistical analysis is well described and appears unremarkable. However, I thought it odd that the authors looked for associations between quality improvement strategies and survey scores in 2019 – would it not have been more logical to look for an association between QI strategies and improvement in scores (i.e. from 2014 to 2019)?

Generally speaking, there was little change in patient experience during the period studied. There was considerable variation between hospitals; a few things got a bit better and a few things got worse. The overall trend was of slight improvement.

Likewise there was little relationship between scores in 2019 and QI strategies. In two domains, hiring external consultants was associated with worse scores in 2019. This rather reinforces my view that the authors have done the wrong analysis here. Surely, it’s likely that the hospitals with the worst scores will have been most likely to hire external consultants. Indeed, the authors comment that one of the three hospitals using external consultants was a ‘strong negative outlier’. I’d have been more interested in looking at QI strategies (and their timing) in relation to changes in scores. At the very least the authors should point out this limitation – it would be even better if they redid the analysis.

The reason why we originally only looked at association with scores in 2019 is because of the limitation that we do not know the date of implementation of the strategies (so looking at changes in scores from 2014 onwards might not be meaningful for strategies implemented only at the end of the study period). Despite this shortcoming, we agree that investigating changes in scores over time might provide additional insight. Therefore, we additionally investigated differences in time trends between hospitals with and without a strategy, by using multilevel models including an interaction term between a binary indicator for strategy implementation and a linear variable for year.

This analysis showed that the percentage of patients recommending the hospital increased over time in hospitals with nursing ward interventions, but deteriorated in hospitals without nursing ward interventions. A similar (Bonferroni-corrected significant) pattern was observed for the dimension dealing with patients and collaboration between healthcare providers. Finally, patient experience scores in the dimension safe care increased more steeply over time in hospitals with board setting strategy than in hospitals without.

4. The lack of improvement is well described in the paper, but hardly surprising. Unfortunately, there is little information about what the hospitals actually did, i.e. the intensity of the interventions, as the data on the interventions was based solely on the answers to 16 binary questions about what strategies the hospitals employed (e.g. “Did you feed back the results to clinicians – Yes/No”). The commonest improvement strategy reported was feedback to nursing ward and clinicians and feedback on its own is well known to be a relatively ineffective strategy for quality improvement. As the authors point out, high-performing hospitals use “multiple and similar concurrent interventions to improve patient experiences”.

As we outline in the limitation section of our revised manuscript (lines 340-344), we lacked specific information on the QI strategies employed by the participating hospitals. We e.g. had no information on the implementation date of the surveyed strategies and we lacked detail on how and on what wards the hospitals chose to implement their strategies. Informal talks with a few of the respondents of the participating hospitals, taught us detailed information on quality improvement initiatives was not always well recorded at the level of quality management departments of the hospitals. Moreover, quality management is characterized by a high turn-over of staff, leading to information being unavailable for a majority of participants.

However, about half of the participants provided us with more detail on some of the surveyed strategies as well as on other employed initiatives within their hospital in the open-ended questions of the survey. While we could not use them in the regression analyses, some examples provided were used in the newly-added Table 1 (line 229), which gives a description of each surveyed strategy. In adding this, we hope to have given you some idea on what the hospitals actually did in terms of interventions.

REVIEWER #2

1. In lines 132-136, the authors should clarify if the survey about strategies specified a time period for the implementation of those strategies as this is important in understanding the strategy analyses.

Like stated above, we outline in the limitation section of our revised manuscript (lines 340-344), we lacked specific information on the QI strategies employed by the participating hospitals. We e.g. had no information on the implementation date of the surveyed strategies and we lacked detail on how and on what wards the hospitals chose to implement their strategies. Informal talks with a few of the respondents of the participating hospitals, taught us detailed information on quality improvement initiatives was not always well recorded at the level of quality management departments of the hospitals. Moreover, quality management is characterized by a high turn-over of staff, leading to information being unavailable for a majority of participants.

2. Lines 143-144: authors should state how the repeated measures within hospitals were accounted for. (In the results, they specify the use of multilevel linear regression, but this should be stated here in the methods for the 1st objective).

We amended the statistical analysis section in lines 147-149, where we further state how repeated measures within hospitals were accounted for. Herein, we state: Linear changes in top-box percentages over time were modelled using a separate multilevel model for each outcome, accounting for repeated measures through a random intercept for hospital.

3. Authors state they use multilevel linear regression when assessing the association between strategy use and patient experience outcomes, but on line 229, they say they are only analyzing results for the 1st semester of 2019. If it is only the patient experience results from the 1st half of 2019, it is not clear why multilevel linear regression is needed.

We are grateful for this comment, as it pointed out an error in the original manuscript. Indeed, the analysis of results from the 1st half of 2019 did not include repeated measurement, so normal instead of multilevel linear regression was used.

4. Did the authors check the underlying assumptions of the models? Particularly when they start analyzing individual questions, I imagine there could be some highly skewed distributions which could pose some issues with the assumptions.

Firstly, we no longer analyze individual questions in the revised version of the manuscript, as -per your suggestion- below the link between strategies and individual questions is not always clear. Instead, we now consider the average top-box score percentages within 8 Flemish Patient Survey dimensions as secondary outcomes. Top-box percentages for the two global questions as well as dimension-specific averages were relatively normally distributed. Model diagnostics were checked are were quite acceptable.

5. To give a better sense of the data, the authors should provide the median and the average number of participants per hospital (and standard deviation) that filled out patient experience across the years (here I don't mean hospital level-data, but rather across the 44 hospitals, what was the mean, median, and sd in number of participants in 2014? in 2015? etc.) This will help understand the percentage level hospital data that are being analyzed.

We provided the average and median number of participants that filled out patient experiences across the 44 participating hospitals in lines 178 to 182 of the revised document. We described how the average has evolved between 2014 to 2018 and 2019. For full disclosure, you can find the descriptives for each study year in the table below.

Year Median Mean Std Dev

2014 520,0 613,2 360,7

2015 596,5 626,8 308,6

2016 566,5 649,7 350,6

2017 650,0 737,3 398,8

2018 648,0 741,4 440,4

2019 384,5 379,7 195,0

6. In many of the analyses, the authors use year as a categorical variable. What is the justification for this? If they want to understand a general trend, would it make more sense to use time as a continuous variable (time since 2014, for example)?

When looking at time trends in FPS scores, we originally treated year as a categorical variable to be able to pick up potential deviations from linearity. We agree, however, that for the purpose of drawing conclusions on general trends, treating year as a continuous variable makes more sense, especially now we have added the slope analyses to assess associations with strategies, which also assumes trends to be linear. Therefore, we added the linear estimate for year as the main result to Table 1 (Supplementary table 1 in the revised manuscript), but we kept the estimates for year as categorical variable (along with the crude percentages by year), as these demonstrate that improvements in FPS scores are most pronounced during the last few years of the study period.

7. In Tables 1, 2, Supplemental Table 1, and the text, authors should include 95% confidence intervals for the beta estimates.

The newly revised manuscript contains beta estimates in Table 2, Supplemental Table 1 and Supplemental Table 2. We always provide 95% confidence intervals per reported estimate.

8. Lines 219-220: the authors state that the number of strategies is independent of hospital size or teaching status, but no data are provided to support this statement.

We have provided additional information to support our statement concerning number of employed strategies in the revised manuscript (lines 223-226). Herein, we took a subsample of the 5 hospitals with the highest number of QI strategies implemented and saw their hospital characteristics varied largely.

9. In the analyses of the association between patient experience and improvement strategies, I wonder if more rationale can be provided, especially for the analysis of the individuals questions (rather than the global questions), since it is not always clear to me why certain improvement strategies might be associated with certain questions.

We fully agree with your comment concerning the unclear relationship between strategies and individual patient survey questions. We therefore no longer analyze individual questions in the revised manuscript. Instead, we now consider the average top-box score percentages of 8 Flemish Patient Survey dimensions as our secondary outcomes. As discussed in lines 320 to 322 of the revised manuscript, our new analyses (looking at changes in time trends) found significant associations between nursing ward interventions and the dimension ‘dealing with patients and collaboration between healthcare providers’ as well as between the board setting strategy and the dimension ‘safe care’. Both associations can be described as logical, considering the high visibility of nursing ward interventions for the patient, as well as the impact of an integrated approach on safety of care respectively.

10. Also, in the experience/improvement strategies analysis, authors use a Bonferroni correction that accounts for the 16 strategies investigated, but not necessarily across all outcomes. I might be misunderstanding what is done here (see point 3 above), but there are many more than just 16 comparisons being made.

Firstly, as stated above, we needed to correct our manuscript (see lines 159-164), as our Bonferroni correction only took 15 instead of 16 strategies into account. We understand the reviewer’s concern towards the multiple testing of several strategies on several outcomes, but we feel our revised manuscript already reduced the risk of false significant results, as we now no longer test for each individual survey question but for patient dimensions (n=8) only. By using a Bonferroni correction, we already apply the most conservative method, as Bonferroni does not take the correlation between outcomes into account. We therefore opted to not further take the tested dimensions into account for the Bonferroni correction. Moreover, although few associations remain significant after Bonferroni correction, our general conclusion is that associations between implemented strategies and FPS scores are weak or non-existing.

11. In the Discussion (line 286), the authors mention a strong negative outlier in the external consultants analysis - what happens to the findings if this outlier is removed from the analysis (to get a sense of the influence of this one observation)?

Our new analysis used a more recent version of the Flemish Patient Survey data, with slight changes in percentage top-box scores. This has led to subtle changes in estimates obtained for the analysis investigating associations between strategies and FPS scores for 2019. The association between external consultancy use and percentage of patients rating the hospital 9 or 10 is no longer significant now, although the estimate (-6.48; 95% CI: -13.68; 0.72) is similar to the previous one (-7.6, p=0.0409). As there are no significant associations observed for the external consultants strategy in the new results, we removed the part on external consultants from the discussion. It should be noted that results for this strategy should be interpreted with caution as only 3 hospital have implemented this strategy, resulting in wide confidence intervals (as can be seen from figure 3).

Minor points:

1. Line 203: in the Figure 1 caption, I believe "on of" should be "one of".

This is corrected and is now stated in line 210 of the revised document.

2. Lines 215-217: the given percentages do not match Figure 2 (nor calculations out of 44 hospitals), so these should be corrected.

The section on implemented strategies to improve patient experiences was revised and corrected. Amendments can be found in lines 217-223 of the revised document.

3. Lines 276-277: it is not clear which FPS outcome this line refers to. Also, no analyses are presented to demonstrate that use (or not) of a strategy was associated with improvement over time.

We provided more clarity in lines 310-313 of the revised document. Additionally, as we have stated above, we added analyses to demonstrate if use of a strategy was associated with improvements over time.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

8 Sep 2020

PONE-D-20-04886R1

Six years of measuring patient experiences in Belgium: limited improvement and lack of association with improvement strategies.

PLOS ONE

Dear Dr. ASTRID VAN WILDER:

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please revise as per reviewer one's comments. As per my suggestion in the previous comments, please provide additional information on your recommendations, particularly on those interventions that had a positive impact on patient experience - "Nursing ward interventions" and "Hospital wide education." Also, as recommended by reviewer one, please add this information on the positive interventions to the abstract and revise the final sentence of the abstract to reflect a more scholarly approach.

Please submit your revised manuscript by October 5, 2020. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Nelly Oelke

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The manuscript is improved from the previous version and the comments I made have been satisfactorily addressed. The only comment I would now add is that more could be made of the two strategies that appear to be associated with improvement, i.e. 'Nursing ward interventions' and 'Hospital wide education' as these are the two practical things that a reader could take from the paper. I would make two suggestions. First, somewhere in the discussion (or an appendix if necessary), the paper should describe what these actually are, to the extent that the information is available. Second, I would recommend that these two findings (almost the only positive findings in the paper) should be included in the abstract. I note that the word limit for an abstract in PLOS ONE is 300, whereas the current word count of the abstract is 246, so they have some space.

Personally, I don't like the last sentence of the abstract which is expressed in rather unscientific language. I also don't know if it's overstated - e.g. in relation to the world literature on quality improvement, is a 5% increase in top scoring hospitals in five years that bad? This could also be addressed in the discussion.

Reviewer #2: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Martin Roland

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

8 Oct 2020

Dear Sir,

Dear Madam,

We are once again incredibly grateful for the thorough synthesis and review of our manuscript by the PLOS ONE editor and external peer reviewers. We are glad our previous changes have overall been well received and feel the additional comments made in this review have merit and will add to an improved manuscript. Outlined below you will find our point-by-point response to all comments to demonstrate how we took them to heart and adjusted our manuscript. The line numbers refer to the revised manuscript document without tracked changes.

Academic Editor’s Comments

1. Please revise as per reviewer one’s comment.

See reply below.

2. As per my suggestion in the previous comments, please provide additional information on your recommendations, particularly on those interventions that had a positive impact on patient experience – “Nursing ward interventions” and “Hospital wide education.”

We added additional information on recommendations for practice in lines 328 to 336 and stated the following from line 332: “The examples provided by some participating hospitals such as e.g. mealtimes between staff and patients or the development of hospital-wide courses, suggest a large variety of ways to execute strategies. We thus encourage hospitals to share and learn from both their positive and negative experiences. By focusing on both nursing ward interventions and hospital wide education, a high visibility for the patient as well as a widespread reach of all healthcare staff can be ensured.”

We hope this information can be applied by hospitals reading this manuscript that are aiming to improve their patients’ experiences.

3. Also, as recommended by reviewer one, please add this information on the positive interventions to the abstract.

We added information on the positive associations between improvement strategies and patient experiences both in the results and conclusion section of the abstract. As such, we wrote the following in line 59: “Still, positive associations were discovered between the strategies ‘nursing ward interventions’ and ‘hospital wide education’ and recommendation of the hospital.” Additionally, in line 62, we wrote: “Hospitals report to have invested in patient experience improvement strategies but positive associations between such strategies and FPS scores are weak, although there is potential in further exploring nursing ward interventions and hospital wide education.”

4. Revise the final sentence of the abstract to reflect a more scholarly approach.

We altered the final sentence in our abstract to “Hospitals should continue their efforts to improve the patient’s experience, but with a more targeted approach, taking the lessons learned on the efficacy of strategies into consideration.” (line 65). As such, we hope to have provided a more nuanced conclusion that takes the improvements already made into account, but also highlights the fact that most strategies have not been associated with improvements in patient experience scores. A further focus on strategies that could potentially benefit the patient’s experience is therefore required.

Reviewers' comments:

REVIEWER #1

1. The only comment I would now add is that more could be made of the two strategies that appear to be associated with improvement, i.e. ‘Nursing ward interventions’ and ‘Hospital wide education’ as these are the two practical things that a reader could take from the paper. I would make two suggestions. First, somewhere in the discussion (or an appendix if necessary), the paper should describe what these actually are, to the extent that the information is available.

As we mentioned in our previous review, as well as in our limitation section (line 351) of our discussion, detailed information on when, how and on what ward the hospitals chose to implement their employed strategies, is unavailable for the majority of participating hospitals. Nevertheless, we understand the need for more detailed information on the surveyed strategies, in particular those with potential benefit towards the patient’s experience. We therefore contacted the quality managers of three participating hospitals that all indicated they employed both nursing ward interventions and hospital wide education and asked them to elaborate on how they employed these strategies. While exact details could not be provided, they could give examples of interventions they had implemented, which add to the understanding of what that particular strategy entails. As such, we added more information in the description section of Table 1 (line 234) for both nursing ward interventions, hospital wide interventions and hospital wide education. We provided extra information in the discussion section as well in line 332: “The examples provided by some participating hospitals such as e.g. mealtimes between staff and patients or the development of hospital-wide courses, suggest a large variety of ways to execute strategies. We thus encourage hospitals to share and learn from both their positive and negative experiences. By focusing on both nursing ward interventions and hospital wide education, a high visibility for the patient as well as a widespread reach of all healthcare staff can be ensured.”

In addition, we put more emphasis on the potential benefits of nursing ward interventions and hospital wide education throughout the discussion section. First, we added lines 315 to 318: “What’s more, both nursing ward interventions and hospital wide education were found to be associated with better 2019 FPS results. Additionally, nursing ward interventions in particular were positively associated with improved global patient experiences over time.” Second, we wrote lines 321-324: “Despite the positive associations between both nursing ward interventions and hospital wide education and 2019 FPS results and the positive relationship between nursing ward interventions and recommendation of the hospital, improvement strategies were overall not or only weakly associated with patient experience ratings.” Last, we made some considerations for practice in lines 328-336: “Considering its potential, further research into the benefits of nursing ward interventions or a hospital-wide educational program is advised. By researching the evidence-base on the interventions that have shown most promise, we hope future healthcare policy and practice might be altered towards a more unified care, instead of the wide spectrum of sometimes ineffective interventions currently implemented. The examples provided by some participating hospitals such as e.g. mealtimes between staff and patients or the development of hospital-wide courses, suggest a large variety of ways to execute strategies. We thus encourage hospitals to share and learn from both their positive and negative experiences. By focusing on both nursing ward interventions and hospital wide education, a high visibility for the patient as well as a widespread reach of all healthcare staff can be ensured.”

By the best of our abilities, we thus hope to have provided you with a deeper understanding of how the surveyed strategies could be implemented in practice.

2. Second, I would recommend that these two findings (almost the only positive findings in the paper) should be included in the abstract.

We added information on the positive associations between improvement strategies and patient experiences both in the results and conclusion section of the abstract. As such, we wrote the following in line 59: “Still, positive associations were discovered between the strategies ‘nursing ward interventions’ and ‘hospital wide education’ and recommendation of the hospital.” Additionally, in line 62, we wrote: “Hospitals report to have invested in patient experience improvement strategies but positive associations between such strategies and FPS scores are weak, although there is potential in further exploring nursing ward interventions and hospital wide education.”

3. Personally, I don’t like the last sentence of the abstract which is expressed in rather unscientific language. I also don’t know if it’s overstated – e.g. in relation to the world literature on quality improvement, is a 5% increase in top scoring hospitals in five years that bad? This could also be addressed in the discussion.

We altered the final sentence in our abstract to “Hospitals should continue their efforts to improve the patient’s experience, but with a more targeted approach, taking the lessons learned on the efficacy of strategies into consideration.” (line 65). As such, we hope to have provided a more nuanced conclusion that takes the improvements already made into account, but also highlights the fact that most strategies have not been associated with improvements in patient experience scores. A further focus on strategies that could potentially benefit the patient’s experience is therefore required.

Lastly, to further nuance the improvements made by the participating hospitals, we removed the comment ‘yet small’ (talking about significant improvement) from the results section (line 53) of the abstract. As you rightly pointed out, a 5% increase over the course of 5 years is still a commendable achievement. We also highlight this in line 296 of the discussion of the manuscript: “The overall improvement, strongest in most recent years, is commendable, yet small.” Still, like our discussion continues, we believe there is still room for improvement, considering the achievements made in other countries and considering the fact most of the surveyed strategies had no impact on the patient’s experience.

We would like to thank you again for considering our manuscript for publication in your journal.

Yours sincerely,

Astrid Van Wilder

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

15 Oct 2020

Six years of measuring patient experiences in Belgium: limited improvement and lack of association with improvement strategies.

PONE-D-20-04886R2

Dear Dr. Van Wilder,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Nelly Oelke

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

20 Oct 2020

PONE-D-20-04886R2

Six years of measuring patient experiences in Belgium: limited improvement and lack of association with improvement strategies.

Dear Dr. Van Wilder:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Nelly Oelke

Academic Editor

PLOS ONE