Improving the paediatric surgery patient experience: an 8-year analysis of narrative quality data.

BACKGROUND: Narrative data about the patient experience of surgery can help healthcare professionals and administrators better understand the needs of patients and their families as well as provide a foundation for improvement of procedures, processes and services. However, units often lack a methodological framework to analyse these data empirically and derive key areas for improvement. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) is aimed at improving the quality of surgical care by collecting patient data and reporting risk-adjusted surgical outcomes for each participant hospital in the programme. Though qualitative data about patient experience are captured as part of the NSQIP database, to date no framework or methodology has been proposed, or reported on, to analyse these data for the purposes of quality improvement. The goal of this study was to demonstrate the feasibility of using content analysis to empirically derive key areas for quality improvement from a sample of 3601 narrative comments about paediatric surgery from patients and families at British Columbia Children's Hospital. STUDY
DESIGN: Thematic content analysis conducted on a total of 3601 patient and family narratives received between 2011 and 2018.
RESULTS: Overall satisfaction with care was high and experiences with healthcare providers at the hospital were positive. Areas for improvement were identified in the themes of health outcomes, communication and surgery timelines. Results informed follow-up interprofessional quality improvement initiatives.
CONCLUSIONS: Recording and analysing patient experience data as part of validated quality improvement programmes such as ACS NSQIP can provide valuable and actionable information to improve quality of care. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Patient experience and patient outcomes play a fundamental role in the quality of healthcare systems. Quantitative and qualitative data about patient experience can help healthcare professionals and administrators better understand the needs of patients and their families as well as provide a foundation for improvement of procedures, processes and services. However, units often lack a methodological framework to analyse these data empirically and derive key areas for improvement. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) is aimed at improving the quality of surgical care by collecting patient data and reporting risk-adjusted surgical outcomes for each participant hospital in the programme. Through medical records and communications with patients, ACS NSQIP collects patient information, such as patient demographics, preoperative risk factors, operative information, and perioperative and postoperative outcomes within a defined time period after surgery.1 This information is used to calculate the expected number of negative outcomes for each institution and this number is compared with the observed number of negative outcomes to pinpoint the areas that need improvement.2 Though qualitative data about patient experience are captured as part of the NSQIP database, to date no framework has been proposed, or reported on, to analyse these data for the purposes of quality improvement (QI). Here we address this critical gap by hypothesising that content analysis is a feasible methodology for this purpose and test our hypothesis using a sample of over 3600 qualitative comments from patients and families.

Patient experience in surgery

Patient experience in medicine is an important driver of healthcare quality. In surgery, including paediatric surgery, most of the research explores the relationship between patient satisfaction and patient outcomes, such as readmission, morbidity and mortality rates.3–7 Many studies collect data using patient satisfaction surveys, with the Hospital Consumer Assessment of Healthcare Providers and Systems survey being the most commonly used patient experience reporting tool.4 6 7 Other studies use preoperative and/or postoperative questionnaires to measure patient experience and outcome,3 as many have found that high patient satisfaction is linked to lower 30-day mortality, morbidity and readmission rates.4–7 However, other studies have shown a non-linear relationship between patient satisfaction and patient outcomes.5 7 As such, harnessing a methodological framework to evaluate patient experience independently of outcomes can assist in establishing quality baselines and in identifying areas for improvement that are more remotely linked to morbidity or mortality outcomes.

The goal of this study was to test the feasibility of using content analysis to conduct a rigorous descriptive quantitative analysis of narrative data to further our understanding of the paediatric surgery experience and to identify key strengths and opportunities for improvement.

Methods

Study context and participants

British Columbia Children’s Hospital (BCCH) has participated in ACS NSQIP since 2011. The quantitative clinical data collected have been used to compare morbidity and mortality rates with other paediatric hospitals, and identify targets for QI work. Using the rates of adverse outcomes as a guide, the ACS NSQIP team has created QI projects to target the most common adverse outcomes for reduction. In addition to collecting quantitative patient data, patient families are given the opportunity to provide feedback on their experience at BCCH at 30 days after surgery. Since 2011, a total of 3601 narrative reports on patient and family experience have been aggregated.

Design and coding

Taking the feedback from patient families, we conducted a content analysis of the narratives that have been received since 2011. We used qualitative description to characterise the experiences of families. A total of 3601 narrative patient experience comments were retrieved from the ACS NSQIP database and the unit of analysis was selected to be each single comment independent of its length. Comments that did not pertain specifically to patient experience (n=111) were excluded from the analysis. The full sample was randomised and an initial 10% of the comments were retrieved for a pilot analysis in order to develop a coding guide to represent the broad themes found using an emergent coding strategy. This coding guide was first refined through discussion between researchers (JMR (F), SCB (F), MTT (F)). To ensure that the coding guide reliably captured all the thematic categories within the data, two coders (SCB, MTT) applied it to a second subsample of 10% (n=360) of the comments. Any discrepancies or disagreements were discussed and resolved through consensus, and the coding guide was refined using an iterative process using additional subsamples of 5% of the data. Following this pilot coding phase, inter-rater reliability reached 93% between the two coders using the coding guide. The remainder of the sample was coded by the primary coder (SCB) using Microsoft Excel.

The final coding guide comprised the following major themes that reflect the types of issues and feedback contributed by patients and families: (1) overall satisfaction with patient experience; (2) health outcomes (eg, pain management, healing, complications); (3) experiences with hospital staff (eg, overall staff, doctors, nurses, trainees); (4) communication (eg, from doctors, between doctors/departments); (5) timelines (eg, wait times); and (6) facilities.

Patient and public involvement

Patients and their families contributed the narrative reports analysed in the present study. The research team consulted with patient representatives when interpreting the results. In response to the findings, the research team is planning a series of workshops with patients, families and hospital staff to improve pain management communication. Patients will contribute to the dissemination strategies by determining the results they deem most important to share with hospital staff in order to encourage improvements of the overall quality of care.

Results

Through content analysis, the large quantity of patient narrative data was reduced to quantitative measures related to the key thematic categories. To demonstrate the feasibility of using content analysis to rigorously capture key areas for improvement in a set of narratives and to illustrate the nature of a data set resulting from content analysis, here we provide descriptive data from our sample organised by major theme, supported by relevant quotes. Henceforth, for each thematic category, numbers are reported as (percentage, numerator/denominator for the subsample of the thematic category). For example, of the total comments that relate to a certain subsample of a thematic category (eg, comments on pain management), a percentage of those comments indicate a specific experience within that subsample (eg, good pain management or poor pain management).

Overall experiences

A majority of comments (77%, 2778/3601) mentioned overall satisfaction with the experiences of families and care their child received (see figure 1 for a detailed breakdown of the comments regarding overall satisfaction). Of these comments expressing an overall level of satisfaction (n=2778), most indicated that families were pleased with the care they experienced, grateful to the hospital, and expressed comments that were characterised as very satisfied (56%, 1543/2778) or satisfied (26%, 731/2778) by the research team. Examples that illustrate satisfaction with patient experience include:

Figure 1

Overall satisfaction. Feedback from families regarding overall satisfaction on experiences at the hospital for surgery.

We’re glad we went to BC Children’s Hospital. Couldn’t have been any better.

Our daughter was treated with the utmost care, respect and love! Thank you for the amazing work you all do!

Health outcomes

A common topic of feedback was regarding the physical health outcomes of children after surgery (26%, 921/3601). General mentions of health outcomes were mostly positive (good=67%, 619/921; bad=33%, 302/921) (see figure 2 for a more detailed outline of comments regarding health outcomes). A majority of families who reported on whether their children had improved following surgery indicated an improvement (92%, 262/285). With regard to wound healing specifically, a majority of the comments were positive stating that they were healing well (82%, 103/126). However, families had a negative feedback related to complications after surgery (52%, 165/315) and pain management (93%, 71/76) (figure 2). Even though parents were happy with some parts of their experience, they were mostly worried about their child’s pain:

Figure 2

Health outcomes. Comments regarding the health outcomes of their children after undergoing surgery.

Satisfied w/ Dr’s. Not happy with discharge after 2 days. Had to return to hospital. Should have kept her in longer to keep close eye on her. She was still in pain. I think they should keep the patient in for a few days longer.

Experiences with hospital staff

A proportion of the total comments (57%, 2040/3601) included a mention of hospital staff, healthcare professionals or others involved in the child’s care (table 1). The majority (91%, 1865/2040) of these comments included positive feedback about the personnel, and about nurses and doctors in particular, families encountered during their hospital visit:

Table 1

Experiences with different hospital staff

	Very positive	Positive	Neutral	Negative
Staff (unspecified) (n=618)	76	21	1	2
Doctors (n=776)	74	20	3	3
Nurses (n=530)	66	17	7	10
Residents (n=22)	45	9	9	36
Surgical team (n=65)	82	18	–	–
Clerical staff (n=26)	46	15	8	31

Numbers are presented as %.

Every single member of the surgical team (attending, fellows, residents, nurses etc.) gave fantastic family and patient centered care to my son. There seems to be amazing communication amongst staff in your department and fantastic modelling of great care to the more junior staff. Thank you. P.S. There actually isn’t one thing that I can think of that would have made my son's surgery or care better or safer. :)

However, the balance between positive and negative comments was not equal for each type of healthcare professional or staff. For example, in comparison to the comments on other medical staff, an increased number of negative comments were reported for experiences with medical residents (positive=55%, 12/22; neutral=9%, 2/22; negative=36%, 8/22) and clerical staff (positive=62%, 16/26; neutral=8%, 2/26; negative=31%, 8/26). Some parents felt as though residents needed more supervision or training in order to care for their child and that some clerical staff were not helpful.

Comments were also coded for the types of qualities the families observed in staff. The most commonly mentioned qualities stated that they were helpful (20%, 228/1162), friendly (14%, 167/1162) and supportive (12%, 140/1162). Negative traits were mentioned in a smaller proportion of comments, for example, not helpful (2%, 18/1162), not efficient (2%, 26/1162). In addition to these qualities, some parents also raised comments about whether they felt that the staff listened to their concerns, with most feeling that they did not listen (61%, 56/92). For those who mentioned their experiences with the bedside manner of doctors and nurses, 67% felt that doctors and nurses had good bedside manner (16/24) and 33% had bad bedside manner (8/24), as illustrated in the following example:

The nurses in the recovery room need to have better bedside manners!

To capture a more specific feature of the patient and family/personnel interactions, a code was generated for comments giving a sense of ‘dignity and respect’ or the lack thereof. This represented those experiences of parents who felt that they were treated respectfully by the people they encountered at the hospital:

[…] not rushing made us feel like we were human, not just part of an assembly line.

Communication

Families had different experiences with the types of information provided to them during their stay at the hospital regarding surgical processes, information about home care after discharge and information about the healing trajectory after discharge (figure 3). A majority (74%, 158/214) of comments indicated that families felt that they were provided with adequate information for them to feel comfortable at the hospital, and some felt that there were communication issues between doctors, departments and other staff which affected the care they received:

Figure 3

Communication. The experiences of families with the communication encountered during their stay at the hospital.

[There was a] hiccup, miscommunication, the right people didn’t know he was scheduled for surgery. Gtube nurse didn’t know he had surgery, so we had to wait to see her, didn’t end up seeing her. Nurse had to sign forms for supplies, but the nurse didn’t have [Patient] on list. So no supplies, and didn’t actually get them for 1.5 wks.

A common discharge issue identified in the comments was that families did not feel they received enough information that prepared them for caring for their child at home after surgery (79%, 67/85) as well as not given adequate details about what would happen during the healing process (89%, 59/66):

We were discharged with no specific follow-up instructions, unless we were concerned or saw infection or other complication. [Patient] was doing very well post-op but I just wanted reassurance by having our GP look at incision.

Another example highlights the emotional aspect of the lack of communication after discharge:

When getting discharged we did not get any instructions regarding possibility of post-surgical seizures. [He] did not experience a seizure until 5 days after surgery; the night we went home to [City] 2 days prior to that he was taken off the steroids used to control the swelling. I wonder if this was why he had seizures & why we were not warned this could happen. We had no knowledge of what to do we were unprepared & frightened.

Timelines

A substantial number of responses about surgery were related to the timelines that families experienced. Some parents reported long wait times in the clinic (35%, 11/31), while comments referring to waiting for surgery (long wait times=79%, 92/117; short wait times=21%, 25/117) were coded when families expressed that they waited a long time for the surgery to be scheduled or when there were long wait times during the day of surgery. Families experienced delays in imaging (eg, MRI, X-ray) before surgery (4%, 6/167) and a few (2%, 3/167) were given less than 24 hours’ notice for surgery which was difficult for those with long travel distances to the hospital. In addition, some parents (20%, 53/269) felt that their children were discharged too soon after surgery:

…discharged you rushed them out, he couldn’t get up, rushing him out. Threw up in car and elevator. Keep them in for longer.

Another example demonstrates the feelings that families have when they are discharged too soon from the hospital:

Our daughter was discharged 12 hours after her emergency lap Appendectomy. This seems a bit too soon. Sure enough she developed a fever within 24 hours post discharge. I would think that patients should be observed for longer after operation of this magnitude.

Facilities

Families who had experiences with the clinic had issues with contact, as some reported that it was difficult to make an appointment (52%, 16/31) or that the clinic did not book a follow-up appointment with them (13%, 4/31).

Safety concerns (9/3601) were mentioned when families felt that they or their child was unsafe due to the provided amenities or the events happening within the hospital:

A very unusual thing happen. There was some kind of child custody issue on ward, I felt extremely unsafe, begged Dr. to let us go home, I felt unsafe. Individual was still allowed in. We should not have been around that, felt uncomfortable. Unsafe around that parent. Told Dr. how we felt. We went home early. Other little girl in room was so upset. A bit scary.

A small proportion of comments (7%, 238/3601) included considerations related to the physical and administrative components of the actual facility. Examples of comments about facilities included issues with the shared rooms being too loud and disruptive for the families or their child, and having to request to move to a single room. Other issues were raised about parking such technical difficulties with the parking metre, stress about topping up the parking metre and the excessive parking costs during their visit and stay.

Discussion

The use of content analysis to analyse healthcare and patient experience data has become increasingly popular over the last decade. While this method has been traditionally used in the context of interview studies8 or media analyses, content analysis is now being applied to large data sets, including social media,9 and clinical databases such as in the present analysis. Results from the 8-year analysis of patient experience narratives demonstrate the potential for empirical content analyses of databases such as ACS NSQIP to successfully uncover key strengths and areas for QI in the paediatric surgery setting.

Methods in paediatric patient experience research usually involve distributing surveys to the parents of patients to better understand their experience, with some studies also involving the paediatric patients directly.10–12 Findings from these active participation studies align with those of our database analysis. For example, survey results revealed that parents reported more positive experiences when information about the surgery was provided ahead of the intervention, and when families encountered staff who were not in a rush to provide care.10–12 In addition, higher patient and family satisfaction was also linked to the involvement of the child patient in conversations about his/her health and treatment.10 11 These findings are supported by the emphasis on quality communication uncovered in the NSQIP data.

With the increasing interest in using patient experience to improve the quality of healthcare, the body of literature assessing the impact of using patient experience data as a driver of QI initiatives is growing. Indovina et al used patient experience data to inform a randomised, QI initiative to tackle low patient satisfaction in physician communication.13 As a result of this intervention, satisfaction with communication from doctors, patient perception of courtesy and respect from physicians, and satisfaction with physicians listening to patients increased by 21%, 53% and 38%, respectively.13 Similarly, a Pittsburgh hospital created a QI project to increase patient involvement and positive experiences at discharge following adult orthopaedic spine surgery.14 Patient satisfaction scores increased when patients and families were involved in goal setting and provided education videos during the discharge planning process.14 In the paediatric setting, Soeteman et al described using surveys before and after introducing a programme to reduce paediatric ambulatory clinic wait times.15 Feedback from patients provided the opportunity to evaluate current operational processes and further improve the quality of services.15 The methodology described in the present study and findings from our analysis offers a foundational process to capture the authentic experiences of families and patients and use these data to proactively address quality and satisfaction of care.

Some of the challenges in patient experience data analysis include recall bias, limited generalisability between populations, response rate and difficulty in translating the data into meaningful care improvements.3 6 7 16 While the analysis in this study provides new applications of analytical methods and detailed insight to the experiences in paediatric surgery, we acknowledge the limitations of the approach. First, families are more likely to comment about a certain aspect of their experience, such as pain management, if it was a negative experience rather than a positive one which creates bias in our findings. Second, there may have been a selection bias as families were contacted via phone for their comments only if they did not have a 30-day follow-up. For complex cases, a letter was sent to complete a survey along with any additional comments they wanted to share. Although these comments were also entered in the NSQIP database, the different data collection modalities may interfere with data quality. Third, all comments were given in English and therefore do not capture the perspectives of patients and families who do not speak or write in English. Fourth, feedback regarding the hospital facilities (eg, rooms) may not be relevant as the building was upgraded during the data collection period. Finally, patient narratives are not a validated metric in ACS NSQIP, therefore the present study focuses on the importance of its inclusion. Future research will explore linkages between patient narratives and surgical outcomes, and the effectiveness of internal and patient-facing dissemination strategies for the data obtained through the present analysis.

Conclusions

Databases such as ACS NSQIP play a critical role in improving the quality of surgical care in paediatric populations. The inclusion of patient experience as an outcome of the surgical encounter enriches the data available for analysis and provides new opportunities for improving care to patients and families along the surgical care pathway.