Usefulness of a national parent experience survey in quality improvement: views of paediatric department employees.

OBJECTIVES: This study presents results from an electronic survey among paediatric department employees, addressing employees' attitudes and use of results from a national parent experience survey carried out in 2005.
METHODS: Electronic questionnaire survey of employees from each of the 20 paediatric departments included in the national survey, with a response rate of 87%.
RESULTS: The employees had favourable opinions of user experience surveys, and the results from the national survey were well known among both managers and other personnel. User experience surveys were considered important, and 56% reported that they had implemented improvement actions addressing problems identified in the national survey. Managers reported more often than staff without managerial responsibility that the results had been informally discussed, and that the survey was useful for their own department. Department leaders were more positive to the usefulness of the survey than non-leaders. Significant differences in attitudes were found between physicians and other health personnel.
CONCLUSION: Employees in the paediatric departments were positive to user experience surveys, and the surveys have a potential to be actively used in quality improvement actions. Effects of the quality improvement initiatives should be assessed in future parent experience surveys.

Introduction

There has been increasing emphasis on the use of patient satisfaction surveys in health services to assess quality of care. The purpose of these surveys is often related to quality improvement. However, a range of methodological challenges threaten the value of using survey data, such as psychometric properties of the measurement instruments,1 non-response2 and case-mix.3 4 Many studies have focused on methodological issues, but few have investigated their active use and utility in quality improvement efforts at the organisational level.5–7

A French study found positive attitudes towards patient experience surveys among employees at a University hospital, but the results were not systematically used in quality improvement.6 Another study found organisational, professional and data-related barriers for active use of patient survey data, and concluded that better use requires the development of cultures that support patient-centred care, quality-improvement capacity, professional receptiveness and leadership, and technical expertise with these data.5 Corresponding conclusions were made in a third study.7

Due to the paucity of studies addressing this issue, we carried out a follow-up survey among health personnel at specialised children departments included in a national parent experience survey. The aim of the study was to explore employees' attitudes to user experience surveys, their opinions on the usefulness of the national survey as well as their reported use of the results.

Method

The head of department at each of the 20 specialised children departments included in the national parent experience survey in 20058 was asked to choose five employees from their own department. Selection was based on the following criteria (ranked by importance): departmental leader, employees with a special role in quality improvement, other physicians, nurses. Contact information for the sample was sent to us by each head of department.

The survey was conducted electronically from June to October 2007. Non-respondents were sent up to four reminders.

Questionnaire

The questionnaire was based on a previous study,6 with modifications conducted by experts on survey methodology to capture issues specifically relevant to this study.

The questionnaire comprised questions concerning attitudes to user involvement and user experience surveys, the national parent experience survey in 2005 and employees' opinions on the use and usefulness of the national survey. Most questions had a five-point response format that ranged from ‘not at all’ to ‘a very large extent.’ Results on 16 of the 18 closed-ended questions and one of the six open-ended questions are reported.

Analysis

Statistical analyses were carried out using SPSS (15.0; SPSS, Chicago, Illinois). Descriptive statistics are presented for the closed-ended questions about attitudes, use and usefulness. The mean and SD for department leaders, head of sections and employees without a management position are reported for questions with a five-point response format. One-way analysis of variance (ANOVA) with Bonferroni post hoc correction was used to test differences between groups. Remaining questions are reported as percentages answering ‘yes,’ and the χ2 test was applied to test differences. The questions measuring attitude were included in an exploratory factor analysis, and the Cronbach α coefficient was applied to evaluate the internal consistency of the scale.

The association between socio-demographic and professional characteristics and attitudes as well as a single item addressing the usefulness of the survey was assessed by bivariate and multivariate linear regression analyses. An exploratory analysis was chosen due to the scarceness of studies. The association between the two dependent variables and each of the independent was explored by bivariate regression models. Independent variables with p<0.1 on at least one of the dependent variables were included in multivariate regression models.

Content analysis was applied on responses to the open-ended question about implemented improvement actions. The responses were categorised into major topics, and a content analysis of each topic was conducted to identify the most important themes.9

Results

Response rate and characteristics of respondents

Nineteen of 20 children departments from the national parent survey participated in the follow-up survey. One hundred employees were included and posted the questionnaire. Nine never received the email and were excluded. Completed questionnaires were returned by 79 (87%). Twelve respondents filled out paper questionnaires.

Seventy-one per cent were female, the mean age was 46 years, and the mean work experience was 14 years. Fifty-three per cent were trained nurses, and 38% were physicians. Sixty-two per cent had a management position; 23% were department leaders, and the remainder were a head of a section or had other management positions below the department level. Forty per cent worked at a district general hospital, 38% at a University hospital and the remainder at a local hospital.

Main results

All attitude items regarding user involvement were rated positively. On a scale from 1 to 5, where 5 represent the most positive attitude, the mean score was around 4 (table 1). The importance of patient experience surveys had the highest total score of the items concerning user experience surveys (4.16), while the question addressing the potential effect on medical quality had the lowest score (3.15). No significant differences were found between department leaders, head of sections or employees without a management position on the attitude items.

Table 1

Descriptive statistics for questions with five-point response format*, grouped by type of position within the department

Main topic/question	Department leaders (n=18)		Head of section/lower level (n=30)		Ordinary employees (n=29)		Total (n=79)
	Mean	SD	Mean	SD	Mean	SD	Mean	SD
User involvement
Do you think it is important to involve patients in decisions regarding treatment and caring?	4.61	0.50	4.23	0.63	4.31	0.66	4.35	0.62
Do you think it is important to involve patients in healthcare quality assessment?	4.33	0.49	4.07	0.58	4.38	0.62	4.25	0.59
How important do you think it is to involve parents in decisions regarding the treatment and caring for their children?	4.33	0.77	4.37	0.67	4.24	0.58	4.33	0.66
In general, do you think it is important to involve parents in healthcare quality assessment?	4.06	0.73	3.83	0.70	3.90	0.67	3.91	0.68
User experience surveys
All in all, do you think patient experience surveys are important?	4.33	0.77	3.97	0.67	4.28	0.65	4.16	0.69
All in all, do you think patient experience surveys can contribute to improvements in the health service?	4.00	0.91	3.80	0.71	4.00	0.80	3.91	0.79
Do you think patient experience surveys can contribute to improving the medical quality of the health service?	3.33	0.77	3.33	0.71	3.41	0.78	3.37	0.74
All in all, do you think parent experience surveys can contribute to improvements in the health service?	3.83	0.71	3.60	0.77	3.69	0.71	3.68	0.73
Do you think parent experience surveys can contribute to improving the medical quality of the health service?	3.22	0.73	3.13	0.68	3.10	0.72	3.15	0.70
Use and usefulness of the national survey
Have the results of the survey been discussed informally in the department?	3.22	0.81	2.90	0.90	2.22	0.93	2.73	0.96
All in all, do you think the national parent survey in 2005 has been useful for your department?	3.29	0.69	2.96	0.88	2.56	0.97	2.89	0.91

Five-point response format ranging from ‘not at all’ to ‘a very large extent.’

Respondents attached a lower value to the assessment of the usefulness of the national survey (2.89) and to the question concerning informal discussion of the results (2.73) than to the attitude items (table 1). Department leaders had significantly higher scores than the other groups as concerns informal discussion (F value: 7.62, p=0.001). Department leaders also had a more positive evaluation of the usefulness compared with employees without a management position (F value: 3.86, p=0.026).

Most respondents (86%) had been informed about the results for their own department, and 78% reported that the results had been discussed (table 2). Almost 20% believed that the national survey had changed the way health personnel behave towards patients. Many (65%) had knowledge of the report from the survey, and 56% reported that improvement actions had been implemented as a consequence of the survey. Department leaders and other leaders gave more positive responses than employees without a management position on all questions in table 2; however, χ2 tests are not reported because the sample size requirement for the test was not satisfied; the expected frequency for many of the cells was less than 5 and could not be interpreted.

Table 2

Percentage ‘yes’ for questions with ‘yes’ and ‘no’ response categories, grouped by type of position within the department

Question	Percentage ‘Yes’
	Department leaders (n=18)	Head of section/lower level (n=30)	Ordinary employees (n=29)	Total (n=79)
Have you been informed about the results for your department? How?*	100	87	76	86
Has your department discussed the results formally (ie, department meeting, employee meetings)?†	100	80	62	78
Has your department implemented initiatives to improve weaknesses identified by the parent survey?†	89	53	38	56
Have the results of the parent survey led to changes in how employees behave towards patients?†	33	17	11	18
Do you have knowledge of the report from the national parent survey in 2005?	83	70	48	65

‘Yes’ includes the following response categories: ‘yes, meetings,’ ‘yes, notice,’ ‘yes, internal notes or email,’ ‘yes, informal meetings with colleagues,’ ‘yes, other.’

These questions included an additional ‘Don't know’ category. The denominator includes this category, and the number of ‘Don't know’ responses ranges from 15 (formal discussion of results) to 56 (changes in behaviour).

The nine questions measuring attitude were included in an exploratory factor analysis, and a unidimensional scale was identified. One of the questions was excluded due to low factor loading and low item–total correlation. The final scale had a Cronbach α coefficient (α=0.89) well above the recommended value of 0.7;10 11 item-total correlations were also satisfactory (0.46–0.77).

Table 3 shows that several variables have a significant bivariate association with either the attitude scale or the question addressing usefulness (p<0.1). However, the multivariate regressions (table 4) identified only two significant associations; physicians were less positive than others on items comprising the attitude scale (p=0.01), and department leaders were more positive to the usefulness of the survey than employees without a management position (p=0.042).

Table 3

Bivariate association between independent variables and the attitude scale and the question about usefulness

Independent variables	Attitude scale		Usefulness for department
	Coefficient*	p Value	Coefficient*	p Value
Male (vs female)	−0.290	0.027	0.362	0.133
Age	0.007	0.305	0.009	0.507
Up to 10 years work experience (vs 10 or more)	−0.231	0.068	−0.261	0.285
Employees with quality functions (vs employees without quality functions)	−0.023	0.879	0.227	0.408
Type of position
Head of section/lower (vs department leaders)	−0.167	0.166	0.123	0.580
Ordinary employees (vs department leaders)	0.054	0.661	−0.533	0.015
Professional background (physicians vs others)	−0.470	<0.001	−0.119	0.597
Type of hospital
Local hospital (vs university hospital)	−0.174	0.224	−0.018	0.946
District general hospital (vs university hospital)	0.062	0.610	0.128	0.562
Experience from local patient experience surveys at own department (vs no such experience)	0.148	0.222	0.240	0.281

Unstandardised regression coefficient.

Table 4

Multivariate regression models on the association between independent variables and the attitude scale and the question about usefulness

Independent variables	Attitude scale		Usefulness for department
	Coefficient*	p Value	Coefficient*	p Value
Male (vs female)	0.041	0.820	0.606	0.074
Up to 10 years work experience (vs 10 or more)	−0.148	0.265	0.034	0.894
Type of position
Head of section/lower (vs department leaders)	−0.114	0.437	−0.242	0.382
Ordinary employees (vs department leaders)	−0.064	0.684	−0.610	0.042
Professional background (physicians vs others)	−0.435	0.010	−0.535	0.088

Unstandardised regression coefficient.

Forty-three employees reported that the department had implemented improvement actions based on results from the national survey, and 36 respondents described the actions more specifically. Major topics were information, role clarification, continuity and routines. Role clarification was related to relationships between employees and parents but also within the department. Predictability was underlined as a main target of efforts addressing the relation and dynamics between health personnel and parents. Continuity was mainly related to health personnel continuity for parents during the hospital stay. The comments also showed that the children departments endeavour to optimise routines to meet the needs for improvement of standardisation.

Discussion

This national study shows that health personnel in paediatric wards report active use of results from a parent experience survey in quality-improvement work at the ward level. We did not intend to assess the national surveys' effect on quality, but sought the opinions of important target groups on the follow-up of the survey. To our knowledge, this is the first study focussing on the use and usefulness of a national parent experience survey conducted at hospital departments for children.

Hospital management have been emphasised as most responsive to data from quality-measurement studies, especially top management, and a common effect is implementation of quality-improvement initiatives.12 13 The effect of these initiatives is debated and beyond the scope of our study, but we did find that leaders gave the most favourable assessment of effect and usefulness. Attitudes were affected by professional background, while assessment of usefulness depended on the type of position. These findings are confirmed by other studies.6 14 A recent study illustrates the importance of this issue, showing that the personal involvement of the department head played the key role in the continuation of the quality-management programme.15 Another study also concluded that it seemed especially difficult to engage medical doctors.16

The study sample was constructed to measure attitudes among leaders and employees involved in quality improvement, because those groups are core personnel in implementing improvement projects based on quality measurement. The survey was national and included all hospitals in Norway with a children department. The response rate was high, implying that potential effects related to non-response are of minor concern. Therefore, the results can be expected to be representative at the national level for the groups included in the study sample. Obviously, testing the effect of the national survey on all personnel at hospital children departments would call for another design. The generalisability of our results to other settings is uncertain. We built our study on the study by Boyer et al,6 the latter being conducted at a French teaching hospital. Our findings resemble the main findings in the study by Boyer et al,6 but the poor knowledge base regarding this topic warrants more research to validate the findings in other settings. We are currently planning a similar study within mental healthcare.

The inclusion of attitudinal questions means that some of the results are not very specific. The reason for including attitudinal questions is twofold. First, we wanted to base and compare our study with the study by Boyer et al,6 and they included several attitudinal questions. Second, the qualitative study by Davies & Cleary5 identified professional barriers as one important barrier to the use of patient survey results in quality improvement. Negative attitudes might impede the active use of survey results, so we wanted to assess the extent of this barrier in our study. Both our study and the study by Boyer et al6 show that health personnel had positive attitudes to patient surveys, indicating that the other barriers in the study by Davies & Cleary5 might be more important in these settings. However, future research might use fewer questions about attitudes making more room for questions with a more specific focus.

It is commonly assumed that feedback from patients on professional performance might lead to improvement activities and, accordingly, a change in behaviour towards the patients. Despite positive attitudes, only 18% reported that the results of the national survey had led to changes in the behaviour of health personnel. However, we believe that the main reason for this result is poor knowledge of the topic addressed by this question. To reduce social desirability bias, we have asked the respondent to assess behaviour change for health personnel in the department but overestimated the respondents' knowledge of this topic; 56 (74%) of the respondents answered that they did not know whether other health personnel had changed their behaviour. In future surveys, this topic should be addressed by improved questions including changes in one's own behaviour in addition to changes for other groups of health personnel.

The national survey found that certain specific aspects related to organisation and nursing were rated most negative.8 The quality-mprovement actions reported in the current study were closely related to these aspects, but we are unable to validate questionnaire responses because we lack other data on actual quality-improvement work at the departments. However, qualitative comments include detailed descriptions of a range of follow-up activities. Another methodological issue is related to social desirability bias: we collected data in a study assessing another of our own surveys. This might have affected responses, especially by increasing the pressure for giving social desirable answers. However, questions related to assessment of our national survey had the most negative responses (ie, usefulness), indicating little social desirability bias.

A major goal for user experience surveys is active use of results in quality-improvement work,17 but few studies explore this systematically, and results are contradictory. This study showed that our national parent experience survey has influenced quality work. Whether or not the initiatives and actions actually improve quality needs to be explored in future studies including new parent experience surveys.18