The Nordic Patient Experiences Questionnaire (NORPEQ): cross-national comparison of data quality, internal consistency and validity in four Nordic countries.

OBJECTIVES: To evaluate the Nordic Patient Experiences Questionnaire (NORPEQ) for data quality, reliability and validity following surveys of patients in Finland, Norway, Sweden and the Faroe Islands. DESIGN, METHODS AND PARTICIPANTS: The NORPEQ was mailed to 500 patients randomly selected after receiving inpatient treatment in Finland, Norway and Sweden. The NORPEQ was also included in a national survey in Norway and in the Faroe Islands. Dimensionality was assessed using principal component analysis and internal consistency by item-total correlation and Cronbach's α. Construct validity was assessed by correlating NORPEQ scores with variables known to be related to patient experiences.
SETTING: Somatic hospitals in Finland, Faroe Islands, Norway and Sweden. PRIMARY AND SECONDARY OUTCOME MEASURES: Item missing, internal consistency reliability and construct validity.
RESULTS: Response rates ranged from 45.8% in Norway to 84% for Sweden. Levels of missing data were low for all items across the surveys. Principal component analysis identified one component with six experiences items. Mean NORPEQ scores ranged from 74 to 79 on the 0-100 scale, where 100 represents the best possible experiences. Cronbach's α ranged from 0.84 in Finland to 0.88 in Sweden.
CONCLUSIONS: The NORPEQ is a brief measure of patient experiences that covers important aspects of the healthcare encounter. It shows good evidence of reliability and validity. PRACTICE IMPLICATIONS: The NORPEQ instrument is recommended for cross-national comparisons of healthcare experiences for the four Nordic countries.

Introduction

The literature relating to the development and testing of healthcare quality indicators that assess patient experiences and satisfaction is extensive.1 2 Such indicators have gained increasing importance following the work of international organisations, such as the Organisation for Economic Cooperation and Development and the WHO, which have emphasised the importance of the patient's perspective by capturing patients' experiences or satisfaction in the evaluation of the quality of healthcare delivery.2–4 Much of this work has been at the local level in relation to local providers, but national governments require comparisons of providers.1 2

All the Nordic countries have a history of measuring patient experiences and patient satisfaction. The patient groups surveyed, survey methodology and questionnaire content have however differed. In Finland, the National Institute for Health and Welfare (THL) questionnaire has been used in different surveys,5 but there has not been a national patient experiences survey. In Norway, national patient experience surveys have included a variety of patient groups, including somatic inpatients,6 outpatients,6–8 psychiatric inpatients and outpatients,7 and parents of paediatric patients.9 In Denmark, national patient experience surveys have been conducted for many years, both among psychiatric and somatic patients.2 In Sweden, smaller scale studies have been conducted with the Picker Patient Experience Questionnaire,10 and since 2009, the Swedish Association of Local Authorities and Regions in Sweden has conducted regular national surveys of users of primary care services and among somatic and psychiatric inpatients and outpatients. The Faroe Islands national patient experience surveys were conducted in 2007 and 2010. Iceland has conducted national surveys of healthcare providers in 2002 and 2005.2

The Nordic Council of Ministers initiated a cross-national collaboration to develop a set of quality indicators, including patient experiences, for measuring the quality of the health services across the Nordic countries. The work follows the Organisation for Economic Cooperation and Development's conceptual framework in which one of the core healthcare quality indicators is patient-centeredness.11 The similarities of the healthcare systems within the Nordic countries make them well suited for cross-national comparisons; care is virtually free at the point of use, governments use monetary incentives and various directives to influence the priorities of service producers and accountability for service provision is delegated to local authorities.12 During the past decade, the Nordic countries have increasingly presented information regarding the performance of healthcare providers. Public reports and internet sites such as the ‘Free Hospital Choice’ in Norway (http://www.frittsykehusvalg.no) and the ‘National Patient Questionnaire’ in Sweden (http://www.skl.se/nationellpatientenkat) have published results of surveys of patient experiences, the intentions being quality improvement and public accountability. Such information can also provide a basis for cross-national comparisons.2 13 Cross-national comparisons of patient experiences in the Nordic countries provide information about quality of healthcare, which may inform the Nordic citizens, patients, politicians and leaders and healthcare personnel thereby promoting a common understanding of factors relating to healthcare quality across the Nordic countries.

Although the CAHPS group has applied the CAPHS-questionnaire across Spanish- and English-speaking inpatients in the USA,14 previous cross-national studies of patient experiences have not demonstrated that the questionnaires perform in the same manner cross-nationally.13 This makes it difficult to determine to what extent any differences in patient evaluations are attributable to differences in healthcare quality or questionnaire performance across countries. Hence, the importance of adequate reporting of questionnaire development and survey methodology has been emphasised.2 11 15 It is important to account for cultural and demographic differences, health problems and potential translation problems in cross-national studies of patient experiences.2 Many healthcare quality indicators have been studied at the local level with local providers, but national governments require comparisons of providers.2 11

The NORPEQ was designed to include a core set of questions covering the most important aspects of inpatient experiences that can be used for cross-national comparison of inpatient experiences alongside existing longer-form national survey questionnaires.13 16 This article describes the cross-national questionnaire development based on surveys undertaken among hospital inpatients in Finland, Norway, Sweden and Faroe Islands. Following a rigorous process of questionnaire development including forward–backwards translation, the NORPEQ questionnaire was assessed for levels of missing data, reliability and validity in the four countries. The development of NORPEQ followed recommended criteria including forward–backwards translation necessary for a questionnaire that is to be used cross-nationally.2 17

Methods

NORPEQ development

The appropriateness of health quality indicators for benchmarking and other aspects of quality improvement is dependent on their reliability and validity,11 18 and criteria for their evaluation have been recommended.3 4 6 11 19–21 These criteria were applied in the development of the eight-item NORPEQ, which was translated into English by two professional translators (online appendix 1). Development followed a literature review of existing questionnaires and consultation with experts within the field of patient experiences in three face-to-face meetings. In the first of these meetings, three main considerations guiding the development of the questionnaire were decided. First, the questionnaire should include the most important aspects of patient experiences following a literature review relevant to patients across the Nordic countries. Second, the questionnaire should be brief so that the questions can supplement existing surveys. Third, the questionnaire should be developed in Norwegian and translated into the other Nordic languages using the forward–backwards methodology.13

The content of questionnaires used in surveys was assessed for appropriateness, and patient involvement in this process was designed to lend the NORPEQ content validity. Analysis of themes and items in existing NORPEQs revealed that the content of the questionnaires was fairly similar,13 16 but that there was some variation in question formulation and the choice of scaling. The analysis was based on the most widely used questionnaires in the Nordic countries,13 16 which included the national patient experience survey in Denmark,22 a 20-item measure of patient experience in Finland,5 a short form of the Quality from the Patient's Perspective questionnaire on Iceland,23 the Patient Experiences Questionnaire in Norway6 and the Picker survey in Sweden.10 Furthermore, the review found that patient experiences with health personnel including whether the doctors were understandable, doctors' and nurses' professional skills, nursing care, whether the doctors and nurses were interested in the patients problems and information related to tests are the most important aspects of patients experiences.13 16

Six of the eight NORPEQ items sum to produce an overall scale from 0 to 100, where 100 is the best possible experience of care. If respondents had missing values on more than half of the items, mean scores were imputed. The NORPEQ is designed to be routinely used alongside longer-form instruments in Nordic and international patient surveys.9 13 18 24

The NORPEQ was tested by means of cognitive interviews with six patients. A Norwegian pilot survey of 500 patients receiving inpatient care at a large University hospital found to have evidence for data quality, unidimensionality, internal and test–retest reliability and construct validity.13

To ensure valid comparison across countries, questionnaires must demonstrate cross-cultural equivalence.2 The NORPEQ questionnaire was developed in Norwegian for translation into the other Nordic languages using methods that adhere to minimum standards recommended for translation of patient questionnaires, including the forwards–backwards methodology.14 25 26 The questionnaire was translated into Danish, Finnish and Swedish by two forward translators fairly acquainted with the area and with some experience in health-related research. Emphasis was put on conceptual rather than literal translation. The backward translators were Norwegians who were not familiar with the original version. The Faroese version went through forward–backward translation from Danish to Faroese and back to Danish. This was because Faroese speak Danish, which is taught at school, and no one from the research group was able to translate from Norwegian to Faroese. Again, this included independent forward and backward translators to ensure that the Faroese questionnaire was conceptually similar to the Danish version. The Norwegian researchers assessed the forward–backwards translations following discussions with those responsible in the different countries, and it was agreed that the instructions and questions had retained their original meaning.

The approved translations were then tested by means of cognitive interviews with six patients in Finland (n=6) Sweden (n=11) and the Faroe Islands (n=27). Patients were asked whether they had omitted any questions, if any questions were difficult or too similar, if questions were acceptable and relevant and if they had any other problems with completion.

Data collection

In October and November 2009, 500 patients were randomly selected from adult inpatients at one University hospital in Finland were sent a questionnaire. Postal reminders were sent 3 weeks after the first questionnaire. Five hundred patients were randomly selected from adult inpatients at one Swedish University hospital within a 3-week period in February–March 2009. Two postal reminders were sent to non-respondents at 1 and 3 weeks. The Norwegian national survey included 24 141 patients randomly selected among patients discharged from 63 hospitals in a 3-month period in 2006. About a week later, a random sample of 270 respondents was asked to complete an identical second questionnaire for purposes of assessing test–retest reliability. The Faroe Islands has only 50 000 inhabitants and hence a large pilot of the NORPEQ was not possible. Instead the NORPEQ was applied alongside a national patient experience survey. In May 2010, 892 inpatients were discharged from three different hospitals on the Faroe Islands, and these patients were mailed a questionnaire: six patients were not eligible. One postal reminder was sent 3 weeks later to non-respondents.

Statistical analysis

Questionnaire evaluation followed recommendations relating to measures of patient satisfaction and questionnaire development more generally.19 20 Items were assessed for levels of missing data. Principal component analysis (PCA) with varimax rotation was used to assess the underlying dimensionality of the six items measuring patient experiences.27 28 Following previous findings,13 it was expected that these items would be unidimensional. Internal consistency was assessed using item-total correlation and Cronbach's α. The former measures the association between the item and the remainder of its scale, the latter determines the overall correlation between items within a scale.13 In the national Norwegian survey, test–retest reliability was assessed by the intraclass correlation coefficient.

Construct validity assesses the extent to which a questionnaire measures what it is intended and is assessed through comparisons with variables that following empirical and theoretical considerations have expected associations with patient experiences or satisfaction.20 Research including systematic reviews has found that patient experiences and satisfaction are associated with general satisfaction, perceptions of incorrect treatment,9 13 24 health status, health outcomes, fulfilment of expectations.1 6 13 18 24 29 We used responses to five additional items to assess construct validity of the NORPEQ scores including the two items relating to general satisfaction and incorrect treatment that have been widely used in Nordic and international research and items relating to general health,9 10 13 24 changes in general health and physical health compared with before admission, and fulfilment of expectations. These items all have 5-point descriptive scales. For an overview of stages in the development and evaluation of the NORPEQ instrument see Table 1.

Table 1

Stages in the development and evaluation of the NORPEQ

1	Item derivation	Literature review, Nordic surveys, expert consensus
2	Translation	Forwards–backwards
3	Patient interviews	Cognitive testing
4	Pilot survey	Data quality—missing data, response distribution
5	Main survey	Dimensionality, internal consistency, test–retest reliability, construct validity

NORPEQ, Nordic Patient Experiences Questionnaire.

Results

Only minor changes were made to the questionnaire on the basis of the forward–backward translations. The results of cognitive interviews showed that the eight items were acceptable, relevant and understandable to Norwegian and Swedish inpatients. One challenge related to the use of the term ‘health personnel’, which in Norwegian includes nurses and licensed practice nurses, whereas in other countries, nurses comprise other groups or they do not have a corresponding concept. While there were no problems for the other patient groups, some Finnish patients found the item about whether doctors and health personnel were interested in patient's problem difficult because it asked about two occupational groups. However, the majority of informants in Finland did not rate the question as difficult, and most found it relevant with appropriate response categories. Therefore, no changes to the NORPEQ were made following cognitive interviews in Finland, Norway and Sweden. Several patients from the Faroe Islands found the question relating to incorrect treatment difficult to understand, and hence, the question was reformulated.

Table 2 shows the response rate and respondent and non-respondent characteristics for each country. Of 496 eligible patients, 383 (77.2%) responded to the Finnish survey, and in the Swedish survey, 412 (84%) patients responded. Of the 24 141 patients included in the Norwegian national survey, 11 079 (45.8%) responded. For the Faroe Islands, 551 (62.2%) responded.

Table 2

Characteristic of respondents and non-respondents

	Distribution, N (%)	Age, mean (SD)	Gender, N (%)		Admission type, N (%)		Length of stay, mean (SD)
			Male	Female	Planned	Emergency
Finland
Respondents	383 (77.2)	59.13 (17.62)**	185 (48.30)	198 (51.70)	174 (45.40)	209 (54.60)	4.65 (4.04)
Non-respondents	113 (22.8)	50.13 (19.63)	66 (58.40)	47 (41.60)	49 (43.40)	64 (56.60)	5.28 (5.10)
All	496 (100)	57.07 (18.43)	251 (50.60)	245 (49.40)	223 (45.0)	273 (55.00)	4.78 (4.29)
Norway
Respondents	244 (48.8)	56.67 (19.42)**	120 (49.38)	123 (50.63)	109 (44.86)	134 (55.14)	3.54 (4.87)*
Non-respondents	256 (51.2)	55.85 (21.81)	133 (52.16)	122 (47.84)	95 (37.25)	160 (62.75)	4.84 (8.16)
All	500 (100)		253 (50.60)	245 (49)	204 (40.80)	294 (58.80)
National Norwegian
Respondents	11 079 (45.8)	60.37 (21.68)**	5187 (46.80)*	5892 (53.2)*	5398 (48.70)**	5635 (50.90)**	5.17 (7.37)**
Non-respondents	13 111 (54.2)	61.01 (17.02)	6329 (48.30)	6782 (51.7)	7382 (56.30)	5668 (43.20)	5.64 (9.81)
All	24 190 (100)	60.66 (19.68)	11 516 (47.60)	12 674 (52.4)	12 780 (53.00)	11 303 (46.80)	5.4 (8.79)
Sweden
Respondents	412 (83.7)	61.13 (18.29)**	206 (50.00)	206 (50.00)	187 (45.40)	225 (54.60)	4.16 (6.85)
Non-respondents	80 (16.3)	52.91 (22.09)	39 (48.75)	41 (51.25)	28 (35.00)	52 (65.00)	4.89 (8.20)
All	492† (100)	59.79 (19.18)	245 (49.79)	247 (50.20)	215(43.69)	277 (56.30)	4.28 (7.04)
The Faroe Islands
Respondents	551 (62.2)	55.96 (23.59)	290 (52.6)	261 (47.4)	350 (63.5)	201 (36.5)	5.73 (10.35)
Non-respondents	335 (37.8)	50.94 (28.65)	163 (487)	172 (51.3)	185 (56.4)	143 (43.6)	4.67 (8.87)
All	886 (100)	54.07 (25.72)	453 (51.12)	433 (48.9)	535 (60.9)	344 (38.8)	5.33 (9.83)

Statistical significance:*p<0.05; **p<0.01.

The questionnaire was sent to 500 adults, eight of these were not eligible and the gross population is 492.

In Finland, the respondents' mean age was 59.1 (SD=17.6) years and 51.7% were women. Respondents were on average approximately 9 years older than non-respondents, which was statistically significant. Respondents in Sweden were on average approximately 8 years older, which was statistically significant. In the Norwegian national survey, the respondents' mean age was 60.35 (SD 16.97) years and 53.2% were women (table 2). There were a significantly slightly greater proportion of female respondents. Compared with non-respondents, respondents were also significantly more likely to be admitted in an emergency and have 0.47 fewer days in hospital. For the Faroe Islands, the respondents' mean age was 57.0 (23.6) years and 47.7% were women.

Missing data for the six NORPEQ items ranged from 0.5% to 3.9% across the three smaller surveys in Finland, Norway and Sweden (table 2). For the Faroe Islands, the level of missing data varied between 6.7% and 10%. Incorrect treatment had the highest level of missing data for all surveys, the only exception was Norway where general satisfaction for the national survey had the highest level of missing data. Item means for the four surveys were generally skewed towards positive experiences of care ranging from 3.7 to 4.4 on the 1–5 scale. The mean ranking of items was very similar across the surveys. Patients had the poorest experiences on the two items information relating to tests or health personnel being interested. Patients had the best experiences with nursing care.

Results of PCA and measures of internal consistency are shown in table 3. The Finnish data gave two components, with eigenvalues of 3.38 and 1.02. One component was found for the Swedish, Norwegian national and for the Faroese data with eigenvalues of 3.82, 3.68 and 3.62, respectively. Given the relatively low value for the second component found for the Finnish data that just meets the criterion of 1.0, a single component that comprised all six items was accepted. The six items had high component loadings between 0.70 and 0.82 on this single component. The component loadings had a similar pattern across the different countries with the items ‘health personnel interested in health problem’ and ‘information on tests’ generally having the highest loadings and ‘nursing care’ and ‘doctors understandable’ generally having the lowest loadings. Item-total correlations for the six NORPEQ items were acceptable and ranged from 0.53 to 0.80. Cronbach's α for the six NORPEQ items ranged from 0.84 to 0.88 for Finland and the Faroe Islands, respectively.

Table 3

Component loadings and internal consistency of the NORPEQ

	Finland		Norway		National Norway		Sweden		Faroe
	Comp load	CA/item total	Comp load	CA/item total	Comp load	CA/item total	Comp load	CA/item total	Comp load	CA/item total
NORPEQ score		0.84	–	0.85		0.87	–	0.88		0.87
Doctors understandable	0.74	0.62	0.71	0.59	0.76	0.65	0.80	0.71	0.75	0.64
Doctors' professional skills	0.76	0.63	0.76	0.63	0.78	0.68	0.73	0.62	0.78	0.68
Nurses' professional skills	0.70	0.56	0.76	0.64	0.79	0.68	0.79	0.69	0.78	0.66
Nursing care	0.68	0.53	0.73	0.60	0.77	0.65	0.77	0.67	0.75	0.63
Health personnel interested in problem	0.82	0.71	0.81	0.71	0.83	0.74	0.87	0.80	0.81	0.72
Information on tests	0.80	0.69	0.78	0.66	0.76	0.65	0.80	0.70	0.79	0.69

NORPEQ, Nordic Patient Experiences Questionnaire.

Of 270 randomly selected Norwegian patients asked to take part in the test–retest survey in the national Norwegian survey, 196 (72.6%) returned a second questionnaire. The intraclass correlations ranged from 0.68 to 0.73 for the items relating to information on test and nurses' professional skills, respectively. The intraclass correlation for the NORPEQ scores was 0.85, which is considered acceptable for group comparisons.9 19

Table 4 shows that the NORPEQ scores for the four countries were skewed towards positive experiences and varied from 75.3 to 78.6 for Finland and Sweden, respectively. Approximately 10% of the patients in both of the Norwegian and the Swedish surveys scored 60 or below. In Finland, 15.7% of Finnish patients scored 58.3 or below, while on the Faroe Islands, 16.3% scored 58.3 or below. Scores for the six items were also similar with all three countries scoring lowest on the item relating to information on test and examinations. Finland, Sweden and the Faroe Islands had the highest scores on the item relating to nursing care, while Norway had very similar high scores for this item and two others. Scores for the satisfaction item were also generally high.

Table 4

Missing data, means (SD) for NORPEQ items and scale scores

	Finland			Norway*			National Norway			Sweden			Faroe
	n	Missing, n (%)	Mean (SD)	n	Missing, n (%)	Mean (SD)	N	Missing, n (%)	Mean (SD)	n	Missing, n (%)	Mean (SD)	n	Missing, n (%)	Mean (SD)
NORPEQ† score	382	1 (0.3)	75.3 (14.92)	243	1 (0.4)	74.37 (15.69)	10 849	230 (2.1)	77.03 (16.3)	400	12 (2.9)	78.57 (16.11)	514	37 (6.7)	75.45 (16.43)
Were the doctors understandable?‡	381	2 (0.5)	3.88 (0.84)	238	6 (2.5)	4.0 (0.80)	10 766	313 (2.8)	4.08 (0.84)	399	8 (1.9)	4.07 (0.81)	507	44 (8)	3.86 (0.91)
Did you have confidence in the doctors' professional skills?	379	4 (1.0)	4.21 (0.72)	239	5 (0.2)	4.13 (0.69)	10 745	334 (3.0)	4.2 (0.8)	400	7 (1.7)	4.31 (0.71)	507	44 (8)	4.09 (0.87)
Did you have confidence in the nurses' professional skills?	376	7 (1.8)	4.17 (0.62)	242	2 (0.8)	4.07 (0.66)	10 752	327 (2.9)	4.21 (0.74)	400	7 (1.7)	4.26 (0.68)	509	42 (7.6)	4.18 (0.72)
Did the nurses take care of you?	378	5 (1.3)	4.27 (0.72)	243	1 (0.4)	4.14 (0.81)	10 773	306 (2.8)	4.2 (0.78)	399	8 (1.9)	4.34 (0.78)	513	38 (6.9)	4.29 (0.76)
Were the health personnel interested in your problem(s)?	375	8 (2.1)	3.83 (0.86)	241	3 (1.2)	3.73 (0.92)	10 684	395 (3.5)	3.92 (0.90)	391	16 (3.9)	3.96 (0.92)	500	51 (9.3)	3.85 (0.85)
Did you receive sufficient information about tests and examinations?	380	3 (0.8)	3.71 (0.95)	241	3 (1.2)	3.8 (0.98)	10 699	380 (3.4)	3.82 (0.92)	399	8 (1.9)	3.93 (0.92)	497	54 (9.8)	3.86 (0.95)
Overall, was the treatment and care you received in the hospital satisfactory?	379	4 (1.0)	4.14 (0.82)	243	1 (0.4)	4.03 (0.86)	10 652	427 (3.8)	4.24 (0.76)	398	9 (2.2)	4.19 (0.77)	510	41 (7.4)	4.15 (0.79)
Was there a time you thought a medical mistake was made in your treatment and care?	368	15 (3.9)	1.39 (0.82)	238	6 (2.5)	1.44 (0.85)	10 558	521 (4.7)	1.39 (0.87)	387	25 (6.1)	1.53 (1.14)	496	55 (10)	1.33 (79)

These results were reported in a previous article.13

The NORPEQ total score is calculated by summing the six items to produce an overall scale of patient experiences from 0 to 100, where 100 is the best possible experience of care.

All items have 5-point descriptive scales of ‘not at all’, ‘to a small extent’, ‘to some extent’, ‘to a large extent’ and ‘to a very large extent’.

NORPEQ, Nordic Patient Experiences Questionnaire.

Table 5 shows the results of validity testing. The correlations across the different surveys are broadly consistent. High levels of correlations were found between the NORPEQ scores and general satisfaction in the range 0.72–0.77 and correlations with incorrect treatment were lower and in the range 0.24–0.39 for all surveys. For Finland, Norway and Sweden, correlations with expectations relating to treatment and care were moderate and in the range 0.51–0.58. Correlations with expectations relating to health outcome were lower and in the range 0.30–0.38. Lower levels of correlation were found for the health-related variables, for example, general health had low correlations with NORPEQ scores in Norway and Sweden, whereas in Finland, general health did not correlate with NORPEQ. The health-related variables were not available for the Faroe Islands.

Table 5

Correlation of NORPEQ scores with patient perceptions of satisfaction, incorrect treatment, expectations, health and outcomes

	Finland	Norway†	National Norway	Sweden	Faroe Islands
General satisfaction	0.77**	0.74**	0.72**	0.73**	0.73**
Incorrect treatment	−0.35**	−0.39**	−0.39**	−0.24**	−0.31**
Expectations: treatment and care	0.53**	0.51**	0.57** ‡	0.58**	–
Expectations: outcomes	0.30**	0.32**	0.37**	0.38**	–
General health	−0.09	−0.19**	−0.24**	−0.24**	–
General health compared with before admission	–	0.22*	0.27**	–	–
Physical health compared with before admission	–	0.27**	0.22**	–	–

Statistical significance:*p<0.05; ** p<0.01.

These results were reported in a previous article.13

This question has a slightly more general wording in the national Norwegian survey than in the other pilots.26

NORPEQ, Nordic Patient Experiences Questionnaire.

Discussion and conclusions

Discussion

The NORPEQ is a short self-completed questionnaire with evidence for data quality, reliability and validity. The NORPEQ includes what are judged to be the most important aspects of experiences for patients.1 Levels of missing data were very low across the eight items for Finland, Norway and Sweden. The results of PCA showed one uniform measure of patient experiences based on the six NORPEQ items, and general satisfaction and incorrect treatment are treated as supplementary items. Overall, this shows that NORPEQ meets the objective to develop a short questionnaire covering the most important aspects of patient experiences within the Nordic countries.

Furthermore, the NORPEQ shows evidence for construct validity in tests that were based on hypothesised associations with variables relating to general satisfaction, incorrect treatment,9 13 24 health status, health outcomes and fulfilment of expectations.1 6 18 24 29 However, although it was negative which followed the hypothesis, the correlation between general health and the NORPEQ for the Finnish data was close to zero. The future inclusion of more specific questions that relate to different aspects of health will help determine whether this finding has implications for cross-cultural validity.

The test–retest reliability of the NORPEQ has been found to be acceptable in two Norwegian surveys at the local and national levels.13 It would be an advantage to have test–retest data for the other countries as well. Previous studies have found comparable results for internal consistency and test–retest reliability for measures of patient experiences.7 24 Therefore, it is reasonable to assume similar levels of test–retest reliability for the other countries given the similarity of the results across the countries.

Compared with other surveys of patient experiences at a national level, the samples for the surveys were small and three of them were pilot studies. However, the different analyses still produced satisfactory results and items performed very similarly across the countries. Mean item scores, component loadings and item-total correlations were similar across countries. Results of validity testing were in line with those following national surveys that have used longer questionnaires comprising scales relating to different aspects of patient experiences and satisfaction.6 30 NORPEQ scores had similar significant levels of correlation with those for the additional items used in validity testing, the exception being general health for Finland. The strong associations between patient experiences as measured by NORPEQ and general satisfaction in each country confirm the importance of the NORPEQ items as a measure of patient experiences across countries.

The measurement of patient experiences is an important component of patients' evaluation of healthcare services.1 During the past decade, measures of patient experiences have gained increasing interest as healthcare quality indicators. Most studies of healthcare quality indicators have taken place at the local level, and few studies have conducted comparisons across countries.2 It is thus difficult to know whether variations in patient experiences reflect differences in culture, health systems or actual differences in quality as perceived by patients.31

Methodological concerns aside, existing research has implied that there is cross-national variation in patient experiences.32 The Commonwealth Fund and the WHO have conducted surveys of the general population32 33 and within primary care.34 Recipients of the survey results have included health ministers and decision-makers in each of the countries. Commonwealth Fund surveys include a large number of single items, and there has been no reporting of data quality, reliability and validity.2 The Picker Institute has compared patients' perceptions of the quality of acute hospital care as well as more focused surveys of patient experiences across Germany, Sweden, Switzerland, the UK and the USA.11 These studies have focused on results of surveys and have not described the translation processes and cross-cultural testing.2 The Consumer Assessment of Health Plan Surveys has published the results of one cross-cultural evaluation of its questionnaires,35 and it is translated for use in the Netherlands.36 However, Consumer Assessment of Health Plan Surveys questionnaires have not been used in international comparisons.2 Finally, one study used existing data for 12 European countries in comparisons of the quality of primary care.37 However, evidence for the cross-cultural equivalence of the included instruments was not given.

The present study is the first to report on the necessary evaluative work to support cross-national comparisons of patient experiences for several countries. One study included a cross-national comparison of public trust in healthcare in Germany, the Netherlands and England and Wales that has followed guidelines for questionnaire development including forward–backward translation and cross-cultural testing.38 In contrast to the similar ranking of patients' confidence in both nurses' and doctors' skills across the NORPEQ countries, this study revealed significant differences across the countries when comparing public trust in healthcare providers including doctors.37 However, patient experiences and public trust are different constructs, and the two surveys of these constructs were undertaken in different populations. The NORPEQ is the first questionnaire developed to compare general hospital inpatients' experiences across countries, which is the most studied and largest group of healthcare users.

Furthermore, patient experiences were assessed in four countries with highly similar healthcare delivery. This similarity may have contributed to the consensus in identifying important aspects of experiences for inclusion in the NORPEQ. The process of item development included experts from the different countries who subsequently monitored all phases of the project. The translation process followed recommended procedures relating to cross-cultural adaptation within health-related research.17 39

The study included two national surveys for Norway and the Faro Islands, but two surveys of just one University hospital for two other countries, Finland and Sweden, which limits the conclusions that can be drawn in terms of the appropriateness of the NORPEQ and cross-cultural equivalence. However, the national survey results for Norway and the Faroe Islands were very similar to those reported in an earlier study of the NORPEQ that included patients from one Norwegian University hospital using the similar survey design that was reported here for Finland and Sweden. Hence, it is reasonable to hypothesise that the NORPEQ will perform similarly in national surveys within these countries and in other Nordic countries following recommended translation procedures and cognitive testing. To further assess the NORPEQ for cross-cultural equivalence, evaluation following national surveys based on representative samples including the other Nordic countries should be conducted. Moreover, the inclusion of NORPEQ alongside existing instruments used in national surveys of inpatient experiences or other patient groups' experiences will further contribute to the cross-national evaluation of validity.

Practice implications

This study has demonstrated that it is possible to cover important aspects of the healthcare experiences in a short-form measure. Furthermore, the 8-item questionnaire has proved relatively easy to translate and evaluate across the Nordic countries. Through the inclusion of the NORPEQ in national surveys of Norway and the Faroe Islands, the current study has shown how the brevity of the questionnaire lends it feasibility for inclusion as part of existing national surveys. Hence, we consider the NORPEQ suitable as a supplement to future national healthcare surveys conducted in the Nordic countries that can facilitate cross-national comparisons to inform Nordic citizens, patients, politicians and leaders and personnel in healthcare about patient experiences as a measure of the quality of healthcare. This will promote a mutual understanding of quality in healthcare across the Nordic countries.

Conclusions

Although NORPEQ is brief, it measures important aspects of patient experiences directly related to the care patients receive in the hospital. Items typically used in longer questionnaires such as hospital access, hospital equipment or standards of patient facilities6 are not included. Still, results of testing for validity including the high level of correlation with responses to the general satisfaction item follow previous findings for longer questionnaires.1 24 The strength of NORPEQ is the focus on important aspects of healthcare summed to form a single score that is, supplemented with two items relating to patient perceptions of incorrect treatment and satisfaction. The NORPEQ is acceptable and feasible for cross-national comparisons of hospital care but should also be considered for applications alongside existing national surveys for the Nordic countries.30 Overall, the NORPEQ scores show that there is considerable scope for improvement in delivery of care in the Nordic countries.

To conclude, the NORPEQ has acceptable data quality, reliability and validity within the Faroese, Finnish, Norwegian and Swedish inpatients. The questionnaire has been translated into other Nordic languages, cognitive interviews have taken place in Denmark and further testing is planned in Iceland. Following further evaluation, the NORPEQ will be used in future national and cross-national surveys. The NORPEQ was developed and tested as part of a Nordic quality indicator project that aims to assess and compare health service performance across the Nordic countries. Such cross-national comparisons of health system performance offer greater accountability and transparency, support strategy review and development, and give potential for mutual learning.40