Patient Experience Monitor (PEM): The Development of New Short-Form Picker Experience Questionnaires for Hospital Patients with a Wide Range of Literacy Levels.

PURPOSE: Several patient-reported experience measures (PREMs) were developed through the years. These questionnaires are frequently found to be inappropriate for people with lower literacy levels. This paper describes the development of patient experience questionnaires for hospital patients with a wide range of literacy levels, while enabling the potential for quality improvement.
METHODS: Mixed methods were used to adapt Picker Institute patient experience questionnaires: selection of items and adaptation towards language level B1 (the language level of which patients can express their own opinion and describe experiences, events and expectations) by expert panels, usability tests with patients, analysis of psychometric properties and member checking. A theory-driven approach was followed for definitive enrolment of items, meaning that the items eligible for exclusion had been carefully reviewed by the expert team and representatives of a patient council before definitive exclusion.
RESULTS: A pilot study was performed in an University Medical Centre in the Netherlands among in- and outpatients after discharge. Two provisional questionnaires of 22 items, designed by an expert panel, were reduced towards a final selection of 14-15 items. This led to two short-form questionnaires, called Patient Experience Monitor (PEM) Adult Inpatient and PEM Adult Outpatient. To illustrate, the results of the PEM Adult Outpatient questionnaire are presented.
CONCLUSION: PEMs are short and valid questionnaires specifically developed to measure patient experiences of hospital patients with a wide range of literacy levels. Acceptance of the questionnaires for both lower and higher educated patients are confirmed by usability tests. The respondents of the pilot study represent both groups. The developed questionnaires should be seen as a dynamic entity and part of a continuous effort to evaluate and improve patient experiences. Future studies are needed to examine the usability of these new questionnaires for quality improvement.

Introduction

The concept of patient-centered care (PCC) is considered fundamental for high-quality health-care systems.1–4 The foundation of PCC lies in understanding and respecting individual patient values, preferences and expressed needs, which should be the basis of the clinical encounter and guide all clinical decisions.3,5–7 Patient experiences have become a key indicator to examine patient centeredness and quality of healthcare.8 To get insight in patient experiences, a number of tools can be used to capture the complexity of hospital care. Questionnaires are commonly used because of their capability to include a large number of patients in a standardized manner.9,10

Through the years, several generic and disease-specific patient-reported experience measures (PREMs) were developed. The choice of an instrument as such should be determined by a balanced consideration of different aspects of utility such as costs (a large standardized sample can be expensive), acceptability by their users (healthcare professionals and patients) and educational impact (can healthcare be improved with the results).10 Also, it should be kept in mind that characteristics of PREMs which may lead to an overall low response rate or non-response bias should be avoided. A frequently found characteristic of questionnaires is that these are less appropriate for people with lower literacy levels.11,12 The language used by healthcare professionals is usually too difficult to understand for this population, and this language often occurs in surveys as well.13 Furthermore, questionnaires often include a large number of questions while a more concise questionnaire has a better response rate.14–16 The result may be an instrument that due to the lack of comprehensibility, length, or low response rates, has a reduced educational impact.17,18

Therefore, short and simple questionnaires that capture different aspects of PCC are essential to evaluate patient experiences with hospital care.19–21 These questionnaires should preferably be developed with extensive involvement of users applying techniques such as interviews, focus groups, and usability tests (cognitive interviewing).22–24 Additionally, the questionnaires must be useful to guide quality improvement (QI) in clinical practice.25–27

In the search for validated patient experience questionnaires that met the aforementioned criteria, Picker Institute questionnaires were selected considering; 1) Picker Institute’s longstanding history and extensive research in developing and validating patient experiences questionnaires, 2) the underlying theory of 8 principles of PCC,21 3) extensive patient involvement in questionnaire development, and 4) permission to adapt the questionnaires.

The development of Patient Experience Monitor (PEM) contributes to patient experience measurements by integrating the various insights from existing questionnaires, the latest insights regarding literacy, the trend towards shorter questionnaires and the focus on QI. The aims of this study were twofold:

a) To adapt this questionnaire for people with limited health literacy by the use of simple language.

b) To shorten existing validated patient experience questionnaires for a better response rate.

This paper describes the process of adapting two existing patient experience questionnaires of Picker Institute for patients with a wide range of literacy levels in a hospital setting, while enabling the potential for QI in the Netherlands and internationally.

Methods

The following methods were used to construct the new questionnaires: expert panel, cognitive interviews, analysis of psychometric properties and member checking (Figure 1). Each method is described below.

Figure 1

Method of survey development.

Expert Panel

An expert panel of ten members was convened to select existing questionnaires and adapt these to shortened versions and to optimize comprehensibility for a broad range of literacy levels. The panel included the original translator, experts in healthcare, quality managers and staff advisors of various university medical centers, as well as experts with experience in questionnaire development and translation. Furthermore, a staff member of Pharos was involved. Pharos is a Dutch institute that specializes in decreasing healthcare disparities related to language, education level, health skills and cultural background of patients (). Given that the original Picker questionnaires consist of 67 to 87 items, the expert panel aimed to reduce the questionnaires to a core set of items that represent the eight key domains of PCC as identified by Picker Institute.21

Cognitive Interviews

Provisional questionnaires, the version compiled by a first reduction of the expert panel, were tested by means of cognitive interviews on a purposive sample (N=28). Participants were informed about the aims of the study and received written information about participating in medical scientific research prior to the start of the study by the principal investigator (CB). All participants (and parents or legal guardians if under 18 years of age) provided written informed consent prior to the interview. The Three-Step Test-Interview procedure (TSTI) was used.28 The TSTI is an observation-based method for pretesting self-completion questionnaires in three steps; 1) Observation of response behavior and concurrent think aloud verbalization, 2) Follow-up interview for clarifying and completing the interviewers interpretation, and 3) Eliciting experiences and opinions of the interviewees. Relevance of the questions and answers was evaluated, as well as the method of online collection. All participants verified the findings and the suggestions of their own interview as part of a member checking process. The findings and suggestions were completely anonymized and documented by the interviewer.

Data Analysis of Psychometric Properties

In order to test whether the number of items of the provisional questionnaires had to be further reduced, a pilot study was performed in a University Medical Centre in the Netherlands between January and April 2019 at departments that were enthusiastic about the study and were willing to participate. All in- and outpatients of 16 years and older who visited or were hospitalized at the neurology, surgery or ear-nose-throat (ENT) department received within two weeks after discharge by email an invitation with a link to complete one of the provisional questionnaires in an online environment. Patients who were not willing to participate could unsubscribe via a link in the received e-mail. A reminder was sent to non-responders two weeks after the initial invitation. Questionnaires were included for analysis only if at least 50% of multiple-choice items were completed. Questionnaire items were eligible for exclusion if they fulfilled one of the following criteria:

Item response: more than 10% missing values, with exemption of routing questions (eg, “if yes go to question …”);

Room for improvement: items with more than 90% of responses in the same extreme category (ie, floor-ceiling effect);

Item relevance: items with more than 40% of response “not applicable”.

Since the questionnaires were developed with the aim of QI, a theory-driven approach was followed for definitive enrolment of items instead of a data driven approach.29,30 A data driven approach, such as performing Rasch analyses or principal component analyses followed by reliability analyses for item reduction, better suits questionnaires with many questions per domain. This is not applicable here, as there are only a few questions for each domain, and we want to maintain all eight domains. The theory-driven approach means, that the items eligible for exclusion had been carefully reviewed by the expert team and representatives of a patient council before definitive exclusion (member checking). For example, the item was not excluded when it represented an essential component of PCC according to Picker institutes principles or was considered to be of great importance for QI, despite a floor-ceiling effect or low relevance. Of the final selection of items, Spearman correlation coefficients were calculated, correlations >0.70 were flagged as an indication that items yield overlapping information.

Member Checking

Member checking is traditionally a technique in qualitative research used to establish the tenet of credibility of data.31 The results have to be returned to participants to check for accuracy and resonance with their contribution. The final questionnaires were presented to all those involved in the adaptation process (quality managers, staff advisors, healthcare professionals and patients) for evaluation, discussion and approval.

Ethical Approval

Ethical approval for the study was given by the Institutional Review Board Erasmus MC Rotterdam in the Netherlands, case number MEC-2018-1714.

Results

PEM Adult Outpatient: Expert Panel

To illustrate, the results of the adaptation process of PEM Adult Outpatient are presented. The process and results of the PEM Adult Inpatient followed a similar path (, ). As required by Picker Institute, their Survey Adaptation Guide was accurately followed.32 This implies that every step in the adaptation process that generates a reduction or adjustment of items was presented to Picker Institute for discussion and approval. The questionnaire was first translated from English into Dutch according to the forward-backward procedure. Based on the results of previously used patient experience questionnaires of various university medical centers in the Netherlands, our knowledge on what patients value in healthcare4 and the suitability of items for QI, the expert panel reached consensus on 22 eligible items out of 87 items of Picker Adult Outpatient (Figure 2). An even representation of Picker Institutes 8 principles of PCC (2–5 items per principle) was taken into account. The pilot study was used to investigate whether the selection should be further reduced. A staff member of Pharos edited the selected items to language level B1, the language level of which patients can express their own opinion and describe experiences, events and expectations ().

Figure 2

Flowchart of excluded questions.

PEM Adult Outpatient: Cognitive Interviews

This provisional questionnaire of 22 items was cognitively tested on a purposive sample of 28 discharged patients for comprehensibility and relevance of the items from a patients point of view. These patients were recruited in a primary care center by their GP or physiotherapist 6 weeks after hospital discharge or visiting an outpatient clinic at the latest. This sample consisted of 13 men and 15 women, of whom eight were aged 16–25, fourteen aged 25–60 and six of them were 60 years and older. Sixteen had a lower education level (≤ lower secondary education) and 12 of them had a higher education level (≥ upper secondary education). On the basis of the interviews, adjustments were made to the text of the introduction and questions and one item (Q22) was removed due to multiple interpretations. Two patients with a higher level of education criticized the simplicity of the language used, but acknowledged that the language chosen was to give priority reaching lower literate people. The vast majority, 26 of the 28 interviewees, did not comment on the simplicity of language used.

PEM Adult Outpatient: Data Analysis of Psychometric Properties

The cognitively tested and modified outpatient questionnaire of 21 items was sent within two weeks after consultation to all neurology, surgery and ear-nose-throat (ENT) outpatients who registered their email and had given permission to use email for communication (N=6806, which is 58.7% of the visiting patients). Of these, 53.3% were men and 46.7% women. Patients who were not willing to participate could unsubscribe via a link in the received e-mail, or simply not respond. The final response rate for this survey was 36.8%. In Table 1 gender, age and education level of the respondents are shown. Of the 2506 respondents, 54.4% were men and 45.6% were women. Regarding educational level, 21.3% of all respondents had a low education level (early childhood, primary and lower secondary education), 35.3% an intermediate education level (upper secondary education) while 43.6% had a high education level (equivalent to tertiary education and Bachelor’s, Master’s or Doctoral level). Of all these respondents, 2384 completed more than 50% of the items and were included for further analysis. Missing values ranged from 0.3% to 5.4% with the exception of the question Q4 “Could other people hear what you were saying to the person at the reception?”, where 17.4% of the answers were missing (Table 2). Seven items had a ceiling effect, items with a floor effect were not found. Four items of the questionnaire were of low relevance for a larger group of patients (>40% category response “not applicable”).

Table 1

Characteristics of Respondents

Sample Description	PEM Adult Outpatients
	N=2506	Valid %
Gender
Male	1253	54.4
Female	1050	45.6
Age
16–20	24	1.0
21–30	94	4.1
31–40	153	6.6
41–50	263	11.4
51–60	550	23.9
61–70	726	31.5
> 70	492	21.4
Educational level
Early childhood education	35	1.6
Primary school	61	2.8
Lower secondary education	379	16.9
Upper secondary education	790	35.3
Tertiary education	230	10.3
Bachelor’s/Master’s or equivalent level	530	23.7
Doctoral or equivalent level	212	9.6

Table 2

Statistical Characteristics of Items

Picker Principle	Questions Selected by Expert Team Adult Outpatient	Ceiling >90% Positive Answers	Floor >90% Negative Answers	Missing >10%	Do Not Know >10%	NA >40%
FA, ESES, CI, FAESESCICC, ETCI, IR, ESETIR, CI, ESETAN, ETCI, CCESIRIF, IR, CCIR, CICC, CICC, CICC, CIAN, ETES	Q1Did you have to wait at the clinic to see the nurse/doctor or was they on time?**Q2Were you told how long you would have to wait? (“go to”)**Q3Were the staff at the Outpatients Clinic reception friendly?**Q4Could other people hear what you were saying to the person on reception?Q5Did the staff at the Outpatients Clinic tell you what was going to happen to you? (“go to”)**Q6Did the doctors or nurses know what was written in your medical records?**Q7If you had a question for the doctor did you understand what they told you?**Q8Did you trust the doctors? (“go to”)**Q9If you had a question for one of the other members of staff at the Outpatients Clinic, did you understand what they told you?Q10Did you trust the other members of staff at the Outpatients Clinic? (“go to”)Q11Did you have enough time to talk to the doctor or other members of staff about your illness or problem?**Q12Sometimes at the Outpatient Clinic, members of staff say different things which may confuse you. Did this happen to you?**Q13Were you able to talk to anyone at the Outpatients Clinic about any worries or fears you had?Q14Were you involved in the decisions about your treatment?**Q15Were your family or someone close to you involved in the decisions about your treatment?**Q16Did anyone at the Outpatients Clinic explain to you the pros and cons of the treatment?**Q17Did a member of staff tell to you about possible side effects that you could have from any new medication?**Q18Did anyone at the Outpatients Clinic explain to you about any problems you needed to be aware of when you got home?Q19Did anyone speak to you about after care? For example: Should you make a new appointment or should you see your GP?**Q20How clean was the Outpatients Clinic?Q21Do you feel that the members of staff at the Outpatients Clinic treated you with respect?	42.822.085.142.480.992.0*91.6*91.3*89.887.193.1*79.576.988.193.4*84.467.477.392.5*98.4*92.8*	3.360.11.18.614.51.00.61.20.51.31.83.412.12.62.31.315.412.52.10.00.3	0.00.0.0.217.4*1.50.40.22.30.420.42.32.81.91.92.32.62.82.62.52.32.8	0.81.20.39.63.11.2n/a0.10.29.6n/an/an/a1.7n/an/an/an/a1.21.6n/a	n/a72.8n/an/an/a0.55.4n/a45.3*n/an/an/a56.9*24.634.923.254.9*46.0*23.2n/an/a

Abbreviations: FA, fast access to reliable healthcare; ET, effective treatment and trusted professionals; CC, continuity of care and smooth transitions; IR, involvement in decisions and respect for preferences, needs and values; CI, clear information and support for self-care; IF, involvement of and support for family and friends; ES, emotional support, empathy and respect; AN, attention for physical and environmental needs; NA, not applicable (it was not necessary/I do not need to/I did not have to/I did not spoke to); n/a, not applicable, go to, routing question; *, item eligible for exclusion; **, final selected item.

These results have been submitted to the expert panel who, after careful consideration, made a final selection of 14 items (Figure 2). This selection was based on the statistical characteristics of items (Table 2), an even representation of the 8 principles of PCC, and the advice of users (healthcare professionals and patients). To illustrate, the item Q7 “If you had a question for the doctor did you understand what they told you?” was eligible for exclusion based on a ceiling score of 91.6%. However, the client council strongly advised to include this item since comprehensibility of healthcare was considered a key prerequisite for patient-centered care. They considered understandable information as crucial for the assessment of quality of care in the long term. Item Q15 “Were your family or someone close to you involved in the decisions about your treatment?” was included despite of a 93.4% ceiling score, as it was a serious issue for the expert panel and was considered to be the best representation of the Picker principle “family involvement”. A final illustration of how the qualitative weighting of items influenced the final selection was the inclusion of item Q17 “Did a member of staff tell to you about possible side effects that you could have from any new medication?”. Even though 54.9% of respondents indicated they had not received new medication, which made the item eligible for exclusion, it was decided to include. Medication is an important topic considering the substantial risk of adverse events through incorrect use of medication. There is considerable room for improvement, only 67.4% of respondents received adequate information about side-effects and 15.4% of the respondents did not receive any information about side-effects at all.

As a final consideration, the Spearman’s inter-item correlation coefficients of the final selection of items were calculated (Table 3). The correlations above the threshold of 0.7 were flagged. This was the case for “Q14 Were you involved in the decisions about your treatment?” and “Q15 Was your family or someone close to you involved in the decisions about your treatment?” with a correlation of 0.708. Despite the strong relation between these items, the expert panel decided not to remove either of the questions due to their various meaning; 1) involvement in decisions and 2) family involvement.

Table 3

Spearman Correlation Coefficients Between Final Selected Items PEM Adult Outpatient

	Q1	Q2	Q3	Q5	Q6	Q7	Q8	Q11	Q12	Q14	Q15	Q16	Q17	Q19
		0.104	0.201	0.169	0.111	0.073	0.123	0.085	0.096	0.097	0.099	0.114	0.113	0.052
Q1	2364		0.165	0.156	0.070	0.075	0.055	0.085	0.101	0.118	0.101	0.115	0.083	0.119
Q2	612	619		0.205	0.235	0.180	0.230	0.232	0.214	0.230	0.207	0.224	0.225	0.195
Q3	1950	497	1967		0.195	0.137	0.173	0.178	0.096	0.247	0.186	0.268	0.237	0.163
Q5	2259	588	1885	2276		0.359	0.386	0.357	0.304	0.416	0.339	0.407	0.223	0.273
Q6	2317	611	1931	2230	2335		0.330	0.315	0.276	0.370	0.249	0.374	0.249	0.225
Q7	2233	582	1868	2154	2217	2251		0.344	0.295	0.403	0.326	0.371	0.294	0.252
Q8	2305	603	1923	2218	2288	2241	2325		0.287	0.392	0.338	0.395	0.297	0.197
Q11	2309	604	1924	2228	2284	2202	2274	2329		0.325	0.258	0.321	0.238	0.216
Q12	2297	602	1918	2216	2272	2187	2260	2293	2317		0.708	0.527	0.369	0.355
Q14	1699	447	1448	1655	1688	1647	1677	1676	1675	1712		0.358	0.237	0.310
Q15	1489	393	1266	1434	1478	1445	1469	1472	1462	1264	1499		0.463	0.294
Q16	1757	463	1502	1696	1739	1699	1735	1732	1735	1479	1310	1770		0.290
Q17	1003	290	876	970	998	977	990	989	994	874	781	951	1009
Q19	1727	448	1484	1684	1712	1674	1704	1710	1700	1356	1207	1425	871	1742

Notes: Correlations are in the right-upper triangle, numbers in the left-lower triangle. Correlations > 0.7 are shaded in red.

PEM Adult Outpatient: Member Checking

The final selection of 14 questions has been evaluated by healthcare professionals and patients (former interviewees) for discussion and were approved. It was discussed whether the final version met the predetermined criteria such as inclusion of each of the 8 principles of PCC, language level B1, a limited length of the questionnaire, the potential to improve quality and whether all advices for adjustments of stakeholders were processed satisfactorily. The final selection was translated back into English and approved by Picker Institute.

Discussion

Patient Experience Monitors (PEMs) were specifically developed for hospital patients with a wide range of literacy levels, while enabling the potential for QI. We want to give the largest possible number of patients the opportunity to share their experiences. A concise questionnaire with accessible language is an important first step. We described the process of adaptation so that colleagues outside the Netherlands could do likewise in their own language.

We aimed to develop questionnaires appropriate for patients with a wide range of literacy levels. Population statistics on educational levels in the Netherlands show that 30.4% has a low education.33 Although education and literacy are different entities, there is a strong positive relation between them.34 As such, we reached both lower and higher educated patients, however we should also acknowledge that lower educated patients (21.3%) are still substantially under-represented. Other studies also found that lower educated people represent the largest group of survey non-respondents.35 However, we also expect this non-response to be related to the method of administration (e-mail). Although the method of administration in the cognitive interview did not cause a problem for lower educated participants, studies on the relationship between health literacy and the use of health information technology shows that lower educated people have a lack of information technology skills.36 Notably, lower health literate patients are less likely to use information technology which is positively associated with trust in health care.37 Accordingly, we will further investigate possible response bias to facilitate extra participation of those with a lower education level.

The provisional selection of approximately 21 items was reduced to 14 items by evaluating relevance, ceiling effects or missing answers in the first place. Also, an expert panel followed a theory-driven approach for the definitive enrolment of items. Methodologically, a common approach is to administer a questionnaire and select items using principle component analysis (PCA) and item response theory (IRT).29,30 With data from Picker Institute original questionnaires we probably could have more easily investigated by Principle Component Analysis or Item Response Theory which item per Picker principle is best to select. However, we did not want to burden patients unnecessarily with the initially long questionnaires of 67 to 87 items. We also expected that these long questionnaires would affect the response rate negatively. Thereby, selecting entirely at statistics obtains items which can be improved on in theory, but may be of little importance for clinical practice (patients or healthcare providers). In practice, a poorly scoring item in statistics can be a key condition for a good experience.

A selected item about privacy was “Could other people hear what you were saying to the person at reception?” This item was removed for final selection based on 17.4% missing values. This could be explained by the fact that the pilot hospital had recently switched to new self-registration desks. This and the aforementioned examples indicate that the location of the pilot study determines which items ultimately prove relevant and that the choice of conducting the pilot at merely three departments of one hospital influences the final results. We also acknowledge that some patients of neurology, surgery and ear-nose-throat (ENT) departments could not fill in the questionnaires due to cognitive problems or other severe diseases which is, among other unit non-response factors, a known problem of PREMS.38 Furthermore, the questionnaires were developed with the input of quality advisors from several university medical centers, the selected questions cover all 8 principles of PCC and represent the patients journey of care, with specific attention for QI. Future studies are required to examine whether the questionnaires are suitable for QI.

Conclusion

Based on Picker Institute Questionnaires, two short-form questionnaires were designed, called Patient Experience Monitor (PEM): PEM Adult Inpatient and PEM Adult Outpatient. PEMs are short and valid questionnaires specifically developed to measure patient experiences of hospital patients with a wide range of literacy levels. Acceptance of the questionnaires for both lower and higher educated patients are confirmed by usability tests. The respondents of the pilot study represent both groups. To enable the potential for quality improvement, the developed questionnaires should not be seen as static, but as a dynamic entity and part of a continuous effort to evaluate and improve patient experiences. The set of questions are constantly liable to changes in healthcare and patient expectations. Annual analysis of survey results with respect to (new) needs of users should lead to improvement of the questionnaire by going over the same cycle (Figure 1). Future studies are needed to examine the usability of these new questionnaires for literacy levels and quality improvement.