Older, vulnerable patient view: a pilot and feasibility study of the patient measure of safety (PMOS) with patients in Australia.

OBJECTIVES: The UK-developed patient measure of safety (PMOS) is a validated tool which captures patient perceptions of safety in hospitals. We aimed (1) to investigate the extent to which the PMOS is appropriate for use with stroke, acute myocardial infarction (AMI) and hip fracture patients in Australian hospitals and (2) to pilot the PMOS for use in a large-scale, national study 'Deepening our Understanding of Quality in Australia' (DUQuA). PARTICIPANTS: Stroke, AMI and hip fracture patients (n=34) receiving care in 3 wards in 1 large hospital.
METHODS: 2 phases were conducted. First, a 'think aloud' study was used to determine the validity of PMOS with this population in an international setting, and to make amendments based on patient feedback. The second phase tested the revised measure to establish the internal consistency reliability of the revised subscales, and piloted the recruitment and administration processes to ensure feasibility of the PMOS for use in DUQuA.
RESULTS: Of the 43 questions in the PMOS, 13 (30%) were amended based on issues patients highlighted for improvement in phase 1. In phase 2, a total of 34 patients were approached and 29 included, with a mean age of 71.3 years (SD=16.39). Internal consistency reliability was established using interitem correlation and Cronbach's α for all but 1 subscale. The most and least favourably rated aspects of safety differed between the 3 wards. A study log was categorised into 10 key feasibility factors, including liaising with wards to understand operational procedures and identify patterns of patient discharge.
CONCLUSIONS: Capturing patient perceptions of care is crucial in improving patient safety. The revised PMOS is appropriate for use with vulnerable older adult groups. The findings from this study have informed key decisions made for the deployment of this measure as part of the DUQuA study. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

This was the first time the patient measure of safety (PMOS) has been used in Australia.

The sample size was targeted and modest, and limited to one site. Nonetheless, the participants were representative of the population that will be surveyed in the forthcoming national Deepening our Understanding of Quality in Australia (DUQuA) study.

Researchers partnered with hospital leaders and healthcare professionals to implement participant recruitment and data collection processes that can be replicated.

This study investigated feasibility of data collection with vulnerable older adult patients, which is of importance given the expected growth in older persons globally.

Background

Over the past two decades, the patient safety movement has urged hospitals to improve the safety of their practices.1–4 Although patients are frequently surveyed on the satisfaction of their experiences in hospital, these surveys do not capture information that can be used to improve safety of care.5 Patients’ own perceptions of the care provided to them is an overlooked yet well-placed resource for understanding the factors responsible for safety of care.6 Patients have demonstrated a willingness and capability to provide valuable feedback on both the safety of care delivered to them, and of the care environment (eg, delays to treatment, faulty equipment, miscommunication)—factors that busy healthcare professionals may not observe or report.5 7 With the right tools, patients can help hospitals to proactively identify safety issues, in order to intervene and prevent incidents from occurring.6

In recognition of this, a group of healthcare quality and safety experts in the UK developed the patient measure of safety (PMOS).5 6 The PMOS is the first valid and reliable tool that asks patients to comment on the factors that influence patient safety, according to a validated contributory factors framework of patient safety.8 It has been designed such that hospitals and wards can use it as a diagnostic guide for the design of patient safety improvement interventions by front-line staff.

While a measure to gain information about patient perceptions of safety can be useful, identifying whether patient perceptions of safety are associated with (1) how hospitals set themselves up for quality and safety, (2) the way care is delivered and (3) health outcomes for patients, is yet to be established. Understanding the extent of these relationships can assist policymakers and hospital leaders in making decisions about the target areas to invest resources to improve the quality and safety of healthcare.9 A large national study, ‘Deepening our Understanding of Quality in Australia’ (DUQuA;9 a modified replication of a European study of 188 hospitals (DUQuE)10), aims to explore the relationships between organisation and department-level quality management systems and patient factors in 60 large public hospitals across Australia.9 One of the factors that will be measured in DUQuA is patient-reported perceptions of safety, and for this the PMOS will be used.[i]

The patient conditions selected for the DUQuA study are stroke, acute myocardial infarction (AMI) and hip fracture. Thirty patients within each of these condition groups will be surveyed in each hospital (estimated total of 5400 patient participants) using the PMOS. However, to date, the PMOS has neither been tested in the Australian setting, nor with a principally vulnerable older adult population (the mean (SD) age of the original validation sample was 54 (18.13) years). Since stroke, AMI and hip fracture predominantly affect older adults in Australia,12–14 they are a particularly vulnerable sample due to the prevalence of condition-related complications or age-related comorbidities.15–19 As such, it is highly probable that these patients may experience pain or fatigue, and may suffer cognitive, hearing or visual impairments; factors that must be considered when planning recruitment, consent and data collection, and which can severely impact the quality and reliability of data.20–22 Therefore, it is important to identify and improve the suitability of the PMOS for use with this population prior to the full deployment of the measure. Finding strategies to overcome the challenges of undertaking research with hospitalised older adults23 is of significance due to an ageing population, and expected global growth of older persons—a factor associated with an increase in health expenditure.24

It is also crucial to ensure that the processes for conducting research with in-hospital AMI, stroke or hip fracture patients are appropriate, risk-free, consistent and effective. The large-scale nature of the national DUQuA study necessitates piloting the PMOS to explore ways to establish efficient, rigorous, pragmatic and ethical processes that can be replicated by the data collectors who will administer the measure in 60 Australian hospitals.9

Objectives

The objective of this study was to investigate whether the PMOS is appropriate to be used for stroke, AMI and hip fracture patients in Australian hospitals in order to ensure the feasibility of using this measure in the DUQuA study. The study had two phases. Phase 1, ‘think aloud’, aimed to test the face validity of the PMOS with the intended population for DUQuA (ie, Australian hospitalised older and vulnerable adults), and identify if amending the PMOS would improve understanding by, and relevance for, this population. Phase 2 aimed to test the revised measure to establish the internal consistency reliability of the revised subscales, and to pilot the PMOS to identify key areas within the recruitment and administration processes that could inform and enhance the feasibility of the DUQuA study.

Methods

Hospital engagement

The research team contacted the quality manager (QM) at a large metropolitan public hospital that has also enrolled in the DUQuA study. The research team advised the QM of two benefits for participating as the PMOS pilot site: provision of a report on patients’ perceptions of safety across the three wards, and that the experience would familiarise the hospital with data collection processes prior to the DUQuA study. The QM agreed that the hospital would participate as the pilot site for this study.

Participants

Sample type and size

Given that (1) the first five participants should usually reveal around 85% of the problems available for discovery in that iteration of an instrument25 26 and (2) an extensive amount of ‘think aloud’ research was undertaken on the originally developed PMOS, five patients with stroke were approached to participate in the ‘think aloud’ (phase 1). Stroke was selected as the most vulnerable cohort of the three conditions, due to the sensorimotor, communication and cognitive complications (eg, speech or hand impairments) faced by hospitalised patients with stroke.27 Recruitment continued until there was evidence of data saturation, consistent with other ‘think aloud’ studies.28 29

For phase 2, guidance was taken from the literature on appropriate sample sizes for pilot studies testing a new scale (n=30).30 As this was a pilot study, it was deemed appropriate to collect 10 PMOS questionnaires per ward, giving a total of 30 participants.

Eligibility

Detailed criteria were required to ensure the exclusion of patients who lacked the cognitive ability to give informed consent or who would be overly burdened by participation. Inclusion and exclusion criteria for both study phases are presented in table 1.

Table 1

Participant inclusion and exclusion criteria

Inclusion criteria	Exclusion criteria
Experienced the care of stroke, cardiac or orthogeriatric wards as a stroke, AMI or hip fracture patient* Deemed well enough by a ward nurse or doctor for the researcher to approach Provided with notification, verbally or in writing, of discharge Willing to give written informed consent Willing to participate in the study	No capacity to give informed consent (eg, suffering from dementia, delirium or confusion) Suffering from physical or emotional distress (eg, in pain, having recently received bad news) Unable to communicate in English (unless a carer or family member would act as an interpreter) Under the age of 18 Currently enrolled in a clinical trial

*Refer to the full DUQuA study protocol9 for primary diagnosis admission codes for each condition.

AMI, acute myocardial infarction; DUQuA, Deepening our Understanding of Quality in Australia.

Measure

The PMOS items were developed based on 33 interviews with inpatients from six units in one UK teaching hospital about their perceptions of patient safety.5 The face validity of the measure was established via 12 patients and 12 healthcare professionals. Following this, 297 patients from 11 hospital wards completed the PMOS questionnaire. Factor analysis revealed nine key domains of safety6 indicated in table 2, which align with a validated framework of contributory factors to patient safety.8 Items on the PMOS are phrased both positively and negatively, and scored using a five-point Likert scale (strongly disagree to strongly agree). Following reverse scoring for negative items, higher total scores represent more favourable responses.

Table 2

Patient measure of safety subscales and example items

Subscale (n items)	Example item
Communication and team work (10)	I always felt staff listened to me about my concerns
Organisation and care planning (5)	Staff gave me different information about my care
Access to resources (4)	Staff seemed to struggle to get help when they needed it
Ward type and layout (11)	Staff were prompt in answering my buzzer
Information flow (3)	After a shift change staff appeared to know important information about my care
Roles and responsibilities (4)	It wasn't clear to me who was in charge of my care
Staff training (2)	On at least one occasion a member of staff was not able to use the necessary equipment
Equipment design and functioning (2)	Equipment needed for my care was always working properly
Delays (2)	My treatment/procedure/operation always happened on time

Data collection

Identifying and approaching patients

The research lead (NT) was introduced to the nurse unit managers (NUMs) for each ward by the QM. The research lead liaised with each NUM to determine a communication process to identify patients efficiently, causing as little disruption to normal working practice as possible. Over a 3-month period, a member of the research team called the wards on 1–2 mornings per week to determine if any eligible patients were due to be discharged from hospital. In the orthogeriatric ward, the research lead was provided with a list of highlighted patient names and bed numbers. In the stroke and cardiac wards, the NUM informed the research lead of the patient name and bed number.

Once identified, the research lead approached the patient with a standard introduction (box 1), which was adjusted (eg, through language simplification, emphasising word pronunciation, reducing the speed of word flow or repetition) based on individual patient responses or needs.

▸ Hi, are you Mrs X? My name is Natalie. I have been told by the nurse in charge that you are due to be discharged soon—is that correct? (Patient response…)

▸ I am doing some research with Macquarie University to help improve this hospital and wondered if you would be willing to take part in a short patient questionnaire? (Patient response…)

▸ It is just to try and understand a bit more about your experience of care and safety since you have been on this [AMI/stroke/hip fracture] ward. It takes about 15 minutes and is confidential. (Patient response…)

▸ I have an information sheet for you to read and if you agree to take part there is just a consent form to sign. It is up to you whether you would like to complete the questionnaire yourself, have me read the questions out to you and you point to the answers, or a member of your family can help you to complete it too. (Patient response…)

Consent and screening

From the initial approach to finalising questionnaire administration, the research lead screened the consenting patients using a list of cues developed by Harris and Dyson22 (box 2). If the patient at any time demonstrated a lack of understanding of consent as per these cues, the patient was excluded from the study, regardless of the provision of consent.

Was the patient able to concentrate on the brief introduction?

Did they appear to listen and engage in what the researcher was saying?

Did they respond appropriately? Did they ask appropriate questions?

Were there any signs of inconsistency in what they said?

Was there a lot of repetition in what the patient was asking?

Was the conversation appropriate or did the patient talk at length, constantly going off at tangents?

Were there other agendas and preoccupations that were far more important to the patient?

If the patient did not appear to understand, how did they deal with this? Did they deny not understanding? Or make appropriate suggestions, for example, using their next of kin to help them?

Did they appreciate the consequences of consenting or not consenting?

Did they appear to understand that there was a need for a decision?

Considerations for using the PMOS with vulnerable older adult patients

The PMOS could be completed either by the patient, or with assistance from a researcher.6 The research team anticipated that in implementing this measure with this group of participants, it would be likely that they would request assistance.31 The research lead used evidence-based techniques for interacting with older adult participants (eg, sit facing the patient, ensure lips are visible to the patient, check patients have their hearing aids or glasses if necessary, speak slowly and clearly, use visuals aids and large fonts).32 To assist patients with visual, speech or physical impairments,31 and to protect privacy of participants who may be concerned about others overhearing their answers on the ward,33 an enlarged Likert scale was designed for patients to point to as a way of indicating answers. The research lead also offered to read questions aloud, and at a slower pace, for patients with visual or hearing impairments, or both. ‘Thank you’ cards were provided for all patients who participated in the study.34

Phase 1—‘think aloud’

The ‘think aloud’ method has previously been used to assess validity and feasibility of health and social care measures for use with older adults,28 and was also used to assess face validity during the development of the original PMOS.5 During the ‘think aloud’ phase, the research lead sat with patients as they worked through the PMOS and prompted them to identify and elaborate on aspects of the questionnaire that were difficult to understand or presented cultural differences in terminology. Patients were asked to consider and discuss the following features with the research lead, who made notes during the conversation: presentation, terminology, content, phrasing and the number of questions.

Phase 2—testing the revised PMOS

Following amendments to the PMOS based on the ‘think aloud’, the revised measure was tested with patients recruited across the stroke, cardiac and orthogeriatric wards. The majority of participants requested assistance from the research lead in the form of reading out the questions and writing down answers. The average length of time to complete was ∼20 min per patient, ranging from 10 to 30 min.

Pilot test for the DUQuA study

From the outset of the study, the research team kept a project log of potentially relevant and important occurrences—any facet that may affect the feasibility of using the questionnaire. This included records of communication with the hospital's research office, conversations with ward staff, the research lead's observations of accessing wards, interacting with patients and family members, and administering the questionnaire. These occurrences were discussed by the research team during regular meetings. This was undertaken to inform and justify key decisions about data collection materials and processes for future use of the PMOS, in particular for the DUQuA study.

Analysis

The notes made by the research lead were collated and categorised, generating a list of recommended changes to the PMOS. Given the extensive development process in the UK, the research team consulted with the UK PMOS lead (RL) to discuss the suggested amendments. A consensus was reached based on findings from additional studies that had been undertaken or were underway in the UK, and revisions were made to the Australian version of the PMOS.

Phase 2—revised PMOS reliability test

The reliability of the PMOS subscales were assessed using interitem correlation for subscales with two items (r>0.2 acceptable),35–37 and Cronbach's α for subscales with more than two items (α>0.7 acceptable).38 For the total sample and the individual wards, mean (M) and standard deviation (SD) scores were calculated for each subscale, as well as a ‘negative index’, representing the mean number of items that each patient scored negatively (ie, ≤2).6

The project log was categorised into a set of key feasibility factors, with an explanation of how each had affected the decisions made for the DUQuA study. These key feasibility factors informed the development of training materials and resources for DUQuA data collectors.

Results

Five patients agreed to participate in the ‘think aloud’ study but on continuous screening during the third consultation, it was clear that this patient was not engaging with the researcher to answer questions appropriately. Therefore, the researcher thanked the patient and informed the NUM that, for this reason, the patient would be excluded. Consequently, four patient consultations were included in this phase. All four patients commented on three main areas: item phrasing, terminology and wording. As no new comments or suggestions were made, no additional patients were approached for further consultation.

Of the 43 questions in the PMOS, 13 (30%) were amended based on areas these patients highlighted for improvement (table 3). Recommended changes included amending healthcare terms that were less common in the Australian context, and rephrasing negatively phrased questions to positively framed questions. Following consultation with the UK team, the decision to reword all negatively phrased questions was not taken, as the research team believed in some instances the intended meaning of the questions would be altered as a result. The proportion of negatively phrased questions was therefore reduced from 56% to 28%. The new structure and reliability levels of the adapted PMOS are reported following phase 2.

Table 3

Modifications to the patient measure of safety following phase 1

Issues and areas for improvement	PMOS amendments made
Item phrasing Patients not comfortable with negatively phrased items that put staff caring for them in a bad light Patients found many negatively worded questions confusing	Phrasing reworded Negatively worded items were reduced from 56% to 28% (24–12 items); did not reword items if this would change the intended meaning
Terminology UK terminology caused uncertainty, for example, patients unsure what ‘porters’ were	Terminology changed UK terms were substituted with Australian, for example, changed ‘porters’ to ‘wardsmen’
Wording Patients suggested improvements to language	Wording modified Changes were made to item wording, for example, ‘staff gave me different information about my care’ was changed to ‘staff gave me conflicting information about my care’

Descriptive statistics and revised PMOS reliability

A total of 34 patients were approached following identification by the NUMs. One patient declined and two agreed to participate and signed a consent form, but later withdrew, citing reasons including feelings of tiredness, or the arrival of a physiotherapist for final assessment before discharge. Patients agreeing to participate (n=31) were from the cardiology (n=8), stroke (n=13) and orthogeriatic wards (n=10). One stroke and one hip fracture patient agreed to participate but on continuous screening by the research lead, it appeared they were confused and consequently these responses were excluded. Therefore, the number of included patients was 29 (n=8 cardiology, 12 stroke and 9 hip fracture).

The mean age of participants was 71.3 (SD=16.39) across all wards (stroke M=69.83, SD=18.89; cardiology M=62.25, SD=12.81; hip fracture M=82.10, SD=9.56). Two items with response rates lower than 25% were removed from analysis: ‘a doctor changed my plan of care and other staff know about it’, and ‘staff were kept waiting for my test results’. The indications were that patients did not know the answers to these questions. All PMOS subscales demonstrated acceptable reliability, based on either Cronbach's α (α>0.7) or interitem correlation (r>0.2), except for the organisation and care planning subscale, which had a Cronbach's α of 0.52. Additional file 1 compares the Cronbach's α, interitem correlations and mean subscale interitem correlations for the originally tested PMOS and our revised version.

Revised PMOS scores

As illustrated in table 4, the mean PMOS negative index for the entire sample was 2.59 (SD=2.72), suggesting that on average people answered 2–3 out of 43 questions negatively (once all items were reverse scored, a low score, ie, ≤2/5 indicates a negative answer). Individual item analysis revealed scores between M (SD)=3.26 (0.99; ‘staff were kept waiting for my test results’) and M (SD)=4.69 (0.47; ‘the drugs I have been prescribed were always available’)—this item also had the lowest SD, indicating limited use of the rating scale. The extent to which items were answered with ‘not applicable’ (NA) ranged from 0 (19 items) to 10 (1 item), with ‘a doctor changed my plan of care and other staff knew about it’, and ‘the following aspects of the ward made it difficult for staff to do their jobs: position of nurses’ station receiving n=8 NAs and ‘staff were kept waiting for my test results’ receiving 10.

Table 4

PMOS scores

Subscales (5-point Likert scale)	Stroke M (SD)	AMI M (SD)	Hip M (SD)	Total
Communication and team work	4.53 (0.61)*	4.50 (0.71)	4.13 (0.90)	4.40 (0.76)
Organisation and care planning	4.45 (0.65)	4.50 (0.94)	4.17 (0.86)	4.38 (0.81)
Access to resources	4.38 (0.80)	4.34 (0.87)	3.74 (1.23)	4.21 (0.97)
Ward type and layout	4.40 (0.83)	4.25 (0.88)	4.20 (1.12)	4.31 (0.93)
Information flow	4.50 (0.59)	4.38 (0.81)	4.29 (0.61)*	4.41 (0.66)
Roles and responsibilities	3.96 (1.13)†	4.13 (1.18)†	3.68 (1.36)	3.92 (1.22)
Staff training	4.46 (0.66)	4.69 (0.60)*	4.18 (0.53)	4.44 (0.63)*
Equipment (design and function)	4.43 (0.88)	4.47 (0.83)	4.00 (0.94)	4.31 (0.84)
Delays	4.30 (0.88)	4.20 (0.94)	3.11 (1.13)†	3.89 (1.11)†
PMOS negative index‡	1.67 (1.56)	2.13 (2.23)	4.22 (3.70)†	2.59 (2.72)

*Ward-based subscale most favourable scores from patients.

†Ward-based subscale with least favourable scores from patients.

‡Represents the mean number of items that each patient scored negatively (ie, ≤2).6

AMI, acute myocardial infarction; M, median; PMOS, patient measure of safety.

Among the nine key domains measured in the PMOS, patients rated staff training most favourably (M=4.44, SD=0.63) and the delays subscale least favourably (M=3.89, SD=1.11). It was noted that the delays subscale only contained two questions, and thus had a larger SD.

Ward-level results

Patients with stroke had the most favourable answers to the communication and team work subscale (eg, patients felt staff listened to their concerns). The roles and responsibilities subscale received the least favourable scores (eg, patients were uncertain about who was in charge of their care, or of the different roles of those caring for them).

Patients with AMI rated staff training most favourably (eg, patients felt staff could use equipment properly and carry out tasks they should be able to do). The roles and responsibilities subscale received the least favourable scores (eg, patients were uncertain about who was in charge of their care, or of the different roles of those caring for them).

Hip fracture patients answered more negatively on average than patients with stroke and AMI. The information flow subscale (eg, staff handed information over between shifts effectively) was the most favourable to hip fracture patients, and delays (eg, to treatment, or having too few staff to get things done on time) received the lowest mean score.

A list of 10 key feasibility factors is presented in table 5. For each factor, the details of findings from the pilot study are provided, alongside an explanation of how each factor has influenced the decisions made for the DUQuA study. In addition, the research lead provided hospital leaders with a report of the results and subsequently met to discuss the results in order to gain feedback about the report (ie, if the tone was appropriate and the information valuable), and to discuss how best to provide the hospital with recommendations for improvement.

Table 5

Key feasibility factors and implications for the DUQuA study

Key factors	Findings from the pilot	Implications for the DUQuA study
Ethics	Identifying possible ethical issues for working with vulnerable populations	Provide training in ethical research and clear recruitment instructions to data collectors coordinating PMOS
Liaising with wards	Timing—understanding likely times for discharge and working around thisAppreciating the nature of the individual working wards and the best way to contact/ engage with the NUM	Design instructions for data collectors to ensure they understand the need to find information about discharge timing to make the process efficient. Provide options in the instructions for data collectors to present to wards to arrange data collection
Questionnaire timing	Identifying how long it takes to undertake the questionnaire	Provide an experience-based estimate of the time required in instructions for data collectors
Identifying patients	Logistics for obtaining eligible patient names	Emphasise importance of awareness of patient's name before approaching them, that is, train data collectors to liaise with the NUMs to create a system suited for the individual wards
Approaching patients	Learning how to approach and talk to these cohorts of patients	Provide scripts for data collectors detailing how to approach the patients, and the order in which information should be provided and requests for participation made
Providing participant with assistance	Assistance from family members or carers, or the data collector	Provide scripts for data collectors to determine whether a family member or carer may be willing to assist the patient to complete the PMOS if the patient is physically unable (eg, reading out the questions, and/or circling the scales), interpret the questionnaire (for non-English-speaking patients), or if the patient would like assistance from the data collector
Presentation of information	Presentation of information sheets and consent forms in an easy to use format	Provide data collectors with colour-coded paper to simplify which documents belong to the patient or data collector
Additional data	Decisions about additional data that will be important to factor into the analysis	Include item on questionnaire form to indicate whether the patient completed the PMOS independently, with assistance from the data collector, or from a family member or carer
Specialised tools	Tested the use of specialised tools (eg, paper-based enlarged Likert scales)	Provide data collectors with laminated and enlarged versions of the questionnaire and Likert scale
Providing hospitals with recommendations	Identifying an appropriate level of practical recommendations for the hospital and the wards based on the findings	Consult with hospital leaders to ensure (1) the hospital is provided with appropriate and feasible recommendations and (2) the format of hospital reports is pitched at an appropriate and valuable level

DUQuA, Deepening our Understanding of Quality in Australia; NUM, nurse unit manager; PMOS, patient measure of safety.

Discussion

Patients have the capacity to signal potential safety incidents early,39 40 and the PMOS has the capacity to reliably and efficiently tap into this resource.6 41 Testing the extent to which the PMOS can capture this information from vulnerable older adult patients in a reliable and risk-free way was a key aim of this study. Following a ‘think aloud’, the revised PMOS was tested with patients with stroke, AMI and hip fracture in one Australian hospital to establish internal consistency reliability of the revised subscales. While the ‘organisation and care planning’ produced a Cronbach's α of 0.52, the comparisons for the Cronbach's α, interitem correlations and mean subscale interitem correlations for the originally tested PMOS and our revised version demonstrate similar findings (additional file 1), despite our smaller sample size in our pilot study. Throughout the study, the research team identified key factors relating to hospital engagement, participant recruitment, questionnaire administration and providing feedback to hospitals that will inform and enhance the feasibility of using this measure for the national DUQuA study.

One key amendment to the PMOS, based on the results of the ‘think aloud’, was the reduction of negatively worded items from 56% to 28% (24–12 items) to reduce the possibility of confusing, distressing or eliciting unreliable answers from participants. The decision to make these changes is supported by previous research with acutely ill hospitalised older adult patients, which found that simpler questions are more effective.31 On testing the revised version in phase 2, internal consistency reliability was established for all but one subscale, and differences were observed for the most and least favourably rated areas of safety on the three wards. However, the modest sample size within each ward did not warrant statistical analysis of these differences. Item analysis revealed three questions had been completed with between 8 and 10 NA answers. It is understandable that patients may not know the answers to these questions (eg, they may have had no way of knowing if their test results had been delayed, or where the nurse’s station is positioned to enable an opinion to be formed about how this might have affected the difficulty of a nurse's job). In the deployment of the PMOS, participants should be encouraged to comment on why a particular question is not applicable to them in order to support the explanation of results.

By piloting the PMOS, the research team have experienced the process of administering the questionnaire and have developed ideas for improving the feasibility of using this measure with an older, vulnerable cohort of patients (the mean age of patients in this study was 71.3 years, 17.3 years older than the sample used for the original PMOS validation) for the DUQuA study. Examples of some of the key feasibility findings include liaising with wards to understand operational procedures and identify patterns of patient discharge. Making initial contact with each of the NUMs to make a collaborative decision about the days and times of day to call to learn of new patient discharges and arrange subsequent visits to the ward was crucial for the smooth recruitment of patients. This was particularly important due to the unpredictable nature of the discharge process, which has been previously highlighted as a challenging point at which to attempt to collect data from patients.32

While specific resources were developed for data collection based on recommendations from previous research with acutely ill hospitalised older adult patients, this work highlighted areas for improvement to the type and format of resources that need to be provided. For example, in this study, patients typically needed the research lead to read aloud and fill out the questionnaire based on their answers, and were also provided with an enlarged (font size 28) Likert scale to point to the rating of their choice. However, findings indicated patients also preferred to see the questionnaire while it was read aloud and often leaned over from their bed towards the research lead's version of the PMOS, causing patients to move into a position that was not ideal for their condition. As the research lead read the questions aloud, it would have been useful to have enlarged versions of the entire questionnaire and scale for patients to view (eg, a minimum font size of 15 to ensure letter height based on visual arc is sufficient for viewing capability)42 to ensure they were comfortable throughout participation, a factor previously identified as important.34 Laminating these resources would enable the research lead to easily identify them among the different sets of documentation (eg, consent forms, information sheets, revocation sheets, the questionnaire itself), and to reuse them with all patients. Colour coding the documentation would also have been beneficial, to enable easy identification of the different paperwork.

Implications for health services research

The experiences in this PMOS pilot and feasibility study have influenced key decisions for the use of this measure in the national DUQuA study (table 5), and will inform the design of future health services research studies measuring patient perceptions. For example, detailed instructions and webinars have been developed for the DUQuA data collectors to ensure they understand the need to develop approaches for (1) engaging with the wards; (2) finding information about timing of discharge; and (3) determining when it is appropriate to visit the ward, to minimise disruption for staff and to make the process efficient. These instructions include strategies that data collectors can present to wards to arrange data collection. Furthermore, the data collection resources (ie, laminated versions of enlarged scales and complete questionnaires) have been modified to maximise the ease for patients to visualise and use the materials while maintaining a comfortable position, and to improve the logistics of the process for the data collector.

This preliminary work also enabled the research team to consider the format of feedback to provide to the participating hospital and respective wards, an important factor for bolstering senior healthcare management decision-making for improvement based on research evidence.43 Throughout the production and dissemination of the hospital report based on this work, the research team liaised and met with senior hospital leaders to discuss the possible approaches in presenting feedback to ensure that (1) the hospital and participating wards were provided with appropriate and realistic recommendations and (2) the reports for the cohort of DUQuA hospitals will be pitched at an appropriate and valuable level.

Strengths and limitations

Factors including selection and investigator bias, randomisation and the use of a control group could have been improved within the study. For practical reasons, we had to rely on the ward managers to identify the patients based on their discharge lists for the day. While patients were not randomly selected, the full list of patients due to be discharged was used, and all patients on this list were approached, removing an element of selection bias. The investigator was chosen as an impartial, unbiased person to administer the questionnaire, rather than members of staff working on the ward, in order to reduce the risk of patients providing responses that were more favourable than their genuine perceptions (eg, for fear of offending staff who may have directly cared for them). Limited timeframes and the difficulty recruiting these types of patients meant the decision to use a control group or comparison questionnaire was not taken.

While only one hospital was used, it is typical of the DUQuA hospital sample (ie, a tertiary metropolitan public hospital that treats all three conditions, with ∼300 beds). Furthermore, although this study used modest samples of patients in each phase, the sample was representative of the forthcoming DUQuA study population (ie, in terms of the inclusion criteria, including the primary diagnosis for admission—fully specified in the DUQuA study protocol9), which bolsters the ecological validity of this feasibility test. The sample size across individual wards did not allow for statistical analysis of ward differences that were observed. Nonetheless, internal consistency reliability was established for all but one subscale, and the DUQuA study aims to test the PMOS with 1800 patients from each condition (5400 total) across 60 hospitals. Therefore, sample sizes to detect differences between wards for patient ratings of these areas of safety will be adequately powered. Despite the small sample size, feedback to the research team from the hospital has indicated value in the results for generating ideas for targeted improvement.

Conclusion

For the first time internationally, a cohort of vulnerable patients provided their perceptions of the care they received on the respective wards using the PMOS, a tool that highlights evidence-based contributory factors to the delivery of safe patient care, and signals areas for hospitals to target for improvement. Ensuring that the questionnaire is appropriate for use with these patient groups is highly significant for the DUQuA study, and will inform the deployment of this questionnaire nationally, and of future health services research projects. With a larger, more representative sample, DUQuA will further strengthen the validity and reliability of the PMOS for the benefit of Australian and international hospitals and institutions. The DUQuA study will also assess the strength of relationships between the PMOS and the way hospitals are set up for quality and safety, the way care is delivered, and patient outcomes.