Validation of the Primary Care Patient Measure of Safety (PC PMOS) questionnaire.

BACKGROUND: The Primary Care Patient Measure of Safety (PC PMOS) is designed to capture patient feedback about the contributing factors to patient safety incidents in primary care. It required further reliability and validity testing to produce a robust tool intended to improve safety in practice.
METHOD: 490 adult patients in nine primary care practices in Greater Manchester, UK, completed the PC PMOS. Practice staff (n = 81) completed a survey on patient safety culture to assess convergent validity. Confirmatory factor analysis (CFA) assessed the construct validity and internal reliability of the PC PMOS domains and items. A multivariate analysis of variance was conducted to assess discriminant validity, and Spearman correlation was conducted to establish test-retest reliability.
RESULTS: Initial CFA results showed data did not fit the model well (a chi-square to df ratio (CMIN/DF) = 5.68; goodness-of-fit index (GFI) = 0.61, CFI = 0.57, SRMR = 0.13 and root mean square error of approximation (RMSEA) = 0.10). On the basis of large modification indices (>10), standardised residuals >± 2.58 and assessment of item content; 22 items were removed. This revised nine-factor model (28 items) was found to fit the data satisfactorily (CMIN/DF = 2.51; GFI = 0.87, CFI = 0.91, SRMR = 0.04 and RMSEA = 0.05). New factors demonstrated good internal reliability with average inter-item correlations ranging from 0.20 to 0.70. The PC PMOS demonstrated good discriminant validity between primary care practices (F = 2.64, df = 72, p < 0.001) and showed some association with practice staff safety score (convergent validity) but failed to reach statistical significance (r = -0.64, k = 9, p = 0.06).
CONCLUSION: This study led to a reliable and valid 28-item PC PMOS. It could enhance or complement current data collection methods used in primary care to identify and prevent error. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Research into patient safety in primary care is growing internationally given some recent estimates about the frequency of incidents currently occurring in this setting. A systematic review from Panesar et al shows that between <1 and 24 patient safety incidents occur for every 100 primary care consultations, with approximately 4% of them resulting in severe harm.1

The types of safety incidents occurring in primary care have been previously published2–9 and the most common centre around administrative and communication incidents, diagnostic incidents, and prescribing and medication management incidents.1 Furthermore, information about the contributory factors that lead to a safety incident in primary care have also been described. There are numerous latent and error-producing conditions in the primary care environment including the professional, patient, team, working environment, task, computer system, organisation and management systems, primary–secondary interface and institutional context.2 9 10

Much of the knowledge regarding patient safety in primary care has been obtained via studies using retrospective review of individual patient records, formal incident reporting or significant event audits (root-cause analysis). While these methods can capture high-quality information, these approaches are often reactive, lack a systematic understanding or implementation of accident causation theory and prevention of error, and limited to one point of view—the primary care provider.11–14

Patient involvement in safety is a developing field with growing interest and advancement internationally. While there are numerous measures and research reporting on patient experience of primary care,15 16 measures that capture patient feedback on safety are limited.17 Patient experience and satisfaction with primary care service use is inherently different from patient perspectives on the contributing factors to safety incidents or experiences of safety incidents themselves.18 Capturing feedback from patients on safety in primary care has traditionally occurred through experience or satisfaction surveys, after event reporting, and incident disclosure.19 20 Existing patient-reported instruments to measure patient safety in primary care17 21 are few in number and primarily focus on the outcome of the safety incident, rather than the error-producing or latent contributory factors to safety incidents.22 However, there is one tool that identifies the contributory factors to safety incidents in primary care from the patients’ perspective: the Primary Care Patient Measure of Safety (PC PMOS).23

The PC PMOS is a 50-item questionnaire covering 15 contributory factors domains. The PC PMOS was developed using both inductive and deductive approaches derived from James Reason’s Swiss Cheese model of accident causation theory,24 qualitative inquiry,25 26 literature exploring patients views of safety2 3 5 6 27–33 and has direct lineage from the hospital Patient Measure of Safety (PMOS) tool.34 35 The PC PMOS has shown good face validity but has not undergone further reliability and validity testing to produce a robust tool intended to improve safety in practice. Therefore, the aim of this study was to establish the validity and reliability of the PC PMOS. The objectives were to explore the factor structure and internal reliability of the scale, the extent to which the scale discriminates between primary care practices (discriminant validity) and the extent to which it converges with staff measures of patient safety (convergent validity).

Method

Study design, setting, sample and recruitment

A cross-sectional study design was used to collect data from primary care patients and staff during April to June 2017 in Greater Manchester, UK. A total of 10 primary care practices were invited to participate in the study. Recruitment of primary care practices was undertaken by author SG using a purposive approach to ensure a heterogeneous sample was obtained regarding practice demographics. Practices were recruited through existing networks and contacts. A total of nine primary care practices agreed to participate in the study, and one practice declined to take part due to an internal investigation taking place at the time. The nine practices included a total of 114 members of staff and 95 844 patients who were connected with them.

All staff at each primary care practice were invited to complete the Agency for Healthcare Research Quality Medical Office Survey (AHRQ MOS) on patient safety culture.36 There was no a priori required sample size for the staff survey; although at a minimum the practice manager or clinical lead at each practice was asked to complete the survey. Completion and return of the AHRQ MOS implied staff consent.

Each practice was asked to recruit a minimum of n=50 patients to complete the PC PMOS patient questionnaire in order to reach the minimum required sample size of n=450 patients for the planned confirmatory factor analysis (CFA). This predicted sample size is based on the item per participant ratio of 1:10 principle.37 A minimum of 30 patients from each practice were required to enable comparisons between practices. Patients included in the study were those presenting to their primary care practice for an appointment during the study period. Patients who were unable to speak English, unable to read or write or who were unable to complete the questionnaire for any other reason were excluded from the study. Patients were approached by practice staff or a member of the research team while in the waiting room and were invited to participate in the study. They were given an information sheet about the study and were then asked to complete the PC PMOS questionnaire while in the waiting room. Waiting rooms were selected as the site for completing the PC PMOS as this was deemed most suitable from both patients and staff during face validity testing.23 While patient preference for completing the PC PMOS in the waiting room was adhered to in this study, there are some challenges with this method. These include time to complete and return the survey, and patients being called to see a health professional while in the middle of completing the questionnaire. Completion of the PC PMOS implied consent. Patient sampling occurred over a 6-week period or until a minimum of n=50 patients had completed the questionnaire. A random subset of patients (n=50) from all nine practices were asked to complete the questionnaire again after 1 week for test–retest reliability purposes. These patients were provided with a second copy of the PC PMOS and a prepaid envelope. They were asked to complete the questionnaire 1 week later and return it to the research team using the prepaid envelope.

Measures

PC PMOS

The PC PMOS assesses patients’ perception of the factors contributing to patient safety in primary care. These include latent and error-producing conditions across a number of primary care safety domains. The specific items and domains contained in the PC PMOS have been published elsewhere.23 Briefly, the PC PMOS is a 50-item questionnaire with a five-point Likert response scale ranging from ‘strongly disagree’ (1) to ‘strongly agree’ (5). The scale also includes ‘don’t know’ and ‘not applicable’ options. The 15 domains in the scale include access, communication, continuity of care, dignity and respect, equipment (design and function), external policy context, information flow, organisation and care planning, patient-related factors, physical environment, primary–secondary care interface, referrals, task performance, team factors, and training and education. Patient demographic information about age, gender, postcode and number of visits to the primary care practice in the previous 12 months was also collected.

In order to perform the analysis, domains containing ≥2 items were reconsidered for inclusion by two researchers (SG, AH). The original 15 domains were reduced to 10 new domains prior to undertaking the analysis. A total of six items from single-item domains were added to conceptually similar domains (online supplementary file 1). Single items pertaining to a construct have been found to demonstrate poor reliability and validity,38 and therefore, these domains were eliminated and items moved to similar domains. Negative items were reverse scored. Items were not collapsed. A high score indicates a positive patient view of safety.

AHRQ MOS on patient safety culture

The AHRQ MOS on patient safety culture was used to determine convergent validity. The AHRQ MOS has been validated and reported elsewhere.36 The patient safety grade was calculated using question G2. Overall rating on patient safety from the AHRQ MOS; ‘Overall, how would you rate the systems and clinical processes your medical office has in place to prevent, catch and correct problems that have the potential to affect patients?’ (five-point Likert response scale: poor, fair, good, very good or excellent).

Data analysis

CFA using AMOS V.19 software was undertaken using maximum likelihood estimation. Guidelines for testing model fit followed guidance by Hooper et al 39: a chisquare to df ratio (CMIN/DF) of <2.00, the goodness-of-fit index (GFI) ≥0.90, CFI≥0.90, SRMR ≥0.05 and the root mean square error of approximation (RMSEA) ≥0.05. Post-hoc analysis was used to improve the model fit by inspecting modification indices (MIs), standardised residuals (SRs) and item content.40

The internal reliability of the factors was examined using Cronbach alphas. Discriminant validity was assessed using a multivariate analysis of variance (MANOVA) test to establish differences between primary care practices. The convergent validity of the new PC PMOS was tested by correlating total score (individual patient level) with the patient safety grade (individual staff level) on the AHRQ MOS. To examine test–retest reliability, data were used from 22 patients who completed the measures again 1 week from baseline. Baseline scores on the PC PMOS scales were correlated with corresponding scales, 1 week from baseline.

Results

The final sample consisted of n=490 primary care patients who completed the PC PMOS from nine general practices. In total, 163 of the patients were male and 322 were female, 5 patients did not state their gender. The average age of the patients was 44 years (SD 18.76) and the average number of visits made to the general practice in the previous 12 months was 7 (SD 7.92). While the exact number of patients approached to complete the PC PMOS was not recorded, approximately 550 of patients were asked to participate in the study. This provides an expected response rate of 89%.

Missing value analysis indicated that data were missing completely at random (x 2(4669)=5258.798, p<0.001); therefore, missing values were imputed. The descriptive scores pertaining to each domain in the PC PMOS pre and post CFA analysis are presented in table 1.

Table 1

Descriptive statistics for the PC PMOS domains

Domain	Pre CFA		Post CFA
	Mean	SD	Mean	SD
Access	3.51	0.67	3.45	0.79
Communication	4.07	0.57	4.07	0.66
External policy	3.57	0.95	No items deleted
Information flow	3.67	0.67	3.81	0.75
Organisation care planning	3.67	0.58	3.90	0.71
Patient-related factors	3.99	0.56	4.26	0.83
Physical environment	4.11	0.70	No items deleted
Referrals	3.82	0.77	No items deleted
Task performance	3.78	0.66	3.69	0.92
Team factors	3.86	0.77	Domain deleted
Total PC PMOS score	3.84	0.49	3.86	0.51

Data were collected in 2017 in nine general practices within Greater Manchester, UK. A total of 490 patients were included in the study.

CFA, confirmatory factor analysis; PC MOS, Primary Care Patient Measure of Safety.

Construct validity of the PC PMOS

A sample of n=490 patients, CFA was used to test a 10-factor model using maximum likelihood estimation. The initial CFA showed data did not fit the model well (CMIN/DF=5.68; GFI=0.61, CFI=0.57, SRMR=0.13 and RMSEA=0.10), thus post-hoc model fitting was conducted. Post model fit analysis involved removal of items on the basis of large MIs (>10.00), and SRs >±2.58, and assessment of item content. This resulted in the removal of 22 items and one entire domain ‘team factors’. This revised model (containing 28 items in nine domains) was found to fit the data satisfactorily (CMIN/DF=2.51; GFI=0.87, CFI=0.91, SRMR=0.04 and RMSEA=0.05). Example items included in the nine domain model can be found in online supplementary file 2.

Internal reliability

To explore the internal reliability of the domains post CFA, Cronbach alphas were calculated for domains containing seven items or more (where ≥0.70 was considered ‘good’) and average inter-item correlations were calculating for domains containing six items or fewer. Briggs and Cheek recommend an optimum range of 0.20–0.50 for average inter-item correlations to demonstrate ‘good’ internal reliability.41 All domains demonstrated ‘good’ internal reliability, with the exception of the domains named ‘communication’ and ‘patient-related factors’ (see table 2).

Table 2

Internal reliability scores for PC PMOS domains

Domain	Pre CFA		Post CFA
	Items, n	Cronbach alpha*	Items, n	Average inter-item correlation
Access	6	0.68	4	0.30
Communication	12	0.86	6	0.84
External policy	2	0.40	No change
Information flow	5	0.36	4	0.47
Organisation care planning	5	0.10	3	0.25
Patient-related factors	7	0.66	2	0.71
Physical environment	2	0.43	No change
Referrals	2	0.46	No change
Task performance	7	0.68	3	0.38
Team factors	2	0.18	Domain deleted

Data were collected in 2017 in nine general practices within Greater Manchester, UK. A total of 490 patients were included in the study.

*Average inter-item calculated for scales with fewer than seven items.

CFA, confirmatory factor analysis; PC MOS, Primary Care Patient Measure of Safety.

Discriminant validity

A total of n=81 practice staff completed the AHRQ MOS on patient safety culture, and n=74 provided a response to the question assessing an overall patient safety grade. The mean PC PMOS scores by general practice are presented in table 3. A MANOVA was performed to assess the extent to which the PC PMOS discriminated among the nine general practices. The nine domains were entered as dependent variables and the variable ‘GP practice’ was entered as a fixed factor. The MANOVA revealed an overall effect of GP practice (F=2.64, df=72, p<0.001 eta=0.05). Tests of between factors revealed some significant differences between general practices (see table 3). Despite the small sample size within each general practice, the MANOVA revealed PC PMOS to have good discriminant validity. Bonferroni adjustment was used to for post-hoc analysis.

Table 3

Mean scores on PC PMOS by general practice

Domain	General practice
	P1a (N=65)	P2b (N=52)	P3c (N=52)	P4d (N=51)	P5e (N=32)	P6f (N=56)	P7g (N=46)	P8h (N=65)	P9i (N=71)
Access	3.63c	3.0acefh	3.59b	3.56	3.42	3.72bh	3.63b	3.23f	3.64b
Communication	4.15	3.86f	3.95	4.09	4.23	4.28h	4.26	3.90f	3.99
External policy	3.74d	3.51	3.76	3.10aefgh	3.87d	3.73d	3.76d	3.68d	3.44
Information flow	3.90	3.67	3.67	3.97	3.84	3.99	3.92	3.73	3.64
Organisation care planning	4.00	3.78	3.85	4.00	4.1	3.99	3.89	3.81	3.79
Patient-related factors	4.42	4.10	3.98fg	4.15	4.46	4.50c	4.51c	4.14	4.15
Physical environment	4.27i	4.19	3.90 f	4.19i	4.30i	4.32ci	4.19	4.07	3.79adef
Referrals	3.92	3.90	3.64	3.90	3.90	3.94	3.84	3.67	3.79
Task performance	3.79	3.61	3.49	3.58	3.88	3.87	3.97	3.69	3.48

Data were collected in 2017 in nine general practices within Greater Manchester, UK. A total of 490 patients were included in the study. General practice mean scores with a subscript letter indicate the scores are statistically significantly different from one another. P1, P2, P3, Practice 1, 2, 3.

PC MOS, Primary Care Patient Measure of Safety.

Convergent validity

In order to assess the convergent validity, the total PC PMOS score was correlated with the patient safety grade from the staff AHRQ MOS. The correlation showed some positive association but the relationship failed to reach significance (r=0.19, p=0.08).

Test–retest reliability

A random sample of n=22 patients returned completed PC PMOS questionnaires again approximately 1 week after baseline completion. Spearman’s rho correlations were conducted between baseline domain scores and time 2 scores. With the exception of the ‘external policy’ domain, all domains demonstrated good test–retest reliability as scores were highly correlated between time points (see table 4).

Table 4

Spearman’s rho correlations between baseline and follow-up scores on each domain

Domain	r	n	P values
Access	0.65	20	0.002
Communication	0.96	18	0.0001
External policy	0.35	19	0.15
Information flow	0.68	20	0.001
Organisation care planning	0.73	20	0.0001
Patient-related factors	0.96	21	0.0001
Physical environment	0.54	20	0.01
Referrals	0.81	18	.0001
Task performance	0.66	20	.002

Data were collected in 2017 in nine general practices within Greater Manchester, UK. A total of 490 patients were included in the study.

Discussion

The results of the study suggest that the PC PMOS demonstrates acceptable levels of reliability and validity, although further development and testing is required to strengthen each construct. The PC PMOS showed acceptable construct validity, good internal reliability, good discriminant validity, adequate convergent validity (although not statistically significant) and acceptable test–retest reliability.

There are two important reasons for developing a valid and reliable version of the PC PMOS tool. First, it engages patients in the safety discourse. The PC PMOS is the only tool of its kind to systematically and routinely collect information from patients about their views of contributing factors to safety incidents in the primary care setting. It builds on work from the secondary care setting to capture patient views about the factors that contribute to patient safety incidents,34 35 42 and the growing recognition that patients are a valuable source of information in the prevention of error in healthcare. Evidence suggests that patients have intimate and detailed knowledge, and experience of their healthcare journey. Patients have been shown to be able to comment on error and patient harm, and can also identify factors that contribute to patient safety incidents.25 26 34 43 They can provide real-time information about patient safety that is a direct reflection of what is important to them regarding potential risks and how they can be prevented.44–47 Patients have also displayed insight into safety issues that professionals or others may not recognise,48 and reported safety incidents that may have gone undetected using other safety reporting methods.47

The second reason for developing a valid and reliable PC PMOS tool is its potential value as an intervention instrument in primary care practice. The data obtained from the PC PMOS could be used to develop, implement and measure the effectiveness of specific safety prevention activities. Existing quality improvement or clinical governance procedures and processes, which use the well-recognised Plan, Do, Study, Act approach, would be well suited for such a patient-informed safety intervention.49 Our research team aims to examine the feasibility of using the PC PMOS as an intervention tool in the future. Furthermore, data collected on the PC PMOS could compliment other patient safety intelligence data collected in primary care, such as staff incident reporting data, significant event data, the friends and family test, formal patient complaint data and data from patient feedback websites, such as ‘NHS choices’, and ‘I want great care’.50 51 The PC PMOS has the potential to reduce the current gap in the systematic collection of data relating to patients’ views of safety within primary care.

Study strengths and limitations

The main strength of the study is the development of a valid and reliable survey instrument that can be used to assist with improving safety in primary care. All the domains had satisfactory internal reliability. Additional benefits of undertaking this study include the construction of a shorter questionnaire containing 28 items covering nine domains of safety. This version of the PC PMOS will be quicker and less burdensome for patients to complete. Furthermore, each item/domain on the PC PMOS is considered to be actionable by primary care practice staff when undertaking safety improvement activity.

Another strength of the PC PMOS tool is that it is the only available measure that collects patient feedback about contributing factors to safety incidents in primary care. Current measures developed or applied in primary care practices commonly assess patient experience or satisfaction with care delivery.15 16 While this information can be useful for quality purposes, much of the data collected does not address the system-level factors that impact on safety.11–14 Furthermore, these data are often considered unusable by practice teams to prevent error.

While this shorter version of the PC PMOS retains both error-producing and latent items and domains that contribute to safety incidents, there were 22 items and one whole domain that were removed during the CFA. The one domain that was removed was the ‘team factors’ domain and contained the items ‘staff didn’t seem to know what they were meant to be doing’ and ‘staff were always able to get help from other staff when they asked for it’. Both of these items were derived from the hospital PMOS questionnaire34 35 which may have limited their relevance to the primary care setting. Patients in primary care often receive episodic care on a one-on-one basis rather than in a team-based environment which is more apparent in a hospital setting.34 The items from the ‘external policy’, ‘physical environment’ and ‘referrals’ domains remained unchanged during the analysis, but one or more items were removed from the other six domains. The largest removal of items was observed in the ‘communication’, ‘patient-related factors’ and ‘task performance’ domains. Many of the questions from these domains were not concerned with contributory factors to safety incidents but rather outcomes of safety incidents (eg, ‘the diagnosis or treatment plan recommended by my doctor, nurse or other health professional was right for me’) or actions patients took to prevent safety incidents (eg, ‘I feel I cannot speak up about certain things with health professionals at the practice’ (negatively worded)). These items need to be considered further about their potential value and use in primary care practice.52

The PC PMOS demonstrated good discriminant validity and is therefore sensitive to differences between primary care practices; however, it failed to demonstrate significant convergent validity. The AHRQ MOS safety culture survey was chosen as the dimensions of safety assessed were similar to the domains on the PC PMOS, and it has shown to be a valid and reliable measure.53 However, most of the available research using the AHRQ MOS has been conducted almost exclusively with US populations,54 55 which may limit its use with a UK population. Moreover, the response rate by practice staff for this survey also varied widely, and in some cases less than the minimum number required for analysis were returned, which also has implications for assessing the convergent validity of the measure.

The predictive validity of the PC PMOS is not reported here. Given the low frequency of error in primary care1 and lack of adequate measures to capture data on the number of safety incidents at each practice, the ability of the PC PMOS to predict error in primary care was unable to be determined. Further research is needed to establish the predictive validity of this tool.

A further limitation includes the low number of patients recruited to complete the PC PMOS a second time 1 week later which limits the test–retest reliability of the measure.

Another limitation of the study was that patients who were unable to speak English, unable to read and write or unable to complete the questionnaire were excluded. Future work will investigate ways to include these groups of patients.

Although key primary care staff members, such as the practice manager or clinical lead, and a minimum of n=5 staff responses were used to calculate the mean staff safety culture scores, we cannot guarantee that their responses are representative of the entire safety culture at each primary care practice.

Conclusion

The PC PMOS is a theory and evidence-derived tool that captures patients’ perspective on contributory factors to safety incidents in primary care. It is a unique tool and the first of its kind showing adequate reliability and validity. Patient data collected using the PC PMOS may help to identify areas of weakness in the primary care system that are unreported or unidentified using other measures. The PC PMOS can also be used as change variable (measure) in continuous safety improvement work undertaken by primary care practices. Further development and testing is required to further strengthen each PC PMOS construct, as well as a feasibility study to inform and refine its use in practice.