Psychometric evaluation of a Patient Satisfaction Survey Questionnaire to assess advanced practice nurse ambulatory services in primary care.

AIM: To determine the construct validity and test-retest reliability of the Patient Satisfaction Survey Questionnaire (PSSQ).
BACKGROUND: Advanced practice nurses (APNs) co-manage patients with non-communicable diseases (NCDs) with public primary care physicians in Singapore. The original Nurse Practitioner Satisfaction Survey (NPSS) instrument was adapted (renamed as PSSQ) to assess satisfaction with APN service.
METHOD: A total of 209 patients self-administered the PSSQ at first visit and repeated it in a retest through phone within the next 2 weeks in 2018. Cronbach's alpha and intra-class correlation (ICC) coefficients were performed to evaluate the internal consistency and test-retest reliability. Construct validity was determined by confirmatory factor analysis (CFA).
RESULTS: The overall Cronbach's alpha (>0.7) and ICC scores (0.697-0.729) indicated excellent test-retest reliability and good internal consistency, respectively. The CFA results of three-factor model showed poor fit. Additional exploratory factor analysis (EFA) and principal component analysis (PCA) with varimax rotation revealed that adding another factor (communication with physician) resulted in a better model (eigenvalue > 1). The ICC for the four-factor model ranged from 0.664 to 0.825, indicating fair to excellent reliability.
CONCLUSION: The PSSQ shows good internal consistency and test-retest reliability. The four-factor model is a better instrument to assess APN service in local populations. IMPLICATIONS FOR NURSING MANAGEMENT: The validated PSSQ can be used as instrument to assess the quality of services provided by various categories of nurses, including registered nurses, nurse clinicians and APNs.

BACKGROUND

In 2018, the World Health Organization (WHO) declared that 41 million people died from non‐communicable diseases (NCDs) each year, equivalent to 71% of all deaths globally (WHO, 2018). Similarly, non‐communicable diseases are still among the top 10 causes of death in Singapore (MOH, 2015). Singapore also faces an ageing population, with a shrinking workforce, including those in the health care sector. Increasingly, nurses are trained to co‐manage non‐communicable disease patients with physicians. Those who attain master's degree in nursing are qualified practitioners to take care of patients with stable non‐communicable diseases (NCDs) independently. They presently helm the advanced practice nurse (APN) service in the local public primary care clinics or polyclinics. The APNs are in the medical frontline to interact and manage patients, and play a pivotal role in supporting the health of the population. In the local fee‐for‐service primary health care system, the continuity of care for patients with NCDs by these APNs hinges on the former's satisfaction with their services at the polyclinics. These APNs are trained to empower patients in self‐care, which is essential in the management of long‐term chronic medical conditions. Satisfaction with care improves adherence to treatment, functional status, overall well‐being and future health‐related behaviours in non‐communicable diseases (Koksvik et al., 2013). Nonetheless, satisfaction with the APN service has yet to be determined.

Satisfaction is broadly defined as the fulfilment of a need or want, a state of being contented and a source of enjoyment (Merriam Webster Online Dictionary, 2017). It is subjective and multifaceted. Thus, evaluation of satisfaction is multidimensional and is preferably rated on a scale. One such scale specifically targeting nursing is the Nurse Practitioner Satisfaction Survey (NPSS) (Agosta, 2009). NPSS is a 28‐item questionnaire in which responses are rated on a 5‐point Likert scale. It was developed to measure satisfaction with nurse practitioner service and the factors associated with this level of satisfaction. NPSS focuses on three main components of the nurse practitioner service: (1) general satisfaction, (2) communication and (3) convenience and accessibility. It was originally developed and validated in a younger population (70% of participants were aged <45 years) in a North American hospital.

The outpatients managed by APN in primary care differ from inpatients served by nurse practitioner in a hospital. Hence, we adopted and adapted the NPSS questionnaire to cater to the predominantly older outpatients with NCD in local primary care setting. The NPSS was modified by a group of Singapore APNs and renamed as Patient Satisfaction Survey Questionnaire (PSSQ).

Aim

This study aimed to determine the psychometric properties of PSSQ, including its construct validity, test–retest reliability and internal consistency. We postulated that PSSQ had equivalent construct validity, internal consistency and reliability as the NPSS.

METHOD

Setting

SingHealth Polyclinics (SHP) is a major public primary health care institution in Singapore. It comprises eight public primary care clinics (polyclinics) located in the eastern region of the island state. It is an accredited academic training centre for student nurses from local nursing schools. This study was conducted from March to April 2018 at seven polyclinics in SHP.

Study subjects

Inclusion criteria

The participants were adult, multi‐ethnic Asian patients with known NCDs in their electronic medical records. They had used the APN services at least twice in the past one year. The NCDs (including hypertension, dyslipidaemia, type 2 diabetes mellitus, hypothyroidism, asthma, chronic renal disease and chronic ischaemic heart diseases) are listed in the International Classification of Diseases Version 10 (2010‐10) (ICD‐10, 2010).

Exclusion criteria

Patients with unstable NCDs, including those with hospital admission and stroke in the past 3 months, were excluded. Those with known cognitive and auditory impairment or those who did not want to be contacted for the retest of PSSQ were also excluded.

Instrument

Permission was granted by Prof Lucie J. Agosta, from Southeastern Louisiana University in the United States to modify the original NPSS instrument to create the PSSQ. The original NPSS is a 28‐item questionnaire to be rated on a 5‐point Likert scale. The questions cover the accessibility, competence, knowledge, trust, interpersonal communication and general satisfaction of the tertiary‐based nurse practitioners. The validation and test–retest reliability study was conducted on 300 participants at the Employees’ Health Services Department of a Hospital in the United States of America. It has good internal consistency: Cronbach's alpha for general satisfaction was 0.98, for communication was 0.83 and for accessibility and convenience was 0.76. Its content and face validity were developed based on literature review and the application of the Cox's Interactional Model of Client Health Behavior. However, the instrument has not been validated in another population.

The NPSS was selected by the authors after an extensive literature search and review of related and appropriate instruments (Agosta, 2009). The APNs modified the NPSS to create the Patient Satisfaction Survey Questionnaire (PSSQ) by: changing the provider from ‘nurse practitioner’ in NPSS to ‘APN’ in PSSQ; setting from the ‘Woman's Hospital Employee Health clinic’ to ‘polyclinic’; and revising ‘My APN knows when to refer to physician’ to ‘APN knows when to refer to a senior physician/specialist’ (Question 17).

Like the NPSS, the PSSQ focuses on three key domains of nursing services: (1) general satisfaction; (2) communication; and (3) convenience and accessibility (Appendix S1). PSSQ also consists of 28 closed‐ended questions, with a 5‐point Likert scale (strongly disagree, disagree, agree, strongly agree and uncertain). Questions 29 to 33 are added to seek information on the number of consultations by APNs and the polyclinic physicians in the past one year and to compare their care by the two providers. The instrument also collects data on patient demographic profiles, general opinion of their health care and education.

Subject recruitment

The APNs at the study sites directed eligible patients to research assistants from a local nursing school in Ngee Ann Polytechnic. The approved patient information sheet was used to provide the patients with information on the study and clarified any doubts or queries before obtaining their written informed consent. The participants were self‐administered or aided by the research assistants in filling up the PSSQ. They were informed that they would be contacted again within the next two weeks for the retest by phone. To minimize any potential difference when the questionnaire was completed in person, as compared to its administration over a phone call, the same research assistant was assigned to conduct the test and retest for the same patient. The patients who completed the questionnaire were provided with SGD5 (USD3.60) voucher as a token of appreciation.

Sample size and power calculation

In order to validate the 28‐item instrument using factor analysis, the required sample size was computed to be 196 based on the COSMIN checklist (Consensus‐based Standards for the selection of health status Measurement Instruments; Mokkink et al., 2010). We further increased the sample size to 210 to buffer for retest defaulters.

Data analysis

Cronbach's alpha and intra‐class correlation (ICC) coefficients, respectively, were used to evaluate the internal consistency and test–retest reliability of the PSSQ (Cronbach, 1951). Cronbach's alpha assesses the close inter‐relationship of a group of items in a questionnaire. A value of more than 0.7 is considered acceptable; if it is more than 0.8, the internal consistency is considered excellent (Cronbach, 1951).

The test–retest reliability assesses the degree to which the test results remain consistent over time. In order to measure test–retest reliability, the same set of questions in the instrument is filled by the same individual on two separate occasions and the scores are correlated.

The intra‐class correlation (ICC) coefficient was used to determine the total and subscale scores obtained at the initial test and the subsequent retest after a short time interval, often about one to two weeks. An ICC score of more than or equal to 0.75 indicates excellent reliability, score of between 0.4 and 0.74 indicates fair to good reliability, and one of less than 0.4 indicates poor reliability (Enderlein & Fleiss, 1986).

To measure the construct validity of the instrument for the local population, confirmatory factor analysis (CFA) was conducted to verify whether the 28‐item PSSQ could fit the original three domains. CFA is a test to confirm or reject the measured theory. CFA determines whether a prespecified factor set influences the data in a predictive fashion (DeCoster & Smith, 2000). It is a multivariate statistical procedure to show how well the measured variables represent the number of concepts or domains. In PSSQ, they are general satisfaction, communication and appointment scheduling.

Assessing the measurement model validity refers to the comparison of the theoretical measurement with the reality model to determine how well the data fit. Such assessment requires a number of the statistical indicators such as the chi‐square test and other goodness‐of‐fit statistics such as Normal Fit Index (NFI), Comparative Fit Index (CFI), Root Mean Square Error of Approximation (RMSEA), the Tucker–Lewis Index (TLI) and the Standardized Root Mean Square Residual (SRMR). The criteria for appropriate model fit include a non‐significant chi‐square statistic, an NFI ≥ 0.90, a CFI ≥ 0.90, an RMSEA ≤ 0.05, TLI ≥ 0.90, and an SRMR ≤ 0.05 (Hu & Bentler, 1999).

As the results did not meet the criteria for the appropriate model fit, we conducted an exploratory factor analysis (EFA) using principal component analysis (PCA) with varimax rotation to determine the minimum number of principal components that account for most of the variations in the data. It reduces the large number of variables into smaller number of interpretable factors that explain the maximum amount variability in the data.

The Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy was used to determine whether the data are appropriate for factor analyses. KMO of more than 0.60 and Bartlett's test of sphericity of less than 0.05 indicate sufficient sample size and adequate correlations between the variables, respectively, so the variables could be reduced to a smaller number of components (Kaiser, 1974). The initial factor analysis has an infinite number of factors. Factor rotation, including varimax rotation, transforms the initial factors into new ones that are easier to interpret. This results in high factor loadings for a smaller number of variables and low factor loadings for the rest. Factor loading is basically the correlation coefficient for the variable and factor. The result is small number of important variables, which facilitates the interpretation of the results (Merenda, 1997). The number of factors to retain can be interpreted from the eigenvalue (>1.0) and the graphical Cattell's scree test. The scree test shows the eigenvalue plot in descending order where they level off. The point where the slope levels off indicates the number of factors that should be generated by the analysis.

Ethics approval

The study was approved by the SingHealth Centralised Institution Review Board (CIRB#2018/2134). The authors received seed grant from the Department of Research in SingHealth Polyclinics (Grant number: SHP‐SEED45‐2017[6]).

RESULTS

Complete data of 209 patients were analysed. One patient was excluded as she was uncontactable for the retest. Baseline characteristics of study participants are summarized in Table 1: the mean age was 64.1 years (SD = 9.9), 54.5% (114) were females and 80.4% had up to secondary education. They had average of three NCDs and were prescribed with an average of three medications. 75.1% of them were attended by the APN in the polyclinic twice or thrice in the past year, while the rest consulted even more frequently.

TABLE 1

Baseline characteristics of study participants

Total	n = 209
Age	64.1 (9.9)
Gender
Male	95 (45.5)
Female	114 (54.5)
Highest educational level attained
No formal education	3 (1.4)
Primary	53 (25.4)
Secondary	112 (53.6)
A‐level/Diploma (ITE/Polytechnic/Private school)	25 (12)
University/Post‐tertiary	16 (7.7)
Diabetes
Yes	88 (42.1)
Hypertension/high blood pressure
Yes	176 (84.2)
Hyperlipidaemia/high cholesterol
Yes	172 (82.3)
Total No. of comorbidities	3 (2–3)
Mean number of medications	3 (2–5)
Mean frequency of APN consultations
2–3	157 (75.1)
4–5	37 (17.7)
6 or more	15 (7.2)

Abbreviation: ITE, Institution of Technical Education.

Construct validity

The indices showed that the original three‐factor model fits well only for SRMR, but generally did not fit well: RMSEA = 0.125, CFI = 0.820, TLI = 0.804 and chi‐square (p < .01; Table 2). The results showed high factor loadings for all items, indicating good convergent validity (Hu & Bentler, 1999).

TABLE 2

Construct validity using confirmatory factor analysis

	PSSQ	Acceptable range
p‐Value, chi‐square	<.01
RMSEA	0.125	<0.06
CFI	0.82	≥0.9
TLI	0.804	≥0.9
SRMR	0.058	≤0.08
CD	0.994	≥0.8

Abbreviations: CD, coefficient of determination; CFI, Comparative Fit Index; PEI, Patient Enable Index; PSSQ, Patient Satisfaction Survey Questionnaire; RMSEA, Root Mean Square Error of Approximation; SRMR, Standardized Root Mean Square Residual; TLI, Tucker–Lewis Index.

Exploratory factor analysis (EFA) was conducted to determine other significant factor, aside from the original three‐factor structure. The KMO measure of 0.938 and significance of Bartlett's test (x 2 = 6,337.129, p<.001) showed that the 28 items in NPSS had adequate common variance for factor analysis (Agosta, 2009).

PCA with varimax rotation revealed four factors (eigenvalue > 1) (Table 3).

TABLE 3

Confirmatory factor analysis

	PSSQ (3 factors)	PSSQ (4 factors)	Acceptable range
p‐Value, chi‐square	<.01	<.01
RMSEA	0.125	0.108	<0.06
CFI	0.82	0.87	≥0.9
TLI	0.804	0.856	≥0.9
SRMR	0.058	0.041	≤0.08
CD	0.994	1	≥0.8

Abbreviations: CD, coefficient of determination; CFI, Comparative Fit Index; PSSQ, Patient Satisfaction Survey Questionnaire; RMSEA, Root Mean Square Error of Approximation; SRMR, Standardized Root Mean Square Residual; TLI, Tucker–Lewis Index.

With the discovery of new factor structure, the CFA was performed again. The results showed slight improvement in the goodness‐of‐fit indices and chi‐square (p < .01). Cronbach's alpha indices for all dimensions were >0.7, which reveals excellent internal consistency (Hu & Bentler, 1999; Table 3).

In this study, CFA identified ‘communication with physician’ as the fourth factor. It affirmed the four‐factor model, which comprised (1) general satisfaction; (2) convenient appointment scheduling; (3) preference for seeing APNs; and (4) communication with physician.

Test–retest reliability

Cronbach's alpha ranged from 0.75 to 0.96 for the three factors (general satisfaction, communication and scheduling), indicating good internal consistency, and average inter‐item correlation ranged from 0.47 to 0.72. The total score had a Cronbach's alpha of 0.957 and item‐total correlation from 0.20 to 0.88 (Table 4). The Cronbach's alpha for the four factors ranged from 0.71 to 0.98, which showed good and excellent internal consistency.

TABLE 4

Internal consistency and test–retest reliability

PSSQ Construct	Cronbach's alpha	Item–total correlation	Ceiling effect (1st)	Ceiling effect (2nd)	ICC
Overall	0.957	(0.203–0.883)	16.7	14.6	0.717
General satisfaction	0.963	(0.44–0.887)	38.8	39.1	0.697
Communication	0.762	(0.416–0.64)	34.9	28.6	0.729
Scheduling	0.771	(0.538–0.632)	51.2	43.8	0.703

Abbreviations: ICC, intra‐class correlation; PSSQ, Patient Satisfaction Survey Questionnaire.

The indices of the three‐factor PSSQ indicated good reliability and stability of the instrument with ICC ranging from 0.697 to 0.729. Similarly, the four‐factor structure also alluded to good test–retest reliability, as ICC ranged from 0.664 to 0.825 (Table 5). Table 6 shows good factor loadings >(=0.4) of the four‐factor model using PCA and varimax rotation.

TABLE 5

Intra‐class correlation (ICC) coefficients for the four factors

PSSQ construct	ICC	An ICC ≥ 0.7 indicates excellent reliability
General satisfaction	0.664	ICC = >0.4 to <0.75 indicates fair to good reliability
Preference for seeing APN	0.680
Communication with physician	0.825
Convenient appointment scheduling	0.696

Abbreviations: APN, advanced practice nurse; ICC, intra‐class correlation; PSSQ, Patient Satisfaction Survey Questionnaire.

TABLE 6

Good factor loadings, with factor loadings ≥0.4, for the four factors of variables using principal component analysis and varimax rotation

Item number	Content	Factor a (component)	Loading
10	APN is a skilled healthcare provider (Q10)	1: General satisfaction	0.62
11	APN discusses methods other than medication to treat my problem (Q11)	1: General satisfaction	0.732
12	Satisfied with how the APN treated me (Q12)	1: General satisfaction	0.761
13	Satisfied with the amount of time APN spent with me (Q13)	1: General satisfaction	0.808
14	APN is caring (Q14)	1: General satisfaction	0.753
15	APN is knowledgeable about health problems (Q15)	1: General satisfaction	0.786
16	I trust my APN (Q16)	1: General satisfaction	0.838
17	APN Knows when to refer to a senior physician/specialist (Q17)	1: General satisfaction	0.666
18	APN listened to what I had to say (Q18)	1: General satisfaction	0.738
19	APN interested in my health concerns (Q19)	1: General satisfaction	0.775
20	APN respected me (Q20)	1: General satisfaction	0.818
21	I can easily talk to the APN about my health concerns (Q21)	1: General satisfaction	0.831
22	I understood what the APN explained to me (Q22)	1: General satisfaction	0.85
23	I understood what the APN taught me (Q23)	1: General satisfaction	0.87
24	The APN explained things in an understandable manner (Q24)	1: General satisfaction	0.798
25	I feel comfortable Asking APN questions (Q25)	1: General satisfaction	0.835
27	I left APN visit with all questions answered (Q27)	1: General satisfaction	0.762
1	Overall, I was satisfied (Q1)	2: Convenient appointment scheduling	0.785
2	Recommend the APN to others (Q2)	2: Convenient appointment scheduling	0.745
3	Likely to schedule appointments with the APN in the future (Q3)	2: Convenient appointment scheduling	0.699
4	APN was not rushed (Q4)	2: Convenient appointment scheduling	0.68
6	Schedule a convenient appointment with the APN (Q6)	2: Convenient appointment scheduling	0.49
8	Polyclinic is easy to access (Q8)	2: Convenient appointment scheduling	0.552
5	Would rather see the APN than my regular physician (Q5)	3: Preference for seeing APNs	0.764
7	When I feel the need to see a healthcare provider, I can get an appointment with the APN without problem (Q7)	3: Preference for seeing APNs	0.683
9	Scheduling an appointment with the Polyclinic APN is easier than scheduling with my usual physician (Q9)	3: Preference for seeing APNs	0.711
26	I feel comfortable asking my personal physician questions (Q26)	4: Communication with physician	0.895
28	I usually leave my personal physician's visits with all questions answered (Q28)	4: Communication with physician	0.879

Abbreviation: APN, advanced practice nurse.

1: General satisfaction; 2: Convenient appointment scheduling; 3: Preference for seeing APNs; 4: Communication with physician.

DISCUSSION

The results reveal the four constructs, which underpin the satisfaction with the APN services: general satisfaction; convenient appointment scheduling; preference for seeing APNs; and communication with physician. Communication with physician was identified as the new fourth factor. The findings alluded to the four‐factor model, which seemed to be statistically superior to the original three‐factor model for our local setting for its excellent internal consistency and good test–retest reliability.

The psychometric properties of the PSSQ appear to be equivalent to NPSS after the minor adaptation. It can be used to assess nursing service for outpatients in primary care, even for those in the geriatric age groups. Compared with other outpatient satisfaction questionnaires (OPSQs) (Goel, Sharma, & Singh, 2014; Khamis & Njau, 2014), the PSSQ in simple English seems to be easily understood by the older patients. Aside from the usual pen‐and‐paper survey, the PSSQ can be digitalized for electronic form of surveys at the polyclinics. The digital format can also be easily administered via smart handphones. The instrument can be used to assess services rendered by other categories of nursing staff, such as nurse clinicians and registered nurses, such as those in mother and child, cancer screening and mental health services.

The results show the desire of patients to communicate with their physician when the need arises. A physician–APN team‐based approach seemed to be the direction forward to manage these patients, especially for those with multiple NCDs, considering their preference to access both providers. Further research is needed to assess this team‐based approach and the resultant levels of patient satisfaction.

The PSSQ is likely to be the first nursing assessment tool to be validated in the English‐literate Asian population. The strength of this study is also reflected in the low dropout rate during the retest segment of the PSSQ. However, the different method used in the test (self‐administration of the questionnaire or assisted by the research assistants) and retest (telephone survey) may potentially confound the results. The limitation is mitigated by deploying the same nursing student to assist the participant in administering the questionnaire during the test and to conduct the subsequent retest by phone interview. This approach was used by the investigators in earlier validation studies of other instrument, which was accepted by the academic fraternity (Koh et al., 2016; Ngoh, Lim, Koh, & Tan, 2017; Seow, Mohamed Yusoff, Koh, & Tan, 2017).

The higher participation rates during the test segment of the study could be related to patients having a sense of obligation to the APNs. The deployment of nursing students from an external nursing school as independent research assistants was a deliberate measure to gather objective responses to the APN services. Nonetheless, participants who used APN services repeatedly could still inflate the performance score in PSSQ due to acquiescence bias or gratitude towards APNs.

In addition, the PSSQ availed to the patients only in the English version. While English is the main official language in Singapore and that English literacy in the population (aged 15 years and older) ranges from 52.5% of Indians to 81.4% of the Malay population, a subset of the older patients may not be able to make use of the instrument. Translation of the PSSQ to the Mandarin, Malay and Tamil languages will be the next step, and psychometric evaluations of these Asian editions will ensue.

CONCLUSION

The PSSQ has good internal consistency and test–retest reliability. The four‐factor model seems to be better suited for the PSSQ, compared with the three factors in the original NPSS. The PSSQ can be extended to assess other nursing services in primary care. Further research is needed to develop the PSSQ in local Asian languages and to assess its routine implementation in the polyclinics.

ETHICS APPROVAL

The study was approved by the SingHealth Centralised Institutional Review Board (CIRB#2018/2134).

Appendix S1

Click here for additional data file.