Validation of a Patient Questionnaire Assessing Patient Satisfaction With Orthopedic Outpatient Clinic Consultation.

Previous qualitative research has identified a number of factors which influence patient satisfaction with orthopedic outpatient clinic visits. To further evaluate these factors, the authors initially generated a number of items or statements representing these factors. This cohort of items was then subjected to analysis by an expert group to assess which 3 items best represented each factor. These items formed the basis of a draft survey which was then administered to 323 orthopedic outpatients to assess these factors as characteristics of patient satisfaction. Items and factors were also assessed against 2 global measures of patient satisfaction. One hundred and one survey responses were returned and subjected to factorial analysis. Results indicated that factors of trust, empathy, and relatedness were not distinguishable and subsequently combined to represent a single factor, the therapeutic relationship. A final 5-factor model is proposed incorporating 3 interpersonal factors (communication, expectation, therapeutic relationship) and 2 environmental factors (clinic wait time, clinical contact time). The factors identified by this study should be considered in surveys evaluating patient satisfaction with orthopedic outpatient services.

Background

The understanding of the factors that influence patient satisfaction with outpatient orthopedic consultation is unclear or at least incomplete. Most studies evaluating patient satisfaction with orthopedic clinic consultation are primarily focused on the outcome of clinical intervention (1,2). The findings of these studies support a view that patient satisfaction is largely influenced by clinical outcomes such as pain severity, range of motion, and functional capacity. However, evaluation of clinical outcomes alone may not capture the factors influencing patient satisfaction with orthopedic clinic consultation (3 -6).

Methods of evaluation of patient satisfaction with orthopedic outpatient services appear to be less developed than other clinical services such as general practice (7,8) or physical therapy (9,10). Some researchers have viewed patient satisfaction as a singular entity using global scales of assessment (11,12). This approach limits the capacity to more specifically determine the factors influencing the patients response (13) and is in contrast to both the complexity and multifactorial nature of patient satisfaction (4,14). Recent studies examining the predictors of patient satisfaction with orthopedic clinic consultations have used generic patient satisfaction questionnaires (15,16).

There are examples of specifically developed survey tools to evaluate patient assessment in orthopedic services, however the process used in the development of these surveys appears to be limited. A common approach is the use of professional groups to determine what factors influence patient satisfaction (17,18). The question raised by this approach is whether those factors are likely to reflect the primary concerns of the professional group rather than a patient cohort and may not capture the breadth of factors which influence patient satisfaction. Other studies have modified or derived survey scales and items without any evidence of validation in an orthopedic context (19,20).

Therefore, the aim of this study was to examine the factorial validity and internal consistency of a patient satisfaction survey, developed using feedback obtained from stakeholder feedback (21) through a rigorous framework and development process.

Methods

Stage 1—Expert Panel Review

Thematic analysis from a focus group had previously identified clinic waiting time, clinical contact time, trust, empathy, communication expectation, and relatedness as potentially influential to patient satisfaction with orthopedic consultation (21). To develop a survey tool to evaluate these themes, the authors generated conceptual definitions of these themes. These provisional definitions were then referenced to the existing literature and enhanced where required to ensure consistency. Next the researchers generated a number of statements to capture the necessary and essential attributes of each dimension of patient satisfaction. Several statements were generated for each dimension to ensure adequate breadth and depth of conceptual coverage. In all, 68 items were generated through this process. Consistent with recommendations for test construction (22), a panel of 8 academic experts was then recruited to provide their perceptions of the degree to which the items were representative of the target construct domain. All participants within the expert group had at least 10 years of experience within their area of professional practice and salient postgraduate qualifications. None of the participants within the panel participated in the focus groups undertaken by Waters et al (21), and none of the authors participated within the expert panel. Each item was reviewed by the expert panel for clarity and relevance against the proposed thematic definition. The expert panel was then asked to rate the quality of the conceptual definitions (eg, comprehensibility) and each item statement for clarity against the proposed thematic definition. The clarity of the definitions and item statements was assessed within a trinomial response of yes, no, or unsure. The relevance of particular statements to the intended domain of patient satisfaction was rated across a 5-point response scale where 0 was very poor and 4 was very good. These ratings were then tabled against a framework to capture the responses of the contributing expert.

Experts’ ratings of the adequacy of the content domain sampling were then subjected to statistical analysis to ascertain the level of agreement (r wg) (23). Items were considered for retention when inter-rater agreement (r wg) was greater than 0.80, and the average rating score was >3. This statistical analysis was complemented by qualitative feedback from the experts in terms of substantive, for example, relevance to intended dimension (24); and grammatical, for example, comprehension, aspects of the statements that could be strengthened (25). The aim was to retain 3 items for each factor based on the statistical criteria and substantive content so as to ensure an adequate breadth of content domain (26).

Stage 2—Questionnaire Development and Evaluation

The retained items were then tested for their factorial structure in an independent sample. At initial presentation to an orthopedic outpatient clinic, each patient was asked for consent to participation by an assistant who was unaffiliated with the study. The study requirements and rationale were explained to potential participants. Patients who agreed to participate were given written information about the study and asked to provide written consent. Clinicians were blinded as to whether patients were study participants. The survey was posted to the patient immediately after the initial consultation along with a prepaid return envelope. No patient identification was possible from the returned surveys. Participation in the study did not change the clinical management of the patient. All items were rated on a 7-point scale (1 = strongly disagree, 2 = disagree, 3 = disagree slightly, 4 = neutral, 5 = slightly agree, 6 = agree, 7 = strongly agree), with the exception of the 1 clinic waiting time item (“Please indicate how long you believe you waited for the clinician”: 1 = 3 hours or greater, 2 = 2 hours, 3 = 1 hour, 4 = 30 minutes, 5 = 15 minutes, 6 = 15 minutes). Factorial validity and internal reliability evidence (ω) (27) of the patient satisfaction scale were examined using confirmatory factor analysis with a robust maximum likelihood estimator. Missing data were handled with full-information maximum likelihood (28). The resultant priori model of interest was a lower-order, 6-factor structure that is consistent with the qualitative findings of Waters et al (21). That is, a latent factor for each for the factors identified, time with clinician, empathy, communication, expectation, trust, and relatedness. Each retained item was assessed by a factor loading indicating the strength of the association between the item and the factor it represents. Alternative configurations were considered on an ad hoc basis where appropriate (eg, poor model-data fit, model convergence issues). Model-data fit was assessed using established indices, namely the χ2 goodness-of-fit index, comparative fit index (CFI), Tucker-Lewis index (TLI), and root mean square error of approximation (RMSEA). According to typical interpretation guidelines for adequate or acceptable model-data fit (29), values of CFI/TLI ≥ .90 and RMSEA ≤ .06 (with the upper bound of the 90% RMSEA confidence interval ≤ .10) provide evidence of adequate or acceptable overall fit. These analyses were performed in Mplus 7.4 (30).

As an initial first look into convergent validity evidence, scores on patient satisfaction were correlated with global satisfaction (“How would you rate your overall level of satisfaction with the orthopedic service”; 1 = strongly dissatisfied to 7 = highly satisfied) and service recommendation (“How likely would you be to recommend the orthopedic service to a friend or family”; 1 = highly unlikely to 7 = highly likely) with a structural equation modelling framework. Four group associations within the data were examined. For continuous variables such as age and duration of condition, regression analysis was undertaken against each of the 5 factors retained. The strength and direction of the association was assessed by a correlation coefficient. For discrete data such as gender and site of data collection, differences were assessed using analysis of variance (1-tailed) and Welch’s t test for unequal variance (31) with a significance threshold of P < .05. All these analyses were performed using SPSS.

Participants

Patients were recruited from referrals to the orthopedic outpatient clinics at Fremantle Hospital Health Service (FHHS) and Sir Charles Gairdner Hospital (SCGH) in Perth, Western Australia. Patients with a compensable injury, acute trauma, malignancy, fracture, previous surgery related to the referred condition, or previous review of the some condition within the past 6 months were excluded from the study. Patients whose communication skills did not allow comprehension of the consent form or ability to complete the survey were also excluded. All participants within the expert group and the patient group offered informed and written consent to participation.

Results

Participant Profile

The total number of surveys sent to patients across both sites was 323 (FHHS = 173, SCGH = 150), with 101 surveys completed and returned (31.3%). The average age of respondents was 48.9 (standard deviation = 14.3) years, with 34% younger than 45 years; 71% of the respondents were female participants, and patients with chronic conditions (>6 months duration) accounted for 66.3% of the cohort. The clinical profile of the survey respondents from the orthopedic clinic was 3.0% spine, 33.7% shoulders, 6.9% upper limb, 9.9% hip/pelvis, 28.7% knee, and 17.8% lower limb.

Adequacy of the Content Domain

Of the original 68 items, the experts indicated that 25 of these statements were relevant for the content domain of the intended target (r wg > .80; M ratings > 3). The experts offered several minor suggestions to enhance the grammatical and conceptual precision of several of the statements that met these statistical criteria. Consideration of the statistical criteria and substantive content resulted in the retention of 21 items (Appendix A).

Item Level Analyses

Missing responses at the item level represented 1.93% of all available data. Survey responses with greater than 15% of missing data on the 21 patient satisfaction items were excluded from the main analyses (n = 3). Thus, the analyses were performed on 98 survey responses. Item level descriptive statistics for the 21 survey items are displayed in Appendix B. The distribution of each item was considered normal based on threshold skewness and kurtosis values of approximately −2.0 and +2.0.

Factorial Structure

An analysis of the lower-order, 7-factor structure represented a good fit with the data, χ2 168 = 280.383, P <.001, CFI = .935, TLI = .919, RMSEA = .082 (90% CI = .065-.099), however, this model was flagged as not positive definite, owing to a correlation between empathy and trust that was greater than 1. Additionally, the latent variable correlations between relatedness and trust (.99) and between relatedness and empathy (.98) approached 1, thereby indicating minimal distinction between these factors. These 3 factors were collapsed as a single latent variable in subsequent analyses, referred to hereafter as therapeutic relationship. The lower-order, 5-factor structure represented a good fit with the data, χ2 179 = 228.97, P < .001, CFI = .937, TLI = .926, RMSEA = .079 (90% CI = .062-.095). Standardized factor loadings and internal reliability estimates (ω) for the final 21-item, 5-factor solution are summarized in Table 1. Item-level descriptive statistics are contained in Appendix B. Descriptive statistics for group associations across factors are given in Appendix C.

Table 1.

Standardized Factor Loadings.

	Item	Therapeutic relationship	Time	Communication	Expectations	Clinic waiting time
1	I felt as though I could trust the clinician who assessed me to do what is best for my condition.	.932
2	I trusted that the clinician has the expertise to identify the best solution for my circumstances.	.910
3	I trusted that this clinician will be able to identify alternative solutions if my circumstances change.	.930
4	The clinician took the time to make me feel at ease.	.928
5	The clinician demonstrated a willingness to understand my concerns.	.982
6	I felt like the clinician was genuinely interested in how my problem was affecting my life.	.938
7	When I arrived, the clinician introduced themselves by name.	.819
8	I got the sense that the clinician I saw respected me as a person.	.953
9	I got the sense that the clinician cared about me rather than seeing me as just another patient.	.929
10	I felt like the clinician dedicated enough time to me during the consultation.		.968
11	I had enough time with the clinician to ask all the questions I had about my condition.		.867
12	The clinician did not seem like s/he was rushed during my consultation.		.400
13	The clinician sought my input during the consultation process.			.902
14	The clinician explained medical terms in a language that easy for me to understand.			.912
15	It was easy to talk to the clinician who assessed me.			.955
16	My experience in the clinic was better than I expected.				.826
17	I expected that more could have been done to assess my problem.				.649
18	My expectations were taken into consideration in the consultation process.				.890
19	Please indicate how long you believe you waited for the clinician					.884
20	I was comfortable with the time I had to wait.					.683
21	I was kept waiting too long to see the clinician.					.699
	ω	.98	.81	.94	.83	.80

Latent factor correlations supported low-to-moderate positive associations between clinical time and therapeutic relationship (r = 0.24, P = .02), clinical time and expectations (r = 0.39, P < .001), clinic waiting time and time (r = 0.47, P < .001), and clinic waiting time and communication (r = 0.28, P = .04). Moderate to high correlations were identified between time and communication (r = 0.63, P < .001) and between expectations and therapeutic relationship (r = 0.82, P < .001). Nonsalient associations were noted between therapeutic relationship and communication (r = .07, P = .502), expectations and communication (r = .14, P = .225), clinic waiting time and therapeutic relationship (r = .09, P = .468), and clinic waiting time and expectations (r = .23, P = .080).

Convergent Validity Evidence

An analysis of the lower-order, 5-factor structure together with observed scores for global satisfaction and service recommendation represented a good fit with the data, χ2 211 = 329.41, P <.001, CFI = .937, TLI = .925, RMSEA = .075 (90% CI = .059-.091). Global satisfaction was associated with higher levels of therapeutic relationship (r = 0.26, P = .005), clinical time (r = 0.49, P < .001), communication (r = 0.48, P < .001), expectations (r = 0.38, P < .001), and clinic waiting time (r = 0.35, P < .001). Similar associations were observed for service recommendation and the patient satisfaction dimensions: therapeutic relationship (r = 0.21, P = .03), time (r = .48, P < .001), communication (r = 0.45, P < .001), expectations (r = 0.36, P < .001), and clinic waiting time (r = 0.35, P < .001). Global satisfaction and service recommendation were strongly correlated (r = 0.87, P < .001).

Discussion

This study, along with our previous research (21), sought to develop a conceptual model of, and validity evidence for, a scale assessing patient satisfaction with orthopedic clinic consultation. Unlike previous studies that have assessed outpatient orthopedic satisfaction, the development of the scale was guided by an inductive, bottom-up approach. Qualitative analysis was used to develop the scale and evaluate the construct validity. Factor validity evidence was assessed through an independent sample with a preliminary assessment of the psychometric properties of the scale.

The results provide an insight into the factors which contribute to orthopedic outpatient clinic satisfaction. A final 5-factor model was established incorporating 3 interpersonal factors (communication, expectation, and the therapeutic relationship) and 2 environmental factors (clinic wait time and clinical contact time). Support was found for willingness to recommend as a global assessment of patient satisfaction. The identification of these factors informs our understanding of the characteristics of patient satisfaction within the context of orthopedic clinic services. This knowledge may inform clinical service design, and the training of orthopedic clinicians looking to understand the clinical interaction beyond the provision of technical prowess.

The results of this study did not support a clear distinction between factors of empathy, trust, and relatedness, a basic psychological need that must be satisfied for people to experience positive development and outcomes (32), which were subsequently collapsed into a single entity of the therapeutic relationship. Conceptually, these factors are similar or at least retain similar descriptors (15,32 -35). Although the development of the survey items was supported by conceptual definitions by an expert group, and the published literature, the results suggest that the patients did not clearly differentiate empathy, trust, and relatedness. While apparent to the expert group, the conceptual subtleties of these factors, and the items generated, were less apparent to the patient cohort. More work will be required to define the therapeutic relationship within a clinical encounter and, in particular, what items or elements may best represent this concept to patients.

Clinic waiting time is included as a factor in a number of patient satisfaction survey tools (36,37) and is recognized as influencing patient satisfaction in orthopedic settings (12,38). Clinic waiting time was moderately correlated with both clinical contact time and communication, such that as waiting time increased, satisfaction with clinical contact time and communication decreased. It is possible that patient experience of waiting time may influence the perception of communication and clinical contact time. Conversely, the time spent with the clinician may counteract any negative influence of waiting time (39).

The results of the present study support the premise that patient expectation influences patient satisfaction, particularly with regard to the process of clinical assessment. This finding is supported by other studies evaluating the role of expectation in orthopedics (20,40,41). Expectation was strongly correlated with the therapeutic relationship suggesting that demonstrations of trust, empathy, and relatedness are interrelated with patients’ expectations of the orthopedic clinic consultation.

The limitation of this study is that the sample size is small relative to typical approaches to scale development in the health sciences. The low survey response rate may have produced some selection bias within the participant cohort. However, the response rate is consistent with those reported in previous orthopedic patient satisfaction studies (12). Although the sample provided calibration in relation to factor analysis and model fit, an argument could be made that the resultant model is data driven or influenced largely by characteristics of the responders.

To establish the psychometric properties of the survey scale, further development will be needed. Male et al (42) proposed 5 criteria to evaluate the performance of a survey instrument through convergent validity, discriminant validity, predictive validity, test–retest validity, and responsiveness. These 5 criteria set a road map for development of the scale. Phone follow-up and mix mode technologies may also improve the response rate from participants (43).

Conclusion

Support is found for a 5-factor model of patient satisfaction within orthopedic consultation: Three interpersonal factors (communication, expectation, therapeutic relationship) and 2 environmental factors (clinic wait time, clinical contact time).

Table A1.

Items Retained Within Draft Survey.

Draf item no.	Survey item no.	Theme/domain, n = 7	Communication	Trust	Relatedness	Sufficient clinica time	Empathy	Wait time	Expectation
		Statement
1	1	Please indicate how long you believe you waited for the clinician						1
2	2	I was comfortable with the time I had to wait.						2
5	3	I was kept waiting too long to see the clinician						5
3	4	I felt like the clinician dedicated enough time to me during the consultation.				3
4	5	I had enough time with the clinician to ask all the questions I had about my condition				4
7	6	The clinician did not seem like s/he was rushed during my consultation.				7
27	7	The clinician sought my input during the consultation process	27
29	8	The clinician explained medical terms in a language that easy for me to understand.	29
30	9	It was easy to talk to the clinician who assessed me	30
12	10	The clinician took the time to make me feel at ease.					12
20	11	The clinician demonstrated a willingness to understand my concerns.					20
24	12	I felt like the clinician was genuinely interested in how my problem was affecting my life					24
15	13	When I arrived, the clinician introduced themselves by name.			15
17	14	I got the sense that the clinician I saw respected me as a person			17
18	15	I got the sense that the clinician cared about me rather than seeing me as just another patient			18
11	16	I felt as though I could trust the clinician who assessed me to do what is best for my condition.		11
13	17	I trusted that the clinician has the expertise to identify the best solution for my circumstances.		13
25	18	I trusted that this clinician will be able to identify alternative solutions if my circumstances change		25
32	19	My experience in the clinic was better than I expected.							32
33	20	I expected that more could have been done to assess my problem.							33
34	21	My expectations were taken into consideration in the consultation process.							34
		n = 21	3	3	3	3	3	3	3

Table B1.

Descriptive Statistics for Item Responses.

	Item	N	Minimum	Maximum	Mean	Standard deviation	Skewness		Kurtosis
		Stat.	Stat.	Stat.	Stat.	Stat.	Stat.	Standard error	Stat.	Standard
1	I felt like the clinician dedicated enough time to me during the consultation.	97	1.00	7.00	5.93810	1.42752	−1.600	0.245	1.933	0.485
2	I had enough time with the clinician to ask all the questions I had about my condition.	98	1.00	7.00	5.83670	1.58403	−1.568	0.244	1.567	0.483
3	The clinician did not seem like s/he was rushed during my consultation.	98	1.00	7.00	5.07140	1.96490	−0.768	0.244	−0.751	0.483
4	The clinician sought my input during the consultation process.	96	1.00	7.00	5.71880	1.58747	−1.410	0.246	1.328	0.488
5	The clinician explained medical terms in a language that easy for me to understand.	96	1.00	7.00	5.85420	1.47241	−1.581	0.246	2.151	0.488
6	It was easy to talk to the clinician who assessed me.	96	1.00	7.00	5.92710	1.62380	−1.793	0.246	2.165	0.488
7	The clinician took the time to make me feel at ease.	98	1.00	7.00	5.31630	2.01334	−1.087	0.244	−0.198	0.483
8	The clinician demonstrated a willingness to understand my concerns	97	1.00	7.00	5.23710	2.01956	−1.044	0.245	−0.217	0.485
9	I felt like the clinician was genuinely interested in how my problem was affecting my life.	98	1.00	7.00	5.12240	2.09696	−0.912	0.244	−0.563	0.483
10	When I arrived, the clinician introduced themselves by name.	98	1.00	7.00	5.76530	1.94142	−1.559	0.244	1.018	0.483
11	I got the sense that the clinician I saw respected me as a person.	98	1.00	7.00	5.45920	1.99571	−1.325	0.244	0.356	0.483
12	I got the sense that the clinician cared about me rather than seeing me as just another patient.	97	1.00	7.00	5.18560	1.95432	0.933	0.245	−0.405	0.485
13	I felt as though I could trust the clinician who assessed me to do what is best for my condition.	98	1.00	7.00	5.35710	2.06218	−1.135	0.244	−0.166	0.483
14	I trusted that the clinician has the expertise to identify the best solution for my circumstances.	98	1.00	7.00	5.50000	1.90631	−1.404	0.244	0.740	0.483
15	I trusted that this clinician will be able to identify alternative solutions if my circumstances change.	98	1.00	7.00	5.21430	2.02179	−1.071	0.244	−0.186	0.483
16	My experience in the clinic was better than I expected.	97	1.00	7.00	4.90720	1.92073	−0.712	0.245	−0.658	0.485
17	I expected that more could have been done to assess my problem.	97	1.00	7.00	4.87630	1.88888	−0.595	0.245	−0.812	0.485
18	My expectations were taken into consideration in the consultation process.	97	1.00	7.00	4.89690	1.90659	−0.706	0.245	−0.654	0.485
19	Please indicate how long you believe you waited for the clinician.	94	1.00	6.00	4.27660	1.23918	−0.234	0.249	−0.664	0.493
20	I was comfortable with the time I had to wait.	99	1.00	7.00	5.41410	1.75552	−1.084	0.243	0.185	0.481
21	I was kept waiting too long to see the clinician.	98	1.00	7.00	5.56120	1.78231	−1.129	0.244	0.133	0.483
	Valid N (listwise)	86

Table C1.

Descriptive Statistics for Group Associations Across Factors.

	Therapeutic relationship		Communication		Expectation		Clinic wait time		Clinical contact time
Statistical measure	β	SE	β	SE	β	SE	β	SE	β	SE
Age	−0.14	0.134	−0.076	0.097	0.11	0.137	0.09	0.115	0.012	0.112
Duration of symptoms	−0.110	0.148	0.134	0.111	−0.129	0.155	−0.012	0.136	0.097	0.067
Gender	−0.039	0.103	0.125	0.12	−0.010	0.111	−0.107	0.101	0.045	0.097
Site	−0.082	0.129	0.058	0.181	−0.077	0.154	0.055	0.175	−0.221a	0.079

Abbreviation: SE, standard error.

Boldface values are statistically significant.