The association between GP and patient ratings of quality of care at outpatient clinics.

BACKGROUND: GPs and patients are frequently asked to evaluate mental health care, but studies including evaluations from both groups are rare.
OBJECTIVE: To assess the association between GPs' and patients' assessment of mental health outpatient clinic in Norway and identify important health care predictors for patient and GP satisfaction with the clinics.
METHODS: Two cross-sectional national surveys were carried out: survey of GPs in 2006 and patients in 2007 evaluating outpatient clinics at 69 community mental health centres in Norway. A total of 2009 GPs and 9001 outpatients assessed the clinics by means of a postal questionnaire. Main outcome measures were correlations between GP and patient ratings of the outpatient clinics at the clinic level and health care predictors for patient satisfaction and GP satisfaction with the clinics.
RESULTS: Clinic scores for GPs' and patients' assessment of waiting time were moderate to highly correlated (0.65), while clinic scores for GP and patient satisfaction had a lower but significant positive association (0.37). Significant positive correlations between clinic scores for GP and patients ratings were found for 38 of the 48 associations tested. The most important predictors for patient satisfaction with the clinics were interaction with the clinician (beta: 0.23) and being met with politeness and respect at the clinic (beta: 0.19), while the most important predictors for GP satisfaction with the clinics were perceived competence (beta: 0.25), rejection of referrals (beta: -0.17) and waiting time for patients (beta: -0.16).
CONCLUSIONS: A consistent positive association between GP and patient ratings at the clinic level was identified. Mental health services aiming at improving GP and patient satisfaction should be sensitive to the fact that the two groups prioritize different health care factors.

Introduction

It is widely recognized that GPs' views are important in the evaluation of mental health care services.1 Together with frequently conducted patient experiences or user satisfaction studies,2,3 such evaluations give insight into how the health care system functions fromthe perspectives of important target stakeholders. However, the validity of patient satisfaction or experiences studies is debated,2,4,5 and evidence for an association with other widely used measures of institutional performance would strengthen the validity of such studies.6 Since health professionals normally are better qualified to evaluate the technical or professional aspects of health care than patients, one fruitful approach may be to compare patient evaluations and GP or other referrer evaluations of mental health care institutions.

Systematic searches of the literature revealed few studies on the association between ratings based on user evaluation and referral evaluation, and the studies found had varying quality.5,7–11 In a study that compared patient satisfaction and other quality indicators,5 it was found that patient satisfaction was more closely associated with the results of an independent review than other quality indicators including referral satisfaction. However, the sample was small for each of the five service units, and no statistical tests were conducted to assess the associations at the individual or service level. In another study of consumer satisfaction at an inpatient child psychiatric unit,7 a low-to-moderate correlation was found between parent satisfaction and referral satisfaction. However, different data collection methods for parent satisfaction and referral satisfaction make it hard to interpret results. The study was also small and included one psychiatric unit. Other referrer and user evaluation studies have also been restricted to one unit and have not included an assessment of the strength of associations between the groups evaluation of the unit.8–11

In our study, we examine the association between GPs and patients assessment of community mental health outpatient clinics in Norway. GPs act as spokesmen for patients in addition to being users of mental health services by referring patients and receiving supervision, and the need for better communication between psychiatric services and GPs has been stressed by others.12 Due to lack of knowledge about the association between patient experiences and GP experiences, the study was exploratory without formal hypothesis about the strength of association. Our first research question is concerned with whether clinics with positive patient assessments also have positive GP assessments. First, we compare clinic scores in two directly comparable areas. General satisfaction and waiting time for patients are measured in the same manner in the two national surveys and can be expected to be correlated because of similar content. Second, we assess the association between clinic scores for GP and patient evaluation across a range of variables. The focus in this part of the analysis is not each separate association but rather to assess if there is a systematic pattern of positive and significant associations between clinic scores for patient and GP evaluations across a range of aspects of health care. Our second research question concerns health care predictors of general satisfaction for both GPs and patients. The objective of this aspect of the analysis is to provide relevant information for quality improvement work aimed at increasing patient satisfaction and/or GP satisfaction.

Methods

Sample

All community mental health centres in Norway were included in both national surveys (n = 77). In the GP study, four centres were excluded because they had been assessed in the pilot study. Six centres reported in the GP study were merged to three institutions to give concurrence between the institutional lists in the two surveys. One centre was excluded because there were of too few respondents in both surveys. Hence, 69 community mental health centres were included in the study.

The GP sample was all GPs registered in Norway who filled in a questionnaire for evaluation of the community mental health centre in their area in a postal survey in January 2006. The questionnaire was answered by 2115 (61%) GPs. Of these, 106 GPs were excluded because a filter question showed that they did not evaluate the outpatient clinic at the community mental health centre. Consequently, our sample comprises 2009 GPs. The national sample has been shown to be representative.13 There were small differences between respondents and non-respondents in relation to demographic and practice variables, and the background variables that were significantly different were not at all or were only weakly related to the main outcome variables. The procedure regarding informed consent, study design and data collection was approved by the Norwegian Social Science Data Services.

The postal patient survey included adults with one or more registered contacts with a mental health outpatient clinic in Norway between August 20 and November 11, 2007. We drew a probability sample of 400 patients for each institution or included all eligible patients in the period if the number of patients were <400. Clinics for elderly or substance abuse clinics were excluded. In our study, patients attending the outpatient clinics at the community mental health centres from the GP survey were included (n = 69 centres). The questionnaire was answered by 9001 (40%) patients. The respondents differed from non-respondents in relation to age, gender, diagnosis and the number of consultations in the inclusion period. To assess non-response bias, we randomly selected 293 postal non-respondents from 10 randomly selected clinics to be included in a telephone follow-up. A total of 110 patients answered a shorter version of the postal questionnaire, including the 11 core items of The Psychiatric Out-Patient Experiences Questionnaire.14 The difference between telephone respondents and postal respondents on the Psychiatric Out-Patient Experiences Questionnaire (POPEQ)-11 scale was small and insignificant (2 on a scale from 0 to 100), indicating little non-response bias. The Norwegian Regional Committee for Medical Research Ethics, the Data Inspectorate and the Norwegian Directorate of Health and Social Affairs approved the survey.

Materials

The General Practitioner Experiences Questionnaire is described in detail elsewhere1 and comprises the following scales with good evidence for data quality, reliability and validity: workforce situation (four items), discharge letter (three items), competence (four items), guidance (three items) and emergency situations (two items). Workforce situation includes items relating to stability in key positions and doctor coverage. Discharge letter includes items relating to quality, further plans and discharge letter time. Competence includes items relating to assessment and treatment skills. Guidance includes items relating to cooperation meetings, organized training and receiving necessary professional support. Emergency situations includes items relating to contact with and help from the centre in emergency situations. All scales met the criterion of 0.7 for Cronbach's alpha and test–retest correlations were 0.72–0.87. The results of construct validity testing were as hypothesized. To secure the comparability between GPs and patients, we included items with similar content from the two questionnaires. For GPs single items related to general satisfaction, perceived waiting time for the patients, and contact with the centre in situations where the GP need advice. The reliability of the five scales is unaffected by the inclusion of the single items. The reliability of single items is poorer than for scales, but following the objective of this study comparability was prioritized. All items have a five-point response format.

The POPEQ is described in detail elsewhere14 and comprises one unidimensional scale with the three following subscales: clinician interaction (6 items), information (2 items) and outcome (3 items). All scales met the criterion of 0.7 for Cronbach's alpha and test–retest correlations were 0.75–0.90. The results of construct validity testing were as hypothesized. As a result of better face validity and psychometric results in 2007, two of three items within the outcome scale were replaced with two new items. The revised subscale includes three questions about outcome related to improvement in health problem, functioning in the family and functioning outside the family, all compared to pretreatment at the clinic. We also included items of similar content as in the GP questionnaire: general satisfaction, assessment of waiting time and ease of getting in touch with the clinic by telephone. All items have a five-point response scale, except waiting time, which has a four-point response scale.

All scales and items were transformed to scores ranging from 0 to 100 where 100 is the best possible rating. For computation of scale score, we set the lowest item response to 0, the highest to 4. We summed these values for individual questions within a scale and transformed them into percentage scores. Patients with missing values on more than half of the items in a scale were excluded. The single items were directly transformed from a five-point scale to a 0–100 scale; 1 = 0, 2 = 25, 3 = 50, 4 = 75 and 5 = 100.

The size of the catchment area and the number of consultations in 2005 for each centre were extracted from the SAMDATA reports with the 2005 national statistics for the mental health services.15 The Norwegian Patient Register provided data about waiting time.

Analysis

The main unit of analysis in this study is outpatient clinic. We present descriptive statistics for the outpatient clinics; mean, SD and range for clinic size, number of inhabitants in the catchment area, size of GP and patient sample and for the scales and items from the evaluations by GPs and patients. Individual scores for all GP and patient evaluations are given at the clinic level. Correlations between GP variables and patient variables at the clinic level were assessed by Pearson's r. Pearson's r can be viewed as an indicator that describes a linear interdependence between two variables, ranging from −1 to 1. It has been argued that Pearson's r is robust also at the ordinal level, at least when there are more than two response alternatives.16 Moreover, several of the variables comprise summated rating scales based on more than one item. All correlation estimates were tested for statistical significance (two tailed). Because of the exploratory nature of our study, we chose not to adjust for multiple testing.17 All analyses were conducted within SPSS15.0.

An earlier multilevel study has shown that there is significant variation at the centre level for all scales measuring GP experiences with the centres.18 Multilevel analysis of the patient experiences data shows less variation between clinics, but most variables in this study also have a significant intraclass correlation coefficient at the clinic level.

We performed two multiple regression analyses to identify health care predictors of GP and patient satisfaction with the clinics. We included 14 health care factors in the analysis of patient satisfaction with the clinics and 12 health care factors in the analysis of GP satisfaction. We used a stepwise procedure to identify significant predictors.

Results

The highest score for GP assessment of the clinics was on the competence scale with 54.9 on a scale from 0 to 100 where 100 represents the best experience (Table 1). The lowest score for GP assessment of the clinics was the guidance scale, with a mean score of 32.0, followed by perceived waiting time for patients with 34.2. The largest variation in mean clinic scores was found for perceived waiting time for patients; the best clinic score was 58.3 compared to 6.1 for the clinic with the lowest score. The least variation in clinic scores was found for the competence scale. All correlations between GP variables at the clinic level were significant and of a moderate-to-high level (Table 2).

T

Descriptive statistics for the 69 community mental health centres

	Mean	SD	Range
Number of consultations per centre	10 421	6677	1995–32 545
Number of inhabitants in catchment area per centre	48 618	30 003	9597–175 661
Number of GP respondents per centre	29.1	19.8	3–123
Number of patient respondents per clinic	130.5	35.1	56–322
GP assessment of outpatient clinicsa (n = 2009)
Competence (four items)	54.9	7.3	40.1–76.0
Discharge letter (three items)	51.2	7.8	32.1–73.6
Guidance (three items)	32.0	11.2	11.1–62.3
Emergency situations (two items)	52.5	9.3	32.4–71.9
Workforce situation (four items)	45.2	9.7	25.0–69.7
General satisfaction (one item)	53.6	10.4	29.2–80.3
Waiting time for patients (one item)	34.2	11.2	6.1–58.3
Contact with centre when needing advice (one item)	54.4	8.3	37.5–75.0
Patient assessment of outpatient clinicsa (n = 9001)
Clinician interaction (six items)	68.4	3.2	60.9–75.5
Outcome (three items)	71.4	2.8	65.8–79.0
Information (two items)	60.2	4.9	51.4–73.4
General satisfaction (one item)	72.0	3.7	64.2–79.7
Waiting time (one item)	63.7	6.6	36.2–77.5
Contact on telephone (one item)	72.1	5.1	60.5–82.1

Scores 0–100 where 100 represent best possible experiences.

T

Clinic level correlation of GP and patient evaluations (n = 69 clinics)

	GP variables								Patient variables
	1	2	3	4	5	6	7	8	9	10	11	12	13	14
Competence (GP)	–
Discharge letter (GP)	0.82**	–
Guidance (GP)	0.66**	0.52**	–
Emergency situations (GP)	0.58**	0.44**	0.37**	–
Workforce situation (GP)	0.81**	0.61**	0.49**	0.51**	–
General satisfaction (GP)	0.80**	0.67**	0.56**	0.59**	0.70**	–
Waiting time for patients (GP)	0.61**	0.58**	0.48**	0.50**	0.48**	0.70**	–
Contact with centre when needing advice (GP)	0.79**	0.62**	0.67**	0.66**	0.64**	0.77**	0.63**	–
Clinician interaction (patient)	0.48**	0.35**	0.31**	0.17	0.49**	0.45**	0.33**	0.49**	–
Outcome (patient)	0.40**	0.30*	0.31**	0.23	0.41**	0.50**	0.34**	0.53**	0.78**	–
Information (patient)	0.32**	0.21	0.24*	0.07	0.37**	0.39**	0.29*	0.33**	0.76**	0.63**	–
General satisfaction (patient)	0.38**	0.29*	0.23	−0.01	0.45**	0.37**	0.27*	0.34**	0.79**	0.65**	0.71**	–
Waiting time (patient)	0.29*	0.17	0.20	0.31**	0.25*	0.36**	0.65**	0.37**	0.14	0.22	0.17	0.17	–
Contact on telephone (patient)	0.32**	0.20	0.32**	0.14	0.39**	0.37**	0.38**	0.43**	0.60**	0.51**	0.53**	0.58**	0.22	–

*P < 0.05 (two tailed), **P < 0.01 (two tailed).

The highest score for patient assessment of the clinics was for the item related to telephone contact (72.1). The lowest score for patient assessment of the clinics was the information scale, with a mean score of 60.2, followed by assessment of waiting time with 63.7. The largest variation in mean clinic scores was found for the item related to waiting time; the best clinic score was 77.5 compared to 36.2 for the clinic with the lowest score. The least variation in clinic scores was found for patients’ assessment of outcome, followed by clinician interaction. Except for assessment of waiting time, all correlations between patient variables at the clinic level were significant and of a moderate-to-high level (Table 2).

The content of two of the items in the GP and patient questionnaire were identical: waiting time and general satisfaction. The strongest correlation between clinic scores for GP and patient evaluation in our study was found for perceived waiting time (Table 2), r: 0.65. Figure 1 shows mean clinic scores for the variables related to perceived waiting time. The patient scores are consistently higher than the GP scores, but the two lines show the good agreement between the two ratings at clinic level. The two clinics with the lowest patient score for waiting time were also the clinics with poorest GP assessment of waiting time. These findings were confirmed by actual waiting time. The two poorest rated clinics had the longest mean waiting time in the period May to August 2007, 197 and 110 days, respectively, compared to a national average of 56 days. Correlations between the two other related items in the questionnaires were lower than between clinic score for GP and patient evaluation of perceived waiting time (Table 2). Clinic scores for general patient satisfaction and general GP satisfaction were significantly but only moderately correlated (0.37).

F

GP and patient evaluation of waiting time at the 69 centres (Pearson's r: 0.65). Scores 0–100 where 100 represent the best possible evaluation. Centres sorted ascending by GP scores (blue markers)

Significant correlations between clinic scores for GP and patient assessments were found for 38 of the 48 associations tested (Table 2).

Multiple regression analysis identified nine significant health care predictors for GP satisfaction with the clinics (Table 3). The most important predictors were perceived competence at the clinic (beta: 0.25), rejection of referrals (beta: −0.17) and waiting time for patients (beta: −0.16). Multiple regression analysis identified 11 significant health care predictors for patient satisfaction with the clinics (Table 3). The most important health care predictors for patient satisfaction with the clinics were interaction with the clinician (beta: 0.23) and being met with politeness and respect at the clinic (beta: 0.19).

T

Regression analysis of GP and patient satisfaction on aspects of health care variables

Variable	Beta	Significance
GP satisfaction as dependent variablea
Competence scale	0.25	<0.001
Do the clinic reject patients you have referred?	−0.17	<0.001
Do the patients have to wait to get an offer from the clinic?	−0.16	<0.001
Emergency situations scale	0.13	<0.001
Do the clinic take your opinions of the patients situation serious?	0.13	<0.001
Workforce situation scale	0.11	<0.001
Guidance scale	0.05	0.011
Negative episodes with the centre	−0.04	0.014
Discharge letter scale	0.04	0.043
Patient satisfaction as dependent variableb
Clinician interaction scale	0.23	<0.001
Met with politeness and respect at the clinic	0.19	<0.001
Information scale	0.10	<0.001
Not having the desired health professional (versus other patients)	−0.08	<0.001
Outcome scale	0.08	<0.001
Far too few consultations (versus other patients)	−0.08	<0.001
Waiting time	−0.04	<0.001
Improvement potential regarding the clinics cooperation with other public agencies on practical issues (versus other patients)	−0.03	<0.001
Talked down to or offended	−0.03	0.009
Degree to which treatment was enforced	−0.02	0.012
Improvement potential regarding the clinics cooperation with next of kin (versus other patients)	−0.02	0.033

Adjusted R2 = 59%. Excluded variables: contact when needing advice, feedback from centre during treatment of patients and received information about services at the centre.

Adjusted R2 = 38%. Excluded variables: telephone availability to the clinic, improvement potential regarding influence on medication issues and improvement potential regarding information about medication side effects.

Discussion

The study found that clinic scores for GP and patient assessment of waiting time for patients had a positive high-level association. Clinic scores for GP and patient satisfaction had a lower but significant positive association, while correlations including several aspects of health care identified a consistent positive association between GP and patient clinic scores. The study also found that the most important predictors of patient satisfaction with the clinics were interaction with the clinician and being met with politeness and respect at the clinic, while the most important predictors for GP satisfaction with the clinics were perceived competence, rejection of referrals and waiting time for patients.

The literature on patient and referral satisfaction is large,1–12,14 but studies about the association between patient and referral evaluation are few and inconclusive.5,7–11 Our study is to our knowledge the only study based on two national surveys examining the association between patient and GP assessment scores at the unit level. It supports the view that a clinics performance in the eyes of GPs is related to a clinics performance in the eyes of patients. This strengthens the validity of using each perspective as an indicator of quality at the clinic level and might be especially useful in cultures where patient views are less highly regarded or are a new source of evaluation and quality indicator information. Several barriers have been identified that limit the active use of results from patient surveys in quality improvement processes including clinical skepticism.19 This evidence for an association between clinician and patient ratings of quality might increase the clinical credibility of results and hence may contribute to the use of results in quality improvement processes within units.

A high correlation between patient and GP evaluation might have justified a less resource intensive data collection, in the sense that assessment from one group could be used as a proxy for the other. This could be especially relevant for patient groups that are hard to reach in surveys20 and might be most relevant in large and resource intensive national indicator systems as in Norway. However, the results of our study do not justify using assessments from one group as a proxy for the other. In spite of a low response rate in the patient sample and the costs of conducting the patient survey, the proxy approach replacing patient satisfaction with GP evaluation would conceal the fact that patients are generally more satisfied with the outpatient clinics than GPs. There are several possible explanations for the difference in satisfaction level, and all have relevance in this context: patients and GPs have different standards and ways of viewing the services;20 patients answers are influenced by a social desirability bias2 and more than GPs as a result of the personal relationship between patients and clinicians and GPs evaluation is more critical because they experience the service provided to many patients, increasing the probability of experiencing errors and negative episodes. Therefore, a balanced evaluation and indicator system should incorporate both patient and GP evaluation, in addition to other validated quality indicators. This would also secure a broader evaluation in a thematic sense since the validated instruments for measuring patient experiences and GP experiences include several supplementary scales for the measurement of clinic quality.

Our study has several limitations. First, we are unable to validate the present findings against clinical quality measures. Strictly speaking, we can only conclude that clinic scores for GP and patient evaluation are related, but whether these scores are associated with clinical quality measures is unknown. Jaipaul and Rosenthal6 reported that hospitals with higher patient satisfaction also tended to have lower mortality. However, this study was restricted to medical patients. Shipley et al.5 found that patient satisfaction was a more accurate quality indicator than standard indicators, validated against an independent review of quality at five mental health units. A problem with the study, however, was that the number of respondents in each group was small, resulting in a questionable generalizability. Therefore, further research relating to the association between patient/referral satisfaction at the clinic level and clinical quality measures is needed. Second, the causes of the association between patient and GP evaluation at the clinic level are unknown. We have identified a consistent association but lack variables about possible explanations for the association including for example organizational processes such as leadership, collaboration and communication.21 These organizational aspects should be explored in future studies. Third, our study was exploratory without any formal hypothesis derived from theory and earlier research. Therefore, the results can only be interpreted as preliminary information that future confirmatory studies could be based on.

Lastly, two limitations in methods should be mentioned. First, GPs evaluated the centres the spring of 2006 while patients evaluated the clinics in the autumn 2007. This would be a methodological problem if many units had substantially improved or worsen their quality in the period. However, national results from 2004 and 2007 indicate small changes in patient evaluations in the period, implying that this is not an important limitation. Second, GPs were instructed to evaluate all services at the community mental health centres, while patients only assessed the outpatient clinic at the centre. However, the outpatient clinics were evaluated by almost all GPs (94%), and the rest of the GPs were excluded in this study. Therefore, GPs mainly evaluated the outpatient clinics, and we believe that this limitation also has small consequences for our study.

In summary, our study found a consistent positive association between GP and patient ratings at the clinic level. To further test our major findings and gain a better understanding of the factors connecting patient and GP evaluations, future research should be conducted at the individual level with patients nested within GPs. Mental health services aiming at improving GP and patient satisfaction should be aware of the importance GPs put on competence, rejection of referrals and waiting time for patients and the importance patients put on interaction with the clinician and being met with politeness and respect at the clinic.

Declaration

Funding: Norwegian Knowledge Centre for the Health Services. Funding to pay the Open Access publication charges for this article was provided by Norwegian Knowledge Centre for the Health Services.

Ethical approval: The Norwegian Regional Committee for Medical Research Ethics, the Data Inspectorate and the Norwegian Directorate of Health and Social Affairs approved the patient experiences survey. The procedure regarding informed consent, study design and data collection in the GP survey was approved by the Norwegian Social Science Data Services.

Conflict of interest: none.