A critical analysis of user satisfaction surveys in addiction services: opioid maintenance treatment as a representative case study.

BACKGROUND: Satisfaction with services represents a key component of the user's perspective, and user satisfaction surveys are the most commonly used approach to evaluate the aforementioned perspective. The aim of this discursive paper is to provide a critical overview of user satisfaction surveys in addiction treatment and harm reduction services, with a particular focus on opioid maintenance treatment as a representative case.
METHODS: We carried out a selective critical review and analysis of the literature on user satisfaction surveys in addiction treatment and harm reduction services.
RESULTS: Most studies that have reported results of satisfaction surveys have found that the great majority of users (virtually all, in many cases) are highly satisfied with the services received. However, when these results are compared to the findings of studies that use different methodologies to explore the patient's perspective, the results are not as consistent as might be expected. It is not uncommon to find that "highly satisfied" patients report significant problems when mixed-methods studies are conducted. To understand this apparent contradiction, we explored two distinct (though not mutually exclusive) lines of reasoning, one of which concerns conceptual aspects and the other, methodological questions.
CONCLUSION: User satisfaction surveys, as currently designed and carried out in addiction treatment and harm reduction services, do not significantly help to improve service quality. Therefore, most of the enthusiasm and naiveté with which satisfaction surveys are currently performed and interpreted - and rarely acted on in the case of nonoptimal results - should be avoided. A truly participatory approach to program evaluation is urgently needed to reshape and transform patient satisfaction surveys.

Introduction

Interventions and programs addressing drug use and related harms have traditionally been evaluated primarily by “hard” indicators or criteria,1 including program retention rates, use of the primary drug of abuse and other psychoactive substances, morbidity/mortality, and criminal activity, among others. These criteria are relevant and necessary to inform health care policies; however, they do not necessarily reflect the priorities of those who use harm reduction or addiction treatment services.2,3 Moreover, there is evidence showing that user perspectives do not correlate with staff perspectives,4,5 a well-described phenomenon known as the “paradox of desynchrony.”6 Therefore, it is clear that a more inclusive and comprehensive evaluation of the process and outcomes of interventions should include users’ assessment of these variables.1

Satisfaction surveys, a key component of assessing the service user’s perspective,7 have long been the most commonly used method of exploring this perspective.8 User satisfaction surveys assessing addiction treatment and harm reduction services have consistently found high skewness toward positive satisfaction scores.9–12 While this could be interpreted with great enthusiasm, other hypotheses should be considered, such as publication bias, lack of clarity of norms against which to compare the treatments, the psychometric properties of the instruments used, and the lack of an underlying theoretical framework of patient satisfaction.13,14 The aim of this paper is to provide a critical overview of user satisfaction surveys as currently carried out in addiction treatment and harm reduction services, with a special focus on opioid maintenance treatment (OMT) as a representative case. First, we argue the lack, relevance, and need for a more patient-centered approach to program evaluation. Then, we analyze the main features of client satisfaction surveys conducted at OMT centers as well as conceptual and methodological issues that may explain why the results of such surveys often diverge from results obtained via other data collection/generation methods. Finally, based on the discussion of some limitations of patient satisfaction surveys as currently performed, we suggest alternative directions for inquiry and research focused on a truly user-participatory approach to program evaluation.

Towards a more user-centered approach to program evaluation

When conducting a user-centered evaluation of OMT services, several indicators or assessment areas should be considered: a) user expectations and objectives for the treatment or intervention,15,16 b) patient beliefs or opinions about medication (primarily methadone or buprenorphine),17–21 c) user appraisal of the relational dynamic with intervention providers22–24 and the degree of perceived participation in decision-making,25,26 as well as patient preferences for a participatory or deferential role,27,28 d) indicators of user-perceived quality,29 e) client perception of unmet needs,30,31 and f) level of satisfaction with treatment.8,11,25,32

Despite the relevance and validity of the aforementioned factors in facilitating adherence to the intervention10,33–35 and in measuring service quality,36,37 very few programs or centers evaluate these factors on a routine basis. This is surprising given that program adherence and service quality are two areas in which the importance of these variables is readily apparent. On one hand, these variables influence program acceptability and given that acceptability can affect adherence, these variables may mediate – at least partially – the efficacy of the interventions. On the other hand, the accreditation process of some types of treatment centers (eg, methadone treatment centers) requires the periodic evaluation of certain variables (generally, patient satisfaction) to assess quality.38,39

These variables (or more generally speaking, the user’s perspective) have generally received scant attention40,41 and have been largely ignored by the community of mainstream researchers,42 who do not acknowledge the relevance of such variables in designing, providing, and/or evaluating addiction treatment and harm reduction services. This lack of research on user perspectives contrasts with the many studies evaluating interventions and treatments. This is particularly worrisome considering that input from service users is essential if we are to define, implement, and evaluate these interventions and programs, ensuring they meet the needs of the users themselves.43 Needless to say, a more in-depth understanding of user perspectives of interventions and centers would allow us to better interpret the efficacy and quality of interventions, as well as to plan and implement measures to improve those parameters. Without this knowledge, the risk of underestimating or distorting the value of treatment services to users through the usual assessment practices is high.44

In recent years, interest in including users’ perspectives to evaluate interventions that address substance use appears to be growing. This interest has resulted in the increasing use of patient-reported outcome measures in both clinical trials and usual clinical practice.45–47 This emerging framework could be related to the current trend toward altering the traditional hierarchical relationship between health care professionals and patients, ie, a relationship based on the classical model of medical practice, which implies a biased and exclusionary perception of drug users as noncompetent persons.48,49

Satisfaction surveys, as stated above, are a key component of assessing the service user’s perspective7 and are the most common and most studied approach to exploring the aforementioned perspective.8 Nevertheless, it is important to keep in mind that most classification systems of the opportunities or levels of participation in addiction treatment and harm reduction services consider user satisfaction surveys to be “low-involvement” activities (Table 1).50–52 In fact, some authors prefer to exclude user satisfaction surveys from the definition of participation because such surveys do not entail the existence of mechanisms to modify planning processes in response to the survey’s findings, nor do they imply that service users participate as equal interlocutors.53

Table 1

Levels of user-participation or opportunities for user-participation in addiction treatment and harm reduction services

Degree of user involvement	Type of participation	Example of activity
High	Activities implying a share in decision-making	User’s representatives involved in service planning committees
		User’s representatives attending staff meetings
		User’s representatives involved in staff recruitment
		User’s representatives involved in staff performance appraisal
Mild	Activities in which service users have nondecision-making roles	Users involved in writing or reviewing informative fact sheets or educational materials
		Users involved in staff training
	Activities promoting and supporting user involvement	Availability of an adequate space and schedule within the service to run users own support groups
		Availability of publications or information about the activities of drug user organizations
Low	Activities related to providing information to or receiving information from service users	User councils
		User forums
		User satisfaction surveys
		Suggestion box
		Complaint process
		Adequate display of information concerning changes on policies or service hours
	User-participation built into the values and policies of the service	Charter of rights

Notes: These levels represent a range of opportunities for user-participation. In practice, and due to very diverse reasons (eg, availability of the activity by the service or user’s awareness and motivation), participation does not occur in all of these areas, in all services, or in the case of all users. Adapted with permission from Bryant J, Saxton M, Madden A, Bath N, Robinson S. Consumer participation in the planning and delivery of drug treatment services: the current arrangements. Drug Alcohol Rev. 2008;27(2):130–137. Copyright © 2009, John Wiley & Sons.51

Assessment of client satisfaction in OMT

The number of validated self-reports specifically designed to assess satisfaction with addiction treatment54,55 and, more specifically, with treatment for opioid dependence,56,57 are quite limited. Client satisfaction with OMT has been primarily assessed through a) specific questions,58,59 b) nonvalidated questionnaires designed ad hoc,60,61 c) nonvalidated adaptations of instruments originally designed to assess satisfaction with mental health services or other types of health care services,62 or d) validated generic scales designed to assess satisfaction with any type of health care center (ie, not specific to addictions).11,63–65

Patient satisfaction research conducted with instruments described above shows that most OMT clients (virtually all, in many cases) are highly satisfied with the care received.11,56,58,60,65 However, these findings clearly contradict a) specific comments made by particular OMT patients on many occasions and situations;66 b) most studies that have used a qualitative methodology to explore the views and experiences of OMT clients, the results of which are more nuanced, or in some cases, reflect a clearly ambivalent attitude toward OMT;66–69 and c) the results obtained by satisfaction surveys that include (in addition to items or a scale to evaluate satisfaction) either a list of potential problems or negative treatment-related aspects from which the respondent must select, or open-ended questions requiring a short response about which aspect(s) of the user’s experience with the center he/she finds least satisfying.25,58,70 Interestingly, these latter mixed-method studies have allowed researchers to identify significant and/or frequent problems reported by clients who, concurrently, showed a high level of satisfaction with OMT.

To explain the reasons behind this apparent discrepancy, two distinct but nonexclusive arguments can be made: one line of argument is more closely related to conceptual aspects, while the other is mostly concerned with various methodological questions.

Conceptual confusion between patient perceptions and patient satisfaction

There is a conceptual confusion originating from the interchangeable use of the terms “patient satisfaction” and “patient perceptions.” Satisfaction refers to the patient’s perception of the extent to which his/her expectations, needs, or desires have been met. Satisfaction is thus, a particular kind of perception, but it is not the only one.71 This confusion has increased as most scales that assess satisfaction with addiction treatment have been developed without an underlying theoretical framework. Although various theoretical models of the “treatment satisfaction” construct exist, it seems to be assumed implicitly – and naively – that this is a construct with a universal meaning, or one that does not require greater clarification.72 In fact, some authors believe that the concept of treatment satisfaction seems to have been imposed on service users, with little consideration of its relevance.73 This situation is even more complex in the case of OMT, in which meanings linked to the construct of “patient satisfaction” and their respective operationalization can lead to even more confusion and ambiguity. For instance, from both a clinical and research perspective, it seems necessary to distinguish, at a minimum, between a) holding dose (ie, the dose that prevents subjective and objective opioid withdrawal symptoms over the 24-hour interdosing interval); b) dose adequacy (ie, the dose that allows patients not to use heroin, not to experience withdrawal symptoms nor heroin craving, not to show symptoms of overmedication, and in the case of heroin use, to hardly experience any subjective effects); c) satisfaction with medication; and d) satisfaction with treatment.74,75

Even when patients report a high degree of satisfaction with treatment, this does not necessarily entail that they have a similarly positive perception of the treatment received. High satisfaction scores can be due to negative initial expectations25,76 or may reflect, among other nonexclusive alternatives, beliefs of the type “they do the best they can”77,78 or “really, that is not their job”,78 or service user preference to demonstrate a positive interaction with the people who care for them (social etiquette),79 or the limitations derived from the user’s dependent position in the health care system.79 Given the consistently positive results from patient satisfaction surveys, it is no wonder that many authors believe these types of survey are little more than rhetorical practices that provide us with the comforting illusion that we are listening to patients,80 or that these surveys are mechanisms by which managers and clinicians seek to legitimize and maintain the status quo of treatment centers.69,81–83

Methodological and procedural issues

The second line of argument concerns diverse methodological and procedural issues related to satisfaction surveys.

Unidimensional versus multidimensional approaches

Patient satisfaction instruments can be classified according to factor structure as either unidimensional (ie, those that report a single overall satisfaction factor) or multidimensional (ie, comprised of several factors corresponding to differentiated facets or dimensions of treatment satisfaction).84,85 Dissatisfaction rates are, in general, higher in surveys that use multidimensional scales compared to those obtained with surveys that have used global or unidimensional instruments,85,86 although this comparison is not completely appropriate due to methodological differences arising from the type of instrument used.

Affective versus factual focus

Patient satisfaction scales can be divided into two types depending on the contents: factual measures (ie, objective reports) and affective measures (ie, subjective rating scales).85,87 Factual instruments, which are designed to avoid value judgments, focus on the service user’s experiences. These types of instruments contain specific questions about those experiences in order to acquire objective data (“How often do you go to the center to take or pick up methadone?”; “How long after the scheduled appointment time do you usually have to wait to be seen?”) or to verify whether certain processes and specific events have occurred or not (“Have the side effects of methadone been explained to you?”). In contrast, affective instruments focus on the patient’s opinion about those experiences which reflects his/her preferences and expectations (“Do you think the frequency of your visits to the center to take/pick up methadone is excessive?”; “Are you satisfied with the waiting time between the scheduled appointment time and the actual start of the appointment?”; “Are you satisfied with the information you have received about the side effects of methadone?”).

There is a lack of validated factual instruments among the satisfaction scales available for use in addiction treatment and harm reduction services. However, when such instruments do become available, it seems likely that the experience observed in other health care settings – ie, affective measures typically detect higher levels of satisfaction with treatment than factual instruments85,88 – will be repeated.

Affective self-report instruments, particularly those with a unidimensional factor structure, are unlikely to provide information that will be useful to improve health care.85,86,89 This is due to the way that most questions are formulated in these instruments; in many cases, it is not clear what elements, aspects, or processes need to be modified when a client’s response indicates dissatisfaction,90,91 and it is even less clear how such changes should be effectuated. However, the use of multidimensional instruments (especially the factual measures) can remediate, at least partially, this weakness, as has occurred in other areas of health care.88,92,93

Early drop-outs and refusals to participate

We must not neglect a discussion of two important variables that have received insufficient attention in many of the patient satisfaction studies carried out to date in the addictions field: a) the number of early drop-outs from the program being evaluated81,85,94,95 and b) the nonresponse rate to the survey.26,96–99 Early drop-outs and no response could be reflecting low levels of satisfaction with treatment. These factors contribute to the uncertainty of the results obtained and are a threat to both the internal validity and generalizability of such results.

A variant of the Hawthorne effect

The Hawthorne effect is a positive and transitory change in a behavior being evaluated under experimental conditions. The change in behavior occurs not because of any alterations in the independent variable, but rather because participants know they are being observed or studied.100 By analogy, levels of satisfaction with treatment could, in some cases, reflect or be sensitive to this effect. Sitzia and Wood101 indicate that both the extra attention implicit in the process of data collection and the apparent interest in the user’s level of satisfaction could lead to a positive perception of the service or center and, consequently, to higher scores.

The phenomenon of response shift

Finally, the response shift phenomenon102 merits discussion as a potential additional explanatory factor for the high levels of patient satisfaction typically observed. This phenomenon has been widely studied in the field of health-related quality of life, although it is equally relevant to any field that uses self-report instruments.103 Villar López et al104 highlighted the importance of response shift in evaluating satisfaction with treatment. The response shift phenomenon in health care settings refers to the process of adaptation to changes caused by the disease or treatment.103 More specifically, Sprangers and Schwartz102 define response shift as a change in the meaning of the self-evaluation of the target construct as a result of a) a change in the internal standards of measure (scale recalibration); b) a change in the values themselves – in other words, in the importance of the domains or components that make up the target construct (scale reprioritization); or c) a redefinition of the target construct (reconceptualization). Many years of substance use and a relatively long OMT could potentially modify the client’s internal standards, values, and/or concept of treatment satisfaction and, thereby, overestimate those levels of satisfaction.

Future directions

Most of the instruments currently available to assess user satisfaction with addiction treatment and harm reduction services are probably unable to adequately measure satisfaction due to how those instruments were developed. In general, the available instruments are based on a theoretical model that is often only vaguely defined and which evaluates areas of interest chosen by the researchers, service providers, or policy makers.105,106 It seems evident that self-reports developed in this manner are more likely to reflect the priorities of the clinicians, researchers, and/or administrators rather than those of the service users.101,107–109 This implies that questions about aspects of treatment and care that are relevant to patients would not be included, even though other variables that patients might consider irrelevant are incorporated.87 If true, to some extent this represents a threat to basic bioethical principles, such as autonomy,110 and to the validity of data gathered with these tools,111 questioning the results obtained with these instruments.

Users of addiction treatment and harm reduction services – like users of any other health care services and programs – have a complex and highly nuanced experience that can be more appropriately and fruitfully captured via more generic, open-ended questions that are formulated in terms of the user’s experience (not the satisfaction) with the service. Additional approaches include other qualitative techniques of data gathering, such as participant observation,112,113 focus groups,31,67 or in-depth interviews.114–116 Likewise, the critical communicative methodology117–119 would seem to be particularly pertinent and promising in this area of research. Although the information provided by these methods can be extremely valuable for improving and redesigning interventions, a periodic quality assessment relying heavily on these approaches appears to be of questionable practicality under current circumstances.

The use of the critical communicative methodology or qualitative data gathering techniques must be an essential requirement for developing the much-needed new scales to measure treatment satisfaction120 or perception of treatment121 in a more patient-centered manner (Table 2).109,122 A genuinely patient-centered assessment of satisfaction with treatment will not be feasible without patients participating in the development of the instrument (eg, identifying the dimensions and variables to be assessed, drafting the items). Moreover, the aforementioned techniques should also play a major role in helping health professionals and researchers to directly, through the patient’s own words, come to gain a thorough understanding of the patient’s perspective (eg, expectations, perceived participation in decision-making). Having a deeper understanding of this perspective is an essential part of any efforts to improve the quality of addiction treatment and harm reduction services. This perspective cannot and should not be limited to user satisfaction with the service, and neither should user-participation in quality improvement efforts be restricted to answering a questionnaire. That said, if planners, managers, and workers involved in addiction treatment and harm reduction services are unwilling to make the necessary effort to achieve an in-depth understanding of users’ perspective – and to act on any new insights obtained in the process – then these methodologies are unlikely to be implemented, and thus, authentic patient-centered satisfaction surveys will not be carried out.

Table 2

Classification of instruments to measure patient satisfaction according to the degree to which they incorporate the patient’s own perspective

Type	Description
Patient satisfaction scale developed by patients	A patient satisfaction scale (PSS) developed by patients (PSS-DP) is a PSS developed completely from the patients’ perspective. The main differentiating characteristic of this type of PSS lies in the fact that all those involved in the different phases of its development are (or have been) recipients of the service to be evaluated.
Patient-centered satisfaction scale	A patient-centered PSS (PSS-CP) is a PSS that explicitly incorporates, to a greater or lesser extent, the patients’ perspective. Even though other stakeholders (eg, clinicians, researchers, family members, and/or administrators) have participated in developing the instrument, the patients themselves have made major contributions, usually in the initial phases of 1) generating the domains or variables to be evaluated, 2) writing the questions, and/or 3) evaluating some characteristic of the initial or pilot version of the instrument (eg, pertinence or comprehensibility of the items).
Patient satisfaction scale valued by patients	A PSS valued by patients (PSS-VP) is a PSS that, developed without direct patient participation, is considered as such (ie, valued or appreciated) by most of them because it reflects, at least in part, their perspectives. To determine whether a PSS developed without patient participation can be considered patient-valued, qualitative research methods such as focus groups or cognitive interviews must be performed.
Patient satisfaction scale irrelevant to patients	A PSS irrelevant to patients (PSS-IP) is a PSS that has been 1) entirely developed without any patient participation, and 2) assessed by patients as irrelevant or of no interest or value. A priori, assuming no additional information is available, any PSS developed without any direct patient participation should be placed, at least provisionally, in this category.

Note: Data from Trujols et al.122

Given this situation, it is not surprising that some authors, both in the drug-use intervention field8 as well as in other areas of health care123 pose this question: Do current satisfaction surveys form part of an emerging process of shared decision-making whose objective is to improve health care delivery and clinical care practices or, to the contrary, are such surveys a mere token gesture (ie, an empty ritual)124 from which no changes will emerge? In accordance, Madden et al25 affirm that “satisfaction surveys will be of only limited value in opioid treatment settings until there are agreed quality standards and formal mechanisms to educate consumers about their rights and the standards of care to which they are entitled”. Without these changes and a commitment to act on the knowledge so acquired, it is unlikely that meaningful, rather than tokenistic, modalities of user involvement will be widely implemented in the near future. In fact, service users should not be asked to participate in a satisfaction survey if their input will amount to nothing. Moreover, it should be also emphasized that the aforementioned changes are unlikely to increase the right of service users to participate if their most basic needs are not met.

Patient participation should extend beyond assessing and improving the quality of care. The involvement of drug/service users in the design and implementation of specific programs and interventions, especially in the field of harm reduction, has been remarkable, with users, at times, displaying a higher level of competence than the public health institutions themselves.125–127 Similarly, considering that users’ rights should not be limited to treatment needs alone,128,129 this dialogical and participatory dynamic should be extended to other areas such as addiction research130 and the development of drug policies.131

Limitations

The aim of this article was to provide a critical review of a comprehensive topic (ie, satisfaction with addiction treatment and harm reduction services) rather than a systematic examination of a focused research question. Therefore, a nonsystematic review was judged as the best approach for covering a wide range of issues relating to the topic reviewed. However, this approach implies not to systematically identify, select, appraise, and synthesize all research on the topic, representing a possible limitation of the present study. In any case, as several authors have pointed out, traditional narrative reviews are more appropriate for comprehensive topics and systematic reviews are better suited for focused topics.132 Additionally, our attempts to identify relevant studies went beyond the efforts usually made in narrative reviews. Spanning the last decade, our experience in research on satisfaction with addiction and harm reduction services5,8,11,26,29,32,57,58,74,75 has rendered us current on this topic. In this regard, the PubMed interface to search MEDLINE has been used. A search strategy favoring sensitivity over specificity is periodically performed using different combinations of addiction (eg, substance abuse, heroin, cocaine) or intervention-specific (eg, methadone, detoxification, needle exchange) terms and words related to the patient’s perspective (eg, satisfaction, perception, view). In addition, reference lists included in the selected articles are normally examined for other relevant studies.

Conclusion

The current approach to user satisfaction surveys does not significantly contribute to the improvement of service quality. Therefore, most of the enthusiasm and naiveté with which user satisfaction surveys are currently conducted and interpreted – and rarely acted on in cases of nonoptimal results – in addiction treatment and harm reduction services should be avoided. A truly participatory approach to program evaluation is needed urgently to reshape and transform patient satisfaction surveys. Similarly, there is a need to implement alternative research and quality-improvement initiatives that foster meaningful, rather than merely tokenistic, service user involvement. In turn, this will surely improve the relevance as well as the psychopolitical validity (both epistemic and transformative)133,134 of research and interventions in the substance use field.