Process evaluation of a technology-delivered screening and brief intervention for substance use in primary care.

Psychotherapy process research examines the content of treatment sessions and their association with outcomes in an attempt to better understand the interactions between therapists and clients, and to elucidate mechanisms of behavior change. A similar approach is possible in technology-delivered interventions, which have an interaction process that is always perfectly preserved and rigorously definable. The present study sought to examine the process of participants' interactions with a computer-delivered brief intervention for drug use, from a study comparing computer- and therapist-delivered brief interventions among adults at two primary health care centers in New Mexico. Specifically, we sought to describe the pattern of participants' (N=178) choices and reactions throughout the computer-delivered brief intervention, and to examine associations between that process and intervention response at 3-month follow-up. Participants were most likely to choose marijuana as the first substance they wished to discuss (n = 114, 64.0%). Most participants indicated that they had not experienced any problems as a result of their drug use (n = 108, 60.7%), but nearly a third of these (n = 32, 29.6%) nevertheless indicated a desire to stop or reduce its use; participants who did report negative consequences were most likely to endorse financial or relationship concerns. However, participant ratings of the importance of change or of the helpfulness of personalized normed feedback were unrelated to changes in substance use frequency. Design of future e-interventions should consider emphasizing possible benefits of quitting rather than the negative consequences of drug use, and-when addressing consequences-should consider focusing on the impacts of substance use on relationship and financial aspects. These findings are an early but important step toward using process evaluation to optimize e-intervention content.

Introduction

Computer-delivered behavioral interventions offer their authors a unique opportunity as well as a unique challenge. In terms of opportunity, they allow creation of an intervention in which every word, image, branch, color, and sound can be controlled—a stark contrast to person-delivered interventions in which only specific principles and techniques can be codified. However, this level of control can also be challenging in its implicit demand on the author to predict the characteristics, reactions, and preferences of future users. Evidence-based models offer little guidance in regard to such details as what choices to provide or when to provide them.

Process research models may offer such guidance. These models examine the content of interactions within intervention sessions, and compare those process variables with outcomes in an attempt to identify key mechanisms of change (e.g., Webb et al., 2012, Feeley et al., 1999). Describing the choices that participants make in completing computer-delivered interventions can aid in development of future interventions; for example, being aware of topics that participants rate as most important can allow for those topics to be emphasized. In doing so, less salient topics can be avoided, increasing the perceived relevance and “fit” of the intervention. Further, evidence that certain kinds of in-session ratings are associated with better outcomes—although not evidence of causation—may provide important clues about possible mechanisms through which computer-delivered interventions may exert their effects.

Computer-delivered interventions are also uniquely amenable to process research. Each choice the participant makes, and the context in which he or she makes it, is available without the need for arduous coding of session tapes by raters trained to reliability. Many eHealth interventions also solicit participant ratings of satisfaction with the intervention and/or intention to change within the intervention itself, providing additional clues as to the user's experience of the intervention process and further data to compare against outcomes. For example, user satisfaction is often seen as a critical factor in the efficacy of an intervention, but its actual association with behavioral outcomes is unclear.

The present study uses a process research framework (e.g., Bertholet et al., 2014, Rhodes, 2012) to examine participant interactions with a computer-delivered brief intervention for drug use, from a prior study comparing computer- and therapist-delivered brief interventions among adults at two primary health care centers in New Mexico (Schwartz et al., 2014). We first describe the pattern of participants' choices and reactions throughout the highly interactive computer-delivered brief intervention. In a subsequent exploratory analysis, we examine associations between those process variables and intervention response at a 3-month follow-up. We were particularly interested in the extent to which problem recognition and self-ratings of the importance of change were associated with actual reductions in substance use at follow-up.

Materials and methods

Participants

Data for this secondary analysis study are drawn from a clinical trial comparing computer-delivered to in-person brief interventions among patients attending one of two primary health care centers in New Mexico; that trial and all related data collection were approved by the Institutional Review Boards of the Friends Research Institute and Christus Health. Participants were adults age 18 and older scoring in the moderate risk range (between 4 and 26) for illicit drug use on the Alcohol, Smoking, and Substance Involvement Screening Test (ASSIST; Humeniuk and Ali, 2006), with self-reported use in the past three months but no treatment in the past year. All participants were recruited from the health centers' waiting area by research assistants and received $20 for completing the baseline assessment interview. The present analysis focused on the 178 participants from that trial who were randomly assigned to and completed the computer-delivered single-session intervention for substance use. Details regarding the sample and the results of the clinical trial have been published previously (Schwartz et al., 2014, Gryczynski et al., 2015). Briefly, the computer-delivered brief intervention subgroup was primarily White (n = 160, 89.9%) and approximately half female and half of Hispanic ethnicity (n = 82 for both; 46.1%). Mean age was 36.6 years (SD = 14.8), and 106 (59.6%) were not employed at the time of study enrollment.

Intervention

Also described in detail in Schwartz et al. (2014), the brief computer-delivered intervention used in this trial was designed to be highly interactive, making use of a talking animated narrator as well as branching and/or reflections in response to participants' choices. Taking approximately seven minutes to complete, the intervention began by offering participants a choice of which substance they wished to hear more about. Participants were then offered personalized, gender-specific normed feedback regarding how their use of that substance compared to adults in the US (using data from the National Survey on Drug Use and Health); this material emphasized the large proportion of lifetime users of each drug that had not used it recently (i.e., to convey that cessation of drug use is in fact normative among lifetime drug users). Participants were then offered the opportunity to endorse, from a list, which negative consequences they had personally experienced as a result of using that substance. Those who endorsed at least one negative consequence were asked to rate the importance of changing, and—if indicating moderate or high importance—were helped to consider possible changes. Those who failed to endorse any negative consequences were asked whether or not they were interested in stopping or reducing their use of that substance, and were helped with setting change goals, as appropriate. All participants were also offered the opportunity to go through the intervention a second time, focusing on a different substance. See Fig. 1 for more detail regarding the structure of the brief intervention.

Fig. 1

Simplified intervention flow highlighting main branches and key process variables.

Measures

Variables for this analysis were taken from the brief intervention itself. Key variables are noted in Fig. 1, and fell into two primary types. First were basic descriptive variables regarding choices participants made during the brief intervention. These included which substance participants chose to discuss, whether or not they endorsed negative consequences of their substance use, and whether or not they chose to go through the intervention a second time focusing on a different substance. The second type consisted of variables that could potentially reflect processes through which the brief intervention could have an effect, including perceptions of the usefulness of the normed feedback, and responses regarding the importance of changing their use. That is, evidence of an association between these variables and later substance use could lead to further research regarding whether interventions focusing on increasing problem recognition or the perceived relevance of feedback might lead to better effects.

Outcome was defined primarily in terms of changes in two variables derived from the ASSIST, which were created by subtracting follow-up from baseline scores on two key measures: (1) the overall ASSIST Global Continuum of Illicit Drug Risk (GCIDR) score, which captures risk associated with any drugs the participant reports using and thereby can measure both polydrug use as well as switching from one drug to another; and (2) participant responses to ASSIST item 2, which asks about the frequency of substance use in the past three months using a Likert scale ranging from “Never” to “Daily or Almost Daily.”

Statistical analysis

The present analyses focused first on descriptive statistics regarding the above-noted process variables, and subsequently on associations between those process variables and change in drug use frequency/consequences. Regarding the outcome variables, for ASSIST item 2 (drug use frequency), we created a difference score representing the participant's baseline and follow-up scores for the substance they chose to examine first (see Fig. 1); thus, this score represented changes in marijuana use frequency for those who chose marijuana as the first drug to consider, changes in cocaine use frequency for those who chose to focus on cocaine use first, etc. For this outcome variable as well as for ASSIST GCIDR, the subtraction of the follow-up response from the baseline response yielded a change score on which a score of zero indicated no change, a negative score indicated increased use or overall risk from baseline to follow-up, and a positive score indicated a reduction in use or overall risk from baseline to follow-up. The GCIDR change score variable was highly skewed and leptokurtotic, and could not be rendered normally distributed via transformations; it was therefore converted to an ordinal variable with ten levels. The pre-post use frequency variable was also treated as ordinal; outlying levels with few cases were collapsed, yielding an ordinal variable with seven levels ranging from − 3 to 3.

After characterizing process variables using descriptive statistics, and after creating the two outcome variables as described above, we subsequently examined associations between process variables and the two ordinal outcome variables using non-parametric statistics (Siegel and Castellan, 1988): the Mann–Whitney U test (a non-parametric analogue of the independent values t test, for use with dichotomous predictors and ordinal outcomes) and Kruskal–Wallis one-way analysis of variance (a non-parametric analogue of a one-way ANOVA, for use with categorical predictors and ordinal outcomes). Correlations between outcomes and ratings of the importance of change/satisfaction with feedback (each being ordinal) were conducted using the Spearman rank-order correlation.

Results

Participant choices and preferences

The flow of participants during their first pass through the intervention is depicted in Fig. 1. (Participants had the choice to go through the software a second time, focusing on a different substance. Because only a minority of participants chose a second substance, we report here only on the results from the first pass; however, whether or not participants opted for a second pass is reflected in Fig. 1, and was considered as a process variable.) When asked which substance they wanted more information about, most participants (114, 64.0%) chose marijuana; 23 (12.9%) chose opiates, 16 (9.0%) chose cocaine, and 11 (6.2%) chose amphetamines; less than 8% of participants chose either sedatives, inhalants, or hallucinogens. Notably, these choices were significantly associated with baseline risk scores on each of these substances, such that the highest baseline ASSIST risk score was marijuana for participants who chose marijuana, cocaine for those who chose cocaine, etc.; this pattern was true for all drug classes except for inhalants and sedatives, which had a combined sample size of 11 (Table 1).

Table 1

Baseline ASSIST subscale scores as a function of participant choice of intervention focus.

Intervention focus chosen by participant (n)	Marijuana	Cocaine	Amphetamines	Inhalants	Sedatives	Hallucinogens	Opiates
Marijuana (114)	11.1	1.0	1.1	0.1	1.1	0.4	1.3
Cocaine (16)	8.4	14.0	0.4	0.3	3.6	0.6	4.4
Amphetamines (11)	8.8	3.1	13.7	1.1	3.2	0.5	6.2
Inhalants (1)	6.0	0.0	0.0	0.0	0.0	0.0	0.0
Sedatives (10)	7.3	2.1	3.9	0.6	5.8	1.1	5.7
Hallucinogens (3)	6.7	2.7	0.0	0.0	0.0	7.7	1.0
Opiates (23)	6.3	1.9	1.2	0.1	2.2	0.3	10.6

Note. Highest ASSIST subscale score in each row is highlighted in bold.

After receiving personalized normed feedback regarding their chosen substance, participants were asked to rate the helpfulness of that feedback. Overall, just over half of participants (103, or 57.9%) rated it as somewhat or very helpful, and another 48 (27.0%) rated it as a little helpful. Only 27 participants (15.2%) rated it as not at all helpful. Of interest, ratings of the helpfulness of the normed feedback were unrelated to ratings of the importance of changing (rs = .15, NS), but were associated with desire to change. Specifically, 12 of 32 participants (37.5%) who rated themselves as interested in changing gave the highest possible rating for the feedback (“very helpful”). In contrast, only 11 of 76 participants (14.5%) who rated themselves as not interested in changing gave the highest possible rating of the feedback (χ2[3] = 10.6, p = .014).

When asked to endorse specific consequences related to their substance use, a majority (108, or 60.7%) indicated that they were not currently experiencing any consequences. Of participants who did endorse negative consequences of their drug use, the most frequently chosen consequences were impaired relationships with friends or family (endorsed by 28 participants, 15.7% of the total sample) and money problems (13 participants, 7.3%). Factors including work, physical health, emotional problems, legal problems, and other problems (unspecified) together were endorsed by 29 participants (16.3%). Baseline ASSIST scores were significantly higher for participants who endorsed a negative consequence during the intervention (mean ASSIST GCIDR of 42.2 vs. 27.3; t (176) = − 5.2, p < .001); the likelihood of endorsing a negative consequence was 64.7% for participants scoring in the top decile on the baseline GCIDR, vs. 12.5% for those in the bottom decile.

As indicated in Fig. 1, participants who endorsed a negative consequence of their drug use were asked to rate the importance of changing their use of the substance they chose to address, using a 0–10 scale. In addition, participants who did not endorse a negative consequence but who did indicate a desire to change were also asked to rate the importance of changing. (Of the 108 participants who denied any drug-related problems, 32 (29.6%) indicated a desire to change their drug use.) Of the resulting 102 participants asked to rate the importance of changing (70 who endorsed a drug-related consequence and 32 who denied any consequences but nevertheless wanted to change), the mean rating for the importance of change was 4.8 (SD = 3.8); 55.9% of participants endorsed a score of 5 or lower on this item. Importantly, this flow only indicates participants' choices their first time through the intervention; as noted in Fig. 1, 45 participants (25.3%) chose to complete the intervention a second time, focusing on a different substance.

Association of process variables with change in drug use risk, baseline to 3 month follow-up

As noted, process variables were compared to within-participant changes in drug use risk between baseline and follow-up, using two ASSIST-related variables: overall drug use risk (GCIDR) and response to ASSIST item 2 regarding drug use frequency, using only their response for the drug class that they chose to focus on in step one of the intervention. Most associations were not significant (not shown). There was a significant association between participants' initial drug category selection (dropping the inhalant and hallucinogen categories, which had 1 and 3 participants respectively) and changes in GCIDR (Kruskal-Wallis χ2[4] = 10.9, p = .027). A similar difference was present between participant choice of drug category and changes in drug use frequency (Kruskal–Wallis χ2[4] = 11.3, p = .024). Post-hoc pairwise comparisons using the Mann–Whitney U test revealed significant differences between participants choosing marijuana and those choosing cocaine for both GCIDR (U = 434.0, p = .004) and frequency of use (U = 477.5, p = .008), with greater (more positive) change for participants choosing cocaine. Pairwise analysis also revealed a specific difference between participants choosing marijuana and those choosing opiates, for frequency of use only (U = 761.0, p = .034), in which participants choosing opiates showed greater (more positive) change than participants choosing marijuana. Table 2 shows the proportion of participants selecting each drug use class who showed positive change in either use frequency or GCIDR.

Table 2

Crosstabulation of intervention focus chosen by participant with positive change in overall risk and drug use frequency.

Intervention focuschosen by participant (n)	Positive changein GCIDR (n, %)	Positive change in drug use frequency (n, %)
Marijuana (114)	57 (53.8%)	27 (25.5%)
Cocaine (16)	12 (80.0%)	9 (60.0%)
Amphetamines (11)	5 (50.0%)	4 (40.0%)
Inhalants (1)	1 (100%)	0
Sedatives (10)	7 (77.8%)	3 (33.3%)
Hallucinogens (3)	2 (66.7%)	2 (66.7%)
Opiates (23)	15 (75%.0)	10 (50.0%)

There was one additional positive association between process variables and outcomes. Considering only the 108 participants who denied any negative consequences as a result of their drug use, those who subsequently chose to make a change in their drug use (n = 32) showed greater reductions in overall drug use risk (U = 725.5, p = .013). However, choosing to make a change was not associated with a reduction in drug use frequency. Other process variables such as problem recognition, perceived importance of changing, willingness to go through the intervention a second time, and satisfaction with personalized normed feedback were unrelated to reductions in drug use risk or frequency at follow-up.

Discussion

Computer-delivered interventions are typically multi-faceted and can contain many different possible paths, with each path determined by participants' choices. Greater understanding of those choices—and how those choices might be associated with outcomes—can assist in optimizing these complex interventions. The present study sought to explore participants' choices within a highly interactive, primary care-based brief intervention for drug use, and to examine the extent to which those choices were associated with short-term outcomes. Overall, we found that participants were primarily interested in interacting with the software with respect to their marijuana use; that most participants felt they had not experienced any problems as a result of their drug use, but that nearly a third of these chose to stop or reduce their use; and that participants tended strongly to focus on the substance for which their ASSIST score was highest. Participant ratings of the importance of change or of the helpfulness of personalized normed feedback were unrelated to changes in substance use frequency.

Several findings from this analysis must be highlighted. First, it appears that non-treatment seeking users of software such as this are likely to make good use of it, even without any guidance from a therapist. For example, when given the choice to learn more about any type of drug, participants tended to choose the one that testing suggested was most problematic for them. This finding is important given potential concerns that although allowing participants to choose the focus of the brief intervention is consistent with Motivational Interviewing principles (Miller and Rollnick, 2013), doing so could allow participants to avoid the substance that is causing them the most harm. The results of this analysis suggest that this avoidance of the substance causing the most risk is unlikely, and that a patient-centered approach to brief intervention seems appropriate. This was even true for marijuana, which despite being increasingly accepted as normal (Miech et al., 2015), was the chosen focus for the majority of participants. In addition, there was a strong linear association between baseline risk scores and the likelihood that the participant would report that drug use interfered negatively with his or her life in some way. Finally, even though there was no incentive to do so at all (and again, this was a proactively recruited, non-treatment seeking sample), 25% of participants chose to go through the program a second time with respect to another substance.

Second, less than half of persons within this non-treatment seeking sample felt that their substance use was interfering with their lives in any way, despite the inclusion criteria of active use and at least moderate risk on the ASSIST; further, endorsement of a negative consequence did not rise above 65%, even among participants with the highest baseline severity. Further, nearly a third of those who did not perceive significant negative consequences of their drug use were nevertheless open to changing their use. This suggests that future interventions should focus on the benefits of changing and should not presume that a failure to perceive negative consequences of drug use means that the participant is not willing to change.

Third, self-reported intention to change was associated with change in drug use frequency at the three-month follow-up. This finding mirrors that of prior research showing that intention to change is associated with later behavior change (Webb and Sheeran, 2006). As with other findings from this study, it appears that participants' choices in the context of this brief intervention were valid indicators of their risks, the extent of those risks, and their likelihood of changing. Among other implications, this suggests that participant-reported intention to change following a brief intervention may be a valid proxy outcome for use in pre-post developmental trials (Ondersma et al., 2011).

Finally, it is notable that other process variables such as problem recognition, perceived importance of changing, willingness to go through the intervention a second time, and satisfaction with personalized normed feedback were unrelated to reductions in drug use risk or frequency at follow-up. This lack of association is particularly interesting in the case of satisfaction with normed feedback, which is a key focus of usability testing and which is often an important outcome in initial intervention development (Whittemore et al., 2013, Kim and Chang, 2007, Ondersma et al., 2012, Ondersma et al., 2007). However, the present findings are consistent with those from other investigations finding no association between satisfaction with the intervention and behavioral outcome (e.g., Huis in ‘T Veld et al., 2010, Solberg et al., 2015).

The study has several limitations. First, data were gathered only from two health center sites in one US state. Findings are limited to the patient population attending those clinics, and may not generalize to other patient populations. For example, although individuals with Hispanic ethnicity were well-represented, the sample included very few African–American participants. Second, the sample size was not large, particularly as we considered process variables that were only available after the sample was repeatedly split as participants proceeded along the intervention's various branches. Third, findings from this study are uniquely tied to the very specific context of the brief intervention that was developed under the previous clinical trial (Schwartz et al., 2014), and may not hold true in the context of another intervention. Finally, this analysis only considered outcomes at the three-month follow-up in an effort to maximize our ability to link process variables to proximate outcomes. Results may be different when considering later time points.

Conclusions

The present findings underscore the utility of patient-centered models of technology-delivered brief intervention. When given the opportunity, participants chose to focus on the substances for which their risk was highest, and—although most chose not to endorse negative consequences of their drug use—they were more likely to do so when their risk was higher. Further, 25% of participants chose to go through the program a second time in order to focus on a different substance. This may reflect participants' engagement with the brief intervention, as well as the tendency for persons at moderate risk to be open to more information about their use.

These findings also highlight several dimensions relevant to the design of future computer-delivered brief interventions for drug use. For example, although fewer than half of this moderate-risk sample reported a specific negative consequence of their drug use, those who did do so were most likely to focus on relationships with friends or family and money problems. Further, nearly a third of those who denied any negative consequences were nevertheless willing to make a change when asked. Future interventions should highlight the role of relationship and financial issues, and should focus more on the possible benefits of changing drug use rather than on perceptions of negative consequences (gain-framed vs. loss-framed messaging; e.g., Mays et al., 2015). Finally, although self-reported intention to change was associated with frequency of use at follow-up, a number of other process variables in the current analysis—including satisfaction with personalized normed feedback—were unrelated to outcomes. More research is needed to identify in-session tailoring/branching variables that are most critical to later outcomes. Such efforts would leverage the unique transparency of technology-delivered brief interventions to maximize their efficacy and efficiency.