Pragmatic Quality Assessment of Brief Health Behavior Change Interventions: Evidence for Criterion-Related Validity of the Behavior Change Counseling Index.

BACKGROUND: Healthcare providers frequently engage patients in conversations about health behavior change and are encouraged to use patient-centered approaches, such as Motivational Interviewing. Training in and sustainment of these skills are known to require feedback based on actual or role-played patient encounters. The behavior change counseling index (BECCI) is a pragmatic measure to assess healthcare providers' patient-centered behavior change counseling skills that was developed as an alternative to resource-intensive "gold standard" measures, which are difficult to use in routine practice. We are not aware of any studies that examine the criterion-related validity of this measure using an alternative gold standard measure. We examined the criterion-related validity of the BECCI as rated by a simulated patient actor immediately after a brief behavior change intervention role-play using objective ratings on the motivational interviewing treatment integrity (MITI) scale.
METHODS: We conducted a secondary analysis of data from a 25-site clinical trial of screening and intervention for posttraumatic stress disorder and comorbidities with patients at level I trauma centers in the USA. Participants were 64 providers representing diverse professional roles trained to deliver a multi-component intervention with study patients. As part of the training, providers role-played counseling a patient to reduce risky alcohol use with a simulated patient actor. These 20-min role-plays were conducted by telephone and audio recorded. Immediately after the role-play, the simulated patient actor rated the quality of the providers' patient-centered behavior change counseling skills using the BECCI. A third-party expert MITI rater later listened to the audio recordings of the role-plays and rated the quality of the providers' patient-centered behavior change counseling skills using the MITI 3.1.1.
RESULTS: All correlations observed were statistically significant. The overall BECCI score correlated strongly (≥ 0.50) with five of the six MITI scores and moderately (0.33) with MITI percent complex reflections.
CONCLUSIONS: This study provides evidence of criterion-related validity of the BECCI with a sample of healthcare providers representing a range of professional roles. Simulated patient actor rating using the BECCI is a pragmatic approach to assessing the quality of brief behavior change interventions delivered by healthcare providers. Copyright 2019, Darnell et al.

Introduction

Training healthcare providers in patient-centered counseling approaches such as motivational interviewing (MI) is increasingly popular, both to improve patient-provider communication in general and better support patients in health behavior change [1-3]. A cornerstone of training is to observe trainees using skills and then provide feedback and coaching to improve on skills where needed; however, traditional approaches that rely on expert trainers to review and rate work samples using time-intensive quality measures are not practical in the busy healthcare setting [4-6]. In the area of behavior change counseling, Lane and colleagues [7] developed a brief measure of counseling quality, the behavior change counseling index (BECCI), to assess healthcare provider skill as a pragmatic alternative to resource-intensive “gold standard” measures of MI that had been developed and used in the research context. We are not aware of any studies that examine the criterion-related validity of the BECCI, or the degree to which BECCI ratings are correlated with an alternative gold standard measure [8]; however, the BECCI has gained popularity and been used to assess outcomes in effectiveness trials as well as studies of training outcomes [9-13]. In the majority of these studies the BECCI is rated by a third-party objective rater; however, in routine training it is more practical for the quality of work samples or role-plays to be assessed by a subjective rater, such as a role-play partner or trainer [14]. Trainer-based ratings have also become more common in pragmatic clinical trials of behavior change counseling [11, 15]. In the present study, we harnessed a subset of data from a pragmatic multisite clinical trial to compare patient-centered alcohol brief intervention quality assessment ratings made by a simulated patient after a role-play using the BECCI with those made objectively by a third-party expert rater using the motivational interviewing treatment integrity (MITI) scale. We hypothesized that these two measures of quality would be correlated.

Materials and Methods

Design

This study is a secondary analysis of data generated through a 25-site clinical trial of screening and intervention for posttraumatic stress disorder and related comorbidities (e.g., risky alcohol use) among traumatically injured inpatients: the Trauma Survivors Outcomes and Support (TSOS) Study [16]. As part of the trial, trauma center providers were trained to deliver alcohol brief interventions using principles of MI. Data for this study come from a simulated patient role-play [17] conducted to assess pre-training skills in patient-centered alcohol brief intervention delivery.

Participants and procedure

Participants were 64 providers at 25 level I trauma centers in the USA identified for recruitment through the site principal investigator (Table 1). Provider recruitment occurred between January 2016 and February 2018. Providers underwent informed consented procedures with University of Washington research staff. Consenting providers agreed to be TSOS care managers, which included delivering alcohol brief interventions, among other treatment components, and complete a 20-min telephone-based simulated patient role-play assessment of alcohol brief intervention counseling prior to a 1-day TSOS workshop training. Providers received $50.00 for the simulated patient role-play. Previous experience with alcohol brief interventions was not required to participate. This study was approved by the Western Institutional Review Board and conducted in compliance with the ethical standards of the responsible institution on human subjects as well as with the Helsinki Declaration. The trial was registered at Clinical trials.gov (NCT02655354).

Table 1

Provider Characteristics (N = 64)

Characteristic	N (%)/M, SD
Gender
Male	8 (12.5)
Female	56 (87.5)
Race/ethnicitya
White	41 (64.1)
Multiracial/ethnic	6 (9.4)
Black	4 (6.3)
Asian	1 (1.6)
Ageb	38.3, 10.3
Professional role
Chemical dependency/mental health counselor	4 (6.2)
Psychologist/psychology trainee	6 (9.4)
Physician/physician trainee	5 (7.8)
Physician assistant	6 (9.4)
Nurse (RN)	8 (11.7)
Nurse practitioner	10 (15.6)
Social worker/social work trainee	21 (32.8)
Research coordinator	4 (6.2)
Education
Bachelors	11 (17.2)
Masters	39 (60.9)
Doctorate	13 (20.3)
Years since receipt of professional degreea	9.0, 9.7

aCategories do not add up to 100% due to missing data for 12 participants. bBased on n = 51 due to missing data. M: mean; SD: standard deviation.

Simulated patient role-play assessments

A research staff member was trained as an actor to role-play a hospitalized trauma patient who was drinking alcohol at the time of the injury event. The simulated patient was trained by the trial behavioral interventions trainer (DD), who herself was trained in alcohol screening and brief interventions using MI by an internationally known expert in these areas and member of the Motivational Interviewing Network of Trainers (CD). Simulated patient training included didactics, demonstration, role-play (from both the perspective of a patient and the simulated patient), and feedback on performance as a simulated patient. The simulated patient was taught to be appropriately responsive to MI-adherent and non-adherent behavior of providers and therefore familiar with the distinction between high and low quality patient-centered alcohol brief intervention counseling. Following this training, the simulated patient was taught to rate the BECCI with careful study of the BECCI manual.

Role-plays were conducted by telephone and audio recorded (for MITI coding). The providers were asked to role-play for 20 min, although actual lengths varied (mean (M) = 12.6, standard deviation (SD) = 6.2). The scenario was initially developed and used in a previous implementation trial of alcohol screening and brief intervention services within trauma centers (Table 2) [18].

Table 2

Simulated Patient Role-Play Scenario Provider Instructions and Additional Details

Scenario instructions

We will be doing a brief 20-min intervention role-play. I am going to give you a warning when we have a few minutes left for each role play and if you feel comfortable giving a summary at that point to close out the session, you may do so.

I am a 21-year-old college woman named Angela. I was in a motor vehicle crash while driving home from a party. There was no blood alcohol test available but I told the nurse I had been drinking. I have a left ankle fracture and a forehead laceration.

You will pretend you are at bedside with me at a trauma center; your goal is to counsel me about alcohol. That is all the information about the patient I will give you to start, any other questions regarding the patient can occur during the role play.

Additional scenario detailsa

Patient engages in periodic binge episodes on weekends at parties; does not drink during the week.

Patient is committed to not driving after drinking and is willing to try counting her drinks at parties and/or consider other means of socializing or relaxing that do not include alcohol.

aThe simulated patient actor is trained to provide these extra details when asked relevant questions during the role-play by the provider.

BECCI ratings

The simulated patient actor completed the BECCI immediately following the role-play, without reviewing the audio. We calculated the overall BECCI score comprised of 11 items rated on a Likert-type scale with 0 = not at all, 1 = minimally, 2 = to some extent, 3 = a good deal, and 4 = a great extent. Examples of items are “Practitioner invites the patient to talk about behavior change” and “Practitioner uses empathic listening statements when patient talks about the topic.” The 11 items were summed and divided by 11 to get an overall BECCI score ranging from 0 to 4; higher scores reflect greater provider use of behavior change counseling skills (α = 0.94).

MITI ratings

Each simulated patient role-play was coded using the MITI system version 3.1.1 [19] independently by an expert MITI coder and MI trainer (CD) who had no contact or involvement with the study participants. Sessions were rated for 4 of the 5 global scores on a scale from 1 (low) to 5 (high) evaluating the entire session, including evocation, collaboration, empathy, autonomy/support and frequencies of individual provider behaviors (i.e. open-ended questions, simple and complex reflections, MI-adherent and non-adherent behaviors). For the study purposes, we examined empathy as well as the five suggested MITI summary scores, which included MI spirit (an average of three global Likert scales: evocation, collaboration, and autonomy, scores range from 1 to 5) and ratio scores derived from counts of counseling behaviors: 1) The percentage of total reflections that are complex versus simple (i.e. percent complex reflections); 2) The percentage of total questions that are open questions versus closed (i.e. percent open questions); 3) The ratio of total reflections to total questions (i.e. reflection-to-question ratio); 4) The percent of MI adherent responses out of the sum of MI adherent plus non-adherent responses (i.e. percent MI adherent).

Plan of analysis

We examined descriptive statistics for the overall BECCI score and MITI empathy and summary scores, depicting the median and interquartile range for skewed MITI variables. We examined correlation coefficients between the overall BECCI score and MITI empathy and the five summary scores, using Spearman’s rho for skewed MITI variables.

Results

Providers were primarily female (n = 56, 87.5%) and all had, at minimum, a bachelor’s degree (majority master’s, n = 39, 61.0%). Social work (n = 21, 32.8%) and nursing (n = 18, 28.1%) were the most common professional roles. Of those reporting race/ethnicity (12 had missing data), the majority reported White/Caucasian (n = 41, 64.1%). Providers varied in pre-training skills per the BECCI and MITI (Table 3). All observed correlations between the overall BECCI and six MITI scores were statistically significant and all but one, the correlation between the overall BECCI and the MITI percent complex reflections were large (≥ 0.50; Table 4).

Table 3

Descriptive Statistics for Overall BECCI and MITI Empathy and Summary Scores

	M	SD
Overall BECCI score	2.3	1.2
MITI scores
MI spirit	2.3	0.9
Empathy	2.3	1.1
% MI adherent	36.3	39.7

BECCI: behavior change counseling index; MITI: Motivational interviewing treatment integrity; MI: motivational interviewing; M: mean; Mdn: median; SD: standard deviation; IQR: interquartile range. Overall BECCI score ranges from 0 (not at all) to 4 (a great extent). MI spirit and empathy scores range from 1 (low) to 5 (high).

Table 4

Correlations Between Overall BECCI and MITI Scores Based on Simulated Patient Performance

	MITI scoresa
	MI spirit	Empathy	% MI Adh	% OQ	% CR	R:Q
	r	r	r	rs	rs	rs
Overall BECCI scoreb	0.74*	0.70*	0.51*	0.56*	0.33*	0.60*

MITI: motivational interviewing treatment integrity; BECCI: behavior change counseling index; MI: motivational interviewing; Adh: adherent; OQ: open questions: CR: complex reflections: R:Q: reflection-to-question ratio. Spearman’s rho (rs) was used to assess the correlation between BECCI and skewed MITI scores. aRated by an expert MITI coder using audio recording of the session. bRated by the simulated patient immediately following the role-play. *P < 0.05.

Discussion

Pragmatic quality assessment is critical to implementing routine healthcare provider training with feedback and coaching in patient-centered behavior change intervention skills. We found evidence of criterion-related validity for the BECCI, specifically, that simulated patient actor ratings of the quality of healthcare provider alcohol brief interventions using the BECCI correlated strongly with objective ratings made by an expert using the MITI. Our findings suggest that having simulated patient actors rate healthcare providers on the BECCI immediately following a role-play is a pragmatic approach to the quality assessment, which can then be used to provide feedback and inform skills coaching efforts. Correlations were particularly strong for the MITI scores corresponding to the spirit of patient-centered counseling. The observation of stronger correlations between the BECCI and MITI scores of MI spirit and empathy than for MITI scores based on frequencies of specific provider behaviors may be due to differences in item-level measurement, specifically, that all of the BECCI items and MI spirit and empathy are based on global impressions.

This investigation capitalized on the availability of a unique dataset generated from a larger study. Although we had a sample of providers representing diverse professional roles, findings may not generalize to types of healthcare providers not captured in our data that engage patients in conversations about behavior change. Further, the simulated patient role-plays focused on the topic of behavior change related to alcohol use and it is unknown whether different correlations would be observed if ratings were based on other behavior change topics. Findings may not generalize to BECCI ratings generated by other types of raters, such as a peer or fellow trainee. Continued research on pragmatic quality assessment and types of raters is warranted, however, given that the time and financial resources needed to use simulated patients may remain a barrier for some routine training environments [20]. Research indicates peer role-plays in patient-centered behavior change interventions can be as effective as those conducted with simulated patients [21], and it may improve the utility of peer role-play to have peers rate each other on the BECCI immediately after role-plays and share the scores as feedback. Research also shows that trainees can improve in their ability to score complex measures like the MITI as a consequence of training in patient-centered counseling [22], which bodes well for the use of the BECCI by trainees as well.