Study protocol: A multisite trial of Work-Related Cognitive behavioral therapy for unemployed persons with social anxiety.

This paper provides a methodological description of a multi-site, randomized controlled trial (RCT) of a cognitive-behavioral intervention for enhancing employment success among unemployed persons whose employment efforts have been undermined by social anxiety disorder (SAD). SAD is a common and impairing condition, with negative impacts on occupational functioning. In response to these documented employment-related impairments, in a previous project, we produced and tested an eight-session work-related group cognitive-behavioral therapy provided alongside vocational services as usual (WCBT + VSAU). WCBT is delivered by vocational service professionals and is designed in a context and style that overcomes accessibility and stigma-related obstacles with special focus on employment-related targets. Our previous project found that WCBT + VSAU significantly improved social anxiety, depression, and a range of employment-related outcomes compared to a control group of socially anxious job-seekers who received vocational services as usual without WCBT (VSAU-alone). Participants in this study were all homeless, primarily African American job-seekers with high levels of psychiatric comorbidity and limited education and employment histories. The present, two-region study addresses whether WCBT + VSAU enhances job placement, job retention and mental health outcomes in a larger sample assessed over an extended follow-up period. In addition, this trial evaluates whether the effects of WCBT + VSAU generalize to a new population of urban-based, racially diverse job-seekers with vocational and educational histories that differ from our original sample. This study also investigates the system-effects of WCBT + VSAU in a new site that will be informative for broad implementation of WCBT + VSAU. Finally, this project involves a refined, technology-assisted form of WCBT + VSAU designed to be delivered more easily by vocational services professionals.

RCT Entities:   Population Interventions Outcomes

Introduction

Social anxiety disorder (SAD) is a common mental disorder, with lifetime morbid risk at 13.0% (SE = 0.5) in the USA [1]. SAD has been linked to notable life impairments [2]. This impairment is particularly evident in occupational functioning. Over 90% of individuals with SAD report significant occupational impairment [3], approximately one in five have turned down a job offer or promotion due to social fears [4], and many report more impairment in work productivity and increased absenteeism compared to non-anxious controls [[5], [6], [7]]. In a longitudinal study of 600 female welfare recipients, SAD was the only mental disorder that was associated with significantly lower rates of employment after controlling for demographic characteristics and human capital variables [8]. Similarly, in another longitudinal study of over 500 primary care patients, individuals with SAD were over twice as likely to be unemployed compared to patients without SAD [9].

Cognitive-behavioral therapy (CBT), provided either individually or in a group format, is the gold-standard research-supported psychosocial treatment for SAD, with large effect sizes compared to a range of control conditions [[10], [11], [12]]. Although CBT is effective in reducing symptoms, functional impairments, such as employment problems, often remain a challenge after treatment [13]. One potential method that could be used to address this shortcoming would be to provide CBT in vocational service settings.

CBT designed to address a range of emotional and practical barriers to employment has been successfully implemented in vocational rehabilitation settings [[14], [15], [16], [17], [18]] and mental health settings with a focus on employment [[19], [20], [21]]. Among the most evaluated vocationally-focused, mental health-related intervention programs is the Winning New Jobs Program (JOBS) [14]. In several RCTs, the JOBS program increased job acquisition and retention, and reduced depression among unemployed job seekers [14,17,22]. However, consistent with prior research indicating limited employments gains after CBT for SAD [13], our unpublished research suggests that JOBS is not effective in improving employment outcomes for persons with SAD.

Given that JOBS is not effective for persons with SAD and the lack of other specialized employment-related programs for unemployed persons with social anxiety, we designed an 8-session, twice weekly, group intervention combining CBT strategies specifically designed to target social anxiety with key techniques from the JOBS program (Work-Related Cognitive-Behavioral Therapy - WCBT). We evaluated WCBT delivered alongside vocational services as usual (WCBT + VSAU) in a pilot RCT involving 58 participants with SAD (many with extensive co-occurring disorders) comparing WCBT + VASU to a vocational services as usual without WCBT (VSAU-alone) control condition [23]. Results indicated that WCBT + VSAU significantly improved social anxiety, generalized anxiety, depression, job search behaviors, and job search self-efficacy at post-treatment and at 3-month follow-up over the VSAU-alone control condition. Of particular note, the study sample was mostly comprised of homeless, African Americans with substantial co-occurring psychiatric disorders.

As a logical next step, the present study is a multi-site RCT testing of the effect of WCBT + VSAU versus VSAU-alone) on SAD symptoms and employment outcomes. We will also explore potential moderators of the effects of WCBT + VSAU including, but not limited to, site, race/ethnicity, education, prior work experience, medication status and comorbid depression. These potential moderators were selected because of their known influence on social anxiety symptoms (e.g., medication [24] and comorbid depression [25]) or because of our interest in exploring the influence of certain demographic or site-specific factors on change with WCBT + VSAU. Finally, this paper describes the methodology of the multisite RCT and provides a detailed discussion of the treatment protocol and methods used to gather information about the systems-level impact of implementing WCBT + VSAU at two participating vocational service agencies.

Method

Study sites

The study is taking place in two sites, JVS Human Services– Detroit (JVSD), the original site of the completed pilot RCT [23] conducted by researchers at the University of Michigan and a second site that involves researchers from UCLA and JVS SoCal – Los Angeles (JVSLA).

Study design

The overall aim of this project is to conduct a randomized Type 1 hybrid effectiveness implementation trial of WCBT + VSAU versus VSAU-alone, across two urban vocational service sites among unemployed job seekers with social anxiety disorder (SAD). Hybrid Type 1 trials are analogous to effectiveness trials in which the primary focus is on determining the impact of the intervention on individual outcomes, with secondary focus on observational data collected on implementation of the intervention [26]. The design was chosen because preliminary data were underpowered to assess potential mediators, moderators, or other mechanisms of effect (e.g., symptoms, employment) that are vital to further dissemination of WCBT + VSAU, and because of the need to better understand the context in which the program is implemented in a second site in a major urban location (JVSLA) serving a more diverse sample of participants compared to JVSD.

We used the Enhanced Replicating Effective Programs (REP) [27] implementation framework to guide study roll out and to ensure that WCBT + VSAU was designed for further dissemination if proven effective. Enhanced REP has been previously used by study investigators to simultaneously implement and evaluate the uptake process for research-supported mental health treatments (R01 MH79994; VA HSR&D 11–232) [[27], [28], [29]]. REP is based on the Centers for Disease Control and Prevention's Research to Practice Framework [[27], [28], [29]] and derives its origins from Social Learning Theory [30] and Rogers' Diffusion of Innovations Model [31]. Enhanced REP consists of four stages (Pre-Conditions, Pre-Implementation, Implementation, and Sustainability). The first two stages were completed prior to the initiation of the current multisite trial as they involve organizational readiness to conduct the intervention and development of intervention materials into lay language in the form of an intervention “package” based on community feedback [32]. The present project extends this work to the Implementation and Sustainability phases of REP as the effectiveness of WBCT will be further demonstrated and key outcomes, including long-term consumer impact and costs, will be ascertained.

Participants are recruited from two urban-based study sites that provide employment assistance to a population of primarily underserved minority group members who are typically underrepresented in studies of mental health interventions [33]. Participants are randomly assigned to WCBT + VSAU or VSAU-alone. Social anxiety, other mental health conditions and employment-related variables are assessed pre-treatment and throughout a one-year follow-up period which is an extension over our previous project which was limited to a 3 month follow-up period [23]. Moreover, our main secondary aim is to describe the implementation of the intervention, including provider and consumer acceptance, leadership buy-in, and feasibility to inform its further dissemination as a multimedia tool that is acceptable to agencies.

Study specific aims

Evaluate the effects of WCBT + VSAU on employment (i.e., hours worked, job search behavior, job search self-efficacy) and mental health (i.e., social anxiety, generalized anxiety, depression) and life functioning compared to VSAU-alone. We hypothesize that WCBT + VSAU will show greater improvements than VASU-alone in employment, mental health symptoms and functional domains across 4, 12, 26 and 52 weeks post study entrance.

Evaluate whether improvements in social anxiety mediates the effect of WCBT + VSAU upon employment-related outcomes. We hypothesize that reductions in social anxiety will lead to more employment-related behaviors and greater employment success. In addition, we hypothesize that specific reductions in social anxiety will better explain the effects of WCBT + VSAU on employment than reductions in depression and generalized anxiety.

Explore moderators of the effects of WCBT + VSAU, including, but not limited to, site, race/ethnicity, education, prior work experience, medication status and comorbid depression.

Describe the uptake of WCBT + VSAU across multiple sites, including agency and employer acceptance of WCBT + VSAU and incremental costs associated with WCBT training, delivery, and maintenance using an established implementation framework.

Subject recruitment

By the end of the project, 300 men and women aged 18–60 who are seeking vocational services at JVSD (n = 96) and JVSLA (n = 204) will be recruited. Utilizing standard programming at each JVS site and our ongoing screening efforts, JVS consumers who score 5 or more on the Mini-SPIN screening tool [34] for social anxiety disorder (see description below) are informed about the research study on their first day of seeking services at JVS. Consumers who screen positive on the Mini-SPIN are asked by their JVS intake professional if they are interested in speaking with a research assistant about a study on social anxiety and unemployment. Interested consumers then complete a contact form so a research assistant can later reach them by phone. In some cases, the research assistant meets immediately with the potential participant at JVS. During these calls or in-person screening visits, the research assistant briefly describes the study and, if the consumer remains interested, a meeting time at JVS is scheduled to review the informed consent and discuss the study in greater detail. At this subsequent meeting, informed consent is obtained and the consumer remains at JVS to complete a pretreatment assessment interview with the research assistant or schedule a pretreatment assessment interview for a later date (one month maximum). JVS consumers who qualify for the study following the pretreatment assessment interview are provided a date to present to JVS for randomization to either WCBT + VSAU or VSAU-alone. Confidentiality is emphasized with each interaction between consumers, agency staff and research study team members.

Based on the highly diverse sample who participated in our previous project [23] and our current participant flow, we anticipate a diverse sample that is mostly middle aged (40–50 years), approximately 50% female and 60% racial/ethnic minority group members. We also anticipate many participants, particularly at JVSD, will have lower educational attainment and that several will have a felony record.

Fig. 1 presents a CONSORT diagram that provides the anticipated number of participants who will need to be screened and subjected to a pretreatment eligibility interview in order to reach our target of 300 participants randomized to either WCBT + VSAU or VSAU-alone. The diagram also provides anticipated attrition rates at various points in the recruitment and enrollment process. Our attrition rates at various stages are informed by our previously completed pilot RCT [23]. Given our current participant flow, we expect to end recruitment of new participants within the first half of 2019.

Fig. 1

Expected rise CONSORT flow diagram.

Inclusion/exclusion criteria

Inclusion and exclusion criteria are designed to fit our goal of designing and testing WCBT + VSAU in a sample that is representative of clients who visit urban-based vocational service agencies.

Inclusion requires participants to be JVS service-seeking adults who are unemployed, between the ages of 18 and 60 (to match typical job-seeking age range), score at least 5 on the Mini-SPIN and meet diagnostic criteria for SAD based on a structured diagnostic interview (see diagnostic interview below). Participants taking psychotropic medications are offered participation without restriction. We are monitoring medication use as a potential moderator of outcomes. Participants must agree to refrain from outside CBT for social anxiety throughout the active 4-week treatment period. Given the extensive number of participants in our previous trial with substantial mental health and substance use disorder comorbidity, for ethical reasons, we did not restrict other outside mental health and substance use treatments throughout the trial.

Exclusion criteria include presence of any of the following: current psychotic symptoms and/or manic symptoms that would interfere with study participation; current anorexia nervosa; prior course of CBT for SAD (at least 8 sessions); concurrent CBT outside of the study; prominent suicidal/homicidal ideation with imminent risk; and cognitive or communication difficulties that would interfere with an individual's ability to engage in WCBT + VSAU. Persons meeting structured diagnostic criteria for substance use disorder are included if they are able to attend sessions while not under the influence, except for those meeting substance use disorder criteria for opiates or freebase cocaine (JVSD and JVSLA do not provide vocational services to persons abusing these drugs). Our prior experience at JVSD revealed that participants with limited psychotic, manic, or substance abuse symptoms were often capable of participating in WCBT + VSAU. Exclusions based on psychotic or manic symptoms, suicidality and risk of harm to others (hostility subscale) were operationally determined by a cut-off score on a well-established measure (see BPRS below). Our approach allowing persons with limited psychotic or manic symptoms and with current substance abuse aligns with standard practices at JVSD and JVSLA where persons with these conditions are able to receive usual vocational services. Since WCBT + VSAU and all study assessment are delivered in English, potential participants who do not speak English are excluded. All participants must read at the 6th grade level. Participants who could benefit from having self-report questions read aloud by computer (reading difficulties, visual impairment) will be provided with this service if requested.

Random assignment

Randomization is by group, following the recruitment and pretreatment assessment of a sufficient number of eligible individuals to comprise a group at a given site. The randomization design specified that 16 groups be randomized at JVSD and 34 at JVSLA. Computer-generated randomization schedules for each site were created by the study statisticians and were not accessible to staff involved in participant recruitment, assessment, or intervention delivery until the day of randomization for each group. The specific randomization strategy includes cohort randomization by site with different random number seeds for each site. The SAS data step program was used to conduct cohort randomization by site. For each site, a different set of random numbers were generated from a uniform [0, 1] distribution. When the random number was < .5, the cohort was allocated to the VSAU-alone group. When the random number ≥ 0.5, the cohort was allocated to the WCBT + VSAU group.

Assessment

JVS consumers who screen positive on the Mini-SPIN with a score of 5 or higher and agree to participate, complete a structured assessment interview to determine the presence of DSM-5 social anxiety disorder and comorbid psychiatric conditions and to assess other psychometric and demographic variables (see Table 1). Diagnostic measures are completed by a trained and reliability certified clinical interviewer and self-report measures are completed by the JVS consumer on a laptop computer under the supervision of the clinical interviewer. Pretreatment assessments take an average of 2–2.5 h to complete. JVS consumers who complete a pretreatment interview receive $60. Potential participants who do not qualify for the study are provided outside referrals to mental health providers in their area.

Table 1

Measure administration schedule.

Measure and Reference	Abbreviation	Time points for Measurement	Measure Type and Person Completing Assessment
Mini-SPIN Social Phobia Inventory [34]	Mini-SPIN	Screening, Sessions 1,3,5,7	Subject-Report Questionnaire
MINI International Neuropsychiatric Interview v. 7.0 [37]	MINI	Pre, Post, 12 wk, 26 wk, 52 wk.	Clinician Administered Diagnostic Interview
Pretreatment Demographic and Work History	NA	Pre	Clinician Administered Questionnaire
Brief Psychiatric Rating Scale [54]	BPRS	Pre, Post, 12wk, 26wk, 52 wk	Clinician Administered Interview
Liebowitz Social Anxiety Scale [61]	LSAS	Pre, Post, 12 wk, 26 wk, 52 wk	Subject-Report Questionnaire
Alcohol, Smoking and Substance Involvement Screening Test [57]	ASSIST	Pre, Post, 12 wk, 26 wk, 52 wk	Clinician-Administered Questionnaire
Sheehan Disability Scale [44]	SDS	Pre, Post, 12 wk, 26 wk, 52 wk	Subject-Report Questionnaire
Social Phobia Inventory [36]	SPIN	Pre, Post, 12 wk, 26 wk, 52 wk	Subject-Report Questionnaire
Interpersonal Competence Questionnaire [58]	ICQ	Pre, Post, 12 wk, 26 wk, 52 wk	Subject-Report Questionnaire
Brief Symptom Inventory-18 [48]	BSI-18	Pre, Post, 12wk, 26 wk, 52 wk	Subject-Report Questionnaire
Multidimensional Scale of Independent Functioning – Work Performance Anchors [43]	MSIF	Pre, 12 wk, 26 wk, 52 wk	Clinician Administered Interview
Job Search Behavior Questionnaire	JSB	Pre, Post, 12 wk, 26 wk, 52 wk	Subject-Report Questionnaire
Acceptance and Action Questionnaire [59]	AAQI	Pre, Post, 12 wk, 26 wk, 52 wk	Subject-Report Questionnaire
Group Cohesiveness Scale [52]	GCS	Sessions 1,4,8	Subject-Report Questionnaire
Cognitive-Behavioral Therapy for Social Anxiety Disorder: Therapist Adherence Scale	TAS	Sessions 1-8	Independent Observer Rating
Medication Use Questionnaire	FUQ	Pre, Sessions 4,8, Post, 12 wk, 26 wk, 52 wk	Clinician Administered Questionnaire
Morisky Medication Adherence Measure [53]	Morisky	Pre, Sessions 4,8, Post, 12 wk, 26 wk, 52 wk	Subject-Report Questionnaire
Expectancy Rating [50]	ER	Pre, Session 3	Subject-Report Questionnaire
Treatment Impressions Rating Scale	TIRS	Post	Subject-Report Questionnaire
Treatment Adherence Rating Scale	TARS	Sessions 1-8	Clinician Rating Scale
Job Search Self-Efficacy [14,17]	JSSE	Pre, Post, 12 wk, 26 wk, 52 wk	Subject-Report Questionnaire
Patient Health Questionnaire 9 [46]	PHQ-9	Pre, Post, 12 wk, 26 wk, 52 wk	Subject-Report Questionnaire
Everyday Discrimination and Major Experiences of Discrimination Scale [60]	EDS	Pre	Subject-Report Questionnaire
VSAU Active Phase Questionnaires	VSAU	Twice weekly between Pre and Post	Subject-Report Questionnaire

Participants assigned to VSAU-alone and WCBT + VSAU complete measures of social anxiety symptoms throughout the active phase of the trial (see below). Other active treatment phase measures related specifically to each condition are described below.

All participants who complete the pretreatment assessment interview and are randomized, complete follow-up interviews at post-treatment (4 weeks) and at 12, 26, and 52 weeks after randomization. All participants who engage in any way in their assigned condition (i.e., completing at least one questionnaire for VSAU-alone, attending at least one group for WCBT + VSAU) are asked to complete follow-up interviews regardless of the number of sessions they attended or the condition to which they were randomized. Follow-up assessments include diagnostic measures completed by a blinded, trained clinical interviewer and self-report measures. Participants randomized to the WCBT + VSAU intervention are asked to complete a treatment satisfaction measure at the 4-week assessment timepoint. Follow-up assessments last an average of 1.5–2 h. Participants receive $40 for each post treatment and follow-up assessment.

Measures

As noted above, JVS consumers complete the Mini Social Phobia Inventory (Mini-SPIN; Connor et al., 2001) as part of their routine intake assessments at JVS. The Mini-SPIN is a three-question measure that yields a total score ranging from 0 to 12. Nearly 90% of persons scoring six or greater on this instrument have been found to meet structured diagnostic interview criteria for generalized social anxiety disorder (Connor et al., 2001). Our experience with our previous project (Himle et al., 2014) revealed that a score of 5 or more points on the MINI-SPIN was an efficient screening threshold to identify potential participants with either generalized or specific social anxiety disorder [35]. Therefore, we are using a threshold of 5 in the present study. The Mini-SPIN is derived from the Social Phobia Inventory (SPIN) which has good test – retest reliability, internal consistency, convergent and divergent validity [36].

Diagnostic assessments are conducted for DSM-5 disorders at pretreatment, post-treatment, and at 12, 26, and 52 weeks follow-up. Diagnostic assessments are completed use the MINI International Neuropsychiatric Interview v. 7 (MINI) [37]. Research associates, blinded to treatment condition, administer the MINI. The MINI is a widely used structured interview with excellent test-retest and interrater reliability [38,39].

The primary symptom measure for assessing treatment outcome is the Liebowitz Social Anxiety Scale (LSAS) [40]. It assesses fear and avoidance of several social interaction and performance situations. The LSAS total score has excellent internal consistency and has demonstrated sensitivity to change following pharmacological and cognitive-behavioral therapy [41,42]. The Social Phobia Inventory (SPIN) [36] is used to measure social anxiety symptoms across assessment time points whereas the Mini-SPIN [34] is used to assess social anxiety weekly throughout active treatment for participants randomized to WCBT + VSAU and weekly during a four-week period following randomization for those assigned to VSAU-alone.

Several methods are in place to assess employment. The primary work-related outcome measure is the reported number of paid work hours per week during the follow-up periods. Number of paid work hours is measured bi-weekly where participants receive a $5 incentive every time they reply to a text message inquiry or return a pre-printed mail-in “time-card” reporting the number of hours worked in the previous two weeks.

Measures of work status and time to re-employment are collected at follow-up using a modified version of the Work Activity Questionnaire [14]. This questionnaire also inquiries about the number of job interviews and job applications completed during the specified follow-up period. Modifications to the Work Activity Questionnaire included adding items related to job searching at employment fairs/networking events and online job search activities. Job search self-efficacy and motivation are measured using the Job Search Motivation Index [14,17], which assesses job search self-efficacy, attitude, subjective norms, and intention to engage in intensive job search. Finally, work productivity and performance are measured by the work performance subscale of the Multidimensional Scale of Independent Functioning (MSIF) [43]. This instrument has good criterion, discriminant, interrater and construct validity along with strong inter-rater reliability [43].

Overall disability is measured using the Sheehan Disability Scale (SDS) [44]. The SDS is a commonly used three item measure of functional impairment and has high internal consistency and construct validity [45].

Since general anxiety and depression may influence outcomes, comorbid depressive symptoms are assessed using the Patient Health Questionnaire-9 (PHQ-9), which has adequate reliability and validity [46], and anxiety symptoms using the Brief Symptom Inventory-18 (BSI-18) [47,48] which also has well-established reliability and validity [49].

Several demographic variables are collected including gender, date of birth, race/ethnicity, education level, living situation, relationship status, and income.

Treatment expectations are measured using the Expectancy Rating [50]. This is a four item self-report instrument designed to assess patient expectations regarding change with treatment. The Expectancy rating is administered for participants randomized to WCBT + VSAU just prior to the first session and then after the third WCBT + VSAU session such that participants can report expectations after they have been well socialized to the treatment. The Expectancy Rating has high internal consistency and high test-retest reliability [51]. Treatment satisfaction is rated using the Treatment Impressions Rating Scale, a Likert-based self-report instrument designed for and utilized in our previously published trial [23]. Ratings on this scale were entered into the computer directly by the participants and were not seen by WCBT group leaders.

The seven-item Group Cohesiveness Scale (GCS) [52] measures participants’ cohesion and engagement with treatment group members using a five-point scale (1 = strongly disagree, 3 = agree, 5 = strongly agree). The GCS demonstrates high internal consistency [52].

Participants in the VSAU-alone (control) condition complete measures at time intervals consistent with session-by-session measures completed by participants assigned to the WCBT + VSAU intervention condition (see administration table below). These questionnaires include a series of sham measures that do not relate to study outcomes (e.g., health, physical activity, sleep) as well as the Mini-SPIN and the Morisky medication use inventory (see below).

Measures of other treatments received include: a) The Morisky Adherence Measure [53] a 4-item self-report measure of medication adherence, with high internal consistency and concurrent and predictive reliability when applied to medication use [53]; b) The specific number and type of JVS services utilized, are tracked using computerized records of client service use at both JVSD and JVSLA; and c) The CBT/Medication Use Questionnaire, an instrument developed for our previous pilot trial, that tracks self-use of CBT, use of additional therapist-directed CBT, and medication use over the follow-up period.

Psychiatric symptoms, particularly symptoms related to psychosis, are assessed using the clinician rated Brief Psychiatric Rating Scale (BPRS) [54]. The BPRS has well established validity and reliability across subscales [55].

Use of substances and related problems (e.g., tobacco, alcohol, cannabis, cocaine, amphetamine type stimulants, sedatives, hallucinogens, inhalants, opiates) is measured by the Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) [56]. The ASSIST is clinician-administered has well established validity and reliability across subscales [57].

Levels of participants’ competency for peer relationships is assessed by the Interpersonal Competence Questionnaire (ICQ) [58], adapted for use in this study.

The Acceptance and Action Questionnaire (AAQ-II) [59] was also adapted for use in this study to measure participant psychological flexibility and avoidance.

Participant-reported experiences of unfair treatment and discrimination are assessed at pretreatment using the Everyday Discrimination and Major Experiences of Discrimination Scales [60] with well-established internal consistency. The measures include items related to the frequency of experiences and how participants understand their experiences based on a list of reasons people are treated unfairly (e.g., ethnicity, gender, race, age, appearance).

Participant adherence to weekly CBT homework is rated using the eight item Treatment Adherence Rating Scale (TARS), a Likert scale-based instrument created for the current study. The TARS measures adherence to six core elements including exposure exercises, cognitive restructuring exercises, homework, homework recording forms, use of CBT principles, and video exposures.

Training interviewers

Clinical interviewers were trained at each academic site to administer the clinician administered measures (i.e., MINI, BPRS, and MSIF – see above). Interviewers completed a three-step training process. First, they underwent standardized didactic trainings. Second, they listened to recordings of MINI, BPRS, and MSIF administrations that had been previously rated by experts. All interviewers were required to successfully “match” with the gold standards (i.e., correctly identifying all diagnoses on three MINIs and scoring within 1 point on all items on three BPRS and MSIFs.) Once this was completed, interviewers moved onto the third and final step, which was a mock interview with a licensed clinician to ensure they were competent in administering the interview. To maintain reliability and minimize interviewer drift throughout the study, all interviewers receive ongoing supervision with a licensed clinician.

Training independent evaluators

Two doctoral-level clinicians with expertise in psychodiagnostic assessment and CBT for social anxiety served as independent evaluators to assess the reliability of diagnostic interviews and WCBT fidelity. In order to assess diagnostic reliability, the independent evaluators were trained to administer the MINI, BPRS, and the MSIF-Work Performance Scale. Evaluators received the same didactic training on all three measures as the interviewers and were similarly required to match the gold standard ratings for three MINI, BPRS, and MSIF assessments. All assessment interviews were audio-recorded and approximately 10% of tapes from each site are randomly selected for rating by independent evaluators. Rates of agreement will be analyzed and significant disparities corrected through additional interviewer training and adherence monitoring. Any potential misdiagnoses revealed via this process will be evaluated by the principal investigators (Drs. Himle or Craske) and confirmed diagnostic errors will be corrected.

The same two independent evaluators assess WCBT fidelity. WCBT fidelity is measured using a manual and ratings scale that were specifically adapted for this study [23]. Adaptations in the instrument from our previous trial include ratings of fidelity to a new pre/post exposure checklist that evaluates adherence to inhibitory learning principles and to new weekly “ice breaker” exposure exercises that have been added to WCBT for the present trial. The manual covers each major component of each session and asks the evaluator to determine whether each relevant section was completed (i.e. counselor adherence) and how well it was completed (i.e. therapist competence). Four session recordings were selected for use as gold standards and adherence and competence ratings were made for each by two study investigators. Where discrepancies were present, they were resolved through a discussion with a third investigator. Independent evaluators had to match within one point of all gold standard ratings for all four recordings before they were certified to start making independent ratings. The independent evaluators are conducting ongoing WCBT fidelity assessments by scoring approximately 37% of WCBT sessions. Session are randomly selected for fidelity assessment with the goal of rating an equal number of sessions 1–8.

Intervention groups

WCBT is an 8-session group intervention that meets twice weekly for 2 h each session. The group format was chosen for its advantages in providing a convenient format for facing challenging social situations in the presence of others and because the majority of programming at JVSD and JVSLA are provided in groups. WCBT + VSAU participants join cohorts with a target size of six to eight individuals who start and end together. WCBT was originally based on Heimberg and Becker's group CBT treatment manual for SAD [62] one of the most extensively studied and widely accepted research-supported psychosocial treatments for SAD and the JOBS program intervention described above [14]. With these intervention manuals as a foundation, WCBT was initially developed for use in an NIMH-sponsored intervention development grant (NIMH R34MH083031). It was developed using an iterative process with a range of key stakeholders including university-based researchers, outside experts in CBT for social anxiety, experts in the delivery of the JOBS intervention, socially anxious job seekers and vocational service professionals. WCBT + VSAU has been adapted for delivery in the current trial (see refinements below).

WCBT is designed to be delivered by trained vocational service professionals with little or no previous training in CBT. WCBT is designed to be delivered alongside a full range of vocational services offered at a comprehensive vocational service center. WCBT's intervention manual uses lay language that is delivered in a user-friendly format to make it more accessible to non-mental health professionals seeking to deliver this intervention. WCBT's group-based delivery format was designed to match the typical way that educational programming is delivered at vocational service agencies (small group format involving extensive participation from consumers). Vocational service professionals were chosen over outside mental health professionals for several reasons including their: deep expertise in employment; experience leading small groups; and their presence at the JVS agencies involved in the study. Utilizing existing vocational service professionals as WCBT group leaders also has the potential to reduce costs and increase sustainability over importing outside mental health professionals to deliver WCBT.

Exposure therapy is the cornerstone of WCBT and involves repetitive exposure to challenging social situations that are mostly work-related, within an inhibitory-based learning model [63]. WCBT relies heavily on group leaders and fellow group members to serve various roles in exposure exercises. Participants complete exposure and cognitive-based homework after each session. Sessions 3–8 include a series of vocationally relevant psychoeducational topics adapted from the JOBS program [14]. These include: reducing self-defeating behavior; asking others for help finding work; recognizing strengths; turning weaknesses into strengths; and talking with co-workers and supervisors. Table 2 presents the key elements in each WCBT session.

Table 2

WCBT - sessions outline.

Session #	Psycho-Educational Content	CBT Content
1	• Orientation to the program • Introduction to social anxiety	• Ice breaker exposure – Member Introductions • Introduce three components of anxiety • Introduce negative cycle of anxiety • Review consequences of avoidance • Exposure homework
2	• How social anxiety presents itself at work	• Ice breaker exposure – Members speak on their expectations for the group • Introduction to anxious thoughts • Identification of thinking errors • Challenging anxious thoughts • Exposure homework
3	• No specific non-CBT psychoeducation	• Review exposure homework • Ice breaker exposure – Members speak about their last job • Building an avoidance list • Individual in vivo exposure exercises • Exposure homework
4	• Self-defeating behavior at work	• Review exposure homework • Ice breaker exposure – Describe a new work-related skill member wishes to acquire • Individual in vivo exposure exercises • Exposure homework
5	• Asking for help finding a job	• Review exposure homework • Ice breaker exposure – How to respond to critical feedback at work • Individual in vivo exposure exercises • Exposure homework
6	• Recognizing strengths and weaknesses • Turning weaknesses into strengths	• Review exposure homework • Ice breaker exposure – Mock job interview • Individual in vivo exposure exercises • Exposure homework
7	• Speaking with supervisors and coworkers	• Review exposure homework • Ice breaker exposure – Describe biggest strengths and weaknesses at work • Individual in vivo exposure exercises • Exposure homework
8	• Progress review and termination	• Review exposure homework • Ice breaker exposure – Impromptu speeches • Individual in vivo exposure exercises • Using exposure and cognitive strategies after the group

WCBT has been adapted for the current multisite trial. WCBT is now delivered in a fully technology-assisted format. All WCBT training and delivery materials are contained within a computer platform. In technology-assisted WCBT, leaders use a practice support, internet-based computer system to guide session content. The internet program is modeled after a program previously developed for a large multisite RCT examining the efficacy of delivering individual CBT with the assistance of novice clinicians in primary care clinics (CALM) [64,65]. As in CALM, computer-assisted delivery supports JVS vocational service professionals leading WCBT groups, all of whom were beginners in CBT before participating in our WCBT trials. Computer/internet guidance of group leaders as they interact with participants reinforces familiarity with WCBT and its principles, while facilitating adherent and competent delivery of WCBT. The computer-assisted format also greatly enhances standardization of WCBT and has potential to encourage sustained use once the current trial is competed.

Specifically, the computer-assisted WCBT program guides group leaders through the content of each session, including outlines, interactive worksheets, and videos. As an example, when WCBT calls for the group leader to work through an exercise with an individual group member, the group leader works through the activity while entering relevant information on the computer which is then projected for all group members to follow along. Then, group leaders assist all group members in completing the exercise. For most WCBT role play activities, the group leader projects demonstrations from the internet program, and then leads role play exercises among group members. At completion of each session, group leaders generate homework assignments based on the goals and progress of the individual participant. The computer-assisted WCBT program is also accessible by group members in between the group sessions for review of session materials and for completion of CBT homework reports. Finally, the computer program provides an opportunity for group members to practice mock job-related social challenges (e.g., asking a friend for a job lead, participating in a job interview) in video-recorded simulation. As an example, a recording of a person asking a typical job interview question is shown to the participant who then responds while being recorded by the computer. The group member is then able to view their response to compare their actual performance to how they imagined they appeared. The simulations supplement in vivo exposures exercises (e.g., applying for a job) that are assigned as between session homework. These exercises that can be conveniently accessed by computer, tablet or smartphone. For participants without access to one of these devices, printed copies of CBT homework reports are provided and JVS-based computers with sufficient accommodations for privacy are made available before or after WCBT sessions to complete the simulations.

VSAU-alone is the most appropriate comparison condition for an effectiveness study such as the present one because CBT for social anxiety has been found to be superior to attention control conditions in several studies [11,12] and because research-supported psychosocial treatments are currently not provided for most unemployed people with SAD. The VSAU-alone condition, like WCBT + VSAU, includes all typically offered services (e.g., resume development support, career coaching, providing job leads) at JVSD and JVSLA but does not include WCBT. Vocational services received by VSAU-alone and WCBT + VSAU participants are measured during the trial using existing service tracking methods employed at JVSD and JVSLA.

WCBT group leader training and supervision

All group leaders began training by completing internet-delivered training programs for research ethics/human protections and on CBT for anxiety disorders [66]. This training includes education regarding CBT theory and principles, case examples, expert clinician videos, animations, and “challenge questions” to reinforce new skills. After satisfactorily completing the computerized training program, all leaders attended a 2-day, in-person, technology-assisted, didactic training with the study PIs that included familiarization with the WCBT treatment protocol, familiarization with the computer-based treatment platform, extensive role-playing of treatment components and developing in-session exposures. Following the initial training, all group leaders completed five remote, cross-site vignette-based training sessions with licensed clinicians in which each group leader role-played vignettes highlighting the key intervention components from the WCBT treatment protocol. Group leader competency for each component of each skill was rated on a scale of 1–7. Group leaders received direct feedback following each vignette and allowed to repeat their practice following this feedback. All group leaders were required to score an average rating above 4 on all components of each vignette. The PIs and Co-Is also conducted live observations of group leaders in all WCBT sessions during the pilot phase of the study. To maintain fidelity in treatment throughout the study, group leaders receive weekly supervision by an expert CBT clinician at each academic site. Oversight and ongoing training for group leaders also includes bi-weekly cross-site conference calls with study investigators in which each academic site alternates presenting on and leading discussion on WCBT sessions that are underway and facilitating role-plays of key WCBT intervention components. To date, we have trained six group leaders to serve at JVSD and twelve at JVSLA.

Data analysis plan

Because random assignment and WCBT intervention delivery occur at the group level, all analyses will use multilevel approaches – longitudinal mixed-effects linear modeling (MLM) and multilevel structural equation modeling (SEM) that accommodate the nested structure of the data. Using SAS v. 9.4 software (SAS Institute, Cary, NC), all primary analyses will be done on the full intent-to-treat sample, using multiple imputation to handle missing observations. The significance level will be set to 0.05 and 95% confidence intervals will be reported. To assess the effects of WCBT + VSAU on employment and mental health outcomes (Aim 1), 3-level models will incorporate repeated assessments (at 12, 26, and 52 weeks post study entrance) of individuals nested within the 50 groups that have been randomly assigned to WCBT + VSAU or VSAU-alone. Individual pretreatment scores on each outcome variable will be incorporated as level two covariates to increase the precision and statistical power of tests of the randomized intervention and reduce regression to the mean artifacts.

To evaluate whether improvements in social anxiety mediate the effect of WCBT + VSAU on employment related behaviors and outcomes (Aim 2), multilevel SEM will be used to assess the significance of indirect effects of WCBT + VSAU on subsequent employment via reduction in social anxiety. This approach provides several advantages that are essential to testing mediation in this design: it accounts for nesting of individuals within randomized groups; it focuses on change by using centering to disentangle between- and within-person effects; it specifies temporal order (i.e., change in social anxiety symptoms precedes change in employment related outcomes); and it provides ways to statistically compare multiple alternative mediation paths, which will be essential to testing our hypothesis that reduction in social anxiety will better explain WCBT + VSAU effects on employment than improvements in depression and generalized anxiety.

To explore whether the effects of WCBT + VSAU vary across potential moderating variables (Aim 3), interaction terms involving these variables (e.g., site, race/ethnicity, education, prior work experience, psychiatric medication status, comorbid depression) will be added to the MLM models testing mental health and employment outcome effects. Because groups are nested within site (16 groups at JVSD and 34 at JVSLA), site-level effect differences will be tested as level 3 interactions with condition; moderation by time-invariant individual characteristics (race/ethnicity, education, pre-intervention work experience) will be incorporated as cross-level interactions.

Sample size and power analysis

The intent-to-treat sample of 300 individuals, nested within 50 groups, with 5 assessments over time, will provide power of .8 to detect (at 2-tailed p < .05) an intervention effect on intercept or slope of at least d = 0.34. Effect sizes in the WCBT + VSAU pilot RCT ranged from 0.46 for hours worked to 1.36 for social avoidance; this suggests that power in this larger RCT will be adequate to detect condition differences on all outcomes, including employment. These estimates assume a minimum intraclass correlation (ICC) between groups of .01, which is larger than the ICCs observed in the pilot RCT. If the ICC for the current study is found to be substantially higher at 0.03, the minimum effect size detectable with power of .8 will rise only slightly, from d = 0.34 to d = 0.36.

Mixed method implementation study

We are using the Enhanced Replicating Effective Programs (REP) implementation framework to guide our observational study of WCBT + VSAU implementation at both sites, including agency and employer acceptance of WCBT + VSAU and incremental costs associated with WCBT training, delivery, and maintenance. Using Enhanced REP as a guide, we are using mixed methods to evaluate WCBT + VSAU implementation. Agency leaders and employees providing WCBT completed an initial in-person interview and will complete annual follow-up interviews using the Organizational Transformation Model (OTM) assessment [67,68] that measures organizational processes found to be associated with improved uptake of new programs, including the agency's readiness to implement new programs, leadership commitment, degree to which frontline providers are involved in the operationalization of a new program, and alignment of new program goals with existing priorities and day-to-day processes and workflow.

A semi-structured interview utilizing the OTM, that includes details regarding the implementation of this specific project, is administered annually. The interviews last approximately 30 min and include general inquiries regarding the various domains of interest followed by specific prompts to enhance the breadth and consistency of material provided. A wide array of JVS staff are interviewed, including those with diverse positions within the organization and with varying degrees of involvement in this specific project (e.g., administrators, managers, counselors, and support staff). As new staff come into the agency and/or onboard the project, initial interviews are conducted. Every effort is made to obtain follow-up interviews from staff members who depart the project or the organization. All staff members provide informed consent for these interviews. Interview responses are transcribed and uploaded into a qualitative data analysis program.

Finally, data on the cost of implementing WCBT, including employee time, training, and technical support are being ascertained using employee interviews in which we will determine total time spent on: 1) REP-based WCBT training and technical assistance; 2) delivering WCBT sessions, including study relevant paperwork and client contacts/follow-up; and 3) time spent with site leadership and other staff members to garner feedback on sustainability issues including barriers and facilitators to uptake and planning future dissemination activities. In addition, we are ascertaining the total time study staff spent conducting REP activities such as WCBT packaging, training, and technical assistance, including note-taking and time spent at the sites. Costs associated with WCBT implementation will be estimated by multiplying the total time spent on each activity by the average salary of the employee responsible for the activity.

Trial status

The randomized trial of WCBT + VSAU versus VSAU-alone is underway as of March 2019. We anticipate that enrollment in the active phase of the study will be completed in the first half of 2019. Follow-up assessments are continuing and will conclude in early 2020. The project was approved by human subjects review panels at University of Michigan and UCLA and the trial was registered on ClinicalTrials.gov. A comprehensive participant safety protocol is in place that provides participants same-day contact with a senior clinician/investigator for any instances of suicidality, homicidality, clinical deterioration and other participant safety-related concerns.

Discussion

This is the first multisite trial of a specialized CBT aimed at improving employment outcomes among unemployed persons with social anxiety disorder. The trial improves upon the previously published pilot RCT of WCBT + VSAU versus VSAU-alone [23] in several ways including: a refined WCBT intervention based on lessons learned from the pilot trial; internet-assisted intervention delivery to enhance fidelity and to aid in future dissemination; a larger sample of more diverse participants whose educational and vocational histories differ from the original study; a second site in a different region of the U.S. with no experience in the delivery of WCBT; an extended follow-up period allowing investigation of the effects of WCBT + VSAU over the long term; and a larger sample that allows for a more comprehensive examination of WCBT + VSAU's effect on number of paid work hours and the investigation of mediators and moderators of change. Finally, this trial will provide critical information about the system effects of WCBT + VSAU that will be informative for broad implementation of WCBT + VSAU nationwide.

If WCBT + VSAU is found to be effective in improving social anxiety symptoms and employment outcomes over time, it will address two important shortcomings in the current literature. First, if WCBT + VSAU enhances employment outcomes over time for persons with SAD, it will address a key question about the durability of WCBT + VSAU that was not answered in our previous project with its short follow up period [23]. Second, data from present project can potentially provide insights in how to modify standard CBT for SAD to make it more effective in addressing employment challenges. Third, if WCBT + VSAU improves social anxiety and perhaps other mental health symptoms over time compared to VSAU-alone, it could be deployed in vocational service agencies nationwide, providing increased access to research-supported therapy for SAD. Finally, if WCBT + VSAU is found to be effective in this trial, it would provide evidence that CBT for SAD can be effectively delivered after a brief training period by non-mental health professionals without prior training in CBT. This potential outcome could help to address critical shortages in the availability of CBT for SAD [69].

The present trial will also provide important information on the effects of CBT for SAD among a racially diverse sample of mostly low income persons with SAD. Our sample will also include many persons with substantial psychiatric comorbidity, low levels of educational attainment, homelessness and previous criminal justice system involvement. This stands in contrast to the large majority of CBT outcome studies that are conducted with mostly Caucasian, highly educated samples [70,71] with limited psychiatric comorbidity [72]. The present trial is designed to test the effects of WCBT + VSAU in a sample that is representative of typical clients who seek employment services in urban-based, vocational service centers.

Conclusion

This multisite trial of WCBT + VSAU will provide a more comprehensive evaluation of the effects of this intervention on client outcomes and vocational service agencies. If this multisite RCT reveals that WCBT + VSAU can be successfully implemented in multiple vocational service agencies and if it reduces unemployment and social anxiety symptoms over the long term, it will be of substantial public health benefit considering the devastating consequences of chronic unemployment [[73], [74], [75]].

We believe that the current trial focused on social anxiety-related barriers to work is particularly significant given that the service sector is the largest area of job growth in the U.S [76]. It is clear that the ability to comfortably and skillfully interact with others is a key element contributing to employment success in this domain and it is likely that many persons with SAD will need specialized treatment to both secure and maintain employment in this sector.

Funding

This work was supported by the National Institute of Mental Health (RO1MH102263, J. Himle, PI; R01MH102274, M. Craske, PI).