Computerized Clinical Training Simulations with Virtual Clients Abusing Alcohol: Initial Feasibility, Acceptability, and Effectiveness.

Although masters-level social work students typically build clinical skills via role-playing with their peers or instructors, several innovative training simulations are emerging in the literature that may enhance existing skill-building methodologies. We evaluated the initial feasibility, acceptability, usability, and effectiveness of three computerized simulations (two cognitive behavioral therapy, one motivational interviewing) during an interpersonal practice course among 22 students in a Master of Social Work program accredited by the Council on Social Work Education. Trainees repetitively practiced their clinical skills with virtual clients while receiving feedback via real-time nonverbal cues, transcript review, and performance assessment across pre-specified theoretical learning objectives. Across the three simulations, at least 86.4% of students completed the required protocol and completed M = 468.95 (SD = 178.27) minutes of simulated sessions. Students improved their scores (range 0 to 100) across all the simulations from M = 63.41 (SD = 11.13) to M = 93.64 (SD = 3.24). Students found the simulations to be acceptable with strong usability. Paired sample t-tests revealed students reported greater self-efficacy in general clinical skills, exploration skills, insight skills, and action skills between pre-test and post-test after completing the simulations (all p < 0.001). Students reported that the clinical skills learned from the simulations translated into successful interactions with real-world clients during their field placements. We discuss the results of this initial feasibility study within the context of simulation-based learning and the potential for broader implementation within MSW programs. © Springer Science+Business Media, LLC, part of Springer Nature 2020.

Recently, the field of social work has stressed the urgency to enhance preparing clinical social work students for their field experience and clinical careers. This urgency stems from evidence that didactic teaching methods are not adequate to facilitate the clinical competence needed prior to engaging in field education (Duron and Davis Giardina 2018). Moreover, this urgency is associated with students seldom engaging in an adequate amount of clinical practice and supervision during field experiences (Beddoe et al. 2011; Bogo 2015). One approach to enhance clinical social work education is through the implementation of evidence-based clinical training simulations that facilitate skill development by integrating conceptual skills with applied practice through a combination of training, observation, feedback, and coaching (Kourgiantakis et al. 2020).

Currently, the vast majority of available simulations are in-person dialogues where an actor is trained to portray a client, a student practices clinical skills with the simulated client, and the student’s performance is appraised, with feedback provided to the student (Kourgiantakis et al. 2020). Importantly, simulation-based learning has several strengths when compared to peer role-plays (e.g., it is research-informed, it provides consistent opportunities for all students, the simulated client is more authentic, and it provides a setting in which the student feels safe to take risks and make clinical mistakes). Moreover, these strengths help translate learning into enhanced competence and yield reliable and valid educational outcomes in several areas of social work practice (e.g., child welfare, mental health, behavioral health; for scoping review, see Kourgiantakis et al. 2020).

More recently, the development and evaluation of technology-based simulations—which use virtual reality or computerized simulations to assess symptoms or deliver therapy to a ‘virtual client’—have emerged in the field of social work. These technologies facilitate an immersive learning experience where students can repeatedly interact with virtual clients to safely practice their skills (Dodds et al. 2018). Specifically, students learn to use psychosocial techniques such as client engagement, mental and behavioral health screening, cognitive behavioral therapy (CBT), and motivational interviewing (MI; Huttar and BrintzenhofeSzoc 2020; Mastroleo et al. 2020; O’Brien et al. 2019; Putney et al. 2019a, b; Washburn et al. 2020).

These technological innovations have a number of advantages over more traditional methods. First, they address emerging challenges related to the practice environment, such as the delay between development and both the dissemination and implementation of evidence-based psychosocial approaches in practice (Bellamy et al. 2006, 2017; McHugh and Barlow 2010), overall limited clinical training and support for providers (Kilbourne et al. 2018), the reality of large caseloads in community mental health practice (Morse et al. 2012), field site expectations that students have some advanced skills when entering their field placement (Williamson 2020), and the recent emergence of telehealth practices (Zhou et al. 2020). Second, the use of technology-based simulations may make the development and refinement of clinical skills more accessible, effective, and convenient (Fairburn and Cooper 2011; Kobak et al. 2013; Liddle et al. 2002) given that in-person simulations may have some limitations to scalability as they are resource-intensive (e.g., required time and costs of training actors, space to conduct simulations). Third, there is an immediate priority for social work education to streamline such innovative methods in the wake of COVID-19 and the need for remote learning activities.

Finally, research suggests that technology-based simulation training used by clinical social workers has the potential to enhance traditional methods of training (Olson et al. 2015), may be equally effective as in-person training (Weingardt 2004), and may be more effective than manual-based self-study (Sholomskas and Carroll 2006). Clinical social work trainees often develop skills in delivering evidence-based psychosocial treatments before or during their field placements via the more established in vivo simulations. Though technology-based simulations are still emerging and require feasibility and effectiveness evaluations, they may be a promising method to enhance students’ accessibility to evidence-based psychosocial treatments. Specifically, these evidence-based training methods may improve the effectiveness (e.g., standardizing its delivery to include feedback and supervision in addition to content; Bennett-Levy and Perry 2009; Kourgiantakis et al. 2020) and feasibility (e.g., streamlining training given documented challenges of access, time, and cost; Kilbourne et al. 2018; Liddle et al. 2002) of clinical social work education.

To further investigate technology-based simulations, we evaluated the initial feasibility, acceptability, usability, and effectiveness of implementing three computerized simulations with virtual clients within a real-world Master of Social Work (MSW) course. While effectiveness is arguably the most essential component to evaluate when determining new methods of instruction, we also examined feasibility, acceptability, and usability because these factors are germane to the field of implementation science, and they are consistent with a well-established taxonomy salient to the early stages of implementation evaluation (Proctor et al. 2011). The first simulation trained students to conduct brief MI. The second and third simulations focused on CBT with the second simulation targeting the introduction of CBT to a client, and the third simulation targeting the conducting of a CBT functional analysis. The authors chose simulations focused on MI and CBT because they are among the most versatile and empirically validated psychosocial treatments (Hofmann et al. 2012; Randall and McNeil 2017). Moreover, the Brief MI and Introducing CBT simulations were recently examined and found to be efficacious (when delivered and evaluated under ideal research conditions) at enhancing clinical skills (Mastroleo et al. 2020; Putney et al. 2019a, b). These existing evaluations informed our decision to implement these simulations in the course. Although the efficacy of the CBT Functional Analysis simulation had not yet been examined, we implemented this simulation because it was developed using the same scientific standards (and by the same team) as the Introducing CBT simulation.

To evaluate the feasibility of the simulations, we collected process data on students’ performance and adherence to the training protocol. To evaluate the acceptability and usability of the simulations, we used a mixed-methods approach, employing a satisfaction survey with open-ended questions. Lastly, we evaluated the preliminary effectiveness of the simulations using a one-group pre-test–post-test design that tested whether students self-reported improvements in their clinical self-efficacy.

Methods

Participants

The participants included 22 graduate students enrolled in the CSWE-accredited MSW program at the University of Michigan.

Design

This study used a one-group pre-test–post-test mixed-methods design to evaluate the initial feasibility, acceptability, usability, and effectiveness of implementing three psychotherapeutic clinical training simulations with a virtual client in an adult interpersonal practice MSW course.

Procedures

All students were enrolled in the first author’s course on interpersonal practice with adult individuals. All students were informed they would be oriented on how to use the simulations during class, and the simulations were required as homework.

Recruitment

During the first class, the first author described the study in detail and presented students with the opportunity to participate. Students were emailed a consent form to be completed privately outside of class and were instructed to direct their questions to the first author prior to signing their consent form. Students were notified verbally and in the consent form that their decision on whether or not to participate would not affect their grade. All students agreed to participate and provided their informed consent. The study was reviewed and approved as exempt by the University of Michigan's Institutional Review Board.

Required Coursework

The course taught the foundation, application, versatility, and limitations of two evidence-based practices: MI and CBT. The 13-week course required students to read two textbooks during the course to help facilitate their learning of MI and CBT. The first book was Motivational Interviewing: Helping People Change (Miller and Rollnick 2013) and the second book was the Essential Components of Cognitive-Behavior Therapy for Depression (Persons 2009). The course also had accompanying lectures that highlighted the versatility of MI and CBT across cultures and disorders (Armstrong et al. 2011; Kar 2011; Lee et al. 2013; Manber et al. 2011; Montgomery et al. 2011; Murphy et al. 2010) and introduced other psychotherapeutic techniques (e.g., terminating psychotherapeutic relationships, exposure therapy; Bourne 2015; Teyber and Teyber 2017).

Simulation Implementation

During the first class, all students attended a 45-min orientation on how to navigate the interface of the simulation and were informed that to earn 5% of their grade, they needed to complete the following: (1) score 90 or better (out of 100 points) at least once for each virtual client, and (2) complete at least four sessions with each of the three virtual clients. The instructor advised (but did not require) students to review the eLearning curriculum within each simulation prior to beginning their simulated clinical sessions, and their eLearning engagement was observed naturalistically.

Simulations

The three simulations used in this study were developed by SIMmersion (www.simmersion.com), which uses a proprietary technology called “PeopleSIM.” The PeopleSIM technology facilitates a simulated conversation using videos of actors and speech recognition software. Within PeopleSIM, the virtual clients are present within an interface that has been scripted with statements that vary in terms of their clinical effectiveness (Fig. 1). Students then choose the direction of their clinical engagement using the script, or if they want an option that is not available from the script, they can request a different statement from a search function. Also, each simulated client is facilitated by a matrix of three characters by three moods (described below), and the student’s clinical statements from throughout the conversation (not just the preceding statement) inform the algorithm that decides how the virtual client’s character and mood will evolve throughout the conversation. Thus, the simulation goes beyond a “choose your own adventure” design and allows trainees to establish rapport with the virtual client who subsequently may be more accepting of a clinical misstep based on their ongoing rapport.

Fig. 1

PeopleSIM Interface. This figure provides a glimpse into the interface that students use to engage the simulation. Panel A Shows the virtual character Gabe and the primary on-screen interface for all of the simulations. The script to the right of Gabe allows students to speak a statement that ranges in clinical effectiveness and influences the ongoing clinical relationship. Additionally, students can interact with Tanisha in the Introducing CBT simulation (Panel B) or Roger in the CBT Functional Analysis simulation (Panel C)

Each simulated session provides three levels of feedback to students. First, students receive nonverbal cues on the effectiveness of their clinical engagement from a coach in the bottom-right-corner of the virtual client’s interface screen (Fig. 1). Second, students get feedback on the effectiveness of their individual statements to the client via a transcript accessed from the interface. Students can engage the transcript during the session or after the session. Third, students receive an overall numerical score (0–100) and a qualitative assessment of their performance after the simulated session has ended. The framework for this assessment is specific to the learning goals associated with each simulation. The learning goals are covered in the eLearning content that students were asked to review prior to beginning their simulated sessions. Overall, this approach reinforces effective clinical statements, but more importantly, students can make mistakes in a judgment-free setting and use what they learn from the feedback to reengage the client and try a different approach.

Brief MI with Gabe Turner

Before engaging in a conversation with Gabe (Fig. 1, Panel A), students were directed to first engage the brief eLearning curriculum to learn about: (1) Gabe’s background; (2) the general skills used within MI such as developing a collaborative relationship with clients, being sensitive, and displaying empathy; (3) the four processes of MI: engaging, focusing, evoking, and planning; and (4) the specific clinical skills used to facilitate motivating conversations such as using open-ended questioning, affirmations, reflective listening, and summarizing as well as using readiness rulers and strategies to elicit change talk. After the student engages in simulations with Gabe, PeopleSIM generates the numerical scores and performance assessments based on the student’s timing and use of general skills, MI processes, and specific skills.

Gabe’s character is randomly selected to begin as “open to change,” “willing to consider change,” or “will not change.” When Gabe is “open to change,” he is open to discussing his alcohol use and making changes in his drinking. When Gabe is “willing to consider change,” he is hesitant to discuss his alcohol use and potential changes to his drinking. Lastly, when Gabe “will not change,” he is hesitant to discuss the details of his alcohol use and is not open to making changes in his drinking.

Gabe’s mood begins “neutral,” where he is cautious about sharing details about his drinking and does not trust the student. If the student successfully builds rapport with Gabe through being collaborative, affirming, and demonstrating active listening, Gabe will transition into an “engaged” mood in which he is more willing to share openly about his life and his alcohol use. However, if the student fails to build rapport with Gabe, he will become guarded and transition into a “disengaged” mood in which he will pacify the student’s questions with minimal information and seek to move through the conversation quickly.

Introducing CBT with Tanisha Mosley

Prior to engaging in a conversation with Tanisha (Fig. 1, Panel B), students were directed first to engage the brief eLearning curriculum to learn about: (1) Tanisha's background, (2) strategies to develop a relationship with a client, and (3) key concepts and topics that should be covered with a client when introducing CBT. Also, students learn that they will be scored based on their implementation of the strategies and key concepts and skills from the eLearning, including creating a collaborative environment, maintaining a conversational tone, displaying empathy, setting an agenda for the session, individualizing CBT for Tanisha, validating that Tanisha understands the process, and assigning homework.

Tanisha’s character is randomly selected to begin as “inquisitive,” “passive,” or a “naysayer.” When Tanisha’s character is “inquisitive,” she is curious about the process of CBT, engages in the discussion, and will ask many questions throughout the session. The student needs to be prepared to answer all of her questions in order to get her buy-in to use CBT during future sessions. When Tanisha’s character is “passive,” she is disengaged from the conversation and relatively apathetic to the process of CBT. The student needs to work to keep her interested in the discussion in order to get her buy-in to use CBT during future sessions. Lastly, if Tanisha is a “naysayer,” she will be skeptical about the CBT process and will push back on its efficacy. The student will need to handle these doubts appropriately in order to get her buy-in to use CBT during future sessions.

Tanisha’s mood begins “neutral,” where she is noncommittal toward using CBT. While she is open to hearing more, her confidence in CBT methods needs to be built before she will agree to use CBT methods. If the student helps Tanisha overcome barriers collaboratively and coherently, then Tanisha will transition into the “buying in” mood in which she has all the information needed to make an informed decision with optimism about using CBT methods in the future sessions. If the student does not respond to Tanisha's pushback or questions, Tanisha will move to the “disinterested” mood in which she has little desire to use CBT methods in future sessions.

CBT Functional Analysis with Roger Ellison

Before engaging in a conversation with Roger (Fig. 1, Panel C), students were directed to first engage the brief eLearning curriculum to learn about: (1) Roger's background, (2) strategies to develop a relationship with a client, and (3) key concepts and strategies to complete a functional analysis. Also, students learn they will be scored based on their implementation of the strategies from the eLearning that include: (1) implementing key CBT strategies (e.g., identify functional analysis as next step; explain the purpose of functional analysis; identify recent drinking episode; introduce worksheet; introduce and describe antecedents, behaviors, and consequences; introduce and model ABC worksheet; describe the purpose of worksheet); (2) developing a positive relationship (i.e., collaboration, conversational tone, empathy); and (3) utilizing core CBT concepts (i.e., individualize CBT to Roger, set agenda, assign homework).

Roger’s character is randomly selected to begin as “cognitive distortions,” “coping skills,” or “drink refusal skills.” When Roger is the “cognitive distortions” character, he describes his reasons for and experiences with drinking. He also demonstrates various incorrect thoughts about alcohol and why he feels he needs to drink. When he is the “coping skills” character, Roger describes his reasons for and experiences with drinking. He also demonstrates a lack of coping skills when it comes to stress and anger management. When Roger is the “drink refusal skills” character, he describes his reasons for and experiences with drinking, and he demonstrates an inability to say no when he is offered a drink.

Roger’s mood begins “neutral” where he participates in the discussion but is hesitant to share vulnerable information. If the student is collaborative, encouraging, and affirming with Roger, his mood will become “engaged” in developing a functional analysis; he becomes vulnerable and then invests in each stage of the discussion. If the student is demanding, judgmental, or too clinical with Roger, his mood will become “disengaged,” and he demonstrates frustration and disengages from the conversation. In this mood, he becomes less likely to share helpful information and seeks to rush the conversation to its conclusion.

Measures

In this feasibility study, we assessed the process data on participants’ performance and adherence to the training protocol, and their views on the acceptability and usability of the simulations.

Simulation Feasibility (i.e., Adherence and Process Variables)

To characterize the students’ adherence to using the simulations, we recorded the total number of completed sessions, the scores for each session, the frequency of students who completed at least one session after completing the course requirements (i.e., completing at least four sessions with each simulated client and scoring a 90 out of 100 at least once), and the total number of minutes engaged with each client and eLearning materials.

Simulation Acceptability and Usability

To assess simulation acceptability and usability, we used an intervention satisfaction scale from the initial efficacy study that evaluated the Introducing CBT simulation (Mastroleo et al. 2020). The scale included two sets of items. The first set included the original four items assessing the “helpfulness” of the simulation (i.e., developing relationships with clients, setting agendas, explaining CBT, and dealing with client resistance). For the CBT Functional Analysis simulation, we added two items to assess “assigning homework” and “conducting a functional analysis.” The Brief MI simulation included the original “developing relationships with clients” and “dealing with client resistance” items. Then we added MI-specific items: “learning Open Ended Questioning, Affirmations, Reflection, and Summarizing (OARS),” “engaging with a client,” “focusing with a client,” and “planning with a client.” All items were scaled from “0 = not at all” to “3 = very.”

The second set of items focused on the student’s experience using the training and included 14 items. Six items focused on acceptability, including: “How engaging was the training?,” “How realistic was the virtual client?,” “How much did you want to try again?,” “How likely are you to recommend this?,” “Overall, how well did this training prepare you to work with a client with a substance abuse problem?,” and “Overall, how well did this training prepare you to use CBT with clients in general?” The remaining eight items focused on usability, including: “Did you use the help coach? (yes or no),” “If yes, how helpful was the help coach?,” “Did you use the scores? (yes or no),” “If yes, how helpful were the scores at the end?,” “Did you read the eLearning training guides? (yes or no),” “If yes, how helpful were the guides?,” “Did you use the speech recognition? (yes or no),” and “If yes, how helpful was speech recognition?” All “If yes” items were scaled from “0 = not at all” to “3 = very.”

Lastly, we evaluated perceived acceptability and usability after completing each simulation using the following open-ended qualitative questions: (1) What did you like best about the simulation?; (2) What were the limitations of the simulation?; and (3) How did the simulation compare with other training that you’ve received?

Effectiveness

We used the counselor activity self-efficacy scales (CASES; Lent et al. 2003) as our proximal measure of effectiveness. The CASES consist of 15 self-reported items (scaled “0 = no confidence” to “9 = complete confidence”) that evaluate one’s confidence in performing counseling skills. In addition to generating a total score reflecting overall confidence in skills, we organized the individual counseling skills into three domains—exploration skills, insight skills, and action skills—based on prior research (Mastroleo et al. 2020). The exploration skills included five basic communication items: attending, listening, reflection of feelings, restating, and asking open-ended questions. Alpha reliability was strong at pre-test (α = 0.84) and post-test (α = 0.80). Insight skills included five items: intentional silence, challenging client inconsistencies, offering interpretations, using self-involving immediacy statements, and self-disclosure for exploration and insight. Alpha reliability was strong at pre-test (α = 0.88) and post-test (α = 0.90). Action skills included four items: information-giving, direct guidance, role-play and behavior rehearsal, and homework. Alpha reliability was strong at pre-test (α = 0.86) and acceptable at post-test (α = 0.76).

Data Collection Procedures

Students completed demographic and educational background and pre-test self-efficacy surveys during the first day of class. They then completed the post-test self-efficacy survey and the simulation acceptability and usability survey during the final session of the course. The feasibility (adherence and process) variables were generated by the simulations and accessed via an administrative portal on the website that facilitated access to the simulations (http://training.simmersion.com).

Data Analysis

We used descriptive statistics to characterize the sample’s demographic and educational background. We evaluated the feasibility of the simulations via reporting on student adherence and process data (i.e., completion of at least four sessions and scored 90 at least once per simulation; completion of at least one session after obtaining a score of 90; the total number of sessions completed; first and high scores; and total minutes engaged with each simulation’s virtual client and eLearning materials) using descriptive statistics. We evaluated the acceptability and usability using the three open-ended questions. To process and analyze the open-ended responses, two coders reviewed responses and used an open coding technique to generate themes across the three questions. Preliminary codes and themes were evaluated by a third coder, and a discussion was held to reach an agreement among the coders for any theme that was not a consensus among the first two coders. Lastly, themes were organized into a final framework, which was confirmed by the coding team. We used a paired sample t-test to evaluate the presence of a change in the student’s self-efficacy of their counseling skills between pre-test and post-test. Additionally, we used Pearson correlations to explore the relationships in this feasibility study between simulation process measures and the change between pre-test and post-test (i.e., post-test measure minus pre-test measure) in one’s self-efficacy of their counseling skills. Based on prior efficacy results with the simulations in this study, we used one-tailed tests.

Results

Background Characteristics

Seventeen students identified as female (77.3%), four students identified as male (18.2%), and one student identified as gender non-conforming (4.5%). Students had a mean age of 28.24 (SD = 7.27). Fifteen students self-reported as White (68.2%); two students self-reported as Latinx (9.1%), two students self-reported as Asian American (9.1%), two students self-reported as more than one race (9.1%), and one student self-reported as African American (4.5%). Fourteen students (63.6%) were in their first year, while eight students (36.4%) were in their second year. One student had prior formal CBT training (4.5%), four students had prior formal MI training (18.2%), two students had formal training in substance abuse treatment (outside of MI and CBT; 9%), and six students had prior counseling training (that was not MI or CBT; 27.3%).

Simulation Feasibility (via Adherence and Process Variables)

The feasibility results regarding adherence and process data are summarized for each simulation in Table 1. We observed that 19 of 22 students (86.4%) adhered to the requirement to complete at least four attempts at each simulation and scored 90 at least once. The three students who did not adhere to the training protocol completed more than eight simulated sessions on the individual simulations but did not obtain a score of at least 90. Moreover, 40.9% of students engaged at least one simulation session beyond the required protocol. In total, students completed M = 28.86 simulated sessions (SD = 8.82; range 15 to 47) across the three simulations, which translated into M = 468.95 min (SD = 178.27; range 269 to 809) of simulated clinical training that included an additional M = 19.68 min (SD = 18.79; range 1 to 63) of eLearning. We observed that across all three simulations, students scored M = 71.21 (SD = 5.96; range 56.13 to 80.50) on their first simulation and scored M = 91.76 (SD = 2.56; range 82.67 to 95.33) at their highest score.

Table 1

Simulation feasibility: adherence and process variables (N = 22)

	Brief MI with Gabe Turner(mean, SD) or %	Introducing CBT with Tanisha Mosley(mean, SD) or %	CBT Functional Analysis with Roger Ellison(mean, SD) or %
% of students who completed at least four sessions and scored 90 at least once	90.9%a	100%	86.4%a
% of students who completed at least one session after requirements completed	22.7%	31.8%	9.1%
Total session completions	13.68 (6.43)	6.91 (2.86)	8.27 (3.57)
First score	63.41 (11.13)	66.45 (20.44)	75.95 (7.57)
High score	90.14 (6.05)	93.64 (3.24)	91.50 (2.97)
Mean score	71.65 (5.55)	74.27 (8.08)	81.42 (4.45)
Minutes engaged in simulated session	182.96 (68.28)	113.21 (46.81)	172.77 (71.70)
Minutes engaged in eLearning	6.50 (7.17)	4.31 (5.71)	8.70 (12.71)

aThree students did not obtain a score of 90 after more than nine sessions with Roger and 13 sessions with Gabe

Simulation Acceptability and Usability

Quantitative Results

The acceptability of each simulation at the item level reflected scores between M = 1.91 (SD = 0.92) and M = 2.86 (SD = 0.35), suggesting they were somewhat to very acceptable (see Table 2 for item-level means [SD]). The usability of each simulation at the item level reflected scores between M = 2.13 (SD = 0.83) and M = 2.68 (SD = 0.48), suggesting they were somewhat to very usable (see Table 3 for item-level means [SD]).

Table 2

Item-level acceptability of computerized simulations (N = 22)

	Brief MI with Gabe Turner(mean, SD)	Introducing CBT with Tanisha Mosley(mean, SD)	CBT Functional Analysis with Roger Ellison(mean, SD)
How helpful were simulations in preparing students to
Develop relationships with clients?	2.09 (0.81)	1.95 (0.90)	1.91 (0.92)
Use OARS?	2.41 (0.91)	–	–
Engage with client?	2.55 (0.67)	–	–
Focus with client?	2.36 (0.73)	–	–
Evoke change talk with a client?	2.59 (0.50)	–	–
Explain CBT?	–	2.86 (0.35)	2.59 (0.59)
Set agendas	–	2.32 (0.72)	2.27 (0.70)
Conduct a functional analysis?	–	–	2.59 (0.50)
Assign homework?	–	2.32 (0.84)	2.59 (0.50)
Deal with client resistance?	2.36 (0.66)	2.00 (0.82)	2.18 (0.79)
Overall, how well did this training prepare students to work with a client with substance abuse?	2.05 (0.65)	1.91 (0.75)	2.14 (0.64)
Overall, how well did this training prepare you to use CBT (or MI for Gabe) in general?	2.32 (0.72)	2.55 (0.60)	2.50 (0.51)

The item level scales ranged from 0 = ‘not at all,’ 1 = ‘minimally,’ 2 = ‘somewhat,’ 3 = ‘very'

Table 3

Item-level usability of computerized simulations (N = 22)

Questions about your experience with the simulations	Brief MI with Gabe Turner(mean, SD) or %	Introducing CBT with Tanisha Mosley(mean, SD) or %	CBT functional analysis with Roger Ellison(mean, SD) or %
How engaging was the simulated client?	2.23 (0.61)	2.31 (0.57)	2.45 (0.60)
How realistic was the simulated client?	2.45 (0.67)	2.36 (0.58)	2.32 (0.72)
How much did you want to try it again?	2.32 (0.65)	2.23 (0.75)	2.14 (0.89)
How likely are you to recommend the simulation?	2.55 (0.60)	2.68 (0.48)	2.55 (0.60)
Did you use the help coach? (% yes)	90.9%	90.9%	90.9%
How helpful was the coach?	2.14 (0.91)	2.45 (0.69)	2.35 (0.75)
Did you use the scores? (% yes)	90.9%	100%	90.9%
How helpful were the scores?	2.19 (0.75)	2.50 (0.52)	2.30 (0.80)
Did you read the eLearning? (% yes)	68.2%	68.2%	72.7%
How helpful was the eLearning?	2.27 (0.70)	1.93 (0.80)	2.13 (0.72)
Did you use the speech recognition? (% yes)	36.4%	31.8%	36.4%
How helpful was the speech recognition?	2.13 (0.83)	2.29 (0.76)	2.13 (0.83)

The item level scales ranged from 0 = ‘not at all,’ 1 = ‘minimally,’ 2 = ‘somewhat,’ 3 = ‘very‘'

Qualitative Results

Two main themes were identified after analyzing the students’ responses to the surveys. The first theme reflected that the simulations provide a strong foundation that prepares beginners for clinical practice, including introductory and foundational components of the CBT and MI modules that were beneficial for beginner therapists. Many students shared that the simulations were a strong initial exposure to CBT and MI and that the simulations were useful for practicing concepts before providing therapy with real clients. The following quote highlights how students found the modules to be helpful and foundational for beginners: “This simulation would be a good first step for people to use to understand how to approach therapy before they are able to work with real clients.” A notable subtheme was that introductory training for CBT was specifically reported as helpful for beginners. The following quote reflects one student’s positive experiences with developing a foundation in CBT through the module: “The simulation did a great job of showing how thoughts, feelings, and behaviors interact, and using ABC worksheets in the simulation was a great demonstration of a useful tool [in this intervention].”

The second theme reflected the presence of technical barriers to the simulation experience. Specifically, some students felt their choice of response options was limited, often not representative of what they would realistically say to a client, and lacked sufficient opportunities to express empathy. A minority of students experienced technical glitches when navigating the modules that impeded the natural flow of the simulated conversation. The following excerpts describe the challenges that were noted: “It was hard to show empathy with the limited answer options provided” and “There were some technical glitches that didn’t always recognize my responses appropriately.” In summary, these two overall themes presented both positive aspects of the use of simulation for clinical training and considerations for improving future designs.

Initial Effectiveness Results

The initial effectiveness results are summarized in Table 4. Overall, students reported significant increases to their self-efficacy between pre-test and post-test concerning general clinical skills (p < 0.001) and specifically their exploratory skills, insight skills, and action skills (all p < 0.001).

Table 4

Initial effectiveness outcomes (N = 22)

	Pre-test	Post-test	T-value
Total clinical skills	71.23 (16.46)	97.55 (15.31)	8.86***
Exploration skills	28.59 (6.26)	37.27 (3.97)	7.69***
Insight skills	24.64 (9.13)	34.36 (9.12)	6.28***
Action skills	18.00 (6.14)	25.91 (5.47)	5.41***

***p < 0.001

Exploratory Correlations Among Feasibility, Acceptability, Usability, and Effectiveness

We evaluated whether simulation feasibility (i.e., via adherence and process data; average scores, high scores, minutes of simulation, minutes of eLearning) were correlated with changes between pre-test and post-test on measures of their general clinical skills, including their exploratory skills, insight skills, and action skills. Given that this study is underpowered, we reported correlations that had a magnitude of r > 0.29 even if the p-value was non-significant, as they may help generate hypotheses for future evaluation in an adequately powered trial.

Overall, the mean scores on the brief MI simulation were correlated with change in self-efficacy of total clinical skills (r = 0.42, p = 0.026), exploratory skills (r = 0.35, p = 0.055), and insight skills (r = 0.47, p = 0.014). Also, the mean scores on the Introducing CBT simulation were correlated with change in the self-efficacy of action skills (r = 0.40, p = 0.035), and the mean scores on the CBT Functional Analysis simulation were correlated with change in self-efficacy of total clinical skills (r = 0.31, p = 0.085) and insight skills (r = 0.38, p = 0.04). Lastly, the mean number of eLearning minutes completed for the CBT Functional Analysis simulation was correlated with change in self-efficacy of total clinical skills (r = 0.30, p = 0.085) and exploratory skills (r = 0.38, p = 0.04). There were no additional correlations with a magnitude of r > 0.29 for high scores, simulation minutes, or eLearning minutes for the simulations.

Discussion

Technology-based simulations are emerging as a promising method to enhance the accessibility of evidence-based psychosocial treatments and improve the effectiveness and feasibility of clinical social work education and training (Kilbourne et al. 2018; Kourgiantakis et al. 2020). This is particularly relevant given the current COVID-19 context of greater accessibility to remote learning and training methodologies. With this in mind, we evaluated the initial feasibility, acceptability, usability, and effectiveness of three psychotherapeutic simulations with virtual clients among master’s students in a CSWE-accredited MSW program.

Simulation Feasibility

As a measure of feasibility, the vast majority of students adhered to the required protocol to complete simulations, with upwards of 40% of students engaging in simulations beyond the required protocol. Moreover, the process data reflected that students spent approximately 2.5 h (i.e., approximately 10 sessions) engaged with each simulated client, and their initial score of 71 increased to a high score of 92. Thus, the students engaged in the tool and made progress toward increasing their scores. We also recognize that achieving a high score above 90 one time does not convey that the simulation skills being taught have been mastered. The first author discussed this concept with students before they engaged with the simulations to provide additional insight that the simulations are not a test and that consistently scoring in a high range may reflect that one has mastered these particular simulation exercises, but that fine-tuning clinical skills requires lifelong learning. Interestingly, several students discussed (anecdotally in class) that they would sometimes engage the simulation with the intention to perform poorly to explore how clients may react to poor clinical performance. Although we did not measure this style of learning in this initial study, future studies may benefit from evaluating the extent to which students intentionally engage in poor clinical performance during the simulations and how the students learn from this experience.

In addition, students minimally used the eLearning and speech recognition tools. Anecdotally, students discussed in class that they engaged less in eLearning, given their concurrent didactic training within the psychosocial modality teachings of this MSW course and their engagement in field education. The choice to minimize the eLearning content presented challenges for some students, as the eLearning covered specific topics that informed how the simulation performances would be scored. Thus, some students who minimally used the eLearning may have otherwise obtained higher scores. Future implementation and evaluations of these simulations within clinical training settings may benefit from emphasizing the potential helpfulness and usefulness of the eLearning content within this technology. As for speech recognition, students anecdotally reported in class that they often engaged the simulations in settings outside a private office or home, which made them uncomfortable speaking to their computer in a public setting, such as a school library. Other students noted they used the simulations in settings that had background noises (e.g., café, common study space) that may have interfered with the effectiveness of the speech recognition software and disrupted the flow of the conversation. As a result, students did not adhere to the recommendation to use the simulations’ speech recognition software and instead clicked their responses. Nevertheless, speech recognition was accessible for most students and is an important aspect of this technology so that students can practice speaking their responses out loud as this adds an extra layer of learning beyond clicking their responses.

Students reported on their surveys that the simulations were generally enjoyable and easy to use, thus reflecting strong acceptability. The qualitative results corroborate the quantitative survey results as students suggested the simulations were particularly useful among those who self-reported they had limited prior clinical experience or training in CBT. Usefulness in the surveys related explicitly to experiences of engagement with the client, realism, the ability to engage in repeated practice, recommending the simulations to other students, using the help coach who provided nonverbal cues in real-time to reinforce the effectiveness (or lack thereof) in clinical statements, and usefulness of both transcripts and scores as a clinical trainee. However, not all students reported that the simulations were helpful in their clinical training during their responses to the open-ended survey questions. Specifically, some students reported that the simulations had too few options for responses and had limited options to express empathy, and that they experienced technical difficulties. Thus, not having enough options for interacting with the virtual client and minimal statements conveying empathy made the experience less realistic for some students and reduced the potential of its generalizability with various clients. While technical intervention aspects of CBT or MI must be delivered per evidence-based theory and empirically supported treatment manuals, it is also essential to build rapport with a client using validation and empathy (Norfolk et al. 2007; Watson 2016). Therefore, future iterations of these simulations should consider greater use of empathic statements in conjunction with technical treatment content.

Initial Effectiveness via Self-efficacy

Students’ self-reported self-efficacy improved from pre-test to post-test for the total score and for each domain score (i.e., exploration, insight, and action). Though self-reported self-efficacy does not directly measure changes in applied clinical skills, clinical self-efficacy is an essential predictor of real-world implementation of evidence-based practice (Abrahamson et al. 2012; Bornheimer et al. 2018). Additionally, we found some initial correlational evidence that simulation performance may be related to changes in self-efficacy. However, these results are preliminary and underpowered. As such, we provide a conservative interpretation that the correlations may provide an initial effect size that future studies can potentially use to power their sample. Thus, future trials evaluating the effectiveness and implementation of these simulations would benefit from a larger sample and a formal measure of skill demonstration.

Limitations and Future Directions

There are several limitations to consider. First, the study’s sample was small and, therefore, underpowered to detect the statistical significance of small-to-medium-sized effects such as the observed correlations between simulation performances and changes in self-efficacy. Second, the vast majority of students in this particular course were White, and future studies need to evaluate the simulations among students from underrepresented and marginalized communities. Third, this study focused on a sample of students in one MSW class, and there was no control group to investigate changes in self-efficacy as compared to students who did not engage with the simulations. Fourth, students were simultaneously developing skills in other MSW classes and their field practicum. Thus, it was not possible to separate the contributions of the simulation to self-efficacy from other classes and field learning experiences, which may also contribute to overall clinical self-efficacy. Lastly, the students’ engagement with topics and content in the MSW course was not measured outside of class. This extracurricular engagement could also explain changes in self-efficacy and the students’ performances on the simulations. Therefore, potential extracurricular engagement must be acknowledged concerning the current findings, and future research should measure and statistically control for these potentially confounding variables.

Overall, the results of this study advance prior evaluations of training simulations by moving the setting of the study from ideal research conditions into the real-world classroom (Mastroleo et al. 2020). Given that a recent systematic review found seven publications focused on virtual reality and computer simulations in social work education (Huttar and BrintzenhofeSzoc 2020), the literature on these topics is still in its infancy. Thus, the current study helps move the field forward toward translating research-informed practices into clinical social work education. That said, future fully powered randomized controlled trials are needed to validate the effectiveness and inform the real-world implementation of these simulations within classrooms and within MSW programs. Moreover, future studies evaluating computerized simulations might also consider stratifying their samples to examine outcomes between students who are and are not concurrently engaged in their field practicum.

Implications for Social Work Education

Research evaluating whether technology-based simulations enhance psychotherapeutic education is still in its infancy. Thus, translating this research to educational practice and scaling up effective simulations can take years if not decades. In the context of social work education, there is a strong literature that speaks to the effectiveness of simulations (Kourgiantakis et al. 2020), which suggests that social work programs may be moving toward piloting or adopting simulations into their curriculum. The technology-based simulations evaluated in this study, and in other studies (Huttar and BrintzenhofeSzoc 2020; Mastroleo et al. 2020; O’Brien et al. 2019; Putney, Levine, et al. 2019a, b; Washburn et al. 2020), provide strong research-informed options for a post-COVID-19 world where more asynchronous training will be needed and simulations that are delivered via the internet are highly scalable when an effective implementation plan is followed. For example, MSW administrators can consider implementing the more basic simulations during their foundational coursework (e.g., Introducing CBT) while reserving simulations such as CBT Functional Analysis for advanced coursework or field practicums.

Another implication to consider with respect to implementation is at what level schools of social work will administrate access to the simulations. Though this particular study was a pilot, the University of Michigan has since conducted a school-wide implementation that required the Office of the Associate Dean of Educational Programming to develop an infrastructure to regulate access to the simulations for faculty, lecturers, and students. This infrastructure required both labor and non-labor costs that go beyond the scope of this pilot. However, these issues must be considered when balancing not only the cost to access the simulations, but also the cost to implement them.

Conclusion

In summary, MSW students improved their CBT and MI simulation scores and self-reported greater self-efficacy over a 13-week adult interpersonal practice course. The students perceived the simulations to be acceptable and usable, while sharing feedback regarding the desire for the simulations to make more persuasive empathic statements and for developers to fix some technological glitches. The results of our initial investigation of this training modality, along with its real-time feedback and technical training (e.g., eLearning), are promising and show that virtual simulations have the potential to increase the effectiveness and implementation of clinical training in social work education and beyond. Next steps require randomized controlled trials to validate the effectiveness of the simulations at enhancing skills, and effective implementation strategies must be developed.