Simon B Goldberg1, Michael Tanana2, Zac E Imel3, David C Atkins4, Clara E Hill5, Timothy Anderson6. 1. Department of Counseling Psychology and Center for Healthy Minds, University of Wisconsin-Madison, Madison, WI, USA. 2. College of Social Work, University of Utah, Salt Lake City, UT, USA. 3. Department of Educational Psychology, University of Utah, Salt Lake City, UT, USA. 4. Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA. 5. Department of Psychology, University of Maryland, College Park, MD, USA. 6. Department of Psychology, Ohio University, Athens, OH, USA.
Abstract
Objective: Therapist interpersonal skills are foundational to psychotherapy. However, assessment is labor intensive and infrequent. This study evaluated if machine learning (ML) tools can automatically assess therapist interpersonal skills. Method: Data were drawn from a previous study in which 164 undergraduate students (i.e., not clinical trainees) completed the Facilitative Interpersonal Skills (FIS) task. This task involves responding to video vignettes depicting interpersonally challenging moments in psychotherapy. Trained raters scored the responses. We used an elastic net model on top of a term frequency-inverse document frequency representation to predict FIS scores. Results: Models predicted FIS total and item-level scores above chance (rhos = .27-.53, ps < .001), achieving 31-60% of human reliability. Models explained 13-24% of the variance in FIS total and item-level scores on a held out set of data (R 2), with the exception of the two items most reliant on vocal cues (verbal fluency, emotional expression), for which models explained ≤1% of variance. Conclusion: ML may be a promising approach for automating assessment of constructs like interpersonal skill previously coded by humans. ML may perform best when the standardized stimuli limit the "space" of potential responses (vs. naturalistic psychotherapy) and when models have access to the same data available to raters (i.e., transcripts).
Objective: Therapist interpersonal skills are foundational to psychotherapy. However, assessment is labor intensive and infrequent. This study evaluated if machine learning (ML) tools can automatically assess therapist interpersonal skills. Method: Data were drawn from a previous study in which 164 undergraduate students (i.e., not clinical trainees) completed the Facilitative Interpersonal Skills (FIS) task. This task involves responding to video vignettes depicting interpersonally challenging moments in psychotherapy. Trained raters scored the responses. We used an elastic net model on top of a term frequency-inverse document frequency representation to predict FIS scores. Results: Models predicted FIS total and item-level scores above chance (rhos = .27-.53, ps < .001), achieving 31-60% of human reliability. Models explained 13-24% of the variance in FIS total and item-level scores on a held out set of data (R 2), with the exception of the two items most reliant on vocal cues (verbal fluency, emotional expression), for which models explained ≤1% of variance. Conclusion: ML may be a promising approach for automating assessment of constructs like interpersonal skill previously coded by humans. ML may perform best when the standardized stimuli limit the "space" of potential responses (vs. naturalistic psychotherapy) and when models have access to the same data available to raters (i.e., transcripts).
Authors: Wolfgang Lutz; Brian Schwartz; Stefan G Hofmann; Aaron J Fisher; Kristin Husen; Julian A Rubel Journal: Sci Rep Date: 2018-05-18 Impact factor: 4.379