Performance of a Supervisor Observational Coding System and an Audit and Feedback Intervention.

Workplace-based clinical supervision is common in community based mental health care for youth and families and could be a leveraged to scale and improve the implementation of evidence-based treatment (EBTs). Accurate methods are needed to measure, monitor, and support supervisor performance with limited disruption to workflow. Audit and Feedback (A&F) interventions may offer some promise in this regard. The study-a randomized controlled trial with 60 clinical supervisors measured longitudinally for 7 months-had two parts: (1) psychometric evaluation of an observational coding system for measuring adherence and competence of EBT supervision and (2) evaluation of an experimental Supervisor Audit and Feedback (SAF) intervention on outcomes of supervisor adherence and competence. All supervisors recorded and uploaded weekly supervision sessions for 7 months, and those in the experimental condition were provided a single, monthly web-based feedback report. Psychometric performance was evaluated using measurement models based in Item Response Theory, and the effect of the SAF intervention was evaluated using mixed-effects regression models. The observational instrument performed well across psychometric indicators of dimensionality, rating scale functionality, and item fit; however, coder reliability was lower for competence than for adherence. Statistically significant A&F effects were largely in the expected directions and consistent with hypotheses. The observational coding system performed well, and a monthly electronic feedback report showed promise in maintaining or improving community-based clinical supervisors' adherence and, to a lesser extent, competence. Limitations discussed include unknown generalizability to the supervision of other EBTs.