Supporting workforce practice change: protocol for a pilot study of a motivational interviewing virtual client software tool for health professionals.

INTRODUCTION: Motivating behavioural change during client consultations is of crucial importance across all health professions to address the growing burden of chronic conditions. Yet health professionals often lack the skills and confidence to use evidence-based counselling interventions to support clients' behavioural change and mobilise clients' resources and self-efficacy for change to address their long-term needs. AIMS: This pre-post pilot study will develop a motivational interviewing (MI) virtual client training tool for health professionals and test the effectiveness of the educational content and usability of the virtual client interaction. METHODS AND ANALYSIS: Postgraduate students across a range of health disciplines will be recruited. Data assessing attitudes towards preventive healthcare will be collected using a modified version of the Preventive Medicine Attitudes and Activities Questionnaire. Conversations with the virtual client will be analysed using the Motivational Interviewing Treatment Integrity code to assess changes in MI skills. The System Usability Scale will be used to assess the usability of the virtual client training tool. ETHICS AND DISSEMINATION: This protocol was approved by the Flinders University Social and Behavioural Research Ethics Committee in May 2019. The results of the pilot study will inform the development of an avatar-based mobile application consisting of MI teaching and interactions with a generic virtual client that can be easily adapted to multiple scenarios. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The study will pilot online motivational interviewing (MI) education to provide health professionals with an accessible, cost-effective training resource for the development and maintenance of MI competence.

Potential limitations of the design include self-selection bias and potentially low response rates given that postgraduate students are often time poor.

This pilot study represents the initial step in the development of a mobile application providing comprehensive MI training and interaction with a virtual client so health professionals can develop, practise and receive feedback on their MI skills.

Introduction

Non-communicable, largely chronic conditions account for around three in five global deaths, attributed in particular to cardiovascular disease, cancer, chronic lung disease and diabetes.1 Lifestyle risk factors (eg, diet, body mass index and smoking) are the leading risk factors for chronic conditions.2 Modifiable health behaviours are key targets of health promotion and interventions to reduce the burden of chronic conditions.3 While it can be difficult to provide interventions for entire populations, primary prevention at the individual client level is recognised as an effective means of supporting behavioural change.3 Motivating behavioural change during client consultations is therefore of crucial importance across all health professions.4 5 Yet health professionals often lack the skills and confidence to use effective, evidence-based counselling interventions to support behavioural change.

Motivational interviewing (MI) is a counselling method that enhances clients’ motivation to change through resolving ambivalence about health behavioural change. Backed up by 40 years of research, MI is a key skill for health professionals internationally for supporting their clients to make and maintain positive behavioural changes.6–9 Research has shown the effectiveness of MI across a range of health professional groups and settings.10 11 Yet it is a complex skill and health professionals have struggled to translate these skills into everyday practice.12

Health professionals seeking to improve their ability and confidence to counsel clients about changing behaviours need access to effective education to facilitate and sustain this practice change. The principal method of MI training is through face-to-face workshops, often including experiential exercises such as role-plays or use of a standardised patient.13 The current approach to MI training has tended to focus on ‘more general or introductory MI training’,13, p106 which fails to adequately develop health professionals’ MI skills, particularly over the long term. Miller and Moyers14 have suggested that in order to develop MI competence, MI training should ideally incorporate eight stages across two phases (phase 1: enhancing clients’ motivation for change; phase 2: strengthening and consolidating commitment to change). This staged approach aims to embed MI skills and support the maintenance of these skills in health professionals’ future practice.14 The application of this staged approach has been explored in a systematic review of training in MI, with the results demonstrating that most of the MI training studies focused primarily on a small number of stages in phase 1 of MI training.13

In addition to limitations in the extent to which MI skills are consistently embedded in health professionals’ long-term practice, MI workshops often fail to address the need for health professionals to develop and reinforce their MI skills in a non-threatening environment in which they can gain confidence, make mistakes safely and learn from them. Health professionals also need to revisit, practise, reflect and test how they operationalise the MI skills they have learnt to their everyday practice. Furthermore, workshops are less accessible to rural, remote and busy health professionals. It is well documented that distance from and access to education providers is a barrier to further education.15 16 Consequently, there is a move towards the development and provision of technology-delivered MI training, using virtual standardised patients, which aims to address these challenges.

For example, Kognito has developed a number of role play simulations using MI communication skills (open questions, affirmations, reflections and summaries; MI Spirit—partnership, acceptance, compassion and evocation), along with a range of communication strategies based on social/cognitive models and neuroscience.17 The simulations focus on specific behaviours and settings, such as a parent talking with a child about substance abuse, a veteran who is worried about a buddy, and a healthcare professional conducting screening and brief interventions with adolescents. Carpenter et al 18 developed a multimedia instructional tutorial to teach MI-based brief intervention strategies for smokers, using a Role-play Internet Simulation. Vicky is another example involving an MI-based brief intervention.19 Vicky is a virtual coach who teaches the Screening Brief Intervention and Referral to Treatment intervention for clients with alcohol or other drug use problems in the emergency room setting.

As these examples demonstrate, current approaches to technology-delivered MI training tend to focus on particular health behaviours and particular settings. Furthermore, there is a lack of MI specific training that teaches MI in depth across all of Miller and Moyers’14 eight stages. No ‘one stop’ technology-based MI training was identified that covers the range of client types, conditions and behaviours usually seen in real-world practice, given working with clients with diverse multimorbidity is the norm for most health professionals. A contributory factor may be the laborious nature of existing virtual client (VC) creation; for example, developing a smoking cessation case study does not make it easier to develop a weight loss scenario, as each case is comprised of the specific words used in the virtual interviews. Integration of advances in techniques of dialogue knowledge representation and management and ontology integration20 can address this problem.

The pilot study discussed here will involve the development and pilot testing of technology-delivered MI training to inform future development of a VC-based MI learning system covering multiple conditions, behaviours and client groups to provide opportunities for health professionals to develop, practise and receive feedback on their MI skills. VCs are increasingly being used in teaching various skills to health professionals, and have been effective in increasing knowledge, confidence and skills.21 22 In the pilot study, we aim to develop and test a virtual client conversation tool (V-CCT) for use in teaching MI. The V-CCT will be integrated within an online MI course following Miller and Moyer’s14 eight stages of MI training. Development of the V-CCT and educational content began in January 2019 with participant recruitment beginning in August 2019. Data collection is due to be finalised by May 2020 with an estimated completion date for the pilot being July 2020.

Methods and analysis

A pre–post pilot study is currently underway at Flinders University, South Australia, testing the following aspects of technology-delivered MI training:

The effectiveness of MI teaching delivered via e-learning following Miller and Moyer’s14 eight stages of MI training.

The usability of a V-CCT.

Development of an online MI course

The MI course has been developed following Miller and Moyer’s14 eight stages of MI training (see table 1).

Table 1

The eight stages of MI training

Stage	Description
The Spirit of MI	Partnership, acceptance (absolute worth; accurate empathy, autonomy support, affirmation), compassion and evocation (evoking the client’s own motivation and resources for change)
OARS	Client-centred counselling skills (open questions, affirmations, reflections, summaries)
Recognising and reinforcing change talk	Identifying and reinforcing desire, ability, reason, need and commitment to change
Eliciting and strengthening change talk	Using OARS skills to elicit change talk and commitment language
Rolling with resistance	Importance of not directly refuting the client’s arguments against change (resisting the righting reflex) and strategies to roll with resistance
Developing a change plan	How to work with the client to develop a change plan
Consolidating client commitment	Eliciting and consolidating the client’s commitment to make the change
Switching between MI and other approaches	MI can be combined flexibly with other counselling methods

MI, motivational interviewing.

The course has been developed by MI training experts within Flinders University. The course involves a series of recorded lectures, links to YouTube videos demonstrating MI, downloadable resources and quizzes for checking and reinforcing knowledge. The course will be delivered using the university’s Learning Management System, a Moodle-based platform.

Development of the V-CCT

For the purpose of the pilot study, the V-CCT focuses on a conversation with a virtual client about diet and weight loss. The tool has been developed based on a case study of a VC, Anna, developed specifically for the study.

Villaume et al 23 identify three classes of interactive VCs. In the first class are VCs that use a restricted set of questions that the user can click on, generating an automated response from the client. The sets of questions are independent and can, therefore, be asked in any order. The second class of VCs is scripted to simulate a natural flow of interaction between user and the VC, while still using a restricted set of questions. This involves the VC responding to the selected question followed by a list of further responses from which the user can select. The third class involves freeform spoken input by the user with the computer determining the exact wording of the VC’s response in light of what the user has spoken (see eg, Hubal et al 24). In this class, VC responses are generally delivered in the form of videos or through the use of avatars. According to Sweller’s cognitive load theory,25 presenting material using visual, auditory and/or gestural interfaces helps to reduce cognitive load, allowing us to either absorb more information or to engage at a higher cognitive level.

In this study, we have used a hybrid of classes two and three, with the VC response being in text form (with a view to developing a VC avatar in the next stage of the project, underpinned by the pilot study results). The user’s interaction with the VC involves typing in a question/response, and then selecting the closest match from a restricted set of responses (or if there is no close match, then selecting whichever response the user deems appropriate). The VC then responds (in text form in the pilot version) and the user again types in their response and selects the closest match from the restricted set of responses. The purpose of adopting this approach is to better facilitate learning of MI by the user actively practising what they would say (typing their response), while still mimicking a conversation flow.

The conversation flow options have been designed to incorporate four conversation flows; an MI-consistent flow, an MI-inconsistent flow and two neutral conversation flows. The neutral conversation flows are designed to be either somewhat MI consistent or somewhat MI inconsistent. This allows us to limit the number of VC responses while still maintaining a logical flow in the conversation. The conversation flows are interconnected (with one response option for each conversation flow presented in response to the VC) such that the user can move between MI-consistent, MI-inconsistent and neutral conversations. This is an instructional process supporting conscious effort by the user and reinforcing the acquisition of MI knowledge through activities that enhance higher order thinking (or deeper learning) rather than the replication of content.26 The interconnection of the conversation flows maximises critical thinking and thinking mindfully, providing opportunities for learners to consider how they can actively use MI skills in engaging with the VC.25 27

Development of the conversation flows

In order to develop the conversation flows we began with the MI-consistent flow, informed by the case study of Anna. The conversation follows the four processes of MI, namely engaging (establishing a working relationship), focusing (clarifying the client’s agenda), evoking (eliciting and reinforcing the client’s own arguments for behavioural change) and planning (guiding the client to develop and commit to a plan of action). This conversation was then used as the basis for developing the health professional text options and the VC responses in the MI-inconsistent and two neutral conversation flows, ensuring that the user could move between the four conversation pathways to enable them to get back on track when diverging from the MI-consistent interaction. A selection of the script flow chart is shown in figure 1. HP refers to health professional, C is the client, the numbers are the set of questions/responses from which the user selects, and the letters refer to the MI-inconsistent (A), neutral (B and C) and MI-consistent (D) conversation flows. While the user always gets to choose from one of the four response options (A, B, C or D), these are presented in the V-CCT in random order to mitigate against the user guessing which option to select.

Figure 1

Script flow chart.

Interaction with the VC

The V-CCT is accessed via a website where the user is introduced to Anna in a short video (see screenshot in figure 2). They are then asked to enter either the precourse dialogue (to be completed prior to completing the MI course) or the postcourse dialogue (to be completed after completing the course). The dialogue screen (showing user inputted text and text selection options) can be seen in figure 3.

Figure 2

Introduction to the virtual client.

Figure 3

Dialogue screen.

The dialogue, derived from the script flow chart (figure 1), was modelled inside of a Standard Query Language database. This database is then queried each time the user makes a response, generating both a response from Anna that is displayed to the user and a list of four multiple-choice options that the user will select from in the next round. While the user is interacting with Anna, each iteration of the conversation is recorded, linking what the user typed in the free text section, their multiple-choice selection and the response from Anna. After the conversation is over, the user’s ‘attempt’ is logged with a date and time for possible retrospective analysis to ensure that the user has not done both conversations straight after each other (ie, without completing the MI course in between).

After progressing through the conversation with Anna, the user then receives feedback about how they went, focusing on the extent to which they selected MI-consistent, MI-inconsistent and neutral text. No feedback is provided during the conversation flow in the pilot version of the V-CCT, nor will users be shown which are the MI-consistent responses, because the V-CCT will be used as part of the pre–post pilot testing of the effectiveness of the MI course content. We believe that provision of this type of formative feedback during the pilot testing would prevent an objective assessment of MI skill development; however, formative feedback will be provided in later versions of the V-CCT.

Recruitment

Participants for the pilot study will be recruited via Flinders University postgraduate courses for students already working within or training to enter into a range of health professions (eg, medicine, nursing, psychology, physiotherapy, nutrition and dietetics, occupational therapy). This will provide a potential participant pool of around 200 students. While interaction with the VC begins with a discussion of blood glucose levels, users are informed that the conversation will be focusing on behavioural change relating to diet and exercise, which has application to all of the health professional student groups, rather than medical management.

Participants will be invited to take part in the online MI course via an announcement through the Flinders University Learning Management System. Consent to participate will be obtained by students signing a consent form and emailing it to administrative staff not involved in the study, who will enrol students in the online course. Participants will interact with the V-CCT prior to going through the online MI educational content and again at the end. Access to the VC will be through a link to a website.

Data collection

In order to test the effectiveness of the online MI course delivered using the eight stages of teaching MI, students will be asked to complete the Preventive Medicine Attitudes and Activities Questionnaire (Modified) (PMAAQ) prior to undertaking the course. They will be asked to complete the PMAAQ again at the end of the course. The PMAAQ in its original form was developed by Yeazel et al 28 and is an 85-item tool that addresses physicians’ health prevention behaviours and preventive healthcare attitudes. The questionnaire has its theoretical underpinnings in social cognitive theory with the key constructs being ‘self-efficacy (confidence in performing a behaviour), situation (one’s perception of the environment), behavioural capacity (knowledge and skill to perform a behaviour), expectancies (values placed on a given outcome) and expectations (anticipated outcomes of a behaviour)’(28, p.87). The tool was initially assessed for validity and reliability in the form of cardiovascular disease subscales29 and the complete instrument assessed for validity, reliability and stability results.28 The PMAAQ has been used in a number of studies, both in its original form and with various modifications for particular settings and research questions.30–32 The modified version we are using was used in previous research to assess the motivation of health providers who were trained to provide SagePlus lifestyle counselling to use MI in their practice.33 This version has been further modified to exclude four questions based on the results of reliability testing we conducted of the questionnaire, leaving nine items relating to preventive healthcare attitudes (see box 1) (the reliability of this modification will be assessed in the pilot study). Responses are on a 5-point Likert scale from strongly agree to strongly disagree.

Question

Behavioural change counselling is an effective use of my time as a healthcare provider.

For most clients, health education does little to promote their adherence to a healthy lifestyle.

I am less effective than professional counsellors in getting clients to change their behaviour.

Clients without symptoms will rarely change their behaviour on the basis of my advice.

Most clients try to change their lifestyle if I advise them to do so.

It is difficult to understand why clients can’t meet the goals they have set with me.

It would be difficult to change my routine of lifestyle counselling to include motivational interviewing.

Clients prefer being told what to do over helping to come up with a plan themselves.

Doing lifestyle counselling using motivational interviewing takes longer than traditional methods.

The effectiveness of the online MI course will be further evaluated by analysing the users’ inputted text from the V-CCT (precourse vs postcourse) using the Motivational Interviewing Treatment Integrity (MITI) code.34 The MITI is a reliable and valid MI fidelity tool that is widely used to assess the degree to which an MI conversation reflects the paradigms of MI.35 36 The MITI is particularly useful in this pilot study because it assesses the health professional’s language (ie, the text the user inputs into the V-CCT) and not the client’s language (which has been predetermined in the V-CCT). The MITI incorporates four global scales (Cultivating Change Talk, Softening Sustain Talk, Partnership and Empathy), rated on a 5-point Likert scale measuring how well or poorly the clinician meets the description of the dimension being measured. Global scores are summarised as Technical Global (the score of Cultivating Change Talk plus Softening Sustain Talk divided by two) and Relational Global (the score of Partnership plus Empathy divided by two). There are also ten behavioural counts, which are summarised as pecentage complex reflections, reflection-to-question ratio, total MI-adherent behaviours and total MI non-adherent behaviours. Proficiency and competency thresholds are provided for each dimension.

In order to test the usability of the V-CCT, the post-MI course survey also includes questions exploring users’ views of the V-CCT using the System Usability Scale (SUS). The SUS is considered to be ‘the most widely used standardised questionnaire for the assessment of perceived usability’.31, p.1 The SUS includes 10 questions assessed on a 5-point Likert scale from strongly agree to strongly disagree (see box 2). The scale has been assessed for validity and reliability37 and used in previous research with VCs.38

I think that I would like to use this conversation tool frequently.

I found this tool unnecessarily complex.

I thought the tool was easy to use.

I think that I would need the support of a technical person to use this tool.

I found the various functions in the tool were well integrated.

I thought there was too much inconsistency in this tool.

I would imagine that most people would learn to use this tool very quickly.

I found the tool very awkward to use.

I felt very confident using the tool.

I needed to learn a lot of things before I could get going with the tool.

Finally, the postcourse questionnaire includes additional closed and open-ended questions about the course itself, as described in table 2.

Table 2

Questions exploring participants’ views of the MI course

Question	Response options*
How likely are you to recommend this course to a friend or classmate?	10-point Likert scale from not at all likely to extremely likely
How challenging was this course?	5-point Likert scale from extremely challenging to not challenging at all
How much did you learn from this course?	5-point Likert scale from a great deal to nothing at all
How reasonable or unreasonable was the workload for this course?	7-point Likert scale from extremely reasonable to extremely unreasonable
How well did this course meet your expectations?	5-point Likert scale from extremely well to not well at all
What did you like most about this course?	Free-text response
What did you like least about this course?	Free-text response
How could this course be improved?	Free-text response

*Questions for this section were amalgamated from a range of educational surveys, hence the variation in response options.

MI, motivational interviewing.

Demographic data will also be collected, as described in table 3.

Table 3

Demographic questions

Question	Response options
What is your age?	18–24; 25–34; 35–44; 45–54; 55–64; 65 or older
What is your gender?	Male; female; other
Have you had motivational interviewing training through another source this semester (eg, through your postgraduate courses) in addition to participating in this course?	Yes; no
Are you currently, or have you previously, worked as a health professional?	Yes; no
If yes, in what role?	GP; RN; dietitian; physiotherapist; occupational therapist; social worker; psychologist; other (please specify)
About how many years have you been, or did you, work in that role?	Less than 1 year; At least 1 but less than 3 years; at least 3 years but less than 5 years; at least 5 but less than 10 years; 10 years or more
What is your main client group?	Free-text response
Which of the following best describes the geographical location in which you work?	Metropolitan; regional; rural; remote
Do you currently, or have you previously, used motivational interviewing in your consultations with clients?	Yes; No

GP, general practitioner; RN, registered nurse.

Analysis

This study will pilot the MI course, the V-CCT and logistical aspects of the study for future research.39 This includes using the results of this study to ‘obtain empirical estimates of statistical parameters to inform power calculations, and other design effects, of a subsequent trial’.39, p.3 Negatively worded items on the PMAAQ will be reverse scored and mean scores calculated for each item. Two members of the research team will code the user inputted text using the MITI and the scores will be averaged. Mean scores will be calculated for each summary domain (global technical, global relational, percentage complex reflection, reflection-to-question ratio, MI-adherent behaviours and MI non-adherent behaviours). Scoring of the SUS will follow Lewis.37 Blank items will be given a raw score of 3. For odd numbered items, 1 will be subtracted from the raw score, with the raw score subtracted from 5 for even-numbered items. The sum of adjusted scores will be computed, which will be multiplied by 2.5 to get a standard SUS score. Mean scores on the questions exploring participants’ views of the MI course will be calculated.

Relevant to the pilot design of this project, statistical analyses will mostly be descriptive and specific techniques will be selected once the nature of the data is known, for example, distribution of outcomes. Analysis of precourse and postcourse questionnaires and MITI scores will be through t-tests or analyses of variances. Cronbach’s alpha’s will be calculated for the PMAAQ (assuming sufficient sample size). Alpha error will be set to p<0.05. Answers to open-ended questions will be analysed using content analysis.40

Patient and public involvement

The research question is informed by the needs of patients to have health professionals educated to support them to self-manage their chronic conditions. No patients were involved in the design of the study or recruitment to and conduct of the study.

Ethics and dissemination

Ethical risk to participants is likely to be minimal. Participants’ university lecturers will not be aware of who has taken part in the study. Pre–post data will be linked using a unique identifier to maintain anonymity, with the spreadsheet of identifiers maintained by administrative personnel with no involvement of the researchers.

The results of the pilot study will be used to inform the development of an avatar-based mobile application consisting of MI teaching using the eight stages identified by Miller and Moyers14 and interactions with a generic VC that can be easily adapted to multiple scenarios.

Discussion

Behavioural change support for chronic condition management is fast becoming one of the most significant challenges for health professionals worldwide. MI can be integrated into health professionals’ consultations to address clients’ ambivalence to health behavioural change as it arises. It can also be used to support retention and adherence with therapies.41 This can facilitate early intervention to address health behavioural factors, rather than waiting for clients to reach a crisis stage in their health.42 43

Health professionals seeking to improve their ability and confidence to counsel and support their clients to change behaviours need access to effective education to support practice change. Development of the MI V-CCT will inform the development of a comprehensive MI training platform to provide an accessible, cost-effective, cross-disciplinary resource that supports the development and maintenance of MI competence beyond basic training levels. While the pilot work is situated in a single scenario (diet and weight loss), results of the pilot study will be used to establish a platform from which to undertake the development of a generic virtual MI client that can be easily adapted to multiple scenarios.

A promising approach to achieving this would be to incorporate abstract rules or ‘recipes’ about the MI process into the software.20 Such a recipe encapsulates an abstract generalisation about the dynamics of an MI session, which holds regardless of the health behaviour that is in focus. In addition, other, lower-level rules define the explicit content of these summary statements for a particular health behaviour scenario. In combination with software that enacts the higher-level recipe, developing a subsequent scenario would involve only modifying portions of the lower-level rules, while the more abstract high-level rules are reused as they are. Bickmore et al 20 show that this approach allowed a second scenario to be developed in only 4% of the person hours that had been required to develop a first scenario, while reusing 22% of the software code. Such a software system will constitute a significant advance in the methodology of virtual client development and will greatly accelerate the ability of MI trainers to create new training material and adapt existing lessons.

Allowing students to type in what they would have said next (their most-preferred ‘dialogue move’) in advance of seeing the multiple choices available, also provides us with valuable data for enriching the existing scenario as well as informing the development of subsequent scenarios. MI-adherent student dialogue moves may be used as a source of alternative verbal formulations that can be added in order to make the system more interesting to use, but we envisage that the main value of student moves will be in suggesting interesting common mistakes that students make, in order to incorporate these into MI non-adherent paths in the dialogue.

Building on the pilot study in this way will ensure the tool is relevant to a range of health professionals and the varied client groups with which they interact. Such technology-delivered training is particularly valuable for health professionals in rural and remote locations where it is difficult to provide the same levels of face-to-face support for training as is available to those in urban settings. Additionally, technology-delivered training provides opportunities for ongoing reinforcement and consolidation of skill development and maintenance that is not generally provided with face-to face delivery of training.