Determinants of usefulness in professional behaviour change interventions: observational study of a 15-year national program.

OBJECTIVE: Educational, and audit and feedback interventions are effective in promoting health professional behaviour change and evidence adoption. However, we lack evidence to pinpoint which particular features make them most effective. Our objective is to identify determinants of quality in professional behaviour change interventions, as perceived by participants.
DESIGN: We performed a comparative observational study using data from the Veterans' Medicines Advice and Therapeutics Education Services program, a nation-wide Australian Government Department of Veterans' Affairs funded program that provides medicines advice and promotes physician adoption of best practices by use of a multifaceted intervention (educational material and a feedback document containing individual patient information).
SETTING: Primary care practices providing care to Australian veterans. PARTICIPANTS: General practitioners (GPs) targeted by 51 distinct behaviour change interventions, implemented between November 2004 and June 2018. PRIMARY AND SECONDARY OUTCOME MEASURES: We extracted features related to presentation (number of images, tables and characters), content (polarity and subjectivity using sentiment analysis, number of external links and medicine mentions) and the use of five behaviour change techniques (prompt/cues, goal setting, discrepancy between current behaviour and goal, information about health consequences, feedback on behaviour). The main outcome was perceived usefulness, extracted from postintervention survey.
RESULTS: On average, each intervention was delivered to 9667 GPs. Prompt and goal setting strategies in the audit and feedback were independently correlated to perceived usefulness (p=0.030 and p=0.005, respectively). The number of distinct behaviour change techniques in the audit and feedback was correlated with improved usefulness (Pearson's coefficient 0.45 (0.19, 0.65), p=0.001). No presentation or content features in the educational material were correlated with perceived usefulness.
CONCLUSIONS: The finding provides additional evidence encouraging the use of behaviour change techniques, in particular prompt and goal setting, in audit and feedback interventions. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The study analyses a large number of discrete interventions over a long period.

The 15-year duration allowed us to isolate the effect of different features on perceived usefulness.

Distinct interventions with distinct goals, compared by perceived usefulness.

Sample size prevented statistical adjustments that could handle potential confounders in observational data.

Introduction

Public health interventions play an important role in translating evidence into practice and promoting health professional behaviour change. Multifaceted interventions are effective in improving medicine prescription patterns,1 radiology referrals2 and screening and preventative practices.3 A common approach to health professional behaviour change is shaping knowledge and promoting better informed clinical decisions. A growing body of evidence suggests that interventions targeting health professional can change clinical performance. Examples of effective interventions include educational interventions and audit and feedback interventions.4–8 However, knowledge shaping interventions are commonly the result of a bundle of different actions in a complex environment,9 which makes it difficult to determine the effect of the individual components. Moreover, audit and feedback interventions may influence behaviour by affecting domains beyond knowledge, such as beliefs about capabilities, motivations and goals and creating social influences.10 Therefore, while there is evidence to determine the effect of both educational, and audit and feedback interventions, there is no evidence to pinpoint which particular features make them most effective.11 A deeper understanding based on empirical data can improve intervention design, which currently relies on theory alone.

Since 2004, the Veterans’ Medicines Advice and Therapeutics Education Services (Veterans’ MATES) program, an Australian Government Department of Veterans’ Affairs (DVA) funded program, has provided medicines advice and promoted physician adoption of best practices by use of a multifaceted intervention, which contains educational material and personalised, patient-specific information and recommendations (audit and feedback component).

The program’s longevity offers an opportunity to empirically evaluate which features of educational and audit and feedback interventions have an impact in its usefulness to physicians. Over the years, the Veterans’ MATES program has gone through changes and, despite keeping the core principles consistent, the features of each intervention have differed slightly from the previous. The changes, coupled within a consistent context of implementation, make interventions comparable.

To improve understanding about which features of educational and audit and feedback interventions contribute to success, we analysed data from all interventions implemented in the Veterans’ MATES program. Our aim is to identify whether content features, presentation features or the use of behaviour change techniques are associated with changes in perceived usefulness by receiving general practitioners (GPs).

Methods

Study design: the Veterans’ MATES program

We performed a comparative observational study12 to evaluate how intervention features from the educational, and audit and feedback components of the program explained the observed variability in intervention success. We used data from the Veterans’ MATES program, comparing 51 distinct interventions that took place between November 2004 and June 2018. The program aims to improve medicine and health services use and health outcomes for all persons in the veteran community by delivering interventions to GPs, pharmacists, other relevant healthcare professionals and veterans.1 The program’s goal is to provide medicine advice to veterans and their primary physicians according to previously identified issues, as detected by epidemiological inquiry.

Since the program’s conception, it has been driven by the use of technology and big data to detect health issues and tailor interventions for at-risk groups. During each intervention’s development process, an epidemiological analysis is performed on a comprehensive database containing administrative claims data (including medicine dispensings) collected by the Australian Government DVA. The analysis is guided by a clear question to explore potential quality use of medicine issues. Examples include: long-term prescription of medicines recommended for acute issues; doses above guideline recommendations and lack of screening tests for an eligible population.

After identifying the problem, a behaviour change goal is defined. This goal typically aims to promote evidence-based practices, for example, encouraging the cessation of medicines without long-term evidence or the optimisation of therapy according to treatment guidelines. With the problem and goal in mind, an intervention is designed. The intervention is informed by social cognitive theory,13 the transtheoretical model14 and the health promotion model Precede-Proceed.15 The full list of topics is available at www.veteransmates.net.au, the influence of the aforementioned theories in intervention development is further discussed in Roughead et al.1 Interventions include three main components: (1) printed and online educational material for health professionals called the therapeutic brief; (2) printed and online educational material for veterans, called the veteran brochure and (3) an audit and feedback document for the GP containing personalised patient information, called the prescriber feedback. An example of the current version of the feedback can be seen in figure 1. The format of this feedback varies depending on the behaviour change goal. Interventions target particular health professionals and veterans according to criteria developed during the epidemiological analysis, meaning that only those individuals likely to benefit will receive the intervention. Along with all veterans identified using these criteria, the interventions target the main GP involved in the veterans’ care.

Figure 1

Example of the audit and feedback document for the ‘recovering from pain’ topic.

Stakeholder (health professionals and patients) perceptions of the service were evaluated using one-page response forms (survey) mailed at the time of the intervention. The surveys have 10 questions designed with a double role of collecting new information and reinforcing important educational content. It contains questions to evaluate intervention quality including perceived usefulness, our main outcome described below and the relevance of this information for patients. In some interventions, the survey also included commitment questions, for example, asking doctors how many patients they would review. Epidemiological postintervention analysis evaluates the actual population-level change in prescription patterns and use of health services, quantifying the effect of interventions. The program has been extensively described elsewhere.1 It has been shown to be effective for changing professional behaviour in different domains,1 including promoting medicine review,16 osteoporosis screening,17 uptake of health services,18 reducing inappropriate proton pump inhibitor use19 and hypnotic use for insomnia.20

Analysis: extracted features from the educational component

Within the educational component of the intervention, composed of the printed and online material, we identified two features that changed over the years: presentation and writing style. Changes in presentation were related to information organisation, given that the visual identity of the materials, including font, colour schemes and general document organisation, remained consistent. The most relevant variations in presentation included adding tables, images and more frequent headlines for section subdivision. The presence of contextually appropriate visual cues has been shown to improve knowledge uptake in health education.21 Visual cues include illustrations and pop-out effects, comprising items that do not match the immediate surrounding environment, such as tables, colour highlights or italic citations. We hypothesised that presentation would affect perceived usefulness due to improved readability and information distribution.

Educational content varied significantly according to the subject being covered making it the most difficult to measure and compare. Three overarching intervention themes were identified: (1) Therapy initiation, which includes the recommendations for prescribing new medicines for known conditions. It usually follows detection of under-use of evidence-based medicines in the health claims database. (2) Therapy adjustment or withdrawal, which includes alerts about common or serious side effects associated with the use of a medicine. It usually includes recommendations for stepping down the dosage, or changing, or stopping medicines. (3) Service initiation, which includes recommendations to refer patients for funded healthcare services (eg, screening tests, home medicines reviews or psychology services for mental health conditions).

Two behavioural economic theories guided our feature selection: loss aversion22 and excess of choice.23 Loss aversion refers to a tendency to give higher weight to loss than to gain, and engage in riskier behaviour to avoid loss. Excess of choice refers to a tendency to make poorer choices when faced with too many options. The relevance of these features was previously demonstrated in health promotion settings.24 In a study evaluating messages to promote hand washing,24 loss and fear inducing (threatening) messages about influenza contagion risk were superior to standard messages. Additionally, messages containing simple instructions (coping messages) were superior to multiple webpages containing detailed information about influenza contagion.

To capture a proxy of loss aversion and excess of choice, we measured features related to writing style using a natural language processing technique called sentiment analysis.25 Sentiment analysis is ‘the computational study of people’s opinions, appraisals, attitudes and emotions toward entities, individuals, issues, events, topics and their attributes’.25 Two common measures of sentiment analysis are polarity and subjectivity. Polarity is a measure of how positive (or negative) a piece of text is, and ranges from +1, meaning positive, to −1, meaning negative. Subjectivity measures how much judgement is embedded in the text by the use of adjectives and adverbs, and ranges from +1 for completely objective to −1 for completely subjective. We performed sentiment analysis using the Python library TextBlob26 to extract polarity and subjectivity score for each educational material. We hypothesised that more negative and objective (what not to do) would be deemed more useful than positive and subjective (best practices).

We also counted the number of times a medicine was mentioned in the educational component. We used the Anatomical Therapeutic Chemical27 classification to identify medicines names and the sum of all mentions was added as feature. We counted the number of external links, as a measure of the importance of providing access to further information, if needed.

Analysis: extracted features from the audit and feedback component

To extract features from the audit and feedback document, we analysed 51 prescriber feedback documents and identified five different behaviour change techniques. To identify relevant codes, one of the authors who has medical background performed manual review to extract behaviour change techniques used in a random sample of prescriber feedback documents (10 documents). The rules and prototypical examples were presented and discussed with the original intervention developers prior to coding (see table 1). The behaviour change techniques were coded using the behaviour change techniques Taxonomy v1.28 Since more than one technique can be present in the same audit and feedback document, the number of different techniques present in an intervention was considered a feature. After consensus, a set of rules for coding was created and applied to each document. Questions and borderline cases were brought back to the group for discussion. We hypothesised that every additional technique would further increase the usefulness of the intervention.

Table 1

Behaviour change techniques and how they were implemented in the prescriber feedback

Behaviour change techniques (code according to behaviour change techniques Taxonomy v1)	Prescriber feedback implementation followed by an example
Prompt/cues (code 7.1)	Display of a message highlighting a patient-specific issue related to the current topic. For example, ‘Patient dispensed an antidepressant and has had no claim for DVA-funded psychological therapies in last 2 years.’
Goal setting (code 1.1)	Display of the suggested action followed by a checkbox. For example ‘Action: consider review.’
Discrepancy between current behaviour and goal (code 1.6)	Display questions about a patient condition that, if answered positively, suggest an action is required. For example, ‘Has adequate pain relief been achieved with current treatment approach? Could adjuvant pain medicine regime be changed?’
Information about health consequences(code 5.1)	Display the rationale for the suggested change. For example, ‘Patient prescribed different medicines that may increase their risk of heat-related illness (see therapeutic brief insert).’
Feedback on behaviour (code 2.2)	Re-interpretation of the prescription data to provide additional data. For example, Translating 20/40 mg for low/high dose; providing drug burden indices.

DVA, department of veterans’ affairs.

Analysis: feature extraction procedure

All educational materials were downloaded from the main program site (https://veteransmates.net.au). We parsed the files to extract text and presentation features, such as number of special formatting (eg, table, headline and external links) (table 2).

Table 2

List of features extracted from the interventions

Feature	Category
Number of images	Presentation
Number of tables	Presentation
Average section size (number of characters/number of headlines)	Presentation
Number of external links	Content
Polarity score	Content
Subjectivity score	Content
Number of medicine mentions	Content
Prompt/cues	Behaviour change technique
Goal setting	Behaviour change technique
Discrepancy between current behaviour and goal	Behaviour change technique
Information about health consequences	Behaviour change technique
Feedback on behaviour	Behaviour change technique

Analysis: outcome

The main outcome of this study was perceived usefulness. Usefulness is a predictor of continued use29 30 and guideline/decision support system use.31 Moreover, it allows the comparison of different types of behaviour change interventions. A postintervention survey was included with mailed material, containing up to 10 questions connected to the interventions content. We extracted information from postintervention surveys asking ‘How useful have you found the therapeutic brief’. Each responder could select one of four options: (1) Very useful; (2) Moderately useful; (3) Slightly useful and (4) Not useful. To create an intervention usefulness score, answers were attributed a score, from 4 (Very useful) to 1 (Not useful). We averaged the scores per intervention, resulting in a final score ranging from 1 to 4.

We used Pearson’s correlation coefficient and Mann-Whitney U test for hypothesis testing. We used Python SciPy package V.1.1.0 for analysis.32

Results

Intervention characteristics

Of the 51 interventions, 48 were evaluated and 3 were excluded due to a lack of outcome data (perceived usefulness). On average, each intervention was delivered to 9667 GPs, for a total of 493 015 individual veterans targeted. GPs eligible for multiple interventions were targeted multiple times. We received, on average, 608 survey responses for each intervention, totalling 29 809 GP responses (6% median response rate (P25=5%; P75=7%)).

Overall, the topics were deemed useful. There were 15 ‘Service initiation’ topics (eg, recommendation of bone mineral density testing), 23 ‘Therapy adjustment’ topics (eg, recommendation to cease unnecessary anticholinergic medicines) and 10 ‘Therapy initiation’ topics (eg, recommendation to use emollients for dermatitis exacerbations). Figure 2 shows the usefulness scores by intervention theme. Usefulness scores ranged from 3.61 (osteoporosis—June 2018) to 2.74 (chronic musculoskeletal pain—March 2014), with a median score of 3.17(SD 0.18).

Figure 2

Usefulness score box plot by intervention theme.

Hypothesis testing

The features exhibited substantial variability over the study period. The exception was the sentiment analysis metrics. Both polarity and subjectivity remained relatively consistent throughout the program duration. This means there was little variation in writing style, regardless of the topic and behavioural goal. Prompt was the most common behaviour change technique, used in 36 interventions (75%), followed by goal setting (33 interventions, 69%). The least common behaviour change technique was information about health consequences, present in four interventions (8%).

We found a significant influence of the number of behaviour change techniques on perceived usefulness. The addition of behaviour change techniques to audit and feedback improved perceived usefulness. Prompt and goal setting strategies were statistically significant (table 3). The effect of behaviour change techniques was summative (figure 3 and table 3). The sum of techniques was considered as an independent variable and was significantly correlated to perceived usefulness (table 3).

Table 3

Association between extracted features and perceived usefulness scores

Behaviour technique	Mann-Whitney U statistics		P value
Prompt	155.0		0.030*
Goal setting	150.0		0.005*
Discrepancy	118.0		0.125
Health consequences	70.0		0.554
Feedback	161.0		0.234
Variable	Mean (SD)	Pearson’s coefficient (SD)	P value
Number of distinct behaviour change techniques	1.79 (1.05)	0.45 (0.19, 0.65)	0.001*
Number of images	0.60 (1.16)	0.18 (0.11, 0.44)	0.226
Number of tables	1.73 (1.23)	−0.16 (−0.42, 0.13)	0.274
Average section size in number of characters	1718 (1094)	0.11 (−0.18, 0.38)	0.447
Number of external links	6.02 (7.07)	0.25 (−0.04, 0.50)	0.088
Text polarity, score +1 to −1	0.079 (0.041)	0.16 (−0.13, 0.42)	0.283
Text subjectivity, score +1 to −1	0.389 (0.057)	0.10 (−0.18, 0.38)	0.475
Number of medicine mentions	31.08 (23.83)	0.17 (−0.11, 0.44)	0.234

*Significant association (p<0.05).

Figure 3

Average usefulness score of topics by number of behaviour techniques.

No presentation or content features were correlated with perceived usefulness (table 3). The number of external links in the educational material showed a tendency to increased perceived usefulness, but was not statistically significant.

Discussion

Our analysis of 48 interventions of an ongoing medicines advice program adds an empirical perspective to designing successful health professional behaviour change interventions. The Veterans’ MATES program is a multicomponent intervention, including an educational component, an audit and feedback component of patient specific information, and a patient mediated intervention. It has repeatedly demonstrated its effectiveness in changing professional behaviour.1 16–20

Overall, the results suggest that changes in the audit and feedback component significantly improve perceived usefulness. Such demonstration of effect of distinct behaviour change techniques is uncommon in the literature, due to the usually small effect and the sensitivity to contextual changes.12 Due to the long duration of the program, we were able to demonstrate statistically significant effects of prompt and goal setting in improving the usefulness of the intervention. This finding also reflects the relevance of adding personalised information in an intervention. Every prompt was generated from patient data and reflected information that may have been unknown to the physician. By bringing accurate information in the right context, the prompt creates a trigger that may increase the likelihood of an appropriate action.

We also found that there was an additive effect of using multiple different behaviour change techniques on usefulness. This effect has been described in the context of health promotion.33 34 This finding suggests that different techniques may independently solve different needs and the use of multiple techniques in the same intervention is not detrimental to effect. In our study, all behaviour change techniques aimed to help physicians in their decision-making process, such as suggesting the evidence-based recommended action, providing scientific information or translating patient information (eg, classifying the dosage as high or low instead of a quantity). This common theme may explain the additive effect of the techniques, as they often complement each other and inform the same choice process.

While we hypothesised that presentation and content features would be determinants of usefulness, none of the studied features were correlated with perceived usefulness. Since the beginning of the Veterans’ MATES program, aesthetic was deemed important and there have been few significant changes in visual identity over the years. However, information organisation has changed considerably without significant impact in perceived usefulness, as it would be expected if changes improved readability or comprehension. There are three possible explanations for this finding: (1) no impact on usefulness—research has shown that, while design improves usability,35 it has limited effect on perceived usefulness;36 (2) threshold effect—aesthetic has been shown to be important in other contexts, such as digital medium,37 38 therefore a possible explanation is that design and presentation may have a threshold effect, over which there are diminishing returns (3) audit and feedback components as confounder—since prescriber feedback changes dominated the intervention effect, it may have masked the impact of educational component features.

With respect to content, our goal was to identify the writing style that conveys a clear health intervention message, for example, content should be more objective rather than comprehensive or normative versus theoretical. Our findings do not support any recommendation concerning writing style. In this study, content effect on perceived usefulness was dominated by the influence of theme. Physicians tended to evaluate topics about service initiation and therapy adjustment (recommendations to remove or reduce medicines) more favourably than topics containing therapy initiation (recommendations to add medicines). This may reflect existing physician knowledge about the topic.

The lack of significant findings regarding content features should be interpreted cautiously, due to study design and the intrinsic complexity of evaluating communication using automated means such as sentiment analysis. The Veterans’ MATES program employs trained and experienced medical writers with multiple peer review processes that maintain consistent text quality. This is evidenced by small variation in the tone and subjectivity scores between different interventions. Also, the sentiment analysis algorithm used in this study is based on the frequency of words deemed as positive/negative and subjective/objective. Therefore, it may not be sensitive enough to detect subtler language features. There is evidence in the health promotion domain that users prefer messages that are correctly spelled and grammatically correct and directive rather than passive or interrogative.39 Emphasis on objectivity and clarity are also recommended for creating medical guidelines,40 41 though empirical evidence is still lacking.

Theories and findings regarding message tone are more conflicting,22 42 which may reflect a dependency on context. It is well established that loss aversion is a driver for insurance purchase, as the fear of rare but large loss is greater than a small but consistent premium.22 In addition, negatively framed messages appear to promote behaviour change by creating stronger cognitive responses.42 A similar reasoning has been applied to medical reasoning giving an important weight to regret during decision-making,43 meaning that a therapeutic decision is only made when the expected usefulness surpasses the potential harm that is always associated with treatment. Other studies have suggested that positive messages may be better received by individuals already motivated to make a behaviour change.39

Translating large-scale health professional behaviour change interventions from a concept to a real-life solution is not a straightforward process. Given the inherent complexity of behaviour change interventions, replicating a successful intervention often fails due to poorly understood factors.44 The strong effect of implementing behaviour change techniques in audit and feedback interventions, particularly prompt and goal setting, suggests a clear recommendation for developers.

Strength and weakness

As an observational research process, the strength of this study is the number of discrete interventions over a long period of time. The context within which the interventions took place remained sufficiently similar, which allowed us to isolate the effect of different features on perceived usefulness.

However, there are important limitations. The response to the surveys was voluntary and the response rate was small, ranging from 12.9% to 2.8% (mean 6.2%). This may have selected those who benefited from the topic and biased the results. Due to the heterogeneity in behaviour change goal, we used perceived usefulness. Finally, despite 15 years’ experience, 51 large-scale interventions are likely underpowered for some of the analyses performed. The small sample also prevented statistical adjustments that could handle potential confounders in observational data.

Conclusion

This study demonstrated that prompt and goal setting components improve perceived usefulness when added to audit and feedback interventions. The addition of multiple different behaviour change techniques in the audit and feedback component improved usefulness and showed a summative effect. None of the content and presentation features extracted from the educational component were correlated with changes in perceived usefulness.