Development of training for medicines-oriented policymakers to apply evidence.

BACKGROUND: Health systems globally promote appropriate prescribing by healthcare providers and safe and effective medicine use by consumers. Rx for Change, a publicly available database, provides access to systematic reviews regarding best practices for prescribing and using medicines. Despite the value of the database for improving prescribing and medicine use, its use remains suboptimal. This study aimed to develop a training program for five medicine-focused organisations in Canada and Australia to facilitate the use and understanding of the Rx for Change database.
METHODS: Four steps were undertaken: 1) key informant interviews were completed across all organisations to understand the knowledge user perspective; 2) a directed content analysis was completed of the interview transcripts and proposed training was developed; 3) a second round of feedback on the proposed training by knowledge users was gathered; and 4) feedback was integrated to develop the final training.
RESULTS: Sixteen key informant interviews with knowledge users were conducted. Themes for training content included the scope of, navigation and strategies for using Rx for Change (generic content) and practical examples on incorporating evidence within their workplace context (tailored content). The final training consisted of an informational video, a 60-minute face-to-face workshop and two post-training reminders.
CONCLUSIONS: A method of engaging knowledge users in the development of a training program to improve the use of an on-line database of systematic reviews was established and used to design training. Next steps include the delivery and evaluation of the training.

Background

Prescribing of medications is a key element of healthcare treatment and a significant driver of healthcare costs [1]. Significant inadequacies in prescribing practices, medication errors and patient reports of prescribing errors by healthcare providers have been documented across most countries [2, 3]. In Canada and Australia, rates for medication errors are reported to be as high as 30% [2]. Medication administration and prescribing errors have been directly linked to increased morbidity [4] and other negative health outcomes such as increased infections involving antibiotic resistant micro-organisms due to overprescribing of antibiotics [5], preventable hospital admissions [6] and high rates of preventable adverse drug events [7]. Suboptimal medication use by patients is also evident, with adherence to medicines typically around 50% [8]. As such, promoting appropriate prescribing practices by healthcare professionals (providers) and appropriate medication use by patients (consumers) for maximizing patient outcomes and better managing of healthcare costs is of high interest for health systems globally [9].

Currently, a substantial evidence base about the effectiveness of various interventions for improving prescribing and medication use is available [10, 11]. Despite its availability, this evidence base is, in effect, ‘hidden’ from knowledge users (i.e. healthcare policymakers and other stakeholders interested in optimal medication use) due to its dispersion across a wide range of journals and other sources, and potential inaccessibility through the need for journal subscription. Systematic reviews, one of the best sources of research evidence, are often reported by decision-makers as not only difficult to access, but also difficult to understand [12]. Combined with constraints on time, resources and capacity for evidence synthesis among knowledge users, significant challenges to using systematic review evidence in healthcare decision-making exists.

In 2007, the Canadian Agency for Drugs and Technologies in Health (CADTH) in collaboration with the Cochrane Effective Practice and Organisation of Care (EPOC) Group, and the Cochrane Consumers and Communication Review Group (CC&CRG) developed Rx for Change, an online database of appraised and summarized systematic review evidence on various interventions to improve health professional behaviour, including prescribing and medication use (http://www.rxforchange.ca) [13]. The purpose of the database was to facilitate knowledge user access to updated systematic review evidence and, most importantly, facilitate understanding of relevant high quality systematic reviews. Target knowledge users included policymakers, decision-makers for drug plans, quality improvement organisations, non-government organisations, healthcare managers responsible for the development of programs to improve prescribing and medication use, as well as healthcare providers and consumers. Despite the introduction of initiatives to facilitate use of the database (e.g. workshops, webcasts), and positive attitudes reported towards the value of Rx for Change [13], use of this evidence resource has remained suboptimal, with users identifying the need for additional training. Internal information collected by CADTH liaison officers, who represent CADTH within provincial Canadian jurisdictions, indicated that users find the database overwhelming, and the educational supports, while helpful, do not allow enough time to become familiar with database structure and content. Further training opportunities to maximise how the database is used to help inform evidence-based decisions was identified as an outstanding need.

The overall objective of this research was to engage knowledge users in order to co-design, implement and evaluate a training program for facilitating use and understanding of the Rx for Change database. This paper describes Phase 1 of this initiative – the development of a knowledge user-informed training program for the Rx for Change database. Phase 2 will include implementation and an evaluation of the training program. Five medicine-focused organisations were involved; three organisations in Canada who were concerned with policy relevant to provider behaviour and two organisations in Australia who were concerned with policy relevant to consumer behaviour.

Methods

Development of our training used an integrated knowledge translation (KT) approach [14] and was guided by the Knowledge-to-Action (KTA) framework [15]. This approach recognizes the inherent benefits to collaborating with knowledge users and including the perspectives of the target audience of an intervention in its development [14]. The KTA framework outlines the following steps as important avenues for targeting knowledge user input when developing an intervention: problem identification, contextualizing evidence and barriers assessment, and intervention development. Guided by the KTA framework, our training development process consisted of four steps: (1) key informant interviews; (2) analysis of interviews and draft training development; (3) confirmation of feasibility of the proposed training by knowledge users; and (4) final training development.

Five principal knowledge user organisations (three Canadian, two Australian) participated. Each organisation provided outreach to either health providers or consumers specific to best practices for the prescribing and use of medicines. These five organisations were recruited to participate in the study based on their interest in using Rx for Change on a regular basis, and each had expressed an identified need for additional Rx for Change training. Indeed, this project grew from the view of the organisations that the database had a lot to offer and that they were not using the database to its full potential when developing and implementing evidence-based medicine use programs within their jurisdictions. Our interest was also in ensuring the inclusion of both provider and consumer perspectives – a direct link to the two Cochrane review groups (EPOC and CC&CRG) that were involved in the development of Rx for Change. Having three provider-oriented organisations in Canada and two consumer-oriented organisations in Australia was a reflection of the international nature of our team (i.e. Canada and Australia) and the countries in which the two Cochrane review groups (EPOC and CC&CRG) were (at the time) situated.

Each of the organisations had a principal knowledge user who was part of the research team and was involved in the entire research process, including the development of the research protocol. The same process of intervention development was followed at all five of the organisations.

The study was approved by the research ethics boards at the Ottawa Hospital Research Institute, Ottawa, Canada, and La Trobe University, Melbourne, Australia (#20130010-01H).

Step 1: Key informant interviews

The principal knowledge user from each organisation was asked to recommend staff members to participate in key informant interviews. Staff members’ selection for the interview was based on (1) the position of the individual within the organisation, ensuring that a diverse sample was achieved and (2) a job role appropriate to the use of Rx for Change ensuring that a targeted but diverse sample was achieved. A semi-structured interview guide developed by the research team and which included the knowledge users was used; the interview guide addressed the following topics: organisation characteristics, use of Rx for Change, use of evidence for decision-making, barriers and facilitators to using Rx for Change, local training needs, and preferred approaches to the training. Following these topic questions, the interview included a 15-minute session in which key informants navigated the Rx for Change database. They were asked to use the database to seek evidence regarding either a relevant project the informant was undertaking or a hypothetical project suggested by the interviewer. This navigation task was intended to provide additional information regarding the process of using the database that could not be collected from questioning. All interviews were audiotaped and transcribed verbatim, including the dialogue from the navigation session.

Step 2: Analysis of data and development of draft training

A directed content analysis [16] of all transcripts was performed and NVivo Version 9 (QSR International, Cambridge, MA) was used for data management and coding. This entailed line-by-line coding into the existing interview categories (e.g. barriers to using Rx for Change, local training needs), followed by a thematic analysis that considered each interview category independently as well as all of the interview categories together. This method provided the opportunity to focus our attention on separate aspects of the intervention design as needed (e.g. local preferences for the training) as well as an overall thematic analysis. The line-by-line coding was conducted by one member of the research team from each country (EH in Canada, DL in Australia) and the thematic analysis was conducted by at least two members of the research team (two of HC, EH, DL). The intent of the separate country coding was to determine if there were any differing perspectives based on provider-oriented and consumer-oriented organisations. The results of the analyses were used to develop a proposed training program.

Steps 3 and 4: Confirmation of feasibility of proposed training by knowledge users and development of final training

The project team confirmed the feasibility of the proposed training program with one target knowledge user in each organisation. These targets were the principle knowledge users that were on the research team or key leaders in the organisation. This feedback was collected verbally from each target after sharing the proposed training. Summary information from each of the targets was discussed with three members of the research team (HC, EH, DL) and final decisions were made regarding the content and delivery of the intervention. The intent of this round was to ensure that the proposed training was consistent with what was feasible for each of the organisations in terms of time and resources and perceptions of what would benefit the organisation most. This step resulted in a final training program for implementation and evaluation that was customized to the needs and audiences for each of the knowledge user organisations.

Results

Characteristics of key informants

Nine face-to-face key informant interviews across all three Canadian organisations (n = 1, n = 3, n = 5) and seven (one face-to-face and six telephone) interviews across two organisations in Australia (n = 3, n = 4) were completed between April and July 2013. All of the staff members who were recommended by the organisation and then approached for an interview (n = 16) agreed to participate. The average duration of the interviews was approximately 45 minutes. Key informants held their positions in their respective organisation for an average of six years (range 1–16 years) and had worked in the area of healthcare for an average of 19 years (range 1–30 years). Eight of nine Canadian and six of seven Australian key informants were aware of Rx for Change. Seven of nine Canadians, and five of seven Australians had used Rx for Change but none of the key informants had received any formal training in using the database. Three of nine Canadians and three of seven Australians reported having been encouraged to use the database within their organisations.

Analysis of data and development of the training

Analysis of the transcripts revealed six key themes to guide the development of the training: (1) need for knowledge on scope and content of database; (2) developing skills for navigating the database; (3) developing skills for incorporating evidence into decision-making; (4) supporting routine use of the database; (5) need for tailoring; and (6) diversity of delivery preferences. Each is summarized below. Notably, the themes identified were consistent; there were no discernible differences found between the provider- and consumer-oriented organisations.

Need for knowledge

The interviews revealed a set of information about Rx for Change for which the key informants were either unaware of or had an incorrect knowledge of the facts. This misinformation included the type of information in Rx for Change, what a systematic review is, how the database is populated, how it is kept up to date, the meaning of the quality assessment, understanding of the taxonomies on which Rx for Change is based [17], and the value of utilising the database. The key informants were expecting Rx for Change to provide them with clear direction on changing prescribing/medicine use behaviours and were often frustrated by the limited practical information that they were able to find.

Developing skills for navigating the database

Key informants indicated in general that Rx for Change was not a user-friendly database, both in terms of layout and the terminology used. The general perception among key informants was that the database would be similar to other general search databases (e.g. PubMed) creating the mistaken perception that one could search the database using phrases, synonyms, limiters and/or Boolean operators to refine the relevance of the search output. When the simplistic functionality of keyword searching proved limiting, users lost interest and stopped searching.

Developing skills for using evidence for decision-making

Despite indicating a belief in the value of using evidence, the key informants expressed difficulty in how to find the evidence and how to incorporate the evidence found into their decision-making processes. Key informants expressed that, often, evidence was difficult to access and they were uncertain how to use the evidence into their day-to-day decision-making.

Routine habit

Key informants expressed the need for support to make the use of the database part of their routine activity. They believed that having reminders to use the database would be helpful.

Need for tailored content

Key informants expressed the need for the training to focus on issues that were currently relevant to the users’ work context.

Diversity of delivery preferences

Significant heterogeneity existed across organisations in terms of preferred approaches for training, with some knowledge users requesting a full day workshop and others indicating they would only access material they could engage with on their own time. Mixed feedback was received in terms of what the training should look like and ranged from a 15-minute instructional video to be viewed at the discretion of the respondents/participants, through to a 2-day workshop.

Although the thematic analysis provided valuable information on the perspectives of the knowledge users and served to guide intervention development, the thematic analysis alone was insufficient to develop the intervention. In order to facilitate the process of designing the intervention, our team engaged in a process of reviewing the thematic analysis while also considering three factors: (1) what the themes meant in terms of the content and delivery [18] of the training; (2) the need for tailoring (both in content and delivery) [19, 20]; and (3) evidence that would support the use of different strategies of the training. This approach was taken to ensure clear links between the interview themes and the training components. We felt that this process was best achieved with a 1-day face-to-face meeting between three of the researchers who were involved in the thematic analysis (HC, EH, DL) with additional input from two other team members who were not involved in the thematic analysis (DB, JG). Table 1 outlines the end result of this meeting with a description of how the themes and the information within the themes were linked to the tailored content and delivery of the intervention, and evidence.

Table 1

Summary of training development process – Stage 1

Theme	Descriptive details of theme	Considerations for training (content, delivery, tailoring)	Rationale (evidence to support training approach)
Need for knowledgeQuote: “I think the main barrier is knowledge. … knowledge of how it works, what it contains and for me would be having a sense of how often it is updated, how comprehensive is it, how much I can trust it, that is the kind for that key information that I am looking for. Has this done enough of the work in this area for me to use or do I have to do additional search”	Targets expressed limited knowledge of the type of information contained within Rx for Change, how the database is populated, how it is kept up to date, the purpose and relevance of the quality assessment, understanding of the EPOC and CCCRG taxonomies on which Rx for Change is based, and the value of database	Content: Include information on as many of the deficits as possibleDelivery: Fact sheets, FAQ sheets, 1:1 didactic, webinar, workshopTailoring: Create an introduction to the workshop or webinar that can be emailed to potential participants to provide basic knowledge that some might need more than others	Knowledge: provision of information [38]
Needed skills
1) Navigating the databaseQuote: “the key one I think in terms of being challenging to find, to understand how to use the system or how to search for things”	Issues for navigation: how to enter search terms, add limits, open or closed search, how to determine how many studies are in the review, how are individual studies assessed, how is drop down menu subdivided, what should one expect to find under each heading, why did this review show up as opposed to other reviews	Content: Use examples from participants or create standard examples. Hands-on navigation using relevant examples. Conduct comparative exercises using Rx for Change versus other databases to gather evidence. Use of a worksheet to facilitate navigation skills. Use screen capture of navigation of the data base that uncovers the layers and explains the components of the databaseDelivery: Workshop, webinar, YouTube videosTailoring: Locally relevant examples for practice	Modelling, demonstration [38]
2) Incorporating evidence into decision-makingQuote: “if you were looking for some suggestions as to a way to structure and implementation project so go here and it will tell you what’s the most helpful, what is the quickest, or what is the whatever, but it doesn’t really do that”.	Incorporating info into decision-making: The limitations of evidence in general related to decision-makingUnderstanding use when evidence is there and strong and when it is notUnderstanding how Rx for Change can help them with their decision-making, how it fits with the other sources of evidence they may use	Content: Use of a guide or rubric to facilitate learning. Use of an example, group discussionDelivery: Workshop, webinar, practical and hands onTailoring: Locally relevant examples	Modelling, demonstration [38]SUPPORT format for applying evidence [21]
Routine habit	The need to make using the database a routine aspect of every prescribing-based project	Content: The use of reminders for the database’s use, value, navigation tips. Follow-up with resources (Frequently Asked Questions, workshop presentation) and a reminder to use Rx for Change at 1 and 3 months post workshopDelivery: Email post-training, flyers	Reminders [42]
Need for tailoring		The tailoring required for each theme is summarized by the themes
Diversity of delivery preferences		Content: n/aDelivery: Differing perspectives on delivery requires a multi-pronged approach in order to engage the groupTailoring: Tailor the training to the different groups by having different options for the content

Confirmation of feasibility of proposed training by knowledge users and development of final training

The second round of input on the proposed training was obtained from one principal knowledge user in each of the five organisations. This feedback mostly focused on available time in each organisation in terms of resources as well as input on relevant projects or upcoming program development that might be informed by evidence. The input highlighted the need to limit the total training to a 1-hour workshop as all organisations were limited in the time they could commit to staff education. Principal knowledge users also provided needed information on how best to ensure that the workshops could accommodate the plan for all participants to have access (i.e. devices and Wi-Fi) to the Rx for Change database during the workshop.

Final training

Table 2 outlines details of the final training intervention using the headings content, delivery and tailoring as well as the associated themes that each aspect of the training addressed. A brief description of the contents of the intervention is included below:

Table 2

Detailed description of delivered training with related themes

Training description	Related themes
1. Video: Invite for the workshop and foundational information on Rx for Change and systematic reviews Delivery: A video (prezi with audio narration) covering general information about Rx for Change and systematic reviews; the video was sent electronically 1 week prior to the workshop with a request for people to review it prior to attending the workshop; duration was 6 minutes Content: Quick facts about the database, information on the type of evidence included, information on when it can be used and how it can help participants in their work; an invitation to attend a free workshop and to explore Rx for Change on-line in a group environment was included	Need for knowledgeDiversity in delivery
2. Face-to-face workshop (1 hour) Delivery: Didactic presentation (15 min) + hands on skills training (45 min) + discussion (15 min) + handouts Content of didactic presentation: Rx for Change was presented as part of the evidence source “puzzle” Knowledge users’ questions (as per interviews) partly guided the session’s themes (e.g. What is Rx for Change? Can I trust the evidence presented? What types of questions are relevant? How is Rx for Change different compared to other evidence sources?); the didactic part of the session focused on emphasizing “How we turn a research question into a “searchable question” in the Rx for Change database” Content of hands-on skills training: A pre-determined example of a research question was used to demonstrate and familiarise participants with strategic navigation strategies for the Rx for Change database; screenshots were used along with an audio or live presentation/demonstration; the three-question framework based on Lavis et al. [21] was used to demonstrate different ways of searching efficiently, the layout of the database, the taxonomy and the layers of the database Participants were then asked to use one of their own work examples and search the database based on the same three-question framework; this was a facilitated exerciseThe session was facilitated by a research team member with expertise in Rx for Change or an external collaborator with knowledge of Rx for Change Content of discussion: The incorporation and applicability of the evidence in one’s context as well as the strengths and limitations of the database and the evidence that it includes was discussed Handouts: Frequently Asked Questions and a “map” of the database layout (screenshots) were given to participants in the form of handouts after the session (for desk use)	Didactic presentation: Need for knowledgeHands on skills training: Need for skill development in navigating the database and integrating evidence into decision-making, tailoringDiscussion: Knowledge and skillsHandouts: Need for knowledge, variety in delivery methods
3. Post-training remindersDelivery: Emailed at 1 and 3 months post-workshopContent: Cues and/or reminders regarding the database’s use, value, updates, and navigation tips	Routine habits

An informational video: This 6-minute video serves as an invitation to the workshop as well as a way to provide basic background information about the database and systematic reviews. The video link is sent electronically to workshop participants who are asked to view the video prior to the workshop. A number of key informants had expressed the need for a brief and practical session, so this component of the training module aims to provide necessary background information about the database that can be viewed on participants’ own time, thus allowing for the main training session to be brief and focus on practical aspects of using and understanding the database.

A 60-minute face-to-face workshop: The workshop consists of a 15 minute didactic portion followed by a 30 minute skills demonstration and practice with the database, and a 15 minute discussion. The skills demonstration and practice takes the participants through the SUPPORT Tool process [21] using a current and local topic provided to the research team by the key knowledge users in each of the organisations. The SUPPORT tool process includes the following steps: Identifying the problem and/or question at hand, “What do I want to achieve?”; identifying the available options to address the problem; identifying the available resources; decision making; and implementation of most suitable options. Each step is conducted with direct connection and relevance to the local and current prescribing-based issue in the organisation. A full set of the slides is available by request through the corresponding author. The hands-on component includes (1) how to navigate the database and demonstrate the structure and different layers of Rx for Change, and (2) how the information within the database can be used to answer questions as related to the example or scenario. Handouts of the content and presentation are given to the participants.

Two post-training reminders: Reminders are sent to workshop participants at 1 and 3 months following the training and consist of an email reminding them about the workshop on Rx for Change and encouragement to use the database as a routine part of their work.

Discussion

We describe the development of a knowledge user-informed training program to increase the use of Rx for Change, an on-line database of quality assessed and summarized systematic reviews focused on best practice for the prescribing and use of medicines. Our process was focused on ensuring that the training incorporated the perspectives of the target knowledge users. We interviewed 16 knowledge users across two countries and five organisations in a process of information gathering and training development and clearly situated the results of the interviews in terms of the content, delivery and tailoring of the training. Next steps include implementation and evaluation.

Policy decisions involve a number of competing factors in addition to evidence [22]. In a qualitative study of how drug policymakers make decisions, Ritter et al. [23] found that evidence is but one of nine potential sources of information used to make decisions including politics, funding and public opinion. In fact, scientific evidence was found to be one of the least preferred sources due to its time consuming and contradictory nature. Policymakers by far preferred summaries of information [24, 25]. While Rx for Change was designed to align with these preferences through the use of evidence summaries, it might not be comprehensive enough, or indeed, even be in a preferred format to meet the needs of policymakers. It is possible that regardless of how well an on-line database of systematic reviews is designed and how well training is aligned to user needs, the lack of importance attached to scientific evidence will prevent optimal use of such resources. A better understanding of how to summarize and integrate systematic review evidence into policy decision-making is needed. Future work could include follow-up with policymakers to assess how information from sources such as Rx for Change impacted their policy decision-making experiences/process.

Engaging knowledge users in the training development process highlighted knowledge gaps and user information needs and preferences. Commonly referred to as integrated KT (iKT) [14], this process of engaging users in the development and testing of KT interventions has been increasing in popularity [26, 27]. As is typical to iKT efforts [28] there was a sharing of perspectives with both the research team and the knowledge users broadening their perspectives on the use of evidence and the Rx for Change database. However, including the perspectives of the knowledge users in training design required time and resources. The entire process of training development took approximately 1 year. Our training is more likely to be relevant to the knowledge users [15] and, therefore, potentially more effective but not without a cost. Although not designed to test this assumption, our implementation and evaluation will provide useful information on the benefits of our training.

The process of integrating a qualitative analysis of user perspectives into the design of an intervention was not straightforward. It was a challenge to make this process systematic, rigorous and replicable. The preferences for training vary widely. Although knowledge users were asked to consider what their organisation might need for database training more generally, it is not clear how different knowledge levels about the database prior to the interviews affected their stated needs for training. Improving the problem of inadequate reporting of interventions [29], as well as their design [30], is needed. Other methodology papers for KT intervention design do exist [31-34] and some outline an integrated KT approach [35], but these are not abundant. Research teams that design interventions should be encouraged to publish methods papers on their processes for intervention design. The use of tailoring and situating the training in a local context was important to our knowledge users, even across organisations that differed in roles and responsibilities. Although modest, some evidence exists for the effectiveness of tailoring KT interventions [19]. Balancing core information with tailored information is an issue that needs attention because training that is relevant has to connect with specific locales or organisational responsibilities. In addition, clarity of the benefits of tailoring would provide justification for its time consuming nature.

Different methods exist for the design of interventions aimed at changing health provider or health behaviour [32, 34, 36]. While consensus on a ‘best’ approach has not been achieved, recommendations are often made to use a theory-based approach. Our approach focused on collecting knowledge user perspectives to design the intervention and drew upon an iKT perspective. We did not, however, specifically draw on other KT or behaviour change theory to design our intervention. Although much encouragement and recommendation exists for the judicious use of theory to design interventions [30, 37, 38], theory use remains low [37]. It is likely that a range of methods are needed, both including the use of theoretical frameworks as well as specific theories for characterizing and designing behaviour change interventions such as the Capability, Opportunity and Motivation model [39]. Understanding of which method produces the best results under which conditions remains to be found. We have presented a relatively straightforward process.

Several limitations warrant mentioning. We only conducted one round of feedback. The phase to confirm feasibility of the intervention was conducted with a manager at the organisation and not with all the individuals who were interviewed. It is possible that additional rounds of feedback would have resulted in a more developed and possibly different intervention. The initial line-by-line coding into the existing interview categories (e.g. barriers information into a barriers code) was undertaken by one person, whereas the thematic analysis was undertaken by two people; therefore, we might have ended up with different coding had two people been used in the earlier stage. Finally, we focused our efforts on the iKT approach and not on other potentially informative concepts such as principles of user-centred design [40] or literature related to web-enabled resources [41]. Future efforts to understand the user experience in relation to on-line databases more broadly would be worthwhile.

Conclusions

We have described a method of engaging knowledge users in the development of a training program to improve the use of Rx for Change, an on-line database of systematic reviews. Evaluation will guide future activities to optimize Rx for Change use by varied user groups and guide our efforts for the use of databases of evidence summaries in general.

Abbreviations

CADTH, Canadian Agency for Drugs and Technologies in Health; CC&CRG, Cochrane Consumers and Communication Review Group; EPOC, effective practice and organisation of care; iKT, integrated knowledge translation; KT, knowledge translation; KTA, knowledge to action