The effectiveness of toolkits as knowledge translation strategies for integrating evidence into clinical care: a systematic review.

OBJECTIVES: The aim of this systematic review was to evaluate the effectiveness of toolkits as a knowledge translation (KT) strategy for facilitating the implementation of evidence into clinical care. Toolkits include multiple resources for educating and/or facilitating behaviour change.
DESIGN: Systematic review of the literature on toolkits.
METHODS: A search was conducted on MEDLINE, EMBASE, PsycINFO and CINAHL. Studies were included if they evaluated the effectiveness of a toolkit to support the integration of evidence into clinical care, and if the KT goal(s) of the study were to inform, share knowledge, build awareness, change practice, change behaviour, and/or clinical outcomes in healthcare settings, inform policy, or to commercialise an innovation. Screening of studies, assessment of methodological quality and data extraction for the included studies were conducted by at least two reviewers.
RESULTS: 39 relevant studies were included for full review; 8 were rated as moderate to strong methodologically with clinical outcomes that could be somewhat attributed to the toolkit. Three of the eight studies evaluated the toolkit as a single KT intervention, while five embedded the toolkit into a multistrategy intervention. Six of the eight toolkits were partially or mostly effective in changing clinical outcomes and six studies reported on implementation outcomes. The types of resources embedded within toolkits varied but included predominantly educational materials.
CONCLUSIONS: Future toolkits should be informed by high-quality evidence and theory, and should be evaluated using rigorous study designs to explain the factors underlying their effectiveness and successful implementation. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

This systematic review on toolkits critically appraises research on strategies to facilitate practice change among health professionals.

Results highlight the importance of evaluating implementation outcomes in addition to behavioural and clinical outcomes.

This review was limited by a lack of an accepted definition for the term toolkit.

Introduction

Knowledge translation (KT) is a complex process occurring between researchers and knowledge users that includes the “synthesis, dissemination, exchange and ethically sound application of knowledge to improve health…provide more effective health services and products, and strengthen the health care system.”1 The degree of engagement in the KT process may be influenced by factors such as the research results and needs of the knowledge user.1 Clinical practice audits have demonstrated that health professionals do not consistently or effectively use current and high-quality research evidence as a basis for clinical care.2 Despite strategies to facilitate the process of implementing research into practice, such as the development and evaluation of clinical practice guidelines, a major disconnect remains between evidence-based practice and actual clinical practice.3

Evidence-based KT strategies for linking research evidence and clinical practice include but are not limited to printed educational materials, educational meetings, educational outreach, the use of local opinion leaders, audit and feedback, and reminders.3 These strategies have been used alone as single KT intervention or as multifaceted KT interventions, which consist of two or more strategies or variations of the same strategies (eg, educational materials) delivered in combination to change practice.4–6 The benefit of multifaceted versus single KT interventions to change clinical and practice outcomes remains unclear, with some investigators reporting they are no more effective.4 7 8

A variation on multifaceted KT interventions is the toolkit. Toolkits offer greater flexibility of use, and for the purposes of this review, are defined as a packaged grouping of multiple KT tools and strategies that codify explicit knowledge (eg, templates, pocket card guidelines, algorithms), and are used to educate and/or facilitate behaviour change.9 Use of KT strategies housed within a toolkit are not necessarily prescribed in any combination or temporality (eg, Strategy A+/or Strategy B+/or Strategy C, etc). The goal is for the user to select KT strategies in the toolkit that are supported by evidence of effectiveness and for use at their own discretion, according to their aims, resources and context. Toolkits differ from multifaceted interventions in which the coupling of more than one KT strategy must be implemented together to comprise the ‘KT intervention’; for example, Strategy A+Strategy B=multifaceted KT strategy.

Evidence-based toolkits can be used to facilitate practice change, and can include strategies for guideline implementation, informing policy, practitioner training, and provide quality audit materials.10 11 Currently, a wide range of toolkits address various clinical disease entities, such as diabetes and cancer care. For instance, the Registered Nurses Association of Ontario offers a toolkit on Best Practice Guidelines for patient care.12 Despite the uncertainty surrounding the effectiveness of multifaceted KT interventions, organisations are investing resources in the development of KT toolkits because they provide a simple, more flexible and expedient method for promoting and utilising best healthcare practices. Whether these toolkits or their components are effectively implemented and positively associated with clinical outcomes remains unknown.

Toolkits comprise KT strategies that can be effective in supporting a range of KT aims if they are based on a clear rationale, quality evidence of their effectiveness, supported by a conceptual framework and built on a careful assessment of contextual barriers.3 To be effective, toolkits should also provide high-quality evidence to guide their use or implementation. Currently, little is known about the effectiveness, feasibility and acceptability of toolkits. The aim of this systematic review was to identify and evaluate the effectiveness of toolkits for facilitating the implementation of evidence into clinical care and to inform future development, implementation and evaluation of toolkits.

Methods

The methods for this review were based on the PRISMA checklist (http://www.prisma-statement.org/2.1.2%20-%20PRISMA%202009%20Checklist.pdf).

Search strategy

A systematic literature search of four electronic databases, MEDLINE (1946–November 2013), EMBASE (1947–November 2013), PsycINFO (1806–November 2013) and CINAHL (1981–November 2013), was conducted by a library information specialist. Search terms included database subject headings and text words for the following concepts: toolkits or toolboxes; evaluation, adherence or outcome assessment; and hospitals and hospitalised patients. The evaluation search terms used in MEDLINE, EMBASE and PsycINFO were based on published optimised search strategies.13–15 CINAHL evaluation terms were based on the optimised MEDLINE strategy. No date, age or language limits were applied (see online supplementary appendix).

Study selection

Study selection was conducted in two stages. First, all titles and abstracts were screened independently by two reviewers (Winnie Lam and Tissari Hewaranasinghage). To establish inter-rater reliability of study selection, each reviewer pilot tested 10 studies using the inclusion criteria. There was 95% agreement on the selected review articles. If necessary, a third reviewer (AS) who was not involved in the selection process resolved any disagreements. In the second stage, the full texts of all selected studies were screened to assess study eligibility and determine the final list of included studies.

Studies were included if: (1) they evaluated the effectiveness of a toolkit to support the integration of evidence into clinical care, either alone or embedded within a larger multistrategy intervention (toolkit +); (2) the KT goals(s) were to inform, share knowledge, build awareness, change practice, change behaviour (in the public), and/or clinical outcomes in healthcare settings, inform policy, or to commercialise an innovation; and (3) they included a comparison group. Studies published in languages other than English, thesis dissertations and studies published in non-peer-reviewed journals or in abstract form only were excluded. All study designs were included. Reference lists from included papers were screened for additional studies.

Methodological quality ratings

The methodological quality of included studies was assessed using the Effective Public Health Practice Project's (EPHPP) Quality Assessment Tool for Quantitative Studies.16 The EPHPP assesses methodological quality in systematic reviews of effectiveness.17 Reliability and content and construct validity of the tool have been established.18

The EPHPP tool can be used to evaluate multiple study designs that include comparison groups. Six categories, each consisting of a series of questions, are used to rate each study: (1) selection bias (two questions); (2) study design (four questions); (3) confounders (two questions); (4) blinding (two questions); (5) data collection methods (two questions) and (6) withdrawals and drop-outs (two questions). Each category is then assigned a rating (strong, moderate or weak), and based on these individual category ratings, a global rating is assigned for the study (strong, moderate or weak). Additionally, the integrity of the study intervention and analyses is also examined; however, they do not contribute to the overall global rating.16

All studies were rated independently by two reviewers (AS and JY) using the EPHPP tool. Prior to rating the studies, the tool was pilot tested on 10 studies. Overall per cent agreement was 88.5% (κ=0.84, 95% CI 0.72 to 0.96). When necessary, consensus meetings were held between reviewers to compare results and reach agreement on all studies. A third reviewer (KW) who was not involved in the quality assessment process resolved any disagreements.

Data extraction and analysis

Utilising a standardised data extraction chart, three reviewers (AS, KW and JY) independently extracted the following data from the studies that received a strong or moderate methodological global rating: study type, type of study participants, toolkit content, KT strategy and clinical outcome measures, including implementation outcomes as defined by Proctor et al19 (ie, acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration and sustainability) and study results. Because many studies embedded the toolkit into a multistrategy intervention (ie, toolkit plus an additional KT strategy(ies)) and did not evaluate the toolkit alone, information regarding all of the components of the KT intervention was extracted. As well, the type of evidence, if any, underpinning the toolkits’ contents (KT strategies, tools) was extracted.

To determine toolkit effectiveness, Lugtenberg et al’s20 method was adopted to assign outcomes from each toolkit to one of three categories: (1) not effective (if no significant effects were demonstrated); (2) partially effective (if half or less of the outcome measures showed significant effects) or (3) mostly effective (if more than half the outcome measures showed significant effects). When study outcomes could not be at least partially attributed to the toolkit (eg, the toolkit was used in the multistrategy intervention and the control group), the study was excluded from detailed reporting.

If similar data from studies were available (eg, means, SDs, proportions), meta-analyses would be conducted. A weighted mean difference, or a standardised mean difference, relative risk, risk difference all with 95% CIs would be conducted using a fixed effects model. If pooling of results would not be possible, a narrative descriptive review of study results would be presented.

Results

The search strategy yielded 39 unique studies for inclusion in this review11 21–58 (figure 1). Given the diversity of studies in terms of participants and outcomes, a meta-analysis was not possible; therefore, we chose to report on all studies with a strong or moderate global ratings rather than focusing only on randomised controlled trials (RCTs) of potentially weak quality.

Figure 1

Study selection flow chart.

The majority of the studies were RCTs (n=11)26 30 33 36 44 45 49 51 53 57 or one-group cohort studies (n=13).21 24 25 27 32 34 37 39 46 50 52 55 58 Eighteen of the included studies had toolkits embedded within a larger multistrategy KT intervention,22 23 26–28 31 32 38 42–45 49 53–55 57 58 and 21 studies evaluated toolkits as standalone KT interventions.11 21 24 25 29 30 33–37 39–41 46–48 50–52 56

Among all of the toolkits, 20 were developed for a specific disease context,21 22 26–28 31–34 40–45 47 51 54 55 57 most commonly for cancer (n=8)27 28 31 40 41 43 54 57 and diabetes (n=3).22 26 51 The remaining toolkits were developed for disease prevention (n=5),23 38 46 52 58 infection prevention (n=2),11 53 postoperative pain (n=1),48 smoking cessation (n=1),49 care in the geriatric population (n=8),24 25 29 30 35 36 39 56 patient safety (n=1)50 and general hospital quality improvement (n=1).37

Toolkits were targeted to health professionals (n=29),11 21 23–27 29–32 35 37–39 42 44–47 49–52 53 55–58 patients (n=10)21 22 28 33 34 40 41 43 48 54 and caregivers (n=1).36 In one study, the intervention included separate toolkits for primary care physicians and patients.18

Only 2611 21 24–32 34 35 37 39–41 43 44 46–51 54 of the included studies specifically indicated the clinical evidence, rationale or theoretical basis underlying the toolkit strategies.

Methodological quality of the studies

The majority of studies (n=26)21–25 27–29 31 32 34 35 37–41 45–48 50 52 55 56 58 were rated as methodologically weak on the EPHPP tool (ie, in terms of study design, selection bias, confounders, blinding, data collection methods and withdrawals and drop-outs); with 8 studies11 26 30 42 44 49 51 53 rated as moderate; and 533 34 43 54 57 as strong. The 13 moderate and strongly rated studies still had some general weaknesses. In 7 of the 13 studies,11 26 30 33 42 44 57 blinding of outcome assessors and/or blinding of study participants to the research question were not explicitly stated. In the selection bias category, only 4 of the 13 studies26 36 44 57 reported the proportion of eligible participants who agreed to participate in the study. As well, in 6 of the 13 studies,11 26 30 36 42 44 raters agreed that the study participants were only somewhat likely, as opposed to very likely to represent the study population, introducing the potential for selection bias.

Evaluation of the effectiveness of the toolkits

In 5 of the 13 moderate to strongly rated studies,11 43 49 53 54 it was not possible to determine if clinical outcomes were attributable to the toolkit because all study participants received the toolkit in some variation. These five studies explored the effectiveness of the toolkit, either alone or paired with minimal additional interventions (multistrategy).A summary of the remaining eight studies is provided in table 1.26 30 33 36 42 44 51 57

Table 1

Effectiveness of toolkits in strongly and moderately rated studies (N=8)

	Study design, participants	Toolkit and intervention	Evidence informing toolkit development	Outcomes measured	Results	Quality
Cavanaugh et al26	Two RCTs at 2 academic medical centresN=198 adult patients with diabetesn=99 (control) n=99 (intervention)Country: USA	Intervention components: enhanced diabetes care programme; training sessions; DLNETControl components: enhanced diabetes care programmeToolkit target: health professionalsToolkit contents: customisable 24 instructive modules about diabetes self-management activities, including blood glucose monitoring, nutrition management, foot care, administration of medications	Incorporated communication principles	Glycaemic control (A1c) Patient-reported self-efficacy of diabetes self-management Self-management behaviours Treatment satisfaction Sustainability	Significant improvements in A1c levels in intervention and control groups at 3 months (adjusted analyses showed greater improvement in the intervention group (p=0.03))Significant improvement in self-efficacy from baseline in both groups (p=0.01,0.02) (but NS differences between groups in adjusted analyses)NS differences between intervention and control for self-management behaviour or treatment satisfactionNS differences between intervention and control groups at 6 monthsImplementation outcomes: sustainability of outcomes measured at 3 and 6 monthsToolkit effectiveness: partially effective	M
Dykes et al 30	Cluster RCTN=8 units in 4 urban US hospitals, N=10 264 patients n=5160 (intervention) n=5104 (control)Country: USA	Intervention components: FPTK; local championsControl components: usual careToolkit target: health professionalsToolkit contents: Morse Falls Scale to assess fall risk; interventions tailored to patient-specific areas of risk; bed poster, patient/family education handout, fall prevention plan (tailored for each patient)	Literature review, focus groups with nurses+nursing assistants, assessment of barriers and facilitators to optimal practice	Patient falls per 1000 patient-days Fall-related injuries Fidelity	Significantly fewer patients with falls in intervention versus control units (p=0.02)Significantly lower adjusted fall rates in intervention versus control units per 1000 patient-days (p=0.04)NS difference in fall-related injuriesImplementation outcomes: protocol adherence >89%Toolkit effectiveness: mostly effective	M
Majumdar et al42	Controlled clinical trialN=14 managed care practicesCountry: USA	Low-intensity intervention components: evidence-based guideline on Helicobacter pylori; ToolkitHigh-intensity intervention components: evidence-based guideline on H. pylori; Toolkit; academic detailing of guideline dissemination by a PCP champion using persuasive educational session; 1 month reinforcement of guideline message; and reminder about eligible patients by a pharmacistControl components: usual careToolkit target: health professionalsToolkit contents: customised list of eligible patients from participating practice; educational materials for patients; patient letters used to arrange for test or follow-up appointment; pre-printed materials including: (A) H. pylori serology test requisitions, (B) preapproved prescriptions, (C) progress notes for patient charts	Not specified	Rate of testing Rate of continued use of acid suppressing medications Sustainability	Significant increase in H. pylori test-ordering in high-intensity intervention versus usual care at 12 months (p=0.02)Significant decrease in proton pump inhibitor use by 9% per year in high-intensity intervention versus usual care (p=0.028)Implementation outcomes: sustainability of outcomes measured at 12 monthsToolkit effectiveness: mostly effective	M
Menchetti et al44	Cluster RCTN=15 primary care groups with 223 PCPsn=8 intervention (128 patients)n=7 control (99 patients)Country: Italy	Intervention components: 2-day intensive training for PCPs; implementation of a stepped care protocol; dedicated consultant psychiatrist; Depression Management ToolkitControl components: usual careToolkit target: PCPsToolkit contents: issues discussed during training with PCPs; diagnostic procedure based on the PHQ-9; treatment algorithm	Based on training program developed by project steering committee	Clinical remission of depression Treatment response PCP behaviour Sustainability	NS differences between groups in remission of depression at 3, 6, 12 months; however in patients with minor/major depression, intervention was more effective than usual care at 3 months (p=0.015). Intervention group showed significantly higher treatment response rates at 3 (p=0.016) and 6 months (p=0.049). PCP increased use of appropriate antidepressants and decreased use of sedatives, hypnotics at 3 months.Implementation outcomes: sustainability of outcomes measured at 3, 6, 12 monthsToolkit effectiveness: partially effective	M
Shah et al51	Pragmatic Cluster RCTn=933 789 adult patients with diabetes (administrative data study)n=1592 patients with diabetes at high risk for cardiovascular disease (clinical data study)Country: Canada	Intervention components: toolkitControl components: usual careToolkit target: family physiciansToolkit contents: introductory letter; tailored 8-page summary of practice guidelines; 4-page synopsis of key guideline elements pertaining to cardiovascular disease risk; small laminated card with simplified algorithm for cardiovascular risk assessment, vascular protection strategies+screening for cardiovascular disease; tear-off sheets for patients with a cardiovascular self-assessment tool; list of recommended risk reduction strategies. Toolkit was packaged in a brightly coloured box with Canadian Diabetic Association branding)	Clinical experts (family physicians, endocrinologists), clinicians with KT expertise	Composite end point of death or non-fatal MI Processes of care	NS difference between groups in death or non-fatal MI (p=0.07) and use of a statin (p=0.26). Decreased use of ECG (p=0.02) and cardiac stress tests (p=0.04) in intervention groupImplementation outcomes: not specifiedToolkit effectiveness: not effective	M
Wright et al57	Cluster RCTN=42 Ontario hospitals (616 patients with stage II colon cancer)Country: Canada	Intervention components: standardised lecture from expert opinion leader; toolkit; academic detailing of local opinion leader; 6-month follow-up reminder packageControl components: standardised lecture from expert opinion leaderToolkit target: physiciansToolkit contents: pathology template; poster and pocket cards emphasising 12 LNs to be assessed in colon cancer	Not specified	Mean number of LNs assessed Proportion of cases staged with a minimum of 12 LNs before and after intervention	Significant increase in mean number of LNs assessed and the proportion of cases with 12 or more LNs retrieved for both groups after standardised lecture (p<0.001). No additional increase noted with academic detailing and toolkit.Implementation outcomes: not specifiedToolkit effectiveness: not effective	S
Goeppinger et al33	4 months RCT and 9 months longitudinal studyN=921 adults with osteoarthritis, rheumatoid arthritis, fibromyalgia or chronic joint symptoms n=463 (control), n=458 (intervention)Country: USA	Intervention components: Arthritis Self-Management ToolkitControl components: no interventionToolkit target: patients with arthritic conditionsToolkits contents (available in English and Spanish): ‘self-test’ to self-tailor the toolkit; information sheets on arthritic-related health issues and on key process components of the Arthritis Self-Management Program (eg, decision-making); Arthritis Help Book; audio relaxation and exercise CDs; audio CD of all material from information sheets	Not specified	Health status Health-related behaviours Medical care utilisation Perceived self-efficacy Sustainability Adoption and appropriateness of toolkit	Statistically significant improvement in 6/7 health status measures, all health-related behaviours and self-efficacy but not in medical care utilisation variables at 4 months postintervention (p<0.01) in intervention versus control groupsResults maintained at 9 months compared with baselineImplementation outcomes: sustainability of outcomes measured at 9 months97% of participants reported use of the toolkit and found it useful. The Book was rated the most useful partToolkit effectiveness: mostly effective	S
Horvath et al36	RCTN=108 dyads of patients with progressive dementia of Alzheimer's type/caregiver n=48(control) n=60 (intervention)Country: USA	Intervention components: Home Safety ToolkitControl components: 1 page standard patient education sheetToolkit target: caregivers of patients with Alzheimer'sToolkit contents: ‘Keep the Home Safe for a Person with Memory Loss’ booklet; low-cost sample items to reduce risky behaviours and accidents	Principles of health literacy, patient-centred care and self-efficacy	Caregiver self-efficacy Caregiver strain Home safety Risky behaviours and accidents Fidelity Costs	Significantly higher caregiver self-efficacy (p=0.002), significantly lower caregiver strain (p≤0.001), significant improvement in home safety (p≤0.001), significantly fewer risky behaviours and accidents (p≤0.001) in intervention versus controlImplementation outcomes: fidelity to protocol achievedCost of toolkit included but not a cost/benefit analysisToolkit effectiveness: mostly effective	S

DLNET, Diabetes Literacy Numeracy Education Toolkit; FPTK, Fall Prevention Toolkit; LN, lymph node; M, moderate; NS, non significant; PCP, primary care physician; PHQ-9, Patient Health Questionnaire-9; RCT, randomised controlled trial; S, strong.

Among the remaining eight studies, three33 36 51 evaluated the toolkit as a single KT intervention against a no KT intervention group, while five26 30 42 44 57 evaluated a multistrategy KT intervention against a no KT intervention group. Only four of five multistrategy intervention studies26 30 42 44 demonstrated partial to mostly effective results. Of the three single KT intervention studies, two33 36 were mostly effective at changing clinical outcomes. Additionally, no studies evaluated the relative effectiveness of each KT strategy (eg, use of audit and feedback); therefore, it was not possible to determine which components contributed to the change in outcomes.

The majority of the studies26 30 33 36 44 51 aimed to evaluate the toolkit's effectiveness for a variety of KT goals. One study focused on changing patient clinical outcomes (eg, myocardial infarction, number of falls); two studies also evaluated change in patient behaviour;26 33 and one evaluated behavioural change in family caregivers.36 Two studies44 51 focused on toolkit effectiveness for changing clinician behaviour in addition to improving patient clinical outcomes, and two studies42 57 were solely focused on improving clinician behaviour.

Implementation outcomes were mentioned in six studies.26 30 33 36 42 44 Dykes et al30 included a process for assessing fidelity of the KT intervention; Goeppinger et al33 examined the adoption, appropriateness and sustainability of the toolkit; Horvath et al36 provided information about the fidelity of the KT intervention and cost of the toolkit but did not conduct a cost/benefit analysis; and Cavanaugh et al,26 Majumdar et al42 and Menchetti et al44 examined the sustainability of improved clinical outcomes over time.

Toolkit content varied across studies. Two studies included self-management toolkits for patients and caregivers with a focus on arthritis33 and Alzheimer's.36 Six studies evaluated toolkits for health professionals on fall prevention,30 gastro-oesophageal reflux,42 depression,44 diabetes26 51 and cancer.57 Toolkit resources included information/handout sheets, posters, pocket guides and educational modules. Wright et al57 included reminder packages for participants comprised of a cover letter from an expert opinion leader, a peer-reviewed article and additional reminder pocket cards. In five studies,26 30 36 44 51 the authors reported that they relied on clinical experts, reviews of the literature or clinical practice guidelines to inform the toolkit components. Dykes et al30 also incorporated an assessment of the barriers and facilitators to optimal practice in falls prevention and designed the toolkit to address the identified barriers.

Discussion

Toolkits, either alone or as part of a multistrategy intervention, hold promise as an effective approach for facilitating evidence use in practice and improving outcomes across a variety of disease states and healthcare settings. There was significant variation in the combination and type of KT strategies contained within the toolkits, a range of diseases for which they were developed, and a variety of intended knowledge users (eg, health professionals or patients/caregivers), all of which contributed to key knowledge gaps.

Most toolkits contained printed educational materials, such as information sheets or guideline summaries, which were intended to fill knowledge gaps. Although feasible and relatively inexpensive, Giguère et al59 reported that printed educational materials tend to have little to no influence on health professional behaviour, and uncertain effects on patient behaviour. Additional efforts are required to ensure that knowledge users actively engage with toolkit materials, moving away from passive diffusion. Wright et al57 utilised reminders within the toolkit. The effects of computer reminders have demonstrated small to moderate benefits; however, further research is needed on other types of reminders, perhaps utilising social media strategies.60 There is currently no definitive evidence for the ideal combination or number of KT strategies and tools that should be used in toolkits. A planful approach (ie, need to identify the KT goal that is being addressed by the toolkit strategy) including evidence-based KT strategies, tailored and planful implementation support, active engagement, and evaluation of KT impacts that include implementation outcomes should be considered for achieving intended KT goals with the targeted audience.

Better understanding of toolkit effectiveness requires more thorough descriptions of the embedded KT strategies/components and how each individual component contributes to study outcomes. Toolkit descriptions and the contents of the eight moderate and strongly rated studies were brief. Dobbins et al61 suggested that use of multiple KT interventions may weaken the key message of the clinical content when compared with single KT intervention strategies, and the same may be true for toolkits. To minimise this potential weakness, each component within the toolkit should have a purpose and rationale3 that is clearly described for toolkit users.

Toolkit components should be based on high-quality evidence, particularly when the goal is to change practice;3 rationale for their inclusion in the toolkit, given the toolkit aims; and guidance on the implementation process62—how they are to be used. Although the eight studies in this review mentioned some form of evidence underlying each component, descriptions were vague and non-descriptive, and few mentioned high-quality evidence, such as systematic reviews. Often, evidence was provided for only one component of the toolkit. Cavanaugh et al26 used communication theory to design the ‘Diabetes Literacy and Numeracy Education Toolkit’, and did not specify any underlying evidence for their content. Shah et al,51 however, provided evidence for using educational materials as a resource within their educational toolkit, which focused on cardiovascular disease screening and risk reduction in patients with diabetes. Nevertheless, their content was not based on a barriers assessment, quality improvement or educational theory.51

Multiple barriers have been identified to account for the knowledge to practice gap, and many are intrinsic to health professionals and their practice environment or context. For example, organisational constraints, such as lack of time or an inability to access resources, are common barriers to KT.2 LaRocca et al63 suggested that the more successful KT intervention strategies were those that were accessible and could be tailored to the needs and preferences of the users. Components of the fall prevention toolkit by Dykes et al30 included patient/family education handouts that were tailored by the nurse based on the knowledge of the patient, thereby capitalising on high tension for change; adaptability, strength and quality of the intervention; and low complexity.64 The effects of tailoring strategies to address identified barriers to change require more clarity, but may improve care and patient outcomes,65 particularly when KT approaches can capitalise on what we know works in implementation.64 Only one of the eight reviewed studies30 assessed barriers and facilitators to inform the toolkit's components.66 Furthermore, determining the influence of modifiable components of context (eg, leadership support, culture, evaluation) would further allow for customisation of KT strategies to facilitate practice change and clinical outcomes.67 Further research is needed on how the toolkit was developed, and the influence of the practice context as these factors may influence study outcomes.

Consideration should also be given to factors implicated in successful implementation.64 Proctor's taxonomy for implementation outcomes19 was extracted from studies where possible, as these outcomes could be used to indicate successful implementation of the toolkit within the healthcare system. Developing toolkits supported by implementation guidance would go a long way in demonstrating how toolkits contribute to good clinical and implementation outcomes. Descriptions of most toolkits lacked details about the implementation process and outcomes. Evidence Based Practice for Improving Quality (EPIQ)68 is an example of a KT intervention that combines evidence, continuous quality improvement, an implementation process and assessment of implementation outcomes. In phase 1 (Preparation), the hospital unit identifies an implementation team, who are trained to review existing unit pain practices, guidelines and research evidence to inform targeted practice changes. In phase 2 (Implementation), the team identifies specific pain practice aims and KT strategies (eg, educational outreach, reminders and audit and feedback) to implement using quality improvement cycles. EPIQ was effective in improving pain process outcomes (ie, pain assessment and management) and reducing the odds of having severe pain by 51%.5

Only two studies reported on fidelity of toolkit implementation. To be clinically effective, healthcare interventions need to be effectively implemented. Yet, implementation outcomes are often overlooked in research and KT practice, creating high potential for type III errors; lack of clarity about whether the intervention or its implementation have been unsuccessful. This type of error can reduce the power to detect significant effects of an intervention.69 Assessing the fidelity of implementing complex interventions addresses type III error and provides evidence of variability in implementation of interventions, which could also contribute to limited effectiveness.70

All eight studies in this review used RCT designs to evaluate toolkit effectiveness. There is a common methodological challenge to RCT studies of KT effectiveness, in that this design could block important contextual factors that now have burgeoning evidence of their importance in successful implementation.64 Caution is required in interpreting which KT strategies are evidence-based, and new studies need to utilise more appropriate mixed methodologies or other types of randomised designs, such as wait listed or stepped-wedged designs, to address what works in implementation of practice changes.71

Several limitations to this systematic review warrant discussion. The term ‘toolkit’ was used in the studies included in this systematic review. However, there is currently no accepted definition for toolkits in existing taxonomies related to quality improvement and behavioural change strategies (eg, Cochrane Effective Practice and Organisation of Care Group). Although we chose a term that had some consistency in the literature, based on the evidence reported in this review, there is no consensus on key content, implementation strategies to promote behavioural change or theoretical approaches that should be included in implementation toolkits. These findings could explain the heterogeneity of the toolkits included in this review. Therefore, capturing all relevant literature was challenging because of the lack of standard terminology used for toolkits. As a result, relevant studies might have been missed by the search. The majority of studies had significant methodological shortcomings and were rated as weak, mostly due to the study designs. One of the limitations was that we focused on studies that evaluated the effectiveness of a toolkit to support the facilitation of evidence into clinical care; therefore, the studies included in this review reported quantitative results.

The literature search was limited to toolkits used in hospital and other clinical settings. Broadening the search to community or public health settings may have yielded additional studies for the review.9

In summary, toolkits have potential as a promising KT strategy for facilitating practice change in healthcare. To fully understand their effectiveness, a systematic approach to planning and reporting their development, the evidence underlying each component, and any direction regarding appropriate implementation is required. Toolkits should have (1) a clearly described purpose, rationale for each component; (2) components that are rigorously developed and informed by high-quality evidence, such as systematic reviews; (3) delivery methods that are guided by a comprehensive implementation process (eg, self-directed, facilitation, reminders) with consideration for fidelity of implementation where appropriate; and (4) a rigorous evaluation plan and study design that can help explain the factors underlying their effectiveness and successful implementation (ie, combining outcome and process measures including context).9

Only a few of the toolkits in this review met all of these criteria.33 51 Ideally future studies of toolkit effectiveness should also be informed by a theoretical approach. In conclusion, this study provides some evidence for the utility of the toolkit.