Scoping review of costs of implementation strategies in community, public health and healthcare settings.

OBJECTIVES: To identify existing evidence concerning the cost of dissemination and implementation (D&I) strategies in community, public health and health service research, mapped with the 'Expert Recommendations for Implementing Change' (ERIC) taxonomy.
DESIGN: Scoping review. DATA SOURCES: MEDLINE, EMBASE, CINAHL, PsycINFO, Scopus and the Cochrane Library were searched to identify any English language reports that had been published between January 2008 and December 2019 concerning the cost of D&I strategies. DATA EXTRACTION: We matched the strategies identified in each article using ERIC taxonomies; further classified them into five areas (eg, dissemination, implementation, integration, capacity building and scale-up); and extracted the corresponding costs (total costs and cots per action target and per evidence-based programme (EBP) participant). We also recorded the reported level of costing methodology used for cost assessment of D&I strategies.
RESULTS: Of the 6445 articles identified, 52 studies were eligible for data extraction. Lack of D&I strategy cost data was the predominant reason (55% of the excluded studies) for study exclusion. Predominant topic, setting, country and research design in the included studies were mental health (19%), primary care settings (44%), the US (35%) and observational (42%). Thirty-five (67%) studies used multicomponent D&I strategies (ranging from two to five discrete strategies). The most frequently applied strategies were Conduct ongoing training (50%) and Conduct educational meetings (23%). Adoption (42%) and reach (27%) were the two most frequently assessed outcomes. The overall costs of Conduct ongoing training ranged from $199 to $105 772 ($1-$13 973 per action target and $0.02-$412 per EBP participant); whereas the cost of Conduct educational meetings ranged from $987 to $1.1-$2.9 million/year ($33-$54 869 per action target and $0.2-$146 per EBP participant). The wide range of costs was due to the varying scales of the studies, intended audiences/diseases and the complexities of the strategy components. Most studies presented limited information on costing methodology, making interpretation difficult.
CONCLUSIONS: The quantity of published D&I strategy cost analyses is increasing, yet guidance on conducting and reporting of D&I strategy cost analysis is necessary to facilitate and promote the application of comparative economic evaluation in the field of D&I research. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This review provides a novel approach to summarise the cost outcomes of dissemination and implementation (D&I) strategies using the ‘Expert Recommendations for Implementing Change’ (ERIC) terms and definitions.

The review was limited to studies conducting D&I activities (ie, D&I strategies), applied evidence-based programmes (EBPs) and have reported costs associated with D&I activities.

The current body of literature does not consistently identify and map the D&I strategies to specific D&I outcomes or sufficiently differentiate the activities of implementation from the activities of an EBP which limits the ability to assign costs consistently across studies.

Some implementation activities in community settings might not be captured or cannot be mapped appropriately because ERIC taxonomies focus primarily on healthcare-related strategies.

Background

Dissemination and implementation (D&I) science is widely recognised as central to enhancing the uptake and use of evidence-based programmes (EBPs), and thus to improving healthcare and health outcomes.1 D&I strategies are methods or techniques used to enhance implementation, sustainment or scale-up of an EBP.2 In the past two decades, significant progress has been made in the explicit identification, development, refining and testing of D&I strategies to facilitate the reach, adoption, implementation and sustainability of EBPs.2 The Expert Recommendations for Implementing Change (ERIC) project was convened to address the challenges of reconciling the terms and definitions of D&I strategies across a wide range of disciplines (eg, implementation and clinical experts, and health services researchers) through expert consensus.3 In spite of these progresses, few studies have applied an economic evaluation of D&I strategies, which may attribute to the fundamental difference between economic evaluation in D&I research and economic evaluation in health service research/healthcare. The former addresses the resources (eg, implementation activities of training or education) needed to increase reach, adoption, high-quality implementation and sustainability of the EBP, whereas the latter focuses primarily on the costs of the delivery of an EBP (eg, behavioural intervention programmes) or the EBP itself (eg, pharmaceutical or surgical interventions).4

In a recent systematic review, Reeves et al5 identified 14 articles examining costs, consequences and cost-effectiveness of strategies designed to influence the adoption of public or population-level interventions in community settings. Their review included D&I strategies that followed the taxonomy of professional, organisational, financial and regulatory strategies developed by the Effective Practice and Organization of Care group. This was compiled nearly 20 years ago, and is likely outdated with the current advances in classifying and reporting D&I strategies. The authors concluded that more applications of cost and/or cost-effectiveness analyses of D&I strategies are needed to understand their economic consequences. Roberts et al6 conducted a systematic review of health economic evaluation focusing on improvement or implementation science studies. These authors included studies covering aspects of quality or service improvement, health and clinical service delivery, staff behaviour change and patient behaviour change in their review. In total, 30 eligible studies were included. The authors noted that their review was limited by the number of studies identified and synthesised, especially among studies associated with implementation or implementation strategies. Similar to Reeves et al’s study, they concluded that though the number of economic evaluations in implementation research is increasing and presented with improved quality, more applications should be widely encouraged considering recent advances within the field and the plethora of terms and definitions to describe D&I strategies.

Recently, Powell and colleagues7 identified the need to increase economic evaluations of D&I strategies as one of five research priorities for advancing D&I science. Comparative economic evaluations of D&I strategies provide critical information for payers, policymakers and providers to make informed decisions and determine if specific strategies are an efficient use of often scarce organisational resources.4 Current areas of challenges in this pursuit include: (1) the lack of a standardised system and methodology for gathering information on D&I strategy costs; (2) inconsistent reporting on costs in the existing literature (eg, <10% D&I studies included information regarding implementation costs)7 8; (3) the lack of standardised D&I outcomes for evaluating the effectiveness of implementation strategies9; and (4) the sometimes fuzzy boundaries between costs of strategies that focus on reach, adoption, implementation and sustainability (a D&I strategy rarely influences one implementation outcome alone). Identification and estimation of costs incurred when adopting and implementing an EBP would be key to address this need.10–13 It is known that D&I strategies can vary in their intensity and resource use as well as their effectiveness in facilitating implementation of EBPs. For example, in Reeves et al and Roberts et al’s reviews,5 6 they both found it difficult to identify the spectrum of D&I strategies.

Accordingly, the objectives of the present study focused on leveraging recent advances in D&I science to facilitate the widespread, routine use of EBPs to bridge the research–practice gap3 4 through answering the following questions: (1) What is the evidence concerning the cost of D&I strategies in community, public health and health services research? (2) What is the available cost information of D&I strategies that focus on reach (ie, dissemination), adoption (ie, scale-up), the implementation process, integration and capacity building? (3) What are the research gaps? Specifically, we considered costs of strategies using the ERIC taxonomy3 to: (1) facilitate the development, testing and comparison of economic analyses; and (2) categorise strategies as those intended to address initial adoption, implementation quality or sustainability. Finally, due to the lack of guidelines in costing of D&I strategies, we proposed a cost analysis guide based on both the best practice recommendations for economic evaluation in healthcare14 15 and the findings from our scoping review for future D&I research projects- that will allow for a more systematic and consistent comparison of costs as they relate to D&I strategies and outcomes.

Methods

We used a scoping review approach, a rapid form of knowledge synthesis where the aim is to map the key concepts underpinning a research area and the main sources of evidence available, to provide an overview of the available evidence.16 17 We followed the Arksey and O’Malley framework16 and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist18 to organise our procedures and this paper. The PRISMA-ScR checklist is available in online supplemental table S1. The framework consists of the following five steps: (1) identify the research questions, (2) identify relevant studies, (3) study selection, (4) chart the data, and (5) collate, summarise and report the results. The research questions were identified in the Background section, steps 2–4 were described in this section and step 5 was reported in the Results section.

Patient and public involvement

Patients or the public were not involved in the design, conduct, reporting or dissemination plans of this research.

Identifying relevant studies

The librarian in the research team (CS) developed the search protocol and strategy in consultation with other team members. We conducted searches of MEDLINE (via OVID, including In-Process & Other Non-Indexed Citations), EMBASE (via embase.com), CINAHL and PsycINFO (via EBSCOhost), Scopus and the Cochrane Library (including the Wiley versions of the Cochrane Database of Systematic Reviews and the Cochrane Central Register of Controlled Trials) on 17–18 December 2019. Subject headings were not helpful in focusing the search on implementation costs so only keywords and key phrases were used in the search strategies. Terms were identified for the four search concepts: ‘cost’, ‘evidence-based program’, ‘strategy’, ‘implementation’. Words or phrases from the names of the 73 discrete implementation strategies included in ERIC3 and synonyms for these terms were among those used for the strategy concept.

Databases were searched for articles that had titles containing one of our implementation terms, a title or abstract containing one of our evidence-based programme terms and a title or abstract that contained one of our cost-related terms within five words of one of our strategy-related terms. Appropriate truncation symbols were used to account for search term variations and maximise retrieval of relevant articles. Of note, because Scopus includes a great deal of non-health-related content and the goal of our review was to identify health-related programmes and cost data, we used filters to limit our results to literature concerning medicine; social sciences; business, management and accounting; biochemistry, genetics and molecular biology; nursing; psychology; pharmacology, toxicology and pharmaceutics; health professions; immunology and microbiology; neuroscience; multidisciplinary; dentistry; and undefined subject areas.

All database searches were limited to articles published between January 2008 and December 2019 due to the proliferation and advances in D&I research in the past decade. When available, English language filters were applied and editorials, conference abstracts, book chapters and dissertations were removed using search filters. However, our search strategy was lengthy, and these additional search steps were more than the PsycINFO and CINAHL search engines (both via EBSCOhost) could handle. Articles published in Trials or Contemporary Clinical Trials journals focused on publishing trial protocols were removed from the EMBASE and MEDLINE results. This step proved overwhelming for the remaining databases. In order for the search to run in the Cochrane Library, the search had to be divided into five separate searches which produced a total of 653 records for 550 unique items. Many of the items represented by the Cochrane Central Registry of Controlled Trials records corresponded to undesired publication types (ie, articles in foreign languages, clinical trial registry records, conference abstracts, articles from Trials and Contemporary Clinical Trials). The CINAHL, PsycINFO and Scopus results also included records for undesired publication types (ie, dissertations, conference abstracts, books, book chapters, editorials).

All the search strategies are available through University of Nebraska Medical Center’s digital repository (https://digitalcommons.unmc.edu/search/8/). As an example, the full search strategy for EMBASE is presented in online supplemental table S2. Researcher CS imported the 1960 CINAHL records, 653 Cochrane Library records, 3027 EMBASE records, 3260 MEDLINE records, 1087 PsycINFO records and 5736 Scopus records into a RefWorks database. RefWorks duplicate detection tools were used to remove 8651 duplicate records leaving 7072 records for unique publications. After duplicates were removed, CS used the RefWorks search and sorting tools to identify 627 records of undesired publication types (ie, foreign language articles, clinical trial registry records, conference abstracts, dissertations, book chapters, books and articles from the journals Trials and Contemporary Clinical Trials). This left 6445 records for title and abstract review.

Study selection

EndNote, CADIMA and an Excel spreadsheet were used to manage records and data throughout the remainder of the review process and a prespecified template was used for data abstraction. All identified studies were uploaded to CADIMA,19 an online platform, which automatically removed 17 duplicates remaining at the time of upload. Titles and abstracts were split and screened independently by four investigators (TLM, GP, CG and EP) in the research team. Similarly, four investigators (TLM, GP, CG and EP) in the research team independently conducted a full-text review. Differences in screening decisions were resolved by consultation with a third investigator (PAE). The inclusion criteria were: (1) the intervention programme adopted or implemented in the study was evidence based; (2) the study applied an implementation strategy/strategies to increase the uptake, implementation or sustainability of an EBP; (3) the study reported costs of the implementation strategy/strategies; and (4) the research was conducted in community, public health or healthcare settings. Studies that did not meet these criteria, conference abstracts (did not provide detailed information) and review articles were excluded. Of note, we defined EBPs as programmes that have been rigorously tested in controlled settings and proven effective.20 The type of programmes/practices may include surgical intervention, pharmaceutical treatment, behavioural intervention, health promotion programme, practice, guideline, policy, process or recommendation.

Charting the data

We created a data charting form using Microsoft Excel 2016. Each eligible article was independently extracted by two investigators. The following information from the selected studies, including appendices and online supplements, was entered into the data chart form: (1) general study information of the EBP, including study design, study setting, length of study, participants, primary outcome and effect size; (2) description of the D&I strategy, including items from the Proctor’s framework (name of D&I strategy, action target, action, implementation outcomes), discrete (involves one process or action) versus multicomponent/multifaceted (applies two or more discrete strategies) strategy,21 classification using ERIC,3 total costs of the D&I strategy, the reported cost of the discrete D&I strategy (if available); and (3) other cost categories (if available), including EBP costs, healthcare cost and participant cost.

There is currently no known standard reporting guideline available for cost assessment of D&I strategies. Therefore, we used a modified template based on questions derived from the Drummond and Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklists14 15 and Chapel and Wang’s review on cost data collection tools22 to extract the details of the costing methodology undertaken in each study. The template included the perspective of the costs analysis/economic evaluation, type of economic evaluation (if any), cost data collection design (prospective vs retrospective), costing approach by which the resource was identified (eg, activity-based costing, microcosting, ingredient method, bottom-up or top-down), cost category, cost data collection tool/mode, cost data collection instrument, cost inflation or currency conversion and cost reference year.

Summarising and reporting the results

To provide an overview of the available evidence, we presented (1) a descriptive numerical analysis of characteristics of included studies, and (2) a narrative summary, including total and unit costs of D&I strategies. Specifically, when total cost estimates were not provided, we used the unit cost results (eg, cost per participant served) combined with participant sample sizes to approximate total costs. Additionally, because the total strategy cost may vary based on different sample sizes and health conditions,23 we converted the overall/discrete strategy costs into unit costs (eg, per person) by dividing the overall/discrete strategy costs by the number of sample size of participants and/or action target to allow for comparison across studies. In addition, we described the cost range of identified discrete D&I strategies mapped to the terminology from the ERIC project.3 We then categorised the discrete D&I strategy into five outcome classifications: dissemination, implementation process, integration, capacity building and scale-up, defined by Leeman et al,24 to facilitate the cost comparison among strategies focused on these five areas. Finally, we summarised the reporting level of the costing approach in each study based on the items abstracted from the Drummond and CHEERS checklists14 15 and Chapel and Wang’s review on cost data collection tools22 to ascertain the degree to which the costing methodology was described. The cost information was converted to purchasing power parity dollars using country-specific exchange rates25 if expressed in local currency. We subsequently inflated all money to 2020 US dollars using the Consumer Price Index.26

Results

The initial search identified 15 723 records, 6445 were subject to title/abstract screening after duplicates were removed and 379 articles were identified as eligible for full-text review. Studies that did not report cost data on D&I strategies (55%) or studies that did not specify D&I strategies (31%) were the predominant reasons for exclusion in the title and abstract screening (figure 1). In total, 52 articles met the inclusion criteria and were included in the study. Figure 1 summarises the results of our search and selection processes.

Figure 1

Search flow diagram for selecting sources of evidence. D&I, dissemination and implementation; EBPs, evidence-based programmes.

Study characteristics

Table 1 summarises the study characteristics across health conditions, study settings and D&I outcomes. Among the 52 studies, 19% focused on mental health, 13% on infectious disease, 12% on cardiovascular disease risk factors (eg, smoking, physical inactivity, obesity and diabetes), 9% on sexually transmitted diseases, 8% on cancer, 6% on maternal health, 6% on low back pain and 19% on others. A large proportion of studies (42%) were observational (eg, cost analysis and programme evaluation) or randomised controlled trials (33%). Thirty-one (60%) examined the effect of D&I strategies on D&I outcomes, and 14 (27%) conducted an economic evaluation of D&I strategies. The healthcare system was the most common study setting, with 44% of studies being conducted in primary care clinics and 21% in hospitals or other healthcare facilities. Eighteen (35%) studies were conducted in the USA, 13 (25%) in Europe, 12 (23%) in Africa, 5 (10%) in Australia/New Zealand, 2 (4%) in Asia and the remaining 2 (4%) studies in Canada and Mexico. Thirty-five (67%) out of 52 studies applied multicomponent implementation strategies and 42 (81%) of the included studies separately reported the costs for discrete implementation strategies (including studies that used discrete D&I strategy). The majority (n=22, 42%) focused on strategies to increase EBP adoption, following 14 studies (27%) on reach/penetration.

Table 1

Characteristics of studies included in the review (n=52)

Characteristics	n (%)
Disease/health condition
Mental health	10 (19)
Other*	10 (19)
Infectious disease	7 (13)
HIV/syphilis/HPV	5 (9)
Cancer	4 (8)
Physical inactivity/obesity/diabetes	4 (8)
Maternal and newborn health	3 (6)
Low back pain	3 (6)
Substance abuse	2 (4)
Cardiovascular disease	2 (4)
Smoking/tobacco	2 (4)
Study design
Observational/cost analysis/evaluation	22 (42)
RCT/cluster RCT	17 (33)
Quasiexperimental/pre-post	7 (13)
HEI/SMART/pragmatic RCT	4 (8)
Qualitative design (focus group/key informant interview)	2 (4)
Examined the effect of implementation strategies
Yes	31 (60)
No (focused on the EBP effect)†	21 (40)
Study setting
Primary care clinics	23 (44)
Hospital/healthcare facility	11 (21)
Community	8 (15)
School	3 (6)
Local health department	2 (4)
Emergency room	2 (4)
Community pharmacy/drugstore	2 (4)
Early learning centre	1 (2)
Country
USA	18 (35)
Europe	13 (25)
Africa	12 (23)
Australia/New Zealand	5 (10)
Asia	2 (4)
Canada	1 (2)
Mexico	1 (2)
Applied multicomponent/discrete implementation strategies
Multicomponent	35 (67)
Discrete	17 (33)
Primary implementation outcomes‡
Adoption	22 (42)
Reach/penetration	14 (27)
Implementation fidelity	6 (12)
Competence	2 (4)
Compliance/adherence to protocol	3 (6)
Appropriateness/feasibility	3 (6)
Acceptability	2 (4)
Conducted economic evaluation of D&I strategy§
Yes	14 (27)
No	38 (73)
Separated costs reporting for discrete D&I strategies
Yes	42 (81)
No	10 (19)

*Included healthcare-associated infections, sepsis, sleep problem, brain injury, adolescent immunisation, antibiotics resistance, frailty and sarcopenia, Huntington’s disease and malnutrition.

†Those that did not examine the direct effect of an implementation strategy were those that included a description and cost information on implementation strategies, but did not examine variability in the strategy with specific D&I outcome variability.

‡Implementation outcomes were derived directly from the included studies (ie, competence and compliance/adherence to protocol) or labelled based on the information provided in the included studies by the research team according to the Proctor’s outcome framework74 if not defined in the study publication.

§Economic evaluation refered to studies that examined cost of strategies relative to change in D&I outcomes. Studies that did not do this most often simply provided cost information on the strategy itself.

D&I, dissemination and implementation; EBP, evidence-based programme; HEI, hybrid effectiveness implementation; HIV, human immunodeficiency virus; HPV, human papillomavirus; RCT, randomised clinical trial; SMART, Sequential Multiple Assignment Randomized Trial.

Characteristics of identified D&I strategies

Among the included studies, the number of D&I strategies per study ranged from 1 to 5 (median=2) and the number of outcome classifications per study ranged from 1 to 4 (median=2). No study reported strategies in all outcome classifications. Scale-up was the most studied outcome classification (85% of studies), followed by dissemination (37%), implementation processes (35%), integration (15%) and capacity building (10%; see online supplemental table S3).

Thirty studies (58%) focused on D&I strategies at the individual level, made up primarily of clinicians (n=18 studies; eg, physicians, nurses, pharmacy technicians, midwives, therapists, psychologists and medical assistants). In addition, at the organisational level, the most common action target was the healthcare system (n=16 studies) including primary care clinics, hospitals and emergency departments (online supplemental table S4).

According to the ERIC taxonomy, in total, 30 discrete implementation strategies were identified from the 52 included studies. The most common D&I strategy reported was Conduct ongoing training (50%), followed by Conduct education meetings (23%), Facilitation (21%), Develop educational materials (15%), Distribute educational materials (13%) and Audit and provide feedback (12%) (table 2).

Table 2

Cost ranges of discrete D&I strategies reported in the included studies

Name of discrete strategy3	Classification*24	Included studiesn (%)	Studies that reported costs (n)	Cost ranges	Cost ranges per action target	Cost ranges per EBP participant
Conduct ongoing training.	Scale-up	26 (50)	22	$389751; $52 758 (GI only), $72 054 (GI+MC)75; $12 06776; $10 70977; $17 84448; $4950 (intervention), $4055 (control)32; $273 ($21/session)29; $19939; $63 53454; $844978; $65 196 (Cenat), $21 386 (Hope), $12 144 (Cata)79; $42 08680; $603127; $17 08981; $33 494 (clinical team), $33 640 (technical team)82; $13 79933; $30 77483; $283084; $50 72745; $105 772 (high), $45 590 (medium), $49 803 (low)/school47; $330–$1864/session85; $37552	$11876; $10577; $15348; $129; $171 (intervention)/$135 (control)32; $439; $6958–$13 97354; $13078; $14 02980; $30227; $11481; $389 (clinical team), $801 (technical team)82; $518930; $234 (active implementation)33; $55083; $724745; $3022 (high), $1628 (medium), $1383 (low)47; $4252	$0.551; $110 (GI only), $147 (GI+MC)75; $0.0248; $1 (intervention), $1 (control)32; $139; $1078; $81 (Cenat), $29 (Hope), $31 (Cata)79; $5227; $41230
Conduct educational meetings.	Scale-up	12 (23)	10	$658 42855; $1.1–$2.9 million/year50; $775286; $22 366 (IAU), $27 873 (IAU+P4P)34; $44 165 (MSD), $176 659 (PCV), $422 894 (RV)69; $1004 (intervention), $987 (control)32; $393135; $35/facility (pilot)46; $67 801 (year 1), $30 134 (year 2)87; $47 38752	$54 86955; $18586; $1598 (IAU), 1858 (IAU+P4P)34; $1699 (MSD), $6795 (PCV), $16 265 (RV)69; $35 (intervention); $33 (control)32; $5535; $35/facility (pilot)46; $526552	$14655; $0.2–$0.450; $3586; $35 (IAU), $52 (IAU+P4P)34; $0.1 (intervention); $0.2 (control)32; $535
Facilitation.	Scale-up	11 (21)	8	$58 43936; $475 (webinar), $755 (in person)/session88; $117 88737; $3388–$5000/session42; $3733 (intervention), $2950 (control)32; $20 57489; $64 464 (high), $583 (medium)47; $266 736 (network A), $261 307 (network C)43	$84 (in person), $95 (webinar)/session person88; $126837; $167 (in person), $109 (webinar)/session person88; $129 (intervention), $98 (control)32; $73589; $1842 (high), $21 (medium)47; $66 684 (network A), $65 327 (network C)43	$11637; $0.5 (intervention), $0.5 (control)32; $9489
Develop educational materials.	Dissemination	8 (15)	5	$75951; $243131; $129527; $131084; $387552	$81031; $6527; $167 (clinical team), $1507 (technical team)82; $41352	$0.151; $4731; $1127
Distribute educational materials.	Dissemination	7 (13)	5	$515336; $11 33953; $609635; $188433; $9991 (year 1), $8793 (year 2), $6580 (year 3)85	$11 33953; $8235; $32 (active implementation)33	$553; $4135
Audit and provide feedback.	Implementation process	6 (12)	4	$3100 (high), $1417 (medium), $3189 (low)/school47; $26 093/A team90; $30 28091; $891 97051	$26 093/A team90; $757091; $89 (high), $51 (medium), $89 (low)47	$11551; $22990; $491
Use train the trainer strategies.	Scale-up	5 (10)	4	$19 66131; $20 07089; $51 89430; $100 (pilot), $184 (roll-out)/facility46	$655431; $71789; $100 (pilot), $184 (roll-out)46	$37831; $9189
Provide local technical assistance.	Scale-up	4 (8)	4	$2056 (clinical team), $12 953 (technical team)82; $8195 (active implementation)30; $11 29633; $62552	$24 (clinical team), $308 (technical team)82; $81930; $191 (active implementation)33; $6952	$6530
Use mass media.	Dissemination	4 (8)	3	$1623/1 month (Google), $1507/3 months (FB)92; $127435; $59 40483	$1635; $106183	$40 (Google), $36 (FB)/initiation of smoking cessation programme92; $6135
Develop a formal implementation blueprint.	Implementation process	3 (6)	3	$41 49954; $16 27080; $277484	$3362–$14 934/site54; $542380	N/A
Inform local opinion leader.	Implementation process	3 (6)	3	$145576; $100435; $166330	$1476; $1435; $16630	$5935; $1330
Identify and prepare champions.	Implementation process	3 (6)	2	$274 20055; $78235	$22 85055; $1235	$6155; $11235
Provide ongoing consultation.	Scale-up	3 (6)	3	$58 69249; $31 02545; $334 252 (year 1), $332 201 (year 2), $278 558 (year 3)85	$97849; $443245	$53449
Conduct educational outreach visit.	Scale-up	2 (6)	2	$51 74036; $67 801 (year 1), $30 134 (year 2)87	$1507 (year 1), $670 (year 2)87	N/A
Conduct local consensus discussion.	Implementation process	2 (4)	2	$93786; $209 (high)47	$2286; $6 (high)47	$486
Change record systems.	Integration	2 (4)	1	$111 66183	$199483	N/A
Develop and organise quality monitoring systems.	Integration	2 (4)	2	$243 41851; $974189	$34889	$3151; $4489
Intervene with patients/consumers to enhance uptake and adherence.	Implementation process	2 (4)	1	$257/SHG40	$257/SHG40	$1640
Organise clinician implementation team meetings.	Capacity building	2 (4)	2	$85 42293; $632630	$14 23793; $63330	$9793; $5030
Purposely re-examine the implementation.	Implementation process	2 (4)	1	$64 99251	N/A	$851
Remind clinicians.	Integration	2 (4)	1	$8794 (clinical team), $105 457 (technical team)82	$102 (clinical team), $2511 (technical team)82	N/A
Use advisory boards and workgroups.	Implementation process	2 (4)	1	$60 923 (clinical team), $62 873 (technical team)82	$708 (clinical team), $1497 (technical team)82	N/A
Build a coalition.	Capacity building	1 (2)	1	$326 12337	$350737	$32237
Create a learning collaborative.	Capacity building	1 (2)	1	$555 588/learning collaborative41	$12 58341	$511541
Make training dynamic.	Scale-up	1 (2)	1	$291 (active implementation)33	$5 (active implementation)33	N/A
Mandate change.	Integration	1 (2)	1	$66 647 (MSD), $266 587 (PCV), $412 362 (RV)69	$2563 (MSD), $10 253 (PCV), $15 860 (RV)69	N/A
Provide clinical supervision.	Integration	1 (2)	1	$60 441 (year 1), $38 120 (year 2)87	$1343 (year 1), $847 (year 2)87	N/A
Recruit, designate and train for leadership.	Capacity building	1 (2)	1	$113 18849	$188649	$102949
Use other payment scheme.	Scale-up	1 (2)	1	$348434	$23234	$634
Assess for readiness and identify barriers and facilitators.	Implementation process	1 (2)	0	N/A	N/A	N/A

All the costs were reported in 2020 US dollars.

*Discrete strategy was categorised into five distinct classes (dissemination, implementation process, integration, capacity building and scale-up) of strategies, proposed by Leeman et al,24 by identifying actors and action targets.

D&I, dissemination and implementation; EBP, evidence-based programme; GI, guideline implementation; IAU, implementation as usual; MC, motivational counselling; MSD, measles; N/A, not available; PCV, pneumococcal conjugate vaccine; P4P, pay for performance; RV, rotavirus vaccine; SHG, self-help group.

Cost estimates of D&I strategies

The cost ranges of the D&I strategies used in the studies varied broadly. Expressed in terms of year 2020 US dollars, the estimates of the total costs of the D&I strategies ranged from $728827 to $3.0 million28 for multicomponent strategies, excluding studies with incomplete cost information for all identified strategies29–31 and studies that were designed to compare the effect of different strategies.32–38 In contrast, the range was $21239–$4.4 million40 for discrete D&I strategies (online supplemental table S4). No association was observed between the number of strategies used and the total cost, given the broad range of the D&I strategies and associated costs.

Further converting to a unit cost basis, the mean costs of discrete D&I strategies per action target ranged from $439 to $12 583 per provider trained41 at the individual level; whereas the organisational level range was $10942–$66 68443 per setting (online supplemental table S4). For multicomponent strategies, the range at the individual level was $2244–$11 67945 per provider and at the organisational level between $13546 and $116 911 per setting.47 Expressed as the unit costs per EBP participant, in studies that applied a discrete D&I strategy, the mean costs were estimated between $0.0248 and $511541 per participant; whereas in studies that applied multicomponent strategies the cost was estimated between $0.328 and $156349 per participant.

For specific D&I strategies there was also a substantial variation in costs (table 2). The total cost varied between $19939 and $105 77247 for Conducting ongoing training, from $98732 to $1.1–$2.9 million per year50 for Conduct educational meetings and from $58347 to $266 73643 for Facilitation. Similarly, there were wide ranges for Develop educational materials ($75951–$387552), Distribute educational materials ($188433–$11 33953) and Audit and provide feedback ($141747–$891 97051). This substantial variation reflects the considerable difference in content even within the D&I strategy types and the scope of the study. For example, Develop educational materials could refer to simple development of participant facing didactic materials51 or a team of four members to researching, discussing and coordinating the development of shared medical appointment-related materials.52 Converting to per-action target/EBP participant basis (table 2), the cost of Conduct ongoing training was estimated between $129 and $13 97354 per provider and $0.0248 and $41230 per EBP participant. Moreover, for Conduct educational meetings, costs were between $3332 and $54 86955 per action target and $0.2–$0.450 and $14655 per EBP participant.

Of note, all included studies reported the total costs of the D&I strategies; however, some studies did not individually report costs for discrete strategies used.28 38 44 56–58 This resulted in a smaller number of studies reporting costs for a discrete D&I strategy than the number of studies that applied that discrete D&I strategy (table 2).

Reporting level of costing approach

A summary of the costing approach used for assessments of the costs of the D&I strategies is presented in table 3. Among the 52 included studies, 60% did not indicate the cost data collection design, 71% did not report analytical perspective, 62% did not report the method in which the resource used for conducting the D&I strategies was identified, 54% did not report the costing approach, 40% did not report the cost collection tools used in the studies, 58% did not indicate where or how the cost data were obtained, 58% did not report the cost reference/inflation year, 58% did not adjust cost data (ie, inflation) and 65% did not separately report unit costs and the resource utilisation quantity (table 3). Overall, there was insufficient reporting across studies suggesting the costing methodologies are of moderate or low quality based on the items abstracted from the Drummond and CHEERS checklists14 15 and Chapel and Wang’s review on cost data collection tools.22

Table 3

Summary of costing approach of the included studies

Item	n (%)
Cost data collection design
Prospectively	9 (17)
Retrospectively	11 (21)
Combined	1 (2)
Not reported	31 (60)
Perspective specified
Healthcare system	8 (15)
Society	4 (8)
Small-scale stakeholders*	3 (6)
Not reported	37 (71)
Method by which resource was identified†
Accounting/financing department	4 (8)
Standardised reporting template	8 (15)
Administrative databases	8 (15)
Direct observation	1 (2)
Not reported	32 (62)
Costing method†
Activity-based costing	3 (6)
Microcosting/ingredient approach	7 (13)
Budget approach (gross costing/average costs)	8 (15)
Cost analysis	6 (12)
Bottom-up approach	2 (4)
Direct/detailed cost calculation	2 (4)
Not reported	28 (54)
IS cost collection instrument†
Activity log	13 (25)
Standardised template/questionnaires	19 (37)
Direct observation	1 (2)
On-site database/records	9 (17)
Time-motion survey/observation	2 (4)
Not reported	21 (40)
IS cost data collection platform/tools†
Computer based (eg, Excel, Microsoft Access)	13 (25)
Paper based (receipt, attendance record)	1 (2)
Telephone	2 (4)
In person	1 (2)
Email	1 (2)
Website based	2 (4)
Electronic database (eg, accounting system, EHR)	4 (8)
Not reported	30 (58)
Cost inflation	22 (42)
Reference year	22 (42)
Using qualitative data to collect cost information	12 (23)
Separate reporting of quantity and unit cost data	18 (35)

*Included implementation organisation/staff or local government/community.

†Some studies reported more than one approach.

EHR, electronic health record; IS, Implementation strategy.

Discussion

To facilitate and promote the application of comparative economic evaluation in the field of D&I research, we conducted a scoping review of the evidence concerning the cost of D&I strategies in community, public health and health service research published between January 2008 and December 2019. We included studies conducting implementation activities to facilitate the reach, adoption, implementation or sustainability of EBPs. We mapped the D&I strategies employed using published/standardised taxonomies. Not surprisingly, among the 52 studies included, we found considerable variation among the description and definition of D&I strategies, which are, by nature, heterogeneous and complex. As most studies did not use the Proctor’s reporting guideline for D&I strategies which we used to assist in the abstraction process and as information provided about D&I strategies was limited, it was sometimes challenging to map a strategy identified from the included studies.2

Still, we found Proctor and colleagues’2 recommendations for specifying and reporting D&I strategies to improve generalisability and comparison across studies helpful. Due to the application of D&I strategies as one of the inclusion criteria, the Proctor’s specification of the D&I strategies was amenable as a template for identifying and monitoring costs.24 59–61 As suggested by Cidav et al,61 the information provided in Proctor’s framework can be used to conduct the cost estimates of D&I strategies using time-driving activity-based costing approach—a modified version of activity-based costing strategy which identifies activities (responsible for producing implementation outcomes) associated with implementation and assigns the cost for each activity.62 In this regard, our review suggests that standardised reporting of D&I strategies using Proctor’s reporting framework and published list of taxonomies of D&I strategies may be helpful for advancing economic evaluation in D&I research. However, more effort is needed to promote consistent use of D&I strategy terms and definitions to promote comparative analysis and development of a cumulative knowledge base about the effectiveness of strategies with similar content but different taxonomies.24 60 63 64

In this review, more than half the included studies did not provide sufficient information on their costing approach (ie, they failed to report data on most of the items listed in table 3). Given the increasing number of cost analyses and/or economic evaluations within implementation research, there is a critical need for an appropriate guidance on reporting, including identification of cost categories (eg, labour and non-labour costs), measurement of costs (eg, quantities and unit costs) and valuation of costs (eg, currency and inflation).5 This guidance should address cost analysis as well as economic evaluation as a large proportion of items related to cost-effectiveness analysis (eg, choice of health outcomes, measurement of effectiveness or measurement and valuation of preference-based outcomes) on the CHEERS checklist—the most commonly used reporting checklist of economic evaluation—may not be applicable in the reporting of costing assessment.65 A more straightforward guideline (eg, Drummond’s checklist14) is suggested over the CHEERS checklist if an economic evaluation is to be carried out.6

Furthermore, we suggest that costs be categorised into two main groups, labour and non-labour costs, to facilitate comparative analyses across studies. Labour costs may include personnel expenses, estimated based on the amount of time that each individual spent on specific activities (ie, micro-costing66 67 or activity-based costing strategy62); whereas non-labour costs may include equipment/information technology, space (or overhead), supplies and travel and can be derived from actual amounts spent and tracked by entry of receipts and payment invoices in an administrative database.68 Based on our findings and experience in abstracting information from these studies, we developed a checklist (table 4) to guide the conduct and reporting of cost analysis of D&I strategies, based on the CHEERS checklist,15 Chapel and Wang’s review on cost data collection tools22 and Proctor’s specification of D&I strategy,2 to improve the quality and generalisability of cost assessments in the field of D&I science. The checklist is intended to serve as a complementary tool for the economic evaluation of implementation studies which may include costs of direct implementation (costs incurred from implementing EBPs), direct service (costs associated with healthcare or other services) and indirect implementation (opportunity costs as a result of implementing EBPs).65 Of note, although some of the checklist items (eg, objectives, intended audience, EBP, strategy specification or implementation outcome) were already described in the Standards for Reporting Implementation Studies guideline, we still included these items with the intention to provide a comprehensive and comparable cost assessment of D&I strategies. For researchers or community implementers who are interested in applying this checklist to conduct cost analyses of D&I strategies, it may be challenging to distinguish the concepts between the EBPs (especially behavioural intervention programmes) and implementation strategies (ie, implementation activities), their corresponding intended audience (patients/individuals at risk vs providers/community facilitators) and associated outcomes (health outcomes vs implementation outcomes).

Table 4

Checklist to guide the conduct and reporting of cost analysis of implementation strategies

Item	Description
Background and objectives	Present the study question and its relevance for health policy or practice decisions for stakeholders.
Intended audience	Describe characteristics of the population that EBPs intended to target.
Evidence-based intervention	Describe the evidence-based programme that is being adopted or implemented.
D&I strategy	Specify each D&I strategy used to facilitate the adoption, implementation or sustainability of evidence-based programme described in the study.
Name	Label strategy according to the ERIC project.
Actor	Specify individuals associated with each implementation activity.
Action	Itemise implementation activities.
Action target	Specify the recipients of the implementation intervention.
Temporality	Specify date and time of each implementation activity.
Dose	Specify the duration of each implementation activity.
Implementation outcome	Describe the outcomes affected by the implementation strategies (eg, acceptability, adoption, appropriateness, feasibility, fidelity, penetration or sustainability).
Study perspective	Describe the perspective of the study to determine which costs and benefits are included (eg, healthcare system, payer, society, patient or small-scale stakeholders).94
Costing study design
Study design	State whether the cost analysis was planned/conducted retrospectively or prospectively.
Costing approach	Describe approaches used to estimate resource use with the implementation strategies (eg, activity-based costing/microcosting, bottom-up, top-down, ingredient approach or TDABC).
Time horizon	State the time horizon over which costs are being evaluated.
Identification of costs
Resource identification	Specify methods by which resources were identified (eg, process map or pathway analysis).
Cost category	Describe cost categories by (1) labour (ie, personnel), (2) equipment/information technology, (3) facility/space/overhead, (4) supplies, (5) travel, and (6) others.
Measurement of costs
Cost data collection tool/mode	Describe what tools/platforms were used to collect/track the data for cost estimates (eg, computer based, telephone, paper based, email or web based).
Cost data collection instrument	Describe what instruments were used to collect the data (eg, activity log, time-motion survey, standardised questionnaire or direct observation).
Quantity	Report the quantity of each activity/item reported in each cost category.
Valuation of costs
Inflation	Describe any adjustments for inflation or currency conversions.
Reference year	State the year the cost data were collected.
Unit costs	Describe the method to value unit costs.

D&I, dissemination and implementation; EBP, evidence-based programme; ERIC, Expert Recommendations for Implementing Change; TDABC, time-driven activity-based costing.

This review also highlights the importance of estimating unit costs, which can be derived from total costs if unit costs were not provided in the original studies. In our case, we described the unit costs as the mean cost per action target of the D&I strategy according to the Proctor’s reporting framework2 as well as the mean costs per EBP participant, aiming to increase the comparability across included studies. This will be helpful to provide context to the total strategy cost which may not be informative and usually differs in size due to the scale of the study, research scope and disease targeted.23 As demonstrated in the present study, the total costs of D&I strategies (eg, ~$0.5–$1.7 million) were genuinely higher in the studies targeting the adoption and implementation of regular immunisation programmes.28 69 The sample size of the intended audience in these studies ranged from 0.5 to 1.7 million. However, the unit cost of per-vaccinated individual was between ~$0.3 and $14.28

Not all included studies provided details about their costing approach. Of those that did, 11 out of 21 included studies reported that cost data were estimated retrospectively (ie, ex-post). While retrospective cost capture is considered a practical and low-burden method,70 it is also of value to plan cost analyses early during implementation to reduce the potential of recall bias and increase the likelihood of disentangling the costs attributed to the implementation activities versus costs associated with the EBPs.

Challenges and limitations

This scoping review has several strengths. These include offering a synthesis on the costs of D&I strategies that have been applied to facilitate the uptake of EBPs and mapping and defining the D&I strategies using existing taxonomies. The challenges faced during production of this review and limitations of this work must also be recognised. First, we only included studies conducting implementation activities (ie, D&I strategies) to facilitate the uptake, implementation or sustainability of EBPs and have reported costs associated with implementation activities. Some studies were excluded due to not having an EBP—as a result, studies that focused on dissemination or implementation of programmes that are not evidence based were not included. Second, the 73 strategies identified in the ERIC compilation3 and included in our search strategy were mainly developed in clinical contexts. Consequently, some implementation activities in community settings might not be captured or cannot be mapped appropriately. As such, work related to strategies based on community-engaged research methodologies that focus on adoption, implementation and sustainability in community contexts is likely under-represented. Further, despite the specific definitions included in the ERIC compilation, a large variability of activities that were employed in the included studies could conceivably fit within a single strategy. An example of this is Develop educational materials; as a specific D&I strategy, this could refer to simple development of participant facing didactic materials51 or broad interactive materials that reflect very different activities and costs relative to D&I outcomes.71 Future research should further investigate this categorisation and clarify the common implementation procedures and resources used to make the taxonomies more useful in comparing work performed in diverse contexts. Because of the wide variation of the degree to which D&I strategies were reported, it was challenging to accurately identify and map the D&I strategies applied in the included studies and sufficiently differentiate the activities of implementation from the activities of an EBP. We may have underestimated or overestimated the total number of D&I strategies included in each study. Similarly, it was also challenging to categorise strategies into different outcome classifications (ie, dissemination, implementation, integration, capacity building and scale-up). Without sufficient context, some strategies (eg, conduct ongoing training) may be applicable across the implementation stages (eg, exploration, preparation, implementation and sustainability). Finally, although we were not able to provide a search update due to the lack of resources, we would expect the number of the eligible studies published after our initial search to be small given the infrequency and scarcity of cost data reporting in the field of implementation research.

Conclusion

To facilitate the application of comparative economic evaluation of D&I strategies, we carried out a scoping review to explore the current state of cost information and assessment of D&I strategies. These results inform the current cost reporting gap and call for the development of a standardised cost reporting template.72 73 We also proposed a checklist that may be of use for future D&I researchers. The checklist combines elements of Proctor’s report framework of D&I strategies, the CHEERS checklist and Chapel and Wang’s review on cost data collection tools.