Toolkits for dissemination and implementation research: Preliminary development.

The Dissemination and Implementation Research Core, a research methods core from the Clinical and Translation Science Award at Washington University in St. Louis Institute of Clinical and Translational Sciences, developed toolkits about dissemination and implementation (D&I) concepts (e.g., D&I outcomes, strategies). This paper reports on the development of the toolkits. These toolkits respond to 3 identified needs for capacity building in D&I research: resources for investigators new to the D&I field, consolidation of tools, and limitations in local contexts.

Introduction

The field of dissemination and implementation (D&I), which aims to accelerate research translation, is growing and emerging as a national priority [1]. D&I is a multidisciplinary science, and as such it presents the advantages and challenges of researchers of different disciplines working together [2, 3]. The Clinical and Translational Science Award (CTSA) [4], through the National Center for Advancing Translational Science, provides an infrastructure, including cores, to support collaboration between investigators from multiple disciplines leading to innovative translational science [5]. A challenge for the cores is to provide consultation to investigators from different disciplines in a timely manner and with high quality.

Several resources for D&I research are scattered across the web or in scholarly products, such as measurement compilations [6, 7], guidance on frameworks [8], sources of information about grants and trainings [8, 9] or books [10]. The quantity and variability of resources is a testament to the importance of D&I science for different disciplines [11]. While these efforts are valuable, researchers new to the field may not know these resources exist or where to find them [12]. To address the unmet educational gap of providing D&I resources for beginners, to provide tools for investigators from different disciplines interested in D&I, and to equip investigators with key D&I language and concepts, our team from the Dissemination and Implementation Research Core (DIRC), a research methods core from the CTSA at Washington University in St. Louis Institute of Clinical and Translational Sciences (ICTS), developed a set of toolkits about D&I concepts.

The ICTS supports our methods core to provide technical assistance to affiliated investigators preparing D&I research grants. We do so primarily through face-to-face meetings. As the effort to support to our core is relatively small and demand for DIRC services has steadily grown, we recognized the need for efficiency. Toolkits offered efficiency to the Core and opportunity for users to review materials at their own pace and schedule.

A “toolkit” can be defined as an action-oriented compilation of information, resources, or tools, to guide users in organizing information about a specific topic [13]. Our team uses toolkits to supplement, not replace, individual consultation. Each toolkit was led by a research assistant from the DIRC team, under supervision of the DIRC Coordinator and Principal Investigator. This article describes the first phase of development of the toolkits aimed to facilitate the knowledge of D&I science research. The topics and focus of the 8 toolkits are shown in Table 1.

Table 1

Topics and goals of the Dissemination and Implementation Research Core dissemination and implementation (D&I) toolkits, available at https://sites.wustl.edu/wudandi/di-toolkits/

Toolkit topic	Goals	Citation
Introduction to D&I	Assist beginner D&I researchers in situating the field of D&I science in the continuum of research Provide key D&I science terms for use in planning and carrying out D&I research. Identify additional introductory D&I science resources beyond this toolkit	Morshed A, Tabak R, Taranhike I, Baumann A, and Proctor E. Intro to D&I [Internet], 2016 October. St. Louis, MO: Washington University. Eight toolkits related to Dissemination and Implementation
D&I Aims	Assist researchers in gaining familiarity with formulation of D&I research aims by research phase and design type Enable researchers to locate examples of funded D&I research online	Morshed A, Baumann A, and Proctor E. D&I aims toolkit [Internet], 2016 November. St. Louis, MO: Washington University. Eight toolkits related to Dissemination and Implementation
D&I Designs	Provide a brief introduction and review of research designs for experimental, quasi-experimental, and observational D&I studies Assist researchers in identifying the most appropriate design(s) to addresses the contextual nuances within their D&I study	Lewis E, Baumann A, Gerke D, Tabak R, Ramsey A, Small S, and Proctor E. D&I research designs [Internet], 2017 July. St. Louis, MO: Washington University. Eight toolkits related to Dissemination and Implementation
Implementation outcomes	Help further one’s understanding of implementation outcomes Help identify those outcomes and corresponding measures that are most relevant to one’s research	Gerke D, Lewis E, Prusaczyk B, Hooley C, Baumann A, and Proctor E. Implementation outcomes [Internet], 2017 July. St. Louis, MO: Washington University. Eight toolkits related to Dissemination and Implementation
Implementation strategies	Provide definitions of strategies, resources on how to map strategies to frameworks and how to report strategies Provides examples of studies that have used strategies	Prusaczyk B, Baumann A, and Proctor E. Implementation strategies [Internet], 2016 October. St. Louis, MO: Washington University. Eight toolkits related to Dissemination and Implementation
Implementation organizational measures	Improve one’s understanding of organizational constructs relevant to D&I research Help in identification of the corresponding organizational measures that are most relevant to one’s research	Ramsey A, Lawrence K, Prusaczyk B, Baumann A, Kryzer E, and Proctor E. Organizational measures. [Internet], 2016 October. St. Louis, MO: Washington University. Eight toolkits related to Dissemination and Implementation
Assessing barriers and facilitators	Help further one’s understanding of factors that affect successful implementation Help identify and measure implementation barriers and facilitators that are most relevant to one’s research and that may improve the impact of an implementation strategy	Lawrence K, Ramsey A, Baumann A, and Proctor E. Dissemination & implementation barriers & facilitators. [Internet], 2016 October. St. Louis, MO: Washington University. Eight toolkits related to Dissemination and Implementation
Guideline	Provide resources for guideline development, guideline testing Gives examples of studies on guidelines effectiveness, guideline appraisal and modifications, and guidelines implementation	Prusaczyk B, Kryzer E, Baumann A, and Proctor E. Guideline implementation [Internet], 2016 October. St. Louis, MO: Washington University. Eight toolkits related to Dissemination and Implementation

Materials and Methods

Participants

Participants in this study included members of our DIRC team, D&I researchers at Washington University in St. Louis, and national experts in the D&I field. The study was reviewed and exempted by the Washington University in St. Louis Human Research Protections Office (Institutional Review Board no. 201508175). The D&I experts that provided feedback on the toolkits received a $50 gift certificate from Amazon.

Theoretical Framework

Our team relied on educational design research methodology to develop the toolkits. The function of educational design research is to develop solutions to complex educational problems through “systematic study of designing, developing, and evaluating educational interventions” [14]. This systematic and iterative methodology conducts assessments of adaptations to the tools and interventions, providing details regarding the process that allows for replication by others [14-16]. We selected the educational design framework, rather than other models that could provide guidance to this process (e.g., user-centered design), because it allowed us to create tools to support our consultation process—which often involves an education component of defining D&I and how D&I can support our customers’ work—while also advancing the knowledge on tool development [14-16].

Fig. 1 illustrates this study’s evaluation process, adapted from McKenney’s [17] model and based on educational design research methodology [14-17]. The process is cyclical and consists of 3 phases. Each cycle represents a small cycle of research, with reflection and documentation taking place during all cycles [13]. Each phase has a separate focus and emphasizes different evaluation criteria [13]. The first phase (preliminary research and initial development) consists of analysis of the educational problem and its context and leads to the development of toolkit prototypes and plans for their implementation. The second phase (development through small-scale evaluation) iteratively evaluates the toolkits with emphasis on content validity and consistency. Finally, in the summative evaluation (not shown in the table), the toolkits are evaluated based on their use as intended. Because the D&I field is constantly evolving, we consider that Phases I and II can be in constant iteration, where after the initial development and preliminary research, the toolkits would be updated after a certain period of time has passed. This study describes the details from Phase I and Phase II of the development of eight D&I toolkits.

Fig. 1

Cyclical evaluation process of the Dissemination and Implementation Research Core dissemination and implementation toolkits (Adapted from McKenney [17]).

Toolkit Reach

We used Google Analytics to examine the number of visitors (new and returning) to the page that hosts the toolkits, and their location.

Results

Phase I: Preliminary Research and Initial Development

Preliminary Research

Table 2 details an overview of the steps taken for the development of the DIRC toolkits. To complete the first phase—analysis of the educational problem and its context—we characterized the likely users, their research experience, current D&I expertise, training preferences, and organizational needs. Discussions among the DIRC leadership led to the decision that the first target audience for toolkits would be the DIRC customers. They range widely in substantive areas (e.g., psychologists, social workers, nurses, cardiologists, pediatricians), career (e.g., assistant, associate and full professors) as well as D&I expertise. We next reviewed existing online D&I materials and tools to identify gaps. Although we found several resources, most were scattered across the web and targeted to researchers with mid-level to high level of knowledge of D&I. Our team concluded that missing in the field was a compilation of resources for researchers with beginner level knowledge in D&I.

Table 2

Overview of the development of the DIRC toolkits

Research phase	Outcomes	Measures/methods	Sources of data
1. Preliminary research and initial development: Context and problem analysis and initial development of toolkits	Description of user and organizational needs	ICTS survey	WU researchers
		Landscape of D&I materials	D&I materials available online
	Description of user audience and their characteristics	Brainstorming	DIRC team
		Meetings with stakeholders	ICTS leadership
	Description of toolkit implementation context	Meetings with ICTS leadership	ICTS leadership
		Brainstorming with WU researchers	WUNDIR members
	Development of prototypes of toolkits and with focus on initial content validity and consistency	Review of literature	Scientific literature
		Internal review	DIRC team
2. Continued development through small-scale evaluation: Iterative piloting and revision of toolkits	Assessment of content validity Assessment of expected usability	Expert review: self-administered checklist	D&I experts ICTS customers
3. Review and update: Update of contents and format of toolkits	Literature review of topics Assessment of what needs to be updated	Review of new literature by research assistants and leaders in the field from the team for internal review	Databases Books Key Web sites with D&I resources

D&I, dissemination and implementation; DIRC, Dissemination and Implementation Research Core; ICTS, Institute of Clinical Translation Science; WU, Washington University in St. Louis; WUNDIR, the Washington University Network of Dissemination and Implementation Researchers.

Phase II: Development of Toolkits

Phase II of the evaluation consisted of several evaluation cycles in which we assessed and revised the toolkits for content validity and consistency [13] and expected practicality and effectiveness. The toolkits underwent 3 main evaluations. To ensure initial content validity, we conducted a systematic review of scientific literature relevant to each toolkit. Next, to ensure initial consistency, we revised each toolkit using a checklist of toolkit characteristics and design specifications, adapted from AHRQ’s Toolkit Content and Usability Checklists [12]. As a result, we added a description of objectives and short overview of the toolkit to the first page of each toolkit. Finally, each toolkit was evaluated by 2 national experts in the D&I field, selected from our network of collaborators. Experts were contacted by email and asked to review the content of the toolkit. The toolkit was attached as a PDF and survey responses were collected in Qualtrics. The survey included quantitative items in Likert scale (e.g., “To what extent do you think this toolkit adequately captures the necessary elements to achieve its objectives?”) as well as open-ended questions (e.g., “Which components of the toolkit would benefit from further development in order to enhance the toolkit’s usefulness for D&I beginners?”).

Table 3 shows the results of the quantitative questions. With regard to inclusion of key information, the evaluations varied considerably. The introduction to D&I toolkit was considered complete, while the other toolkits needed to be rearranged, updated, or expanded. However, overall experts thought the toolkits would be very useful for someone in the D&I field, that the toolkits had reasonable goals, and that they would recommend the toolkit to others.

Table 3

Results of the evaluation of the Dissemination and Implementation Research Core toolkits

Toolkit	Aims	Intro	Guidelines	Strategies	Outcomes	Designs	Organization	Facilitators and barriers	Average (n=2)
After reading the toolkit, I understand its content	88	100	90.5	74	72.5	90	61	56	79 (17.3)
This toolkit is easy to understand	84	100	98	70.5	79	85	71	35	75 (22.0)
A person may need help from a D&I expert to use this toolkit	80	n/a	59	52	81.5	55	80	100	78.7 (15.6)
This toolkit provide all the resources/information that I need to know for this particular D&I topic	76	100	39.5	74	55	75	21	18	58 (32.3)
I think this toolkit would be very useful for someone in the D&I field	n/a	100	60	83.5	67	100	50	100	87.5 (21.3)
This toolkit is reasonable to use for the goals described	n/a	100	46	76	68	95	50	100	86.2 (21.0)
I would recommend this toolkit to others	n/a	100	80	79	72	95	51	100	86.5 (20.6)

Scale of 10–100, with 10 being “completely disagree” and 100 being “completely agree.” D&I, dissemination and implementation.

Experts were asked “Overall, do you think that this toolkit is a good resource for (check all that apply): D&I beginners, D&I intermediate, D&I experts”. Most of the toolkits were considered a good fit for all 3 levels, with the exception of the Strategies (intermediate) and Designs toolkits (expert and intermediate). Evaluators suggested adding more examples in order to make the toolkits more useful for beginners in the field. For example:

“For intermediates and experts really, I think it might be helpful to include more examples from the literature of each type of design. For myself, I really like to see examples of these designs in action, and with the number of protocol papers being published, it seems that there are a lot to choose from. Perhaps a bibliography link for each type with multiple examples?”

“This is a useful resource. It would be strengthened by more information at the beginning about when in the proposal process to think about these strategies, and more on how to select strategies that align with certain interventions or other needs. If you wanted to really go to town, you could give more examples in the appendix of how specific strategies have been used, when there are more available. It would also be helpful to see a reference for what research has been done on the effectiveness of each strategy, or note when there is no such evidence.”

One evaluator stated, “I like the idea and would use this in my teaching and mentoring.”

Toolkit Editing

The team gathered the feedback from the expert evaluators and edited the toolkits accordingly. Edits included adding examples, rearranging the flow of information, editing the decision trees, updating, and adding additional resources. Some feedback was conflicting; for example, the Strategies toolkit was considered to be too advanced for a D&I beginner investigator by one expert, and perfect for a beginner by another expert. In this case, the team decided not to make major changes pending further use and feedback.

Toolkit Reach

The toolkits are currently available online at https://sites.wustl.edu/wudandi/, a page that hosts information about the D&I work at Washington University in St. Louis in general. In 2017, the page had more than 1000 views, with about 290 new users and more than 300 returning users (data from Google Analytics). The majority of the visitors were from the United States but a number of them are from Australia (n=10) and other countries such as United Kingdom and India (n=4), Canada, Guatemala, and Ireland (n=3) showing a potential international reach of our materials. As our local ICTS investigators were the primary target, about 40% of the visitors were from Missouri.

Discussion

Although the rapid growth of the D&I science field has triggered high demand for capacity building [18], demand for research training far outstrips the supply of slots in current programs [19]. This project provides insight into how toolkits with key D&I concepts can be developed to support investigators moving into the field. These toolkits respond to 3 identified needs for capacity building in D&I research: resources for early-stage investigators, consolidation of tools, and limitations in local contexts [20]. While the D&I literature has expanded rapidly, many resources are geared to those with intermediate or advanced knowledge. Moreover, most are scattered across the literature or internet. While more introductory resources have been developed, there are calls to consolidate resources in order to lower barriers to entry for those new to the field [20].

Chambers et al. [20] note that a challenge in meeting the needs of D&I research trainees is limited support and local capacity. Resources that are available virtually and in real time may help address this challenge. Even in an environment such as ours that is relatively rich in D&I resources, including an ICTS core dedicated to providing technical assistance [21], our DIRC team is often stretched to meet demand for service. For example, the number of customers served has grown from 11 in 2009 to 65 in 2017. The toolkits have expanded our team’s capacity by providing resources investigators can use before, in conjunction with, or following one-on-one consultation and have proven useful to our Center for D&I Research’s annual D&I grant bootcamp [21]. While they have helped us support our customers, the rapid pace of the D&I field requires the toolkits be periodically updated. Our current procedure includes review and update of all toolkits every 6 months.

This is the first step in the development of the toolkits and is limited by its preliminary nature and by the involvement of a relatively small number (n=2) of stakeholders. The toolkits were developed in response to needs directly experienced in one CTSA. However, the project relied on national experts in D&I for toolkit development and followed a systematic approach to iterative product development and refinement.

Future Directions

The toolkits are currently being updated by our team and will be evaluated in Phase III [13]. Our goal is to involve investigators affiliated with CTSAs nationwide to engage a broader, more diverse audience of users and to test the usability and D&I knowledge gained among CTSA customers. We will also engage D&I experts from other CTSAs to develop new toolkits, including advanced and specialized topics. Possible topics for new toolkits are adaptation in D&I science, deimplementation, and D&I planning early in intervention development. We will also work with other CTSAs to make some toolkits interactive, such as the ones developed by other D&I Centers [22, 23]. These efforts will be evaluated and reported so others can replicate and add to the process.