Aligning Theory and Design: The Development of an Online Learning Intervention to Teach Evidence-based Practice for Maximal Reach.

BACKGROUND: Online educational interventions to teach evidence-based practice (EBP) are a promising mechanism for overcoming some of the barriers to incorporating research into practice. However, attention must be paid to aligning strategies with adult learning theories to achieve optimal outcomes.
METHODS: We describe the development of a series of short self-study modules, each covering a small set of learning objectives. Our approach, informed by design-based research (DBR), involved 6 phases: analysis, design, design evaluation, redesign, development/implementation, and evaluation. Participants were faculty and students in 3 health programs at a complementary and integrative educational institution.
RESULTS: We chose a reusable learning object approach that allowed us to apply 4 main learning theories: events of instruction, cognitive load, dual processing, and ARCS (attention, relevance, confidence, satisfaction). A formative design evaluation suggested that the identified theories and instructional approaches were likely to facilitate learning and motivation. Summative evaluation was based on a student survey (N=116) that addressed how these theories supported learning. Results suggest that, overall, the selected theories helped students learn.
CONCLUSION: The DBR approach allowed us to evaluate the specific intervention and theories for general applicability. This process also helped us define and document the intervention at a level of detail that covers almost all the proposed Guideline for Reporting Evidence-based practice Educational intervention and Teaching (GREET) items. This thorough description will facilitate the interpretation of future research and implementation of the intervention. Our approach can also serve as a model for others considering online EBP intervention development.

BACKGROUND

Evidence-based practice (EBP) has become a recognized standard for improving healthcare quality worldwide.[1,2] Providing a framework for healthcare decision making, EBP integrates the best available research with patient values and preferences and the experience and judgment of the health professional.[3,4] However, while there has been a growing awareness and appreciation of EBP across all health professions, there remains a disappointing lack of implementation.[5]

Educational interventions promoting foundational EBP skill development are an important and required step for the complex and dynamic process of translating research into practice.[6,7] Systematic reviews have examined the effectiveness of EBP educational interventions to undergraduate and postgraduate medical, nursing, and allied health students, as well as practicing professionals.[6,8-11] There is general agreement that a variety of strategies for teaching EBP can be effective, at least in improving knowledge and skills. However, actual use and uptake of research remains limited.[5] Several common barriers to EBP implementation have been identified across health professions, including lack of training, skills, resources, and time[5,12,13] and are important to consider when designing EBP educational interventions.

To date, no one method for teaching EBP has emerged as the gold standard.[6,8,9,14] However, one consistent finding supported by a substantial body of evidence is that “elearning” is as effective as in-person instruction.[6,8,15,16] Online interventions are of particular interest because they can overcome some of the barriers to EBP. By providing consistent delivery independent of instructor skill, online, self-study EBP educational interventions can overcome issues related to lack of resources and training. They can also provide sustainable solutions for teaching large numbers of learners over time and geographic location. Additionally, online approaches offer flexibility in terms of availability and pacing, which can address the commonly cited barrier of practitioners' lack of time.

One limitation consistently identified in the growing body of research is the inadequate description of EBP educational interventions.[6,8-10,17-20] This is particularly true for studies of online learning, which often limit the description of the intervention to “elearning” and provide insufficient details regarding the applied instructional techniques, delivery methods, and learning theories underlying the intervention design.[14] These limitations may lead to a misalignment of learner needs and optimal learning strategies, which could explain disappointing educational outcomes. Gaps in intervention reporting also limit the ability of others to apply effective educational approaches and reproduce them in further studies. Recognizing the need to standardize the reporting of EBP educational interventions, the Guideline for Reporting Evidence-based practice Educational intervention and Teaching (GREET) guidelines have recently emerged to facilitate the design and interpretation of EBP educational research.[6,9,17,18]

The purpose of this article is to describe the development of an EBP online learning intervention, providing sufficient context and detail regarding specific educational strategies to inform future interpretation and application.[18] The online intervention was part of a larger multifaceted EBP education implementation project for complementary and integrative health (CIH) disciplines funded by the National Center for Complementary and Integrative Health (R25AT003582).

METHODS

The online intervention consists of a series of short self-study modules, each of which covers a small set of learning objectives about foundational research knowledge. This aspect of the larger EBP research education implementation project took place from 2007 to 2012.[21] The initial implementation site was Northwestern Health Sciences University (NWHSU), one of the largest accredited CIH educational institutions in the midwestern United States and home to chiropractic, massage therapy, and acupuncture/Oriental medicine. Participants were faculty and students in the 3 health-related programs. The project received approval from the institutional review boards at NWHSU and the University of Minnesota, Minneapolis.

Our approach to developing the EBP online learning intervention took place over 5 years and involved 6 main phases (Table 1). It was informed by a design-based research (DBR) framework[22,23] characterized by the following practices: (1) following a continuous cycle of analysis, design/development, implementation, and formative evaluation; (2) basing design decisions on pedagogical theory and using research to evaluate both the underlying theory and the educational intervention; (3) using mixed methods (qualitative and quantitative) for the formative evaluation; (4) evaluating the educational intervention in a real-life learning context; and (5) taking a team approach with designers, researchers, and practitioners working closely together.

Table 1

Six-phase Process and the Methods Used at Each Phase

Phase	Goals	Methods
1: Analysis	Identify audiences, learning objectives, available resources, and constraints to inform the intervention design	Conducted document review, focus groups, surveys, and interviews with key administrators and representatives from the potential audiences
		Examined the results from initial surveys conducted with faculty and students on their perceptions of the importance of EBP for themselves and the institution and their EBP-related attitudes and skills
2: Design	Choose learning theories and a design approach that addresses the goals and constraints identified in the analysis	Applied design and usability guidelines and instructional and motivational theories to graphic and functionality design
	Design technical and graphical standards and templates
3: Evaluate design	Assess if a pilot online module (1) was capable of effectively presenting content and practice to address the learning objectives; (2) met the goals of being easy-to-use, relevant, and motivating to different learners; and (3) was technically feasible for multiple audiences
		Experts reviewed the content of the initial module according to a set of criteria
		Small but representative group of students and faculty participated in usability testing of a pilot module and completed a survey on the usefulness and clarity of content
		Other faculty and staff participated in functionality and performance testing
4: Redesign	Modify theoretical approach and design, if indicated by results from phase 3	Results from phase 3 indicated that no major changes were needed to the theoretical approach or overall design
		Made minor adjustments to template functionality and graphic design
		Finalized instructional and style guidelines
5: Develop and implement	Create online modules to address identified learning objectives and deliver them via the learner management system	Online modules were developed using modified templates, along with documented standards for instructional approach and style
		Experts reviewed content of each module and representative learners reviewed for content clarity and technical performance
6: Evaluate modules	Assess the effectiveness of the online modules as an instructional tool and evaluate the selected educational theories that informed the design	Tracked participation and completion
		Conducted student surveys about their perceptions of what helped them learn

Abbreviation: EBP, evidence-based practice.

Phase 1: Analysis Outcomes

Phase 1 took place over the course of 18 months: 12 months prior to the project grant award and in the initial 6 months of the project. The goal of this phase was to identify the audiences, learning objectives, available resources, barriers, and other challenges or constraints to inform the intervention design. We used several methods to gather information, including document review, focus groups, surveys, and interviews with key administrators and representatives from the potential audiences.[21] We also examined the results from initial surveys conducted with faculty and students on their perceptions of the importance of EBP for themselves and the institution, as well as their EBP-related attitudes and skills.[21]

Early in our analysis phase, we determined the need for a multifaceted educational intervention for 2 main reasons. First, given the need to educate a large and varied audience with a limited number of EBP experts, we decided the most feasible approach would be a blend of online and in-person strategies. Second, to cover the full range of EBP competencies,[21] a variety of instructional methods was needed, including classroom discussions, activities, and assignments for students and workshops, case conferences, and seminar series for faculty. The online modules were intended to provide foundational knowledge about research concepts as a basis for the practical skills of asking appropriate clinical questions and finding and appraising relevant research. The following are key factors with implications for the design phase of the online modules.

Audience characteristics: Survey results indicated that all potential audiences believed it was an advantage for CIH practitioners to be able to locate, evaluate, understand, and integrate research into teaching and clinical practice.[21] However, some felt that research was threatening, might reduce third party–payer coverage, and sometimes didn't match CIH paradigms. All audiences rated their own EBP skills in the low to moderate range. Experience with online learning varied widely, and some faculty and practitioners were not comfortable with technology.

Barriers and challenges: These included the pressure of large course loads on students and faculty, which made them very time sensitive; variability in previous education and training; and professional differences.

Available resources: An R25 grant from the National Center for Complementary and Integrative Health at the National Institutes of Health financially supported a lead content expert and 4 others to develop content and faculty members at large to review the content. Importantly, the grant supported the effort of an instructional designer and a programmer who created the templates.

Phase 2: Design of the Online Self-study Modules

Phase 2 took place in the project's second year over 6 months. A main goal of this phase was to identify learning theories and a design approach that addressed the goals and constraints identified in the analysis. It was essential that the design met the needs of a variety of audiences with different levels of education, professions, and learning goals. The modules needed to be motivating enough to hold interest in the face of multiple competing demands and instructionally effective in the presence of time pressures. Since some faculty had limited computer skills, it was crucial that the interface was intuitive. A second goal of phase 2 was to design the technical and graphic standards and templates.

Learning Theories and Models

To accommodate the multilevel factors and complex issues identified in our analysis, we recognized that a multi-theory approach would be required to design the EBP online learning intervention (Table 2). We choose a reusable learning object (RLO) approach based loosely on the Cisco Systems (2003) model[24] to build our online EBP self-study modules. RLOs are small units of instruction that cover a limited set of related learning objectives. As self-contained units, they can be used independently in a variety of settings with different audiences.

Table 2

Learning Theories Mapped to RLO Structure

Theory	Overview	Body	Summary	Assessment
Events of instruction	Gain attention Inform learner about objectives Stimulate recall of prior learning	Present material Provide guidance Provide practice and feedback Enhance retention and transfer	Provide guidance Enhance retention and transfer	Assess
Cognitive load	Minimize extraneous cognitive load with focused content, minimal text, and physical integration of different information sources			N/A
Dual processing	Present information using both text and images			N/A
ARCS	Attention Relevance	Attention Relevance Confidence Satisfaction	Relevance Confidence Satisfaction	Confidence Satisfaction

Abbreviations: ARCS, attention, relevance, confidence, satisfaction; N/A, not applicable; RLO, reusable learning object.

The RLO approach allowed us to apply 4 main theories: events of instruction,[25] cognitive load,[26] dual processing,[27] and the ARCS (attention, relevance, confidence, and satisfaction) theory of motivation.[28] The RLO structure includes an overview, body, summary, and assessment, and each theory was mapped to these components (Table 2).

Application of Theories

In designing the technical templates, we applied common graphic design and usability principles, such as use of color and whitespace to highlight key content, increase readability, and reduce “noise.” Standardized conventions, such as a menu bar and forward arrows, were used to facilitate navigation.

Consistent with cognitive load theory, we limited the amount of text on each screen and gave each a title, labeling the content to enhance mental processing. We physically integrated captions and graphics. We sought to reduce the extraneous cognitive load as much as possible by focusing content on a small set of learning objectives, which reduced the amount of content and overall size of each online module. Using dual processing theory, we included images to convey key messages in visual form. An example of the graphic design application of these theories is illustrated in Figure 1. Full access to all online modules is available at http://www.csh.umn.edu/evidenceinformedpracticemodules/index.htm.

Figure 1

Example of reusable learning object content screen.

To cover all the events of instruction, we designed a set of interactive practice and quiz templates that provide meaningful practice and feedback and allow us to assess students. (We cover other events with intentional choice of content on text screens.)

Paying attention to the ARCS motivational theory, we sought to enhance relevance to multiple audiences by designing functionality that allowed us to offer different versions of a content screen for different types of learners. A number of additional strategies were used to meet the ARCS theory. This included capturing attention using a visually appealing professional design that instills confidence in the reliability of the content; creating relevance through content targeted to the learners' needs and extensive use of real-life examples; enhancing confidence through an easy-to-use interface and practice questions that allow learners to test their understanding; and building satisfaction with multiple opportunities for learners to succeed, including practices with supportive feedback and straight-forward quizzes.

Phase 3: Evaluate Design

Phase 3 took place in the second year over a 5-month timeframe. The goals of this phase were to assess if a pilot online module (a) was capable of effectively presenting content and practice to address the learning objectives; (b) met the goals of being easy-touse, relevant, and motivating to different learners; and (c) was technically feasible for multiple audiences. To answer these questions, we first needed to determine if the content, practice activities, and quiz were addressing the learning objectives and at the right level of complexity and detail. Multiple mixed methods were used involving a total of 25 representative learners, content experts, and technical performance testers.

EBP experts reviewed the content, exercises, feedback, and assessment questions and assessed their match against objectives and learners' incoming level of knowledge. Usability testing with 5 representative learners (a mix of students and faculty) explored ease of use. We observed users as they performed a “think-aloud” protocol to complete set tasks and explore the module. This was followed by a short survey querying the usefulness and clarity of content. Finally, performance testers used a task script to test the technical performance of the module with the most popular browsers on both PCs and Macs at intra- and extramural sites.

Design Implications

Overall, the evaluation of the online module design provided us with a “proof of concept” that our identified theories and approach were likely to be motivating and educational. Further, we could successfully develop and deliver this EBP online intervention.

Expert review found that the content and exercises matched the learning objectives. In some cases, experts recommended simplifying the language and adding additional practice to better match learner needs. They also found the content relevant to the multiple target audiences and agreed that customized examples were not needed. Usability testing showed that representative learners had little difficulty accessing the online module, navigating through it, answering questions, and participating in other practice activities. However, a need to enhance the visibility and clarity of some screen instructions was identified. With respect to the chosen motivational and instructional strategies, all testers rated the module easy or very easy to use, with the right amount of content on the screen. A number of testers remarked on the attractiveness of the design. All testers appeared engaged when going through the module and did not demonstrate fatigue or loss of motivation. Most of the testers did all the practice interactions, and when asked specifically about this, the majority said they would do them all to enhance their learning. The overall results of our performance testing found no issues with the pilot module and determined that the university had the technical capacity to deliver the online modules.

The development team also found that the predefined RLO structure of the online modules and carefully defined learning objectives greatly facilitated module creation.

Phase 4: Redesign

Phase 4 took place in the second year of the project over a period of 1 month. The goal of this phase was to use feedback from phase 3 to adjust the theoretical and design approach if necessary. Based on the feedback received, it was determined that no major changes to our approach were needed. However, we did learn that it appeared unnecessary to customize examples, practice questions, or other content for the 3 targeted health disciplines, because the testers found the content relevant and motivating as it was. Minor adjustments were made to enhance the effectiveness of the onscreen instructions, including the use of a brighter font color and some rewording for greater clarity. At the end of this phase, we finalized our graphic design and programming templates, as well as our instructional and style guidelines.

Phase 5: Develop and Implement

Phase 5 occurred in the second and third year of the project. Development and implementation occurred almost simultaneously with modules released as they became available. The goal of this phase was to create a set of online modules that addressed the identified EBP learning objectives and deliver them via the learner management system (LMS). Module details and their alignment with the GREET criteria are found in Table 3. The content for 22 online modules was developed by EBP experts following the instructional guidelines and programmed using the templates. Other experts reviewed each module and representative learners spot-checked content for clarity and relevance.

Table 3

Description of Self-study Online Modules With GREET[17,18] items identified

	Overall Characteristics	GREET Criteria
Learning environment	Online Presents same set of content and interactive screens to all Self-study: no live instructor feedback or peer interaction Asynchronous: learners can complete the modules when and where they want and at the pace desired	Settings where teaching/learning sessions undertaken Process used to ensure fidelity of teaching/delivery Face-to-face time with instructors Frequency of the teaching/learning sessions Timing of intervention
Learner characteristics	Chiropractic, acupuncture/Oriental medicine, and massage therapy faculty and students Prior knowledge about EBP and research generally low Attitude towards EBP moderate to high Some suspicion around relevance of research Time constraints and pressure extremely high Some had limited technical skills and confidence	Whether any identified barriers were targeted Whether a systematic method was used beforehand to identify barriers—see also analysis phase Whether program will be compared across different sites
Learning objectives	11 modules had ≤5 learning objectives, 11 modules had > 5 objectives[a]	Learning objectives
Content	No prerequisites required Foundational knowledge about EBP and research principles Types of research Five steps of EBP Commonly used statistical concepts Content difficulty ranges from fairly simple EBP model to complex statistical concepts	Any pre-intervention readings/activities required Aims and objectives of the educational intervention Detail of EBP components/content What method was used to decide content—see also analysis phase
Content creators	One lead and 4 other authors wrote all content, following design guidelines All authors were clinical researchers: 4 were chiropractors and 1 a physician Instructional designer edited content and added graphics and practice interactions and created final quizzes	Training required for instructors to teach the intervention Number of instructors/teachers involved Whether the same instructor was used for all teaching Description of teaching experience/expertise Profession of instructors Who was involved in designing the content Who designed the intervention
Underlying Theories
Instructional and motivational theories	Reusable learning object (RLO) approach Gagne's theory of the events of instruction Sweller's cognitive load theory Paivio's dual coding theory Keller's ARCS theory of motivation (attention, relevance, confidence, and satisfaction)	Theoretical basis/educational framework used
Graphic design	Graphic design and usability principles Cognitive load theory Dual coding theory
Number and Size
Number of modules	22 were used in the faculty development track 20 were used in the student courses	Number of face-to-face teaching/learning sessions
Length of modules	17 modules were ≤ 20 screens; average ∼17 screens/module (range: 13-23 screens) Time to complete a module averaged ∼30 minutes (range: 15 to 40 min) Total time for all 22 modules: 610 minutes	Duration of each session Duration of each entire educational program Non–face-to-face contact time with learners
Design
Graphic	Experienced graphic designer created a professional look Look standardized for all modules, providing a consistent experience
Content, interactivity, assessment	Minimal text “Content graphics” to convey key ideas visually Use of labels and captions to cue learners Multiple opportunities throughout each module for practice Feedback designed to enhance understanding and motivate learners 4 to 11 multiple choice, true/false, or short answer quiz questions, linked to 1 or more of the learning objectives at the end of each module	Teaching/learning strategies The specific educational materials/resources used

a All learning objectives are online at (see ‘Worksheets and Other Resources'): http://www.csh.umn.edu/evidenceinformedpracticemodules/index.htm

Abbreviations: EBP, evidence-based practice; GREET, Guideline for Reporting Evidence-based practice Educational intervention and Teaching; RLO, reusable learning object.

Implementation was done at the student and faculty levels. A total of 20 modules were included in a 2-credit student course entitled “Foundations of Evidence-Informed Practice” that was required for students in all 3 health programs.[29] This course used a flipped classroom approach, whereby the modules provided the “lecture” content and initial practice activities, and the classroom time allowed for in-depth discussion and hands-on practice. The modules were also offered to faculty in 2 tracks as part of an overall faculty development program: 6 core modules were required and 16 other modules were recommended. EBP experts moderated online discussions around each of the 6 core modules. In addition, faculty offerings included workshops, seminar series, and collaborative practice opportunities.[21]

RESULTS

Phase 6: Evaluate Online Module Intervention

Phase 6 took place in the second through fifth years of the project. Consistent with DBR, the goal of this phase was to collect data to assess the effectiveness of the online modules as an instructional tool and to evaluate the selected educational theories that informed our design.[22,23] Summative evaluation was based on a student survey (N=116) that specifically addressed how the underlying theories (events of instruction, cognitive load, dual processing, and ARCS) supported learning. In addition to the formative evaluation described in phase 3, we looked at module completion rates for students and faculty to assess motivation.

Results of the student survey are displayed in Table 4. Events of instruction were addressed by 4 items. Students strongly agreed or agreed that the text (80%), practice questions and feedback (86%), examples (88%), and quizzes (82%) helped them learn. Cognitive load theory was assessed using 3 items. Overall, students strongly agreed or agreed that the modules covered the right amount of information (86%), were of a length that helped them learn (80%), and the amount of text was easy to get through (79%). Dual processing theory was addressed with 1 item. Seventy-four percent of students felt the content graphics and captions helped them learn. ARCS theory was addressed with 7 items. Most students felt the modules were easy to use (95%) and understand (95%). Approximately three-quarters expressed strong agreement or agreement that the interactive elements enhanced interest (79%); the modules made EBP seem professionally relevant (77%); the overall design (72%) and graphics (71%) were appealing. Less than half expressed the idea that modules held their interest (41%).

Table 4

Results of Student Survey Assessing Underlying Educational Theories Applied in Online Modules

Item	Strongly Agree/Agree	Neutral	Disagree/Strongly Disagree
Events of instruction	%	%	%
The text helped me learn.	80.2	15.5	3.4
The practice questions and feedback helped me learn.	86.2	12.9	1.8
The examples helped me learn.	87.9	7.8	1.7
The quizzes helped me learn.	81.9	12.9	3.5
Cognitive load theory
The modules covered the right amount of information.	86.2	11.2	2.6
The length of the modules helped me learn.	80.1	15.5	3.5
The amount of text on each screen made it easy to get through.	79.3	13.8	6
Dual processing theory
The graphics and captions helped me learn.	74.2	17.2	8.7
ARCS theory of motivation
The modules were easy to use.	94.8	2.6	1.7
The modules were easy to understand.	94.9	3.4	0.9
The modules made EBP seem relevant to my profession.	76.8	19.8	3.5
The modules held my interest.	41.4	33.6	24.1
The overall design of the modules was appealing.	72.4	22.4	3.4
The graphics were appealing.	70.7	24.1	5.2
The interactive elements (questions, scenarios, clickable areas) made the module more interesting.	79.3	12.9	7.8

Abbreviations: ARCS, attention, relevance, confidence, satisfaction; EBP: evidence-based practice.

Two additional open-ended questions queried what was liked most and least about the modules (Figure 2). These were analyzed using qualitative methods. The features that were commented on the most were length and content with approximately equal numbers liking most and least (59%). Comments related to relevance (27%) and quizzes (23%) were the next most frequently mentioned.

Figure 2

Percentage of students citing what they liked most and least about the online modules.

Module completion rates among 848 participating students were high, ranging from 94% to 99% depending on academic program. Completion rates for faculty (N=117-129) were similarly high for the core faculty modules (89%); 37% completed the additional optional modules.

DISCUSSION

This article describes the process used to develop an EBP-focused, online self-study educational intervention. By incorporating a DBR perspective, we evaluated the intervention in specific contexts and the educational theories for general applicability. Overall, our multi-faceted evaluation found that our EBP online intervention was an appealing and efficient way to teach basic conceptual knowledge. Implementation of the online modules proved viable and offered a method for teaching a large and varied audience in a sustainable, standardized, and cost- and resource-efficient way.

Our goal was to develop the educational strategies that would work in our given context to improve EBP knowledge, skills, and attitudes in our identified audiences. We chose an RLO approach that was further informed by Gagne's events of instruction, Sweller's cognitive load theory, Paivio's dual processing model, and Keller's ARCS theory of motivation.[24-27] By being explicit about our chosen theories and application, we were able to align our learners' needs with optimal learning strategies (Table 2).

Overall, the DBR process helped us identify and document the intervention at a level of detail that meets almost all the proposed GREET criteria[17,18] and addresses limitations in the existing literature of EBP educational interventions (Table 3).[6,8-10,17-20] Further, the DBR methodology helped us design what looks to be a promising approach for teaching foundational EBP and research concepts and exploring the impact of selected learning theories. We found that the rigor of this approach produced a thorough understanding of the audience, environment, and learning goals, which in turn supported an effective instructional design. The feedback from the various formative evaluations likewise strengthened the design. Our expert review indicated that the online modules covered the stated learning objectives and were a feasible way to present foundational research concepts and assess learners. The usability testing (phase 3), while done with a small sample, consistently indicated ease of use. The high completion rates (phase 6), while likely influenced by various motivations and incentives such as course grade or dean-required faculty development, are encouraging and support our choice of theory and design elements.

The student survey results (phase 6) indicate that overall students felt the online modules helped them learn EBP concepts and that specific theory-based design elements enhanced their learning and motivation. An exception was the relatively low percentage of students (41%) agreeing with the statement “The modules held my interest.” Possible explanations appeared in the open-ended comments, with students expressing a high sensitivity to module length (Figure 2). As expressed by 2 individuals: “Some were too long, and seem to just drag on and on,” “I lost interest in the modules that were longer than 15 pages.” Students who indicated that the modules did not hold their interest may be reacting to the longer modules or the overall amount of work with 20 modules. It may also be that the length of the modules, while contributing to learning with an appropriate cognitive load, may not be optimal for motivation. Content was also mentioned frequently as something liked most and least by learners (Figure 2). This suggests that the modules might have been too basic for some due to prior EBP training. Alternatively some modules on more complex topics, such as statistics, may have been too difficult for others and hence failed to hold interest. Thus the content itself may be another reason for the lower percentage of students saying that the modules held their interest. We intentionally designed the content to be foundational in nature. A suggestion for going forward might be to offer more advanced content for students with prior knowledge. Thirteen percent of students mentioned that they least liked the redundancy between the modules and classroom. While our intention was that the modules would provide foundational knowledge and the classrooms advanced practice, it is possible that the classroom activities were too basic for some students.

There is evidence that multifaceted EBP educational approaches using a combination of teaching methods are more likely to improve EBP attitudes, knowledge, and skills.[6] The online EBP educational intervention described in this paper represents one aspect of a larger, multifaceted EBP educational program reported elsewhere.[30] We intentionally attempted to cover lower-level learning objectives in the online intervention, in part because the self-study aspect limited the type and depth of the practice and feedback opportunities. The modules were offered as starting point, followed by other effective instructional strategies including collaborative learning, personalized guidance and support, increased time on task, and iterative cycles of learning in classes and workshops, all which addressed higher-level objectives. We feel that an online module approach such as ours is best suited to content that provides a foundation and allows for further exploration with peers and instructors as well as hands-on application, and this is consistent with the current literature.[6]

Our DBR approach did have some limitations. Since we used multiple learning theories, it was not always clear which specific theory was impacting learners in our formative assessments. Some of our formative evaluation data collected in phase 3 is limited by small sample sizes. Additionally, while phase 6 evaluation included a summative evaluation comprising student surveys (N=116) assessing underlying theories, we had insufficient resources to perform similar assessments on faculty and field practitioners. Finally, while we did assess our learners' pre- and post-intervention EBP attitudes, skills, and behaviors as part of the larger, multifaceted EBP education project[21] we did not specifically measure these as they related directly to the online intervention. Rigorous randomized trials of EBP educational interventions, particularly those that are online and informed by appropriate adult learning theories, are very much needed.[14] To that end, our EBP online modules have recently been tested in a randomized trial in a nationwide sample of chiropractic clinicians and will be published at a later date.

CONCLUSION

We found a DBR approach to be extremely useful for developing and assessing an EBP online learning intervention. This process helped us define and document our specific learning strategies and underlying instructional and motivational theories, thus addressing some of the common limitations of educational intervention reporting that currently exist in the literature.[6,9,14,19] By describing many of the items recommended by the GREET guidelines for EBP educational intervention reporting, we have provided important information for others who may consider using this EBP online learning intervention or designing their own.

The online learning modules described in this manuscript are available at no charge to the user at http://www.csh.umn.edu/evidenceinformedpracticemodules/index.htm.

Authors' Contributions

LD was a co-investigator on the larger evidence-based practice (EBP) educational program under which the online modules were developed (R25AT003582). She initiated the design-based research process and participated in all aspects of online module design, development, implementation, and evaluation, including assisting in the analysis of the survey data. LD wrote the first draft of the manuscript, worked with the other authors to revise, and critically reviewed and approved the final manuscript. CV was a co-investigator on the larger EBP educational program under which the online modules were developed (R25AT003582). She coordinated R25 program activities for the last 2 years; assisted with online module implementation and evaluation; analyzed survey data and prepared tables/figures; assisted with the interpretations of the results; and critically reviewed the drafts of the manuscript. RE was the principal investigator of the larger EBP educational program under which the online modules were developed (R25AT003582). She was responsible for securing grant funding and program administration. She participated in all aspects of online module design, development, implementation, and evaluation. She worked with the other authors on all sections of the manuscript and critically reviewed and approved the final manuscript.