"ACEing" the Evidence Within Physical Medicine and Rehabilitation (PM&R).

Introduction: While evidence-based medicine (EBM) is important in all fields of medicine, it can be specifically challenging for the field of physical medicine and rehabilitation (PM&R), a rapidly developing field where the standard hierarchy of evidence does not always apply and randomized controlled trials can be difficult to design. We developed an EBM curriculum for residents that improved EBM competency and was specific to the field of PM&R.
Methods: We developed a blended learning longitudinal approach to EBM designed specifically for PM&R residents, with a pre- and postcourse assessment by the Evidence-Based Practice Questionnaire (EBPQ) and Assessing Competency in EBM (ACE) tool. Interactive presentations paired with structured presession assignments were held for five introductory sessions, followed by monthly EBM and journal club sessions over 1 academic year.
Results: Fourteen residents of varying postgraduate years of training participated in the EBM curriculum from 2018 to 2019. EBPQ scores after completion of 1 academic year of this EBM curriculum were significantly improved compared to precurriculum EBPQ scores. Comparison of pre- and post-EBPQ and ACE tool scores stratified by postgraduate year did not show a significant correlation between resident levels and self-reported prior EBM education. Discussion: This longitudinal blended learning EBM curriculum resulted in an increase in residents' self-reported behaviors and knowledge/skills regarding EBM. The curriculum was also effective in advancing competency of the residents to an EBM Advanced level using the ACE tool. The curriculum can be easily replicated in other PM&R residency programs.

Educational Objectives

By the end of this activity, learners will be able to:

Formulate clinically relevant questions that guide the search for specific knowledge to inform clinical decisions within physical medicine and rehabilitation (PM&R).

Search and select appropriate evidence-based information tools, understanding the hierarchy of evidence, to answer specific clinical questions within PM&R.

Effectively appraise evidence for its validity and applicability to individual patient care within PM&R.

Increase the use of evidence-based research and tools to inform clinical decisions within PM&R.

Introduction

In order to provide excellent patient care and engage in lifelong learning within the field of physical medicine and rehabilitation (PM&R), physiatrists need a clear and comprehensive understanding of the evidence underlying their clinical care. This requires them to stay current on the best evidence to guide clinical practice and to deliberately maintain up-to-date knowledge and skills in PM&R. Thus, a key objective of a PM&R residency program is to support the development of a framework and skill set to regularly use evidence-based medicine (EBM). The importance of this is underscored by the development of PM&R Milestone for Practice-Based Learning and Improvement 2, which includes the ability to locate, appraise, and assimilate evidence from scientific studies related to patients' health problems.[1]

EBM is taught variably in medical school and during preliminary or transitional years for residency training and often focuses on evidence-based resources that are relevant to general medical issues.[2] Often, residents entering into their PM&R-specific training years have difficulty knowing how to search for the best evidence within PM&R knowledge areas and then how to regularly use that evidence-based information in their rehabilitation-specific clinical care.

To address this need, our goal was to develop an effective longitudinal EBM learning curriculum for our residency program. The Journal of the American Medical Association has described the 5 A's of EBM (ask, acquire, appraise, apply, and assess) as the core competencies of EBM in order to better inform the development of curricula, learning outcomes, and assessment strategies.[2,3] Knowledge, skills, attitude, and behavior all play a key role in implementing these core competencies into practice effectively.[4]

In a 2014 systematic review, a lack of EBM knowledge and skills was one of the most commonly reported barriers to practicing EBM.[5] Ilic and colleagues hypothesized that a blended learning approach would help improve not only knowledge and skills but also attitudes and behaviors with regard to EBM.[6] Blended learning integrates a variety of learning approaches (lecture, tutorial, online, problem based, and clinical) in an attempt to account for different learning styles and requirements and create an optimal learning environment. A systematic review looking at the efficacy of different types of educational methods for EBM concluded that there was no difference in learner outcomes across a variety of educational modes, including lecture versus online, direct versus self-directed, multidisciplinary versus discipline-specific groups, and lecture versus active small-group facilitated learning.[7]

A direct comparison between blended learning and traditional didactic learning showed that there was no significant difference in EBM competency; however, perceived self-efficacy, attitudes, and behaviors toward EBM were significantly higher in students who received the blended learning approach.[6] Ilic and colleagues concluded that blended learning can increase medical trainees' self-confidence and help bridge the gap between theory and practice of EBM. In order to develop an effective EBM curriculum for our residency program, we chose a blended learning approach to integrate core knowledge with clinical practice.

Several publications have showcased the development of different EBM curricula for users ranging from general practitioners to pediatric residents to nurses to medical students, but none specifically for PM&R.[8-14] The number of studies published within the field of PM&R is growing at a rapid pace as the field itself grows and develops; however, EBM in PM&R is uniquely challenging for a variety of reasons. The field of PM&R is based on a biopsychosocial model that utilizes interdisciplinary teamwork to promote new motor, psychological, and social behaviors. Because PM&R is often more focused on activity and impairments rather than the disease itself, there is the challenge of standardizing terminology and outcome measures for the field. The highly interdisciplinary nature of PM&R can also make a “standard rehabilitation medicine” comparison in research difficult as variances in team dynamics and individual members of the team can make a significant difference. For example, research studies often focus on compliance and adherence to a therapy without taking into account the therapist's competence, attitude, and exercise program or the patient's beliefs and attitude, all of which can significantly impact the rehabilitation outcome of a patient.[15]

Lastly, EBM has historically evolved to value randomized controlled trials (RCTs), systematic reviews, and meta-analyses at the top of the pyramid of evidence. In some cases for medicine, specifically rehabilitation medicine, it is not always practically possible to produce research at the level of RCTs because of inherent methodological or ethical issues. Smith and Pell wryly used the example of trying to determine whether parachutes are effective in preventing major trauma when jumping out of a plane.[16] Likewise, in PM&R, there are certain interventions for which there is clear evidence that the intervention is helpful without the need for an RCT.

Although there have been many EBM curricula developed for health care professionals and trainees, none have specifically been developed for the field of PM&R, which comes with the unique aforementioned challenges. Also, many of these proposed curricula have been either a seminar series or short course condensed over several weeks.[8-14] Extended EBM instruction throughout clinical experiences was rarely reported in our literature review. A blended longitudinal learning experience developed specifically for PM&R residents would be the first of its kind.

Methods

Study Design

We developed a blended learning longitudinal approach to EBM designed specifically for PM&R residents, using a pre- and postcourse assessment by the Evidence-Based Practice Questionnaire (EBPQ)[17] and a postcourse-only assessment with the Assessing Competency in EBM (ACE) tool.[18] The curriculum was designed with the goal of achieving competency in the 5 A's of EBM and effecting change in the knowledge, skills, attitude, and behavior domains of evidence-based practice. The outcome assessments were selected to evaluate levels 1–3 in the Kirkpatrick model of learning (reaction, learning, and behavior).[19]

Course Curriculum

This curriculum consisted of three 1-hour EBM sessions and two 1-hour journal clubs spread over the span of 3 months, which were then followed up by a series of monthly paired EBM and journal club sessions throughout the rest of the year to practice using learned skills more efficiently. The EBM sessions focused on the EBM core competencies of ask and acquire while the journal clubs focused on appraise and apply. All residents, regardless of postgraduate year, participated in this annual curriculum.

EBM session 1 was a focused didactic utilizing a PowerPoint presentation (Appendix A) to discuss the background and context of EBM and the importance of formulating clinically relevant questions in order to guide searches in the literature. Residents were given a brief tutorial on how to search for evidence using an EBM session worksheet (Appendix B) designed for this curriculum that guided them through different levels of evidence with PM&R-specific resources. Prior to this session, each resident was asked to share one clinical question in the PICOTT (patient, intervention, comparison, outcome, type of question, type of study) format (Appendix C). One clinical question was selected from those shared, and residents were assigned the task of completing the EBM worksheet with the selected question, focusing mainly on utilizing preappraised sources.

Objectives for EBM session 1 included the following:

Be able to name the five steps of EBM.

Be able to self-assess which mode of EBM is being practiced (e.g., doing, using, replicating).

Be able to ask a foreground question in the PICOTT format.

Understand which type of study is preferred for specific question types.

For EBM session 2, residents participated in an interactive guided discussion of their search process and shared their search results for the selected clinical question using the preappraised sources section of the EBM session worksheet. A second PowerPoint presentation (Appendix D) was used to discuss the hierarchy of preappraised evidence and how to utilize preappraised sources to make efficient clinical practice decisions. The top two articles found by residents were selected for subsequent journal club review, which focused on how to critically appraise articles and was resident led. Residents were asked to submit another PICOTT question with their journal club assignments, and one of those questions was selected for EBM session 3.

Objectives for EBM session 2 included the following:

Understand the preappraised hierarchy of evidence.

Know which resources are available when searching for preappraised evidence.

Know how to access preappraised evidence resources.

Journal club session 1 was held between EBM sessions 2 and 3. This was an interactive didactic with a PowerPoint presentation (Appendix E) using one journal article selected in EBM session 2 as an example throughout. The presentation walked through the basic concepts for appraising a therapy article.[3] The session ended with an interactive appraisal discussion of the selected journal article with the assistance of an appraisal worksheet (Appendix F).

Objectives for journal club session 1 included the following:

Be able to assess the validity of the results and how to detect bias.

Be able to calculate different forms of treatment effect.

Understand how to interpret study results.

Be able to assess how those results affect patient care.

EBM session 3 was similar to EBM session 2 in that it was an interactive discussion of search processes and results but with a focus on individual peer-reviewed studies. A third PowerPoint presentation (Appendix G) was used to review the hierarchy of evidence for individual studies and the preferred type of study for certain clinical questions. Search strategies utilizing medical subject heading (MeSH) terms and PubMed tutorials were also reviewed in this session.

Objectives for EBM session 3 included the following:

Understand the hierarchy of evidence for individual peer-reviewed studies.

Understand how to use MeSH headings in PubMed.

Understand how to use Boolean operators and other tools to perform manual searches via PubMed.

Understand how to apply filters and utilize clinical queries to narrow a search and retrieve targeted results in PubMed.

Journal club session 2 (Appendix H) was held after EBM session 3. It was similar to journal club session 1 in that it comprised an interactive didactic portion discussing basic concepts for appraising a systematic review/meta-analysis followed by an interactive appraisal discussion using the critical appraisal tool introduced during the lecture (Appendix F). For journal club session 2, the group appraised a therapy article (to review skills learned in journal club session 1) and a systematic review article (requiring use of information newly covered in journal club session 2).

Objectives for journal club session 2 included the following:

Understand how to assess for bias in and the validity of systematic reviews/meta-analyses.

Understand how to interpret flow diagrams, forest plots, and I2 values.

Be able to assess how the results apply to patient care.

The remaining monthly EBM sessions were resident-led discussions that focused on efficient search strategies, with selected articles being chosen for the next associated journal club. The facilitator guide (Appendix I) provided a more detailed description of what each EBM and journal club session covered and how to lead the session.

Outcome Assessment

The ACE tool presented a patient scenario and then asked a series of questions to test the trainee's knowledge and skills in EBM across the first four steps of the EBM process (ask, acquire, appraise, and apply). The fifth step of the EBM process, assess or evaluate, was excluded since self-evaluation relied on users' personal reflection on the EBM process and clinical scenario.[18] The EBPQ was designed to gather information and opinions on the use of evidence-based practice in health care professionals. It essentially measured attitude, knowledge, and implementation of EBM.[17] In order to assess the efficacy of our blended learning EBM curriculum, we chose to utilize the ACE tool and the EBPQ to evaluate knowledge, skills, attitudes, and behaviors with regard to evidence-based practice.

Results

Fourteen PGY 2-PGY 4 PM&R residents from a single program participated in this study during the academic year starting in 2017 and ending in 2018. Pre- and posttest scores for the EBPQ were analyzed using a paired t test. Scores for the ACE tool were analyzed using an analysis of variance to compare test scores amongst the residents.

EBPQ scores after completion of 1 academic year of this EBM curriculum were significantly improved compared to precurriculum EBPQ scores, with the most significant change coming from the knowledge and skills domains while behaviors and attitudes scores remained relatively unchanged (Table 1). Nine of the residents (five PGY 3s and four PGY 4s) had prior exposure to partial aspects of the proposed curriculum. Comparison of pre- and post-EBPQ scores and ACE tool scores stratified by PGY level did not show a significant correlation between resident levels or self-reported prior EBM education (Table 2).

Table 1.

Comparison of EBPQ Scores (Total and Three Domains) Before and After Evidence-Based Medicine Curriculum

Table 2.

Comparison of EBPQ Scores by Domain and PGY Level Before and After EBM Curriculuma

The ACE tool was scored on a 15-point scale: EBM Novice (<2 years of training) = 8.6 (SD = 2.4), EBM Intermediate (3 years of training) = 9.5 (SD = 1.8), and EBM Advanced (4 years of training) = 10.4 (SD = 2.2).[18] The average of all 14 residents' ACE scores was 10.5 (SD = 1.91), with no significant difference in scores when stratified by PGY level (Table 3).

Table 3.

Average Postassessment ACE Score for All Residents and by PGY Level

Discussion

In order to assess the efficacy of this longitudinal blended learning EBM curriculum, we utilized the ACE tool and EBPQ to measure changes in knowledge, skills, attitudes, and behaviors. The curriculum resulted in an increase in residents' overall knowledge of and skills in EBM. An improvement in self-reported behaviors was not statistically significant. Interestingly, a slight decrease in the EBPQ attitudes domain was seen for PGY 3s and PGY 4s, although it was not statistically significant. The main driver of this decrease was residents' perception of their heavy workload and the additional time required to incorporate new evidence. However, at the same time, their perception of the importance and practice-changing effect of evidence-based practice increased. Increased practice of EBM skills may result in increased efficiency, which may ultimately result in an increase in EBPQ attitudes. The curriculum was also effective in advancing overall competency of the resident participants to an EBM Advanced level using the ACE tool. This increase in self-reported behaviors and knowledge/skills signifies an improvement in evidence-based practice and self-efficacy, which are essential aspects of EBM competency. These increases were seen across all resident levels despite prior exposure to partial aspects of the EBM curriculum among some residents, indicating that ongoing EBM competency required regular use of knowledge, skills, and practice.

Although our results are limited to 1 academic year of evaluating this EBM curriculum, it will be continued as part of our residency program's education curriculum with annual pre/post assessments of subsequent incoming PGY 2 residents utilizing both the EBPQ and the ACE tool. A larger and more universal effect on EBM competency may be seen with a longer curriculum, following residents across multiple years of residency, allowing for increased and regular practice of EBM-related knowledge and skills. In addition to the limited time frame and sample size, we did not obtain a precourse ACE tool assessment when this curriculum was first developed and implemented due to potential difficulty in interpreting test-retest reliability as the tool provided a single case example. Implementation of this course curriculum may be better assessed with pre- and postcourse assessments of both the ACE tool and the EBPQ. Future implementers of this curriculum should also consider including a postcourse evaluation for resident feedback on the quality of the curriculum, achievement of objectives, and overall satisfaction with the course.

The combination of traditional didactics, interactive resident-led sessions, at-home participation, and integration with associated journal clubs in a longitudinal format allows for continual learner engagement and accommodates multiple types of learning styles. Introduction of key concepts in the beginning of the year with the subsequent monthly hands-on practice of the EBM and critical appraisal worksheets help to not only solidify learning but also improve efficiency. The limited volume of information included in each session combined with the interactive components of the sessions and precourse assignments was perceived by the PM&R residents as easily digestible, and they felt that it supported their learning and practice of the relevant knowledge and skills. The PM&R resident learners appreciated the structured approach and tools that were utilized in scaffolding their knowledge and skills regarding evidence-based practice. Providing a time estimate of 45 minutes for initial precourse assignments was helpful in setting expectations. Providing a framework of initiating with more time-consuming comprehensive searches to understand how to utilize each of the specific information technology EBM resources and gradually progressing over the academic year to focused and less time-intensive efficient searches was also helpful.

This curriculum was developed specifically for PM&R residents, highlighting PM&R-specific information technology EBM resources. The residents found it particularly helpful to know which resources were more likely to yield high-quality evidence for PM&R-specific topics, which often differed from their prior experience in other medical specialties. A limitation of this curriculum is that it was implemented in a residency program that had 3 hours of dedicated lecture time per week. Residencies that are limited in didactic time may not wish to dedicate a large percentage of their didactics to EBM and journal club, in which case, they could choose to do the paired EBM and journal club sessions bimonthly or quarterly. The resources needed are a classroom or meeting space with a projector to display slides plus computer and internet access for the participants outside of the EBM didactic sessions. The original implementation of this curriculum utilized two different faculty members to teach the EBM and journal club sessions; however, in subsequent years, the curriculum was able to be led by a single facilitator utilizing the facilitator guide (Appendix I).

EBM Presentation 1.pptx

EBM Session Worksheet.docx

EBM Question Worksheet.docx

EBM Presentation 2.pptx

Journal Club Presentation 1.ppt

Critical Appraisal Worksheets.docx

EBM Presentation 3.pptx

Journal Club Presentation 2.ppt

Facilitator Guide.docx

All appendices are peer reviewed as integral parts of the Original Publication.