A prospective mixed-methods study evaluating the integration of an evidence based medicine curriculum into an internal medicine residency program.

INTRODUCTION: Evidence based medicine (EBM) skills are often lacking in the general internal medicine physician population. Our aim is to evaluate the impact of our internal medicine EBM curriculum on the residents' EBM skills and knowledge through mixed methods.
METHODS: A prospective study was performed that evaluated the EBM curriculum: quantitatively, with pre/posttests and qualitatively through a focus group that addressed residents' opinion on the educational and clinical impact of each aspect of the curriculum.
RESULTS: A total of 60 internal medicine residents were surveyed. Short-term EBM skills: therapy workshops (N=25) median pretest score of 8 (IQR): [6-9]) vs posttest 8 (IQR: [8-9]), (p=0.006); diagnosis (N=16) pretest score of 6 (IQR: [3-6]) vs posttest 7 (IQR: [6-9]), (p=0.006); systematic review (N=13) pretest score of 4 (IQR: [4-6]) vs posttest 7 (IQR: [6-8]), (p=0.002); and harm (N=16) pretest score of 6 (IQR: [5-7]) vs posttest 7 (IQR: [7-8]), (p=0.004). Long-term EBM skills: Fresno test of competence in EBM, pretest median score of 110.5/212 (IQR: 96.0-124.0) and a median posttest score of 115/212 (IQR: 100.0-130.0) (p=0.60). Having previous EBM training, being actively involved in research and being the first author on a publication was associated with higher Fresno test scores. Focus group provided qualitative feedback on the residents' EBM curriculum perception.
CONCLUSION: This curriculum adds a significant contribution to the current field of medical education as it fills an important educational gap, through defining ways of effectively delivering EBM concepts which led to improvement in residents' ability to evaluate and apply medical literature. The EBM curriculum was overall well received by the residents.

Introduction

Evidence-based medicine (EBM) is a term used to describe a structure for practicing medicine. This structure incorporates three fundamental components: the best available clinical research, a clinician’s judgement, and patient’s values and beliefs.1 Dr. Gordon Guyatt and his colleagues at McMaster University were the first to identified the need to foster clinician skills in medical literature retrieval, critical appraisal, and thoughtful application to a patient.2 Soon after, the first edition of the JAMAevidence, Users’ Guide to the Medical Literature was created,3 which has been the backbone of many EBM curricula worldwide.4–6

In recent years, with the development of a multitude of online databases and search engines, acquiring the ability to access and interpret reliable and applicable medical literature is an exceedingly integral part of the daily practice of medicine. Teaching young physicians the skills and knowledge necessary to perform these tasks remains a challenge.7 Journal club, the most common vehicle used to teach evidence based practice to resident physician learners, is a highly variable experience that does not always deliver the intended learning outcomes.8 Other EBM education tools have been studied, but with much less frequency, and none have shown superiority.9,10

A recent Accreditation Council for Graduate Medical Education (ACGME) competency states that

the residents are expected to develop skills and habits to be able to meet the following goals: locate, appraise, and assimilate evidence from scientific studies related to their patients’ health problems.11

Currently, this competency must be integrated in all residency programs and is expected to improve on the previously described lack of formal EBM curricula in postgraduate training programs.10–12

Considering this, in 2013, our institution developed an EBM curriculum based on the adult-learning theory and integrated it into the internal medicine residency. The first year after implementation, the curriculum successfully increased residents’ comfort with EBM concepts and self-reported application of EBM skills and knowledge to patient care.13 In 2014, our EBM curriculum was adjusted based on the results from the previous year. The adjustment reflected the addition of four short workshops that aimed at teaching residents to critically appraise four study types: randomized controlled trials, systematic review, harm  studies, and diagnostic studies. After evaluating this intervention, it was concluded that teaching EBM critical appraisal concepts in four short workshops was an effective method that improved residents’ short term EBM skills and knowledge.14 Based on this previous evaluation, the internal medicine resident EBM curriculum was combined and received this final structure that we are currently reporting.

Aim

The aim of the current study is to systematically and scientifically evaluate the internal medicine residency EBM curriculum through mixed-methods.

Methods

A prospective single center, longitudinal, study was conducted to evaluate the implementation of the EBM curriculum. The study was conducted at Beaumont Hospital, Royal Oak, Michigan, from July 2015 through August 2016. Sixty categorical internal medicine residents, 16 transitional year residents, and 16 internal medicine/pediatric residents rotate through and participate in the educational internal medicine ambulatory clinic and noon conference lectures throughout an academic year. All residents were allowed to participate in the lectures and workshops, but only data from categorical internal medicine residents were included in this study. Transitional year and internal medicine/pediatric residents were excluded from the study because they were not exposed to all the EBM curriculum components, as they are frequently on rotations outside of the internal medicine department.

Curriculum description

Our current internal medicine resident EBM curriculum was designed and later modified, by a core group of EBM faculty, with the aim of increasing internal medicine (IM) residents’ ability to critically appraise and apply current medical literature to enhance patient care and promote lifelong learning. Four main curriculum components were designed to meet this goal and included, (1) a monthly EBM workshop, (2) a monthly journal club, (3) a resident-lead morning report named “senior morning report (SMR)”, and (4) a teaching rounds exercise referred to as “chief rounds (CR)”.

Curriculum structure

Each of the four main components of the curriculum follow the same basic workflow. First, a patient is identified and a relevant clinical question is formulated in the standard population/patient, intervention, comparison, outcome (PICO) format. Depending on the activity, a literature search is performed and a relevant article is selected either as a group or by the individual participant. The article is then critically appraised using the critical appraisal forms found in the JAMA evidence Guide.3 Workflow is depicted in the Figure 1. The critical appraisal is then reviewed in a group, with group size varying based on activity. There is at least one EBM faculty that moderates the group teaching. The structure of our EBM curriculum relies heavily on the adult learning theory to better facilitate the knowledge gain and the skill development.13

Figure 1

EBM workflow.

Curriculum integration

Each of the four activities were incorporated into the current didactic and rounding schedule and have different levels of residents active involvement throughout the year. Briefly, the residents have 2.5 hrs of protected educational time daily. Depending on the residents’ rotation, not all residents are expected to attend every lecture, every day, but having daily protected time for educational activities provides the framework. Figure 2 demonstrates an example of the weekly schedule during core internal medicine rotations.

Figure 2

Example daily lecture schedule, EBM weekly view for in-patient and out-patient perspective.

The following section details each of these EBM activities and briefly describes their integration in the overall academic internal medicine residency curriculum at Beaumont Health, Royal Oak. (Figure 3)

Figure 3

Structure of EBM curriculum and the level of resident participation.

Curriculum components

First, the monthly EBM workshops, in which residents rotate through quarterly, were created to improve IM residents’ comprehension of the basic EBM and statistical concepts needed to appraise the four major study designs found in medical literature: therapy (randomized-control trial), harm, diagnosis, and systematic reviews and meta-analyses. These workshops are comprised of three 30 min sessions held on three consecutive days, during a regularly scheduled ambulatory internal medicine lecture called “clinic talk”. Each of the four workshops repeats every month for four consecutive months to allow all the IM residents the opportunity to rotate through the course.14

Second, a monthly journal club was modified from its initial form to include a greater emphasis on practicing EBM skills. The current journal club is conducted in two discrete parts. First, a small group (6–10 residents), led by a faculty member, select a recent patient case that raised a clinical question. After the PICO question is defined, the participants independently search for the best available evidence. The residents independently analyze the article and then meet as a group and discuss the critical appraisal. Later that month, at a daily noon conference, each member of the small group is then expected to guide a group of residents who has not yet appraised the article, through the critical appraisal. The EBM faculty also teaches EBM concepts related to the study design.

The third component, senior morning report (SMR), is a resident led, 60 min conference, whereby, residents, currently rotating through a general medicine inpatient service, identify an interesting patient, formulate a clinical question in the PICO format, and select and appraise a relevant article. Every Thursday, three residents prepare and present the patient, the article, and the critical appraisal of the article to the entire IM residency program. The entire IM program then has the opportunity to discuss the article in terms of its validity, results and applicability.

The final component, chief rounds (CR), consists of teaching rounds performed with every general internal medicine inpatient team once per month. In attendance are a senior faculty member, not associated with the patient’s active treatment team, a chief medical resident, as well as the residents and medical students on the active treatment team. Again, an interesting patient is chosen, a clinical question is asked, and a relevant article is selected from the primary literature and appraised. The patient case and article is then presented to the group and thoroughly discussed.

Curriculum evaluation

Our prospective evaluation of the curriculum included mixed-methods, qualitative and quantitative, with the aim of evaluating three aspects: overall resident satisfaction with the curriculum, short-term advancement of EBM knowledge and skills, and long-term application of EBM skills to daily practice. To assess overall satisfaction and subjective improvement of skills, a combination of surveys and a focus group was conducted. To objectively assess residents’ short term EBM skills, a pre- and posttest was administered before and after each EBM workshop. To objectively assess residents’ long-term application EBM skills, the Fresno test of competence in evidence based medicine, a validated evaluation tool,15 was administered to all residents at the beginning and the end of the academic year.

Resident overall satisfaction and subjective improvement of skills

A focus group was conducted six months into the academic year, to allow enough time for the interns to become familiar with all of the EBM curricular components. The focus group goal was to better explore residents’ subjective evaluation regarding the EBM curriculum and to explore their views on how the curriculum improved their ability to more confidently apply EBM skills.16–18 Preparation and execution of the focus group was based on the checklist published by Tong et al.18

Six internal medicine residents who were randomly selected using a computer program, two residents from each postgraduate year (PGY), were invited via email to participate in the focus group. The focus group was led by an EBM-experienced, associate professor from the hospitals’ affiliated medical school, Oakland University William Beaumont School of Medicine. The selected residents were not required to participate in the focus group, but they were compensated for their participation with a nominal gift card to a local coffee shop and a catered dinner during the focus group. The leader of the focus group was chosen because of her prior experience leading focus groups, knowledge of EBM, and lack of personal or professional interest in the study curriculum or IM residency program. The discussion was audio recorded and notes were taken during the session in real time. Transcription and analysis was conducted by a third party that had no connection with our IM residency program and no knowledge of the IM residents. The transcripted focus group was free of names and identifiers at the time the study personnel and the authors of this manuscript had access to read it. This was done in order to maintain the EBM faculty blinding to the comments from the focus group.

Short term knowledge

In order to accurately evaluate the impact of the EBM workshops on the IM residents’ short-term retention, a pre- and posttest, created by the EBM faculty was administered to the residents. The test was 9-to-10-questions and was based on the 15-question validated Berlin questionnaire.4 The test was instructor created to accurately evaluate the information taught in the workshops. The pretest was administered in the first seven minutes of the first session of the workshop. The posttest was administered in the last seven minutes of the last session. Residents were asked to create an 8-digit unique identifier that could not easily be linked back to a particular resident. The example of the numerical value of their mother’s birthday was given. The pre and posttest were matched to allow for paired analysis of the workshops.

Long term knowledge

The “Fresno test of competence in evidence based medicine” was administered at the beginning and at the end of the academic year to assess long term growth of EBM skills. This test is a reliable and validated tool of 12 multipart questions with a maximum score of 212 points. Partial credit could be awarded. The test and the rubric are available online.18 The residents were asked to use the same 8-digit identifier that the residents chose for the short term knowledge test which allowed residents’ short term and long term knowledge to be paired and correlated. (Appendix 1) The test was graded by two of the authors, AA and AH, independently. Conflicts were resolved with discussion. In the event a solution could not be reached, AA and AH re-graded the answers together.

IRB approval

The current study was reviewed and approved by the Beaumont Hospital Research Institute Institutional Review Board for ethical research prior to initiation. IRB number: 2015-103

Informed consent

Informed consent was obtained from participants though an information sheet that stated that by filling out the initial Fresno survey, the participants agree to participate in the study.

Blinding

All investigators, data collectors, data analysts, and adjudicators of outcomes were blinded with respect to the EBM workshop and Fresno test results. Each resident’s identity was linked to a unique identifier. This unique identifier, along with other demographic information including, post-graduate-year, age, sex, and year and country where the resident graduated medical school, were all used to match residents pretest and posttest for analysis.

The focus group was transcribed by a third-party professional transcription company and was de-identified at the time of transcription. The audio file was later destroyed. All data collected were stored with limited access to certain investigators to maintain anonymity. Microsoft Sharepoint, an institutionally endorsed online, encrypted, files sharing system was used to store the data. All investigators except AA were blinded from data collected from the focus group. AA could not be blinded because he was present during the focus group to help recording the session.

Statistical analysis

Comparing improvements in short-term knowledge, categorical variables were summarized by counts and percentages. The difference between total test scores for pretest and posttest was assessed using a test for marginal homogeneity. Correct answers on individual questions were analyzed using the McNemar’s test as well as odds ratios. Comparing changes in long-term knowledge improvement, categorical variables were again summarized by counts and percentages. Categorical variables including resident demographics were compared with the chi-squared/Fisher’s exact test. Continuous variables were analyzed with Wilcoxon-rank-sum test. All analyses were either primarily done by or reviewed by a biostatistician. Analysis used StatXact 11 (Cytel, Inc, Cambridge MA) and the SAS System for Windows version 9.3 (SAS Inc, Cary NC).

Results

Qualitative evaluation

During the focus group, the residents were asked about each component of the curriculum including chief rounds, EBM workshops, journal club, and senior morning report. (Table 1) Briefly, the residents felt that the most useful component offered is the EBM workshops stating that the sessions were clear and concise, focusing on one specific study design and teaching all the relevant components of its critical appraisal. There were mixed reviews over the timing of the session. Two of the six residents preferred to combine the three short (30 min) sessions into one longer 90 min session. Regarding journal club, residents found that it was most beneficial for the 8–10 residents presenting the article with only minor benefit for the residents who were analyzing and reviewing the article during the 1 hr noon conference. There were no recommendations for how to improve this portion of the curriculum. This same criticism was also expressed when referring to the senior morning report, stating that there is great value for the resident preparing the article and only very minor benefit to the rest of the residents in the audience. One interesting point was that the residents did not completely understand that senior morning report and chief rounds were actually a deliberately designed portion of the EBM curriculum. They stated “I don’t get the point” when referring to chief rounds. They continued “We do not always have a patient on our service where a good clinical question could be asked and where there is available evidence to appraise”. One resident also pointed out a similar complaint when preparing for SMR, stating, “I often either make up a clinical case so I can present an article I am interested in or I end up presenting an article that is not very good or interesting to the rest of the program”. Residents for the PGY1 class also requested a focused session for interns regarding the statistics like hazard ratio, relative risk, p-values, etc. All the residents expressed gratitude for the faculty willingness to organize and teach such a robust curriculum. Overall, the most common phase used to describe the curriculum was “helpful”. Many residents expressed, “I had no idea how to appraise a scientific article before”.

Table 1

Evidence based medicine curriculum focus group summary

Overall EBM Learning Experience	Positive: “Some aspects are very helpful”; “Everybody is so involved and enthusiastic about it”; “It has helped”; EBM is a very good concept and it’s great that they pursued this type of curriculum for us”
Negative: “Fragmented, which makes it difficult to follow”; “Should have more structure with clear objectives”; “Needs a little more organization”
Journal Club	Positive: “It did help me like break it down because I had no idea how to read an article before”; “Reviewing an article is still new and having that outline of the critical appraisal is helpful”
Negative: “Boring; needs to be separated from EBM”; “there’s just not enough time to like talk about the topic of the article and then to talk about how to read the article itself”
Chief Rounds	“Not necessary, honestly, really, I’m not learning anything”; “I don’t really know exactly what the point of Chief Rounds is”
EBM workshops	“Very helpful for me”; “Best part of the curriculum”
Do you think you’re practicing EBM	“[I] Try to, I think it is very important … I have a case, I have a patient. I go look it up (Uptodate), so in theory I do practice EBM”
Suggestions	“Teach basic statistics: RR, NNT, set aside 6 months of EBM intern lectures to avoid redundancy”; “Create database of trials of what we base standard of care on, it will probably also help us remember”; “Journal Club should be educational for us to get up to date on the knowledge … and in the beginning, if someone could break it down for us really quickly that would be better”

Quantitative analysis - short term quantitative knowledge

The total number of internal medicine residents that were in the program and could have been surveyed was sixty. For the therapy workshop, the pretest and posttest comprised of 10 multiple choice questions regarding topics relevant to the study design. Twenty-five internal medicine residents completed both the pretest and posttest and their testes were matched. (Table 2) There were twelve (48%) residents who were PGY1, two (8%) PGY2, and 6 (24%) PGY3. The median pretest score was 8 with interquartile range (IQR: 6–9). The median posttest score was 8 with (IQR: 8–9). (Figure 4) The median difference between the two test was 1 (IQR: 0–2) with a 95% confidence interval (CI: 0–2). The difference between the pretest and posttest scores was statistically significant using the signed rank test with p=0.006 and using the test for marginal homogeneity with p=0.005. (Table 3) Only three residents had lower scores on the posttest. All 8 residents who received a score of 6 or less on the pretest improved their score on the posttest. As a group, the residents demonstrated improved knowledge in two topics in particular: question 3 (Appendix 2) relating to the concept of evaluating the validity of a randomized control trial with 40% who answered correctly on the pretest compared with 80% on the posttest (p=0.006) and question 5 (Appendix 2) related to the concept of intention-to-treat analysis with 60% answering correctly on the pretest compared with 92% on the posttest (p=0.008). When analyzing resident scores by level of training there was a median pretest score of 6 PGY1, 8 PGY2, and 8 PGY3 and median posttest score of 8.5 PGY1, 8 PGY2, and 9 PGY3. The Jonckheere Terpstra test provided evidence of increasing scores with increasing time in residency (p=0.04). Residents further along in their training tend to have higher pretest scores than first year residents.

Table 2

Comparing correct and incorrect responses for individual questions on therapy, diagnosis, SR and harm workshops

Therapy
Question	Prop.correct – Pretest (n=25)	Prop.correct – Posttest (n=25)	p-value (McNemar)	Odds ratio	95% CI for odds ratio
Q1	1.00	0.96	1.0	0	(0,19)
Q2	0.92	0.96	1.0	∞	(0.05, ∞)
Q3	0.40	0.80	0.006	11	(1.87, 233)
Q4	0.76	0.88	0.38	4	(0.52, 97)
Q5	0.60	0.92	0.008	∞	(1.86, ∞)
Q6	0.64	0.68	1.0	1.25	(0.34, 4.93)
Q7	0.76	0.68	0.73	0.60	(0.12, 2.46)
Q8	0.88	0.88	1.0	1	(0.11, 9.24)
Q9	0.72	0.84	0.38	4	(0.52, 97)
Q10	0.64	0.72	0.77	1.4	(0.42, 4.52)
Diagnosis
Question	Prop.correct – Pretest (n=16)	Prop.correct – Posttest (n=16)	p-value (McNemar)	Odds ratio	95% CI for odds ratio
Q1	0.38	0.75	0.03	∞	(1.49, ∞)
Q2	0.62	0.56	1.0	0.75	(0.15, 3.44)
Q3	0.69	0.81	0.62	3	(0.33, 77.5)
Q4	0.25	0.44	0.51	2	(0.43, 9.23)
Q5	0.19	0.50	0.12	6	(0.80, 136)
Q6	0.19	0.56	0.07	7	(1.0, 156)
Q7	0.81	1.0	0.25	∞	(0.58, ∞)
Q8	0.50	0.81	0.12	6	(0.80, 136)
Q9	0.88	0.88	1.0	1	(0.11, 9.24)
Q10	0.56	0.88	0.12	6	(0.80, 136)
Systematic review
Question	Prop.correct – Pretest (n=13)	Prop.correct – Posttest (n=13)	p-value (McNemar)	Odds ratio	95% CI for odds ratio
Q1	0.92	1.00	1.00	∞	(0.05, ∞)
Q2	0.62	0.92	0.12	∞	(0.90, ∞)
Q3	0.38	0.31	1.0	0	(0, 19)
Q4	0.46	0.85	0.06	∞	(1, ∞)
Q5	0.62	0.62	1.0	1	(0.11, 9.24)
Q6	0.31	0.69	0.06	∞	(1, ∞)
Q7	0.54	1.0	0.03	∞	(1.49, ∞)
Q8	0.69	0.85	0.50	∞	(0.29, ∞)
Q9	0.62	0.92	0.12	∞	(0.90, ∞)
Harm
Question	Prop.correct – Pretest (n=16)	Prop.correct – Posttest (n=16)	p-value (McNemar)	Odds ratio	95% CI for odds ratio
Q1	0.81	1.0	0.25	∞	(0.58, ∞)
Q2	0.50	1.0	0.008	∞	(1.86, ∞)
Q3	0.94	1.0	1.0	∞	(0.05, ∞)
Q4	0.88	1.0	0.50	∞	(0.29, ∞)
Q5	0.25	0.25	1.0	1	(0.24, 4.18)
Harm
Question	Prop.correct – Pretest (n=16)	Prop.correct – Posttest (n=16)	p-value (McNemar)	Odds ratio	95% CI for odds ratio
Q6	0.38	0.50	0.62	3	(0.33, 77.5)
Q7	0.81	0.94	0.50	∞	(0.29, ∞)
Q8	0.81	0.81	1.0	1	(0.11, 9.24)
Q9	0.50	0.56	1.0	1.5	(0.23, 12.1)

Note: ∞, infinity.

Abbreviation: Prop.correct, proportion correct.

Figure 4

Median pre- and post- tests scores from the four short EBM workshops (therapy, diagnosis, systematic review (SR), and harm) Error bars represent inter-quartile range.

Table 3

Short-term quantitative knowledge from the four EBM workshops

Workshop (n)	Total questions	Pretest- median score (IQR)	Posttest- median score (IQR)	Median Difference (IQR)	95% CI	p-value (signed-rank test)
Therapy (25)	10	8 (6–9)	8 (8–9)	1 (0–2)	0–2	0.006
Diagnosis (16)	10	6 (3–6)	7 (6–9)	1.5 (0.5–4)	0–4	0.006
Systematic Review (13)	9	4 (4–6)	7 (6–8)	2 (1–3)	1–3	0.002
Harm (16)	9	6 (5–7)	7 (7–8)	1 (0.5–2)	1–2	0.004

For the diagnosis workshop, the pretest and posttest comprised of ten multiple choice questions regarding topics relevant to the study design. Sixteen internal medicine residents completed both the pretest and posttest and their testes were matched. (Table 2) There were seven (44%) residents who were PGY1, 5 (31%) PGY2, and 4 (25%) PGY3. The median pretest score was 6 with interquartile range (IQR: 3–6). The median posttest score was 7 with (IQR: 6–9). (Figure 4) The median difference between the two test was 1.5 (IQR: 0.5–4) with a 95% confidence interval (CI:1–4). The difference between the pretest and posttest scores was statistically significant using the signed rank test with p=0.006 and using the test for marginal homogeneity with p=0.008. (Table 3) Only two residents had lower scores on the posttest. All 5 residents who received a score of 40% or less on the pretest improved their score on the posttest. As a group, the residents demonstrated improved knowledge on two topics in particular: question 1 (Appendix 3) in regard to pretest probability, with 38% answering correctly on the pretest compared with 75% on the posttest (p=0.03) and question 6 (Appendix 3), regarding positive-likelihood-ratio, with 19% answering correctly on the pretest compared with 56% on the posttest (p=0.07). Two topics were particularly difficult for the residents on both the pretest and posttest question 4 and 5 (Appendix 3), aimed at evaluating validity, where less than 50% of residents answered correctly on either the pre- or posttest. When analyzing resident scores by level of training there was a median pretest score of 4 PGY1, 6 PGY2, and 6 PGY3 and median posttest score of 7 PGY1, 9 PGY2, and 6 PGY3. Residents further along in their training tend to have higher pretest scores than first year residents. The median scores were 4 for PGY1 and 6 for both PGY 2 and PGY 3. The Jonckheere Terpstra test did not provide statistically significant evidence of increasing scores with increasing time in residency (p=0.15). It was also notable that 25% of the residents scored a 20% on the pretest.

For the systematic review course, the pretest and posttest comprised of 9 multiple choice questions regarding topics relevant to the study design. Thirteen internal medicine residents complete both the pretest and posttest and their testes were matched. (Table 2) There were six (46%) residents who were PGY1, 3 (23%) PGY2, and 4 (31%) PGY3. The median pretest score was 4 with interquartile range (IQR: 4–6). The median posttest score was 7 with (IQR: 6–8). (Figure 4) The median difference between the two test was 2 (IQR: 1–3) with a 95% confidence interval (CI:1–3). The difference between the pretest and posttest scores was statistically significant using the signed rank test with p=0.002 and using the test for marginal homogeneity with p=0.002. (Table 3) Only 1 residents had lower scores on the posttest. All 11 residents who received a score of 7 or less on the pretest improved their score on the posttest. As a group, the residents demonstrated improved knowledge on two topics in particular: question 7 (Appendix 4), in regard to heterogeneity, with 54% answering correctly on the pretest compared with 100% on the posttest (p=0.03) and question 4 (Appendix 4), a question about the exhaustive literature search in a systematic review, with 46% answering correctly on the pretest compared with 85% on the posttest (p=0.06). When analyzing resident scores by level of training there was a median pretest score of 4.5 PGY1, 4 PGY2, and 7 PGY3 and median posttest score of 8 PGY1, 6 PGY2, and 7 PGY3. Analysis with Jonckheere Terpstra test did not show a trend for significance according to level of training.

For the harm course, the pretest and posttest comprised of 9 multiple choice questions regarding on topics relevant to the study design. Sixteen internal medicine residents complete both the pretest and posttest and their testes were matched. (Table 2) There were 9 (56%) residents who were PGY1, 4 (25%) PGY2, and 3 (19%) PGY3.The median pretest score was 6 with interquartile range (IQR: 5–7). The median posttest score was 7 with (IQR: 7–8). (Figure 4) The median difference between the two test was 1 (IQR: 0.5–2) with a 95% confidence interval (CI: 1–2). The difference between the pretest and posttest scores was statistically significant using the signed rank test with p=0.004 and using the test for marginal homogeneity with p=0.004. (Table 3) Only 2 residents had lower scores on the posttest. All 9 residents who received a score of 6 or less on the pretest improved their score on the posttest. As a group, the residents demonstrated improved knowledge on one topics in particular: question 2 (Appendix 5), regarding case-control study design, with 50% answering correctly on the pretest compared with 100% on the posttest (p=0.008). When analyzing resident scores by level of training there was a median pretest score of 5 PGY1, 6 PGY2, and 7 PGY3 and median posttest score of 7 PGY1, 8 PGY2, and 7 PGY3. Analysis with Jonckheere Terpstra test did not show a trend for significance according to level of training.

Quantitative analysis of long term knowledge improvement - Fresno test

The Fresno test was administered to all 60 internal medicine residents at the beginning and the end of the academic year. Forty-two residents took either the pretest, posttest, or both. Only 5 residents took both the pre- and posttest. Therefore, only their pretest score was included in the analysis of independent samples. One resident did not answer any of the questions so they were excluded. A total of 36 tests were analyzed (Table 4), with 18 pretest and 18 posttest. The Fresno test has a total score of 212 points. Median pretest score was 110.5 points [IQR 96.0–124.0] and median posttest score was 115.0 points [IQR 100.0–130.0] with p=0.60. Baseline demographics were similar in residents taking the pre- and posttests. Specifically there were no differences in regard to gender, primary language of English, attending an allopathic, osteopathic, or international medical school, year of completion of medical school, whether they received EBM training in medical school, received EBM training outside of medical school or residency, prior statistical training, prior research experience, plan to do a fellowship after residency, or if they were a first author on a publication.

Table 4

Fresno test of competence in evidence based medicine: Pre-/Posttest scores

Questions 1–7: Subjective question from Fresno test with partial credit awarded*
Question Number	Maximum Points	Median (25th–75th) Pretest Score	Median (25th-75th)Posttest Score	p-value
1	24	18.5 (14.00–20.00	15.5 (10.00–23.00)	0.64
2	24	13 (6.00–20.00)	6 (4.00–18.00)	0.18
3	24	14 (8.00–18.00)	12 (12.00–15.00)	0.99
4	24	14 (8.00–16.00)	14 (12.00–15.00)	0.86
5	24	9.5 (5.00–10.00)	5 (5.00–10.00)	0.48
6	24	10 (5.00–18.00)	18 (18.00–24.00)	0.03
7	24	7 (5.00–10.00)	14 (9.00–18.00)	0.00
Questions 8–12: Objective questions from Fresno test with no partial credit awarded**
Question Number	Maximum Points	Correct responses (%) Pretest	Correct responses (%) Posttest	p-value
8a	4	13 (72.2%)	9 (50%)	0.17
8b	4	9 (50%)	8 (44.4%)	0.74
8c	4	9 (50%)	11 (61.1%)	0.50
8d	4	10 (55.6%)	8 (44.4%)	0.51
8e	4	5 (27.8%)	2 (11.1%)	0.02
9a	4	10 (55.6%)	14 (77.8%)	0.16
9b	4	7 (38.9%)	7 (38.9%)	1.00
9c	4	8 (44.4%)	10 (55,6%)	0.51
10	4	7 38.9%)	14 (77.8%)	0.02
11	4	3 (16.7%)	2 (16.7%)	1.00
12	4	12 66.7%)	15 (83.3%)	0.44
	Maximum Points	Median (25th–75th) Pretest Score	Median (25th–75th) Posttest Score
Overall Score	212	110.5 (96.00–124.00)	115 (100.00–130.00)	0.60

Notes: *Results displayed as median pretest and posttest score by question. Pretest and posttest were compared using Mann-Whitney U test. **Results displayed as number of residents who answered correctly and percentages. Pretest and posttests where compared using Chi-square or Fisher exact test.

Two trends were also identified: residents who were a first author on a publication vs residents who were not first author on a publication at the time they took the Fresno test, median test score 116.0 [IQR: 104.0–130.0] vs 102.0 [IQR: 65.0–123.0] p=0.05. Additionally, residents who were actively involved in a research project at the time they took the Fresno test, had better median test scores than residents who were not; median test score 116.5 [IQR: 105.5–127.5] vs 99.0 [IQR: 67.5–127.5] p=0.06.

Discussion

The current EBM curriculum has shown a statistically significant improvement in residents’ short-term knowledge and had a qualitative improvement in residents’ EBM skills. This curriculum has also subjectively improved residents confidence applying EBM concepts to patient care. Additionally, this curriculum has demonstrated a longitudinal impact of residents knowledge and skills which was demonstrated by PGY3 residents having the best quantitative scores, followed by PGY2 residents, followed by PGY1. This was likely due to PGY3 residents’ exposure to this current curriculum prior to this current study’s initiation.

Although directed educational time is difficult to coordinate in an internal medicine residency curriculum, a major emphasis has been placed on teaching and preparing residents with the skills needed for them to continue their medical education long after formal training has completed. With the recent mandate by the ACGME,11 residences across the country are looking to implement formal EBM curricula.10 The current study offers a new, modular EBM curriculum that has demonstrated a statistically significant improvement in short-term knowledge gain and a qualitative improvement in implementation of EBM skills into their daily practice. As this curriculum is modular and has an emphasis on critical appraisal skills, it provides a platform on which residency programs can easily and effectively start to modify their current curriculum to meet the need for residents’ education.

Comparison with prior literature

Several prior studies evaluated EBM curricula implemented in the undergraduate medical education19–23 however, few have targeted graduate medical education in the United States. There are a great number of differences in teaching at the undergraduate and graduate levels. One major difference between educating medical students versus residents is that the latter has much greater work requirements, imposing greater time constraints.

With the inherent time constraints of graduate medical education, prior studies have also evaluated teaching EBM via a short workshop. The current results affirm the notion that these short courses and EBM related activities are appreciated by residents training in a variety of specialties.4,5,14 Ubbink et. al. concluded that short courses are appreciated and useful to residents in surgical specialties with an improvement in short-term knowledge after a short workshop.5 Nasr et al found that teaching EBM in four short workshops improved EBM knowledge and critical appraisal skills related to the four topics: therapy, harm, diagnosis and systematic review.14 This study was done in the same residency program as current study and using the same workshop structure and very similar workshop surveys. The findings of this current study are consistent with the findings in Nasr et al, proving that the workshop had a sustained effect over time when repeated every year for 2 years in a row. Additionally, the current study has better validity due to the rigorous blinding and paired data analysis which allowed for the identification of a statistically significant improvement in test scores. Another prior study evaluating an EBM short-course, which is similar to the current study, also demonstrated EBM skills are acquired throughout residency, with PGY-3 residents doing better than PGY-2 and better than PGY1.24 However, contrary to the currently reported study, Feldstein et. al. failed to show an improvement in internal medicine residents’ EBM knowledge.24 The currently reported intervention, although similarly concise, was effective in improving short term knowledge with a statistically significant improvement in critical appraisal of therapy, diagnosis, systematic review, and harm article types.

The current study was unable to detect a statistically significant increase in residents overall long term knowledge and skills using the Fresno test which is in contrast to other published work.25 There are several reasons that may contribute to the lack of improvement in the Fresno posttest scores. First, the test is long with several questions requiring a long written answer. In the current, technological age, where most residents prefer to take a computerized test, it is difficult to type out the answers and create an organized response to the questions in an electronic manner. Second, with the exceeding time demands on residents, finding the time required to complete the test thoroughly to score maximally on the Fresno test is very difficult. Furthermore, many of the questions had multiple components and required a detailed answer, however, there was no indication in the question stem how many items were required to obtain maximal credit on the test. For example, one of the questions asks: “list as many possible sources a clinician may search to find the answer to a clinical question”. To obtain maximal credit for the question, one must list 4 discrete sources, and discuss at least 2 issues related to convenience, relevance, and validity to one source or one issue related to 2 sources from the list. There was no indication in the question stem how many elements were required to achieve maximal credit. As such, several residents likely did not complete such a lengthy response because they did not know it was required. Additionally, it is also noteworthy that several of the points are awarded by performing a calculation. While this was previously stressed in medical education, there has been a recent culture shift to place more emphasis on the concepts of evidence based practice and not the formulas associated with them.

This study also provided significant insight into the EBM areas that are harder for residents to understand. From the focused group the EBM faculty identified the need to clearly define the structure of our evidence based medicine curriculum for the trainees to understand the context and its importance. As suggested throughout the focus group, dedicated sessions for the interns were scheduled in the first 6 months of their internship to review basic EBM concepts like PICO question formulation, searching strategies led by the librarians, pyramid of evidence, and detailed explanation of study designs. From the workshops, a few key concepts needed more attention. Regarding the therapy workshop, concealment and relative risk calculation, the harm workshop, validity of a harm study, impact of surveillance bias, and study design and for the diagnosis workshop, pretest probability, reference standard, and likelihood ratios. Lastly, regarding the systematic review workshops, the following topics were identified: how to recognize a sensible question for a systematic review, publication bias, and Funnel plots. From the Fresno test of competence in EBM, we learned that the residents struggled explaining the magnitude of effect and significance, likelihood ratio calculation, and identifying the type of study design in diagnostic tests. The topics identified as difficult in the workshops and fresno test were subsequently better addressed in the curriculum to allow more time and in depth understanding for the learners.

Weaknesses

Study limitations include low response rate. There was a limited response rate in both the pre- and posttests in the workshops and on the Fresno test. The pre/posttests administered during the workshops were short and non-intrusive, therefore the low response rate was likely due to overall poor daily attendance. Whereas, the poor response rate on the Fresno test was likely due to the time consuming and involved nature of the Fresno test itself. The Fresno test is comprised of several long answer questions and can take up to 30–45 mins to complete. The authors tried to mitigate this limitation by awarding $15 gift cards to any resident who completed both the pre- and post- Fresno tests. However, this only marginally improved response rates, which was not enough to match the individual residents pre/post Fresno test. Furthermore, there was no way to control for completeness of the test. This was particularly apparent when grading the Fresno test. Several of the tests were only partially completed. On the portion that was complete, the residents did well, while the portion that was not complete, the resident was given no credit. This brought the average score down and was not an accurate representation of the overall long term residents’ EBM knowledge and skills.

Strengths

There are several strengths of this study. First is the comprehensive nature of this study, including mixed-methods, to quantitatively and qualitatively evaluate residents’ short-term knowledge gain (pre/post workshop scores), long-term skills (pre/post Frenso test), and residents’ subjective perception of improvement in EBM skills and comfort with implementation of evidence-based practice.

Strengths of the curriculum include its modular nature. Each component of the curriculum can be implemented independently of the other. This allows for flexibility to modify the curriculum. For example, the structure of chief rounds was changed based off the resident and attending feedback to incorporate more of an emphasis on appraising the article rather than creating a PICO question and searching for a relevant article. This change did not affect the other components of the curriculum. Additionally, the modular nature of the curriculum would also allow for easier adoption of all or part of the curriculum by a program looking to start or modify their current curriculum. Furthermore, the entire curriculum was developed around teaching residents the fundamentals of analyzing medical literature, using the published book, JAMAevidence User’s Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice (JAMAevidence Guide). This curriculum outline can easily be modified and can be applied to many residency curriculums, especially internal medicine residency programs. Another strength is that EBM faculty who created the curriculum have all undergone advance training in Evidence Based Clinical Practice (EBCP) by participating in the McMaster University Evidence Based Clinical Practice course. One of the authors, AH, is also one of the course tutors.

Conclusion

The current EBM curriculum has demonstrated a significant improvement in internal medicine residents’ fundamental knowledge and skills in evaluating and applying medical literature. The curriculum has also affected residents’ long-term attitude, knowledge, and behavior, which will allow for safer and better quality of patient care.