A year-round evidence-based medicine-learning course organized by medical students at Ehime University.

To resolve the problem that evidence-based medicine (EBM) courses are not sufficiently taught in Japanese medical schools, we organized a year-round EBM-learning course. This study was an observational study and was designed to evaluate the participants' understanding of EBM using an original survey. The survey was given three times. In total, 18 students responded to our survey. Of those 18 students, six students answered both the first and the last surveys, and their mean score increased 1.17 of 4.00 (95% CI: 0.72-1.65). These results suggest our course improved students' ability to read clinical articles.

Introduction

Evidence‐based medicine (EBM) is an essential skill for physicians. The global standard medical school curriculum, WFME Global Standards for Quality Improvement for Basic Medical Education, treats EBM as a scientific method which medical schools must teach throughout their curriculums.1 In contrast, Japanese medical schools’ core curriculums merely describe EBM as a tool, of which “medical students can explain the necessity.”2 This situation might cause Japanese physicians not to use EBM as part of their daily clinical activities.

Therefore, we organized scenario‐based EBM‐learning study meetings to improve students’ ability to use EBM. The objective of this study was to evaluate the participants’ understanding of keywords related to EBM. There are some studies that have showed the effectiveness of short‐term EBM‐learning courses organized by faculties3 or medical students.4 However, the effectiveness of a year‐round EBM‐learning course organized by medical students was unknown. Therefore, we created an original survey to evaluate how well students understood keywords related to EBM. The survey results suggest that our course, even though it was planned by medical students, increased students’ ability to read articles about EBM.

Methods

Research design

This study was an observational study of educational intervention at Ehime University Medical School.

EBM‐learning course

We planned an EBM‐learning course once or twice per month. Each meeting was a 60‐90 minute session, led by a doctor who was familiar with EBM. Prior to the meeting, the participants were given an article and a worksheet.5 The worksheet contained a case scenario and some questions related to the article. The participants were expected to read the article and fill out the worksheet before the meeting. During each meeting, the participants and the doctor discussed the article, after the doctor gave a short lecture.

Participants

Before the beginning of the first session, we announced the course to all second through sixth‐year medical students using a student mailing list. After the first session, we announced our course schedule to the medical students who were interested in our learning course.

Outcome and measurement

The outcome we aimed to measure was the participants’ understanding of keywords related to EBM. To test the participants’ knowledge and skills related to EBM, some previously validated methods like the Fresno test6 or Berlin Questionnaire7 were used. However, those methods deal with all five steps of EBM (Step 1: converting the need for information to an answerable question, Step 2: tracking down the best evidence with which to answer that question, Step 3: critically appraising that evidence for its validity, impact, and applicability, Step 4: integrating the critical appraisal with our clinical expertise and with our patient's unique biology, values, and circumstances, Step 5: evaluating our effectiveness and efficacy in executing steps 1‐4 and seeking ways to improve them both for next time).8 As our course focused only on Step 3, we made an original survey with 23 questions (Table 1). Each question (except Q.23) was scored from 1 to 5, with 5 being the best understanding. Q.23 was scored from 1 to 3.

Table 1

Rubric type questionnaire

	Score	1	2	3	4	5
Goals
Q1	Your motive to join our meetings	Invited by friends	To study English	To read academic articles	To read academic articles critically	To apply evidence to practice
Experience
Q2	Knowledge of statistics	None	Studied in a lecture	Have credits in university	Have experience in statistical work	Wrote articles using statistical methods
Q3	Experiences in EBM study	None	Studied in Ehime Univ. (within a year)	Studied in Ehime Univ. (over a year)	Studied outside Ehime Univ.	Learned in some systematic course
Step 1: Formulate clinical question
Q4	PICO	Don't know	Know just the name	Have studied in some lectures	Used PICO in some lectures	Use PICO daily
Step 2: Searching literature
Q5	Searching literature	Never searched and don't know PubMed	Never searched but know PubMed	Have searched for articles about basic medicine	Have searched for articles of clinical medicine in some lectures	Search for articles daily
Q6	Clinical information tools (UpToDate, Dynamed, and so on)	Don't know	Know just the name(s)	Have seen the website(s)	Have searched	Use them daily
Step 3: Critical appraising
RCT
Q7	RCT	Don't know	Know just the name	Have read in some lectures	Have read on your own	Read them daily
Q8	Randomization	Don't know	Know just the name	Have been taught in some lectures	Can find the keyword in the article	Can judge its validity when you read the article
Q9	Baseline	Don't know	Know just the name	Have been taught in some lectures	Can find the keyword in the article	Can judge its validity when you read the article
Q10	Withdraw	Don't know	Know just the name	Have been taught in some lectures	Can find the keyword in the article	Can judge its validity when you read the article
Q11	ITT analysis	Don't know	Know just the name	Have been taught in some lectures	Can find the keyword in the article	Can judge its validity when you read the article
Q12	Blind	Don't know	Know just the name	Have been taught in some lectures	Can find the keyword in the article	Can judge its validity when you read the article
Q13	Outcome	Don't know	Know just the name	Have been taught in some lectures	Can find the keyword in the article	Can explain true/surrogate outcome, primary/secondary outcome
Q14	RR, RRR, ARR	Don't know	Know just the name	Have been taught in some lectures	Can find the keyword in the article	Can calculate by myself
Q15	NNT	Don't know	Know just the name	Have been taught in some lectures	Can find the keyword in the article	Can calculate by myself
Q16	95% CI, P value	Don't know	Know just the name	Have been taught in some lectures	Can find the keyword in the article	Can interpret by myself
Diagnosis
Q17	Sensitivity, specificity	Don't know	Know just the name	Have been taught in some lectures	Can explain them	Use them in daily practice
Q18	Likelihood ratio	Don't know	Know just the name	Have been taught in some lectures	Can explain it	Use it in daily practice
Meta‐analysis
Q19	Meta analysis	Don't know	Know just the name	Have been taught in some lectures	Have read on your own	Read them daily
Q20	Funnel plot	Don't know	Know just the name	Have been taught in some lectures	Can explain it	Can judge its validity when you read the article
Q21	Forest plot	Don't know	Know just the name	Have been taught in some lectures	Can explain it	Can judge its validity when you read the article
Q22	Heterogeneity	Don't know	Know just the name	Have been taught in some lectures	Can find the keyword in the article	Can judge its validity when you read the article
Step 4: Applying evidence to practice
Q23	Applying evidence to practice	Have never applied evidence to practice	Have applied evidence to practice only in a lecture	Have applied evidence to practice in a real clinical situation

PICO, Patients Intervention Comparison Outcome; RCT, randomized controlled trial; ITT analysis, Intention to treat analysis; RR, Relative Risk; NNT, Number needed to treat; 95% CI, 95% confidence interval.

These twenty‐three questions are about keywords related to evidence‐based medicine. The first three questions (Q.1‐Q.3) were about the participants’ motivation and experience, while the other 20 (Q.4‐Q.23) observed how well the participants understood keywords related to EBM. Participants self‐assessed their understanding of the keywords. We conducted the survey using this sheet three times: at the beginning, middle, and end of the term. The participants were informed that these will be used to evaluate this EBM‐learning course and medical education.

The surveys were conducted three times: at the beginning, middle, and end of the term. To assess the participants’ understanding of the keywords, we calculated their average scores as well as a 95% confidence interval for 19 questions (Q.4‐Q.22). We did not use Q.23 for the average score calculation. In addition, we calculated the average score of each question (Q.4‐Q.22) on the last survey. We used the Wilcoxon signed‐rank test for calculation of the P‐value using IBM SPSS Statistics (ver.23). All tests were two‐sided and significance was set at 5%.

Ethical consideration

We notified the survey respondents that we would use the survey results for the following analysis and for publication. We informed the participants that they were agreeing to our research by returning their surveys to us. This study was approved by the Ethics Committee of Ehime University Hospital.

Results and Discussion

Study meetings and participants

We organized 12 course meetings and dealt with seven randomized controlled trials (RCTs), four meta‐analyses, and one other topic.5 We chose articles (i) changed the history of EBM and (ii) all the students (including lower years students) could understand them. The course was held from April 2014 to January 2015. We did not record the number of participants, but by analyzing the pictures of the classes, about thirty medical students came to the first session and around ten students came to the other sessions.

Participants’ score increases on the survey

Figure 1 shows the average score on each of the 19 questions for each student. Of these 18 students, six students answered both the first and the third surveys. Their mean score increase was calculated as 1.17 of −4.00 to +4.00 (95% CI 0.72‐1.65). The average score of the last survey was significantly higher than the first survey (P=.028).

Figure 1

Score increases on the survey. Each point shows the average score of the 19 questions (Q.4‐Q.22 of the Table 1). The participants’ scores got higher with each survey

Our course improved the participants’ understanding of some keywords related to EBM (Figure 1). Understanding these keywords helps medical students read medical articles that use EBM.

Average score of each question on the third survey

We calculated the average scores of each question on the third survey (n=10). The keywords with the highest scores (>4.0 of 5.0) are randomization, withdraw, blind, relative risk, 95% confidence interval, sensitivity, and specificity. The keywords which received the lowest scores (<3.5 of 5.0) are evidence‐based clinical information tools, intention‐to‐treat analysis, meta‐analysis, and heterogeneity.5

This indicates that the participants understood keywords related to RCT, but did not understand keywords related to meta‐analysis. The participants’ understanding would be improved if we held more course meetings about meta‐analysis. These results imply the limitation of a one‐year course.

Limitations

There are some limitations to this study. First, as the participants in this study were highly motivated students, our results cannot be directly applied to the general population of Japanese medical students. A larger scale study, for example, a study which is targeted to every medical student in one year of medical school, is needed. Secondly, the main outcome of our study was the participants’ understanding of keywords related to EBM, but understanding keywords alone is not sufficient training to help medical students read journal articles that use EBM. And finally, the rating scores on our survey have no continuous variation and thus are not suited for simple averaging. We also calculated the average score of different questions, but it is not appropriate to add different questions’ scores.

Conflict of Interest

The authors have stated explicitly that there are no conflicts of interest in connection with this article.