Pediatric Online Evidence-Based Medicine Assignment Is a Novel Effective Enjoyable Undergraduate Medical Teaching Tool: A SQUIRE Compliant Study.

Evidence-based medicine (EBM) is delivered through a didactic, blended learning, and mixed models. Students are supposed to construct an answerable question in PICO (patient, intervention, comparison, and outcome) framework, acquire evidence through search of literature, appraise evidence, apply it to the clinical case scenario, and assess the evidence in relation to clinical context. Yet these teaching models have limitations especially those related to group work, for example, handling uncooperative students, students who fail to contribute, students who domineer, students who have personal conflict, their impact upon progress of their groups, and inconsistent individual acquisition of required skills. At Pediatrics Department, Faculty of Medicine, Cairo University, we designed a novel undergraduate pediatric EBM assignment online system to overcome shortcomings of previous didactic method and aimed to assess its effectiveness by prospective follow-up during academic years 2012 to 2013 and 2013 to 2014. The novel web-based online interactive system was tailored to provide sequential single and group assignments for each student. Single assignment addressed a specific case scenario question, while group assignment was teamwork that addressed different questions of same case scenario. Assignment comprised scholar content and skills. We objectively analyzed students' performance by criterion-based assessment and subjectively by anonymous student questionnaire. A total of 2879 were enrolled in 5th year Pediatrics Course consecutively, of them 2779 (96.5%) logged in and 2554 (88.7%) submitted their work. They were randomly assigned to 292 groups. A total of 2277 (89.15%) achieved ≥ 80% of total mark (4/5), of them 717 (28.1%) achieved a full mark. A total of 2178 (85.27%) and 2359 (92.36%) made evidence-based conclusions and recommendations in single and group assignment, respectively (P < 0.001). A total of 1102 (43.1%) answered student questionnaire, of them 898 (81.48%) found e-educational experience satisfactory, 175 (15.88%) disagreed, and 29 (2.6%) could not decide. A total of 964 (87.47%) found single assignment educational, 913 (82.84%) found group assignment educational, and 794 (72.3%) enjoyed it. Web-based online interactive undergraduate EBM assignment was found effective in teaching medical students and assured individual student acquisition of concepts and skills of pediatric EMB. It was effective in mass education, data collection, and storage essential for system and student assessment.

INTRODUCTION

Evidence-based medicine (EBM) is defined as the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.[1,2] The lag between the development of medical sciences and practice spans several years.[3] It was found essential to incorporate EBM in curricula of medical schools, with stressing upon decision analysis.[4,5] EBM is delivered through a didactic, blended learning, and mixed models. Students are supposed to construct an answerable question in PICO (patient, intervention, comparison, and outcome) framework, acquire evidence through search of literature, appraise evidence, apply it to the clinical case scenario, and assess the evidence in relation to clinical context.[6]

Faculty of Medicine, Cairo University is challenged by the massive number of students enrolled in its undergraduate years, with a total of 9664 to 8687 undergraduate students across its 6 undergraduate years in the academic year 2004 to 2005 and 2010 to 2011, respectively.[7]

Pediatrics Evidence-Based Medicine Assignment (PEBMA) was delivered didactically by competent pediatrics staff members trained in EBM teaching in Department of Pediatrics, Faculty of Medicine, Cairo University until 2011 to 2012 academic year. Students worked in 10 to 15 student groups, yet group work has its limitations in handling uncooperative students, their impact upon progress of their groups, students who fail to contribute, students who domineer, students who have personal conflict, and inconsistent individual acquisition of required skills.[8]

We developed a web-based online system to allow undergraduate medical student individual acquisition of PEBMA skills, allow work in group and overcome limitations of group work.[9] This study aimed at assessing the effectiveness of the newly developed online system as an educational tool of EBM skills among 5th year students of Pediatrics Course, Cairo University.

MATERIAL AND METHODS

Design, Setting, and Participants

PEBMA is a part of curriculum of Pediatrics Course delivered in 5th undergraduate year. Shift from PEBMA didactic model to website was approved by Department of Pediatrics Staff Council in July 2012. We followed this educational intervention prospectively in Pediatrics Department, Cairo University during the academic years 2012 to 2013 and 2013 to 2014 to assess effectiveness of PEBMA website as a tool for EBM education for 5th year Pediatrics Course students.

Participants of this study were undergraduate medical students of 5th year Pediatrics Course.

In a given academic year, students were divided into 2 independent groups to join November or March semesters. Participants of this study were 2897 students. PEBMA spanned 7 weeks.

Interventions

Structural

We developed an interactive E-medical assignment portal that is a web-based information system, which uses Internet web technologies to deliver information, services, and statistics to the system stakeholders (students, demonstrators/facilitators, and system administrator). The power of the system resides in the lack of need to install software to navigate the system. It functions by a web browser and an Internet connection which exist already in all contemporary computers and smart devices.[9]

The portal consisted of 3 different layers; the first layer is the database layer, which contains all related information about the system that includes the medical case scenarios, questions, and information about stakeholders and their transactions using Open Source Database Technology (My Structured Query Language). The second layer is the interface design using web technology (Hypertext Markup Language and Cascading Style Sheets), this technology was used to build the portal interface screens for system stakeholders. The third layer was the system workflow transactions using Open Source Development Language (Personal Home Page Tools), this layer contains the brain of the system to guide and interact with system stakeholder to accomplish their tasks throughout the system.

Functional

Students received a 2 hour orientation lecture of EBM of website and its functions. Optional hands-on training involving system navigation and expected tasks were offered.

Students login by their identification card number as a username and password to be changed by student as wished. Difficulty to login was responded by an automatic message to admin requesting username and password to be sent to an email address.

PEBMA comprises acquisition of knowledge and skills that undergo criterion-based assessment. To assure individual acquisition of skills, students accessed through a login screen, filled personal profile page that was followed by multiple screens to access tutorials. Tutorials were accessible without login as well.

Students found an external link to PICO question tutorial and model answers of single and group assignments. Screens were designed to provide scholar knowledge of EBM, that is, describe concept of EBM, how to formulate a search question (PICO question), search literature, and define the results of search in terms of level of evidence. Each student was expected to submit a single assignment comprising a structured referenced summary of maximum 400 words that contains her/his argument substantiated and decision.

After deadline for submission of the single assignment is met, students were made familiar with their group members, that is, the others who answered the other questions in a same case scenario. Students were made familiar with their achievement in the single assignment often with feedback from demonstrators/facilitators. Groups were assigned randomly by e-system. Group members had to nominate a leader and work together to prepare a structured summary. Only the leader was allowed to submit group assignment on behalf of the group members, and all had to submit their contribution in a discussion platform that was visualized by demonstrator. Students were expected to work in groups and submit written referenced structured summary of maximum 800 words, design and describe a plan for future research relevant to their case scenario subject. The EBM should foster moral-based choices, and students were encouraged to check their work for plagiarism; make individualized choices to suit the patient care in the specific scenario within a given time limit; makes weighted decisions by comparing evidence and benefits; and build self-confidence by supporting decision by argument. Students put their contribution in the discussion panel to allow delineate those who shared and annulled the effect of uncooperative students. The total assignment duration was divided into a single assignment and a group assignment. Case scenario questions had 3 types of answers, positive with evidence to support, negative with evidence against, or evidence was not enough.

The system accumulated 180 case scenarios with a total of 1804 questions. Case scenarios were added twice yearly. For every case scenario there are a group of questions related to clinical picture, management, and prognosis. The system allows the student to login, read instructions, and read a specific case scenario. Each student has to answer a single question which is computer generated. The student finds links to databases[10-13] and can submit the PICO question that identifies patient population/problem (P), intervention (I), comparison parameter (C), and measurable outcome (O), referenced summary, make changes, and save them. A student can get help through the get help box, where the student can compose the questions and receive answers. Students were encouraged to comment or make suggestions to improve the system and/or the assignment.

Students were allowed to engage in conversations in the group assignment through “discussion platform” online function. In group assignment, students have to describe their contribution in discussion platform in group assignment to get the mark in the group assignment.

The system allowed demonstrators/facilitators to login. For each of the 4 pediatrics courses 24 demonstrators were assigned to PEBMA. Each demonstrator had a group of computer-assigned students. Demonstrators coached and facilitated through the get help box, met students upon request from students, and corrected the submitted single and group assignments through a criterion-based system.

The Professor of Pediatrics in charge of coordinating the PEBMA during the Pediatrics Round is the admin of the e-system. Admin screens provide reports and link to all transactions on web including number of visits of all, demonstrators’ time to correct single and group assignments, comments report, final mark list, and allow search for individual specific student results, change password and/or username upon student request, and view in box messages and responses. Students were invited to seek help from admin through admin e-mail address.

Main Outcomes and Measures

Main Outcomes

Students acquisition of the following EBM skills:

Understanding concept of EBM.

Formulating the relevant question in the PICO framework.

Searching literature for evidence systematically and efficiently.

Appraising the evidence.

Applying the evidence to the specific clinical scenario.

Decision making and providing argument to support decision.

Students acquisition of the following presentation skills:

Structuring answer and argument in abstract form abiding by word count (400 word abstract for single assignment and 800 words for group assignment), putting a title to work, formulating an introduction, describing their search methodology, describing results, and making an evidence-based recommendation and conclusion.

Citing references.

Abiding by instructions.

Building an e-system that:

Suits average student and delivers the intended learning outcomes.

Delivers 24 hours/7day synchronized consecutive functions.

Effective in mass education.

Allows criterion-based assessment of students’ performance.

Delivers assessment feedback to students.

Delivers timely reports and statistical data.

Delivers students’ comments and anonymous questionnaire.

Measures

We assessed the system through the students’ grades, number of engagements online, and students’ end of assignment feedback questionnaire.

Students’ performances in single and group assignments were subjected to analysis.

For single assignment (total 2.5 marks) 0.5 marks were assigned for each of the following:

Student formulated a PICO question and defined the PICO elements correctly.

Student answered his/her relevant question.

Student followed instructions in preparing the 400 words abstract.

References content is congruent with results and cited in Vancouver style.

Conclusion and student recommendation tailored to case scenario are evidence based.

For group assignment (total 2.5 marks) 0.5 marks were assigned for each of the following:

Students answered their relevant question.

Students followed instructions in preparing the 800 words abstract.

References content is congruent with results and cited in Vancouver style.

Conclusion and recommendation tailored to case scenario are evidence based.

Students forwarded a future research plan with defined patient population, intervention, comparison, and measurable outcome.

Students received a total of 5 marks divided as 2.5 for single and 2.5 for group assignment. Students were warned against plagiarism and instructed that they get a 0 in case of plagiarism, irrespectively. Our assessment meets reported validation properties,[14] that is, content validity, internal reliability, item difficulty, item discrimination, and construct validity. Interrater reliability was not studied in PEBMA and is a part of future work.

Students were encouraged to answer an assignment anonymous feedback questionnaire by end of pediatrics course regarding website navigation, login, single, and group assignment, and the subjective impressions left upon the students. The screen of feedback anonymous questionnaire appeared only after students’ completely submitted their work and it was assessed to assure lack of bias or intimidation of students.

Statistical Analysis

All the statistical analyses in this study were conducted using Statistical Package for Social Sciences version 16 (SPSS, Chicago, IL). Descriptive analysis was used for data summarization. The parametric quantitative data were expressed as mean ± standard deviation (SD).

The simple frequency and cross-tabulation were employed for the qualitative data. Suitable tests of significance (t-test for parametric data and Chi-square [χ2] tests for nonparametric numbers N5) were used as indicated. Comparisons were carried upon number of students in single and group assignment.

RESULTS

A total of 2879 were enrolled in 4 pediatrics courses during 2012 to 2013 and 2013 to 2014 academic years. Of them 2779 (96.5%) logged in and only 2554 (88.7%) submitted their work. They were randomly assigned by system to 292 groups. Forty nine demonstrators participated in PEBMA across both academic years.

Students’ Performance in Single and Group Assignments

Total number of login of students was 55,886 (mean ± SD = 38.8 ± 15 per student), total number of demonstrators login was 374 (mean per demonstrator = 15 ± 6.9), and admin was 622. Students and demonstrators engaged in 1880 online conversations, of them 1023 (54.4%) were composed messages by students, and 857 (45.58%) replies by demonstrators. Students requested login information (username/password) and emailed 4349 email messages to admin on the admin announced email address. Admin responded to all, but only 3783 (86.98%) were communicated while admin failed to communicate response to 566 (13%) messages requesting username and password to login, as student correspondence email address was not valid. Table 1 shows various numbers of e-system transactions.

TABLE 1

System Recorded Transactions During the Academic Years 2012 to 2013 and 2013 to 2014

Assignment marks comprised a total of 5 marks divided as 2.5 for single and 2.5 for group assignment. Only 20 (0.78%) students of those who submitted their work failed to achieve a total of 2.5/5 (50%), while 2277 (89.15%) achieved 80% or more of total mark (4/5), of them 717 (28.1%) achieved a full mark. Table 2 presents assessed single and group assignment parameters. Mean ± SD of single assignment was 2.16 ± 0.49 and of group assignment was 2.38 ± 0.23 (P < 0.001). The individual activity of the students reflected by number of student login, correlated positively with all the 5 single assignment parameters (P < 0.001), their total grade (P < 0.001). On the contrary, frequency of login did not correlate with group assignment parameters except grade for answering relevant questions (P = 0.049). A total of 225 students did not submit their work and got a 0.

TABLE 2

Students in Single and Group Assignments Assessed Parameters

Students’ Responses to Anonymous Questionnaire

Students who responded to invitation to answer questionnaire were 1102 (43.1%). A total of 898 (81.48%) the e-educational experience satisfactory, 175 (15.88%) disagreed, and 29 (2.6%) could not decide. Figure 1 describes the various responses of student's. It is interesting that 794 (72.3%) found the overall experience enjoyable and 707 (72.2%) found the website links to databases enough.

FIGURE 1

Student's responses to end of assignment feedback questionnaire.

DISCUSSION

The novel interactive electronic system is invaluable in teaching EBM in 5th year Pediatrics Course, Faculty of Medicine, Cairo University.

It assured that each student understood, practiced, and applied the knowledge and skills of EBM, presentation skills, and worked in group. The PEBMA capitalized on “the need to know,” fostered information acquisition, decision making, developed presentation skills, and supported arguments abiding by limits of instructions. It allowed students to construct new knowledge, assured individual active participation and acquisition of skills, allowed participation in group work, enforced a code of moral conduct, and enabled confident decision-making. PEBMA targeted the average student and was efficient in mass education without compromising quality of education. It allowed continuous surveillance and monitoring affording prompt intervention. It accommodated a large bank of questions that did not allow duplication of scenarios, giving each student a different task, stores student responses, provided future research plans bank, and allowed students to participate with their comments and feedback. The sequence of tasks allowed students group-work to follow mastering skills by individual student.

The newly devised e-system exploited the advantages afforded by the accurate, accessible, and continually updated sources of evidence such as the Cochrane Library and Clinical Evidence,[15] PubMed database, and others that contain all types of reports, even those containing contradictory evidence that augment students’ power to appraise and of EBM e-learning sites.[15-19]

It is interesting to note that frequency of students login correlated with single assignment parameters, which highlights the engagement of students in self-acquisition of concept, search, appraisal, application, assessment, and presentation of EBM assignment. PEBMA challenged a minority of students (14.7%) who were inclined to resistance to precision and tended to ambiguity in tailoring their conclusions and recommendation for their specific case scenario.

Students’ higher marks in group assignment might be attributed to group effect education, learning from mistakes in single assignment and gaining more insight by practice.

The newly designed e-system is technically demanding, requires space, effort, maintenance, feeding of question bank twice annually and needs continuous surveillance for server malfunction, bugs, password losses, and student personal information inaccuracy. It demands too much work of the PEBMA staff team when they are overloaded by other teaching and training duties for both under and postgraduates. It pressurized students to shift from didactic into interactive and mostly self-education. The PEBMA team had to face students’ uprising frustrations and resistance at initial implementation that faded by time.

It is rewarding that 794 (72.3%) students found the novel e-system enjoyable.

Sustainability of PEBMA remains a great challenge, and the impact of PEBMA upon personal attitudes of students in their postgraduate practice remains to be known and cannot be foreseen. Reinforcement by subsequent practice remains the barrier against loss of acquired skills, that is, workshops, seminars, journal clubs (or any combination), standalone, and integrated methods to improve knowledge, skills, attitudes, behavior,[20] and on-the-job EBM training.[21,22]

In conclusion, we found that PEBMA website is effective in teaching undergraduate medical students. It assures individual student acquisition of concepts and skills of pediatric EBM especially in courses enrolling large number of students.