Effect of Interactive Lectures and Formative Assessment on Learning of Epidemiology by Medical Undergraduates - A Mixed-Methods Evaluation.

BACKGROUND: Previously, we had a course in epidemiology for medical undergraduates that was based on traditional lecture methods with no formal formative assessment (FA). We found poor uptake of our course in terms of learning and attendance by students.
OBJECTIVE: The objective was to assess the effect of improved course (interactive lectures and formal FA) in epidemiology on student learning and attendance.
MATERIALS AND METHODS: It was a triangulation type of mixed-methods program evaluation, where both quantitative (quasi-experimental design) and qualitative (open-ended responses) analysis was done. This study was carried out in the department of community medicine in a tertiary care teaching hospital, Puducherry. We improved the quality of the course material, interaction in lectures and included formal structured FA in the last course. Kirkpatrick's framework was used for the course evaluation. We compared the performance of three batches to check the effect of our revisions on students' learning and their attendance.
RESULTS: Student's learning outcome was measured using end-of-course assessment scores (Level-2). The percentage of students successfully completing the course improved from 39% to 81% and attendance status of ≥90% improved from 50% to 57%. Learner's immediate reactions (Level-1) were captured using open-ended questions, and content analysis was done. Students appreciated the course material, FAs, and in-class activity.
CONCLUSIONS: Little improvement in a traditional epidemiology course for undergraduates in the form of interactive lectures and formative feedback and providing the student with course material led to significant gains in students' knowledge and attendance. Copyright:

INTRODUCTION

Learning epidemiology improves critical thinking and the problem-solving ability of students. However, medical undergraduates pay less attention to this subject, assuming that skills in this subject are not related to practicing medical service in future.[12] Though there are various methods of teaching epidemiology, it is predominantly taught by traditional lecture system in India.[3] Lectures make student passive recipients of information, leading to poor engagement of the learning process.[4] These things make teaching epidemiology a real challenge to them.

Considering the large intake of students in medical colleges, lectures still continue to be the primary mode of teaching. Hence, it is important to look at ways in which the traditional lectures can become a more effective environment for engaging students in gaining adequate knowledge and skills. Scientific literatures recommend that lectures can be made effective by modifying it following good teaching–learning principles and assessment format.[5] However, there exists a paucity of evidence, especially at local context, showing the combined effect of interactive teaching and formative assessment (FA) on learning epidemiology by medical undergraduates.

In the department of community medicine (DCM), we have been running a course on epidemiology for medical students for the last 4 years. It has been found that nearly 40% of the students successfully complete the course, which was thought to be far from satisfactory. The course was predominantly lecture based, and there was no mechanism to identify and offer assistance to students facing any learning difficulty. Thus, it was decided to improve the existing epidemiology course and to assess its effectiveness in terms of uptake of the course by students.

MATERIALS AND METHODS

Setting and participants

The study was carried out in a private medical college located at the eastern border of Puducherry union territory for a period of 10 months from August 2018 to May 2019. The college admits 150 undergraduate students during every academic year and is affiliated to the Pondicherry University. Epidemiology is a part of medical undergraduate's curriculum which is taught during posting in the DCM. The improved epidemiology course was delivered to 145 sixth-semester students of the academic year 2016, over a period of 3 months from January 2019 to March 2019. Two previous academic batch students of the year 2015 (n = 122) and 2014 (n = 129) for whom epidemiology was taught using traditional lecture formed the historic comparison cohorts.

Design and intervention

It was a triangulation type of mixed-methods evaluation, where both quantitative (quasi-experimental design) and qualitative (open-ended responses) methods were involved simultaneously for evaluating the improved course on epidemiology.[6] Quasi-experimental design was adopted to evaluate the effectiveness of intervention, which was ascertained by the uptake of the epidemiology course by medical undergraduates, and the qualitative component included content analysis of written feedback given by the students about the course. Uptake of the epidemiology course was measured in terms of the students' learning scores and their overall course attendance. Improved lecture system is a package of educational interventions that aimed to improve the learning of epidemiology that includes development of course material, lecture with preplanned interaction in the form of self-check exercises (multiple-choice questions [MCQs] and epidemiological problems), and FA with feedback tailored to every student.

As there were modifications in the traditional teaching–learning format, the proposal was presented to the members of the Medical Education Unit and was initiated after obtaining the clearance from the Research Committee and Ethics Committee (IEC code no. 20/2018). The flow of study procedure for the development, delivery, and evaluation has been explained in Figure 1.

Figure 1

Overview and timeline of the educational project

Development of course and course material

We ensured that the overall course objectives were in alignment with the Pondicherry University curriculum. The intended learning outcomes of each session of the course were in alignment with the overall course objectives. The course content consisted of study design, measures of association, interpretation of findings, and causal models. All the three batches were taught the similar content, however the recent batch additionally received interactive lecture and feedback. The revised course included authentic examples from real worksetting to help them understand its relevance to practice. The course material was written in simple English language, and it was made available to all the students at the beginning of the course. The course book included details on course goals, session objectives, support strategy, and criteria for certification.

Planning and delivery of interactive lectures

Ten classes (2 h/week) were delivered over the period of 3 months. The interactive sessions were facilitated by a team of trained faculty and postgraduates in community medicine. The lesson plan of each session included set induction, recapitulation of the previous session to reactivate their knowledge, and plan for self-check exercises. At the end of each ten interactive lectures, self-check exercises in the form of MCQ of the single response type that test learners remembering and understanding levels were used. Students were encouraged and offered clarifications on concepts, if required.

Formative assessments and feedback

Students were subjected to two written FAs during the course. The question paper consisted of essay-type questions, short-answer questions, and epidemiological exercises of a total of 30 marks. Questions testing all domains of cognition were included. Postgraduates (n = 6) and faculties (n = 5) were trained to give effective feedback. They provided written feedback to all the students stating their strengths and areas for further improvement. This feedback was given within a week of FA. Those students who scored <50% mark were given feedback in a face-to-face meeting based on the Pendleton's model.[7]

The Pendleton method is a simple way to give feedback to learners in a constructive manner.[7] The positives aspects are identified first by the learner, which is followed by the facilitator reinforcing these positives. Then, self-assessment by the learner on what could be done differently is done, and the alternate skills required for improvement are then suggested by the facilitator. Recognizing the positives and avoiding a discussion of weaknesses right at the beginning prevents defensiveness and allows reflective behavior in the learner.

End-of-course assessment

Students' learning of epidemiology was assessed by a written examination conducted at the end of course that was similar to the pattern of FA. Questions testing the factual knowledge, comprehension, reasoning and critical analysis, and synthesis of information related to contents delivered in the course were included. The answer sheets were also checked by the faculty and postgraduates who were not the part of this evaluation study. They assessed the students' responses against the standard answer key. The scores of the current batch of students were compared with the scores of previous batches of students. In order to ensure alignment between course objectives and assessment plan, a test blueprint was prepared.

Evaluation of the course

The evaluation of our improved epidemiology course was carried out to analyze its effectiveness. Because the course was evaluated in terms of learning and as it was backward in nature, we used Kirkpatrick's framework to evaluate it.[8] We used students' end-of-course scores (Level-2) and their feedback given for the course (Level-1) to decide the effectiveness of the course.

Data analysis

SPSS statistical package version 24 (SPSS Inc., Chicago, IL, USA) was used for data analysis. Proportion of students as per se x, grades of end-of-course assessment score, and level of attendance was compared between the present batch of students and the previous batches using Chi-square test. Intention-to-treat analysis was carried out to check the effectiveness of improved course, as all students in the intervention group, not just those who attended all the sessions of the course, were included. The level of significance was set at 5%.

Manual content analysis was carried out. After reading the responses of learners to the open-ended questions, similar responses across the study participants were coded. Next, the codes related to similar areas were clustered together to form the categories. Finally, similar categories were grouped under the selected themes. The guidelines by UCLA Center for Health Policy Research were used for analysis.[9] The “Consolidated Criteria for Reporting Qualitative Research” guidelines have been followed while reporting.[10]

RESULTS

There were 145 students in the present batch (2016) that was exposed to the improved course on epidemiology, of which 44.1% were male and 55.9% were female. In the previous batches, 53.3% and 58.1% were female students of batches 2015 and 2014, respectively. Gender distribution was not statistically significantly different among all the three batches of students (P = 0.74). The result of the present study was described under two levels of Kirkpatrick evaluation model.

Students' reaction (Kirkpatrick level-1)

The content analysis of student's feedback about the course was carried out into two themes, the facilitating factors and the suggestions to improve. Four categories emerged under each theme, namely FAs conducted, course material provided, additional academic support provided, and the quality of teaching methods and media [Table 1].

Table 1

Content analysis of students’ reaction to the course

Categories	Facilitating factors	Suggestions for improvement
Formative assessments	Useful to revise the topics covered (15)	Conduct at more interval (4)
	Discussion on the problems were useful (12)	Lesser portion can be given (2)
	Encourage us to learn (9)	MCQs can be discussed in between the class (5)
	Helpful to score more marks at final exam
	Made us feel confident about the topics
	It keeps us enthusiastic (5)
Course material	Informative and illustrative (23)	Include more exercises to solve (3)
	Easy to understand (22)	Remove high-level contents (2)
	Simple language (28)
	Easy to revise (43)
Additional academic support provided	Helped those who missed to learn (3)	Don’t keep any additional class
	Compelled us to attend the regular class
	Showed your interest in teaching us (2)
The quality of teaching methods and media	Clear message (5)	Include more videos (2)
	Contains all information (4)	Keep it more colorful (2)
	Diagrams were good	Keep class more interactive (2)
	Concepts well explained (8)	Monotonous voice (3)
	Problems solved nicely on blackboard (4)	Conduct quiz in between class (3)
	Your teaching is nice (9)
	Well organized (4)

Figures within bracket indicate number of students who mentioned that feedback. MCQ: Multiple-choice questions

Students' learning (Kirkpatrick level-2)

In the end-of-course assessment, 49% of the present 2016 batch of student's scored ≥70%, whereas only 3.1% and 4.9% of the previous 2014 and 2015 batch students scored ≥70%, respectively. In the current batch, only 18.6% of the students scored <50%, whereas 68.2% of the 2014 batch and 61.5% of the 2015 batch students scored <50% in the end-of-course assessment. The improvement in learning among the current batch of students was statistically significant (P < 0.001). Percentage of students who had ≥90% attendance during the entire course improved in the current batch of students compared to the two previous batches, however this was not statistically significant (P = 0.24).

The median (interquartile range) score of end-of-course assessment of all students in the current batch was 70 (55–80.6) and that of two previous batches namely 2014 and 2015 batch was 31.4 (18.6–51.4) and 36.5 (23–51), respectively. There was significant improvement in the median score of the current batch of students irrespective of gender [Table 2].

Table 2

Gender-wise comparison of median score of students between the previous batches (2014 and 2015) and the current batch (2016)

Gender of students	End-of-course assessment scores in median (IQR)			P#
	2014 batch (n=129)	2015 batch (n=122)	2016 batch (n=145)
Female	42.8 (27.1-57.1)	50 (29-54)	77.5 (65-87.5)	<0.001*
Male	22.1 (16.1-36.4)	29.5 (21-50)	57.5 (50-70)	<0.001*
Total	31.4 (18.6-51.4)	36.5 (23-51)	70 (55-80.6)	<0.001*

#P value based on Kruskal-Wallis test, *Statistically significant. IQR: Interquartile range

DISCUSSION

This basic course in epidemiology for medical undergraduates covered the complete cycle of course planning, designing, implementation, and evaluation. The academic performance of the current batch of learners has shown significant improvement than that of the previous two batches of students. Though there was improvement in the overall course attendance, the difference was not statistically significant compared with two historic groups. The revised course was well received by the students.

The current project demonstrated significant knowledge gains in epidemiology among medical undergraduates. Two aspects of this study are particularly noteworthy. First, nearly 50% of the students scored more than 70% mark at the end-of-course exam. Second, the failure rate had reduced significantly compared with the previous two batches of students. Traditional lecture is effective in the transmission of information to a large group of students, but it has its own demerits namely teacher centeredness, long lecture hours, didactic, and passive learning.[11] We found that little interaction in traditional lectures and ongoing assessment and feedback improved the course uptake among students. These findings were similar to the results of an educational model where traditional lectures were supplemented with cognitively engaging tasks.[12] Interaction promotes student's engagement in the learning process, heightens attention and motivation, gives feedback to the teacher and student, and increases satisfaction for both.[131415] Previous studies showed that the students exposed to formal FA with immediate feedback showed better academic achievement.[16171819] It helps to identify at-risk learners who require additional academic support.[2021]

Multiple assessment methods were used at multiple times in our course. The use of multiple observations and different assessment methods over time can partially compensate for flaws in any one method.[22] The test papers were in alignment with the course objectives. The assessment content and the method chosen need to be in alignment with the learning objectives and activities in order to make the test scores reliable and valid.[2324]

Strengths and limitations

Our educational project, being an improved traditional lecture, had no additional cost involved in designing, implementing, and evaluating the course, thereby making this model feasible and cost-effective. Being an educational research in real practice setting, it was not possible to carry out randomized controlled trial on feasibility and ethical grounds. Hence, the effectiveness of intervention was compared with that of the previous two cohorts of students taught by traditional methods. The previous two batches were taught on the same course content by the same set of faculty and the same course duration and had similar pattern of end-of-course assessment, thus ensuring the comparability. For all the three batches, we had used a test blueprint, a standard answer key for assessment, and the same set of examiners.

CONCLUSIONS AND RECOMMENDATIONS

Thus, we conclude that little improvement in traditional teaching with respect to interaction with students and ongoing assessment can help us better achieve students' learning outcome and satisfaction. This finding is particularly important for resource-poor developing countries where most of the curriculum is delivered in traditional format with limited technical capacity and resources for a major change in curriculum.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.