Impact of immediate feedback on the learning of medical students in pharmacology.

INTRODUCTION: Providing feedback to students is an essential component in medical education and has been shown to improve the students' learning. The purpose of this study is to evaluate the effect of computer-based immediate feedback on the medical students' learning in a pharmacology course.
METHODS: In this prospective intervention study some feedback modules in pharmacology (FMP) were prepared in two topics: the cardiovascular system (CVS) and chemotherapy, using blank templates on "Hot Potatoes" software. The FMP included MC-based questions and two versions were developed: one with feedback (FMP-1) and the other without feedback (FMP-2). The FMP-1 module provided immediate feedback for each option the student chose. The students (n=48) were randomized by computer generated random number table to two groups A and B to receive the module in CVS, i.e., FMP-1 and FMP-2, respectively. A cross-over design was adopted to expose all students to immediate feedback modules. The test scores were compared and feedback was obtained from students and faculty using a validated questionnaire. A focus group discussion was conducted to clarify the issues raised by the students.
RESULTS: The module with immediate feedback was much better appreciated by the students than the module without feedback. The students spent more time on FMP-1 (42±7.00 minutes vs 27±12.36 minutes; p<0.001 in chemotherapy and 40±12.11 minutes vs 24±6.01 minutes; p<0.001 in CVS). However, there was no statistically significant difference in mean test scores. The qualitative data collected provided important information on the value of immediate feedback. The students believed that immediate feedback was an excellent way for self-assessment and improved their deeper understanding of content areas. They also felt that it supplemented their traditional learning habits and stimulated them to read more. The students enjoyed its non-threatening nature.
CONCLUSION: Immediate feedback improved the deeper understanding of pharmacology and its relevance to medicine for the two topics although immediate feedback did not improve test scores. Overall, immediate feedback had a positive impact on the students' self-directed learning.

Introduction

Providing feedback to students is considered to be a central and essential component in medical education (1). The feedback has been shown to improve the students’ performance and learning (2). However, despite the usefulness of feedback, students usually do not receive feedback during their learning process (3-5). A review of educational research by Hattie, et al. (2007) on feedback further substantiates these facts (6). We face a similar situation in India, too. A study by Ram, et al. (2014) highlights the concern about the appropriate use of feedback (7). The written assessment system in undergraduate medical teaching in India is traditional and comprises of long and short answer questions (8). Answer sheets are manually corrected and the marks are posted on the notice board after a varying interval. Rarely students are offered feedback, and when the feedback is provided, it is usually quite late or non-specific, losing its formative value. There is a need to study the effect of feedback on the learning of the students as part of a strategy to improve the learning. To be effective, the feedback should be immediate, structured and related to the task, and provided at the time when students have the opportunity to make use of it (3,9).

Many of these objectives can be achieved by the use of information technology (10). “Hot Potatoes” is one such free software, which can be used not only to provide immediate feedback but also remediation (11). Computer-based MCQs have been used in various scenarios to provide feedback. These studies mainly focused on the module or the acceptability but provided very few MCQs (12,13). Hence, this study was designed to exploit the effect of immediate feedback on the learning of the students using information technology.

Methods

This prospective study enrolled 48 students of second professional year of undergraduate medical course. This study was of a mixed nature having quantitative as well as qualitative data. The study was approved by ethics committee of the institute.

Intervention

Development stage:

Feedback modules in pharmacology (FMP) were prepared in two topics, i.e. the cardiovascular system (CVS) and chemotherapy using blank templates on “Hot Potatoes”. The software (Hot Potatoes), available free online, was used (11). Two faculty orientation sessions were held to sensitize the faculty to the usefulness of feedback and use of hot potatoes software. The FMP were developed by collaborative efforts of the departmental faculty. The FMP included MC-based questions. The MCQs were constructed by the faculty. A pilot testing of modules was done on a few students and feedback was also obtained from the faculty. Item analysis of the MCQs was done using the excel sheet developed for the purpose. The final MCQs were carefully evaluated by the faculty and edited if there was need for it. Two versions of the FMP were developed: one with feedback (FMP-1) and the other without feedback (FMP-2). Each module consisted of 60 MCQs. The FMP-1 module provided immediate feedback for each option as the student clicked, irrespective of whether the right or wrong option was selected. For the right answers, reinforced feedback was provided while for the wrong answers corrective feedback was provided with explanations. These feedback comments pop up immediately when the student clicked any choice (Figure 1). The appropriate suggestions were incorporated in the module. FMP-2 had the same questions and answers, but without the feedback comments.

Figure1

Screen shots of CVS FMP-1 and FMP-2 with pop up FMP-Feedback module in pharmacology; CVS-Cardiovascular

Implementation stage:

After the conventional teaching-learning sessions in CVS, the second professional medical undergraduate students were randomized by computer generated random number table to two groups, A (n=24) and B (n=24), to receive the module in CVS, i.e. FMP-1 and FMP-2, respectively. To take care of ethical concerns, all students were exposed to immediate feedback comments using a cross-over design. They were informed that there was no time limit for attempting these modules and they needed to go through all the four choices. If they hit the right answer in the first attempt, they should then look at alternate options, too. The next module on chemotherapy was administered to the students after they had been taught the same system. Therefore, group A was administered FMP-2 and group B was administered FMP-1 in chemotherapy FMP.

Evaluation

Both groups completed the traditional tests containing short answer questions and MCQs after completing each module. The students’ performance scores in these written examinations after the completion of modules were compared. Feedback was obtained from students and faculty using a validated questionnaire. A five-point Likert scale was used in the questionnaire (14). Semi-structured interviews and surveys were used to evaluate the improvement in the understanding of the need and importance of immediate feedback from faculty (15).

A focus group discussion (FGD) with students (n=6) was conducted to clarify the issues raised by the students. The information obtained from FGD was analyzed and some themes were generated (16). At the end of each FMP session, the students got a cumulative score in that session. These scores were also compared. The emphasis was, however, on immediate feedback. We also recorded the time spent by students in each module.

Statistical analysis

Repeated measure ANOVA with Bonferroni correction was used to compare the scores and the time spent on modules. The carry-over and intervention effects were also evaluated. The p value <0.05 was considered as statistically significant. SPSS 20 was used. The qualitative data analysis was used to analyze the FGD and the interview data. Some common themes were generated from this data.

Results

Figure 2 and 3 show the information on the time spent in CVS and chemotherapy FMPs. The students spent more time on FMP-1 (42±7.00 minutes vs 27±12.36 minutes; p<0.00001 in chemotherapy and 40±12.11 minutes vs 24±6.01 minutes; p<0.00001 in CVS). There was a significant carry-over effect for the time spent in each module with feedback and without feedback (p= <0.001) and there was no intervention effect for the time spent in each module with feedback and without feedback (p=0.203) for both CVS and chemotherapy module.

Figure2

Various parameters (mean value) obtained in CVS FMP FMP-Feedback module in pharmacology; CVS-Cardiovascular; MCQ-Multiple choice question

Figure3

Various parameters (mean value) obtained in chemotherapy FMP FMP-Feedback module in pharmacology; MCQ-Multiple choice question

There was no statistically significant difference in mean test scores as depicted in figure 2. As indicated, the scores of the tests, i.e. short answer questions (p=0.035) as well as MCQs (p=0.530) were not significantly different. There was no carry-over effect for the scores in test papers in each module with feedback and without feedback (p=0.487) but there was a significant intervention effect for the scores in test papers in each module with feedback and without feedback (p=0.003) for both CVS and chemotherapy module. There was no carry-over effect for scores in MCQ tests in each module with feedback and without feedback (p=0.313) and there was no treatment effect for scores in MCQ tests in each module with feedback and without feedback (p=0.313) for both CVS and chemotherapy modules (Figure 3).

The qualitative data collected provided important information about the immediate feedback. Figure 4 depicts the responses of students on the questionnaire. The Likert scale values were consistently in strongly agree/agree part for FMP with feedback. Table 1 shows the major themes that emerged after FGD with students; e.g., the students suggested that the immediate feedback was an excellent way for self-assessment and improved their deeper understanding of content areas. It supplemented their traditional learning habits, stimulated them to read more and the students enjoyed its non-threatening nature. Faculty interviews highlighted the utility of the immediate feedback for the students, faculty learning while making MCQs with feedback and the increased eagerness of students to know the feedback comments.

Figure4

The students’ response of on the modules used FMP-Feedback module in pharmacology

Table 1

Major themes of focus group discussion with students on the modules used

Themes
● Improved conceptual learning and clinical performance. I was able to answer in otolaryngology clinical classes due to FMP.
● It was a quick review of the whole topic.
● It stimulated us to go back and read more and correlate what we leant during the feedback module.
● We have a better understanding of the topic; however, we do not expect our performance scores to improve immediately.
● We remembered better and for a longer time as we ourselves searched for the answers.
● No fear, it was short and crisp. We felt more confident.
● It was fun in learning and we enjoyed it.
● More useful than the conventional feedback which I receive
● Like to get more immediate feedback modules in more topics in pharmacology and other subjects as self-learning modules
● Will recommend immediate feedback modules to other students
● It is a motivating and learning strategy.

FMP-Feedback module in pharmacology

Discussion

Giving feedback is a critical skill for effective teaching and learning, the “heart of medical education” (17). Timely feedback has been shown to motivate the students in the right direction. The module with immediate feedback was much better appreciated by the students than the module without feedback. Although there was no significant difference in the test scores, immediate feedback made learning more acceptable to the students. Qualitative data indicated that immediate feedback improved the deeper understanding of pharmacology and its relevance to medicine although it did not increase the test scores. Earlier studies also indicated that immediate feedback improved qualitative aspects of learning. Even improvement in retention can happen, but it takes a longer time (18,19). It is expected that changes in scores will take time like the present intervention which fostered a habit of using feedback to improve the self-directed learning. In our study, the students even stated that they did not expect their scores to improve immediately after the module.

The students spent more time in the module with feedback comments (FMP-1) indicating that the module did interest them and they were trying to read and reason out the answer. Even, reading the four feedback comments takes time as compared to no comments in FMP-2.

However, students stated that the immediate nature of feedback in the module was thought-provoking and they were stimulated to go back and read. In earlier studies also students did appreciate immediate feedback as more useful (9,19). The students felt that both modules helped them to fill knowledge gaps. The validity of FMP modules can be considered good as the modules were able to measure the knowledge part related to topics. We cannot comment on reliability as this module was used once for the study purpose. Since we advise the use of FMP for formative purposes, a good validity is an important factor here (20).

Limitations of the study might include cross talk amongst students about the module designs. However, the two topics were taught at different times and the students were exposed to the modules at different times allowing a sort of wash out period. Similarly, the modules were administered in the central library of the institute and after that they were not having access to modules. However, there can be a carry-over effect.

Conclusions

The students and faculty felt that immediate feedback helped to learn better. The students and faculty understood the need, importance and usefulness of immediate feedback to improve learning in pharmacology. Overall, immediate feedback had a positive impact on their self-directed learning.