Summative program evaluation of a Student-Led Seminar Series in the subject of physiology: an outcome-based study.

PURPOSE: Active involvement of students in their learning process is a recommended andragogical approach to increase student engagement. Many new teaching-learning strategies based on active learning have been implemented, but their efficacies to achieve the proposed benefits of attaining knowledge, skills, and attitude have not been evaluated, especially in the field of medical education. We substituted passive learning in the conventional tutorial classes with an active-learning strategy of Student-Led Seminar Series (SLSS) in the subject of physiology over 4 months and performed program evaluation for the SLSS.
METHODS: Sixty-four first-year medical undergraduate students volunteered to participate, who were divided into groups to present seminars on the allocated topics under the guidance of a mentor. At the end of 4 months, program evaluation was done using Kirkpatrick's model of evaluation-levels 1 and 2, which correspond to reaction and learning, respectively.
RESULTS: Statistically significant improvement was observed in students' satisfaction, and the self-perceived increase was observed in knowledge, skills, and attitude.
CONCLUSION: Program evaluation of SLSS not only established the significant impact of SLSS as an andragogical approach but also helped us in the improvisation of the program for the next cycle.

Introduction

Methods of teaching other than conventional methods of teaching have shown promising results with an increased level of intellectual performance shown by the students in recent times. Various studies have argued that traditional teaching methods lead to a lack of critical thinking and problem-solving skills which could be enhanced by teaching methods having active involvement of students [1].

The benefits of active learning outweigh those of being a passive recipient of learning instructions; therefore, educators must strive to incorporate activelearning strategies within the class to ascertain a grasping environment [2].

Consequently, many student-centric activities (programs) implemented so far have followed the evidence-based approach for active learning [3]. Very few studies have been conducted to evaluate the effectiveness of active learning strategies for improved outcomes [4,5]. Often, the evaluation of these learning strategies is based on student feedback which corresponds to level 1 of Kirkpatrick’s model of evaluation, while there are limited studies that evaluate change in learning, behavior, or results (corresponding to levels 2, 3, and 4 of the Kirkpatrick’s model, respectively) [6].

Further, it is also known that evaluating a technique only based on post-session feedback leaves little scope for quality improvement. Hence, evaluating the impact and effectiveness of any novel approach in the teaching-learning methodology is essential to identify its strengths and weaknesses, so that the necessary improvements can be made [7].

The present study was focused on examining the Student-Led Seminar Series (SLSS) as an educational intervention which inherently, by design, warrants active involvement of the targeted students. The basic premise for initiating such activity at Padmashree Dr D Y Patil Medical College was the dismal trend in students’ attendance in the conventional tutorial classes held during the previous academic year, which employed a passive teaching approach. Therefore, it was decided to substitute passive teaching tutorials with SLSS to promote the active-learning andragogical approach.

Additionally, it was proposed that the effectiveness of SLSS should be assessed by program evaluation using Kirkpatrick’s model of evaluation. Program evaluation in educational settings determines whether the program fulfills its proposed goals and objectives. Thus, the aim of our study was to determine the level of student satisfaction with SLSS and students’ perceptions of the role of SLSS in the acquisition of relevant knowledge, generic skills, and attitudes. Specific evaluation questions that were addressed during program evaluation are: (1) Do students feel satisfied with the SLSS? and (2) Does SLSS help in the self-perceived acquisition of knowledge, skills, and attitude?

Methods

1. Ethics statement

Ethical approval was sought from the Institutional Ethics Committee of Padmashree Dr D Y Patil Medical College (PDYPMC/Ethics/May-06/2016). All the class students were provided with a participant information sheet and informed consent form.

2. Study design and participants

This educational research was conducted at the Department of Physiology, Padmashree Dr. D Y Patil Medical College, Navi Mumbai, India over a period of 8 months (4 months for program administration, 4 months for program evaluation using Kirkpatrick’s model). Out of a batch of 150 first-year medical undergraduates (MBBS [Bachelor of Medicine, Bachelor of Surgery] students), 64 volunteers participated in the study. Other students (non-participants) only attended the seminar series that was presented by the participating students.

3. Educational intervention: Student-Led Seminar Series

This study for evaluating the efficiency of SLSS was conducted along with the progression of the curriculum when a few topics were already covered. An announcement was made in the class regarding SLSS; the topics were announced, and students were encouraged to participate voluntarily. Using audio-visual aids for the presentation was at the discretion of the presenting student. One faculty was designated as a mentor for the participating students.

The whole batch (n=150) was divided into three batches of 50 students each. The seminars were scheduled on Tuesdays, Thursdays, and Saturdays, with one batch presenting each day. Topics from nine main systems were considered for SLSS. Each system was divided into sub-topics and each sub-topic was presented by a group of 7–8 students; in the case of more than eight volunteer students, two groups were formed. All groups presented their seminar after discussion with the designated mentor and each topic was further subdivided within the group. Other students of the batch who did not volunteer for SLSS attended the seminar presentation with all other faculty members. A tabular representation of the implementation of SLSS is given in Table 1.

Table 1.

Tabular Representation of the Implementation of SLSS

	1st month				2nd month					3rd month				4th month
	Week 1	Week 2	Week 3	Week 4	Week 1	Week 2	Week 3	Week 4	Week 5	Week 1	Week 2	Week 3	Week 4	Week 1	Week 2	Week 3	Week 4	Week 5
Activity	A		A		A		A		A		A		A		A		A
	M	SLSS	M	SLSS	M	SLSS	M	SLSS	M	SLSS	M	SLSS	M	SLSS	M	SLSS	M	SLSS
System	1st		2nd		3rd		4th		5th		6th		7th		8th		9th
Sub-topics/system	1		3		3		3		3		3		3		3		3

SLSS: Student-Led Seminar Series, A: Announcement of the topic in the class, M: Mentoring of the students.

4. Program evaluation

The SLSS program was evaluated using levels 1 and 2 of Kirkpatrick’s model of evaluation which is a four-step evaluation model for evaluating and improving the quality and efficiency of the implemented educational programs [8]. Various methods are employed to evaluate the four levels viz: level 1 (reaction)—such as using a questionnaire survey immediately after attendance; level 2 (learning)—such as evaluation of learners’ achievement by written tests or reports; level 3 (behavior)—evaluation of behavior change, such as through interview with students; level 4 (results)—using an index of four grades for evaluating the degree of improvement in learners and workplace performance after taking a training course [6].

In our study, level 1 was evaluated using the administration of a 5-point Likert questionnaire to evaluate the students’ satisfaction with the SLSS [9] (Table 2). Level 2 was evaluated using a 5-point Likert questionnaire regarding the self-perceived acquisition of knowledge, skills, and attitude. Both the questionnaires had responses ranging from “strongly disagree” marked as 1 to “strongly agree” marked as 5 [10] (Table 3).

Table 2.

Items of Questionnaire 1 [9]

Item no.	Items of questionnaire 1
1.	Seminars were useful.
2.	Seminars were enjoyable.
3.	Seminars improved my knowledge.
4.	I was satisfied with the seminars.
5.	I learnt better by seminar series.
6.	It helped to improve my problem-solving skills.
7.	It motivated me to pay attention better.
8.	It stimulated my thinking process.
9.	I think it is feasible to incorporate seminar series for the whole curriculum.
10.	I recommend these series for other classes.

Table 3.

Items of Questionnaire 2 (Knowledge Related [n=8], Skill Related [n=4], and Attitude Related [n=3]) [10]

Item no.	Category	Items of questionnaire 2
1.	K	Seminars helped me better understand the topics covered.
2.	K	Seminar has made me a self-directed learner.
3.	K	I can better link knowledge of basic physiology with applied physiology.
4.	K	I can integrate my prior knowledge in the context of the current topic.
5.	K	I can evaluate the information that I have collected related to a topic.
6.	K	This form of learning encourages the development of my decision-making skills.
7.	S	I have learnt to voice my ideas effectively to the group.
8.	S	Seminar gives me the opportunity to improve my leadership skills.
9.	K	Seminar improves my information processing skills.
10.	S	I am able to communicate effectively to the group.
11.	S	Seminar helps me to participate without always having to lead.
12.	A	I have learnt to respect others’ views within the group.
13.	A	I am able to identify my ethical and moral obligations to other group members.
14.	A	I have become aware of my limitations while functioning in a seminar group.
15.	K	I have learnt to critically analyze information brought by other group members for discussion.

K: Knowledge related, S: Skill related, A: Attitude related.

The authors decided to consider the outcome as favorable for both the questionnaires if at least 75% of the students respond positively to the elements in the questionnaire i.e., Likert score ≥4.

5. Statistical analysis

Frequency distribution of responses to different questions related to satisfaction with the SLSS, and self-perceived increase in knowledge, generic skills, and attitude were obtained. A null hypothesis of 50% correct response (p=0.5) to each question was considered and tested using the exact binomial test. The p-values corresponding to each question indicated rejection of the null hypothesis, showing that the proportion of correctness of responses was significantly higher. All analyses were performed using IBM SPSS ver. 20.0 (IBM Corp, Armonk, USA) and the statistical significance was tested at a 5% level.

Results

1. Students’ responses to different questions related to satisfaction with SLSS

Descriptive statistics for responses of participating students to questionnaire 1 are depicted in Table 4. The students’ responses related to satisfaction with SLSS showed that the mean score for question 1 was found to be <4 (3.89±0.87), and for the remaining questions (question no. 2 to 9), the mean scores were found to be >4. This suggested a good level of agreement among students towards the questions.

Table 4.

Descriptive Statistics for Responses of Participating Students to Different Questions Related to Student Satisfaction with Seminar (Questionnaire 1)

Question no.	Strongly disagree (1)	Disagree (2)	Neutral (3)	Agree (4)	Strongly agree (5)	Mean±SD	Response		p-value[a)]
							Unfavorable (Likert score 1, 2, 3)	Favorable (Likert score 4, 5)
1.	1	2	20	21	20	3.89±0.87	23 (35.9)	41 (64.1)	0.033
2.	1	1	12	25	25	4.12±0.76	14 (21.9)	50 (78.1)	<0.0001
3.	1	2	6	25	30	4.26±0.76	9 (14.1)	55 (85.9)	<0.0001
4.	0	0	9	20	35	4.41±0.52	9 (14.1)	55 (85.9)	<0.0001
5.	0	2	2	30	30	4.37±0.48	4 (6.3)	60 (93.8)	<0.0001
6.	1	0	7	20	36	4.41±0.65	8 (12.5)	56 (87.5)	<0.0001
7.	0	2	9	25	28	4.23±0.65	11 (17.2)	53 (82.8)	<0.0001
8.	2	0	10	22	30	4.22±0.86	12 (18.8)	52 (81.3)	<0.0001
9.	0	2	12	20	30	4.22±0.73	14 (6.3)	50 (78.1)	<0.0001
10.	1	0	3	25	35	4.45±0.53	4 (6.3)	60 (93.8)	<0.0001

Data are presented as number, mean±SD, or number (%).

SD: Standard deviation.

Obtained using exact binomial test.

2. Student’s response to a self-perceived increase in knowledge, generic skills, and attitude

Descriptive statistics for responses of participating students to questionnaire 2 are depicted in Table 5. In the case of a self-perceived increase in knowledge, generic skills, and attitude, the overall mean score across questions was 4.30±0.10, suggesting that the students agreed with the benefits of SLSS. The questions were categorized according to knowledge, skills, and attitude. The mean score for knowledge-related questions was 4.28±0.1, for skills questions was 4.25±0.07 and for attitude was 4.36±0.06. Overall, there was good agreement among students regarding the benefits of SLSS.

Table 5.

Descriptive Statistics for Responses of Participating Students to Different Questions Related to Self-Perceived Increase in Knowledge, Generic Skills, and Attitude (Questionnaire 2)

Question no.	Strongly disagree (1)	Disagree (2)	Neutral (3)	Agree (4)	Strongly agree (5)	Mean±SD	Response		p-value[a)]
							Unfavorable (Likert score 1, 2, 3)	Favorable (Likert score 4, 5)
Knowledge						4.28±0.1
K1	0	1	7	26	30	4.33±0.53	8 (12.5)	56 (87.5)	<0.0001
K2	1	2	5	25	31	4.29±0.74	8 (12.5)	56 (87.5)	<0.0001
K3	1	1	8	30	24	4.17±0.67	10 (15.6)	54 (84.4)	<0.0001
K4	0	0	10	29	25	4.23±0.49	10 (15.6)	54 (84.4)	<0.0001
K5	0	1	3	25	35	4.47±0.44	4 (6.3)	60 (93.8)	<0.0001
K6	1	2	6	15	40	4.42±0.81	9 (14.1)	55 (85.9)	<0.0001
K9	1	3	4	27	29	4.25±0.78	8 (12.5)	56 (87.5)	<0.0001
K15	2	3	7	25	27	4.12±0.98	12 (18.8)	52 (81.3)	<0.0001
Skills						4.25±0.07
S7	0	3	8	16	37	4.36±0.76	11 (17.2)	53 (82.8)	<0.0001
S8	1	1	5	25	32	4.34±0.66	7 (10.9)	57 (89.1)	<0.0001
S10	0	0	8	28	28	4.31±0.46	8 (12.5)	56 (87.5)	<0.0001
S11	1	0	10	11	42	4.45±0.75	11 (17.2)	53 (82.8)	<0.0001
Attitude						4.36±0.06
A12	1	2	7	19	35	4.33±0.81	10 (15.6)	54 (84.4)	<0.0001
A13	0	2	9	28	25	4.18±0.62	11 (17.2)	53 (82.8)	<0.0001
A14	1	1	9	24	29	4.23±0.74	11 (17.2)	53 (82.8)	<0.0001
Total mean score of K, S, A						4.30±0.10

Data are presented as number, mean±SD, or number (%).

SD: Standard deviation, K: Knowledge, S: Generic skill, A: Attitude.

Obtained using exact binomial test.

Discussion

Herein, this study we had implemented SLSS to increase the students’ engagement in their studies. The participating students used different ideas other than the conventional PowerPoint presentation to present their seminars. One group of students used flashcards to denote various components of the cardiac conducting system while explaining each component. Another group depicted the cardiac cycle in a very explanatory manner by using the idea of a human formation to show the atria and the ventricles. Two students held a red-colored cloth to depict the blood flow and two students represented valves while clapping of hands depicted the valvular function. To explain the mechanism of breathing, one group of students made a model using a rubber sheet to depict the diaphragm movement, besides other materials like thermocol and cardboard. Many students employed various other novel audio-visual mediums like Prezi presentation software and animations for making presentations. The results of the program evaluation conducted at the end of the seminar series were reflective of the success of the SLSS.

Our findings of level 1 of Kirkpatrick’s model showed that the students found SLSS useful in improving the knowledge and problem-solving skills of the students. These seminars were found to be enjoyable, provided better understanding of the subject to the students and students were more attentive towards this mode of study. Similar to our findings, a study conducted at the Department of Community Medicine, Jawaharlal Institute of Postgraduate Medical Education and Research, Puducherry assessed the effect of implementing the “student-centered learning” methodology by the reaction level of Kirkpatrick’s training evaluation model using a rating scale [9]. The majority of students found the sessions on innovative teaching-learning and assessment techniques enjoyable, useful, and informative. Also, students actively participated in curriculum planning, execution, and evaluation. Likewise, another study evaluated the effectiveness of modified seminars as a teaching-learning method in Pharmacology through feedback for traditional seminars, so that new interventions in the form of modified seminars can be introduced and compared. Participants of this study opined that interaction was the most important component of a seminar which was also in line with the findings of our study [2].

A few studies have evaluated student-led seminars emphasizing the active-learning component. In our study also, students agreed to the fact that their active participation in their own learning, process helped them to develop better information processing and analytical skills. Students confirmed in their replies to the questionnaire that they became a self-directed learner and that this form of learning improved their decision-making skills. A similar study was conducted to investigate the effect of active learning among first-semester general biology students by implementing a flipped active-learning model in the course. It was observed that flipped active-learning pedagogy was a more effective teaching method as compared to traditional didactic methods for teaching general biology concepts [11].

Our findings of level 2 of Kirkpatrcik’s model showed that students who participated in SLSS confirmed that this mode of learning helped them to voice their ideas effectively to the group and increased their acceptance towards ideas/views presented within the group. While there are very few similar studies which evaluated the SLSS beyond level 1 of Kirkpatrick’s model, we found an interesting study in concordance to our study which evaluated a pedagogical intervention, the student-led tutorial (SLT), using a mixed-methods approach for the data collection. This was implemented within a first-year undergraduate university module based on the concept of tutor-less tutorials and presents a platform for student learning which was designed to enhance active learning. The findings demonstrated that SLTs can potentially facilitate active learning and aids comprehension and understanding. Students particularly value the social aspect of the SLTs, which enables extended peer-to-peer interaction. The results suggested that students develop a sense of responsibility and ownership of their learning [12].

Similar to our findings of level 2 of Kirkpatrick’s model, a study conducted for assessing the perception of the medical students about seminars in the curriculum compared to conventional teaching-learning activity showed seminars to be more effective in a deep understanding of the subject along with improving self-confidence and communication skills among the students [13].

The present study had few limitations. One limitation was that no control group was considered to compare the satisfaction of students who attended SLSS. Since, we assessed the satisfaction of students who attended the seminar; the control group who did not attend the seminar would have given no relevant information for comparison as assessment was based on satisfaction/reaction of the student after attending SLSS, so we did not take any control group. Similarly, while assessing the self-perceived increase in knowledge after attending SLSS, a control group that did not attend the seminar would not have added any relevant information or comparison. Had we followed these students up to their second year and documented their attendance, then this could have been a level 3 evaluation as per Kirkpatrick’s model, which registers a change in behavior.

In conclusion, it is important to conduct a program evaluation of the new teaching-learning strategy so that feedback can assist the stakeholders to modify and accommodate the necessary changes. The program evaluation of SLSS established the impact of the implementation of active learning strategies in increasing the engagement of students in their learning process. It also resulted in the incorporation of the lessons learned to improve the quality of the teaching-learning strategy in the next cycle. It was decided that SLSS should start at the beginning of the next academic year and should continue throughout the year, unlike for a short period as in our study. A sensitization and orientation programme for the students can be conducted at the beginning of the academic year to increase the participation of students. It is worthwhile to plan the evaluation of level 3 by following up with the same set of students in their next academic year.