Course quality management based on monitoring by students at a medical school.

PURPOSE: This study aims to develop a system of course monitoring by students and evaluate the course quality management system (CQMS) implemented as an educational assessment tool.
METHODS: This research was conducted in accordance with the ADDIE model which is a well-known instructional design model. The ADDIE process includes needs analysis, design of the course monitoring and course evaluation, development of evaluation forms, implementation of course monitoring, and evaluation of the program.
RESULTS: To meet the need for a system that can replace the traditional lecture evaluation approach, this study developed and implemented a new course evaluation system. In comparison with the quantitative evaluation method, course monitoring by students provided more qualitative information on classes and courses from the students' perspective. The students' realistic description helped know how student felt the atmosphere of class and what kinds of teaching style students preferred. However, some view that the evaluation by the four monitoring members was less reliable.
CONCLUSION: This study proposed a course quality management based on students' monitoring which emphasized the narrative evaluation to help to identify the strengths and weaknesses of the classes and gather qualitative information from the students' perspective that can be used to improve the courses. It is expected that providing the monitoring members with better orientation could help manage the quality of the courses using the monitoring system.

Introduction

Lecture evaluation by students is an important tool to evaluate instructional activities, and it is implemented in all Korean medical schools [1]. However, many studies reported problems of quantitative lecture evaluation and a need for improvement was identified [1-5].

Historically, lecture evaluation began in the United States in the 1950s. At that time, universities only emphasized a professor’s research capability, regarding teaching ability as less important. In response, students called for a need to evaluate lectures. That led to a beginning of teaching evaluation [6]. In Korea, lecture evaluation began to spread at universities as the government announced the “Review of New Educational Reform Plan” in 1995, emphasizing the effectiveness of education [2]. The government desire to measure educational effectiveness through lecture evaluation at universities and began to link the results with financial support to schools. The lecture evaluation system served as a measure of university education evaluation [6]. At this point, the most commonly used evaluation method was the student satisfaction survey.

To meet this demand, the first lecture evaluation at domestic medical schools was conducted in 1990. In 2006, lecture evaluation became a requirement of the assessment and accreditation of medical education [7], and medical schools were required to conduct lecture evaluations. Individual medical schools developed their own lecture evaluation tools or used standardized forms offered by the universities they are part of [2,8]. By 2007, the practice was adopted by all medical schools in Korea [2,9]. However, as the practice of lecture evaluation spread, problems with quantitative evaluation system began to be reported [2-5,10-13]. In research conducted by a medical school to analyze lecture evaluation responses during three academic years, it was found that 20% of the respondents gave identical responses to all survey questions [2]. Another study of students from other majors also reported that over 30% to 50% of students “responded without reading the questions” or “gave identical responses to all questions” [3-5]. Previous studies confirmed that multiple-choice evaluations by all students are only a formality and evaluation results cannot be used as an indicator for lecture improvement.

To solve these problems, some researchers analyzed response patterns or problems of traditional lecture satisfaction surveys and then suggested alternative lecture evaluation methods. The proposed methods include verbal interaction analysis [14], instructional supervision using classroom assessment [15], mid-semester evaluation by students [16], student interviews after the evaluation survey [17], and the “think-aloud interview” [18]. However, these were experiments based on one-time evaluation or methods for small groups. Until now, there have been few studies on proposing course evaluation methods that can be used to obtain qualitative and practical data necessary for improving course quality as well as applied to all courses on a regular basis.

Therefore, this study aims to develop a system of course monitoring by students, analyze the experience of operating, identify its advantages and disadvantages, and propose a course quality management system (CQMS) as a curriculum assessment tool. Specific objectives are as follows. First, this study proposes a course monitoring system by students that can address disadvantages of the existing methods. Second, a CQMS is suggested using the system. Third, this research identifies characteristics of the course monitoring system by students in comparison to the traditional quantitative evaluation and analyzes strengths and weaknesses of the monitoring methods based on implantation experience of this program.

Methods

1.The course monitoring system development process

The course monitoring system was developed in accordance with the ADDIE model which is the same major steps of many educational program development [19]. ADDIE model includes phases of analysis, design, development, implementation, and evaluation. This study followed these development process. In the analysis phase, a needs analysis was carried out to set the program goals. In the design stage, the operation method of course monitoring, the participants and the period of implementation were devised. In the development stage, the assessment form and course monitoring evaluation system were developed. In the implementation stage, the selected monitoring members were trained, courses were monitored, the evaluation results were collected, the course operation reports were completed, and the course review meetings were held. In the evaluation stage, the monitoring method’s applicability was assessed, a comparison was made between the proposed and traditional course evaluation systems, and areas of improvement were identified. Fig. 1 illustrates the process of course monitoring system development. The followings are specific procedures and practices. This study was approved by the Institutional Review Board of Kyungpook National University Industry Foundation (IRB approval no., 2018-0040).

Fig. 1.

The Process of Developing Course Evaluation System Based on Monitoring by Students

2. Analysis

A needs analysis for a new course evaluation method was conducted. Course operation directors were asked to give opinions on problems of current course evaluation practice.

3. Design

One education major and one doctor of medicine drew up a draft based on design principles. The course monitoring operation method, the participants, the period of implementation and the developmental direction of system were decided after medical education office meetings. The system was designed to be easily applicable, give incentives for students, and provide course operation directors with specific information for improvements.

4. Development

The monitoring evaluation form was developed. The course management system through course monitoring was devised.

1) Development of the course monitoring evaluation form

The researcher prepared a draft form through literature review. The evaluation form was finalized after a series of review meetings with two doctors of medicine to ensure content validity.

2) The course evaluation and quality management system

It was developed by referring to the general course procedures suggested in the New Instructional System (NIS) model of the Korean Educational Development Institute (KEDI) [16]. The NIS model is a model developed and verified by the KEDI since the 1970s in order to maximize the productivity and efficacy of teaching. It consists of a system of teaching and a school support system [16]. Here, a system of teaching refers to a whole teaching process model through planning–diagnosis–guidance–development–evaluation. The school support system refers to a series of teaching support activities that support, plan, implement, and evaluate a new class system to function effectively in the teaching practice. Thus, the researcher considered the quality management activities of teaching needed at the medical school. This study schematized the teaching process of the medical school based on the NIS model, and then suggested the supporting activities needed for the quality management of the teaching at each teaching stage. This support activities include course monitoring. The developed system was reviewed by a doctor of medicine and a doctor of education to obtain content validity.

5. Implementation

1) Selecting and managing the monitoring student members

First to fourth-year medical school students have participated in the course monitoring since 2012. Second-year premedical students have joined the student monitors since 2016. All the courses during spring and fall semesters were evaluated. The monitoring members were selected among students who had applied at the beginning of the semester. The student monitors included four students from the class of each school year. In 2012 and 2013, the student members had to monitor the courses on a yearly basis. However, some students suggested that year-long monitoring activities could be too burdensome. For this reason, the monitoring program has been run on a semester basis since 2014. As a compensation, the monitoring students were rewarded with volunteer service points which are one of the graduation requirements. Each semester, 15 points were given to a student. Participants were allowed to serve as monitors for multiple semesters. A student selected as a monitors received a detailed orientation at the beginning of the semester on how to write the course evaluation report and what to be cautious about (Table 1). It was emphasized that the student monitors should not record their own assessment but collect peer students’ opinions. A 1-hour orientation was held during lunchtime. At the meeting for orientation, the monitors of the previous semester also attended to share their experiences with the new monitoring members.

Table 1.

Contents of the Orientation for Monitoring Staff

Contents of the orientation
Describe peer students’ opinions rather than the staff’s personal opinions
Provide detailed description of the class
Describe strengths, weaknesses, characteristics of each professor
Submit the report within a week after each course ends

2) Implementation of the course monitoring system

The monitoring system had been applied to first to fourth-year courses in School of Medicine, Kyungpook National University since 2012. The fourth-year courses were monitored only in the spring semester. Since 2016, the monitoring system has also been applied for second-year premedical courses.

3) Collecting the course evaluation data, and preparing and delivering the reports

The monitoring students prepare and submit the evaluation reports via email to the Department of Medical Education office within a week after the course completed. When all four reports for a course are submitted, opinions of the four monitoring students are combined. At the end of the semester, the materials necessary for the course evaluation are collected in addition to the course monitoring results. In the course operation reports, the analysis of the test item evaluation, the results of lecture satisfaction survey by all students, the evaluation of individual faculty members, the percentage of the non-lecturing class time, and the course monitoring data. The complete course operation reports are delivered to the course operation director and deputy director who decide whether to inform the lecturers of the evaluation results.

4) Course review meetings

Once the course operation report is delivered to the director and deputy director, course review meetings are held at the end of each semester. Two to three courses are reviewed at a meeting during lunchtime. In the meeting, the course director makes general remarks, reflects on the problems of the courses and presents plans for improvement. The meeting also reviews whether improvement plans from previous semesters were successfully implemented.

6. Evaluation

Faculty members who participated in the course evaluations and the program managers share their experiences and opinions. Based on this data, the applicability of these course assessment system using students’ monitoring and teaching quality management system were assessed, and areas of improvement were identified. In addition, a comparison was made between this new evaluation method and the traditional approach.

Results

1. Needs analysis for setting the goals

At School of Medicine, Kyungpook National University, all students evaluate courses immediately provided by the medical school itself after computer-based tests which is method of testing using a computer. Analysis of students’ response patterns indicated that they did not take the questions seriously and gave sloppy answers. It was also found that the students tended to give higher ratings to classes they felt easier to understand. As a result, course directors and lecturers doubted the credibility of the evaluation results and pointed out that the 5-point Likert scale evaluation did not provide qualitative information for lecture improvement. The school authorities concluded that it was the problems that the traditional average-rating-based evaluation method did not provide detailed information on how individual lectures were practically conducted in the classroom and how students felt about them. To solve these problems, it was necessary to develop a new evaluation method and a sustainable teaching quality management system. In response to such expressed and identified needs from school members, School of Medicine, Kyungpook National University decided to enhance the course evaluation to collect data showing improvement of its curriculum. At a workshop in 2012, course operation directors agreed to develop a system of course monitoring by students. Under the system, selected monitoring students conduct evaluations based on criteria to identify the courses’ advantages, problems, and improvements.

2. Operation of the course monitoring system

1) Evaluation form for course monitoring

Two types of the course evaluation form were developed: one for lectures and the other for clinical practices. The course evaluation form consists of 5-point Likert scale questions and open-ended questions. The structured question items provide evaluation categories for respondents when answering the open-ended items. Although the survey has the structured question items, the respondents were guided to offer detailed descriptions of the strengths, weaknesses, overall opinions, and other comments about the courses. The evaluation of lectures included 24 items on syllabus and lecture note (four items), lecture contents (four items), class management (nine items), basic science laboratory class (three items), and examination (four items). The clinical practice evaluation form included items on orientation for clinical practice (three items), practice process (12 items), professor in charge of practice (three items), practice place (seven items), and evaluation method (seven items). The same form as the lecture evaluation was used to evaluate the premedical courses. And then, four items on course outcome and lecture outcomes were added to the survey in 2016. The evaluation form for lectures was modified for assessment of the premedical courses and has been used since 2016. The course evaluation items had been supplemented and added according to the important evaluation agenda of the year.

2) Selecting monitoring students

A total of 131 students participated in the monitoring program from 2012 to 2017. Among them, 44 students served as monitors for more than a semester. Thirty-nine students served 2 semesters. Three students served 3 semesters, and two students served 4 semesters. Out of the 39 students who served 2 semesters, 69% (27 people) participated in the program during the same school year while 31% (12 people) served during different school years.

3) Collecting monitoring data and reporting course management results

Once the courses ended, four evaluation reports from each course were submitted to the Department of Medical Education within a week. However, collecting the evaluation documents was challenging. Some students submitted their overdue reports all together at the end of the semester. To address the issue of late submission, the program adopted a discriminative point policy in 2016. Monitoring members received different volunteer service points depending on their submission statuses at the deadline: 15 points for full submission, 10 points for 66% submission, 5 points for 33% submission. As a result, most evaluation reports were submitted before the deadline (one student’s report was submitted after the deadline).

The lecture monitoring report which sent to the course director at the end of semester includes not only the general evaluation of the classes but also the detailed comments and evaluation of the individual lecturer. Regarding such a detailed evaluation of individual teaching, some professors expressed discomfort, particularly, when their names were exposed. Since it was not appropriate to reveal lecturers’ names on the evaluation data, some course director requested for the evaluation results without professor names. When lecturers’ names were included in the open data, the program managers got phone calls from the lecturers asking why they had received such evaluation results. In this case, the officer in charge deleted the professor names from the report and sent them the data again.

3. Proposal of course quality management system based on a course monitoring by students

This study suggested the CQMS using the course monitoring system. The CQMS is a teaching management system that suggests specific activities to improve the efficiency of teaching in the stages of preparing, proceeding, and evaluating of instruction. The instructional stage adopting in the study was theoretically based on the instructional model developed by NIS. The seven steps presented in NIS model and were adopted as standard instructional procedures in this CQMS. In the CQMS, instruction activities were specifically divided into seven steps: course plan, teaching delivery, assessment of student achievement, course evaluation, feedback on the results of course evaluation, and improvement plan. In each step, this approach identified quality management activities that need support from other departments. Course monitoring is a new method of course evaluation by students. It can be used during the course evaluation step. The monitoring system is one of the quality management activities carried out by students who are consumers of education in the entire CQMS (Fig. 2).

Fig. 2.

Course Quality Management System Using Course Monitoring by Students

4. Comparing traditional course evaluation and course monitoring

The implementation of the course monitoring system indicated the following characteristics compared with the traditional rating evaluation method. In traditional course evaluations, almost all students responded to multiple choice questions. However, the main characteristic of the monitoring method is that a small number of students voluntarily evaluate courses focusing on the open-ended questions rather than multiple-choice items. These characteristics of the new method help to identify special episodes in the class, feelings of the students, the class atmosphere, and other comments without attending the lecture. The traditional evaluation methods mainly use multiple-choice items which are easy to process. However, as the course monitoring system is based on open-ended questions, it consumes more time and human resources to classify and analyze the data manually (Table 2).

Table 2.

Comparison between Traditional Evaluation and Monitoring Evaluation Methods

	Traditional method	Monitoring method
Evaluator	All the students in the class	A few selected students
Evaluation tool	Course evaluation survey	Course evaluation survey
Main evaluation method	Likert scale evaluation	Narrative description
	Description of other opinions	Likert scale evaluation is a supplementary info
Evaluation point	Immediately after the test	Immediately after the class
	Mandatory evaluation before checking grades
Characteristics of the collected data	Students’ satisfaction level on the 5-point Likert scale	Able to identify episodes during the classes
Easy to obtain the "Results"	Able to produce results immediately	Need to organize the responses manually
		Time-consuming
Operation	Easy to operate (computerized operation)	Need time and human resources

1) Strengths and weaknesses of the course monitoring system

As a result of conducting the course monitoring system in practice, the author found that there are several strengths and weaknesses of this system. The first advantage is that the study can obtain qualitative course evaluation data which is difficult to acquire using quantitative approaches. The monitoring reports provided information about why the students like or dislike individual courses. It was also possible to identify the types of ‘good instruction’ students prefer. Another benefit is that courses with high impact on students can be understood in more detail. Students’ descriptions about the courses help understand the atmosphere of the classroom.

The study also found two weaknesses of the system. First, if the orientation message is not delivered well to the monitoring students, the evaluation results may reflect the monitoring students’ personal opinions rather than their peers’. For this reason, the orientation emphasized that the monitoring students should collect peer students’ feedback, not their own. The other one is that some lecturers did not give credit to the monitoring results because only a small number of students participated in the monitoring. When the monitoring data were shared with the lecturers, some professors made requests to increase the number of monitoring members. In particular, when the evaluation showed conflicting opinions among four students on specific matters, the lecturers expressed difficulty interpreting the results. For example, two students marked ‘strongly agree’ on a questionnaire item, but the rest marked ‘strongly disagree.’

In summary, this system also provides opportunities for lecturers to identify classes students favor or elements of good classes through feedback from students despite this weakness. Thus, lecturers can customize classes to match the students’ current levels. A challenge is that it may be difficult for the director/deputy director of this program to open the evaluation data to all the lecturers (Fig. 3).

Fig. 3.

Evaluation of Implementing the Course Monitoring by Students

2) Characteristics of the students’ description in monitoring reports

Analysis of the collected monitoring data provided useful information to improve the quality of teaching. Such information included repeated requests, the students’ preference for teaching methods, comments on the order of courses, redundant contents, the class atmosphere felt by students, levels of tests, and difficulties in class environment (Table 3).

Table 3.

Characteristics of “Course Quality Improvement” Information from Course Monitoring

Characteristic of teaching quality improvement information	Student comments
Repeat every year but no improvement	The letters on the slides are too small
	There are too many lecture slides
Episodes during the class	We never used some textbooks we were told to buy
	There were too many class schedule changes
Teaching method	A short learning objective was presented but it was too simple and vague.
	Please don't schedule a field trip on a day before an examination
	A recap at the end of the class helped to understand
	The lecturer talked too fast to catch up
	The process and structure of the class was impressive
Order of the courses	The timetable has changed so much that it didn't match the original syllabus
	We have good curriculum learning anatomy at the beginning of the semester
	I hope Introduction to Medicine would remain as a semester-long course for a long time, unlike other block courses.
	It is a good idea to place pathology behind required courses like this considering non-medicine majors
	I hope this course would remain as a semester-long course for a long time, unlike other block courses.
Identifying the class atmosphere	The class atmosphere was somewhat distracting as it was the first class this semester. But, I felt comfortable, because the subject was not so tricky
Regarding evaluation	Taking exams on Friday was a good idea. We could focus on the exam and take rest during the weekend. So, everyone was happy about it.

Specifically, the analysis could identify which requests are made every year by students and which ones have not been improved over time. For example, a professor uses lecture slides with letters too small to recognize. But no improvement was made despite repeated requests by students. In contrast, some professors received an excellent evaluation every year. Second, course director and educational office could recognize episodes that occurred in class. It was possible to identify small, class-related episodes such as abrupt cancellation of a lecture, changing class schedule, and missed break time. Third, it was possible to identify teaching methods students prefer. For instance, students offered such opinions as “It was difficult to focus on the lecture because the slides were different from handout materials,” “I prefer video materials,” or “The terms in the handout manual are in both English and Korean. I’m confused which one to memorize.” These comments helped recognize specific needs about the courses. Fourth, we could identify the students’ preferences or difficulties regarding the order of the courses. There were student responses such as, “Clinical diagnosis & radiology was a good starter for the semester,” and “Learning anatomy in the first semester helped motivate me as a medical student.” Fifth, the students’ realistic description helped understand how they felt about the class. For example, students’ accounts like “The class atmosphere was somewhat distracting as it was the first class this semester. But I felt comfortable, because the subject was not so tricky” helped understand the atmosphere of the class. Last, lecturers’ own teaching styles can be identified. Student ratings on lecturers’ voice, slide presentation, and teaching methods helped identify preferred and non-preferred styles.

3) Applicability and limitations

Based on the analysis, the results of the course monitoring have great potential to be used as relevant information for ‘managing teaching quality.’ First, the monitoring system provided the students with a window of opportunity to regularly communicate information about class satisfaction, class difficulty, and test difficulty to the lecturers and the school. In addition, as students are given opportunities to participate in the course development, it is possible to develop a course reflecting their needs. Second, lecturers can establish improvement plans for the following semester, considering detailed monitoring results such as redundant class methods and contents. The course director can indirectly identify classroom atmosphere through data on the class. Last, the school curriculum development and operation department can monitor courses. They can also collect improvement plans and check whether the order of the courses is appropriate. Based on the information, the departments can request that the curriculum should be reorganized or ask the lecturers to improve their courses (Table 4).

Table 4.

Application of Course Monitoring Data by Professor-Faculty-School

Applied by	Applications
Student	Opportunity to evaluate individual classes and professors
	Opportunity to actively participate in developing school curriculum
Faculty	Able to establish improvement plan based on the detailed class monitoring data
	Able to identify classroom atmosphere and teaching method
School	Able to identify unusual episodes that happened during class
	Use as supporting data when requesting the professor to improve the course
	Use as supporting data for improving curriculum

Discussion

All medical schools in Korea are currently conducting course evaluation by students. It is the most primary activity in the course evaluation process. However, the practice is not reliable in use, and this study developed and implemented a new course evaluation method. Many related studies have suggested alternative course evaluation methods to date. Previous research include verbal interaction analysis [14], instructional supervision using classroom assessment [15], mid-semester students’ evaluation [16], a student interview after the evaluation survey [17], and a course evaluation of the “thinkaloud interview” at the University of Massachusetts Medical School, United States [18]. However, these studies were based on one-time evaluation and evaluation for small groups. Therefore, it was unreasonable to apply the methods to all the courses continuously. The course monitoring method is a new case that has not been adopted by other medical schools so far. The course monitoring system was rarely used at the universities both domestically and abroad. According to the course evaluation research by Chae and Kim [2], the compulsory evaluation system should fully reflect the purpose of quality improvement, the results should be continuously fed into the quality improvement process, and there need to be meetings for reporting the results or improvements to help lecturers improve their classes. The suggested course evaluation model based on students’ monitoring can meet the existing demands for evaluation methods to improve course quality. Because both quantitative and qualitative evaluations are conducted, this method allows lecturers to gather concrete data to improve their courses. This system may be able to help other medical schools to gain course evaluation results including detailed opinions of students, which are hard to obtain through quantitative methods. Compared with the traditional quantitative method, the course monitoring by students can help gather more useful data to manage the quality of courses. However, there is also a negative view that an evaluation by four monitoring members is less reliable. Therefore, it is necessary to prepare complementary measures.

Such measures can include increasing monitoring members, strengthening orientation for the monitoring personnel, and extending the scope and depth of the narrative evaluation. Course evaluation methods currently used by medical schools have limitations. They are less applicable to improving the class and do not provide rich information necessary to boost course quality. This study is meaningful in that it suggests a course quality improvement system and a new course evaluation method applicable at medical schools. Future studies will have to analyze the evaluation descriptions in more detail to identify such information as what roles students expect their lecturers to play or what students think constitutes a good class. Based on these results, more advanced course management methods suitable for medical schools may be proposed.