Evaluation of Medical Humanities Course in College of Medicine Using the Context, Input, Process, and Product Evaluation Model.

BACKGROUND: Seoul National University College of Medicine has provided a new educational curriculum on basis of the competency-based curriculum since 2016. The new curriculum included the medical humanities course (MHC) to potentiate humanities of medical students. The present study applied the context, input, process and product (CIPP) evaluation model to the MHC in order to confirm the feasibility of the CIPP model and to improve the MHC by questionnaire survey and analysis of teaching materials.
METHODS: This study analyzed the MHC provided to the freshmen in 2016 and to the freshmen and sophomores in 2017 by the CIPP model. Firstly, evaluation criteria and indicators were developed according to the CIPP classification. The materials collected from the course were analyzed by quantitative and qualitative analysis according to the evaluation criteria. In the quantitative analysis, an independent sample t-test was performed to verify the difference in the responses between the students (n = 522) and the professors (n = 22). In addition, content analysis was conducted for qualitative evaluation.
RESULTS: There were significant differences in perceptions of MHC between students and professors about the results of almost all objective survey questions through the t-test, such as score 3.64 in students and 4.48 in professors in response to the item of 'provision of appropriate feedback.' As a result of the content analysis, 7 categories and 20 subcategories were derived. There were the most responses to various instructional methods (students, 20%; professors, 21.5%).
CONCLUSION: The CIPP evaluation model was acceptable for the MHC analysis. The first task is to raise students' awareness of the MHC in order to improve the MHC quality.

INTRODUCTION

The importance of improving the quality of education cannot be overemphasized to help medical school students be equipped with the various capabilities required by the society.

For this purpose, the Seoul National University College of Medicine (SNUCM) has provided a new educational course on basis of the competency-based curriculum since 2016. Currently, the competency-based curriculum is at the center of medical education.123 In addition, it is important to evaluate the new educational course in order to identify its impacts and the pros and cons of the education course for quality education.4

Previous studies were conducted to evaluate and improve education in the field of medicine.45678 However, most of the studies have been conducted to verify partial appropriateness without a systematic framework.

The context, input, process and product (CIPP) evaluation model has been introduced in many references of education evaluation as a recommendable methodology.9 This model offers a systematic analysis of education, and above all, allows educators to review the overall process of education for decision-making, including various and comprehensive indices in the individual phases of evaluation.10

The CIPP evaluation model consists of four sections: context, input, process, and product. The context section provides the grounds for determining the goals of a program. The input section provides the information needed to determine the approaches to accomplish the goals and to utilize the resources. The process section is to monitor the process and provides the information that helps to identify problems. Lastly, the product section is to accomplish the goals and to measure the product.1112

In the present study, the new Medical Humanities Course (MHC) provided at the SNUCM was evaluated using the CIPP evaluation model to identify the areas requiring improvement and support. In addition, the usefulness of the CIPP model was reconfirmed and the model was applied to the education course of the SNUCM to provide a case of applying the CIPP model, the most representative model in education evaluation, to the field of medical education.

METHODS

In this study, the MHC provided at the SNUCM in 2016 and 2017 was analyzed. In 2016, the MHCs 1 and 2 were provided to the freshmen. In 2017, the MHCs 1 to 4 were provided to the freshmen and sophomores. Each of the MHCs includes 24 hours of education.

The educational objectives of MHC are cultivating the essential knowledge and attitudes necessary for developing as medical personnel in the global society. This education aims to make learners achieve graduate competency in communication, medical history, medical ethics, doctor and society, self-development and leadership, disease prevention, environmental medicine, and international health. These subjects have one credit and 24 hours of lectures, one time per semester. The MHC 1 teaches understanding medicine and humanity, MHC 2 introduces medical history and medical ethics, MHC 3 focuses on health and society, and MHC 4 includes evidence-based medicine and medical ethics.

This new curriculum was implemented in SNUCM; MHCs 1 and 2 for freshmen in 2016, MHCs 1 and 2 for freshmen and MHCs 3 and 4 for sophomores in 2017.

The research process was as follows; Development of CIPP evaluation indices, collection of materials relevant to educational course, and analysis of individual CIPP sections.

Development of CIPP evaluation indices

The educational goals, design principles, and evaluation criteria of the SNUCM were classified in accordance with the CIPP steps. Among the criteria generally employed by the CIPP evaluation model for education evaluation,1112 the detailed CIPP criteria suitable for the evaluation of the MHC of the SNUCM were selected as shown in Table 1. Two researchers, who are experts in education economics (over 10 years of experience in medical education) and human resources development (about 2 years of experience in medical education), participated in the selection of the criteria.

Table 1

CCIP evaluation indices and evaluation materials for MHC of the SNUCM

CIPP classification	Detailed CIPP criteria	Focuses of curriculum of SNUCM (evaluation criteria)	Materials for analysis
Context evaluation	Have the learning goals of the course been prepared well?	Consistency with the education goals of the SNUCM	Syllabus
			Meeting minutes
			Curriculum
			FGI
	Do the learning goals well reflect the objectives of the classes or the demand of the students?	Reflection of students' demand	Survey
Input evaluation	What are the available input resources? (human and material resources)	Participation by professors	Time table
			Survey
		Learning support for students	FGI
	How is the educational environment? (facilities, apparatuses, etc.)	Formation of learning environment	Meeting minutes
Process evaluation	Has the program (class) been implemented as planned?	Increase of students' participation in classes	Survey
	How do the students evaluate the program?	Extension of small group-based learning	Meeting minutes
		Utilization of various student rating methods depending on their capabilities	FGI
		Increase of feedback to students	Syllabus
Product evaluation	Have the initial goals been accomplished?	Accomplishment of learning goals for each course (subject)	Survey
			Meeting minutes
			Grades
			FGI

CIPP = context, input, process, and product, MHC = Medical Humanities Course, SNUCM = Seoul National University College of Medicine, FGI =focus group interview.

In the context evaluation, the questions, “Have the learning goals of the course been prepared well?” and “Do the learning goals well reflect the objectives of the classes or the demand of the students?,” were respectively substituted for the curriculum evaluation criteria of the SNUCM, which were “consistency with the education goals of the SNUCM” and “Reflection of students' demand,” to be used as the evaluation indices. In the input evaluation, the questions, “What are the available input resources?” and “What are the approaches to the accomplishment of the educational goals?,” were respectively substituted to be used as the evaluation indices for the evaluation of the “participation by professors,” “learning support for students,” and “formation of learning environment,” which are emphasized in the curriculum of the SNUCM. In the process evaluation, the questions, “Has the program (class) been implemented as planned?” and “How do the students evaluate the program?,” were used as the evaluation indices for the “increase of students' participation in classes,” “extension of small group-based learning,” “utilization of various student rating methods depending on their capabilities,” and “increase of feedback to students,” which are the curriculum evaluation criteria of the SNUCM. In the product evaluation, the question, “Have the initial goals been accomplished?,” was used as an evaluation index for the “accomplishment of learning goals for each course (subject),” which is one of the curriculum evaluation criteria of the SNUCM.

Collection of materials related to educational course

To evaluate the MHC, the course materials, including the time table, syllabus, meeting minutes, focus group interview (FGI), and grading, were collected as shown in Table 1 before, during, and after the educational course.

Among the materials, the survey questionnaire consisted of questions based on the CIPP evaluation indices. The questionnaire included questions asking about satisfaction and anticipation regarding the educational course, appropriateness of educational methods, close connection between classes, encouragement of positive participation of students, provision of feedback, and degree of satisfaction (Table 2). The questions included multiple choice questions based on Likert six-point scale and four-point scale as well as essay form questions. The questionnaire survey was performed with the students and professors of the SNUCM participating in the MHC. The survey was carried out through e-mail on the last day of the educational course with the students and after the completion of the educational course with the professors.

Table 2

The structure of the survey regarding whether the education went well or not organization of questionnaire

Section	Questions	Likert scale
Context	1. Satisfaction of anticipation about the educational course	1–6
Input	2. Appropriateness of educational methods	1–4
	3. Close connection between classes	1–6
Process	4. Encouragement of positive participation of students	1–6
	5. Provision of appropriate feedback	1–6
Product	6. Degree of satisfaction to the educational course	1–6
Etc.	7. Opinion about the whole educational course of MHC	Descriptive item

The survey for evaluating students' and professors' perception of whether the education went well or not had six objective items and one descriptive item. The contents of the evaluation were as follows: item number 1 asked how far the expectation and reality of the educational courses are aligned; number 2 asked whether the methods of instruction were appropriate; number 3 asked whether each class of MHC classes was closely connected; number 4 asked if the instructors encouraged voluntary participation from students; number 5 asked whether appropriate feedback of discussion and inquiry was provided to students; number 6 asked about the students' satisfaction with the whole educational course of MHC. A four-point Likert scale (1, very relevant to 4, very irrelevant) was used at number 3 which asked about the relevance of the educational method. The other questions, from number 1 to number 6 except number 3, used a six-point Likert scale (1, strongly disagree to 6, strongly agree). The respondents were asked to write their opinions about the construction of the class, its contents, and connectivity for the descriptive item, number 7 freely.

Every item was developed with each component of CIPP due to fulfilling the validity of survey items. When the researchers had different ideas about developing items, they reached an agreement by repetitive discussion. The reliability of the surveys, measured by Cronbach α, was 0.800 to 0.871.

The subjects of the survey included 87 students and 8 professors participating in the MHC 1 and 100 students and 4 professors in the MHC 2 in 2016, and 106 students and 4 professors in the MHC 1, 122 students and 3 professors in the MHC 3, and 109 students and 3 professors in the MHC 4 in 2017.

The evaluation procedure of MHC 2 conducted in 2017 was simplified due to internal affairs. Even though the survey was not conducted, students' opinions were researched by deep FGI as a supplement. More FGI participants were recruited to supplement the questionnaire that was not conducted in MHC 2 in 2017. As a result, the FGI of MHC 2 in 2017 was conducted with eight participants. (Each MHC, except for MHC 2 in 2017, performed FGI with 6 participants.) In addition, the FGI of 2017 MHC 2 was performed an hour longer than the FGI of other MHCs.

Every FGI (2016 MHC 1-MHC 2 in 2016, MHC 1-MHC 4 in 2017) was conducted with students after credit handling was complete. For recruiting panels for the FGI, after the purpose of the study, getting students' opinion for evaluating the curriculum of MHC, was explained to students by the research assistants unrelated to students' credits, the applications of those who wanted to apply were accepted. The researchers proceeded with the FGI. Six to eight students participated at once, and it ran for two to three hours. For more systematic analysis, semi-structured questions were used as the contents of the Supplementary Table 1. Students confirmed the organized FGI data as correct and as they intended.

Analysis of collected materials

Collected quantitative data were analyzed as follows: the results obtained from the multiple-choice questions were analyzed using the SPSS software program (version 23.0, IBM Corp, Armonk, NY, USA). An independent sample t-test was performed to verify the difference in the responses between the students and the professors.

Collected quality data were analyzed by content analysis method. Data were analyzed with the standard of CIPP as in Table 1. Data were read repeatedly, and the subcategories were determined again after consultation with researchers at every step due to fulfilling to ensure the validity of the categories. The success and failure of content analysis depends on whether the categories are clearly defined and whether the content is classified for problem identification.13 Therefore, we carefully defined the qualitative data categories through discussions to find the implicit significance of qualitative data. Students' data were analyzed by their responses to descriptive item surveys and FGI results, and professors' data were analyzed by their responses to descriptive item surveys and meeting minutes.

The other types of qualitative data, such as educational competency, syllabus, time table and grading on a curve, were analyzed after verifying which part of the CIPP criteria those matters related. Moreover, educational competency was studied by contents analysis, the syllabus was studied by the number of objectives and evaluation method, and the number of subjects was studied by the time table.

Ethics statement

This study used results of the surveys and interviews conducted after the MHC were completed to evaluate the curriculum. The study was approved by the Institutional Review Board (IRB) of Seoul National University Hospital (IRB No. C-1901-044-1001). In addition, the IRB of Seoul National University College of Medicine and Seoul National University Hospital authorized the exemption of the informed consent of this study since it was conducted to evaluate the educational curriculum.

RESULTS

Quantitative evaluation

Objective questions response

Of the total 750 students who participated in the MHC, 522 responded (response rate 69.6%) and 22 of 56 professors responded (response rate 39.3%). In detail, 186 of 294 students who participated in the MHC in 2016 (63.3%) responded, and 12 of 31 professors responded (38.2%). In addition, 336 of 456 students who participated in the MHC in 2017 (73.7%), 10 of 25 professors responded (47.2%). The results of the survey are summarized in Table 3.

Table 3

Results of the surveys by year

Items	2016, mean ± SD				2017, mean ± SD				Both 2016 and 2017, mean ± SD
	Students (n = 186)	Professors (n = 12)	t-value	P value	Students (n = 336)	Professors (n = 10)	t-value	P value	Students (n = 522)	Professors (n = 22)	t-value	P value
Context; satisfaction of anticipation about the educational course (6-point Likert scale)	3.44 ± 1.52	4.50 ± 1.38	−2.36	0.02	3.83 ± 1.26	5.00 ± 1.49	−2.87	0.00	3.69 ± 1.37	4.73 ± 1.42	−3.46	0.00
Input; appropriateness of educational methods (4-point Likert scale)	2.70 ± 0.71	3.36 ± 0.48	−3.04	0.00	2.85 ± 0.57	3.26 ± 0.86	−2.18	0.03	2.80 ± 0.63	3.31 ± 0.67	−3.66	0.00
Input; close connection between classes (6-point Likert scale)	3.56 ± 1.35	4.27 ± 1.10	−1.71	0.09	3.94 ± 1.09	4.80 ± 1.23	−2.44	0.02	3.81 ± 1.20	4.52 ± 1.17	−2.68	0.01
Process; encouragement of positive participation of students (6-point Likert scale)	3.94 ± 1.38	3.83 ± 1.59	0.26	0.80	3.97 ± 1.25	5.11 ± 0.60	−2.73	0.01	3.96 ± 1.30	4.38 ± 1.40	−1.46	0.14
Process; provision of appropriate feedback (6-point Likert scale)	3.74 ± 1.36	4.08 ± 1.56	−0.84	0.40	3.58 ± 1.20	5.00 ± 0.50	−3.55	0.00	3.64 ± 1.26	4.48 ± 1.29	−3.00	0.00
Product; degree of satisfaction to the educational course (6-point Likert scale)	3.39 ± 1.48	4.67 ± 1.07	−2.95	0.00	3.83 ± 1.28	5.43 ± 0.53	−3.29	0.00	3.67 ± 1.37	4.95 ± 0.97	−4.02	0.00

SD = standard deviation.

The survey question asking about Context evaluation; whether the classes of the MHC satisfied their prior expectations or not, both the students (3.69 ± 1.37) and the professors (4.73 ± 1.42) responded that the educational course satisfied their expectations. However, the t-test showed that there was a significant difference in the degree of satisfaction between the students and the professors (P < 0.001). The response was more positive in the professors than in the students.

Among the questions about Input evaluation, to the question about ‘the connectivity between the teaching methods and classes,’ both the students (2.80) and the professors (3.31) responded that the teaching methods were connected with the classes. Students' response increased in 2017 (2.85) compared to 2016 (2.70); however, professors' response decreased in 2017 (3.26) compared to 2016 (3.36).

To the survey question asking about the close connection between the classes of the MHC, both the students and the professors responded positively in 2016 and 2017 (Table 3). The t-test showed a significant difference in the response between the students and the professors.

Among the questions for Process evaluation, the results of the survey showed that both the students and the professors responded that the students' participation in the classes was increased, and the positive evaluation was higher in the professors (4.38 ± 1.40) than in the students (3.96 ± 1.30). In particular, while no significant difference was found in the responses between the students and the professors in 2016, the professors' responses were significantly more positive (5.11 ± 0.60) than those of the students (3.97 ± 1.25) in 2017 (P < 0.01).

To the question asking about the provision of appropriate feedback to the students, the students responded that the feedback was moderately or slightly satisfactory (3.64 ± 1.26), and the professors responded that the feedback was moderate to very positive (4.48 ± 1.29). In addition, the difference was not significant between the students and the professors in 2016, but the evaluation was significantly more positive in the professors (5.00 ± 0.50) than in the students (3.58 ± 1.20) in 2017 (P < 0.001).

Among the questions for Product evaluation, to the question asking about the overall degree of satisfaction with the MHC, the students' responses were from ‘moderately’ to ‘slightly satisfactory’ (3.67 ± 1.37), while the professors' responses were ‘satisfactory’ (4.95 ± 0.97). The t-test showed that there was a significant difference in the degree of satisfaction between the students and the professors in both 2016 (P < 0.001) and 2017 (P < 0.001). The degree of satisfaction was higher in the professors than in the students in both 2016 and 2017.

There is a significant difference between students' and professors' perception of the MHC in the result of almost every objective survey question.

Qualitative evaluation

Results of content analysis of students' and professors' responses

The result of students' response to the descriptive questions and FGI result, and professors' response and meeting minutes analyzed with four categories of the CIPP model is shown in Table 4; the result of analyzing students' and professors' data, including quotations, is shown in Supplementary Table 2. There were 259 significant responses from students' answers to descriptive questions and FGI as they were differentiated to meaningful units; there are 139 significant responses from professors' answers to descriptive questions and proceedings. The main categories were seven and subcategories were 20 through the CIPP model. The major categories were like these: ‘reflecting goals and students' demands’ and ‘evaluation’ in context, ‘available human resource,’ ‘available material resource,’ and ‘strategy’ in the side of Input, ‘evaluation of instruction’ in terms of Process, and ‘achieving the goals’ in the aspect of Product.

Table 4

Results of content analysis by categories

Theme	Categories	Subcategories	Students (n = 295)		Professors (n = 135)
			No. (%)	Contents	No. (%)	Contents
Context	Reflecting the objective and students' demand	Advance preparation before instruction	1 (0.4)	Prior consultation between professors required, before class	3 (2.2)	The effectiveness of prior consultation between professors before class
		Supplying the instructional objective	5 (1.9)	Non-presenting instructional objective	0 (0.0)	-
		Students' demand and level	8 (3.1)	Fail to correspond with students' demand and level	2 (1.4)	Fail to correspond with students' demand and level
	Evaluation	Announcing evaluation method	2 (0.8)	Advanced notice required for evaluation method and grade	1 (0.7)	Advanced notice required for evaluation method and grade
		Evaluation criteria	47 (18.1)	Absence of evaluation principle	26 (18.7)	Propose a grading method
				Propose a grading method
Input	Available human resources	Support for developing instructional competency	0 (0)	-	4 (2.9)	Usefulness of professor development workshop
		Professors' efforts	4 (1.5)	Appreciating professors' efforts for the class	0 (0.0)	-
	Available material resources	Classroom	6 (2.3)	Absence of space for group discussion	6 (4.3)	Absence of space for group discussion
	Strategy	Educational contents	11 (4.2)	Various educational contents	0 (0.0)	-
				Need broadened variety of points of view for educational contents
		Educational contents connection	9 (3.5)	Sufficient educational contents connection	7 (5.0)	Insufficient educational contents connection
				Insufficient educational contents connection
		Amount of educational contents	33 (12.7)	Excessive amount of educational contents	12 (8.6)	Excessive amount of educational contents
		Instructional method	52 (20.1)	Interest in various instructional methods	30 (21.6)	Utilization of various instructional methods
				Inefficiency because of time shortage with various methods		Pressure of various instructional methods
		Timetable	15 (5.8)	Class time adjustment required	14 (10.1)	Class time adjustment required
		Learning support	13 (5.0)	Insufficient advanced notice for class	3 (2.2)	Insufficient advanced notice for class
Process	Class evaluation	Process	15 (5.8)	Not keeping class hours	6 (4.3)	Not keeping class hours
				Progress unsatisfactorily		Progress satisfactorily
		Small group	7 (2.7)	Need to improve the way of grouping	2 (1.4)	Need to improve the way of grouping
		Student participation	11 (4.2)	Increased student participation and learning educational contents insufficiently	9 (6.5)	Active student participation
		Feedback	8 (3.1)	Insufficient feedback	5 (3.6)	Insufficient feedback
Product		Curriculum	11 (4.2)	Satisfied with the educational course	7 (5.0)	Satisfied with the educational course
						Need to improve the perception of educational course
		Achieving the educational goal	1 (0.4)	Uncertain achieving educational goals	2 (1.4)	Achieving educational goals

In the subcategories, students remarked that prior consultation for the preparation of the subjects before the class was necessary, at the educational objective and reflecting students' demand; by contrast, professors remarked that prior consultation of the subject before the class worked well. Also, both students and professors responded that the MHC should more actively reflect students' needs. Students and professors required giving advanced notice well and proposed a method of giving a grade at the evaluation. Students also requested that professors disclose the principles of evaluation.

Regarding available human resources of Input, professors mentioned that the workshop helped develop professors' competency and students answered that they appreciated professors' efforts. Students and professors both pointed out that they need the space for group discussion as part of the available material resource. In the section of strategy, students responded that various points of view were required in the class but not various enough in spite of various educational contents. Students and professors pointed out the shortage of connection between educational contents, but some of them answered that those were enough. Students and professors remarked that the quantity of education was large and expressed positive opinions about the utilization of various educational methods, but students answered that the various methods did not work well due to a shortage of time and the various methods of education were burden for professors. Besides, class time should be adjusted, and advanced notice about the class should be supported more, said both students and professors.

In the section of Progress, students and professors commonly stated that there was not enough time in class. Students reported that the classes' progress did not work out well, but contrary to this, professors stated that it did progress well. In the small group learning, students and professors mentioned that the way of grouping needs to be improved and more useful feedback was also necessary. Both groups of subjects agreed that more active student participation was needed; due to this, students pointed out that they could not acquire the contents.

In the section of achieving the goal of Product, students and professors thought that they were satisfied with the curriculum, although some professors answered that improving the perception of the MHC was necessary. Students hesitated to express their ideas about achieving the educational objectives.

Instructional materials analysis

The results of analyzing the collected syllabus, timetable, competency, and grades are shown in Table 5. First of all, in the section of the educational objective of Context, it turned out that two to four educational objectives were presented, except for one subject, through reviewing the syllabus of every MHC subject. Moreover, as in Table 6, the competency categories of the MHC (6 competency categories) are matched with those the curriculum of the SNUCM (10 competency categories).

Table 5

CIPP evaluation result

CIPP classification	Detailed CIPP criteria	Focuses of curriculum of SNUCM (evaluation criteria)	Evaluation result
			Evaluation details (evaluation tool)
Context evaluation	Have the learning goals of the course been prepared well?	Consistency with the education goals of the SNUCM	Syllabus
				-Two to four educational goals were proposed for all the subjects except one subject. Educational goals for field trips were absent
			Educational course
				-The competencies of the MHC (6 competencies) are matched with those the curriculum of the SNUCM (10 competencies)
	Do the learning goals well reflect the objectives of the classes or the demand of the students?	Reflection of students' demand	Time table
				-The number of class topics was 22 in 2016, but it was decreased to 11 in 2017
Input evaluation	What are the available input resources? (human and material resources)	Participation by professors	Time table
	What are the approaches to the accomplishment of the educational goals?			-The number of class topics was 22 in 2016, but it was decreased to 11 in 2017
Process evaluation	Has the program (class) been implemented as planned?	Utilization of various student rating methods depending on their capabilities	Syllabus
	Has the program been implemented efficiently?			-Two or more evaluation methods were provided for each subject
Product evaluation	Have the initial goals been accomplished?	Accomplishment of learning goals for each course (subject)	Grading
				-The grading had a negatively skewed distribution as more than 97% of the students fulfilled the course

CIPP = context, input, process, product, SNUCM = Seoul National University College of Medicine, MHC = Medical Humanities Course.

Table 6

Comparison of the competencies involved in the MHC and those of the curriculum of the SNUCM

Competencies of the curriculum of SNUCM		2016, Medical Humanities No.		2017, Medical Humanities No.
Level l	Level 2	Medical Humanities 1	Medical Humanities 2	Medical Humanities 1	Medical Humanities 2	Medical Humanities 3	Medical Humanities 4
Medicare competencies as a physician	Understanding of human body and diseases	8	-	6	-	1	-
	Collection of clinical information and diagnosis	-	-	-	-	-	-
Research competencies	Critical and creative thinking	-	-	-	-	-	-
	Ability to perform researches	1	-	-	-	-	6
Leadership and international view	Empathetic understanding and communication	5	6	5	6	-	-
	Understanding of society and culture	-	-	-	-	-	-
	Understanding of international public health	-	4	-	4	6	-
Professionalism	Observation of ethics and laws	-	6	-	2	1	5
	Continuous self-development	-	-	-	-	-	-
	Social contribution	-	2	-	2	5	-

MHC = Medical Humanities Course, SNUCM = Seoul National University College of Medicine.

At the part of Input, the themes were lessened from 22 in 2016 to 11 in 2017, according to analyzing instructional materials shown in the timetable.

In the section of utilizing various evaluation methods of Process, two or more evaluation methods were presented for every subject on the syllabus.

In achieving the goal of Product, students' grades showed as a negatively skewed distribution and over 97% of students completed the courses.

DISCUSSION

According to the limitations of medical education due to the changes of the time, the first MHC was created in 1996 and the number of courses increased significantly in 2000.1415161718 The present study was conducted to inspect the effectiveness and improvement of MHC in the competency-based curriculum newly introduced to Seoul National University 20 years later. Students and professors have been interested in the results, such as the effectiveness and problems of operating the courses, since the improved course was introduced.419

In the present research, MCH was analyzed using the CIPP evaluation model. In the Context evaluation, although the educational objective was presented clearly on the syllabus, the results of the content analysis of students' opinions showed that students wanted the educational objective and evaluation principal to be provided more clearly. Moreover, according to the content analysis, it is necessary to reflect students' demand and level with the class. Therefore students have to narrow the distance between what they want to learn and what they should learn. Because students experience conflicts among these. The previous study indicated that the problems came from a shortage of understanding the necessity of curriculum related to MHC. 20 On the other hand, competency of SNUCM such as ‘critical and creative thinking,’ ‘Empathetic understanding and communication,’ ‘understanding of social and cultural,’ ‘Continuous self-development’ and ‘social distribution’ were reflected on the competency of MHC due to the competency-based curriculum. This is the same concept as cultivating the ability as a human sociologist like a “morality and a sense of ethics,” a “sense of accountability,” “communication skills,” and “empathic ability” to train future doctors in developed countries. However, the previous MHC only emphasized having a good personality.3

To sum up the results of the Input evaluation analysis, this section could be described as having various and extensive educational contents and educational methods. In this part, despite professors' high passion, desire and efforts, there were different perceptions between students and professors in the section of Input evaluation. Positive support is required21 with finding the need for improving the educational course to improve the curriculum since professors' expectation level has been higher. Moreover, this fact was supported by the result of content analysis that the workshop of developing instruction was effective for the professors. Since students' learning achievement was influenced by instructors,21 professors' enthusiasm for education should be encouraged and supported. However, professors should keep in mind not to increase the amount of educational contents due to the high enthusiasm for education.

In the section of Process evaluation, students and professors generally gave a positive review. However, in 2017, a significant difference between students' and professors' opinions was shown compared to those in 2016. There were offers, observing the class hours, expanding practical students' participants, and active feedback. Students' engagement in class, practice/expending small grouping, and intensifying feedback were important requisites22 for a good class so that positive support is required. These active students' engagements in the course enhance communication, and the difference of perceptions between students and professors could be reduced.23 As proved in previous research, the most crucial factor was the process24; the efforts for intensifying educational process should be required.In the section of Product evaluation, students' academic achievement was excellent. Their satisfaction with the education was also reasonable, therefore it would be appropriate to maintain the educational course of MHC. However, according to the result of the present study, improving the perception of students should be targeted, because there were students who had prejudice against MHC which was not good and necessary. In addition, students were not sure about their academic achievement. Because the MHC, which included holistic attributes such as knowledge, technology, and attitude, were intrinsically not the same as medicine, which is based on the field of natural science,24 it could be assumed that students did not understand MHC itself and they had difficulty recognizing their academic achievement.

To summarize, in order to improve the MHC quality, it is first necessary to conduct a student demand analysis and provide training objectives and evaluation principles (in the context). It is also important to control the amount of education, strengthening demand analysis and practical support for professors (in the input). It should be continuously managed to ensure that these efforts are carried out faithfully (in the process). Above all, t-test results showed that there was a difference between students and professors throughout CIPP. The reason for this result is the low perception of MHC in students. Therefore, the most important task is to raise students' awareness of the MHC.

This study has a few limits. Firstly, the results cannot prove that the curriculum of MHC was evaluated correctly since the continuative educational course23 was researched with a segmental perspective. Secondly, these collected and analyzed data hardly can be said to be representative of the whole educational course of MHC. Thirdly, even if intensifying FGI complemented the research, because the survey of MHC 2 was not conducted in 2017, this study does not include the survey result of MHC 2 in 2017.