Students' self-assessment of achievement of terminal competency and 4-year trend of student evaluation on outcome-based education.

PURPOSE: This study was designed to allow a student at School of Medicine, Kyungpook National University (KNUSOM) to self-assess how well they had achieved competency and to analyze the differences and trends of the scores by year. Furthermore, students are asked to evaluate the need for curriculum improvement by competency, the tendency of the score is analyzed by year, and the results are reflected in the improvement of the curriculum.
METHODS: We conducted a questionnaire survey for fourth-year students of KNUSOM who took medical education classes from 2015 to 2018. Questionnaire items were evaluated on the basis of their current achievement of 30 subordinate competencies of competency and the degree of necessity with respect to revising the curriculum. One-way analysis of variance was performed for the yearly difference analysis.
RESULTS: The students' scores on the graduation competency were 2.03 to 4.06. In the yearly difference analysis, there was no significant difference in the average of 30 total competencies, but 16 of the sub-competencies showed significant year-to-year differences. The scores for the 30 graduation competencies were different for each year, but the competencies showing high scores and low scores were found to be similar each year.
CONCLUSION: We found that the achievement level of the students was approximately 60% to 70%. We were able to confirm the contents of the education that the students continuously demand. The curriculum trend graphs for each year showed that the students' scores improved when the curriculum was being revised. We found that it is necessary to accept the students' self-evaluation reliable as the students indicated that the contents of the curriculum should be added to the areas where the contents were lacking in the present curriculum.

Introduction

Korean medical schools have adopted outcome-based curriculums and developed terminal competencies that students have to achieve at the time of graduation. However, how the students’ achievement of the terminal competencies will be measured and who will evaluate the attainment remain an unsolved problem [1]. When we divide the domain of terminal competencies into the areas of knowledge, clinical skills, and attitude, we can identify examination or measurement tools that can determine whether students have achieved competencies in each of these areas. To measure competencies in the medical knowledge domain, exams such as the basic medical science summative examination, the clinical medicine summative examination, and various written exams are used as indexes. For assessment in the skill domain, objective structured clinical examination (OSCE), clinical performance examination (CPX), and other exams are used as indexes. There are various competencies associated with professionalism in the attitude domain, but it is difficult to find standardized testing instruments or reliable instruments to measure the attitude domain [2,3]. For example, competencies such as communication skills and empathy are difficult to measure because there are few standardized instruments or methods to measure them. Even in the knowledge and attitude domains that have relatively objective and concrete assessment methods, there are neither commonly used instruments nor methods to measure specific competencies, such as critical thinking and clinical reasoning competencies, nor methods to measure the students’ current progressive levels. Thus, in the absence of a recommended set of assessment instruments for each of the terminal competencies in each school, the self-assessment method in which the students themselves evaluate their own competence is recommended as a useful and meaningful competency evaluation method [4,5].

Previous Korean studies on competency measurement reported about clinical performance competency improvement based on the results of OSCE and CPX [6-8] and medical professional competency by self-assessment [9]. There are no previous studies that have assessed the students regarding the achievement of all terminal competencies at a medical school. In other countries, most studies also report on the measurement of clinical performance competency [4,10-13].

Competency self-assessments are used as a method to identity improvement in clinical skills or professionalism [4,5,10,11]. Although there are some negative views on self-assessment [3,5,14], a real performance level is reflected in self-assessment [4,15] and this method is recommended [2,4,16,17] as it has its own meaning since it is an assessment of the self-perceived ability by the students themselves. Moreover, some argue that accurate self-assessment [10,14] is one of the competencies necessary for physicians in the era of lifelong learning [18]. Based on these previous studies, the authors regarded students’ self-assessment of their competencies at the end of their education as a meaningful competency assessment method. With this rationale, this study adopted a self-assessment method that uses surveys to assess students’ competencies.

The purpose of this study is to assign the graduating senior students at School of Medicine, Kyungpook National University (KNUSOM) self-assessment of their own achievement of terminal competencies. The subjects will also be tasked with assessing the need for curriculum improvement with regard to each competency each year with the aim of analyzing the annual trend. In addition, the results of this study will be reflected in the improvement of the curriculum.

Methods

The research subjects were seniors at KNUSOM who took a medical education class from 2015 to 2018. The number of students who took the class was 62 and 65 in 2015 and 2016, respectively, and the number dropped to 35 both in 2017 and 2018. A total of 197 students took the class from 2015 to 2018, and the students who submitted inadequate questionnaire responses were excluded from the research. The total number of research subjects was 186 (Table 1). The number of students who took the medical education class varied because the class was an elective course in the senior year. In 2015 and 2016, there were two elective courses for approximately 120 students. In 2017 and 2018, there were four elective courses, with each class having fewer students.

Table 1.

Participant and Respondents of This Study

Year	Participant	Respondent
2015	62	57
2016	65	62
2017	35	35
2018	35	32
Total	197	186

The questionnaire was composed of 30 questions about the specific sub-competencies of graduate competencies of KNUSOM (Table 2). The 30 questionnaire items were categorized into five domains of competency: medical knowledge competency (six items), research competency (two items), clinical competency (11 items), professionalism competency (nine items), and global competency (two items). Each item was then scored on a 5-point Likert scale.

Table 2.

Items of Terminal Competencies in Questionnaire

Domain of competency	Ten abilities (sub-competencies)	Specific sub-abilities (questionnaire items)
A. Medical knowledge competency	Ability to utilize medical knowledge	1. Scientific principles and concepts underlying medicine
		2. Normal structure and function of the human body
		3. The reaction of the human bodyand the pathogenesis of the disease
		4. Generation and life cycle
		5. Principle of examination and treatment
		6. Prevention of disease and promotion of health
B. Research competency	Research ability	7. Understanding the research method & design
		8. Experiment and analysis of results
C. Clinical performance competency	Clinical decision making	9. History taking
		10. Physical examination
		11. Diagnostic and therapeutic skills
		12. Interpreting examination and diagnostic laboratory test results
		13. Diagnosis–treatment–prevention planning
		14. Prioritization and first aid
	Problem solving ability	15. Clinical reasoning
		16. Critical thinking
		17. Use of evidence-based medicine
	Holistic patient care	18. Social and cultural considerations of the patient
		19. Understanding and responding to the patient’s pain
D. Professionalism	Ethical behavior & legal responsibility	20. Ethical behavior
		21. Legal responsibility
	Life-long learning	22. Exploring and leverage information
		23. Self-reflection & life-long learning
	Social responsibility	24. Social responsibility
		25. Understanding and application of health care system
	Communication and cooperation	26. Communication skills
		27. Writing medical records
		28. Patient education
E. Global competency	Global competency	29. Understanding and cooperating with international health
		30. Foreign language ability

The competencies of KNUSOM reflected in the questionnaire were developed and announced in 2006. The survey was conducted during the medical education class, and students were asked to assess their achievement of the competencies as well as the need for curriculum strengthening or supplementation in each competencies domain on a 5-point Likert scale. The sub-abilities and competencies were assigned serial numbers so that the serial numbers were used for convenience in lieu of the actual sub-abilities when the results of the analysis were released. The authors explained the purpose of the survey to students and asked them to perform honest self-assessments. The students voluntarily participated in the survey and then the written informed consent to participate in the study was obtained from them.

Data analysis was conducted as follows. First, the mean scores of students’ self-assessments were calculated with respect to each 30 sub-ability. Second, a one-way analysis of variance (ANOVA) with Scheffe’s post-hoc test was conducted to analyze the differences in competency scores of students who graduated in different years. Third, the students’ annual self-assessment scores were represented in a graph to analyze the trend in relation to each sub-ability.

The trend was analyzed by using a graph reasoned that the analysis of years of accumulated data would make it possible to estimate the students’ achievement levels with respect to each competency based on their self-assessments after they completed a new curriculum. We expected that the competencies of low and high achievement levels would be derived from years of accumulated data. We considered the results of 1-year assessments by students would be insufficient to prove the effectiveness of the curriculums. Instead, we judged that reporting on years of students’ assessments would be reliable. Fourth, in order to identify the score differences between the students’ self-assessment of competencies and the need for curriculum strengthening or supplementation, the mean scores of the two domains obtained for 4 years were compared with respect to each competency. Because the scores of the two domains showed an inverse relationship, it was inconsequential to conduct a one-way ANOVA for the purpose of identifying the score differences. Consequently, the mean scores of the two domains were compared in a graph so that the score differences could be visually interpreted. Fifth, with respect to curriculum improvement, the top five items with the highest scores were analyzed. The item with the fifth highest score was set as a limit because there was a noticeable gap between items with the fifth and sixth highest scores among the 30 items. If more than five items were selected for analysis, it was difficult to detect a trend because most competencies were included in the range for analysis. The top five items were selected annually based on relative rankings, irrespective of scores, because the mean score was different every year. This selection method was to ensure that the analysis would reflect the opinions of all students who answered the questionnaire for the 4 different years. All data were analyzed with IBM SPSS ver. 22.0 for Window (IBM Corp., Armonk, USA) and p<0.05 was considered significant. This study was approved by the Institutional Review Board of Kyungpook National University Industry Foundation (IRB approval no., 2018-0207).

Results

1. Results of students’ self-assessment of graduation competencies

The students’ self-assessment scores of graduation competencies for the 4 years ranged from 2.03 to 4.06 points. The competency with the lowest score was ‘No. 29 understanding and cooperating with international health’ of 2017, while the competency with the highest score was ‘No. 26 communication skills’ of 2018. When the 4-year mean score of each competency was considered, the five competencies with the highest scores in descending order were ‘No. 9 history taking (3.74 points),’ ‘No. 22 exploring and leverage information (3.69 points),’ ‘No. 26 communication skills (3.67 points),’ ‘No. 19 understanding and responding to the patient’s pain (3.63 points),’ and ‘No. 23 self-reflection & life-ling learning (3.62 points).’ The five competencies with the lowest scores in descending order were ‘No. 29 understanding and cooperating with international health (2.74 points),’ ‘No. 4 generation and life cycle (2.76 point),’ ‘No. 7 understanding research method & design (2.86 points),’ ‘No. 25 understanding and application of health care system (2.98 points),’ and ‘No. 30 foreign language ability (3.13 points)’ (Table 3).

Table 3.

Competencies with a Significant Difference by Year

Specific sub-abilities		Students' achievement of competencies					Competencies with a significant difference by year	Mean score of 3.5 points or higher over 3 years	Average score of 3.0 points or lower over 3 years
		2015	2016	2017	2018	2015-2018
1.	Scientific principles and concepts underlying medicine	3.32	3.47	3.06	3.31	3.32
2.	Normal structure and function of the human body	3.44	3.45	3.20	3.31	3.38
3.	The reaction of the human body and the pathogenesis of the disease	3.44	3.55	3.26	3.34	3.42
4.	Generation and life cycle	2.65	3.00	2.57	2.69	2.76			0
5.	Principle of examination and treatment	3.49	3.52	3.31	3.38	3.45
6.	Prevention of disease and promotion of health	3.26	3.31	3.03	3.25	3.23
7.	Understanding the research method & design	2.77	3.13[a)]	2.40[a)]	3.00	2.86	0		0
8.	Experiment and analysis of results	3.19[a)]	3.27[b)]	2.60[a),b),c)]	3.31[c)]	3.13	0
9.	History taking	3.77	3.76	3.54	3.84	3.74		0
10.	Physical examination	3.39	3.65[a)]	3.14[a)]	3.69	3.48	0
11.	Diagnostic and therapeutic skills	3.39	3.56	3.29	3.38	3.42
12.	Interpreting examination and diagnostic laboratory test results	3.61	3.61	3.37	3.75	3.59		0
13.	Diagnosis-treatment-prevention planning	3.51	3.53	3.09	3.56	3.45		0
14.	Prioritization and first aid	3.26	3.47[a)]	2.97[a)]	3.38	3.30	0
15.	Clinical reasoning	3.40	3.58	3.20	3.69	3.47
16.	Critical thinking	3.33	3.45[a)]	2.89[a),b)]	3.53[b)]	3.32	0
17.	Use of evidence-based medicine	3.54	3.60	3.14[a)]	3.72[a)]	3.52	0	0
18.	Social and cultural considerations of the patient	3.58[a)]	3.58[b)]	2.91[a),b),c)]	3.75[c)]	3.48	0	0
19.	Understanding and responding to the patient’s pain	3.74	3.61	3.29	3.84	3.63		0
20.	Ethical behavior	3.49[a)]	3.58[b)]	2.91[a),b),c)]	3.56[c)]	3.42	0
21.	Legal responsibility	3.25[a)]	3.66[b)]	2.63[a),b),c)]	3.31[c)]	3.28	0
22.	Exploring and leverage information	3.72	3.76	3.40	3.84	3.69		0
23.	Self-reflection & life-long learning	3.56	3.76	3.31	3.81	3.62		0
24.	Social responsibility	3.39	3.68[a)]	3.11[a)]	3.69	3.48	0
25.	Understanding and application of health care system	2.96	3.34[a)]	2.54[a)]	2.78	2.98	0		0
26.	Communication skills	3.72	3.52[a)]	3.49	4.06[a)]	3.67	0	0
27.	Writing medical records	3.44	3.40	3.03[a)]	3.78[a)]	3.41	0
28.	Patient education	3.54	3.53	3.11[a)]	3.81[a)]	3.51	0	0
29.	Understanding and cooperating with international health	2.84[a)]	3.11[b)]	2.03[a),b)]	2.63	2.74	0		0
30.	Foreign language ability	3.16	3.35[a)]	2.71[a)]	3.09	3.13	0
Mean		3.37	3.49	3.02	3.47	3.36

p<0.05

When annual differences in competency scores were analyzed, no difference was found in the annual mean scores of the 30 competencies. However, there were significant variations in 16 among the 30 competencies (Table 3). Competencies with high and low assessment scores in each year were identified, and there were ten sub-abilities that scored more than 3.50 points over 3 years in 4 years: six sub-abilities in clinical performance competency and four in professionalism competency. Medical knowledge, research capabilities, and global competencies were not included in this category. On the other hand, there were four sub-abilities that scored 3.0 or lower over 3 years (Table 3).

We calculated the achievement level of the students’ graduation ability on the basis of 100%. The average score of achievement by year was calculated as 100%. As a result, the achievement level of the students was approximately 60% to 70%. When the circle shows the achievement of the graduation ability, the area of the circle is the total target that the students must achieve. Outside the line, this area can be seen as the amount of competency that students have not yet achieved, and the amount is calculated as 30%–40% (Fig. 1).

Fig. 1.

Student’s Achievement of Competencies by Year When We Calculate the Total Achievement Target as 100%

2. Results of the analysis on the annual trend of students’ self-assessment of terminal competencies

Although the students’ self-assessment scores of the 30 sub-abilities and the mean score of the sub-abilities differed every year, the sub-abilities with high and low scores were similar every year (Fig. 2). The five subabilities with the highest and lowest scores were identified for each year. ‘No. 9 history taking’ and ‘No. 22 exploring and leverage information’ ranked in the top five every year. ‘No. 26 communication skills’ ranked in the top five 3 times. ‘No. 12 interpreting examination and diagnostic laboratory test results,’ ‘No. 19 understanding and responding to the patient’s pain’ and ‘No. 23 self-reflection & life-long learning’ ranked in the top five twice. Sub-abilities of the professionalism competencies domain ranked in the top five most frequently, while none of the sub-abilities of global competency ranked in the top five (Table 4).

Fig. 2.

Annual Distribution of Students’ Self-Assessment Scores of Terminal Competencies

Table 4.

Comparison of the Five Competencies with the Highest Scores and the Bottom Scores in Relation to the Achievement of Terminal Competencies by Year

Students’ achievement of competencies	Year	Domain of competencies
		A (N=6)	B (N=2)	C (N=11)	D (N=9)	E (N=2)
Top 5	2015			9, 19, 12	22, 26
	2016			9	21, 22, 23, 24
	2017	5[a)]		9, 12	22, 23[a)], 26
	2018			9, 19	22, 23[a)], 26, 28[a)]
Bottom 5	2015	4	7		25	29, 30
	2016	4, 6	7, 8			29
	2017	4	7, 8		25	29
	2018	4	7		25	29, 30

Same mean score.

On the other hand, ‘No. 4 generation and life cycle,’ ‘No. 7 understanding research method & design,’ and ‘No. 29 understanding and cooperating with international health’ continuously ranked in the bottom five every year. ‘No. 25 understanding and application of health care system’ ranked in the bottom five 3 times. ‘No. 8 experiment and analysis of results’ and ‘No. 30 foreign language ability’ ranked in the bottom five twice. Subabilities of research competency and global competency ranked in the lowest five most frequently (Table 4).

3. Competencies requiring curriculum improvement and education strengthening and its annual trends

A total of 12 sub-abilities (No. 8, 10, 11, 13, 14, 20, 21, 24, 25, 27, 29, 30) ranked in the top five competencies that required curriculum improvement and education strengthening for the 4 years (Table 5). The yearly trends in competency education, which needed to improve and complement the curriculum, were reflected in those of the students’ competency achievement results. We found that students responded indicating that they needed more education about the competencies in the response areas in which they had low achievement. Although the educational improvement requirement scores for 30 items were different every year, the competencies showing high scores and low scores were similar year by year (Fig. 3).

Table 5.

The Top Five Competencies Requiring Curriculum Improvement and Education Strengthening by Year

Year	Domain of competencies
	A (N=6)	B (N=2)	C (N=11)	D (N=9)	E (N=2)
2015				21, 25, 27	29, 30
2016		8	11, 14	20	29
2017			13	20, 21, 24, 25
2018			10, 11, 13	21, 25

Fig. 3.

Annual Need for Curriculum Improvement in Relation to Terminal Competencies (2015–2018)

The sub-abilities that ranked in the top five competencies requiring education strengthening were identified. A total of two sub-abilities, ‘No. 21 legal responsibility’ and ‘No. 25 understanding and application of health care system’ ranked in the top five 3 times for the 4 years. A total of four sub-abilities (No. 11, 13, 20, 29) ranked in the top five twice. Among them, the students responded that global competence requires improvement of education in all of two total subcompetencies (Table 5).

4. Score gaps between students’ self-assessment and the need for education improvement

The annual average scores of the students’ self-assessment of competencies and the annual average scores of the competencies that require education improvement were compared, we found that the average scores of the two areas showed inverse patterns (Fig. 4). That is, the terminal competencies with lower self-assessment scores had higher scores of the competencies that require education improvement than self-assessment scores (Fig. 4).

Fig. 4.

Comparison of the 4-Year Average Scores between Students’ Self-Assessment of Terminal Competencies and the Need for Curriculum Improvement

Discussion

This study is to assess the graduating students’ achievement of terminal competencies by using a self-assessment, to identify the competencies that require curriculum improvement and supplementation and to reflect the results in the improved curriculum.

The five sub-abilities with the highest achievement scores among the 30 sub-abilities for the 4 years were ‘No. 26 communication skills,’ ‘No. 9 history taking,’ ‘No. 22 exploring and leverage information,’ ‘No. 19 understanding and responding to the patient’s pain’ and ‘No. 23 self-reflection & life-long learning.’ The authors did not interpret this result as an indication that the students fully acquired these competencies, but we considered because the survey was conducted on senior students. In this period, intensive CPX education was provided to seniors, who met with simulated patients to receive intensive CPX communication education and role-play studied to prepare for the national examination. We concluded that the students’ self-assessment of competencies was higher in the case of the competencies that were recently taught in the curriculum. This results is consistent with the result of a study conducted with third-year medical students regarding ‘bedside teaching.’ In that study, the students gave themselves higher scores for performance competency when they had more frequent performance sessions [11]. That is, students tend to assess themselves higher when they are given more education and training opportunities.

The results of the analysis on the annual differences in competency scores showed that there was no difference in annual average scores but that the sub-abilities with high and low scores were similar every year. The results were considered meaningful. Particularly, the subabilities of ‘global competency’ never ranked in the top five in terms of the achievement of terminal competencies, which is interpreted as a fairly reasonable result. The reason is that KNUSOM does not offer classes or extra curricula activities in its official curriculum to help students develop global competency. The medical school only provides a ‘students exchange clerkship program’ for students who wish to gain practical clerkship experience at Mahidol Medical School in Thailand, Hamamatsu University and Hokkaido University in Japan and other overseas medical schools. As this program is offered to select applicants rather than the entire student body, the fact that the majority of students have underestimated their global competence is interpreted as a fairly credible self-assessment result.

It is also reasonably valid that sub-abilities of research competency was continuously included in the bottom five terminal competencies. The reason is that research strengthening education was not offered to the current senior students. In 2016, KNUSOM completed a list of terminal competencies, including research competency. Subsequently, a curriculum to help students achieve these competencies was designed and research competency strengthening education was first included in the new curriculum of 2017.

The result of including the research competencies in the lower five is also interpreted as a reasonable result reflecting the characteristics of the curriculum. In the fourth grade curriculum, senior students can make one selection between ‘clinical clerkship’ and ‘basic medical experiment’ for 4 weeks as an elective clerkship. Most students choose a specialized clinic and some choose basic medical experiment and research. Consequently, students who did not choose a basic medical science research course inevitably reported low scores of ‘No. 7 understanding the research method & design’ and ‘No. 8 experiment and analysis of results,’ belonging to research competency. In other words, the students’ self-assessment differed depending on the selective subject they chose. That is, ‘No. 8 experiment and analysis of results’ was included in the bottom five not because the curriculum was insufficient but because the characteristics of the elective course affected the results of the survey.

Next, the results of the analysis of the score gaps between the students’ self-assessment and the need for education improvement show that self-assessment scores and the need for improvement had an inverse relationship. This can be interpreted as an indication of reliability and consistency of students’ self-assessment. However, we believe that the basic medical science domain requires careful analysis since its scores are high both in self-assessment and the need for improvement. Basic medical science is a mandatory course for students to learn and all students passed the course without fail. Therefore, it is reasonable that the students’ self-assessment scores regarding this competency are high.

As for the result indicating that the need for improvement is also high, it also is necessary to analyze lecture evaluation results of basic medical science courses. The results of the lecture evaluation in the basic medical science courses show consistent demand for improvement in terms of lesson scope, evaluation methods, the degree of test difficulty, and the validity of evaluations. Therefore, the high need for improvement is also reasonable. In conclusion, we interpreted that basic medical science requires continuous improvement in terms of education methods.

This study is significant in that it is the first Korean study that has reported on medical students’ achievement of terminal competencies. However, it can be noted it is difficult for a self-assessment of terminal competencies to have an adequate level of reliability. In several studies, self-assessment is widely used as a method to identify competency achievement [5] and it is a meaningful method to assess competencies [4,16]. Furthermore, some argue that self-assessment is a surrogate method to assess objective improvement [10]. Leonardi et al. [4] in 2018 used self-assessment for the competency assessment of obstetrics and gynecology residents in their study. As indicated by the results of the study, they reported that the residents’ perceived competencies were quite meaningful in recognition of the validity of self-assessment. Of course, it has also been reported that there are gaps between self-assessment scores and real assessment scores of performance. However, self-assessment scores can represent the students’ confidence. While confidence does not necessarily lead to competency, it can follow the trajectory if much time is invested [4]. Meanwhile, MacDonald et al. [13] in 2003 argued that self-assessment is critical to encouraging and improving the ability of professionals. The reason is that students and residents should be able to accurately self-assess their skills [13]. Of course, there is a difference between the students’ self-assessment and the experts’ real assessment of performance, and students’ self-assessment scores were generally higher than objective assessment scores. In conclusion, we believe that the use of self-assessment is an appropriate, relatively efficient and simple method to evaluate the achievement of competencies, to assess students’ achievement and to evaluate the curriculum. This study presents evidence that self-assessment can be used in other schools as a means to suggest methods to improve students’ competencies and curriculums.

In further studies, it would be meaningful to examine the students’ competency levels as perceived by the professors and thus identify the gaps between the students’ self-assessment and the professor’s assessment. For example, a professor may have strengthened a curriculum, thinking that the students’ competency in a specific area is insufficient. However, students, in response, might raise a complaint, thinking that they have sufficient competency in that area. Therefore, as the gap between the competency levels perceived by the professors and the students is examined and offered to professors and students, the exchange would facilitate communication among education participants. As they can better understand one another, it is expected that they can set the right directions for curriculum improvement and harmoniously find the balance in terms of the educational need between educators and learners. In particular, if a gap between the students’ self-assessment and the real performance is examined with respect to clinical skills or performances, and thus it is found a gap between the students’ confidence and their real and insufficient competencies. Based on these results, we expect that educators will be able to improve the curriculum to direct students’ confidence toward their real competencies.