An analysis of verbs used in the course outcomes of outcome-based integrated courses at a medical school based on the taxonomy of educational objectives.

PURPOSE: The purposes of this study were to analyze the course outcomes of integrated courses of a medical school that introduced an outcome-based curriculum and to suggest ways to improve techniques in the description of outcomes.
METHODS: In order to investigate the course outcomes of 39 courses in the first and second grades at Kyungpook National University School of Medicine in Korea, verbs for the course outcome were classified according to Bloom's taxonomy and frequency analysis was completed. The appropriateness of use of verbs in description of the outcomes was also analyzed.
RESULTS: The total number of course outcomes for both grades was 497. The number of course outcomes per credit ranged from a minimum of 0.9 to a maximum of 22.0. Most of the course outcomes were classified as the comprehension domain according to Bloom's taxonomy. The most frequently used verb was "explain," accounting for 61.0% (n=303) of the entire course outcome verbs. Some verbs in the outcomes, however, were unmeasurable.
CONCLUSION: The major findings of this study include the followings. First, the number of course outcomes varied according to courses. Second, several course outcomes included unmeasurable verbs. Finally, most of the verbs used to describe course outcome belonged to the two lower levels of Bloom's taxonomy-knowledge and comprehension. In order to improve the description of course outcomes, this study suggests that it is necessary to adjust the number of course outcomes, applicate overt behavior verbs, and elevate the level of course outcomes from the lower memorizing level to the higher application level.

Introduction

It is important to describe learning outcomes appropriately in an outcome-based curriculum. This is because the learning outcomes clearly state what learners should achieve at the end of the learning as well as what are to be evaluated [1,2]. To describe learning outcomes is an important education designing activity that determines education and instructional directions [1]. Therefore, a medical school that implements the outcome-based curriculum should first inform its members about its macroscopic educational directions by establishing exit and phase outcomes. Second, instructors should be provided with guidelines that help them to describe learning outcomes of their courses and lessons appropriately. Third, the instructors should be provided with appropriate education corresponding to the guidelines. It is also necessary for the school to review the learning outcomes described by the instructors and to provide feedback to them about the relevance of their descriptions [3].

In Korea, outcome-based education (OBE) began to be expanded in 2011, when the Korean Institute of Medical Education and Evaluation (KIMEE) announced the post-second period Accreditation Standards of KIMEE. Since then, developing learning outcomes to operate OBE has become a mandatory task for all medical schools to meet the Accreditation Standards [3].

Based on this educational and social background, most medical schools in Korea developed exit outcomes, phase outcomes, course outcomes, and lesson outcomes [4]. Exit outcomes indicate final achievements that students at graduation should accomplish. Phase outcomes may differ between the schools. Achievements that students should accomplish at the end of a period, either classified according to grades or contents of education, are defined as phase outcomes. If periods were classified according to educational contents, for example, premedical, basic medical science, and clinical medical curriculum can be classified into phase 1, phase 2, and phase 3 outcomes, respectively. Course outcomes indicate achievements that learners should accomplish at the end of a course. Finally, lesson outcomes, or instructional learning outcomes, are the smallest outcomes that indicate achievements that learners should accomplish at the end of a class [3]. Developing such learning outcomes is the starting point of developing a successful outcome-based curriculum.

While developing learning outcomes, many instructors have been confused about how to describe learning outcomes that are distinct from learning objectives [4]. Although “learning outcomes” itself can be used as a substitute for learning objectives or lesson outcomes for practical use in development of outcome-based curriculum [4,5], several literatures point out the conceptual differences [4,5].

Conceptually, the learning objective refers to a collective statement that describes new abilities acquired by learners through a lesson [6]. It is a term mainly used in the era when schools operate “subject-centered” or “discipline-centered” curriculum. Learning outcomes began to replace learning objectives with the emergence of OBE in the 1970s and many universities in the United States started to emphasize “ability to perform” rather than “acquiring knowledge [2].” Learning outcomes are defined as the competencies of a learner to integrate knowledges, skills, and attitudes in an authentic context after achieving specific learning objectives [2,6]. In summary, the learning outcomes emphasize more comprehensive competencies of the learners to perform than learning objectives do and are used as a higher concept than learning objectives. Learning outcomes used in this study are defined as the term that can replace learning objectives for practical purposes and emphasized practical performance ability of learners more comprehensively than that of learning objectives.

Despite the difference in the concept of the two terms, the method of describing learning outcomes in medical schools has developed based on the method of describing learning objectives [1]. Different methods of describing the learning outcome were proposed by researchers including Tyler, Mager, Gagné, and Gronlund with different degrees of specificity of statements [7]. One of the simplest and most widely used methods is Tyler’s method. This method expresses a learning objective with a statement composed of the subject as the learner, the object as the educational contents, and a measurable behavior verb as the learner’s final behavior [7]. Kyungpook National University School of Medicine (KNUSOM) adopted Tyler’s method in the development of its learning outcome guidelines. The most significant reason was that the simplest method was suitable to spread the concept of learning outcomes. Even with this simple method it is still possible to state the contents of learning outcomes sufficiently.

In this context, studies regarding what learning outcomes actually are and how to describe them have been conducted based on papers about the concept of competency [3], description of learning outcomes, and an association of learning outcomes and evaluation [8] in outcome-based curriculum. In other study fields, research has been conducted on the analysis of educational objectives according to taxonomy [9,10]. However, it has not been possible to find previous studies at medical schools that analyzed how appropriate the learning outcomes were written according to the guidelines although the description of an outcome statement is of paramount importance in outcome-based curriculum.

Therefore, this study aimed to suggest possible ways to improve strategies in the description of learning outcomes by analyzing learning outcomes developed in one medical school based on the taxonomy of educational objectives. The specific purposes of this study are: (1) to identify whether the number of developed outcomes of the courses is appropriate; (2) to classify verbs used in describing the course outcomes according to Bloom’s taxonomy to determine whether courses were developed at appropriate levels to achieve competencies; and (3) to determine if suitable behavior verbs were selected to describe the course outcomes.

Methods

This study analyzed the course outcomes of 39 courses including 16 courses for the first grade and 23 courses for the second grade at KNUSOM. The course outcomes of all the courses were collected for the following analysis. First of all, the courses were classified into three categories (basic medical science, clinical medical science, and medical humanities) and then the number of course outcomes and the number of course outcomes per credit according to the course classification were calculated. Second, the number of course outcomes of courses and the number of course outcomes per credit in each grade were calculated. Third, the verbs used to describe the course outcomes were classified based on Bloom’s taxonomy of cognitive educational objectives [11].

Bloom’s taxonomy was designed by Bloom [11] during the 1950s and is a model to categorize the levels of cognitive abilities required in classroom situations. The taxonomy is divided into six levels: knowledge, comprehension, application, analysis, synthesis, and evaluation. From level 1 to 6, the level of cognitive ability increases. Knowledge in Bloom’s taxonomy indicates the ability of a student to memorize or remember what he/she has learned such as facts, concepts, principles, methods, types, and structures. Comprehension refers to the ability for learners to understand what is learned. Beyond simply recalling or memorizing facts, this ability refers to that of understanding and discovering the meaning even when the content of the materials is replaced with something else. Application means the ability to apply what students have learned to actual or new situations. Analysis is the ability to disassemble the given material of a phenomenon into an organization, structure, or component and discover their interrelationships. Students will actually find patterns that they can use to analyze a problem. Synthesis indicates the ability to arrange, organize, and integrate the elements of a given material to establish new and creative forms, principles, relationships, and structures, or to make predictions. Evaluation is the highest level of Bloom’s taxonomy which refers to the ability to assess information to reach to a conclusion or to judge values based on some criteria [7]. In this study, the reason why the categorization of learning outcomes was based only on the cognitive/ knowledge domain was because course outcomes of lecture-centered courses mainly involve contents of the cognitive/knowledge domain.

In addition, the reasons why the present study applied Bloom’s taxonomy, which is treated as a traditional taxonomy of educational objectives [6,12], instead of the newly revised Bloom’s taxonomy were as follows. First, Bloom’s taxonomy of educational objectives is still being widely used in most educational fields [7,10,12]. Second, the relevance of the revised taxonomy is still controversial even if it supplemented the knowledge domain [10,12]. Third, although the revised taxonomy provided a more detailed high-level cognitive domain [6], the level of course outcomes analyzed in this study was rarely in the high-level cognitive domain so that it was unsuitable to apply the revised taxonomy as the criteria for the classification of course outcomes in this study. Among the courses analyzed in this study, course outcomes of “patient doctor relationship,” “medical ethics,” “clinical skills,” and “problem-based learning” are supposed to be classified according to the psychomotor or affective domains of the taxonomy of learning objectives. However, considering the small number of outcomes and large amount of the content of the outcomes corresponding to the knowledge domain, the course outcomes of these courses were classified according to the knowledge domain of the taxonomy. This study was approved by the Institutional Review Board of Kyungpook National University Industry Foundation (IRB approval no., 2018-0163).

Results

1. Number of course outcomes according to grades

This study identified 226 and 271 course outcomes in 16 first grade and 23 second grade courses, respectively. In total, 497 learning outcomes were identified (Table 1). The first grade curriculum included nine basic medical science, five clinical medical science, and two medical humanities courses and the number of course outcomes per course ranged from a minimum of three to a maximum of 25 (Table 2). The number of course outcomes per credit ranged from 0.9 to 22.0 (Table 2). The second grade curriculum included 21 clinical medical science and two medical humanities courses and the number of course outcomes per course ranged from a minimum of five to a maximum of 44 (Table 3). The number of course outcomes per credit ranged from 2.0 to 22.0 (Table 3).

Table 1.

Summary of the Number of the Course Outcomes and the Number of the Course Outcomes per Credit Based on Grade and on the Three Categories of Course

Category	No.of courses	Total credits	Total no. of course outcomes	The course outcomes per credit			Bloom’s taxonomy
				No.	Min no.	Max no.	Knowledge	Comprehension	Application	Analysis	Synthesis	Evaluation
Grade
Year 1	16	46.5	226	4.9	0.9	22.0	10	174	17	1	5	19
Year 2	23	43.0	271	6.3	2.0	22.0	23	128	41	8	19	52
Course
Basic medical science	9	29.0	128	4.4	1.8	7.0	5	119	2	0	0	2
Clinical medical science	26	54.5	324	5.9	0.9	22.0	24	160	49	7	21	63
Medical humanities	4	6.0	45	7.5	2.5	22.0	4	23	7	2	3	6
Total	39	89.5	497	5.6	0.9	22.0	33	302	58	9	24	71

Table 2.

The Number of Outcomes for the First-Year Students and Classification of Learning Outcome Verbs Based on Bloom’s Taxonomy

Course	Category of course	Credit	No.of course outcomes	No. of course outcomes per credit	Bloom’s taxonomy
					Knowledge	Comprehension	Application	Analysis	Synthesis	Evaluation
1	A	3.0	21	7.0		21
2	A	3.0	6	2.0		6
3	A	3.0	16	5.3		16
4	A	3.0	17	5.7		17
5	A	3.0	18	6.0		17	1
6	A	3.0	11	3.7	3	8
7	A	3.5	19	5.4		17				2
8	A	3.5	13	3.7	2	11
9	A	4.0	7	1.8		6	1
10	B	3.0	5	1.7			5
11	B	3.0	13	4.3	1	10	1	1
12	B	3.0	21	7.0		7	2		4	8
13	B	3.0	25	8.3		15	3			7
14	B	3.5	3	0.9		3
15	C	1.0	9	9.0		5	3			1
16	C	1.0	22	22.0	4	15	1		1	1
Total		46.5	226	4.9	10	174	17	1	5	19

Category of course: A (basic medical science courses), B (clinical medical science courses), and C (medical humanities courses).

Table 3.

The Number of Outcomes for the Second-Year Students and Classification of the Learning Outcome Verbs Based on Bloom’s Taxonomy

Course	Category of course	Credit	No. of course outcomes	No. of course outcomes per credit	Bloom’s taxonomy
					Knowledge	Comprehension	Application	Analysis	Synthesis	Evaluation
1	B	1.0	6	6.0		2	3	1
2	B	1.0	9	9.0		2	6		1
3	B	1.0	6	6.0		1	1		2	2
4	B	1.5	11	7.3		1	9			1
5	B	1.5	7	4.7		6	1
6	B	1.5	11	7.3	1	4	4	2
7	B	1.5	18	12.0	4	7	6			1
8	B	1.5	27	18.0	2	23				2
9	B	1.5	-	-
10	B	2.0	7	3.5		1	5			1
11	B	2.0	17	8.5	1	3			3	10
12	B	2.0	7	3.5			2		3	2
13	B	2.0	5	2.5	1	2	1			1
14	B	2.0	44	22.0	5	36		2		1
15	B	2.0	37	18.5	9	10		1	1	16
16	B	2.0	14	7.0		13			1
17	B	2.0	-	-
18	B	2.0	5	2.5						5
19	B	3.0	6	2.0					3	3
20	B	3.0	6	2.0					3	3
21	B	3.0	14	4.7		14
22	C	2.0	9	4.5		3	3	1	1	1
23	C	2.0	5	2.5				1	1	3
Total		43.0	271	6.3	23	128	41	8	19	52

Category of course: B (clinical medical science courses) and C (medical humanities courses).

2. Number of course outcomes according to the three course categorization: basic medical science, clinical medical science, and medical humanities courses

The first and second grade curricula included 128, 324, 45 course outcomes in nine basic medical courses, 26 clinical medical courses, and four medical humanities courses, respectively (Table 1). The number of course outcomes of nine basic medical science courses ranged from a minimum of six to a maximum of 21, and that of the course outcomes per credit ranged from 1.8 to 7.0 (Table 2). The number of course outcomes of 26 clinical medical science courses ranged from a minimum of three to a maximum of 44 and that of the course outcomes per credit ranged from 0.9 to 22.0 (Tables 2, 3). The number of course outcomes of four medical humanities courses ranged from a minimum of five to a maximum of 22 and that of the course outcomes per credit ranged from 2.5 to 22.0 (Tables 2, 3). The summary is shown in Table 1.

3. Classification of course outcomes according to Bloom’s taxonomy

The classification of the total 497 course outcomes according to Bloom’s taxonomy resulted in 33, 302, 58, 9, 24, and 71 outcomes in the “knowledge,” “comprehension,” “application,” “analysis,” “synthesis,” and “evaluation” domains, respectively. The detailed results according to grades were showed in Table 1. Table 1 also represents detailed results according to the course classification. The domain in which most course outcome verbs were distributed was the comprehension domain (n=302, 60.8%). The number of verbs classified as belonging to the “knowledge” and “comprehension” domains in Bloom’s taxonomy accounted for 67.4% (n=335) of the 497 verbs used in description of the course outcomes. The number of verbs that belong to the two lower levels in the Bloom’s taxonomy of educational objectives accounted for more than a half of the total number of verbs, exceeding those in the three higher order cognitive levels: application, analysis, and evaluation.

4. Analyzing types of verbs in the course outcomes

The frequency analysis of the verbs used in the description of 497 course outcomes revealed that 78 different Korean verbs were used and that eight of them were unmeasurable verbs (Table 4). The top three types of verbs most frequently used were “understand (21 times),” “recognize (5 times),” and “acquaint (3 times).” Regarding course outcomes verbs, the most frequently used verb was “explain.” Of the total 497 course outcome verbs, 54.9 % (n=273) used this verb.

Table 4.

Usage Frequency of the Unmeasurable Verbs

Unmeasurable verbs used in the outcomes	Frequency
Understand	21
Recognize	5
Acquaint	3
Examine	2
Think	1
Realize	1
Inquire	1
Obtain	1

Discussion

This study aimed to analyze how course outcomes were developed in the process of transforming the existing learning objectives into learning outcomes in the establishment of OBE. The primary goal of this study was to suggest ways to improve strategies in the description of learning outcomes and therefore help professors who are responsible for providing lectures to describe learning outcomes appropriately.

The major findings of this study include the followings. First of all, the number of course outcomes per credit varied according to courses, ranging from 0.9 to 22.0. The courses with a small number of course outcomes were either abstract or beyond the level that can be achieved during the course. The courses with a large number of course outcomes were more like a list of learning contents that should be dealt with during the lessons. In this case, many outcomes described as learning outcomes should be used as lesson outcomes and it is much better to create new course outcomes as competencies that can be acquired through a series of lessons in the course. Second, it was learned that several course outcomes were described with unmeasurable verbs. Even though the school provided professors with guidelines for writing learning outcomes, unmeasurable verbs were still being used. Third, most of the verbs used to describe course outcomes belonged to the two lower levels of Bloom’s taxonomy of educational objectives. The “knowledge” and “comprehension” domains in Bloom’s taxonomy require low degree cognitive activities such as memorization from the learners. The “application,” “analysis,” and “evaluation” domains require learners to practice higher level cognitive activities such as comprehensive, integrated, and creative thinking. This study demonstrated that the lessons were focused on delivering knowledges rather than encouraging utilization of knowledge since verbs associated the level of memorization accounted for 67.4%.

Based on the results of this study, several suggestions regarding the development of better learning outcomes can be made. First of all, the number of course outcomes should be adjusted appropriately in proportion to course credits. The fact that the course outcomes should describe competencies that learners can perform at the end of the course, not list all the topics covered in the lessons, needs to be emphasized. Therefore, the course outcomes should be created at a level that can cover a variety of lesson outcomes. Lee and Yang [1] also pointed out that caution should be exercised when creating learning outcomes in consideration of the inclusiveness of the learning outcomes and not to set learning outcomes that are either too narrow or low. Second, unmeasurable verbs used in description of course outcomes should be changed to overt behavior verbs that enable assessment of the learning results. The course outcomes are sometimes referred to as the evaluation of learning achievements. In other words, the course outcomes should be stated in forms which enable their measurement. This is also emphasized in several literatures that deal with the description of course outcomes [4,5,13]. Finally, it is necessary to elevate the level of the course outcomes from the knowledge acquisition and comprehension level to the application level including “analysis,” “synthesis,” and “evaluation” in order to implement successful OBE in medical school. The primary purpose of OBE is to make learners perform what they have learned, aiming to produce the best performance. Therefore, setting the course outcomes to the level of application, evaluation, and performance, rather than staying at the level of comprehension, is consistent with the goal of OBE. As mentioned in several literatures which pointed out that OBE goes beyond gaining knowledge or training physicians who can perform [2,14-16], course outcomes should be set at a level that aims to achieve actual performances.

This is the first published study in the field of medical education to analyze the course outcomes according to taxonomy of learning objectives. The results of this study can serve as useful material in the development of outcomes since they provide information about how instructors in medical school develop or revise outcomes based on the OBE. This study, however, had some limitations. The results cannot be generalized without considering different educational conditions and curricula since it was conducted only for one medical school. Even though the method applied in organizing educational contents may vary from school to school, the major strength of this study is derived from the fact that all the Korean medical schools have similar educational contents and pursue similar exit competencies.

In further studies, it would be necessary to analyze the content of learning outcomes based on the results of this study. It is expected that the learning outcomes that are more objective and that are more related to exit competencies can be developed when the results of the present study are provided as feedback to the professors who actually develop learning outcomes.