Setting school-level educational goal and objectives with the modified delphi method.

PURPOSE: The aim of this study was to set the school-level educational goals and objectives, based on the needs analysis, by modified Delphi method.
METHODS: A needs analysis and workshops were performed to establish educational goals and objectives. The needs analysis comprised 3 rounds of questionnaires and a panel and reference group that compared the results. Various workshops were held to set, outline, develop, and promote the educational goals and objectives and perform a satisfaction survey.
RESULTS: In the needs analysis, we identified 8 keywords for 'ends' and 12 for 'means' with regard to educational goals and 25 keywords for educational objectives, which were summarized in 5 factors (categories). There were significant differences between the panel and reference groups. Through the workshops, we established new educational goal and objectives that met with high satisfaction among members.
CONCLUSION: The developmental process with which the educational goals and objectives were established through a needs analysis and workshops was effective, efficient, and supportive in medical education.

INTRODUCTION

Educational goal and objectives are descriptions of desirable change in learners. Educational goals and objectives are categorized broadly by nation, society, and school and narrowly by instruction and learning objectives. Instructional objectives, behavioral objectives, and performance objectives have similar meanings as educational objectives, but educational objectives constitute a superordinate concept [1]. Instructional objectives are used commonly, focusing on the outcomes of learners in instructional activities. Educational goal and objectives can be developed at the school, curriculum, and instructional levels. School-level educational goal and objectives should express the core values of an organization and guide school policy. School-level educational goal and objectives tend to be represented in very general terms and as global statements [2] and comprise 2 parts; an educational goal and educational objective. Goals are relatively abstract, and educational objectives are specific [3]. At the school level, an educational goal is similar to a mission, and educational objectives are similar to core values. Mission has 2 components: ‘ends’ and ‘means’ [4].

There are many reports on educational objectives, but those on school-level educational goals and objectives are uncommon. Some schools cooperate with outside professionals on a large scale to establish goals and objectives, whereas others establish or modify educational objectives with merely several staff members. Desirable educational objectives reflect the school’s reality and are supported by members and upgraded periodically. Large-scale projects require large budgets and do not easily reflect the conditions of a specific school. Small-scale projects that are run by a select few tend to receive little support from staff members.

Several methods have been used to develop educational objectives, such as the Delphi method [5], crosspurpose matrix [6], Q-sort technique [7], trend extrapolation, force field analysis, scenario writing, and fishbowl analysis [1]. Workshops or meetings with experts are often held to develop educational objectives [8].

The traditional Delphi method involves 3 rounds, in all of which the same experts participate. Variations on the Delphi method have been reported-increasing the rounds from 2 to 5 and varying the number of participants in each round [9]. The size of groups in the Delphi method ranges from 7 to 350 people [10]; however, the most effective expert panel size is 10 to 15 members [11]. The modified Delphi method has been reported by many groups. The Delphi method has been used to examine common factors across theories [12]. By exploratory factor analysis, 771 variables that are considered to be essential to supervisory outcomes were subdivided into 5 categories and 37 conceptual clusters [13]. The modified Delphi method has also been used to identify and prioritize issues in information systems research, in which peer debriefing and member checks are included to ensure the reliability and validity of the process. A panel group had a reference group to compare opinions and improve the reliability of the results [14]. Qualitative interviews have been conducted with selected panels to clarify discrepancies and interpret data. An internetbased Delphi survey has been used to identify the communication skills that are deemed to be most important for practicing physicians [14].

We developed school-level educational goals and objectives for 2 reasons. First, 5 years had passed since the existing educational goal and objectives were set, requiring a periodic review. Second, we had hoped to include the opinions of our staff, especially professors, on the new educational goals and objectives so that faculty members could actively apply them in their teaching.

Two research questions were posed.

1) What are the preferred keywords that describe the educational goals and objectives with regard to the results of the needs analysis?

2) Are staff members satisfied with the new educational goals and objectives?

SUBJECTS AND METHODS

Fig. 1 shows the research process: phase 1 is the needs analysis, and phase 2 is the establishment of the educational goals and objectives.

Fig. 1.

Process for Setting the Educational Goal and Objectives

1.Phase 1: need analysis

1) The needs analysis

Phase 1 of the needs analysis used a modified Delphi method to analyze the keywords that the panel considered valuable. There were 3 rounds, each with specific aims. The aim of round 1 was to identify keywords that were related to the educational goal and objectives, which were collected from an open-ended questionnaire that was completed by the chief professors of each subject and by student representatives. In round 2, the panel’s opinions were gathered using a structured questionnaire that was developed by the research committee, based on the results of round 1. The aim of round 2 was to analyze the priorities that were attached to the keywords, compare the importance that was given to the keywords by the panel and reference groups [14], and summarize the keywords. A statistical analysis was performed to identify the members’ needs [13]. All professors, students, alumni, and parents were surveyed. The aim of round 3 was to reach a consensus, in which the panel’s opinions (all professors) were obtained using the same questionnaire from round 2.

In this study, medical professors who were employed at the same medical school were chosen for the panel. We selected professors from the school for 3 reasons. First, they are experts in their fields and in medical education. Second, they play a key role in the planning, teaching, and evaluation of medical education. As discussed, one purpose of this study was to encourage professors to apply the new school-level educational goals and objectives to their instruction-level educational goal and objectives. Third, we intended to use the questionnaires to promote and disseminate the development and content of the educational goals and objectives to professors.

2) The modified Delphi method questionnaire

The questionnaire comprised two parts on the educational goal and objectives. The educational goal was described in terms of ‘ENDS’ and ‘MEANS’ [5]. Our questions on ENDS and MEANS were: (1) ‘What should the ENDS of our school be?’ and (2) ‘What are the appropriate MEANS to achieve the ENDS of our school?’

The questions on educational objectives were: ‘Who would make a good doctor?’ (round 1) and ‘How important are these qualities in a doctor?’ (rounds 2 and 3). The importance of items was rated on a 5-point Likert scale: ① very unimportant, ② somewhat unimportant, ③ neutral, ④ somewhat important, and ⑤ very important.

3) Analysis method

We analyzed the data using frequency analysis for all variables and an independent samples t test and one-way analysis of variance (ANOVA) to identify differences between groups. The keywords for educational objectives were identified using exploratory factor analysis by varimax rotation with SPSS version 18.0 (SPSS Inc., Chicago, USA)

2. Phase 2: setting achool-level educational goals and objectives

In Phase 2, the goal and educational objectives were established. The educational goal and objectives were developed in 3-step workshops. In Step 1, current and former professors attended a core staff workshop, during which they reviewed the results of the needs analysis and shared information about trends in medical education. The purpose of the core staff workshop was to introduce the research process and develop an outline of the educational goal and objectives. In Step 2, committee workshops were held to describe the educational goal and objectives systematically and specifically, based on the results of the core staff workshop. The committee contained some faculty members of the current administrative and medical education office. In Step 3, the new educational goal and objectives were introduced and verified with a satisfaction survey. The questionnaire explored the participants’ degrees of satisfaction with the educational goal and objectives and their opinions on the aspects of medical education that they considered to be valuable currently and in the future.

RESULTS

1. Data

We applied this developmental model to the medical school of Konkuk University, Korea, from 2008 to 2010. In the first phase, in the needs analysis by modified Delphi method, the response rates for the panel group were 77% (chief professors, n=23/30) in the first round, 25.76% (all professors, n=51/198) in the second round, and 31.31% (all professors, n=62/198) in the third round. The response rates for the student reference group were 100% (student representatives, n=16/16) in the first round and 41.95% (all students, n=73/174) in the second round. In the second round, the response rate of parents was 26.44% (n=46/174), and that of alumni was 3.77% (n=22/583).

In the second phase, there were 17 current and former professors in Step 1 and 11 committee members in Step 2, and 80 professors (40.40%) completed the satisfaction survey in Step 3.

2. Phase 1: needs analysis

1) What should the ENDS of our school be?

In the first round, the chief professors in each division and student representatives answered open-ended questions. Researchers selected 8 keywords from the responses that were used in the round 2 questions. Table 1 shows the highest priorities of ENDS among the panels between rounds 2 and 3. The panels’ responses changed after they received the responses, including the reference group in round 3: ‘promotion of health’ (37.3% to 56.5%), ‘respect the value of life’ (25.5% to 19.4%), and ‘healthy lifestyle’ (17.6% to 16.1%).

Table 1.

The Top Priority Keywords on the ‘ENDS’ of the Educational Goal (%)

	Panel group			Reference group
	Round 2	Round 3	Change %	Round 2
Promotion of health	37.3	56.5	19.2	32.6
Respect the value of life	25.5	19.4	-6.1	19.9
Healthy lifestyle	17.6	16.1	-1.5	10.6
Protection of lives	3.9	3.2	-0.7	14.9
Medical progress	5.9	3.2	-2.7	9.2
Human happiness	3.9	1.6	-2.3	5.7
Human respect	5.9	0.0	-5.9	5.0
Realization of a just society	0.0	0.0	0.0	0.7
Missing	0.0	0.0	0.0	1.4
Sum	100.0	100.0		100.0

2) What are the appropriate MEANS to achieve the ENDS of our school?

From the results of the first round, the research committee identified 12 keywords. Table 2 shows the highest priorities in both rounds. The response rates also changed: ‘give priority to the patient’s health’ (21.6% to 41.9%), ‘develop competency for excellence’ (11.8% to 22.6%), and ‘develop good doctors’ (19.6% to 11.3%).

Table 2.

The Top Priority Keywords on the ‘MEANS’of the Educational Goal (%)

	Panel group			Reference group
	Round 2	Round 3	Change %	Round 2
Give priority to the patient’s health	21.6	41.9	20.4	24.8
Develop competency for excellence	11.8	22.6	10.8	17.0
Develop good doctors	19.6	11.3	-8.3	11.3
Work as a primary care physician	15.7	6.5	-9.2	0.7
Be a warm-hearted doctor	9.8	6.5	-3.4	11.3
Develop future-oriented experts	3.9	3.2	-0.7	4.3
Provide creative research and education	3.9	1.6	-2.3	6.4
Educate for the whole person	3.9	1.6	-2.3	7.8
Be a lifelong educator	5.9		-5.9	7.1
Cultivate doctors’ talents	2.0		-2.0	2.1
Demonstrate international insight	2.0		-2.0	4.3
Develop expertise in different fields			0.0	1.4
Missing		4.8		1.4
Sum	100.0	100.0		100.0

3) ‘Who would make a good doctor?’ and ‘How important are these qualities in a doctor?’

The committee identified 25 keywords from the results of the first round. Table 3 shows the mean scores for importance. By ANOVA, we noted significant differences in the ratings of some keywords concerning the qualities of good doctors between groups: ‘mission’, ‘social leader’, ‘promotion of the medical system’, ‘medical society leadership’, ‘diversity of knowledge’, ‘medical educator’, ‘devotion’, and ‘global leader’. Of these qualities, by Scheffe post hoc test, students and parents scored the following keywords significantly higher than professors and alumni: ‘global leader’, ‘medical educator’, ‘promotion of the medical system’, and ‘medical society leadership’.

Table 3.

Keywords for Educational Objectives: Results of Factor Analysis and ANOVA in Delphi 2

	Factor analysis					ANOVA
						Panel group	Reference group
	Factor 1	Factor 2	Factor 3	Factor 4	Factor 5	Professors	Graduates	Students	Parents
Honesty	0.800	0.107	0.085	0.053	0.021	4.45	4.64	4.42	4.53
Understanding the patient	0.783	0.170	0.177	0.092	0.241	4.35	4.59	4.48	4.53
Priority given to the patient’s life vs. profit	0.728	0.103	-0.012	0.356	0.159	4.29	4.45	4.42	4.58
Mission*	0.722	0.031	0.153	0.199	0.232	4.33	4.73	4.45	4.69
Warm-heartedness	0.707	0.096	0.174	0.056	0.133	4.31	4.43	4.36	4.38
Sincerity and patience	0.663	0.149	0.390	-0.072	0.115	4.10	4.38	4.18	4.24
Rational thinking	0.624	0.377	0.137	-0.127	–0.183	4.48	4.41	4.33	4.20
Promotion of public health	0.574	0.421	0.066	0.179	0.118	3.92	3.86	4.18	4.11
Physically and mentally healthy doctor	0.543	0.217	0.285	0.260	0.069	4.24	4.50	4.32	4.27
Self-management	0.539	0.154	0.226	0.277	0.065	3.78	4.23	4.03	4.16
Social Leader*	0.138	0.753	0.011	0.256	0.139	3.47	3.55	3.90	3.73
Understanding of business	0.144	0.750	0.214	0.047	0.046	3.48	3.43	3.63	3.51
Promotion of the medical system**	0.140	0.736	0.121	0.228	0.238	3.47	3.64	4.16	3.84
Medical society leadership**	0.074	0.716	0.057	0.213	0.431	3.41	3.59	3.97	3.78
Cooperation with other health professionals	0.329	0.565	0.153	0.186	0.029	4.06	3.86	4.04	3.73
Happy doctor	0.489	0.511	0.334	-0.315	0.050	3.86	4.05	4.19	3.80
Knowledge of the primary care physician	0.224	0.195	0.773	-0.003	0.004	3.90	4.14	4.01	4.12
Diversity of knowledge*	0.182	0.083	0.753	0.312	0.070	3.77	3.81	4.10	4.20
Communication skills	0.416	0.334	0.483	0.168	0.046	4.14	4.14	4.38	4.09
Outstanding research ability	0.089	0.229	0.075	0.741	0.191	3.84	3.71	4.05	4.11
Medical educator*	0.144	0.185	0.177	0.660	0.061	3.69	3.48	3.90	4.04
Devotion*	0.502	0.274	-0.019	0.569	0.022	3.84	4.32	4.07	4.22
Challenge and creativity	0.199	0.146	0.052	0.152	0.784	3.98	4.05	4.07	4.38
Global leader**	0.086	0.305	-0.015	0.118	0.775	3.58	3.43	3.88	4.20
Competent doctor	0.425	0.038	0.449	-0.100	0.525	4.52	4.68	4.48	4.60

p<0.05,

p<0.01.

An exploratory factor analysis was performed to identify the themes of the keywords for educational objectives. The analysis yielded 5 meaningful factors: (1) basic competency (22.62% of variance explained), (2) social responsibility and relationships (14.57%), (3) knowledge and skills (9.09%), (4) academic qualifications (8.47%), and (5) competency for excellence (8.02%). Table 3 presents the loading of each item under these factors, following varimax rotation. Considering the ANOVA results, students and parents rated keywords in ‘social responsibility and relationships’ higher than professors and alumni.

4) Did the panel group reach a consensus between Delphi rounds?

The professors participated in Delphi rounds 2 and 3 as a panel. As Tables 2 and 3 indicate, the percentages of the highest priority increased (ENDS 19.2%, MEANS 20.4%) and the number of response keywords decreased (ENDS, 1 keyword; MEANS, 3 keywords) between rounds 2 and 3.

Of the 25 keywords for educational objectives in Table 4, some were rated significantly lower in round 3 than in round 2: ‘rational thinking’, ‘social leader’, ‘understanding of business’, ‘cooperation with other health professionals’, and ‘medical educator’.

Table 4.

The Mean Differences in the Panel Group between Rounds 2 and 3

		Round 2	Round 3	t	p-value
1	Honesty	4.45	4.23	1.53	0.129
2	Understanding the patient	4.35	4.33	0.13	0.895
3	Priority given to the patient’s life vs. profit	4.29	4.31	-0.14	0.891
4	Mission	4.33	4.31	0.12	0.909
5	Warm-heartedness	4.31	4.17	0.92	0.359
6	Sincerity and patience	4.10	3.95	0.99	0.325
7	Rational thinking**	4.48	4.14	2.83	0.006
8	Promotion of public health	3.92	3.97	-0.37	0.713
9	Physically and mentally healthy doctor	4.24	4.03	1.34	0.185
10	Self-management	3.78	3.86	-0.54	0.588
11	Social Leader*	3.47	3.17	2.10	0.039
12	Understanding of business**	3.48	3.10	2.69	0.008
13	Promotion of the medical system	3.47	3.40	0.61	0.543
14	Medical society leadership	3.41	3.41	-0.04	0.968
15	Cooperation with other health professionals*	4.06	3.72	2.60	0.011
16	Happy doctor	3.86	3.71	0.80	0.425
17	Knowledge of primary care physician	3.90	3.97	-0.39	0.698
18	Diversity of knowledge	3.77	3.81	-0.24	0.812
19	Communication skills	4.14	4.12	0.15	0.884
20	Outstanding research ability	3.84	3.56	1.89	0.062
21	Medical educator**	3.69	3.26	3.09	0.003
22	Devotion	3.84	3.80	0.27	0.787
23	Challenge and creativity	3.98	3.91	0.43	0.669
24	Global leader	3.58	3.34	1.46	0.147
25	Competent doctor	4.52	4.39	1.00	0.321

p<0.05,

p<0.01.

3. Phase 2: establishing educational goals and objectives

Three-step workshops were held to establish new educational goals and objectives: 1 for the core staff, 1 for committee members, and 1 for all faculty members.

1) Core member workshop

Seventeen core staff members were divided into 4 groups. These staff members checked, shared, and discussed the results of the needs analysis, information about changes in medical education, and various types of educational goals and objectives that are pursued worldwide. After brainstorming in their respective groups, each group suggested educational goal and objectives. After sharing the group results, all participants reached a consensus on keywords to describe 3 new items: ‘competency for excellence’, ‘human kindness’, and ‘social responsibility’.

2) Committee workshops

After considering the outcomes of the core staff workshop, 11 committee members selected 4 main keywords and 3 subcategories in each category of the new educational goals and objectives during 8 workshops, not including preparation meetings.

3) New educational goals and objectives

The new educational goals and objectives were articulated as follows.

a. Goal

To educate students to be competent and creative professional doctors who respect the value of life according to the school philosophy: ‘Sincerity, fidelity, and righteousness’.

b. Educational objectives (MOVE)

Medical leader (M): The medical leader’s mission is to foster leadership, self-development, and self-management.

Others first (O): Doctors give priority to the patient’s life and health, communication, and honesty.

Value diversity (V): Doctors value diversity by showing an understanding of others and other cultures, the demands of other jobs, and international viewpoints.

Excellent competency (E): Excellence includes problem solving, lifelong learning, and international competitive competency.

The goal was created by reviewing the results of the ENDS and MEANS that were identified in the needs analysis, considering changes in medical education. The goal of ‘realizing the life-respect value’ was classified as an END, and ‘students to be competent and professional doctors’ was classified as a MEANS. The intent of the new educational objectives—under the acronym MOVE— was to express dynamism and action.

4) Satisfaction survey

In the workshop, the new educational goal and objectives were introduced to all professors, who were surveyed on their satisfaction. Forty percent of the professors answered the questions, and most were positive (‘somewhat agree’ or ‘strongly agree’): (1) ‘Are you satisfied with the educational goal and objectives?’ (86.3%), (2) ‘Do you agree that the new educational goal and objectives reflect the members’ opinions?’ (80.1%), (3) ‘Do you agree that the educational goal and objectives contain items that are valuable for the present and future?’ (87.6%), and (4) ‘Do you agree with the general intent and specific content of the educational goal and objectives?’ (85.1%).

DISCUSSION

We established school-level educational goals and objectives, based on a needs analysis by modified Delphi method, for a medical school. Evidence that support its use in this context was obtained.

Including professors in the needs analysis and establishment of educational goals and objectives effected high levels of satisfaction, which will aid in realizing new educational goal and objectives in medical education. In this project, we created school-level educational goals and objectives, but if we want to realize them, we must apply curriculum-and instruction-level educational goals and objectives. The professors play a significant role at both levels of education; thus, they were actively involved in the panels in all Delphi rounds and workshops. More than 80% of the professors agreed with the new educational goal and objectives. This high satisfaction with the educational goal and objectives, as with the mission, will help incorporate educational goals and objectives into the curriculum and teaching. A high degree of employee satisfaction with the mission is an important precursor to his commitment, and the more that a company’s mission influences an employee’s behavior, the greater the performance (product) [5]. Consequently, a school-level project with a needs analysis and several workshops is a reasonable means of promoting educational goals and objectives and persuading all staff members of the outcomes.

It was an effective and efficient method for staff members to ask directly about the keywords for the educational goal and objectives. We asked staff members about the keywords for ENDS and MEANS to create the educational goals and objectives, and we analyzed the results by priority and summary. These results provided practical information about establishing educational goals and objectives to committee members. Fundamentally, medical educational trends were considered fully in all processes.

We used a reference group to compare the panel’s opinions with those of the students, parents, and alumni. Coico et al. [14] used an internet-based Delphi method to establish guidelines for preclerkship bioterrorism curricula using 64 medical educators as panel members and 12 bioterrorism experts as a reference group. In this study, although the keywords on the goal did not differ significantly between groups, some keywords on educational goals and objectives varied between groups. Of the 5 factors that were identified in the exploratory factor analysis (‘basic competency’, ‘competency for excellence’, ‘social responsibility and relationships’, ‘academic qualifications’, and ‘knowledge and skill’), the student and parent groups displayed greater interest in the ‘academic qualification’ and ‘social responsibility and relationships’ items. We infer that these results reflect the students’ and parents’ hopes that the students will secure a job after graduation. Students and parents desired for the student to work eventually as a paid doctor in a university hospital (students, 32.5%; parents, 27.8%), a university professor (students, 2.4%; parents, 56.5%), a physician at an independent health care center (students, 29.6%; parents, 13%), and in a medicinerelated position (students, 2.8%; parents, 2.6%) [15].

The new educational goal and objectives have similarities and differences from those in previous reports. The new educational goal and objectives are described by the acronym MOVE: ‘medical leader’, ‘others first’, ‘value diversity’, and ‘excellent competency’. Factors that are in common with other reports are ‘medical leader’, ‘others first’, and ‘excellent competency’. ‘Value diversity’ in this study was distinct. A good doctor must understand the diversity of his patients’ backgrounds, including differences in gender, race, culture, and employment. An understanding of diversity improves communication skills, and valuing diversity is essential to maximizing one’s capacity to design approaches to progress [16]. By understanding and valuing diversity, students can improve their understanding and creativity.

Some (round 1) and all (2 and 3 round) faculty members in a school participated as panel members in each process. In this case, the key points were sufficient to develop the educational goal and objectives and practical enough to apply them in the curriculum, but there is no sufficient evidence to demonstrate their professional competency as medical educators. Thus, it will be better to adopt professional medical educators as a reference group for comparison. We also surveyed all staff members on their satisfaction of the new educational goal and objectives, but it will be more valid to involve those who participated in the needs analysis—students, parents, and alumni.

In summary, implementation of these consensus learning goals and objectives by modified Delphi method drew a high level of faculty satisfaction and recognition. We hope that faculty members will apply the new learning goals and objectives to their curriculum and instructions actively, for developing medical students into good doctors.