Program evaluation of an Integrated Basic Science Medical Curriculum in Shiraz Medical School, Using CIPP Evaluation Model.

INTRODUCTION: In recent years curriculum reform and integration was done in many medical schools. The integrated curriculum is a popular concept all over the world. In Shiraz medical school, the reform was initiated by stablishing the horizontal basic science integration model and Early Clinical Exposure (ECE) for undergraduate medical education. The purpose of this study was to provide the required data for the program evaluation of this curriculum for undergraduate medical students, using CIPP program evaluation model.
METHODS: This study is an analytic descriptive and triangulation mixed method study which was carried out in Shiraz Medical School in 2012, based on the views of professors of basic sciences courses and first and second year medical students. The study evaluated the quality of the relationship between basic sciences and clinical courses and the method of presenting such courses based on the Context, Input, Process and Product (CIPP) model. The tools for collecting data, both quantitatively and qualitatively, were some questionnaires, content analysis of portfolios, semi- structured interview and brain storming sessions. For quantitative data analysis, SPSS software, version 14, was used.
RESULTS: In the context evaluation by modified DREEM questionnaire, 77.75%of the students believed that this educational system encourages them to actively participate in classes. Course schedule and atmosphere of class were reported suitable by 87.81% and 83.86% of students. In input domain that was measured by a researcher made questionnaire, the facilities for education were acceptable except for shortage of cadavers. In process evaluation, the quality of integrated modules presentation and Early Clinical Exposure (ECE) was good from the students' viewpoint. In product evaluation, students' brain storming, students' portfolio and semi-structured interview with faculties were done, showing some positive aspects of integration and some areas that need improvement.
CONCLUSION: The main advantage of assessing an educational program based on CIPP evaluation model is that the context, input, process and product of the program are viewed and evaluated systematically. This will help the educational authorities to make proper decisions based on the weaknesses and strengths of the program on its continuation, cessation and revision. Based on the results of this study, the integrated basic sciences course for undergraduate medical students in Shiraz Medical School is at a desirable level. However, attempts to improve or reform some sections and continual evaluation of the program and its accreditation seem to be necessary.

Introduction

In recent years, curriculum reform and integration was reported in many medical schools. The integrated curriculum is a popular concept in medical schools all over the world. The main aim of integration is to break the boundaries between basic and clinical sciences. Integration supports maintenance of knowledge by development of thought and the application of concepts (1).

In Shiraz medical school, the integrated program was initiated by establishing a central curriculum committee for revision of basic sciences courses in 2007 (2). Considering different types of integration models, analyzing the situation, doing library-based studies, studying the viewpoints of defenders and detractors and after a visit some professors from Canada, Britain, Malaysia and Singapore universities, the horizontal integration model was confirmed and Early Clinical Exposure (ECE) for vertical integration was considered (3).

For program evaluation, the Context, Input, Process and Product (CIPP model) was used to measure the difficult features of medical education programs and provide formative evaluation data to stakeholders with the aim of improvement of the program (4-6). The first component, context evaluation, is useful when an established program is going through a planned change or must adjust to the conditions changed. The second part, input, helps to establish an appropriate program model to assure the identified needs. Process evaluation provides formative data for guiding revisions whilst the program is running. The last component, product evaluation, produces valuable information in order to judge program outcomes (7,8).

This article elaborated the use of the CIPP evaluation model as a comprehensive framework to help to initiate, develop, install, and evaluate a new undergraduate medical education program in a medical school in the south of Iran.

As the CIPP model is based on this concept that the most important aim of assessment is to improve and modify but not to confirm a plan, in this paper we aimed to provide some data for decision makers to improve the quality of the integrated medical curriculum, using four parameters (context, input, process and product).

Methods

Study context

This study was done in Shiraz Medical School and Education Development Center of Shiraz University of Medical Sciences. Shiraz Medical School is one of the largest and oldest schools in the south of Iran. The program of this school was composed of two and half years of integrated basic sciences, one year of pathophysiology of diseases, a one-year externship, and 18-month internship. The program contains Early Clinical Exposure (ECE) for integrated basic sciences curriculum.

Data collection

The present study is a triangulation mixed method study carried out in Shiraz Medical School in 2012 to look into the quality of the integrated basic sciences course curriculum for undergraduate medical students based on the views of basic sciences professors and first and second medical students, using CIPP evaluation model. In this study, both qualitative and quantitative tools were used. Table 1 shows four complementary sets of evaluation phases which were compatible with the four components of the CIPP evaluation model.

Table 1

Components of CIPP evaluation model for evaluation of integrated basic sciences curriculum in Shiraz University of Medical Sciences

Domain under study	Variables measured	Tools		Target group
		Qualitative	Quantitative
Context		Review of current evidence on integration and consultation with experts	Modified DREEM questionnaire	Students, faculty and administrators
Input	Content of curriculum	-	A researcher made questionnaire	Faculties and curriculum committee
Process	The process of learning	-	1-Researcher-made questionnaires for evaluating the quality of each integrated course	First and second year medical students
	Process of teaching	-	2-Researcher-made questionnaires for evaluating the quality of ECE
Product	Students’ Performance The process of teaching and learning	1-Learner centered integrated basic science portfolios	-	First and second year medical students
		2-Brainstorming (students)
		3- Semi- structured interview (professors of basic sciences)		Professors of basic sciences

Phase 1: Context

In order to understand the need for change, we conducted a complete context evaluation in 2012. The projects included review of the existent literature on integration, consultation with experts in curriculum design and assessment of the educational environment from the students’ perspectives, using the modified Dundee Ready Education Environment Measure (DREEM) (2,9).

Phase 2: Input

A researcher made questionnaire based on the five point Likert scale was used for input evaluation. The questionnaire consists of some questions about facilities in laboratories, available equipment, etc. The content validity of the questionnaire was determined by medical education experts. Reliability of the questionnaire was determined after a pilot study (r=0.83).

Phase 3: Process

A researcher made questionnaire based on the five point Likert scale was used to evaluate the quality of 9 integrated modules including the genital system, digestive system, skeletal system, nervous system, endocrine system, visual and auditory systems, cardiovascular system, renal system, and pulmonary system. The validity and reliability of the questionnaires are determined in our previous study (2).

A researcher made questionnaire was used to assess the quality of the ECE, containing 7 questions on the viewpoints of first-year medical students. The validity and reliability of the questionnaire were confirmed in our previous study (2).

Phase 4: Product

For measuring the product, we designed a portfolio for assessing the students’ learning and reflection. The students reported their reflection in their portfolio and also brainstorming sessions with students was done to obtain their view and pros and cons about integration systems. Another method used for process evaluation was the semi-structured interview with basic sciences faculties. These semi-structured interviews about integration were done by an expert researcher through the conversational style. The view of basic science was gathered. Some of the samples of the interview questions are mentioned below:

To what extent was the integration of clinical cases in the courses of basic sciences effective? How can the integration plan be improved? Was the volume of course content decreased in your department? If yes, how and if no, what is your suggestion? And what is your general opinion about the integration phases and your suggestions for further changes?

Statistical analysis

The statistical analysis was done on quantitative data using SPSS software, version 14. For analysis of qualitative data, content analysis method was used. For trustworthiness of data in the qualitative section, member check and peer check were done.

Results

Context Evaluation

In this part, reviewing the current evidence on integration and consultation with experts were done. Also, the modified DREEM questionnaire was used to measure the educational environment. Table 2 shows the results of modified DREEM questionnaire (Table 2).

Table 2

Students’ viewpoints about the educational setting

No.	Items	Percent of students that are agree
1	This educational system encourages me in active class participation.	77.75
2	Teachers are knowledgeable enough to teach courses in the integration program.	35.91
3	Teachers value the students’ viewpoints.	42.73
4	Presentation of the materials in the integration system results in my progress now and in future.	43.62
5	Courses are well scheduled in the integration system	87.81
6	Examination time is well scheduled.	79.12
7	Manner of presentation and arrangement of the integrated courses are boring.	5.10
8	In this system, teachers establish a good relationship with the students.	42.11
9	Education in this system reinforces my self-confidence.	45.51
10	For the education of each course in this system, a specific time period is assigned.	51.54
11	This education system emphasizes real learning.	66. 22
12	I can put most of the materials in my long-term memory.	62.76
13	Professor advisors provide the students with proper education feedback.	58.69
14	The integration system has made me more interested in education.	46.29
15	The office of vice-chancellor for education has provided a proper situation for constructive criticism.	76.72
16	Teachers provide proper situations for constructive criticism.	45.41
17	Teachers give clear examples for learners’ better understanding.	55.57
18.	Learning objectives in the integration system are clear for me.	35.28
19	Teachers precede class sessions based on educational objectives.	63.56
20	Teachers are dissatisfied with this education system.	31.21
21	Despite the stresses in the new system, I enjoy more when participating in it.	67.72
22	The atmosphere of the class sessions motivates me more as a learner	83.86
23	The educational setting is favorable.	54.81
24	Long-term learning is more valued than short-term learning.	61.22
25	Education is more teacher-oriented.	19.11

Input Evaluation:

The result of input evaluation is shown in Table 3.

Table 3

The result of input evaluation

Variable	Number of responses	Mean±SD (from 5)
Faculties interest in teaching	157	3.63±1.14
Competent instructors	157	3.26±1.87
Complete teaching facilities in classrooms	157	3.46±1.84
Capability of laboratories	157	3.10±1.78
Sufficient facilities in libraries	157	3.19±1.81
Allocation of budget and financial resources	157	3.25±1.12
Sufficient cadavers	157	1.95±1.19
Need assessment of modules	157	3.89±1.22
Training program for faculties before stating the courses	157	3.65±1.06
The length of basic science education	157	1.84±.98

Process Evaluation

Process of learning- teaching the ECE with patient:

The results of students’ views about early clinical exposure sessions are shown in Figure 1.

Figure1

Students’ viewpoints about the quality of Early Clinical Exposure program

The results of quality of education of each integrated module:

Table 4 shows a sample of the results of the quality of education in the digestive system from the students’ viewpoint (Table 4).

Table 4

The results of quality of digestive system integrated module

Variable	Number of respondents	Min	Max	Mean±SD (From 5)
Introduction to Anatomy and physiology of abdomen	136	3	5	3.83±1.97
Nerve roots, veins, arteries in abdomen	136	3	5	4.16±1.69
Integrated anatomy and physiology of Inguinal canal	136	3	4	3.58±1.10
Integrated anatomy and physiology of liver	136	3	5	4.19±1.78
Integrated anatomy and physiology of GI tract	136	2	5	3.66±1.75
Integrated anatomy and physiology of spleen	136	3	5	4.21±1.15
Integrated anatomy and physiology of nervous system in GI tract	136	2	4	3.27±1.32

Product Evaluation:

Brainstorming sessions

Discussion at brainstorming session with students focused on 2 primary areas: determining the positive and negative points of the integrated curriculum and defining the necessary changes needed within this integrated curriculum to address the future demands of medical students.

Some of the positive and negative points expressed by students are mentioned below:

Drawing the attention of students toward basic sciences

Active and in depth learning

Better application of theoretical knowledge in basic sciences

More joy from studying basic sciences

Relevance of presented topics to each other

▪ Curriculum overload

▪ Confusion and stress about new integrated modules

* More cooperation between faculties and students

* Consistent course template to facilitate access to content

* Advanced study guides for each integrated model

* Technology enhanced teaching and learning facilities

* Alternative education materials such as a well defined course syllabus

Educational portfolio

One instrument that was designed in this study for evaluating the product was a learner centered integrated basic science portfolio. Students are expected to complete the portfolio and especially mention their educational activities and their reflection. Most students mentioned their satisfaction of learning in integrated curriculum in their portfolio and believed that the process of learning in this new curriculum is better and they are able to learn the materials of different disciplines as a whole. Students ascertained that consolidation experiences such as using problems, quizzes and formative assessments help them to solidify their understanding and improve their learning.

Semi structured interview with faculties

Semi-structured interview was done with basic science faculties that were responsible for teaching and learning. The following results were obtained:

o After implementation of horizontal integration, the amount of duplicate material in different disciplines decreased.

o To some extent, separating the core curriculum was done in some modules.

o Faculties from different disciplines are aware about their colleagues’ activities.

* Basic sciences comprehensive exam should be redesigned based on the integrated modules and Early Clinical Exposure program.

* There should be an opportunity to teach some topics at patient bed side.

* The volume of curriculum should be lowered more.

Discussion

The major aim of this study was to evaluate the integrated curriculum for basic sciences in undergraduate medical students in Shiraz Medical School, based on the CIPP evaluation model. The CIPP is a useful model for stakeholders for evaluating their educational program.

The results of the present study showed that in content evaluation, educational environments in Shiraz Medical School are satisfying in comparison to our previous studies (2,10). The results of a study on evaluation of integrated curriculum in Tehran University of Medical Sciences showed that there was no significant difference between traditional and integrated curriculum in overall DREEM questionnaire, but in some domains students in the integrated curriculum evaluated the environment better than those in the traditional curriculum (8).

In the input evaluation

The duration of training period was mentioned as the most important obstacle that was similar to other published article in Iran (11,12). It seems necessary that the duration of basic sciences should be decreased to 2 years.

As to facilities, the shortage of cadavers for education was mentioned as a problem in Shiraz Medical School. In recent years, it has been recognized that anatomy laboratory education using cadavers for teaching human morphology and also some aspects of professionalism such as dealing with death should be emphasized in the curriculum (13).

In the process evaluation

We used a valid and reliable researcher made questionnaires based on the Likert scale for each course. As compared to our previous study, the results of the quality of integrated courses are improved over the time (2).

Another questionnaire used for process evaluation in this study was the Early Clinical Exposure (ECE) questionnaire that showed improvement in these sessions in comparison to our previous studies (2,3). Similar results were reported from other published articles (14-17).

In the product evaluation, we used the students’ brainstorming session and portfolios and semi- structured interview with basic science faculties. In comparison to our previous study, the results were better than before. Students and faculties are now more familiar with integration system in Shiraz Medical School (2). Well informed faculty members ensure the successful implementation of every curriculum. Regular communication between faculties is very necessary for the success of integration programs (18). Faculties also recommended changes in assessment methods, especially for comprehensive exams. One of the tips in integration program that is mentioned in Melik et al.’s study is selecting appropriate assessment method (18).

The results of students’ portfolios and brainstorming sessions showed that students believed that horizontal integration was positive and it seems that presentation of basic science in a simultaneous and integrated manner led to a better understanding of the subjects. The students ascertained the need for decreasing the amount of educational content and using new methods of teaching and learning in new curriculum. Sobral reported similar results (19).

Also, Brauer and Ferguson emphasized that integrated curricula should be designed to separate the core content from unnecessary content but should not diminish the importance and necessity of basic sciences (1).

Yamani et al. reported that the concepts of core curriculum and integration are closely linked together and indeed integration is one of the models of designing core curricula and both concepts address the need for creating meaningful learning (20).

This study was done because it was one of the important priorities in medical education research in Iran and Eastern Mediterranean Region (21,22). One of the limitations of our study was that data were collected based on the student and faculties’ viewpoint that may not show a precise description of situation. Another limitation was that we did not have control group for comparing the results.

Conclusion

The main advantage of assessing an educational program based on CIPP evaluation model is that the context, input, process and product of the program are viewed and evaluated systematically. This will help the educational authorities to make proper decisions based on the weaknesses and strengths of the program on its continuance, cessation and revision. Based on the results of this study, the integrated basic sciences course for undergraduate medical students in Shiraz Medical School is at a desirable level. However, attempts to improve or reform some sections and continue the evaluation of the program and its accreditation seem necessary. The following suggestions are required for improving the integrated program in future:

- Using a wider range of factors for evaluation in the domains of context, input, process and product.

- Using a wider range of interest groups, involved in the program evaluation such as top managers and executives, graduates, supervisors in the health system and the university staff, in order to evaluate the quality of the program more precisely.

- Evaluating the teaching-learning methods for each of the integrated courses based on the professors and students’ viewpoints, using a qualitative study.

- Designing the same standard assessment tool with usability to all universities is recommended to evaluate the integrated educational programs internationally which contributes to comparison of the quality of integrated educational programs in all universities with the standard.