Identifying weaknesses in undergraduate programs within the context input process product model framework in view of faculty and library staff in 2014.

PURPOSE: Objective of this research is to find out weaknesses of undergraduate programs in terms of personnel and financial, organizational management and facilities in view of faculty and library staff, and determining factors that may facilitate program quality-improvement.
METHODS: This is a descriptive analytical survey research and from purpose aspect is an application evaluation study that undergraduate groups of selected faculties (Public Health, Nursing and Midwifery, Allied Medical Sciences and Rehabilitation) at Tehran University of Medical Sciences (TUMS) have been surveyed using context input process product model in 2014. Statistical population were consist of three subgroups including department head (n=10), faculty members (n=61), and library staff (n=10) with total population of 81 people. Data collected through three researcher-made questionnaires which were based on Likert scale. The data were then analyzed using descriptive and inferential statistics.
RESULTS: Results showed desirable and relatively desirable situation for factors in context, input, process, and product fields except for factors of administration and financial; and research and educational spaces and equipment which were in undesirable situation.
CONCLUSION: Based on results, researcher highlighted weaknesses in the undergraduate programs of TUMS in terms of research and educational spaces and facilities, educational curriculum, administration and financial; and recommended some steps in terms of financial, organizational management and communication with graduates in order to improve the quality of this system.

Introduction

Fundamental part of good decision making is evaluation which its information will be used in developing a precise perception of outcomes of a program and input requirements like personnel and financial [1].

Evaluation of higher education increasingly is affecting academic quality, accountability of authorities, and universities ranking. Faculty members at universities are considered as effective and main factors in quality insurance of teaching–learning process in educational systems [1]. In many evaluation literatures, the CIPP model or Context, Input, Process, Product model is a very recommendable method for educational evaluation [2,3,4,5]. Complete definition for evaluation in the CIPP model is “Evaluation is a process of specifying, acquiring and providing descriptive information and making judgments about value and utility of goals, plans, performance and results to guide decision making, providing accountability, and greater understanding of phenomena under the study [6].”

From the CIPP model point, we can breakdown these eight evaluated factors into four sections:

(1) Context

- Goals, management and organization area

(2) Inputs

- Students

- Human resources

- Research and educational spaces and equipment/spaces and facilities

(3) Process

- Research and educational courses and programs, teaching and learning process

- Administration and financial

- Program evaluation

(4) Product (output)

- Graduates

This study aimed to examine faculty and library staff perspectives on undergraduate program quality and weak points within universities and factors that may facilitate program quality–improvement. Although undergraduate programs within Tehran University of Medical Sciences (TUMS) are the focus of this paper, one must also keep in mind the broader context of undergraduate programs and other school’s essential needs which are common among all universities and schools. Objective of this research is to find out weaknesses of undergraduate programs in terms of personnel and financial, organizational management and facilities in view of faculty and library staff, and determining factors that may improve its quality.

Subjects and methods

This is a descriptive analytical study conducted in four selected faculties at TUMS in 2014. We selected our sample only from Public Health School, Faculty of Nursing and Midwifery, Faculty of Rehabilitation and Faculty of Allied Medical Sciences that have undergraduate programs. Statistical population were consist of three sub-groups including department head (n=10), faculty (n=61), and library staff (n=10) with total population of 81 people. After receiving the permission from Ethics Committee of Tehran University of Medical Sciences with number (IR.TUMS.REC.1394.2156), sampling was done as a census among head of departments, faculty and library staff.

We used three separate researcher-made questionnaires inspired from the CIPP model and internal evaluation literatures to collect data, based on eight factors in four area of CIPP evaluation model, consist of demographic data such as place of work, educational degree, teaching status, scientific ranking and multiple choice questions that were used based on Likert scale rating 1 to 5 digits, grading from very low, low, medium, high, and very high. Thus, if the score of a factor became less than 49.9%, it was assumed undesirable; if the score was between 50% and 74.9%, it was considered as relatively desirable situation; and if the score was more than 75%, the factor had a desirable situation. The accuracy of the questions, topic, and research goals were validated by consulting expert, supervisor, and advisor. Cronbach α coefficient formula was used to determine department head, faculty and library staff questionnaire’s reliability which estimated at α=0.97, α=0.92, and α=0.87, respectively that suggests the questionnaires have had the required reliability.

The questionnaires were distributed and collected via face to face approach. The data were then analyzed using descriptive and inferential statistics including distribution of population, frequency, percentage, and Fisher exact test. Collected data was analyzed by SPSS version 20.0 software (IBM Corp., Armonk, USA).

In this study first, the context (goals, management and organization area) was evaluated and then, the education system indicators were determined, by using facilitate decision-making model of CIPP. These factors are including: input (students, human resources, educational and research spaces and equipment/spaces and facilities); process (educational courses and programs, teaching and learning process, program evaluation, and administration and financial); and product (graduates).

Results

The result showed that highest response rate among faculty members is from public health faculty with 38.3%. Also assistant professor is the most frequent ranking among faculty members (43.9%) and most of faculty members teach full time in their faculties. Among library staff, most of them (60%) have bachelor degree. Most frequent scientific ranking and degree among department head is associate professor with 44.4% and PhD with 88.9%, respectively.

1. Context

In this area, goals, management and organization factor was evaluated. Fifty percent of department head (Table 1) and 55.7% of faculty members believed that this factor has a relatively desirable situation (Table 2).

Table 1.

Frequency Distribution of Units under Research among Department Dead according to Evaluated Factors

CIPP component	Evaluated factor for head of department	Evaluation spectrum
		0-49.9 (low)	50-74.9 (middle)	≥75 (high)
Context	Goals, management, and organization area	1 (10)	5 (50)	4 (40)
Input	Facility and spaces	5 (50)	3 (30)	2 (20)
Process	Educational courses and programs, learning and teaching process	0	3 (30)	7 (70)
	Administration and financial	7 (70)	1 (10)	2 (20)
	Program evaluation	3 (30)	3 (30)	4 (40)
Product	Graduates	4 (40)	5 (50)	1 (10)

Data are presented as frequency (%).

CIPP: Context input process product.

Table 2.

Frequency Distribution of Units under Research among Faculty Members according to Evaluated Factors

Component of CIPP	Evaluated factors for faculty members	Evaluation spectrum
		0-49.9 (low)	50-74.9 (middle)	≥75 (high)
Context	Goals, organization and management area	19 (31.1)	34 (55.7)	8 (13.1)
Input	Learner or student	20 (32.8)	32 (52.5)	9 (14.8)
	Spaces, research and education facility	46 (75.4)	13 (21.3)	2 (3.3)
Process	Courses, research and educational programs, teaching and learning process	25 (41)	32 (52.5)	4 (6.6)
	Evaluation	27 (45)	27 (45)	6 (10)

Data are presented as frequency (%).

CIPP: Context input process product.

2. Input

In this area, three factors were evaluated. Fifty percent of department head (Table 1), 40% of library staff (Table 3), and 75.4% of faculty members (Table 2) stated that research and educational spaces and equipment/facility and spaces are in undesirable condition. Sixty percent of library staff believed that human resource is in desirable situation (Table 3) and 52.5% of faculty members specified that students are in relatively desirable situation (Table 2).

Table 3.

Frequency Distribution of Units under Research among Library Staff according to Evaluated Factors

CIPP component	Evaluated factors for library staff	Evaluation spectrum
		0-49.9 (Low)	50-74.9 (Middle)	≥75 (High)
Input	Human resources	1 (10)	3 (30)	6 (60)
	Facility and spaces	4 (40)	2 (20)	4 (40)

Data are presented as frequency (%).

CIPP: Context input process product.

3. Process

In this area, also three factors were assessed. Seventy percent of department head expressed that educational courses and programs, learning and teaching process is in desirable situation (Table 1) and 52.5% of faculty members pronounced that this factor is in relatively desirable situation (Table 2). Seventy percent of department head believed that administration and financial is in undesirable condition and 40% of department head stated that program evaluation is in desirable condition (Table 1), but 45% of faculty members specified that this factor is in relatively desirable situation (Table 2).

4. Output

Fifty percent of department head expressed that graduates has a relatively desirable situation through these four faculties (Table 1).

The Fisher exact test showed statistically significant association between sampling location and research and educational spaces and equipment/spaces and facilities and between teaching status and research and educational spaces and equipment among faculty members that it means there are differences in these factors among selected faculties (Tables 4-9).

Table 4.

Frequency Distribution of Units under Research according to CIPP Components and Sampling Location among Faculty Members

CROSS TAB
CIPP components			Faculty				p	Fisher exact test
			Public Health	Nursing and Midwifery	Allied Medical Sciences	Rehabilitation
Context	Goals, organization and management area	Low	6 (26.1)	6 (42.9)	4 (36.4)	3 (25)	0.285	7.15
		Middle	16 (69.6)	7 (50)	4 (36.4)	6 (50)
		High	1 (4.3)	1 (7.1)	3 (27.3)	3 (25)
Input	Students	Low	8 (34.8)	6 (42.9)	2 (18.2)	4 (33.3)	0.44	5.81
		Middle	12 (52.2)	8 (57.1)	6 (54.5)	5 (41.7)
		High	3 (13)	0	3 (27.3)	3 (25)
	Research and educational spaces and equipment	Low	22 (95.7)	10 (71.4)	9 (81.8)	4 (33.3)	0.001	16.84
		Middle	1 (4.3)	4 (28.6)	2 (18.2)	6 (50)
		High	0	0	0	2 (16.7)
Process	Educational courses and programs, teaching and learning process	Low	8 (34.8)	9 (64.3)	3 (27.3)	4 (33.3)	0.31	6.55
		Middle	14 (60.9)	5 (35.7)	6 (54.5)	7 (58.3)
		High	1 (4.3)	0	2 (18.2)	1 (8.3)
	Program evaluation	Low	11 (47.8)	8 (57.1)	3 (27.3)	5 (41.7)	0.34	6.49
		Middle	11 (47.8)	6 (42.9)	5 (45.5)	5 (41.7)
		High	1 (4.3)	0	3 (27.3)	2 (16.7)

Data are presented as frequency (%).

CIPP: Context input process product.

Table 5.

Distribution of Units under Research according to CIPP Components and Scientific Ranking among Faculty Members

CROSS TAB
CIPP component			Scientific ranking				p	Fisher exact test
			Assistant professor	Associate professor	Professor	Lecturer
Context	Goals, organization and management area	Low	6 (24)	3 (33.3)	3 (33.3)	6 (42.9)	0.92	2.37
		Middle	16 (64)	5 (55.6)	5 (55.6)	6 (42.9)
		High	3 (12)	1 (11.1)	1 (11.1)	2 (14.3)
Input	Student	Low	6 (24)	4 (44.4)	2 (22.2)	6 (42.9)	0.73	3.84
		Middle	13 (52)	4 (44.4)	6 (66.7)	7 (50)
		High	6 (24)	1 (11.1)	1 (11.1)	1 (7.1)
	Research and educational space and equipment	Low	16 (64)	8 (88.9)	7 (77.8)	11 (78.6)	0.87	3.12
		Middle	7 (28)	1 (11.1)	2 (22.2)	3 (21.4)
		High	2 (8)	0	0	0
Process	Educational courses and programs, teaching and learning process	Low	7 (28)	3 (33.3)	3 (33.3)	9 (64.3)	0.21	7.44
		Middle	17 (68)	6 (66.7)	5 (55.6)	4 (28.6)
		High	1 (4)	0	1 (11.1)	1 (7.1)
	Program evaluation	Low	9 (36)	3 (33.3)	5 (55.6)	8 (57.1)	0.83	3.23
		Middle	13 (52)	5 (55.6)	4 (44.4)	5 (35.7)
		High	3 (12)	1 (11.1)	0	1 (7.1)

Data are presented as frequency (%).

CIPP: Context input process product.

Table 6.

Distribution of Units under Research according to CIPP Components and Teaching Status among Faculty Members

CIPP component			Teaching status		p	Fisher exact test
			Full time	Part time
Context	Goals, organization and management area	Low	13 (28.3)	5 (35.7)	0.38	1.94
		Middle	28 (60.9)	6 (42.9)
		High	5 (10.9)	3 (21.4)
Input	Students	Low	14 (30.4)	6 (42.9)	0.40	2.12
		Middle	26 (56.5)	5 (35.7)
		High	6 (13)	3 (21.4)
	Research and educational spaces and equipment	Low	39 (84.8)	6 (42.9)	0.002	11.04
		Middle	7 (15.2)	6 (42.9)
		High	0	2 (14.3)
Process	Educational courses and programs, teaching and learning process	Low	19 (41.3)	6 (42.9)	1.00	0.284
		Middle	24 (52.2)	7 (50)
		High	3 (6.5)	1 (7.1)
	Program evaluation	Low	19 (42.2)	7 (50)	0.61	1.10
		Middle	22 (48.9)	5 (35.7)
		High	4 (8.9)	2 (14.3)

Data are presented as frequency (%).

CIPP: Context input process product.

Table 7.

Distribution of Units under Research according to CIPP Component and Degree among Library Staff

CIPP component			Degree		p	Fisher exact test
			Diploma	Bachelor
Input	Human resource	Low	0	1 (14.3)	1.00	0.81
		Middle	1 (33.3)	2 (28.6)
		High	2 (66.7)	4 (57.1)
	Facilities and spaces	Low	1 (33.3)	3 (42.9)	1.00	0.92
		Middle	1 (33.3)	1 (14.3)
		High	1 (33.3)	3 (42.9)

Data are presented as frequency (%).

CIPP: Context input process product.

Table 8.

Distribution of Units under Research according to CIPP Components and Scientific Ranking among Department Head

CIPP component			Scientific ranking			p	Fisher exact test
			Assistant professor	Associate professor	Professor
Context	Goals, organization and management area	Low	1 (100)	0	0	0.25	5.17
		Middle	0	3 (60)	2 (50)
		High	0	2 (40)	2 (50)
	Research and educational spaces and equipment and human resources	Low	1 (100)	3 (60)	1 (25)	0.85	2.97
		Middle	0	1 (20)	2 (50)
		High	0	1 (20)	1 (25)
Process	Educational courses and programs, teaching and learning process	Low	0	0	0	0.41	2.33
		Middle	1 (100)	1 (20)	1 (25)
		High	0	4 (80)	3 (75)
	Administration and financial	Low	1 (100)	3 (60)	3 (75)	1.00	2.80
		Middle	0	1 (20)	0
		High	0	1 (20)	1 (25)
	Program evaluation	Low	1 (100)	2 (40)	0	0.45	4.21
		Middle	0	1 (20)	2 (50)
		High	0	2 (40)	2 (50)
Product	Graduates	Low	1 (100)	2 (40)	1 (25)	0.85	3.78
		Middle	0	2 (40)	3 (75)
		High	0	1 (20)	0

Data are presented as frequency (%).

CIPP: Context input process product.

Table 9.

Distribution of Units under Research according to CIPP Components and Degree among Department Head

CIPP component			Degree		p	Fisher exact test
			PhD	Master
Context	Goals, organization and management area	Low	1 (100)	0	0.1	5.05
		Middle	0	5 (55.6)
		High	0	4 (44.4)
	Research and educational spaces and equipment and human resources	Low	1 (100)	4 (44.4)	1.00	1.42
		Middle	0	3 (33.3)
		High	0	2 (22.2)
Process	Educational courses andprograms, teaching and learning process	Low	0	0	3.00	0.52
		Middle	1 (100)	2 (22.2)
		High	0	7 (77.8)
	Administration and financial	Low	1 (100)	6 (66.7)	1.00	1.51
		Middle	0	1 (11.1)
		High	0	2 (22.2)
	Program evaluation	Low	1 (100)	2 (22.2)	0.60	2.26
		Middle	0	3 (33.3)
		High	0	4 (44.4)
Product	Graduates	Low	1 (100)	3 (33.3)	0.50	2.28
		Middle	0	5 (55.6)
		High	0	1 (11.1)

Data are presented as frequency (%).

CIPP: Context input process product.

Discussion

Comparing our study and ones which have been conducted in past in same university highlights the significance of periodic evaluation of universities and reflection of its results to top authorities in order to identify gaps and improve universities quality accordingly.

In context area, goals, management, and organization factor was evaluated and has relatively desirable situation in view of department head and faculty members. Our findings is in opposition to results of Farzianpour et al.’s study [7] in Environmental Health and Engineering School of Public Health and Farzianpour et al.’s study [8] in Endodontic Department, Faculty of Dentistry at TUMS which showed desirable situation in management and organization. Findings of Farzianpour et al. [9] showed undesirable situation of goals and rather desirable situation of organizational and management structure in Epidemiology and Biostatistics Department in Public Health School at TUMS.

In input domain, our result showed that research and educational spaces and equipment/facilities and spaces factor was in undesirable situation which is against to result of studies in Biochemistry Department of Medicine Faculty, and in Healthcare Management Department of Allied Medical Sciences Faculty and in Epidemiology and Biostatistics Department in Public Health School at TUMS which showed relatively acceptable situation regarding this factor [9,10]. But findings of evaluation studies in Department of Environmental Health and Engineering, School of Public Health and in Endodontic Department, Faculty of Dentistry, TUMS, are similar to our results and showed undesirable situation in research and educational spaces and equipment [7,8].

Factor of human resources in input area is in desirable situation in view of library staff which is opposed to the results of Farzianpour et al. [10], which showed relatively acceptable situation in Biochemistry Department, Faculty of Medicine, TUMS. Farzianpour et al. [7] also showed relatively acceptable situation in manpower in Department of Environmental Health and Engineering School of Public Health, TUMS. Our finding is similar to results of Farzianpour et al. [8] in internal evaluation of Department of Endodontic, Faculty of Dentistry, TUMS.

In our study, students have relatively desirable situation which is aligned with findings of evaluation studies in Epidemiology and Biostatistics Department, Public Health School at TUMS and in Healthcare Management Department, Allied Medical Sciences Faculty [9]; but this result is in opposition of finding of evaluation studies in Department of Environmental Health and Engineering, School of Public Health and in Endodontic Department, Faculty of Dentistry at TUMS [7,8].

In process domain, in view of department head, educational courses and programs, teaching and learning process is in desirable situation, but in view of faculty members this factor is in relatively desirable situation that is aligned with the findings of Farzianpour et al. [11] in Healthcare Management Department, Allied Medical Faculty, TUMS and finding of Farzianpour et al.’s study [8] in internal evaluation of Endodontic Department, Faculty of Dentistry, TUMS. Results of Farzianpour et al. [9] showed rather desirable situation of educational courses and curriculum in Epidemiology and Biostatistics Department at TUMS.

Administration and financial is in undesirable situation.

Program evaluation is in desirable situation in view of department head and in relatively desirable situation in view of faculty member which is similar to findings of Farzianpour et al. [10], which showed relatively acceptable situation in Biochemistry Department, Faculty of Medicine, TUMS.

In output scope, graduates are in relatively desirable situation that is similar to findings of Farzianpour et al. [9] which showed rather desirable situation of graduates in Epidemiology and Biostatistics Department at TUMS. These similarities and contradiction between their findings and ours in some factors confirm the result of Fisher exact test which showed there are some significant differences between four faculties regarding to evaluated factors in view of faculty and library staff. This comparison also indicates that situation of some factors were desirable or relatively desirable in past but now have undesirable situation, it means that top authorities do not have had equal view to all faculties in terms of financial and providing facilities at TUMS or there has been some neglects during this time. Some similar findings also indicate that despite of undesirable situation in some faculties regarding to evaluated factors in past, they did not take them into consideration and did not act accordingly which affected quality and situation of those faculties in terms of education, research and facilities and they are still in undesirable situation in view of faculty and library staff in our study. These factors have been determined which may facilitate quality–improvement of program: development of a system for reflecting of student and faculty member’s criticism, satisfaction and views about department head; improvement of goals and mission of programs in dealing with graduates; development of a report on changes made during past 3 years by department head; development of documentary regulation in the department; allocating sufficient budget to increase and improve the quality of academic spaces for faculty members and students; establishing a system evaluation in library regarding to services, collection, facilities, quality and appropriateness of text book, internal and external scientific journals based on the field and availability of print and copy in the library; development of framework to estimation of needed human resources; establishing a systematic mechanism for allocation of facilities and services to faculty members in the department; increasing cooperation between universities regarding to research project, consultant services and executive activities. Besides, there should be more seminar and scientific conference for students and graduates and continuous assessment and evaluation for courses and programs; community needs, professional problems, proportional between theoretical and practical work should be considered in curriculum. Also there should be a system to record needed information about graduates, their skills, their satisfaction and their situation after graduation and for communication with them.

In this study, we faced with limitations such as lack of cooperation of faculty due to time constraint that prolonged the time to collect data. Besides, faculty wanted to modify the questionnaires according to their will.