Factors influencing the results of faculty evaluation in Isfahan University of Medical Sciences.

OBJECTIVE: This study aimed to explore factors influencing the results of faculty member evaluation from the viewpoints of faculty members affiliated with Isfahan University of Medical Sciences, Isfahan, Iran.
MATERIALS AND METHODS: This qualitative study was done using a conventional content analysis method. Participants were faculty members of Isfahan University of Medical Sciences who, considering maximum variation in sampling, were chosen with a purposive sampling method. Semi-structured interviews were held with 11 faculty members until data saturation was reached. The interviews were transcribed verbatim and analyzed with conventional content analysis method for theme development. Further, the MAXQDA software was used for data management.
RESULTS: The data analysis led to the development of two main themes, namely, "characteristics of the educational system" and "characteristics of the faculty member evaluation system." The first main theme consists of three categories, i.e. "characteristics of influential people in evaluation," "features of the courses," and "background characteristics." The other theme has the following as its categories: "evaluation methods," "evaluation tools," "evaluation process," and "application of evaluation results." Each category will have its subcategories.
CONCLUSIONS: Many factors affect the evaluation of faculty members that should be taken into account by educational policymakers for improving the quality of the educational process. In addition to the factors that directly influence the educational system, methodological problems in the evaluation system need special attention.

Introduction

Faculty member evaluation is the one which determines the level of faculty member achievements in the quality of education and leads to equity in academic settings.[1] Adams believes that faculty member evaluation is one of the critical indicators of progress in the educational system. Furthermore, in those institutions that evaluation is continuously performed, the quality of education is improved.[2] The evaluation system for faculty members in (the field of) medical sciences is of high importance because graduates are responsible for human health and life.[3] Clearly, the position of evaluation in the educational system, approaches toward evaluation, the method and process of evaluation, and the application of its results affect the quality of the evaluation process. There are two aims for evaluation as formative and summative, which determine the aim and method of the evaluation system, respectively.[4]

The results of other studies show that the viewpoints of faculty members can help with the improvement of the evaluation system quality. Bastani et al. in a study on the perspectives of faculty members regarding the evaluation system called for the clarification of the aim of the evaluation system, a constructive use of its results, the rectification of the current evaluation system using a multi-method approach, and the collection of the students’ perspectives.[5] According to a study conducted in 2011, top universities performed the evaluation system using the student and administrator rating method by both students and administrators.[6] The result of a study by Joibari et al. showed that faculty members believed that the results of evaluation were subjective routinized and invalidated.[7] Javadi and Arab Baferani in a study on the pathological aspects of the evaluation system, teaching quality, and performance quality from students’ perspectives reported the importance of the evaluation methods, internal and external factors affecting evaluation, and inability to assess all related factors.[8] In addition, some studies indicate the ineffectiveness of the results of the evaluation system and dissatisfaction of faculty members with evaluation.[89] In addition to faculty members’ negative perspectives on the methods of evaluation, quantitative studies have compared faculty members’ perspectives with the students’. They have suggested various methods for removing pitfalls in the evaluation system.[31011]

This is over 15 years (from 2001 to 2017) that the evaluation process is performed in Isfahan University of Medical Sciences, Isfahan, Iran. Initially, evaluations were made on paper manually and only by students. Advances have happened in the evaluation system including the implementation of online evaluation by students at the end of each academic semester. Further, academic authorities including the Department Manager, Deputy of Education, and the Dean individually perform the evaluation for each faculty member. The results of faculty member evaluation are privately sent to each person for future rectifications in educational processes. Since the perspectives of faculty members and their suggestions are helpful for the improvement of the evaluation system, this study aimed to explore factors influencing the results of faculty member evaluation from the perspectives of faculty members affiliated with Isfahan University of Medical Sciences, Isfahan, Iran.

Materials and Methods

Study design

This was a descriptive exploratory qualitative study using a conventional content analysis method.

Participants

This study was conducted in 2015–2016 with faculty members affiliated with Isfahan University of Medical Sciences, Isfahan, Iran. The participants were chosen using purposive sampling method with the consideration of maximum variation in sampling in terms of different departments, faculties, and evaluation scores. The faculty members with the lowest and highest scores were recruited based on the following inclusion criteria: the experience of work at least for 10 years (because of enough experience and encounter period with evaluation process) and willingness to take part in this study.

Data collection

After obtaining the permission from the research council affiliated with the university to conduct the study, the faculty members were recruited. They were provided with some information about the aim and method of the study, the data collection process, the confidentiality and anonymity, and the possibility of withdrawing from the study without being penalized. Next, those who agreed to take part in this study were asked to sign the written informed consent form. The participants determined the convenient time for interviews and gave the permission of setting down their words and tape-recording their voices, except one faculty member. They were ensured that the results of the study would be provided to them in case of their request. The interviews were performed in at least 11 sessions and continued until data saturation was reached as long as the data did not add to the variation of the findings. The semi-structured interviews were conducted in a face-to-face manner, except one interview that was held via the phone. Our main open-ended questions used in the study were as follows: what is your perspective of the evaluation process? What factors affect your score during the evaluation system? What is the effect of the evaluation system on your performance? Probing questions were asked to follow their perspectives as follows: What do you mean by …? Will you explain it more? Do you want to add more details?

Data analysis

The interviews were transcribed verbatim and read several times to get a sense of the whole. The collected data were analyzed using a conventional content analysis method[12] with the following steps: coding the data, sorting codes with similar meanings, and extracting themes and patterns. Furthermore, constant comparison of the data helped with the development of themes and categories. This process was performed continuously until the quality of the data analysis was ensured.

Trustworthiness and rigor of the study

The method suggested by Guba and Lincoln (1985) was used to ensure the rigor. For the sake of more credibility, a doctoral student in the field of medical education who was familiar with qualitative research undertook data collection. The interviewer spent enough time with participants to attract their trust and collect in-depth data. Moreover, the processes of data collection and analysis were checked and discussed in a team. For dependability, data collection and analysis were performed concurrently and forward and backward movements between the findings and the transcriptions were performed.

With regard to confirmability, a summary of codes and transcriptions was sent to a couple of participants and their perspectives and feedbacks were taken into account during the data analysis. As peer checking, a qualitative research expert was also invited to check the coding and analyzing processes that led to some minor rectification of the findings. The maximum variation in sampling, consideration of various perspectives, the use of direct quotations of the participants during the description of findings all helped with transferability and consistency of the study.

Ethical considerations

The permission to conduct this study was obtained from Educational Development Center, and the participants were chosen based on their evaluation scores from the lowest to the highest. Their anonymity was ensured throughout the study process. Before the interviews, the required information about the study was given to the participants, and their questions were answered. The permission to tape-record their words was obtained, and the voluntary nature of the study was described to all of them. The confidentiality of the data collection and management processes was also guaranteed. They were ensured that the findings of this study would be provided to them in case of their request.

Results

The participants were 11 faculty members consisting of five women and six men. The age range of the majority (54.5%) of them was 41–50 years and the rest of them were more than 50 years old. In addition, 36% of them were assistant professors and 36% were associate professors. The remaining were professors. Their educational work experience (54.5%) was 21–30 years and the rest were <21 years.

The data analysis on 421 statements and sentences extracted from the interviews led to the development of two main themes, seven categories, and fourteen subcategories. The data analysis led to the development of two main themes, namely, “characteristics of the educational system” and “characteristics of the faculty member evaluation system.” The first main theme consists of three categories; “characteristics of influential people in evaluation,” “features of the courses,” and “background characteristics.” The other theme has the following as its categories “evaluation methods,” “evaluation tools,” “evaluation process,” and “application of evaluation results.” The descriptions of the themes, categories, and subcategories are presented in Table 1.

Table 1

Themes, categories and subcategories developed from the data analysis

Characteristics of the educational system

From the faculty members’ perspectives, some factors affecting the educational system were the “characteristics of influential people in evaluation,” “features of the courses,” and “background characteristics.” These factors were related to the importance and usefulness of the evaluation with the aim of personal development and social responsibility in the university.

Characteristics of influential people in evaluation

This category consisted of the following subcategories: “evaluator's individual characteristics” and “faculty member's personal characteristics.”

Evaluator's individual characteristics (students and administrators)

One of the sources for the evaluation of faculty members was students. In spite of the importance of their perspectives, their age, educational level, cognitive maturation, personality, assiduousness, and responsibility affected how they communicated with and evaluated faculty members.

“If the faculty member reports students’ absence from class sessions, his/her evaluation is endangered by students. Students do not like to accept educational rules and to come to terms with them.” (Participant 11)

Lack of motivation for precisely and completely filling out the evaluation forms (by the students) and doubt about the effectiveness of evaluation hindered an appropriate evaluation of faculty members.

“I am sure that <10% of students read carefully the evaluation form items. Many of them are not interested in filling out the forms and find no benefit for it.” (P 11)

In the eyes of faculty members, students’ educational level and cognitive maturity can affect the evaluation process.

“For instance, the evaluation process performed by medical residents is more validated. They care about it and mainly have cognitive maturation. I have seen that they spend enough time to fill out the evaluation forms…” (P 6)

If the student cares about learning and feels the responsibility of what he/she does, he/she will fill out the evaluation questionnaire carefully. If not, the result of the evaluation system would lead to downgrading the faculty member's dignity.

“Some students are more assiduous and take care of their own academic behaviors. They perform evaluations well. Some other students use the evaluation process as a tool to disturb faculty members. Such behaviors are insulting and downgrade their dignity.” (P 11)

Another source for the instructor evaluation was administrators. Their characteristics such as responsibility and beliefs affected the results of the evaluation system.

“If you could find faculty members and administrators who believe in God and ethical principles, then you could rely on their evaluation….” (P 1)

“If we were enough mentally grown and developed to prevent the interference of our own personal preferences in the evaluation process, the result of evaluation then would be reliable and valid. Of course, it does not happen frequently.” (P 10)

“Some administrators do not do what they believe. They describe their perspectives about the faculty member in some manner, but report it in the opposite manner.” (P. 3)

Faculty member's personal characteristics

In addition to knowledge, attitudes, experiences, interpersonal communication skills, commitment and responsibility, seriousness and discipline that could affect the process of faculty member evaluation, his/her appearance, behavior, personality, permissiveness or strictness, engaging the students in teaching and becoming too much intimate with them influenced the results of faculty member evaluation.

“An undergraduate student stated that she was not in the mood to go to the classroom, but she attended the class because the instructor was handsome and wear nice clothes. Therefore, they would give high scores to the instructor during the evaluation.” (P 1)

“There are differences among instructors. Some instructors have stable personalities and students communicate with them easily, they respect students and keep their distance and therefore are given good scores.” (P 3)

The method and strictness of class management by the faculty member affected the results of the evaluation process.

“The instructor receives lower score because of respecting class management rules and regulations. If the faculty member is strict in the class management, he/she receives lower score during the evaluation process.” (P 11)

“I felt that those faculty members that are given lower scores are in two types: those who are unable to manage the class and those who are too much strict and respectful of class regulations. Perhaps sometimes their strictness is illogical.” (P 2)

Features of the courses

The educational course also affected the evaluation process. The theoretical or practical nature of courses, the relationship between the course and students’ academic disciplines, and the level of the course difficulty affected the evaluation process. While the process and tools were the same in the evaluation, various factors affected the results of faculty member evaluation.

“If the instructor management is too serious and tough in the classroom due to heavy workload of educational materials, students give negative scores to instructors.” (P 3)

“I am so busy with educational tasks and have no time to ask questions from the students. However, I try to make them involved in the classroom.” (P 4)

Background characteristics

This category consisted of two subcategories as follows: “culture and atmosphere in the faculties and departments” and “existing educational and evaluation policies and laws.”

Culture and atmosphere in the faculties and departments

Variations in the culture and atmosphere of educational environments affected the evaluation results. Theses consisted of the type of communication and interpersonal relationships in the workplace, its history, type of management, and educational levels, which were different in various places and even sometimes from year to year. Therefore, such conditions can affect the results of the faculty member evaluation. In this respect, faculty members believed that the evaluation process should be made compatible with the atmosphere and culture of each workplace.

“There are differences in the atmosphere of faculties. Therefore, the evaluation process should be performed differently according to such differences.” (P 1)

“There are differences in each classroom… sometimes education is more important and sometimes informal relationships are more important.” (P 3)

According to religious doctrine, the position of the instructor is greater than the students though the evaluation of the faculty member by students seemingly could damage this position, because such an evaluation system was designed based on the principle of the customer satisfaction.

“The evaluation process is not based on the national-religious culture and context, because the student should always respect faculty members and in the evaluation system, the faculty member may get undermined.” (P 3)

Characteristics of the faculty member evaluation system

From the instructor's point of view, this system could potentially affect the results of evaluation. This theme consisted of the following categories: “evaluation methods,” “evaluation tools,” “evaluation process,” and “application of evaluation results.”

Evaluation methods

This category consisted of the following subcategories: “diversity of methods” and “technical limitations.” The participants mostly believe that new and validated evaluation methods are not adopted in the Iranian educational system. In this respect, they noted the need for a variety of methods and pointed out the probable limitations of each method. They suggested revision of the evaluation system and adoption of the appropriate methods from the international universities.

“The current system belongs to 25 years ago, which was translated into Farsi and has been used here. However, it has never been updated and has remained unchanged. There is a need to new ideas with the consideration of our own value systems.” (P 3)

“For an appropriate, complete and reliable evaluation of the faculty member,” there is a need to sufficient contact between the student and faculty member.” (P 2)

“… the student is asked to evaluate the faculty member, but he/she has never seen that instructor.” (P 3)

Evaluation tools

This category consisted of the following subcategories: “validity of the instrument” and “reliability of the instrument.” Some issues in the current evaluation tools made the faculty members believe that the tools were not valid and reliable.

“…. the problem is that evaluation tools are not valid. I think they are mixed with bias.” (P 11)

The incompatibility between the questions and the faculty member's field of practice, especially theoretical teaching and clinical practice, was the main problem mentioned by the participants.

“The evaluation that has been performed during these years has been different from realities… the questions in the evaluation tool have been copied and pasted from other tools. Also, some of them are incompatible with our field of practice.” (P 3)

“…the questions should be related to the field of practice of faculty members.” (P 7)

“…if there are too many items, students will lose their concentration and carelessly fill out the evaluation forms.” (P 11)

The number of students participating in the evaluation process was mentioned critical. The limited number of students, especially from the postgraduate degree, can affect the results of evaluation.

Evaluation process

This category consisted of the following subcategories: “time of evaluation” and “process of evaluation.”

“Students are requested to fill out the evaluation form of about 20 faculty members simultaneously. Therefore, they would do it carelessly.” (P 8)

“Evaluation should take place in an appropriate time. But, for instance, during the final examination, students are asked to evaluate the faculty member. If they have good feelings about the examination, the evaluation process is performed well. This is important and should be taken into account during the evaluation process.” (P 10)

Application of evaluation results

“Evaluation feedback to instructors” and “the use in administrative decisions” were the subcategories of this category. They indicated that inapplication of the evaluation the results would lead to no improvement in the educational system. Thus, the evaluation process is considered something useless and disturbing.

Evaluation feedback to instructors

“…the feedback should be sent in a way that faculty members are treated respectfully, for instance via a private letter or an e-mail. Many years ago, the faculty dean sent us the evaluation result using a postal card full of kindly and friendly statements.” (P 7)

“The feedback should be informative notpunitive. An educational expert should send the feedback. Such a feedback should be evidence-based. For instance, a scientific article should accompany the feedback…” (P 7)

The use in management decisions

The use of the evaluation process for managerial decisions could improve the importance of the faculty member evaluation and attract their attention to evaluation.”

“…given all problems in the educational system, the results of the evaluation system cannot do anything, especially for the faculty member who carries out unethical activities or pays no heed to the educational principles.” (P 3)

“If the faculty member does need the evaluation score, for instance for his/her promotion, then, she/he will pay enough attention to it, and try to do her/his best.” (P 10)

Discussion

This study aimed to explore factors influencing the evaluation results of faculty members from the viewpoints of faculty members affiliated with Isfahan University of Medical Sciences, Isfahan, Iran. The data analysis led to the development of two main themes, namely, “characteristics of the educational system” and “characteristics of the faculty member evaluation system.” The first main theme consists of three categories, i.e. “characteristics of influential people in evaluation,” “features of the courses,” and “background characteristics.” The other theme has the following as its categories: “evaluation methods,” “evaluation tools,” “evaluation process,” and “application of evaluation results.”

The findings of this study showed that factors influencing the evaluation process were related to the characteristics of the educational and the faculty member evaluation systems. Among the important factors were the characteristics of evaluators including students and administrators. The use of students’ perspectives in faculty member evaluation has been discussed over the time.[13] Various studies on students’ competencies for the evaluation of instructors are controversial. Some studies report that students are the first and immediate customers of the educational system.[14] Some others believe that students are not stakeholders of the educational system, but they are its outcomes.[15] Nevertheless, students are the receivers of the educational services and the best choices for the evaluation of faculty members in educational settings. Cashin and McKeachie in some review studies reported that the evaluation of the faculty members by the students was statistically valid, reliable, and empty of bias. Nevertheless, it should be used along with other evaluation methods for drawing a more complete picture of this phenomenon.[1617] In most studies on the evaluation process, student educational level, age, academic achievements, personal development, ability to judge the content of the course, and personality traits such as maturity influenced the evaluation results.[1418] In this study, the need for the revision of the evaluation process was also highlighted.

The study of Azizi et al. showed that the majority of students due to laziness and seeking comfort were interested in getting the professor's lecture notes, while this way of educational method (presentation) at the different levels, especially postgraduate is not suitable and leads to the superficial transfer of knowledge to the students.[19] However, in students view if the professor does not give a lecture note to the students, he/she will not a good professor, because they do not prefer to study a large volume and difficult text book. The study participants attributed the current insufficiencies of the evaluation process to the laziness of and lack of the feeling responsibility among students. It is evident that the faculty member evaluation should be done in such a manner that does not endanger the dignity of that person.[82021] It is believed that students should not perceive that the professional identity of their instructors is fully depended on the evaluation process performed by them.[19]

One method for the reduction of students’ bias during the evaluation process is the description of the reasons for the evaluation process, its mission, aims, and quality. In the study of Kamali et al., the description of the aims of the evaluation to students led to more reliable evaluation results.[6] Similarly, Sanagoo and Joibari stated that some students filled out the evaluation forms carelessly and hastily.[15] In the study of Keriter and Lakshman, the evaluation of the instructor was performed only by a few students chosen via a random and purposive sampling method that led to valid and reliable results.[22] Some studies used successful students for the evaluation of the instructor to provide more reliable and valid evaluation results.

It was found that the characteristics of the faculty member such as appearance and clothing could affect the results of the evaluation. However, the results of other quantitative studies showed that these factors had no direct effects on the evaluation process.[2324] Harandi Zadeh et al. found that appearance was an important factor for attracting students to effectively communicate with the faculty member.[25] In this study, one of the characteristics of the instructor was mentioned as the level of the faculty member's strictness. Greenwald believed that the faculty member's strictness or flexibility could affect students’ perspectives during the evaluation process.[26] Dargahi and Mohammadzadeh, Amini, and Dadkhah et al. also believed that too much stringency and assertiveness increased the gap between the student and instructor.[102728] Tamizifar negates the presence of such an effect.[29] In some studies, students after graduation found the importance of (quality and the worth of) the instructor's performance and endorsed his/her stringency during education.[3031] The evaluation by cognitively mature students can remove such problems. Therefore, some educational centers use students with higher academic levels to take part in the evaluation process to achieve more valid and reliable results.[3032] In their view, these students have more experience and understanding of academic and educational issues in previous grades or levels.

Based on the differences among disciplines and courses, some believed that the results of the evaluation could be different. For instance, course difficulties and volumes could affect teaching methods and class management, and hence, students’ satisfaction with the educational process could be influenced by. Brady and Eisler and Goldberg and Callahan showed that the course characteristics and the number of students affected evaluation results and scores.[3334] Marsh stated that the course difficulties have both positive and negative aspects that could lead to learning progression and postponement, respectively. There was also a positive correlation between positive aspect of difficulties and teaching evaluation. Conversely, a negative correlation was reported between the negative aspect of difficulties and teaching evaluation.[35]

Kamali et al. and Amini and Honardar reported that faculty member evaluation was severely affected by the background and cultural aspects of the educational system, as well as human and administrative interactions in the educational setting.[1036] In this study, participants believed that culture and atmosphere of the educational system could influence the evaluation results. It means that positive atmosphere in the educational system and lower expectations of students from instructors lead to more positive evaluation results. Sharifi et al. stated that the informal evaluations by students about teaching methods and the behavior of faculty members during daily communications can influence the formal evaluation process.[37] Dunkin believed that according to psychological studies, people's perceptions and judgments are influenced by general environmental characteristics.[38] Such a finding highlights the relative nature of the evaluation process, i.e. it depends on both the evaluator and the student.[1539] There is a need for an evaluation system with the least impact of the above-mentioned factors.

It was found that some features of the evaluation process, including evaluation methods, evaluation tools, and evaluation processes, could affect the results. The participants believed that student-centered evaluation could not provide a complete picture of the instructor's abilities.[15] Therefore, the use of various evaluation methods can remove the insufficiencies of one method.[3032] Furthermore, it is sometimes necessary to take into account the limitations of each evaluation method.[32]

Other characteristics of the evaluation process were the type and the number of evaluation tools, which from the viewpoints of the participants were needed to enhance the validity and reliability of the evaluation process. In a study by Javadi and Arab Baferani, unreliable tools hindered an appropriate evaluation system.[8] In the Tootoonchi et al.'s study, tools, timing, and method of the faculty member evaluation were inappropriate.[40] The validity of an evaluation tool means that this tool can take all faculty member's tasks into account and is used by those who are unfamiliar with the faculty member's field of practice.[8]

The process of evaluation was related to the participants’ perspectives regarding the timing and process of evaluation. Many studies showed the most inappropriate time for evaluation to be exactly before and after the final examinations.[83640] Javadi and Arab Baferani in an article on the pathology of the faculty member evaluation stated that the time and method of evaluation should be in such a manner that the safety of the student and faculty member is not endangered.[8]

The importance of the faculty member evaluation is that it improves their performance and decisions made by administrators. In our studies, some participants believed that the results of the evaluation process are not used, and therefore, faculty members and other stakeholders undermine its significance. A study by Javadi and Arab Baferani showed that inapplication of the evaluation results can demotivate evaluators and reduce the validity and reliability of the evaluation process.[8] On the other hand, the application of the results in serious administrative decisions including the suspension of services has been low, which has reduced the motivation to perform the faculty member evaluation.[15] In spite of the problems in the evaluation process, timely and complete feedbacks based on the evaluation results can guide faculty members to improve their educational activities.[41]

Conclusions

Many factors affect the faculty member evaluation that should be taken into account by educational policymakers to improve the quality of the education process. In addition to the factors that directly influence the educational system, some other factors such as the methodological problems in the evaluation process, the application of the evaluation results, and the faculty member evaluation system need special attention. In this respect, future research can provide us with more knowledge and insight in how the evaluation process can be improved. Furthermore, strategies such as designing specific questionnaires for evaluation, reviewing tools and methods of data collection, use of multi-source evaluation, more effective application of evaluation results, and adapting the evaluation process with cultural and social characteristics can be beneficial.

Financial support and sponsorship

Isfahan University of Medical Sciences supported the study.

Conflicts of interest

There are no conflicts of interest.