Identifying challenges for effective evaluation in nursing education: A qualitative study.

BACKGROUND: Although nursing education in Iran has a positive trend in growth; it is still facing with multifaceted challenges. This study aims to explore the challenges for effective evaluation of nursing education perceived by academic managers.
MATERIALS AND METHODS: A qualitative study was performed by using 21 face-to-face, in-depth interviews with academic managers in medical universities and at the Ministry of Health and Medical Education in Iran. All interviews were recorded digitally, transcribed verbatim, and analyzed by qualitative content analysis.
RESULTS: The main challenges could be categorized under 3 themes, each included 3 subthemes: managerial issues (inefficacy of management, inadequacy of policies and strategies, ineffective evaluation planning); administrative issues (inefficient and affected evaluators, inappropriate implementation, and inefficacy of approaches and tools); and structural issues (inappropriate culture, clinical education complexity, lack of alumni follow-up system).
CONCLUSIONS: The results emphasize the need for educational evaluation development in nursing, including systematic and regular educational evaluation planning focusing on efficient feedback system and regard to excellence models. The comprehensive educational evaluation requires participation, involvement, and collaboration among the Nursing Board, Nursing ministerial office, faculties of nursing, and Nursing Organization. Thus, it is necessary to better designate current educational evaluation systems, policies, approaches, methods, and procedures.

INTRODUCTION

Medical education always had faced with complex challenges, such as educational quality improvement, educational evaluation, and demand to community health promotion.[1] Medical and nursing schools must be accountable to respond to the enormous changes and to compete with other national and international schools by general and specific evaluation systems.[2-4] Nursing education as a part of medical education has developed very rapidly in 2 recent decades with regard to the number of general undergraduate and specific postgraduate courses and has caused some concerns about the quality of education.[56] Nursing education also had faced with some challenges, including the competency of graduated nurses and quality of nursing education as well.[7]

Moreover, the nursing education is very important due to close ties with people health and should lean toward quality improvement.[8] Therefore, the quality of nursing education should be evaluated continuously by the local, national, and international evaluation systems.[7] Evaluation of nursing education guarantees quality of nursing education and using in most countries.[9] Evaluation of nursing education can be done through two different approaches: first internal evaluation or self-assessment, which is executed by schools, departments, and faculty members; and next external evaluation, which is done by outsiders and usually includes nongovernmental organizations (NGOs) or governmental agencies.[1011] Continuous and systematic evaluation develops nursing education quality by collecting organized data, which reflects nursing education performance and educational goals achievement[11-13] and directly or indirectly improves the nursing education.[14]

Nursing education and educational evaluation in Iran

In Iran nursing education has transferred from hospital training to higher education since 1979 and has had a positive trend of growth in recent decades, similar to that in other countries.[1516] At present, nursing education in Iran has undertaken 3 levels of education; bachelors (BSc), masters (MSc), and doctorate (PhD). There are 152 nursing education centers offering the BSc degree, 30 of the total centers offering the MSc degree, and 11 of them offering the PhD degree in nursing.[17] The nursing programs were accredited by the High Council of Medical Education of the Ministry of Health and Medical Education (MoHME). The MoHME regulates all programs, licenses schools, and determines their curriculum; however, there is flexibility for curriculum modifications.[15] Iranian nursing education still faces with a multifaceted challenge named educational evaluation.[16]

In Iran, evaluation of nursing education also was done through 2 methods: internal evaluation, which is executed by the same school's faculty members; and external evaluation, which is done by other governmental schools’ faculty members.[18] Internal evaluation plan in nursing education is accomplished at departmental levels. However, the results of it are reported to school, university, and educational deputy of MoHME, but the internal evaluation cycle is not completed by feedback.[1920]

External evaluation plans are conducted more or less as ranking program in universities and schools at national level every 5 years with incomplete approaches. Ranking program categorizes according to the size and resources of the universities.[913] Accreditations plans are carried out not only incompletely, irregularly, scattered, and temporarily by MoHME but also the same organization accredits its institutions and programs.[421] These accreditations are conducted in the form of establishment and approval of programs of nursing bachelors, masters, and doctoral with preliminary evaluation by the National Nursing Board and final approval by Secretariat of Supervision, Evaluation and Expansion Council of Medical Science.[1516]

Review of studies in Iran

However, several studies have been carried out in the field of educational evaluation in higher education.[92122] These studies have been investigated on the national standards in nursing and midwifery postgraduate education system; identifying strengths and weaknesses and suggesting modifications on national standards,[23] accreditation standards in nursing education by documenting process,[8] and analyzing teaching effectiveness in nursing education by qualitative approach.[24] Although some studies on educational evaluation have been carried out in nursing education in Iran, there are scarce qualitative and quantitative studies in this area, especially for challenges of effective evaluation in nursing education.[82324] This study aimed to explore the perception of academic managers in Iranian nursing education on the challenges involved in providing effective evaluation in nursing education.

MATERIALS AND METHODS

A qualitative study was conducted using face-to-face, semi-structured interview between April and July 2010. This study used a qualitative approach to gain a deep understanding of the current state evaluation and finding evaluation challenges in nursing education in Iran.[25] The subjects were chosen with stratified and purposeful sampling. They were included until saturation was reached. Qualitative conventional content analysis was used to analyze the data.[26]

Participants

The participants were academic managers and experts in the Evaluation Committee of Nursing Education or National Nursing Board, who were selected through purposive sampling. Including criteria for participating in this study were having more than 5-year work experience in nursing education evaluation and holding a managerial position, such as dean, vice-dean, evaluation committee member, educational evaluation expert, or national nursing board member. Participation was voluntary and the individuals who agreed to participate in the study included 5 deans, 4 vice-deans at nursing schools, 8 educational evaluation committee members, 4 educational evaluation experts in the Schools of Nursing and Midwifery in Tehran, Shahid Beheshti, Welfare Sciences, Tarbiat Modarres, Mashhad, Kerman, and Tabriz University of Medical sSiences for a total of 21 interviewees. Eight of them, at the same time were national nursing board members as well. Their ages ranged between 38 and 57 years, their work experience in general between 5 and 32 years. All were PhD in nursing education with associate professors and assistant professors degree.

Data collection

Data were collected through semi-structured interviews. The interview duration ranged between 20 and 106 min. The interview guide included a list of general questions about the current situation for educational evaluation in Iranian nursing education. By using questions or statements, such as “Please explain your own perceptions about current situation for educational evaluation” and “What are its strengths and weaknesses?”

The timing and wording of each question was individualized in order to capture the perspectives of each informant in his or her own words. The interviews were recorded digitally.

Data analysis

All of the interviews were recorded completely and then were carefully listened and transcribed. All the transcribed interviews were read several times, and analyzed by qualitative conventional content analysis.

First, the key words and important phrases in the text were noted. During this phase more than 580 primary codes were extracted. The codes that seemed having overlapped were revised to one refined code and finally 68 refined codes from the primary codes were resulted. Then, the codes were compared for similarities and differences. Similar codes were put in one category with the same title, and formed subthemes. At the end, the main themes were emerged and finally, 68 codes, 9 subthemes, and 3 themes were created.

To validate the findings, 2 transcripts and the primary results (codes and subthemes) were checked by 2 participants. Also the transcripts were read and coded independently by 2 external researchers as well. Furthermore, in order to increase transferability of the findings, the participants were chosen from 8 nursing and midwifery schools. Also in this study, data analysis was performed through in-depth prolonged engagement with the data, and the process of data analysis was constant comparison analysis, which increases the validity and reliability of data.

Ethical considerations

The study was part of a larger project approved by Shahid Beheshti University of Medical Sciences Ethics Committee. All the participants were informed of the purpose and design of the study and the voluntary nature of their participation.

RESULTS

The results revealed that effective evaluation in nursing education is confronted with multifaceted and major challenges. These challenges can be summarized under 3 themes concerning managerial, administerial, and structural issues [Table 1].

Table 1

Challenges for evaluation system in Iranian nursing education

Managerial issues

The main challenges of managerial issues in nursing education evaluation were elicited as: inefficacy of management; inadequacy of policies and strategies, and ineffective evaluation planning.

Inefficacy of management

Iranian academic management is established by the government, which proves governmental governance with some specifications, such as managements’ instability, lack of authority, lack of accountability, and inefficient supervision. In fact, the informants drew attention to the fact that the academic management would be affected by lack of commitment to evaluation requirements, unwilling to evaluate, distrust to evaluation results, and nonutilization of evaluation results. Another important issue emphasized by the participants concerned the inefficacy of governmental educational management in matters of lack of evaluation knowledge, inexperienced managers, and individual-based management.

Participants stated

“Evaluation in nursing education is governmental which involve with managers changes and new graduated managers without considering of enough authority and accountability.” (Participant 5 (P5): national board member and dean). “Some managers are unfamiliar with evaluation's concepts, models, and even goals and do not trust to evaluation and its results consequently; they do not employ the evaluation results.” (P4: evaluation committee member). “They do not interest in be evaluated, but if they know an evaluators group are coming for evaluation, they try to do their best for good pseudoevaluation.” (P9: national board member).

Inadequacy of policies and strategies

The participants expressed discontent with lack of scientific policy making, ministerial monopoly to evaluation, political attitude to evaluation, and to establish nursing courses related to evaluation of nursing education. The participants’ central perception was that the inadequacy of policies and strategies in evaluation of nursing education characterized by centralized evaluation policies and strategies, ambiguity in evaluation strategies, and instability in evaluation policies and strategies.

Following are some statements

“Educational evaluation conducting and planning is only upon Ministry of Health with its centralized and political approach. The Ministry does education, evaluation, ranking, and even does accreditation” (P11: evaluation committee member). “Ministerial evaluation policies and strategies are unscientific, unstable, and with paradox.” (P7: evaluation expert). “Influential members’ pressures interfere on approving and disapproving of nursing courses” (P9: national board member).

Ineffective evaluation planning

Participants stressed that nursing educational evaluation planning is ineffective and proves by unsystematic plan, lack of applied plan, nonemploying evaluation models, and disregard to excellence model in nursing education.

Interviewees pointed out that the evaluation system in nursing education faced with ambiguity in evaluation's aims, uncertainty in evaluation expectations, lack of goal-based plan, lack of efficient feedback plan, and consequently late feedback.

Participants said

“Educational evaluation plans prepares without considering of scientific evaluation, excellence models and systematic approaches” (P1: evaluation committee member). “Present evaluations are performed with uncertainty in aims and expectations and evaluation plans sometime are not basically goal-based” (P11: evaluation committee member). “However, the feedback mechanism was considered in evaluation plans but in action we do not have efficient and on time feedbacks” (P18: vice-dean).

Administerial issues

The results showed that nursing education evaluation when trying to achieve its objectives and to complete the tasks that are entrusted face 3 administerial issues as challenges, concerning: inefficient and affected evaluators, inappropriate implementation, and inefficacy of approaches and tools.

Inefficient and affected evaluators

The interviewees pointed out that a major challenge for evaluation in nursing education is facing with unqualified evaluators regarding inexperience, inexpertise, disinterest, nonindependent, and biased evaluators, which consequently cause inefficacy of evaluation process. Moreover, the intraevaluators and interevaluators variations would be increased because of inefficacy of evaluators and consequently cause false documentation. Participants also stressed that the inefficient evaluators will be more affected to influential outsiders’ and insiders’ inferences and recommendations.

“Evaluator teams have not specialty, qualification, and experience in evaluation field” (P2: national board member and dean). “Some evaluators have no belief and disinterested in evaluation so they do not keep independency, unbiased, neutrality in their evaluations” (P1: evaluation committee member).

Inappropriate implementation

The interviewees emphasized that in order to conduct effective educational evaluation, there are important challenges that one should try to deal with, such as evaluations with long intervals, lack of tradeoff in programs and evaluators, ignorance of course outlines coverage, inattention to effective student evaluation, inattention to effective faculty evaluation, and consequently inattention to some domains evaluation. Furthermore, the participants also pointed out a lack of stakeholders’ collaboration and that individual-based evaluation may lead to a faulty measurement in educational evaluation.

Participants stressed

“Educational evaluations usually are conducted in too long intervals with some defects; lack of assessing students’ and faculties’ evaluation and lack of course outlines coverage evaluation, and in each time still there are the same defects” (P7: evaluation expert). “The numbers of acceptable nursing educational evaluators are limited but nursing programs and courses are too many to evaluate appropriately” (P3: national board member).

Inefficacy of approaches and tools

Interviewees emphasized that educational evaluation face with major challenges, including inefficient internal evaluation, unrealistic external evaluation, insufficient accreditation, distrust ranking methods, and lack of qualitative approaches.

Moreover, participants also pointed out employing evaluation tools as a major challenge for conducting educational evaluation, for example, lack of consensus on evaluation tools, old-fashioned tools, and insufficient evaluation tools. Consequently, educational evaluation process faces with ambiguity and uncertainty in criteria, and disregard to nursing care standards as criteria.

Some participants said

“Internal evaluation is conducted incompletely and is useless. However, external evaluations occasionally are carried out by Ministry of Health and lead to accreditation or ranking results, which are not acceptable and efficient because it is done by the same organization that conducts the programs.” (P11: evaluation committee member). “Current evaluation tools are old, unspecific, and single faceted with only quantitative approach, which needs to be revised but there is no consensus on evaluation tools revisions (P19: evaluation expert).

Structural issues

Three types of challenges concerning the structural issues were cited in evaluation of nursing education: inappropriate culture, clinical education complexity, and lack of alumni follow-up system.

Inappropriate culture

The interviewees emphasized that the educational evaluation should try to deal with inappropriate culture challenge, which caused by lack of belief in evaluation values, lack of scientific attitude to evaluation, and fear of evaluation results. Consequently, it caused disinterest in evaluation and resistance to be evaluated. Furthermore, the participants pointed out that educational evaluation is faced by cultural challenges, including disregard to rule of laws, lack of accountability, conservativeness, and low morale as well.

Participant stated

“Some managers and faculty members look at educational evaluation as inquisition or strictness rather than feedback for performance improvement and accountability (P17: evaluation committee member). “In organizational culture, evaluation is not well established; faculties don't like supervision and performance evaluation. Academic managers are disinterested to evaluate and be evaluated then they try to keep present situation in any way even by disregard to laws” (P18: vice-dean).

Clinical education complexity

Interviewees highlighted that clinical education environment has some challenges concerning integration of medical education with health services, unscientific clinical governance, and noncollaborative atmosphere, which acts as a barrier for evaluation in nursing education. They also pointed out that complexity of nursing practice as a major challenge regarding the multiplicity of clinical courses, multiplicity of clinical wards, and multifaceted clinical setting.

“In clinical education settings, we don't see scientific governance and supervision.” (P13: evaluation committee member). “In nursing profession, educational evaluators face with lack of educational documentation in nursing practice and a variety and numerous clinical courses” (P10: national board member and dean).

Lack of alumni followup/tracking system

Participants pointed out that there is ambiguity for realizing responsible organizations for alumni assessment, including lack of alumni supervision organizations, lack of alumni associations, inaccessibility to alumni, lack of alumni competencies assessment, and lack of alumni impact assessment on workplace. The other challenges in which participants focus on those are unawareness of alumni job situation, lack of alumni job satisfaction measurement, and unawareness of alumni academic upgrading.

Participants expressed

“There is no association or organization to gather information about alumni and their job situation, satisfaction, and competency or even academic achievement” (P8: evaluation committee member). “The ministerial nursing office doesn't evaluate graduates. The work places should provide documented feedbacks to nursing schools about nursing graduates” (P9: national board member).

DISCUSSION

We analyzed the perception of participants about challenges that educational evaluation in Iranian nursing education currently faces and found 9 that were pressing enough to urge the attention of the Iranian academic managers.

One challenge is that evaluation management is governmentally appointed in Iran. Cabinet change was caused to change managers in MoHME and universities of medical sciences. Consequently, management instability and inexperienced managers do not have commitment to evaluation requirement in nursing education, which is in line with other researchers’ works.[19] Another challenge is that centralization and hierarchical structure in MoHME and government has produced more centralized policies and strategies and unscientific policy making. Moreover, ministerial monopoly to evaluation leads to political attitude to evaluation. Researchers[927-30] have found that educational evaluation policies are sometimes ineffective, centralized, and unstable as well.

A third challenge deals with ineffective evaluation planning. Educational evaluation plan lacks regard for excellence model in evaluation, being unsystematic and goal based. Furthermore, evaluation plans are more focused on only minimum standards of evaluation with ambiguity of evaluation objectives and lack of appropriate feedback system. Our results are in line with other researchers’ works.[31-33] Another challenge to evaluation in nursing education refers to the inefficient and affected evaluators. In the implementation stage of educational evaluation, evaluators have a crucial role to effectiveness and efficacy of evaluation plan. However, educational evaluators were appointed by MoHME with not enough consideration to experience, expertise, interest, and independency. Moreover, the inefficient evaluators will be more affected by influential outsiders’ and insiders’ inferences and recommendations, which are in line with other works.[28]

A further challenge relates to the inappropriate implementation of educational evaluation. Effective educational evaluation in practice faces lack of essential requirements, such as regular time, faculty member and student evaluation, and trade-off between programs and specified evaluators. However, in action evaluators evaluate educational programs individual-based and without stakeholders’ collaboration lead to faulty measurement. Other investigators have found similar results regarding these issues.[24293435] Another challenge refers to the inefficacy of approaches and tools. Educational evaluation approaches should be integrated, including internal and external evaluation plans, accreditation, and ranking evaluation. In Iran, educational evaluation approaches were conducted parallel, isolated, sectional, and sometimes with individual-based and nonqualitatively. However, it would be wrong to think that Iran is a unique case.[28] In addition, Iranian evaluation tools are consider old fashioned without experts’ consensus, which is in line with other works.[3233] Another concerning challenge to educational evaluation in Iran is disinterest and resistance to evaluate and to be evaluated as a cultural issue resulting in lack of accountability. Other Iranian investigators also have found cultural resistance to evaluation.[936] Another worrying challenge is complicated and multifaceted clinical setting. These structural issues affect on nature of clinical education and lead to incomplete and tough educational evaluation. Some researchers stated that overly extensive set of missions and responsibilities is the result of merging health services and medical education in Iran.[129323738]

Another finding showed that there is no appropriate mechanism for access to job situation, job satisfaction, and academic upgrading of alumni. Furthermore, educational evaluation faces lack of alumni associations and supervision organizations. It seems the alumni followup system to be a requirement for effective educational evaluation, which is in line with other researcher's works.[1618]

This study has some limitations that are worth mentioning. The participants do not cover the complete spectrum of nursing education and management population. Clearly, the present study does not identify all challenges related to educational evaluation in Iranian nursing. Despite the negative picture of evaluation in nursing education depicted by this article, educational evaluation in Iranian nursing education has promoted establishment of nursing courses and postgraduate programs.

CONCLUSION

In this study, we endeavor to clarify the challenges of effective educational evaluation in nursing that require dynamic changing and dealing at school, university, and the ministerial levels. According to Schwartz and Westerheijden, the dynamic educational system demands dynamic evaluation methods.[39] Our results emphasize the need for educational evaluation development in Iran, including systematic and regular educational evaluation planning focusing on efficient feedback system and regard to excellence models. Moreover, it is important to consider the scientific policy making, all stakeholders’ participation, and collaboration, academic management development, trust, and employing the evaluation results as educational evaluation requirement. Educational evaluation system seems to be focus on integrated and effective internal evaluation with emphasis on qualitative approaches. Despite the presence of private and governmental nursing education, formal educational evaluation is completely on MoHME regarding external evaluation programs, including accreditation system and ranking plans. The comprehensive educational evaluation requires participation, involvement, and collaboration among the Nursing Board, Nursing Office management, faculties of nursing, Nursing Organization, and other involved organizations in order to goals setup, planning, implementation, and evaluation.