The effectiveness of clinical teaching of mental health courses in nursing using clinical supervision and Kirkpatrick's model.

BACKGROUND: Clinical experience associated with the fear and anxiety of nursing students in the psychiatric unit. Mental health nursing instructors find it challenging to teach nursing students to deal with patients with mental disorders in an environment where they need to provide patient teaching and clinical decision-making based on evidence and new technology.
OBJECTIVE: To measure the effectiveness of clinical teaching of mental health courses in nursing using clinical supervision and Kirkpatrick's model evaluation in the psychiatry unit of Imam Reza Hospital, Bojnurd, Iran.
METHODS: This cross-sectional study was carried out from 2011 to 2016 on 76 nursing students from a university as part of a clinical mental health course in two semesters. The students were selected by a non-probable convenient sampling method. After completing their clinical education, each student responded to checklist questions based on the four-level Kirkpatrick's model evaluation and open questions relating to clinical supervision. Finally, all data was analyzed using the SPSS version 16.
RESULTS: The students have evaluated clinical supervision as a useful approach, and appreciated the instructor's supportive behavior during teaching and imparting clinical skills. This has made them feel relaxed at the end of the clinical teaching course. In addition, in the evaluation through Kirkpatrick's model, more than 70% of the students have been satisfied with the method of conducting the teaching and average score of nursing students' attitude toward mental health students: Their mean self-confidence score was 18.33±1.69, and the mean score of their performance in the study was evaluated to be 93.74±5.3 from 100 points.
CONCLUSION: The results of clinical mental health teaching through clinical supervision and Kirkpatrick's model evaluation show that the satisfaction, self-esteem, attitude, and skill of nursing students are excellent, thereby portraying the effectiveness of clinical teaching. But this program still needs to be reformed. To establish long-term goals and obtain knowledge and clinical skills of nursing, it is recommended to develop a curriculum and evaluate it appropriately.

1. Introduction

The learning process for students of psychiatric departments is very complicated. Fear is a common reaction of many students as they become influenced by media advertising about patients with mental disorders (1). For mental health nursing instructors, it is challenging to teach nursing students to deal with patients with mental disorders in an environment where they need to provide patient teaching and clinical decision-making based on evidence and new technologies. After entering the psychiatric unit, nursing students feel distrustful and unable to make therapeutic interactions. There are three ways in which instructors can impart self-esteem to nursing students in the psychiatry department: 1. By providing positive feedback. 2. By establishing some kind of honest relationship so that students are comfortable in answering questions. 3. Presentation of teaching and learning through clinical experiences (2, 3). Mental health educators tend to transfer their valuable experiences to nursing students, thereby making them ready for long-term aims of clinical knowledge and skills, while the curriculum resources of any mental health nursing course is limited to clinical teaching of nursing students. Instructors and students search ways to construct meaningful knowledge (4, 5). To provide education and teaching through the clinical experiences of writers in the field of local mental health education in the clinic of psychology department of Bojnurd, and based on the programs of North Khorasan University of Medical Sciences, Iran, two studies were conducted in 2012–2013. They used role-play scenarios, problem-solving, and films to teach mental health-related issues to nursing students at the clinic (6, 7). According to the results obtained from these two studies, the main questions are: How can we evaluate the knowledge, skills, and attitudes of students by implementing different educational methods? Will the clinical environment create rich experience for students in the future? And, whether we have chosen a suitable strategy for learning as well as evaluation purposes. Next, we found a clear picture of the curriculum to understand the strengths and weaknesses of the program as well as to determine the value and effectiveness of clinical education programs. Several models and patterns have been proposed, most notably the clinical monitoring and evaluation model. The Kirkpatrick model can be noted in this regard. Clinical supervision is an important strategy, especially in the psychiatric unit, for learning and refining skills which provide the opportunity for mental health nursing students and mentors to participate actively in reflecting teaching experiences. If clinical supervision is conducted by the instructor at the clinic, it can provide an opportunity to process and re-think about the clinical experiences. In clinical supervision, nursing students are scheduled to receive assignments that should be documented during their training sessions on their clinical activities—for example, in reflective journals—and then the instructors will review the written assignments and provide feedback to the students in the training sessions (8–10). The study by Silvia et al. shows that writing in a notebook is a valuable tool to reflect on the learning experiences and abilities of the students. In fact, the students describe the meaning of their experiences, recognize their feelings and those involved, and can assess the effectiveness of planned measures (11). The Kirkpatrick evaluation model is appropriate to assess the results of teaching studies and to investigate the field of nursing teaching. Yardley and Duran (2012) suggest that this model can be used for relatively short periods of training or short-term medical training programs. This model is not appropriate for long-term training programs with long-term training results that are important in the evaluation process of long-term results (12). It seems that the Kirkpatrick model should be used just in corporate training programs, but this kind of evaluation can be used both in clinics and in schools. Although this evaluation model was classified, it is not based on the value or importance of any classification. This model provokes more questions from students about training methods and educational contents (13). After compiling an appropriate content of clinical training with three semesters for a mental health course in nursing, an appropriate method to determine the effectiveness of training content seems necessary. So, this study aims to measure the effectiveness of a clinical teaching mental health course in nursing using clinical supervision and Kirkpatrick’s model evaluation in the psychiatry unit of Imam Reza Hospital, Bojnurd, Iran.

2. Material and Methods

2.1. Study Design

This is a cross-sectional study because the purpose of this study, just like other studies (14, 15) is to create learning opportunities and evaluation for all students. Therefore, this study lacks a control and experimental group. This study was conducted from the first semester of 2011 to final-year nursing students for five semesters (2011–2016) in the psychiatry ward of Imam Reza Hospital in Bojnurd, Iran. Thus, the needs assessment, along with the formulation and revision of a proper training content, was conducted for a mental health clinical training course of nursing students from the semester of 2011 to 2014. Then, the formulated training content was evaluated by clinical supervision and Kirkpatrick’s model in two semesters of 2014 to 2015.

2.2. Setting and Sample

The seventh-term nursing students, as part of the clinical mental health teaching course participated in this study. The selection of students was carried out by a non-probable convenient sampling method. The seventh-term nursing students who completed their homework assignments in mental health teaching were enrolled in this study, and students who first experienced the clinical mental health teaching course were excluded. All the final-year nursing students totaling 36 members in the first semester of 2014–2015 and 40 members in the first semester of 2015–2016 from October to early January participated in this study. In each semester, the students were arranged in five groups of eight to nine people. Each student was trained for 51 hours approximately a nine-day period in accordance with the nursing curriculum at the seventh semester, to learn about psychiatric disorders and their symptoms as well as related cares in the psychiatric ward of Imam Reza Hospital in Bojnurd, Iran.

2.3. Instruments

2.3.1. Clinical Supervision

To evaluate the educational contents of the clinical mental health teaching course by the clinical supervision method, a reflective journal was first prepared by the mental health instructor, and was approved by the Board of the Nursing and Midwifery Faculty, Bojnurd, Iran. This reflective journal contained assignments based on the students’ experiences and practices, including the description of the clinical teaching course of mental health, educational rules and regulations, training objectives, introduction of educational resources, notes of daily performance, individual interviews with patients, various types of mental disorders, conferences, nursing reports, psychiatric medication, physical examination of patients, patient education, and clinical experiences of the students. Then, the clinical mental health instructor taught each student during the clinical mental health course, and gave feedback. After completing the clinical teaching course for each group of students, they answered three questions regarding the assessment of the homework assignments of clinical students

2.3.2. Kirkpatrick Evaluation Model

To evaluate the clinical teaching mental health course immediately at the end of the course (Day 9), each student group was assessed using the four-area Kirkpatrick evaluation checklist. The content validity of the checklist was confirmed by 10 professors in the nursing and midwifery school. The checklist contained 26 questions relating to the four areas of reaction, learning, performance, and result. To measure the reaction, three questions were designed—an open answer question and two questions that were ranked by Likert method to measure the students’ satisfaction and interest in the content of instruction, how to implement clinical education, and how to learn. The Likert scale ranged from very dissatisfied (= 1), dissatisfied, neither satisfied nor dissatisfied, satisfied, very satisfied (= 5). The level of learning of the students in the clinical mental health teaching course was examined in three sections: attitude (six questions), self-esteem (four questions), and knowledge (13 questions based on subjects and objectives of the course). The Likert-type scale ranging from 1 to 5, from “strongly agree” to” strongly disagree”, and knowledge questions were measured using a 0–100 score system so that the students scored 0–100 on their subjects and the goals of the clinical mental health teaching course. The performance level of each student was evaluated from the perspective of the clinical instructor, students, peers, clinical nurses, and patients. Students, clinical instructors, peers, and clinicians received a checklist of 20 questions that assessed the students’ practical and communication skills from very low (1 score) to very high (2 score). The checklist of 15 questions, which was scored based on the Likert method (yes, to some extent, no), was received. The results of the training will be used in four sections: cost savings, savings, strategic benefits, and results. In this study, the results index was used in a way such that at this level, the contents of the teaching clinical mental health course, including student-sponsored care plans, student clinical journals, and patient education sheets, were presented to the supervisor and the head nurse of the psychiatric department to be evaluated by hospital auditors, based on hospital accreditation measures. Data were analyzed using the SPSS statistics version 18.

2.4. Ethics of the research process

The consent form of the voluntary company in the study was obtained from all students and evaluators. The students were assured that their responses had no effect on their grades and were completely confidential and anonymous. An encoding system was used to keep the evaluators’ information confidential. Also, this process was conducted after receiving permission from the training council of the Nursing and Midwifery School of North Khorasan University of Medical Sciences.

3. Results

Nearly 76 undergraduate nursing students participated in the present study. Of them, 53 (70%) were women and 23 were men (30%). The average age of participants was 20 years. Clinical supervision evaluation and Kirkpatrick’s model of evaluation outcomes are as follows:

3.1. Clinical supervision evaluation

As much as 70% of the students were not aware of the teaching method of clinical supervision in the clinical mental health teaching course. However, the evaluation results of this methodology by mental health nursing students according to their notes in the clinical journal revealed that they appreciate communication and interactions with their instructor, and the skills learned in this manner. It also made it clear that the students believe that the teaching course was purposeful and rich. They also explained that this teaching course was interesting in terms of learning and establishment of independence and responsibility. Although at the beginning of the experiment they felt worried and were afraid of the patients at the psychiatry department, they eventually gained more self-confidence and felt more satisfied with this instructor. Also, the teaching contents, such as film distribution, psychiatric drugs, psychotherapy, differential diagnosis of psychiatric disorders, they were a variety of activities that increased learning more than other activities. The psychiatric facility was inadequate in the interview room: The lack of an adequate number of desks and chairs to sit down or write a nursing report caused dissatisfaction. The need to present entertainment for patients, such as the publication of a journal, was also stressed. Other experiences mentioned by the students were listed in Box 1.

3.2. Kirkpatrick’s Model of Evaluation

3.2.1. Step One: Reaction

To measure the first level of the Kirkpatrick evaluation model (reaction), two factors, namely the Students’ emotion about quality of the implementation of clinical mental health teaching and their satisfaction with the implementation of the clinical mental health teaching course, were assessed.

3.2.2. Emotion Evaluation

As many as 70% of the nursing students had anxiety at the onset of clinical teaching. Only 7.5% of the students felt pleased to attend the clinical mental health teaching course, while 22.5% of those who participated in the mental health clinical teaching course, felt resistance within themselves. Moreover, in response to a question that asked students about their emotion at the end of the clinical mental health teaching course using a sticker, more than one-third (35.5%) of the students selected the “Sticker OK” to describe their emotional state.

3.2.3. Satisfaction Evaluation

The nursing students were satisfied with what they had learned. Combining excellent and very good choices — 82% and 68% in two semesters of 2014–2015 and 2015–2016 respectively — it was clear that the nursing students were highly satisfied.

3.3. Step Two: Learning

The evaluation results of the second level (learning) are as follows: Level 2 evaluation was performed in two stages (a, b)

3.3.1. Level a

This level relies on the measurement of attitude and self-confidence. The mean scores of the attitude and self-confidence of nursing students at the end of the mental health clinical teaching course was 31.82±3.47 and 18.33±1.69 respectively, which were evaluated to be high.

3.3.2. Level b

At this level, nursing students at the end of the clinical mental health teaching course scored each part of the theoretical and clinical content on a scale of 1–100, with 100 being the maximum, which is summarized in Table 1.

Table 1

Nursing students’ evaluation of theoretical and clinical components relating to the content taught in mental health clinical courses on the basis of 0–100 scoring

Question	Theoretical and clinical components	Percentage of frequency
What theoretical and clinical components relating to the content taught in mental health clinical courses help you to learn and improve skills?	Group discussion about mental illness	71.75
	Individual interviews	70.63
	Conferences	65
	Group discussion about drugs	71.13
	Movies	70.63
	Neurological examinations	63.38
	Mental health terminology in English	69.38
	Nursing reports	80.45
	Patient education	82.5
	The evaluation method of nursing students	65.95
	Psychotherapies (cognitive, relaxation, visualization, deep breathing, etc.)	88.76
	Clinical supervision	88.83
	Rethinking of obtained clinical skills	63.03

3.4. Step Three: Performance

Evaluation was done in two levels (a, b)

3.4.1. Level a

Assessing the performance of nursing students was done by using skill in patient education. The maximum score for the patients’ study checklist was 28 and the minimum was eight. The average score was 20.82±4.3, and was obtained by the students at this level.

3.4.2. Level b

Assessing the performance of nursing students using a 360-degree assessment method (Table 2).

Table 2

Mean scores obtained by nursing students in the 360-degree evaluation

Evaluator score	Mean and Standard deviation
Instructors’ score	97.25±3.05
Undergraduate nursing students’ score	89.55±7.2
Peers’ score	94.22±4.4
Nursing staff score	93.95±1.08
Total evaluation	93.74±53

3.5. Step Four: Results

The evaluation results of the fourth level of assessment using Kirkpatrick’s model are as follows:

3.5.1

Provide clinical training results to patient education and care plan, logbooks, translations, psychotherapy, conferences with the clinical/psychiatric head nurse and educational supervisor at Imam Reza Hospital.

3.5.2

An evaluation of this level through accreditation measures of clinical training in educational hospitals was conducted on February 20, 2016, but the results have not been published by the hospitals.

4. Discussion

This paper evaluates the results of teaching nursing students a clinical mental health course using the two methods of clinical supervision and Kirkpatrick’s evaluation model. The results of this study show that the students were concerned about conducting the mental health training, and had no idea about the reflection journal. They showed a weak reaction to their first experience of the mental health course in nursing. However, after the implementation of the clinical teaching of the mental health course, the support provided by the instructor increased their learning, self-esteem, satisfaction, responsibility, and sense of independence. Thus, their negative feelings eventually turned into positivity during the training course on mental health. In line with the results of this study, a review of other studies conducted by Nadler-Moodie and Loucks (16), Happell and McAllister (17), Melrose (4), Penn (2), Mahoney et al. (3) and Chawla et al. (18), shows that it is not easy to train nursing students in psychiatric wards, because the required clinical skills in other departments are not helpful in the psychiatric ward. Thus, most nursing students suffer from anxiety, anger, fear, and loss of confidence, and experience conflict in playing their role in the psychiatric ward. In addition, it is challenging for nurse instructors to educate nursing students to deal with mental disorders in an environment that requires education and clinical decisions based on evidence and new technologies. Upon arrival to the psychiatric ward, nursing students are often plagued by a lack of confidence, and unable to establish therapeutic interactions. In this study, three ways have been described in which the psychiatric nursing instructor could impart self-confidence and motivation in the students and change their reaction to the psychiatric ward and mental health training course: 1. Provide positive feedback 2. Make an honest relationship so that students become comfortable in answering questions. 3. Provide education and training through partnerships along with discipline, feedback, role playing, games, evidence-based education, training principles and clinical experience, and group discussions (19).

This study benefits from four supportive components related to the students’ learning. The first component was the use of small groups of students comprising seven to eight members, who attended the training for three weeks based on the planning devised by the nursing school. A review of studies showed that small groups in clinical training is helpful to clear the meaning of new contents and to impart it to the minds of the students (20). The second important component is the application of clinical supervision that has been emphasized as a tool for nursing students’ training in the mental health course. Echternacht and Youngs state that clinical supervision is an important strategy, especially in the psychiatric ward, that provides the necessary opportunity for students and psychiatric nurse instructors to actively participate to reflect on the learning experience; it is an important strategy to acquire and develop clinical competence and skills (10). The learning of students will be complete with clinical supervision of each student’s performance in a psychiatric ward along with real patient experience. Clinical supervision in a psychiatric ward means a high level of clinical learning. If clinical supervision is performed by the faculty instructor in the clinic, it creates an opportunity to process and rethink the obtained clinical experiences. Clinical supervision limits the gap between theory and practice. This is particularly beneficial in the concepts and principles of communication therapy to assess the mental health status of patients (21). The third level of Kirkpatrick’s evaluation model relates to behavior measurement — it refers to the changes in the behavior of learners (19). The idea is that the training program would be offered at the same time and the same place, as the offered program is not measured, but the learners come back to the location of the education at a later date to encounter the training program and be evaluated (22). In this study, there are some limitations relating to the students’ behavior measurement: Each student trained in the psychiatry ward for nine days, and enrolled in other training courses at the end of the period. So, there was no possibility to measure their behavior later. Moreover, since one of the duties of nurse instructors is to create learning opportunities for all students, there was no control group in this study. In line with Roos et al, we created the facilities and prerequisites for implementing educational interventions in the clinical area, including nurses’ participation in the psychiatric ward with the instructor and monitoring the students’ performance at the time of the mental health status of patients, and nursing reporting and implementation of drugs. Furthermore, to prevent the antithetic behavior with the educational program, the instructor should explain to the students what should they do and why they should do so, based on certain principles (23). The fourth level of Kirkpatrick’s evaluation model relates to the evaluation of results. At this level, we want to know whether the students’ performance based on this educational method would result in useful and verifiable results. The results of training can be divided into four areas: avoidance of the cost, benefit, cost reduction, and saving. This model does not exactly emphasize any particular component (13, 22). In this study, the researchers could not evaluate the results of the training at a later date because of the students’ graduation. Thus, the results of the clinical education of the mental health training course, including patient education documents, reflection journals, and patient care plans, are given to the psychiatric head nurse and the educational supervisor of the hospital to be evaluated and scored in the evaluation program of the training hospitals of the health system reform plan. This study has limitations, including the fact that the participants in this study are only from one nursing school, similar to all the studies on this subject. Therefore, the information of this study may not be generally applied to the mental health training programs in other nursing schools. It is recommended to conduct this research with a larger sample size. In addition, in this study, the perspective of other instructors has not been evaluated about this type of evaluation. So, a similar study can be conducted by considering the perspective of other instructors about this kind of evaluation. Self-assessment by students, especially in the measurement, measurement of learning rate, attitude and confidence, may have limitations and therefore the evaluation of each student may be different to those of others. However, the findings of this study and the presented educational content relating to mental health lessons could be a useful clinical training guide for other instructors to effectively and comprehensively present the mental health lesson to nursing students, and evaluate them.

5. Conclusions

In summary, the findings of the evaluation of nursing students’ education in the mental health teaching course by Kirkpatrick’s evaluation model and clinical supervision show that the satisfaction, attitude, learning, and skills of the students are excellent, thereby presenting the effectiveness of the training imparted to nursing students on mental health. Clinical supervision and Kirkpatrick’s evaluation model can promote discipline in clinical education and provide a new method to educators, along with the traditional method, which would be effective in reducing the distance between theoretical and clinical education. However, this approach still needs to be reformed. It is recommended to develop an educational curriculum with the appropriate evaluation method in order to achieve the long-term goals of nursing students to acquire knowledge and clinical skills in mental health teaching courses.

“I intend to develop and codify a magazine, Mental Health, to improve the state of entertainment and patient education” “I had a sense of independence” “The best clinical teaching method for learning was the clinical teaching about mental health” “All clinical education was mental health” “The mental health teaching course was very purposeful” “We mastered to conduct appropriate interviews with patients” “Clinical teaching of mental health experiences differed from other clinical teachings” “First, I was afraid and worried, but then I entered the room of patients without any worries and interviewed” “We learned interesting things from the educational movies on mental health” “Good interaction with staff and friends in the psychiatric unit” “We had kept a better relationship with our instructor” “I know my responsibilities” “First, I was afraid of mental health clinical teaching, but at the end of my clinical mental health experience I felt I missed my patients” “The clinical mental health teaching course was very attractive to me because the instructor was always full of practical and experimental training” “The hard work and seriousness of the instructor in clinical teaching of the mental health course has been a source of admiration for everybody” “I would like to go to the psychiatry unit to complete my nursing postgraduate education”