A half-day stroke workshop based on the Kirkpatrick model to improve new clinical staff behavior.

INTRODUCTION: The present study aimed to determine the validity and usefulness of scales and training programs for clinical staff to evaluate nerve signs as an initial response to stroke. We developed a stroke workshop, using the analysis, design, development, implementation, and evaluation (ADDIE) model method based on instructional systems design theory.
METHODS: The workshop aimed to improve the basic first aid skills of clinical staff for stroke. The participants (n=46) were randomly assigned to conventional Cincinnati Pre-hospital Stroke Scale (CPSS) or modified CPSS groups (simple randomization). Short-term case simulation was conducted immediately after the training as well as 6 months later to evaluate the nurses' skills. We conducted evaluations, using an instructional framework throughout the ADDIE process. We used the Kirkpatrick model to evaluate the educational effect of up to level 3 in this study. The Wilcoxon signed-rank test was used to analyze differences between the pre-test and post-test groups.
RESULTS: The evaluation of the new clinical staff stroke emergency training program, either using the conventional CPSS or the modified CPSS, showed that the participants were highly satisfied and exhibited improved knowledge and skills (conventional CPSS: 3.05 ± 0.73 vs 3.64 ± 0.59, P = 0.012 and modified CPSS: 2.95 ± 0.97 vs 3.61 ± 0.49, P = 0.111, before training vs after training, respectively). On the other hand, it was difficult for the participants to evaluate neurologic conditions using the modified CPSS compared with the conventional CPSS.
CONCLUSION: These results demonstrated that stroke care training is effective in reaction, learning, and behavior. The modified CPSS could be useful as with the conventional CPSS. In future, evaluation of neurological conditions should be improved. Copyright: © Shiraz University of Medical Sciences.

Introduction

The present study aimed to determine the validity and usefulness of scales and training programs developed for new clinical staff members to evaluate nerve signs as an initial response to stroke. The training program was developed using the analysis, design, development, implementation, and evaluation (ADDIE) model method ( 1 , 2 ) based on instructional systems design theory ( 3 ). The present workshop design mainly included the adoption of applying the Cincinnati Pre-hospital Stroke Scale (CPSS) ( 4 ) to simulated patients. Netz, et al. reported that CPSS could be also useful in hospital for stroke with acute onset ( 5 ). The CPSS is a validated pre-hospital stroke screening tool that has been easily and widely adopted by emergency medical services. The CPSS is scored from 0 to 3, with one point given for each of the following physical exam findings: facial droop, arm drift, and slurred speech ( 4 , 5 ).

The Kirkpatrick model of evaluation is used to evaluate whether a training program is likely to meet the needs ( 6 - 8 ). It distinguishes among four outcome levels following educational guidelines (Figure 1). Many studies on faculty development have focused on Kirkpatrick level 2 outcomes (changes in knowledge). Some studies that investigated level 3 or 4 outcomes (changes in behavior or outcomes) following an educational intervention failed to find any important effects ( 9 , 10 ). Especially, it is difficult to evaluate effects by level 4 ( 11 ).

Figure1

Kirkpatrick model of evaluation

We conducted a preliminary survey on the degree of understanding for clinical staff in our hospital ( 12 ). Neurological findings of consciousness level, pupil, and paralysis evaluation are essential for the early detection and response of cerebral signs ( 13 , 14 ). Among these, findings of paralysis can rule out symptoms other than those of cranial nerve disorder, such as hypoglycemia and peripheral symptoms, and a simple scale is required for evaluation. Some studies have reported the training for stroke care ( 13 - 15 ).

The present study aimed to determine the impact of a stroke workshop on clinical faculty knowledge and behaviors. This workshop is targeted for all clinical staff.

Methods

We conducted a study on 46 new clinical staff members (medical doctor: 0, nurse: 46, other staff: 0) in Mitoyo General Hospital, Kagawa, Japan, from 2015 to 2017. The participants took a stroke workshop aiming to improve their basic first aid skills for stroke.

Research flow diagram using an instructional framework

Pedagogical approach, using ADDIE model (

In the analysis phase, the instructional problems were clarified, the instructional goal was established and the participants’ knowledge and skills were identified. We conducted a preliminary survey on the degree of comprehension for the clinical staff in our hospital. Our survey revealed the degree of comprehension of 50% for airway problems, 49% for breathing problems, 40% for circulation problems, and 29% for disability ( 12 ). The goal is to get knowledge and skills to find stroke early. Our training objectives were to 1) understand the contents of CPSS and 2) evaluate neurological conditions, using the CPSS scale. Forty-six new nurses were randomly assigned to the traditional CPSS or modified CPSS. Training took place in 2015 and August 2016. Short-term case simulation was performed shortly after training in 2016 and 2017, wherein the skills and quality were evaluated.

The design phase established learning content, lesson planning and media selection. We gathered feedback from the analysis phase and resources on the topic provided by information with evidence. The contents of this workshop were designed to augment the Basic Life Support (BLS) resuscitation paradigms through the addition of a structured approach to early stroke assessment based on the Immediate Stroke Life Support (ISLS) course. To allow the participants to practice smoothly, we recruited three qualified, clinically experienced and ISLS certified facilitators. The facilitators explained the objectives and goals of the training ( 15 ). Stroke scale practice was first observed and then practiced.

By applying the CPSS used in pre-hospital settings, we developed a scale that can detect the severity and abnormality of nerve signs at an early stage and can lead to the early response to and treatment of stroke. Measurement items were the same as those of the CPSS, namely facial paralysis, upper limb paralysis, and language impairment. The evaluation scale was used in pre-hospital settings without modifications. Each item was measured using the NIHSS measurement method: facial paralysis, 0–3; upper limb paralysis, 0–4; language disorder [1]; aphasia 0–3 [2]; and articulation disorder 0–2. We modified the scale to a constant of 12 points and refer to the modified scale as modified CPSS.

During the development phase, we received the reviewed design document. We developed modified CPSS based on conventional CPSS. The conventional CPSS is presented in Figure 3A. The modified CPSS evaluated the neurological condition of the simulated patients, using Figure 3B. We revised case simulations to introduce the modified CPSS. After practical training, the subjects were explained the importance of comprehensively judging diseases and symptoms other than stroke, and the intention of index of scale evaluation in evaluating stroke scale as a whole. We developed a training DVD (digital versatile disc) of normal cases during the training sessions and distributed it as a self-learning teaching material. We expected the learning effect by video ( 16 ).

Figure3

(A) Conventional Cincinnati Prehospital Stroke Scale (CPSS) and (B) modified CPSS. We designed the modified CPSS by incorporating the aspects of convenience of CPSS and quantitative measurement of NIHSS

During the implement phase, we received the reviewed design document and used the designed scale. One staff member worked with two to four students. The students worked in pairs and alternated between the role of patient and clinical staff. In the scenario, after performing the evaluation alternately in normal cases, we evaluated one stroke case, and we set a time for debriefing for each case in the group. We expected the learning effect by video ( 16 ). Three cases of stroke, facial nerve paralysis, cervical spine injury, and hypoglycemia were prepared in the case simulation test immediately after training and again 6 months later. For the case simulation test, a room separate from the training venue was set up. Each case took 2 min; feedback and movement after the simulation took a total of 5 min per person. We also prepared a waiting room so that the participants who completed the simulation test and those waiting to complete the examination would not come in contact.

One staff member worked with two to four students. The students worked in pairs and alternated between the role of a patient and clinical staff. In the scenario, after performing the evaluation alternately in normal cases, we evaluated one stroke case, and set a time for debriefing for each case in the group. Three cases of stroke, facial nerve paralysis, cervical spine injury, and hypoglycemia were prepared in the case simulation test immediately after training and again 6 months later. For the case simulation test, a room separate from the training venue was set up. Each case took 2 min; feedback and movement after the simulation took a total of 5 min per person. The conventional CPSS is presented in Figure 3A. The modified CPSS evaluated the neurological condition of the simulated patients, using Figure 3B. We also prepared a waiting room so that the participants who completed the simulation test and those waiting to complete the examination would not come in contact.

Educational evaluation is important for improving educational programs’ quality and efficiency. Kirkpatrick’s 4-step evaluation model is an important tool for measuring and evaluating educational programs ( 16 , 17 ). This model evaluates the satisfaction level of training at four levels: level 1 (reaction), using a questionnaire survey immediately after attendance, etc.; level 2 (learning), using evaluation of learners’ achievement by written tests, reports, etc.; level 3 (behavior), using evaluation of behavior change by interview with students or evaluation of others; level 4 (results), using an index of four grades for evaluating the degree of improvement in learners and in workplace performance after taking a training course. In this study, levels 1, 2, and 3 were evaluated. Level 1 was conducted via a questionnaire. To confirm the level of comprehension of the training, questions on the same level of accuracy judgment were given before and after the training for level 2. As a behavioral evaluation, short-term case simulation tests, using the conventional CPSS and the modified CPSS, were conducted immediately after training and 6 months later. Skill evaluation was performed using the independently created evaluation table. Level 3 was assessed, using a questionnaire with four Yes/No questions about actions and changes in consciousness after training.

The conventional CPSS is reliable and widely prevalent as a hospital scale. Mackey, et al. reported that the conventional CPSS could be useful even on bedside use (class II evidence study) ( 18 ). Although we modified the reliable conventional CPSS, we kept the basic principle of the conventional CPSS. We have developed it as a modified CPSS that can evaluate neurological findings level.

All data were reported as mean±SD. The Wilcoxon signed-rank test was used to analyze differences between the pre-test and post-test groups. The χ² independent test was used to compare the conventional CPSS group and the modified CPSS group. A statistically significant difference was defined as a P value less than 0.05. All statistical analyses were performed, using StatMate V (ATMS Co., Ltd., Tokyo, Japan).

Ethical consideration

This study was conducted at the Mitoyo General Hospital after being approved by the hospital’s ethics committee. The purpose of the research study was explained to all subjects, and all subjects gave their consent to participate. The test results and consciousness survey results were carefully managed to maintain anonymity.

Results

The average age of the participants was 24.3±6.8 years. Two were teenagers, 34 were in their 20s, nine were in their 30s, and one was in 40s. Out of 46 new clinical staff, 43 participated in the training course. Four participants from the conventional CPSS group and one participant from the modified CPSS group were absent during the simulation test due to poor business and physical condition. Therefore, we evaluated the skills of the remaining 17 participants of the conventional CPSS group and 21 of the modified CPSS group.

Evaluation of pedagogical approach by ADDIE model

We describe the inputs and outputs during each phase of stroke workshop development, using ADDIE model (Table 1).

Table1

Pedagogical approach for stroke workshop using ADDIE model.

ADDIE	Inputs	Outputs
Analysis	Preliminary survey	Learning needs data Pilot workshop program
Design	Learning objective	Design document Established learning content
Development	Learning tool applied to design document	Modified CPSS Development of video
Implementation	Improvement of teaching methods	Refinement of workshop
Evaluation	Evaluation by Kirkpatrick’s model	Level 1-3 evaluation results

Kirkpatrick Level 1: Reaction

The level of satisfaction with the training contents in the conventional and modified CPSS groups is shown in Table 2. No significant difference was observed in scale understanding, scale evaluation, or training satisfaction scores between the conventional and modified CPSS groups. In response to the question of whether the outcome of training can be utilized in future nursing, the conventional CPSS group scored 4.12 ± 0.49 and the modified CPSS group scored 4.14 ± 0.64, indicating no significant difference (Table 3).

Table2

The visual analog scale scores of satisfactions with the training contents

Question items	Conventional CPSS group (n=17)	Modified CPSS group (n=21)	P
	Mean±SD	Mean±SD
Understanding of scale	79.4±10.7	75.7±16.4	0.88
Evaluation using scale	79.4±10.7	75.7±16.4	0.94
The scores of satisfaction of the rating	88.8±10.9	88.5±17.4	0.63

Table3

Responses to whether this training can be used for future nursing based on a 5-point Likert scale

Conventional CPSS group (n=17)	Modified CPSS group (n=21)	P
Mean±SD	Mean±SD
Mean±SDMean±SD4.12±0.49	4.14±0.64	0.89

Kirkpatrick Level 2: Learning

Evaluation of understanding was conducted before (pre) and after (post) training (Figure 3). In the conventional CPSS group, the average points were 3.05 ± 0.73 before and 3.64 ± 0.59 after the course (P = 0.012, Figure 4, left). In the modified CPSS group, the average points were 2.95 ± 0.97 before and 3.61 ± 0.49 after the course (P = 0.011, Figure 4, right).

Figure4

Comparison of knowledge before (pre) and after (post) the training course. Evaluation of the degree of comprehension of the training in the conventional CPSS group (left) and the modified CPSS group (right)

Results of the test using simulated patient evaluation are shown in Table 4. In post-training difficult cases except for stroke, facial weakness evaluation was at 100% for the conventional CPSS group and 67% for the modified CPSS group (P = 0.008). Dysarthria evaluation was at 100% for the conventional CPSS group and 71% for the modified CPSS group (P = 0.016). However, no significant difference was observed between the scores immediately and 6 months after training.

Table4

Results of test using simulated patient evaluated using chi-square test

Test using simulated patient		Convetional CPSS group (n=17)	Modified CPSS group (n=21)	p
		Mean±SD	Mean±SD
Post training course
Stroke case
	Facial weakness	76±0.43	57±0.50	0.212
	Motor arm	82±0.38	90±0.29	0.461
	Dysathria	76±0.43	95±0.21	0.089
Applied case
	Facial weakness	100±0.00	67±0.47	0.008
	Motor arm	94±0.23	86±0.35	0.401
	Dysathria	100±0.00	71±0.45	0.016
After 6 months
Stroke case
	Facial weakness	63±0.49	52±0.50	0.368
	Motor arm	75±0.43	71±0.45	0.159
	Dysathria	75±0.43	95±0.21	0.074
Applied case
	Facial weakness	94±0.24	71±0.45	0.086
	Motor arm	94±0.24	71±0.45	0.086
	Dysathria	100±0.00	90±0.29	0.204

Kirkpatrick Level 3: Behavior

We investigated behavior change of 43 new nurses participating in the training after 6 months and 18 months. We also investigated 177 staff members in the training participation department as a control. In the behavior change survey, the results for four of the five items were high (P = 0.039, Table 5).

Table5

Behavior transformation after 6 months and 18 months

Trainee participants new face nurses (n=43) Mean (%)	Nurses other than training participants (n=177) Mean (%)
Did you read a book related to the cranial nervous system
72	44
Did you at the time of sudden change, attempted to evaluate the consciousness level after checking ABC
58	59
Did you at the time of a sudden change, attempted to evaluate the nervous system
40	31
Did you at the time of a sudden change, in case of 2 digits or more I attempted to evaluate nervous system
67	51
Reporting to the doctor including neurological findings
56	45

Discussion

The present stroke workshop was a half-day program designed to augment the BLS resuscitation paradigms through the addition of a structured approach to early stroke assessment. The training course developed for the new clinical staff members is based on valid guidelines and provides the ability to diagnose patients in a hospital ( 15 ). An optimal instructor-to-learner ratio should be approximately 1:4. The hands-on and simulation activities should be mentored and facilitated by experienced instructors who provide formative assessments and feedback throughout each exercise. The participants alternately role-played patients with specific neurologic deficits and clinical staff member to practice and self-assess the skills learned and apply the neurologic rating scales. Role-playing serves to engage and activate thinking about the disease process and course content and influences the healthcare awareness of the participants acting as simulated patients.

According to the questionnaire survey, many participants thought that the present program was useful to learn about the initial management of stroke patients. The evaluation of the stroke first aid training program for new clinical staff members showed high participant satisfaction and improved the participants’ knowledge and skills. The results of the present study demonstrated that training of stroke care could improve nurses’ reaction, learning, and behavior.

The evaluation of the new clinical staff stroke emergency training program, either using the conventional CPSS or the modified CPSS, showed that the participants were highly satisfied and exhibited improved knowledge and skills. On the other hand, it was difficult for the participants to evaluate neurologic conditions, using the modified CPSS compared with the conventional CPSS. As a future task, we have to improve the training method of neurological evaluation. A blended learning, which incorporates video-assisted resources, might be a useful tool to teach clinical skills to students ( 19 ).

Most studies based on the Kirkpatrick model have evaluated only levels 1 and 2 of this model ( 9 , 10 ). In the present study, the participants’ comfort levels with practicing skills for stroke improved, indicating changes in Kirkpatrick level 1 (reaction) and 2 (learning). Most of this improvement was maintained 6 months after the training, indicating a change in Kirkpatrick level 3 (behavior). In order to maintain skills in the long term, continuous learning using videos could be effective ( 20 ). Video-based learning has shown to improve participants’ learning output in various disciplines ( 21 - 23 ). On the other hand, three factors present challenges at the behavior level. First, the participants could find an opportunity to change behaviors. Second, the time for change in behavior could not be predicted. Finally, the atmosphere could have an impact on changing behavior ( 17 , 24 ).

A limitation of the present study was the lack of evaluation of Kirkpatrick level 4, which should be addressed in future studies. No study has assessed the effectiveness of training of clinical staff members for stroke care based on all four levels of the Kirkpatrick model.

Conclusion

Comfort among new clinical staff members in practicing stroke management could be improved with a half-day workshop, and most results of the training could be maintained 6 months later. Although the modified CPSS is more complicated than the conventional CPSS, there was no significant difference in the test. The modified CPSS could be useful as with the conventional CPSS. In future, it is necessary to improve training, using the modified CPSS.