Evaluation and Effectiveness of Dementia Staff Training Programs in General Hospital Settings: A Narrative Synthesis with Holton's Three-Level Model Applied.

BACKGROUND: People with Dementia (PwD) are frequently admitted to hospital settings. The lack of proper dementia knowledge, poor communication skills, negative attitudes toward dementia, and lack of confidence affects the quality of care, thus development of dementia trainings has increased. Nevertheless, literature regarding the effectiveness of training implementation is limited.
OBJECTIVE: The aim of this narrative synthesis is to 1) identify the characteristics of training programs and 2) explore the effectiveness of these training programs in everyday clinical practice.
METHODS: A systematic search in PubMed, PsycINFO, CINAHL, and Cochrane was conducted, including qualitative and quantitative peer-reviewed studies. Holton's evaluation model with its three outcome levels (learning, individual performance, and organizational results) was adopted. 14 studies were included.
RESULTS: The synthesis of the results was divided into two parts: 1) to describe the characteristics and content of trainings 2) to evaluate the effectiveness of training programs according to the three outcome levels of Holton's model, taking into consideration its construct domains: ability, motivation, and environment. Learning outcomes were assessed in all selected studies: 13 studies observed changes in individual performance, four studies reported changes within the organizational level, and only five showed sustainable changes over time.
CONCLUSION: Person-centered care (PCC) approaches, interactive and varied teaching methods, supporting conditions like champions, action plans, and setting care policies, are all characteristics of effective trainings. Successful programs should be sustainable over time, demonstrating positive outcomes across the organization. Based on current findings, there is a lack of adequate evaluation with regard to training programs on the organizational level.

INTRODUCTION

The world’s older population continues to increase, with the number of People with Dementia (PwD) rising in parallel [1]. PwD are more frequently admitted to hospital settings than older people without dementia [2]. It is estimated that 12.9–63.0% of people admitted to general hospitals have dementia [3]. As PwD can be intensive users of health services, caring these patients merits increased attention.

PwD are likely to stay in hospital longer [4] and their clinical outcome is predominantly negative. Malnutrition, dehydration, loss of functionality, increased disorientation, depression and delirium, low detection and diagnosis of dementia, frequent use of antipsychotic drugs, and a high mortality rate are some of the consequences of hospital admission [5]. Furthermore, the longer PwD stay in hospital, the stronger patients’ symptoms of dementia become. Up to 75% of PwD admitted to an acute hospital will experience Behavioral and Psychological Symptoms of Dementia (BPSD) [6], including agitation, aggression, wandering, hallucinations, and sleep disturbances [7] that further cause difficulties in managing and significant distress in caregivers [8-10]. The hospital surrounding appears to be unsuitable for PwD, as it can worsen BPSD [5] due to the fact that hospital admission is considered frightening and confusing for PwD [11].

The lack of proper dementia knowledge by all hospital care team members has recently been characterized as one of the major factors affecting care quality in hospital settings [5]. Moreover, poor communication skills, negative attitudes toward dementia, poor management strategies, and lack of confidence by nursing and medical staff, healthcare assistants, allied healthcare professionals, and other support staff further contribute to poor quality of PwD care [5, 12, 13]. Due to these factors, staff training is highly recommended for hospital surroundings [14].

Adaptation of the way hospital staff providing care for PwD is a complex procedure due to multiple personal, social, and organizational factors prompting different responses to training interventions and, therefore, different outcomes [12]. For instance, staff’s professional focus may affect the way the information about a person’s social or medical history will be used. Also, the expectations of employers for dementia care may have a further impact on how staff members prioritize their work with PwD. In addition, lack of long-term studies and methodological differences of studies make it difficult to compare outcomes.

Although there are reviews describing the current care regarding environment, culture of care, attitudes, and challenging behavior or exploring dementia care best practices including educational staff trainings [4, 5], only few reviews aimed on evaluation of educational programs in hospital settings. A realist review by Handley, Bunn, and Goodman [15] focused on what works about dementia-friendly interventions for PwD in hospital settings and how to achieve desired patient outcomes, including well-being, reduced anti-psychotic medication, reduced distress and challenging behaviors, adverse incidents (such as falls or hospital-acquired infection), and length of stay. A more recent review aimed to evaluate the effectiveness of educational interventions for general hospital staff to improve the care and management of older people with delirium and/or dementia [16]. Although there is limited evidence about the features that characterize effective programs, some suggestions for successful training implementations do exist. Specifically, two recent literature reviews explored the most effective approaches of dementia training and education for hospital staff using a model (Kirkpatrick’s four-level model) to evaluate the outcomes [17, 18]. Thus, the evaluation of training programs' effectiveness is recommended for developing future training programs.

Holton’s evaluation and effectiveness model

To understand influencing factors of educational research outcomes is among the most critical issues in Human Resource Development (HRD). Among several evaluation models, there are four models which are well established in the research fields [19]. The first is Kirkpatrick’s [20] four-dimensional measurement typology (i.e., reactions, learning, behavior, results), which may be the simplest method for understanding training evaluation and the most widely used technique. The second from Tannenbaum and colleagues [21] is based on Kirkpatrick’s typology by adding post-training attitudes and dividing behavior into two outcomes for evaluation: training performance and transfer performance. The third provided by Kraiger [22] has three multidimensional target areas for evaluation: training content and design, changes in learners, and organizational payoffs.

The fourth, Holton’s Human Resource Development Evaluation Research and Measurement Model [23], is a more comprehensive framework than Kirk-patrick’s four level taxonomy. This model integrates not only training evaluation like the others, but also training effectiveness [19]. The model hypo-thesizes that HRD outcomes are primarily a function of ability, motivation, and environmental influences at three outcome levels learning (trainee’s reaction, motivation to learn, ability), individual performance (motivation to transfer, intervention fulfillment, learning outcomes, job attitude, transfer conditions, transfer design), and organizational performance (link to organizational goals, expected utility/or payoffs, external factors). Learning and individual performance represent individual behaviors that an intervention is intended to create. Organizational outcomes can be conceptualized as a function of organizational goals, individual motivation, and environmental factors [23]. Unfortunately, there is no possibility to full test the model because there are no available tools for some of the constructs. Some years ago, the model was modified by delineating specific constructs based on new research or theory [24]. Among several modifications: two separate constructs of “Motivation to Learn” and “Motivation to Transfer” completely re-conceptualized motivation by creating the construct of “Motivation to Improve Work Through Learning” [25]. Besides, the constructs of “Transfer Conditions” and “Transfer Design” were re-conceptualized to one construct: “Learning Transfer System Inventory”, including all factors in the person, training, and organization that influence transfer of learning to job performance. The modified model is more appropriate for empirical testing [24].

This narrative synthesis gives an overview of the established training programs in general hospitals. Its purpose is to 1) identify the characteristics of all the selected training programs relevant to dementia (qualitative and quantitative studies), and 2) explore the effectiveness of each training program according to Holton’s Model (learning, individual performance, organizational changes), interpreting the outcomes in regards to its three constructs: ability, motivation, and environment.

METHODS

Search strategy

A systematic search of research literature was carried out with the help of four commonly used databases: PubMed, PsychInfo, CINAHL, and Coch-rane. The search strategy was conducted by using a combination of terms and subject headings for the following themes: Dementia, Hospital, and Education (Supplementary Material). The first systematic search was conducted in January 2017, and a second systematic search was conducted in September 2019 to identify papers published since January 2017.

Procedure, selection process, and data extraction

This paper follows the PRISMA Statement guidelines [26]. After removing duplicates, articles were separately screened by title, abstract, and relevance by two reviewers, namely MG and JS. Full-text articles that seemed relevant for inclusion in this study, were screened. The following inclusion criteria were implemented: 1) all studies took place in general hospital(s)’ settings/wards; 2) studies were all reported as interventions in the framework of a training or educational program about dementia for staff members involved in the care of PwD; and 3) all studies were written in English. Papers that met the following criteria were excluded: 1) training interventions targeting mainly in hospital staff who were specialists or working in special dementia care units (or psychiatric wards) in hospital settings; 2) reports/projects without intervention/recommendations for projects; 3) reviews/case studies; 4) programs that focus predominantly on staff training for delirium or other geriatric syndromes or issues irrelevant for dementia. Disagreements concerning the criteria for inclusion were resolved by a third party (BT). Data extraction was performed independently by the same two reviewers (MG, JS) using a standard table with the following data: 1) country of the study; 2) type of program; 3) study design; 4) participants (characteristics) and setting; 5) method of intervention (procedure and content); 6) frequency and duration of the program; 7) its evaluation; 8) main results; 9) quality rate of the study.

Quality assessment

A quality assessment of the selected studies was conducted. The quality scores are derived from Spector et al. [27]. Kmet et al. [28] introduced 14 criteria for assessing higher or lower quality of study designs. In this review, eleven items were used (Table 1), as in Scerri et al. [17], since only one included study was a randomized control trial (RCT) and three out of 14 items in this checklist focused on randomization and blinding studies. Each study was rated according to whether it fully met the criteria (2 points), partially met the criteria (1 point), or did not meet the criteria at all (0 points). A calculation was made and the total score summary was divided by the maximum score of 22. By this calculation three levels were categorized: low (<0.6), medium (0.6–0.8), and high (>0.8) quality studies [27]. In the current work, after grouping the studies according to their quality, the researchers set a tighter inclusion criterion, because of the great heterogeneity of the studies and to strengthen their results. Thus, only studies of medium and high quality were included.

Table 1

Quality rating assessment. From Kmet et al. [28]

Items
1.	Question or objective sufficiently described?
2.	Design evident and appropriate to answer study question?
3.	Method of subject selection or source of information/input variables is described and appropriate?
4.	Subject characteristics or input variables/information sufficiently described?
5.	Outcome and exposure measure(s) well defined and robust to measurement/misclassification bias? Means of assessment reported?
6.	Sample size appropriate?
7.	Analysis described and appropriate?
8.	Some estimate of variance is reported for the main results/outcomes?
9.	Controlled for confounding?
10.	Results reported in sufficient detail?
11.	Do the results support the conclusion?

Data synthesis strategy

The synthesis of the results was divided in two parts. In the first part, characteristics of studies, participants, and content of training were described. In the second part the evaluation and effectiveness of each training program was described, using the three primary domains of Holton’s Model [23], analyzing also the suggested factors of influence according to its three construct domains ability, motivation, and environment in an attempt to interpret the outcomes.

RESULTS

The search resulted in 6,735 hits. After removing duplicates, title and abstract were initially screened to identify relevant articles. 301 full-text articles were scrutinized. 14 articles met the inclusion criteria, also taking into consideration the quality assessment (Fig. 1).

Fig. 1

PRISMA flowchart of the search strategy. From Moher, Liberati, Tetzlaff, and Altman [74].

Table 2 presents an overview of the data extraction, with the main focus lying upon the outcome measures and the results of the selected studies.

Table 2

Data extraction

Study	Country	Program	Study design	Participants/setting	Intervention	Frequency/duration	Evaluation	Main results	Quality rate
Banks et al. [39]	UK (Scotland)	“The Dementia Champions Programme”	Pre-post design	N = 104,n = 93 became champions/4 sites of an acute setting (14 NHS Boards)	A blended learning approach based on Waugh et al. [73] plus an additional online virtual learning environment.2 sessions per study day including lectures (hospital environment and fundamentals of care, communication, stress and distress, planning for discharge, vulnerability issues, working with caregivers, legislative frameworks, end-of-life care) and group activities with a person-centered approach, visits from a number of invited speakers and guests.After completing the program, they became champions with a) submitted 3 written assignments fulfilling satisfactory standards, b) 80% attendance on the study days.	5-day program and a half-day spent in a local community setting.	A) Pre-post measurements in Approaches to Dementia Questionnaire (ADQ): n = 83 day 1, n = 89 day 5. After the half day spent in the community setting: completion of 3 work-based tasks including the submission of reflective reports about perceptions.B) Post measurement (5th day of training) with self-efficacy scale.C) Post measurement evaluation with a questionnaire about course materials and satisfaction.	The overall score of ADQ was significant (sig.) increased from day 1 to day 5.Positively altered perceptions about their community colleagues, volunteers, and informal caregivers.77% of participants felt confident to recognize the learning items. The delivery of the program was positively evaluated from the vast majority of participants (96.6%), who were also highly satisfied with the online sources.	15/22= 0.71 (M)
Elvish et al. [36]	UK	“Getting to Know Me” training program	Pre-post design	N = 72/6 wards in a general hospital (complex care, trauma orthopedic, and orthopedic wards)	Phase 1: evaluation with new psychometric scales (CODE and KIDE).Phase 2: completion of the “Getting to Know Me” intervention.Content (6 modules): introduction to dementia, seeing the whole person, communication, the impact of the hospital environment, knowing the person, a person-centered understanding of challenging behavior.Training materials: 2 DVDs, communication skills mini-guide, “Getting to Know Me” card, manual for trainers, and booklet for staff.	4 sessions/with a duration of 45’ to 90’ (6 h).	Phase 1: 115 participants in pre-measurements of three new scales: Confidence in Dementia Scale, Knowledge in Dementia Scale, and additionally the Controllability Beliefs Scale. Phase 2: 72 participants in pre-post-measurements of the same scales.Attrition rate: 37%	All three scales were valid tools (good internal consistency, face and content validity).The program had a sig. impact on staff knowledge; confidence and caring shifted towards a more person-centered perspective on challenging behavior.	18/22= 0.82 (H)
Elvish et al. [37]	UK	Phase 2 of “Getting to Know Me” training program	Pre-post design	N = 607 (n = 35 trainers)/3 NHS Trusts.n = 517 signed a consent form	Involvement of a “train-the-trainers” phase in the “Getting to Know Me” program.N = 607 completed the training.Content: same as in Elvish et al. [36] with a difference in delivery and number of sessions.	2 full-day courses for trainers (by the authors) and 1 full-day (6 h) course from trainers to staff.	Pre measurements for KIDE, CODE, and Controllability Beliefs Scale (N = 607). Post measurements for CODE (n = 480), KIDE (n = 476), Controllability Beliefs Scale (n = 471).Attrition rate: 21%	All three scales were valid tools confirming the previous study (good internal consistency, face and content validity).The same strong impact in knowledge and confidence but greater impact on Controllability Beliefs Scale compared to the initial evaluation. That means consistency and stronger conclusions towards challenging behavior than in the initial evaluation. The one-day workshop can be more feasible than small short sessions of previous study.	20/22= 0.90 (H)
Galvin et al. [34]	USA	“Dementia-friendly Hospital (DFHI)” training	Pre-post-follow-up (120 days) design	N = 540,(n = 143 in 2 pilot-studies and n = 397 in 1 of 2 sessions)/4 hospitals	5 modules divided into 8 sessions.Content: medical overview, approaches to communication and behavior, dementia friendly care, and connecting the caregiver.	A full-day training program/7 h.	Pre-post measurements: medical dementia knowledge test (multiple choice), confidence in providing care, and various practice behavior/attitudes questionnaires.Post measurements: a standard program quality rating, an overall evaluation of the program questionnaire.Follow-up measurements: knowledge, confidence, and attitudes. Low response rate (14%). 1 out of 4 hospitals sustained the knowledge and 3 out of 4 sustained the confidence.	The knowledge, confidence, and practices/attitudes towards hospitalized dementia patients, were sig. improved after the training. This sustainable impact was reported in 3 out of 4 hospitals. The greatest behavioral change was to involve the families to a greater extent and to include a family questionnaire in their assessments. The staff also recognized the need for improved communication skills, and they listed strategies to improve the hospital environment.	18/22= 0.82 (H)
Hobday, Gaugler, and Mittelman [31]	USA	“CARES Dementia -Friendly Hospital” (CDFH) online training	Pre-post design	N = 25 nurse assistants and allied hospital workers/6 metropolitan and 6 rural hospitals	4 prototype online modules.Based on Galvin et al. [34] with new elements of audio-narrated text, graphics, video interviews, video scenarios, interactive text-entry, and case studies, “What would you do?” scenarios asking learners to synthesize knowledge learned in real case scenarios.Content: introduction to dementia-friendly care, communicating with patients, dementia-related behavior, and wandering and falls.	Not mentioned.	Pre-post measurements: Dementia Care Knowledge questionnaire (multiple choice test).Post measurement: open-ended questionnaire about technical issues and reaction to the prototype, closed-ended items questionnaire about satisfaction with the program.	Staff were satisfied with the interactive online training program (96.2% in several items) and they also sig. gained knowledge (80%), however 12% decreased their knowledge. They also reported an improvement in perceived skills, compassion, confidence (100%), and generally adoption of a holistic approach when providing care for PwD.	15/22= 0.68 (M)
Hunter et al. [42]	Canada	An interactive, geriatric education program	Post design	N = 84 nursing staff, 3 medical units in a teaching hospital	Content: understanding cognitive impairment, differentiating dementia, delirium and depression, and how to use evidence-informed strategies to meet responsive behavior.Interactive delivery: role playing, discussions, and intranet resources.	2 h sessions repeated six times in 3-days.	Post measurements: six questions with a 5-point Likert scale (n = 44) and three open-ended questions (n = 17).Online survey 1 month after the training program, also for staff who did not participate in the training program: the context and implementing evidence into practice was examined with the Alberta Context Tool (ACT) (n = 47). The tool covers the following dimensions: leadership, culture, feedback, formal interactions and informal interactions, connections among people, structural and electronic resources, and organizational slack. The questions are rated on a 5-point Likert scale.	Participants rated the training program as very helpful, appropriate to their work, and that would improve their work. Non-pharmacological and pharmacological strategies were also rated as highly useful. The implementation of strategies was rated lower.Identified barriers to implement learned knowledge were: the limited time and staff, the unpredictable nature of the workload, and variations in the way individuals implement strategies.ACT: participants rated culture, leadership, and connections among people as most positive factors for a successful implementation. Overall, implementation of evidence-based strategies was identified as challenging.Additional identified barriers were: little contact to research and quality improvement experts, as well as limited access to evidence-based resources.	14/22= 0.64 (M)
Jack-Waugh et al. [40]	UK (Scotland)	A Dementia Champions Program	Pre-post design	N = 430 health professionals in an NHS hospital completed the program (n = 524 enrolled the program)	Training included pre-reading, and five face-to-face study days. Participants spent a half day in a community setting, and they had to complete three written assignments, while a distance learning was offered. The teaching and learning approaches are described in detail in Banks et al. [39].	5-day program and a half-day spent in a local community setting, over a period of 8 months.	Pre-post measurements: the Approaches to Dementia Questionnaire, the Knowledge of Dementia Scale, and a self-efficacy scale.Not all participants completed pre-and post-questionnaires.	Sig. increase occurred in all three measurements. Training had a positive impact on participants’ attitudes towards, and knowledge of dementia, and increased participants confidence to reach all of the program learning outcomes.	16/22= 0.73 (M)
O ’Brien et al. [41]	UK	VOICE. A dementia communication skills training course	Pre-post design	N = 45 Health Care Professionals (HCP), 2 acute hospitals	Content: based on the experiential learning theory.Optional computer-based learning modules as preparation for the training courses.Day 1: introduction in typical HCP-initiated interactions, small-group simulation workshops, role-play scenarios between participants and simulated patients (actors), including structured feedback for the participants.Reflective diary in between the two training days.Day 2: reflective group workshop using the diary, sessions about person-centered care and how to avoid “elder speak”, video, and a second simulation workshop.	2-day courses 1 month apart/6 courses over a period of 5 months.	Pre-post measurements: Confidence in Dementia Scale, a dementia communication knowledge test, video-recorded simulated exercise to measure changes in communication behavior.Additionally, after the training: five questions asking participants to rate their confidence on a 0-10 scale on awareness and use of communication skills, questions about the learned skills, usage and usefulness, and an evaluation of the training.1 month after the training: again, questions about the learned skills, usage and usefulness.	HCPs sig. increased confidence and knowledge about dementia.The training program met participants’ expectations (95%) and 98% would recommend it to other HCPs.One month later participants reported using the skills learned in clinical practice.Sig. changes occurred when closing interactions, while no sig. changes related to requests were measured. Furthermore, an increase in controlling, bossy, and dominating communication after training was observed.	17/22= 0.77 (M)
Palmer et al. [35]	USA	DFHI program	Pre-post-follow-up (3-month) design	N = 355/5 hospitals	Phase 3 of the DFHI program by Galvin et al. [34].They only added: several active learning strategies, videos, case studies, exercises, involvement of administrators, and they shared learning materials.	A full-day educational program/-.	Pre (n = 355)-post (n = 325)-follow-up (n = 88) measurements: knowledge, confidence, attitudes questionnaires, same as Galvin et al. [34]. Post measurements: evaluation about the satisfaction and the program itself. Open-ended questions about further needs and challenges in working with PwD	The attitudes scores were more positive on the post measurement and remained positive even after the 3-month follow-up compared to the baseline.Confidence in caring for PwD increased sig. for everyone. Overall, participants rated the program as effective on the evaluation form. They received new information, understood the communication and other special needs of their patients and caregivers.	15/22= 0.68 (M)
Pfeifer et al. [32]	USA	A “docudrama” training program	Pre-post design	N = 447 certified nursing assistants from 1 Midwest teaching hospital	Three frameworks were used for planning the content delivery of the dementia training. Content:Part 1 incidence, stages, and symptoms of the Alzheimer’s disease using the YouTube video “Understanding Alzheimer’s Disease in Three Minutes”.Part 2 patient-centered approaches and strategies for anxiety reduction by using patient scenarios depicting PwD (15 min).Part 3 and 4 techniques to manage challenging behaviors using the worksheet “Things I Would Like You to Know About Me” (10 min) and four video vignettes based on the Scenariation Model (15 min).Participants had the opportunities for interaction, discussion, share experiences, and answer questions.	21 sessions over a period of 2 months/1h.	Pre-post measurements: Three Likert-type statements which evaluated participants’ capability to: identify dementia related behaviors, use communication techniques, and manage situations with agitated PwD.Two open-ended questions asked about two techniques participants would use while caring for PwD and how they can implement these techniques.428 CNAs completed the evaluation tool.	Sig. increases occurred in all three Likert-type statements: identification of dementia related behaviors, the use of different communication techniques, and the management of situations with agitated PwD.Techniques that participants would use were: focus on the patient, stay calm, keep the eye contact, and within patients’ view. Participants stated that they would use the worksheet “The Things I Would Like You to Know About Me” to care more individual, to distract agitated patients and to reduce challenging behavior.	15/22= 0.68 (M)
Sampson et al. [29]	UK	A “train-the-trainer” program	Pre-post (3 months) design	N = 2,020 (“Tier 1” level)/8 acute care hospitals	Dementia training curriculum: 24 modules/30’ to 1 h interactive and experiential dementia training modules (classroom teaching, in the ward training or one-to-one coaching in practice). In these 24 modules, the ’Tier 1’ level (Barbara’s Story) was integrated as a basic module for all employees.	Staff training program: minimum 1 h “Tier 1” level training.Train-the-trainer program: 2 full-day workshops.	Pre (n = 1688)-post (n = 456) measurements in 4 levels:1) Individual: The Sense of Competence in Dementia Care staff questionnaire.2) Group/team (ward) level evaluation: Person, Interactions, and Environment qualitative tool (2 h observational data collection after the training by reflection in all 3 domains).3) Organization (hospital trust) level evaluation: a hospital trust level data questionnaire.4) System level evaluation: staff evaluation of the overall content, the quality of the training materials, the presentation of the material by the trainer and the usefulness of the training.	Sig. improvement in staff sense of competence in dementia care after the training.The quality and quantity of person-focused interactions improved, in “building relationships”, being more confident and sensitive in responding to non-verbal cues of PwD.Sig. improvements in provision of caregiver information leaflets on dementia and leaflets for PwD, in gathering personal information using “This Is Me” document and environmental changes. Routine delirium screening and use of delirium care pathways were also increased.Train-the-trainer courses were completed by 52 staff members and 63% of those became active trainers. 79% rated the training as “very” useful, by changing the way they worked with PwD.	19/22= 0.86 (H)
Scerri et al. [30]	Malta	A person- centered program (Appreciative inquiry (AI))	Pre-post-follow-up (4 month) design	N = 68/2 hospital wards in a rehabilitation setting	Workshops about a PCC approach to dementia (a narrative story).Content: intro to AI, overview of the main findings of the discovery phase, a morning with Mary, creating a vision for the future, from dreaming to delivering.	24 workshops/ lasting 1.5 h.	Pre-post-follow-up measurements about: the patients’ quality of care with Dementia Care Mapping, Behavior Category Coding was part of this and used for recording the behavior of each PwD, for observations of personal detractions, and personal enhancers (observations for 6 h every 5’).Post-follow-up measurements with interviews about the relevance and acceptability of the AI workshops.	The overall impact was positive. Staff worked in a more person-centered way and developed action plans to improve the quality of care for PwD.Great value of storytelling in healthcare education and organizational development. Case scenario based on their own positive care experiences helped positively change staff attitudes (becoming more tolerant, patient, empathic, calmer, more assured in caring, better in communication) and provided pragmatic solutions on how to deal with day-to-day challenges in the best manner. Workshops initiate and sustain interdisciplinary collaboration.The number of staff interactions that enhance the need for comfort, doubled after the training.	14/22= 0.64 (M)
Schindel Martin et al. [33]	Canada	Gentle Persuasive Approaches (GPA) program	Controlled pre-post-follow-up (8 weeks) design	N = 745 nurses: Intervention Group (IG) (site A) n = 468,Control Group (CG) (site B wait listed group - standard education) n = 277.7 hospital wards (medicine, surgical oncology, orthopedic surgery, intensive care unit, cardiac care, and the emergency department)	IG: interactive session (learning exercises, case studies, video vignettes, small group work). A manual was given to all participants.Content of GPA: 4 modules: person-centered care principles, brain changes common in dementia and delirium, communication and interpersonal strategies, staff-specific, self-protective skills and team/patient/family debriefing, and reassurance techniques.CG: standard educational support consisted of clinical educators providing advice on management of Need-driven Dementia-compromised Behaviour (NDB) in PwD when requested by staff in specific cases.	Intervention group: 1 day/7.5 h.	Pre-post-follow-up measurements for IG and pre-follow-up measurements for CG.Both groups completed the “Self-Perceived Behavioural Management Self-Efficacy Profile” (SBMSEP) questionnaire (in time points given above).Post measurements for IG: a set of semi-directed questions about the most effective strategy in practice setting after applying GPA.Follow-up measurements for IG: reports of their experience.	IG had a sig. improvement in self-efficacy (SE) comparing to the wait-listed group from baseline to post-training measurement while in the follow-up measurement a small, but sig. decrease occurred.IG also showed a sig. improvement in SE levels in each of the 3 measurements. Greater confidence, as reported by the IG, may decrease staff feelings of vulnerability.After the training, IG stopped using physical restraints and psychotropic medication while the CG reported the above as a first line treatment. Instead of these IG used the best strategies in communication and behavior.Qualitative findings: increased competence and effectiveness in the practice of the IG after the GPA implementation.An overall positive impact of GPA in person-centered care approaches.	20/22= 0.91 (H)
Surr et al. [38]	UK	Person-centered Care Training for Acute Hospitals (PCTAH) program	Pre-post-repeated design (T1-T2-follow-up, 3-4 months)	N = 40/in an NHS hospital (medical, surgical, and orthopedic wards, accident and emergency)	Foundation level: seven 30’ modules. Content: PCC, types and impact of dementia, identification and meeting people’s emotional needs, effective communication, the impact of the physical environment, identifying and meeting physical health needs, and supporting challenging behaviors.Intermediate level: 6 modules. Content: more in-depth knowledge, care needs, delivery, and support of staff group.“A train-the-trainer” day for PCTAH attendances following after the program where they can deliver sessions to peers.	3.5-day training over a 3-4 month period. Foundation half-day, intermediate 3-days/ foundation 3.5 h program, intermediate half-day modules.	Pre measurement: questionnaire about previous dementia training and demographics.3 measurements: immediately prior to training (T1), after completion of foundation level training (T2: 4-6 weeks post-baseline) and following intermediate level training (T3: 3-4 months post-baseline) with: Approaches to Dementia Questionnaire, Staff Experiences of Working with Demented Residents questionnaire, Caring Efficacy Scale.	Sig. positive changes in all three questionnaires in the post measurements of the intermediate training.A sig. positive effect was found only on the ADQ between pre-and post-completion of foundation level training.The foundation program of learning was also sufficient to change staff attitudes and in particular to produce a greater sense of hope in staff members who care PwD. A greater depth of knowledge was observed around dementia PCC, as staff felt more efficient in providing care to PwD. The greater depth of knowledge in the intermediate level of training seems essential in improving staff’s satisfaction over time regarding the provision of care PwD.Sustainability was observed in satisfaction, feeling of efficacy, and staff attitudes over 3-4 months.	19/22= 0.86 (H)

Characteristics of studies

Eight of the studies were of medium and six of high quality. The quality scores ranged from 0.64 to 0.91. Seven out of the 14 articles came from the UK, while four came from the United States, two from Canada, and one from Malta. Eight studies used pre-post measurement designs, five used pre-post-follow-up measurement designs, while one used only post measurements. Most of the studies used a mixed-methods approach (quantitative and qualitative measurements, n = 10), and four of them used only quantitative measurements. While almost all studies (n = 14) used self-report measurements, two of them conducted observations [29, 30] (Supplementary Table 1).

Characteristics of participants

The study samples ranged from a small group of 25 participants [31] to 2,020 participants [29]. Most of the studies reported interdisciplinary groups of attendees from a wide range of professions. The majority of participants came from the nursing profession. One study included only certified nursing assistants [32], while two studies did not report participant characteristics in detail [30, 32]. Six studies reported the participant’s previous education in dementia [33-38].

Characteristics of the training programs

The included studies described eleven different training programs. The duration of training programs varied from one hour [32] to five and half days [39, 40].

There was a great variation of different teaching methods. Almost all studies used in-classroom or face-to-face-training, four combined this traditional training approach with online resources [39-42], and one used an online training only [31]. Interactive approaches were a crucial component of methodology for the majority of trainings. In specific, videos were among the most frequent/dominant approaches (n = 9), followed by group activities (n = 5) and case studies/scenarios (n = 4). Reflections/reflective diaries (n = 2), roleplaying (n = 2), brainstorming (n = 1), and learning exercises (n = 1) were also used. One study offered one to one coaching in practice [29]. Some studies additionally invited either speakers and guests [39, 40], or integrated actors [41]. To continue learning or to read up what has been learned, participants received learning materials [35] or manuals/booklets at the end of the training [33, 36, 37].

The content of trainings varied. The majority of trainings included content regarding the various aspects of knowledge about PwD, such as topics about general knowledge of dementia (n = 10), communication (n = 11), impact of the physical/hospital environment (n = 5) and fundamentals of dementia care (n = 4), dealing with challenging behaviors (n = 6), working with relatives (n = 5), separate modules relevant to PCC (n = 8), topics about screenings and assessments (n = 5), discharge planning (n = 4), and modules about knowing the person (n = 3). However, two of them were based only on one basic module relevant to PwD’s perspective (Barbara’s story) [29] or a narrative story [30].

Evaluation and effectiveness of training programs according to Holton’s model

Learning

Learning outcomes were assessed in all selected studies (n = 14). The most common learning outcome was dementia knowledge (n = 8, 3 validated tools), followed by outcomes relevant to dealing with challenging behavior (n = 3, all validated tools), measured mostly by self-reports. Level of knowledge is argued to be linked to confidence (n = 10, 6 validated tools) or self-efficacy/competence (n = 3, all validated tolls) as successful learners are expected to feel confident and therefore more motivated to transfer the acquired knowledge. Moreover, knowledge has an influence in trainee’s attitude toward dementia (n = 7, 3 validated tools). Learning behavior is also affected by trainee reactions or perception in the learning environment through the complex interaction between motivation to learn and learning. Thus, satisfaction with the program (n = 8, none validated tool), contributed also to learning evaluation, and were measured by self-reported scales.

Most of the studies used pre-post measurements to evaluate learning outcomes, and the majority of post measurements were conducted immediately after the training. However, sometimes participants’ understanding and perceived behavioral changes are not captured effectively as they often overestimate their knowledge before the training, making measurements of change suspect [43]. Thus, in one study both pre and post measurements were assessed after the program to capture the perception of learning more effectively [32], while other studies collected the post data one or three months after the end of the training program [29, 41]. All included studies with pre-post measurement or only post-measurement design demonstrated significant changes in learning outcomes (n = 9). However, one medium quality study, using a not validated tool, showed a significant increase of knowledge (80%) in the post measurement (after 8 days), but this was not the case by all of the participants in the sample. 12% of the participants indicated a decrease in knowledge related to dementia care, and none possible explanation for that decrease was provided by the authors [31]. According to Holton’s model, the factor of participant’s “ability” to learn may have been influenced by the heterogeneity in job or differences in cognitive ability. Besides, some “environmental” influences such as the lack of a control group, the small sample size (N = 25), and no formal psychometric testing of the knowledge scale, may explain that decrease in knowledge.

It has been suggested that only pre-post and follow-up measurements can directly evaluate real changes through reports/questionnaires or specific tools, otherwise reports about changes immediately after the training may be less strong [30, 33]. Concerning our included studies, sustainability of changes through pre-post-follow-up measurements was tested only in five training programs. Two medium and one high quality programs suggest positive changes in staff attitudes, or/and satisfaction with care, or/and confidence and sense of competence were sustainable, over three months [30, 35, 38]. Two of them used validated tools [30, 38] while one study used tools which were not validated [35]. In one study, despite the overall increase of knowledge, attitudes, and confidence, a decline in one item of the attitudes scale was observed in the follow-up measurement [35]. The “environmental” factor of low psychometric properties of the scale (Cronbach’s alpha is 0.39), may have influenced this result. It is possible that participants misunderstood this PCC item. In two high quality studies, a decline in confidence, knowledge [34], and self-efficacy [33] was observed after four and two months respectively. In Galvin et al. [34] that decline happened only in one out of three included hospitals and the used tools were not validated. Among the possible reasons of those declines may be the uncontrolled “environmental” factors or participants “ability”. According to demographic variables of participants responding to the follow-up measurements, heterogeneities in job, different education background, full or part time job, working in different medical departments including work experience, and high attrition rate may have influenced the sustainability of learning changes [33, 34]. A further possible reason for the decline, according to the applied model, is the staff’s “motivation” to learn. In one study, a high percentage of those who answered the follow-up measurements had received previous dementia education. Participants who received such training may have relied on previously learned information and had limited uptake of new knowledge or they had low expectations/interest in a similar training topic. Thus, receiving three or more hours of dementia training in the last two years before the intervention was one of the strongest predictors of this decrease in knowledge [34].

Furthermore, the workplace as an “environmental” factor may have an impact on the person capacity to “transfer design” by using new learned care approaches. Thus, the limited or lack of time and staff [34, 42], the unpredictable nature of the workload in acute care [33, 42], and variations in the way individuals implement strategies, appeared to be barriers to implement knowledge and sustainable results [42].

Performance outcomes

Learning is expected to lead to individual changes in performance. Changes in attitude can motivate participants to transfer knowledge and implement it into clinical practice. Thus, 13 studies measured changes in individual performance or in clinical practice after the training program. Use of communication techniques (n = 7, 2 validated tools), shift to PCC (n = 6, 5 validated tools), and clinical skills (n = 5, 1 validated tool) were among the most common changes in individual performance. Action plans (n = 4), creating champions/mentors (n = 3), use of tools (n = 3), and decrease of chemical or physical restrains (n = 1) were also included in individual performances outcomes. The majority of these were based on self-reported measurements (n = 12), while a few of them relied on direct observations of care practices (n = 2).

Four medium and one high quality studies stressed an overall change in participant’s clinical skills (n = 5) [30, 31, 33, 35]. Open-ended questions or Likert scales were often assessed to explore the use or possible use of skills in clinical practice immediately after the training, after one/three/or more months [31–33, 41]. Only one study used a validated tool of patient-staff interactions [30].

Further changes in practice were brought about action plans or individual initiatives in clinical practice [30, 39]. Setting action plans or goals for newly acquired knowledge, are important factors, according to Holton’s model, to “transfer the design” of the program into the practice. It may also be a linkage between individual performance and organizational outcomes, according to the applied model. In one study, participants evaluated their own clinical area using a workbook [39].

Most of the studies (n = 6), including almost all high-quality studies (n = 5), reported an overall positive impact of person-centered approaches so as to demonstrate changes in attitudes or in behavior and values. Changes shifted to PCC measured either directly, through open-ended questions [33] or indirectly, being part of validated questionnaires [36-38] or through observation by validated tools [29, 30]. In a pre-post design, PCC shifts were minor [36] and in another pre-post-follow-up study, the observed activities were not always based on PCC [30]. A possible explanation of this may be that the participant’s “ability” regarding job, education, and cognitive abilities varies, and maybe the “motivation to learn” also varies due to their different personalities’ characteristics. Moreover, PCC as a “transfer design” approach may need more time to be adapted by trainees or achieved through consistent long-term relationships with patients [44]. Thus, immediately after the training or few months later is actually a short-term period to implement what they have learned.

Staff also reported that they used the most effective strategies regarding appropriate communication with PwD, in four medium and three high quality studies (n = 7). In some studies, communication changes were evaluated through validated observation tools [29, 30] or through blind rating of simulated assessments [41]. In other studies, the outputs were presented through reports, like open-ended questions [32-34] or reflection reports [39]. In one pre-post experiential learning design there were neither changes of making requests, nor changes in voice’s tone after the training but there was an increase of controlling, bossy, and dominating communication [41]. Again, a possible explanation may be the PCC approaches used in the communication techniques. However, PCC is important to “transfer design” in order to motivate participants, time is needed to change individual performances. The period of one month may be insufficient for participants to demonstrate all the new communication skills learnt on the course. Moreover, there was no agreement in rating, demonstrating a poor inter-rater reliability, which may possibly be an “environmental factor” that influences the outcomes.

Champions or mentors are important for the transfer of knowledge. Although five studies included champions or mentors, only two medium and one high quality studies had an outcome in this respect. Champions or mentors have an influence to the “Learning Transfer System” as part of the “transfer climate” of learning in order for staff to change practice. Champions express a willingness to teach others, to strengthen relationships between services [39] to support and lead changes in the workplace, to improve the experience, care treatment and outcomes [40], and this impacts the way they work with PwD and contributes to the overall training sustainability [29].

Changes in clinical practice may be also reflected by an increased use of assessment or practical tools, measured by open-ended questions and reports in two high and one medium quality studies. Tools may facilitate the “Learning Transfer System” through the “transfer design” according to Holton’s model. In some studies, the routine of screening for delirium, dementia, pain assessment [29, 34], and the gathering of patient information through documentation forms [29, 39] were increased, providing changes in clinical practice. In one high quality study, the use of chemical and physical restrains decreased, demonstrating the applicability of training on their daily practice [33].

Organizational outcomes

Outcomes at the organizational level have been suggested as a consequence of changes in individual performance. Outcomes on the organizational level were identified in four studies. Changes in the hospital environment (n = 3) and setting care polices (n = 3) were among the main outcomes.

Environmental changes affected the organizational outcome in one medium and two high quality studies [29, 34, 39]. Thus, the creation of special rooms for PwD, pictures and signs on bathroom doors for patients with orientation difficulties or useful information on how to adjust physical environments (e.g., light, noise) to the needs of PwD, led to organizational changes immediately after training [39]. However, some participants, in the follow-up measurement, reported significant challenges for implementation, including lack of resources and inflexible plans, all “environmental” factors that can affect the organizational outcomes [39]. Some trainees, using self-report questionnaires in a follow-up measurement, experienced positive changes by using better signage and eating/drinking utensils or by using different colored crockery when offering food to PwD, but some discrepancies have arisen when observations conducted in the wards. Specifically, neither the use of colored cups and jugs or specialist cutlery changed, nor the signage or labels around the ward according to observers’ notes [29]. Moreover, due to some “environmental” influences of the complex hospital workplace, the decrease of noise level could not be managed [29].

A number of changes were reported in the frame of culture, care pathways, and policies in three high quality studies. Some hospitals planned to use activity kits about dementia and one hospital established volunteer teams to assist the care of hospitalized PwD or to place patient at risk in “green gowns” in order for staff to respond appropriately in a dementia-friendly behavior [34]. Besides, some hospitals developed care pathways such as the “carer passport” (allows carers visits out of the normal visiting hour), and non-drug approaches [29]. Other reported organization initiatives were the delivery of the program to all clinical managers and to all acute sites of hospitals, and a refresh of training as a mandatory annual skills review for staff [33].

The expected utility or payoff within an organization can also affect training outcomes. Although there were some reports about cost effectiveness of the programs [39] or indirect impact of cost effectiveness due to decreased physical, and/or chemical restraints [29, 33], there were no specific qualitative and quantitative measurements identified.

DISCUSSION

The aim of this narrative synthesis is to discuss the outcomes of dementia training programs according to an effectiveness model. Outcomes were evaluated according to Holton’s three outcome levels (learning, individual performance, and organizational results) taken into consideration its three basic construct domains: ability, motivation, and environment.

Among other reviews in dementia trainings [15, 16, 45–50] only two examined training programs in hospitals according to an evaluation model like Kirkpatrick’s [17, 18]. Although this model is widely applied, it has been criticized for the incomplete and oversimplified understanding of the transfer process from learning into practice [18, 23]. Recognizing the complexity of trainings in environments such as hospitals is difficult. Thus, Holton’s model includes an independent and highly relevant “organizational level” as an additional outcome.

According to our results, four studies fulfilled the three outcome levels of Holton’s model, while only two of them were of high quality [29, 33]. A learning effect could be observed in all studies, as well as changes in individual performance, which were observed in almost all studies. PCC approaches, interactive and varied teaching methods, champions/mentors and generally supporting conditions, action plans and implementation of care policies, all seemed to be among the successful characteristics of training programs for employees, facilitating “Motivation to Learn” and “Learning Transfer System” according to Holton’s model. However, all these factors are effective only if linked with organizational goals and evaluated across the organizational environment. According to our results, there is a lack of adequate cross-level evaluations (n = 4), perhaps due to the fact that long-term changes on the organizational level are time consuming and longitudinal evaluations are needed. Interestingly, none of the studies directly measured patient outcomes as a result of education and training for staff, which is in accordance with recent review results [16]. Only few of the studies measured interactions between staff and patients through observations [29, 30, 41] and some of their outcomes indicated that patients had a passive engagement, conducted no activities or were socially withdrawn after the trainings [29, 30].

Although there was a gain in learning for all included studies, acquiring knowledge or facts about dementia is not enough to capture practice change resulting in improved care experiences. Given that only 20% of what is taught during training is successfully applied in the working environment [51], a great challenge for researchers is to bridge the gap from knowing to doing.

Applicability of good practices in caring for PwD is one of the most important elements of a successful implementation. Providing PCC approaches, support by champions/mentors, use of specific tools (like dementia/delirium screening, documentation forms), strategies or initiatives like action plans, could lead to changes in staff’s individual performance by “environmental” and “ability” influences like “transferring condition and design” of the training into practice. Moreover, organization factors like transferring knowledge or strengthening the collaborative spirit in third parties (e.g., among providers, expert associations, academics), reducing psychotropic medication and physical restraints, creating a special environment for PwD and applying care policies across organizations, all lead to a high training impact according to our results. In an attempt to formulate a pragmatic “community of practice” approach, one study included an evaluation for the organization (hospital trust) level [29] which seems to be linked with the organization goal according to Holton’ s model.

Moreover, the design, content, and delivery of trainings are all features that influence its applicability. The use of interactive and varied teaching methods, such as action learning, experiential learning, supervision and support seemed to be a strength among the effective trainings [52, 53]. Most of our included studies used blended methods and were face-to-face programs including interactive parts, which are more successful than online programs. Videos, group activities, and case studies relevant to general knowledge about dementia, communication, dementia care, and managing challenging behaviors were the most frequent training characteristics. “Motivation” factors like “participants’ readiness” and “fulfillment of content” are also important for successful learning and implementation. Apart from the exploration of educational needs expectations, caregivers experiences, interviews, discussions, and reflective accounts with and from staff members are always effective, taking into account their experience and perception, which is in agreement with other studies [53].

Some of our results correspond with recent reviews, demonstrating specific features of effective dementia training programs, such as trainings relevant to specific roles and duties in the workplace, learning through group activities/interactive approaches, face-to-face learning, and the development and support of in-service “experts” who facilitate training implementation [18, 54].

Although there is no doubt that PCC approaches are effective in changing the clinical practice [33] with more than half of the studies including person-centered components, in some cases [30, 41], the shift was not strong or PCC was not implemented well, because it presupposes to know the person at a deeper level which needs time. Additionally, organization priorities usually are the rapid diagnosis and therapeutic intervention with short lengths of stay. This highlights the idea that PCC cannot be implemented with a one-off “quick fix” [55] but requires a change in the values of both the acute healthcare system and the individual practitioner [44]. This can be achieved if a “whole system approach” [56] links all the persons concerned, including the PwD, their family members, direct care workers, and administrative and managerial staff. A strategy to this direction is the Appreciative Inquiry approach, used by three studies [30, 39, 40] and was related to organizational development for transformative change.

Dementia champions have been identified as one of the best models for providing higher-quality care in general hospitals. Champion’s role is an attempt to “transfer climate” influencing the “environment” factors of training. Champions promote good practices making them sustainable, and ensure that staff is fully supported to provide PCC [57-59]. Five studies in total included champions in their training programs. Moreover, the use of action plans, can support the sustainability of champion’s role as a change agent in further improving the care of PwD in their practice area [40]. Nevertheless, it has been suggested that while champions may be effective in the early stages of introducing a change in local settings, their influence may become diminished over time as the spread of change goes organization-wide and incorporates different disciplines [60]. Thus, the factor of work environment, cannot be easily influenced [40] suggesting the great impact of the organization in sustainable programs. Moreover, visits by the trainer to continue implementation may not be sufficient [61]. Training providers or stakeholders must find other solutions for long-term implementation such as extending training possibilities to clinical wards and departments [33, 62]. This is in agreement with the fact that workplaces of hospital systems are complex and to transfer learning into practice needs not only supporters but many other organizational, environmental, and cultural factors [5].

Thus, sustainability seems to require more than just a good implementation. The supporting environment/conditions in the frame of strong leadership, care culture, initiative approaches, connections among people including teamwork [29, 30], continuous action, and vision are the most valuable factors for successful implementation [42, 63]. Without continued in-service training and hospital wide changes, the changes achieved through trainings will be short-lived [34]. Dementia trainings cannot solve all the problems of failing services. Thus, limited time, lack of staff or the unpredictable nature of the workload in acute care, lack of financial resources, all appeared to be barriers to sustainable changes [29, 33, 34, 42, 64, 65]. Moreover, bureaucracy [65] and hierarchical culture focused on internal stability, models of working with adherence to rules being resistant to flexibility, innovation, and openness are all factors that make the change difficult in organizations [66]. No matter how good a dementia training is, changes in the organizational level, and a transformation of care culture are imperative for training sustainability over time.

Limitations

This review has some limitations. There was a high heterogeneity of study methods. The inclusion of both quantitative and qualitative study designs made the data extraction and conclusions difficult, but quantitative measurements alone are not always sensitive to changes [67] and some changes are only reflected in qualitative measurements [68]. The bias of the small sample size of participants, the lack of RCTs [33], and discrepancies in evaluation methods [29] further made the conclusions difficult. However, there is still a debate on whether RCTs should be considered as the “gold standard” in health education research [69, 70] because the complexity of health care settings may require simpler solutions or other types of randomization [71]. Furthermore, most of the studies used pre-post measurements immediately after the training making the outcomes suspect or less strong. Participants sometimes overestimate their understanding and perceived behavioral changes are not effectively captured [43]. Thus, the additional lack of follow-up measurements makes it difficult to conclude which programs were successfully in changing long-term clinical practice. Lastly, only articles written in English were included.

Conclusion and future research

This narrative synthesis demonstrates the characteristics and effectiveness of dementia training programs in general hospitals according to Holton’s model. Effective trainings focused not only on individual changes, such as learning and performance, but also on changes on the organizational level. Moreover, PCC approaches, a variation of teaching methods including an interactive face-to-face design, supporting conditions like champions and mentors, and action plans are all invaluable characteristics of effective trainings. In addition, trainings where stakeholders adopted initiatives, plans or care policies to continue systematic changes and foster long-term implementation in the organization or across organizations were more effective.

The high heterogeneity of studies regarding the content and delivery of trainings leads us to make further suggestions for the design of future training programs. Guidance or frameworks such as the Medical Research Council (MRC) for developing complex training interventions are necessary for future interventions [72]. Moreover, trainings must use multi-level evaluations not only among hospital wards, but also across the organization in order to measure the sustainability of changes in learning, staff behavior/attitudes, and clinical practice towards caring for PwD. More follow-up designs and longitudinal evaluations of the programs are required in order to conclude which programs were successfully able to make a difference in long-term clinical practice. Sustainable changes in clinical practices means more than just having well-trained and qualified employees. Dementia staff education should be adapted to specific training goals (raising awareness, train experts, and commissioners) with flexibility in the content, length, and type of intervention.

Click here for additional data file.