Evaluation of iLead, a generic implementation leadership intervention: mixed-method preintervention-postintervention design.

OBJECTIVES: The present study aimed to evaluate the iLead intervention and to investigate whether or not transfer of training can be supported by contextualising the intervention (recruiting all managers from one branch of the organisation while focusing on one implementation case, as well as training senior management).
DESIGN: A pre-evaluation-postevaluation design was applied using mixed methods with process and effect surveys and interviews to measure the effects on three levels.
SETTING: Healthcare managers from Stockholm's regional healthcare organisation were invited to the training. PARTICIPANTS: 52 managers participated in the iLead intervention. Group 1 consisted of 21 managers from different organisations and with different implementation cases. Group 2, representing the contextualised group, consisted of 31 managers from the same organisation, working on the same implementation case, where senior management also received training. INTERVENTION: iLead is an intervention where healthcare managers are trained in implementation leadership based on the full-range leadership model. PRIMARY OUTCOME MEASURES: Reactions, knowledge and implementation leadership are measured.
RESULTS: Quantitative and qualitative analyses indicate that iLead was perceived to be of high quality and capable of increasing participants' knowledge. Mixed effects were found regarding changes in behaviours. The contextualisation did not have a boosting effect on behaviour change. Hence, group 2 did not increase its active implementation leadership in comparison with group 1.
CONCLUSIONS: iLead introduces a new approach to how implementation leadership can be trained when knowledge of effective leadership for implementations is combined with findings on the importance of environmental factors for the transfer of training. Even though managers reported general positive effects, transfer was not facilitated through the contextualisation of the intervention. There is a need to further develop approaches to help participants subsequently apply the learnt skills in their work environment. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The present study is based on a rigorous evaluation process of the iLead intervention using mixed methods, where the quantitative evaluation method is followed up by interviews to get a deep understanding of the effects.

Effects of the iLead intervention are measured on different levels based on a thorough theory-based evaluation plan.

Effects of the iLead intervention are, in addition to self-reports, measured through employee ratings, where employees report on their managers’ implementation leadership related to a current implementation.

Multilevel modelling is applied to account for the nestedness of data, which is the case for longitudinal data.

Drop-out was more prominent in one intervention group and the response rate decreased over time.

Background

Implementing the ever-growing number of evidence-based methods into practice is an integral part of daily work in healthcare organisations. For implementation to be successful, leadership has been identified as a central factor.1–10 However, many managers lack formal training in leadership and leading change, as they have often been promoted for their work as frontline providers (cf. McMillen and Raffol11). In addition, existing studies on leadership during implementation have often lacked a theoretical underpinning,4–6 12 which prevents knowledge about how and why leadership is important for successful implementation. Accordingly, there is little research on how to train managers in leadership that facilitates the implementation process (eg, Reichenpfader et al 5). Whereas there is some evidence for the effectiveness of training leaders in implementing evidence-based practice (eg, Aarons et al13) or specific evidence-based methods (eg, preventing diabetic foot ulcers14), little is known about how to train generic implementation leadership (ie, implementation leadership that can be used across various implementation efforts), a skill that is needed when leaders are expected to lead multiple simultaneous implementations as part of their daily work. The present study is an evaluation of the iLead intervention that aims to train managers in these generic implementation leadership skills15 answering to calls highlighting the need to provide and evaluate trainings directed at individuals in implementation roles and therefore focusing on implementation practice.16 17

The iLead intervention

A large amount of leadership research has been based on the full-range leadership model (FRLM)18 19 that describes both desired active leadership behaviours (ie, transformational leadership and contingent reward) and undesired passive behaviours (ie, management by exception and laissez-faire). Active leadership has been related to positive organisational and employee outcomes7 20–24 and fostering change.7 9 25 Even though the FRLM has only been used in a few studies that investigated implementation (eg, Aarons et al 13), systematic reviews have identified leadership activities important for implementation that map well on the active leadership behaviours of the FRLM.5 6 10 26 Based on this work, the FRLM was used in the iLead intervention (for the study protocol, see Richter et al 15).

Fostering transfer through a supporting organisational context

Even though leadership development in general has been found to result in positive effects,7 27 28 it has been acknowledged that these often are limited to proximal outcomes such as reactions and knowledge.27 29 Only 10% of training expenditure has been estimated to translate into behavioural change.30 This highlights the transfer gap, the difficulty in translating knowledge and skills to the work setting.31

Three primary factors influence the transfer of training: trainee characteristics, intervention design and delivery, as well as the post-training work environment.32 Trainee characteristics include personality, the motivation to participate and existing skills, whereas intervention design and delivery define the objectives of the training and the applied pedagogical methods that are used to bring about skills. The post-training work environment refers to the organisational context of participants, such as social support, transfer climate and the opportunity to perform, and follow-up of the new skills. Even though trainee characteristics could be used for the selection of participants, this is often not possible in practice; hence, the intervention design and the post-training work environment are factors that can be proactively tackled by interventionists to leverage transfer (cf. Blume et al 33). Therefore, in designing the iLead intervention, pedagogical tools to facilitating transfer were focused on (ie, how the iLead workshops were brought about) (see upper part of table 1). Moreover, a feature that sets iLead apart from other interventions is its effort to further foster transfer by incorporating a contextualised intervention group to also modify the training work environment. Here senior management (ie, a team of individuals at the highest level of the organisation) and all first-line managers from one organisational branch participated in the training and worked on the same implementation case (see lower part of table 1). Involving senior management is important as they not only allocate resources and have the authority to restructure processes and structures to make the implementation work,34 but also generate and help maintain managers’ and employees’ commitment35 and compliance with an intervention.36

Table 1

Intervention design and post-training work environment factors to facilitate transfer of training and the operationalisation in the iLead intervention

	Facilitators for transfer of training	Elements in the iLead intervention	Intervention group
Intervention design	Behavioural modelling	Role play, planning their actions and practicing between workshops	1 and 2
	Error management	Role play, practicing between workshops and revising the action plan, one workshop on handling resistance and continuous problem solving	1 and 2
	Realistic training environment	Working on an ongoing implementation, practicing between workshops, examples from healthcare in the workshops	1 and 2
Training work environment	Peer and supervisor support	All first-line managers from one organisation, in addition to a senior manager intervention	2
	Transfer climate	Interventions on different levels in the organisation to create a shared mental model about implementation	2
	Opportunity to perform	One common implementation and the support of senior management to create alignment and direction	2
	Follow-up structure	One common implementation and the support of senior management to create alignment and direction	2

The present study

The overall aim of this study was to examine the primary outcomes of iLead, an intervention based on the FRLM,18 19 to train healthcare managers’ generic implementation leadership. Based on Kirkpatrick’s four-level evaluation model,37 four questions are addressed with a mixed-method evaluation:

How do managers perceive iLead?

Does iLead increase managers’ knowledge related to implementation leadership?

Does iLead increase managers’ skills in leading a current implementation?

Furthermore, we investigate under which conditions the iLead intervention has greater impact by studying the contextualisation of the intervention. Thus, two intervention conditions were compared: an individualised group (group 1) and a contextualised group (group 2). We expected no difference between the intervention groups regarding to their reactions and learning because both groups were exposed to the same intervention content and pedagogy. In contrast, we expected that contextualisation (group 2) will facilitate the transfer of training resulting in the fourth question:

4. Does iLead result in a larger change of the behavioural outcome, that is, generic implementation leadership, in group 2?

Method

A mixed-methods pre-evaluation–postevaluation approach was applied with a two-armed, non-randomised intervention design in which managers—based on their organisational belonging—were assigned to one of the two intervention groups.

Setting and participants in the intervention

Healthcare managers from Stockholm’s regional healthcare organisation, which offers primary, psychiatric, rehabilitation and habilitation services, as well as acute hospital care, were invited to participate in an implementation leadership training. More detailed information about the recruitment process can be found in the study protocol.15 In total, 52 managers participated (see table 2). The majority of managers worked as first-line managers (ie, worked closest to and had managerial responsibility over operating staff) having responsibility for staff, budget as well as administration for one unit. The majority of managers had responsibility for one unit, whereas some managers had leadership responsibility for several small units (less than five employees). In intervention group 1, two managers had second-line responsibility.

Table 2

Descriptive statistics of managers in the two intervention groups

	Intervention group 1 (individualised group)	Intervention group 2 (contextualised group)
Number of participants	21	31
Total attrition	11	4
Drop-out
Before the start of the intervention	3	2
After WS1/2	3	1
After WS3	1	1
After WS4	1	–
Women (%)	92.3	96
Age (years)	50 (9.1)	50.8 (8.3)
University education (%)	73.3	81.3
Years being a manager	3.3 (2.09) (0.2–9.0)	4.4 (3.9) (0.5–13.0)
Number of employees	25.15 (12.70) (5–50)	21.83 (7.78) (8–39)

Means and SD are presented for age, years as manager and number of employees. Range (minimum–maximum) is presented for years as manager and total number of employees.

Group 1 consisted of 21 managers from different branches of the healthcare organisation who work with different implementation cases during the intervention. Group 2 consisted of 31 managers from one division of the regional healthcare organisation, where senior management (the chief operating manager together with second-line managers) made participation in the training mandatory. In practice, that meant that first-line managers in group 2 were given time to participate in the intervention as a form of competence development. In reconciliation with senior management, one first-line manager decided to not participate in the training due to an ongoing major reorganisation of his/her unit. With some exceptions, first-line managers worked with the same implementation case, which was determined by senior management.

The two groups of managers had similar demographic characteristics, which are representative of employees in the Swedish healthcare sector38 (see table 2).

Attrition was greater for group 1 (for details and time of drop-out, see table 2). On average, managers from group 1 participated in three out of the four training occasions (SD=0.84), whereas managers from group 2 participated on 3.5 occasions (SD=0.79).

Intervention

The iLead intervention consists of five half-day workshops, which were provided at four occasions. The intervention content was the same for intervention groups 1 and 2. More detailed information about the development and content of iLead can be found in the study protocol15 as well as in online supplementary file 1.

Patient and public involvement

When the iLead intervention was designed, a good fit between the intervention, the healthcare context and participants’ needs was ensured through the involvement of 5 national experts in implementation and leadership training (consults or researchers in the area), 31 first line managers and 9 senior managers who participated in a cocreated programme logic process, generating attitudes, skills and behaviours of successful implementation leaders. The output was used to define intervention goals and activities (for more information, see von Thiele Schwarz et al 39). Patient involvement was not applicable in this study.

Data sources for the evaluation

A sequential exploratory design was used.40 Quantitative surveys were conducted prior and twice after the intervention followed with qualitative interviews to enhance our understanding of the training impact. Shorter process evaluation surveys were also conducted after each individual workshop. To strengthen the research design, the participating managers, as well as their employees, were included in the data collection (for an overview, see figure 1).

Figure 1

Evaluation design for iLead. WS, workshop.

Table 3 shows the response rates for the effect and process evaluations. Response rates decreased over time, which is common in longitudinal studies.41

Table 3

Response rates for managers and employees

	Process evaluation (self-rated manager data)				Effect evaluation(employee data)
	WS1/2	WS3	WS4	WS5	Pretest	Post-test 1	Post-test 2
Group 1	15/18 (83.3%)	10/15 (66.6%)	8/14 (57.1%)	10/10 (100%)	252/477 (52.8%)	160/368 (43.4%)	132/268 (49.2%)
Group 2	26/29 (89.5%)	23/28 (82.1%)	22/27 (81.4%)	22/27 (81.4%)	432/607 (71.1%)	313/562 (55.6%)	292/544 (53.6%)

WS, workshop.

Nine individual semistructured interviews were conducted by a researcher who was not involved in the intervention. The interview guide was developed based on Kirkpatrick’s evaluation model and Baldwin and Ford’s transfer of training model32 37 (for the interview guide, see the online supplementary appendix). Interviews, which lasted for approximately 1 hour, took place at the respondents’ work places and were recorded and transcribed verbatim by an external transcription service.

Measures in the process evaluation and pre-effect–posteffect evaluation surveys

Measurements are described in table 4.

Table 4

Constructs in the process evaluation and preintervention and postintervention surveys

Research question	Construct	Content	Items (n)	Response alternatives	Reference	Time of measurement	Cronbach’s alpha
	Process evaluation (self-rated manager data)
1	Appraisal of the intervention as a whole	Complexity, relevance, novelty and valence involvement	10	10-point continuum for each adjective pair	61	WS5	0.810.680.840.600.29
2	Knowledge about implementation and implementation leadership		6	1 (strongly disagree)–10 (strongly agree)	Especially constructed to match the iLead intervention	WS1/2, WS3, WS4 and WS5	0.900.970.930.97
	Preintervention and postintervention surveys (employee data)
3	Changes in implementation and leadership	Extent of perceived changes in the implementation of the new method as well in the manager’s leadership during the last 6 months	2	1 (big impairment)–5 (no change) to 10 (great improvement)	62 63	T2T3	0.790.74
4	Active implementation leadership	Leadership behaviours in line with FRLM related to the implementation	13	1 (strongly disagree)–5 (strongly agree)	50	T2T3	0.950.96

FRLM, full-range leadership model; T2, postmeasure 1; T3, postmeasure 2; WS, workshop.

Analyses

Multilevel modelling was used to analyse data based on three or more repeated measurements to account for the hierarchical nature of the data.42 Two-level models with the repeated measure at the first level and the individual person at the second level—the individual employee at the first level and the group belongingness at the second level, respectively—were constructed. Nested models were compared by using full maximum likelihood estimation.42 Time was centred on the baseline, respectively WS1/2, whereas the group remained uncentred (0=intervention group 1 and 1=intervention group 2). The multilevel models were run in Mplus V.7.2, whereas all other analyses were conducted in SPSS V.24.

Interviews were analysed using thematic data analysis.43 A semantic approach was used (ie, the explicit meaning of the data was analysed). Patterns in the narrative material that captured something important in relation to the above-outlined evaluation models were selected.32 37 Next, the themes were reviewed by the research team. A few themes were revised or excluded because they overlapped with other themes or were less prevalent (raised by less than three respondents).

Results

Reactions to the intervention

Participants were satisfied (ratings over 7 out of a maximum of 10) with the training’s complexity, relevance, valence, their involvement and the novelty of the content. No group differences were found (see table 5), which is in line with our expectations. The quantitative results were strengthened by interview data (for quotes, see the bottom of table 5). In the analysis, two themes emerged. First, managers emphasised that they were able to work hands-on with their implementation cases, which differed from other trainings they had attended (quote 1). Moreover, they highlighted the usefulness of the action plan guiding their implementation work during the iLead intervention, which made their intuitive knowledge of the implementation process more explicit (quote 2) and helped them clarify the implementation for employees. Second, the use of role-play was perceived to be influential on the managers’ development and understanding (quote 3).

Table 5

Reactions to the intervention and related quotes

	Complexity	Relevance	Valence	Involvement	Novelty
Group 1	9.15	9.35	9.15	8.85	7.85
Group 2	8.52	9.06	8.63	8.56	7.09
Difference	t(30)=0.99	t(30)=0.58	t(30)=0.90	t(30)=0.55	t(30)=1.63

Independent t-test did not reveal significant differences between the two groups.

Improvements in implementation leadership knowledge

Managers reported an increase in knowledge about implementation and how to lead this process over time, and no differences between intervention groups was detected (see table 6), which is in line with our expectations. In the interviews, managers expressed increased knowledge concerning implementation leadership as a generic skill, the structure and the iterative nature of the implementation process (see table 6, quotes 4 and 5) and the possibility to lead an implementation decoupled from knowledge about the specific content of the implementation (quote 6).

Table 6

Multilevel models predicting change in knowledge and related quotes

	Knowledge (ICC=0.44)
	Model 1	Model 2	Model 3
Intercept	6.92*	5.97*	6.37*
Time		0.41*	0.42*
Group			−0.62
σ2 e	1.39*	1.10*	1.10*
σ2 u0	1.12*	1.20*	1.11*
−2*log(lh)	497.62	474.3	471.8
df	3	4	5
Δ−2*log(lh)		23.3*	2.5
Δdf		1	1
Pseudo R1 2			0.21
Pseudo R2 2			0.01

Table entries represent unstandardised parameter estimates. Individual level: n=128–140, group level: n=42. Time is centred at WS1/2; intervention group is coded 0=intervention group 1 and 1=intervention group 2.

*P<.05.

ICC, intraclass correlation coefficient.

Improvements in implementation leadership behavior

When reviewing the last 6 months, employees experienced an improvement in implementation and their manager’s leadership practices. No difference was found between the intervention groups (see table 7, left side). Active implementation leadership at T2 did not differ between groups nor did group 2 have a steeper increase in implementation leadership between T2 and T3 (see table 7, right side).

Table 7

Multilevel estimates for models predicting implementation leadership (employee ratings)

	CP T2 (ICC=0.035)		CP T3 (ICC=0.16)		AIL T2 (ICC=0.26)		AIL T3 (ICC=0.49)
	Model 1	Model 2	Model 1	Model 2	Model 1	Model 2	Model 1	Model 2
Intercept	6.10*	0.6.36*	5.93*	3.36*	3.76*	3.70*	3.64*	0.97*
CPa/AIL T2 b	–		–	0.42*a	–		–	0.73C*b
Group		−0.42		0.06		0.10		−0.12
σ2 e	2.63*	2.64*	2.65*	2.33	0.65*	0.65.23*	0.62*	0.35*
σ2 u0	0.09	0.04	0.51*	0.39	0.24*		0.60*	0.18*
−2*log(lh)	660.67	658.3	728.4	645.6	436.74	426.5	486.46	336.0
df	3	4	3	5	3	4	3	5
Δ−2*log(lh)		2.4		82.8*		0.2		150.5*
Δdf		1		2		1		2
Pseudo R1 2		−0.003		0.12		0.002		0.43
Pseudo R2 2		0.55		0.23		0.004		0.69

Unstandardised coefficients.

*P<0.05.

AIL, active implementation leadership; CP, change in leadership procedures; ICC, intraclass correlation coefficient.

To sum up, employees experienced a positive change in both the implementation process and their manager’s leadership practices, but no difference between groups could be found regarding an increase in active implementation leadership. Interviews provided a deeper insight in what participants perceived as particularly valuable and provide examples on altered ways of leading implementations. However, the boosting effect of the contextualisation, which should facilitate a transfer of training for group 2, was absent. It became clear that varying attitudes toward the common implementation case (table 8, quote 7), the timing of the iLead intervention in relation to a concurrent major organisational change (table 8, quote 8) and a perceived lack of support from senior management and peers (table 8, quote 9) may have mitigated the impact that the contextualisation had on the outcomes.

Table 8

Quotes related to the contextualisation

Interview quotes
Quote 7	ID11: It was in the midst of this reorganization when managers were dealing with crying employees who were going to be transferred and so on. And then one was asked to focus on implementing the new [common] program. There must have been a lot of other cases that we could have implemented that would have been more appropriate to implement at this moment in time…
Quote 8	Id 7: I think that it was unfortunate that we were in the midst of the reorganization while the training program was simultaneously running. I think that it was very interesting to participate in the training and that it is very important for all of us to do this. However, I think that employees may have been in a slightly different mindset as a result of the reorganization, and were more concerned about how things would change in their daily job (eg, who they were going to collaborate with later that year, what unit they would belong to, etc.). Change happens, but on this scale – once in a decade, maybe, so it is not very often.
Quote 9	ID14: I feel that they [the senior management] have not been able to fully handle the situation [with supporting line managers as part of the training), which I believe—yet again—is the result of the timing. If it was not for the reorganization that was occurring in the midst of everything, then I think the senior management would have focused more on supporting us.

Discussion

This study focuses on the outcomes of iLead, an intervention training on healthcare managers’ generic implementation leadership. Results showed that managers perceived the content, as well as the pedagogy of the intervention to be relevant and of high quality. Moreover, they perceived that their knowledge about implementation leadership had increased throughout the intervention. However, behavioural effects were mixed. The employee transition ratings on the progress regarding the implementation and the leading of it indicated an improvement. This was mirrored in the interviews in narratives about altered ways of thinking about implementation and how to lead it. Despite our attempt to facilitate transfer by contextualising iLead, by offering interventions to both first-line and senior managers (chief operating manager and second-line managers) from one organisation and working on the same implementation case, no difference between the two intervention groups in implementation leadership or its increase over time could be found.

According to previous literature, transfer of training may be facilitated when there is a common understanding about implementation, alignment across hierarchical levels and social support among colleagues and from senior managers.34–36 44 Based on this, the interviews with managers provided insight into why the contextualisation of iLead might not have resulted in the anticipated boosting effect.

First, first-line managers’ attitudes regarding the common implementation case (decided by the senior management) were clearly mixed. Some embraced it, others were opposed to it, and that had been so for a long time: the implementation had been ongoing for some years with several setbacks. The fact that senior management made it mandatory to focus on this specific implementation case in the iLead intervention caused frustration. Thus, it seems likely that the readiness for the implementation case differed between the intervention groups. Managers in group 1, who were free to choose their implementation case, possibly experienced higher readiness for their implementation case than managers from group 2, who were expected to work with a particular implementation. This may have decreased the managers’ ability to make the most out of the exercises in the iLead intervention, which resulted in reoccurring discussions about the feasibility of the implementation case in the workshops for group 2. This presents the challenge of separating attitudes and experiences of the leadership training and its contextualisation from the attitudes and experiences of the implementation case. Nevertheless, it also points toward the importance of the fit between the perceived needs of the organisation and the evidence-based practice that is implemented (eg, Aarons et al,2 Fixsen et al 45 and Damschroder et al 46). Moreover, there also needs to be a shared perception of managers on different levels regarding the importance of implementing the evidence-based practice under question. Hence, even when the focus of an intervention is on implementation leadership such as iLead, rather than a specific evidence-based practice (eg, Aarons et al 13 and Gifford et al 14), it may still be necessary to offer support to the organisations and participating managers to ensure the feasibility of the implementation case before accepting participants for this kind of intervention.

Second, major organisational change concurrently occurred with the intervention. In group 2, managers described conflicting focus, both for themselves and for employees, due to major organisational change (merging or closure of units, change in first-line managers as well as change of employees within units). However, managers from group 1 also experienced organisational changes, yet they reacted differently. They mentioned the changes but did not pay as much attention to them nor did they describe them as a major hindrance in participating in the intervention and conducting the implementation. Yet, in group 1, attrition was higher, which might have been a consequence of a conflicting focus.

The impact of managers’ attitudes towards the common implementation and the timing of the iLead intervention and organisational change in group 2 may be elucidated by research on mental models.34 Mental models concern underlying psychological beliefs, which affect participants’ reactions and behaviours. Even though the quantitative evaluation of the iLead intervention revealed positive reactions, the interviews indicated mixed—in some cases, critical—beliefs regarding the implementation case and the timing of the organisational change. For an intervention and its implementation to be effective, the participants should believe that there is a problem that the intervention is suitable to address, which motivates them to participate in the intervention activities.47 Whereas no difference in intrinsic motivation to participate in iLead was found between the two intervention groups, extrinsic motivation was higher in group 2 (analysis can be obtained from the authors). This is possibly a consequence of senior management making both the training and the implementation case mandatory for the first-line managers.

Third, when whole organisations undergo an intervention, the group dynamics and existing organisational culture is brought into the intervention. Consequently, sceptical or conflicting mental models about the intervention or the implementation case can receive more attention and need to be addressed. For example, for group 2, workshop leaders had to spend more time on managing issues that originated from the organisational context (eg, the sceptical attitude towards the common implementation case). In addition, in the contextualised group, senior management took part in an intervention of their own, aiming to support first-line managers. However, this support was only partly perceived by first-line managers. Even though senior management themselves developed through this intervention (for more information, see Hasson et al 48), it did not result in a sufficient alignment between organisational levels. The timing of the senior management intervention in relation to first-line managers’ intervention may have been suboptimal. Important discussions that would have had the potential to facilitate the implementation process, if issued earlier, emerged among senior management during their intervention. More preparation time to define the implementation case and senior management’s role in supporting first-line managers in their implementation process might have been beneficial and should be adjusted in future multilevel interventions.48

In sum, although contextualisation may theoretically have several benefits, such as providing social support, direction and alignment of the implementation to boost transfer, this study highlights several impeding factors that may have outbalanced these potentially beneficial effects. A more thorough organisational analysis prior to the intervention to identify barriers for the intervention and the implementation case is recommended. Hence, the general implementation and group climate, the history with the implementation case and the structure and opportunities to perform in line with the implementation should be investigated, along with participants’ capacity and readiness for this implementation. Based on this analysis, preparatory workshops for the actual intervention should be provided. Even though the content of the parallel first-line manager and senior management interventions should be retained, more elements fostering the dialogue between the different managerial levels should be included.48

Strengths and limitations

This study has several strengths that should be highlighted. First, iLead is a generic intervention that is theory driven and has been developed involving relevant stakeholders (eg, line and senior management). It is based on the FRLM, which mirrors relevant leadership behaviours that were also previously identified in implementation research.5–10 26 As it has been highlighted that general active leadership is not sufficient to reach specific results (eg, a successful implementation),20 49 iLead focuses on active implementation leadership. Second, to evaluate iLead, a sophisticated longitudinal multisource design has been applied using both quantitative and qualitative data, which made it possible to capture the intervention context and ongoing process to understand the effects of iLead. Third, evaluation was facilitated by the iLead scale,50 capturing implementation leadership of the specific implementation case. The scale was specifically developed for this purpose, as it has been highlighted that the used evaluation criterion needs to be aligned with the intervention content.51 The iLead scale can also be a useful tool in practice to provide managers with feedback regarding their implementation leadership.

The current study also has some shortcomings that must be acknowledged. First, the recruitment processes for the intervention and assignment to the two intervention groups might have introduced a systematic bias. Randomisation of managers was not possible, and we cannot exclude that intervention groups differed systematically. Moreover, drop-out varied between the groups, which might have affected the generalisability of results, particularly for group 1. Furthermore, the lack of randomisation makes it impossible to separate effects of time from effects of the intervention; hence, an evaluation framework and multiple data sources were used to mitigate the risk of erroneous conclusions. Second, some outcomes (reactions and learning) relied on self-reports, which can be biassed through common method bias.52 53 Third, to investigate behaviour change as an effect of the iLead intervention, transition rating questions where used. Transition ratings are ascribed to overestimate effect sizes54 as well as being influenced by the present state bias.55 56 These biases could, however, not be found in a recent study comparing different ways of assessing change.57 A traditional pre-evaluation–postevaluation measurement was not feasible for several reasons. First, the iLead scale50 could only be administered at the two follow-up measurements because managers were still undecided regarding their implementation case when the baseline measurement was conducted preintervention. Moreover, a comparison of overall mean changes preintervention–postintervention might not to be feasible in iLead, where each manager’s work took its starting point in her/his specific stage of her/his specific implementation case to assure the perceived usability of the intervention. Even in the contextualised intervention group, where the same implementation case was a focus, local conditions varied and led to different time plans. Hence, timely alignment of measurement with managers’ individual change processes58 is challenging with individualised interventions when the implementation process does not follow the time frame of the intervention, that is, when managers differ in their implementation progress and, therefore, vary in their ability to show implementation leadership. In addition, managers set individual leadership goals based on their strengths, weaknesses and work group needs. While probably beneficial for the individual participant, tailoring the intervention to the participants created a large variation of goals and pace in the implementation. Fourth, healthcare organisations are fast-moving entities with high turnover,59 resulting in changes in the work unit composition across measurement times manifesting in different sample sizes for the analyses. Only a smaller group could be followed up across all three time points. In addition, whereas iLead focused on active implementation leadership, recent research shows that destructive leadership has detrimental effects20 60; hence, including how to decrease passive leadership in leadership trainings is another avenue for future research.

Conclusions

This study shows that a generic implementation leadership training that is based on the FRLM may lead to positive outcomes in participating managers’ reactions and implementation knowledge. However, it also shows how hard it is to achieve transfer from training to behavioural change. Efforts to support transfer through contextualisation was not successful. Potential explanations are offered by interview data, which suggest a countereffect of impeding organisational factors. Hence, contextualisation may not be sufficient to counterbalance such factors, calling for a thorough organisational analysis to identify hindering factors for the implementation beforehand.