A comparison of face-to-face and online training in improving managers' confidence to support the mental health of workers.

BACKGROUND: In recognition of the important role managers play in the well-being of the staff they supervise, many workplaces are implementing specialised training for leaders to help them better understand and support the mental health needs of their staff. This training can be delivered through face-to-face or online training sessions. Evaluation of such programs have found positive results for each format when compared to a control group, but to date, face-to-face and online manager mental health training have not been compared with one another. AIMS: This study brings together results from two trials evaluating the same program content, each employing a different mode of content delivery. Both types of training aimed to change managers' confidence to better support the mental health needs of the staff they supervise.
METHODS: Utilising data derived from two previously conducted trials, mean change in manager confidence from baseline at both post-intervention and follow-up were examined for each method of content delivery. An identical way of measuring confidence was used in each study.
RESULTS: Managers' confidence improved from baseline with both methods of training. A greater change was observed with face-to-face training than for online, although both methods had sustained improvement over time. Analyses indicate that at follow-up, improvements in confidence were significant for both face-to-face (t 18  = 5.99; P < .001) and online training (t 39  = 3.85; P < .001). Analyses focused on managers who fully completed either type of training indicated very similar impacts for face-to-face and online training.
CONCLUSIONS: Both face-to-face and online delivery of manager mental health training can significantly improve managers' confidence in supporting the mental health needs of their staff. This change is sustained over various follow-up periods. However, lower retention rates common in online training reduce the relative effect of this method of delivery.

Introduction

Within the working population, rates of functional impairment due to treatable and often preventable mental illnesses such as anxiety and mood disorders have been steadily increasing (Harvey et al., 2017a). This increase corresponds with a shift in the leading cause of long-term sickness absence and work incapacity from musculoskeletal injuries to mental illness throughout most high-income countries (Harvey et al., 2009; McLellan et al., 2001; Tsutsumi, 2011; Petrie et al., 2017). Being absent from work can negatively impact on self-esteem, increase social isolation, and lead to financial hardship (Modini et al., 2016; Ormel et al., 1994) which can further delay the recovery process for individuals experiencing mental illness. Research has confirmed the importance of employment in helping individuals maintain positive mental health or facilitate recovery from an illness (Modini et al., 2016). However, the relationship between work and mental health can be complex, with evidence suggesting that in some circumstances an individual's work environment can increase their risk of developing mental health problems and act as a barrier for them returning to or remaining at work after difficulties arise (Henderson et al., 2011). High levels of job demands and strain, low levels of job control or decision latitude, and poor collegial relationships and support, have been found to be associated with poor productivity for employees and increase the risk of mental health problems (LaMontagne et al., 2014; Harvey et al., 2018; Harvey et al., 2017b). When not adequately addressed, work-related mental health risk factors can result in substantial costs to the individual, their workplace, and to the economy (Harvey et al., 2017b; Harvey and Henderson, 2009; Nieuwenhuijsen et al., 2004; Nishiuchi et al., 2007; CIPD, 2010; Harvey et al., 2014; HSE, 2009). Therefore, there is an increasing consensus that organisations need to undertake actions at the individual employee level, within the team and across the organisation, to ensure the impact of these work-based mental ill-health risk factors are reduced as much as possible (Harvey et al., 2014).

Managers are in a key position to ensure changes can be implemented to modify these risk factors in order to create a mentally healthy workplace for the staff they supervise. There is increasing evidence that manager behaviour and attitudes can predict mental health outcomes amongst employees (Petrie et al., 2018). In addition, the recovery process and occupational outcomes for an employee experiencing mental ill health, regardless of the underlying cause of the illness, can be influenced by the way their manager responds (Nieuwenhuijsen et al., 2004) with evidence suggesting a positive association between early and regular manager contact and employees' recovery and return to work (Nieuwenhuijsen et al., 2004; CIPD, 2010). Despite this potential for managers to have a positive impact, managers continue to report feeling uncertain about how to best support employees' mental health needs (Henderson et al., 2011; Harvey et al., 2014). This is a major barrier, as managers' confidence to discuss mental health issues with employees has been found to be the key predictor to inititating and maintaining conversations with staff (Bryan et al., 2018).

Several studies have been conducted evaluating various training programs designed to upskill managers to better support the mental health needs of the staff they supervise. To date, the majority of evaluations have involved face-to-face training. Certain studies have shown such training delivered face-to-face to be effective in improving managers' confidence and the implementation of appropriate responses to support employee mental health (Milligan-Saville et al., 2017; Dimoff et al., 2016; Moffitt et al., 2014), as well as a reduction of absenteeism amongst staff (Milligan-Saville et al., 2017). However, an ongoing limitation with this mode of training is the potential for it to be expensive and logistically difficult to deliver with little opportunity for scalaibilty. With time-pressures and financial factors often influencing the capacity of a workplace to deliver training to staff, face-to-face training may not be a viable option for many organisations, particularly small-to-medium enterprises. This has led to the development and evaluation of similar training for managers delivered via an online format (Gayed et al., 2018a; Kawakami et al., 2005; Kawakami et al., 2006; Gayed et al., 2019). Although value has been found in some online training (Gayed et al., 2018a; Gayed et al., 2019) face-to-face and online training managers in mental health matters have not been directly compared to one another.

The aim of this study is to bring results from a trial evaluating face-to-face training together with results from a separate study evaluating online delivery of the same educational content (Milligan-Saville et al., 2017; Gayed et al., 2019) to explore any apparent differences in their effectiveness.

Methods

Results from two separately conducted studies that delivered the same program content to managers via different learning methods were compared to establish whether a difference in change in confidence to support employee's mental health could be determined between face-to-face and online learning. One study evaluated the impact of a four-hour face-to-face training session called RESPECT (Milligan-Saville et al., 2017) while the second study looked at the effectiveness of the same content delivered via an online format through a program called HeadCoach (Gayed et al., 2019). Both studies used an identical scale to measure managers' confidence in supporting the mental health needs of staff they supervise and initiating conversations about mental health matters.

Study 1 (face-to-face training)

Study 1 was conducted with duty commanders from a large Australian state fire and rescue service to evaluate the effectiveness of RESPECT Manager Training, a 4-hour face-to-face educational program designed to upskill managers' mental health literacy and improve their communication with staff about mental health issues. Details of the program and full trial have been published by Milligan-Saville et al. (Milligan-Saville et al., 2017).

The level of duty commanders was selected as appropriate managers for this study as this rank oversaw the operation of several fire stations and were responsible for managing sickness absence amongst staff they supervise. The 4-hour program was delivered in a single session by a clinical psychologist or psychiatrist to seven separate groups of 7–15 managers. The first part of the training focused on helping managers identify common mental disorders such as depression, anxiety, post-traumatic stress and alcohol mis-use amongst their staff. The second part aimed to upskill managers in appropriate response and positive communication strategies with the opportunity to practice new skills through interactive group discussions and role plays. At the commencement of the session, managers completed a baseline questionnaire. Additional assessments were conducted immediately after the 4-hour session was completed and 6-months following their training. In the 8-weeks following the 4-hour training session, participants in the intervention group received a single telephone call from an employee assistance programme representative specialising in assisting managers to address any questions or concerns.

Study 2 (online training)

Study 2 was conducted with two state ambulance services and an Australia-wide building and construction hire company to evaluate the potential effectiveness of an online manager mental health training program called HeadCoach (Gayed et al., 2019). Similar to RESPECT, HeadCoach aimed to improve managers' communication with staff about mental health issues. In addition, this online training program also included content discussing preventative strategies valuable in creating a mentally healthy workplace. This online program consisted of a series of 10-minute modules aimed to be completed at the individual participants' own pace over a 6-week period. Modules consisted of learning material presented via text, videos, case vignettes and interactive exercises.

Overlapping content in both the RESPECT and HeadCoach programs included introducing managers to common mental illnesses they might come across in the workplace and the signs and symptoms that might help them identify these conditions amongst staff. Both programs also aimed to upskill managers in appropriate strategies to better communicate with employees experiencing mental illness. Table 1 details the content and learning objectives of the two programs. Further details of the content development and evaluation of HeadCoach have been published previously (Gayed et al., 2018a; Gayed et al., 2019; Gayed et al., 2018b).

Table 1

Program characteristics and participant rates of the two interventions.

	RESPECT (face-to-face)	HeadCoach (online)
Content/Iearning objectives	The aims of the 4-hour face-to-face session is to increase managers' knowledge and skills to:- Recognise the key features of common mental health issues, and the impact of these disorders on the individual and the organisation; - Implement strategies in approaching and supporting firefighters/officers who develop depression or other mental illness in the workplace; - Better support firefighters/officers experiencing mental ill-health and implement successful return to work strategies.	The aims of this online program is to increase managers' knowledge and skills to:- Better recognise and understand mental health in the workplace - Initiate conversations with and facilitate help-seeking for staff with mental health concerns - Better support staff experiencing mental ill-health and implement successful return to work strategies. - Help keep staff as healthy and productive as possible.
Duration of intervention period/mode of delivery	A single 4-hour face-to-face session in groups of 7–15 managers	18 × 10-minute modules completed online at the individual's own pace over a 6-week period
Participants (Consent and baseline received)	N = 45- Fire and Rescue = 45	N = 86- Ambulance = 54 - Construction Hire = 32

Within the two ambulance services included in the evaluation of HeadCoach, managers were defined as those responsible for the operational, clinical and financial activities of the individual stations on a day to day basis, in addition to overseeing front line paramedic staff. Managers at the construction hire company were those in supervisory roles responsible for branches of employees. Managers were recruited to participate in the online trial via email. They were informed of the study by their organisation prior to the researchers contacting them. Managers who opted into the study completed an online consent form and created an account. It was within this account the baseline questionnaire was available and the HeadCoach program content was accessed. The post questionnaire was available immediately after all training modules had been completed. For managers who did not complete all modules, notification of the post questionnaire was emailed to them at 6-weeks following their baseline questionnaire with a hyperlink directing participants to their account login page. All managers were invited by email to complete the follow-up questionnaire at 4-months following their baseline assessment.

Outcome

An identical scale measuring managers' self-reported confidence in supporting employee mental health issues was used in the evaluation of both RESPECT face-to-face manager training (Study 1) and HeadCoach online manager training (Study 2). This was a modified version of a previously published supervisor scale (Nieuwenhuijsen et al., 2004) evaluating confidence, which has been found to be a key predictor of manager behaviour (Bryan et al., 2018). In this scale, managers are asked to indicate their level of confidence in dealing with the five different workplace scenarios. Ratings are provided on a five-point Likert scale ranging from not at all to extremely confident, resulting in an overall confidence score ranging from 5 to 25. Scenarios included: “Initiating contact with staff on sickness absence leave that you believe might be due to mental illness” and “Identifying barriers preventing a staff member returning to work and coming up with solutions to those barriers” (Gayed et al., 2018a).

Statistical measures

All analysis was conducted using SPSS version 23. Given the data from the two study groups were collected through separate trials, with different samples and varying follow up timing, direct statistical comparisons between the two interventions were not conducted as part of this evaluation. Rather, mean change in manager confidence from baseline at both post-intervention and follow-up were examined for each study group, with the overall patterns of response compared. The differences in the mean percentage confidence scores between baseline and follow-up for each intervention were assessed using paired samples t-tests.

In addition to the main analysis, which included all participants regardless of the amount of the training they completed, an additional analysis was completed which included only participants who completed all training requirements of either program.

Results

Baseline assessment data was available for 46 managers in the face-to-face training (Study 1) and 86 managers in the online intervention (Study 2). All managers who attended the face-to-face training completed the single training session. For managers in the online intervention group, 30 (37.5%) completed the entire training program.

For the face-to-face training in Study 1, post-intervention data was available for 44 (97.8%) managers and follow-up data for 19 (43%) managers. Of the group of managers who began the online training (Study 2), post-intervention and follow-up data was available for 44 (52.4%) and 40 (47.6%) respectively.

The mean change from baseline in managers' confidence for the two studies at each occasion of measurement is displayed in Fig. 1. There was a larger change from baseline with face-to-face training than for online however, as shown in the figure, improvements in confidence occurred for managers in both face-to-face and online training at post-intervention, with this improved confidence remaining at each of their respective follow-up assessments. Further analyses indicate that at follow-up, these improvements in confidence from baseline are significant for both face-to-face (t = 5.99; P < .001) and online training (t = 3.85; P < .001).

Fig. 1

Demonstration of the change in confidence from baseline with ±standard error at each assessment point for both online and face-to-face training.

Within Study 2, of the 30 managers who completed the online training, all provided post-intervention data and 27 (90%) provided follow-up data. At baseline, this group had a mean confidence score of 17.8 (SD = 3.10) and a range of observed values between 8 and 24. For completers in the face-to-face training in Study 1, post-intervention and follow up data was available for 44 (97.8%) and 19 (43%) managers respectively. At baseline, the mean score for confidence was 15.70 (SD = 3.76) with values ranging between 5 and 21. The mean change in manager confidence from baseline at each occasion of measurement in each group for managers who completed their respective training is displayed in Fig. 2. Once again, significant improvements in confidence were found for both face-to-face (t = 5.99; P < .001) and online training (t = 4.20; P < .001) between baseline and the completion of training. However, amongst completers, the improvements seen with face-to-face and online training appear very similar. As with the main analysis, both types of training appeared to result in sustained improvements. These improvements correspond with effect sizes (standardised mean differences) of 0.64 and 0.56 for face-to-face and online training respectively.

Fig. 2

Change in confidence from baseline with ±standard error at each assessment point amongst managers who completed all training components of either the face-to-face or online intervention.

Further analyses were conducted to evaluate whether the impact of online training differed between managers based on industry type. While the change observed amongst those in the construction industry was greater than that seen in emergency service workers (increase of 4.2 vs 1.4, p = .02), there was still a significant difference in the mean change in confidence at 4-month follow-up for both construction workers (t5 = 4.776; P = .005) and emergency service workers (t20 = 2.828; P < .01).

Discussion

Our findings suggest both face-to-face and online modes of delivery provide effective means of improving managers' confidence to initiate conversations and address mental health issues within their teams. Both modes of delivery appear able to create sustained changes in confidence levels. However, a key challenge of online modes of delivery appears to be retention rates and program adherence. Although not dissimilar to completion rates in other studies of internet-based interventions (Christensen et al., 2009; Fear et al., 2010; Cook et al., 2000), adherence rates in the online training program used in this study were low, which reduced the overall effect compared to face-to-face. Thus, at present, it appears that online mental health training can result in similar increases in managers' confidence to that observed with face-to-face training, but only when managers complete the online training program. Additional strategies involving more engaging means of improving retention in online training need to be determined to maximise the impact of new online training programs.

A key methodological strength of this study was the comparison of two training programs delivering the same content through different methods of learning with the outcome of manager confidence evaluated using an identical scale. However, this examination utilised data from two separately conducted trials, each containing different recruitment strategies, samples, length of program delivery, follow-up timing, and response rate at follow-up. Although the results provide valuable preliminary data supporting the comparable benefits of both face-to-face and online training, these differences between the two trials make direct comparison problematic. A further strength of the study was the inclusion of similar high-risk, male-dominated industries in both trials. However, although both studies included emergency service workers, the sample evaluating online training also included high-risk workers from the building and construction industry. Additional analyses suggested that the benefits of online training may differ according to industry groups, with greater gains of confidence found amongst construction workers. Further investigations are required with a larger sample size to ascertain whether there are a range of industries that may be more responsiveness to an online training program. An additional point of consideration is the risk for bias due to the different way in which each group was recruited; the face-to-face training group was recruited at the training sessions, while the online training group was recruited via email. This creates the potential for different rates of participant commitment at the commencement of the training program, which may impact on adherence and bias the observed effect. When analysing the intention to treat data (Fig. 1) which included all managers who commenced the program regardless of their adherence to the program, the mean change from baseline to follow-up was greater for the face-to-face group compared to the online group. This difference was reduced in the secondary analysis (Fig. 2) which included only program completers. However, this potentially creates a strong selection bias that should be considered when interpreting the results. Furthermore, during the follow-up period, the face-to-face training groups were provided with support calls by an employee assistance programme representative which may have influenced changes observed. Nevertheless, the mean change in confidence from baseline to post-intervention in the face-to-face and online intervention groups both corresponded to a medium effect size. It is uncertain what value this corresponds to within the workplace setting, and future studies would benefit from analysing this further to determine the impact of this effect size on actual behaviour change as reported by both the manager and their employees. The absence of a control group in this analysis may have also created further bias, by increasing the risk of a Hawthorne Effect. However, as previous RCTs have shown that both face-to-face and online manager training generate a significant improvement in confidence compared to control groups (Milligan-Saville et al., 2017; Gayed et al., 2019), a Hawthorne Effect is unlikely to explain all the observed effect in the improvement in confidence. In addition, although there are three measurements of occasion for both interventions, it is important to note the potential impact of the different time points data collection occurred in each group. For face-to-face training, post-intervention data was collected on the same day as the baseline with the final follow-up occurring 6-months post training, whereas for the online training, post-intervention data was collected when managers completed the program, which could have occurred at any point during the 6-weeks following baseline with final follow-up at 4-months. There was also the opportunity for managers who did not complete all online program requirements to still complete the post-intervention and follow-up questionnaires. As with any trial where loss to follow-up occurs, the program adherence and varied response rates in these studies, particularly for the online group, may have led to biases in the results. Furthermore, the follow-up response rate for the managers who completed the face-to-face was markedly lower than the managers who completed the online training. Due to logistical reasons, data-collection for the face-to-face group switched from paper-and-pencil for baseline and post intervention on the same day as the training session, to electronically via email at follow-up. This change in data collection method may have affected the response rate and thus impacted the results with managers who responded to the email possibly more attentive or invested in the study. Similarly, the possibility exists that managers who felt more confident with the program content completed the online intervention in full and may have been more likely to answer follow up questionnaires, thus impacting on the results for that sample. Further exploration of the motivations of managers to participate in and complete mental health training could be valuable to examine in future investigations. Finally, the generalisability of the results from the included sample may be limited. The construction company and first responder agencies involved in the investigations are high risk industries with specific work conditions and characteristics that may not be representative of the wider population. However, a one-size-fits-all approach is not relevant to manager training and thus a key feature incorporated into the design and development of these programs were tailoring of resources to each sample group included in the evaluation. This may have the potential to enhance the effectiveness of the core program content across industries when additional tailored content specific to that organisation is integrated.

Although both face-to-face and online training delivering the same content appear to be effective means of training managers in mental health matters, it is uncertain whether they both offer the same type and degree of benefit to managers, and due to the absence of employee level data, it is uncertain whether these benefits flow onto their staff. In traditional statistical analyses, tests are conducted to examine the difference between two interventions. This is the type of analysis shown in this paper demonstrating that both modes of training resulted in improvements from baseline. In studies of equivalence, statistical measures are applied to determine whether the efficacies of two interventions are similar enough that neither one can be considered superior or inferior (Jones et al., 1996; Walker and Nowacki, 2011; Le Henanff et al., 2006). It is hoped that in the future, larger controlled trials can be established with analyses of equivalence to gain more insight into the relative benefits of online and face-to-face training, in terms of both manager confidence and behaviour. Given the opportunity to role-play and practise skills learned during face-to-face learning, it is possible that a greater difference between methods of delivery may be evident for more behavioural outcomes. Analyses of employee level data may also help determine if equivalent effects flow onto the staff the managers supervise. Outcomes evaluating perceptions of supervisor support and psychosocial climate, as well as employee rates of sickness absence (Milligan-Saville et al., 2017) would provide valuable information about the effect of these methods of manager training for their staff they supervise.

Although the equivalence of face-to-face and online manager mental health training cannot yet be established with certainty, the available evidence suggests that both methods of training managers can be recommended as important and effective ways which help managers feel more confident to support the mental health needs of the staff they supervise. Until more data on equivalence is available, which also incorporates objective data from the managers' employees to ascertain the flow on effects of manager training to their staff, workplaces should be reassured that both methods of training are able to improve managers' confidence in initiating conversations with staff and addressing mental health matters in the workplace, and that these improvements appear to be sustained. However, when selecting a method of training for their managers, workplaces should expand their selection criteria beyond the cost benefits and scalability opportunities offered through online training. Other factors such as the likelihood workers will engage with an e-Learning platform should be considered as this may determine the benefits of training for an organisation. If employees do not utilise technology on a regular basis as part of their work, it may be that face-to-face training remains a better fit for their organisation. Therefore, decisions regarding which type of training are most suitable for different workplaces will need to continue to be made on the basis of which is likely to be the best logistical and cultural fit for each organisation, with the caveat that users of online training need to find ways to ensure adequate completion rates.