Simulation-based training for increasing health service board members' effectiveness: protocol for a cluster-randomised controlled trial.

INTRODUCTION: Research indicates that health service boards can influence quality of care. However, government reviews have indicated that board members may not be as effective as possible in attaining this goal. Simulation-based training may help to increase board members' ability to effectively communicate and hold hospital staff to account during board meetings. METHODS AND ANALYSIS: To test effectiveness and feasibility, a prospective, cluster-randomised controlled trial will be used to compare simulation-based training with no training. Primary outcome variables will include board members' perceived skill and confidence in communicating effectively during board meetings, and board members' perceptions of board meeting processes. These measures will be collected both immediately before training, and 3 months post-training, with boards randomly assigned to intervention or control arms. Primary analyses will comprise generalised estimating equations examining training effects on each of the primary outcomes. Secondary analyses will examine participants' feedback on the training. ETHICS AND DISSEMINATION: Research ethics approval has been granted by Monash University (reference number: 2018-12076). We aim to disseminate results through peer-reviewed journal publication, conference presentation and social media. TRIAL REGISTRATION NUMBER: Open Science Framework: http://osf.io/jaxt6/; Pre-results. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

RCT Entities:   Population Interventions Outcomes

This is the first study to test an intervention to improve health service boards’ functioning.

Extensive evidence review and stakeholder consultation informed the development of the intervention.

Study will examine the impact of simulation-based training on board members’ skills and confidence in communicating effectively in health service board meetings, and perceptions of board meeting processes.

Examining the impact on quality and safety outcomes was not possible within study budget and timeframe, but results can inform the development of a future large-scale randomised trial that includes these measures.

Introduction

Existing research indicates that hospital boards have the ability to influence patient safety and quality of care.1–4 However, government reviews have indicated that board members may not be as effective as possible in attaining this goal. For example, limitations to health service boards’ effectiveness were found to have contributed to negative health outcomes in Victoria, Australia,5 where a cluster of perinatal deaths were, in part, attributable to suboptimal board processes.

To influence patient safety, health service boards must have the necessary skills and confidence to actively lead and pursue quality healthcare.6 However, there is often a gap between required skills and actual skills on boards, especially given the variability of board composition across states, from large metropolitan hospitals to small rural locations.5 Within the public health service in Victoria, there are 85 independent health service boards with directors appointed by the Minister of Heath (of the Victorian parliament) and governed by the Department of Health and Human Services (DHHS). Victorian health service boards are responsible to the Health Minister for the effective and efficient governance of their health service. This includes monitoring and improving quality and safety, and risk management.

The Review of Hospital Safety and Quality Assurance in Victoria (the ‘Duckett Review’) found that in health service board meetings there was a lack of attention to quality performance compared with financial performance.5 Although there are continuous efforts in Victoria to improve data, health services currently have access to numerous data sources on quality and safety outcomes, including operational performance, clinical outcomes and the experience of patients receiving care. However, it was found that one in five boards did not have quality performance as a standing agenda item. Furthermore, the report concluded that, ’Consistent with the literature on boards and their impact on safety and quality, this review found that gaps in board skills, information and oversight are a key priority for strengthening governance of patient safety in hospitals' (5, p. 27). The review articulated 19 recommendations to enhance the skills, independence and effectiveness of health service boards, as well as recommendations pertaining to information, accountability, risk management and promoting a culture of safety and quality. One of these recommendations was that all boards must be highly skilled, independent and effective, and the flow of information in the health system must ensure deficiencies in care are identified and focus attention on opportunities for improvement. It is therefore important that all board members have the communication skills and confidence necessary to ensure that they receive and consider relevant information.

Qualitative investigations into the culture of Victorian health service boards found that there was a perception that boards were becoming more professional and more engaged with quality issues, changes which were attributed to increased awareness that quality and safety were core functions of the board.1 However, there was a significant variation across boards, highlighting the need to better understand internal board processes. There is a paucity of research that has measured internal health service board culture.

There are both formal and informal practices within boards and board meetings. Many problems highlighted by the literature occur within the informal sphere of the board, such as information asymmetries between management and directors, relational difficulties and tensions, and a lack of open boardroom discussions.7 Furthermore, informal power dynamics and alliances can influence how board meetings run. Such informal networks within boards influence decision-making processes. Dominant members may form an alliance of like-minded individuals who then set the tone for how decisions are made. However, this may not be conducive to well‐informed decisions. In managing the numerous viewpoints of a board, time allocation and team management become important. In a study of two corporate boards, it was found that a single participant accounted for around half of all speaking time in any one item, and the top three contributors accounted for more than 80% of the speaking time.8 The development of an ‘inner board’ (a subset of board members with high influence) that grows among some of the board members outside of the boardroom is likely to have profound effects on information-seeking, cognition and decision-making. Research has found that informal networks developed within boards, and these networks, partly shaped by the formal governance structure within which they were formed, had effects on cognitive agreement within the board meetings.9

Boards need to reflect accurately on their own competencies to identity skills or knowledge gaps. A 2012 survey of Victorian boards found that virtually all respondents believed that the overall safety and quality of care delivered at their health service was as good as, or better than, the typical Victorian health service.2 This self-assessment bias reflected a lack of benchmarking data, which meant that many hospitals did not actually know how the safety and quality of their care compared with other hospitals’ performance.5

Simulation-based training has increasingly been used to foster adult education in the medical field, and there is emerging evidence that simulation practices are more effective than traditional learning methods.10 11 Simulation-based training is an educational approach that involves placing learners in realistic situations that provide an opportunity to practice and learn in a safe environment.10 12 Studies have found that exposing student health practitioners to simulations can foster the development of knowledge and skills,13 including communication skills.14 Debriefing and allowing participants to reflect on the simulation practice has also been identified as an integral factor to success of this type of training.15 However, there are few studies that report on using simulation to address complex systems or issues, or group processes, including in health service board settings.

This study aims to ascertain the feasibility and effectiveness of using simulation-based training for improving board members’ skills and confidence in communicating effectively in health service board meetings. The study is designed as a cluster randomised controlled trial with two parallel groups: an intervention group that receives the training, and a control-waitlist group that does not receive the training until after the trial is completed. It is hypothesised that providing simulation-based training on how to navigate difficult conversations will improve board members’ skills and confidence in communicating effectively during health service board meetings and perceptions of board meeting processes.

Methods and analysis

Intervention development process

The intervention was developed through a structured three-step process. First, an expert panel was convened to steer the topic development. Second, an evidence review was undertaken of the academic evidence, complemented by practice interviews. Third, a structured stakeholder dialogue was convened to deliberate on the evidence and prioritise a feasible intervention that could be piloted.

Expert panel

A panel of four experts with extensive experience in healthcare governance was convened to steer the development of the topic. The responsibilities of this expert panel were to identify a broad focus within the topic of governance that could improve patient safety and quality of care. This refined the topic to ‘Optimising health service board meeting processes and behaviours to better meet governance objectives’. Following this, the expert panel provided input into recruitment for the stakeholder dialogue and reviewed the evidence brief.

Evidence review

A review of relevant evidence was undertaken using a rapid review methodology. The evidence review included one narrative review and 27 primary studies. The review covered board meetings and board members, and cognitive and group processes. In addition to reviewing the academic evidence, six practice interviews were conducted including; one Chief Executive Officer (CEO), one Director of Quality and Safety, two board chairs and two board directors. The practice review identified key implementation challenges and reinforced the findings of the literature review that the gap in skills is key to the functioning of the board. The review also included a list of extracted potential interventions. The evidence review was circulated to all dialogue participants prior to the dialogue.

Structured stakeholder dialogue

A day-long, structured stakeholder dialogue was conducted in September 2017 to address the issue of ‘Optimising health service board meeting processes and behaviours to better meet governance objectives’. The dialogue was attended by 11 people representing government, research, health service management, health service boards, board evaluation and consumer sectors. Through facilitated discussion, suggested interventions were added to the initial list of 38 presented in the evidence review. This list was then collapsed into 15 key areas by reducing conceptual overlap between the interventions identified.

A structured process drawing on key principles of prioritisation was used to identify an intervention from 1 of the 15 key areas. Through this process, an intervention that developed soft skills in board members in navigating difficult conversations to obtain and follow-up the information that board members require to make decisions was identified as a key area where there was a gap in skills and there is a paucity of available training.

Patient and public involvement

Although no patients or members of the public were directly involved in the design of the trial, one healthcare consumer consultant represented these perspectives in the structured stakeholder dialogue described above. This person had previously provided consumer input in several major government health initiatives and had substantial experience in advocating for healthcare consumers. They assisted in identifying and prioritising possible interventions and outcome measures, but were not otherwise involved in the trial design or in recruiting participants for the trial.

Study design

The study will use a cluster-randomised trial design with a simulation-based training intervention group and a waitlist-control group (who will not receive the training until after the trial has been completed). Primary outcome measures will be collected at baseline (before any training has occurred), and at 3 months in both intervention and control arms. Health service boards will be randomly allocated to intervention or control arms by central computer randomisation, which will be performed by the first author. No blinding after assignment will be used. Stratified randomisation will be used to ensure similar numbers of regional and metropolitan boards are allocated to each arm of the trial.16

Participants

Members of Victorian health service boards will be recruited. All members of health service boards are eligible to participate. No exclusion criteria apply. The participant timeline is shown in figure 1.

Figure 1

Participant timeline.

Recruitment strategy

Health service boards will be recruited jointly by the DHHS and the Victorian Managed Insurance Authority (VMIA), who will approach boards in both metropolitan and regional areas.

Sample size determination

Sample size has been determined primarily by time and budget constraints. We anticipate including 12 boards in the trial, each of which will have an average of approximately 7 board members. Assuming that 70% of board members provide data at both pre-time and post-time points (with multiple follow-up reminders), we expect to receive usable data from 5 board members per board, resulting in a total sample of 60 individuals across 12 boards. Given the limited sample available, we will use a p<0.1 significance level in all tests. Additionally, we will use one-sided significance tests because we are only interested in whether simulation training improves outcomes: for practical purposes, the resulting course of action (ie, discontinue training) will be the same if the simulation training has a negative effect or has no effect.17 Using General Linear Multivariate Model Power & Sample Size (GLIMMPSE) software,18 we estimated that this sample size will provide 80% power to detect a 0.39 mean difference between the intervention and control conditions in Time 2 outcomes, controlling for Time 1 outcomes (on a scale with a standard deviation equal to 1, and assuming equal sample sizes in the treatment and control groups, an intracluster correlation coefficients (ICC) equal to .05, an r = .4 correlation between Time 1 and Time 2 responses, a total of 12 boards with five individuals per board, and a one-sided p < 0.1 significance level). This effect size is smaller than that found in existing research examining the effect of other forms of training on physicians’ communication skills,19 and smaller than the average effect of patient simulation training in nursing education, as identified in a meta-analysis.10

Intervention

The intervention involves immersive, simulation-based training of health service boards to increase their confidence in asking targeted questions and ensuring that they are satisfied with responses. Training sessions will run for 2 hours, and will involve all Board Directors and Board Chairs. Each participant will have the opportunity to engage in a scenario-based simulation exercise with a facilitator and trained actor for approximately 5–8 min, and will also observe other participants completing the simulations. This exercise will include considering the specific roles of designated board directors, particularly the Board Chair, as well as the roles of others that may be present, for example the CEO/hospital executive. The scenarios have been developed in partnership with key stakeholders and a trained actor to develop exercises that will challenge participants and give them opportunities to employ relevant skills. Scenarios will be developed based on three themes informed by key stakeholders: (1) asking difficult questions; (2) dealing with pushback and (3) refocusing the conversation. During the training session, participants will have opportunities to reflect on feedback, and incorporate feedback into practice scenarios. To aid participants in responding to the scenarios, a small set of guiding questions will be provided to each participant. This discrete set of guiding questions has been codesigned with relevant stakeholders, including the VMIA, SaferCare Victoria, the Victorian Healthcare Association, researchers, consumers and Board representatives.

Data collection

Data will be collected using an online survey that will be administered to participants at two time points: immediately before the intervention is delivered to boards allocated to the intervention arm (Time 1), and 3 months postintervention (Time 2). Participants will be sent emails at each time point inviting them to complete the survey. Additionally, participants in the simulation training condition will receive a face-to-face reminder and opportunity to complete the Time 1 survey immediately before the training occurs.

Outcomes

The primary outcomes are perceived skills and confidence in communicating effectively in health service board meetings, and perceptions of board meeting processes. These outcomes will be measured using a series of survey items designed specifically for this study. These items are shown in table 1.

Table 1

Survey items for measuring primary outcomes

Outcome	Item wording	Response scale
Skills and confidence	I can communicate convincingly, yet diplomatically in board meetings.	1=Strongly disagree to 6=Strongly agree
Skills and confidence	I am confident in my ability to get the information I need in board meetings	1=Strongly disagree to 6=Strongly agree
Skills and confidence	I respond to information I have received in board meetings in a timely manner	1=Strongly disagree to 6=Strongly agree
Skills and confidence	I am confident in requesting information during board meetings	1=Strongly disagree to 6=Strongly agree
Skills and confidence	If information is not provided after being requested in a board meeting, I know what to do	1=Strongly disagree to 6=Strongly agree
Skills and confidence	I am very confident in raising difficult issues during board meetings	1=Strongly disagree to 6=Strongly agree
Skills and confidence	Even when other board members disagree with me, it is easy to express my opinions	1=Strongly disagree to 6=Strongly agree
Board meeting processes	Board meetings are conducted in a manner that encourages open communication and meaningful participation from all board directors.	1=Strongly disagree to 6=Strongly agree
Board meeting processes	Sufficient time is provided during board meetings for thoughtful discussion and interrogating issues.	1=Strongly disagree to 6=Strongly agree
Board meeting processes	There is adequate time in board meetings to address all agenda items thoroughly	1=Strongly disagree to 6=Strongly agree
Board meeting processes	All directors ask questions during board meetings	1=Strongly disagree to 6=Strongly agree
Board meeting processes	All directors make robust contributions to discussions	1=Strongly disagree to 6=Strongly agree
Board meeting processes	All directors are able to disagree without being disagreeable	1=Strongly disagree to 6=Strongly agree
Board meeting processes	All directors display respect for each other	1=Strongly disagree to 6=Strongly agree
Board meeting processes	There is a culture of openness at our board meetings	1=Strongly disagree to 6=Strongly agree

Secondary outcomes are self-reported perceptions of the relevance and utility of the training. These measures will only be included in the Time 2 survey in the intervention arm. They include both Likert-response questions and open-ended questions designed to examine participants’ qualitative reflections on the training, as shown in table 2.

Table 2

Survey items for measuring secondary outcomes

Item wording	Response scale
The training was relevant and useful to my role as a board member	1=Strongly disagree to 6=Strongly agree
The training has helped the board better achieve its objectives	1=Strongly disagree to 6=Strongly agree
The training has helped the board manage its relationship with the CEO and executive	1=Strongly disagree to 6=Strongly agree
I have used what I have learnt at the training to asking questions at board meetings	1=Strongly disagree to 6=Strongly agree
The training has changed the way I conduct my role in board meetings	1=Strongly disagree to 6=Strongly agree
The training has enabled me to ask better questions	1=Strongly disagree to 6=Strongly agree
The training has enabled me to follow-up on requested information	1=Strongly disagree to 6=Strongly agree
What are your reflections on the training you received?	Open-ended
Did the training alter your behaviour in board meetings, and if so, how?	Open-ended
Is there anything you would change about the training?	Open-ended
What is one thing you would retain about the training?	Open-ended

Data analysis

Computing multi-item scales

Items with Likert-response categories will be subjected to exploratory factor analysis (using principal axis factoring and promax rotation) to identify a plausible factor structure that will be used to generate multi-item scales. The items reflecting ‘Board meeting processes’ and ‘Skills and confidence’ (as labelled in table 1) will be entered into separate analyses, as will the Likert items measuring secondary outcomes (as shown in table 2). For all analyses, the number of factors will be determined using Cattell’s scree test criterion.20 Items with factor loadings≥0.32, and no cross-loadings≥0.32, will be retained for inclusion in multi-item scales.21 Items will be averaged to form composite scales for each factor identified. Internal reliability will be assessed using Chronbach’s alpha for scales with three or more items, and correlation coefficients for scales with two items.

Quantitative analyses

Generalised estimating equations will be used to estimate the impact of the simulation training on primary quantitative outcomes (and account for nesting of individual participants within boards). These analyses will adjust for remoteness area (metro, regional) because it will be used as a balancing variable in the stratified randomisation.22 As such, the independent variables included in these analyses will be: Time 1 scores; experimental condition (0=control, 1=treatment) and remoteness area (0=metro, 1=regional and remote). The dependent variables will be the Time 2 scores. All analyses will use board as the clustering variable. Generalised estimating equations usually use a Huber-White sandwich estimator that requires a large number of clustering units (e.g. more than 30–50 boards) to generate accurate and unbiased estimates of standard errors.23 24 In contrast to many cluster- randomised controlled trials that ignore this issue, we will use a small sample, bias-corrected estimator to avoid this potential limitation.25–27 Descriptive statistics will also be reported for all quantitative measures.

Qualitative analyses

Open-ended questions will be analysed using thematic analysis28 to identify common qualitative themes in responses.

Discussion

Research has identified health service boards as an important influence on the quality of care in hospitals, but no research has investigated techniques for improving how these boards function. There is a pressing need for this research, with government reviews indicating that poor health service board functioning has contributed to increased mortality in hospitals.5 This research develops and tests the impact of simulation training as a technique for improving how health service boards function, and improving the skills and confidence of health service board members. Previous research has indicated that simulation training is a useful form of adult education,10 13 but this study is the first to use it for improving health service board members’ ability to effectively communicate and hold hospital staff to account during board meetings.

It should be noted that this study is not without limitations. First, it would be desirable to include long-term monitoring of patient outcomes, in addition to the self-reported measures of board members’ confidence and perceptions of how board meetings function. Unfortunately, timeframe and budget constraints prevent such measures from being monitored at the scale and timeframes necessary to estimate effects on these outcomes. However, the findings of this trial could be used to inform the development of a subsequent, larger scale randomised controlled trial. Second, for practical and ethical reasons, our sample will only include boards who are willing to receive the training. As such, we will only be able to estimate the effect of the intervention on hospital boards that are willing and interested in participating. It is possible that the effect may be different if boards were forced to receive the training, but our trial will not be able to examine this possibility.

The quantitative and qualitative findings of this study will be used to inform the future development and large-scale implementation of simulation training for health service board members in Victoria, Australia. The study has the potential to improve the functioning of health service boards, which may result in improved patient care outcomes.

Ethics and dissemination

Research ethics approval has been granted by Monash University (reference number: 2018-12076). Results will be disseminated through peer reviewed journal articles and conference presentations.