Effectiveness of knowledge brokering and recommendation dissemination for influencing healthcare resource allocation decisions: A cluster randomised controlled implementation trial.

BACKGROUND: Implementing evidence into clinical practice is a key focus of healthcare improvements to reduce unwarranted variation. Dissemination of evidence-based recommendations and knowledge brokering have emerged as potential strategies to achieve evidence implementation by influencing resource allocation decisions. The aim of this study was to determine the effectiveness of these two research implementation strategies to facilitate evidence-informed healthcare management decisions for the provision of inpatient weekend allied health services. METHODS AND
FINDINGS: This multicentre, single-blinded (data collection and analysis), three-group parallel cluster randomised controlled trial with concealed allocation was conducted in Australian and New Zealand hospitals between February 2018 and January 2020. Clustering and randomisation took place at the organisation level where weekend allied health staffing decisions were made (e.g., network of hospitals or single hospital). Hospital wards were nested within these decision-making structures. Three conditions were compared over a 12-month period: (1) usual practice waitlist control; (2) dissemination of written evidence-based practice recommendations; and (3) access to a webinar-based knowledge broker in addition to the recommendations. The primary outcome was the alignment of weekend allied health provision with practice recommendations at the cluster and ward levels, addressing the adoption, penetration, and fidelity to the recommendations. The secondary outcome was mean hospital length of stay at the ward level. Outcomes were collected at baseline and 12 months later. A total of 45 clusters (n = 833 wards) were randomised to either control (n = 15), recommendation (n = 16), or knowledge broker (n = 14) conditions. Four (9%) did not provide follow-up data, and no adverse events were recorded. No significant effect was found with either implementation strategy for the primary outcome at the cluster level (recommendation versus control β 18.11 [95% CI -8,721.81 to 8,758.02] p = 0.997; knowledge broker versus control β 1.24 [95% CI -6,992.60 to 6,995.07] p = 1.000; recommendation versus knowledge broker β -9.12 [95% CI -3,878.39 to 3,860.16] p = 0.996) or ward level (recommendation versus control β 0.01 [95% CI 0.74 to 0.75] p = 0.983; knowledge broker versus control β -0.12 [95% CI -0.54 to 0.30] p = 0.581; recommendation versus knowledge broker β -0.19 [-1.04 to 0.65] p = 0.651). There was no significant effect between strategies for the secondary outcome at ward level (recommendation versus control β 2.19 [95% CI -1.36 to 5.74] p = 0.219; knowledge broker versus control β -0.55 [95% CI -1.16 to 0.06] p = 0.075; recommendation versus knowledge broker β -3.75 [95% CI -8.33 to 0.82] p = 0.102). None of the control or knowledge broker clusters transitioned to partial or full alignment with the recommendations. Three (20%) of the clusters who only received the written recommendations transitioned from nonalignment to partial alignment. Limitations include underpowering at the cluster level sample due to the grouping of multiple geographically distinct hospitals to avoid contamination.
CONCLUSIONS: Owing to a lack of power at the cluster level, this trial was unable to identify a difference between the knowledge broker strategy and dissemination of recommendations compared with usual practice for the promotion of evidence-informed resource allocation to inpatient weekend allied health services. Future research is needed to determine the interactions between different implementation strategies and healthcare contexts when translating evidence into healthcare practice. TRIAL REGISTRATION: Australian New Zealand Clinical Trials Registry ACTRN12618000029291.

Introduction

Healthcare systems worldwide continue to grapple with unwarranted variation in quality and safety. On average, an estimated 60% of care is delivered according to recommended guidelines [1-4], while up to 30% is considered low-value care [5-7]. Underuse of effective treatments and overuse of those with questionable benefit are arguably responsible for substantial inefficiency and lost opportunity to improve patient outcomes, particularly when considering improvements that could be realised by reallocating resources to high-value care.

The evidence-to-practice gap often manifests through decisions and negotiations within and across multiple levels of the health system [8]. These phenomena create a complex ecosystem, where the delivery of care can be dependent upon policy and managerial decisions regarding the organisation of resources. One highly topical area of healthcare policy and management decision-making is the provision of weekend allied health services to inpatient wards [9]. Internationally, there is substantial variation in access to allied health professionals within hospitals (e.g., physiotherapists, occupational therapists, speech pathologists dietitians, social workers, and podiatrists) [10-14]. This variability is typified by service provision during weekends, where some hospitals extend limited provision of allied health services [15,16]. A recent systematic review and meta-analysis reported that additional weekend allied health services in subacute rehabilitation units can reduce hospital length of stay by over two days and is a cost-effective approach to improve function and health-related quality of life [17]. However, the benefits were less clear for acute general medical and surgical wards. Despite this evidence, subacute rehabilitation units often provide less allied health during weekends, compared to acute general medical and surgical wards, both in absolute and relative terms [11].

Accessing and applying research evidence to guide resource allocation decisions can be difficult for healthcare managers [18,19]. Therefore, it is imperative to evaluate strategies that promote evidence-informed decision-making to ensure that the benefits of research on weekend allied health service models can be translated to improved health outcomes. Guidelines and recommendations are widely used to disseminate concise instructions regarding patient care. Recommendations have been found to increase awareness of key messages, change attitudes and knowledge [20-22], and impact practice in some circumstances [23-26]. However, they do not always lead to meaningful changes in behaviour [27-30]. Given the feasibility and low cost involved, dissemination of evidence-based practice recommendations may be an efficient way to change practice under certain conditions, even if it is less effective than more interactive alternatives. A more interactive, and resource-intensive, approach is the use of knowledge brokering to support dissemination and implementation of recommendations into evidence-informed decision-making [31-36]. Knowledge brokers are intermediary agents who build relationships between decision-makers and researchers, by sharing expert knowledge and establishing communication channels [37]. Much of this work occurs informally [38]. Yet, these roles are increasingly being formalised and institutionalised [39,40], despite limited evidence to support their effectiveness [36,41]. The substantial cost and resources required to deliver formalised knowledge broker roles [42,43] require evidence of both effectiveness and cost-effectiveness to justify investment. This is because the health system is characterised by finite funding, which must be allocated to competing needs: providing funds for activities with unknown levels of effectiveness such as knowledge brokering is difficult to justify, as those same funds can no longer be used for alternative activities that are known to effectively improve health outcomes elsewhere in the health system.

The aim of this study was to determine the effectiveness of knowledge brokering and dissemination of evidence-based practice recommendations for weekend allied health resource allocation decisions by hospital managers.

Methods

Monash Health Research Ethics Committee approved this research (HREC/17/MonH/44). The study protocol has been published (S1 Text) [44] and registered (Australian New Zealand Clinical Trials Registry ACTRN12618000029291); Universal Trial Number (UTN): U1111-1205-2621.

Context

Allied health professionals routinely deliver inpatient services Monday to Friday for hospitals in high-income countries. In certain parts of the world, these services are also extended during Saturday and Sunday, with Saturday physiotherapy services being the most provided [10,45]. In Australia, approximately 60% of acute hospital wards and 30% of subacute wards provide physiotherapy during weekends [11], which contrasts with Level-I evidence indicating that the benefits are clearer in subacute rehabilitation units [17]. A recent study by Haines and colleagues demonstrated that weekend allied health services could be removed from acute general medical and surgical wards without impacting health or service delivery outcomes and that redesign and reinstatement of these services also did not change these outcomes [16].

Trial design

In this multicentre study, we conducted a blinded (data collection and analysis) three-group, parallel cluster randomised controlled trial with concealed allocation to compare two alternate research implementation strategies with a control. Clustering occurred at the organisation level where weekend allied health staffing decisions were made within each healthcare organisation to avoid the potential risk of contamination between units of randomisation (e.g., network of hospitals or single hospital). Hospital wards were nested within these decision-making structures. For example, organisations made up of geographically distinct hospitals that made independent decisions in relation to allied health staffing were randomised as separate clusters; those making decisions across hospitals within the organisation were randomised as a single cluster. The diversity in decision-making structures across Australian and New Zealand hospitals constrained the ability to prespecify the number of potentially eligible hospitals/wards to be included within each cluster. Randomisation was stratified based on self-reported geographical classification, as either metropolitan or rural (including regional and remote). This study design is considered the most suitable to address questions of effectiveness, avoid potential contamination across study conditions, and capture outcomes at the system levels where changes were expected to occur.

During the conduct of this study, there was one randomisation error and one modification to the statistical methods described in our study protocol. The randomisation error occurred when one cluster was randomised to the recommendation group but mistakenly did not receive it because of human error. Data were still collected from this cluster and analysed according to the group to which they were assigned. Modifications to the analysis are outlined in the statistical methods section. A CONSORT Extension for Cluster Trials checklist for this study is provided in S2 Text, and the trial was also reported according to the Standards for Reporting Implementation Studies Statement (StaRI) in S3 Text.

Participants and setting

This study took place across a sample of Australian and New Zealand hospitals. Eligible hospitals were those providing acute or subacute services, with either public or private funding arrangements. Specific wards of interest were general medical and surgical and subacute rehabilitation. Specialist hospitals, including maternity, paediatric, cancer, mental health, and palliative care, were excluded, as no research regarding weekend allied health provision had been identified in these settings. Hospital managers who were responsible for inpatient weekend allied health resource allocation decisions at each cluster were eligible to receive the interventions on behalf of the cluster after providing written informed consent.

Interventions

A detailed description of the three study conditions according to the Template for Intervention Description and Replication (TIDieR) guidelines [46] is provided in the published protocol [44], and specification of the implementation strategies delivered to the study groups is reported in Table 1 [47]. Each strategy was commenced at the time of randomisation for a period of 12 months to implement specific recommendations regarding weekend allied health provision, derived from a systematic review and meta-analysis [17] and summarised in Box 1. The full evidence-based practice recommendations are provided (S4 Text).

Table 1

Specification and reporting of each implementation strategy.

Domain	Recommendation strategy: Written evidence-based practice recommendations	Knowledge broker strategy: Webinar-based knowledge broker in addition to recommendations
Actor	EviTAH consortium.	EviTAH consortium. Additionally, a single knowledge broker with a PhD-level qualification, from an allied health professional background, with research experience, employed as a postdoctoral research fellow.
Action	An evidence-based practice recommendation document provided via email.	An evidence-based practice recommendation document provided via email. Additionally, knowledge broker support for the facilitation, transfer, and exchange of information to enable alignment of practice with the recommendations. Prompting questions informed by the COM-B model [49].
Target of the action	Hospital managers responsible for weekend allied health resource allocation decisions.	Hospital managers responsible for weekend allied health resource allocation decisions.
Temporality	Approximately within one week following randomisation.	Approximately within one week following randomisation.
Dose	Single occasion (although recommendation resent if requested).	(1) Initial individualised contact made via email or phone to confirm receipt of the written recommendations, discuss local needs, and discuss a plan over the next 12 months; (2) within six months (according to hospital manager availability), a group webinar was arranged; (3) the group webinar was followed up by individualised contact via email or phone (according to hospital manager preference); (4) a final group webinar was arranged; (5) follow up individualised contact thereafter on an “as needs” basis. Contacts were made over a 12-month period with dose varying according to levels of participant engagement.
Implementation outcome affected	Primary outcome—practice alignment with recommendations: capturing implementation outcomes—adoption of evidence-based practice recommendation, penetration among eligible hospital wards, and fidelity to the recommendation.Economic, process, and qualitative measures: capturing implementation outcomes—appropriateness of the recommendation as a source of information for the decision, acceptability of the trustworthiness and sufficiency of the recommendation, feasibility of the evidence-base to guide clinical practice, sustainability of the intervention and how it was provided, and cost to make the decision. To be reported in other publications.	Primary outcome—practice alignment with recommendations: capturing implementation outcomes—adoption of evidence-based practice recommendation, penetration among eligible hospital wards, and fidelity to the recommendation.Economic, process and qualitative measures: capturing implementation outcomes—appropriateness of the recommendation as a source of information for the decision, acceptability of the trustworthiness and sufficiency of the recommendation, feasibility of the evidence-base to guide clinical practice, sustainability of the intervention and how it was provided, and cost to make the decision. To be reported in other publications
Justification	Evidence-based practice recommendation documents are one of the few implementation strategies that have been evaluated for hospital managers [41,50,51], which have the potential to increase engagement with research implementation [52].	Multifaceted and interactive implementation strategies are thought to improve evidence-informed decision-making, particularly for organisations without a strong research culture [36]. Many public health organisations have adopted knowledge broker roles [53].

COM-B, capability, opportunity, motivation, and behaviour; EviTAH, The Evidence Translation in Allied Health; PhD, post-honorary doctorate.

Reduce weekend allied health staffing to a criterion of clinical priorities and exceptions that ensure between 0% and 0.1% of total allied health service events on acute general medical and surgical wards are delivered on weekends.

Increase weekend allied health staffing for subacute rehabilitation units to provide physiotherapy or a combination of physiotherapy and occupational therapy on Saturdays and physiotherapy on a case-by-case basis for stroke patients and, occasionally, to other patients when a clear need is evident on Sundays. This would ensure that between 10% and 20% of total service events for these professions are provided on weekends.

Briefly, three study conditions were delivered at the cluster level: (1) usual practice wait list control; (2) dissemination of written evidence-based practice recommendations; and (3) access to a webinar-based knowledge broker in addition to the recommendations. The waitlist control group experienced usual practice conditions according to their local setting. Upon study completion, they received the evidence-based recommendations for weekend allied health provision. Participants in the recommendation group were provided with an evidence-based weekend allied health practice recommendation (detailed and summarised) document via email. This document contained specific recommendations for the proportion of total allied health services that should be delivered during weekends. The document was constructed with an outline of key messages, executive summary, and presentation of the full research methods and findings [48]. The knowledge broker group was provided the same recommendation document and additional access to a knowledge broker who facilitated the transfer of relevant information to promote evidence-informed decision-making. The knowledge broker offered support via interactive online webinar, telephone, or email, in one-on-one and group settings. Prompting questions by the knowledge broker were informed by the COM-B (capability, opportunity, motivation, and behaviour) behaviour change model [49]. This support included individual needs assessments and developing a 12-month plan to address the recommendations. These supports followed an iterative process depending on participant needs, based on factors perceived to be associated with effective strategies from a recent systematic review [41]. The dosage and duration of the interventions were based on a similar study evaluating a knowledge broker role [36], which aligned with our hypothesis that a 12-month intervention duration would prove sufficient time for hospital managers to develop and implement a business case for change.

Outcomes

The primary outcome of interest was whether weekend allied health service provision at both the cluster and ward level aligned with the recommendations at 12-month follow-up. This outcome addressed the adoption of evidence-based practice recommendation, penetration among eligible hospital wards, and fidelity to the recommendation. Alignment with the recommendations was determined according to the number of allied health service events occurring during weekends, as a proportion of the total allied health service events for the cluster or ward, over a one-month period. A ratio of allied health full-time equivalent staffing was used, where service event data were not available (n = 3 clusters). Allied health service event data were collected at the time of randomisation for the preceding calendar month and the same calendar month, 12 months later. For cluster-level analysis, each cluster received a single classification as either (1) fully aligned with the recommendations for both acute wards and subacute units; (2) partially aligned with the recommendations (if acute wards are aligned but subacute units are not aligned, or vice versa); or (3) not aligned with policy recommendations. For the ward-level analysis, each acute ward or subacute unit received a single classification as either (1) fully aligned with the recommendations; or (2) not aligned with policy recommendations.

The secondary outcome was the mean hospital length of stay at each ward for the calendar month 12 months after study entry. We also collected equivalent data for each ward from the same calendar month 12 months earlier. Hospital length of stay was extracted from administrative data sources [54]. Detailed process and economic outcomes described in our protocol are planned for other publications.

Randomisation

Study investigators consulted with each healthcare organisation to determine their decision-making structure for allied health staffing to reduce the risk of contamination between study groups. Healthcare organisations that made allied health staffing decisions across multiple hospitals were treated as a single unit of recruitment and randomisation, where hospitals with independent decision-making processes within a broader healthcare organisation were treated as separate units. Hospital wards were nested within clusters and randomised using a random number sequence, generated by a single investigator (TPH) using an online software application [55] with permuted blocks within randomisation strata of sizes of 3, 6, or 9. Investigators conducting recruitment, data collection, and analysis (MS and MJ) were blinded, as per procedures outlined in the published protocol [44].

Statistical methods

Initially, the use of the nonparametric rank-based classical hypothesis test was planned to compare data for the primary outcome. However, changes were made to account for the observation that several clusters and wards were already aligned with the evidence-based recommendations. We changed our analysis approach to an ordered logit regression at the cluster level and logistic regression analysis at the ward level, so we could statistically adjust for the baseline status of each unit of analysis when comparing 12-month follow-up alignment with the recommendations between groups. These ANCOVA style analyses are aligned with recommendations from the Cochrane Handbook for Systematic Reviews of Interventions [56] for how baseline values can be appropriately incorporated into an interventional analysis framework. We used robust variance estimates to account for dependency of clustering across wards that had the same decision-makers, such that this analysis was still analysing data relative to the level of randomisation. The mean hospital length of stay was compared between groups in a ward-level analysis. Linear regression using baseline mean hospital length of stay as a covariate and robust variance estimates at the level of the decision maker was conducted.

Analysis of cluster-level and ward-level data was undertaken according to the group to which they were assigned by an analyst (MS) blinded to group allocation, using three mock codes representing different sequence allocation patterns. Multiple imputation using chained equations was performed to impute 50 datasets for missing values in both baseline and follow-up alignment with the recommendations. Geographical classification was used as the independent variable when imputing missing data at the cluster level; geographical classification and ward classification were used at the ward level. Two sensitivity analyses were conducted: The first was a complete case analysis for comparison with the analysis of imputed data, and the second was a multivariable analysis to adjust for other potential baseline confounders (cluster level: geography and full-time research staffing; ward level: geography and ward type). All analyses were adjusted for the number of weekday days and weekend days within the month to ensure that calendar years changes in the proportion of day types did not impact the results. Analyses were undertaken using Stata version 13.1 (StataCorp, College Station, Texas). Sample size estimates are reported in our study protocol [44].

We planned two levels of analysis, one to be conducted at the cluster level (one unit of data per cluster) and the other to be conducted at the ward level (nested within cluster). The study sample size was determined based on the most conservative unit of assessment for our primary outcome at the cluster level (organisation-level where weekend allied health staffing decisions were made), as an adequate sample size at the cluster level would also prove sufficient at the ward level. Further power analysis at the ward level was not conducted because there was no reasonable way to estimate the likely intracluster correlation coefficient (ICC) for this healthcare context and our main sample size constraints pertained to practicality of recruitment at the cluster level. A sample size of 25 clusters per group was estimated to provide greater than 80% power, assuming that 50% of clusters in either intervention group and 10% in the control group completely aligned with the policy recommendations. The diversity in decision-making structures across Australian and New Zealand hospitals constrained our ability to prespecify the number of hospitals/wards within each cluster and recruit sufficient clusters per group, as often allied health staffing decisions were made across hospitals/wards within organisations resulting in fewer potential units of randomisation. Study data used in these analyses are provided (S1 and S2 Data).

Results

The first cluster was recruited on 7 February 2018, and the final cluster completed their intervention in January 2020; final data collected for that period on 30 April 2020. A total of 45 clusters were randomised to either the control (n = 15), recommendation (n = 16), or knowledge broker (n = 14) groups. There was one cluster that did not provide any outcome data and three that did not provide follow-up outcome data (recommendation n = 2; knowledge broker n = 2) leaving a rate of 9% loss to follow-up (Fig 1). No adverse events were recorded. Within the clusters, there were n = 132 hospitals, n = 833 wards, and n = 204 hospital managers, whose baseline characteristics are provided in Tables 2 and 3.

Fig 1

CONSORT study flow diagram.

One cluster did not provide baseline or follow-up data; three clusters did not provide follow-up data only.

Table 2

Hospital manager baseline demographics.

	Control n = 71	Recommendation n = 44	Knowledge broker n = 89	Total n = 204
Geographical classification n (%)
Metro	44 (62)	27 (61)	51 (57)	122 (60)
Rural	25 (35)	17 (39)	36 (40)	78 (38)
Mix	2 (3)	0 (0)	1 (1)	3 (1)
Unknown	0 (0)	0 (0)	1 (1)	1 (<1)
Hospital classification n (%)
Acute	63 (89)	37 (84)	75 (84)	175 (86)
Subacute	7 (10)	4 (6)	8 (9)	19 (9)
Mix	1 (1)	3 (4)	5 (6)	9 (4)
Missing	0 (0)	0 (0)	1 (1)	1 (0.5)
Age (years) mean (SD)	46 (8.8)	47 (9.6)	46 (9.1)	46 (9.1)
Sex (female) n (%)	53 (75)	35 (80)	71 (79)	159 (78)
Professional background n (%)
Physiotherapy	22 (31)	13 (30)	18 (20)	53 (26)
Occupational therapy	10 (14)	11 (25)	16 (18)	37 (18)
Social work	12 (17)	4 (9)	13 (15)	29 (14)
Dietetics	10 (14)	6 (14)	12 (13)	28 (14)
Speech pathology	11 (15)	6 (14)	13 (15)	30 (15)
Podiatry	2 (3)	2 (5)	6 (7)	10 (5)
Other	4 (6)	2 (5)	11 (12)	17 (8)
Healthcare policy or management experience (years) mean (SD)	12 (7.6)	14 (10.6)	12 (8.9)	12.4 (8.8)
Highest qualification n (%)
Diploma	1 (1)	0 (0)	0 (0)	1 (<1)
Bachelor	24 (34)	9 (20)	35 (39)	68 (33)
Graduate or Honours	14 (20)	14 (32)	24 (27)	52 (25)
Master	30 (42)	19 (43)	29 (33)	78 (38)
Doctorate	2 (3)	2 (5)	1 (1)	5 (2)

Percent (%) values subject to rounding error and refer to group totals.

n, sample; SD, standard deviation.

Table 3

Healthcare organisation baseline demographics.

	Control	Recommendation	Knowledge broker	Total
Clusters (hospital or hospital network) n (%)
Total	15	16	14	45
Metropolitan	9 (60)	9 (56)	8 (57)	26 (58)
Rural	6 (13)	7 (44)	6 (43)	19 (42)
Allied health research staffing n (%)
Full-time academic	6 (40)	5 (31)	2 (14)	13 (29)
Clinician-researcher	9 (60)	7 (44)	6 (43)	22 (49)
Number of clusters providing inpatient allied health services n (%)
Total weekend	15 (100)	15 (94)	9 (64)	39 (87)
Acute	15 (100)	15 (94)	9 (64)	39 (87)
Subacute	7 (47)	10 (63)	5 (36)	22 (49)
Hospitals n (%)
Total	45	48	39	132
Metropolitan	20 (44)	26 (54)	18 (46)	64 (48)
Rural	25 (56)	22 (46)	21 (54)	68 (52)
Ward classification n(%)
Total	335	285	212	833
Acute	288 (86)	227 (79)	175 (83)	690 (83)
Subacute	47 (14)	59 (21)	37 (17)	143 (17)
Ward type n (%)
General medical and surgical	250 (75)	191 (67)	141 (67)	582 (70)
Orthopaedic	22 (7)	16 (5.6)	10 (5)	48 (6)
Neurological	16 (5)	15 (5)	13 (6)	44 (5)
Rehabilitation	46 (14)	56 (20)	30 (14)	132 (16)
Mixed	1 (<1)	7 (2)	18 (8)	26 (3)

Percent (%) values calculated relative to total values per group and are subject to rounding error.

n, sample.

The summative raw data for the primary and secondary outcomes are presented in Tables 4 and 5. Proportion of total allied health service events provided on weekends is presented for each group, along with the corresponding number and percentage of clusters or wards aligned with the recommendations. None of the clusters in the control or knowledge broker groups transitioned from “not aligned” at baseline to “partial” or “full alignment” at 12-month follow-up. Three clusters (21%) that did not align to the practice recommendations, from the written recommendation document only group, transitioned to partial alignment within the study period. The flow of cluster alignment from baseline to follow-up, by implementation strategy group, is presented in Figs 2–4 [57]. The most common reasons for missing follow-up data across all three groups were (1) that a different contact person was used (due to staff turnover) who did not know how to extract all the required data; and (2) the contact person did not have time to extract all the required data.

Table 4

Cluster-level summative raw data for primary outcome.

	Control (n = 15)		Recommendation (n = 16)		Knowledge broker (n = 14)
	Baseline	Follow-up	Baseline	Follow-up	Baseline	Follow-up
Cluster-level allied health service events, per day and ward mean (SD), obs
Acute ratio	0.13 (0.06), 15	0.14 (0.07), 15	0.13 (0.08), 15	0.11 (0.09), 14	0.09 (0.14), 14	0.08 (0.09), 12
Subacute ratio	0.10 (0.26), 15	0.04 (0.08), 15	0.04 (0.08), 15	0.06 (0.13), 14	0.06 (0.13), 14	0.04 (0.07), 12
Cluster-level alignment with recommendations n (%)
Full alignment	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)
Partial alignment	0 (0)	0 (0)	1 (6)	3 (19)	8 (57)	5 (36)
Not aligned	15 (100)	15 (100)	14 (88)	11 (69)	6 (43)	7 (50)
Missing	0 (0)	0 (0)	1 (6)	2 (13)	0 (0)	2 (14)

Percent (%) values calculated relative to total values per group and are subject to rounding error; acute and subacute ratios were calculated as the number of allied health service events occurring during weekends, as a proportion of the total allied health service events for the cluster, over a one-month period; acute ratio alignment with the recommendation is between 0 and 0.001; subacute ratio alignment with the recommendation is between 0.1 and 0.2; missing values removed from analysis.

n, sample; obs, observations; SD, standard deviation.

Table 5

Ward-level summative raw data for primary and secondary outcomes.

	Control (n = 335)			Recommendation (n = 286)			Knowledge broker (n = 212)
	Baseline		Follow-up	Baseline		Follow-up	Baseline		Follow-up
Ward-level allied health service events, per day and ward mean (SD), obs
Acute ratio	0.12 (0.15), 214		0.11 (0.15), 246	0.11 (0.18), 199		0.10 (0.17), 155	0.13 (0.18), 147		0.09 (0.12), 123
Subacute ratio	0.06 (0.10), 32		0.05 (0.08), 43	0.06 (0.17), 52		0.03 (0.08), 48	0.09 (0.16), 25		0.08 (0.20), 26
Ward-level alignment with recommendations n (%)
Acute aligned	70 (21)		72 (21)	67 (23)		51 (18)	63 (30)		47 (22)
Acute not aligned	144 (43)		173 (52)	132 (46)		104 (36)	84 (40)		76 (36)
Acute missing	74 (26)		43 (15)	28 (12)		72 (32)	28 (16)		52 (30)
Subacute aligned	3 (1)		7 (2)	1 (<1)		2 (<1)	3 (1)		4 (2)
Subacute not aligned	29 (9)		36 (11)	51 (18)		46 (16)	22 (10)		22 (10)
Subacute missing	15 (32)		4 (11)	7 (12)		11 (19)	12 (32)		11 (30)
Ward-level hospital length of stay mean (SD), obs	*	†		*	†		*	†
Acute	7.1 (9.1), 111	6.77 (7.5), 101	7.4 (7.4), 108	12.6 (13.1), 81	12.5 (13.3), 76	13.2 (18.0), 82	9.8 (7.8), 45	5.6 (4.4), 25	5.1 (3.3), 40
Missing n (%)	177 (61)	187 (65)	180 (63)	146 (64)	151 (67)	145 (64)	130 (74)	150 (86)	135 (77)
Subacute	19.3 (15.1), 14	19.3 (15.1), 14	24.1 (15.0), 16	27.4 (25.1), 27	28 (26.0), 25	26.6 (21.7), 25	20.6 (13.6), 8	11.6 (5.9), 5	14.3 (6.5), 6
Missing n (%)	33 (70)	33 (70)	31 (66)	32 (54)	34 (58)	34 (58)	29 (78)	32 (86)	31 (84)

Percent (%) values calculated relative to total values per group and are subject to rounding error; acute and subacute ratios were calculated as the number of allied health service events occurring during weekends, as a proportion of the total allied health service events for the cluster, over a one-month period; acute ratio alignment with the recommendation is between 0 and 0.001; subacute ratio alignment with the recommendation is between 0.1 and 0.2; missing values removed from analysis.

n, sample; obs, observations; SD, standard deviation.

*Includes all baseline wards where data provided.

Only includes baseline wards where follow-up data also provided.

Fig 2

Flow of clusters from baseline to follow-up policy recommendations alignment, by implementation strategy group.

Fig 4

Flow of subacute rehabilitation units from baseline to follow-up policy recommendations alignment, by implementation strategy group.

The effect size estimates for the implementation strategies compared to the control are presented in Table 6. Adjusted for baseline recommendation alignment, there was no significant difference for the primary outcome of alignment with the recommendations between the groups at the cluster level using ordered logit regression (recommendation versus control β 18.11 [95% CI −8,721.81 to 8,758.02] p = 0.997; knowledge broker versus control β 1.24 [95% CI −6,992.60 to 6,995.07] p = 1.000; recommendation versus knowledge broker β −9.12 [95% CI −3,878.39 to 3,860.16] p = 0.996). These findings were similar with a multivariable sensitivity analysis that included potential confounding baseline variables: geographic location and whether a full-time allied health researcher was employed (recommendation versus control β 19.65 [95% CI −18,116.13 to 18,155.43] p = 0.998; knowledge broker versus control β 1.93 [95% CI −22,189.52 to 22,193.38] p = 1.000; recommendation versus knowledge broker β −9.28 [95% CI −4,601.58 to 4,583.01] p = 0.997). There was no significant difference for the primary outcome of alignment with the recommendations between the groups at the ward level using logistic regression (recommendation versus control β 0.01 [95% CI 0.74 to 0.75] p = 0.983; knowledge broker versus control β −0.12 [95% CI −0.54 to 0.30] p = 0. 581; recommendation versus knowledge broker β −0.19 [−1.04 to 0.65] p = 0.651). These findings were similar with a multivariable sensitivity analysis that included potential confounding baseline variables: geographic location and ward type (recommendation versus control β 0.42 [95% CI −0.62 to 1.46] p = 0.430; knowledge broker versus control β 0.15 [95% CI −0.47 to 0.76] p = 0.639; recommendation versus knowledge broker β 0.04 [95% CI −0.96 to 0.87] p = 0.925).

Table 6

Effect size estimates for primary and secondary outcomes using imputed data.

	Recommendation vs. control	Knowledge broker vs. control	Recommendation vs. knowledge broker	ICC*
Primary
Cluster level
Alignment with recommendations coefficient (95% CI)	18.11 (−8,721.81 to 8,758.02) p = 0.997	1.24 (−6,992.60 to 6,995.07) p = 1.000	−9.12 (−3,878.39 to 3,860.16) p = 0.996	NA
Ward level
Alignment with recommendations OR (95% CI)	0.01 (0.74 to 0.75) p = 0.983	−0.12 (−0.54 to 0.30) p = 0.581	−0.19 (−1.04 to 0.65) p = 0.651	C: 0.31H: 0.40
Secondary
Ward level
Mean hospital length of stay coefficient (95% CI)	2.19 (−1.36 to 5.74) p = 0.219	−0.55 (−1.16 to 0.06) p = 0.075	−3.75 (−8.33 to 0.82) p = 0.102	C: 0.53H: 0.56

*ICCs partitioned at the C and H levels.

C, cluster; H, hospital; ICC, intracluster correlation coefficient; NA, not applicable.

For the secondary outcome of hospital length of stay, there was no significant difference between the groups at the ward level using linear regression (recommendation versus control β 2.19 [95% CI −1.36 to 5.74] p = 0.219; knowledge broker versus control β −0.55 [95% CI −1.16 to 0.06] p = 0.075; recommendation versus knowledge broker β −3.75 [95% CI −8.33 to 0.82] p = 0.102. These findings were similar with a multivariable sensitivity analysis that included potential confounding baseline variables: geographic location and ward type (recommendation versus control β 1.92 [95% CI −1.85 to 5.70] p = 0.308; recommendation versus knowledge broker β −5.32 [95% CI −13.95 to 3.31] p = 0.213), although a significant difference was identified for the knowledge broker versus control (β −0.71 [95% CI −1.38 to −0.05] p = 0.037).

The primary and secondary outcomes, at the cluster or ward level, were largely unaffected by a complete case sensitivity analysis presented in Table 7.

Table 7

Complete case effect size estimates for primary and secondary outcomes sensitivity analysis.

	Recommendation vs. control	Knowledge broker vs. control	Recommendation vs. knowledge broker	ICC*
Primary
Cluster level
Alignment with recommendations coefficient (95% CI)	17.29 (−5,508.16 to 5,542.76) p = 0.995	0.510 (−7,925.09 to 7,925.09) p = 1.000	−16.68 (−7,035.87 to 7,002.51) p = 0.996	NA
Ward level
Alignment with recommendations OR (95% CI)	1.64 (0.62 to 4.33) p = 0.315	0.78 (0.48 to 1.28) p = 0.330	0.39 (0.14 to 1.11) p = 0.078	C: 0.34H: 0.38
Secondary
Ward level
Mean hospital length of stay coefficient (95% CI)	2.19 (−1.35 to 5.73) p = 0.218	−0.55 (−1.16 to 0.06) p = 0.074	−3.75 (−8.30 to 0.79) p = 0.101	C: 0.36H: 0.31

*ICCs partitioned at the C and H levels.

C, cluster; H, hospital; ICC, intracluster correlation coefficient; NA, not applicable.

Process outcomes

The knowledge broker strategy dose varied from that specified in the protocol across sites, due to differing levels of participant engagement. There were 17 interactive online knowledge broker support webinars (duration one to two hours) conducted in total to facilitate the transfer of relevant information to promote evidence-informed decision-making. Healthcare decision maker participation was higher in the initial (range 6 to 18) compared to follow-up (range 2 to 8) webinars. Typically, the first webinars were attended by representatives from each allied health profession; however, the second webinar was predominantly attended by decision-makers from the physiotherapy and occupational therapy professions because the evidence and recommendations mostly pertained to these roles. Only two clusters (14%) participated in more than two webinars with the knowledge broker. A desire to make an internal decision regarding weekend allied health service provision was the most common reason for nonparticipation. Two clusters (14%) did not engage with the knowledge broker without providing a reason. Further detail regarding the knowledge broker webinar support sessions is published elsewhere [58]. Other process, economic, and qualitative outcomes specified in our protocol are planned to be reported in other publications.

Discussion

This study was unable to identify differences between the knowledge broker and dissemination of written evidence-based practice recommendation strategies employed for hospital managers to improve the alignment of weekend allied health services with current evidence. These findings were possibly driven by the high rate of baseline alignment with the recommendations in the knowledge broker group and inadequate statistical power at the cluster-level analysis, despite including 132 hospitals and 833 wards. All 15 clusters assigned to the control group continued providing weekend allied health that was not aligned with the recommendations. Three clusters (21%) from the dissemination of written recommendations only group transitioned from nonalignment at baseline to partial alignment 12 months later. None of the clusters who received the knowledge broker in addition to the recommendations transitioned to partial or full alignment with the recommendations. The reduction in hospital length of stay for the knowledge broker group compared to the control indicates that there may have been an effect but the precise mechanism is unclear without an observed change in recommendation alignment.

Our study results are concordant with the only other three-arm trial published on the dissemination of recommendations and knowledge brokering for implementing evidence into healthcare policies and programs [36]. This earlier research was also unable to demonstrate a positive effect of their knowledge broker implementation strategy; however, the authors reported that impacts may have been more pronounced for health departments with low baseline organisational research culture. Achieving our desired outcomes at only a few organisations aligns with this, and other previous research indicating the effect of research implementation strategies may be contextually specific to organisational characteristics, such as strategic priority, leadership, and readiness [59,60]. The knowledge broker group in our study was characterised by higher baseline rates of recommendation alignment, despite pilot work conducted prior to study commencement indicating that most healthcare organisations were unlikely to be aligned with the evidence-based practice recommendations. While our analysis adjusted for baseline alignment, it is possible that a ceiling effect was reached in this group or other potentially confounding variables may have influenced differences in follow-up recommendation alignment. Formal assessment of evidence alignment and other potential confounders prior to study commencement was considered impractical within this study due to the data collection burden for time-limited hospital managers. Although, future research may benefit from these baseline assessments if information can be captured via routinely collected data for administrative or other purposes.

The decision to use an externally based, centralised knowledge broker to deliver support across multiple hospitals via interactive online webinar may have been an important factor influencing the strategy’s impact. Our findings align with a similar cluster randomised controlled trial conducted by Minian and colleagues, which compared generic emails with a remote knowledge broker to integrate mood management into a smoking cessation program [61]. In their study, the more intense and personalised remote knowledge broker strategy was no more effective at enabling healthcare professionals to provide their patients with mood management resources. The “more is better” theory, suggesting that a higher implementation strategy dosage through frequency of interactions or longer duration leads to greater success, is difficult to reconcile with this emerging empirical evidence. Hospital managers and healthcare professionals are time and resource constrained, facing multiple competing priorities that limit their ability to engage in more active strategies to facilitate evidence implementation [41]. The limited frequency and duration of the knowledge broker interactions may have impeded observed effectiveness of the strategy. However, it is unlikely that providing additional opportunities for engagement or a longer duration (e.g., 24 months) would have changed the dosage received, given the limited voluntary hospital manager engagement throughout the 12-month period and the desire to make an internal decision being the most common reason for limited engagement.

Most clusters only attended one or two group-based knowledge broker support sessions, which were delivered online rather than face-to-face. The limited dosage and mode of delivery may have constrained the ability to build relationships with the hospital managers. Knowledge broker roles are inherently relationship based [62], whose theory of change is premised on interpersonal contact [63], development of rapport [64], and building linkages and exchange between research producers and end users [31,39]. These social influence mechanisms of change [65-68] imply that the effectiveness of a knowledge broker may be largely individual dependent [53] and could be enhanced by embedded brokerage roles within organisations. For example, interactions between different hospital managers and knowledge brokers can result in varying levels of “relationship capital,” which is instrumental to fostering use of knowledge [63]. Less formal roles that are internal to organisations, such as opinion leaders or clinical champions, could instead be considered to leverage preexisting, peer-to-peer relationships and channels of persuasion [63]. It is important for informal brokers to consider group affiliation with the hospital managers, through occupation and professional legitimacy, to ensure that brokerage is not considered as “outsider expertise” [38]. These localised models of knowledge brokering, such as the National Institute for Health Research (NIHR) Collaborations for Leadership in Applied Health Research and Care (CLAHRCs), may provide a more situationally relevant, flexible, and collaborative approach to implement research into practice [69], albeit at a higher cost of investment.

The small number of clusters in this study was primarily driven by the need to cluster multiple geographically distinct hospitals to avoid contamination. During protocol design and sample size calculation, it was not possible to anticipate how many wards would be included in each cluster prior to recruitment, as we first needed to ascertain the decision-making structure for each healthcare organisation to in order to understand how many wards were nested within each decision-making structure (e.g., single hospital or network of hospitals). We acknowledge that our efforts to understand and cluster according to these decision-making structures prevented prespecification of cluster size. This design element was imperative to avoid the potential risk of contamination between units of randomisation. Recruitment challenges and difficulty ensuring adequate statistical power in analyses have been reported in other studies seeking to implement evidence into health system policy changes [70], particularly for pragmatic “real-world” projects at the level of organisations rather than individuals [71,72]. Considering the challenges experienced in this study and others, some have questioned the appropriateness of empirical designs for evaluating the effectiveness of implementation strategies, such as knowledge brokering [36]. Instead, more exploratory approaches oriented towards improving the understanding of how, why, when, and in what circumstances these strategies address more subjective, intermediate outcomes (e.g., capability for evidence engagement) have been advocated. These types of pluralistic, discursive, and iterative methods hold considerable value for understanding the processes by which the institutionalisation of evidence-based practice can occur. However, the decision to delegate and allocate healthcare resources to specific individual knowledge brokers requires consideration of its effectiveness and cost-effectiveness, in relation to competing priorities that are known to effectively improve health outcomes elsewhere in the health system. We contend that there are design-based solutions for overcoming the challenges in empirically studying the effectiveness of implementation strategies. For example, the use of counterbalanced implementation study designs has been proposed as a solution to these challenges, requiring smaller sample sizes through the concurrent investigation of multiple implementation strategies across different health context areas [73,74].

Weekend allied health service provision presented a uniquely challenging contextual area to implement evidence into practice. Healthcare professionals tend to incorporate local “tacit” or “codified” knowledge [75] into resource allocation decisions [76], particularly when presented with recommendations to reduce weekend allied health in acute general medical and surgical wards where the evidence base was considered less clear [58]. These findings align with previous research reporting that disinvestment from weekend services in these wards has been perceived as a threat to professional identity [77]. Conversely, recommendation to increase weekend services in subacute rehabilitation units was met with enthusiasm where this aligned with previously held attitudes, beliefs, and values [58]. Future research on these knowledge translation strategies is therefore needed across multiple contextual areas to determine the interaction effect between these strategies and context areas [73,74]. Further, an updated knowledge broker strategy with greater internal organisational relationships, which incorporates a component of cognitive debiasing to assist managers and policy makers when confronted with evidence that conflicts with current service delivery models, might reduce potential barriers to change and facilitate active implementation of evidence into practice [78-80].

Conclusions

Owing to a lack of power at the cluster level, this trial was unable to determine whether the use of a webinar-based knowledge broker was more effective than dissemination of recommendations or usual practice for promoting evidence-informed healthcare management decisions for inpatient weekend allied health services. The implication of this research is that more intense and interactive strategies for implementing evidence into practice do not always enable changes in resource allocation. Future research is needed across multiple contextual areas of healthcare recommendations to understand the context dependency of implementation strategy success, using counterbalanced implementation study deigns.

Cluster-level per day values (blinded).

(XLSX)

Click here for additional data file.

Ward-level per day values (blinded).

(XLSX)

Click here for additional data file.

Study protocol.

(PDF)

Click here for additional data file.

CONSORT extension for cluster trials.

(DOCX)

Click here for additional data file.

Standards for Reporting Implementation Studies statement (StaRI).

(DOCX)

Click here for additional data file.

Weekend allied health recommendation.

(PDF)

Click here for additional data file.

15 Mar 2021

Dear Dr Sarkies,

Thank you for submitting your manuscript entitled "Effectiveness of knowledge brokering and recommendation dissemination on resource allocation decisions: cluster randomised controlled trial" for consideration by PLOS Medicine.

Your manuscript has now been evaluated by the PLOS Medicine editorial staff and I am writing to let you know that we would like to send your submission out for external peer review.

However, before we can send your manuscript to reviewers, we need you to complete your submission by providing the metadata that is required for full assessment. To this end, please login to Editorial Manager where you will find the paper in the 'Submissions Needing Revisions' folder on your homepage. Please click 'Revise Submission' from the Action Links and complete all additional questions in the submission questionnaire.

Please re-submit your manuscript within two working days, i.e. by March 18, 2021.

Login to Editorial Manager here: https://www.editorialmanager.com/pmedicine

Once your full submission is complete, your paper will undergo a series of checks in preparation for peer review. Once your manuscript has passed all checks it will be sent out for review.

Feel free to email us at plosmedicine@plos.org if you have any queries relating to your submission.

Kind regards,

Beryne Odeny

Associate Editor

PLOS Medicine

6 May 2021

Dear Dr. Sarkies,

Thank you very much for submitting your manuscript "Effectiveness of knowledge brokering and recommendation dissemination on resource allocation decisions: cluster randomised controlled trial" (PMEDICINE-D-21-01199R1) for consideration at PLOS Medicine.

Your paper was evaluated by a senior editor and discussed among all the editors here. It was also discussed with an academic editor with relevant expertise, and sent to independent reviewers, including a statistical reviewer. The reviews are appended at the bottom of this email and any accompanying reviewer attachments can be seen via the link below:

[LINK]

Considering these reviews, we would be grateful if you could please revise your manuscript to respond to comments raised by reviewers. We would strongly recommend that you pay special attention to the statistical reviewers’ comments regarding your randomization procedures and would suggest toning down your conclusions. Please note that this is not a guarantee that we will accept the manuscript and that further consideration is dependent on the submission of a manuscript that addresses all reviewer concerns. We will carefully review your manuscript upon revision, so please ensure that your revision is thorough.

Please also check the guidelines for revised papers at http://journals.plos.org/plosmedicine/s/revising-your-manuscript for any that apply to your paper. In your rebuttal letter you should indicate your response to the reviewers' and editors' comments, the changes you have made in the manuscript, and include either an excerpt of the revised text or the location (eg: page and line number) where each change can be found. Please submit a clean version of the paper as the main article file; a version with changes marked should be uploaded as a marked up manuscript.

In addition, we request that you upload any figures associated with your paper as individual TIF or EPS files with 300dpi resolution at resubmission; please read our figure guidelines for more information on our requirements: http://journals.plos.org/plosmedicine/s/figures. While revising your submission, please upload your figure files to the PACE digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at PLOSMedicine@plos.org.

We expect to receive your revised manuscript by May 27 2021 11:59PM. Please email us (plosmedicine@plos.org) if you have any questions or concerns.

***Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.***

We ask every co-author listed on the manuscript to fill in a contributing author statement, making sure to declare all competing interests. If any of the co-authors have not filled in the statement, we will remind them to do so when the paper is revised. If all statements are not completed in a timely fashion this could hold up the re-review process. If new competing interests are declared later in the revision process, this may also hold up the submission. Should there be a problem getting one of your co-authors to fill in a statement we will be in contact. YOU MUST NOT ADD OR REMOVE AUTHORS UNLESS YOU HAVE ALERTED THE EDITOR HANDLING THE MANUSCRIPT TO THE CHANGE AND THEY SPECIFICALLY HAVE AGREED TO IT. You can see our competing interests policy here: http://journals.plos.org/plosmedicine/s/competing-interests.

Please use the following link to submit the revised manuscript:

https://www.editorialmanager.com/pmedicine/

Your article can be found in the "Submissions Needing Revision" folder.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Please ensure that the paper adheres to the PLOS Data Availability Policy (see http://journals.plos.org/plosmedicine/s/data-availability), which requires that all data underlying the study's findings be provided in a repository or as Supporting Information. For data residing with a third party, authors are required to provide instructions with contact information for obtaining the data. PLOS journals do not allow statements supported by "data not shown" or "unpublished results." For such statements, authors must provide supporting data or cite public sources that include it.

We look forward to receiving your revised manuscript.

Sincerely,

Beryne Odeny,

PLOS Medicine

plosmedicine.org

-----------------------------------------------------------

Requests from the editors:

- Please revise your title to indicate that this is an implementation science study. Your title must be nondeclarative and not a question. It should begin with main concept if possible. For example, please place the study design ("A cluster randomized controlled implementation trial,") in the subtitle (i.e., after a colon).

- Abstract summary - At this stage, we ask that you reformat your non-technical Author Summary. The Author Summary should immediately follow the Abstract in your revised manuscript. This text is subject to editorial change and should be distinct from the scientific abstract. The summary should be accessible to a wide audience that includes both scientists and non-scientists. Please see our author guidelines for more information: https://journals.plos.org/plosmedicine/s/revising-your-manuscript#loc-author-summary.

- Abstract:

1. Please structure your abstract using the PLOS Medicine headings (Background, Methods and Findings, Conclusions).

2. Please combine the Methods and Findings sections into one section, “Methods and findings”. Please ensure that all numbers presented in the abstract are present and identical to numbers presented in the main manuscript text.

3. Please quantify the main results (with p values in addition to 95% CI).

4. Please include a summary of adverse events if these were assessed in the study.

- When completing the CONSORT checklist, please use section and paragraph numbers, rather than page numbers.

- In addition to the CONSORT checklist please ensure that your implementation research is reported according to Standards for Reporting Implementation Studies statement (STARI). The STARI guidelines can be found here: https://www.equator-network.org/reporting-guidelines/stari-statement/

- Please clearly specify and report your implementation strategies. Consider using the guidelines published by Proctor et al. to improve your reporting: Proctor EK, Powell BJ, McMillen JC: Implementation strategies: recommendations for specifying and reporting. Implement Sci 2013, 8:139

- To improve the clarity and definition of your primary implementation outcomes, please use standard implementation science terminology such as adoption, fidelity, penetration, sustainability etc.

- Please present p-values along with 95% CI in the tables and main text. Please specify the statistical test used to derive p values

- Please use the "Vancouver" style for reference formatting and see our website for other reference guidelines https://journals.plos.org/plosmedicine/s/submission-guidelines#loc-references. Please ensure that weblinks are current and accessible

Comments from the reviewers:

Reviewer #1: The evaluation of knowledge brokering is important to contribute to the knowledge translation literature. As the authors point out there have been relatively few empirical studies conducted to date evaluating the impact of knowledge brokering on the use of evidence-informed practices and/or recommendations. I enjoyed reading this article and believe, with revisions, it will be an important contribution to the literature. However, I have a few concerns that I feel are important to be addressed prior to publication.

Major Concerns:

1) greater justification for the use of a cluster randomized controlled trial given previous suggestions in the literature to avoid the use of such a design to evaluate knowledge brokering.

2) More detail needs to be provided about the knowledge brokering intervention as this is very sparse.

3) However, my understanding from what is written about the knowledge brokering intervention is that in entailed interactive webinars. While this is certainly one activity that knowledge brokers engage in, if this was the only activity conducted during this study, I would not classify this as knowledge brokering. As the authors point out in the discussion, knowledge brokering involves building relationships with end users and considerable interaction. It does not appear from the description provided that these activities occurred, which might suggest that what was actually evaluated in this study was webinars as a knowledge translation strategy delivered by a knowledge broker.

4) Also, previous research has suggested that knowledge brokering likely needs to be implemented for more than 12 months to be effective, so perhaps some more justification can be provided for why a 12 month intervention was implemented in this study.

5) A significant challenge for this study is the extent to which it is under powered to detect even a large difference. I have two issues here. The sample size calculation is based on a very large effect (50% alignment with the allied health allocation recommendation). Justification from previous knowledge brokering studies for such a large effect should be provided. However, I am not certain that there is evidence to suggest that knowledge brokering evaluations in the past has found such large effects, in which case, a much more conservative effect size should have been used in the sample size calculation. The challenge this creates is that the required sample size would be much larger than it already is, which makes this study even more under-powered.

6) With the study being so significantly under-powered, the conclusion is worded much too strongly, that knowledge brokering is no more effective than dissemination of recommendations or standard practice. Given the lack of power to detect a difference, the only conclusion that can be drawn is that there was insufficient power to observe statistically significant differences between groups.

7) The ceiling effect in this study requires more attention in the discussion. While a regression analysis was conducted to take into account differences at baseline with half of the knowledge brokering group already being in alignment with the recommendations, this may not adequately adjust for these baseline differences. It is also worth noting that the knowledge brokering group had more highly educated participants than the other two groups, and could this explain the baseline difference in the primary outcome?

8) I believe a much more developed discussion about the lessons learned about what went wrong in this trial would be extremely helpful in helping others fall in to the same challenges in evaluating knowledge brokering. This includes: appropriate designs for evaluation, a more fulsome knowledge brokering intervention implemented for a longer period of time, ensuring it is feasible to be adequately powered to detect statistically significant differences, and ensuring the outcome is not present prior to intervention. A comprehensive discussion of these issues, drawing from what is currently known in the literature, with suggestions on how to avoid these pitfalls, will provide an important contribution to the literature.

Minor comments:

1) In the abstract it is stated the study occurred between Feb 2018 and Jan 2020, but in the Results section of the paper, it is stated data collection occurred until April 2020.

2) Provide justification for why at the ward level, partial alignment with the recommendations was not included as an option for measuring the outcome.

3) It is stated in the methods that an intention to treat analysis was conducted. However, 4 organizations either did not provide any data or follow-up data and therefore were not included in the analysis. This is not consistent with intention to treat analysis. For an intention to treat analysis to have been conducted, those organizations that did not provide data should have still been analyzed in the group to which they were allocated.

4) I may have missed it but I did not see statements as to there being statistically significant differences between groups at baseline. From the data provided it appears as though there could be.

5) there is a formatting issue with Ref #30

Reviewer #2: This is an interesting cluster randomised controlled trial on the effectiveness of knowledge brokering and recommendation dissemination on resource allocation decisions. However, the trial was poorly designed and conducted with quite a few major issues needing attention.

1) Cluster design. Throughout the paper, there is no detailed description of the cluster RCT design especially on cluster size. Also It's very confusing as to what was ultimately randomised? decision makers or wards?

2) Sample size calculation. On page 10 and 11, it said "a sample size of 25 clusters per group..." but in the results it said 15 clusters per group. Also, cluster size or ICC was never mentioned. And, why effect size is on cluster level rather than paticipants level? In short, the sample size calculation was very confusing and not informative therefore not adequate at all.

3) Primary outcome. It says on page 9 "The primary outcome of interest was whether weekend allied health service provision at both the cluster- and ward-level aligned with the recommendations at 12-months follow up". However, wards were not in the sample size calculation so not powered therefore should not be used as a primary outcome.

4) Intention to treat. It's claimed on page 10 "Analysis was undertaken according to intention-to-treat principles...". However, as shown in Figure 1 and table 1, the randomisation was disrupted at cluster level with error and empty clusters and also at healthcare decision maker level as shown in table 1 with huge imbalance. Basically, the trial analysis was not intention to treat at all, neither at cluster level nor paticipant level. The imbalance at both levels mean the trial was poor conducted and lost the purpose of effective randomisation. It basically becomes an observational study.

5) Statistical analysis. As the trial was not balanced at both levels, the simple summary statistics as shown in table 1-4 becomes inadequate and potentially misleading. Multivariable analysis adjusted for all confounders is essentially needed however it's not done or seen at all in results section although vaguely mentioned in the stats methods section. All these methodological inadequacies can lead to unreliable and unbelievable results and conclusions.

6) Randomisation. On page 10, not clear what was randomised? clusters or healthcare managers?

Reviewer #3: I wish to thank the editor for their invitation to review the manuscript, titled "Effectiveness of knowledge brokering and recommendation dissemination on resource allocation decisions: cluster randomised controlled trial." The aim of this study was to determine the effectiveness of two research implementation strategies to facilitate evidence-informed healthcare management decisions for the provision of inpatient weekend allied health services. Given that I am not a statistician and I would recommend the editors consider a statistical review prior to publication. I present my concerns in the order they appeared:

1. The Title: I find it a bit confusing and not specific enough

2. The abstract: Might want to be specific as to which were the two implementation strategies used.

3. The background section they might want to describe in more detail what is meant by "the dissemination of of evidence-based practice recommendations " (line 114-115), and the literature behind this strategy.

4. How can the participants be blinded to receiving the support of a knowledge broker?

5. How was the COM-B model used?

6. Discussion: How do their finding relate to what previous researchers have found?

7. What are the implications to the field?

Reviewer #4: This study tried to address an important issue of the impact of evidence dissemination and knowledge brokering on implementation of health system evidence. It is a well designed cluster randomized trial, which is a strength. However there are a few significant shortcomings that affect the validity and usefulness of its conclusions.

-the interventions seem to be multi component and complex. The evidence-based recommendation states the desired objective but suggested complex activities to managers in order to achieve them. Activities such as development of a business case, identification of local priorities, engagement of diverse stakeholders including allied health professionals, nurses, and even patients in the process, and development of communication and negotiation channels with staff. It seems unlikely that email dissemination of an evidence-based guideline or one or two online webinars by the knowledge broker would be sufficient to change the behavior of hospital managers to engage in these complex and resource-intensive activities.

-Since the authors did not report any information regarding the adoption and implementation of such activities (e.g how many managers initiated those activities, how many actually implemented them in their routine practice, how many were successful in engaging staff in planning and priority setting, etc), it is difficult to attribute the study findings to the lack of effectiveness or unsuccessful implementation. This is a widespread shortcoming in implementation studies, where the emphasis is on comparative effectiveness rather than investigating the conditions that facilitate or impede utilization and ultimate effectiveness of an implementation strategy. Ideally, a mixed methods study would provide better opportunities to address the contextual and procedural complexities of these interventions. But even as a proper RCT, a more comprehensive assessment of implementation outcomes was needed.

-ward-level analysis of alignment to the guidelines at study arms (table 4) shows that the number of wards with missing status increased dramatically at follow up in knowledge broker and dissemination arms. This along with the lack of engagement of sites in knowledge broker meetings implies that there were challenges in preserving the adherence of hospital mangers to the interventions. More clarification is needed regarding the potential reasons for this lack of adherence and the increase in non-response in intervention arms.

Any attachments provided with reviews can be seen via the following link:

[LINK]

3 Jun 2021

Submitted filename: Response to reviewer comments 02062021.docx

Click here for additional data file.

14 Jul 2021

Dear Dr. Sarkies,

Thank you very much for submitting your manuscript "Effectiveness of knowledge brokering and recommendation dissemination for influencing healthcare resource allocation decisions: A cluster randomised controlled implementation trial" (PMEDICINE-D-21-01199R2) for consideration at PLOS Medicine.

Your paper was evaluated by a senior editor and discussed among all the editors here. It was also discussed with an academic editor with relevant expertise, and sent to independent reviewers, including a statistical reviewer. The reviews are appended at the bottom of this email and any accompanying reviewer attachments can be seen via the link below:

[LINK]

In light of these reviews, I am afraid that we will not be able to accept the manuscript for publication in the journal in its current form, but we would like to consider a revised version that addresses the reviewers' and editors' comments. Please pay particular attention to comments from reviewers #2 and #4. Obviously we cannot make any decision about publication until we have seen the revised manuscript and your response, and we plan to seek re-review by one or more of the reviewers.

In revising the manuscript for further consideration, your revisions should address the specific points made by each reviewer and the editors. Please also check the guidelines for revised papers at http://journals.plos.org/plosmedicine/s/revising-your-manuscript for any that apply to your paper. In your rebuttal letter you should indicate your response to the reviewers' and editors' comments, the changes you have made in the manuscript, and include either an excerpt of the revised text or the location (eg: page and line number) where each change can be found. Please submit a clean version of the paper as the main article file; a version with changes marked should be uploaded as a marked up manuscript.

In addition, we request that you upload any figures associated with your paper as individual TIF or EPS files with 300dpi resolution at resubmission; please read our figure guidelines for more information on our requirements: http://journals.plos.org/plosmedicine/s/figures. While revising your submission, please upload your figure files to the PACE digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at PLOSMedicine@plos.org.

We expect to receive your revised manuscript by Aug 04 2021 11:59PM. Please email us (plosmedicine@plos.org) if you have any questions or concerns.

***Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.***

We ask every co-author listed on the manuscript to fill in a contributing author statement, making sure to declare all competing interests. If any of the co-authors have not filled in the statement, we will remind them to do so when the paper is revised. If all statements are not completed in a timely fashion this could hold up the re-review process. If new competing interests are declared later in the revision process, this may also hold up the submission. Should there be a problem getting one of your co-authors to fill in a statement we will be in contact. YOU MUST NOT ADD OR REMOVE AUTHORS UNLESS YOU HAVE ALERTED THE EDITOR HANDLING THE MANUSCRIPT TO THE CHANGE AND THEY SPECIFICALLY HAVE AGREED TO IT. You can see our competing interests policy here: http://journals.plos.org/plosmedicine/s/competing-interests.

Please use the following link to submit the revised manuscript:

https://www.editorialmanager.com/pmedicine/

Your article can be found in the "Submissions Needing Revision" folder.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Please ensure that the paper adheres to the PLOS Data Availability Policy (see http://journals.plos.org/plosmedicine/s/data-availability), which requires that all data underlying the study's findings be provided in a repository or as Supporting Information. For data residing with a third party, authors are required to provide instructions with contact information for obtaining the data. PLOS journals do not allow statements supported by "data not shown" or "unpublished results." For such statements, authors must provide supporting data or cite public sources that include it.

We look forward to receiving your revised manuscript.

Sincerely,

Beryne Odeny,

PLOS Medicine

plosmedicine.org

-----------------------------------------------------------

Comments from the reviewers:

Reviewer #1: The authors have comprehensively addressed all of my suggestions.

Reviewer #2: Many thanks authors for their great effort to improve the manuscript. I can appreciate the challenges in this implementation trial that investigators have faced and the fact that investigators tried their best to deal with the situations, however, I am still not convinced/statisfied with the response and revision. Remaing issues:

1) This is not an "intention to treat" analysis. There are 3 clusters without 12-month follow up (primary) so not in the main analysis, as shown in Figure 1. Also multiple imputation for missing data is not ITT analysis. So far, it was effectively performed with a complete case analysis plus sensitivity analysis for missing values (imputation).

2) Design issue. Cluster RCT without pre-specified or uncertain cluster size? Sample size only for cluster level but not for ward level? Primary outcome for both cluster (powered) and ward (not powered)? All these deviate from the definition of a cluster RCT. Perhaps this is the pragmatic nature of implementation trial but how to address the methodological issues and potential biases which could impact on the results?

3) Statistical analysis. Again, the trial was not balanced at both levels, the simple summary statistics becomes inadequate and potentially misleading. Multivariable analysis comprehensively adjusted for all confounders is essentially needed. Otherwise, the results may be subject to scrutiny due to design, sample size and trial conduct issues.

Reviewer #3: The authors have addressed all the comments in a satisfactory manner.

Reviewer #4: The authors tried to address most of the comments by reviewers. The revised manuscript is better organized and is more informative.

My main remaining concern, which is now more prominent since the term 'implementation' was added to the title, is lack of a guiding framework and evaluation metrics related to the implementation. Alignment of practice with the guidelines does not capture the complexities of the implementation. It seems that some data have already been collected that should be added to the Results.

-For example webinar attendance rates at ward and organization levels may address the notion of 'adoption' of the intervention by managers. Furthermore, the authors can include ward-level attendance data in the regression analyses to assess its impact on study outcomes.

-Also, given the large non-response, it is important to provide some contextual information about the characteristics of non-respondents, in comparison to the participating organizations. This comparison may reflect the notion of 'reach' (who was missed?), using RE-AIM terminology.

-Providing some temporal participation and adoption statistics (how did the participation differ during the first few months vs later dates), if available, would address the notion of sustainability ('maintenance' in RE-AIM).

Any attachments provided with reviews can be seen via the following link:

[LINK]

22 Jul 2021

Submitted filename: Response to reviewer comments 230721.docx

Click here for additional data file.

13 Sep 2021

Dear Dr. Sarkies,

Thank you very much for re-submitting your manuscript "Effectiveness of knowledge brokering and recommendation dissemination for influencing healthcare resource allocation decisions: A cluster randomised controlled implementation trial" (PMEDICINE-D-21-01199R3) for review by PLOS Medicine.

I have discussed the paper with my colleagues and the academic editor and it was also seen again by two reviewers. I am pleased to say that provided the remaining editorial and production issues are dealt with we are planning to accept the paper for publication in the journal.

The remaining issues that need to be addressed are listed at the end of this email. Any accompanying reviewer attachments can be seen via the link below. Please take these into account before resubmitting your manuscript:

[LINK]

***Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.***

In revising the manuscript for further consideration here, please ensure you address the specific points made by each reviewer and the editors. In your rebuttal letter you should indicate your response to the reviewers' and editors' comments and the changes you have made in the manuscript. Please submit a clean version of the paper as the main article file. A version with changes marked must also be uploaded as a marked up manuscript file.

Please also check the guidelines for revised papers at http://journals.plos.org/plosmedicine/s/revising-your-manuscript for any that apply to your paper. If you haven't already, we ask that you provide a short, non-technical Author Summary of your research to make findings accessible to a wide audience that includes both scientists and non-scientists. The Author Summary should immediately follow the Abstract in your revised manuscript. This text is subject to editorial change and should be distinct from the scientific abstract.

We expect to receive your revised manuscript within 1 week. Please email us (plosmedicine@plos.org) if you have any questions or concerns.

We ask every co-author listed on the manuscript to fill in a contributing author statement. If any of the co-authors have not filled in the statement, we will remind them to do so when the paper is revised. If all statements are not completed in a timely fashion this could hold up the re-review process. Should there be a problem getting one of your co-authors to fill in a statement we will be in contact. YOU MUST NOT ADD OR REMOVE AUTHORS UNLESS YOU HAVE ALERTED THE EDITOR HANDLING THE MANUSCRIPT TO THE CHANGE AND THEY SPECIFICALLY HAVE AGREED TO IT.

Please ensure that the paper adheres to the PLOS Data Availability Policy (see http://journals.plos.org/plosmedicine/s/data-availability), which requires that all data underlying the study's findings be provided in a repository or as Supporting Information. For data residing with a third party, authors are required to provide instructions with contact information for obtaining the data. PLOS journals do not allow statements supported by "data not shown" or "unpublished results." For such statements, authors must provide supporting data or cite public sources that include it.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript.

Please note, when your manuscript is accepted, an uncorrected proof of your manuscript will be published online ahead of the final version, unless you've already opted out via the online submission form. If, for any reason, you do not want an earlier version of your manuscript published online or are unsure if you have already indicated as such, please let the journal staff know immediately at plosmedicine@plos.org.

If you have any questions in the meantime, please contact me or the journal staff on plosmedicine@plos.org.

We look forward to receiving the revised manuscript by Sep 20 2021 11:59PM.

Sincerely,

Beryne Odeny,

Associate Editor

PLOS Medicine

plosmedicine.org

------------------------------------------------------------

Requests from Editors:

Thank you for responding to editorial requests. Before we proceed, please address the following comments:

1) Please highlight in the conclusion (both abstract and main text), that you did not observe a difference imparted by the implementation strategies, owing to lack of power. For example, you could state “owing to lack of power, our study was unable to find whether…”

2) In the author summary, please revise line 110... should this be “are” instead of “are represent”?

3) In the abstract, last sentence of “Methods and Findings” please clearly highlight the limitations of the study. The statement, “Limitations of this study include…” can be useful.

4) References #55, #57: please include access dates for the weblinks, e.g., Accessed July 15, 2021.

Comments from Reviewers:

Reviewer #2: The authors have addressed all my concerns professionally. I am satisfied with the response and revision. No further issues needing attention. Thanks.

Reviewer #4: I'd like to thank the authors for their efforts to address the comments.

I think that the researchers could use more suitable implementation theoretical frameworks to guide the design of the study, as well as more relevant and sensitive implementation outcomes to capture the impact of individual, team-level, and organizational factors that could determine the success/failure of the intervention. Despite this, I do not object the publication of this study, given its rigorous design and potentially important implications for further research on knowledge broker interventions.

However, I re-emphasize that, it should be clearly stated in the study limitations that the observed lack of evidence to support knowledge broker intervention could simply be due the fact that the intervention was not successfully implemented (for which insufficient evidence was provided). I recommend that the authors suggest potential solutions for more theoretically-informed assessment of the implementation of KB interventions.

Any attachments provided with reviews can be seen via the following link:

[LINK]

17 Sep 2021

Submitted filename: Response to comments 140921.docx

Click here for additional data file.

4 Oct 2021

Dear Dr Sarkies,

On behalf of my colleagues and the Academic Editor, Dr. Elvin Hsing Geng, I am pleased to inform you that we have agreed to publish your manuscript "Effectiveness of knowledge brokering and recommendation dissemination for influencing healthcare resource allocation decisions: A cluster randomised controlled implementation trial" (PMEDICINE-D-21-01199R4) in PLOS Medicine.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. Please be aware that it may take several days for you to receive this email; during this time no action is required by you. Once you have received these formatting requests, please note that your manuscript will not be scheduled for publication until you have made the required changes.

In the meantime, please log into Editorial Manager at http://www.editorialmanager.com/pmedicine/, click the "Update My Information" link at the top of the page, and update your user information to ensure an efficient production process.

PRESS

We frequently collaborate with press offices. If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximise its impact. If the press office is planning to promote your findings, we would be grateful if they could coordinate with medicinepress@plos.org. If you have not yet opted out of the early version process, we ask that you notify us immediately of any press plans so that we may do so on your behalf.

We also ask that you take this opportunity to read our Embargo Policy regarding the discussion, promotion and media coverage of work that is yet to be published by PLOS. As your manuscript is not yet published, it is bound by the conditions of our Embargo Policy. Please be aware that this policy is in place both to ensure that any press coverage of your article is fully substantiated and to provide a direct link between such coverage and the published work. For full details of our Embargo Policy, please visit http://www.plos.org/about/media-inquiries/embargo-policy/.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Thank you again for submitting to PLOS Medicine. We look forward to publishing your paper.

Sincerely,

Beryne Odeny

Associate Editor

PLOS Medicine