Supporting Policy In health with Research: an Intervention Trial (SPIRIT)-protocol for a stepped wedge trial.

INTRODUCTION: Governments in different countries have committed to better use of evidence from research in policy. Although many programmes are directed at assisting agencies to better use research, there have been few tests of the effectiveness of such programmes. This paper describes the protocol for SPIRIT (Supporting Policy In health with Research: an Intervention Trial), a trial designed to test the effectiveness of a multifaceted programme to build organisational capacity for the use of research evidence in policy and programme development. The primary aim is to determine whether SPIRIT results in an increase in the extent to which research and research expertise is sought, appraised, generated and used in the development of specific policy products produced by health policy agencies. METHODS AND ANALYSIS: A stepped wedge cluster randomised trial involving six health policy agencies located in Sydney, Australia. Policy agencies are the unit of randomisation and intervention. Agencies were randomly allocated to one of three start dates (steps) to receive the 1-year intervention programme, underpinned by an action framework. The SPIRIT intervention is tailored to suit the interests and needs of each agency and includes audit, feedback and goal setting; a leadership programme; staff training; the opportunity to test systems to assist in the use of research in policies; and exchange with researchers. Outcome measures will be collected at each agency every 6 months for 30 months (starting at the beginning of step 1). ETHICS AND DISSEMINATION: Ethics approval was granted by the University of Western Sydney Human Research and Ethics Committee HREC Approval H8855. The findings of this study will be disseminated broadly through peer-reviewed publications and presentations at conferences and used to inform future strategies. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

RCT Entities:   Population Interventions Outcomes

Introduction

Internationally, there is a growing commitment to better use of evidence from research in policy, with a view to improving outcomes and optimising resource allocation.1–4 As a result, considerable attention has recently been given to assisting government agencies to better use evidence from research in policy and programme design.

Health policies and programmes have been a particular focus, and a number of approaches to building skills and organisational capacity to use evidence from research have been developed.5 6 The target for these programmes is government agencies who develop national or regional policy and programmes rather than organisations that provide health services and clinical care. Despite the proliferation of such programmes and their cost to government, there have been few examinations of their impact on the use of research. A recent comprehensive review7 found only five studies examining the impact of strategies to increase the use of evidence in policy and since that time, to the best of our knowledge, only one other relevant study protocol has been published.8

The paucity of trials may be partly a result of the considerable challenges in evaluating the impact of strategies designed to increase the use of research in policy and programme design. Agencies involved in policy and programme development, even within health, are heterogeneous—they will be different in remit, size, proximity to central government and in their need for and current capacity to use evidence from research. These agencies are also often complex organisational systems and may respond very differently to the same strategy programme. Effective strategies will therefore most likely be multicomponent and highly tailored to meet the needs of different agencies. Study designs will need to be capable of assessing complex interventions and of carefully measuring the context in which the programme is delivered; it is most likely that the programme will need to be delivered at the level of the health policy agency, which means a cluster design will be required. It may not be feasible to recruit and intervene with sufficient agencies to power a randomised controlled trial. A further challenge lies in the lack of established validated measures of the use of research in policy and programme development; in a trial context, an objective measure of the impact of the programme is required.

This paper describes SPIRIT (Supporting Policy In health with Research: an Intervention Trial), an evaluation of the impact of a multifaceted programme designed to build the capacity of health policy agencies[i] to use research in policy and programme development. The trial is based on the SPIRIT Action Framework described in detail elsewhere; the Framework hypothesises that the use of research in policy is mediated by the capacity of the organisation to use research (including the extent to which the agency values research, has systems in place to use research and the skills of its staff). It predicts that greater capacity will lead to more research engagement actions (demonstrated by accessing and appraising existing research, generating new research and interacting with researchers) and that, in turn, more engagement actions will result in a greater use of research evidence.

SPIRIT is a stepped wedge trial evaluating a complex intervention and using an objective measure for its primary outcome. It extends previous studies which often evaluated only one strategy,10–12 did not use objective outcome measures10 11 13 and were unable to include a detailed process evaluation.10–14 In addition, SPIRIT is the only trial designed to bring about organisational change in policy and programme agencies and that has the capacity to assess the long-term impact of the programme (up to 18 months after its completion).

Aims

Objective and aims

The objective of the trial is to examine what impact the SPIRIT intervention has and how it works in the field. Our primary outcome measure is an objective assessment made by an expert panel, our secondary measures utilise self-informant or key informant reports and our process evaluation includes coded and descriptive accounts of the intervention delivery and participation made during observations of sessions, semistructured interviews and written participant feedback. We will examine these data across the components of the SPIRIT Action Framework (capacity, research engagement actions and use of research evidence); this will enable us to examine the impact of the SPIRIT intervention programme and to test the causal relationships hypothesised by the model. By capacity, we mean the extent to which the agency values research, has systems in place to use research and the skills of its staff. By research engagement actions, we mean the extent to which research is accessed, appraised and generated, and where interaction with researchers occurs, in relation to the development of a specific policy or programme document. By research use, we mean the extent to which research is sought and used in developing the policy or programme document, taking into account barriers and facilitators.

The specific aims are:

Primary outcome

To determine, using an objective measure, whether the SPIRIT intervention results in an increase in the extent to which research engagement actions are undertaken, and research is used in the development of policy and programme documents.

Secondary outcome

To determine, using the self-report of staff members, whether the SPIRIT intervention results in an increase in:

The capacity of health policy agencies to use research in terms of: (A) the value placed on the use of research evidence by individual policymakers and by the organisation; (B) the confidence of policymakers in undertaking research actions and using research and (C) the systems and tools the organisation has in place to support research use;

Research engagement actions;

The use of research.

Process evaluation: The aim of the process evaluation is to complement the outcome measures by exploring how and why the intervention worked or did not work in different contexts. This includes descriptions of:

The implementation of the intervention programme, including the delivery of the essential elements (fidelity);

Participation in and responses to the intervention (eg, how people interacted with the programme, how they evaluated different aspects of it and what sort of change they experienced, if any);

Contextual factors that might have affected the programme and responses to it (eg, agency priorities, practice norms, other training or organisational change initiatives, relationships with external bodies, legislative reform).

Cost analysis

To document the cost of delivering the intervention.

Methods

SPIRIT started in October 2012, and at the time of writing, approximately one-half of the total intervention period of 30 months has been completed.

Trial design

SPIRIT is a stepped wedge cluster intervention trial (figure 2) involving six agencies. Two agencies are randomly assigned to each step. This design is a variation on a cross-over design where each intervention unit is measured in the control and intervention phases, except that the step wedge design involves a cross-over in one direction only. The key feature of this design is that the intervention is rolled out sequentially over a number of time periods, such that all trial units have received the intervention by the end of the trial, although the order in which they receive it is determined at random.15 16 Outcome measures are obtained at the same time in all sites, at baseline and after implementation in each site. This design was selected in accordance with the UK Medical Research Council17 recommendations because the intervention is delivered at the level of the health policy agency (cluster) and recruiting and intervening with a large enough number of policy agencies to power a traditional randomised controlled trial was unfeasible due to the resource-intensive nature of SPIRIT and its status as a new, untested programme.

Figure 2

SPIRIT design. LP, liaison person; SPIRIT, Supporting Policy In health with Research: an Intervention Trial.

Participants

Recruitment of health policy agencies

The inclusion and exclusion criteria for policy agencies were established prior to recruiting trial participants. An agency was eligible to participate in SPIRIT if: (A) a significant proportion of its work was in health policy or programme development; (B) at least 20 staff members were involved in policy or programme design, development or evaluation; and (C) it was located in Sydney (for ease of provision of the programme).

Government websites which list all New South Wales and Australian government health policy and programme agencies were used to identify potentially eligible agencies located in Sydney, New South Wales (where most of the major health policy agencies in the state are located). Information from the website of each agency was reviewed by members of the investigator team with policy experience to exclude those agencies without a significant focus on health and on policy or programme design, development or evaluation. Seventy-five agencies were classified as potentially eligible and email and/or phone contact was made with each agency to determine the number of relevant staff members. Following this step, the 16 agencies still regarded as potentially eligible were ranked to determine those with the greatest specific focus on health and the largest numbers of relevant staff. A visit was made by two of the authors (SR and NL) to the top six ranked agencies, all of whom agreed to take part. The agencies which agreed to participate in SPIRIT comprise a major centre within the New South Wales Ministry of Health and five agencies (four state and one national) that develop policy and programmes about specific aspects of health.

Intervention

Development of the intervention programme

The intervention programme is underpinned by the programme logic shown in figure 1.

Figure 1

The programme logic for Supporting Policy In health with Research: an Intervention Trial (SPIRIT).

The intervention was designed to be appropriate for policy agencies, taking into account that they are sophisticated, complex organisations with skilled staff and diverse policy priorities. The design of the programme was strongly influenced by cognitive behavioural theory,18 19 system science,20 21 organisational change theory22–25 and adult learning theories.26 27 We used these approaches, together with information from interviews with policymakers and experience from existing programmes in Australia28–30 and internationally,5 6 10 14 to identify factors that have been associated with significant individual or organisational change. Telephone interviews were then conducted with nine senior Australian health policymakers to canvass their views about the kinds of strategies being considered for inclusion in the intervention. They endorsed our approach, indicating that the proposed change strategies were acceptable and would most likely make a difference in improving the use of research evidence in policymaking.

Components of the intervention and all three measures were piloted in a policy agency that was not part of SPIRIT prior to the start of the study. This pilot broadly confirmed that the proposed methods were acceptable and appropriate and allowed fine tuning of some aspects of the intervention.

SPIRIT: the programme

The SPIRIT intervention is a six-component programme (table 1):

Table 1

Overview of the intervention components and subcomponents, their delivery mode and goals

Components	Intervention subcomponents/sessions	Format/mode	Goals
Audit, feedback and goal setting	Feedback forum*	Interactive feedback (from the preintervention measures) forum for executive staff. Opportunities for improvement are identified. Further feedback sessions are provided mid-intervention and at the conclusion of the trial	Leaders make informed decisions about opportunities for growing their agencies’ capacity to use research
Leadership programme	Leading organisational change* Supporting the use of evidence in the team*	Interactive forums for executive staff. Forum 1 presents feedback about the agency's use (see above). Forums 2 and 3 explore how leaders can build capacity. Forums provided by policy experts and leaders in knowledge exchange. Real world examples and collaborative problem-solving used	Leaders can identify modifiable barriers to research-informed policy/programme development and identify strategies for reducing them in their agencies
Agency support for research	Quarterly email endorsements from the participating agency CEO* Web CIPHER Tools and resources	CEO email to all staff supporting SPIRIT, promoting the use of research in agency and providing links to relevant resources Access via Web CIPHER to new research, evidence libraries and research expertise Provision of useful information at R4P sessions, such as key publications from leaders in knowledge exchange	1. Staff see that their senior leader values the use of research in policy/programme work and that they (staff) have increased regard for the value of research in policy/programme work2. Staff are given access to research resources3. Staff attending R4P events receive hardcopy resources to guide action responses
Opportunity to test systems for accessing research and reviews	A brokered rapid review or the development of an evaluation plan or a brokered analysis of local linked data†	Intervention officers and nominated staff negotiate the product requirements (research questions, timelines, deliverables, etc). Expert researchers develop the product or consult on its development. Agency staff may be as involved in the process of developing the product as they wish	Staff gain experience commissioning a review/an evaluation/a piece of linked data analysisAgencies have a research product that directly addresses a current priority or concern and use it to inform their policy/programme work
Research exchange	1. Research exchanges Δ 2. E-bulletins Δ	A tailored interactive form that brings together researchers and policymakers around a topic specified by the agency Receipt of a summary of recently published systematic reviews relevant to the agency's work	Staff receive accessible research information that directly addresses current priorities or concerns and use it to inform their policy/programme work
R4P symposia for staff	Using research in policy/programme development* Accessing and applying systematic reviews‡ Skills for appraising research‡ Policy and programme evaluation‡ Commissioning research‡	Educational workshops for all policy/programme staff provided by respected policy experts (session 1) and content experts (sessions 2 and 3)Use of real-world case examples (including agency cases), small group discussion and feedback and participant questions	Staff understand and value the role of research in informing policy/programme developmentStaff have increased knowledge and skills in research use in areas selected by the agency

*This is a standard subcomponent with some tailoring. For example, agency leaders edit pro forma emails (leadership endorsement), feedback focuses on that agency's measures and their response (feedback forum) and agency-nominated case examples and concerns are explored to increase relevance and applicability (all others).

†This is an entirely tailored subcomponent. For research exchanges, agencies identify the topic, the questions to be addressed and the delivery format (eg, provider type, group size, level of interactivity). For e-bulletins, they identify the topic, scope and target audience of the information. Agencies may choose three research exchanges or three e-bulletins or a mixture of both. For brokered services, agencies select one service and identify its focus, working with knowledge brokers to develop a product which meets their needs.

‡This is a selected subcomponent. Agencies are provided with a menu of options and select those that will best meet their needs. Agency case examples and concerns are explored to increase relevance and applicability.

CEO, chief executive officer; CIPHER, Centre for Informing Policy in Health with Evidence from Research; SPIRIT, Supporting Policy In Health with Research: an Intervention Trial.

Audit, feedback and goal setting: SPIRIT begins with the provision of a facilitated discussion of the findings from the agency's preintervention measures (see below) to the individuals that each agency nominates to be part of their ‘leaders’ group. One of the authors (SR) facilitates these sessions and encourages the leaders to reflect on and discuss the current strengths of the agency in using evidence from research, and opportunities for improvement based on the preintervention data. The discussion is used to (A) agree on priorities for change; (B) select the elective elements of the SPIRIT programme (see table 1) and (C) identify other actions outside the scope of the SPIRIT programme that the agency might wish to undertake to increase its use of evidence in policy.

Leadership programme: This component provides an opportunity for agency-nominated leaders to consider the value of research in their work, current approaches to knowledge exchange and how to bring about a greater use of evidence from research in their agency. It draws on the CFHCI EXTRA programme,5 and consists of two interactive workshops led by experts in knowledge exchange and policy development.

Agency support for research: This component is designed to strengthen the perception among staff that their organisation values the use of evidence and to provide tools for structural change. It includes (A) a quarterly email sent to all staff in each agency by the chief executive (CE). The email aims to demonstrate CE support for the use of evidence in the agency's work and is drafted by the research team and contextualised by the CE; (B) access to Web CIPHER (Centre for Informing Policy in Health with Evidence from Research), an online resource for policymakers where users can access research, hear from opinion leaders, share knowledge and debate ideas about the use of research in policymaking and programme development and (c) provision of relevant fact sheets and key publications.

Systems for accessing research, commissioning a review and analysing relevant data: This component gives agencies an opportunity to trial existing systems for helping agencies to use research evidence. The agency elects to trial the system for one of: a brokered rapid review of research evidence around a topic of interest; a brokered analysis of local data or the development of an evaluation framework related to an aspect of the agency's work. A tailored research product is produced for each agency as a result of this process.

Research access: In this component, agencies choose three occasions of facilitated access to research or researchers—this might be (A) a tailored interactive forum that brings together researchers and policymakers around a topic specified by the agency; or (B) receipt of a summary of recently published systematic reviews relevant to the agency's work.

Staff training and skill development: This component is designed to increase the value placed on the use of research in policy (symposium 1, received by all agencies) and to increase skills in areas selected by the agency (symposia 2–3, topics selected by the agency). The workshops are provided by individuals with expertise appropriate to the topic, include tailored case studies and draw on adult learning principles. Symposium 1 includes feedback on the current use of evidence from research by the agency and symposia 2 and 3 might address any of the following depending on the interests of the agency: accessing and applying systematic reviews; skills for appraising research; policy and programme evaluation (introduction to evaluation or evaluation in practice); working with researchers or commissioning research for policy and programme development.

The active phase of the intervention runs for 12 months with around one activity in each agency per month. There are no costs to agencies for any component, other than the opportunity cost associated with the time taken for their staff to participate. The CE of each participating agency was asked to nominate a senior member of staff to function as the liaison person (LP) and to be the primary point of contact for the SPIRIT team. An individual with extensive knowledge of brokering experience was appointed as the SPIRIT officer for each agency and worked with the LP to ensure that SPIRIT was provided effectively at that agency.

Timing of recruitment, intervention delivery and follow-up

As shown in figure 2, the SPIRIT programme runs for 12 months in each health policy agency. The total measurement period is 30 months with measures being collected at the beginning of the study period, and then every 6 months thereafter in each agency. There are thus six measurement points in total. The first measurement point is a preintervention measure for all agencies and the final measurement point is 6 months after the completion of the SPIRIT programme in the last randomised pair of agencies.

Outcomes

Primary outcome

The primary outcome is an objective measure (SAGE, described below) of whether the SPIRIT intervention results in an increase in the extent to which research engagement actions are undertaken and research is used in the development of policy and programme documents.

Secondary outcomes

To determine, using the self-report of staff members (SEER and ORACLe, described below), whether the SPIRIT intervention results in an increase in:

The capacity of health policy agencies to use research in terms of: (A) the value placed on the use of research evidence by individual policymakers and by the organisation; (B) the confidence of policymakers in undertaking the research actions and using research; and (C) the systems and tools the organisation has in place;

Research engagement actions;

The use of research.

Process evaluation: The third set of outcomes was collected as part of a process evaluation and describes the implementation of SPIRIT in health policy agencies including data around:

SPIRIT components selected by each agency;

Participants’ engagement;

The fidelity of delivery;

The cost of delivery.

Outcome measurement

All outcome measures used in SPIRIT have been developed specifically for the trial as we were unable to identify any suitable measures in the literature. All measures were extensively piloted prior to the start of the trial and papers outlining their development and validation will be published in peer-reviewed journals. Table 2 provides a summary of the primary and secondary study outcomes and the accompanying outcome measurement tools.

Table 2

SPIRIT outcome measures

Outcome	Level of assessment	Tool	Data collection method	Participants	Domains
Primary outcome measure
Use of research in the creation of policy documents	Policy document	SAGE	Interview (face-to-face or phone) plus collection of documentation	One or two policymakers involved in the development of the policy document being considered×four documents at each measurement point	Seeking research Research access efforts Appraisal of relevance Appraisal of quality Using research Conceptual research use Instrumental research use Tactical research use Imposed research use Influence of research Generating research Plans to generate or advocate new research Influences on research use in policy Barriers to research use Enablers of research use
Secondary outcome measures
Policymakers’ self-assessments of their research use capacity, actions and outcomes	Individual policymaker	SEER	Online survey	All policymakers from within participating agencies	Access to, and appraisal, generation and use of research Engagement with researchers Value placed on research Confidence in using research Perceived value organisation places on use of research Organisational tools and systems available to support research use
Organisational capacity to use research as measured by the existing tools and systems to support research use	Policy agency	ORACLe	Interview (face-to-face or phone) plus collection of documentation	One senior member of each policy agency, nominated by agency's leaders	Policies that encourage or mandate the examination of research in policy and programme development Tools and programmes to assist leaders of the organisation to actively support the use of research in policy and programme development Strategies to provide staff with training in using evidence from research in policy and in maintaining these skills Strategies to help staff to access existing research findings Methods to generate new research evidence to inform the organisation's work Methods to ensure adequate evaluations of the organisation's policies and programmes Strategies to strengthen research relationships

ORACLe, Organisational Research Access, Culture and Leadership; SAGE, Staff Assessment of enGagement with Evidence; SEER Seeking, Engaging with and Evaluating Research; SPIRIT, Supporting Policy In health with Research: an Intervention Trial.

Staff Assessment of enGagement with Evidence (SAGE) is an objective measure of the primary outcome—the extent to which agencies apply research findings in the development of the policy and programme documents they produce. An expert panel will be assembled to assign SAGE scores on the basis of interview data and document review. At each of the six data points, agencies are asked to provide the four policy documents finalised during the past 6 months that best represent the use of research in their policy and programme development work. The level of measurement is the policy document. Information about each document is collected via a structured qualitative interview with an individual who was heavily involved in the development of the document, nominated by the LP. The interview runs for approximately 1 h. A separate interview is conducted for each nominated document. The domains SAGE assesses are: (A) research engagement actions (accessing research, appraising research (for quality and relevance), generating new research or analysis and interacting with researchers)—each of these six dimensions receives a score from 0 to 9. Scores can be summed or reported separately; (B) outcome: research use (four types of research use are considered: instrumental, tactical, conceptual and imposed). Each of these four dimensions receives a score from 0 to 9. Scores can be summed or reported separately.

Seeking, Engaging with and Evaluating Research (SEER) measures individual policymakers’ capacities, research engagement actions and research use. SEER is administered via an online self-report survey to eligible policymakers identified by the LP. At each of the six measurement points, the LP from each agency provides a list of eligible policymakers to the research team.

Health policy or programme staff are regarded as eligible to complete SPIRIT measures if:

They write health policy documents or develop health programmes, or make or contribute significantly to policy decisions about health services, programmes or resourcing;

They are employed at a mid-level or higher in their agency;

They are over 18 years of age and consent to participate in the study.

Individual agency staff members are excluded if they are contractors and/or work across several of the participating agencies. All nominated policymakers are emailed an invitation to participate in the SPIRIT trial outcome measure in question along with standard participant information and consent forms. SEER measures (A) capacity (four subscales: predisposing factors related to individual values, perceptions of organisational values, perception of organisational systems and individual knowledge); (B) research use actions (accessing research, appraising, generating new research or analyses and interacting with researchers); and (C) research use (conceptual, instrumental, tactical and imposed research use) and extent of research use.

Organisational Research Access, Culture and Leadership (ORACLe) is used to assess each agency's capacity to use research findings. ORACLe data are collected via a structured qualitative interview with a senior staff member from each agency, nominated by the agency LP. The qualitative data are transcribed and used to assign scores to seven dimensions related to an organisation's capacity to use research. Two researchers separately score the responses (0=no, 1=yes—some or to a limited extent, 2=yes—very much so) of each participant to the 21 questions which make up the ORACLe scale. Inter-rater reliability has thus far been high (95%). Scores for each domain and a total score are then generated using an algorithm developed via a discrete choice experiment. This experiment harnessed the views of leaders in policy and knowledge exchange on the most important agency-level factors influencing the use of evidence.

ORACLe assesses agencies across the following seven domains: (A) processes that encourage or require the examination of research in policy and programme development; (B) tools and programmes to assist leaders of the organisation to actively support the use of research in policy and programme development; (C) strategies to provide staff with training in using evidence from research in policy and in maintaining these skills; (D) organisational strategies to help staff to access existing research findings; (E) methods to generate new research evidence to inform the organisation's work; (F) methods to ensure adequate evaluations of the organisation's policies and programmes; and (G) strategies to strengthen research relationships.

Process evaluation

A mixed methods process evaluation is conducted in parallel with the intervention in each of the six agencies. It collects data about, and analyses the interactions between, three domains:

The delivery of the intervention, including implementation of essential elements (fidelity). All intervention sessions are observed and audio recorded by process evaluation staff. Initial essential elements (ie, aspects of each intervention component that are considered critical to its success) are identified, and codes developed to monitor their delivery descriptively and, where possible, quantitatively. Essential elements and codes are tested in the early stages of the programme delivery, refined and further tested throughout the trial. Revised elements are applied retrospectively to earlier sessions to enable comparison across intervention sites.

Participation in and responses to the intervention: Participation attributes are recorded during session observations, and self-report evaluation forms are used to collect feedback on the relevance, applicability and value of each intervention session from the participants’ point of view, including whether they are likely to use knowledge/skills developed in the session. Following the intervention, five to seven semi-structured interviews are conducted in each site which explore participants’ responses to the programme, including changes in relation to individual and organisational capacity to use research, research engagement actions and actual research use.

Contextual factors that might have affected the intervention and responses to it: Session observations are used to collect data on participants’ views and experiences of research use in their work. Preintervention interviews with participants explore workplace priorities and culture, how they perceive their organisations’ support for research use, the role of research within the mix of other information and any other contextual factors that affect their use of research or may affect how they will engage with and respond to the intervention, including exposure to other research use drivers.

The process evaluation will complement the outcome measures by describing how and why the intervention worked or did not work, including why the same intervention strategies may have had different effects in different contexts.31–33 In addition, it will aid future programme improvement by identifying beneficial design and implementation strategies and suggesting contextually responsive programme adaptation for different recipients and settings. The full protocol for the SPIRIT process evaluation is described in detail elsewhere.

Cost analysis

A cost analysis is being conducted as part of SPIRIT because: the intervention will include non-market goods for which standardised cost estimates are less likely to exist. This cost analysis is conducted from the perspective of the intervention provider; detailed data on the resources used to deliver the intervention and their values are collected via interviews with central intervention staff, logging of the costs involved in hiring session facilitators, interviews with agency liaison people regarding the time taken to complete their SPIRIT tasks, recording the length of interviews, monitoring the time taken to complete the SEER online surveys and careful record keeping regarding the duration of sessions. Microcosting (a bottom-up approach methodology) is being used.34

Randomisation

The health policy agency is the unit of randomisation and intervention in the SPIRIT trial, with two sites randomised to start the intervention at each ‘step’. The six participating agencies were each assigned a code. The list of agency codes was sent to a biostatistician who was not involved in any other aspect of the trial, who assigned each agency code a computer-generated random number sequence and resorted them. The first two agencies in the list were allocated the October 2012 start date (step 1), the second two the April 2013 start date (step 2) and the final two the October 2013 start date (step 3).

Blinding

The SPIRIT staff and those investigators involved in delivering aspects of the intervention, by necessity, are not blind to allocation. All other investigators are blind to allocation, although analysis will need to know when sites start the intervention. They will not be informed which agencies are in the active intervention phase, and all questionnaire responses and interview records that they receive will be de-identified for individual participants. The agency allocation is concealed from those scoring SAGE and ORACLe to limit bias; this is particularly important since the intervention team cannot be blinded to the active intervention agencies.

Statistical methods

Analysis will be undertaken using generalised linear models with a link function and error distribution appropriate to the outcome measure. In each model, the intervention effect will be estimated as the difference between the postintervention and preintervention levels of the outcome after adjusting for time. The unit of analysis will be the agency for the SAGE and ORACLe outcome measures and the individual for the SEER outcome measure. Since there are too few clusters to employ either a linear mixed model or a generalised estimating equation (GEE) approach, all models will adjust for the clustering by using agency as a fixed effect.

For the study design outlined in figure 2 and with the assumption of an intraclass correlation coefficient of 0.01, a total sample size of 144 documents, 4 per site per time period, will give 89% power to detect a 1 SD difference in the mean SAGE total score at the 5% significance level.16

Data management and monitoring

Data for the SEER measure are entered online by participants. SEER is primarily multiple choice. In order to enhance data quality, the online survey has built in skip functions and participants are unable to progress from one page to the next until all questions are answered. Data from SAGE and ORACLe are transcribed (the scoring and checking procedure was described in the outcome measures section).

All data and documents are stored on password protected computers and on a network server accessible only to the researchers. Any paper documents are stored in a securely locked filing cabinet located at the Sax Institute, which has day-to-day management of CIPHER. The Sax Institute is a secure facility requiring a unique staff access code to enter; visitors are escorted on-site and are required to sign in.

A data monitoring committee was not required for this trial as the risk of harm to participants is negligible.

Ethics and dissemination

All participants provide electronic consent to participate in outcome measures via return email, and signed consent for process evaluation data collection. All participating agencies and individuals are free to decline to participate in any and all aspects of SPIRIT at any time with no explanation required (example participant information and consent forms in online supplementary appendix 1).

Participating agencies receive feedback on their results at regular intervals during the active phase of the intervention and again at the conclusion of the trial. The findings of this study will be disseminated more broadly through peer-reviewed publications (authorship determined by BMJ guidelines) and presentations at conferences and used to inform future strategies.