in-DEPtH framework: evidence-informed, co-creation framework for the Design, Evaluation and Procurement of Health services.

With a multitude of variables, the combinations of care, health program activities and outcomes are infinite, and this renders improvement efforts to complex health service interventions particularly intricate. Here, we describe a framework that seeks to incorporate research evidence and the multi-faceted considerations of stakeholders, context and resources to co-create sustainable health solutions that improve the health outcomes of patients and communities. This evidence-informed, co-creation framework for the Design, Evaluation and Procurement of Health services (in-DEPtH) is a systematic approach to support health agencies to commission services that are evidence-informed, contextually relevant and stakeholder engaged. The framework consists of several steps from defining the research question, health outcomes and search inclusion criteria, to the synthesis of evidence, and to co-creation and Delphi consultations with stakeholders. In this paper, we describe the various steps of the framework and explain the theoretical methods underpinning the framework. The approach of the framework is context neutral and can be applied to healthcare systems of different countries. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The evidence-informed, co-creation framework for the Design, Evaluation and Procurement of Health services (in-DEPtH) is a systematic approach that can support health agencies to commission services that are evidence-informed, contextually relevant and stakeholder engaged.

The approach of the framework is context neutral and can be applied to healthcare systems of different countries.

The value of the framework is influenced by the quantity and quality of available research evidence, but it can guide decision making in any context.

The framework incorporates a Delphi process, which supports convergence of views, but participants with vested interests could potentially confound the Delphi process and skew its outcome to reflect the perspective of the dominant participant representation.

Introduction

The performance of healthcare systems has stagnated and there are no easy means to improve a complex and adaptive healthcare system.1 Healthcare systems are characterised by many complex variables, such as intricate funding models, clients with diverse needs, various intervention options for a medical condition, clinical processes that need to be individualised to each patient, presence of numerous stakeholders with different roles and interests and uneven regulations that are too strict in some areas or too lax in other areas.1 For example, in Australia health programmes for residents of aged care facilities are commissioned by primary health networks (not-for-profit companies funded by Department of Health), while services are delivered by private aged care providers (for or not-for profit). Within aged care facilities, there are residents who are still healthy, while others are towards end-of-life, thus necessitating different levels of care and clinical interventions. When residents require acute care and are transported to hospital emergency departments, the provision of aged care straddles the primary and secondary care sectors. The service environment is consequently complicated with primary care funded by state governments, while secondary/acute care is funded by the commonwealth (ie, national) government. With such a multitude of variables, the combinations of care, health programme activities and outcomes are infinite, and this renders improvement efforts to complex health programme interventions particularly difficult and intricate. There are also other factors to consider when attempting to improve complex healthcare.1 First, cost effectiveness of the changes. Second, acceptance of the changes by stakeholders. Third, deliverability of the changes by health organisations. All these considerations contribute to the overall sustainability of the improvement changes.

Commissioning of health services is a relatively new healthcare concept and there is limited experience and published literature to guide health agencies on the best way to conduct commissioning.2 Commissioning is defined broadly as the process of planning, purchasing and monitoring health services for a population, subpopulation or individual client.2 In the review by Gardner et al,2 the authors found insufficient evidence to identify a preferred commissioning model, that commissioning impacts were highly context-dependent and there was still significant work to be done to support commissioning. The authors also highlighted several considerations for successful commissioning, such as the need for engagement of stakeholders (patients/consumers, clinicians and providers), and localised priority setting and decision making.2

With a lack of evidence for a preferred commissioning model and the need for highly contextual local considerations, it is requisite to have a framework that is able to address the aforementioned issues and support health agencies to prioritise and make sound decisions by incorporating research evidence and the multi-faceted considerations of stakeholders, context and resources (such as cost, skills and capacity), with the ultimate aim to improve the health outcomes of patients and communities. The usual systematic review approaches (quantitative, qualitative or mixed methods) of using quantitative outcome data for statistical meta-analysis and/or abstracting qualitative data into categories/themes may not be sufficiently comprehensive to incorporate the multi-faceted considerations inherent in a complex health programme. Instead, a broader and more comprehensive approach is needed. Here, we present an evidence-informed and locally contextualised priority setting framework, which aims to support improvement in the commissioning and delivery of complex health programmes, changes that not only lead to successful commissioning, but are also acceptable, feasible and sustainable.

Description of in-DEPtH framework

We developed a framework that can be used to incorporate locally relevant data and multi-factor considerations from various stakeholders to inform priority setting and decision making in the context of complex health programme interventions. We term this the in-DEPtH (evidence-informed, co-creation framework for the Design, Evaluation and Procurement of Health services) framework. in-DEPtH aims to provide a systematic approach to incorporate evidence on health programmes, local context realities and stakeholder multi-factor considerations to co-create specific health programme features that are relevant and applicable to individual health agencies and their areas of service. It is envisaged that prioritised health programme features can be directly used as specifications for the procurement of health services.

The framework has several steps. First, a search for relevant studies (both quantitative and qualitative) is conducted. Quantitative studies, if appropriate, undergo meta-analysis while qualitative studies are used to identify barriers and facilitators to the health programme of interest. The identified barriers and facilitators are converted into pertinent programme features. Through engagement and discussion with stakeholders, the pertinent features are synthesised (ie, research evidence and expert opinions are aggregated) and refined to suit local context and realities at the coalface. For each synthesised programme feature, published and grey literature are searched to inform the costs and effects of each feature, which is then shared back with local stakeholders. The synthesised features are then prioritised through a series of Delphi rounds to identify the most important features. Finally, the ranked programme features are then recommended to decision-makers so that they can decide in an informed and transparent manner, with the assurance that the considerations and assessments of various Delphi participants (stakeholders, clinicians, users and communities) have been systematically incorporated.

The individual steps of the framework are explained as follows:

Define research question and search inclusion criteria: Population, intervention, comparison and outcomes for quantitative effectiveness studies, and population, intervention of interest and contextfor qualitative studies. This should be jointly defined with the commissioning health agency to ensure the right and relevant research question and health outcomes are addressed.

Search for studies (quantitative and qualitative) that are relevant for the research question, inclusion criteria and context.

For included quantitative studies conduct meta-analysis (if appropriate).

For included qualitative studies:

Identify and extract barriers and facilitators (until data saturation is reached).

Group similar barriers and facilitators.

Condense and convert to statements on programme features (be as specific as possible).

Relate features back to included quantitative studies, and whether these studies reported positive or negative results (this is to understand success/failure factors and to incorporate learning points back into the extracted programme features, eg, factors on dosage/intensity).

Synthesise extracted programme features from primary evidence together with features included in corroborated positive trials, and existing programme specifications of the health agency, if any (ie, co-create desired health programme features via synthesis of data and engagement with stakeholders).

For each synthesised programme feature:

Conduct a search for evidence (published or grey literature) on outcomes and costs.

Analyse data to generate estimates of the costs and effects of each programme feature.

Incorporate inputs from stakeholders via Delphi process to rank programme features for priority setting:

Two initial Delphi rounds: (a) First round: rank based on level of benefits to patients/consumers. (b) Second round: rank based on level of difficulty to implement (ie, deliverability).

Third round: Results of the first two rounds will be shared with Delphi participants to inform the third round. Taking into account results of prior two rankings, features to be finally ranked.

Recommend prioritised programme features to senior executive management for decision making.

Figure 1 shows the steps of the framework in a graphical representation.

Figure 1

Graphical representation of the evidence-informed, co-creation framework for the Design, Evaluation and Procurement of Health services.

The structured synthesis of evidence and local stakeholder considerations provides a transparent and systematic basis for stakeholders to contribute to the commissioning of health programmes, thus improving stakeholder satisfaction and acceptance of the changes. Also, with the assurance that research evidence, local context and inputs of various stakeholders (health agencies, healthcare providers, clinicians, users and communities) have all been systematically incorporated and prioritised, decision-makers will be better empowered and equipped to act.

Methodologies applied in framework

The in-DEPtH framework uses concepts from the methodologies of realist review, qualitative descriptive analysis and integrative review. The adaptation of these methodologies to the framework is described as follows.

Realist review

A realist review is based on a realist philosophy of science and considers the interaction between context, mechanism and outcome.3 The realist reviewer seeks to answer the question: ‘What works for whom under what circumstances, how and why?’3 4 In a realist review, the following steps are involved. First, the main ideas behind the intervention is elicited from literature (ie, the intervention programme theory is formulated). Next, the relevance and effectiveness of each theory idea is verified using various types of evidence (such as qualitative, quantitative, comparative and administrative) from both published and grey literature sources. Searches are conducted iteratively and are purposive and targeted in approach to answer or test specific questions or theories.4 For each theory idea, the reviewers aims to understand the contextual factors that triggers the mechanisms which, in turn, generates the outcome of interest. By comparing the intervention programme theory to empirical evidence, the reviewers seek to determine the many contingencies and circumstances (ie, what works for whom in what situations) that affect the ability of the intervention to generate the intended outcomes. The information gained from a realist review helps policy makers to interpret and use an explanation of why a programme works better in one context than another.4 The last step in a realist review is to implement, test and evaluate recommendations with stakeholders in particular contexts.

Complex health programme interventions can be viewed as complex systems consisting of multiple service delivery features/components and being subject to various dynamically changing and interacting factors such as human behaviours/perceptions, skills level/capacity, macro/local policies, physical structures and resource allocations. This social reality cannot be measured directly, but can be known indirectly3 and because a realist review adopts an approach that sits between positivism and constructivism,5 it is particularly suited for examining and analysing complex health programme interventions.3 4

For our framework, we adapted the approach of a realist review to suit the nature and scope of our research aim. The realist review approach serves to guide us in examining complex health programme interventions, in conducting iterative, purposive searching to understand specific aspects as new information is uncovered, and in testing and evaluating the recommendations. The realist approach also enables us to understand contextual and social relational factors surrounding the programme features. However, unlike the realist review, we do not formulate a programme theory and our approaches towards data appraisal and synthesis is based on methods of qualitative descriptive analysis6 7 and integrative review.8

Using the approaches of descriptive analysis, we identify and extract barriers and facilitators of a health programme from qualitative primary studies and, due to quality considerations, these primary studies are searched from published, peer-reviewed journals (framework steps 4i and 4ii). The analysis and synthesis of data follows an integrative review approach, where similar data is grouped, reduced into summary statements, compared with other data and displayed in tables and matrices (framework steps 4iii, 4iv and 5). In framework step 6, we again adopt the realist approach of searching for evidence for each synthesised feature, that is we use iterative and purposive searches to answer specific questions and find data for economic costing and outcome effect sizes. For these searches, both published and grey literature are included to maximise data collection and refinement.

Qualitative descriptive analysis

The process to identify specific health programme features/components starts with the descriptive analysis of qualitative research evidence. Qualitative descriptive analysis involves the identification of findings that are close to the data, with minimal transformation (ie, with low imputation of meaning by researchers).6 9 The descriptive analysis approach enables us to identify and extract data to describe specific barriers/facilitators, what works/does not work as they are presented, that is, manifest context is described with low interpretation of data.7 We use the descriptive analysis approach for this step of the framework, as it allows us to understand the actual nature of the difficulties and challenges faced by those involved in specific local contexts, thereby giving us a sharper resolution of the problems that are being encountered on the ground, without any loss of details or contextual meaning.

The qualitative research evidence is usually generated from interviews and focus group discussions with various ‘actors’ of the health programme intervention being examined. The qualitative research evidence used in descriptive analyses could be studies that are conducted before the start of a health programme, after implementation of a health programme or even quantitative trials assessing the fidelity and effectiveness of health programmes when they contain relevant information on what works/does not work. The pre-program studies serve to examine local contextual factors and are particularly useful in understanding the barriers and facilitators at ‘ground zero’. The post programme studies examine the out-workings of certain implemented health programmes and serve to describe the programme processes, identify what worked/did not work and suggest potential improvements. To maintain the quality of evidence, such studies should come from published, peer-reviewed sources.

Integrative review

The next stage of the framework involves the analysis and synthesis of data. For these steps, the data analysis and presentation stages of an integrative review are applied, as it enables us to summarise data, compare iteratively across data, synthesise data elements into an integrated summation and to present conclusions with a logical chain of evidence.8 Integrative review has the potential to capture the complexity of varied perspectives and provide a comprehensive understanding of problems relevant to healthcare and policy,8 10 11 which makes the principles of an integrative review particularly applicable for this stage of the framework.

The application of the integrative review (data analysis and presentation stages) for steps 4iii to 4iv and step 5 is described as follows. From the descriptive analysis, a list of barriers/facilitators or problems/solutions is identified. This list of data is sorted and those that addressed a similar aspect are grouped together. See table 1 for an illustrated example. Relevant data (ie, components) about a similar aspect is extracted from each study, and studies with no relevant inputs are left empty. The extracted information is then condensed into a programme feature on the right. During the formulation of the programme feature, attention should be paid to retain as much specific details as possible, as this will facilitate the use of these programme features as specifications in procurement documents. In table 2, an exemplar case study has been applied to further illustrate how a programme feature is extracted from individual studies.

Table 1

Condense and convert to statements on programme features

Study 1	Study 2	Study 3	Identified programme features
Components that relate to aspect A		Components that relate to aspect A	Programme feature A
Components that relate to aspect B	Components that relate to aspect B		Programme feature B
Components that relate to aspect C	Components that relate to aspect C	Components that relate to aspect C	Programme feature C
	Components that relate to aspect D	Components that relate to aspect D	Programme feature D

Table 2

Condense and convert to a programme feature (exemplar case study)

Stokoe et al21	Conway et al22	Crilly et al23	Arendts et al24	Codde et al25	Shanley et al26	Arendts and Howard27	Arendts et al28	Identified programme features
Has sufficient access to GPs (ie, GPs are able to make unscheduled visits and when they do come, they allow for sufficient consultation time)	Reduce the number of GPs who come to RACF and running regular GP clinics at RACF for all residents		Streamline processes: reduce the amount of paperwork involved for GPs and provide flexibility for GPs to treat residents when they become unwell		GPs need to be willing or able to spend the time to undertake the assessment and follow-up of their sick patients that are necessary when the patients are not transferred to hospital		Needs timely visit by GPs for acute cases when nurses call them	Timely and sufficient access to GPs, such that GPs are able to make unscheduled visits and when they do come, they allow for sufficient consultation time Suggestions identified: reduce the number of GPs who come to RACF and running regular GP clinics at RACF for all residents streamline processes: reduce the amount of paperwork involved for GPs and provide flexibility for GPs to treat residents when they become unwell

GP, general practitioner; RACF, residential aged care facility.

In the exemplar case study, a programme feature on access to general practitioners (GPs) by aged care residents is illustrated. Relevant data about access to GPs was extracted from the qualitative studies identified in a review of evidence on multi-component interventions (three studies did not report relevant inputs on access to GPs). In the example above, the data was condensed and aggregated to an identified programme feature that captures the essence of GP access across all studies. Factors reported in the reviewed studies were included in the identified programme feature to provide as much specific details as possible.

The quantitative studies included for meta-analysis are then compared against each of the identified programme features. Programme components that address the identified programme features are extracted from each quantitative study and grouped according to the identified programme feature. This generates a table that links identified programme features with components of the programmes evaluated by the identified quantitative studies. Please see an illustrated example under table 3.

Table 3

Comparison between identified programme features and quantitative studies

Identified programme features	Quantitative study 1	Quantitative study 2	Quantitative study 3
Programme feature A		Components that relate to feature A
Programme feature B	Components that relate to feature B	Components that relate to feature B
Programme feature C		Components that relate to feature C	Components that relate to feature C
Programme feature D			Components that relate to feature D

This approach allows us to iteratively compare the quantitative studies against the identified programme features: horizontally we can compare the different intensities/dosages of the programme components that relate to each programme feature and vertically we can compare the extent to which studies addressed the identified programme features. The aim is to generate an understanding of the reasons why studies did or did not report positive trial results. This comparative understanding can also help us to interpret the result of the meta-analysis, either reinforcing or contradicting it. In the case that the aggregated result of the meta-analysis is inconclusive, we can compare the trialled outcomes of individual studies with the comparative understanding gained. If the comparative understanding explains why a particular study should be effective and this is also corroborated by the positive trial outcome of that study, we have a good concurrence between the two. Table 4 shows an exemplar case study to illustrate the comparison between identified programme features and quantitative studies.

Table 4

Comparison between an identified programme feature and quantitative studies (exemplar case study)

Identified programme features	Hullick et al18	Fan et al12	Connolly et al19	Boyd et al20
Has readily available clinical expertise and advice for management of illnesses within the facility, such as telephone support line, adding external clinical resources to RACFs Suggestions identified: telephone support line to organise alternatives to hospital transfer such as a medical or nursing consultation in the nursing home or an urgent outpatient appointment the next day	Telephone advice to RACF staff; working with them to define the purpose of transfer and the goals of care	HINH allocates clinical staff to manage aged care residents with actual or potential acute symptoms in the RACF HINH program manager assesses whether HINH or hospital admission was most appropriate. Daily review of HINH patients Developing individualised treatment plan for the patient in collaboration with the patient’s general practitioner and RACF nursing staff	Resident review by GNS. GNS’s time commitment was 20% across all intervention facilities (18 facilities) Only 23% of residents were discussed in multidisciplinary team meetings	Regular, proactive bimonthly GNS visits Telephone consultation and site visits as needed

GNS, gerontology nurse specialist; HINH, hospital in the nursing home; RACF, residential aged care facility.

In the exemplar case study, a programme feature on availability of clinical expertise was compared across corresponding components extracted from four quantitative studies. From the comparison, we found that Fan et al12 had the most components related to the feature (ie, higher intensity/dosage). By comparing with other identified programme features (ie, by comparing vertically [not illustrated here]), we found that the Fan et al12 addressed most of the identified programme features. The Fan study also reported positive trial results, and all this led us to conclude that the Fan trial was a positively corroborated study.

Next we compare the list of identified programme features against corresponding programme components of any identified corroborated studies that have positive trial outcomes. At this stage, if current programme specifications from the health agency are available, they should be incorporated into the analysis framework. Data from the health agency programme specifications have to be extracted and grouped according to the list of identified programme features. This process enables us to horizontally compare the descriptions of identified programme features across corresponding component descriptions of the corroborated studies and the health agency specifications. We then abstract and synthesise these similarly grouped component descriptions into synthesised programme features. Please see an illustrated example under table 5. Table 6 shows an exemplar case study to illustrate the synthetisation of a programme feature.

Table 5

Synthetisation of programme features

Identified programme features	Corroborated study	Current programme specifications of health agency	Synthesised programme features
Programme feature A	Components that relate to feature A		Synthesised component that relate to feature A
Programme feature B	Components that relate to feature B	Components that relate to feature B	Synthesised component that relate to feature B
Programme feature C		Components that relate to feature C	Synthesised component that relate to feature C
Programme feature D	Components that relate to feature D		Synthesised component that relate to feature D

Table 6

Synthetisation of a programme feature (exemplar case study)

Identified programme features	Fan et al12	Current programme specifications of health agency	Synthesised programme features
ACDs to facilitate communication between resident, family and RACF staff to incorporate patients' wishes into treatment plan during emergencies Have explicit notes in the medical records about care decisions and a commitment to stay the course of care	Study has no relevant ACD data	End of life care — incorporating the 7 step pathway - community version into eldercare’s palliative care model pathways to support end of life care and associated decision making	All residents to have ACDs to facilitate communication between resident, family and RACF staff to incorporate patients' wishes into treatment plan during emergencies Have explicit notes in the medical records about care decisions (such as using the 7 step pathway - community version) and a commitment to stay the course of care

ACDs, advanced care directives; RACF, residential aged care facility

The exemplar case study shows the synthetisation of an identified programme feature on advanced care directives (ACDs). We compared the ACD programme feature across corresponding descriptions from the corroborated Fan et al12 study and the current programme specifications of a health agency. Here, the Fan et al12 study had no relevant data on ACD. However the current programme specifications stated the use of an ACD tool (7 Step Pathway - community version), which was subsequently added to the ACD programme feature to form the synthesised version. In an actual case study, further inputs would have to be sought from stakeholders in order to solicit their views on the syntax of the synthesised feature and incorporate their expert opinions into the synthesised programme feature.

In this way, local contextual factors reflected in programme specifications of the health agency, positively trialled components from corroborated quantitative studies and identified programme features extracted from qualitative evidence sources are all incorporated into a synthesised feature that is both locally relevant and evidence-based. To further enhance its relevance and acceptability, stakeholders should be engaged to review the descriptions of synthesised programme features jointly with researchers. Another benefit of this approach is that it allows both stakeholders and researchers to understand the differences between the various sources of data, especially the gaps between programme specifications of the health agency and identified programme features that have been extracted from primary evidence sources. This enables a richer and more thorough discussion during the synthesis of programme features.

Patient and public involvement

The research question is developed from the prevalent public health understanding that there is a translational gap between evidence and the practicalities of health service commissioning. During the co-creation process, we aim to involve patients and health consumers, together with clinicians, service providers and the commissioning health agencies to achieve a better translation from evidence to practice. For example, the patients and consumers could be involved to formulate the research question, jointly review the evidence with commissioning health agencies and participate in the stakeholder Delphi process.

Discussion

Co-creation with stakeholders

The co-creation process of the framework is critical to ensuring that changes to the health programmes are acceptable, feasible and sustainable. By systematically and thoughtfully incorporating various local contextual factors and stakeholder considerations, we seek to achieve priority setting and decision making that is localised and relevant for the specific population area, where the health programme is to be commissioned.

The co-creation process with stakeholders takes place systematically over steps 1, 5, 6 and 7 of the framework. In step 1, the research question and scope are jointly defined with the commissioning health agency and stakeholders so that the purpose, programme context and outcomes of the research are correctly understood and agreed to by all involved parties. Next, under step 5 synthesis of the programme features occurs. Here, data extracted from evidence is aggregated with the existing programme specifications of the commissioning health agency, and contextual limitations and enablers are also incorporated into the synthesised programme features. We expect synthetisation to occur via workshops with stakeholders, where the research team will present on the findings and a facilitator will assist to guide the discussion to reach consensus. Specifically, stakeholders would jointly deliberate with the research team on the specific phrasings to include/exclude for a particular programme feature. Through such facilitated workshops, the various stakeholders will have to consider the relevance and applicability of the research evidence to the local context. The facilitated workshops will encourage stakeholders to be engaged and be explicit about the rationale for their decisions. Through this process of joint dialogue and exchange of ideas, the desired health programme features are refined and ‘polished’. This stage of the process is expected to be the most time consuming, but it would be prudent not to rush through step 5 and instead devote more time to reach an understanding and consensus, as the subsequent steps 6 and 7 will depend on the outcome of this step.

How the programme features are organised for the Delphi process is also determined at this stage. For example, if there is a positive corroborated study of programme features that are relevant to the local context, the health agency may well choose to simply adopt those features which are reflected in the study. For those remaining synthesised features that are not present in a corroborated study, these could then proceed for prioritisation via the Delphi method. This is just an illustration of what might occur at this stage of the framework. The actual organisation of the synthesised programme features will vary, depending on how the co-creation with the health agency progresses.

It is also to note that steps 5 and 6 are inter-connected and can be iterative. The cost and outcome data obtained for each synthesised feature could influence the aggregation of the programme features. There should be flexibility and regular communication with the health agency and stakeholders for steps 5 and 6 in order to maximise the synergies of data and stakeholder inputs.

In step 7, a wider range and quantity of stakeholders is engaged in a Delphi process to seek their expert opinions in order to prioritise the synthesised features. The wider group can extend to health agencies, healthcare providers, community and consumers. This stage of the framework serves to co-create the combination of features that are beneficial and truly valued by the specific population area. At the same time, the co-creation provides insights into the levels of deliverability by the healthcare providers.

Application of diverse analysis methodologies

We used several methods of review and analysis in our in-DEPtH framework, namely: realist review, descriptive analysis and integrative analysis. The reasons to do so are twofold. First, the nature of health programme interventions is complex and we need to be able to ‘dissect’ the programme system into components of data that can be worked and analysed. Second, we need to be able to generate specific details of evidence-informed programme features that both reflect on-the-ground realities and which can be synthesised to incorporate further evidence and stakeholder inputs for priority setting. These synthesised features should be clear and specific, with the intention that these features can be used directly as procurement specifications. Given the outputs required, we find that a ‘one size fits all’ approach of using one particular method of analysis is not suitable. Instead, using a combination of methods allows us to achieve outputs that are practically useful for commissioning activities.

Combining quantitative meta-analysis findings with qualitative findings

In steps 3 and 6ii of the framework, meta-analysis is conducted for quantitative data. However meta-analysis may not be appropriate if the trial data pertain to intervention programmes that are not homogeneous.13 14 This would apply particularly to step 3, as at this stage of the framework, it is likely that quantitative studies are trialling complex programme interventions which consist of different components. If the trial programme components are different between studies, then in essence each trial is testing different aspects of the intervention, even though the trial population, context and outcome measures are similar. In our experience, given the complex nature of health programmes, it is likely that meta-analysis would not be appropriate at step 3.

At step 6ii, however, it may be appropriate to conduct meta-analysis, as at this stage, we are looking at individual programme features and not the health programme as a whole. At this ‘smaller-scope’ level, the trials would be measuring outcomes and cost for a more specific intervention scope. Hence it is likely that the trial studies will be more homogeneous and hence possibly suitable for meta-analysis. The challenge, though, is the availability of data as there might not be studies which focuses on specific areas, given that trials are resource intensive and usually aim to test a wider scope. The availability of empirical cost data for individual programme features is also likely to be limited. Nevertheless, if quantitative data is available, it can provide useful intelligence to the synthesised programme features.

Limitations of framework

Broad evidence base of a realist review

In a realist review, the reviewer is able to draw on a broad range of information from various sources. At the same time, the search for and analysis of evidence is not linear in approach, but iterative.4 This enables a comprehensive search for elusive data and allows for a deeper understanding of the review question as information and knowledge is built up gradually over successive iterations of searching and analysing. Such an approach to reviewing enables researchers to understand the diverse contexts, mechanisms and outcomes that are characteristic of complex health programme interventions.15

While broad encompassing, the use of various evidence or only parts of an evidence can lead to inaccurate findings or self-fulfilling findings. As critical appraisal of the quality of studies in a realist review is mainly based on the concepts of relevance and rigour,15 it is highly dependent on the ability of the reviewer to make a sound judgement. In our framework, to maximise the quality of evidence on which identified programme features are based, we seek to use evidence from published, peer-reviewed sources (ie, for framework step 2). However in the search for cost and effects data to further inform the identified programme features (during framework step 6), we widen the sources of evidence to include not only published articles but also grey literature. At this stage, data is usually harder to come by and so we use a broader search to collect more data.

Our approach lies somewhere in-between the highly defined search of a traditional systematic review and the broad approach of a realist review, which could potentially open us up to including low-quality evidence, especially during the search for cost and effects data.

Confounding of Delphi method

The Delphi technique was developed by The RAND Corporation in the 1950s as a method to solicit the opinions of experts through a series of questionnaires and opinion feedback in order to establish a convergence of opinion.16 It arose as there was inadequate theoretical foundations which could forecast the outcomes of socio-economic and political problems, and where there was still a need to rely on expert judgments and opinions in the absence of reliable predictive theory.17 The Delphi method is thus a systematic and constructive way to obtain and converge relevant and intuitive insights of experts. One potential issue with sourcing expert opinions is how best to reach a true consensus. The usual round-table committee discussion can be subjected to various confounding factors such as dominating influences of experts with higher standing, the need to stick to publicly expressed opinions or just adopting the majority viewpoint.17 The Delphi method avoids such complexities by using individual questionnaires and group opinion feedback, such that the experts formulate their answers independently and anonymously, while still being able to take into consideration the summarised opinion of the entire group from prior Delphi rounds. This systematic, unbiased way of converging the opinions of experts makes the Delphi method an attractive tool that is able to generate ranked potential programme components to inform decision making.

For an optimal Delphi run, the creators of the Delphi method also advised, first, to select the appropriate expert participants wisely and, second, to explain clearly the problem setting to the experts so that there is no misinterpretation in their answers.17 These are very valid considerations, of which the former advice brings us to a potential confounder in our Delphi process. Under step 6 of the framework, a search is conducted to find cost effects data for each synthesised programme feature. While the availability of such data can help to provide more information for a synthesised feature, it can potentially confound the Delphi process. In a Delphi process, participants representing various segments of the stakeholder group are sourced. As each stakeholder segment has its own potential priorities and vested interests, it is important to have a balanced group, such that the different perspectives are distributed evenly. For example, for a synthesised programme feature, there could be different cost perspectives (eg, hospital, residential aged care facility [RACF], commonwealth/state government, resident/patient), depending on which party is financially impacted. Depending on the mix of Delphi participants, the ranking of the features could be skewed towards a certain cost perspective, for example, if too many participants representing party ‘A’ are involved, features with less cost impact to party ‘A’ could potentially be higher valued than others.

Similarly, there are also different perspectives on deliverability (ie, complexity of implementation), for example, implementation by hospital of particular features can be complex, but implementation by RACF could be simple. Again, depending on the mix of Delphi participants, the ranking of such features could be skewed towards a certain perspective, for example, if too many participants from party ‘A’ are involved, features with less implementation difficulty to party ‘A’ could potentially be higher prioritised than other features.

The presence of imbalanced different perspectives could potentially confound the Delphi process and skew its outcome to reflect the perspective of the dominant participant representation. Care should be taken to ensure that expert participants who come with different perspectives and vested interests are carefully selected and adequately balanced for each Delphi round.

Prioritising research efforts and maximising use of existing evidence

Under steps 2 and 6i of the framework, searches for research evidence are performed. If there is insufficient primary evidence to address the pertinent research question defined by health agencies under step 2, the need for primary research is highlighted. Also, the framework, through the use of evidence to value programme features under step 6i, searches for evidence on cost and effects data, and where features warrant further evidence, future research efforts can be directed to generate data. On the other hand, where evidence is available and sufficient (both in terms of quantity and quality), then a particular feature has been adequately addressed. In this manner, the framework identifies evidence gaps and the need to prioritise research efforts to generate more data to address these gaps, while also maximising the use of existing evidence for those areas where there are adequate evidence.

Insufficient research evidence

In the absence of sufficient evidence to address the research question, the contribution of other sources of data (such as expert opinions and local intelligence) would gain greater emphasis. The framework is still expected to inform the definition and selection of programme features, even in the presence of minimal research evidence. Though not fully evidence based, the framework outputs could still be considered valid, in view of the contribution of expert opinions. Over time, as research efforts are directed towards addressing the gaps, the framework could be updated to incorporate the new evidence.

Generalisability to other healthcare systems

It is envisioned that the framework can be used in the healthcare systems of different countries, as the methods of the framework are context neutral. The important starting point is to define the research questions, health outcomes and search inclusion criteria, such that they fit the nature of the intervention to be investigated at the local contextual setting. The subsequent steps of evidence synthesis and Delphi consultations will then be relevant and meaningful to local stakeholders, such that the outputs from the framework will be appropriate to the specific environment where the health programme intervention is to be adopted.

Engagement with health partners

We are currently engaging health service partners to apply the framework in their commissioning activities. Through this process of real-world testing for specific health priorities (such as aged care, mental health, etc), we hope to evaluate the feasibility and acceptance of such a framework and identify areas for improvements. On completion of the engagement, we aim to publish our findings in later articles.

Conclusion

The in-DEPtH framework seeks to provide a systematic approach to improve the success of health services commissioned by health agencies. It aims to combine research evidence and the expert knowledge of local stakeholders to jointly create practical, feasible and sustainable solutions that are appropriate to local contextual settings. The framework consists of several steps from defining the research question, health outcomes and search inclusion criteria, to synthetisation of evidence, and to co-creation and Delphi consultations with stakeholders. In this paper, we have described the various steps of the framework and explained the theoretical methods underpinning the framework, namely: realist review, qualitative descriptive analysis and integrative analysis. The approach of the framework is context neutral and can be applied to healthcare systems of different countries, depending on the availability of relevant primary evidence.