What was the impact of a participatory research project in Australian Indigenous primary healthcare services? Applying a comprehensive framework for assessing translational health research to Lessons for the Best.

OBJECTIVES: To (1) apply the Framework to Assess the Impact from Translational health research (FAIT) to Lessons from the Best to Better the Rest (LFTB), (2) report on impacts from LFTB and (3) assess the feasibility and outcomes from a retrospective application of FAIT.
SETTING: Three Indigenous primary healthcare (PHC) centres in the Northern Territory, Australia; project coordinating centre distributed between Townsville, Darwin and Cairns and the broader LFTB learning community across Australia. PARTICIPANTS: LFTB research team and one representative from each PHC centre. PRIMARY AND SECONDARY OUTCOME MEASURES: Impact reported as (1) quantitative metrics within domains of benefit using a modified Payback Framework, (2) a cost-consequence analysis given a return on investment was not appropriate and (3) a narrative incorporating qualitative evidence of impact. Data were gathered through in-depth stakeholder interviews and a review of project documentation, outputs and relevant websites.
RESULTS: LFTB contributed to knowledge advancement in Indigenous PHC service delivery; enhanced existing capacity of health centre staff, researchers and health service users; enhanced supportive networks for quality improvement; and used a strengths-based approach highly valued by health centres. LFTB also leveraged between $A1.4 and $A1.6 million for the subsequent Leveraging Effective Ambulatory Practice (LEAP) Project to apply LFTB learnings to resource development and creation of a learning community to empower striving PHC centres.
CONCLUSION: Retrospective application of FAIT to LFTB, although not ideal, was feasible. Prospective application would have allowed Indigenous community perspectives to be included. Greater appreciation of the full benefit of LFTB including a measure of return on investment will be possible when LEAP is complete. Future assessments of impact need to account for the limitations of fully capturing impact when intermediate/final impacts have not yet been realised and captured. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

It is the first study reporting on the application of the Framework to Assess the Impact of Translational health research (FAIT) on an Australian-based study.

Although designed for prospective application, a retrospective application of FAIT yielded impacts beyond traditional academic impacts such as publications.

The use of three proven methods of impact assessment provides a comprehensive understanding of the impact of LFTB from multiple perspectives.

The calculation of an accurate, evidence-based ‘return on investment’ was not possible because the benefits of LFTB were not fully realised at the time of assessment.

This issue of lag in the health research translation process, and a delay in impact, is an ongoing challenge that needs to be recognised when undertaking impact assessments.

Introduction

Despite expectations, a substantial proportion of health and medical research does not translate into practice or policy, and does not generate wider benefits.1 This means research findings are not optimally implemented by healthcare providers, do not always contribute to policy and practice and, hence, fail to realise their potential effectiveness or impact in the wider, non-academic community.2 The Framework to Assess the Impact from Translational health research (FAIT) was developed specifically for health and medical research (HMR) to improve research translation and to optimise and assess the impact from research investments.3 FAIT recommends prospective application, mapping a pathway from research need to impact and combines a modification of three existing, validated methods of impact assessment (Payback, Economic analysis and Narratives) to present a multidimensional, comprehensive view of research impact.

Helping drive the increased focus on impact from research is the growing pressure for increased accountability in public spending across all sectors, including health.4 In Australia, the focus on greater accountability includes investments made to improve Aboriginal and Torres Strait Islander (hereafter respectfully referred to as Indigenous) health outcomes. This focus is driven by unacceptable health disparities between Indigenous and non-Indigenous Australians.4 5 An additional concern is that Indigenous Australians report being over-researched without corresponding evidence of improvements in health outcomes.6

Comprehensive PHC is central to managing the growing burden of chronic disease and addressing socioeconomic and environmental factors affecting health.7 Having a well-PHC sector improves the overall health and well-being of the population and reduces acute hospitalisations.8 Quality improvement (QI) is one mechanism for enhancing the overall quality of care provided within a health system.3 4 In Indigenous healthcare, there are existing data that can be leveraged to help improve the quality of PHC and the overall health of Indigenous Australians. The Audit for Best Practice in Chronic Disease National Research Partnership and One21seventy used data and continuous quality improvement (CQI) strategies to improve the quality of PHC provided to Indigenous Australians.9 From 2010–2014, 175 primary healthcare centres around Australia participated and agreed to share their CQI data for research purposes.

From 2014–2017, Lessons from the Best to Better the Rest (LFTB) used these audit data to identify six Indigenous PHC centres showing consistent high improvement in care quality.10 LFTB then used (1) longitudinal CQI systems assessment data11; (2) qualitative data obtained through interviews with health centre staff, health service users (Indigenous community) and external stakeholders; (3) health centre and workforce survey data; and (4) observations by members of the LFTB team as recorded in field notes12 13 to uncover how contextual factors integrated to facilitate the success of CQI initiatives within each centre and to identify common factors for success. The team also used a participatory method of research that involved researchers working with health centre staff and the Indigenous community to co-create new knowledge on a range of factors that appeared to be associated with CQI success.13 LFTB aimed to use these learnings to empower Indigenous communities and striving PHC centres to improve their systems and service delivery performance and share those lessons widely, including with policy makers.

Over a similar timeframe (2015–2019), the Australian National Health and Medical Research Council funded the Centre for Research Excellence for Integrated Quality Improvement in Indigenous Primary Healthcare (CRE-IQI). The CRE-IQI vision was to improve Indigenous health outcomes by accelerating and strengthening large-scale PHC improvement efforts. LFTB was selected as one of five CRE-IQI flagship projects and a portion of CRE-IQI funding was allocated to adopting an impact framework (FAIT)14 that would both encourage research translation and measure the impact of its five Flagship projects. This paper (1) reports the impact from the investment into the LFTB programme, not the process the research undertook; and (2) based on data collected by the independent evaluator (SAR), reports the feasibility, experience and outcomes from a retrospective application of the FAIT tool to LFTB. Previous papers reported on the research process and results of the LFTB project.10–13 15

Methods

Setting

The setting for the impact assessment was the LFTB coordinating office at the Anton Breinl Research Centre for Health Systems Strengthening at James Cook University, the distributed LFTB learning community including three of the six high improving health centres and the broader CRE-IQI network. Project coordination was distributed between Townsville, Darwin and Cairns.

Participants and ethics

Participants in the impact assessment were researchers associated with LFTB and PHC staff at the three Northern Territory-based services involved in LFTB. The other services (two in Queensland and one in Western Australia) were excluded because ethical approval lapsed and a reapplication process (including site-specific research governance approvals) was not feasible from a resource perspective. Oversight of the application of FAIT to LFTB was undertaken by an independent assessor (SAR).

Patient and public involvement

There were no patients (Indigenous community members) involved in this study. Prospective application of FAIT would have allowed involvement of health service users in the design and application of the impact assessment.

Key components of the methods and application of the three FAIT methodologies are summarised here but a detailed description of FAIT and its application can be found elsewhere.3 16

In phase 1, a modified programme logic model was developed retrospectively based on project documentation available at the time and appears as figure 1. ‘Retrospective’ refers to the logic model being developed based on actual pathways that had been followed, instead of a ‘prospective’ logic model that would map the intended pathway at the start of the research project.

Figure 1

Modified programme logic model. AMSANT, Aboriginal Medical Services Association of the Northern Territory; CQI, continuous quality improvement; NACCHO, National Aboriginal Community Controlled Health Organisation; QAIHC, Queensland Aboriginal and Islander Health Council.

The logic model documented the pathway between LFTB and its impact and informed identification of suitable impact metrics. This work was assisted by a scoping review of published information on potential impact metrics. The review (undertaken for three of the CRE-IQI Flagship Projects) resulted in lists of potential benefits such as the Becker Medical Library Model for Assessment of Research Impact,17 the Ideas Book from the RAND Corporation18 and other examples.19–24 However, many of the published impact metrics were not directly applicable to LFTB. A process was undertaken to map the activities of LFTB to suitable published metrics then supplement them with customised metrics that related directly to LFTB; these supplementary metrics were determined by the independent assessor in consultation with LFTB researchers.

Phase 2 proposed to involve an evaluation of the implementation of FAIT in LFTB.25 However, this was not undertaken because the retrospective application of FAIT did not allow the application of the full functionality of FAIT such as planning for impact and the encouragement of translation.

In phase 3, the three FAIT methodologies were applied.

Modified Payback3 26

The Payback methodology was used to assess impact using quantitative metrics that sat in three of the five domains of benefit in Payback—knowledge advancement, policy and legislation and economic impacts. Following discussions between the independent assessor (SAR) and LFTB researchers, an additional domain of benefit ‘capacity strengthening’ was included to capture the education, training and professional development impacts of LFTB. Data for the impact metrics were obtained from project documentation, through interviews with project researchers and staff from participating high improving services, and through online searches of relevant websites. A cut-off date of 20 February 2020 was used for all publication statistics found on the internet.

The economics

Although a cost–benefit analysis from a societal perspective is ideal to determine social return on investment, it was unsuitable for application to LFTB because (1) there was no intervention with which to compare the outcomes (ie, a counterfactual); (2) many of the benefits for LFTB are yet to be realised; and (3) many of the benefits could not be easily monetised. Instead, a form of cost-consequence analysis (CCA) that presents, in a disaggregated form, an array of consequences and costs, was undertaken.27 Rather than combine costs and outcomes into a cost–benefit ratio, the decision about whether LFTB was a good use of resources has been left to the reader.

Costs

Direct research costs were captured from the LFTB research proposal and disaggregated to provide greater transparency. In addition, indirect research costs via in-kind contributions from chief and associate investigators were costed using a bottom-up micro-costing methodology.28 The lead investigator was assessed separately. Two other investigators were asked to estimate the number of hours they spent on the project from inception to completion. The largest difference in cost arose from the location of the investigators, which translated to greater travel time to all face-to-face project activities. Location of the furthest and the closest investigators (to Townsville) were used to cover the bandwidth of potential travel time for investigators. In addition, attendance data were used to account for variance in the number of teleconferences, workshops and site visits that investigators were involved in which was another point of variation. In the absence of a national academic staff salary schedule, academic wages were costed using the Australian National University Academic Staff Salary Schedule as a proxy.29 Wages for non-academic investigators were costed using the average wage rate for Australia as a proxy.30 A sensitivity analysis (SAR) was employed to account for the variance in the hours contributed by the different investigators (average: 75 hours; range: 30–160 hours). An additional 30% was added to cover oncosts. All costs were converted into 2019 values using standardised Reserve Bank of Australia inflation rates.31

Implementation costs from the point of view of the health centres, service users and external stakeholders were also costed using a similar bottom-up micro-costing methodology based on (1) opportunity cost; (2) documentation of implementation activities; and (3) interviews with LFTB staff to estimate time spent on those activities. Examples of implementation activities included CQI-related site visits, face-to-face meetings and workshops, monthly teleconferences and in-depth interviews. The time allocation to each activity was reported in hours and multiplied by the number of participants involved, then costed at appropriate published wage rates for participating health centre staff32 and the average wage rate for Australia for health service users and external stakeholders.30 The addition of 30% oncosts and conversion into 2019 values31 were also applied. Each health centre received $10 000 in recognition of the contribution of health centre staff (particularly a local project champion) to cover meeting and interview coordination, facilitating access to existing documents/data, hosting meetings, catering and enhancing research capacity at each centre. It was also to reimburse the health centre for the time of participating healthcare staff. The total reimbursement figure was reported in the CCA.

Consequences

The direct monetised consequences from LFTB were limited because LFTB was a ‘knowledge creation’ study with no identified ‘intervention’ being implemented. The translation of learnings from LFTB is ongoing through LEAP which is currently in the last year of implementation (notwithstanding a COVID-19-related extension). Hence, downstream consequences enabled by LFTB but realised through LEAP are not reported in this study. However, two of the consequences could be monetised—the funding secured for LEAP and the next iteration of CRE-IQI—the Centre of Research Excellence in STRengthening systems for InDigenous healthcare Equity (CRE-STRIDE).

LEAP is an essential step on the pathway to impact from LFTB and stands to have significant consequences for striving services involved in LEAP and for the Indigenous PHC sector and community more broadly. The leveraging of LFTB findings attracted additional resources to: (1) assist striving services reach their CQI goals; (2) create a learning community to advance the application of CQI in Indigenous PHC; and (3) create a set of resources to help all services improve the success of their CQI initiatives; all of which open up the possibility of further downstream consequences from having conducted the LFTB project. The funding of LEAP was entirely attributable to the learnings from and relationships developed during the LFTB Project.

In addition, the work undertaken, learnings generated and wide engagement from LFTB and the ongoing translation through LEAP also contributed to securing funding for CRE-STRIDE. Attributing the level of contribution is inherently complex given the multifactorial nature of that contribution. This includes the fact that LFTB/LEAP is one of about five streams of work that have flowed from CRE-IQI into CRE-STRIDE. LFTB brought new partnerships, achieved wide engagement and impact, pioneered the adoption of Indigenous co-leadership within LEAP, all of which were leveraged for the CRE-STRIDE application. Based on these assumptions, contribution of LFTB was estimated by two LFTB researchers at between 10% and 20% of the overall grant.

The narrative

The narrative was based on interviews and discussions with three LFTB team members and three health centre staff, one each from three participating health centres, who were nominated by LFTB researchers as being most closely involved. One nominee was no longer employed at the respective health centre. A current member of staff offered to be interviewed as a substitute. Two of the six interviewees were health service managers, one was a clinician manager, three were academic researchers (including one clinician researcher) and one identified as an Indigenous Australian. All interviewees signed consent forms prior to their interviews.

Interviews were transcribed and analysed to identify key themes. These interviews formed the basis of the narrative and captured key aspects of the story from ‘the need for the research’ through to specific ‘impacts’ that were realised. The results for the application of FAIT to LFTB are summarised and presented in a scorecard format by each method.

Results

Aim 1: application of FAIT to LFTB

Payback

Table 1 presents the results from the application of the modified Payback method of assessment, expressed as metrics, grouped within four ‘domains of benefit’—knowledge advancement, capacity strengthening, policy engagement and economic benefit. Knowledge advancement included peer-reviewed publications, but also captured grey literature such as project briefs that are a powerful non-academic pathway to translate LFTB findings into useable products to inform key end-user groups.

Table 1

Impact scorecard

Domains of benefit	Categories of impact	Results
Advance knowledge	Publications	4 articles published in peer-reviewed journals including original research, protocols and editorials
		17 citations in other peer-reviewed publications
		33 citations in Google scholar
		100% of articles that are on an open access platform
		9261 views as recorded on the journal website
		3975 downloads as recorded on the journal website
	Conferences	1 plenary presentation as an invited speaker
		6 presentations at concurrent sessions
		2 posters
	Grey literature	3 project briefs—community, policy makers and health services
		6 project reports to each specific LFTB health centre
	Media	1 article in local/regional/national newspapers (LEAP)
		1 radio interview in local/regional or national programme (LEAP)
Capacity strengthening	Collaborations	17 unique authors on peer-reviewed publications
		9 unique organisations represented on peer-reviewed publications
		4 non-academic institutions represented on peer-reviewed publications
		2 health centre staff representing LFTB/LEAP at conferences
	Indigenous representation	12 researchers/academics who identify as Aboriginal or Torres Strait Islander involved in LFTB or LEAP
		Indigenous co-leadership of LEAP
		16 Indigenous health centres involved in either LFTB and LEAP or both
		1 Indigenous lead author on a peer-reviewed publication
		5 unique Indigenous authors on peer-reviewed publications
		1 Indigenous researcher presenting LFTB Masterclasses and RCS teleconference
	Community engagement	51 health service users/consumers interviewed as part of the LFTB project
		6 newsletters sent out about LFTB
		3 newsletters sent out about LEAP
	Project team visits to each site	20 visits to individual health centres (LFTB and LEAP)
		55 health centre staff interviewed during team visits
	Face-to-face (F2F) meetings of team and services	4 face-to-face meetings conducted as part of LFTB and LEAP
		38 health centre staff involved in F2F meetings
	Monthly teleconferences	38 teleconferences with both the LFTB and LEAP Projects
		10 participants per teleconference (average)
	CRE_IQI Masterclasses	1 Masterclass delivered based on LFTB research/data
		31 participants attended this Masterclass
	RCS Teleconferences	1 RCS Teleconference presentation based on LFTB research/data
		19 participants who have attended these teleconferences
Policy	Policy engagement	2 meetings involving policymakers (eg, government, AMSANT, QAIHC)
		2 policy makers on the Management Committee for LFTB (and LEAP)
Economic	Research grants leveraged	2 grants directly leveraged from result/findings (LEAP and CRE-STRIDE (STRengthening systems for InDigenous healthcare Equity))
	Resources invested in health centres	$60 000
	Value of research grants leveraged	$A1.41 million

AMSANT, Aboriginal Medical Services Association of the Northern Territory; CRE-IQI, Centre of Research Excellence in Integrated Quality Improvement in Indigenous Primary Healthcare; LEAP, Leveraging Effective Ambulatory Practice; LFTB, Lessons from the Best to Better the Rest; QAIHC, Queensland Aboriginal and Islander Health Council; RCS, Research Capacity Strengthening.

Using an all teach, all learn approach,33 the LFTB project enhanced the capacity of high improving services through opportunities to learn from each other and about research from the researchers within the project team. Simultaneously, the researchers learnt about the challenges and opportunities for applying CQI within the Indigenous PHC context and the factors that enabled success. There was also capacity strengthening of researchers as they engaged with the Indigenous community and vice versa.12 Learnings from LFTB were also used to strengthen capacity in Indigenous community engagement among academic researchers through CRE-IQI masterclasses and reflection. An Indigenous researcher was also asked to co-lead the LEAP project and gain experience and skills which demonstrated her leadership capabilities and contributed to her taking on the position of co-lead investigator of CRE-STRIDE.

The LFTB team were also invited to meet with the NT Health Minister and the Aboriginal Medical Services Association of the Northern Territory (AMSANT) CQI Collaboratives to speak about LFTB. These meetings, in addition to including two policy makers on the Management Committee for both LFTB and LEAP, are important precursors to impact within the policy domain. Within the economic domain, LFTB was able to leverage nationally competitive funding from the Australian Commonwealth Government to translate the findings from LFTB to help striving services reach their CQI goals and continue wider CQI collaborations.

Economic analysis

Table 2 presents a cost consequence tabulation based on achievements of LFTB to date. The total research costs were valued at $A847 663 (SA $A800 680 to $A936 409). The total implementation costs were valued at $A66 242. The total research and implementation costs were $A913 904 (SA $A866 921 to $A1 002 650).

Table 2

Cost-consequence analysis

		2019 Australian dollar value
Costs	Direct research costs
	Staff salaries	$391 851
	Travel expenses for site visits, accommodation, catering (20 site visits, 3 workshops)	$201 996
	Contribution to participating centres	$63 788
	Indirect research contribution
	Opportunity cost for non-paid researcher time	$190 028
	Total direct/indirect research costs (Sensitivity analysis (SA)—min: $800 679.54; max: $936 408.51)	$847 663
	Implementation costs—health service
	Face-to-face meeting and workshops (3 sessions×2 days per session)	$76 280
	Site visits (20 sessions×1 day)	$21 189
	Teleconferences (26 sessions)	$11 805
	Interviews costs (54 interviews)	$4086
	Total health service costs	$113 361
	Reimbursement from research grant ($A60 000)	−$60 000
	Total health service in-kind contribution	$49 572
	Opportunity costs for health service users (51 members)	$11 335
	Opportunity costs for external stakeholders (24 members)	$5334
	Total direct and indirect implementation costs	$85 378
	Total costs (SA—min: $886 057.60; max: $1 021 786.57)	$933 041
Consequences	Consequence of LFTB
	LEAP funding leveraged to (1) assist striving services reach CQI goals; (2) develop resource tool kit to assist all Indigenous PHC services improve their quality of care; (3) build a learning community	$1 160 330
	Contribution to CRE-STRIDE (STRengthening systems for InDigenous healthcare Equity) (claiming 10%–20%)	$250 000–$500 000
	Total consequences	$1 410 330−$1660.330
	Professional development for health service staff	Benefit unable to be monetised
	Improvement in quality of care provided by striving services	Not yet available
	Improvement in corresponding health outcomes for the community	Not yet available
	A comprehensive list of consequences is not possible as translation and implementation of learnings is still ongoing and these consequences have not yet been realised

CQI, continuous quality improvement; LFTB, Lessons from the Best to Better the Rest; PHC, primary healthcare.

The consequences that could be monetised at this point included $A1 160 330 which represents the value of the grant that was leveraged from LFTB to further this work (100% attribution) and a further $A250 000 to $A500 000, which represents 10%–20% of the $A2.5 million awarded to CRE STRIDE. The total value of the consequences that could be monetised was between $A1 410 330 and $A1 660 330. Some of these leveraged funds will be used to employ Indigenous Australians, to develop CQI capacity in Indigenous health centres (including community-controlled services) with the aim of improving the quality of care provided to Indigenous Australians. Indigenous co-leadership is also a hallmark of both the LEAP project and CRE-STRIDE.

Narrative

A narrative describing the pathway from research need to impact for LFTB is displayed in box 1. This approach presents a brief story of the research and incorporates the more nuanced impacts on health centres, the staff who were involved, and on the communities they serve.

Need: High quality primary healthcare is essential to close the gap in Aboriginal and Torres Strait Islander health outcomes. Performance on key measures relating to quality of care provided to Indigenous communities shows wide variation between primary healthcare (PHC) centres. The effectiveness of continuous quality improvement (CQI) initiatives also varies across these centres. This variation was identified through data available from the Audit for Best Practice in Chronic Disease National Research Partnership (ABCD-NRP) conducted from 2010 to 2014. What was not understood was what factors facilitated the success of CQI initiatives within each service. Further analysis indicated that the variable trends in quality of care as reflected by CQI audit tools did not appear to be related to easily measurable health service characteristics suggesting the need for a deeper or more nuanced understanding of factors that moderate the effect of CQI on health service performance.

Research response: Lessons from the Best (LFTB) used a mixed-method participatory approach to understand what and how contextual factors interact to facilitate the success of CQI initiatives within a service. They interviewed 134 people who worked at the high improving health services and service users (Indigenous community members) to understand what works to improve quality of care. This involved three site visits per centre as well as numerous team meeting, workshops and monthly teleconferences involving the research partners and participant services.

Key findings: LFTB found that each of the 6 high improving services put CQI into practice differently. Overall, however, some common themes were that CQI was supported through collaborative decision-making; being embedded across orientation, training, health service operational systems, IT systems, team meetings, regional partnerships; and having strong links with community. There was also a ‘strong force’ of Aboriginal Health Practitioners and an appropriate Indigenous/non-Indigenous staff mix at each centre and the Indigenous workforce was valued and supported in engaging with the community to ensure that improvement processes were embedded in culture. LFTB also highlighted some strategies to improve care including involving all staff in CQI, listening and responding to community needs, ensuring the purpose of quality improvement is explicit and shared with the health centre team with a focus on improving client care and health outcomes.

Impact: LFTB advanced knowledge in this field via peer-reviewed publications, conference presentations and project briefs for community, health centres and policymakers. Reach of its publications are wide, having been read in places like South Korea, Iceland, Chile and Canada.

The participatory action research design and two-way learning approach ensured that there was a strong capacity strengthening component to LFTB which resulted in health centre staff strengthening new networks and skills in research as expressed by one Health Centre Manager: “talking to researchers and other services about what they do around quality because we are quite insulated in our own service – it was great for getting new ideas.” A Clinician Manager spoke about how this project “was very different–we were really engaged in it right from the start…we got a lot of support from the research team and we were made to feel a part of it”. This highlights how end-user engagement was undertaken upfront, a hallmark of sound research practice that optimises translation and downstream impacts. Another staff member spoke about the legacy that the project left within the health service, “Once you’ve participated in a process and moved forward – what was done becomes part of the foundation of what goes on. So I’m totally confident that it contributed to our ability now to pass our ISO accreditation, to maintain our AGPAL accreditation, and inform our current CQI activities.” Another impact of LFTB expressed by health centre staff was the strengthening of the health system. “The research that you guys did and the evaluation is probably the more beneficial stuff. We want research that’s going to benefit our communities in the medium to short-term. What I’m saying is the benefits have become entrenched in the organisation and has benefitted how we deliver services to the community and ultimately how we have structured ourselves to provide these services.”

In addition to health centre staff, LFTB also built capacity among researchers, government bureaucrats and policy makers. LFTB Project members ran Masterclasses and workshops to deliver Indigenous community engagement skills to CRE-IQI affiliates. Indigenous researchers were also invited on the team and given prominent roles thereby enhancing the capacity of Indigenous researchers.

LFTB also had economic impacts as the findings were used to leverage further funding to apply the lessons learned to help striving services. The Leveraging Effective Ambulatory Practice (LEAP) received $A1.1 million in funding from the National Health and Medical Research Council (NHMRC) between 2017 and 2020 to work with striving services who were engaging with CQI but not achieving their quality improvement goals. The money is being used to create a learning community and a toolkit of customable tools and processes to address key challenges with implementing CQI. The learning community includes staff from the striving services, the research team, partners and staff from the LFTB high improving services. The team will also be rigorously assessing the effectiveness, impact and acceptability of the intervention to ensure it is improving the quality of services provided to Indigenous Australians. In addition, LFTB has made monetary contributions to each of the striving services to acknowledge and value the contribution of those services to the research—“One of the really big benefits is that the Health Centre was given $10 000 and the Health Department agreed to allow us to spend it on new equipment … so we got more emergency gear and we got things to examine skin with and we got quite a few different things that the Health Centre had wanted but the Health Department couldn’t afford to buy so that was absolutely brilliant.” In addition, LFTB also contributed to securing funding for CRE-STRIDE (STRengthening systems for InDigenous healthcare Equity) which, among other things has actively sought to ensure Indigenous representation on the research team and will be continuing this work into the future.

The following section briefly draws out some of the main impacts that were expressed qualitatively by LFTB participants:

On a personal level, the experience of participating in LFTB was satisfying, both professionally and personally for some.

I’m sure (they) got a lot of kudos, personal satisfaction and professional satisfaction out of being part of a system that was successful…and by reflection, I guess, was positive about their own efforts. (speaking on behalf of staff who had been more directly involved in LFTB) (Clinician 1)

For others, it was an opportunity to be immersed in a research project which had varying impacts on different participants:

It’s just not my thing. But it was completely different for AA because she likes research and she loved going down to the meetings, she thought it was the most wonderful experience ever and she learnt a lot. (Health Service Manager 1)

The ability for high improving services to be engaged right from the start and to feel part of the research team was an important aspect of LFTB that built research capability and understanding among participating services.

I think one benefit was experiencing quite a different model of research in our space,… we were really engaged in it right from the start. We got a lot of support from the research team. We were also made to feel part of it, that our perspective on how we would like to see (the research) go was certainly well-received. (Health Service Manager 2)

LFTB also provided opportunities for health centre staff to present the results from the project, something normally reserved for academic research staff. This was a professional development opportunity not normally associated with participation in research.

We had the opportunity to present LFTB at the CQI Collaborative in Alice Springs. It’s run by AMSANT. So AA really wanted to do it. So she did everything and she loved it! (Health Service Manager 1)

LFTB also facilitated the sharing of knowledge and ideas among services:

Talking to researchers and other services about what they do around quality because we are quite insulated in our own service – it was great for getting new ideas….and we could apply these ideas to other sites that we managed who might not have been doing as well. (Health Service Manager 2)

One health centre staff member talked about the fact that projects like LFTB (and the ABCD CQI processes in general) have positively impacted how they deliver services to the community and this has had benefits for the community in the short to medium term:

The research that you guys did…the benefits have become entrenched in the organisation and this has benefitted how we deliver services to the community and ultimately how we have structured ourselves to provide these services. (Clinician 1)

A final serendipitous impact was the redirection of a portion of the $A10 000 (provided to each health centre as a form of reimbursement for their time) to other needs within their centres:

One of the really big benefits is that the Health Centre was given $10 000 and the Health Department agreed to allow us to spend it on new equipment…so we got more emergency gear and we got things to examine skin with and we got quite a few different things that the Health Centre had wanted for a long time but the Health Department couldn’t afford to buy so that was absolutely brilliant! (Health Service Manager 1)

Aim 2: feasibility, experience and outcomes from a retrospective application of the FAIT tool to LFTB

Although the logic model, a key component of the FAIT tool, was applied retrospectively, it proved to be useful for (1) documenting the path between LFTB and impact; (2) identifying metrics that could evidence impact from LFTB; and (3) raising awareness within the LFTB study team as to other benefits that had not been previously considered.

The Payback Framework captured the many capacity strengthening impacts of LFTB including uplift in the skills and competency of researchers. It also captured capacity strengthening among Indigenous researchers and healthcare providers and the co-leadership opportunities via the LFTB project and its spin-off, the LEAP Project.

Although a return on investment was unable to be completed, the economic analysis gave transparency to the cost involved in undertaking this project and the consequences of the project, some of which were unable to be monetised or reported in their natural units. Ultimately, this is an impact story that is still unfolding as the knowledge gains in LFTB are currently being implemented through LEAP. The narrative articulated the pathway from the need for the research through to the impact and expressed benefits from LFTB that could not be expressed in quantitative terms such as involvement in and understanding of the research process and the repurposing of LFTB funds such as the purchase of testing equipment to improve the delivery of health checks.

Discussion

Previous publications have focused on the outcomes and co-created learnings from the LFTB project11–13 but this is the first time that an attempt has been made to understand the contribution this project made to the end-users of the research—individuals, organisations and society. In the case of LFTB, the immediate end-users are the Indigenous PHC centres involved in the project, downstream end-users are the striving services involved in the LEAP project and the ultimate beneficiaries are the Indigenous communities served by these centres. The focus of the study was on the immediate end-users but given that active translation is ongoing (via LEAP) and a participatory two-way learning approach was used, we could also consider some benefits to striving services and the Indigenous community. Broadly, the application of FAIT to LFTB evidenced its impact on knowledge advancement, policy and the economy, but were mainly in capacity strengthening.

From the perspective of participating health centres and their staff, LFTB had many benefits. Staff at these centres benefitted through their own personal development in relation to CQI practice and research, their involvement as coproducers of the research, the opportunity to present the findings of the research, the networking and learning opportunities through exposure to like-minded professionals and the fact that the learnings had become entrenched in their ways of working. Health centres benefitted through recognition of their ongoing high performance, exposure to additional strategies for CQI success and the contribution of funds by LFTB some of which had been used to extend PHC provision to their patients. While not a common research practice, the contribution to health services for involvement in research is becoming more commonplace and should be considered in future impact assessments.

The main strength of this study is that it provides a different lens—one that focusses on the benefits of the research and its impact rather than the results and learnings. This type of information is useful for policy makers and funders of research who want to understand not just what the research found but what difference it made and to whom. Another group interested in the benefits are the Indigenous community. In the paper, No one is discussing the elephant in the room, Bainbridge and colleagues question whether ‘the abundance of research conducted; purportedly to improve health of Indigenous people, is justified and benefits Indigenous people in ways that are meaningful and valued by them’ p2.6 The LFTB project interviewed 51 health service users (Indigenous community members) which helped to ensure that the ‘communities’ views were embedded in learnings and reflections from LFTB. This participatory, two-way learning approach to research and knowledge generation also helped engage Indigenous communities in quality improvement processes with their local services in a way that was meaningful and ensured that their views of ‘quality’ are incorporated in future CQI activities.

Second, the mixed-methods approach was a good fit with the strengths-based participatory research approach of the LFTB project. It allowed for impacts to be described holistically: quantitatively in terms of suitable metrics, qualitatively through the voices of the beneficiaries and in economic terms through a CCA. This multimethod approach to impact assessment was a second key strength of this study.

There were, however, some key limitations that need to be acknowledged.

Timing

The timing of this impact assessment (post-LFTB and pre-LEAP) meant that it is too soon to report the impacts on striving services involved in LEAP and on potential downstream impacts such as better outcomes for patients, their families and communities. This limitation of impact assessment using the available evidence can be addressed to some extent by the inclusion of simulation modelling to estimate possible downstream health outcomes. Given the highly specialised setting of LFTB-CQI within Indigenous PHC—and its discovery phase of research, such economic modelling was not suitable. The underlying assumptions for key parameters would be extensive. Hence, in the absence of simulation or other related modelling that estimates future outcomes, the timing of this impact assessment limited the ability to fully report impact from LFTB. This issue of lag in the health research translation process, and a delay in impact, has been previously highlighted34 and is an ongoing challenge. Funders, policy makers and society want to know what the impacts and returns have been for funded research, the reality is that the answers they seek may not be immediately forthcoming and evidence-based answers may take longer than they would like.

Retrospective versus prospective application

Prospective application of FAIT was not possible in LFTB because the research was largely complete at the start of the impact assessment. The retrospective application made two important impact-related strategies impossible. First, though possible, it was too costly and impractical to interview a sample of health service users who participated in LFTB to understand the impact that experience had on them. This information, had it been financially and practically feasible to collect, would have identified potential benefits and costs that were not immediately obvious to either the independent assessor (SAR) or the LFTB study team. Prospective data collection would have avoided this limitation. Second, it is important that the end-user’s understanding of what benefit looks like is included and considered in an impact analysis. This is imperative in Indigenous health research where an understanding of what impact looks like for individuals, families and community is critical as these views on benefit may differ from the views of non-Indigenous Australians. In a retrospective impact analysis, it is not possible to consult relevant end-users, ensure their view of impact is captured and reflected in the impact metrics that have the potential to help shape the research pathway that will link to those impacts. Prospective application would have allowed the impact assessment to capture impacts that are important to end-users, so research managers have the opportunity to guide the research to optimise the realisation of these impacts.

Attribution

Research is often iterative with discrete projects forming part of a larger programme of work that, over time, covers the entire research pipeline from discovery to application, through to widespread adoption and scale-up. LFTB heavily leveraged findings from the ABCD programme of work which contributed data and expertise to LFTB. Similarly, the findings from LFTB were the catalyst for attracting funding for LEAP. The application of FAIT to LFTB was an isolated impact assessment in a chain of connected events. Hence, the impact assessment does not account for (1) the investment in the ABCD Programme of work which provided the collateral (data and research question) on which LFTB was based; and (2) benefits from LFTB that can legitimately be claimed once LEAP is completed. Figure 2 presents a graphical representation of this issue of lag and attribution. Future applications of FAIT may wish to consider applying the framework to multiproject research programmes or portfolios (like LFTB and LEAP combined) rather than at the project level given how difficult it is to disentangle the learning and funding from one project which is closely linked to another. There is certainly a precedent for a portfolio-type approach in the field of impact assessment.35

Figure 2

Timeline for implementation of FAIT to LFTB. ABCD-NRP, Audit for Best Practice in Chronic Disease National Research Partnership; CRE-IQI, Centre for Research Excellence for Integrated Quality Improvement; FAIT, Framework to Assess the Impact from Translational health research; LEAP, Leveraging Effective Ambulatory Practice; LFTB, Lessons from the Best to Better the Rest.

Linear approach to research

The FAIT framework assumes a linear, sequential approach to research that maps a pathway from need to impact. This is not an accurate representation of how the LFTB project (or any participatory research project) is implemented in the field. Translation can and did occur at different stages of the research process. The application of FAIT simplifies this for the purposes of assessing impact. This poses a limitation for readers who want a clear understanding of the LFTB project and its inherent complexities. Other publications from the LFTB project10–13 are better sources for this type of process information.

The assessment of the implementation of the FAIT tool was informed by interactions with others in the LFTB study team, and the CRE-IQI evaluation team including a lead author of the FAIT framework (AS). However, the perspective reported in this paper is predominantly that of the lead evaluator charged with implementation of the FAIT tool, collecting relevant data and writing up the results (SAR).25 This perspective may not reflect that of the broader pool of evaluators but will only be addressed as implementation of FAIT is scaled up.

Conclusion

As a participatory, comprehensive knowledge-creation project, LFTB (1) advanced knowledge in the field of CQI in Indigenous PHC; (2) strengthened CQI and/or research capacity of participating health centres, their staff, academic researchers and the community; and (3) leveraged between $A1.4 and $A1.6 million to further translate the knowledge from the research to assist striving services improve the care they provide to Indigenous Australians and continue to extend application of CQI processes through CRE-STRIDE.

Despite significant limitations in its application, FAIT was found to be suitable for undertaking an impact assessment of LFTB and the multimethod approach provided a holistic view of impact. A subsequent impact assessment applied after LEAP is completed (and potentially incorporating the larger programme of work) may yield a more comprehensive assessment of impact including an estimate of the return on investment that was missing from this study.

Future impact assessments using FAIT should consider implementing the framework prospectively prior to the commencement of the research so end-users’ views of impact can be incorporated; where beneficial, apply FAIT to a programme of work rather than a discrete project; and ensure the timing of the assessment is such that there has been an opportunity for downstream impacts to be realised to the point where simulation modelling of benefits can be undertaken.