Economic evaluation of clinical quality registries: a systematic review.

OBJECTIVES: The objective of this systematic review was to examine the existing evidence base for the cost-effectiveness or cost-benefit of clinical quality registries (CQRs).
DESIGN: Systematic review and narrative synthesis. DATA SOURCES: Nine electronic bibliographic databases, including MEDLINE, EMBASE and CENTRAL, in the period from January 2000 to August 2019. ELIGIBILITY CRITERIA: Any peer-reviewed published study or grey literature in English which had reported on an economic evaluation of one or more CQRs. DATA EXTRACTION AND SYNTHESIS: Data were screened, extracted and appraised by two independent reviewers. A narrative synthesis was performed around key attributes of each CQR and on key patient outcomes or changes to healthcare processes or utilisation. A narrative synthesis of the cost-effectiveness associated with CQRs was also conducted. The primary outcome was cost-effectiveness, in terms of the estimated incremental cost-effectiveness ratio (ICER), cost savings or return-on-investment (ROI) attributed to CQR implementation.
RESULTS: Three studies and one government report met the inclusion criteria for the review. A study of the National Surgical Quality Improvement Programme (NSQIP) in the USA found that the cost-effectiveness of this registry improved over time, based on an ICER of US$8312 per postoperative event avoided. A separate study in Canada estimated the ROI to be US$3.43 per US$1.00 invested in the NSQIP. An evaluation of a post-splenectomy CQR in Australia estimated that registry cost-effectiveness improved from US$234 329 to US$18 358 per life year gained when considering the benefits accrued over the lifetime of the population. The government report evaluating five Australian CQRs estimated an overall return of 1.6-5.5 times the cost of investment.
CONCLUSIONS: Available data indicate that CQRs can be cost-effective and can lead to significant returns on investment. It is clear that further studies that evaluate the economic and clinical impacts of CQRs are necessary. PROSPERO REGISTRATION NUMBER: CRD42018116807. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This review provides an updated overview of published literature regarding the economic value of clinical quality registries (CQRs).

There was considerable variation in participants, comparators and outcomes across the studies.

There was limited transferability of cost-effectiveness or cost-benefit results across the studies.

The methods used for the assessment of the true clinical and cost impacts attributed to CQR implementation is poorly established.

Few studies or reports were identified, highlighting the need for more economic evaluations of CQRs.

Introduction

Clinical quality registries (CQRs) are clinical registries that collect data for the purposes of feedback to healthcare systems and facilitating quality improvement. They are increasingly being used to inform improvement of healthcare processes, adherence to evidence-based guidelines and standards, and maximisation of healthcare safety and quality.1 2 Previous studies have demonstrated that CQRs contribute to improvements in patient outcomes across a variety of healthcare settings comprising surgical, psychological and medical care.2–5 The opportunities to gain real-world evidence from CQRs facilitate continuous efforts to improve patient care, clinical research, benchmark hospital performance, and health service planning and prioritisation.1 4 6 Such attributes have led to an increasing number of economic evaluations of health interventions based on CQR data.7–9 Additionally, registries are increasingly used as platforms to facilitate randomised controlled trials (RCTs), which allows for lower trial costs, more rapid enrolment of representative patients and more complete follow-up.10–12

Despite their benefits, few studies have assessed the health economic impact of CQRs. In a recent systematic review, CQRs were found to improve measures of healthcare processes and patient outcomes.4 13 14 Further, Hoque et al contended that CQRs were likely to reduce health resource use based on two exploratory studies. However, no cost-effectiveness or cost-benefit studies were identified, and studies performing economic evaluations of CQRs were not a primary consideration of the review. Hoque et al 4 noted the development and implementation of CQRs is associated with significant costs, and at present, their cost-effectiveness has not been well studied. Robust methods for the economic evaluation of CQRs are also warranted in order to inform government decision-making and to ensure that CQRs are developed in the most sustainable and effective manner. The objective of this systematic review was to examine the existing evidence base for the cost-effectiveness or cost-benefit attributed to CQRs.

Methods

We conducted a systematic review for studies that involved economic evaluations of one or more CQRs.

Definition of a CQR

CQRs are a subgroup of clinical registries intended for the provision of feedback to registry participants and healthcare providers for improving quality of care.1 4 Clinical registries collect a well-defined, minimum dataset of patient and clinical characteristics.15 To allow for ongoing benchmarking and feedback, CQRs must collect a well-defined clinical dataset of measures using standardised processes.6 Such data should be collected within close proximity to the initial time of treatment, and include sufficient details pertaining to patient characteristics, risk factors and predetermined clinical outcomes or performance indicators. Additionally, CQRs must have adequate patient follow-up to allow for the identification of temporal trends in key risk-adjusted outcomes. This is necessary to improve the clinical utility of risk-adjusted outcomes data for the purposes of benchmarking and quality improvement.

Inclusion and exclusion criteria

Study design and intervention

Studies were included if they were conducted in a clinical healthcare setting. This includes acute care (hospital inpatient and outpatient care), subacute care (patient rehabilitation) and community care (general practice or aged care).4 Studies were excluded if the CQR did not provide ongoing feedback or only collected data from one hospital or care centre.

Various study designs were considered, but a key criterion for study inclusion in the present review was that the CQR was considered a health intervention for the economic evaluation. Although economic evaluations using data from RCTs are associated with the greatest rigour, randomisation and selection of an appropriate control group for assessing CQRs as health interventions is rarely performed. Hence, different study designs, such as cohort and ‘before and after’ studies, were considered in this review, as various methods have been used to assess the clinical and cost impacts of CQRs in lieu of an RCT setting.4

A description of the population, intervention, comparison and outcome (PICO) for our systematic review, and a summary of the inclusion and exclusion criteria are presented in table 1. The PICO was based on the systematic review by Hoque et al 4 of CQRs, with the primary outcome adjusted to focus on CQR cost-effectiveness based on incremental cost-effectiveness ratios (ICERs), cost savings or return on investment (ROI).

Table 1

Description of the PICO of the systematic review

PICO	Description
Population	Patients across clinical care environments, including: Acute care (inpatient and outpatient); Subacute care (rehabilitation); OR Community care (general practice or aged care)
Intervention	Inclusion criteriaCQRs, defined based on the following inclusion criteria: A clinical registry which collects data on a procedure, disease or healthcare resource; AND Data is systematically collected on an ongoing basis from the population of interest; AND Provide continuous feedback on health system performance; AND Collect data from more than one hospital or care centre Exclusion criteriaStudies were excluded: if the CQR: Only collected and reported data from one hospital or care centre* Did not provide ongoing feedback If they were written in a language other than English If published prior to start date on 1 January 2000 or after the end date of 15 June 2018 Review articles Not an economic evaluation of CQRs
Comparison	Comparators included: Data collection tools other than registries to monitor health outcomes Population-based or administrative data or medical records Studies without comparators were included
Outcome	Cost-effectiveness based on ICERs/cost savings/ROI (primary outcome) Clinical outcomes (secondary outcome)

*Economic evaluation studies based on the perspective of a single centre were eligible for inclusion.

CQR, clinical quality registry; ICER, incremental cost-effectiveness ratio; PICO, population, intervention, comparator, outcome; ROI, return on investment.

Comparator(s)/Control

Comparators included data from hospital or insurance databases, or clinical information systems measuring key outcomes of interest such as mortality or survival. Studies without comparators were also eligible for inclusion.

Outcome measures/Outcome of interest

The primary outcome was the cost-effectiveness of a CQR in terms of the ICER, cost savings or ROI. Studies assessing the effectiveness attributed to CQRs consider prespecified measures of change in key performance indicators (KPIs) collected by registries as indicators of the benefits directly attributed to registry operation. These include reductions in patient mortality or improved clinical outcomes, improvements to healthcare quality or more efficient utilisation of healthcare resources. These indicators of CQR effectiveness are used to inform the cost savings resulting from CQR operation, and are considered alongside the costs of implementing the registry in economic evaluations.4 Additionally, for the purposes of this systematic review, these indicators of CQR effectiveness are presented as secondary outcomes.

Discrepancies between cost outcomes presented in the identified studies were corrected through applying relevant purchasing power parities provided by the Organisation for Economic Co-operation and Development, and adjusted to 2017 US$ using the relevant Consumer Price Index.16–19 This facilitates the comparability or transferability of results across the identified studies.

Patient and public involvement

No patients or the public were involved in this review.

Search strategy

The search strategy was developed by the research team and refined through peer review by a senior medical librarian in accordance with the Peer Review of Electronic Search Strategies (PRESS) statement.20 Records of each search strategy were maintained throughout the review process, including details of how each search was undertaken and search output.

The following electronic databases were searched: MEDLINE, EMBASE, Cochrane Central Register of Controlled Trials (CENTRAL), Cochrane Database of Systematic Reviews; Cochrane Database of Methodology Reviews, The Database of Abstracts of Reviews of Effectiveness (DARE); the National Health Service (NHS) Health Technology Assessments, NHS Economic Evaluation Database and The American College of Physicians Journal Club to identify studies in English covering the period January 2000 to August 2019. The search strategy included key words describing studies involving clinical registries as the intervention. The medical subject heading (MeSH) terms relating to ‘registry’ or ‘registries’ were combined with the MeSH terms referring to ‘cost’ or ‘economic evaluations’ in the search (see online supplementary appendix table S1).

Additionally, a search of grey literature resources such as government websites and conference abstracts was performed. Finally, hand-searching and reference-checking of citations and reference lists were also performed.

Study screening and selection

All titles and abstracts that potentially met the inclusion criteria were screened for inclusion by two independent authors in duplicate (PL and KC). Full-text articles of potentially suitable studies were retrieved and assessed for eligibility by two team members (PL and KC). Any disagreement between the two reviewers was resolved through discussion between the reviewers and with other members of the team.

Data extraction and assessment of quality

Five criteria for the assessment of CQR quality were identified by Black et al.4 21 These key areas include the nature of CQR recruitment, data completeness, the independent collection of explicitly defined outcomes data and the validation of data collected by CQRs. Attributes that are associated with these areas, such as the extent of registry coverage and database management processes, were extracted. This method of data extraction was also used in a prior systematic review on CQRs as an intervention.4 21

Two authors (PL and EZ) independently reviewed and extracted data from eligible full-text studies following title and abstract screening in duplicate. A standardised data extraction form based on the Cochrane data abstraction form was used to extract data regarding study design and methods, country setting, economic perspective, characteristics of patients or participants and interventions and study outcomes. Discrepancies were resolved by consensus with a third reviewer (DL).

Study quality, risk of bias and transferability

Study quality and risk of bias were assessed by two authors (PL and EZ), independently and in duplicate. The reporting quality of each study was assessed using either the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement, or the Authority, Accuracy, Coverage, Objectivity, Date, Significance (AACODS) checklist.22 23 Where possible, risk of bias was assessed using the Consensus Health Economic Criteria (CHEC)-extended checklist.24 In systematic reviews of economic evaluations, the CHEC-extended checklist is a validated method for the appraisal of the internal and external validity, and risk of bias, of economic evaluations.25 As discussed above, costs were converted to 2017 US$ to improve the comparability, or transferability, of results among studies. In systematic reviews of economic evaluations, an assessment of the transferability of results is recommended to facilitate the interpretation of results across a variety of populations or healthcare settings.25 26 Due to the heterogeneity of data collected between different CQRs and across different diseases or health conditions, it was unlikely that a more rigorous assessment of transferability of study results, as described in Welte et al,16 would be feasible.

Data synthesis and analysis

Narrative synthesis

A narrative synthesis of findings across the identified studies was performed, in accordance with a published guidance on narrative synthesis for systematic reviews.27 The research team had planned a narrative synthesis of results given the likelihood of high heterogeneity across the identified studies, as noted in Hoque et al.4 First, a narrative synthesis was performed around key attributes of each CQR, such as the referenced patient population and collection of data. A narrative synthesis of study findings on the cost-effectiveness or ROI around CQRs, and around key patient outcomes or changes to healthcare processes or utilisation was then conducted.

Results

Search results and description of the included studies

Figure 1 summarises the selection process used in this systematic review in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram.28 A primary database search returned 2376 articles, and 5 articles were identified through additional hand-searching and reference-searching of citations. Following removal of duplicate articles, the number of studies was reduced to 2057, of which 1916 were excluded after review of titles and abstracts. After reviewing 141 full-text articles, three studies and one report were selected for the final synthesis. All selected articles were published between 2006 and 2018. Other studies considered in full-text screening were excluded as they were not economic evaluations of a CQR (n=81), not CQR-based (n=26) or were review only (n=30). These reviews described CQR characteristics or assessed the potential utility of CQRs in research and in improving patient outcomes but did not include economic evaluations.

Figure 1

Results of the search strategy. CQR, clinical quality registry.

Table 2 presents details of the included studies and key study characteristics. Three studies describe CQRs which were established in the USA (n=1), Canada (n=1) and Australia (n=1).29–31 Two studies described CQRs aiming to improve surgical outcomes, and the Australian study described the impact of a registry monitoring health outcomes following splenectomy.29–31 The study by Hollenbeak et al was included as although the evaluation was assessed from the perspective of one hospital site, the registry itself captured data nationally in the USA.29 One report identified in the grey literature search evaluated the following Australian CQRs: the Victorian Prostate Cancer Registry (PCR), the Victorian State Trauma Registry (VSTR), the Australia and New Zealand Intensive Care Society Adult Patient Database (ANZICS), Australia and New Zealand Dialysis and Transplantation Registry (ANZDATA) and the Australian Orthopaedic Association National Joint Replacement Registry (AOANJRR).13 In this report produced by the Australian Commission on Safety and Quality in Healthcare (ACSQHC), referenced populations included patients with prostate cancer, trauma patients, patients undergoing renal dialysis and transplantation, patients in intensive care and patients with knee or hip joint replacement.13

Table 2

Characteristics and findings of studies included in the systematic review

Characteristic	Study
	Hollenbeak (2011)	Thanh (2018)	Woolley (2006)	ACSQHC (2016)
Registry	NSQIP	NSQIP	Victorian Spleen registry	Five CQRs in Australia and New Zealand
Condition of interest	Surgical outcomes (general and vascular)	Surgical outcomes	Sepsis	Prostate cancer (VIC PCR), trauma (VSTR), intensive care (ANZICS-APD), renal transplantation (ANZDATA), joint replacement (AOANJRR)
Country	USA	Canada	Australia	Australia
Perspective	Hospital	AHS	Australian healthcare sector	Societal
Reference population	2229 adult general and vascular surgical inpatients at a single academic centre	Patient data from five acute care facilities across all 5 AHS operational zones (QEII, UAH, RDRH, RGH, CRH)	Asplenic patients in Victoria	Victorian Prostate cancer (VIC PCR), Trauma (VSTR), Australian and New Zealand Intensive care (ANZICS-APD), Renal transplantation (ANZDATA), Joint replacement (AOANJRR)
Coverage	National	5 AHS operational zones in Alberta	Victoria	Australia and New Zealand
Sources of data	NSQIP	NSQIP (AHS in Canada)	Registry cohort Victorian Spleen Registry Non-registry cohort Literature	KPIs collected by each registry
Database management	Clinical nurse reviewer	ND	ND	Clinical data managers or custodians, or data management centres
Follow-up	30 day postoperative outcomes	Intraoperative and postoperative events within 30 days of surgery for patients in five pilot sites before and after QI intervention	ND	Varied from no follow-up to 24 months
Reporting and feedback	Annual report  Development of interventions to target performance measures	Twice-yearly reports  Different interventions are initiated and customised by each pilot site, based on key indicators	ND	Varied between quarterly, biannual and annual reporting to key audiences, and contribution of data towards research and benchmarking initiatives
Time period	2007–2009*	2015–2016	Hypothetical 2-year period from introduction of registry	VIC PCR: 2009–2013VSTR: 2005–2013ANZICS: 2000–2013ANZDATA: 2004–2013AOANJRR: 1999–2014
Clinical outcomes	Reduction in postoperative events:Analysis 1: 3.27%Analysis 2: 3.63%	Reduction in:  risk of SSI: −5.55%–2.45%  risk of UTIs: 1.01%–1.89%  risk of blood transfusion: 3.63%  LOS (days): 6  risk of readmissions†: 18.87%	Over 2 years:Years of life gained: 1.2OPSI cases avoided: 1.6Over lifetime:Years of life gained: 81.8OPSI cases avoided:12.5	Overall reduction in:  Patients with a PSM following radical prostatectomy  Patients receiving unnecessary active treatment  Registry-specific length of stay and mortality measures  Transplant graft loss  Peritonitis hospitalisations  Hip and knee replacement revisions
Costs outcomes	Change in total costAnalysis 1: US$945Analysis 2: US$302	Total gross savings: US$9 130 312Total cost of CQR: US$2 109 507	Over 2 years:Additional costs for registry cohort: US$271 999Cost per case of OPSI avoided: US$173 656Over lifetime:Additional costs for registry cohort: US$1 499 858Cost per case of OPSI avoided:US$119 655	Gross attributed benefit:US$4.1–US$50.9 millionCosts avoided:US$0.9–US$28.2 millionQALYs preserved: US$3.0–US$34.5 millionRegistry costs:US$2.1–US$10.2 million
Cost-effectiveness/ROI‡	Analysis 1ICER§: US$28 927Analysis 2ICER: US$8312	ROI (total gross savings/costs of the CQR): 3.4	Cost per life year gained over 2 years:US$234 329Cost per life year gained over lifetime:US$18 358	Benefit to cost ratio:1.6–5.5Internal rate of return:23%–52%

*Two analyses were performed: Analysis 1 measured the change in costs and patient outcomes associated with the 6-month start-up of the NSQIP for 1 year, while Analysis 2 measured the change in costs and events 1 year following the full implementation of the NSQIP.

†Cystectomy patients.

‡All costs were reported in 2017 US$ values.

§ICER based on cost per postoperative event avoided.

AHS, Alberta Health Services; ANZDATA, Australia and New Zealand Dialysis and Transplant registry; ANZICS-APD, Australian and New Zealand Intensive Care Society – adult patient database; AOANJRR, Australian Orthopaedic Association National Joint Replacement Registry; CRH, Chinook Regional Hospital; DMAC, Data Management and Analysis Centre; ICER, incremental cost-effectiveness ratio;KPI, key performance indicator; LOS, length of stay; ND, not discussed; NSQIP, National Surgical Quality Improvement Programme; OPSI, overwhelming post-splenectomy infection; VIC PCR, Victorian Prostate Cancer Registry; PSM, positive surgical margin;QALY, quality-adjusted life years; RDRH, Red Deer Regional Hospital; RGH, Rockyview General Hospital; ROI, return on investment; SSI, surgical site infection;UAH, University Hospital of Alberta; UTI, urinary tract infection; VSTR, Victorian State Trauma Registry.

A ‘before-after’ study design, where the benefits and costs of the registry were compared with a defined preregistry period, was used in two studies.29 30 A hypothetical experimental study design was considered for the study by Woolley et al and the ACSQHC report.13 31 With this design, the benefits and costs of a CQR compared with the hypothetical situation where no CQRs exist were estimated based on trends in patient outcomes and evidence from literature or external data sources.

The economic evaluations performed across the three studies were highly varied. In an evaluation of the National Surgical Quality Improvement Programme (NSQIP) by Hollenbeak et al, the costs per patient admission and registry costs were estimated from the hospital perspective.29 The operating costs per patient admission were obtained through aggregating all department-level costs estimated as a percentage of hospital charges from a hospital accounting database.29 The costs of the surgical CQR were inclusive of administrative and training costs and other costs associated with CQR adoption and maintenance. Registry costs were added to the costs of patient admission in the post-CQR period, allowing for the evaluation of the cost-effectiveness of the CQR following its implementation. An ICER was determined by dividing the difference in mean costs between a pre-CQR and post-CQR period with the difference in average postoperative events.29 The primary outcome of cost per postoperative event avoided was used in the determination of registry cost-effectiveness.29 It should be noted that although the NSQIP is implemented across the USA, the economic evaluation adopted the perspective of a single institution.

In the evaluation presented in Thanh et al,, a decision analytical model was used to estimate the potential cost savings and ROI associated with a CQR collecting data on postsurgical outcomes.30 Five acute care facilities across all five Alberta Health Services (AHS) operational zones were considered in this study (table 2). Briefly, this model was based on the difference in probability of a surgical event occurring before and after the introduction of a surgical CQR, multiplied by the number of patients undergoing surgery and the unit healthcare costs per event.30 A health system perspective (AHS) was considered for this evaluation. The ROI associated with the CQR was estimated through determining the ratio of total gross savings to the total costs of the CQR. In the model considered in this study, changes in the rate of surgical site infections, urinary tract infections and reductions in cystectomy readmissions and length-of-stay were considered as key measures of efficacy in this study.30 These measures were based on site-specific interventions developed in response to NSQIP data recommendations. Correspondingly, the costs considered in the model include the direct and indirect unit costs for surgical site infections, urinary tract infections, cystectomy length-of-stay and readmissions, orthopaedic blood transfusion and registry costs.

A Markov decision model was used in Woolley et al to evaluate the cost-effectiveness of a post-splenectomy registry.31 The methodological approach in this study relied on external data sources rather than data collected directly from the CQR. The estimated uptake rates of vaccination, chemoprophylaxis and educational programme for patients enrolled in the registry, and for patients with no registry involvement, were included in the model to explore the comparative efficacy of the CQR in reducing the rate of overwhelming post-splenectomy infection and mortality.31 The unit costs of recommended interventions and unit costs associated with overwhelming post-splenectomy infection cases were estimated using cost data from a single hospital site. The ICER determined in this study was based on the cost per case of overwhelming post-splenectomy infection avoided and cost per life year gained.

In the ACSQHC report, KPIs for each registry were assessed for temporal changes.13 Control groups providing data on KPIs, but with differential (or no) application to a registry, were used to adjust for confounding factors to assess benefits attributed to CQRs rather than extraneous benefits captured by a registry.13 Subsequently, the economic value attributed to the benefits gained through the operation of the registry was assessed against registry costs using established methods for valuing improvements to patient survival and quality of life.13 For example, the reduction in incidence of positive surgical margins following radical prostatectomy, and in active treatment for low-risk prostate cancer patients were selected as key indicators of registry efficacy in the evaluation of the Victorian PCR.13 Mortality and patient quality-adjusted life years were considered as key outcome measures, as reductions in the incidence of positive surgical margins and rate of active treatment in low-risk patients are associated with reduced mortality and improved quality of life. The rate of improvement in these indicators for early contributors to the PCR were compared with late contributors to determine the influence of the registry on observed changes to rates of PSM and to active treatment rates for low-risk patients. In the analyses of the PCR, only the incremental improvement in early contributing hospitals was attributed to registry operation; furthermore, the mode of treatment was observed to be constant over the period of analyses, and changes in surgical practice were assumed to be uniform across hospitals. The cost of a statistical life year ($A182 000), costs associated with secondary treatment for positive surgical margins, and costs of active treatment for low-risk patients were used in the evaluation to estimate the cost savings associated with the PCR.13 The expected ROI and benefit-to-cost ratio were then estimated through comparing the cost savings associated with the PCR with the costs of establishing and running the registry.

Quality of the included studies

The reporting quality of the three published studies was assessed against 24 checkpoints using the CHEERS statement.22 The risk of bias for the three published studies was assessed using the 20-item CHEC-extended checklist.24 32 A risk of bias assessment using the CHEC-extended checklist was not performed for the ACSQHC report; rather, the quality of the ACSQHC report was assessed against 14 checkpoints using the AACODS checklist.23 A score of 1 was allocated for each point met in full, 0.5 for each point partially met and 0 for each point which was not met across the CHEERS and CHEC checklists. The total score was summed and converted to a percentage ranging from 0% to 100%, with the maximum possible score as the denominator. At least 83% of the CHEERS criteria were met by each study, and the report identified in the grey literature search met all of the AACODS criteria (see online supplementary appendix tables S2 and S3). Overall, the reporting quality was very high across the studies. Out of a maximum total score of 20, Woolley et al scored 16 (80%) and both Hollenbeak et al and Thanh et al had reported 17 (85%) of items in the CHEC-extended checklist (see online supplementary appendix table S4). Overall, the risk of bias across the three studies as appraised using the CHEC-extended list ranged from moderate to low.24 32

Common limitations were noted to affect the internal and external validity of each identified study. A common key issue noted across the studies was the nomination of an appropriate comparator arm. In the two ‘before-after’ studies, it was recognised that there might be heterogeneity in characteristics between the pre-CQR and post-CQR populations.29 30 Limited data for a suitable comparator arm contributed to uncertainty in the interpretation of findings for the study presented by Woolley et al and the ACSQHC report.13 31 Furthermore, there was uncertainty around the extent to which a CQR contributed to improved patient outcomes and cost savings; this may be attributed to the issue of determining an appropriate comparator. Finally, the decision to include ongoing benefits varied across studies and contributed to uncertainty around the true level of cost-effectiveness or ROI attributed to CQRs.

Registry impact on processes of care or clinical outcomes

Table 2 presents the results of each economic evaluation performed across the three identified studies and the ACSQHC report. The three studies described quality improvement initiatives implemented through the CQR. In the study by Hollenbeak et al, data from the NSQIP led to the development of a glucose control protocol and a patient risk assessment for the development of venous thromboembolism to reduce the number of postsurgical events.29 This resulted in a reduction in postoperative events of 3.27% over 1 year (July 2008–December 2008) following the 6-month introductory period of the NSQIP (July 2007–December 2007), and 3.63% between the second (July 2008–June 2009) and first year (July 2007–June 2008) of full registry implementation.29

Tailored measures were implemented to reduce orthopaedic surgical site infections (SSIs) and urinary tract infections in the study by Thanh et al.30 These interventions were implemented based on key areas of improvement identified with NSQIP data across five hospitals in Canada, and led to reductions in SSI risk across three hospitals.30 The risk of urinary tract infections, orthopaedic blood transfusions, cystectomy readmissions and length-of-stay were also reduced due to these interventions.

Each of the five CQRs described in the ACSQHC report led to the implementation of initiatives to improve patient outcomes and reduce costs. Interventions include the development of a case review group to facilitate the management of major trauma patients (VSTR) and the provision of venous thromboembolism prophylaxis (ANZICS).13 Overall, these interventions coincided with a reduction in registry-specific length-of-stay and mortality measures, and improved patient outcomes (see table 2).

Financial impact of CQRs

In the study presented by Hollenbeak et al, two analyses were performed to assess the cost-effectiveness of the NSQIP from a provider perspective of a single academic centre in the USA.29 In the first analysis, the change in costs and patient outcomes between the 6-month start-up period of the NSQIP and the year following this period were measured. In the second analysis, the change in costs and events occurring between the second and first year following the full implementation of the NSQIP were measured. The ICER ranged from US$28 927 (US$25 471 in 2009) per postoperative event avoided (Analysis 1) to US$8312 (US$7319 in 2009) per postoperative event avoided (Analysis 2).29 The authors concluded that the NSQIP was cost-effective and continued to improve in cost-effectiveness over time. Similarly, the cost-effectiveness attributed to a CQR for reducing patient mortality following splenectomy improved from US$234 329 ($A205 931 in 2006) per life year gained in the initial 2 years of patient registration, to US$18 358 ($A16 113 in 2006) per life year gained when extending benefits over the lifetime of the cohort in the analysis presented by Woolley et al.31 The value of investing in a CQR was further supported in an evaluation of the NSQIP across five hospitals in Canada by Thanh et al; a return of US$3.43 ($C4.30 in 2017) was expected for each US$1.00 ($C1.00) invested in the NSQIP based on an ROI of 3.4, which was adjusted from an ROI of 4.3 estimated by the authors.30 The ACSQHC report on five Australian CQRs determined that investment in a CQR would lead to an overall return of 1.6–5.5 times the cost of investment, adjusted from ROIs ranging from 2 to 7 estimated in the report.13

Discussion

The primary objective of the present systematic review was to evaluate the economic impact of CQRs. Globally, the usage of CQRs has rapidly increased, with over 100 extant in Sweden alone.14 33 Outside of Sweden, the implementation of CQRs such as the American Heart Association/American Stroke Association Stroke Registry and heart failure registries in the USA, and the National Hip Fracture Database in the UK have led to significantly improved patient outcomes.5 Despite the significant rise of CQRs worldwide monitoring a diverse array of clinical and procedural outcomes, we identified only three studies which performed an economic evaluation of a CQR, and an Australian report evaluating the economic impact of five CQRs. All identified studies demonstrated that CQRs can be cost-effective and can lead to significant returns on investment.13 29–31 This was attributed to the impact of CQRs on patient outcomes and processes of care. The three identified studies described interventions which were introduced in response to CQR data feedback and benchmarking practices. These led to reductions in postsurgical events and improved patient outcomes.29–31 Similarly, all five CQRs described in the ASCQHC report demonstrated significant improvements in registry-specific KPIs, and reductions in patient LOS and mortality.13

There were a limited number of studies which considered CQRs as a health intervention for an economic evaluation. This may be attributed to several factors. First, it is difficult to distinguish between the clinical impacts attributed to CQRs and the impact of registries developed as part of larger quality improvement initiatives.4 Furthermore, population-based registries may have been developed for purposes other than benchmarking and quality improvement. Hence, there are comparatively few registries capable of collecting and disseminating relevant data for quality improvement. Finally, given the significant resources required for establishing registries, researchers and key stakeholders may also be reluctant to evaluate the impact of CQRs in a rigorous manner. These factors were identified in a systematic review on CQRs by Hoque et al.4 We found that none of the studies identified in this review were considered by Hoque et al; this may be attributed to differences in search criteria, with our review considering studies which conducted economic evaluations of CQRs.

A common limitation across the studies identified was the difficulty in assessing the extent of contribution of a CQR to improvements in patient outcomes over time. Such improvements may occur temporally due to factors unrelated to CQRs, such as the introduction of improved medications.13 29 30 The extent to which the benefits attributed to CQRs continue over time are also uncertain and vary depending on the patient population of interest and the purpose of the economic evaluation itself.4 13 The difficulty in distinguishing between the benefits attributed to CQRs, as opposed to secular trends, were highlighted in both Thanh et al and Hollenbeak et al.29 30 Finally, the choice of the most appropriate comparator for an economic evaluation of a CQR may be difficult, which hinders the assessment of the true clinical and cost impacts of CQRs. For example, studies using a ‘before-after’ design consider the period prior to registry introduction as the comparator. However, as discussed above, the improvements observed in ‘before-after’ studies may be attributed to secular trends rather than CQRs.4 13 31 Additionally, there is often a paucity of suitable alternative data sources for establishing a control group in studies using a hypothetical, experimental study design.4 This further exacerbates the uncertainty around the true clinical impact of a CQR, and limits the applicability and generalisability of the economic evaluation. Hence, the limited number of studies focusing on economic evaluations of CQRs may be attributed to difficulties in developing a rigorous study design to assess the true clinical impacts attributed to CQRs.

The present systematic review has several limitations that warrant discussion. First, the studies used either a ‘before-after’ or hypothetical experimental study design, limiting the degree of certainty around the true level of clinical benefit and cost-efficacy attributed to CQRs.4 13 29–31 Second, it was not possible to pool study results given the limited number of studies available and heterogeneity. There was a high degree of variation with respect to the study populations, coverage, study duration and design, and outcomes of interest considered in each economic evaluation. Characteristics of each CQR, including the reporting mechanism and management of data, also varied across studies. Third, as discussed above, there were considerable differences in the methodological approach used to estimate cost-effectiveness and ROI across the studies. Importantly, the analyses presented by Hollenbeak et al were based on the perspective of a single institution; hence, the generalisability of clinical and cost impacts attributed to the NSQIP across other settings and institutions is limited.16 29 There are considerable differences between the health systems in Canada, Australia and the USA; when coupled with the significant heterogeneity identified across the studies, the transferability of results was limited. Hence, a rigorous assessment of transferability was not possible.16 Finally, as with other systematic reviews, our review may have been subject to publication bias.

Conclusions

There has been a dramatic increase in the role of CQRs worldwide for informing interventions to improve healthcare processes and patient outcomes. At present, there is very limited evidence of the cost-benefits and cost-effectiveness of CQRs, and such studies have limited comparability. Available data indicate that CQRs can be cost-effective and can lead to significant returns on investment. The global implementation of CQRs, coupled with emerging evidence of clinical benefits attributed to CQR operation, clearly warrants further studies evaluating the economic impact of CQRs.