Economic evaluation of Australian acute care accreditation (ACCREDIT-CBA (Acute)): study protocol for a mixed-method research project.

INTRODUCTION: The Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork-Cost-Benefit Analysis (ACCREDIT-CBA (Acute)) study is designed to determine and make explicit the costs and benefits of Australian acute care accreditation and to determine the effectiveness of acute care accreditation in improving patient safety and quality of care. The cost-benefit analysis framework will be provided in the form of an interactive model for industry partners, health regulators and policy makers, accreditation agencies and acute care service providers. METHODS AND
DESIGN: The study will use a mixed-method approach to identify, quantify and monetise the costs and benefits of accreditation. Surveys, expert panels, focus groups, interviews and primary and secondary data analysis will be used in cross-sectional and case study designs. ETHICS AND DISSEMINATION: The University of New South Wales Human Research Ethics Committee has approved this project (approval number HREC 10274). The results of the study will be reported via peer-reviewed publications, conferences and seminar resentations and will form part of a doctoral thesis.

This study uses economic evaluation techniques to assess the costs and benefits of acute care accreditation in Australian health services. The objective is to provide an interactive model of the costs and benefits from the perspective of a broad range of stakeholders. The model can also be used to assess the effectiveness of accreditation in improving patient safety and quality of care.

Despite its widespread implementation, the costs and benefits of acute care accreditation have not been clearly defined, identified and quantified.

Economic evaluation techniques such as costs–benefit analysis can help determine whether accreditation is an effective driver of patient safety and quality of care.

This protocol provides a unique, specifically designed framework and a number of purpose built tools to systematically assess the costs and benefits of acute care accreditation.

The strength of this study lies in using economic evaluation techniques to establish the role of acute care accreditation as an effective audit tool; this has not been accomplished previously.

One limitation in determining the impact of accreditation in Australia is the lack of a suitable control group given the widespread implementation of accreditation.

In keeping with the tenets of an inclusive societal framework,62 stakeholders from group (2A) will be included in the list of identified stakeholders. Those in group (2B) will be closely reviewed for inclusion based on their knowledge and perspective. The group of identified stakeholders will be used throughout the study and referenced when considering the individual costs and benefits to ensure that the broader social framework is addressed.

Introduction

Despite its widespread implementation, the costs and benefits of acute health services accreditation have not been clearly defined, identified and quantified.1–5 An economic framework is needed to systematically assess and compare these costs and benefits. This study protocol applies economic evaluation techniques using a purpose-designed framework to answer our research question as to whether acute care accreditation in Australia is effective in improving patient safety and quality of care.

Accreditation of acute health services in Australia

The International Society for Quality in Health Care (ISQua), the peak body for health services accreditation, defines accreditation as ‘public recognition of the achievement of standards by an organisation demonstrated through independent assessment in relation to set standards’.6 Accreditation has been widely implemented following the establishment of the Joint Commission on Accreditation of Hospitals (now, the Joint Commission) in the USA in 1951.7 8 In Australia, accreditation was first adopted for acute care services in 1974,9 with the Australian Council on Healthcare Standards (ACHS) as the main accrediting agency. Over subsequent years, ACHS developed the Evaluation and Quality Improvement Program (EQuIP). This accreditation programme comprises two external surveys within a 4-year accreditation cycle.10 Facilities are assessed by trained external surveyors using standards developed by ACHS in consultation with healthcare industry experts.10 As part of the wider health system reforms implemented by the Australian Commonwealth Government, recently approved legislation requires all hospitals and day procedure services in Australia to be assessed by an accreditation provider approved by the Australian Commission on Safety and Quality in Health Care (ACSQHC) using newly developed National Safety and Quality Health Service (NSQHS) standards.11 Australian acute care accreditation is the focus of this study. We include acute and high-risk inpatient activity in our definition of acute care facilities (ACFs) in this protocol.

Economic evaluation of health interventions

Health costs are a significant proportion of gross domestic product, averaging 9.6% in 2010 for Organisation for Economic and Co-operation Development countries.12 This, combined with persistent evidence of harm during health service delivery,13–16 has resulted in an increasing international focus on accountability and safety in healthcare.17 18 Economic evaluation addresses these issues by using a systematic framework to identify and compare the costs and benefits of a policy or intervention to determine whether implementation is effective in achieving stated aims and also to compare different policy proposals and interventions.19–21 In cost–benefit analysis (CBA), the costs and benefits are each expressed in monetary terms. This contrasts with other techniques such as cost-effectiveness analysis (CEA) or cost-utility analysis (CUA), which are used in assessing health technologies. In CEA and CUA, the common outcome denominator is not monetised, but expressed in terms of a common utility measure such as Quality Adjusted Life Years.22–25 CBA is advocated where there is a broader range of outcomes and is usually a requirement for submitting a regulatory impact statement for Australian government approval.19 CBA seems justified as the most appropriate model to use in an Australian health services context as the NSQHS standards cover both organisational and clinical outcomes, which are best measured using a common monetary denominator. Although CBA can be used as an allocative efficiency tool for comparing different projects, the requirement to make the costs and benefits explicit in the analysis framework can help clarify the goals, costs and benefits, providing input into the design of future accreditation systems in healthcare.26

Modelling the costs and benefits of a complex intervention, such as accreditation, in a complex system, such as an ACF, is a significant undertaking.27 Given the lack of precedent discussed above, we have developed a unique framework and a number of purpose-built tools specifically designed for evaluating acute care accreditation. Our SIQNS framework is synthesised from several sources,20 21 28 29 and comprises five discrete activities (1) scope and objectives; (2) identify costs and benefits; (3) quantify costs and benefits; (4) calculate net social benefits and (5) sensitivity analysis (figure 1).

Figure 1

SIQNS framework.

Scope and objectives

The aim of the study is to create an interactive model that can be accessed by health service providers, accreditation bodies, quality and safety agencies, governments and researchers to both test the assumptions in the model and to determine the cost–benefit calculations of acute care accreditation at both the national and local service levels.28 The lack of research in this field, as well as the complexity of both the intervention (accreditation) and system (acute healthcare), indicates that an important objective will be to make explicit the costs and benefits of accreditation.1 2 27 29 30

Establishment of the study parameters will be informed by a review of the research literature and analysis of the characteristics of accredited ACFs. These activities will also help determine the indicator selection process to identify and quantify the benefits of accreditation. A critical element is to identify the stakeholders involved in the acute care accreditation process. Although accreditation agencies and ACFs are the most obvious groups affected, a broader societal framework is required when assessing regulatory impact to ensure that equity and impact are meaningfully accounted for in the analysis.20

Our initial analysis has identified a key constraint, in that ACFs do not account for accreditation activities as a separate cost item.31 Our approach, outlined below, is to accept that large-scale data gathering on costs would not be possible within the scope of the study and will be a challenge over time. Instead, we will use a smaller study sample and have this validated by an expert panel. Accreditation benefits, in terms of both clinical and organisational outcomes, are more likely to be assessed using secondary data, but are more difficult to identify partly owing to a lack of clarity in terms of measurable endpoints. In addition, a full impact analysis of all benefits for all stakeholders will be outside the study scope. A further constraint is that an economic appraisal would ideally be conducted with a control group (either randomly assigned or with data collected for a before and after comparison) in order to compare different outcomes.32 This is not possible in Australia where acute care accreditation is widely implemented (93% of public hospital beds in 2010 and 84% of private hospital beds in 2008–2009).33 However, we can analyse data before and after introduction of new standards, review indicator activity against accreditation scores and analyse changes over time.

Identify costs and benefits

To identify the incremental costs of accreditation, we need to determine costs that are only incurred due to the accreditation process.34 We will first review the stakeholders for potential costs and exclude transfer payments to avoid double counting. We will then assess costs for a small number of ACFs (n=10) using our purpose designed assessment tool based on questions posed in similar cost surveys,35–38 and the Business Cost Calculator which has been designed to help Australian businesses compute business compliance costs.7 10 39 40

International accreditation agencies, for example, the Joint Commission, ACHS and Accreditation Canada, identify improvements in patient safety and quality of care as the main benefits of health services accreditation.7 10 40 Within an Australian context, we reviewed the benefits outlined by ACSQHC in a recent regulatory impact statement on the new NSQHS standards.41 In addition, the Australian National Health Performance Authority's (NHPA's) performance and accountability framework has defined the indicators used to assess effectiveness of care in ACFs. We will map our stakeholder analysis with the ACSQHC benefits and the NHPA adverse and sentinel event measures to identify quality and safety indicators.

In our choice of indicators, we need to distinguish whether a lack of change in the indicator is due to a lack of compliance with the accreditation standard, or whether compliance with the standard results in change in an indicator. We have adapted the approach used in measuring quality outcomes in US hospitals.42 This recommends that indicators must meet four key accountability criteria: research—robust evidence; accuracy—whether the process has been carried out satisfactorily; proximity—a clear and direct link between accreditation and the indicator; and, no adverse effects—no unintended or unwanted actions. As accreditation is usually just one facet of a quality and safety framework, we have added a further criterion—specificity—to determine how easy it is to isolate the effects of accreditation from other safety and quality measures. This is important when determining the effectiveness of accreditation versus other safety and quality initiatives.43 We specify the type of indicator, as process indicators are often a preferred measure of quality over outcome indicators. This is due to a more direct link between the indicator and the process being measured (the issue of proximity in Chassin's accountability criteria),42 but this can also lead to estimation problems as only a narrow range of factors is considered. Outcome indicators have an advantage in that the data are often collected routinely, but may need to be adjusted for other factors such as patient acuity and complexity.43–45 We will need to consider whether the chosen indicators need to be weighted to reflect both suitability (in terms of adherence to the accountability criteria) and applicability (quality of data collection and adjustment for patient mix). We will ask an expert panel to review and validate our indicator selections prior to quantifying the indicators for further analysis.

Quantify costs and benefits

Once the costs have been identified for each ACF in our sample, we will need to ensure that the results can be scaled up and assessed on a national basis. Although costs for initial accreditation can be higher than for ongoing accreditation,38 46 we assume ongoing accreditation costs for our sample, but include questions on estimating the cost of implementing the new national standards in the survey.

As the CBA model uses monetary values as the common denominator, we will need to monetise the benefits identified. The techniques used will depend on the availability of pricing and market data. For example, clinical outcome indicators (such as hospital-acquired infection rates, or complications resulting from inpatient falls) can be matched to activity-based costing codes. These data can be used to determine the potential cost savings from a reduction in infection or fall rates. Where we identify an indicator but do not see a change in measurement, we will include this in our sensitivity analysis, for example, the costs associated with reducing hospital-acquired infections by a stated amount. Where indicators can be identified and quantified but lack pricing or market data, we will monetise the effects using techniques such as revealed and stated preference methods, where possible.21 26 47

Calculate net social benefit

Given that costs and benefits are likely to occur at different times, we will need to adjust the timing differentials using an appropriate discount rate. For CBA, both the net social benefit (NSB) and the benefits–cost ratio (BCR) are calculated. The NSB uses a net present value method to derive an absolute measure of whether the discounted (net) benefits are greater than the discounted (net) costs when assessing proposals in a regulatory impact statement.19 22 The BCR is derived by dividing the net benefits by the net costs to determine an effective return on the costs and is used where the absolute size of the investment is a determining factor, for example, in a resource-constrained environment. However, the BCR is subject to more variation depending on how the outcomes are treated.19 20 22

With questions on the table about whether accreditation is more of an audit tool or quality improvement tool,1 8 CBA can address both these issues by comparing accreditation not only with other safety and quality programmes, but also with other methods of regulatory compliance. Although there is no good counterfactual to accreditation due to widespread implementation,33 we can estimate the relative effectiveness of accreditation versus alternative forms of audit. Examples include the additional requirements that a private ACF would need to meet to qualify for private health funding, or additional auditing that would be required for public ACFs in the absence of accreditation.

Sensitivity analysis

A sensitivity analysis is an essential tool to describe the impact of changes in assumptions and variables on our SIQNS framework,20 especially given the constraints discussed. We will review the assumptions in the model and run the NSB and BCR calculations over a range of values. For some costs and benefits, including qualitative outcomes, monetisation will be beyond the scope of this study and these items will not be included in the NSB or BCR calculations. Nevertheless, these costs and benefits can still be included in the final model and used to compare other programmes with similar NSB or BCR outcomes, but where the non-monetised items may be a deciding factor.

ACCREDIT project overview

The ACCREDIT CBA (Acute) study is the 6th of 12 studies under the ACCREDIT (Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork) research collaboration,28 48 49 funded by the Australian Research Council through its industry Linkage Program.50 The ACCREDIT collaboration involves researchers in the Centre for Clinical Governance Research and Centre for Health Systems and Safety Research in the Australian Institute of Health Innovation (AIHI) at the University of New South Wales (UNSW), Australia. The ACCREDIT research team benefits from a high-profile international advisory group containing leading researchers in health safety and quality from the UK, Spain and Sweden. The collaboration includes two leading health safety and quality bodies (ACSQHC and the Clinical Excellence Commission (CEC)) plus three of the major Australian health services accreditation agencies: ACHS, Australian General Practice Accreditation Limited (AGPAL) and the Aged Care and Standards Accreditation Agency (ACSAA).

Methods and analysis

The methods to obtain and analyse data needed to build the CBA model are discussed below using each of the five SIQNS activities. For each survey tool, focus group and expert panel described in the following sections, the ACCREDIT research team will send an electronic invitation to potential participants containing the study information and consent forms approved by the UNSW Human Research Ethics Committee (HREC).51 Those individuals wishing to participate will be asked to contact the UNSW research team and will be invited to either attend the relevant research activity, at UNSW, or be interviewed via telephone. The research activities will last approximately 1 h and will be digitally recorded and professionally transcribed.

SIQNS activity 1: scope and objectives

The first activity is to review the scope and constraints of the study design and to establish the study parameters. This will be completed through three tasks: literature reviews, analysis of accredited ACFs and stakeholder identification.

A literature review covering two general issues will be conducted. First, an investigation of the cost and benefits of accreditation, in health and related fields, and second, the compliance costs and methodologies used in non-health industries.

Analysis of accredited ACFs will be conducted using data from the ACHS national accreditation database, using accreditation survey data from 2003 to 2011. The purpose is to assess the characteristics of accredited ACFs: bed size, ownership and funding structure, geographic location (by state or territory), type of survey and whether these are linked to accreditation outcomes. During 2007, ACHS made changes to the mandatory criteria assessed in the surveys with the changeover from EQuIP3 to EQuIP4. Therefore, econometric analysis of the main study variables will be over the full period (2003–2011), accompanied by the analysis of individual standards as predictors of accreditation outcomes in the different EQuIP programmes. This part of the study will provide guidance for reviewing the structure of the ACFs targeted in the costs study.

To evaluate the stakeholder base, the ACCREDIT research team will send an electronic invitation to senior health services researchers at AIHI, UNSW, to participate in a focus group.52 Those willing to participate will be asked to contact the ACCREDIT team. The focus group will take place at UNSW using the general demographic and content questions from the Stakeholder Analysis Tool (box 1).

General demographic questions

What is your age?

What is your gender?

What are your highest qualifications?

What is your research discipline?

How many years have you worked in health services research?

How many years have you worked as a healthcare professional?

How many first author peer-reviewed publications do you have?

How many second or subsequent author publications do you have?

Content questions

1. Who are the key stakeholders in acute healthcare in Australia?

2. Please classify these as either:

A. Key stakeholders in acute care services who either contribute to, or influence, the costs and benefits of accreditation or

B. Key stakeholders in acute care services who do not contribute directly to accreditation costs or benefits

3. Please identify the costs and benefits for each stakeholder identified in 2A

SIQNS activity 2: identify costs and benefits

The aim of this activity is to estimate the incremental costs and the benefits of accreditation using a variety of purpose-built tools to both assess and validate our results. One of the main costs identified in previous research has been the preparation for external surveys in the accreditation cycle.9 35–38 53 To recruit subjects, the UNSW research team will send a study invitation to the accreditation partners to forward on to the ACFs they accredit. The ACFs agreeing to participate will be characterised according to: facility specialisation (eg, teaching hospital); location (metropolitan or rural) and size (large, medium or small). One from each category (n=10) will be randomly selected and approached by the UNSW research team. Semi-structured interviews will be conducted with Finance Managers, Quality Managers and General Managers as directed by the ACF, using the questions set out in the Incremental Costs Audit Tool (box 2).

1. Demographic details:

A. Bed size of facility.

B. Location (state or territory and whether rural or metropolitan).

C. Funding structure (public or private).

D. Specialisation of the facility (eg, teaching hospital, mental health, correctional facility or other).

E. Job description of the respondent.

2. Is this your first accreditation cycle?

3. Please provide details, where possible, by year incurred, of activity-based costing for the following activities in relation to the accreditation cycle:

A. Notification: implementing recommendations from the survey.

B. Education: staff meetings, working groups and education sessions for new staff and to accommodate changes in standards.

C. Record-keeping: developing and maintaining policies and documents.

D. Enforcement: staff allocated to assist surveyors and in presurvey and postsurvey briefings.

E. Publication and documentation: survey and self-assessment preparation.

F. Procedural: collecting consumer feedback relating to accreditation standards.

G. Other: details of staff involved in surveying other facilities.

H. Purchases: survey fees and consultancy fees relating to accreditation.

4. What do you estimate will be the difference in costs (if any) for complying with the new National Safety and Quality Health Service Standards?

Given the relatively small size of the survey sample, we will validate the results using an expert panel consisting of: accreditation surveyors, leading accreditation researchers, accreditation agencies, health quality consultants and government health quality improvement agencies. We will also invite staff and surveyors from ACFs who agreed to participate, but were not selected for interviews, to take part in the panel. A de-identified costs summary will be given to participants in advance, and the panel will be asked to discuss the questions set out in the Incremental Costs Validation Tool (box 3).

1. Do you think the results from the audit tool are representative of the current accreditation process?

2. If these costs are not representative, what is your estimate of the costs?

3. Are there other incremental costs that have not been included?

4. What would be your estimate of costs identified in question 3?

The outcomes of both the incremental costs audit and validation tools will be assessed against results of the stakeholder analysis. Total costs will be estimated at the national level.

To determine the specific benefits of accreditation, we will use our Indicator Assessment Tool (table 1) to identify the main topics in the new NSQHS standards and identify the relevant standard in the ACHS EQuIP5 survey. We will then review a range of process and outcome indicators including adverse events highlighted in the NHPA framework,54 as well as the outcome indicators and sentinel events collected by the Australian Institute of Health and Welfare.55 56

Table 1

Indicator Assessment Tool

Intervention topic
Accreditation Standards	List relevant standards relating to the intervention topic
NSQHS standards	Identify the relevant NSQHS standard
ACHS EQuIP5	Whether the intervention is a mandatory criterion in EQuIP5
Indicator	Description of indicator
Indicator type	Whether the indicator is process based or outcome based
Research	What is the evidence base that compliance with the standard affects the indicator?
Accuracy	How accurate is the indicator in terms of measuring compliance with the accreditation standard?
Proximity	How close is the link between the standard and the indicator? Is there a causal chain?
No adverse effects	What is the risk of avoiding adverse effects?
Specificity	Is it possible to isolate the effects of accreditation on the indicator from other safety and quality programmes?
Associate programmes	Related programmes initiated by state or federal governments, or healthcare agencies

The results of our indicator selection process will be validated using the established expert panel. A summary of the potential indicators and scores from the Indicator Assessment Tool will be given to participants in advance of the panel. The interviews with the panel will use the semistructured questions in the Indicator Validation Tool (box 4).

1. Do you think the indicators selected using the Indicator Assessment Tool are representative of the benefits of accreditation?

2. If these are not representative, what indicators would you add and why?

3. For the indicators you have identified, would you recommend attaching a weighting to the monetised values and if so, what weighting would you recommend?

The results of the panel will be used to formulate a list of benefit indicators. These will then be mapped to appropriate databases to look for evidence of improvement over accreditation cycles or following the introduction of a relevant standard. Indicators with low scores from the Indicator Assessment Tool, especially for the proximity and specificity criteria, will be evaluated for inclusion or rejection.

As this is a high-level description of the protocol, we have not detailed the exact methods for statistical analysis as these will depend on the indicators selected. The main difficulty is in determining a causal effect where there is no control group given the widespread implementation of accreditation and we will need to select the appropriate economic evaluation techniques depending on the results.

SIQNS activity 3: quantify costs and benefits

Costs will be scaled up nationally according to facility-type both by an actual dollar amount and as a percentage of total costs. Identified benefits will be quantified and monetised using a number of techniques depending on the type of indicators validated by our expert panel. For example, clinical outcome indicators can be matched to the ACSQHC's Costs of Hospital Acquired Diagnoses activity-based costing codes.57 58 Process measures are more difficult to quantify, but techniques for valuing non-market costs such as revealed and stated preference will be used to monetise the benefits where possible.21 22 47

SIQNS activity 4: calculate net social benefit

Our interactive CBA model will be populated with the costs and benefits identified and quantified in SIQNS activities 2 and 3. Costs will be added to the model and allocated by the year in which they were incurred during the accreditation cycle. The benefits will be allocated depending on the type of indicator used. For example, for clinical indicators such as hospital-acquired infection rates, the cost savings from a reduction in infection rates per year nationally can be monetised and modelled. Where the timing is not clear, we will assign equal weights over the expected time horizon and discount accordingly. Australian Government approved discount rates will be applied to the model in order to discount the cost and benefit cash flows back to a baseline year.19 59 Both the NSB and BCR (equations 1 and 2) will be calculated for the costs and benefits that have been monetised. Non-monetised costs and benefits will be included for comparative analysis if they can be quantified.

Equations 1 and 2: Bt is the sum of benefits in year t; Ct the sum of costs in year t; n the lifetime of the accreditation cycle or expected time horizon, in years; and r the discount rate used.

SIQNS activity 5: sensitivity analysis

In order to determine the sensitivity of inputs into the model, NSB and BCR will be recalculated for a range of values (±1, 5 and 10% of the total values) for each individual cost and benefit that is more than 10% of the total. In addition, the model will be run with discount rates at ±2 and 5 percentage points from the base discount rates used in order to test the duration sensitivity of the model.

Ethics and dissemination

The UNSW HREC has approved the ACCREDIT-CBA (Acute) study proposal (approval number HREC10 274). The study will be conducted in accordance with the UNSW Research Code of Conduct and the Australian National Health and Medical Research Council (NHMRC) guidelines.60 61 As such, all project data will be de-identified prior to publication and stored securely for a minimum of 7 years. Contact details of the research team will be given to participants in the study so that any complaints or concerns can be addressed. The results of the study will be submitted for publication in selected journals and presented at national and international conferences and seminars. The findings will also form part of a doctoral thesis.

Conclusion

Although accreditation of acute health services has been widely adopted in Australia, little is currently known about the costs and benefits of the process and whether accreditation is a cost-effective tool in improving patient safety and quality of care. This study aims to create a framework to answer these questions and to make the costs and benefits of accreditation explicit. This will, in turn, inform debate on the important issue of how best to monitor and improve patient safety and quality of care in acute health services.