Understanding how and why audits work: protocol for a realist review of audit programmes to improve hospital care.

INTRODUCTION: Many types of audits are commonly used in hospital care to promote quality improvements. However, the evidence on the effectiveness of audits is mixed. The objectives of this proposed realist review are (1) to understand how and why audits might, or might not, work in terms of delivering the intended outcome of improved quality of hospital care and (2) to examine under what circumstances audits could potentially be effective. This protocol will provide the rationale for using a realist review approach and outline the method. METHODS AND ANALYSIS: This review will be conducted using an iterative four-stage approach. The first and second steps have already been executed. The first step was to develop an initial programme theory based on the literature that explains how audits are supposed to work. Second, a systematic literature search was conducted using relevant databases. Third, data will be extracted and coded for concepts relating to context, outcomes and their interrelatedness. Finally, the data will be synthesised in a five-step process: (1) organising the extracted data into evidence tables, (2) theming, (3) formulating chains of inference from the identified themes, (4) linking the chains of inference and formulating CMO configurations and (5) refining the initial programme theory. The reporting of the review will follow the 'Realist and Meta-Review Evidence Synthesis: Evolving Standards' (RAMESES) publication standards. ETHICS AND DISSEMINATION: This review does not require formal ethical approval. A better understanding of how and why these audits work, and how context impacts their effectiveness, will inform stakeholders in deciding how to tailor and implement audits within their local context. We will use a range of dissemination strategies to ensure that findings from this realist review are broadly disseminated to academic and non-academic audiences. PROSPERO REGISTRATION NUMBER: CRD42016039882. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

This review goes beyond considering the effectiveness of audits by building an understanding of how and why audits work within various contexts.

This review uses a systematic screening protocol.

The main limitation is that realist reviews are dependent on the transparency and adequacy of the reporting of data on the context, the mechanisms and their relationship to the produced outcomes of individual studies by the original authors. The potential lack of adequate data in this regard might hamper developing a full understanding of how and why audits are effective and might restrict the full development of the programme theory.

Introduction

In recent decades, quality and safety issues have become increasingly important in hospital care because of their direct effect on both clinical outcomes and patient satisfaction. However, hospital care still suffers from a quality gap between the ideal care, in line with the best available medical evidence, and the actual care provided to patients.1 To close this gap, health authorities and organisations currently prioritise quality improvement (QI) strategies, which are seen as systematic, data-driven monitoring and evaluation activities to improve the quality of hospital care.2 A widely used QI strategy within hospitals is the audit.

In this review protocol, we will focus on audits that address quality. Such audits are commonly used within hospital care aiming to promote quality improvements by evaluating the delivered care against standards, controlling and/or changing healthcare processes and healthcare providers’ performance.3 However, it is unlikely that audits work in the same way in every setting. Accordingly, it is important to understand how and why audits might lead to quality improvements. A realist review, as outlined in this protocol, will contribute to this understanding.

The range of possible audits can be roughly divided into (1) external audits, used to gain insight into hospitals’ compliance with external criteria (eg, accreditation, certification, external peer reviews); (2) internal audits, often in preparation for an external audit; and (3) clinical audits, carried out on a voluntary basis by healthcare professionals.3 4 Although there are differences, such as in the scope and the approach used, between the various types of audits, they all serve the same objective: to improve the quality of hospital care.

Externally driven audits (ie, accreditation, certification, external peer reviews and preparatory internal audits) seem to be more strongly anchored in quality assurance (QA), referring to initiatives designed to assure compliance with minimum (national) quality standards.5 6 These external audits are used to assess certain dimensions or characteristics of a healthcare-providing organisation against specified standards.7 As such, the implementation of an external audit requires an external standard and collaboration from beyond the hospital—and this distinguishes them from internal audits.

Internal audits are conducted by internal auditors of the hospital’s own organisation, such as quality officers or healthcare professionals from another department than the one being audited to guarantee some level of independent judgement. Internal audits are used to evaluate the delivered care against standards with different purposes. On the one hand, healthcare organisations use internal audits to continuously improve the quality of healthcare. In this way, one could expect that, compared with external audits, threats to quality can be more quickly revealed, allowing the organisation to regularly adapt its processes to improve quality at the local level. Internal audits are also frequently used in the framework of external audits and are conducted to avoid performance standards dropping between two external audits. These audits are designed to evaluate and improve the effectiveness of the organisation’s quality management system and focus more on organisational conditions and less on the behaviour of healthcare professionals and patient outcomes.4

Clinical audits differ from other types of audits in that they are mostly undertaken and initiated by healthcare professionals. Moreover, clinical audits represent a shift from QA to a QI process, with a focus on seeking to improve care, and prevent poor care. This process takes place continuously as part of everyday routines.6–8 As such, healthcare professionals work together to collect data and evaluate their own practices. Following this, they develop and apply improvements in their daily practices, and then the audit cycle is repeated to demonstrate improved and sustained improvements.8 As such, clinical audits do not necessarily use external criteria and are not carried out in response to external demands as the initiative comes from the healthcare professionals themselves.9 A considerable amount of literature addresses the effectiveness of audits and reports mixed results.10–12 A systematic review on audit and feedback13 demonstrated a positive overall effect of audits on clinical practice. Further, the authors noted differences in the design and the effectiveness of audits. This variety can be attributed to at least two issues. First, audits are used to improve specific aspects of healthcare and can be targeted at different levels. For example, external audits are performed to induce changes at the organisational level (eg, in organisational policy or procedures), whereas clinical audits are performed to alter local healthcare practices (eg, clinical day-to-day practices or local guidelines). Second, audits are used in different contexts, and this considerably complicates the evaluation of their effects. For example, an audit could be effective in one organisation, or department, but not in another because of, for instance, the amount of support offered for quality improvements, as part of their differing contexts (see online supplementary file 1). The literature on QI strategies recognises that the mixed effects are partly due to the differing contexts in which interventions are planned.14 15

The variety in the levels of audits, together with the heterogeneity of their contexts, suggests that it is unlikely that audits work in the same way in every setting. This creates challenges when attempting to synthesise evidence in a systematic review. Given this situation, more information about why and how audits work is needed.16 17 A detailed understanding of the contextual factors and the mechanisms that influence the effectiveness of audits is a prerequisite for understanding the mechanisms through which audits might lead to quality improvements. More importantly, a better understanding of ‘how and why audits might work’ will inform decision-making on how to tailor quality improvements at the local level.

A useful approach for explaining how and why audits might work, and investigating the interactions between context, mechanism and outcome, is the use of a realist review.18 19 The value of a realist review is that it is concerned with how an intervention works, rather than whether an intervention works, which is the focus of the conventional systematic review approach. Furthermore, the realist review methodology is specifically designed to cope with the intervention heterogeneity (in both the chosen study design and the used outcome measures) present in previous research on audits. Finally, this method is appropriate for the current research because audits are complex context-sensitive interventions.10 15 20 Within the past decade, similar studies in other contexts have used realist reviews to understand how complex interventions work and are put into practice (eg, ref. 21).

The objectives of the current review are (1) to understand how and why audits might, or might not, work in producing the intended outcome of improved quality of care and (ii) to examine under what circumstances audits could potentially be effective by formulating and refining underlying programme theories. Consequently, this review focusses on three research questions:

What are the mechanisms through which audits deliver their intended outcomes?

What contextual factors determine whether the identified mechanisms produce the intended outcomes of audits?

In what circumstances (ie, which combination(s) of mechanisms and context) are audits most likely to be effective?

Method

A realist review aims to clarify, from observed data, the outcomes (O) of particular interventions in relation to context (C) and mechanisms (M). This ‘CMO’ configuration is based on the philosophical assumption that an intervention in a specific context (C) evokes mechanisms (M) that generate an outcome (O). Consequently, the underlying mechanisms can be expected to produce a broad range of outcomes (O) when performed in different contexts (C).18 19 22 The philosophical basis is realism, which is positioned between positivism and constructivism and assumes the existence of an external reality (a ‘real world’) that is ‘filtered’ (ie, perceived, interpreted and responded to) through the inputs of individuals. Consequently, it is not the intervention in and of itself that causes outcomes but the individuals who initiate a process of change and as such have an effect on whether and how the intervention works.19 One of the key outputs of a realist review is the development of programme theories that set out how and why an intervention is thought to ‘work’ to generate certain outcomes.23

Study design

This review follows Pawson’s steps for conducting realist reviews, namely (1) clarifying the scope, and programme theory development; (2) searching for evidence; (3) appraising primary studies and extracting data; and (4) synthesising evidence and drawing conclusions.18 We completed the PRISMA-P checklist (see online supplementary file 2). The reporting of the review will follow the ‘Realist and Meta-Review Evidence Synthesis: Evolving Standards’ (RAMESES) publication standards23, since this is the most relevant and method-specific reporting guidance. In line with these standards, data extraction and synthesis will be an interpretive process, driven by reflection and discussion by the review team.23 This process requires repeated reading of primary studies because, as the synthesis progresses, new or refined elements of theory are expected to emerge. The protocol outlined below was written after the first steps had already been initiated or completed. Accordingly, both the past tense (steps that have been completed) and the future tense (steps that have yet to be initiated) are used.

The review team represents a range of disciplines and professions, which enables us to consider multiple perspectives and insights on the data gathered within this realist review. LH-M has a nursing background and is a PhD candidate. GW is an implementation fellow and has several years of experience as a quality manager. KA has a background in economics and business, is a professor of healthcare management and has numerous publications related to quality and patient safety. RG is a medical specialist, professor of internal medicine, chair of the Dutch Training Program of Internal Medicine and President of the Dutch Society of Hospital Medicine. He is also involved in the training of hospitalists, which are conducting clinical audits as part of their training. Further, all the members of the review team are experienced in qualitative research.

Scope of the review and programme theory development

The first step of this review process has already been executed with the aim of building a programme theory that would explain how and why audits might work. The unit of analysis in a realist review is not the intervention itself, but the contexts, mechanisms and outcomes that underpin the intervention. Given this situation, the initial step in formulating a programme theory draws on the literature on the effectiveness of QI strategies. As audits are QI strategies, we would assume that the contexts, mechanisms and outcomes uncovered in the QI literature might also play a role in the effectiveness of audits. The initial programme theory explains how audits are supposed to work by framing the interrelationships between context, mechanism and outcome (see figure 1).18 An exploration of programme theories was initiated through ongoing conversations within the review team and by a preliminary search of the literature. In addition, key terms were defined to guide the review and to ensure a common understanding (see online supplementary file 1).

Figure 1

Initial programme theory for the effectiveness of an audit.

After a number of iterations and discussions, we developed an initial programme theory regarding how and why audits might work. This suggests that having an organisational culture that is supportive of QI, a leadership committed to quality and previous audit experiences are important contextual factors in the success of an audit.24 These contextual factors trigger mechanisms, including a focus on continuous improvement rather than auditing for assurance and compliance,25 26 bottom-up initiatives as a prerequisite for ownership11 27 and the active involvement of healthcare professionals in audit processes,13 25 28 that in turn lead to improvements in the quality of healthcare. WHO describes the quality of healthcare quality as follows: ‘quality of care means that a health system should seek to make improvements in six areas or dimensions of quality’.29 These dimensions are effectiveness, efficiency, accessibility, acceptability, equity and safety.

This initial programme theory provides only a provisional structure for the review, and additional contextual factors, mechanisms and outcomes will be identified as the review progresses. The initial programme theory will be expanded, tested and refined using data from studies included in the review.

Search for evidence

As a second step, a search strategy was developed and performed in collaboration with an experienced university librarian. To ensure that all relevant articles were identified, a systematic literature search was conducted in MEDLINE, Embase, PsycINFO, Academic Search Premier, Business Source Premier, EmeraldInsight, Cochrane Library and Web of Science for the period 2005–2015. These databases were selected because they contain the core of quality and patient safety studies in the field of healthcare management as well as the biomedical view on quality of healthcare. The search strategy was piloted first in MEDLINE and later adapted for searching the other databases (see online supplementary file 3). The search included appropriate indexing terms (ie, MeSH terms and keywords) on descriptors of audits (eg, clinical audit, accreditation, certification, peer review, quality improvement, quality assurance), outcomes (eg, efficiency, effectiveness, improvement) and hospital care (eg, academic medical centres, health organisations). The reference lists of the uncovered review articles were studied to identify additional primary studies.

A realist synthesis approach to searching for evidence is iterative and evolves as the understanding of the subject matter deepens. Consequently, as the review progresses further, we will also search for unpublished and grey literature (eg, websites, national guidelines, policy documents and information reported in specialist conferences) on the assumption that the literature on this topic may be diverse and dispersed. In addition, our expectation is that not all the included publications will adequately report on all aspects of an audit. We will therefore identify papers, and other research outputs, that relate to the same study by using ‘cluster’ searching.30 For example, a search can be based on the members of a research team of an included article to identify all other refereed journal articles and related documents. Further, an additional iterative search may be necessary if it is determined that more data are required to refine a specific part of the programme theory or if new prospective theories are identified during data extraction or synthesis.

Appraise primary studies and extract data

The selection of appropriate primary studies has already been executed. First, one reviewer (LH-M) identified and removed duplicates. Next, two reviewers (LH-M and GW) independently screened all titles and abstracts for suitability for inclusion. The focus was on empirical studies that evaluated the effects of audits in hospital settings within high-income countries, without restrictions on the type of study design. Only studies published in English were included to avoid misinterpretation of the content of an article due to language barriers (see box 1).

Research on accreditation, certification, peer review/Dutch visitatie model* or local clinical audit

Hospital setting

High-income country

Published in English

English abstract available

Description of the medical or technical content

Description of the process of how the audit was conducted

Description of the impact of audit on medical and process outcomes

*This is a system of peer review that is led and owned by doctors and designed to assess the quality of care provided by groups of hospital-based medical specialists. Practices are surveyed every 3–5 years by a group of peers.40

Second, to ensure consistency of judgement, the full texts of a random 10% of the articles were independently reviewed by LH-M and GW and retained if they were deemed relevant (ie, the article could provide data on the context, mechanisms or outcomes of an audit). One reviewer (LH-M) reviewed the remaining 90% for their relevancy. In practice, a number of these articles required discussion or joint reading by two reviewers as it was sometimes difficult to decide between inclusion and rejection. Disagreements were recorded and discussed to ensure that decisions were made consistently. When disagreements remained, the matter was resolved through discussion involving the entire review team.

The next section describes activities that have yet to be started. Quality appraisal and data extraction will be undertaken using prespecified Excel spreadsheets (available on request from the first author). As the aim of the data extraction process is to evaluate and refine the initial programme theory, the contents of the data extraction sheets will be developed by the review team based on the content of the initial programme theory. To test the usability of the data extraction sheets, the file will be pretested on two purposefully selected articles.31 For each study, the quality will be appraised and general characteristics extracted concerning the study’s setting, the unit of analysis (including type of organisation) along with sections of the text that relate to context, mechanisms and their relationship to the produced outcomes.

Realist reviews amount to mixed-method reviews in that they incorporate both quantitative and qualitative studies, as well as grey literature. Consequently, different approaches are required to assess the quality of the included studies. Two reviewers (LH-M and either GW, KA or RG) will assess the quality of each included study. Any disagreement will be resolved through consensus-based group discussions within the review team. First, following realist synthesis principles, the evidence will be appraised using the concept of rigour.18 19 Rigour will be assessed by describing fidelity and nuggets (ie, the potential match with the initial programme theory and valuable observations presented in primary studies) and trustworthiness (ie, whether the methods used to generate the data are credible and trustworthy).32 33 Second, to make the concept of trustworthiness more concrete, and to ensure transparency in decision-making, the quality of the evidence of each individual study will be presented in the form of an evidence-level table based on criteria established by the Cochrane ‘Effective Practice and Organisation of Care’ (EPOC) review group (see table 1).34 35 These criteria range from systematic reviews (A1) to descriptive, non-analytical studies (D).

Table 1

Levels of evidence quality

Level	Description
A1	Systematic review Review of data from multiple randomised controlled trial studies
A2	Randomised trial Comparative study with (random) intervention and control group design
B	Controlled trial Trial with intervention and control group and comparisons of outcomes B1 multiple measurement points B2 single measurement point
C	Non-controlled study C1 multiple case, multiple measurements points C2 multiple case, single measurement point C3 single case, multiple measurements point C4 single case, single measurement point
D	Descriptive, non-analytical D1 multiple projects D2 single project D3 literature review

In addition, the  ‘Quality Improvement Minimum Quality Criteria Set’ (QI-MQCS) will be used to assess the completeness of the reporting of each study.36 This tool includes 16 content domains to critically appraise QI intervention publications and determine whether a minimum quality standard has been met (see table 2).

Table 2

Quality Improvement Minimum Quality Criteria Set (QI-MQCS) domains36

Domain	Minimum standard (see supplementary file 2 online in ref. 37)
1. Organisational motivation	Names or describes at least one motivation for the organisation’s participation in the intervention
2. Intervention rationale	Names or describes a rationale linking at least one central intervention component to intended effects
3. Intervention description	Describes in detail at least one specific change including the personnel executing the intervention
4. Organisational characteristics	Reports at least two organisational characteristics
5. Implementation	Names at least one approach used to introduce the intervention
6. Study design	Names the study design
7. Comparator	Describes at least one key care process
8. Data source	Describes the data source and defines the outcome of interest
9. Timing	Describes the timing of the intervention and its evaluation to determine the presence of baseline data and the follow-up period after all intervention components have been fully implemented
10. Adherence/fidelity	Reports fidelity information for at least one intervention component or describes evidence of adherence or of a mechanism ensuring compliance to the intervention
11. Health outcomes	Reports data on at least one health-related outcome
12. Organisational readiness	Reports at least one organisational-level barrier or facilitator
13. Penetration/reach	Describes the proportion of all eligible units that actually participated
14. Sustainability	Describes the sustainability or the potential for sustainability
15. Spread	Describes the potential for spread, existing tools for spread, or spread attempts/large-scale rollout
16. Limitations	Reports at least one limitation of the design/evaluation

Two reviewers (LH-M and either GW, KA or RG) will independently undertake the data extraction and, in this way, data from all the included articles will be extracted by two reviewers. Following this, the review team will discuss the data extracted so that data are not simply categorised but are used to begin to develop a reasoning that provides input to the final synthesis phase. Furthermore, relevant sections of the articles, that is, relating to context, mechanisms and their relationship to the produced outcomes, will be coded. This coding will be both inductive (codes emerge and are created during the data extraction) and deductive (codes created in advance of data extraction and informed by the initial programme theory).

Synthesise evidence and draw conclusions

Evidence will be synthesised by examining the relationships between contexts (eg, organisational culture), mechanisms (eg, bottom-up initiative) and outcomes (ie, intended and unintended consequences and the impact of audits) to determine what works, in what circumstances, how and why. Rycroft-Malone and colleagues31 have developed a five-step approach for a realist synthesis, incorporating the work of Pawson,18 as follows:

Organise the extracted data into evidence tables: The data extraction sheets from each individual study will be summarised and organised into one or more evidence tables. The evidence tables will also include a link back to the source papers.

Theme the data: Themes will be developed from the initial codes based on recurring contexts, mechanisms or outcomes. Identified themes will then be discussed among the reviewers, and contrary evidence will be sought.

Formulate chains of inference from the identified themes: Through an iterative process, we will search for chains of inference (connections) across extracted data and themes. For example, the ‘leadership and competency’ chain of inference might incorporate multiple themes including, for example, active engagement, competencies in QI, strong legitimacy within the organisation and a sound knowledge of quality issues. First, in order to support and formulate such chains of inference, patterns of similar mechanisms will be sought across different contexts to see if emerging patterns of outcomes (‘demi-regularities’) are identified. Second, since we expect context-outcome regularities to be easier to identify than mechanisms, because mechanisms are underlying and hence often unobservable or ‘hidden’, context-outcome regularities will be used as a basis for uncovering mechanisms.19 37 Cases in which the contexts are restrictive or supportive will be identified and this will help in formulating the chains of inference and in recognising and explaining interactions between context, mechanisms and outcomes. Third, we will not overlook the possibility that there may be more than one mechanism in play at the same time. The chains of inference so formulated will function as a basis for the CMO configurations to be developed. Two reviewers (LH-M and either GW, KA or RG) will jointly formulate the chains of inference, and this information will be shared and discussed in the review team.

Link the chains of inference and formulate CMO configurations: The chains of inference will be linked together to develop CMO configurations, which will then be linked back to themes or theories emerging from the literature (eg, commitment, organisational culture). The CMO configurations will be confirmed by returning to the source evidence. This iterative process will be guided by the research questions and the aims of the review. Following this, the generated CMO configurations will be used to either form new programme theories or to test, refine and supplement the initial programme theory. All these processes will be performed through discussions and agreement within the review team.

Refine the initial programme theory: Following the above four steps, a cumulative picture will be developed around the programme theories that summarises the nature of the context, mechanism and outcome, and links to the characteristics of the individual studies included. This cumulative picture will be based on hypotheses. For example, our review may suggest that hospitals that have a supportive culture for QI (context) and that seek the active participation of healthcare professionals in audits (mechanism) generate improved safety as part of the quality of care (outcome). A narrative will be developed around each hypothesis that will describe the characteristics of the supportive evidence.

Pawson et al 18 argue that stakeholders should be involved in both the process of confirming the emerging findings and in dissemination activities.18 To that end, emerging findings, supporting evidence and CMO configurations will be shared and discussed during a focus group session involving researchers, managers, policymakers and clinicians. The focus group will have 10–12 participants who will be selected to ensure some degree of homogeneity since this will enable them to share and discuss ideas by having comparable relevant knowledge in the field of audits.38 This process will help to refine the focus and the presentation of the narrative stemming from the CMO configurations.

Ethics and dissemination

One of the key contributions of this review, compared with the majority of audit evaluations and systematic reviews, is that it focuses on how and why audits might work, rather than just on the impact of audits. To really understand how and why audits might work, or might not, we believe that a clear picture of the underlying processes that lead to the outcomes is essential. By providing this, this review will extend the current literature by providing knowledge on how, and why, audits may lead to sustainable quality improvements.

This review has important practical implications. Along with the increasing emphasis on patient safety and healthcare quality, controlling rising healthcare costs has become a top policy priority in many countries. Research programmes, such as the review proposed here, can provide a basis for identifying appropriate strategies for quality improvements in healthcare. A better understanding of how these audits ‘work’, and how context might impact on the intended outcome of improved healthcare quality, will inform stakeholders in their decision-making about how to tailor and implement audits within their local context.

It has been argued that the theory and emerging evidence about how best to design audits (and what should be avoided) should be incorporated in the development and reporting of audits.11 39 However, such theoretical underpinnings are rarely reported in articles about audits, and this might hamper a full understanding of how and why audits are effective, and further impose restrictions on the ability to fully develop the programme theory and the applicability of the programme theory.

We will use various dissemination strategies to ensure that findings from this realist review are broadly disseminated to academic and non-academic audiences. First, we will submit the findings of this realist review to a peer-reviewed journal. In addition, review results will be disseminated through public websites, publications in professional journals and by presenting our work at relevant national and international conferences, and at conferences for practitioners. The outcomes of this realist review will be disseminated through events organised by The Netherlands Federation of University Medical Centres (Nederlandse Federatie van Universitair Medische Centra) and at a national symposium for hospitalists who conduct clinical audits as part of their training. As part of a more active dissemination strategy, we also intend a follow-up meeting with the focus group participants to discuss the findings and key messages.