Evaluation of current Australian health service accreditation processes (ACCREDIT-CAP): protocol for a mixed-method research project.

INTRODUCTION: Accreditation programmes aim to improve the quality and safety of health services, and have been widely implemented. However, there is conflicting evidence regarding the outcomes of existing programmes. The Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork-Current Accreditation Processes (ACCREDIT-CAP) project is designed to address key gaps in the literature by evaluating the current processes of three accreditation programmes used across Australian acute, primary and aged care services. METHODS AND
DESIGN: The project comprises three mixed-method studies involving documentary analyses, surveys, focus groups and individual interviews. Study samples will comprise stakeholders from across the Australian healthcare system: accreditation agencies; federal and state government departments; consumer advocates; professional colleges and associations; and staff of acute, primary and aged care services. Sample sizes have been determined to ensure results allow robust conclusions. Qualitative information will be thematically analysed, supported by the use of textual grouping software. Quantitative data will be subjected to a variety of analytical procedures, including descriptive and comparative statistics. The results are designed to inform health system policy and planning decisions in Australia and internationally. ETHICS AND DISSEMINATION: The project has been approved by the University of New South Wales Human Research Ethics Committee (approval number HREC 10274). Results will be reported to partner organisations, healthcare consumers and other stakeholders via peer-reviewed publications, conference and seminar presentations, and a publicly accessible website.

This project involves three studies that aim to evaluate current health service accreditation processes in three domains: acute, primary and aged care. Specific issues to be examined include: the strengths and consequences of different accreditation models; the critical elements of accreditation processes that stimulate behavioural change in health services and healthcare professionals; and the development and use of accreditation standards.

Accreditation programmes are implemented worldwide to drive and transfer quality and safety improvements across a variety of health service organisations.

Evidence supporting the capacity of accreditation programmes to promote high-quality and safe organisational and clinical performance is both limited and contested.

This project will involve multimethod approaches to help build the evidence base regarding CAP.

The strength of this project is its examination of accreditation programmes across acute, primary and aged care domains, and comprehensive multimethod approach designed to produce results of practical utility across the health system.

A limitation is the project's focus on accreditation programmes from one country. The results may require additional interpretation to be transferable to other countries.

Introduction

Health service accreditation

Health service accreditation programmes are complex system-level interventions that aim to improve the quality and safety of patient care by monitoring the performance of health services against predetermined standards.1 2 Assessments are usually made by external accrediting or certifying agencies using standards developed in consultation with healthcare industry experts.3 The participation of health services in accreditation programmes can be voluntary, legislatively mandated or encouraged through the use of financial incentives offered by governments or insurance agencies.4 While originally designed for use within acute care settings, accreditation programmes are now available for a range of organisations, including primary and aged care services.5 Accreditation cycles commonly last between 3 and 5 years and involve all, or a mixture of the following activities: organisational self-assessment; on-site assessment by accreditation surveyors; the provision of a summative report to organisations that may include improvement recommendations and subsequent actions implemented by health services to address recommendations.1 2

Health service accreditation evidence base

Given the large number of countries with accreditation systems, there is a large yet undetermined global investment in health service accreditation.6 7 Despite this expenditure, there is limited research that has demonstrated the effectiveness of programmes in achieving their stated aims.8 Recent literature reviews have found accreditation may promote changes to organisational processes associated with quality of care.4 5 8–11 However, several factors reduce the generalisability of these findings: the small number of empirical studies; limited use of experimental study designs; difficulty of aggregating results from individual studies due to their varied research contexts, that is, different country, healthcare domain and accreditation programme settings; and the infrequent use of patient outcome indicators for evaluation.4 5 8–11 Furthermore, there is limited understanding of the specific processes within accreditation programmes that are most capable of promoting quality and safety improvements within health service organisations.5 These issues highlight the pressing need for additional rigorous accreditation research, which has prompted calls to strengthen the evidence base.12 13 This project aims to link researchers, policy bodies and industry partners to provide a response to the global need for further, high-quality health service accreditation research.

Project overview

The project is one component of the ACCREDIT (Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork) research collaboration.14 ACCREDIT involves research and industry partners: researchers in the Centre for Clinical Governance Research and Centre for Health Systems and Safety Research in the Australian Institute of Health Innovation at the University of New South Wales (UNSW); three major Australian healthcare accreditation agencies (Australian Council on Healthcare Standards (ACHS), Australian General Practice Accreditation Limited (AGPAL) and Aged Care and Standards Accreditation Agency (ACSAA)); leading quality improvement policy bodies (the Australian Commission on Safety and Quality in Health Care and the Clinical Excellence Commission); and an international advisory group consisting of prominent quality and safety researchers from Spain, Sweden and the UK. The research is funded by the Australian Research Council through its industry Linkage Programme,15 which aims to promote collaboration between researchers and industry groups to generate rigorous research with practical implications for Australian society. The collaboration has successful experience in this form of research.16 This research protocol details an evaluation of current processes employed by Australian health service accreditation programmes within the ACCREDIT collaboration; known as the Current Accreditation Processes, ACCREDIT-CAP protocol.

Methods and analysis

Three studies will be undertaken to evaluate CAP across acute, primary and aged care domains (figure 1). Each study has specific questions and uses multiple methods to address them.

Figure 1

Research studies and questions.

As illustrated in table 1, a range of research techniques will be applied, including documentary analyses, focus groups, interviews and surveys. The use of different research methods will address the potential limitations associated with any single study tool or measure. The collation of varied data sources, that is, triangulation,17 will strengthen the credibility and generalisability of results by facilitating comparative confirmation of findings.

Table 1

Study research questions, approaches and methods

Study	Research questions	Summary of research approaches, tasks, and scope	Summary of methods, sample sizes, data requirements, analysis and design features
Accreditation models	What are the relative strengths and consequences of different accreditation models?	Undertake a multimethod evaluation of three accreditation programmes: those of ACHS, AGPAL and ACSAA	Interview key stakeholders of three accreditation agencies (n=18) Conduct a web-based survey of staff from accredited acute, primary and aged care services (n=∼300)
Critical elements of accreditation	What are the critical elements of the accreditation process that stimulate improvement?	Assess programme elements (eg, self-assessment, clinical indicators, surveyor visits and accreditation reports) and describe their role in promoting improvement	Run focus groups of stakeholders drawn from accreditation agencies (n=6 focus groups) and jurisdictional health departments (n=8 focus groups) and 15 focus groups involving staff from accredited acute (n=5), primary (n=5) and aged care services (n=5) Conduct a web-based survey of staff from accredited acute, primary and aged care services (n=∼300) Gather examples of how the respective accreditation elements drive change
	What drives behaviour change in health service organisations and their staff?
Standards and their impact	How are standards developed and used?	Examine the development of standards and their application across different accreditation programs, selecting and investigating a sample of standards to determine their sources (eg, public inquiries, adverse events, international guidelines) and how they were developed and applied	Use documentary analysis and focus groups (n=3) to retrospectively analyse the development of standards, assessing the use of evidence and engagement of stakeholders Conduct a survey of staff from accredited organisations, investigating how standards are applied and how they promote change (n=∼300). From these data, identify for detailed analysis case study sites in which standards have promoted measurable change Conduct case studies (n=5) of specified key standards (evaluation of care, documented policies, the quality improvement system, health records and infection control) to identify factors related to organisational change. Employ focus groups (n=15) and obtain organisational data to measure the extent of this change
	How do standards incorporate evidence, and influence the expertise of clinicians, managers and policymakers?
	How does the application of standards promote change in organisational performance and clinical practice?

Study 1: accreditation models

Study 1 aims to examine the relative strengths and consequences of the main accreditation programmes of ACHS, AGPAL and ACSAA.

Interviews with accreditation agency stakeholders

Six one hour interviews will be held with stakeholders of each partner accreditation agency (n=18 interviews in total). Each agency will provide the contact details of representatives from organisations they consider key stakeholders. The following groups are likely to be nominated: federal and state government agencies; healthcare professional colleges and associations; accreditation agency councils, boards, staff and surveyors; and consumer representatives. A sample of stakeholders will be purposively selected from the lists provided by each agency in order to include representatives from a variety of groups. Based on the research team's prior accreditation research experience,18 19 it is probable that thematic saturation20 (ie, when no new themes or categories emerge from the data) will be reached within the proposed sample size.

Specific issues to be examined within interviews will be finalised through collaboration between researchers and industry partners. Based on previous accreditation literature reviews5 and studies21 22 undertaken by the research team, the interview schedule is likely to include a combination of the following topics: philosophical positions underlying accreditation programmes (eg, regulatory compliance or continuous quality improvement); effects of different accreditation agency management structures and administrative processes; the roles and influence of key stakeholders; intended and unintended impacts of accreditation programmes on health services; incentives encouraging health service participation in accreditation programmes; standards development processes; the roles of surveyors; issues surrounding the public disclosure of accreditation results and the role of short notice and patient journey survey methodologies.

Recruitment will occur via emails, in which the study and participant roles within it will be explained using information and consent forms approved by the UNSW Human Research Ethics Committee.23 If responses are not received within 2 weeks, the research team will follow up nominated stakeholders via telephone to request their participation. Interview locations will be determined through discussion with potential participants, and are likely to occur across several Australian cities, states and territories.

Where possible, face-to-face rather than telephone interviews will be arranged to increase participant comfort in speaking openly regarding potentially sensitive issues. Other advantages of the face-to-face approach include increased respondent attentiveness to questions and the opportunity for interviewers to detect non-verbal feedback.24 Two experienced research team members (RH and DG) will conduct interviews. Training will be organised prior to data collection to ensure a uniform approach, which will produce higher reliability and more internally valid cross-comparisons of data.25

The semi-structured interview technique will be employed to balance uniformity against variation of responses. Data resulting from each interview will be sufficiently similar to allow stakeholder views to be compared. However, use of a non-prescriptive interviewing approach will allow respondents to discuss issues not explicitly raised within the interview schedule. New concepts can be included for subsequent interviews. The continual process of adapting the interview schedule will be critical, as the aim is not validation of pre-existing hypotheses, but identification of emergent issues in order to generate new understandings. Interviews will be digitally recorded and professionally transcribed.

Thematic analysis of interview transcriptions will be undertaken inductively, supported by the use of the textual grouping software, NVivo V.9 (Doncaster, Australia).26 Such programmes are used to facilitate systematic classification of the data.27 Three general principles will guide the analysis: include all major rival interpretations, address only the most significant aspects of the data and use prior knowledge and experience to inform analytical decisions.20 Preliminary development of thematic categories (ie, key topics discussed by respondents) and initial coding will be conducted by the first author, who has extensive qualitative data analysis experience.28–31 After discussion among the research team to cross-check the validity and relevance of developed themes, categories will be refined and adjusted until a coherent scheme is developed that admits all instances and is applied to all interview data. To determine the reliability of categorisation by the first author, 50 randomly selected coded sentences will be assessed by a second researcher issued with category definitions. The level of inter-rater reliability will be assessed using Cohen's Kappa statistic, which measures the inter-rater agreement, accounting for that expected by chance.32 A result less than 0.7 will necessitate further refinement of categories until a higher level of agreement is achieved.

Survey of staff from accredited health services

A minimum of 300 web-based surveys using KeySurvey33 software (WorldAPP, Braintree, MA) will be collected from a purposive sample of staff from accreditation agency member organisations, that is, acute, primary and aged care services enrolled in accreditation programmes. Advantages of web-based surveys compared to paper-based surveys are cost reduction, speed and facilitation of more complex questionnaire routing.34 Respondents will be asked to evaluate 25 statements concerning key characteristics of accreditation programmes using a five-point Likert scale35 ranging from ‘strongly agree’ to ‘strongly disagree’. Survey questions will be drawn from the interview schedule and informed by analysed interview data. Participant demographic information will also be collected, including: age, professional background, level of experience, highest qualification, occupational position and years in current position, organisation and industry. In total, the survey will take approximately 25 min to complete.

A pilot study will be undertaken to ensure that instructions and questions are comprehensible (eg, appropriate terminology and jargon are used), the length of time required for completion is feasible and software functions effectively. Senior management teams of partner agencies (n=15 individuals in total), and colleagues of the research team with healthcare professional experience (n=10) will comprise the pilot sample and provide critical feedback. The survey design will be amended as necessary.

Following the pilot study, industry partners will provide a catalogue of their member health services, including details regarding organisational locations (metropolitan, regional or remote), sizes (number of staff) and type of ownership (public or private). Lists will be stratified based on these characteristics, with at least 100 services from each healthcare domain purposively selected to ensure a range of organisations are included in the sample. Through prior collaboration between researchers and industry partners, the sample size was determined to ensure the study is feasible, yet remains capable of producing industry and policy-relevant results. Individuals responsible for accreditation processes in each selected health service (eg, quality managers) will be invited to participate via emails from partner accreditation agencies. Information and consent forms will outline the study and potential participant's role within it. Follow-up of non-respondents via telephone will occur after 1 month to maintain an adequate response rate (over 65%).36 A review of accreditation literature by the research team5 identified considerable variation in the response rates of similar studies, which impedes rigorous estimation of likely rates that may be achieved for this and other surveys within the project. Nonetheless, the facilitation of recruitment by partner agencies and the targeting of individuals with a professional focus on accreditation should ensure a positive outcome.

Several methods will be used to analyse the survey data, including: generalised linear modelling applied to Likert-type outcomes, such as proportional odds modelling,37 categorical data analysis techniques,38 supervised and unsupervised statistical learning,39 and factor analysis.40 Exact and efficient inferential procedures will be used to ensure statistical validity of obtained analytical outcomes and maximum utilisation of available sample sizes.41 42 Empirical patterns will be examined by stratifying quantitative data with respect to key alternative factors based on respondent and organisational characteristics. Participant responses will be compared both within and between stakeholder groups and accreditation domains to identify variations regarding views of key themes. Potential confounding factors and causal links will be examined based on empirical evidence.

Study 2: critical elements of accreditation

Study 2 aims to deduce the critical elements of accreditation processes that stimulate improvement, and identify factors that drive behaviour change in health services and their staff.

Focus groups with accreditation agency stakeholders, jurisdictional health department representatives and staff from accredited health services

Focus groups will be conducted with accreditation agency stakeholders (n=6 focus groups), jurisdictional health department representatives (n=8 focus groups) and staff from accredited acute, primary and aged care services (n=15). Each group will involve between 5 and 10 participants and last approximately 1 h. Focus groups were considered an ideal methodology for this study activity due to their ability to capture communication between research participants, which can help generate additional insights to one-on-one interviews.43

Accreditation agency stakeholders will be purposively drawn from the lists used to guide the interview sample selection for study one. Where possible, different groups will be invited to participate, but the same recruitment strategy will be employed. Staff from accredited health services will be recruited at educational workshops run by accreditation agencies in metropolitan and regional locations throughout Australia. Focus groups regarding each accreditation programme will be arranged in different states to obtain a national perspective. Two experienced researchers (RH and DG) will conduct the focus groups.

Specific accreditation elements to be explored within focus groups will be informed by research literature5 44 and the combined accreditation knowledge accumulated by research partners. Topics may include: accreditation agencies (eg, their credibility and transparency), accreditation standards (eg, development processes), performance measures/indicators/outcomes (eg, their validity and associated collection and collation difficulties), surveyors (eg, reliability and training/educational support issues) and levels of consumer involvement (eg, the role of consumer surveyors and issues surrounding public disclosure of accreditation results). Focus groups will be digitally recorded and transcribed. Transcriptions will be analysed using the same methods as for the interview data in study 1.

A web-based survey will be implemented to triangulate and supplement the focus group results. The survey will ascertain respondents’ views of the relative importance of accreditation elements and their role in promoting organisational improvement. Participants will be drawn from staff of accredited organisations across acute, primary and aged care settings (∼n=300). Equivalent survey methods will be employed as for study 1, including: sampling frame and rationale (ie, a purposively selected sample of a sufficient size to produce industry-relevant results), questionnaire design (ie, 25 Likert scale questions), recruitment strategy (email invitations from accreditation agencies) and analytical techniques. A free-text section will allow participants to describe examples where specific accreditation elements have promoted measurable change in their organisation.

Study 3: standards and their impact

Study 3 aims to examine: how standards are developed and used; how they incorporate evidence and influence the expertise of clinicians, managers and policymakers; and how their application promotes change in organisational performance and clinical practice.

Retrospective analysis of standards development processes

The research team will retrospectively analyse the activities and processes used to develop standards for the main accreditation programme of each partner agency. For data collection, systematic examination of accreditation agency standards development committee and workshop reports, that is, ‘documentary analysis’,45 will be employed in combination with focus groups (n=3) involving between 5 and 10 individuals involved in standards development processes for each agency. Focus groups will be conducted by two experienced researchers (RH and DG). The main topic will be factors promoting standards development. Based on prior work of the research team,5 key factors likely to be discussed include public inquiries, reporting of adverse events or International Clinical Guidelines. Focus groups will be recorded, transcribed and thematically analysed. Sample selection, recruitment and access to relevant reports will be facilitated by partner agencies.

Survey of staff of accredited health services

A web-based survey will investigate the views of staff from accredited health services regarding the application of standards and how they promote change (∼n=300 respondents). Specific standards-related issues to be explored will be drawn from prior studies conducted by the research team1 5 and may include: use of empirical evidence for development; clarity of organisational requirements; revision processes; impacts on administrative processes, clinical care and patient/client satisfaction; roles of surveyors; and whether standards should be developed independently of assessment agencies. A free-text section will ask participants to describe examples where standards have produced measurable change. Survey software and design, sample selection, recruitment and analysis methods will mirror those used for prior survey studies.

Case studies of key accreditation programme standards

Case studies (n=5) of specified key standards—evaluation of care, documented policies, the quality improvement system, health records and infection control—will be conducted. Through prior literature reviews, studies and collaboration with industry and policy partners, these standards were identified as critical influences on the utility of accreditation programmes. For each case study, factors related to organisational change resulting from standards will be examined within a single health service from acute, primary and aged care domains. The sample will be purposively selected using the free-text section of study 3 survey results.

Organisational performance data will be drawn from accreditation and clinical indicator reports where available. The attitudes and beliefs of health service staff will be obtained through focus groups (n=15). Between 5 and 10 individuals from each organisation will be invited to participate in a 1-h focus group, in which participants will be questioned regarding the impacts of key standards. The focus group schedule will be developed based on results from previous research activities for study three. Two experienced team members (RH and DG) will conduct the focus groups. The resulting data will be analysed using the same techniques as for other qualitative studies previously described in this protocol.

Ethics and dissemination

The UNSW Human Research Ethics Committee has approved the proposals for each study, as well as related information sheets and consent forms summarising the research and roles of potential participants. All project data will be de-identified by removing statements identifying participants prior to their use in published materials. All survey, interview, focus group and case study data will be stored securely in password-protected folders accessible only to the research team. In accordance with Australian National Health and Medical Research Council (NHMRC) guidelines,46 all project data will be destroyed after a minimum of 7 years. Participants will be provided with contact details of the research team to enable them to address any potential queries or concerns. Complaints will be systematically recorded and actioned in accordance with UNSW guidelines.

Participants will be informed of their right to access their own results, and the overall results of the research, in accordance with NHMRC guidelines.46 Journal articles developed from the research will be sent to all project partners for distribution to participants, and de-identified results will be published on a publicly accessible website. Conference and seminar presentations will be made to Australian and international healthcare stakeholders.

Conclusion

The project involves a collaborative partnership between major Australian healthcare accreditation agencies, quality improvement bodies and researchers. The three multimethod, triangulated studies presented in this protocol will evaluate CAP. The research and professional knowledge, skills and experience harnessed by the research partnership provides sufficient capacity to design and implement a project capable of producing wide-ranging industry and policy-relevant results. The project will address knowledge gaps concerning the utility and reliability of health service accreditation, and increase the transparency and credibility of accreditation processes.