Short-notice (48 hours) ACCREDITATION trial in Australia: stakeholder perception of assessment thoroughness, resource requirements and workforce engagement.

BACKGROUND: External, independent accreditation assessments of healthcare organisations are necessary to ensure the nationally legislated minimum standards of quality and safety (QS) are met. The predetermined scheduling of the assessments continues to be criticised due to the high level of organisational emphasis on preparing for accreditation.
OBJECTIVES: To determine the stakeholder perception of assessment thoroughness, staff resource requirements and workforce engagement changes if only 48 hours' notice is given to an organisation prior to an accreditation assessment, compared with the standard-notice accreditation process.
METHODS: Logan and Beaudesert Hospitals in Brisbane, Australia, trialled the 'Short-Notice Survey Accreditation Assessment Process' (SNAAP) between August 2017 and December 2018. The organisation was given just 48 hours' notice prior to an accreditation assessment. Staff perception of the standard-notice accreditation process and short-notice process was assessed using a 5-point Likert scale repeated measures questionnaire (pretrial, 6 and 12 months after SNAAP launch).
RESULTS: There was a statistically significant stakeholder opinion that SNAAP more effectively identified the true strengths and achievements of the organisation's QS compared with 'standard-notice' survey (p=0.033). There was a significantly lower overall perceived proportion of staff resources required for SNAAP preparation in contrast to 'standard-notice' process (Baseline Av=21.38% vs Follow-up 1 and 2 Av=9.75%-6.25%, p=0.021). The questionnaire results reflected that SNAAP increased staff engagement in QS activities (Av=3.75 and 3.69, 95% CI=3.45-4.05 and 3.45-3.94).
CONCLUSIONS: With sufficient cultural and operational preparation to move to SNAAP, hospitals can potentially use SNAAP as a truer validation of QS standards, require less staffing resources to prepare for accreditation assessments and improve staff engagement in QS assurance and improvement. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Background

In Queensland, Australia, it is mandatory that all hospitals are accredited against a set of National Safety and Quality Health Service (NSQHS) standards. Independent accreditation organisations perform assessments of healthcare organisations in relation to their performance against NSQHS standards, and ‘accredited’ status is achieved when the standards are met. In 2018, Australian hospitals were assessed on clinical governance (CG), partnering with consumers, healthcare-associated infections, medication safety, patient identification and procedure matching, clinical handover, blood and blood products, preventing and managing pressure injuries, and recognising and responding to clinical deterioration in acute healthcare (Standards 1–10, respectively).1

Accreditation of healthcare organisations has been positively correlated with clinical performance and achieving the status of ‘accredited’ has traditionally been celebrated as the achievement of organisational excellence.1–4 Despite the positive association, the value and impact of structured, scheduled accreditation programmes (‘standard-notice’ accreditation process) and the culture of celebrating the achievement of minimum acceptable quality and safety (QS) standards is becoming more widely criticised.2 5–7

Predetermined timing of the onsite accreditation survey is commonly accompanied by a high level of organisation emphasis on preparation and improvement in the months leading to the planned accreditation date.7–9

With such thorough priming for the survey assessment, there is a significant risk that the accreditation surveyors may fail to identify true weaknesses in the hospital’s day-to-day activity, and the assessment results may not reflect the genuine level of QS in the organisation.2 5 10 11

The opportunity cost associated with staffing resources allocated to accreditation preparation is the day-to-day clinical and non-clinical activities such as direct patient care and workforce development.2 5 7 10

The concentrated focus and engagement in QS assurance and improvement is also very difficult to sustain after accreditation date.6 8

To overcome these perceived limitations of the ‘standard-notice’ accreditation process, there have been several international organisations that have trialled/implemented and researched unannounced or short-notice accreditation versus standard-notice/announced accreditation processes.

Research from Danish hospitals trialling unannounced surveys investigated whether unannounced surveys were more effective at identifying QS deficits in hospitals, however found no statistical difference between standard notice and unannounced surveys.7 10 In contrast, when Greenfield et al researched a trial of short-notice surveys in 20 healthcare organisations in a 2012 Australian study, they found that a significant number of the organisations may not have successfully passed accreditation despite passing their previous ‘standard-notice’ assessment.12

The Joint Commission is an international accreditation organisation that introduced unannounced surveys in the USA in 2004.13–15 Research on hospital performance has found that accredited hospitals under the unannounced survey process have performed better in indicators associated with best practice treatment of acute myocardial infarction, pneumonia and congestive cardiac failure, and have shown a reduction in patient mortality.11 14

Logan and Beaudesert Hospitals within Metro South Hospital and Health Service (MSHHS) trialled and passed accreditation under the short-notice process which we considered a validation that the organisation had successfully achieved a ‘Quality Care Everyday’ culture. The subsequent research intended to evaluate stakeholder perception of the short-notice process. The hospitals partnered with the Australian Commission on Safety and Quality in Health Care (The Commission), the Australian Council on Healthcare Standards (ACHS—the Accreditation Agency), Queensland Health, and Wide Bay Hospital and Health Service (WBHHS) to trial the ‘Short-Notice Survey Accreditation Assessment Process’ (SNAAP).

Under SNAAP, the organisation was notified only 48 hours prior to each accreditation survey date.

Objectives

The aim of this research was to evaluate stakeholder perceptions of SNAAP (compared with standard-notice process) in the objective indicators: (1) perceived level of thoroughness, rigour and accuracy of the survey (Are the results from accreditation an accurate reflection of the true day-to-day level of QS in the organisation?) (2) Does SNAAP change staffing resources required for accreditation preparation? (3) Does moving from standard-notice accreditation process to SNAAP change staff engagement with QS assurance and improvement?

Methods

Setting

Logan and Beaudesert Hospitals are situated mid-way between Brisbane and Gold Coast in Queensland, Australia. Logan Hospital is a 435-bed secondary teaching hospital, offering surgical, medical, women's health and children and neonatal services. Beaudesert Hospital is a 22-bed rural hospital located 45 km south-west of Logan Hospital. Over the two sites, there are over 2100 full-time equivalent staff members.

Interventions and study design

ACHS performed the first SNAAP assessment in November 2017 with National Standards 1, 2 and 3 assessed. Within 24 hours of the first assessment, the organisation was notified that

National Standards 4 and 7 were to be assessed 48 hours later. The third assessment occurred in May 2018, with Standards 5, 6 and 9 assessed and the fourth and final assessment for the trial was undertaken in October 2018 with EQuIP National Standards 11 and 12 assessed. Both Logan and Beaudesert Hospitals were assessed against these Standards during each of the four surveys.

The accreditation body validated the organisation had met all QS standards and action items assessed during the four visits.

Under the SNAAP process, the assessment team could assess any additional standards if a potential or significant patient safety risk was identified during the survey assessment or request an additional follow-up visit.

To evaluate the stakeholder perceptions of SNAAP, Queensland Health commissioned an external consulting company (Lirata) to collaborate with MSHHS, WBHHS and ACHS to develop and undertake the 18-month multicentre SNAAP trial research.

The objectives of the research were evaluated using a repeated measures questionnaire design before and after implementation of SNAAP, incorporating Likert scalar questions. The participant responses were not matched between pre-SNAAP and post-SNAAP implementation questionnaires. The baseline questionnaire was emailed out to eligible participants in August 2017, with the first follow-up questionnaire sent out in December 2017 and second (and final) questionnaire sent out in September 2018.

Participants

Ethics approval and consent to participate

Eligible participants were invited via email to participate in the study by completing the anonymous questionnaires. Consent was implied by questionnaire completion (as approved by The Prince Charles Hospital (EC00168) Human Research Ethics Committee in Queensland, Australia, under the ethics application HREC/17/QPCH/270)

There were three inclusion criteria for the selection of questionnaire participants: (1) the participant must be an employee of Logan or Beaudesert Hospital; (2) the participant must have accountability for preparing and maintaining the documentation required for accreditation evidence (executives, standard leads, team leaders/clinical leads, managers, subcommittee members, Clinical Governance Unit (CGU) staff members); (3) the participant must have contributed to a ‘standard-notice’ survey and are due to contribute to SNAAP (baseline questionnaire)/have contributed to SNAAP (follow-up questionnaires).

Statistical analysis

The ordinal Likert response categories were assigned an ascending binary numerical value (eg, Strongly disagree=1) to allow for a more comprehensive range of statistics. Using parametric testing for Likert data is often debated, as the assignment of values to the ordinal categories can be deemed arbitrary without a relationship between the categories defined. The statistical analysis of the data was run using parametric and non-parametric testing, which produced non-significant difference of results, and thus parametric testing, Student’s t test, one-way analysis of variance and χ2 test, was chosen as a viable analysis method of this research. These tests assumed normal distribution of the mean, as the sample size was greater than 10.16 This choice was supported by literature investigating appropriate statistical analysis methods of Likert scale data.17–19

Results

Participant selection

Of the 115 eligible questionnaire participants, 52 completed the baseline questionnaire (46%), 31 completed the follow-up questionnaire sent out in November 2017 (28%) and 39 completed the final questionnaire (39%). The participant selection flow chart is illustrated in figure 1.

Figure 1

Participant selection flow chart.

Did stakeholders perceive a difference in the level of thoroughness, rigour and accuracy of the SNAAP process survey compared with standard-notice process?

The questionnaire contained three questions pertaining to participant views on this objective (table 1). The overall sentiment in both follow-up questionnaires was primarily that SNAAP was the same as a ‘standard-notice’ survey in perceived thoroughness of the assessment, however analysis suggests results in follow-up questionnaire 2 reflected an increased perception of comprehensibility (Av scored response=3.25, 95% CI 2.87 to 3.63).

Table 1

Did stakeholders perceive a difference in the level of thoroughness, rigour and accuracy of the SNAAP process survey compared with standard-notice process?

Question		Major decrease	Minor decrease	About the same	Minor increase	Major increase
What effect has the SNAAP model had on the degree to which the survey process comprehensively covered the areas of the HHS in scope for the survey?	Follow-up questionnaire 1 (n=20)	20% (4/20)	5% (1/20)	50% (10/20)	15% (3/20)	10% (2/20)
	Follow-up questionnaire2 (n=24)	4% (1/24)	8% (2/24)	58% (14/24)	17% (4/24)	13% (3/24)
What effect has the SNAAP model had on the accuracy of survey results with respect to the day-to-day level of safety and quality of your HHS?	Follow-up questionnaire1 (n=23)	− (0/23)	13% (3/23)	43% (10/23)	26% (6/23)	17% (4/23)
	Follow-up questionnaire2 (n=31)	− (0/31)	13% (4/31)	55% (17/31)	16% (5/31)	16% (5/31)

HHS, hospital and health system; SNAAP, Short-Notice Accreditation Assessment Process.

The majority of participants felt that there was either no change or increase in the accuracy of the SNAAP survey results in reflecting the day-to-day QS standard of the hospital (Av scored response=3.48 and 3.35, 95% CI 3.09 to 3.87 and 3.03 to 3.68).

When specifically asked about how effective both accreditation processes identified strengths and achievement within the organisation related to QS, there was a greater consensus that SNAAP was more effective than ‘standard-notice’ accreditation in the follow-up 1 questionnaire (Baseline Av=3.79 vs follow-up questionnaire 1 Av=4.24, p=0.033). Overall it can be summarised and suggested there was a positive perception of SNAAP’s effectiveness (Av=3.59 and 3.57, 95% CI 3.31 to 3.86 and 3.38 to 3.76) in accurately assessing the true day-to-day QS standards in the organisation when compared with the ‘standard notice’ process.

Does the SNAAP process change staffing resources required for accreditation preparation?

The questionnaire evaluated the average percentage of time each week participants spent on accreditation preparation at various intervals prior to—and after—both ‘standard-notice’ surveys and SNAAP (figure 2). There was a significantly lower overall proportion of time spent preparing for SNAAP in contrast to the ‘standard-notice’ accreditation assessments (Baseline Av=21.38% vs Follow-up 1 and 2 Av=9.75%–6.25%, p=0.021), however the time spent preparing was similarly distributed in both follow-up surveys compared with the baseline (Follow-up questionnaire 1, p=0.639; Follow-up questionnaire 2, p=0.875).

Figure 2

Average proportion (%) of time spent per working week on activities related to accreditation preparation.

Most responders answered that SNAAP decreased the proportion of the week that was required to prepare for accreditation compared with the ‘standard-notice’ process (Av=4.07 and 3.81, 95% CI 3.75 to 4.39 and 3.49 to 4.45).

Seventy-five per cent (follow-up 1, 9/12) and 69% (follow-up 2, 9/13) of responders felt that SNAAP reduced the opportunity costs to the hospital of staffing resources allocated to accreditation preparation. (table 2) (Av=4.17 and 3.81, 95% CI 3.69 to 4.64 and 3.67 to 4.45). The data were broken down into staff member professional role to determine whether there were any significant outliers influencing the average values, however it was determined that all staff roles followed the same trend with regard to time allocated to preparing for accreditation, except for the committee members (who were responsible for rearranging documentation within the folder structure a year prior to the trial). Considering number of respondents in this group, their responses did not significantly influence the overall distribution of time spent.

Table 2

Does SNAAP process change staffing resources required for accreditation preparation?

Question		Major decrease	Minor decrease	About the same	Minor increase	Major increase
What effect has the SNAAP model had on the extent of staff resources required for accreditation administration and preparation?	Follow-up questionnaire1 (n=28)	36% (10/28)	39% (11/28)	21% (6/28)	4% (1/28)	− (0/28)
	Follow-up questionnaire2 (n=36)	31% (11/36)	25% (9/36)	39% (14/36)	6% (2/36)	− (0/36)
To what extent did the change in resourcing required for SNAAP enable the organisation to provide resourcing elsewhere?	Follow-up questionnaire 1 (n=12)	− (0/12)	− (0/12)	25% (3/12)	33% (4/12)	42% (5/12)
	Follow-up questionnaire2 (n=16)	− (0/16)	− (0/16)	31% (5/16)	31% (6/16)	44% (6/16)

HHS, hospital and health system; SNAAP, Short-Notice Accreditation Assessment Process.

Does moving from standard-notice accreditation process to SNAAP change staff engagement with QS assurance and improvement?

One hundred per cent of participants agreed that SNAAP had either the same or increased level of staff engagement in QS activities (Av=3.75 and 3.69, 95% CI 3.45 to 4.05 and 3.45 to 3.94). No participants believed it caused a decrease in engagement (table 3).

Table 3

Does moving from standard-notice accreditation process to SNAAP change staff engagement with QS assurance and improvement?

Question		Major decrease	Minor decrease	About the same	Minor increase	Major increase
What effect has the SNAAP model had on the consistency of engagement by your HHS in safety and quality activities throughout the accreditation cycle?	Post-SNAAP1 (n=28)	− (0/28)	− (0/28)	46% (13/28)	32% (9/28)	21% (6/28)
	Post-SNAAP2 (n=36)	− (0/36)	− (0/36)	47% (17/36)	36% (13/36)	17% (6/36)

HHS, hospital and health system; QS, quality and safety; SNAAP, Short-Notice Accreditation Assessment Process.

Discussion

Logan and Beaudesert Hospitals aspired to trial SNAAP to validate that our organisation had successfully achieved a ‘Quality Care Everyday’ culture, with embedded QS standards in the day-to-day practice of the workforce. Prior to commencement, we carefully considered the conclusions made in the research article by Greenfield et al,12 and therefore made a conscious decision to allow accountability and engagement cultural changes to be implemented and to mature over 3 years before seeking permission from The Commission and Queensland Health to trial a short-notice accreditation process.

By passing each accreditation survey, we interpreted that the organisation had effectively embedded adequate QS standards into ‘business as usual’ and could provide timely evidence of these standards to pass accreditation with only 48 hours’ notice.

The results from the questionnaires indicated that staff members perceived the SNAAP process to effectively detect the true level of QS in the organisation, and therefore we suggest the SNAAP process can be used to identify opportunities for improvement in the organisation’s QS systems and processes. An accreditation assessment can be used as a ‘spare set of eyes’ to identify gaps or better ways that the organisation may be unaware of and could then remedy through quality improvement methodology.

The questionnaire results also indicated that our staff members felt they spent a more consistent amount of their time, and on average a lower proportion of time per week on accreditation/QS documentation under SNAAP in the year leading up to the survey. This suggests that these activities are now embedded into the day-to-day scheduled activities of the stakeholders. This contrasts with the increasing amount of time spent on accreditation-related activities in the months/weeks leading up to the standard-notice accreditation process. A concern with the ‘standard-notice’ process resource requirement is the opportunity costs of direct patient care, workforce development, quality improvement activities and other daily duties, particularly in the month leading up to a survey assessment where on average 30%–40% of stakeholder time is spent on accreditation-related activities alone (see figure 2).

Potentially due to the perceived reduced time resources required for accreditation preparation, the questionnaire responders indicated that they felt the same, or more engaged in QS improvement after the SNAAP process was implemented. Changing the culture from one that sees QS documentation as evidence required for accreditation only, towards one that views it as part of day-to-day business allows the workforce to engage more actively in projects and initiatives to improve patient outcomes.

Before making conclusions as to the perceived benefits of SNAAP over a ‘standard-notice’ accreditation survey, the amount of new systems and process implemented in the organisation, and time spent embedding those processes, needs to be taken into consideration.

Logan and Beaudesert Hospitals started preparing 3 years before the trial, by implementing and maturing a ‘Quality Care Everyday’ culture through:

1. Evaluation and improvement of the clinical governance framework (CGF).

The CGF detailed the systematic management processes through which the health service was accountable for continuously monitoring, evaluating and improving the quality of clinical performance and patient outcomes. It was developed with the engagement of the executive leadership team and relevant stakeholders to articulate accountability relationships between inter-related systems and processes of CG.

2. Clear accountability assignment.

For the past decade, our organisation had performed well in accreditation assessments and this was perceived by executive stakeholders to be a measure of success. Despite this, a large proportion of senior clinicians, particularly medical staff, continued to express dissatisfaction with the orchestrated ‘event’ management of the accreditation survey, and were critical of the dissonance between documentation provided for accreditation and the day-to-day provision of safe and quality care, which has also been expressed internationally through research on clinician opinion of accreditation.7 20 Two of the systematic reviews investigating 83 studies of health sector accreditation corroborated this opinion, with physicians criticising the process as time consuming, costly and irrelevant to their practice.4 5 21–23 To overcome disengagement in our organisation, sponsorship of each NSQHS Standard was assigned to executive team members. The executive sponsor was accountable for providing strategic support to the relevant standard subcommittee by supporting committee activities to ensure compliance and best practice. Senior clinicians were encouraged to apply for the most relevant committee lead role related to their practice area. The senior standard leads promoted responsibility for QS encouraging better outcomes for patients.

To ensure effective communication between executives, standard leads, subcommittees and clinical units, Clinical Nurse Consultants-Quality (CNC Quality) positions were created. The CNC Quality roles were divisional-based positions that provided support to operational units to ensure the CGF was effectively embedded and action points of each national standard were maintained in operational units. The CNC Quality worked closely with the various subcommittees and the CGU ensuring strategic and operational alignment.

3. Standardised audit and evaluation.

While most performance audits were undertaken by designated front-line clinicians, the health service audit and evaluation team (MSHHS Clinical Services Excellence Team) held predominant accountability for developing, implementing and maintaining a standardised audit and evaluation schedule across the district that provided evidence of compliance with National Standards. The implementation of this standardised methodology facilitated benchmarking between health service hospitals and units. Aggregated data collated into the health service CG scorecard were reported to all levels of governance, from the local unit to the Board QS Committee.

One year prior to the trial, the CGU introduced a simple electronic filing system to store QS-related documentation. A set structure of subfolders ensured all evidence of compliance was easily accessible and stored in a standardised way. Documentation of QS activity and monitoring became embedded in the day-to-day quality management process and the folder system provided sufficient information for the accreditation assessors.

The organisation also spent a significant amount of time using change management techniques to prepare the workforce for the new process, with the aim to increase engagement in changing their practice, and to reduce risk of change fatigue. Healthcare is a rapidly changing industry, with numerous strategic priorities undertaken concurrently at all time. The accumulation of energy, effort and stress associated with the implementation of change can lead to fatigue, resistance and burnout of a workforce.24–26

Three key strategies were adopted based on research of change management to manage the major change of implementing SNAAP:

Relating the change to patients and day-to-day care. Rather than a new process implemented for accreditation purposes, SNAAP was part of a bigger strategy to embed a robust QS culture.

Shared accountability including empowering the front-line staff. The shared ongoing workload ensured that no individuals or small groups had increased responsibility.15 27

Targeted communication. We did not saturate the workforce with numerous communication as it was recognised this could lead to desensitisation to the information.15 26

We acknowledge that the extended time frame used for preparation of the SNAAP process trial could heavily influence the organisation’s success in ‘passing accreditation’ under SNAAP, and also influence the staff sentiment towards SNAAP, as the stakeholders were given ample opportunity to query or express concerns in the lead up to the study prior to answering the questionnaires. This is an inevitable limitation of the research.

To strengthen the conclusions made about staff resource requirements, it would be advisable to conduct this study using time sheets or logbooks, rather than use memory recall of time spent preparation for accreditation.

It is recognised that there are also limitations to certainty of suggested assumptions made during the statistical analysis. There can be no guarantee ascertained surrounding independence of the baseline and follow-up surveys, as the responses of questionnaires were not paired. The characterisation of participants was not investigated at this time to determine contributing factors and variables and therefore no sensitivity analysis was undertaken to assess the impact of the final assigned scores.

It is recommended that more healthcare sites trial the SNAAP process and conduct research to further determine feasibility of SNAAP in sites of different characteristics such as workforce number, differing patient demographics and acuity, governance systems and QS culture. The new National Accreditation Assessment Framework has been released since the trial ended and the inter-relationship between it and SNAAP remains to be tested.

Over 100 healthcare organisations across Australia have now applied to undertake SNAAP and international interest is being shown. These organisations considering transitioning to the SNAAP should consider the potential benefits and costs discussed in this paper.

Conclusion

It is essential that an external, independent assessment of the QS of healthcare organisations reflects the true QS standards of the organisation. Significantly increasing staffing resources to prepare for accreditation is a financial burden on the organisation and can contribute to staff disengagement in QS activity and improvement initiatives. It is important for better patient care that a hospital maintains a high level of QS regardless of the organisation’s notice period prior to assessment.

Through sufficient planning, development of clear systems and process for accreditation and improving engagement of clinicians, we moved from an organisation that produces evidence of compliance with standards for accreditation to an organisation that uses the ‘Quality Care Everyday’ as a springboard to achieve consistent and replicable high levels of QS successfully validated through the SNAAP.