Repeat Auditing of Primary Health-care Facilities Against Standards for Occupational Health and Infection Control: A Study of Compliance and Reliability.

BACKGROUND: The elevated risk of occupational infection such as tuberculosis among health workers in many countries raises the question of whether the quality of occupational health and safety (OHS) and infection prevention and control (IPC) can be improved by auditing. The objectives of this study were to measure (1) audited compliance of primary health-care facilities in South Africa with national standards for OHS and IPC, (2) change in compliance at reaudit three years after baseline, and (3) the inter-rater reliability of the audit.
METHODS: The study analyzed audits of 60 primary health-care facilities in the Western Cape Province of South Africa. Baseline external audits in the time period 2011-2012 were compared with follow-up internal audits in 2014-2015. Audits at 25 facilities that had both internal and external audits conducted in 2014/2015 were used to measure reliability.
RESULTS: At baseline, 25% of 60 facilities were "noncompliant" (audit score<50%), 48% "conditionally compliant" (score >50 < 80%), and only 27% "compliant" (score >80%). Overall, there was no significant improvement in compliance three years after baseline. Percentage agreement on specific items between internal and external audits ranged from 28% to 92% and kappa from -0.8 to 0.41 (poor to moderate).
CONCLUSION: Low baseline compliance with OHS-IPC measures and lack of improvement over three years reflect the difficulties of quality improvement in these domains. Low inter-rater reliability of the audit instrument undermines the audit process. Evidence-based investment of effort is required if repeat auditing is to contribute to occupational risk reduction for health workers.

Introduction

Accreditation or certification of health-care facilities using auditing has been recommended by many organizations to improve patient safety and quality of care. Audit has been defined as “a quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change” [1]. The audit cycle involves five stages: choosing a topic, specifying practice standards, testing actual practice against these standards, corrective action, and finally demonstrating improvement in practice through subsequent data collection and closing the loop [2]. Given the significant financial and personnel investment required to conduct accreditation programmes, research is needed to ascertain the effectiveness and reliability of health-care facility accreditation and/or certification in improving patient safety and quality outcomes [3].

South Africa is an upper middle income country characterized by a high level of wealth inequality, with 82% of the population dependent on public sector health services, while private health services exist in parallel, catering for a minority [4]. Health-care policy is centrally administered by the national Health Ministry, but health care is run by the nine provincial governments. Providing universal health-care coverage through a National Health Insurance (NHI) system is a major political goal of the current government [5]. Accreditation of health facilities is one of the means proposed to improve quality of care in such a system. For this purpose, the document National Core Standards for Health Establishments in South Africa was published in 2011, outlining expectations for safe, quality care in both the public and private sectors [6].

The main purpose of the National Core Standards is to create a benchmark against which health-care facilities can be evaluated and to provide for the national certification of compliance of health establishments with compulsory standards. The Office of Health Standards Compliance has been established to monitor and enforce compliance with the standards (http://ohsc.org.za/publication/). Certification of compliance may be a prerequisite in application for NHI accreditation or funding in the future. The general approach is to use repeat auditing, with feedback on corrective actions required, to improve quality. Proposed regulations will establish sanctions for noncompliance, including a warning, certification revocation, fine, and/or criminal prosecution [7].

The National Core Standards, in line with the World Health Organization, recognize that achieving the goals of universal quality care requires a healthy, productive, and safe workforce. In pursuit of this broad goal, health workers need to be protected against risk of injury and other occupational hazards which in low- and middle-income countries prominently include occupationally acquired infection, particularly tuberculosis (TB) [8,9]. South Africa is a high burden country for both TB and human immunodeficiency virus (HIV) infections, making protection of health workers through infection prevention and control (IPC) and occupational health and safety (OHS) priority areas of health-care management [10]. Control of infection risk is both a worker issue via the right to protection from occupational disease and a patient care issue in ensuring the availability of a healthy workforce capable of delivering health services safely.

The professional practice of IPC has long been a responsibility of health-care facilities, although typically considered in relation to patient protection. Based strongly on the common law Duty of Care principle, IPC aims to prevent health-care facility–acquired infections, whether transmitted through inhalation or contact with body fluids or tissue. A number of required IPC practices appear in the National Core Standards.

Independent of health-care policy, worker protection is a statutory requirement in South African labor law as in many countries. The Occupational Health and Safety Act 85 of 1993 requires that employers provide and maintain a working environment that is safe and without risk to the health of their employees [11]. Employers are also required to protect persons other than their employees from harm, by implication including patients, visitors, students, volunteers, and contractors. The professional practice of OHS thus covers the health, safety, and well-being of all persons in the workplace and aims to foster a healthy and safe work environment.

As many of the requirements of the Occupational Health and Safety Act 85 of 1993 such as risk assessment, education and training of staff, and provision of personal protective equipment are also found in the National Core Standards, there is considerable synergy between IPC and OHS activities. However, there is evidence to suggest that the quality of both IPC and OHS are in need of improvement in the South African public sector. In nationwide external baseline audits conducted in South Africa in 2011–2012 by a nongovernmental organization (described further in the following), the proportion of fixed public health-care facilities “fully compliant” with IPC standards was reported to be very low at 0.82% (32 of 3880) [12]. With regard to IPC and OHS, the national mean facility IPC score (averaged over all facilities) was reported as 47% for primary health-care (PHC) facilities and that for the subset domain of OHS as 76% (PHC facilities and hospitals combined). Both were less than the compliance targets set by the National Department of Health [12]. These findings indicate varying performance across different metrics and raise the question of whether such data are a valid reflection of the actual situation in South African health-care services.

There is, however, a dearth of studies evaluating OHS and IPC compliance with standards in low- and middle-income countries and particularly in PHC facilities. A related question is the quality of the instrument or process used to make such evaluations. Specifically, the reliability or reproducibility of such assessment performed by different agencies or staff groups is under-researched.

The availability of audit data collected by different agencies in public sector PHC facilities in the Western Cape Province of South Africa over a four-year period provided an opportunity to contribute to such knowledge. The objectives of this study were to determine (1) the compliance of public sector PHC facilities with the South African National Core Standards for OHS and IPC, (2) changes in compliance at follow-up audits three years after baseline audits as a measure of impact, and (3) the inter-rater reliability of audits, in this case of audits performed internally by health facility staff versus that performed by external agencies.

Materials and methods

Study design

This was a cross-sectional study with a longitudinal component, involving analysis of a subset of data collected during three different audits using the National Core Standards instrument. These are labeled (1) baseline external, (2) follow-up internal, and (3) follow-up external.

Population and sampling

All fixed public PHC facilities operated by the Western Cape Government Department of Health were included in the sampling frame (N = 194). Facilities were eligible if they had both an external baseline audit conducted in 2011/2012 (for Objective 1) and a follow-up internal audit conducted between April 01, 2014, and March 31, 2015 (for Objective 2). Facilities that changed functions or moved during this time period were excluded. To measure audit reliability (Objective 3), all facilities that had both internal and external follow-up audits conducted within the same follow-up period between 01 April, 2014 and 30 June, 2015, were eligible.

A multistage sampling strategy was used, set out in Table 1. The Western Cape Province is divided into six health districts which are further divided into 32 health subdistricts, with a total of 194 facilities. Of these, 90 (46%) met the aforementioned eligibility criteria. Sampling from eligible facilities involved selecting one of each type of facility (clinic, community day center, community health center1) within each of the 32 subdistricts. Where there was more than one of a certain type of eligible PHC facility in a subdistrict, at least 50% of these were selected using a random number generator. These facilities were requested to submit their audit data. If there were no eligible facilities in a particular subdistrict that subdistrict was excluded. For Objective 3 because only a proportion of eligible facilities had both an internal and external audit carried out in the study follow-up period, all those facilities were requested to submit their external audit reports (refer following sections).

Table 1

Sampling of primary health-care (PHC) facilities in the Western Cape province, by health district, 2011

Districts (number of subdistricts)	Number of PHC facilities in 2011	Number of eligible PHC facilities	Sampled	Data received and facility included in study
District A (8)	46∗	17	16	15 (94%)
District B (5)	40	28†	18	17 (94%)
District C (7)	49	4‡	4	4 (100%)
District D (3)	24	16	11	10 (91%)
District E (5)	26	25	14	14 (100%)
District F (4)	9	0§	0	0
Total	194	90 (46% of 194)	63 (70% of 90)	60 (95% of 63)

PHC facility, primary health-care facility.

No clinics operated by the Western Cape Government Department of Health.

No community center audits conducted in the study period.

No clinic internal audits conducted in the study period.

No PHC facility internal audits conducted in study period.

Instruments and audits

All baseline audits were external and conducted by different audit teams from the Health Systems Trust, a nongovernmental organization contracted by the National Department of Health, using the National Core Standards baseline tools (version 2011) described in detail elsewhere [6]. The follow-up internal audits in 2014/2015 were conducted by the Western Cape Government Department of Health staff, consisting of peer audit teams conducting audits at facilities other than their own or a team from the district office, using the National Core Standards, version 2013 tools, an updated version (personal communication — AM Van den berg). Follow-up external audits in 2014/2015 were carried out by audit teams from the Office of Health Standards Compliance, also using National Core Standards version 2013 tools. It is not known whether the same audit team performed the follow-up external audits at all the facilities. After each audit, the facility received a feedback report and had to generate a quality improvement plan to be implemented with the goal of improving annual audit performance results.

Two separate National Core Standards audit tools were used respectively for clinics and community day centers/community health centers. In these tools, each individual measure (question) is classified into one of four (declining) combined levels of risk: “extreme”, “vital”, “essential”, and “developmental”. Each tool is also divided into functional areas (e.g., clinic manager, clinical services, pharmacy) depending on the type of facility (clinic or community center).

Some itemized measures have an associated multi-item checklist, for example, a checklist of 20 items with regard to whether a policy exists that covers all aspects of IPC. The score is the fraction of these questions classified positive, for example, 15/20 = 0.75. In the present study, this score was then converted to a binary response (compliant or not: 1 or 0) by comparing the score to a compliance threshold for that category of measure (refer in the following). The remaining itemized measures require a binary response, for example, whether the facility has a reporting system for needle stick injuries or not, scored 1 or 0 accordingly.

Although specific measures were amended, added, or deleted over the three years, the majority remained the same.

The most notable change was in the risk rating categories of specific measure mentioned previously. The National Core Standards baseline 2011 version had three risk categories (excluding “extreme”), and the 2013 version had the four risk categories listed previously, with some individual items being reclassified accordingly by the developers.

For this study, copies of baseline external audit questionnaires and reports, follow-up internal audit questionnaires and checklists, and follow-up external audit reports (of the selected sample of facilities) were requested. The full set of audit tools used for both clinics and community centers were scrutinized for items/questions that pertained to IPC and/or OHS. Only these items were included in the data extraction sheets.

IPC–OHS itemized measures had to be present in the audit tools from the baseline National Core Standards 2011 version and the National Core Standards 2013 version to be included in the baseline analysis and follow-up comparison (Objectives 1 and 2). To enable this comparison, baseline measures were reclassified by the authors into one of the four risk categories of the 2013 version (extreme, vital, essential, and developmental). For the follow-up external versus follow-up internal audit comparison (Objective 3), the data extraction sheet included OHS and IPC measures/variables found in the National Core Standards 2013 version only.

A number of combined measures were calculated using study-specific criteria.2 For each facility, a mean score for each of the four risk-rating categories was calculated from the itemized measures. Target compliance cut-off levels per risk category were applied to these mean scores to determine (binary) compliance status per risk category in each facility as follows: > 0.7 for developmental measures, > 0.8 for essential measures, > 0.9 for vital measures, and 1.0 for extreme measures. The four risk category scores were then combined (Table 2 for weights) in a facility score. If this score was less than 0.5 (50%), the facility was classified as noncompliant overall (Grade E), while a score of 50% or more resulted in various conditional compliance grades at intervals of 10% (Grade D = 50-59%, C = 60-69%, B = 70-79%) and score of Grade A equal to 80% or more which signified fully compliant. Lastly, a pooled mean score for all facilities was determined, reported as a percentage. These itemized and combined measures are reported across all the sampled facilities in the following tables and figure.

Table 2

Proportion of primary health-care (PHC) facilities compliant with measures and each risk rating category in 2011/12 and 2014/15 (N = 60)

Variable/measure	Baseline (external) audits (2011–2012) Facilities compliant: n (%)	Follow up (internal) audits (2014–2015) Facilities compliant: n (%)	Difference% (95% CI)
Functional area: clinic/CHC manager
IPC policy (E checklist)∗	18 (30%)	32 (53%)	23% (4; 43)†
The annual in service education & training plan includes IPC (especially TB and universal precautions) (E)	26 (43%)	42 (70%)	27% (7; 46)
There is educational material available for staff on universal precautions: hand washing/respirator use/sharps/PPE/cough etiquette (E)	44 (73%)	47 (78%)	5% (-10; 20)
There is educational material available to patients on prevention of the spread of TB (E)	49 (82%)	55 (92%)	10% (-4; 24)
Appropriate types of masks and FDA-approved respirators available and at risk staff fit tested (X)	50 (83%)	36 (60%)	-23% (-40; -7)
Rooms used for patients with infectious TB are separated by adequate physical barriers from non-TB patients (X)	42 (70%)	44 (73%)	3% (-12; 19)
Rooms used for accommodation/consultation of patients with respiratory infections have adequate natural or mechanical ventilation (E)	47 (78%)	55 (92%)	-14% (-0.2; 27)
A comprehensive policy on standard precautions is available (E checklist)	41 (68%)	46 (77%)	9% (-9; 26)
Reporting system for needle stick injuries (V)	50 (83%)	54 (90%)	7% (-8; 21)
Randomly selected clinical area: sharps safety (V checklist)*	44 (73%)	50 (83%)	10% (-7; 27)
Annual hand washing/hygiene campaign/drive held (V)	21 (35%)	25 (42%)	7% (-12; 25)
Up-to-date decontamination policy (E checklist)*	15 (25%)	23 (38%)	13% (-3; 29)
Staff able to explain used instrument sterilization procedure (E Checklist)*	31 (52%)	33 (55%)	3% (-17; 23)
Records show staff with NSI received PEP and have been retested (V)	24 (40%)	31 (52%)	12% (-5; 29)
The fire certificate for the facility is available (E)	7 (12%)	24 (40%)	28% (12; 47)
There are quarterly emergency drills (E)	0	6 (10%)	10% (0.7; 19)
Pooled mean overall facility score (weighted mean‡, SD)	66.08% (20.15)	66.26% (21.60)	0.18 (-6.50; 6.86)§

CHC, Community Health Center; CI, confidence interval; D, developmental; E, essential; FDA, Food and Drug Administration; IPC, infection prevention and control; IQR, interquartile range; NSI, needle stick injuries; PEP, postexposure prophylaxis; PPE, personal protective equipment; SD, standard deviation; TB, tuberculosis; V, vital; X, extreme.

Numerical variable based on checklist. Compliant on Essential measure if numerical score 80% or greater; compliant on Vital measure if 90% or greater.

McNemar test for all binary variable comparisons.

Weighting: X = 40%, V = 30%, E = 20%, D = 10% (none in this study).

Difference between means, paired data.

Longitudinal and inter-rater comparison

The proportions of facilities compliant in the baseline external audit in 2011/2012 were compared with those of follow-up internal audit carried out in 2014/2015, using the McNemar test for paired data. A 95% confidence interval (CI) was calculated for the difference between proportions.

Cohen kappa statistic [13] and percentage agreement, with 95% CI, were used to determine the reliability/agreement of the results between follow-up internal audits and follow-up external audits conducted in the same period (2014/15). Kappa statistics were interpreted according to the descriptions used by Viera and Garrett [14]. All data were analyzed using Stata statistical software, version 12 (StataCorp, Tx, USA).

Results

The total number of fixed PHC facilities existing in 2011 were 194 (Table 1) consisting of 136 clinics and 58 community centers, of which 185 (95%) had a baseline external audit conducted. Ninety facilities (46% of 194) had a follow-up internal audit conducted in 2014/15 (67 clinics and 23 community centers) and were therefore eligible for inclusion for purposes of Objective 2. Sampling as described previously resulted in 63 (70% of 90 eligible) facilities. Of these, 60 (95% of those sampled) provided data—40 clinics and 20 community centers. One rural district (F) was not represented because it did not have the follow-up internal audits required for eligibility at its PHC facilities. District A, a densely populated urban district, had only community centers (i.e., no clinics were represented as clinics in this district are operated by the municipality rather than the province).

A total of 30 follow-up external follow-up audits were conducted by the Office of Health Standards Compliance at PHC facilities (of a total of 194) between April 2014 and June 2015. Twenty-six of the 60 responding clinics aforementioned were eligible for the reliability analysis (Objective 3), which was thus based on a smaller sample, but with a response rate of 96% (25/26).

Table 2 summarizes the 2011/2012 baseline audit results. Proportions of facilities compliant on the 16 items ranged from 0 to 83%. Particularly low compliance practices were having an adequate IPC policy, an annual education/training plan that included IPC, an annual hand washing/hygiene campaign, an up-to-date decontamination policy, records of post-exposure prophylaxis after needle stick injuries, an available fire certificate, and quarterly emergency drills.

Fig. 1 shows the proportions of facilities at the different degrees of overall compliance, and separately, compliant within each risk category. The proportion of facilities fully compliant overall (>80% compliance) at baseline was low (27%). Compliance with essential and vital measures was poor, whereas for extreme measures it was 60%.

Fig. 1

Proportion (%) of facilities (n = 60) compliant overall and with each risk rating measure category at baseline (external assessment) and follow-up (internal assessment). D indicates absolute difference in proportions with 95% confidence interval in parentheses.

Table 2 also presents the results at follow-up internal (self-assessment) audits (2014/15) There was a general increase in the proportion of facilities compliant on all items except one extreme measure—having appropriate masks and Food and Drug Administration (FDA)–approved respirators available and fit tested on at risk staff. There was a statistically significant decline from 83% at baseline to 60% for this measure.

However, on the items that showed a positive trend, the change was statistically significant in only three, all off a low baseline. These were having an IPC policy, an annual education/training plan that included IPC, and a fire certificate.

Fig. 1 shows that the proportion of facilities compliant with essential measures showed the greatest improvement, from 2% to 25%, which was statistically significant. The proportion of facilities compliant with vital measures stayed the same, whereas for extreme measures it decreased. Although at follow-up the proportion of facilities previously noncompliant overall decreased from 25% to 20% and those fully compliant increased from 27% to 35%, neither difference was statistically significant. Notably (Table 2), the pooled mean overall facility score of 66% was the same at baseline and follow-up.

In general, clinics were worse off at baseline than were community centers and showed the most improvement at follow-up internal (self-assessment) audits (not shown). Community centers in general either showed no improvement or declined in compliance. The number of clinics (of 40) that were compliant overall doubled from 8 (20%) to 16 (40%) in contrast to community centers (of 20), of which the number compliant decreased from 8 (40%) to 5 (25%) (not shown).

Table 3 shows the level of inter-rater agreement between assessors who conducted the follow-up external audit (Office of Health Standards Compliance) and those who conducted the follow-up internal audit at the same clinic using the same tool in the same 15-month period. (As this is a smaller sample than used in the interperiod comparison, the compliance proportions are different.) The median duration that elapsed between internal and external audits was three months (interquartile range 3 - 8; range: 1 - 14). All internal audits were conducted before external audits.

Table 3

Clinic audits: inter-rater comparison of reported compliance between follow-up internal and external audits at same facilities in 2014–2015 (N = 25)

Variable/measure	Follow-up (internal) auditsFacilities compliant n (%)	Follow-up (external) auditsFacilities compliant n (%)	Percentage agreement (95% CI)	Kappa (95% CI)
Functional area: Clinic manager
IPC policy (E checklist)∗	14 (56%)	0	44% (24; 65)	N/C
The annual in service education & training plan includes IPC (especially TB & universal precautions) (E)	19 (76%)	3 (12%)	36% (18; 57)	0.08 (-0.03; 0.19)
There is educational material available for staff on universal precautions: hand washing/respirator use/sharps/PPE/cough etiquette (E)	23 (92%)	9 (36%)	44% (24; 65)	0.09 (-0.04; 0.23)
There is educational material available to patients on prevention of the spread of TB (E)	24 (96%)	23 (92%)	88% (69; 97)	-0.06 (-0.17; 0.06)
Appropriate types of masks and FDA-approved respirators available and at risk staff fit tested (X)	14 (56%)	24 (96%)	52% (31; 72)	-0.08 (-0.23; 0.07)
Rooms used for patients with infectious TB are separated by adequate physical barriers from non-TB patients (X)	19 (76%)	21 (84%)	76% (55; 91)	0.26 (-0.18; 0.69)
Rooms used for accommodation/consultation of patients with respiratory infections have adequate natural or mechanical ventilation (E)	21 (84%)	21 (84%)	84% (64; 95)	0.41 (-0.08; 0.88)
A comprehensive policy on standard precautions is available (E checklist)∗	19 (76%)	3 (12%)	28% (12; 49)	-0.03 (-0.21; 0.14)
Reporting system for needle stick injuries exists (V)	25 (100%)	13 (52%)	52% (31; 72)	N/C
Randomly selected clinical area: sharps safety (V checklist)*	23 (92%)	8 (32%)	32% (15; 54)	-0.04 (-0.22; 0.13)
Annual hand washing/hygiene campaign/drive held (V)	9 (36%)	3 (12%)	60% (39:79)	-0.02 (-0.32; 0.29)
Up to date decontamination policy (E checklist)* (N = 20†)	8 (40%)	0	68% (46; 85)	N/C
Staff able to explain used instrument sterilization procedure (E checklist)∗ (N = 19†)	12 (63%)	4 (21%)	68% (46; 85)	0.27 (0.01; 0.53)
Evidence of medical examinations on at risk staff (V)	15 (60%)	0	40% (21; 61)	N/C
Records show staff with NSI received PEP and have been re-tested (V) (N = 19†):	11 (58%)	5 (26%)	68% (46; 85)	0.22 (-0.13; 0.56)
The fire certificate for the facility is available (E)	12 (48%)	1 (4%)	56% (35; 76)	0.089 (-0.08; 0.25)
There are quarterly emergency drills (E)	2 (8%)	0	92% (74; 99)	N/C
Functional Area: Clinical Services
Appropriate types of masks and FDA approved respirators available and at risk staff fit tested available (X) (N = 24†):	13 (54%)	23 (96%)	52% (31; 72)	-0.08 (-0.24; 0.08)
Randomly selected clinical area: Sharps safety (V Checklist)∗ (N = 23†):	21 (91%)	11 (46%)	44% (24; 65)	-0.02 (-0.23; 0.19)
Lighting & ventilation adequate (E) (N = 24†):	20 (83%)	23 (96%)	88% (69; 97)	0.36 (-0.16; 0.88)
No obvious safety hazards (V) (N = 24†):	20 (83%)	20 (83%)	84% (64; 95)	0.40 (-0.08; 0.88)
Cleaning material/equipment available, appropriately labeled and stored (V checklist)∗ (N = 23†)	5 (22%)	1 (4%)	76% (55; 91)	-0.08 (-0.23; 0.07)
Summary measures
Extreme measures	9 (36%)	20 (80%)	48% (28; 69)	0.11 (-0.13; 0.35)
Vital measures	2 (8%)	0	92% (74; 99)	N/C
Essential measures	7 (28%)	0	72% (51; 88)	N/C
Pooled score across facilities (weighted mean, SD)‡	68% (19)	64% (10)	N/A	N/A
No. of facilities noncompliant (<50%)	5 (20%)	3 (12%)	76% (55; 91)	0.12 (-0.32; 0.55)
No. of facilities conditionally compliant (≥50 < 80%)	13 (52%)	22 (88%)	48% (55; 91)	-0.073 (-0.33; 0.19)
No. of facilities fully compliant (≥80%)	7 (28%)	0	72% (51; 88)	N/C

CI, confidence interval; D, developmental; E, essential; FDA, Food and Drug Administration; IPC, infection prevention and control; NSI, needle stick injuries; PEP, postexposure prophylaxis; PPE, personal protective equipment; SD, standard deviation; TB, tuberculosis; V, vital; X, extreme.

Numerical variable based on checklist. Compliant on Essential measure if numerical score 80% or greater; compliant on Vital measure if 90% or greater.

“Not applicable” and missing data excluded.

Weighting: X = 40%, V = 30%, E = 20%, Developmental = 10% (none in this study).

Table 3 shows that the percentage agreement on the 16 itemized measures ranged from 28% (whether a comprehensive standard precautions policy was available) to 92% (whether quarterly emergency drills took place). Percentage agreement between internal and external audits appeared good for overall facility noncompliance (76%) and for overall full compliance (72%). However, when the proportion of agreement expected owing to chance was taken into account, agreement on itemized and summary measures was poor to moderate, with kappas ranging from -0.08 to 0.41. Only one itemized measure achieved moderate agreement beyond chance (defined as k = 0.41-0.60): assessment of adequate natural or mechanical ventilation for rooms used for accommodation/consultation of patients with respiratory infections (k = 0.41) but with a wide 95% CI (-0.08; 0.88).

Overall, external auditors rated fewer facilities compliant with itemized measures than did internal auditors. The three exceptions were adequate lighting and ventilation in facilities (96% versus 83%, k = 0.36), whether rooms used for patients with infectious TB were separated by adequate physical barriers from those for non-TB patients (84% versus 76%, k = 0.26), and whether facilities had approved masks and FDA-approved and fit-tested respirators (85% versus 56%, k = -0.08). This latter divergence had a disproportionate influence on the pooled facility score for extreme measures (80% vs 36%, k = 0.11).

Discussion

In this study of compliance of PHC facilities in South Africa with the OHS and IPC measures of the National Core Standards, the proportion of facilities compliant overall at baseline (2011/12) was low at 27%. This was predictable, given that facilities were just starting accreditation programmes [15]. This finding is in keeping with that of a 2009 study of ten PHC facilities in the Western Cape Province—while eight had adequate supplies of respirators, only two had infection control plans and five had a designated infection control officer on site [16]. Using a scoring rather than a threshold metric, the mean facility combined IPC/OHS score was 66%, close to that of country-wide public health facility audits performed in 2011 which reported a mean of the national IPC and OHS score of 62%. However, both sets of figures fall short of the facility target compliance level of 80% [12]. Equivalent data for the elements of interest in this study were not available for provincial level in the national study, limiting any statement about generalizability to other provinces.

One reason for incomplete compliance is likely to be the historical neglect of OHS and IPC in PHC facilities, where they are generally regarded as ancillary activities with a low level of accountability among senior management [17]. In addition, there is no provincial OHS or IPC unit or manager and a lack of district OHS–IPC–qualified personnel to coordinate and support OHS/IPC activities in the districts of this province, with the majority of the limited OHS and IPC qualified staff attached to large urban hospitals. Although there are policies, their implementation is lacking [17].

Studies evaluating reliability of IPC or OHS audits in PHC facilities are scarce. The poor inter-rater reliability found in this study is consistent with a systematic literature review in 2010 on the measurement properties of OHS management audits. The aforementioned review found inter-rater reliability to be frequently unacceptably low [18]. There are exceptions. A study in Ecuador comparing internal with external audit for measuring compliance with quality standards in hospitals, found kappa statistics ranging from fair to almost perfect and percentage agreement ranging from 71 to 95% [19]. Where there were disagreements, internal auditors were inclined to report more positive findings than external auditors.

In the follow-up component of the present study, the external (Office of Health Standards Compliance) audits scored facilities lower on all measures except three, the most numerically influential being the extreme measure of FDA-approved respirators and fit testing. This might be explained by the time lapse between internal and external audits (mean 3 months) with interval correction of this measure. It would have been easier to purchase equipment such as N95 respirators than updating an IPC–OHS policy, changing infrastructure, starting an education/induction programme, or providing medical surveillance without the necessary expertise or resources available.

While underlying factors were not assessed in this study, poor reliability may be owing to an inadequate measurement scale/tool and/or inadequate selection and training and supervision of auditors [20]. Of the methods of assessment required across all domains, in the IPC and OHS domains studied here document review, observation, and staff interview were the ones used by auditors. It is likely that audit team variation across facilities and variability in the training on the use of the audit tool between internal and external auditors contributed to the poor inter-rater reliability.

The external audits by the Office of Health Standards Compliance could not be viewed as the “gold standard” at the time of the study as this agency was still in the process of conducting audits and making final amendments to the audit tools. The validity of such statutory audits still needs to be determined. Although these external audits are considered by the agency to be of greater validity than internal audits, this study was not designed to confirm this view. Large discrepancies (in both directions) may exist between what health-care workers perceive to be in place in their hospital with regard to OHS/IPC and what external auditors report [21].

The poor reliability of the audit has implications for the interpretation of Objective 2 of the study, namely change in compliance over time. While some individual facilities did show an improvement in this study, the mean facility overall score was identical at baseline and follow-up internal audit, indicating that on average, there was no improvement overall. However, if the audit process is unreliable, it is not possible to make meaningful interpretations of the true impact of National Core Standards audits and feedback on facility compliance levels, resulting in a waste of financial and personnel resources required to conduct these audits. To the extent that internal auditors drawn from the department being assessed may have “overscored” their facilities, the actual impact may be even weaker than indicated in Table 2.

Reports and studies of quality audits in high income countries have shown gradual improvement over time in compliance, although these were mainly in hospitals [[22], [23], [24]]. The completion of a full audit cycle that includes monitoring implementation of changes and follow-up audits has been shown to improve impact [22,25]. With regard to the PHC a study in the Netherlands has evaluated the determinants of the impact of a primary medical care practice accreditation programme. The factors perceived by primary care professionals to be enablers of impact were as follows: designating one person responsible for the programme, having clear lines of communication and having enthusiasm for quality improvement [26].

Similarly, in across hospital comparison, good infection control performance has been associated with having resources such as full time IPC practitioners [27]. The lack of this qualified resource in a PHC setting may be one explanation for the relatively poor compliance and lack of improvement found in this study. Because only one nursing staff member may be trained in IPC, given the fewer number of staff at PHC facilities, turnover of such staff would disrupt continuity of IPC. Furthermore, good leadership at ward or operational level of staff who share the vision of the organization, who develop and stimulate others, and who are active is associated with effective action on IPC measures [28]. However, achievement of this positive type of leadership is adversely affected by direct supervision of a large number of staff, which may be another reason for a lack of improvement in our setting [28].

There are some positive results from low- and middle-income countries. A 2001 study in Mali to determine the impact of internal audits on compliance with quality of care standards reported a significant difference between the intervention group and the control group in overall compliance, suggesting that internal audits can have a significant effect [29]. A 2013 Iranian study to determine the compliance with the Joint Commission International organization–based standards for IPC in 23 hospitals using a self-reported questionnaire with hospital staff found an excellent (>75%) pooled mean hospital IPC score of 79% [30]. Whittaker et al. [15] have shown how facilitated gradual improvements in quality were beneficial in a large South African public sector hospital with a poor baseline and large room for improvement, which took up to three years to reach acceptable levels for accreditation. The relative effects of clinical audit and feedback are thus likely to be larger when baseline compliance with standards is low [ [31]. However, overall there is lack of studies in both high- income countries and low- and middle-income countries reporting on the impact of IPC or OHS auditing or accreditation in PHC as opposed to hospital settings.

Strengths of our study include representative sampling of PHC facilities which had actually undertaken audits, under the control of a single provincial department of health. The response rate among sampled eligible facilities was very high.

Limitations include the constraint imposed by the limited number of PHC facilities (90/194) that met the inclusion criteria, particularly that of having had a baseline audit. This resulted in wide CIs for the period comparison results. Although there were some textual changes across different versions of the audit instrument, most of the measures remained the same. However, some legally required OHS practice items were moved from the PHC National Core Standards facility audit tool to the “district/subdistrict office” tool. The latter managerial level tool was used once (but not covered in this study) and subsequently dropped (AM Van den berg—personal communication). This reflects reversion to a focus on operational or “proximate” measures and neglect of systemic or “upstream” factors which may, however, play an important role in determining compliance with protective standards. Finally, a major limitation of the National Core Standards audit tools is that they focus heavily on structure and process measures. As there were no outcome measures, it remains to be seen whether compliance leads to actual improvement in patient or staff outcomes.

Given the poor interaudit reliability, change might have been better assessed by comparing internal vs internal, or alternatively, external vs external audits over time. However, internal audits were not available at baseline. In addition, external audits at follow-up were performed only at about half the eligible facilities, as selected by the Office of Health Standards Compliance. Accordingly, to maximize the sample for measuring change over time, we used external audit at baseline and internal at follow-up. This in fact accords with the intention of the Office of Health Standards approach which is that facilities will effect quality improvement and their own (internal) audit after an external audit. The follow-up external audit results were thus used only for the reliability study, although the smaller sample resulted in wide CIs around the agreement metrics.

Conclusion

To our knowledge, this is one of the first studies to examine the performance of OHS and IPC audit in PHC facilities. The findings of this study add to the scarce literature on PHC facility compliance with IPC or OHS standards and inter-rater reliability of audit process. This subject is particularly important in low- and middle-income countries where the burden of infectious diseases such as TB and HIV is high and health workers are at significant risk of occupational TB.

The system of monitoring in South Africa has evolved since this study, specifically with the launch of an Ideal Health Facility programme aimed at improving the quality of PHC facilities (https://www.idealhealthfacility.org.za). However, monitoring of PHC facilities against criteria aligned with an updated version of the National Core Standards is set to continue as a means of quality improvement (personal communication—Dr G. Labadarios). Whatever final form the system takes, it needs to be evidence based as far as possible, given the attendant investment of time and effort.

The findings of this study are thus relevant. Baseline PHC facility compliance with OHS–IPC measures was low. Poor inter-rater reliability indicates a large amount of measurement error making it difficult to interpret changes over time as reflecting real improvement or deterioration in quality. Taking into consideration the limited improvements found and the possible upward bias in internal auditing, it is unlikely that there was significant improvement in compliance over the three years of observation. It is possible that feedback alone cannot be relied upon to improve IPC and OHS standards in PHC facilities and that the penalties mentioned earlier may need to be invoked. This may hold lessons for other low- and middle-income countries.

The first step in making auditing meaningful is to improve the reliability of the audit process itself. Remedying this problem requires continuous monitoring of inter-rater reliability and a quality improvement feedback mechanism for auditors [32]. This is an essential step in achieving confidence that corrective actions following on audit feedback have in fact been implemented. In low- and middle-income countries, where human resources for health are already scarce and at high burden of infectious diseases such as TB and HIV, it is crucial that health services improve their IPC and OHS practices. However, the use of repeat auditing to achieve such improvements in compliance and/or outcomes must itself be evidence based.

Disclaimers

None.

Authors' contributions

BC designed the study, carried out data acquisition, collection, extraction, analysis, interpretation, and the drafting of the manuscript. AY contributed to design of the study and critically revised the intellectual content of the manuscript. RE contributed to design of the study and critically revised the intellectual content of the manuscript. All authors gave final approval of the submitted version.

Ethical approval

Ethics approval was obtained from the Human Research Ethics Committee of the Faculty of Health Sciences, University of Cape Town (Ref. 075-2015). The study was approved by the Western Cape Government Department of Health. Consent to participate by individuals was not applicable.

Conflicts of interest

All authors have no conflicts of interest to declare.