Effects of patient safety auditing in hospital care: results of a mixed-method evaluation (part 1).

OBJECTIVE: To evaluate the effectiveness of internal auditing in hospital care focussed on improving patient safety. DESIGN, SETTING AND PARTICIPANTS: A before-and-after mixed-method evaluation study was carried out in eight departments of a university medical center in the Netherlands. INTERVENTION(S): Internal auditing and feedback focussed on improving patient safety. MAIN OUTCOME MEASURE(S): The effect of internal auditing was assessed 15 months after the audit, using linear mixed models, on the patient, professional, team and departmental levels. The measurement methods were patient record review on adverse events (AEs), surveys regarding patient experiences, safety culture and team climate, analysis of administrative hospital data (standardized mortality rate, SMR) and safety walk rounds (SWRs) to observe frontline care processes on safety.
RESULTS: The AE rate decreased from 36.1% to 31.3% and the preventable AE rate from 5.5% to 3.6%; however, the differences before and after auditing were not statistically significant. The patient-reported experience measures regarding patient safety improved slightly over time (P < 0.001). The SMR, patient safety culture and team climate remained unchanged after the internal audit. The SWRs showed that medication safety and information security were improved (P < 0.05).
CONCLUSIONS: Internal auditing was associated with improved patient experiences and observed safety on wards. No effects were found on adverse outcomes, safety culture and team climate 15 months after the internal audit.

Introduction

Hospitalized patients suffer from unintended harm caused by healthcare management [1-3]. Interest in effective and sustainable interventions for reducing patient harm is growing [4, 5]. Interventions regarding patient safety that aim to intervene at an organizational level and that actively engage workers in preventing patient safety risks are presented as promising solutions [6]. In the Netherlands, almost all hospitals have implemented internal auditing to monitor and improve patient safety in hospital care [7]. We defined internal auditing as an independent, objective assurance activity that uses a peer-to-peer evaluation approach to engage healthcare professionals in the plan-do-check-act (PDCA) quality improvement cycle [8]. PDCA is a model for improvement used in improvement science. It improves the quality of care by continuously and cyclically measuring and giving feedback of quality information to (and by) healthcare providers [9]. A team of trained auditors well-versed in the standards of professional practice and quality assessment methodology provides objective data regarding discrepancies between current practice and target performance over a specified period and the implications for patient safety. Verbal and written feedback regarding suboptimal performance can lead to recognition and act as a cue for action, encouraging healthcare professionals to take action to reduce the discrepancy and improve patient safety [10]. The effect of feedback information is based on the belief that healthcare professionals are prompted to modify their practice when given performance feedback showing that their clinical practice is inconsistent with a desirable target resulting in improved patient safety outcomes [11]. Auditing and feedback is one of the most widely used interventions in quality improvement for monitoring and changing health professionals’ behavior; however, it is also variably effective in professional practice [11]. Most studies focus on the influence of auditing and feedback on healthcare professionals’ performance on the proper use of treatments, tests or patient management [11]. Uncertainty remains as to when internal auditing, as a hospital-wide approach, leads to improved patient safety. The aim of this study was to evaluate the effect of internal auditing on patient safety outcomes, safety culture and team climate in hospital care.

Methods

We developed a conceptual framework based on the theories from the field of quality improvement [9, 12] implementation science [13] and Kirkpatrick’s learning model [14], to describe the relationship between auditing and effects on professional practice and patient safety outcomes (Fig. 1) [8]. Audits can be seen as the ‘check’ stage of the PDCA quality improvement cycle: auditors check whether quality standards have been established (‘plan’) and applied in practice (‘do’). Based on the audit results, improvement actions are implemented (‘act’) to improve safety outcomes on patient, professional and department levels [8]. The study protocol of the present study was published in detail in advance [8]. A summary of relevant aspects of the design is given below.

Figure 1

Conceptual framework.

Study design and setting

We carried out a mixed-method evaluation study with a before–after design between July 2011 and November 2014. We collected data on eight departments of a 953-bed university hospital, which were representative for hospital care, 3 months before and 15 months after the internal audit. The included departments were cardiology, general internal medicine, general surgery, neurosurgery, obstetrics and gynecology, orthopedics, pediatrics and pulmonary medicine. We used the Standards for QUality Improvement Reporting Excellence (SQUIRE) guidelines for reporting the methods used and the main findings [15].

Patient safety auditing

A trained team of six healthcare providers of the university hospital audited the included hospital departments separately between October 2011 and August 2014 according to a fixed procedure and scheme (Table 1) [8]. On the audit day, the audit team interviewed 35–55 auditees (especially healthcare providers and managers) using a predefined program. The internal audit resulted in a report with patient safety problems prioritized based on urgency by an independent Board of the Institute for Quality Assurance and Safety. The heads of the audited departments established and implemented an improvement plan based on the audit report and discussion within the department. The audit team revisited the departments 15 months after the internal audit to assess the progress and effects of the implementation of their improvement plans. The hospital’s board of directors monitored the progress of the improvement plans in regular accountability meetings with the department heads.

Table 1

Execution of patient safety auditing

Audit process	Activities by the audit team (A) and department (D)	Planning
1. Initiate the internal audit	Form audit team (A)	−6 Months
2. Introduce the internal audit procedure	Establish initial contact with the head of the department to be audited (A) Explain the audit process (A)	−3 Months
3. Prepare for audit visit	Department heads fill in self-assessment survey (D) Document study, observations, quality and safety measurements, formulating audit visit program and constructing interview framework based on audit focus (A) One-day specialized audit visit (pre-audit), e.g. medication safety and infection prevention (A)	−1.5 Months
		−0.5 Months
4. Audit visit	Interview according to framework (A) Oral presentation of the audit findings and conclusions (A)	AUDIT VISIT
5. Write and correct audit report	Prepare audit report (A) Correct factual inaccuracies in audit report (D)	+2 Months
6. Prioritize and submit audit report	Prioritize audit findings (A) Submit audit report (A)	+3 Months
7. Prepare and implement improvement plan	Writing of improvement plan (D) Implementation of improvement plan (D)	+6 Months
8. Examine improvement plan	Examine improvement plan (A) Submit feedback (A)	+7 Months
9. Revisit the audited department (follow-up)	Delivery of requested documents (D) Interview, observe and visit by plan (A) Generate revisit findings (A) Write revisit report (A) Correct factual inaccuracies in revisit report (D) Submit revisit report (A)	REVISIT
		+15 Months

Outcome measures

To evaluate the effects of internal auditing on patient safety, the outcome measures were defined at patient, professional, team and department level. The measurement instruments used were patient record review, questionnaires, analysis of routinely collected administrative hospital data and structured observations (Table 2).

Table 2

Outcome variables and data sources for evaluating internal auditing

Variable	Source	Method	Before–after measurement	Sample size (n)	Unit and type of analysis
Primary outcome
(Preventable) Adverse events	Records of hospitalized patients	Retrospective record review based on the Harvard Medical Practice Study	−3 Months +15 Months	451 Records 419 Records	Patient Linear mixed model with department as a random effect
Secondary outcomes
Patient-reported experiences	Consumer Quality Index	Survey	−3 Months +9 months +15 Months	442 Patients 468 Patients 432 Patients	Patient Linear mixed model with case-mix adjusters for patient characteristics and department as a random effect
Standardized mortality rate	Administrative hospital data	Routinely collected administrative hospital data	Monthly	41 394 Admissions/ 8 Departments	Patient Linear mixed model adjusted for patient characteristics and department as a random effect
Team climate	Team Climate Inventory	Survey	−3 Months +15 Months	23 Teams/ 588 Professionals 23 teams/ 612 Professionals	Team of professionals Linear mixed model adjusted for occupation with department as a random effect
Patient safety culture	Hospital Survey on Patient Safety Culture	Survey	−3 Months +15 Months	408 Professionals 428 Professionals	Professional Linear mixed model with department as a random effect
Observed patient safety	Safety walk rounds	Structured observation with a checklist	−3 Months +15 Months	8 Departments	Department Paired t-test (by department)

Patient level

Trained independent physician experts not involved with patient care in the university hospital determined the primary outcome, i.e. the presence of adverse events (AEs) and preventable AEs (PAEs) during hospitalization, for the period of July 2011 through November 2014, using structured record review based on the Harvard Medical Practice Study [3]. We defined an AE as an unintended injury resulting in temporary or permanent disability, death or prolonged hospital stay and which is caused by healthcare management rather than by the patient’s underlying disease [3]. We included records of patients with a higher risk of AE based on the seriousness of their disease and treatment upon hospital admission (e.g. colon cancer, pneumonia or acute myocardial infarction), deceased hospital patients and/or length of hospital stay >10 days [8]. For each department and measurement, we randomly selected a minimum of 50 records that met the inclusion criteria, resulting in a total of 870 reviewed records. Originally, we included eight departments, but as one was divided at the start of the record review, an eventual nine departments were included for AE assessment. Physicians double-reviewed 10% of the records to assess the inter-rater agreement.

We used the consumer quality index (CQI) questionnaire [16] based on the Consumer Assessment of Healthcare Providers and Systems [17] to measure patient-reported experiences (PREs). Patients who stayed at least one night in the hospital were, after discharge, invited to share their experiences by filling in a written or web-based questionnaire anonymously. We administered the CQI 3 months before and 9 months (= implementation start month) and 15 months after the internal audit. Patients rated their positive experiences with hospital care on a 4-point Likert scale ranging from 1 (Never/No) to 4 (Always/Yes).

The patient safety indicator, the standardized mortality rate (SMR), was measured on all admissions to the eight departments in the period of 2010–16.

Professional and team level

We used the COMPaZ questionnaire [18] and the Team Climate Inventory (TCI) [19] to measure changes in patient safety culture and team climate, respectively. The COMPaZ questionnaire is the validated Dutch version of the Hospital Survey on Patient Safety Culture [20]. Using the COMPaZ questionnaire, we measured multiple dimensions of patient safety culture, such as openness of communication, teamwork within and across clinical wards, and non-punitive response to error [18]. The TCI is a valid, reliable and discriminating self-report measure of team climate in hospital teams [19]. Using the TCI, we measured the collaboration of teams within a clinical ward. Team climate suggests that four climate factors, i.e. participative safety, support for innovation, vision and task orientation, are essential for developing and implementing improvements after an internal audit [19, 21]. Healthcare providers (physicians, nurses and physiotherapists) and managers who worked in the clinical wards of the eight departments were invited to fill in the COMPaZ and TCI questionnaires anonymously. In both web-based questionnaires, respondents reported their experiences with the patient safety culture and team climate on a 5-point Likert scale ranging from 1 to 5. A 5-point scale was used for most items in the COMPaZ and TCI questionnaires, ranging the positive experiences from ‘strongly disagree/never’ (1 point) to ‘strongly agree/always’ (5 points), with a neutral category ‘neither’ (3 points).

Department level

An experienced internal auditor together with a senior nurse or physician of a clinical ward observed patient safety topics in the clinical ward by conducting safety walk rounds (SWRs) [22]. They used a standardized form with 71 patient safety items to report their findings on a 3-point scale: ‘safe’, ‘unsafe’ or ‘not observed’. The maximum score was 100%. The SWRs were unannounced and conducted over ~1 h. The SWRs conducted before and after the internal audit were similar in terms of day, time and observers. The day and time within the working hours were randomly selected.

Statistical analyses

The data were checked to identify out-of-range answers, inconsistent responses and missing data. If we could not correct or complete the inconsistent and missing data, we excluded these data from the analyses (<3.5% excluded data). We used descriptive statistics to describe baseline characteristics of the patients, professionals, departments and outcome measures. Before analysis, we calculated the mean scores per dimension and the mean overall scores of the CQI, COMPaZ, TCI and SWR according to the corresponding manuals [21, 23, 24]. Negatively-worded items in the CQI and COMPaZ were reverse-coded so that all high scores could be interpreted as more positive answers.

We used linear mixed models using random effects to account for department-level clustering and repeated time measurements, while differences in patient, professional or team outcome of interest before and after the intervention (internal auditing) were adjusted for using fixed effects [25]. If this multi-level model did not fit, we first omitted the random effect for measurement time within the department, and when this linear mixed model still did not fit, we accounted for the nesting of patients or professionals within departments by including departments as fixed effects in the model. We adjusted for the patient characteristics sex, age, education, and self-reported general and mental health in the mixed-model analysis of the CQI [23] and for occupation in the linear mixed-model analysis of the TCI [19]. The SMR was analyzed using a linear mixed-model adjusted for factors that affect in-hospital mortality rates: patient age, sex, primary diagnosis, socioeconomic status, comorbidities, admission status, admission to a palliative care specialty and procedures [26]. The difference in percentage patient safety scores, observed by the SWRs before and after the internal audit, was analyzed using the paired t-test (by department).

We estimated the difference in AEs and PAEs before and after patient safety auditing with a precision of 7% based on the percentage AEs found by Zegers et al. [27], a sample size of 50 patient records from each of the eight departments and the presence of clustering in the before-and-after measurements. We calculated the intra-class correlation coefficient (ICC) that describes department-level clustering by including random effects for each department [28]. The inter-rater agreement between the physician reviewers regarding the determination of AEs and their preventability is described with percentage agreement and kappa statistics [29].

The statistical software IBM SPSS V.22 was used for all statistical analyses and data processing. A P-value of ≤0.05 was regarded as statistically significant.

Results

Patient level

Adverse events

Patient and AE characteristics before and after the internal audit are reported in Appendix A. After the internal audit, patients with at least one AE decreased from 36.1% (163/451 patients) to 31.3% (131/419 patients) (Table 3). Patients with at least one PAE decreased from 5.5% to 3.6%. In seven of the nine departments, the mean percentage of AEs decreased (Fig. 2). After the internal audit, the percentage AEs was decreased by 4.8%; however, it was not statistically significant (P = 0.12, 95% confidence interval [CI]: −0.11 to 0.16). The ICC was 11.7%; PAEs were decreased by 1.9% (P = 0.77, 95% CI: −0.10 to 0.08). The percentage agreement between reviewers in the determination of AEs was 72.4%, and the kappa was 0.43 (95% CI: 0.20–0.63) (Appendix B). The percentage agreement on the preventability of AEs was 92.1%, and the kappa was 0.58 (95% CI: 0.21–0.85).

Table 3

Effects of internal auditing on patient safety outcomes

	Descriptive statistics					Linear mixed-model analyses
	Before	After		Relative change		Δ t0 – t1a (SE)		95% CI		P-value
Patient level
% of patients with one AE (n)	36.1 (163)	31.3 (131)		−4.8		−0.05 (0.03)		−0.11 to 0.16		0.12
% of patients with one preventable AE (n)	5.5 (25)	3.6 (15)		−1.9		−0.01 (0.04)		−0.10 to 0.08		0.77
				Absolute change
Mean score (SD) of perceived patient safety and quality of hospital care by patients (CQI)b		+9 mo	+15 mo	+9 mo	+15 mo	+9 mo	+15 mo	+9 mo	+15 mo	+9 mo	+15 mo
Patient safety	3.36 (0.66)	3.51 (0.57)	3.61 (0.55)	0.15	0.25	0.14 (0.04)	0.25 (0.40)	0.06 to 0.22	0.17 to 0.33	0.001*	0.001*
Overall quality of hospital care	3.29 (0.43)	3.38 (0.43)	3.45 (0.41)	0.09	0.16	0.05 (0.03)	0.13 (0.03)	−0.01 to 0.10	0.07 to 0.18	0.07	0.001*
SMRc (n = 8 departments)	83	102		19		−0.20 (0.13)		−0.46 to 0.06		0.13
Professional and team level
Mean score (SD) of patient safety culture (COMPaZ)	3.39 (0.39)	3.35 (0.37)		−0.04		−0.04 (0.04)		−0.13 to 0.06		0.38
Teamwork across hospital units	2.94 (0.57)	2.90 (0.56)		−0.04		−0.03 (0.06)		−0.18 to 0.13		0.71
Teamwork within hospital units	3.71 (0.74)	3.72 (0.74)		0.01		−0.04 (0.06)		−0.17 to 0.10		0.53
Hospital handovers and transitions	3.43 (0.77)	3.37 (0.75)		−0.06		−0.08 (0.08)		−0.27 to 0.12		0.39
Frequency of event reporting	3.32 (0.74)	3.33 (0.71)		0.01		0.13 (0.06)		−0.14 to 0.17		0.84
Non-punitive response to error	3.64 (0.71)	3.58 (0.68)		−0.06		−0.07 (0.07)		−0.23 to 0.09		0.35
Openness of communication	3.46 (0.50)	3.35 (0.49)		−0.11		0.02 (0.03)		−0.06 to 0.10		0.59
Feedback and communication about error and organisational learning	3.46 (0.61)	3.50 (0.59)		0.04		0.04 (0.06)		−0.11 to 0.18		0.59
Supervisor/manager expectations and actions promoting patient safety	3.49 (0.63)	3.43 (0.64)		−0.06		−0.08 (0.05)		−0.21 to 0.04		0.17
Hospital management support for patient safety	3.39 (0.68)	3.33 (0.69)		−0.06		−0.02 (0.04)		−0.11 to 0.07		0.63
Adequate staffing	3.17 (0.66)	3.05 (0.70)		−0.12		−0.10 (0.10)		−0.35 to 0.14		0.35
Overall perceptions of safety	3.40 (0.64)	3.29 (0.64)		−0.11		−0.08 (0.05)		−0.21 to 0.05		0.20
Mean score (SD) of team climate (TCI)d	3.95 (0.36)	3.83 (0.35)		−0.12		−0.12 (0.09)		−0.34 to 0.10		0.24
Participative safety	3.84 (0.33)	3.79 (0.34)		−0.05		−0.06 (0.10)		−0.29 to 0.17		0.55
Support for innovation	3.66 (0.41)	3.54 (0.41)		−0.12		−0.13 (0.21)		−0.35 to 0.17		0.22
Vision	3.84 (0.34)	3.67 (0.33)		−0.17		−0.17 (0.18)		−0.34 to −0.01		0.045*
Task orientation	4.43 (0.48)	4.33 (0.44)		−0.10		−0.11 (0.12)		−0.40 to 0.18		0.41
Department level
% (SD) observed patient safety culture (SWRs)	92.0 (6.2)	90.5 (6.6)		−1.5		−0.01 (0.02)		−0.06 to 0.03		0.48
Medication safety	92.0 (11.2)	99.1 (3.0)		7.1		0.07 (0.03)		0.00 to 0.14		0.046*
Infection prevention	98.9 (3.7)	94.1 (6.5)		−4.8		−0.05 (0.04)		−0.12 to 0.03		0.21
Environment	85.9 (7.9)	88.5 (5.8)		2.6		0.03 (0.03)		−0.04 to 0.09		0.36
Protocols and procedures of care	90.9 (30.1)	80.0 (4.2)		−10.9		−0.11 (0.16)		−0.52 to 0.30		0.52
Information security	83.8 (15.9)	94.4 (6.5)		10.6		0.11 (0.05)		0.00 to 0.21		0.046*
Sterile medical aids	100 (0)	97.7 (7.5)		−2.3		−0.02 (0.02)		−0.73 to 0.28		0.34
Medical devices	94.1 (10.2)	88.7 (14.7)		−5.4		−0.05 (0.05)		−0.17 to 0.06		0.34
Patient identification	100 (0)	92.4 (17.2)		−7.6		−0.76 (0.05)		−0.20 to 0.05		0.20
Food safety	86.4 (17.2)	90.9 (23.1)		4.5		0.45 (0.07)		−0.12 to 0.21		0.55
Reserved procedures	80.3 (32.3)	72.7 (35.9)		−7.6		−0.76 (0.13)		−0.36 to 0.20		0.56
Overall safety	100 (0)	96.4 (12.0)		−3.6		−0.04 (0.04)		−0.12 to 0.04		0.34

SD, Standard deviation; NA, not applicable; SMR, standardized mortality rate; SWR, safety walk rounds.

*P ≤ 0.05; statistically significant results are in bold.

aThe crude model was adjusted for random effects for departments. Difference in change and corresponding interval does not necessarily reflect difference in absolute change because of inclusion of random effects and covariates in the models tested.

bThe crude model was adjusted for sex, age, education, and self-reported general and mental health.

cThe crude model was adjusted for the skewness of the data.

dThe crude model was adjusted for occupation.

Figure 2

The percentages of AEs 3 months before and 15 months after the internal audit.

Patient-reported experience measures

Respectively, 60.0% (442/737), 49.9% (468/937) and 36.6% (432/1180) of the included patients responded 3 months before and 9 and 15 months after the internal audit to the invitation to fill in the CQI. Nine and 15 months after the internal audit, these patients experienced significantly increased patient safety during their hospital stay: 0.14 (95% CI: 0.06–0.22, P = 0.001) and 0.25 (95% CI: 0.17–0.33, P = 0.001), respectively (Table 3). They also experienced a significant increase in the overall quality of hospital care 15 months after the internal audit (0.13, 95% CI: 0.07–0.18, P = 0.001)

Standardized mortality ratio

There was no significant change in the SMR after the internal audit (Table 3).

Professional and team level: patient safety culture and team climate

Before the internal audit, 408 of the 836 included healthcare providers (48.8%) responded to the COMPaZ; after the internal audit, 428 of the 882 included healthcare providers (48.5%) did. Overall, the patient safety culture in the clinical wards of the included departments did not change after the internal audit, according to the healthcare providers (−0.04, 95% CI: −0.13 to 0.06, P = 0.38) (Table 3). No dimensions changed significantly.

The response to the TCI before the internal audit was 58.8% (588/1000); after the internal audit, it was 58.7% (612/1043). According to the healthcare providers, the team climate did not change either (−0.12, 95% CI −0.34 to 0.10, P = 0.24), except ‘vision’, which decreased significantly (−0.17, 95% CI −0.34 to −0.01, P = 0.045) 15 months after the internal audit.

Department level: observed patient safety

Medication safety (0.07, 95% CI: 0.00–0.14, P = 0.046) and information security (0.11, 95% CI: 0.00–0.21, P = 0.046) in the clinical wards improved both significantly. Other aspects of patient safety culture, such as up-to-date protocols and procedures of care, patient identification and periodically examining reserved procedures (e.g. venipuncture, inserting urinary catheter), did not improve. AEs related to medication decreased by 4% after the internal audit (Appendix A).

Discussion

Our study showed that overall internal auditing was not effective in decreasing AEs and PAEs within 15 months, although we observed a reduction in both patient safety outcomes. Patients’ experiences regarding patient safety and observed patient safety on wards in terms of medication safety and information security were significantly improved. Over time, we did not observe an effect on SMR, patient safety culture and team climate, except for a significantly decreased team vision.

We know from the literature that a well-established patient safety culture and functioning patient-care team are crucial for providing safe care [19, 30]. The lack of change in patient safety culture and team climate over time may partly explain why we observed no reduction in AEs and PAEs. The fact that one dimension of team climate, namely team vision, decreased after internal auditing, supports this explanation. Due to a poorer result of the outcome of the implementation of the improvement actions than expected, the healthcare providers and managers come to realize that their vision on how to improve patient safety was less realistic and achievable than they initially thought. Benning et al. [4] also found no changes in patient safety culture or team climate after an organisational patient safety intervention on frontline care processes in clinical specialties.

In the present study, we measured patient safety outcomes 15 months after an internal audit. However, change in patient safety culture takes time [31]. Therefore, this follow-up timeframe was possibly too short for capturing the effect of internal auditing on patient safety. Another potential explanation is, that it may have been difficult to translate audit feedback of patient safety deficiencies into effective improvement actions. Well-known factors such as poor planning of improvement actions, limitations in expertize, insufficient staff capacity, lack of ownership and management support result in a gap between patient safety investigation and improving patient safety [12]. To gain insight into which factors may have influenced our study results, we carried out a process evaluation and the results are reported in a separate paper, called part 2 [32]. Besides, patient safety outcomes such as (P)AEs and safety culture are very difficult measures to improve over time. Also, other deliberate or targeted interventions produce no or small patient safety changes [1, 4, 5]. Lastly, the low statistical power of our study could mask some existing true effects of internal auditing [33]. Detecting a reduction of 30% in the primary outcome, AEs, with 80% statistical power would require 40–60 departments and 50–100 records per department and measurement, which was not feasible for practical reasons [8]. Therefore, we analyzed 50 patient records per department and measurement for (P)AEs and calculated the corresponding precision of 7%.

Our study has several strengths, including the use of a predefined protocol for evaluating the effects of patient safety auditing. We used a small group of independent record review experts (n = 5) who have more than 10 years’ experience with medical record review [8]. We used validated instruments (CQI, COMPaZ and TCI) to measure PREs, safety culture and team climate. Some limitations concerning this study must be addressed. First, our study design was a before-and-after design without a control group. It is therefore difficult to be confident that the changes can be attributed to the intervention. Nevertheless, before-and-after designs are useful for studies that are part of local quality improvement projects such as audits, PDCA cycles, and action research [34]. Second, the record review method has its limitations. The reviewers had to rely on information available in the records. Only information registered by the hospital staff could be used for assessing the presence of an AE. The inter-rater agreement regarding AE assessment was moderate, and is an issue in other retrospective review studies [1, 2, 27]. Finally, the patients’ CQI response rate 15 months after the internal audit was low compared with that 3 months before the internal audit, as there were more web-based questionnaires than written questionnaires. Compared to written questionnaires, web-based questionnaires result in a lower response, with the potential for non-response bias [35]. We accounted for the representativeness of the CQI data by case-mix adjustment of relevant patient characteristics in the mixed-model analysis of the CQI [23].

Click here for additional data file.

Click here for additional data file.