Evaluation of the organisation and effectiveness of internal audits to govern patient safety in hospitals: a mixed-methods study.

OBJECTIVES: Hospital boards are legally responsible for safe healthcare. They need tools to assist them in their task of governing patient safety. Almost every Dutch hospital performs internal audits, but the effectiveness of these audits for hospital governance has never been evaluated. The aim of this study is to evaluate the organisation of internal audits and their effectiveness for hospitals boards to govern patient safety. DESIGN AND
SETTING: A mixed-methods study consisting of a questionnaire regarding the organisation of internal audits among all Dutch hospitals (n=89) and interviews with stakeholders regarding the audit process and experienced effectiveness of audits within six hospitals.
RESULTS: Response rate of the questionnaire was 76% and 43 interviews were held. In every responding hospital, the internal audits followed the plan-do-check-act cycle. Every hospital used interviews, document analysis and site visits as input for the internal audit. Boards stated that effective aspects of internal audits were their multidisciplinary scope, their structured and in-depth approach, the usability to monitor improvement activities and to change hospital policy and the fact that results were used in meetings with staff and boards of supervisors. The qualitative methods (interviews and site visits) used in internal audits enable the identification of soft signals such as unsafe culture or communication and collaboration problems. Reported disadvantages were the low frequency of internal audits and the absence of soft signals in the actual audit reports.
CONCLUSION: This study shows that internal audits are regarded as effective for patient safety governance, as they help boards to identify patient safety problems, proactively steer patient safety and inform boards of supervisors on the status of patient safety. The description of the Dutch internal audits makes these audits replicable to other healthcare organisations in different settings, enabling hospital boards to complement their systems to govern patient safety. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

This is the first study that evaluates the organisation and the effectiveness of internal audits to govern patient safety in hospitals.

We performed a mixed-methods study consisting of a questionnaire sent to all Dutch hospitals and interviews with stakeholders in the governance and audit process of six Dutch hospitals.

The use of qualitative data collection enabled us to gain insight into the experiences of boards with internal audits.

As we studied internal audits in Dutch hospitals, generalisation to other countries or healthcare settings might be limited.

Introduction

Patient safety should be the top priority of every hospital board.1–3 Hospital boards are legally accountable for the quality and safety of the delivered care in their hospitals.4 5 However, while the need for board safety oversight has been growing,6–8 healthcare is still often unsafe and boards experience difficulties in overseeing safety risks.9–12 In order to fulfil their governance role, hospitals boards need methods and tools that provide monitoring information to mitigate or prevent adverse events.13–15

There are several sources for gathering information that helps boards with the governance of patient safety, and information from internal audits might be one of them. The internal audit is an ‘objective assurance and consulting system for detecting patients’ risks of adverse events early’, which ‘should encourage the continuous improvement of patient safety’.16 It is a systematic evaluation of the quality system of a hospital which aims to improve patient safety by measuring performance of healthcare providers and preconditions for safe care and comparing these outcomes with (national) standards and guidelines. The measurements are performed by an audit team existing of internal peers (ie, employees of a hospital who audit colleagues of other departments). The method was implemented in the 1990s to measure whether organisational preconditions for safe care are in place and to induce improvements when safety problems are detected. Internal audits are initiated by hospital boards and implemented top-down.

Several studies regarding the effectiveness of clinical audits on professional practice have been performed.17 The found effects are small and differ per study. This can be partially explained by the differences in study population, form and content of studied audits and used research methods and outcomes.18 Knowledge regarding the effectiveness of internal audits for internal patient safety governance by hospital boards is, however, scarce and therefore subject of this study.

The reason that almost all Dutch hospitals use internal audits for governance purposes is a combination of the 1996 Care Institutions Quality Act and the constitution of the Netherlands Institute for Accreditation in Healthcare (NIAZ). Hospitals are obliged by the Care Institutions Quality Act to have a quality management system in place, including the assurance that quality activities are undertaken.19 Since the 1990s, many hospitals are using the quality assurance standards of NIAZ.19 In order to be accredited by this institute and to give the assurance of safe care to third parties (eg, healthcare consumers and healthcare insurers), an internal audit system should be in place.20 21 External accreditation parties such as NIAZ have their own audits (ie, external audits that they perform to see whether a hospital is ready for external accreditation). This study does not focus on these external audits, but on the internal audits that are performed by employees from a hospital itself.

Our study focuses on governance within a hospital from a board of directors’ point of view: the need to oversee and to steer patient safety (deriving information from the work floor) and the need to account for patient safety (sending information towards the board of supervisors). We are interested in whether the internal audit assists the board of directors of hospitals in this task. Figure 1 shows examples of tools to govern patient safety, the stakeholders in Dutch hospital governance and the position of internal audits in it (see figure 1).

Figure 1

Positioning of internal audit in governance of Dutch hospitals. Framework is based on two studies: the ‘Deepening our Understanding of Quality Improvement in Europe’ (DUQuE) project24 and the ‘Quality and Safety in Europe by Research’ (QUASER) study.25

Almost every hospital in the Netherlands uses internal audits. However, research regarding the effectiveness of internal audits for boards to govern patient safety is lacking. As internal audits are widely used in hospitals, we wondered whether and how the information coming from internal audits is effective for the governance of patient safety (ie, has a place in hospital governance). Our study has two aims. First, to describe the internal audits in Dutch hospitals, so that, if regarded as being effective, this audit system is replicable to other countries or different healthcare organisations. Second, to describe the views of hospital boards regarding the effectiveness of internal audits to assist them in their task of governing patient safety. We aimed to answer the following questions:

How are internal audits organised in Dutch hospitals?

Are internal audits regarded as effective for the governance of patient safety by hospital boards?

Methods

Study design and setting

We performed a mixed-methods study on the organisation of and experiences with the effectiveness of internal audits in the Netherlands, consisting of a questionnaire survey and individual interviews.

The questionnaire was sent to all Dutch hospitals (n=89) and interviews took place in six hospitals, which were selected among the 89 hospitals. Selection was based on various criteria (see appendix 1). The six hospitals represented both the different types of hospitals in the Netherlands and the different aspects of internal audits. The participating hospitals were located across the country and ranged in size from 536 beds up to 1003 beds. All six hospitals were accredited or were in the process of being accredited, for example, by the NIAZ or the Joint Commission International (JCI).

By ‘hospital boards’, we mean a board of directors. Hospital boards across all participating hospitals (questionnaire and interviews) were structured according to the Care-wide Governance Code.22 In the Netherlands, a board of directors and a board of supervisors represent two independent bodies; a board of directors is responsible for patient safety governance and a board of supervisors supervises a board of directors. A board of directors is accountable for the quality and safety of care to a board of supervisors and external parties such as the Dutch Healthcare Inspectorate (which promotes public health and is a part of government oversight of public health).23 The pressure to have a patient safety governance structure in place comes from the Ministry of Health (see figure 1).24 25

Our study focused on internal audits; ‘audits organised at hospital level and directed at several levels of patient care, including policy, patient safety culture, guideline adherence of professionals and outcomes at the patient level16 ’, looking at every department of a hospital, initiated by the board of directors and implemented top-down. We did not focus on corporate audits (mainly focused on financial aspects) or clinical audits (initiated by healthcare professionals and implemented bottom-up).

Internal audits have a broader scope than patient safety alone. Information coming from audits is used for various purposes: (1) for continuous quality improvement; (2) to control, adjust and secure quality improvement processes and (3) to account for the quality and safety of provided care.26 Our research focuses on patient safety. Safety of care, no patient harm, is one of the most important domain of quality of care.27 Hospital boards in the Netherlands are legally responsible for safe healthcare and over the past few years, (critical) incidents have become ‘public events’ for which boards are held accountable.8 This led to the necessity for board safety oversight and, subsequentially, the focus of our research.

Participants

The questionnaire was sent to the chairmen of the boards of directors of every Dutch hospital (n=89), with the option to forward the questionnaire to a person responsible for internal audits at operational level.

The targeted number of interview partners was six members of boards of directors, six members of boards of supervisors, six quality and safety directors, 12 quality officers (including auditors) and 12 heads of departments or clinical managers (auditees). Participants for the interviews were selected based on purposive sampling to ensure diversity (eg, experience with audits, auditing and type of job) and convenience sampling (for availability purposes).28

Data collection

The research question regarding the organisation of the internal audit was studied with both the questionnaire and the interviews. Issues from the questionnaire were used as input for the interviews in order to gain in-depth information on this subject. The research question regarding the effectiveness of the internal audit was studied with interviews only.

Questionnaire

An invitation to participate in the questionnaire to study the organisation and content of internal audits was sent in 2012 by email. The email included the link to the online survey, the purpose of the study and a statement that anonymous and confidential handling of data was ensured. Informed consent was implied by completing and sending in the questionnaire. A reminder was sent after 2 weeks. The questionnaire consisted of multiple choice questions; six general questions and eight questions regarding content and organisation of the audit (see appendix 2). The questionnaire was developed based on meetings with experts on auditing (n=3) and brainstorming sessions (n=4) with the research team. The questionnaire was pilot tested by target participants (n=3) and adapted accordingly.

Interviews

Interviews took place between May of 2012 and November of 2014. All interviews were audio-recorded with the participants’ consent and transcribed verbatim according to a standardised format. Data collection and analyses of interviews were performed according to the ‘Consolidated criteria for reporting qualitative studies’ (COREQ)29 (appendix 3) and based on thematic analysis. Interviews were in-depth, face-to-face interviews. All interviews were conducted by experienced interviewers (SCvG and a postdoc researcher) and were guided by a topic guide. The topic guide was developed based on results from the questionnaire (regarding the organisation and content of internal audit only) and brainstorming sessions with the research team and was adapted after each interview. Topics for guiding the interviews included the following themes (see appendix 4):

Organisation and content of internal audit.

Effectiveness of internal audit information for boards:

Use of internal audit for identification of safety risks.

Use of audit information to steer patient safety.

Use of internal audit to account for patient safety towards the board of supervisors.

Questions regarding the content and organisation of the internal audits were addressed to all interviewees. Questions regarding the regarded effectiveness of internal audits for hospital boards to govern patient safety were addressed to the boards of directors, boards of supervisors, quality and safety directors and heads of department or clinical managers. On each hospital site, interviews were held until saturation was reached.30

Data analysis

The questionnaire data were analysed using IBM SPSS Statistics version 20. Transcripts of the interviews were coded using Atlas.ti software version 7.0 (ATLAS.ti Scientific Software Development Company, GmbH, Berlin, Germany). The transcriptions of the interviews were analysed using thematic analysis.30 Two researchers (SCvG and MZ) independently analysed and discussed the content of the first (n=3) interviews, which formed the basis of a coding framework. One researcher (SCvG) analysed the rest of the interviews by applying the coding framework and modifying it through an inductive and iterative process. Codes that related to the same phenomenon were grouped into categories and, finally, themes were identified. Differences were resolved by consensus.

Results

Response and characteristics of respondents

Of the 89 questionnaires sent, 69 were returned (76%). In one hospital, no internal audits were carried out. One questionnaire was filled in by a chairman of the board of directors. The other questionnaires were filled in by employees related to the quality departments. The response rate varied per type of hospital: 88% for university hospitals (n=7/8), 82% for tertiary teaching hospitals (n=23/28) and 70% for general hospitals (n=38/54).

In total, 43 interviews within six hospitals were performed. In two cases, the requests for interviews were not granted because of time constraints. Five members of boards of directors were interviewed, as were five members of boards of supervisors, seven quality and safety directors, 14 quality officers (including auditors) and 12 medical specialists or clinical managers (auditees) (see table 1).

Table 1

Interview participants and their characteristics (n=43)

	n	%
Hospital type
University hospital	16	37
Tertiary medical teaching hospital	15	35
General hospital	12	28
Function title
Member of the board of directors	5	12
Member of the board of supervisors	5	12
Quality and safety directors	7	16
Quality officers	14	32
Head of department or clinical manager (auditees)	12	28
Gender
Female	24	56
Male	19	44
Work experience in the current function (years)
1–5	26	60
6–10	12	28
11–15	5	12

Description of the organisation of internal audits in Dutch hospitals

Internal audit cycle and responsibilities of stakeholders

In all six hospitals, the internal audits were performed according to the plan–do–check–act cycle. Figure 2 shows the cycle of the internal audits (see figure 2). In all studied hospitals (n=6), the boards of directors initiated the internal audits and delegated the execution to committees or departments. These departments or committees were responsible for the entire organisation of the internal audit cycles, including the appointment of the audit teams. The audit teams performed the internal audits and were recruited among employees from the organisations. In some hospitals, these auditors were volunteers, while in other hospitals being an auditor was (part of) an employee’s job and they were paid for it.

Figure 2

Internal audit cycle for governance purposes.

Internal audits focused on hospital departments and in some cases also on healthcare pathways and/or patient safety themes. An audit team prepared the audit, which included the analysis of policy documents, medical record reviews and self-evaluation forms filled in by the departments. Preparations also involved, for example, looking at outcomes of earlier performed audits (see table in appendix 5). In audit team meetings, the outcomes of the preparations were discussed and the focus of the audit (eg, which specific safety risks would receive extra attention and which topics were included in the audit) was determined. Audit topics ranged from the existence of soap dispensers to interrelation topics such as department culture, communication and cooperation. During an audit day, the audit team visited the departments or involved departments in cases of auditing healthcare processes/pathways or patient safety themes. Subsequently, the audit team wrote a report in which the impressions of the departments were reflected. With this report, the audit teams provided feedback regarding improvements that must or could be made to increase patient safety (recommendations). Audit results were fed back to the boards of directors for governance purposes and fed back to the audited departments. Department heads were obligated to make improvement plans. Follow-up of the audit results and recommendations was the responsibility of department heads and was monitored by the boards of directors or delegated to committees. This audit cycle was repeated periodically.

Organisation and content of internal audit

Internal audits were performed once every 4 years in 66% of the hospitals, once every 3 years in 13% of the hospitals, once every 2 years in 10% of the hospitals and once every year in 9% of the hospitals (see appendix 5). The time frame of an internal audit ranged from 1 month (9%) to 7 months (2%), with a time frame of 2 months being most common (27%). Members of the audit teams were nurses (present in audit teams in 96% of the hospitals), management employees (present in audit teams in 84% of the hospitals), allied healthcare professionals (present in audit teams in 75% of the hospitals) and medical specialists (present in audit teams in 68% of the hospitals). One hospital (1%) had 5–10 auditors in total, while 65% of the hospitals had more than 20 auditors in total. In 81% of the hospitals, auditors received training, and in 74% of the hospitals auditors were evaluated. Of the 68 hospitals, six did not train nor evaluate their auditors (9%). Hospitals used standards of accreditation institutes (97%), standards set by law (eg, national safety themes, including adherence to sepsis bundles, protocols for medication reconciliation at hospital admission and hospital discharge) (66%), the hospital itself (eg, Team Climate Inventory to measure the improvement climate of teams of healthcare providers) (32%) and the profession (eg, guidelines from medical associations) (27%) for auditing. As inputs for the internal audit outcomes, the following were used: document analysis by audit team (100% of the hospitals), interviews by audit team (100% of the hospitals), site visits by audit team (100% of the hospitals), self-evaluation by audited department (59% of the hospitals), ad hoc measures by audit team (34%) and other measures (37%). These other measures were as follows:

Outcomes of other audits when present, such as audits by external experts, external audits for accreditation and audits initiated by the professional association of medical specialties (visitations).

Outcomes of surveys among employees of partner departments (such as surveys among an orthopaedic department when a radiology department is being audited).

Outcomes of medical record reviews.

Feedback of audit results

The ways in which audit results were fed back to the boards of directors differed per hospital. Three options have been found:

The board of directors receives a report on the headlines deriving from internal audits (aggregated results; found in 50% of the hospitals).

The board of directors receives letter with recommendations based on audit results and has the option to ask audited department for the entire audit report (found in 17% of the hospitals).

The board of directors receives entire audit report (found in 50% of the hospitals).

In one hospital, multiple options have been found. In that hospital, the board received a report on the headlines deriving from all internal audits and the entire report of the executed audits on supportive departments.

In interviews, members of the boards of directors of all hospitals stated that it is the responsibility of the departments to implement improvement actions, except in the following situations: (1) when patient safety is immediately threatened—in this case, a board of directors uses the internal audit as a ‘forced improvement’ instrument and (2) when improvements cannot be made without support from a board of directors, for example, when equipment is out of date and cannot be replaced without consent of a board of directors.

Experiences with effectiveness of internal audits

Use of internal audits for identification of safety problems

Board members perceived internal audits as effective for the identification of safety problems for three reasons: (1) broad, multidisciplinary scope; (2) soft signals and (3) in-depth approach (see table 2 for illustrative quotes).

Table 2

Themes, categories and quotes relating to effectiveness of internal audit

Theme	Category	Quotes
Use of the internal audit for identification of safety risks	Broad, multidisciplinary scope	‘Well, an internal audit is department-wide and multidisciplinary, you talk with professionals of that department, but also with its customers. Like, if there’s an audit at radiology, you talk to specialists of other departments that use the services of radiology as well. With tracers, you don’t have that complete overview.’ ‘And the internal audit contributes to an overall view of a department, of a group of people who are active within a certain discipline: clinicians, nurses, allied healthcare professionals… and yes, you get insight into the department on an aggregated level.’
	Soft signals	‘[With the internal audit] you can get a global impression regarding the actual performance, so to say, and cooperation as well (…). The question could be: ‘how is everything going,’ and they would say: ‘perfect, we are doing the best we can’. Well, show me!’ ‘Well, I think that the soft signals… Partially they derive from the internal audit (…) but it is just very complicated, you know. I think an internal audit should begin with the question: ‘Why do you like working here?’ That is not something you can write down in three sentences in the audit report. It is as soft as you can get, but it is very significant for how people are feeling in their job.’
	In-depth approach	‘So it shows where the need for improvements lie and what you should prioritize… It prioritizes in the way, like, what is going right and what is going wrong?’ ‘Especially when auditors don’t just score, but ask questions, you get more information, like ‘why is it going wrong?’” ‘Maybe it’s even more important to be open and susceptible to signals from within the organization. We talk to chairmen of staff and divisions very often. You cannot wait for the internal audit to take place in a few years to feel safe regarding the functioning of a department.’
Use of audit information to steer patient safety	Monitoring	‘And in our quarterly report, in our four-times-a-year cycle, we ask every unit to report on the audits that have been performed; what were the results, what did you encounter and which actions did you think of, and these actions, are they implemented and are they leading towards results (…).” ‘For us, as board of directors, it’s about knowing what the conclusion regarding the department is, in the context of general functioning, leadership and yes, quality and safety. And that is one of the sources we need to, so to say, feel secure in how the organization is doing.’
	Incentive for change	‘Something in which we intervened immediately was the double check on medication. That was a moment in which we said: this is unacceptable. These answers are unacceptable. So I went there, and the medical directors as well, and we said, this is out of the question. This has to change by tomorrow.’ ‘We’ve also looked at what is the reason that that happens; don’t we need to alter the procedure? So it is about steering, saying: ‘this is unacceptable’, but it is also a moment of thinking: ‘did we organize it correctly?’”
Use of audit information to account for patient safety	Use of internal audit information in regular meetings with the board of supervisors	‘Yes, to inform. By the way, our board of supervisors is very on top of it, they will ask: what did you do about it? So you need to have that answer as well.’
	Use of audit information to inform the board of supervisors regarding critical incidents	‘And if we say: ‘This is critical, these people have to be informed before the report is finished’, then I will call them and they will come over.’ ‘It’s not good for internal monitoring because that will cause a mix-up between the responsibilities of the board of directors and the board of supervisors. Negative results of an audit should be solved in the management line of the hospital. Having said that, the existence of an internal audit system, that they are performed accurately, that there are reports on the results and the system as a whole, helps us to trust the board of directors.’

Interviewees stressed that with internal audits every department of a hospital, including supportive departments, is evaluated periodically. Furthermore, an internal audit has a multidisciplinary focus (eg, involving the opinions of other departments regarding the audited department and not only focusing on clinicians or nurses, but all employees both in an audit team and as auditees). Board members noted that this broad scope of an internal audit provided a complete overview of the performance of all departments in an entire organisation.

According to board members, the use of qualitative methods of gathering information (eg, interviews and observations) makes an internal audit a suitable instrument not just to establish that things are going wrong, but most of all reveal why these things happen. Board members stated that as a result, an internal audit was able to provide information regarding the soft side of an organisation, such as cooperation and communication problems. Nonetheless, they noted these soft signals are not easily translated into facts that can be reported in an audit report.

Board members indicated that an internal audit is an instrument with a very structured and in-depth approach. The quality and safety of an entire department are being evaluated, from cleaning to medical treatment and from medical chart reviews and policy documents to interviews with employees, after thorough preparations of both an audit team and an audited department. Board members stated that this reveals patient safety problems in a structured way, and because of the fact that improvement actions are suggested, audits help boards prioritise what should be done to improve patient safety. Board members indicated that an internal audit is especially suited for generating patient safety information in a planned and prepared manner, but mentioned that because of the low frequency of internal audits (because of the time-consuming, in-depth approach) there is a lack of real-time information.

Use of audit information to steer quality and safety

Two categories emerged within this theme: (1) monitoring and (2) incentive for change.

Board members mentioned that embedding internal audit results in a planning and control cycle gives them the opportunity to have a dialogue regarding the status of quality improvements made by departments. They noted that monitoring and frequently discussing audit results and improvement plans contribute to the feeling of being in control. A caveat mentioned by one board member regarding being in control with the internal audit is that, as with every instrument, critical incidents can still happen.

Board members mentioned that audit results can be an incentive for them to adjust hospital policy and culture. Different interviewees stated that internal audit results were a reason for them to make changes happen as soon as possible, whether these were adjustments needed to be made by the board itself or by departments. Boards used internal audits to start a conversation with staff and as an incentive to visit departments themselves.

Use of internal audits to account for patient safety

Two categories derived from this theme: (1) the use of internal audit information in regular meetings with boards of supervisors and (2) the use of audit information to inform boards of supervisors regarding critical incidents.

Board members stated that an internal audit is one of the information sources used to inform a board of supervisors about the status of patient safety. It is used on a regular basis, as part of the regular meetings between boards of directors and boards of supervisors, on an aggregated level. Interviewees stressed the latter: an internal audit is not discussed in detail, as it is not the job of a board of supervisors to monitor departments in detail. Members of boards of directors stated that instead, they use the headlines of the performed audits to show boards of supervisors whether a hospital is able to learn and improve.

There is an exception when discussing an internal audit on an aggregated level. Members of different boards mentioned that when critical incidents derived from internal audits, boards of supervisors are informed of these incidents.

Discussion

The aim of this study was to describe the organisation of internal audits in Dutch hospitals and to evaluate the effectiveness of these audits for boards to govern patient safety. Our quantitative and qualitative findings revealed that there were similarities in how internal audits were organised in the hospitals. The audit cycle, following the plan–do–check–act cycle, the methods used to gather information (eg, interviews, site visits and document analysis) and the responsibilities of the boards of directors and the committees responsible for audits were in general the same in all hospitals throughout the Netherlands. Aspects that differed among hospitals were related to feasibility and included the number of auditors and disciplines such as medical specialists present in an audit team, the duration of an internal audit and the amount of methods used to get insight into the quality and safety of departments.

Interviewed hospital board members stated that internal audits help them to identify safety problems and to steer patient safety. Internal audits provide a complete overview of departments, prioritise safety problems, are an incentive to adjust policy and are used to monitor safety improvements. A study regarding governance activities of Australian hospital boards showed four tools that are similar to the effective aspects of an internal audit, namely ‘shaping culture’, ‘measuring progress’, ‘setting priorities’ and ‘ensuring accountability’.31 The use of qualitative methods (eg, interviews and observations) makes an internal audit unique compared with quantitative instruments (eg, quality indicators) that boards have for governance. However, since internal audits are highly structured and standardised, their frequency is low which results in limited real-time information. Another disadvantage of an internal audit’s formal character is that soft signals deriving from the qualitative methods are difficult to include in the audit report—even though board members mentioned that these outcomes especially say a lot about the quality and safety of care. These outcomes give insight into problems regarding patient safety culture, communication and collaboration.

The attention to effective board oversight and tools to assist boards in this task is ever growing.2 6 12 31–33 This study contributes to research in this field by evaluating an instrument that is already used in almost every hospital in the Netherlands. Internal audits in Dutch hospitals have existed since the 1990s and are initiated by hospital boards. However, to our knowledge, the Dutch internal auditing process has never been evaluated before and there is little-to-no literature on the evaluation of the effectiveness of internal audits for hospital governance.19 34 35 Our study indicates that an internal audit might be a promising instrument for hospital boards. Internal audits provide a complete, multidisciplinary and periodic overview of quality and safety problems, their underlying causes and needed improvement actions. Research regarding dashboards or scorecards using measurements such as the Hospital Standardised Mortality Ratio (HSMR) or complications and lengths of stay showed that a disadvantage of these measurements is that they do not always provide relevant information regarding the underlying problem(s) or causes related to how to improve quality and safety.12 36–39 Indicators such as HSMR do not go further than revealing that there is a problem, while an internal audit is able to reveal why there is a problem because of the use of qualitative methods.40 41 Our finding that the soft signals deriving from these qualitative methods are important to gaining insight into the underlying causes of quality and safety problems has been found in other studies regarding board oversight as well.36 42 43 Research regarding patient safety is focusing more and more on whether quality management systems (preconditions and performance) are able to give true insight into patient safety.44 In this light, we feel it is important to keep developing internal audits to include cooperation, culture and communication so that this instrument is not just a tick box activity that looks at preconditions only.

A strength of this study is that we used a mixed-methods approach. The questionnaire enabled us to get a complete overview of the organisation of internal audits in all Dutch hospitals and the qualitative measures provided us with in-depth information on the experiences of boards regarded to the effectiveness of internal audits to govern patient safety. Another strength of this study is the high response rate to both the questionnaire and the interviews; there were only two interviewees who did not grant the request for an interview, resulting in a high internal validity of the results. Moreover, we interviewed every stakeholder in the audit process, including members of boards of directors and boards of supervisors, even though this a group that is not easy to involve (in research).

A limitation of our study is that we studied internal audits only in Dutch hospitals. Therefore, generalisation to other countries or healthcare settings might be limited. We know that internal audits exist in hospitals in other countries; however, we have not found literature in which the use of internal audits in these countries has been described or evaluated19 34 and therefore we could not make comparisons between Dutch internal audits and internal audits in other countries. Another limitation is the number of members of boards of directors that we interviewed for information on their experiences with internal audits. Most of the information on the regarded effectiveness of internal audits came from the interviews with members of boards of directors. Because of time constraints and the time-consuming qualitative methods we used, we could only perform an in-depth study in six hospitals. However, we have reached saturation and found substantial consistency in experiences of boards with internal audits. This led us to believe that their experiences are representative for other hospital boards. Moreover, we were able to use the interviews with quality and safety directors and heads of departments or clinical managers to validate the boards’ experiences. Finally, effectiveness in this study has not been established in terms of ‘hard numbers’ like changes in healthcare outcomes. In this study, we were interested in perceptions of effectiveness to govern patient safety by hospital boards (qualitative research is preferred to explore experiences in depth30).

Conclusion

This is the first study in which the organisation and the perceived effectiveness of internal audits to govern patient safety in hospitals are evaluated. Our findings showed that internal audits were regarded as effective for the governance of patient safety, as they help boards to identify patient safety problems, steer patient safety and account for patient safety. Internal audits provide boards with structured, standardised, formal and periodic overviews of quality and safety problems and underlying causes in all departments in a hospital, helping boards prioritise improvement actions and giving them a sense of being in control. Furthermore, the use of qualitative methods to identify soft signals makes an internal audit a unique instrument in the entire spectrum of governance strategies for boards. Hospital boards can use the description of Dutch internal audits given in this paper to complement their systems to govern patient safety.