Registration of surgical adverse outcomes: a reliability study in a university hospital.

OBJECTIVE: Accurate registration of adverse surgical outcomes is essential to detect areas for improvement of surgical care quality. One reason for inaccurate adverse outcome registration may be the method to collect these outcomes. The authors compared the completeness of the national complication registry database (LHCR) as used in our hospital with relevant information from other available resources.
DESIGN: Retrospective reliability analysis.
SETTING: University hospital. PARTICIPANTS: From the 3252 patients admitted to the surgical wards in 2010, the authors randomly selected a cohort of 180 cases, oversampling those with adverse outcomes. The LHCR contains adverse outcomes as reported during morning hand-offs or in discharge letters. The authors checked if the number and severity of adverse outcomes recorded in the LHCR agreed with those reported in morning hand-offs, discharge letters and medical and nursing files.
RESULTS: In 135 of 180 patients, all resources could be retrieved completely. Fourteen per cent of the patients with adverse outcomes were not recorded in the LHCR. Missing adverse outcomes were all reversible without the need for (re)operation, for example, postoperative pain, delirium or urinary tract complications. Only 38% of these adverse outcomes were reported in the morning hand-offs and discharge letters but were best reported in the medical and nursing files.
CONCLUSIONS: Registration of surgical adverse outcomes appears largely depending on the reliability of the underlying sources. For a more complete adverse outcome registration, the authors advocate a better hand-off and additional consultation of the patient's dossier. This extra effort allows for improvement actions to eventually avoid 'mild' adverse outcomes patients perceive as important and undesirable.

Background

Of the patients admitted to a clinical department of surgery, approximately 10% is at risk of having a treatment-related complication and for some extensive gastrointestinal procedures even up to 50%.1 A substantial part of these complications is preventable and thus epitomises suboptimal care.2 Accurate and routine registration of these adverse outcomes is an important starting point from which to take action,3–5 in order to reduce or even prevent these events and lower hospital mortality due to diminishing flaws in the care system.6 Hence, professional societies and governmental institutions have urged to accurately record postoperative complications and to use this as a quality indicator. In the Netherlands, the Dutch Society of Surgeons already introduced a national surgical complication registry (LHCR) for this purpose in 2003.7 8

However, information is needed on the performance of hospitals' adverse outcome reporting systems.9 Inaccurate registration and thus under-reporting of adverse outcomes, as shown in previous studies,10 11 seems to be rewarded with an erroneously high score for quality of care.

A reason for an inaccurate registration of adverse outcomes could be the method chosen to collect and record these outcomes. The events entered into the registration database may be as complete as the resources from which these events are drawn. These resources can be daily verbal hand-offs, regular (multidisciplinary) meetings, medical and nursing dossiers or the discharge letter. A previous comparison between daily reported adverse outcomes with those documented in medical dossiers showed considerable discrepancy.11 Hence, even a uniform structural complication registration may have flaws to be improved. However, the effort to achieve a (nearly) complete registration should be weighed against its surplus value.

The aim of this study was to assess the accuracy of the surgical complication registration database we are using routinely and a comparison with the source documents in order to detect areas for improvement of the adverse outcome registration in clinical surgical care.

Methods

Patients

This survey was undertaken in the Department of Surgery of a tertiary referral university hospital in Amsterdam. From the admissions to any of the surgical wards during the year 2010, we randomly selected a sample of 180 patients (5.5%) from the LHCR database by means of a random number generator, while ensuring that at least half of the patients had suffered at least one adverse outcome according to the LHCR information. This was achieved by sampling half of the patients from the LHCR after selecting those in whom at least one adverse outcome had been recorded. Thus, we ensured a sufficient number of admissions with adverse outcomes to analyse. Patients admitted more than once during that year were included only once. We excluded patients whose resources could not be retrieved completely. The resulting sample was considered valid because we compared the various resources rather than the true incidence of adverse outcomes.

The definition of a surgical adverse outcome used in this study was ‘an unintended and unwanted outcome or state occurring during or following medical care that is so harmful to the patients' health that it requires (adjustment of) treatment or leads to permanent damage’, according to the Dutch Society of Surgery.12 These could include adverse outcomes due to medical management errors,13 as defined in the WHO reporting guidelines, but the recording of events took place before a conclusion regarding its causality (ie, medical management error or disease complication) could be given. The definition, its interpretation and the method of registration did not change during the study period.

Patients without adverse outcomes according to the LHCR were used to check whether the absence of events was in agreement with the other resources. The patient set with adverse outcomes was used to check whether the events as recorded in the LHCR were complete when compared with the other resources.

Resources

For each patient included, we retrieved and studied the medical and nursing files, the discharge letters relevant to that admission period, the documented morning hand-offs and the complication database (LHCR). Adverse outcomes entered into the LHCR were derived from the daily surgical morning hand-offs and the discharge letters. During these hand-offs, every discharged patient was reported. Adverse outcomes documented were those reported by the surgical residents or attending surgeons.12 The discharge letters were screened to find any additional events. The content of the discharge letters used during the study period was predefined in a local protocol, in which the reporting of adverse outcomes that had occurred during the patient's admission was compulsory.

As reference standard for the true number and type of adverse outcomes occurring during the hospital admission period of each patient, we used the combination of all resources consulted, that is, LHCR, morning hand-off, discharge letter, medical file and nursing file. The discharge letter, medical and nursing files were judged separately within the patient's dossier as they were being kept separately and produced by different care givers. At the time of the study, the medical and nursing files were not yet digitalised but contained daily reports of the patient's condition and well-being.

Study procedure

From each of the resources, except the morning hand-offs, two investigators independently extracted the documented adverse outcomes that had occurred in the selected patients and entered these in a database. In case of uncertainties interpreting the texts of the resources, the investigators consulted each other or their supervisors.

The various types of adverse outcomes were first categorised based on the national classification as used by the Dutch Surgical Society. Because these categories in our sample were too fragmented, we regrouped the events by similarity (type) and number of appearance (box 1). The grading of the severity of each event was based on the classification of Clavien et al14 and was divided into four classes: (1) temporary health disadvantage recovering without (re)operation, for example, wound infection; (2) recovery after (re)operation, for example, anastomotic leakage; (3) (probably) permanent damage or function loss, for example, stroke; and (4) death during admission. In retrospect, we also categorised recorded events that had no adverse health effects, for example, a cancelled operation, as ‘class 0’.

1. Abscess

2. Surgical procedure cancelled

3. Cardiac complications

Atrial or ventricular tachycardia

Brady/tachycardia

Asthma of cardiac origin

Myocardial infarction

Heart failure

Arrhythmias

4. Pneumonia

5. Bleeding

Aneurysm

Haematoma

Dissection

6. Shock

Haemodynamic instability

7. Anastomotic dehiscence

8. Miscellaneous leakages

Chyle

Gall

Wound

9. Pressure ulcer

10. Delirium

11. Electrolyte derailment

Anaemia

Hyperglycaemia

International Normalised Ratio derailment

12. Gut complications

Gastroparesis

Ileus

Derailed stoma output

Ischaemia of sigmoid

13. Fluid collections

Seroma

14. Pain

Correction of epidural analgesia

15. Pulmonary complications

Pneumothorax

Respiratory insufficiency

Atelectasis

Respiratory depression

16. Overinfusion

17. Wound or fascia dehiscence

18. Thrombosis

19. (Wound) Infection

Sepsis

Poor wound healing

Wound infection

20. Bladder complications

Retention

Urinary tract infection

Urethritis

21. Fistula

22. Vascular complications

Phlebitis

Shunt occlusion

Cellulitis

23. Cerebral complications

Cerebrovascular accident

Infarction

Neuropraxia

Neural compression

24. Other complications

Kidney infarction

Allergy

Ascites

Contractures

Disturbed liver function

Paresis

Wrong K-wire removed

Secondary dislocation

Rhabdomyolysis

Addison's crisis

Hernia

Temporary hoarseness

Small intestinal perforation

Data analysis

Data were transferred from the various resources into Excel 2003 (Microsoft Corp., Seattle, Washington, USA) for further analysis. Descriptive statistics were expressed as means including SDs or medians with IQRs, whenever appropriate. Agreements between the adverse outcomes recorded in the LHCR and in other resources were expressed as percentages. Similarly, we calculated the agreements for each event severity group.

Results

During 2010, a total of 4196 admissions (of 3252 patients) to the gastrointestinal, vascular or trauma surgery wards were recorded. In 705 (16.8%) of these, one or more adverse outcomes were documented in the LHCR. Of the 180 selected admissions, the resources of 135 different patients admitted could be analysed. Forty-five admissions were excluded because these concerned readmissions of the same patients (n=3) or the data from one of the resources could not be retrieved (n=42). These reasons for exclusion were not likely to be related to the completeness of the adverse outcomes as stated in the various resources. Hence, we considered the remaining set of 135 patients as valid for our purpose.

Of the 135 patients included, 60.7% were men, with a mean age of 59.3 years. Median length of stay was 8 days. As shown in table 1, their characteristics did not differ significantly from the whole group of patients admitted in 2010, except for a significantly longer length of stay and higher number of American Society of Anesthesiologists (ASA)-2 patients, obviously because we oversampled patients with adverse outcomes. In 70% of the 135 patients, one or more surgical procedures were performed, resulting in a total of 208 procedures in these patients. Based on the summary of all events from all resources, 275 adverse outcomes were recorded in total. A total of 98 of 135 patients had suffered one or more adverse outcomes.

Table 1

Characteristics of the 135 selected patients versus all patients admitted in 2010

Characteristic	Included patients (N=135)	Patients admitted in 2010 (N=3252)
Men, n (%)	82 (60.7)	1808 (55.6)
Age (years)
Mean (SD)	59.3 (17.0)	55.1 (18.0)
Median (IQR)	62.0 (47.5–71.8)	57.6 (42.7–68.1)
Length of stay (days)
Mean (SD)	14.9 (27.6)	7.9 (13.9)
Median (IQR)	8.0 (3.0–16.5)	4.0 (1.0–9.0)
Underwent surgery, n (%)	104 (70)	2276 (70)
General (%)	22.4	26.2
Oesophago-gastro-intestinal (%)	34.5	30.4
Hepato-pancreato-biliary (%)	14.3	8.9
Trauma (%)	11.3	20.6
Vascular (%)	17.5	13.9
ASA classification* (%)
1	6.8	24.2
2	72.7	45.6
3	13.6	26.0
4	6.8	3.3
5	0.0	1.0

American Society of Anesthesiologists (ASA) class at admission of patients who underwent surgery.

The proportions of patients with one or more adverse outcomes as recorded in the different resources as well as in the official LHCR are summarised in table 2. In 86% of the cases, the LHCR was in agreement with the reference standard as to the total number of patients with one or more events. In other words, 14% of admissions with adverse outcomes were not recorded in our official registry. Table 3 shows the severity categorisation of the events as recorded in the various resources. Virtually all events missing in the LHCR were mild (class 1) events that could be treated with non-surgical interventions, including pain, delirium and bladder complications. The six missing class-2 events were categorised as haemodynamic instability (n=2), wound abscess (n=1), gastroparesis (n=1) and miscellaneous complications (n=2). The one class-3 complication missed was a pressure ulcer.

Table 2

Percentages and absolute numbers of patients with one or more adverse outcomes as recorded in each resource compared with the reference standard

Resource	Patients with adverse outcome(s)
Reference standard	100% (98)
LHCR*	86% (84)
Morning hand-offs	80% (78)
Discharge letter	78% (76)
Medical file	78% (76)
Nursing file	77% (75)

Dutch national surgical complication registry.

Table 3

Percentages and absolute numbers of adverse outcomes as recorded in each resource, categorised per severity class

Severity class*	Reference standard	LHCR†	Morning hand-offs	Discharge letter	Medical file	Nursing file
0	100% (11)	73% (8)	9% (1)	18% (2)	55% (6)	64% (7)
1	100% (221)	44% (97)	32% (71)	38% (85)	60% (132)	67% (148)
2	100% (31)	81% (25)	66% (20)	77% (24)	68% (21)	58% (18)
3	100% (9)	89% (8)	56% (5)	22% (2)	0% (0)	0% (0)
4	100% (3)	100% (3)	100% (3)	67% (2)	67% (2)	33% (1)
Total	100% (275)	51% (141)	36% (100)	42 % (115)	59% (161)	63% (174)

Severity classes: 0, event without adverse effect on health; 1, temporary health disadvantage recovering without (re)operation; 2, recovery after (re)operation; 3, (probably) permanent damage or function loss; 4, death.

Dutch national surgical complication registry.

Adverse outcomes related to medical management errors (‘class 0’) occurred rarely but were poorly registered, particularly in the morning hand-offs and discharge letters. This is mainly due to the fact that these events mainly concerned ‘cancelled operations’. Although these were recorded during the morning hand-offs, they were considered to be of limited information to include in the discharge letters.

The vast majority (80.4%) of the adverse outcomes in the reference standard was reversible and mild (class 1). The morning hand-offs and discharge letters omitted most of these events. Only 38% of these mild events were registered in these resources. Also the LHCR missed most of the mild events, which were best reported in the nursing and medical files.

Surgical complications requiring a reintervention (class 2) seemed to be under-recorded in most resources. However, this may be influenced by the fact that in some patients more than one class-2 event had led to a single reintervention, but only one of these events was recorded as reason for the reintervention. Unfortunately, there was no consensus on how this should have been recorded.

The more serious adverse outcomes (classes 3 and 4) occurred less frequently. The medical and nursing files did not state class-3 events, probably because at that time the permanent effects of such events could not yet be assessed. Strikingly, even the discharge letter did not mention many of the events leading to permanent damage or function loss. A few deaths were not documented in the nursing and medical files because the patient died on the intensive care unit, which was documented in a separate discharge letter not included in this study.

To discover which types of adverse outcomes in particular might be under-recorded, we investigated which events had a documentation rate of less than 50% compared with the reference standard. The events are listed as following: abscess, shock, pressure ulcers, delirium, fluid collections, pain, pulmonary complications, overinfusion, urinary tract-related complications, fistulas and vascular complications (ie, phlebitis or dialysis shunt occlusion).

Discussion

Registration of surgical adverse outcomes appears valuable but is largely depending on the reliability of the underlying sources. In many hospitals, a complication registration system, such as the LHCR in the Netherlands, heavily depends on the accuracy of the reporting and documentation of adverse outcomes through various resources. The usage of the available resources might be different in the various Dutch hospitals using the LHCR and is not well defined.

The present study showed that adverse outcomes are under-reported with the LHCR system and also during the morning report. The less severe events tended to be reported less frequently, except in the nursing file, which was not designed to serve as input for the LHCR. Nevertheless, all likely resources should be incorporated for an optimum registration of adverse outcomes. The medical rather than the nursing file seems the most appropriate additional resource for this purpose.

One of the reasons for under-reporting might be a reluctance or negligence among doctors to report adverse outcomes. Particularly strong disincentives for reporting are shame, fear of liability, loss of reputation and peer disapproval.15 The awareness that medical errors, and also surgical complications, are frequently system errors rather than an individual liability has helped abandoning a shame-and-blame culture and has harnessed the medical professional to report errors and adverse outcomes.16 Furthermore, increasing societal demands as to safety and transparency in healthcare have created more awareness of the importance of, and willingness to contribute to, and a better quality of care.17 18

The completeness of the complication registration may also vary with the types of adverse outcomes a hospital decides to record. Should adverse outcomes with a low severity, for example, such as delay of surgery, be omitted, that is, a ‘light’ version of complication registration, a higher accuracy would be achievable. A drawback of this would be that other, but common, events, such as wound infections or pressure ulcers, are not monitored properly and cannot be acted upon. Moreover, particularly for relatively minor surgical interventions, patients will still perceive ‘mild’ adverse outcomes as important and undesirable.

Conversely, registration of all possible adverse outcomes requires more effort to extract these from the various complementary resources. When pursuing this policy, the nursing file may be included as an important source of more ‘mild’ events, such as pressure ulcers, insufficiently controlled pain or urinary tract infections. Besides, recording the number of postponed or cancelled surgical interventions can be useful as indicator for a change in the organisation process of care and thereby an improvement of the quality of care.

The low number of adverse outcomes included in the discharge letter may be due to selection of items considered relevant to the general practitioner or follow-up institution. However, any permanent damage or function loss acquired during admission surely needs more attention than it appears to receive, based on this study, in particular in the early phase after discharge and management of the adverse outcomes by the general practitioner. A predefined format and content of these letters, for example, a computer-generated summary, can improve quality and safety of hand-off communication and subsequent care.19

A limitation of this study could be that even the reference standard may have been an underestimation of the true number of adverse outcomes that had actually occurred. If so, the various sources leave even more events untracked. However, this does not seem likely, as all possible sources were studied in retrospect. We did not study the events that might have occurred (shortly) after discharge, which was beyond the scope of this research. Second, the random sample of admissions investigated may have been relatively small. Nevertheless, the trends we found are quite conspicuous and seem reliable since two investigators independently reviewed the resources for events.

In conclusion, the registration, management and prevention of surgical adverse outcomes are not to be neglected in daily clinical practice. It may also impact the selection of patients to be treated and procedures to be performed. Therefore, hospitals and clinicians should be willing to put effort in a structural and reliable means to register not only the beneficial but also the harmful effects of their professional activities or clinical management to improve the quality of care for their patients.