Systematic review of hospital-wide complication registries.

BACKGROUND: An institutional registry covering all surgical specialties could be an implementation tool in quality benchmarking between hospitals and aid determination of their cost-effectiveness. The objective of this systematic literature review was to evaluate original articles on existing prospective surgical registries that can be used by single institutions across surgical specialties.
METHOD: A systematic review of the literature using PRISMA guidelines was conducted for articles focusing on hospital-wide surgical registries. Single-specialty retrospective registries, non-defined outcome measures or system protocols, and studies not in English were excluded.
RESULTS: Five articles were included for analysis. Evaluation of the articles revealed wide methodological heterogeneity in the classification and categorization of complications and data collection methods.
CONCLUSION: Ideal surgical quality monitoring systems should be real-time, contain patient-related risk factors, and encompass all surgical specialties. At present, such institutional registries are rarely reported and no consensus exists on their standard definitions and methodology.

Introduction

Reporting on surgical quality and outcomes remains an issue. More than a century ago, Ernest Codman wrote: ‘The common sense notion [is] that every hospital should follow every patient it treats, long enough to determine whether or not the treatment has been successful, and then to inquire, ‘If not, why not?’ with a view to preventing similar failures in the future’1. Codman's idea still applies today; there is a need for surgical outcomes to be followed and made public. However, to date, there seems to be no consensus on how surgical quality should be measured and reported.

Surgical quality is a heterogeneous concept. Donabedian2 suggested that the concept of quality should be divided into three domains: outcome, structure and process. Outcome can be measured in many ways, including functional gain or health benefit, patient satisfaction, economical gain, quality‐of‐life measurements, and complications or adverse event frequency. Surgical complications cause a major economical and human burden, and can be used as an outcome quality indicator3, 4, 5.

Complications and other quality data have been monitored at various levels of the healthcare system. Claims and mortality data reflect quality for a broad (often national or regional) spectrum of the healthcare system and may provide data for crude benchmarking. Within single surgical specialties, diagnoses or procedures, there are numerous examples of quality and complication registries, the earliest of which started in the field of thoracic surgery (Society of Thoracic Surgery Registry)6. Such registries provide information and feedback on specific patient populations, and serve as process development efforts7.

The development of institutional registries that combine all surgical specialties has been challenging. An institutional registry could serve as a benchmarking and quality control tool just as a single‐specialty registry, but extend this perspective further and could provide data on cost–benefit and health‐gain aspects of the healthcare provider as a whole.

The American College of Surgeons (ACS) now uses a wide, standardized platform called the National Surgical Quality Improvement Program (ACS‐NSQIP), which was initially instituted by the Veterans Health Administration in response to the need for quality improvement8, 9. However, such registries are costly and require dedicated staff.

Follow‐up of complications is clearly needed for surgical quality control. An optimal surgical quality monitoring system should encompass all specialties, have a high degree of coverage, be real‐time, and contain patient‐related risk factors while requiring as few resources as possible. The aim of this systematic literature review was to identify and evaluate original articles on existing prospective surgical registries that can be used by single institutions across surgical specialties.

Methods

The PRISMA statement10 was used as a guideline for this study.

Eligibility criteria

Studies describing surgical monitoring systems that aimed to identify, record and monitor surgery‐related complications, morbidity and mortality within different surgical specialties at single institutions were included. Studies that evaluated registries for a single surgical specialty or indication were excluded. Surgical records on paediatric patients and reports that did not comply with the Patient, Intervention, Comparator and Outcome (PICO) criteria were not included.

Literature search and study selection

Four medical bibliographical databases for published literature were searched systematically: Ovid MEDLINE® In‐Process and other non‐indexed citations and Ovid MEDLINE® from 1946 to 19 February 2015; EBM Reviews – Cochrane Database of Systematic Reviews between 2005 and January 2015 (OVID); PubMed (only ahead‐of‐print articles to February 2015) and Web of Science – Core Collection to February 2015 (Core Collection, Indexes = SCI‐EXPANDED, SSCI).

Searches consisted of three search aspects, each including both Medical Subject Headings (MeSH) terms and text words: search terms related to surgical complications; search terms related to hospital information systems, registries, databases and records; and search terms related to risk adjustment and risk assessment, quality, safety and economic aspects (Fig. 1; Appendix S1, supporting information).

Figure 1

Literature search strategy. MeSH, Medical Subject headings

Records were retrieved through electronic databases (Table S1, supporting information). Two independent researchers began the initial study selection by screening article titles and then their abstracts. Three exclusion criteria (non‐original data, retrospective study and single‐specialty register) were applied initially to the title screening and then to the abstract screening. Eligible studies included original data on surgical, multidisciplinary (surgical subspecialties), prospective monitoring systems to identify, record and monitor surgery‐related complications using validated outcome measures and well described system protocols and parameters. Of articles originally identified, those not in English were excluded. Two reviewers divided the remaining abstracts into three categories: online/feedback construction and protocols; quality assessment methods and cost‐effectiveness; and patient‐related risk factors.

Inclusion criteria applied at this stage were: single‐institution monitoring systems or multi‐institutional systems that could be used on a single‐hospital level, and whether they recorded complications continuously, prospectively or online. Articles were further excluded based on pure cost or pure risk factor analyses. The remaining full‐text articles were then reviewed by two independent reviewers and excluded if they did not meet PICO criteria. Remaining studies were discussed by all three reviewers and retained if they were single‐hospital, prospective complication registries covering all surgical specialties.

Data extraction

The following data were extracted from the registries and categorized: country, hospital type, duration of follow‐up, standard definitions, a denominator from which incidence rates were calculated, inclusion of risk factors, number of patients, output and feedback, study design, coverage, data monitoring, data processing and findings.

Data were recorded on a predesigned collection form. Discrepancies were resolved by discussion within the group of reviewers.

Synthesis of results

Based on the heterogeneity of articles, meta‐analysis was not applicable, and the qualitative evidence was synthesized.

Risk‐of‐bias assessment

Because all the studies were observational, a recently presented method for assessing risk of bias was used11 12. The ten main methodological issues and description of how to assess whether these issues possess a risk of bias are shown in Table 1.

Table 1

Assessment of validity of surgical clinical complication registry studies according to Malmivaara11

	Veltkamp et al.13	Veen et al.14	Bilimoria et al.15	Khuri et al.8	Rebasa et al.16
Power calculated (differences indicated)	n.a.	n.a.	n.a.	n.a.	n.a.
Selection of patients described*	Partially	No	No	Partially	Partially
Valid and sufficient documentation of baseline characteristics	Yes	No	No	Yes	No
Baseline comparability acceptable	n.a.	n.a.	n.a.	n.a.	n.a.
Sufficient documentation of surgical procedures	Yes	Yes	No	Yes	No
Valid and sufficient documentation of outcomes	Yes	Yes	Yes	Yes	Yes
Drop‐out rate acceptable	Yes	No	n.r.	No	No
System‐related features documented†	Yes	Yes	Partially	Partially	Yes
Documentation of staff competence‡	No	No	No	Yes	No
Appropriate statistical analyses and risk adjustment	Yes	No (no risk adjustment)	No (no risk adjustment)	Yes	No (no risk adjustment)
Total of validity points (0–10)	6	4	2	7	3

Yes, if well described or covers whole catchment area;

checklists, quality improvement systems, resources, volume, etc.;

description of experience, etc. n.a., Not applicable; n.r., not reported.

Results

Of 2322 articles originally identified, 224 abstracts were screened and categorized. A further 165 articles were excluded based on pure cost or risk‐factor analyses. The remaining 59 full‐text articles were reviewed by two independent reviewers. Of these, 45 with the following criteria were excluded: studies of single‐specialty registers, articles not meeting PICO criteria, and lack of validated outcome measures and described system protocols. The remaining 14 studies were discussed by all three reviewers; five studies8 13, 14, 15, 16 were finally included (Fig. 2).

Figure 2

PRISMA diagram for the study. PICO, Patient, Intervention, Comparator and Outcome

The risk‐of‐bias assessment showed at least some deficiencies in the description of patient selection in all five studies, and insufficient documentation of patients' baseline characteristics, surgical procedures and staff competence in three, two and four studies respectively (Table 1). Valid and sufficient documentation of outcomes was found in all five studies, as well as full or partial description of system‐related features. Drop‐out rate was acceptable in one study and risk adjustment was provided in two studies.

Five surgical clinical complication registry studies on existing prospective surgical registries that could be used by single institutions across surgical specialties were found. All studies were designed to track adverse events and report them online in an electronic database or prospectively. The registries were created independently between 1991 and 20058 13, 14, 15, 16. The specific focus of the articles varied: patient‐related risk factors, adverse event reporting behaviour (coverage), trends in complication frequency, and feedback loop of the results. Numerous reports on the data from the ACS‐NSQIP registry exist, whereas the methods and principles of the registry itself have remained the same. The reports from the predecessor Veterans Administration (VA) NSQIP, and later the ACS‐NSQIP, use the same registry platform. Therefore only one article on the registry was representative for the present study, and the first one describing the original VA‐NSQIP was chosen8. The structural characteristics of the five studies8 13, 14, 15, 16 were identified; information on patients, duration of follow‐up, definition of outcome, data coverage, monitoring and processing, feedback and findings according to the study design were gathered (Table 2; Table S2, supporting information).

Table 2

Structural characteristics of the surgical clinical complication registry studies

Reference	Country	Study period	Patients and surgical indications	Duration of follow‐up	Definition of outcome	Inclusion of operative and patient risk factors	Study design	Coverage	Data monitoring
Veltkamp et al.13 (2002)	Netherlands	1 year (1996–1997)	All surgical ward patients (also non‐operative) (n = 3075)	30 days after discharge	Complications according to severity (Clavien–Dindo classification)	Yes (emergency, minor or major surgery, and ASA grade, age, sex, co‐morbidities, BMI)	Data collection of risk factors and complications for a risk model	1 hospital surgical ward	Responsible medical team
Veen et al.14 (2005)	Netherlands	> 15 years (1986–2001)	Patients admitted to surgical department for operation (n = 24 201 + 31 161*)	Care on ward after surgery	Complications according to ASN	No	Study of definition and registration methods (real‐time register)	1 hospital surgical department	Physician who noticed the complication
Bilimoria et al.15 (2009)	USA	2 years (2005–2007)	All surgical (also non‐operative) patients (n = 15 524)	Surgery and care on ward	All complications (categorized)	No	New system for reporting adverse events	1 hospital surgical unit	Medical team
Khuri et al.8 (1995)	USA	October 1991 to December 1993	Non‐cardiac operations (n = 83 958)	30 days after discharge	21 postoperative adverse events and mortality	Yes (17 preoperative risk variables (ASA grade, serum albumin level), urgency and duration of surgery)	Prospective study with collection of data in 44 medical centres	44 hospitals	Surgical assessment nurse
Rebasa et al.16 (2009)	Spain	1·5 years during 2005–2006	Patients admitted to surgical department for operation (n = 3807)	30 days after discharge	Adverse events (Harvard Medical Practice Study Group classification†)	No	Prospective surveillance of adverse events and errors in surgery department	1 hospital	Any staff member

This study was conducted in two phases, before and after the system was computerized.

Adverse event, unexpected consequence or lesion caused to the patient as a result of treatment rather than underlying illness; preventable adverse event, adverse event or event attributable to error; error of assistance, error produced by mistakes in the planning or execution of diagnosis and treatment. ASN, Association of Surgeons of the Netherlands.

Patients

The five articles8 13, 14, 15, 16 reported all surgical operations that were performed during the study period (Table 2). There was variation in whether minor and ambulatory surgery was included or not. The VA‐NSQIP register reported excluding patients for surgical operations with very low mortality rates (parathyroidectomy, orchidectomy, carpal tunnel repair)8. Two studies14, 16 excluded minor surgery (all dermatological surgery) and major ambulatory surgery (haemorrhoidectomy, groin hernia surgery). Two studies13, 15 registered all surgical (also non‐operative) patients.

Definition of a complication

The classification and description of complications varied notably (Table 2). Three registries8, 13, 16 used all 30‐day surgical complications as the standard definition. Two studies14 15 collected data on complications only during the hospital stay. In each of the five registries, complications were measured and categorized differently. These complications were either described generically or recorded with a standardized index or scale (Clavien–Dindo17 and the Harvard Medical Practice Study Group18) or with a national classification system (Association of Surgeons of the Netherlands). Generic categorization included the type of morbidity (for instance, thrombosis or infection) or type of error (such as diagnosis, judgement, technique or system error). A standardized index can measure complications based on severity (for example the Clavien index17).

There was heterogeneity in the reporting of unexpected consequences (such as readmissions, reoperations and transfers to the ICU), adverse event category (for example anaesthetic, gastrointestinal, haematological, cardiac or infectious problem, remaining insufficient result or disturbed function), anatomical location of the complication (muscles, nerves, skeleton, arterial/venous, lymphatic system, subcutaneous)14 15 and additional description (such as management problem or materials left in the wound)16.

Staff involved

In all the five studies the method of data collection was designed differently (Table 2). Data were collected as part of the process of care (by all staff), by responsible medical teams (surgical trainees and consultants, nurses and project researchers), or by dedicated monitoring staff.

Risk factors

Risk factors were patient‐related and operation‐related (Table 2). Patient‐related risk factors were collected and measured differently based on patient status (age, sex, ASA grade, functional/self‐supporting status, BMI, smoking, weight loss and wound infection), medical tests (such as laboratory variables and electrocardiography results) or co‐morbidities (either separately or with an index). Operation‐related factors referred to the classification of operation complexity and whether the operation was performed as planned (elective) or emergency. Two studies8 13 collected data on both types of risk factor.

Data coverage

Four8 13, 14, 15 of the five studies measured the coverage of data collection and outcome reporting. Veltkamp and colleagues13 reported missing data for only 5 per cent of patients. Khuri and co‐workers8 reported that 49·7 per cent of operations were included in the register. In the study by Bilimoria et al.15, complications were reported in 25 per cent and inpatient deaths in 42 per cent of cases. In the report by Veen and colleagues14, an increase from 7 to 33 per cent was observed in the rate of registering complications following the introduction of an electronic database.

Discussion

The assessment and reporting of surgical outcomes as quality metrics have gained increasing importance as part of quality improvement and cost containment. It seems reasonable that surgical units should record their results and monitor the frequency of adverse events and complications. Ideally, such monitoring systems would be real‐time, contain patient‐related risk factors, and encompass all surgical subspecialties13 15, 19. The aim of this systematic review was to determine how widely such surgical registries have been reported in the literature.

Despite an extensive review of the literature, only five original articles8 13, 14, 15, 16 were found. Due to heterogeneity between the interventions, settings and outcomes, it was not possible to perform a meta‐analysis.

The classification and categorization of complications were different in all the included studies, emphasizing the need for international standards on institutional quality control systems and complication classification. It seems likely that a classification system according to complication severity would be most applicable to a cross‐specialty surgical registry20.

Two8 13 of the five studies collected data on surgery‐related as well as patient‐related risk factors. However, which risk factor data are sufficiently relevant to be collected is still unclear. The NSQIP has provided much information regarding the impact of patient‐related risk factors21, 22, 23. The risk factors have also been studied specifically for different subspecialties, but overall the variables carrying the greatest risk were albumin, ASA grade, surgical complexity score and emergency class, functional status and wound infection. Mortality, age, disseminated cancer and ventilator‐dependence also played a major role.

In the included studies, data were collected by all the staff, medical teams or dedicated monitoring staff. The process of how and by whom complications should be registered is an unresolved matter. Dindo et al.24 reported that surgical residents recorded outcomes poorly and unreliably, and concluded (along with several other studies) that surgical outcomes should be evaluated by dedicated personnel19. The drop‐out rate was deemed unacceptable in four8 14, 15, 16 of the five studies, referring to the coverage of data collection and outcome reporting, as in the checklist of methodological issues for assessing validity of benchmarking controlled trials the proportion of withdrawals and drop‐outs should not exceed 10 per cent12. In one15 of the studies data coverage was not mentioned, and in three studies8 14, 16 data coverage on complications or operations was below 50 per cent. In the study by Bilimoria and colleagues15, interventions to improve reporting were largely unsuccessful. Automated processes would undoubtedly solve the problem of unreliable data collection.

In contrast to earlier studies19 25, 26, this systematic review assessed whether ideal surgical multispecialty complication registries existed, preferably with risk adjustment, that could be used in single institutions. Although the focus was prospective clinical registries, other data collection methods that were reported in the literature included retrospective cohorts, data mining, trigger tools for electronic language processing, and computerized screening of administrative data. Prospective clinical data collection can provide a tool for quality improvement efforts within clinics, with continuous feedback. Clinical registries have also been considered more reliable than administrative data, due to factors such as accurate diagnoses27, 28, 29. However, studies that assessed the reliability of outcome measures documented that the lack of a sufficient caseload limits the usefulness of clinical registry data in single hospitals30. An option to increase the reliability of outcome measures would be to use composite indicators that combine quality signals, such as outcomes from multiple or related procedures, length of stay and reoperation rate31 32.

Although a comprehensive systematic methodology was employed throughout the study to minimize bias and error in the study selection, data extraction and quality assessment phases, the possibility of publication bias could not be excluded. As many complication registries may be used only for hospital quality management, many may have not been reported in the literature. Benchmarking of hospitals will become feasible once reporting of complications based on standardized methods is implemented.

Appendix S1 Database searches

Table S1 Retrieved references by database

Table S2 Structural characteristics of the surgical clinical complication registry studies

Click here for additional data file.