The Value of Aggregated High-Resolution Intraoperative Data for Predicting Post-Surgical Infectious Complications at Two Independent Sites.

Surgical procedures carry the risk of postoperative infectious complications, which can be severe, expensive, and morbid. A growing body of evidence indicates that high-resolution intraoperative data can be predictive of these complications. However, these studies are often contradictory in their findings as well as difficult to replicate, suggesting that these predictive models may be capturing institutional artifacts. In this work, data and models from two independent institutions, Mayo Clinic and University of Minnesota-affiliated Fairview Health Services, were directly compared using a common set of definitions for the variables and outcomes. We built perioperative risk models for seven infectious post-surgical complications at each site to assess the value of intraoperative variables. Models were internally validated. We found that including intraoperative variables significantly improved the models' predictive performance at both sites for five out of seven complications. We also found that significant intraoperative variables were similar between the two sites for four of the seven complications. Our results suggest that intraoperative variables can be related to the underlying physiology for some infectious complications.

Introduction

Surgical procedures carry the risk of postoperative infectious complications, which can be severe, expensive, and can put patients’ lives at risk [1-5]. With the ubiquity of decision support tools integrated into electronic health record (EHR) systems, perioperative decision support capabilities represent a promising direction for reducing postoperative risk or tailoring care including interventions aimed at individualized anticipation and management of the complication risk.

Initial surgical planning is primarily based on a preoperative assessment. While clearly essential, this assessment is insufficient to capture dynamic risks or changes in patient status, which may occur in the operating room. Preoperative assessment is, by definition, based on preoperative data and the planned procedure. Surgeries can deviate from the original plan for a variety of reasons or may involve unexpected physiologic changes such as bleeding, rendering the original risk estimates less valid. While the original estimates reflect the general risk of the patient and the planned surgery, they reflect neither new findings, changes in patient status, nor an altered course of surgery.

A growing body of published literature indicates that high-resolution intraoperative data can be predictive of complications [6-9]. However, the findings in these studies often contradict each other, are difficult to replicate at other sites, and are often based on very limited use of intraoperative data. For example, several studies have reported intraoperative hypothermia increases the risk of post-surgical infectious complications [6-8]. On the contrary, other studies found that there were no differences in the minimum, maximum, or ending temperatures between patients with surgical site infection (SSI) and those without – potentially due to an institutional effect: better temperature control at certain institutions. While temperature was unassociated with the surgical site infection, other intraoperative factors, most prominently blood loss, showed a significant association [9]. Besides institutional differences in clinical practice, differences in findings may also arise due to differing analytical approaches. Studies differ in the way they aggregate intraoperative variables: some studies use moments and extrema (minimum and maximum), while others use the ending measurement (the last measurement or the average of the last few measurements). Finally, differences in findings can also stem from discrepancies in definitions of the variables and outcomes.

Several high-quality registries form the backbone of surgical quality improvement research aimed at understanding and reducing postoperative complications. The National Surgical Quality Improvement Project (NSQIP) registry [10, 11] uses high-quality manually curated data [1, 12] and stands out as the gold-standard for surgical quality improvement and surgical outcomes research. NSQIP offers standardized definitions for exposure/risk variables and outcomes, helping to ensure consistency across studies. Unfortunately, it lacks detailed information regarding intraoperative risk factors, including physiologic data, laboratory data, medications, or other treatments, limiting its use for our purpose.

We conducted this study at two independent sites, Mayo Clinic and University of Minnesota-affiliated Fairview Health Services. These are two large Midwestern academic health systems with tertiary centers providing a wide range of surgical services. Both sites are members of the NSQIP registry; so, we were able to use outcome data from NSQIP patients at each site, serving as a high quality gold-standard. We used preoperative and intraoperative EHR data collected from the clinical data repositories of the respective sites and standardized the data elements across the two sites to facilitate direct comparison of the models.

The study aims to answer the following questions: (i) In the context of perioperative decision support, does the use of intraoperative data improve the performance of 30-day postoperative risk models? (ii) Do significant intraoperative variables in the risk models pertain to the same physiological concepts across the two sites? We proceed with the assumption that when significant intraoperative variables differ despite having been defined identically, the models may capture institutional differences; when the significant variables coincide, they are more likely to relate to the physiological processes underlying the postoperative complications.

Methods

Setting and cohort definition.

We consider two independent Midwestern health care systems: Mayo Clinic (MC) and Fairview Health Services (FHS). We include all patients from MC and FHS between 2010 and 2017 who are part of the NSQIP sample. For these patients we collected all available information about their NSQIP index surgery and a 30-day history before the index surgery from the respective institutions’ EHR repositories. The NSQIP registry collects complications within a 30-day postoperative window. If a patient had another surgery in the 30-day postoperative window, we used the index surgery and measured the 30-day postoperative window for the outcome from the index surgery. For each complication, patients with the same pre-existing complication at the time of surgery were excluded.

Independent variables.

We primarily rely on known risk factors of infection [13-15]. Independent variables were divided into three groups: demographic, preoperative, and high-resolution intraoperative. We limit ourselves to basic demographic information, such as age, sex, and body mass index (BMI) that are generally available. Preoperative variables are historic diagnoses including the problem list, procedures, and medications, as well as the preoperative indication for surgery. We use laboratory results and vitals to establish a preoperative baseline. Data from the preoperative assessment were preferentially used; if the data is not available, we use measurements from no more than 30-days before surgery. Diagnosis codes are rolled up into complications using the Clinical Classification Software [16].

Aggregating intraoperative variables.

The intraoperative variables include orders, medications, and high-resolution vitals and labs. The stream of high-resolution variables needs to be divided based on the three stages of anesthesia. During the first (approximately) 15 minutes, called the induction phase, the vitals drop and deviate heavily from the normal. The last 15 minutes is called the emergence phase, where vitals are expected to return to close to normal but also involves significant changes as the patient is transitioned from anesthesia—often with full ventilation—to no anesthesia. The operation takes place between these two phases, in the so-called maintenance phase, where the vitals are expected to remain stable, although different from the preoperative baseline. In this work, we focus on the maintenance phase and use the mean value of labs and vitals during this phase.

Outcomes.

We consider seven infectious outcomes: sepsis, septic shock, urinary tract infection (UTI), pneumonia (PNA) and the three kinds of SSI (superficial, deep tissue, and organ space SSI). We extract the outcome information from the NSQIP registry.

Modeling.

Each outcome was modeled independently using logistic regression. For each outcome, two models are constructed. The “pre” model only uses demographics and preoperative data (30-day history of complications, baseline labs and vitals), while the “pre+intra” model uses aggregated intraoperative measurements on top of the preoperative and demographic data.

All models are logistic regression models. Missing labs and vitals were imputed using the middle of the normal range for the measurement. This assumes that the measurement is missing because it was deemed unnecessary to measure. The initially high number of independent variables was reduced by causal variable screening [19]. We used the PC-Simple algorithm [20] with a maximal condition set size of three [21]. This algorithm discards all variables that are independent of the outcome given at most three other variables. The rationale is that these variables do not affect the outcome directly, they only affect the outcome through other variables. Subsequent backwards elimination was applied to the remaining set of independent variables with a significance level of .05. The R statistical computing environment was used for all modeling. The PC-Simple algorithm is available in the pcalg R package [20].

Evaluation.

Consistent with the intended use in perioperative decision support, we evaluated the models based on their predictive performance using concordance as the metric. Concordance is the probability that between two randomly selected patients, among which one has the complication in question while the other does not, the one with the complication has the higher predicted risk. Concordance is equivalent to the commonly reported area under the receiver-operating-characteristic curve (AUC).

Internal validation and effect of the intraoperative variables.

Bootstrap estimation with 200 replications was used to estimate the concordance of the models. For each complication, two models were built on the same bootstrap replication: one with and one without the intraoperative variables. We measured the effect of the intraoperative variables as the mean difference in concordance between the models with and without intraoperative variables across the 200 bootstrap replications. The statistical significance of the difference was determined through a paired t-test.

External comparison.

If the intraoperative variables capture a valid physiological phenomenon, then the variables selected at the two sites relate to the same quantity. We manually examined the models with the intraoperative data across the two sites to determine whether the selected variables coincide or at least relate to the same physiological concept. For example, oxygen saturation and ventilator settings relate to the same physiological concept (oxygen in the blood). When the selected variables relate to different concepts, there is a risk that the model simply captures an institutional artifact.

Results

Cohort Description

Table 1 contains a description of the analytic cohort at the two health systems. It contains demographic information, outcomes, history of the outcomes, history of relevant complications, and baseline (preoperative or at most 30 days before the index surgery when preoperative measurement is unavailable) lab results and vitals. Due to the high number of lab results and vital signs, we only report those that were significant in at least one of the models. Categorical variables are reported as count and percentage in parenthesis; continuous variables are reported as median and interquartile range in parenthesis.

Table 1 –

Cohort Description.

	Mayo Clinic	Fairview
Number of patients	38,045	9,044
	Demographics
Age	61 (49, 71)	55 (40, 66)
Gender (male)	18,769 (48.7%)	4053 (45%)
	Outcomes
SSI Superficial	586 (1.5%)	107 (1.2%)
SSI Deep Tissue	186 (0.5%)	45 (0.5%)
SSI Organ Space	417 (1.1%)	119 (1.3%)
UTI	282 (0.7%)	147 (1.6%)
PNA	453 (1.2%)	111 (1.2%)
Sepsis	494 (1.3%)	88 (0.98%)
Septic Shock	186 (0.5%)	40 (0.44%)
	History of Complications
SSI	1,232 (3.2%)	91 (1.01%)
UTI	1,271 (3.3%)	127 (1.4%)
PNA	777 (2.0%)	73 (0.81%)
Bacteremia	133 (0.3%)	23 (0.25%)
Infections	3,953 (10.4%)	165 (1.8%)
Opport. Inf.	432 (1.1%)	45 (0.50%)
Malnutrition	792 (2.1%)	60 (0.66%)
Cancer	13,547 (35.6%)	1413 (16%)
Metastatic Disease	4,359 (11.5%)	112 (1.2%)
Transplant	624 (1.6%)	224 (2.5%)
Diabetes (T1&2)	5,215 (13.7%)	462 (5.1%)
COPD	2,372 (6.2%)	94 (1.04%)
	Baseline labs and vitals
BMI	28.4 (24.8, 32.9)	27.9 (24.6, 33.6)
Pulse	70.0 (62.6, 80.0)	75.0 (66.3, 85.0)
Respiration	9.5 (8.1, 10.8)	16 (16, 18)
Bilirubin	0.45 (0.45, 0.45)	0.67 (0.4, 1)
BUN	26 (18, 35)	14 (10, 20)
RBC	3.79 (3.13, 4.36)	4.27 (3.77, 4.70)
WBC	7.85 (4.45, 11.5)	8.50 (6.23, 12.1)
MCV	88.8 (87.2, 94.1)	89.4 (86.0, 93.0)
RDW	14.8 (13.5, 16.6)	13.9 (13.0, 15.3)
Hematocrit	33.1 (27.6, 39.7)	38.2 (33.8, 41.6)
PH	7.38 (7.32, 7.42)	7.38 (7.32, 7.42)
CO2	40.5 (36.5, 43.5)	25.5 (24.0, 27.5)
Ca	4.71 (4.55, 4.92)	4.59 (4.30, 4.78)
K	4.15 (3.90, 4.50)	3.9 (3.7, 4.2)
Na	138 (137.8, 138.4)	139 (137, 141)

(1) Abbreviations: UTI: urinary tract infection; PNA: pneumonia; SSI: surgical site infection; COPD: chronic obstructive pulmonary disease; BMI: body mass index; BUN: blood urea nitrogen; RBC: red blood cell [count]; WBC: white blood cell [count]; MCV: mean corpuscular volume; RDW: red blood cell distribution width; Ca: calcium ion concentration; K: potassium ion concentration; Na: sodium ion concentration. (2) Only variables that were significant in at least one model are presented.

Model performance and effect of the intraoperative variables

Table 2 presents the predictive performance of the models for each of the outcomes. For each outcome, the table contains three rows: the first one shows the performance as measured by AUC of the “pre” model (that uses only demographics and preoperative data), the second one shows the performance of the “pre+intra” model, which uses the intraoperative data in addition to the demographics and preoperative data, and the third row shows the difference in performance between the “pre” and “pre+intra” models. For the “pre” and “pre+intra” models, the standard deviation of the performance (obtained from 200 bootstraps) is shown in parenthesis; for the difference, the p-value of the paired t-test is shown in the parenthesis using the scientific notation (e.g. 2e-8 is 2*10−8). All differences are statistically significant at 0.05 confidence level, but superficial and deep tissue SSI are not significant on FHS side after Bonferroni correction.

Table 2 –

AUC performance of the “pre”, the “pre+intra” models, and the difference in performance.

Outcome	Model	Mayo Clinic	Fairview
PNA	Pre	.877 (±.021)	.693 (±.044)
	Pre+intra	.881 (±.022)	.734 (±.044)
	Difference	.004 (1e-8)	.041 (1e-16)
Sepsis	Pre	.736 (±.024)	.695 (±.045)
	Pre+intra	.755 (±.021)	.711 (±.046)
	Difference	.019 (1e-16)	.016 (8e-13)
Septic Shock	Pre	.827 (+.031)	.715(+.O78)
	Pre+intra	.834 (+.030)	.755(+.O79)
	Difference	.007 (1e-13)	.040 (1e-14)
SSI Superficial	Pre	.667 (+.018)	.560 (+.044)
	Pre+intra	.688 (+.017)	.563 (+.042)
	Difference	.021 1e-16)	.003 (0.022)
SSI Deep Tissue	Pre	.660 (±.041)	.568 (±.069)
	Pre+intra	.678 (+.035)	.575 (+.064)
	Difference	.018 (1e-7)	.007 (0.3)
SSI Organ Space	Pre	.732 (±.026)	.657 (±.037)
	Pre+intra	.750 (±.022)	.699 (±.039)
	Difference	.021 (1e-16)	.042 (1e-16)
UTI	Pre	.665 (±.022)	.717 (±.031)
	Pre+intra	.672 (±.021)	.727 (±.032)
	Difference	.007 (1e-13)	.010 (5e-14)

External Comparison of the Significant Variables

Table 4 displays the number of bootstrap iterations in which the variable was selected for each complication at each site. We only list variables that were selected in at least 100 of the 200 bootstrap iterations for at least one outcome at one site.

Table 4 –

Frequency at which each intraoperative variable was selected (out of 200 bootstrap iteration)

	PNA		SSI Superf.		SSI Deep T.		SSI Organ		Sepsis		Septic Shock		UTI
	MC	FV	MC	FV	MC	FV	MC	FV	MC	FV	MC	FV	MC	FV
Calcium Ion	7	17	31	10	62	9	124	66	93	38	7	41	2	94
CO2	120	18	1	20	19	3	4	24	80	5	1	2	0	0
Glucose	8	1	16	0	19	11	160	0	105	0	115	36	66	0
Lactate	158	144	5	21	1	1	43	27	29	23	28	163	7	1
MCV	5	153	27	6	48	64	41	16	10	41	25	5	0	2
PO2	42	90	6	17	24	63	11	192	12	95	40	12	4	110
CVP	11	8	136	12	43	40	122	34	30	8	46	5	31	137
Isoflurane Exp.	1	0	71	1	4	1	127	61	190	0	0	0	0	1
FIO2	115	6	1	13	12	0	53	1	1	5	6	120	9	8
PEEP	16	49	200	107	141	2	136	53	156	4	86	7	75	129
Pulse	31	88	173	22	28	13	167	63	197	142	152	0	4	9
Steroid	22	32	5	36	115	50	3	8	1	8	7	0	106	6
Abx	162	6	72	4	136	0	147	8	88	0	190	0	178	13

Notes: (1) Only variables selected in at least 100 iterations are displayed. (2) Abbreviations; MC: Mayo Clinic; FV: Fairview Health Services; MCV: mean corpuscular volume; CVP: central venous pressure; FIO2: Fraction of Inspired Oxygen; PEEP: Positive End Expiratory Pressure; PO2: Partial pressure of oxygen; Abx: Antibiotic use.

Table 3 offers a summary of the information in Table 4. It lists the most important intraoperative variables that are common between the two sites for each outcome.

Table 3 –

Selected variables

	Significant intraoperaitive variables
Outcome	Mayo Clinic	Fairview
PNA	antibiotic use, lactate, CO2, FIO2	Lactate, MCV, pulse, SpO2
Sepsis	Glucose, isoflurane expired, PEEP, pulse	PO2, RBC, pulse, hematocrit
Septic Shock	Antibiotic use, glucose, WBC, pulse, PEEP	Lactate, FIO2
SSI Superficial	CVP, PEEP, pulse	PEEP
SSI Deep Tissue	Antibiotic use, steroid use, PEEP	PO2
SSI Organ Space	Antibiotic use, bicarb, Ca, glucose, CVP, isoflurane expired	Hemoglobin, PH arterial, PO2, pulse
UTI	Antibiotic use, steroid use	Ca, PO2, CVP, PEEP

Abbreviations: FIO2: Fraction of Inspired Oxygen; PEEP: Positive End Expiratory Pressure; PO2: Partial pressure of oxygen; WBC: white blood cell [count]; CVP: central venous pressure; Ca: calcium ion in the blood; MCV: mean corpuscular volume; SpO2: blood oxygen saturation; RBC: red blood cell [count].

Discussion

We view EHR-integrated perioperative clinical decision support as a key avenue towards reducing postoperative complications or towards better anticipating and managing them. Central to such systems are accurate risk models with the ability to produce risk estimates on demand based on contemporaneous data, including intraoperative data. In this work, we sought to answer the following two questions: (i) Do intraoperative data elements improve our ability to predict postoperative complications? and (ii) Are these models specific to institutions or do they describe possible physiological phenomena that are institution-independently predictive of postoperative complications?

Regarding the first question, we found that the inclusion of intraoperative variables led to statistically significant improvements in the risk models’ predictive ability for five of the seven infectious complications at both sites. These complications were pneumonia, sepsis, septic shock, organ space SSI, and urinary tract infection. At the Mayo Clinic site, the inclusion of the intraoperative variables led to statistically significant (albeit not necessarily clinically relevant) improvements for all complications, likely due to the much larger sample size.

The lack of statistically significant improvement for the other two SSI types, superficial and deep tissue, is unsurprising. These diagnoses have more ambiguity and our previous work on detecting SSI retrospectively also suggests that predicting superficial and deep tissue SSI is difficult [17, 18].

Regarding the second question, we found that some of the intraoperative variables overlap across the two sites for all four of the complications in which intraoperative variables improved the predictive performance. In the cases of pneumonia, sepsis, and organ-space SSI, some of the variables (lac-tate, pulse) matched exactly, while others referred to a shared concept. For example, FIO2 and SpO2 are related to the sufficiency of oxygen in the bloodstream: the first one is the ventilator setting (to supply the oxygen into the bloodstream) while the second one is the direct measurement of oxygen saturation. For sepsis, partial pressure from oxygen and PEEP have an analogous relationship; and for organ space SSI, bicarbonate and PH both measure the acidity of blood. The fact that many of the significant variables are common across the two sites suggests that these variables can be related to the physiological process(es) underlying the outcomes.

Conclusion

Intraoperative variables were found to statistically significantly improve the performance of 30-day postoperative risk models for five of the seven infectious complications at both sites. There was considerable overlap in the significant intraoperative variables across the two sites and the overlapping variables were related to lactate, acidity, and blood oxygen.