An Ounce of Prediction is Worth a Pound of Cure: Risk Calculators in Breast Reconstruction.

Preoperative risk calculators provide individualized risk assessment and stratification for surgical patients. Recently, several general surgery-derived models have been applied to the plastic surgery patient population, and several plastic surgery-specific calculators have been developed. In this scoping review, the authors aimed to identify and critically appraise risk calculators implemented in postmastectomy breast reconstruction.
Methods: A systematic review of the literature was conducted. Included studies described the development of a novel risk calculator, or validation of an existing calculator, in postmastectomy breast reconstruction.
Results: In total, 4641 studies met criteria for title and abstract screening. Forty-seven were eligible for full-text review, and 28 met final inclusion criteria. The most common risk calculators included the Breast Reconstruction Risk Assessment score (n = 6 studies), modified frailty index (n = 3), Caprini score (n = 3), and ACS NSQIP calculator (n = 2). Calculators were applied to institutional data (n = 17), NSQIP (n = 6), and Tracking Outcomes in Plastic Surgery (n = 1) databases. Predicted outcomes included general postoperative complications (n = 17), venous thromboembolism/pulmonary embolism (n = 4), infection (n = 2), and patient reported outcomes (n = 2). Model accuracy was reported in 18 studies, and it varied significantly (accurate risk calculator 0.49-0.85). Conclusions: This is the first study to provide a systematic review of available risk calculators for breast reconstruction. Models vary significantly in their statistical basis, predicted outcomes, and overall accuracy. Risk calculators are valuable tools that may aid in individualized risk assessments, preoperative counseling, and expectation management in breast reconstruction.

Takeaways

Question: How have risk calculators been implemented in postmastectomy breast reconstruction?

Findings: This scoping review identified several publicly available automated risk calculators; however, the utility of these tools may be limited to a certain subset of breast reconstruction patients. More recently, machine-learning-based calculators have been shown to provide more accurate and generalizable predictions.

Meaning: In the future, automated risk calculators may become a routine part of each preoperative encounter and help identify high-risk patients, aid preoperative discussions, decrease costs, and improve patient outcomes.

INTRODUCTION

In the age of personalized medicine, machine learning, and artificial intelligence, automated individualized preoperative risk assessment has become a reality. In recent years, risk calculators have emerged providing evidence-based risk stratification for surgical patients. In the fields of neurosurgery and orthopedics, automated risk calculators have been shown to accurately predict the likelihood of revision after total knee arthroplasty, prosthetic joint infection, and even mortality.[1,2] These tools are particularly efficacious in elective surgery, where opportunities exist to identify and address modifiable risk factors before operative intervention. Although risk calculators have become increasingly prevalent in general surgery and several surgical subspecialties, they are relatively scarce in the plastic surgery literature.

Several models developed and validated in the general surgery population have been applied to plastic surgery patients. For example, the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) calculator has been used to predict risk of postoperative complications in extremity reconstruction following sarcoma resection.[3] Although rare, several plastic surgery–specific calculators do exist. In a 2018 study, risk calculators were shown to successfully predict surgical site infection, reoperation, and wound dehiscence after abdominoplasty in a retrospective national cohort.[4] Although these findings are promising, the statistical basis, indication for use, and accuracy of each calculator are highly variable and deserve scrutiny before application in clinical practice.

Postmastectomy breast reconstruction has also garnered the attention of risk calculator developers. Several risk calculators have been developed or implemented in this patient population including the Breast Reconstruction Risk Assessment (BRA) score, ACS-NSQIP calculator, modified frailty index, and many others.[5-8] An understanding of the indications, efficacy, and statistical basis of the available risk calculators for this patient population is essential for implementation, and for the development of novel risk calculators in the future. In this scoping review of the literature, the authors aimed to identify and critically appraise risk calculators implemented in postmastectomy breast reconstruction, reviewing the statistical bases, accuracy, and indications for each tool.

METHODS

Search Methodology and Selection Criteria

A review of the literature was conducted to identify articles that described the development of a novel risk calculator or validation of an existing calculator in postmastectomy breast reconstruction patients. A scoping review of the Medline, Embase, and Cochrane databases was conducted. A comprehensive query was developed with assistance from a professional research and education librarian at the Duke University Medical Center Library. Specific search terms and methodology can be found in Supplemental Digital Content 1. (See appendix, Supplemental Digital Content 1, which displays the full electronic search strategy. http://links.lww.com/PRSGO/C31.)

Selection criteria were defined before data collection. To be included in this review, studies were required to describe the development or implementation of a risk calculator in postmastectomy breast reconstruction patients (either autologous or implant-based). Oncologic breast studies without reconstruction and studies detailing risk calculators used to predict oncologic outcomes were excluded. Of note, numerous breast reconstruction studies described individual factors or groups of factors that were predictive of a complication but did not provide a discrete scoring system or risk calculator. These studies were also excluded. Included studies were uploaded into the Covidence online systematic review platform and abstract screening was performed by two authors (N.O. and S.B.). The screening process was conducted in accordance with the 2009 PRISMA Statement guidelines, and a detailed description of search and screening methodology can be found in Figure 1.

Fig. 1.

PRISMA diagram summarizing search.

Data Extraction and Analysis

Data extracted from included studies were as follows: author, journal, date of publication, risk calculator (name), dataset used for validation/development, statistical basis of calculator, model accuracy (area under the curve, c-statistic, sensitivity/specificity, etc.), outcomes predicted, and publicly available links to calculators.

RESULTS

A total of 10,867 articles were identified in our initial search of the Medline (PubMed), Embase, and Cochrane Library databases. After removal of 6226 duplicates, 4641 studies remained and were eligible for title and abstract screening. Each study was reviewed for inclusion by two independent authors (N.O. and S.B) and disagreements were resolved through consensus discussion. Ultimately, 47 studies were eligible for full-text review, of which 28 met final inclusion criteria. Fifteen were excluded due to “wrong study design,” three due to “wrong patient population” and one due to “abstract or incomplete study.”

The included studies were published between 2013 and 2021. The most frequent journals of publication were Plastic and Reconstructive Surgery (n = 5 studies), Journal of Plastic, Reconstructive, and Aesthetic Surgery (n = 5), Annals of Plastic Surgery (n = 4), Plastic and Reconstructive Surgery Global Open (n = 3), and Journal of the American College of Surgeons (n = 3). Institutional datasets were used most frequently for development and/or validation of the described risk calculator (72.4%), followed by NSQIP data (21.4%). The majority of described risk calculators were based on traditional statistical models such as multivariate logistic regression analysis (57.1%), while others utilized simple risk scores (28.6%). Three studies (10.7%) utilized machine learning–based algorithms to develop their predictive models.

The most frequently identified risk calculators were the BRA score (n = 6 studies), Caprini score (n = 3 studies), Modified Frailty Index (n = 3 studies), and the ACS/NSQIP calculator (n = 2 studies). The remaining (n = 14 studies) included unnamed calculators reported by individual studies. Risk calculators and simple risk scores were most commonly used to predict “postoperative complications” (61% of studies), thromboembolism (14%), surgical site infection (7%), patient reported outcomes (7%), and flap failure (2%). These data are displayed visually in Figure 2. Calculator accuracy was most commonly measured with the Area Under the Receiver Operating Curve (AUC) or c-statistic. The AUC is a common performance measure used in machine learning that is based on the ability to reliably distinguish between two groups. The AUC ranges for the most commonly reported risk calculators were 0.49–0.71 (BRA score; six studies), 0.55–0.62 (ACS/NSQIP calculator; 2 studies), 0.7 (modified frailty index; one study). Studies assessing the Caprini score did not report the AUC. A comprehensive list of all reviewed studies and available accuracy assessments via AUC/c-statistic can be found in Table 1.

Fig. 2.

Outcomes predicted by risk calculators/risk scores in each individual study included in review. Percentages represent percentages of studies describing a risk tool used to predict a specific outcome.

Table 1.

Summary of Risk Calculators Detailed by Each Included Study, including Dataset Used for Development, Model Name, Outcome Predicted, and Accuracy (AUC)

Study	Study	Dataset	Risk Calculator	Outcomes Predicted	AUC/c-Statistic
1	Blough et al[9]	Institutional data	BRA score XL*	Postoperative complications	0.64–0.739
2	Casella et al[10]	Institutional data	PBRA score****	Postoperative complications	NA
3	Cuccolo et al[11]	ACS-NSQIP	Modified frailty index	Postoperative complications	NA
4	Enajat et al[12]	Institutional data	Unnamed	VTE	0.587–0.814
5	Fischer et al[13]	ACS-NSQIP	IBRRAS***	Postoperative complications	NA
6	Frey et al[14]	Institutional data	Unnamed	Postoperative complications	0.668
7	Hansen et al[9]	Institutional data	BRA score XL*	Postoperative complications	0.712
8	Hermiz et al[15]	ACS-NSQIP	Modified frailty index	Postoperative complications	0.7
9	Kato et al[16]	Institutional data	Unnamed	SSI	0.734
10	Khavanin et al[17]	Institutional data	BRA score XL*	Postoperative complications	0.69–0.78
11	Kim et al[5]	ACS-NSQIP	Unnamed	SSI	0.682
12	Kim et al[18]	TOPS	BRA score XL*	Postoperative complications	0.603–0.677
13	Kim et al[19]	Institutional data	Caprini score	VTE	NA
14	Martin et al[20]	Institutional data	BRA score*	Postoperative complications	0.54–0.75
15	Modarressi et al[21]	Institutional data	Caprini score	VTE	NA
16	Moss et al[22]	ACS-NSQIP	Modified frailty index	Postoperative complications	NA
17	Myung et al[23]	Institutional data	Unnamed	Donor site complications	0.81
18	Nelson et al[24]	Institutional data	Unnamed	Postoperative complications	NA
19	O’Neill et al[25]	Institutional data	ACS-NSQIP calculator**	Postoperative complications	0.538–0.548
20	O’Neill et al[6]	ACS-NSQIP	ACS-NSQIP calculator**	Postoperative complications	0.62
21	O’Neill et al[26]	Institutional data	BRA score*	Postoperative complications	0.49–0.59
22	O’Neill et al[27]	Institutional data	Machine learning prediction model for flap failure in microvascular breast reconstruction	Flap failure	0.67
23	Park et al[7]	Institutional data	Unnamed	Postoperative complications	0.731
24	Park et al[7]	Institutional data	Samsung Medical Center Risk score for bulge/hernia	Donor site complications	NA
25	Pfob et al[28]	Institutional data	Unnamed	Pros (BREAST-Q)	0.81–0.83
26	Roy et al[8]	Institutional data	Unnamed	Postoperative complications	NA
27	Sidey-Gibbons et al[29]	Institutional data	Unnamed	Pros (financial burden)	0.85
28	Subichin et al[30]	Institutional data	Caprini score	VTE	NA

IBRRAS, Immediate Breast Reconstruction Risk Assessment Score; TOPS, Tracking Outcomes in Plastic Surgery.

*Breast Reconstruction Risk Assessment Score; **American College of Surgeons National Surgical Quality Improvement Program; ***Immediate Breast Reconstruction Risk Assessment Score; ****The Prepectoral Breast Reconstruction Assessment Score.

DISCUSSION

Accurate, individualized, risk stratification tools provide tremendous value to the preoperative breast reconstruction patient. Complications after breast reconstructive procedures are relatively commonplace and may impact the final aesthetic and reconstructive outcome, or potentially even influence the timing of adjuvant breast cancer therapy.[31,32] Compared with a population-based assessment, automated risk calculators provide a more nuanced view of an individual patient’s risk and may help guide preoperative discussions and treatment decisions. In this study, we review the existing risk calculators in the breast reconstruction literature, identify several traditional statistical models and simple risk score tools, and describe promising advanced machine learning–based tools currently in development.

A Primer on Risk Calculator Statistics

To contextualize the results of included studies, a brief discussion of the common statistical tests used to analyze predictive models is indicated. Model accuracy is assessed with various statistical tests, the most common of which is the concordance statistic (c-statistic). This test determines how well a model is able to distinguish between patients who did and did not experience a particular outcome.[33] For binary variables, c is equal to AUC, which is why several studies in our review used these two terms interchangeably. A “perfect” or ideal model has AUC of 1.0, which maximizes true positives and minimizes false positives. The closer a model’s c-statistic is to 0.5, the worse the discriminatory capability.[11,26,33] The Brier score was also reported by several studies in this review to assess model accuracy. This score is a representation of the difference between the actual and predicted outcome for each patient. In this case, a perfect model has a score of zero, indicating no difference between the two.[17,33,34] Finally, the Hosmer-Lemeshow goodness of fit test was implemented in several studies to assess model calibration. This is determined by the agreement between observed and predicted event rates in a population. In a well-calibrated model, a nonsignificant difference exists between the two.[33] Taken together, these three tests are a good representation of model calibration, discrimination, and accuracy.[17]

The Breast Reconstruction Risk Assessment Score

The Breast Reconstruction Risk Assessment score, or BRA score, was the calculator most frequently implemented by studies identified in this review. This is a publicly available multiple logistic regression–based calculator designed to predict a variety of complications in postmastectomy breast reconstruction patients. In 2014, Kim et al described the development of the BRA score utilizing ACS/NSQIP data to predict 30-day surgical site infection risk after immediate autologous or implant-based reconstruction. The model included ten NSQIP variables such as reconstructive modality, age, and ASA class, as well as several medical comorbidities. The authors report an AUC of 0.685 and a nonsignificant Hosmer-Lemeshow score, indicating good model calibration. This study is described as a “proof of concept” by the authors, and in the following years several modifications and additions have been made to the BRA score.[5]

In 2015, the Tracking Outcomes in Plastic Surgery database was used to expand the predictive capabilities of this tool. In addition to prediction of SSI, the calculator could now be used to predict seroma, dehiscence, flap failure, and implant explantation with an AUC of 0.644.[18] Additional modifications were made in 2018 when institutional data from Northwestern University was implemented to further enhance calculator functionality. This included the addition of radiation therapy as a predictive variable, and the expansion of complication risk estimates out to one year. This was termed the “BRA score XL,” and was reported to have improved accuracy (AUC 0.7) as well as enhanced discriminatory power (Brier score 0.15).[9] These studies describe the development of a novel risk calculator with accuracy comparable to commonly used clinical risk scores in medicine; however, external validation studies have demonstrated mixed results.

In the model-development studies previously described, the calculator was tested on the same data from which it was developed. Several steps were taken to account for this potential source of bias. For example, Kim et al performed a bootstrapping validation to estimate model accuracy on outside data and found model accuracy to be nearly identical (AUC 0.681 versus 0.685).[18,34] However, true external validation is necessary to determine a model’s accuracy. In a 2017 external validation study of 850 immediate subpectoral implant reconstructions, Khavanian et al found that the BRA score accurately predicted surgical site infection risk, but tended to over-estimate a patient’s risk of explantation and seroma.[17] However, two more recent validation studies found the BRA score to have limited discriminatory capability in predicting flap failure in autologous reconstruction (c-statistic 0.51) and overall complication occurrence in prepectoral expander-based reconstruction (c-statistic <0.6). These studies highlight the limitations of the BRA score and BRA score XL and indicate that this tool is likely most valuable in specific circumstances such as prediction of SSI in immediate, subpectoral implant-based reconstruction. Despite these limitations, this tool appears to be the most robust and well-studied breast reconstruction–specific surgical risk calculator publicly available, and is certainly a step in the right direction. Additional modifications of the BRA score including data from additional autologous reconstruction patients would further improve its utility.

ACS/NSQIP Calculator

The ACS-NSQIP surgical risk calculator was implemented by two studies identified in our review. The National Surgical Quality Improvement Program is a surgical outcomes database created by the American College of Surgeons. This dataset is generated from over 500 participating institutions collecting standardized surgical patient data, including 30-day surgical outcomes.[35] This data have been used to create a logistic regression–based universal risk calculator that is publicly available, user friendly, and has been previously validated in the colorectal and general surgical literature.[6,36,37] However, validation studies examining the predictive utility of this tool in the surgical subspecialties have yielded mixed results.[6,38,39] In two individual studies identified in this review, O’Neill et al examined the predictive capacity of the ACS/NSQIP calculator in both autologous and expander/implant-based breast reconstruction patients.[6,26] In autologous reconstruction, the authors found that this tool was relatively accurate in predicting the proportion of patients within the cohort that would develop a complication (14% predicted versus 11.1% actual), but was very poor at distinguishing individual patients who were likely to develop a complication (c-statistic 0.55).[26] In a separate 2016 study, the authors found that the tool performed even worse in the implant-based reconstruction group, with an actual and predicted complication rate of 16.2% and 5.2%, respectively, and a c-statistic of 0.62.[6] A possible explanation for the ACS/NSQIP calculator’s poor performance in this patient population is the variables considered by this tool. Several of the variables considered by the calculator (including hemodialysis, heart failure, and ventilator dependence) are largely irrelevant to the majority of breast reconstruction patients, while critical factors such as radiation therapy, reconstructive timing, and mastectomy weight are not considered. For this reason, the authors of these two studies advocate for the development of plastic surgery-specific risk calculators.[6,26]

Machine Learning–based Risk Calculators

Machine learning modalities such as decision tree classification and neural networks are well equipped for interpreting large amounts of raw data, and are capable of leveraging nonlinear relationships between variables to improve prediction accuracy. Several studies in this review described the development of novel machine-learning-based risk calculators for breast reconstruction. In a 2016 study, O’Neill et al developed a decision tree classification model to predict flap loss in abdominally based autologous breast reconstruction. Although the predictive capabilities were limited (c-statistic 0.67), this tool was the first of its kind and served as a proof of concept for future studies.[27] More recently, a neural network model was shown to accurately predict abdominal donor site complications after autologous breast reconstruction in a 2021 retrospective cohort study from Seoul, Korea.[23] Although a publicly accessible machine learning–based calculator has yet to become available, these early indicators of improved accuracy and prediction capability suggest that this technology may play a major role in the risk calculators of the future.

Although the majority of studies identified implemented these tools in predicting surgical complications, two unique studies utilized machine learning models to predict patient-reported outcomes after breast reconstruction. In a 2021 study, Gibbons et al used decision tree classification and neural network models to predict a patient’s likelihood of experiencing financial hardship as a result of breast reconstruction.[29] The authors created a highly accurate risk calculator (AUC 0.85) and identified neoadjuvant chemotherapy and autologous reconstruction as significant predictors of financial toxicity.[29] In a separate multiinstitutional study, three machine learning–based models were developed to predict a patient’s likelihood of being satisfied with their reconstruction 1 year after surgery.[28] Again, these models were found to be highly accurate in predicting significant improvements in BREAST-Q scores postoperatively (AUC 0.81–0.86). In the future, tools like these could be implemented in an electronic medical record to help identify patients at risk of financial hardship after surgery, or aid preoperative discussions regarding likelihood of satisfaction with the available surgical options.

Risk Scores

Finally, several studies in this review assessed the utility of specific “risk scores” in the breast reconstruction population. While an automated risk calculator uses individual patient characteristics to provide predictive estimates, simple risk scores subcategorize patients into a group based on specific criteria. Although these tools may not provide the same individualized risk estimate as an automated calculator, they may help identify high-risk patient subgroups. The Caprini score was developed in 1991 and modified in 2005 to predict a patient’s likelihood of developing venous thromboembolism (VTE) after a surgical procedure.[21] Patients are grouped into five risk categories based on several demographic characteristics, medical comorbidities, and procedural specifics, and anticoagulation recommendations are made based on which risk category a patient is stratified into. Studies analyzing the Caprini score found that the vast majority of breast reconstruction patients fell into the high or highest Caprini risk group (87%–92%).[21,40] Overall, the predictive capacity of the Caprini score in these studies was variable. Subachin et al found that the Caprini score was not a significant predictor of VTE occurrence, while the method of reconstruction was.[40] Conversely, Modarressi et al found that the Caprini score value was the only statistically significant difference between patients who did and did not develop VTE.[21] In addition to these mixed results, interpretation of these studies is further complicated by the lack of a viable control group. The Caprini score was designed to predict VTE risk in the absence of intervention; however, patients in these studies were invariably receiving either mechanical or pharmacologic prophylaxis postoperatively. Further investigation is required to determine if the Caprini Score is an accurate and useful tool in the breast reconstruction population.

Implications and Future Directions

Although several risk calculators for breast reconstruction exist, they tend to be limited in their scope or unreliable. The machine learning–based predictive models described in this review are still in their infancy, but currently show tremendous promise for future application. Ultimately, these tools have the potential to revolutionize preoperative patient counseling and optimization in breast reconstruction. As viable preoperative risk calculators improve and become widely available, they will undoubtedly become an integral part of surgical practice, but until then, continued investment and inquiry into optimizing these tools is necessary to maximize patient safety.

Limitations

Several limitations of this study exist. First, while a comprehensive search was conducted of the Medline, Embase, and Cochrane Library databases, there is certainly a possibility that additional studies of risk calculator applications in breast reconstruction were not captured by this search. Additionally, as machine learning–based calculators are an emerging phenomenon in the plastic surgery literature, it is possible that additional studies have been published since the completion of our search, before publication. Another limitation is the lack of aggregate accuracy assessment of individual calculators, or direct statistical comparison between calculators. Although an average c-statistic for each tool can easily be calculated based on our data, heterogeneity across studies in terms of outcomes predicted, reconstructive modality assessed, and overall study design would likely make this value irrelevant. Similar differences across studies also precluded a head-to-head statistical comparison of each individual risk calculator. Finally, the overall quality of evidence of each study is variable. Several studies were based on relatively small single institutional datasets and lacked external validation. Although it is valuable to review this literature to determine the current state of risk calculators in breast reconstruction, the findings of each individual study should be taken in context of their level of evidence.

CONCLUSIONS

Individualized preoperative risk assessment is an invaluable tool that has recently become a reality due to the advent of automated risk calculators. In complex surgical procedures like breast reconstruction, the ability to accurately predict a patient’s likelihood of postoperative complication would have significant surgical outcome, patient satisfaction, and healthcare cost implications. In this review‚ we critically appraise the available risk calculators found in the breast reconstruction literature and provide a summary of the statistical basis, indication, and accuracy of each of these tools. Accessible and user-friendly automated risk calculators such as the BRA score and the ACS/NSQIP do exist; however, their utility may be limited to a certain subset of breast reconstruction patients. Machine learning–based risk calculators are in development, and though not yet publicly available, have shown promising results in their ability to predict both postoperative complications and patient-reported outcomes. In the future, automated risk calculators may become a routine part of each preoperative encounter and help identify high-risk patients, aid preoperative discussions, decrease costs, and improve patient outcomes.

ACKNOWLEDGMENT

This study is exempt from institutional review board evaluation due to the nature of the study design.