The PERCEIVE quantitative study: PrEdiction of Risk and Communication of outcome following major lower-limb amputation: protocol for a collaboratiVE study.

BACKGROUND: Accurate prediction of outcomes following surgery with high morbidity and mortality rates is essential for informed shared decision-making between patients and clinicians. It is unknown how accurately healthcare professionals predict outcomes following major lower-limb amputation (MLLA). Several MLLA outcome-prediction tools have been developed. These could be valuable in clinical practice, but most require validation in independent cohorts before routine clinical use can be recommended. The primary aim of this study is to evaluate the accuracy of healthcare professionals' predictions of outcomes in adult patients undergoing MLLA for complications of chronic limb-threatening ischaemia (CLTI) or diabetes. Secondary aims include the validation of existing outcome-prediction tools.
METHOD: This study is an international, multicentre prospective observational study including adult patients undergoing a primary MLLA for CLTI or diabetes. Healthcare professionals' accuracy in predicting outcomes at 30-days (death, morbidity and MLLA revision) and 1-year (death, MLLA revision and ambulation) will be evaluated. Sixteen existing outcome-prediction tools specific to MLLA will be examined for validity. Data collection began on 1 October 2020; the end of follow-up will be 1 May 2022. The C-statistic, Hosmer-Lemeshow test, reclassification tables and Brier score will be used to evaluate the predictive performance of healthcare professionals and prediction tools, respectively. STUDY REGISTRATION AND DISSEMINATION: This study will be registered locally at each centre in accordance with local policies before commencing data collection, overseen by local clinician leads. Results will be disseminated to all centres, and any subsequent presentation(s) and/or publication(s) will follow a collaborative co-authorship model.

Introduction

Major lower-limb amputation (MLLA) is a life-changing event with significant risk of morbidity and death,. Poorly informed decision-making around MLLA can dramatically reduce quality of life and can be very costly. Sometimes patients who, in retrospect, are in the last few months of their life proceed with an amputation, a choice which is often regretted by surviving relatives. In contrast, in some select patients (younger, often diabetic, patients with chronic foot wounds which drastically limit mobility), an ‘early’ MLLA can potentially provide improved ambulation on a limb prosthesis associated with an improved quality of life.

Shared decision-making involving the patient, clinical team, and family or carers (if requested) is considered standard care,. Healthcare professionals estimate likely risks (including death, need for revision surgery, surgical morbidity) and benefits (including chance of surviving and ambulating), which are used to inform and facilitate decision-making. A recent systematic review of risk perception in surgery (in general) has shown that surgeons predict short-term clinical outcomes reasonably well but are poor at predicting longer-term outcomes. Risk-scoring tools that use patient data to estimate outcome generally outperform surgeons’ estimates, however no studies were identified which evaluated surgeons’ accuracy in predicting outcomes in the context of MLLA. A recent systematic review of prediction tools used to estimate outcomes following MLLA identified 16 tools and most studies were judged to be at high risk of bias. In addition, only a few tools were validated externally. It is an unfortunately common occurrence in the medical literature for outcome-prediction tools to be developed and not validated subsequently in an independent cohort or found to be inaccurate in patient populations other than the development population.

This study addresses a pertinent question to vascular practitioners; the UK James Lind Alliance Priority Setting Partnership’s foremost research priority for vascular surgery as identified by clinicians is: ‘What can be done to improve outcomes in CLTI [chronic limb-threatening ischaemia] (including how best to identify those who would benefit from revascularisation and those who would be best managed with primary amputation or palliation)?’.

The results of the quantitative analyses in this study will be explored in conjunction with results from the separately reported PERCEIVE (PrEdiction of Risk and Communication of outcome following major lower limb amputation—a collaboratiVE study) qualitative study (which will include interviews with healthcare professionals and patients) and will be triangulated to provide an overarching narrative of outcome prediction, decision-making and risk/benefit communication in MLLA.

The primary aim of the PERCEIVE quantitative study is to evaluate the accuracy of healthcare professionals’ (surgeons, anaesthetists, specialist physiotherapists and vascular nurse practitioners) predictions of short- and long-term outcomes for adult patients undergoing MLLA for chronic limb-threatening ischaemia (CLTI) or diabetes. The secondary objectives are to evaluate the accuracy of existing outcome-prediction tools in predicting short- and long-term outcomes in this patient cohort, and to explore differences in the predictive accuracy of prediction tools, and of healthcare professionals, between different geographical centres.

Methods

Design

PERCEIVE is an international, multicentre prospective observational cohort study coordinated by the Centre for Trials Research, Cardiff University and disseminated by the Vascular and Endovascular Research Network (VERN),. The collaborative methodology has been used successfully by VERN previously.

Setting

Any hospitals in Organisation for Economic Co-operation and Development upper and middle-income countries providing elective and/or emergency vascular surgery can participate. Many vascular services are based on a ‘hub and spoke’ model; the hub site may undertake data collection for spoke sites, without registering the spoke sites separately, if practical and congruent with local policies.

Participants

Adult patients undergoing MLLA for CLTI or diabetes (including patients who have previously undergone MLLA of the contralateral limb) are eligible for inclusion. Exclusion criteria are patients under the age of 18 years, those undergoing MLLA for causes other than CLTI or diabetes (such as trauma, cancer) and patients undergoing MLLA revision surgery at the same or higher level of amputation.

Primary outcome

The primary outcomes will be the predictive performance metrics of healthcare professionals’ predictions of the following outcomes: death, morbidity and MLLA revision at 30 days, and death, MLLA revision and ambulation at 1 year.

Death, morbidity and MLLA revision predictions will be provided by healthcare professionals before surgery as a percentage probability using either a visual analogue scale or a verbal rating scale, both from 0 to 100 per cent. Healthcare professionals who routinely use specific existing prediction tools to aid estimations of risk/benefit in their practice can give verbal/visual analogue scale predictions that are informed by those tools. Data on whether healthcare professionals used a prediction tool as an aid, and which tool(s), will be captured. Ambulatory predictions will be provided by healthcare professionals as categorization into: bedbound/chairbound; able to use wheelchair only; able to use a prosthesis to stand/transfer only (equivalent to Special Interest Group in Amputee Medicine (SIGAM) score B); and able to use a prosthesis for ambulating (equivalent to SIGAM score C or greater). Morbidity will be defined as a surgical complication meeting the criteria for Clavien–Dindo grade III or higher; these are detailed in , along with all other definitions. Surgical revision will be defined as a return to theatre for any of the following: evacuation of haematoma/control of haemorrhage, soft tissue revision, re-amputation at the same level and re-amputation to a higher level.

Secondary outcomes

Secondary outcomes will include the discriminatory, calibration and overall predictive performance of 16 existing outcome-prediction tools’ predictions of the following outcomes: death and morbidity at 30 days, and death, MLLA revision and ambulation at 1 year.

Details of the existing outcome-prediction tools and their respective outcomes are shown in .

Table 1

Details of outcome-prediction tools and their respective outcomes

Outcome (predicted by risk-scoring tool)	Risk-scoring tool
Death (30 day)	*Feinglass et al., 200118 *Nelson et al., 201219 Patterson et al., 201222 Easterlin et al., 201323 Jolissaint et al., 201924 Ambler et al., 202025
Death (1 year)	Tang et al., 200926 Norvell et al., 201927
Morbidity (30 day)	Wied et al., 201628 Ambler et al., 202025
MLLA revision (1 year)	Czerniecki et al., 201929
Ambulation (1 year) (defined according to SIGAM mobility grades)30	**Czerniecki et al., 201720 Bowrey et al., 201921

MLLA, major lower-limb amputation; SIGAM, Special Interest Group in Amputee Medicine.*Describe two risk-scoring tools (one for below knee amputations, one for above knee amputations).**Describe two risk-scoring tools (both predicting different a level of ambulation as an outcome).

*Feinglass et al., 2001

*Nelson et al., 2012

Patterson et al., 2012

Easterlin et al., 2013

Jolissaint et al., 2019

Ambler et al., 2020

Tang et al., 2009

Norvell et al., 2019

Wied et al., 2016

Ambler et al., 2020

**Czerniecki et al., 2017

Bowrey et al., 2019

Other secondary outcomes will be: rates of morbidity, and Clavien–Dindo grade of morbidity, at 30 days, rate of surgical-site infection at 30 days, rate of blood transfusion at 30 days, rate of COVID-19 infections at 30 days, time from procedure to surgical revision, time from procedure to death, rate of deaths attributable to COVID-19, rate of healthcare professional use of an existing outcome-prediction tool to aid predictions, and details of which outcome-prediction tools were used to aid predictions. Surgical-site infection will be defined according to the Centers for Disease Control and Prevention criteria.

Patient identification

Patients will be identified by any member of the local study team which will comprise one lead clinician and a maximum of seven other team members (including medical trainees or allied healthcare professionals). The local study team will be members of the patient’s normal clinical team. The resources used will include electronic theatre lists and current inpatient lists. Patient or disease registries will not be screened for eligible patients. Any member of the local study team may confirm eligibility. Any queries will be directed to the local lead clinician and non-resolution referred to the study coordinator.

Data collection

The PERCEIVE quantitative study launched on 1 October 2020. The initial data collection comprised collecting demographic, operative and prediction data; this continued until 1 May 2021. Participating centres could begin collecting data prospectively on consecutive patients undergoing MLLA that met the inclusion criteria on any date during this 7-month interval.

Follow-up data will be collected at 30 days and 1 year following MLLA. The follow-up period will end on 1 May 2022; all data must be returned to the study team by 1 June 2022.

Source data will be captured and uploaded electronically using a secure web application for building and managing online databases; Research Electronic Data Capture (REDCap),. It is encouraged that data will be uploaded directly to REDCap as close to the time of surgery as possible. Paper case report forms (CRFs) will be provided to centres to facilitate data capture when direct upload to REDCap is not possible at the time of surgery. No personally identifiable information will be collected. All variables that will be collected during the entire study period are listed in .

Data management

Cases uploaded to REDCap will be assigned an anonymous study-identification number automatically. Local teams will keep a secure database on-site that includes the local hospital-identification number and corresponding anonymous study-identification number. The lead clinician at each site will be responsible for ensuring data are only stored on site, that this is done securely, and CRFs are disposed of appropriately following upload of all follow-up data to REDCap.

All data uploaded to REDCap will be held securely until the end of the study period. The Centre for Trials Research, Cardiff University will be responsible for data cleaning and analysis.

Screening logs

The local lead clinician at each centre will be required to review (or delegate review of) UK National Vascular Registry (or equivalent) data at the end of their 7-month data-collection period to determine patient identification rates. Patients undergoing MLLA who have not had a preoperative prediction of outcomes made should still be included in the study; their inclusion will provide a larger, consecutive cohort of patients for the evaluation of existing outcome-prediction tools.

Data completeness and accuracy

Data completeness will be quantified following the initial data-collection period. Individual patient records with less than 95 per cent completeness of mandatory datapoints will be returned to the centre for completion of data collection; if this is not possible the patient will be excluded from analysis as per previous international collaborative studies,,. All centres will be required to validate data accuracy in 20 per cent of their uploaded cases (randomly selected); 25 per cent of datapoints (randomly selected) per case will be validated equating to 5 per cent of total datapoints captured in the study. Any centre reporting accuracy of less than 95 per cent will be required to validate a further 20 per cent of their cases, and the lead team member will be asked to investigate and report back to the PERCEIVE study management group. Data validation will be undertaken independently by a team member not involved in the initial data collection.

Based upon UK National Vascular Registry data, 25 centres collecting data during the study period will identify at least 400 to 500 MLLAs. By identifying at least 85 per cent of MLLAs performed, it is expected that centres will collect data on 340 to 425 patients.

Statistical analysis

The accuracy of predictions (by healthcare professionals and existing outcome-prediction tools) will be characterized and compared using various performance metrics. These will include measures of discrimination (receiver operating characteristic (ROC) curve and C-statistic), calibration (calibration slope and Hosmer–Lemeshow test), reclassification (reclassification table and net reclassification index) and overall performance (Brier score). The C-statistics of healthcare professionals/outcome-prediction tools will be compared using DeLong’s test.

Secondary exploratory subgroup analyses will use the above metrics as appropriate, focusing on visual comparisons of ROC curves and will evaluate the accuracy of predictions of different geographic areas (UK versus non-UK centres), different groups of clinicians (surgeons, anaesthetists, specialist physiotherapists and vascular nurse practitioners), clinicians who use outcome-prediction tools and those who do not, CLTI and diabetic patients, and different outcome-prediction tools.

The existing outcome-prediction tools were all developed prior to the COVID-19 pandemic, therefore further sensitivity analyses will incorporate analyses excluding patients positive for SARS-CoV-2 in the perioperative period to account for this confounding factor.

Missing data will be analysed and defined as missing completely at random (MCAR), missing at random (MAR), or not missing at random. Multiple imputation will be used where data missingness conforms to an MCAR or MAR pattern, when case-wise deletion sensitivity analysis will also be conducted. If variables required to calculate outcome predictions using existing prediction tools are systematically missing, predictive performance analysis will be limited to discriminatory performance only for that tool using ‘worst-case’ imputation.

Presentation of results

Descriptive summaries of baseline demographic and observed outcome data will be presented in tables. The results describing the accuracy of predictions (and validation of existing tools) will include graphical representations of discrimination and calibration: ROC curves and calibration slopes. Where possible, multiple ROC curves will be presented in the same graph to demonstrate differences in performance visually: this will be applicable to comparisons of different prediction tools and subgroup analyses.

Ethical and governance approval

The study protocol was approved as a service evaluation (Ref: SA/1188/20) by the Research and Development department of Aneurin Bevan University Health Board, Newport, UK, thus not requiring review by a UK Research Ethics Committee in accordance with the Health Research Authority’s online decision tool and Defining Research Table.

Each participating centre was required to submit the study protocol through the relevant local permission system before commencing data collection. Approval was obtained from the host care organization which considered local governance requirements and site feasibility.

Dissemination and authorship

A writing team, including those involved with the design, implementation and dissemination of this study, and those contributing to data analysis, will be responsible for both presentations and publications. For both presentations and publications, an inclusive authorship model will be used. Criteria to qualify for collaborative authorship are detailed in . Owing to the large number of prediction tools being evaluated (most of which predict short-term, 30-day outcomes), two separate manuscripts reporting results will be produced: one for 30-day outcomes and one for 1-year outcomes.

Discussion

The PERCEIVE quantitative study will provide valuable insight into the accuracy of healthcare professionals’ predictions of outcomes and assess the utility of existing outcome-prediction tools in this patient cohort. PERCEIVE will be the first study to evaluate healthcare professionals’ accuracy in predicting outcomes in MLLA surgery. The insight that will be gleaned from this study has the potential to improve risk/benefit communication with patients and their family or carers, leading to better-informed and shared decision-making. This study will also quantify how frequently healthcare professionals use existing outcome-prediction tools to aid decision-making in contemporaneous, real-world practice. Several of the outcome-prediction tools specific to MLLA described in the literature currently lack sufficient evidence of validation to support their routine use in clinical practice; despite this it is unknown whether clinicians are using these tools in practice.

The wider applicability of outcome-prediction tools is complex—there are several confounding factors such as differences in populations and medical practice, that influence their predictive performance between different patient cohorts. Stand-alone studies validating these tools externally in a specific cohort of patients do not fully address wider applicability. PERCEIVE aims to contribute to the much-needed body of evidence concerning the wider applicability of these tools by aiming to validate externally, and compare, 16 outcome-prediction tools prospectively.

The study has been disseminated by VERN via email contacts and social media; a method successfully used in previous studies delivered by VERN. Based on current interest, it is predicted that over 40 centres will contribute data to the study. The large number of centres increases the generalizability of the findings and will allow identification of variation in practice. Case ascertainment should be high since all centres are required to use screening logs to ensure cases are not missed from initial data collection. Studies evaluating surgeon accuracy in predicting outcomes and the accuracy of outcome-prediction tools in other surgical procedures and specialties have relied heavily on discriminatory performance (using the C-statistic), often neglecting measures of calibration, reclassification and overall performance. By including these additional performance measures, this evaluation of healthcare professionals’ accuracy in predicting outcomes and existing outcome-prediction tools’ performance should be robust. Geographical variation in practice (including volume of procedures), level of clinician seniority, and profession are factors that may influence healthcare professionals’ accuracy in predicting outcomes. Factors that may influence the accuracy of prediction models include validation in a geographical region different to that in which they were developed and confounding from variables ‘unknown’ to the model, such as COVID-19 status. The planned subgroup analyses aim to explore and quantify these potential biases.

There are limitations to the methodology of the PERCEIVE quantitative study. Firstly, healthcare professionals are not mandated to provide predictions of outcomes; engagement could therefore vary between participating centres, potentially introducing participation bias to this specific result. Similarly, engagement may vary between different groups of healthcare professionals and level of seniority, again, potentially introducing participation bias. Despite designing a prospective study that will capture all datapoints needed to validate the existing outcome-prediction tools, there is potential for some data to be missing, and some outcome-prediction tools include variables that may not be collected routinely at all centres. The planned subgroup analyses will probably yield lower estimates of accuracy with less precision owing to the inherent smaller sample size; this should be considered when interpreting these results. Some patients for whom MLLA is considered will receive alternative treatment (such as palliation) and not be eligible or able to be included in this study. The accuracy of predictions for these patients not captured by the study cannot be evaluated; for this reason, regional/national differences in practice may introduce bias. Similarly, variation in practice may lead to a data set that is overweighted to one area/country, reducing generalizability of the results.

The evaluation of outcome-prediction tools will be subject to confounding as a result of the COVID-19 pandemic: mortality and pulmonary morbidity rates in patients with perioperative SARS-CoV-2 are high, the existing tools do not account for this variable. A recent study has shown that the COVID-19 pandemic has resulted in a drastic change in practice within vascular services worldwide, with most centres offering a greater proportion of amputation or palliation compared with revascularization, a finding that is congruent with the UK National Vascular Registry’s short report of UK vascular practice during the COVID-19 pandemic. Additionally, patient outcomes following vascular surgery were worse during the COVID-19 pandemic whether patients had evidence of SARS-CoV-2 infection or not.

Awareness of strengths and limitations in predicting outcomes as healthcare professionals and knowledge of the utility of outcome-prediction tools are key to improving shared decision-making, and ultimately overall patient care, in high-risk surgery such as MLLA. Further research should aim to contextualize the findings of this study by exploring the decision-making process and risk/benefit communication with patients. It is anticipated that this study will provide much-needed evidence and its success will further the ever-growing network of vascular collaborative researchers.

Collaborators

The Vascular and Endovascular Research Network: G. K. Ambler (University of Bristol, Bristol, UK); R. Benson (University of Birmingham, Birmingham, UK); D. C. Bosanquet (Royal Gwent Hospital, Newport, UK); N. Dattani (Queen Elizabeth Hospital Birmingham, Birmingham, UK); G. Dovell (University of Bristol, Bristol, UK); R. Forsythe (University of Edinburgh, Edinburgh, UK); B. L. Gwilym (Royal Gwent Hospital, Newport, UK); L. Hitchman (Hull York Medical School, Hull, UK); S. Nandhra (Newcastle University, Newcastle, UK); S. Onida (Imperial College Healthcare NHS Trust, London, UK); A. Saratzis (University of Leicester Department of Cardiovascular Sciences, Leicester, UK); J. Shalhoub (Imperial College Healthcare NHS Trust, London, UK).

Funding

The study was awarded funding by the Research for Patient and Public Benefit (RfPPB) programme, Health and Care Research Wales (Ref: RfPPB-19–1642).

Supplementary material

Supplementary material is available at BJS Open online.

Click here for additional data file.