External Validation and Modification of a Predictive Model for Acute Postsurgical Pain at Home After Day Surgery.

OBJECTIVES: In 2009, Gramke and colleagues have described predictive factors to preoperatively detect those at risk for moderate to severe acute postsurgical pain (APSP) after day surgery. The aim of the present study is to externally validate this initial model and to improve and internally validate a modified version of this model.
MATERIALS AND METHODS: Elective patients scheduled for day surgery were prospectively enrolled from November 2008 to April 2010. Model discrimination was quantified using the area under the receiver operating characteristic curve (AUC). Model calibration was assessed by visual inspection of the calibration plot. Subsequently, we modified (different assignment of type of surgery, different cutoff for moderate to severe APSP, continuous of dichotomized variables and testing of additional variables) and internally validated this model by standard bootstrapping techniques.
RESULTS: A total of 1118 patients were included. The AUC for the original model was 0.81 in the derivation data set and 0.72 in our validation data set. The model showed poorly calibrated risk predictions. The AUC of the modified model was 0.82 (optimism-corrected AUC=0.78). This modified model showed good calibration.
CONCLUSIONS: The original prediction model of Gramke and colleagues performed insufficiently on our cohort of outpatients with respect to discrimination and calibration. Internal validation of a modified model shows promising results. In this model, preoperative pain, patient derived expected pain, and different types of surgery are the strongest predictors of moderate to severe APSP after day surgery.

Despite increased awareness and improvements in postoperative pain management over the last decades, the prevalence of outpatients reporting moderate to severe acute postsurgical pain (APSP) at home still remains high and varies from 9% to 40%.1–5 Particularly in the ambulatory setting, good postoperative analgesia is challenging because patients are responsible for controlling their pain at home by themselves6 and the types of analgesics (ie, no strong opioids) as well as the route of administration (ie, no epidural, intravenous, subcutaneous or intramuscular route) is limited compared with the inpatient setting.

Obviously, identification of patients at increased risk for APSP provides new opportunities: Tailored pain therapy to specific patient needs, assistance with coping and planned overnight stay can prevent the development of prolonged moderate to severe pain. Therefore, Gramke et al6 identified predictive factors for the development of moderate to severe APSP after day surgery. This model, however, was not validated in a new data set.

Before considering use and implementation of a prediction model, the generalizability of the model needs to be evaluated in a new population by external validation.7 External validation may be performed by either (partly) the same authors or by completely different teams.8 Furthermore, the data set can be retrieved either in the same center (ie, temporal validation) or in a different one (independent validation).7 To assess the performance of a previously described prediction model on a new data set, predicted and observed risks should be compared (ie, calibration) and the ability of the model to differentiate between patients with or without the event of interest should be quantified (ie, discrimination).7,9,10 Unfortunately, external validation of predictive models is still very uncommon,8 but highly desirable. Hence, the primary objective of this study is to externally validate a previously described predictive model of APSP after ambulatory surgery.6

In this previously described model,6 different types of surgery were assigned into 2 groups according to anticipated level of postoperative pain (ie, minor or intermediate). Obviously, division into two broad categories is not precise. Therefore, recently it has been advocated to assign types of surgery to a wide range of surgical procedures (or groups of closely related procedures).11 Furthermore, in the previously described model7 the cutoff point for moderate to severe pain was set on a Visual Analog Scale (VAS, 0 to 100)>40 and the variables were dichotomized. However, recent studies have identified a threshold of Numeric Rating Scale (NRS, 0 to 10)>3 between mild and moderate to severe postoperative pain.12,13

Therefore, the second aim of this study is to modify the previously described prediction model of APSP after ambulatory surgery,7 not only by assigning the types of surgery to a wide range of surgical procedures (or groups of closely related procedures), but also by setting the cutoff point for moderate to severe pain on an NRS>3. Furthermore, continuous variables instead of dichotomized variables were used and the predictive power of additional variables, like American Society of Anesthesiologists physical status classification (ASA-level), work status, preoperative analgesic use, and baseline quality of life (QOL) was included in the analysis.

MATERIALS AND METHODS

Patients

A prospective longitudinal cohort study was used for external validation and modification of a previously published prediction model.6 The study was approved by the institutional Ethics Committee of the Maastricht University Medical Center+ in 2008, and all patients gave informed consent to participate. All patients undergoing day surgery were eligible to participate, regardless of the type of surgery. Exclusion criteria were (1) patients age below 18 years, (2) inability to express themselves, (3) visual dysfunction, or (4) insufficient understanding of the Dutch language.

Questionnaires

Patients were asked to complete 2 successive questionnaire packages. First, a baseline questionnaire package was used to measure demographics (eg, age, sex, educational level, work status, highest level of education), average and present pain intensity, expected postoperative pain intensity by the patient, preoperative analgesic use, previous surgery (related or not) and baseline QOL. The EuroQol (EQ-5D) questionnaire was used to analyze QOL.14 All questions regarding pain were measured on an 11-point NRS (where 0=no pain, and 10=worst pain imaginable). Furthermore, psychological variables (ie, catastrophic thinking, personality trait optimism, fear of potential short-term and long-term consequences of surgery) were analyzed based on 3 validated questionnaires: the Pain Catastrophizing Scale (PCS), the Life Orientation Test Revised (LOT-R) and Surgical Fear Questionnaire.15–18 For the PCS and LOT-R, shortened versions were used to diminish patient burden.15,19 In the PCS questionnaire, 6 of the 13 original items were used. These were 2 questions loading highest on each of the 3 subscales (ie, items 5 and 12 for helplessness, items 9 and 11 for rumination and items 6 and 13 for magnification).15,16 In the LOT-R, 4 of the originally 10 questions were used. Four filler questions were omitted and the 4 questions (2 positively phrased and 2 negatively phrased) loading highest on, respectively, the optimism and pessimism factor were selected.15,19 Second, a follow-up questionnaire was used to measure APSP related to the surgery on an 11-point NRS.

Procedure

Between November 2008 and April 2010, patients planned for day surgery and presenting at the outpatient clinic for preoperative assessment at the Maastricht University Medical Center+, were asked to participate. If consent was obtained, the patient received an envelope containing an informative letter about the study, the 2 questionnaire packages and 2 return envelopes. Patients were instructed to complete the baseline questionnaire package 1 week before the surgical procedure. Patients who did not return this questionnaire package were considered to be unwilling to participate and no further attempts to contact them were made. The follow-up questionnaire package had to be completed at the fourth day after the surgery. Patients who returned the baseline questionnaire package, but did not return the follow-up questionnaire package, were reminded by regular mail or telephone 2 weeks after surgery. Only patients who returned both the baseline and the follow-up questionnaire packages were included into our analyses. All clinical information (eg, ASA physical status, surgical procedure, type of anesthesia, duration of the procedure, unplanned admission, and readmission) was acquired by systematic chart review.

Statistical Analysis

Missing data of potential predictor variables were imputed using multiple imputation according to the method described by Van Buuren et al.20 To compare APSP after various types of surgery, homogenous surgical groups were created. Surgical groups were selected when they contained at least 20 procedures.11

External Validation of the Prediction Model

For each individual in our cohort, the predicted probability of moderate to severe APSP, defined as NRS >4,6 was computed using the regression coefficients of the previously published model.6 To derive the regression coefficients from their tables, we computed the natural logarithm of the odds ratios that they presented. The probabilities were computed using the formula:

P(event)=1/(1+e(-(linear predictor))), in which the linear predictor is the sum of the regression coefficients multiplied by their respective predictor variable values. The intercept was not presented in the manuscript, but is necessary for the calculation of the linear predictor. Therefore, we estimated an intercept based on our cohort.

Briefly, the predicted probabilities were subsequently used to evaluate the discriminative ability of the model, and the model’s calibration. The discriminative ability refers to the model’s ability to discriminate between patients who will, and those who will not develop APSP, and is expressed as the area under the receiver operating characteristic curve (AUC). The AUC ranges from 0.5 (ie, no discriminative ability) to 1.0 (perfect discriminative ability). The calibration of the model refers to the agreement between predicted probabilities and observed frequencies of the outcome. In studies of external validation the calibration of a model is usually examined by computing the calibration in-the-large (ie, a comparison of the average predicted probability for the whole cohort and the proportion of patients with postoperative pain) and by visually inspecting a calibration plot. Because we estimated the intercept for the model on our own data, calibration in-the-large will be spot-on. Therefore, we will confine ourselves to an inspection of the calibration plot.

Modification of the Prediction Model

Potential predictor variables for the modified prediction model consisted of the initial variables comprised in the previously published model before dichotomization and additional variables (ie, ASA-level, work status, preoperative analgesic use, and baseline QOL). Furthermore, type of surgery was assigned to a wide range of surgical procedures (or groups of closely related procedures). Finally, moderate to severe APSP was defined as NRS>3.12,13

A multivariable logistic regression analysis was performed to estimate the regression coefficients of all variables. Only variables with a P-value<0.1 were included in the final model. A stepwise forward multivariable logistic regression analysis was performed to determine predictors for APSP. Only variables that were significant in more than halve of the imputed data sets were considered as significant predictors in the pooled regression model.

The development of the prediction model was based on 3 consecutive steps. In a first step, those variables that are easily to obtain (ie, sex, age, and surgical procedure) were included. In a second step, variables based on items that are relatively easy to obtain during the preoperative assessment (ie, ASA-level, work status, education level, previous surgery, preoperative pain, and preoperative analgesic use) were incorporated. In the third and final step, psychological variables were added to the model (ie, expected postoperative pain by the patient, short-term and long-term surgical fear, pain catastrophizing, optimism, and preoperative QOL).

Internal Validation of the Modified Prediction Model

It is a well-known artifact that a prediction model performs considerably less well in future patients, as compared with the patients in the cohort the model was originally derived. Therefore, we internally validated the model. Standard bootstrap validation was used with 1000 bootstrap samples on each of the imputed data sets.21 Results from the bootstrap averaged over the 10 data sets yielded a measure that was used to subtract from the computed AUC to obtain a conservative estimate, and a shrinkage factor used to multiply the regression coefficients by. The shrunk regression coefficients will produce less extreme results for future patients to counteract the too extreme predictions that are often observed when using a model that had not been internally validated.

All analyses were performed using SPSS version 20.0 (SPSS Inc, Chicago, IL) and R version 3.2.2.

RESULTS

General Characteristics

Patient data are given in Figure 1. A total of 1118 patients were included for the final analysis.

FIGURE 1

Flowchart depicting the inclusion and exclusion.

Baseline patient characteristics are summarized in Table 1. Patients included into the study of Gramke et al6 are slightly younger and less educated as compared with the patients included in our cohort. Furthermore, in our cohort, more patients had moderate to severe pain in the preoperative phase, pain medication was more often used in the week before surgery and more patients were included in the anticipated postoperative pain level “intermediate.” In addition, more patients received general anesthesia as compared with the patients included in the study of Gramke et al.6

TABLE 1

Baseline Characteristics of the Validation Data Set (Stessel et al19) and of the Derivation Data Set (Gramke et al6)

The following regression models could be constructed from the results of Gramke et al,6 after estimating intercepts specific for our cohorts.

Step 1

Predicted probalility APSP=1/{1+EXP (−[−2.125+(0.956×Anticipated pain level)+(0.531×Age<45 vs. 60+)−(0.357×Age 45-59 vs. 60+)+(0.336×Sex)])}.

Step 2

Predicted probalility APSP=1/{1+EXP(−[−2.878+(0.956×Anticipated pain level)+(0.531×Age <45 vs. 60+)−(0.357×Age 45-59 vs. 60+)+(0.336×Sex)+(1.131×Preoperative pain)])}.

Step 3

Predicted probalility APSP=1/{1+EXP(−[−3.481+(0.956×Anticipated pain level)+(0.531×Age <45 vs. 60+)−(0.357×Age 45-59 vs. 60+)+(0.336×Sex)+(1.131×Preoperative pain)+(1.099×Expected pain)])}.

For each individual in our cohort, the predicted probability of APSP was computed using these formulas, leading to the AUCs shown in Table 2. The AUCs in all 3 steps are much lower in our validation data set compared with the AUCs in the derivation data set, presented by Gramke et al.6

TABLE 2

Discriminative Ability of the Previously Published Model (Gramke et al)6 in the Validation Data Set (Stessel et al19) Versus the Discriminative Ability of this Model in the Original Data Set (Gramke et al)6

For the previously published prediction model, the agreement between the predicted risk and the observed incidence of APSP applied to our data are shown in Figure 2. Although the risk-based groups lie close to the ideal 45-degree line, the relative spread of the groups around the average frequency of APSP is limited.

FIGURE 2

Calibration curves of the external validation of the previously published prediction model (Gramke et al6) . Triangles indicate the observed frequency of moderate to severe acute postsurgical pain per decile of predicted risk. The solid line shows the relation between observed outcomes and predicted risks. Ideally, this line equals the dotted line that represents perfect calibration. The histogram on the x-axis shows the distribution of predicted risks in the external validation data.

Results of the modified prediction model are shown in Table 3. The AUC of step 1 (age, sex, and surgical procedure) is 0.73. After correction for optimism (ie, the likely performance of the model in future patients) the AUC reduced to 0.70 (Table 4).

TABLE 3

Results of the Modified Prediction Model: Associations Between Predictor Variables and Acute Postsurgical Pain

TABLE 4

Regression Coefficients (=Beta’s) of the Modified Prediction Model Corrected for Overfitting (ie, they were Penalized, or Shrunk Toward 0, by Multiplying them With the Shrinkage Factor Resulting From the Bootstrap Validation)

Adding ASA status, paid work, level of education, preoperative pain, and preoperative analgesic use to the regression model, the AUC improves to 0.79, with a reduction to 0.75 after correction for optimism (Table 4). Finally, the addition of expected pain, long-term surgical fear and optimism, resulted in an AUC of 0.82, 0.78 after correction for optimism (Table 4).

The regression formulas for each step of the modified regression model, with a correction for optimism are the following.

Predicted probalility APSP=1/{1+EXP(−[−0.85+(Beta step 1×Surgical procedure)+(−0.02×Age)+(0.24×Sex)])}.

Predicted probalility APSP=1/{1+EXP(−[−0.71+(Beta step 2×Surgical procedure)+(−0.02×Age)+(0.18×Sex)+(−0.69×ASA 1 vs. 3)+(−0.80×ASA 2 vs. 3)+(0.30×Paid Work)+(0.52×Low vs. High education)+(0.27×Middle vs. High education)+(0.84×Preoperative pain)+(0.44×Preoperative analgesic use)])}.

Predicted probalility APSP=1/{1+EXP(−[−0.30+(Beta step 3×Surgical procedure)+(−0.17×Age)+(0.15×Sex)+(−0.73×ASA 1 vs. 3)+(−0.89×ASA 2 vs. 3)+(0.33×Paid Work)+(0.49×Low vs. High education)+(0.28×Middle vs. High education)+(0.60×Preoperative pain)+(0.37×Preoperative analgesic use)+(0.67×Expected pain)+(0.03×Long-term surgical fear)+(−0.07×Optimism)])}.

The regression coefficients or beta’s for surgical procedure for the 3 steps of the regression formula are given in Table 4.

For example, if a 60-year-old male patient received anal surgery, using the regression formula of step 1, his individual probability of APSP will be: Predicted probalility APSP=1/{1+EXP(−[−0.85+(1.66×1)+(−0.02×60)+(0.24×1)])}=0.46=46%. If, in addition, we know that his ASA status is 2, he has a paid job, a low level of education, experienced preoperative pain and used analgesics before the surgical procedure, his individual probability of APSP, using the regression formula of step 2, will be:

Predicted probability APSP=1/{1+EXP(−[−0.71+(1.41×1)+(−0.02×60)+(0.18×1)+(−0.80×1)+(0.30×1)+(0.52×1)+(0.84×1)+(0.44×1)])}=0.73=73%. Calibration curves for each consecutive step of the prediction model (ie, step 1 through 3) are shown in Figure 3. These curves indicate good calibration of the prediction model because the risk-based groups are all close to the ideal 45-degree line and they cover the whole range of probabilities (ie, between 0 and 1), especially for step 2 and 3.

FIGURE 3

Calibration curves of the modified prediction model. Triangles indicate the observed frequency of moderate to severe acute postsurgical pain per decile of predicted risk. The solid line shows the relation between observed outcomes and predicted risks. Ideally, this line equals the dotted line that represents perfect calibration. The histogram on the x-axis shows the distribution of predicted risks.

DISCUSSION

In the present study, we externally validated and subsequently modified a previously developed model6 to preoperatively predict the risk of moderate to severe APSP in surgical outpatients on the fourth postoperative day. Finally, we internally validated the modified prediction model.

The predictive accuracy of the 3-step model described by Gramke et al6 was substantially lower in our validation data set of 1118 patients than in the original data set that was used to develop this model. The ability of this model to discriminate between the presence and absence of APSP was poorer with, after the third step, an AUC of 0.81 in the derivation data set6 and an AUC of only 0.72 in the validation data set (Table 2). Furthermore, the calibration plot of the third step of this model (Fig. 2) shows a risk prediction that is too extreme; that is, an underestimation of the predicted low risks and a distinct overestimation of the predicted high risks.

Modification of the original model consisted of assigning type of surgery to a wide range of surgical procedures (or groups of closely related procedures), defining moderate to severe pain as NRS >3, use of continuous variables instead of dichotomized variables, and testing the predictive power of additional variables (ie, ASA-level, work status, preoperative analgesic use, and baseline QOL). Our modified model showed that preoperative pain, patient derived expected pain, and certain types of surgery are the best predictors of moderate to severe APSP on the fourth day after day surgery (Table 3). Other predictors are younger age, higher ASA status, paid work, low level of education, preoperative analgesic use, long-term surgical fear, and pessimism. Moreover, our modified 3-step model is able to discriminate between patients with and without moderate to severe APSP with an AUC of 0.82 and after correction for optimism still an AUC of 0.78 (Tables 3 and 4). The calibration curves shown in Figure 3 indicate good calibration of the modified model.

When applied to new individuals, the performance of a prediction model is generally lower than the performance observed in the population from which the model was initially developed.22 Poor performance in new patients may be due to overfitting of the model and can also arise from differences in patient characteristics, distribution of predictor values between both data sets, methods of measurement, and health care system.7 In our study, the flattened slope of the calibration plot of the original model (Fig. 2) and the observed decrease in AUC (Table 2) are clear signals of overfitting the model and optimism in the performance parameters.23 Furthermore, pain intensity was measured using the VAS in the derivation study in contrast to the NRS in the validation study. Finally, our patient cohort included more patients with preoperative pain and an “intermediate” level of anticipated postoperative pain and they were slightly older and more highly educated (Table 1) as compared with the cohort used by Gramke et al.7 These differences can be explained by the recent evolution of day surgery toward more complex surgical procedures on older and higher risk patients.24

Various predictors of postoperative pain have been reported in literature. On the basis of a systematic review preoperative pain, anxiety, psychological distress (ie, pain catastrophizing, pessimism, depression), younger age, and type of surgery were reported to be the 5 most significant predictive factors for postoperative pain.25 This systematic review did not include preoperative expectations of postoperative pain (by the patient) as a possible predictor. Nevertheless, a positive correlation between preoperative expectations of pain and the occurrence of postoperative pain has been reported in literature.6,26,27 Preoperative expectations by the patient on postoperative pain are influenced by many factors including previous experiences with surgery or other traumatic injuries, the individual memory, and psychological profile of the patient.6 Unlike other studies, pain catastrophizing was not found to be a significant predictor in our model. The predictive value of pain catastrophizing may have been reduced in our model by the inclusion of another psychological predictor “preoperative expectations of postoperative pain” as these 2 variables seem to be associated.28 Our model also showed higher ASA status to be a predictor of APSP and a similar correlation was reported by Caumo et al.29

Furthermore, our analysis showed that patients with a paid job reported a higher APSP as compared with patients without. It can be hypothesized that patients with a paid job desire longer sick leave and therefore tend to overestimate their pain levels.

Limited data exist on the effect of educational level on APSP.6,30 In the present study, we report that a low level of education is a significant predictor for APSP and a similar correlation was found in 2 previous studies.6,30 This association might be related to differences in the ability to cope with pain.6

The observed correlation between preoperative analgesic use and APSP is in line with a previous study.31 Although the relation between preoperative analgesic use and APSP is not clear, 3 possible mechanisms might be involved: tachyphylaxis, opioid-induced hyperalgesia (in case of preoperative use of opioids), and neuroplastic changes in the spinal cord due to chronic noxious input.27

The modified prediction model can be valuable when implemented in the regular preoperative anesthesia evaluation of the outpatient. Identification of patients at high risk for moderate to severe APSP enables physicians to plan a tailor-made effective postoperative analgesic regimen and a more comprehensive follow-up program for these patients. In practice, this includes use of multimodal analgesic techniques, regular telephone follow-up, and even planned overnight stay. Moreover, it enables better patient information provision and adequate use of resources for selected patients with increased risk profile. Implementation of the modified model in daily practice can be achieved with the development of a convenient medical software application. After input of patient demographics, other patient-related predictive factors and data on type of surgery, this application can easily calculate the risk for APSP with the regression formula of the modified prediction model.

This study also has some limitations. First, Gramke et al6 assessed postoperative pain at the day of operation and at postoperative days 1 to 4. In contrast, our validation study only assessed postsurgical pain at the fourth postoperative day. Hence, validation of the predictive model of Gramke and colleagues in the present study is limited to the fourth postsurgical day. Still, the discriminative power of the predictive model of Gramke et al6 did not vary over the 4 postoperative days. Second, pain intensity was measured using the VAS in the derivation study in contrast to the NRS in the validation study. However, it has been proven that VAS and NRS scores correspond well.32,33 Furthermore, the generalizability of a prediction model can only be proven if this model has been tested in a more diverse setting.34 Third, we performed a temporal validation (data set for validation was collected at the same center). As a result, the extrapolation of the predictive performance of the model to other institutes or countries cannot be proven.22 Still, the variation between the 2 different data sets has been enlarged by the fact that the validation has been performed by a different team with overlapping authors and by the large time frame between data collection (more than 6 y). More specifically, within this time frame, different strategies in postsurgical pain therapy have been implemented and improvements in surgical techniques and antiemetic therapy made it possible to perform more complex surgical procedures in an older and higher risk patient population in the outpatient setting.24 Our results confirm these recent developments. Finally, the complexity of this modified model can impede his implementation in clinical practice. However, as mentioned above, a convenient medical software application can overcome this obstacle.

In conclusion, we could not validate the use of the prediction model of Gramke and colleagues on a large cohort of outpatients as both discrimination and calibration were considerably less than expected. Internal validation of our modified version of this model, however, shows promising results with good discrimination and calibration. In this modified model, preoperative pain, patient derived expected pain and certain types of surgery are the best predictors of moderate to severe APSP after day surgery. Other predictors are younger age, higher ASA status, paid work, low level of education, preoperative analgesic use, long-term surgical fear, and pessimism.