Effect of changing patient mix on the performance of an intensive care unit severity-of-illness model: how to distinguish a general from a specialty intensive care unit.

OBJECTIVE: To analyze the effects of patient mix diversity on performance of an intensive care unit (ICU) severity-of-illness model.
DESIGN: Multiple patient populations were created using computer simulations. A customized version of the Mortality Probability Model (MPM) II admission model was used to ascertain probabilities of hospital mortality. Performance of the model was assessed using discrimination (area under the receiver operating characteristic curve) and calibration (goodness-of-fit testing).
SETTING: Intensive care units. PATIENTS: Data were collected from 4,224 ICU patients from two Massachusetts hospitals (Baystate Medical Center, Springfield, MA; University of Massachusetts Medical Center, Worcester, MA) and two New York hospitals (Albany Medical Center, Albany, NY; Ellis Hospital, Schenectady, NY).
INTERVENTIONS: Random samples were taken from a database. The percentage of patients with each model disease characteristic was varied by assigning weights (ranging from 0 to 10) to patients with a disease characteristic. Three simulations were run for each of 15 model variables at each of 16 weights, totaling 720 simulations.
MEASUREMENTS AND MAIN RESULTS: The area under the receiver operating characteristic curve and model fit were assessed in each random sample. Removing patients with a given disease characteristic did not affect discrimination or calibration. Increasing frequency of patients with each disease characteristic above the original frequency caused discrimination and calibration to deteriorate. Model fit was more robust to increases in less frequently occurring patient conditions. From the goodness-of-fit test, a critical percentage for each admission model variable was determined for each disease characteristic, defined as the percentage at which the average p value for the test over the three replications decreased to < .10.
CONCLUSIONS: The concept of critical percentages is potentially clinically important. It might provide an easy first step in checking applicability of a given severity-of-illness model and in defining a general medical-surgical ICU. If the critical percentages are exceeded, as might occur in a highly specialized ICU, the model would not be accurate. Alternative modeling approaches might be to customize the model coefficients to the population for more accurate probabilities or to develop specialized models. The MPM approach remained robust for a large variation in patient mix factors.