Adaptation of Predictive Models to PDA Hand-Held Devices.

Prediction models using multiple logistic regression are appearing with increasing frequency in the medical literature. Problems associated with these models include the complexity of computations when applied in their pure form, and lack of availability at the bedside. Personal digital assistant (PDA) hand-held devices equipped with spreadsheet software offer the clinician a readily available and easily applied means of applying predictive models at the bedside. The purposes of this article are to briefly review regression as a means of creating predictive models and to describe a method of choosing and adapting logistic regression models to emergency department (ED) clinical practice.

INTRODUCTION

Articles reporting clinical prediction models that employ regression techniques are appearing with increasing frequency in the medical literature.1 One of the principal reasons for carrying out a regression analysis is to generate a tool that will help predict an outcome (dependent variable) from available clinical information (independent, explanatory or predictive variables). When validated, such tools may be used to supplement clinical judgment and improve patient care. Comparisons of clinical performance with and without the use of such models support the contention that improved diagnostic accuracy and standardization of care can result from their use in the workplace.2–8

Two major problems associated with predictive models are the complexity of computations associated with their application, and their lack of availability when they are needed at the bedside. Investigators typically modify their derived model to a simplified scoring system using whole numbers. This facilitates calculations, but at the cost of reduced precision of their model, and does not solve the problem of availability.

Use of personal digital assistant (PDA) hand-held devices as a readily available source of medical information is expanding.9 Programs for such devices which incorporate predictive instruments and medical calculators in a “user friendly” format may be downloaded, in many instances for a fee. Unfortunately, such programs are limited in scope and ability to incorporate ongoing changes in medical evidence, and are not easily modified by the user to meet the needs of differing clinical settings.

This article briefly reviews regression as a vehicle for creating predictive models, provides a system for evaluating an article with a predictive model, and describes a relatively simple method for translating the results of studies that use logistic regression (the most common of the regression techniques for creating predictive models) to a PDA spreadsheet format that can be easily applied at the bedside.

Simple Linear Regression

Regression techniques, at their most basic level, employ simple linear regression that describes the straight-line relationship between two variables. A single explanatory (independent) variable (X) is used to predict another (dependent) outcome variable (Y). In the regression model, the slope of the regression line is symbolized by “b”, called the regression coefficient, and “a” denotes the Y-Intercept of the regression line, a constant for that model.

An example of the application of this most elementary form of regression analysis was published by Després et al who examined the ability of waist circumference measurements (in cm) to predict the amount of deep abdominal adipose tissue noted on CT scan (in cm2).10 In their study, waist circumference was the independent variable (X), and the amount of adipose the dependent variable (Y). The regression formula obtained was:

Thus, if the waist circumference of an individual was found to be 70 cm, the predicted amount of adipose on CT was:

Multiple Linear Regression

Many problems in medicine involve multiple predictive variables, all of which must be taken into consideration to predict a single outcome, measured in terms of continuous data, as in the above example. The multiple regression model incorporates two or more independent variables to explain, or predict, an outcome or response. The model, an extension of simple regression, may be represented as follows:

where Y is the dependent (outcome, response) variable, “a” is the intercept of the regression line (a constant for that relationship), X1 is the value of the first independent (predictor) variable and b1 is the regression coefficient associated with it, X2 is the value of the second independent variable and b2 is the regression coefficient associated with it, and so on.

The independent variables typically have continuous numerical values (e.g., weight in kilograms, age in years, etc.). These variables may also have dichotomous (“either/or”) values in which case the presence of that variable is represented by one and its absence by zero. Ordinal values, in which the variable is stratified and ranked in order of increasing severity or exposure, may also be used.

A study published by Benowitz et al illustrates this type of regression analysis.11 They found that the nicotine intake while smoking a cigarette was predictable given the total particulate matter per cigarette and the number of puffs taken. The relation is described by the equation:

where Y is nicotine intake (in mg), X1 is the number of puffs per cigarette, and X2 is the total particulate matter in mg per cigarette, and −0.75 is the constant (intercept) for the model.

With multiple linear regression, while the independent variables may take on dichotomous values, the outcome variable may not. In the frequent case in which the clinician is interested in an outcome with a dichotomous value (e.g., the presence or absence of a particular disease; survival versus death; cure versus treatment failure) the technique of logistic regression is most often employed.

Logistic Regression

In studies using multiple logistic regression the outcome of interest is dichotomous (“either/or” type data) and is expressed within a derived model in terms of the odds that one outcome or the other will occur. Unfortunately, when using odds, the range from zero-chance to even-chance (odds ranging from zero to one) is disproportionate to the range from even-chance to 100% chance (odds ranging from one to infinity). In order to correct for this imbalance, the outcome is expressed in terms of the natural logarithm (ln) of the odds. Ln (odds) can range from minus infinity when the odds are zero, to zero when the odds are one, to positive infinity when the odds are very large. The natural logarithm of the odds is also known as the logit, hence the term logistic regression.

Multiple logistic regression uses the following general formula:

When used in a study seeking to formulate a predictive model, “a” is a constant (analogous to the Y-intercept of the simple linear regression model) generated by the results of the study; b1, b2, b3, and bn are regression coefficients for each independent variable, also generated by the study; and X1, X2, etc. represent the values of each variable for a particular patient. Some authors use α or β0 the regression constant, and β1, β2, etc. to represent the coefficients. Numerical values for the constant (intercept) and the regression coefficients for each variable are often included by the author in the results section. When they are, the reader can reconstruct a formula that will allow precise calculation of probability of outcome, given the values of the variables for a particular patient. This reconstruction of a predictive formula proceeds as follows.

From the above relationship, the ln (odds) of outcome for a particular patient is first calculated by substituting the values for each variable for that patient. Next, by exponentiation, the odds may be found as follows:

where e represents the base of the natural logarithm (equal to 2.71828). Finally, the probability of disease is determined:

Such calculations are impractical for the bedside. However, standard spreadsheet applications created for PDAs may be fairly easily programmed to rapidly carry out these computations as described in the appendix.

SELECTION OF AN APPROPRIATE PREDICTIVE MODEL

Before adapting a predictive model to the ED bedside, an appropriate model must first be selected and evaluated systematically to ensure the accuracy and proper interpretation of the data. Various guidelines have been suggested for the execution, interpretation, and reporting of multivariate methods. However, as of yet, no consensus exists.12–13 We evaluated the literature, and suggest a set of seven criteria to evaluate a predictive logistic regression model for adaptation to the bedside PDA application (Table 1). We do not propose that if an article does not fulfill all seven criteria it would be deemed unworthy of use in clinical practice. Rather, our goal is to provide a method to evaluate the literature, so that the interpretation of the data and its worth can be appraised by the individual practitioner. The results calculated from the model itself can then be added to the clinician’s fund of knowledge to make a sound clinical decision.

Table 1

Evaluating a Logistic Regression Model

Appropriate study population Inclusion of regression coefficients and regression constant Description of variable coding and selection Effect modification reporting Goodness of fit and Validation of the model Overfitting Nonconformity to a linear gradient

Appropriate study population

When interpreting or applying the result of a study, the study population must be taken into account. For a predictive model to be applied in the emergency department, the study must have a patient population representative of an ED population. This strengthens the external validity of a study, i.e. how generalizable the findings are outside of the study population. Similarly, spectrum bias may distort the accuracy of diagnostic tests and apparent effectiveness of treatments when studied using samples of patients with disease severity more advanced or less advanced, than that in your own clinical population. For example, a model derived from an outpatient medicine clinic or an ICU population may not be applicable to your ED. Examination of the characteristics of the study participants, as well as the inclusion and exclusion criteria must take place, to determine whether the results would be relevant to your patient population.

Inclusion of regression coefficients and regression constant

In order to fully implement the logistic regression model, the regression coefficients and the regression constant (intercept) for the ratios are typically reported, which can be transformed into a regression coefficient by taking the ln(odds ratio), the regression constant is often omitted, making the calculation of probability (the raison d’etre of the model) impossible for the reader.

Description of variable coding and selection

Proper description of the coding of individual variables must be included and examined. The apparent effect of a variable can depend on how the variable is coded. For example, the regression coefficient for the impact of age (independent variable) on long-term mortality (dependent variable) will be different if age is coded in one-year increments versus 10-year intervals, or as a dichotomous variable (less than or greater than 65 years). If a dichotomous (or other categorical) variable is included, the coding method must be replicated to generate accurate results. For example, most authors code presence of the variable as “1” and absence as “0.” Others may use “2” and “1.” The method used by the author must be followed in creating a bedside tool.

Additionally, the reasoning behind selecting independent variables for the model must also be considered, including the level of significance at which the variables were included into the model. An assessment must be made as to whether variables were considered based on prior results, clinical experience, or based on an automated algorithm (“forward” or “backward” selection).12 Proper coding and selection can add to the strength of the result of the model, and thus your clinical practice.

Effect of modification reporting

Interactions between independent variables may influence the coefficients associated with those variables. 12 This interaction, known as effect modification, refers to variation in the magnitude of effect by the variable with varying levels of exposure of another variable.14 Consider, for example, a logistic regression model with lung cancer as the binary outcome variable with two independent (exposure) variables: smoking and asbestos exposure. If the interaction between the exposure variables is not considered, a deceptive regression coefficient estimate for smoking would result. This is because the effect modification of asbestos exposure on smoking is synergistic with respect to lung cancer. When evaluating a model with potential effect modification between variables, mention should be made of the testing for such interactions.

Goodness-of-fit and validation of the model

The validity of inferences drawn from logistic regression techniques depends on the assumptions of the model being satisfied. A critical step in assessing the appropriateness of a logistic regression model is to examine how well the model describes the observed data.15 In other words, if an estimate of the outcome is calculated using the model, how well does this estimate “fit” with an actual patient with similar characteristics from the dataset? This is known as “goodness-of-fit,” and is measured by various indexes, including the Hosmer-Lemeshow statistic or reporting of a percentage of the dependent variable that was correctly identified by the model. 16

Validation or retesting of a model in a population different from that used in creating the model is especially important with predictive models to assess model success outside of the derivation study population.12 Internal validation, using the “jackknife” or “bootstrap” procedures, perform the analysis on subsets of the data used to derive the model, investigating the stability of coefficients and predictive ability of the model.12 A better method includes validation analysis on a separate subset of patients not used in the creation of the model. Most desirable is external validation of the model in a population independent of and external to that used in deriving the model, preferably at a separate institution. Reporting of any of these techniques is essential in assessing the validity of model being evaluated.

Overfitting

Overfitting implies that the model has been so refined to conform to the study sample that it has lost general usefulness in application to different populations. A model must have enough outcome events per independent variable in order to have a reliable estimate of risk. Though controversial, studies having fewer than 10 outcome events per independent variable may result in questionable accuracy.16 With overfitting, the resulting regression coefficient may represent spurious associations, or the effects may be estimated with low precision.12

Nonconformity to a linear gradient

In logistic regression modeling, while the dependent variable is binary (dichotomous), the independent variable may be ordinal or even continuous. When a regression coefficient is established for an independent variable, the assumption is that the relationship between the variable and the outcome is linear in nature. That is, a unit change in that variable should always have the same effect on the outcome, regardless of where that unit change occurs in the range of that variable. This may be a problem if the independent variable does not act according to a linear gradient. As an example, the impact of left ventricular ejection fraction on mortality depends not only on the unit change in ejection fraction, but also where the baseline ejection fraction stands. A decrease of 10%, from 30% to 20%, carries greater risk than a decrease from 50% to 40%.12 An article should report the evaluation of conformity to a linear gradient for such variables. Not doing so may overestimate or underestimate the effect depending on the value for an independent variable.

Example

An example below is taken from an article by Shapiro et al,8 which describes a model for predicting 28-day hospital mortality among septic patients based on nine independent dichotomous variables available in the emergency department. Our evaluation of the clinical model based on our seven criteria can be found in Table 2. The study did meet the majority of criteria. The basics of setting up the PDA spreadsheet can be found in the appendix. The variables and their associated coefficients were as follows:

Table 2

Evaluating Predictive Model for Severe Sepsis [From Shapiro et al8]

Evaluation Criteria	Result
Appropriate Study Population	Population consisted of patients > 18 years presenting to the ED at an urban, academic teaching hospital with 50,000 visits annually
Inclusion of regression coefficients and regression constant	The proper coefficient and intercept were reported in the results
Description of variable coding and selection	There was adequate description of the variables in the model. Variables were eligible for inclusion into a forward selection model at a level of p<0.1. Presence of a dichotomous variable was coded as “1” and absence as “0.”
Effect modification reporting	Effect modification and interactions were not mentioned in the article.
Goodness of fit and Validation of the model	Goodness of fit was assessed using Hosmer-Lemeshow goodness-of-fit test. Validation of the model was done by the bootstrap method as well as creating a separate validation set to test the final model created from the derivation set.
Overfitting	Also assessed using the bootstrap method. There were greater than 10 events per independent variable
Nonconformity to a linear gradient	Not mentioned in the article, and often difficult to assess.

The value of the constant (a) for their model was −5.45.

The logistic regression formula can be reconstructed as follows (taking care to enter plus and minus signs accurately):

Recall that the presence of a dichotomous independent variable in this study was designated by the value 1, its absence by the value 0. For example, if a patient had tachypnea, was in septic shock, had bands of >5%, was 68 years old, had clinical pneumonia and was a nursing home patient (other variables being normal), the calculation becomes:

Setting up this predictive model on a PDA spreadsheet can be found in the appendix.

DISCUSSION

The frequency of use of multivariable methods in the medical literature has steadily increased over the years. One study revealed an 8% increase over a five-year period between 1985–1989.12 Computer-assisted predictive tools as described in this article offer several potential benefits to take advantage of this trend. The results of investigations that produce predictive models may be directly translated into clinical practice without alteration, thereby maintaining their original precision. Physicians using PDAs may be more likely to use the tools when they can be easily tailored to their clinical practice.9 Using widely available spreadsheet software, newly validated models may be quickly translated into a form that can be used at the bedside (see appendix), without waiting for third-party software creation and distribution.

There are some important limitations regarding predictive tool adaptation to PDAs. First, predictive models yield only a probability of outcome (e.g., risk of disease, likelihood of benefit from therapy). Establishing thresholds for purposes of diagnostic and therapeutic decision making is a matter of clinical judgment.

Second, a prediction model should be evaluated carefully before it is used in any form. However, proper evaluation is often limited by poor reporting by the author. Bender et al investigated logistic regression in several journals (BMJ, JAMA, the Lancet, and the New England Journal of Medicine) from 1991–94. They found that goodness-of-fit was rarely assessed. Of 111 papers, only seven papers reported a valid assessment of the adequacy of their regression model.17 Other studies have shown similar need for improvements in the reporting and perhaps conducting of multivariable analysis.12 Violations included overfitting of data, a lack of testing for conformity of variables to a linear gradient, no report of testing for interactions, and unspecified coding or selection of independent variables. In the critical care literature, 65% of published articles properly reported coding of pertinent independent variables; 12% referenced whether effect modifications were examined; 1% tested for colinearity; 16% included a goodness-of-fit analysis; and 39% may have overfitted the model, leading to potentially unreliable regressioncoefficients16 In the obstetrics and gynecology literature 51.8% of articles inadequately described the process of variable selection, 85.1% did not report assessment of conformity to linear gradient, only 6.8% tested for goodness-of-fit, and interactions between variables were not assessed in 86.4% of articles.13

Part of the problem with the application of multivariate statistical methods is proper understanding by the study author of these procedures. To this end, editorial guidelines for reporting would improve the ability to interpret a study. If strict reporting guidelines were in place, the methodological flaws in a particular study as well as limitations of model output could be better appreciated.18 Detailed and complete reporting and peer review of such research as well as informed analysis by the clinician evaluating the paper is necessary. The publication of study alone should not ultimately prompt a change in your clinical practice without proper examination of that study.

PDA spreadsheets formatted as predictive tools have been used successfully within our residency for over five years. Our residents have found this to be an effective way to incorporate evidence-based medicine into daily clinical practice. Once created, these tools can be readily shared with other physicians and are easily modified and/or updated as new research is published.

Variable	Coefficient (b)
Terminal illness (<30 days)	1.80
Tachypnea or hypoxia	0.98
Septic shock	0.98
Platelets <150,000/mm3	0.93
Bands >5%	0.82
Age >65	0.77
Lower respiratory infection	0.66
Nursing home resident	0.62
Altered mental status	0.50