Evolving forecasting classifications and applications in health forecasting.

Health forecasting forewarns the health community about future health situations and disease episodes so that health systems can better allocate resources and manage demand. The tools used for developing and measuring the accuracy and validity of health forecasts commonly are not defined although they are usually adapted forms of statistical procedures. This review identifies previous typologies used in classifying the forecasting methods commonly used in forecasting health conditions or situations. It then discusses the strengths and weaknesses of these methods and presents the choices available for measuring the accuracy of health-forecasting models, including a note on the discrepancies in the modes of validation.

Introduction

Forecasting is the process of predicting future events based on foreknowledge acquired through a systematic process or intuition.1,2 It requires data, information, and advanced knowledge. Forecasting has evolved over the years and now has wide applications in many fields, including economics and commerce,1,3 sports,4,5 the environment (including meteorology),6,7 technology and politics,8–10 and health.11–14

Health forecasting predicts health situations or disease episodes and forewarns about future events. It is also a form of preventive medicine or preventive care engaged in public health planning, and it is aimed at facilitating health care service provision in populations.12,14–16 One of the least developed branches of forecasting, health forecasting is a useful tool for decision making in health services provision. Health forecasting has been commonly applied to emergency department visits, daily hospital attendance, and admissions.17–20

Various methods and approaches have been applied in forecasting events, but some outstanding issues are yet to be addressed. Even though a comprehensive classification of all forecasting approaches and methods would serve as a useful guide to forecasters searching for appropriate forecasting methods, there have been limited discussions and debates around this need.21 Health forecasting studies have often adapted statistical techniques used by other well-established areas of forecasting, such as econometrics and finance. However, little has been said about the strengths and weaknesses of these techniques when they are applied to health forecasting.22 Another important issue that has not been explicitly presented in the literature relates to approaches used to determine the accuracy and validity of health-forecasting models. The applications available for measuring and determining the validity and accuracy of health forecasts have not been compared or presented as guides for health forecasting or even used to stimulate discussions that can contribute to improve health forecasting. This paper therefore aims at presenting a brief overview of the evolution of forecasting classifications and methods. It discusses the strengths and weaknesses of various health-forecasting techniques and methods, and then presents the choices available for validating and measuring the accuracy of health-forecasting models. Because of the new approach it brings to medical and health sciences, health forecasting is important for practices in these fields. Advances in health-forecasting research will facilitate the decision-making processes that are associated with healthcare planning and management.

In preparing this review, a search of the literature on health forecasting and statistical methods used in the analysis of health conditions was conducted in popular medical databases, such as PubMed (Medline) and Google Scholar. Additional literature searches were done through citation mapping of key papers. The selected papers and documents were synthesized and summarized according to the objectives of this paper.

An enumeration of forecasting typologies

Although several authors have made attempts to schematically classify the wide variety of forecasting techniques, many have not been adequately exclusivea or both concise and exhaustiveb enough to meet the needs of forecasters across all fields.21 A good classification system for forecasting methods can facilitate the process of choosing an appropriate method for forecasting, in addition to providing a better understanding and organization of the methods involved in designing a forecasting system. The enumeration of forecasting classifications presented below reveals the typologies and methods that have been involved. These classifications also justify the need for further research involving health-forecasting methodologies, since the latter have played a minimal role in shaping contemporary theory and methods in this area.

In 1971, Cetron and Ralph developed one of the earliest classifications of forecasting methods and approaches. It consisted of five categories, including intuitive methods, trend extrapolation, trend correlation, analogy, and dynamic predictive models.23 Generally, intuitive methods in forecasting are based on individual opinion, whether structured or unstructured. Trend extrapolation is an approach that uses known existing trends, whereas trend correlation forecasts are based on the causal links between a dependent factor and another factor or factors. Cetron and Ralph also used the category, Analogy, to describe forecasting approaches that used similarity in patterns for forecasting. They also classified dynamic predictive models (also later known as structural models), which describe simulation procedures that involve high impact causal factors. Although Cetron and Ralph’s classification is concise, it has been criticized for being neither exhaustive nor exclusive enough.21

Similarly, in 1972, in his classification of forecasting methods, Martino provided a five-category scheme consisting of the following: intuitive, consensus (ie, obtaining results from several experts), analogy, trend extrapolation, and structural models.24 Although Martino’s classification was concise and exclusive, it was not sufficiently exhaustive to meet the needs of forecasters. In 1978, another classification of forecasting methods by Bright considered as many as eight different categories, some of which were later thought contentious.21,25 His classification included intuitive forecasting, trend extrapolation, dynamic modeling, morphological analysis, normative forecasting, monitoring, cross-impact analysis, and scenarios.25 The key strength of Bright’s classification was that it added an entirely new concept of scenarios, and could be viewed liberally as exhaustive. However, it was neither exclusive nor concise. Furthermore, as mentioned earlier, some categories, such as monitoring, have been challenged because they are inappropriate as forecasting methods or approaches.21

In 1985, Armstrong published his first “forecasting methodology tree,” which was based on three assumptions or decisions. More recently, his Methodology Tree for Forecasting (2010) assumed that before arriving at an appropriate choice of analytical forecasting method, it is first necessary to decide on whether to use intuitive (judgmental) or objective (statistical) methods. Second, if the choice of approach is statistical, then a choice between causal and noncausal approaches is required. After a causal approach is chosen, the final decision is whether to select either linear or nonlinear (classification) statistical approaches.26 Hence, Armstrong introduced five categories in his maiden classification: judgmental, bootstrapping, extrapolation, econometric, and segmentation. Armstrong’s classification was concise and contributed new approaches to forecasting (ie, naïve/causal continuum), as well as providing guidance on the selection of forecasting approaches, which previous classifications lacked. However, Armstrong’s classification was neither exhaustive nor exclusive.

In 2001, Armstrong revised the classification of his forecasting methods and provided eleven categories of methods that could be derived from the Methodology Tree (including role playing, intentions, conjoint analysis, expert opinions, judgmental bootstrapping, analogies, extrapolation methods, rule-based forecasting, expert systems, econometric models and multivariate models).1 His classification further illustrated the primary distinction between methods that rely on judgmentc and those that estimate relationships using statisticald approaches or quantitative data.1 The classification was however not concise because there were too many categories (eleven in all). It was also not exclusive because of the subclassification extrapolation, which has a much wider application in statistical forecasting.

In 2006, Gentry et al proposed an entirely new form of categorizing forecasting approaches and methods in the form of a grid. In Forecasting Classification Grid,21 argue that two independent dimensions can determine forecasting approaches, which are on the continuums of Opinion and Empirical and Causal and Naïve. This classification helps to distinguish between opinions and ideas that can be empirically verified, and it is also simpler and more logical compared with earlier descriptions.21 The Grid has just four classifications and is therefore a concise scheme. It also is exhaustive because it is designed to fit in as many forecasting methods as are available. Even though the authors believed that the classifications were exclusive, grey boundaries could exist between the continuums. The key challenge in this classification is that the grid appears to be an uncompleted template, so the exact relative positioning of a forecasting method could be the subject of debate.

In a 2010 revision of the Methodology Tree for Forecasting,e Armstrong and Green further extend the list of forecasting methods. The authors differentiated between structured and unstructured approaches related to judgmental forecasting and further classified the theory-based approaches of forecasting into the categories, linear and classification.27 Armstrong and Green provided guidance on choosing suitable forecasting approaches and methods based on specific contexts. However, there are still flaws in their classification because some methods that had multiple applications were not explicitly illustrated in the Methodology Tree. For instance, both univariate and multivariate approaches can be applied in data mining or causal modeling.28 Furthermore, noncausal (black box) approaches, such as those involving data mining and neural nets, are equally applicable to causal modeling and share similar methods and techniques (eg, regressions and segmentation).

All the classifications of forecasting approaches and methods discussed above have significantly contributed to the organization of forecasting. Even though most of these developments have taken place in nonhealth-related areas (eg, marketing, management and finance/econometrics), they have direct applications to health forecasting. For instance, health forecasting has used neural networks,29–31 and many emergency department forecasts use one or more forms of regression analysis. It is therefore imperative that the lessons learned from previous forecasting topologies should inform any development of a typology of health-forecasting approaches and methods. Some related health-forecasting methods involved in the typologies listed above are exemplified in subsequent sections of this essay, which considers their strengths and limitations, their accuracy, and their validation procedures.

Health data and forecasting

Although data is vital in forecasting, what constitutes “health data” is poorly defined in the literature. Health data can be defined as: records of health conditions and situations that refer to individuals or populations and carry information about disease prevalence, incidence, diagnoses, treatments, prognosis, preventive strategies, and health systems. Moreover, these records are categorized by demographics and factors that directly affect health and are collected systematically or otherwise. For example, this definition could apply to hospital attendance or admission records that contain a variety of measures that are recorded in their respective units – age in complete years, for example.

In the practice of medicine, the diagnosis of disease is focused on determining the presence or absence of a condition so that the appropriate treatment can be given. However, the measures taken to facilitate this process are selected from a continuum. For example, diastolic blood pressure and pulse rate are taken to help determine whether a person is hypertensive or not (ie, more than or equal to 90 mmHg). Other factors that may have an effect on health status and whose levels are measured to generate health data include environmental exposure (eg, weather and air quality). At the point of measure or use, health data could be classified as either continuous (ratio or interval scales) or categorical (ordinal, nominal, or dichotomous scales).32 The definition and classification of health data determine how data are accumulated over time in addition to the method or methods of analyses that can be employed in analyzing this information.

An emerging form of health data – electronic health records (EHR) – refers to digital health data that is stored in secured repositories and shared only among authorized users.33 Hayrinen and colleagues identified the following as components of electronic health records: daily charting, medication administration, physical assessment, admission nursing notes, nursing care plan, referral, present complaint (eg, symptoms), past medical history, life style, physical examination, diagnoses, tests, procedures, treatment, medication, discharge, history, diaries, problems, findings, and immunization.34 These kinds of structured records have applications beyond health forecasting because they can be used to make predictions about the occurrence of future health events.

Strengths and limitations of health-forecasting techniques

Many reported studies on health forecasting adopted statistical techniques and methods, the theories of which are described in the standard literature. The choice of method depends on the purpose of forecasting and the nature of the data that are available. The strengths and limitations of these methods pertaining to health forecasting are discussed in the following paragraphs.

Linear regression methods are commonly used because they provide reasonably accurate results, are easy to interpret, and have wide applications in modeling trends and seasonality. However, like most regression methods, linear regression uses the method of ordinary least squares to derive estimates, which may wrongly assume that but for the dependent variable, the independent variables or regressors have no error.35,36 Hence, to account for this problem in modeling, there is always a need to factor in an error component. Linear regressions also require large amounts of data on all variables for parameter estimations.35

Logistic regressions provide a means for analyzing binary or categorical dependent variables, but they are not useful for count data.37 Logistic regressions can thus be applied to forecast the presence or absence of an event in a dichotomous (categorical) state. Poisson and negative binomial regressions are generally used for analyzing count data, and the latter is particularly suitable for analyzing count data that have a skewed distribution with a considerable number of zero entries.38–42 For instance, Negative Binomial Models (NBMs) were used in previous work to investigate the determinants of asthma in the length of stay in hospitals, for which the dependent variable bore the aforementioned characteristics.43,44 NBMs were also used to compare various statistical forecasting models for predicting the number of daily admissions of asthmatic patients in London (personal work yet to be published).

Moving average methods, which include autoregressive- integrated moving average (ARIMA), seasonal autoregressive- integrated moving average (SARIMA), and exponential smoothing (eg, Holt–Winters) are widely used forecasting approaches. They have the advantage of modeling trend and seasonal variations, as well as accommodating multivariable models.45,46 The exponential smoothing methods used in health forecasting are effective with data that change over time.22 However, the main challenge in using these complicated methods is that they require specialist knowledge and expertise.

Time series regressions generally have a much wider application and capabilities in forecasting than all the other nontime-series approaches mentioned here. Time series regressions provide easily interpretable outputs that can be more consistent than ordinary linear regressions.45,46 The use of time series approaches in forecasting ideally requires sufficient data for not only the dependent variables, but also the matching independent variables.

Quantile regressions and fractional polynomials are rarely mentioned in the health-forecasting literature, but they provide a means for predicting and forecasting peculiar events.47,48 For instance quantile regression models allow the modeling and forecasting of anticipated extreme events based on data distributions that are outside the normal range, which is more useful than linear regressions whose forecasts are based on the overall mean distribution. One major limitation of these approaches is that they deal with only the relevant or specific category of the data, and hence some information that could affect the accuracy and statistical power of the analysis is lost.48,49

Artificial neural network (ANN) is a black box modeling procedure known to provide more reliable results than the traditional causal approach.50 ANN is capable of modeling complex and random systems by automatically controlling adjustments to the changes in time series based on the design of the experiment.22,51 The major challenge of these models is that they are difficult to interpret and, unlike the other approaches described earlier, very few statistical software packages are available.52–54

Measuring forecast accuracy and validation

Forecasting is generally aimed at predicting future events in order to inform and guide precautionary measures. It is an art in as much as it is a science, and therefore the degree of certainty of every forecast is imperative. A number of techniques and approaches are used to determine the accuracy or validation of a forecast. The main purpose of measuring the accuracy of a forecast model is to assist in choosing the best model.55 This can be done in several ways: traditional forecasting accuracy measures; model discrimination approaches like receiver-operating characteristic (ROC) curves; and the use of model fit statistics, eg, R-square, Akaike information criterion (AIC), Schwarz information criteria (SIC) and Bayesian information criteria, which are discussed below.

Forecast accuracy

Forecast accuracy is a quantitative measure of the efficiency of the forecasting process, and it is performed by comparing the forecast to the actual situation. Forecast accuracy measures and parameters are usually supplied alongside the forecast as constituent elements to aid in decision making.

The accuracy of any forecast depends on objective features of the environment, such as the nature of the variable being forecast, or the length of the forecast horizon. Accuracy also depends on attributes of the forecast relating to the theories involved.56 Hence, the choice of accuracy measures may depend on the method of forecasting. However, there has been considerable discussion about appropriate measures of forecasting accuracy, which have a wide range.55,57–61

Measures of forecasting accuracy have three main categories: (a) scale-dependent measures (ie, accuracy measures whose scale depends on the scale of the data); (b) percentage error measures (ie, independent measures that can compare forecast performance across different datasets); and (c) relative error measures (ie, scaled errors based on error measured from a reference standard forecast), including the relative measures of each type of error measure.59 Examples of these measures are listed in Table 1.

Table 1

List of forecast accuracy measures

B. Scale-dependent measures Mean square error (MSE) Root mean squared error (RMSE) Mean absolute error (MAE) Median absolute error (MdAE)

C. Percentage error measures Mean absolute percentage error (MAPE) Median absolute percentage error (MdAPE) Root mean square percentage error (RMSPE) Root median square percentage error (RMdSPE)

D. Relative error measuresa Mean relative absolute error (MRAE) Median relative absolute error (MdRAE) Geometric mean relative absolute error (GMRAE)

Notes:

The relative error measures are obtained by dividing each forecast error by the error obtained using a benchmark procedure, such as the grand mean (ie, a reference or benchmark average, which could be determined by taking the average of all averages of several subsamples). The accuracy measures of GMRAE and MDRAE, for instance, were presented by Armstrong and Collopy (1992)61 and Fildes (1992).86 Even though both reports recommend the use of forecast accuracy measures based on relative errors, they express these measures in different and complicated forms. Hyndman and Koehler (2006)59 have however noted that these relative error methods could have some deficiencies that are associated with the difficulty of dealing with extremely small benchmark forecast error measures, resulting in the relative error measures having infinite variances.

Subsequent discussion focuses on selected scale-dependent and percentage error measures (Table 2), which are commonly used in health-forecasting studies. Scale-dependent error measures have been recommended for the comparison of different methods that are applied to the same set of data and scales.59 For example, the root-mean-square error (RMSE) has traditionally been widely used for forecasting evaluation62 and specifically for comparing models of the same series.61,63 Even though some scale-dependent error measures, such as the MSE and RMSE, have been theoretically more relevant in statistical modeling, they have also been found more sensitive in detecting outliers than the mean absolute error (MAE) or median absolute error (MdAE).59 Mean absolute scaled error (MASE) is also another scaled error approach recommended for comparing forecast accuracy across series on different scales.59,64 According to Hyndman and Koehler, MASE provides the most reliable approach because compared to others, it has a meaningful scale, is widely applicable, and is not subject to “degeneracy” problems.59 Moreover, MASE is seen less sensitive to outliers and is more easily interpreted. It shows smaller variations, even with small samples, than other measures in the same category.59,65

Table 2

A comparison of scale-dependent error measures

Scale-dependent measures	Definition	Error spread	Error weights
Mean square error (MSE)	Mean(Ot–Ft)2	Yes	Yes
Root mean squared error (RMSE)	√MSE	Yes	Yes
Mean absolute error (MAE)	Mean|(Ot–Ft)|	Yes	No
Median absolute error (MdAE)	Median|(Ot–Ft)|	–	–
Mean absolute scaled error (MASE)	Mean|(Qt)|	Yes	Yes

Notes: Error spread refers to the ability of the measure to capture an error that is not localized and not widely distributed in the dataset. Error weights refers to the ability of the measure to differentiate the error at different points in history.

Abbreviations: t, at a time; O, observation; F, forecast; Q:A, scaled error independent of scale of data.59

Measures based on percentage errors are not dependent on the scales of data and hence can be used to compare forecast errors across different datasets. However, their results tend to be infinite or undefined if a given forecast result equates 0 at any given time or has an extremely skewed distribution when the forecast is close to 0.59 A further challenge in this category of error measures, particularly for mean absolute percentage error (MAPE), is that they tend to over penalize positive errors compared to negative ones and thereby create an unbalanced symmetry in the measures.66

Model discrimination test (ROC curve)

The ROC curve is another measure of forecast error that is associated with discrimination and has been used in health related forecasting studies. ROC provides a means of measuring and comparing the accuracy of predictive models. It is a graphical plot of Sensitivity versus 1-Specificity in a binary classifier system, and it is constructed to assess the varying thresholds for discrimination of comparable statistical predictive models.67–70 The accuracy of prediction is measured by comparing the true positives against false positives.67,68 The ROC curve has very wide applications in many fields, and its use in forecasting has been described by many authors. It was for example used by Classen and Hokayem to compare various econometric models and to select a suitable model for forecasting “Childhood influences on youth obesity.”71

Model fit statistics

Widely used statistical model fitness tests include R-square, adjusted R-square, AIC and SIC. These model parameters are defined and estimated as follows:

where RSS is the residual sum of squares; TSS is the total sum of squares; n is the sample size; k is the number of parameters in the fitted model.

The use of R2 as a measure of fit or model variability in health-forecasting-related studies is very common in the literature.20,72–75 A higher value of a model’s R2 could be interpreted as having a better fit, which tends to increase with the addition of every extra explanatory variable. However, using R2 as a measure of fit can be unreliable in forecasting because the R2 of a model can be high or equal to 1 and yet be consistently wrong.57 Like the R2, the adjusted R2 also increases with every additional explanatory variable, but this test is more reliable because it tends to penalize the model for every additional explanatory variable as long as the new addition does not significantly reduce the RSS (Equation 2). The AIC is superior to the adjusted R2 because it has a harsher penalty and is preferred in forecasting models as a measure of fit.62,76,77 This technique is based on the maximum likelihood and the number of independently adjusted parameters within a predictive model.76 Compared with AIC and R2, SIC gives the best model diagnostic fit because it imposes the highest penalty on the model.45 However, in forecasting, given the balance between the need for a predictive model that has a good fit and a high explanatory power, AIC is currently the most popular and recommended technique for model fit statistics,78 and it is commonly used in model selection.43,63,79–81

Forecast validation

Forecast predictions are rarely perfect, so validation or cross-validation is an essential process that allows estimation of the extent to which a predictive model emulates the natural phenomenon that produces the data.50,60,82 Validating a forecast requires appropriate techniques and reliable measures. In developing a health-forecasting predictive model, two types of validity can be examined: model validity and predictive validity. Both are important and can be used to generate a useful and reliable forecast. Model validity represents the extent to which the model fits the data that was used for the model development (ie, the fit of the model to the experimental sample). This type of validity test is also referred to in the literature as internal validity. The second type, predictive validity (also known as external validity), represents the extent to which the predicted forecast values fit the observed values (ie, the fit of the model to the test sample).61 Predictive validity is usually carried out through a process described as cross validation.

Cross validation is a statistical technique commonly used in forecasting for estimating the performance of a predictive model. It is usually carried out using a similar, but separate, sample of the data that was used in developing the forecast model. The health-forecasting literature does not provide standard procedures for conducting cross validation. Hence, the proportion of an evaluating sample (compared to the test or model development sample of data) that is suitable and sufficient for validating a health-forecasting model remains unclear. A scan of the literature revealed a wide range of arbitrary choices. As illustrated in Table 3, the relative proportion of a data sample used for cross validation of health-forecasting models could range from 1:1 to 12:1. For example, studies conducted by McCarthy et al18 and Hoot et al17 on forecasting emergency department visits used similar proportions of data for a cross validation of their output. However, other researchers have done this differently. To develop and to test their forecasts, Wargon et al83 and Rotstein et al74 used three quarters of their data as a training sample and the other one quarter as an evaluating sample.

Table 3

Varying ratios of period of training to period of evaluation of health forecasting models

Author	Ratio of period of training: evaluation	Analytical techniques used in forecasting and study purpose
Hoot et al 200817	1:1	ARIMA; to predict ED operation conditions within 8 hours
McCarthy 200818	1:1	Poisson regression; to predict hourly ED presentations
Boyle 201119	1:1/2:1/3:1/4:1	ARIMA, regression, ESM; to predict ED presentation and admission
Hoot et al 200751	2:1	Logistic regression and ANN; to predict ED overcrowding
Wargon et al 201083	3:1	Regression model; to predict daily ED presentation
Reis and Mandl, 200384	4:1/5:1	ARIMA models; to predict daily pediatric ED presentation
Schweigler et al 200985	7:1/14:1	SARIMA, hourly historical averages; to predict hourly ED bed occupancy
Jones et al 200822	8:1	SARIMA, regression, ESM, and ANN; to predict daily presentation
Batal et al 200175	9:1	Stepwise linear regression; to predict daily presentation
Champion et al 200720	12:1	ARIMA, ESM; to predict aggregate monthly ED presentations

Abbreviations: ANN, Artificial Neural Networks; ARIMA, autoregressive-integrated moving average; SARIMA, seasonal autoregressive-integrated moving average; ESM, exponential smoothing; ED, emergency department.

Currently, there are no common scales for validating health forecasts based on a particular forecasting horizon, and the information available suggests that any appropriate cross validation strategy should be considered case by case. Thus, further research is necessary to help define and streamline the process of validating health-forecasting models.

Conclusion

The review identifies a number of knowledge gaps in health forecasting, which presents a challenge for further studies.

These gaps include the following:

Typologies that classify health-forecasting approaches and methods;

A clear definition for health data, which nonetheless is an important ingredient for health forecasting;

Discussions on the strengths and limitations of statistical methods that are applicable to health forecasting, particularly for extreme health events;

A classification and ranking of various accuracy measures applicable to health forecasting; and

A clearly defined approach to cross validation of health-forecasting models.

The classifications of forecasting approaches have evolved over time. Several researchers have attempted to classify forecasting methods into typologies that are concise, exclusive, and exhaustive for all purposes. Lessons learned from these attempts will serve as useful guides in developing health-forecasting classification topologies and schemes, which are currently nonexistent. Few statistical methods have been identified to forecast extreme health events. Compared with percentage and relative error measures, scale-dependent error measures are easier and more frequently used in health forecasting. Because no common guidelines are available for cross validation in health forecasting, the current practice is quite irregular. Therefore, detailed studies are needed to help define standard classifications and applications for health forecasting.