Quantifying the Impact of Unobserved Heterogeneity on Inference from the Logistic Model.

While consequences of unobserved heterogeneity such as biased estimates of binary response regression models are generally known; quantifying these and awareness of situations with more serious impact on inference is however, remarkably lacking. This study examines the effect of unobserved heterogeneity on estimates of the standard logistic model. An estimate of bias was derived for the maximum likelihood estimator βˆ, and simulated data was used to investigate a range of situations that influence size of bias due to unobserved heterogeneity. It was found that the position of the probabilities, along the logistic curve, and the variance of the unobserved heterogeneity, were important determinants of size of bias.

1. Introduction

Theoretical models, such as health modes, generally conceptualize outcomes as a result of interaction among a complex set of components including biological, genetic, behavioral, and socio-economics (Mosley and Chen, 2003; World Health Organization, 2001). In practical situations of data analysis, however, it is not possible to account for all variables that result in an outcome by including these as explanatory variables in a statistical model. Even in the richest model specification, several factors would be unobserved, immeasurable or unknown, and some of these would be of high importance to the resulting outcome (Lee and Lee, 2003; Zohoori and Savitz, 1997). Nonetheless, it is not uncommon in many specialized journals to find some conclusions that were reached on the basis of inference from observed variables, assuming unobserved heterogeneity is of little relevance. Economists were puzzled for nearly two decades by the spurious positive association between drinking alcohol (medically known as drug with depressant properties and is unlikely to positively affect productivity) and high wage, where drinkers were persistently found to earn more than alcohol abstainers, the association was reversed by the introduction of individual specific fixed effect, which rid results of bias due to time invariant unobserved heterogeneity (Peters, 2004).

Unobserved factors whether environmental or personal may have a large effect on outcomes of relevance (for example, health, economics, social). Ignoring such effects may lead to the identification of incorrect risk factors, in addition the magnitude of the association of these may be so seriously biased and as a result conclusions may be misleading.

The objectives of this study were to quantify bias of the maximum likelihood (ML) estimate, of a standard logistic model due to un-modeled unobserved heterogeneity, and to highlight situations where the impact of such bias was more serious. Using Taylor series theory, an approximate estimate of bias was derived then simulation was used to investigate situations that were thought to affect the size of bias. The importance of higher-order terms of the derived approximation was also examined under various situations, including different variance of unobserved heterogeneity and differing positions for the probabilities of outcome within the logistic curve. The estimation described was confined to the case of a single binary explanatory variable x, a binary response y, and unobserved heterogeneity that was linked to each individual.

It was found that in most situations the first-order approximation defined as δ1, provides an adequate approximation of bias due to unobserved heterogeneity. At special situations with large variance and large difference between the two probabilities, however, the first-order approximation becomes inadequate, and does worse as the difference between the two probabilities increases.

2. Methods

2.1. Assumptions and Models

We consider a hypothetical simple example, i.e., lung cancer, as an outcome, and smoking as the only explanatory variable. It was also assumed that there are other factors that were likely to cause lung cancer but these were either unobserved, difficult to measure, or were totally unknown. These may include some genetic factors, personal differences in diet, childhood exposures, lifestyle, or other individual specific factors. In a statistical model, the relationship may be expressed as: where y was the outcome, x was the observed binary explanatory variable, and ϵ was the unobserved variable or variables. The conditional expectation of y given x and ϵ, may be written as:

Under the linear model assumptions, the estimates of β1 will be unbiased whether ϵ, was considered by the model or not. At other situations, where nonlinear models such as the logistic regression model (Agresti, 2002) were used, the effect of the omission of ϵ on estimates was, however, different. The situation with a missing binary variable was described before (Gail et al., 1984). Here, we shall consider the situation, where the omitted variable was continuous which is quite common in many research areas, such as health. The probability of positive outcome (response) in our example with one explanatory variable x, may be written as: and the logit as:

The maximum likelihood (ML) estimate equation for such a model may be written as: where n1 and n0 were the numbers of subjects exposed (X = 1) and unexposed (X = 0), respectively, and the conditional probabilities, and , are the probabilities of positive response, among the exposed and unexposed subjects, respectively.

2.2. Estimation of Bias, Using Taylor Series Expansion: The Uncorrelated Case

We consider the situation where sampled observations were identically independently distributed (IID), expected value of y, E(Y) exists, X was a binary variable (0/1), and ϵ was a random variable that was unobserved but has an influence on the response y. The unobserved term was assumed to follow a normal distribution, with mean zero and a variance . The maximum likelihood estimator, for the logistic model, with one explanatory variable x, was first obtained ignoring unobserved heterogeneity; the effect of unobserved heterogeneity, ϵ on the response was then brought in, and the expected value of the (ML) estimator was evaluated again. Taylor series expansion was used and the estimates were compared (ignoring unobserved heterogeneity and including the influence of the term on the response y). The derivation showed that the (ML) estimator is biased due to un-modeled unobserved heterogeneity and the bias may be approximated by a first order term δ1, of Taylor series as follows:

The bias may also be approximated by the first- and second-order terms, δ1 + δ2, and/or by first-, second-, and third-order terms, δ1 + δ2 + δ3, where δ for i = 1, 2, and 3 together with details of the terms g1(p1) and g1(p0) were fully described in Appendix A, was the derivation steps.

2.3. Simulation

Simulation was used to examine situations that were thought to affect the severity of bias. Data was simulated from a logistic model with a response y, an explanatory variable x, and an extra term representing unobserved heterogeneity. A Fortran program was written to generate data from a logit function of the form: where x is a vector of a binary variable for subjects j = 1, 2,…, n, pj = p(yj = 1 | x) is the conditional probability, β0 and β1 are the logistic regression parameters, and ϵ an extra term representing unobserved heterogeneity, assumed to be normally distributed random variable with zero mean and a variance . Several parameter values were considered, so that they cover a range of probabilities, including situations where one of the probabilities was 0.5, the two probabilities were lying on one side of 0.5, and where they were lying further apart on the two sides of 0.5. We also considered a range of values for the variance of the unobserved heterogeneity term. The standard logistic model was then fitted to predict the response y, using x as the only explanatory variable, in order to detect the effect of ignoring unobserved heterogeneity on the estimated parameter, . Estimates were calculated from 100 simulations, each based on a sample of 1,000 of identically independently distributed (IID) individuals.

3. Results

3.1. Theoretical Bias: The Effect of Position of the Two Probabilities

We examined the size of bias with focus on the importance of first-, second-, and third-order terms in the overall bias approximation. Table 1 illustrates the contribution of each term under varying situations. Different positions of the two probabilities were examined, while the variance of unobserved heterogeneity was kept fixed at 1.0. The positions considered include: (i) the two probabilities lie on one side of 0.5 of the logistic curve; (ii) one of the probabilities is 0.5; and (iii) the two probabilities lie on the opposite side of 0.5.

Table 1

Theoretical contribution of first-, second-, and third-order terms to the approximation of bias due to unobserved heterogeneity, with variance = 1.0

p0, p1	δ1	δ2	δ3
0.73,0.82	−0.08	0.02	0.0
0.88,0.99	−0.28	0.13	−0.03
0.73,0.88	−0.15	0.04	−0.01
0.50, 0.38	−0.08	0.0	0.0
0.50, 0.73	−0.18	0.0	0.0
0.50, 0.82	−0.26	0.02	−0.01
0.50, 0.92	−0.39	0.09	−0.02
0.38, 0.62	−0.20	0.0	0.0
0.27, 0.73	−0.36	0.01	0.0
0.12, 0.88	−0.67	0.10	−0.03

For all three situations, the first term δ1 was dominant, the second-order term is of less importance, and the third-order term has a very small contribution that may be safely ignored at almost all situations. There are, however, special situations where the second-order term becomes more important, for example, as one probability approaches 0.9.

3.2. Simulation Results: Theoretical and Numerical Bias

Table 2, reports the average estimates of 100 simulations, each based on a sample of size 1,000. The table examined various positions for the two probabilities, and a range of variance for unobserved heterogeneity (1.0−0.01) including the special case where unobserved heterogeneity, was almost constant ( = 0.01). Three situations regarding the position of the two probabilities were covered as described earlier. A comparison was then drawn between the theoretical bias, and the numerical bias, which was calculated from the simulation. The 95% confidence intervals for the simulated bias were reported and the number of times the confidence intervals of the estimate , fail to cover the true parameter β was reported as percentage rejected (Rej%). The main findings may summarized as:

Table 2

A comparison between theoretical bias and numerical (simulation) bias, for the (MLH) estimator , of the logistic model, for various probabilities, and for a range of variance (0.01–1.0) for the unobserved heterogeneity term ϵ

Variance	Theoretical bias				Simulation bias
σε2	δ1	δ2	δ3	Total	Bias βˆ – β	95% C.I	Rej%	(p0,p1)
1.0	−0.12	0.0	0.0	−0.12	−0.12	−0.15,-0.09	13	0.62, 0.73
0.75	−0.07	0.01	0.0	−0.06	−0.07	−0.10,-0.04	12
0.5	−0.04	0.0	0.0	−0.05	−0.05	−0.06,0.0	7
0.1	−0.01	0.0	0.0	−0.01	−0.01	−0.04, 0.02	6
0.01	0.0	0.0	0.0	0.0	0.01	−0.03,0.03	3
1.0	−0.39	0.11	−0.03	−0.31	−0.33	−0.36, -0.30	36	0.62, 0.95
0.75	−0.29	0.06	−0.01	−0.24	−0.23	−0.27, -0.19	17
0.5	−0.19	0.02	0.0	−0.17	−0.15	−0.19,-0.11	15
0.10	−0.03	0.0	0.0	−0.03	−0.04	−0.09,0.01	6
0.01	0.0	0.0	0.0	0.0	0.05	0.0, 0.10	4
1.0	−0.09	0	0	−0.10	−0.08	−0.10,-0.06	12	0.5, 0.62
0.75	−0.07	0	0	−0.07	−0.06	−0.08, -0.04	8
0.5	−0.05	0	0	−0.05	−0.05	−0.07, -0.03	5
0.1	−0.01	0	0	−0.01	0	−0.03,0.03	4
0.01	0.0	0	0	0.0	0.01	−0.02, 0.02	2
1.0	−0.34	0.05	−0.01	−0.30	−0.29	−0.32, -0.26	49	0.5, 0.88
0.75	−0.27	0.03	−0.01	−0.24	−0.23	−0.27, -0.21	27
0.5	−0.18	0.01	0	−0.17	−0.16	−0.19,-0.13	14
0.1	−0.04	0	0	−0.04	−0.03	−0.06,0.0	4
0.01	0	0	0	0.0	−0.01	−0.02, 0.02	3
1.0	−0.48	0.11	−0.03	−0.40	−0.39	−0.43, -0.35	49	0.5, 0.95
0.75	−0.36	0.04	−0.01	−0.32	−0.31	−0.35, -0.27	32
0.5	−0.25	0.04	0	−0.21	−0.20	−0.24,-0.16	21
0.1	−0.05	0	0	−0.04	−0.03	−0.08, 0.02	7
0.01	0.0	0	0	0	0.06	0.01, 0.11	5
1.0	−0.20	0	0	−0.20	−0.17	−0.19,-0.15	20	0.38, 0.62
0.75	−0.15	0	0	−0.15	−0.14	−0.17,-0.11	18
0.5	−0.10	0	0	−0.10	−0.10	−0.13,-0.07	13
0.10	−0.02	0	0	−0.02	−0.03	−0.06, 0.0	10
0.01	0	0	0	0	.03	−0.01, 0.04	3

Note: 95% C.I : upper and lower 95% confidence intervals for the simulated bias.

Rej%: the number of times, in percentage that the true parameter β, lied outside the 95% confidence intervals of the simulated estimator .

The bias was well approximated by the first-order term at most of the situations considered.

The bias becomes more serious as the difference between the two probabilities gets larger, especially as one probability gets closer to 0.9 (the situation with one probability approaching 0.1 is identical) or where the two probabilities lie further apart at the two sides of the logistic curve.

The bias was more serious for relatively large variance of unobserved heterogeneity, 1.0 and 0.75, from the range of values considered.

Contributions from the second-order term to the bias approximation become of some importance at special situation where the difference between the two probabilities was large, and the variance of unobserved heterogeneity was relatively large.

For small variance of unobserved heterogeneity, the bias was small and the percentage of rejections was modest.

For the special case of unobserved heterogeneity with variance = 0.01, the bias either disappeared or became negligible, and the percentage of estimates outside the coverage property of was particularly small, less than 5% in most cases.

4. Discussion

Much attention in the 1980's, and after, was given to the asymptotic bias of the maximum likelihood (ML) estimators of binary response regression models, that are widely used to describe associations between binary outcome and explanatory variables in trials and surveys. In general, ML estimators may not hold in small and finite samples, as shown by Anderson and Richardson (1979) where a simulation was used to investigate bias of the logistic model estimates, the study found that bias can be substantial if the sample size is small, a formulae for correction was developed. Another similar study (Griffiths et al., 1987) examined the bias and other sample properties such as mean square error based on three alternative covariance matrix estimators for the Probit model, also reached the same conclusion with regard to the bias. A simpler formula using Taylor series expansion for correction of bias in logistic ML estimate was also developed by Copas (1988). For exponential family such as the logistic model, the bias was of order 0(1/n) suggesting that for large samples it was negligible relative to the standard errors of the estimates, the bias was treated by Jeffery's priors' as reported in McCullagh and Nelder (1989) and Firth (1992). A set of GLIM macros was developed to reduce bias (Firth, 1993; Steyerberg and Eijkemans, 2004), but the reduction achieved was reported to be small.

Another issue of importance to the (ML) estimation was the deviation of data from the assumed identical independent distribution, that was addressed in survey methodology and procedures for correction of estimates were developed (Skinner and Smith, 1989).

Of no less importance was the problem of unobserved heterogeneity, although awareness of the problem has recently increased (Aprahamian et al., 2007; Arana and Leon, 2006; Cramer, 2007), but still many influential articles continue to report important findings ignoring the possibility of any impact of unobserved heterogeneity on these findings. In practice, in almost any biological investigation, there are factors (exogenous or endogenous, independent of a biological process, or part of it, time varying or time invariant, personal or contextual) that would be unobserved. Using nonlinear models under such situations lead to biased estimates of population parameters.

Here we present the case with unobserved heterogeneity that was linked to individuals (for example, taste, charisma, emotions), another scenario is where unobserved heterogeneity was correlated, that may occur with repeated measurements within individual, or due to clusters such as household (family related gene, for example), more details on the correlated case were reported in an earlier study (Ayis, 1995) where it was shown that the leading term of bias approximation for the correlated case was the same as that for the non correlated. An extra term, due to replications, however, becomes important if the number of clusters was small, and where the two probabilities lie further apart within the logistic curve.

While there are situations where estimates from the logistic model may be fairly robust to unobserved heterogeneity, there are others where the problem deserves more attention. For situations with outcomes such as fertility or incidence of disease, where all of the probabilities were on the same side of 0.5, the potential for bias was there, but perhaps not as bad as where extreme probabilities occur in both tails, for example (p0 ≤ 0.2, p1 ≥ 0.8). The work by Copas (1988) is also relevant to the latter situation of extreme probabilities, although the assumption was that extreme values occur due to mis-recording, that is where the values of the response “y” was being transposed in error between 0 and 1, rather than due to the nature of the association between the response and the explanatory variable we present. Mont Carlo simulation was used to examine the sensitivity of different binary response models to such extreme values of probabilities, a model was proposed to allow for robust estimation where a small number of outcomes was being mis-recorded, and techniques for diagnostics where developed. For situations like the one we present in this study, extreme values will be more common, but detection of such values may help in assessing the quality of the estimates and possibly in whether to use alternative methods at situations were the bias is serious.

The misspecification bias of the logistic model, due to unobserved heterogeneity described in this study, is similar to the misspecification bias due to incorrectly assuming the error term was logistically distributed when it was not, as described in Horowitz (1993), where the effect on estimates was measured using simulation. The bias was found to be small as long as the assumed error distribution has the same qualitative shape as the true distribution (unimodal for the logistic case considered) and more serious when the true distribution of the error term was bimodal or heteroskedastic. These findings suggest the need to explore the effect of other forms of distribution on the bias derived in this study. Similar findings were shown by Arana and Leon (2005) where a Monte-Carlo simulation was used to test the performance of a Bayesian mixture normal distribution (semi parametric model), with other parametric models (including the logit) and nonparametric models, using alternative assumptions for the error distribution and using different sample sizes, when data exhibit unobserved heterogeneity. The mean square error (MSE) in all models and for all sample sizes was found to be considerably large reflecting the difficulty in modeling this type of data. The Bayesian specification, however, performed better than the competing models for small as well as for large samples, and substantially reduces the bias and the MSE, the improvements in bias was larger for small samples.

Misspecification bias due to unobserved heterogeneity can seriously affect estimates from the logistic model as well as other binary regression models. Using panel data and suitable random effect models that allow for individual fixed effect adjustment is a solution, but obtaining such data may not always be easy. Attempting other semiparametric models such as Bayesian normal mixture model seems to be an appealing approach, especially as suitable software are rapidly developing. Further work is needed to examine the impact on estimates in real situations where there are several explanatory variables often correlated. Methods of detection may also be developed to assess the need for alternative more flexible models, at situations where unobserved heterogeneity is likely to have more serious impact on the estimates due to data structure, before drawing inference that might be misleading.