The impact of a misspecified random-effects distribution on the estimation and the performance of inferential procedures in generalized linear mixed models.

Estimation in generalized linear mixed models (GLMMs) is often based on maximum likelihood theory, assuming that the underlying probability model is correctly specified. However, the validity of this assumption is sometimes difficult to verify. In this paper we study, through simulations, the impact of misspecifying the random-effects distribution on the estimation and hypothesis testing in GLMMs. It is shown that the maximum likelihood estimators are inconsistent in the presence of misspecification. The bias induced in the mean-structure parameters is generally small, as far as the variability of the underlying random-effects distribution is small as well. However, the estimates of this variability are always severely biased. Given that the variance components are the only tool to study the variability of the true distribution, it is difficult to assess whether problems in the estimation of the mean structure occur. The type I error rate and the power of the commonly used inferential procedures are also severely affected. The situation is aggravated if more than one random effect is included in the model. Further, we propose to deal with possible misspecification by way of sensitivity analysis, considering several random-effects distributions. All the results are illustrated using data from a clinical trial in schizophrenia.