Fixed-effect variance and the estimation of repeatabilities and heritabilities: issues and solutions.

Linear mixed-effects models are frequently used for estimating quantitative genetic parameters, including the heritability, as well as the repeatability, of traits. Heritability acts as a filter that determines how efficiently phenotypic selection translates into evolutionary change, whereas repeatability informs us about the individual consistency of phenotypic traits. As quantities of biological interest, it is important that the denominator, the phenotypic variance in both cases, reflects the amount of phenotypic variance in the relevant ecological setting. The current practice of quantifying heritabilities and repeatabilities from mixed-effects models frequently deprives their denominator of variance explained by fixed effects (often leading to upward bias of heritabilities and repeatabilities), and it has been suggested to omit fixed effects when estimating heritabilities in particular. We advocate an alternative option of fitting models incorporating all relevant effects, while including the variance explained by fixed effects into the estimation of the phenotypic variance. The approach is easily implemented and allows optimizing the estimation of phenotypic variance, for example by the exclusion of variance arising from experimental design effects while still including all biologically relevant sources of variation. We address the estimation and interpretation of heritabilities in situations in which potential covariates are themselves heritable traits of the organism. Furthermore, we discuss complications that arise in generalized and nonlinear mixed models with fixed effects. In these cases, the variance parameters on the data scale depend on the location of the intercept and hence on the scaling of the fixed effects. Integration over the biologically relevant range of fixed effects offers a preferred solution in those situations.