The distribution and hypothesis testing of eigenvalues from the canonical analysis of the gamma matrix of quadratic and correlational selection gradients.

Canonical analysis measures nonlinear selection on latent axes from a rotation of the gamma matrix (gamma) of quadratic and correlation selection gradients. Here, we document that the conventional method of testing eigenvalues (double regression) under the null hypothesis of no nonlinear selection is incorrect. Through simulation we demonstrate that under the null the expectation of some eigenvalues from canonical analysis will be nonzero, which leads to unacceptably high type 1 error rates. Using a two-trait example, we prove that the expectations for both eigenvalues depend on the sampling variability of the estimates in gamma. An appropriate test is to slightly modify the double regression method by calculating permutation P-values for the ordered eigenvalues, which maintains correct type 1 error rates. Using simulated data of nonlinear selection on male guppy ornamentation, we show that the statistical power to detect curvature with canonical analysis is higher compared to relying on the estimates from gamma alone. We provide a simple R script for permutation testing of the eigenvalues to distinguish curvature in the selection surface induced by nonlinear selection from curvature induced by random processes.