Linearity and the limitations of least squares calibration.

The magnitude of errors that can arise in practice from the limitations of the least squares method of calibration is estimated. Data generated from y = xn (0.7 < or = n < or = 1.3 and 1 < or = x < or = 30, or < or = 60) was analysed by least squares regression. Each y-value was then presented to the linear model and an x-value predicted. The relative errors on small x-values reached 70% of the concentration value even when r2 exceeded 0.999. Estimates of the errors on each predicted x-value, determined from the standard errors of the slope and intercept failed to reveal large errors at small x-values. Reducing the range over which linear regression is performed improved the errors. Other data sets with a heteroscedastic error distribution show that linear regression by least squares can also lead to the rejection of methods that performed sufficiently well for their application. Heteroscedastic data may be treated by repeated measurements at the lower end of the range. Data from a validation of an HPLC method for isoflavones in legumes is used to show the errors in recovery when a check sample is presented to the instrument using a calibration which satisfies the linearity tests. It is recommended that both y- and relative x-residuals are inspected. It is proposed that over-reliance on linear calibration supported by r2 may make a major contribution to large, hitherto unexplained, inter-laboratory errors.