Assessing reproducibility for interval data in health-related quality of life questionnaires: which coefficient should be used?

In contrast to other reliability estimates, test-retest reliability (or reproducibility) captures not only the measurement error of an assessment instrument, but also the stability of the construct measured. Consequently, one would expect any departure from identity (Y = X) of measurement pairs (X first, and Y second measurement) to be treated as 'error' by the respective reproducibility statistic, even if 'true' changes happened, e.g. worsening of a disease due to its natural course. The Pearson correlation, still often advocated for continuous measures in test-retest reliability studies, however captures the degree of linearity (Y = bX + a): perfect relationship can be computed, even if the measurement pairs differ not only by a additive constant 'a', but also because of a multiplication of the X-values with the slope 'b'. Therefore, intraclass correlation coefficients (ICCs) have been proposed as alternative statistics for reproducibility. However, only ICCs with absolute agreement definition of concordance capture the degree of identity. ICCs with a consistency definition of concordance measure the degree of additivity (Y = X + a). ICCs are calculated from repeated measures analyses of variance (ANOVAs), and a common population variance must be is assumed for the different measurements. Given this assumption, an ICC computed from a one-way ANOVA seems to be the best choice for this purpose. Otherwise, Lin's concordance correlation coefficient is recommended as identity measure.