Response to: Reliability and validity of telomere length measurements.

In his letter on our recent commentary, Dan Eisenberg argues that Southern blot (SB) and quantitative polymerase chain reaction (qPCR)-based methods have similar reliability of leukocyte telomere length (LTL) measurement, a suggestion that runs counter to the literature going back more than a decade. This literature shows that the SB-based method better captures associations of LTL with a host of variables than does the qPCR-based method. For example, a recent meta-analysis concluded that the influence of sex on LTL was apparent when using SB, but absent when using qPCR, and we see a difference in measurement reliability as the most parsimonious explanation for this finding. We consider this meta-analysis important because it compares how these two methods perform in actual epidemiological studies. Such a comparison cannot be derived from the results achieved in small samples designed to assess reliability in laboratory settings (a point elaborated later). Likewise, with respect to LTL dynamics (LTL and its age-dependent change), lower reliability is also the most likely explanation we can see for the much larger proportions of individuals with counter-intuitive elongation of LTL as they age (i.e. at the second measurement time point) in qPCR-based than in SB- based studies. Moreover, lower reliability primarily explains the difficulties in detecting the longer LTL in African Americans than in Whites of European ancestry when using the qPCR-based method. Thus, the epidemiological telomere literature strongly suggests higher reliability of SB compared with qPCR.

That said, we welcome Dan Eisenberg’s letter since it provides a forum to further address highly relevant issues related to LTL measurements by the SB-based method, which measures LTL in kilo-bases (kb), and the qPCR-based method, which measures telomere DNA content as the ratio of telomeric PCR product to single copy gene PCR product (T/S)., Below we respond to Eisenberg’s points on: (i) the coefficient of variation (CV) as a tool to compare measurement reliability between LTL data generated by SB versus by qPCR; and (ii) the need for external validation of LTL measured by the two methods. We also address other validity-related matters.

The coefficient of variation

In our commentary we suggested that it is not appropriate to use the CV to measure reliability for qPCR, nor to use the CV to compare reliability of qPCR and SB. We pointed out that the heteroscedasticity assumption of the CV does not hold for qPCR measurements. This reason alone is sufficient to require an alternative method to quantify and compare measurement reliability, for which we proposed considering the intraclass correlation coefficient (ICC), a well-established method in other research fields.

However, Eisenberg makes the additional compelling point that when using the CV to compare two methods, one assumes that both measures have the same ‘natural zero points’. He goes on to argue that this assumption does not hold for SB, because the terminal restriction fragments (TRFs) include sub-telomeric regions, and we agree. In making this assertion about the dependence of CV on equal zero points, Eisenberg also makes incorrect assertions, and these are unfortunately conflated because they are illustrated together in his Table 1. He suggests that the higher CVs reported for the qPCR than the SB are entirely due to the different measurement scales used. To equate the two measurement scales, he (i) converted T/S from a ratio to an absolute measure of LTL, and (ii) estimated the sub-telomeric length, which is included in the terminal restriction fragments that are measured by the SB. Both of these procedures are fraught with problems and we believe that his conversions are erroneous. Moreover, Eisenberg’s assertion that qPCR-based TL measurements have a natural zero point may not generally be correct, due to the way it is calculated. We discuss these issues in the Supplementary Data (available as Supplementary Data at IJE online) as it would require us to digress from the main points by including details and complexities that many readers may not find germane.

External validation of LTL measured by the two methods

Validity of measurement depends upon measurement reliability, but reliable measurement does not ensure validity. Eisenberg suggests, therefore, a way to gauge the validity of LTL measures. He proposes that ‘well established correlates of LTL can be utilized as markers of external validity’. We agree that when LTL is measured by multiple methods that differ only in random measurement error (reliability), one does generally expect stronger and more valid correlations for the more reliable method with ‘well-established correlates’. Nevertheless, we do not think that his proposal, a comparison of reported correlations, should serve to assess either validity or reliability. First, Eisenberg’s approach cannot be used to assess validity because both methods could be biased; an independent benchmark criterion is needed to gauge validity. We do not yet have such a benchmark. Thus, one correlation may be stronger than another, but how much closer the best correlation is to the (unknown) benchmark value remains unknown. Second, Eisenberg’s approach is in practice not informative with respect to measurement reliability. Typically, the sample sizes in studies that compared different methods to measure TL are too small to detect relevant differences in measurement reliability using this approach.

We illustrate this concept in Figure 1, where it can be seen that even very large differences in measurement reliability are difficult to detect with sample sizes of 50-200. Thus, the estimates that Eisenberg reports for TL associations with age for sample sizes of 50-190 (Table 2 in his letter) may be viewed as random samples from the broad distributions in our Figure 1 (albeit they are not entirely independent because they are based on the same set of samples within each study). Differences in the distribution of age (frequently non-normal) will also skew correlations between studies, also arguing against the use of age as a comparative measure for validity between studies. Due to the limited sample sizes, possible differences in age distribution between studies and consequently large sampling variation, we attribute little value to these point estimates or Eisenberg’s conclusion that there was on average no difference between SB- and qPCR-based studies.

Figure 1.

Estimated correlations r and 95% confidence intervals (Y axis) depending on the true value of the correlation being estimated (X axis) for different levels of measurement reliability (ICC values 0.5, 0.75 and 1). The estimates of r deviate more from the true value when the ICC is lower. Increasing sample size (50, 100 or 200) decreases the confidence interval of the estimate, but does not affect the average point estimate. Note that confidence intervals are large relative to the effect of the ICC differences on the r estimates, and hence comparing correlation coefficients will have low statistical power to detect effects of measurement reliability on r. Note further that correlations in epidemiological telomere studies will reach values as high as the upper limit (0.7) in this example only in exceptional circumstances.

Validity-related matters

Eisenberg notes that the qPCR assay is considerably influenced by DNA extraction techniques and sample handling. In our commentary, based on theoretical considerations, we also suggested that the integrity of the amplifiable sequences by the qPCR might be critical for obtaining accurate results. This point has recently been confirmed in a study showing that DNA integrity has a considerable effect on TL measurements by qPCR. The requirement to test DNA integrity in each sample, which is a routine for SB, is typically not done in qPCR-based studies of LTL and reduces the high-throughput advantage of the qPCR-based method.

Finally, there is more to understanding the role of telomeres in health and disease than just knowing that telomeres are longer (or shorter) in one individual versus another. Consequently, many studies have transformed their qPCR data, expressed in T/S units, to kb, because the information in T/S data is limited to relative differences that cannot be compared between studies. The pitfalls of transforming the T/S to kb become apparent in Eisenberg’s own work. His transformation of T/S to kb produced an implausible range of LTL across European populations, i.e. from 5.1 kb in Naples to 18.6 kb in Ghent. In his letter, Eisenberg attributes this to the high inter-laboratory variation in generating T/S, which can be considerable according to the paper that was the subject of our commentary. However, if that was the only reason, the range of variation in LTL resulting from transforming qPCR measures generated in a single laboratory to LTL data should be similar to measures generated by SB. This is hardly the case, since the range of transformed qPCR data is considerably larger than the range generated by SB., A likely explanation is that transforming qPCR to SB data is based on linear models, whereas the relation between data generated by qPCR and SB is often curvilinear (see Supplementary Data, available as Supplementary Data at IJE online).

Conclusions

The SB- and the qPCR-based methods of LTL measurements have been important partners in gaining insight into the role of telomeres in human health and disease. Together they have been instrumental, for instance, in deciphering the LTL-SNPs that opened the door to remarkable advances in understanding the role of telomeres in a host of human diseases. The lower measurement reliability of the qPCR-based method renders it less suited to studies with small sample-sizes, but the high throughput of the method makes it a powerful tool in large epidemiological studies where increasing sample size can offset higher measurement error. Note, however, that lower measurement reliability and the attendant non-differential misclassification also attenuates effect size estimates, and the attenuation effect is independent of sample size (Figure 1). Thus, as we stated before, we see a role for both methods, depending on characteristics of the study design. However, in the light of the epidemiological literature, we consider attempts to show that the qPCR-based method is equal to the SB method in reliability to be misleading.

Eisenberg and we agree that the CV is a suboptimal indicator of measurement reliability, supporting the use, for example, of the ICC to evaluate measurement reliability in future telomere studies. Notably, the ICC can be calculated in different ways that provide different information, and it is important therefore to report exactly how the ICC was calculated. Should inter-assay ICCs or other substitutes for CV become standard in telomere epidemiology, the data will be available to make a more quantitative assessment of the reliability of the different techniques as they are applied in the field.

Click here for additional data file.