Standardizing total kidney volume measurements for clinical trials of autosomal dominant polycystic kidney disease.

BACKGROUND: The ability of unstandardized methods to track kidney growth in clinical trials for autosomal dominant polycystic kidney disease (ADPKD) has not been critically evaluated.
METHODS: The Tolvaptan Efficacy and Safety Management of ADPKD and its Outcomes (TEMPO) 3:4 study involved baseline and annual magnetic resonance follow-up imaging yearly for 3 years. Total kidney volume (TKV) measurements were performed on these four time points in addition to the baseline imaging in TEMPO 4:4, initially by Perceptive Informatics (Waltham, MA, USA) using planimetry (original dataset) and for this study by the Mayo Translational PKD Center using semiautomated and complementary automated methods (sequential dataset). In the original dataset, the same reader was assigned to all scans of individual patients in TEMPO 3:4, but readers were reassigned in TEMPO 4:4. Two placebo-treated cohorts were included. In the first (n = 158), intervals between the end of TEMPO 3:4 and the start of TEMPO 4:4 scan visits ranged from 12 to 403 days; in the second (n = 95), the same scan (measured twice) visit was used for both.
RESULTS: Growth rates in TEMPO 3:4 were similar in the original and sequential datasets (5.5 and 5.9%/year). Growth rates during the TEMPO 3:4 to TEMPO 4:4 interval were higher in the original (13.7%/year) but were not different in the sequential dataset (4.0%/year). Comparing volumes from the same images, TKVs showed a bias of 2.2% [95% confidence interval (CI) -5.2-9.7] in the original and -0.16% (95% CI -1.91-1.58) in the sequential dataset.
CONCLUSIONS: Despite using the same software, TKV and growth rate changes were present, likely due to reader differences in the transition from TEMPO 3:4 to TEMPO 4:4 in the original but not in the sequential dataset. Robust, standardized methods are essential in ADPKD trials to minimize errors in serial TKV measurements.

INTRODUCTION

Autosomal dominant polycystic kidney disease (ADPKD) is the most common inherited kidney disease and the fourth leading cause of end-stage renal disease in adults throughout the world [1-4]. Typical progression of the disease is characterized by the growth of fluid-filled cysts, which cause bilateral kidney enlargement, and in many cases lead to renal failure due to the destruction of healthy renal tissue [4, 5]. Markers such as glomerular filtration rate (GFR) are commonly used to measure the kidneys’ ability to filter waste products from the blood [6]. However, these markers start declining in the late stages of ADPKD [7-9]. Drug trials and other longitudinal studies are oftentimes focused on the earlier stages of the disease and therefore more sensitive measures are needed to effectively predict and measure disease progression [10].

Due to the nature of ADPKD, total kidney volume (TKV) has become the most trusted image-based biomarker for tracking its progression, especially in the early stages [6, 8, 11–13]. TKV also plays a key role in choosing patients for enrollment in clinical studies. The US Food and Drug Administration recently issued guidance on the use of TKV, in combination with age and GFR, as a qualified prognostic biomarker for use in clinical trials investigating treatments for ADPKD [14]. Many measurement techniques have been developed throughout the years, including ellipsoid, planimetry (tracing), stereology, semiautomated and fully automated techniques [15-21].

A number of recent clinical research studies have relied on TKV. The Tolvaptan Efficacy and Safety Management of ADPKD and its Outcomes (TEMPO) 3:4 study sought to compare the disease progression of patients taking the drug tolvaptan (vasopressin V2 receptor antagonist) versus patients on placebo for 3 years. Disease progression was primarily tracked using TKV by means of serial outlines (tracings) from a third-party service and involved 1445 patients (961 received tolvaptan and 484 received placebo) [12, 22–24]. The patients completing TEMPO 3:4 (except those from Japan) were invited to enter an open-label extension study for an additional 2 years (TEMPO 4:4); 871 patients were enrolled in this study. The Consortium for Radiologic Imaging Studies of Polycystic Kidney Disease (CRISP) was established to develop studies that use and validate TKV measurement techniques to track disease progression in patients with ADPKD. Initially CRISP tracked volume progression (stereology) and cyst progression (thresholding) in 241 patients and validated measurement methods using ADPKD phantoms to simulate cystic kidneys [5]. HALT Progression of Polycystic Kidney Disease (HALT PKD) Study was a 5-year study, involving 558 patients, comparing the efficacy of rigorous versus standard blood pressure control and of dual versus single renin–angiotensin blockade in early-stage ADPKD. The primary outcome measure was the change in TKV [25, 26]. More recent studies have begun using classification methods for prognostic enrichment to select patients at high risk of early disease progression [18, 27]. Volumetric analysis methods such as the ellipsoid method were used as a quick and easy way to estimate TKV for classification purposes.

Since many of the ADPKD trials use TKV as an inclusion criterion, an endpoint or data points in between, standardized methods for calculating TKV are becoming essential to accurately track kidney growth and disease progression, especially in multicenter, long-term drug trials. The current study was conducted to assess potential risks involved in using different readers from one portion of a study to the next when serial TKVs are being measured in short follow-up increments. Two sets of data were compared. The original data were obtained by a central reading facility that checked the images for quality and measured volumes. Readers were assigned to a particular group of patients but were reassigned between the TEMPO 3:4 and 4:4 trials [24]. The same images from both studies were retrospectively measured by the Imaging Core of the Mayo Translational PKD Center (sequential data) using two semiautomated TKV measurement techniques that work complementary to one another [16, 18]. Data on the placebo-treated patients in TEMPO 3:4 who enrolled in TEMPO 4:4 were then pulled to evaluate how this shift in measurement centers might affect the overall data trends and growth rates throughout both studies, particularly during the interval between the end of TEMPO 3:4 and the start of TEMPO 4:4.

MATERIALS AND METHODS

Study population

For analysis purposes, we split the time frame into four periods of time; TEMPO 3:4 baseline to 12, 12–24, 24–36 and 36 months to TEMPO 4:4 baseline. Only patients on placebo during TEMPO 3:4 were evaluated for this study to exclude the effect of tolvaptan on growth rates. Additionally, only patients who had magnetic resonance (MR) scans of sufficient quality for accurate measurements for all TEMPO 3:4 time points (baseline and 12, 24 and 36 months) and for TEMPO 4:4 baseline were included (Figure 1). Two separate cohorts were included for the purpose of analysis: patients with end-of-study TEMPO 3:4 and baseline TEMPO 4:4 MR scans obtained on different dates (with intervals ranging from 12 to 503 days) and patients whose end-of-study TEMPO 3:4 MR scan was also used as the TEMPO 4:4 baseline scan. Figure 1 displays the way in which these cohorts were determined.

FIGURE 1

Method for selecting the patient population from the TEMPO study.

MR imaging data

TEMPO 3:4 was a multicenter, double-blind, placebo-controlled, 3-year trial in which 1445 participants were randomly assigned 2:1 to split-dose tolvaptan or placebo. The primary endpoint of the TEMPO 3:4 trial was the rate of TKV change (normalized to percentage) for tolvaptan relative to placebo. TKV was calculated from standardized MR scans, which were obtained at baseline and at Months 12, 24 and 36. The magnetic resonance imaging (MRI) acquisition protocol included coronal 4-mm slice thickness covering the entire kidneys during breath holds acquired with the following contrast weighting: T2-weighted single-shot fast spin echo with fat saturation and T1-weighted images without fat saturation. In addition, T2/T1-weighted fast imaging with steady-state precession series were added at some sites to aid in kidney border definition [23].

TEMPO 4:4 was a 2-year extension of TEMPO 3:4. Images were acquired in a similar manner in both trials [22].

Original measurement methods

All MR images from TEMPO 3:4 were sent to a central reading facility for quality control and evaluation of TKV using T1- and T2-weighted MR images. Alice software (Perceptive Informatics, Waltham, MA, USA; www.perceptive.com) was used to calculate TKV from serial kidney outlines that had been verified by independent radiologists familiar with ADPKD. Readers were blinded to patient name, site identifiers, assessments, determinations and sequence of acquisition [24]. All sequential images from each patient were read by the same reader in TEMPO 3:4 but the images from each patient were not necessarily measured in sequential order.

All MR images from TEMPO 4:4 were measured in the same manner but the readers were reassigned, meaning each patient could have been assigned to a different reader in TEMPO 4:4 versus the one in TEMPO 3:4. As described above, the second cohort consists of patients whose TEMPO 3:4 36-month scans were also used as their TEMPO 4:4 baseline scans. Even though they were identical, they were measured twice, first with the 3:4 set as the 36-month scan and then a second time with the 4:4 set as the baseline scan.

Sequential measurement methods

A trained medical image analyst performed kidney segmentations on all baseline images using a semiautomated method called MIROS [16]. The analyst provided a crude polygon contour of each kidney every third slice (every 12 mm). After MIROS automatically completed and refined the segmentation, the interactive tools were used to perform quality assurance and to finalize the segmentation. This process was followed by a complementary automated registration program that uses the completed baseline segmentation data to segment any follow-up scans from the same patient [18]. In some cases, the same MRI series was not available for a follow-up scan compared with its preceding scan. In these cases, the follow-up image was measured manually. The identical TEMPO 3:4 36-month and TEMPO 4:4 baseline scans from the second cohort were also measured twice using the sequential method, along with quality control edits.

Statistical analyses

The sequential automated and semiautomated methods were used for comparison with the original study data.

For the first cohort, comparison statistics were generated from TEMPO 3:4 annualized growth rates (percentage change per year) and the annualized growth rates during the intervals between the end of TEMPO 3:4 and the start of TEMPO 4:4. The annualized percentage growth rates for the four periods were calculated by dividing the percent growth during each period by the duration of that period. The three TEMPO 3:4 intervals were roughly 1 year but the interval period between TEMPO 3:4 and TEMPO 4:4 ranged from 0.033 to 1.4 years (with an average of 0.35 years). Inconsistent measurements between TEMPO 3:4 and 4:4 would be expected to result in a difference in growth rates when the switch occurred (during the interval period). Significance was determined through analysis of variance (ANOVA). Due to a relatively short mean interval period, Bland–Altman plots were generated to assess the effect a short interval period may have on the measurement variances.

Bland–Altman plots were also generated using data from the second cohort (cohort without an interval period). The purpose of this analysis was to observe the TKV measurement variance of the same image when read by two different readers at two different time points. Again, the original study data were compared with the sequential study data.

RESULTS

The growth rates for each time point in the first cohort are displayed in Figure 2 using box plots. The original data were compared with the new sequential data. As determined by ANOVA, annualized growth rates between TEMPO 3:4 and the interval period in the sequential data are not significantly different. However, in the original data, the annualized growth rate during the interval period is significantly larger compared with those during the three periods of TEMPO 3:4. There is not a significant difference in growth rates during the three periods of TEMPO 3:4 between the two datasets. The original data show that in this cohort the median annualized growth rate was 5.5% during TEMPO 3:4 and 13.7% during the interval period. The data from the sequential TKV measurement method show that the average annualized growth rate was 5.9% during TEMPO 3:4 and 4.0% during the interval period.

FIGURE 2

The box plot shows the growth rate during each TEMPO 3:4 period and the growth rate during the gap period between TEMPO 3:4 and TEMPO 4:4. The original study data are shown along with the sequential data.

The interval period between the end of TEMPO 3:4 and the start of TEMPO 4:4 ranged from 12 to 503 days. Bland–Altman plots were generated to display the measurement variability depending on the duration of the interval period. Variability in growth rate between TEMPO 3:4 and the interval period was highest when the latter was shortest, suggesting an inverse relationship (Figure 3). The bias was 9.5% [95% confidence interval (CI) −75–94] in the original study data and 3.0% (95% CI −65–71) in the sequential measurement method data. When interval periods ≤30 days are excluded from the datasets, the bias decreases to 9% (95% CI −41–59) for the original study data and −1.9% (95% CI −37–33) for the sequential study data.

FIGURE 3

Bland–Altman plots show growth rate variability among all patients from TEMPO 3:4 that continued onto TEMPO 4:4. The plots compare the growth rate of TEMPO 3:4 with the growth rate of the gap period for each delayed-treatment patient. The gap period represents the number of days between the end of TEMPO 3:4 and the beginning of 4:4. Variability is relatively high in patients with a short gap period because we have annualized the gap growth rate, which thereby magnifies the measurement error proportionally inverse to the duration of the gap. Each point represents one patient. The bias for the original study data was 9.5% (95% CI −75–94). The bias for the sequential study data was 3.0% (95% CI −65–71). When a gap period ≤30 days is excluded from this dataset, the bias reduces to 9% (95% CI −41–59) in the original dataset and −1.9% (95% CI −37–33) in the sequential dataset.

Bland–Altman analyses of the repeated measurements of TKV using the same scans in the second cohort (cohort without an interval period) are shown in Figure 4. The bias was 2.2% (95% CI −5.2–9.7) for the original data and −0.16% (95% CI −1.91–1.58) for the sequential measurement method data.

FIGURE 4

Bland–Altman plots comparing calculated volumes of repeat scans. The bias for the original study data was 2.2% (95% CI ;−5.2–9.7). The bias for the sequential study data was −0.16% (95% CI −1.91–1.58).

The scans as well as TKVs from individual patients displayed in Figure 5 illustrate the higher variance of TKV between TEMPO 3:4 and TEMPO 4:4 in the original compared with the sequential measurements.

FIGURE 5

Images and TKVs from three patients in this study representing common trends seen in the data. Original growth rates during the gap period where a change in readers occurred appear to be high compared with growth rates from the sequential data. The reason for the high variability between trials in the original study data is likely due to interreader variability.

DISCUSSION

The annualized growth rates from individual time point to time point in TEMPO 3:4 were not different between the two sets of measurements (Figure 2). However, the original data showed a significant increase in the annualized growth rate during the interval between the end of TEMPO 3:4 and the start of TEMPO 4:4, which was not seen in the data using the sequential measurement method (Figure 2). This was likely due to changes in analysts between TEMPO 3:4 and TEMPO 4:4 in the original data, while the sequential measurements used the same standardized, automated method throughout both trials (TEMPO 3:4 and TEMPO 4:4). The sequential measurement method used prior segmentations from the same patient to create segmentations for any follow-up images of the same MRI scan type. In contrast, the original measurement method was designed so that a blinded read of each image was performed randomly instead of sequential reads. Nonsequential reads can sometimes lead to inconsistent boundary definition. For example, an undefinable cyst may have been attributed to a kidney at one time point and the liver at a different time point.

Both sets of measurements found that TKV growth rate measurements can be highly variable when the time between MR scans is very short (Figure 3). This is likely due to the small denominator in the growth rate calculation. For example, if two scans from the same patient are taken within 15 days of each other, a measurement percent error of only 1% (percent error for the registration approach is <2%) [18] results in a 24% error when converted to percent per year. The plots also show variability in the sequential measurements decreases more quickly compared with the variability in the original measurements.

In the original study data, measurements of the same scan were performed at different times and by different readers. This method showed much larger variability overall compared with a method that uses an automated registration program to segment follow-up scans from the same patient (Figures 4 and 5).

In many studies, the same measurement tools are being used and an inclusion/exclusion agreement is established (e.g. whether or not to include the renal pelvis) [28]. However, when measurements are performed across multiple centers using varying tools for the same study, method biases and user biases become magnified. Even in highly controlled follow-up studies spanning many years it may be difficult to keep the analysts and software completely consistent throughout the entire time frame due to the dynamic nature of the field. As shown in the data, these inconsistencies may lead to skewed conclusions on the specifics of growth rate. Consistency is most important in longitudinal study designs and semi- to fully automated methods appear to be the solution. Not only are these automated methods objective, unlike human-dependent methods, they allow for robust measurements across readers and even across centers.

A measurement technique often used in this field is based on approximating the kidneys as ellipsoids (by measuring individual dimensions of the kidneys, such as width, depth and length). Although the ellipsoid equation is good enough for prognostication, it is not sufficient for measuring outcomes where more precision is needed, such as the longitudinal drug study described in this current study, or for the purpose of following patients in the clinic, especially those in the early stages of the disease. In other words, the ellipsoid method can be used as a starting point for classifying a patient but the measurements are not sensitive enough to monitor annual volume changes and therefore are not a good technique to properly track disease progression.

As measurements within this field become more complex, standardized measurement methods along with standardized measurement practices are essential, especially when comparing across measurement sites in longitudinal studies. While there exists a potential for bias when a reader/software has and uses a baseline reference for measurement, as long as this reader is blinded to treatment, using that reference may be important or even critical to producing an accurate estimate of the individuals and/or the population’s true growth rates. Standardized methods for tracking cyst growth have been of more recent interest in the field [29-31]. This is both extremely time consuming and more variable than TKV due to the size, shape or intensity of the cysts, meaning set standards for these measurements are necessary in order to make accurate comparisons [32]. TKV has been a long-standing disease progression marker and continues to play a key role in tracking disease progression. However, overall, TKV as a global measure ignores the intricacies of the changes at the cyst level and associated pathologies such as inflammation and fibrosis [32, 33]. Automated texture feature analysis has recently begun to address these limitations [34]. Additionally, more standardized and automated cyst segmentation methods need to be developed similar in theory to the algorithms discussed in this study.

CONCLUSION

The demand for precise, robust and quick organ measurement methods is growing rapidly, especially in longitudinal disease progression studies where total organ volume is the primary endpoint and oftentimes a major factor in inclusion criteria. Caution is warranted when different readers are measuring images from the same patient in one study, because this can have a significant effect on important statistics such as growth rate. By using standardized semi- or fully automated methods, the variance among users and throughout the duration of the study is greatly reduced and study volumes remain consistent.

FUNDING

This work was supported in part by the Mayo Clinic Robert M. and Billie Kelley Pirnie Translational PKD Center and the National Institute of Diabetes and Digestive and Kidney Diseases (grants P30DK090728 and K01DK110136), as well as the PKD Foundation (grant 206g16a).

CONFLICT OF INTEREST STATEMENT

V.E.T. is a member of the TEMPO 3:4 and 4:4 Steering Committee and has received research funding from Otsuka Pharmaceuticals. J.D.B. and F.S.C. are employees of Otsuka Pharmaceutical Development and Commercialization.