Measuring True Change in Individual Patients: Reliable Change Indices of Cardiac Rehabilitation Outcomes, and Implications for Quality Indicators.

BACKGROUND: Mediated by outcomes such as improved exercise capacity, cardiac rehabilitation (CR) reduces morbidity and mortality. For accuracy, an individual CR patient's change must be measured reliably, an issue not typically considered in practice. Drawing from psychometric theory, we calculated reliable change indices (RCIs), to measure individual CR patients' true clinical change, apart from that from error and test practice/exposure, in exercise capacity, anxiety, and depression.
METHODS: Indirectly calculated exercise capacity (peak metabolic equivalents [METs]) and psychological symptoms were each measured twice, 1 week apart, by administering treadmill tests or the Hospital Anxiety and Depression Scale (HADS) to separate samples of 35 (mean age: 59.0 years; 6 women) and 96 (mean age: 64.4 years; 32 women) CR patients, respectively. Using test-retest reliability and mean difference scores from these samples to estimate error and practice/exposure effects, we calculated RCIs for a separate cohort (n = 2066; mean age: 62.0 years; 533 women) who completed 6-month CR, and compared change distributions (worsened/unchanged/improved) based on critical RCIs, mean and percent changes, cut-off scores, and standard deviations.
RESULTS: Practice/exposure effects were nonsignificant, except the mean HADS anxiety score decreased significantly (P ≤ 0.013; d = 0.17, small effect). Test-retest reliabilities were high (METs r = 0.934; HADS anxiety score r = 0.912; HADS depression score r = 0.90; P < 0.001). Among 2066 CR patients, RCI distributions differed (P < 0.001) from those of most other change criteria.
CONCLUSIONS: Change ascertainment depends on criterion choice. A Canadian Cardiovascular Society CR quality indicator of increase by 0.5 MET may be too small to assess individuals' functional capacity change. RCIs offer a pragmatic approach to benchmarking reliable change frequency, and pending further validation, could be used for feedback to individual patients.

Cardiac rehabilitation (CR) improves morbidity and mortality among cardiovascular patients, mediated by improved exercise capacity. Depression and anxiety are associated with cardiovascular risk, so psychological functioning is an important CR outcome, and psychosocial interventions may contribute to morbidity and mortality improvements., Comparisons of means, for example, of a CR cohort at discharge vs entry, might elucidate programmatic outcomes, while revealing little about individual participants. Clinically therefore, consistent with personalizing health interventions, it is important to determine whether individual CR patients improve meaningfully in exercise capacity and psychological symptoms.

Suppose an individual patient's exercise capacity increased from 6.5 to 7.5 metabolic equivalents (METs; 1 MET = 3.5 ml O2 /kg body-weight/min), 1.0 MET or 15%, from CR entry to discharge. This increase matches a mean change measured directly with cardiopulmonary exercise testing, or a mean between-groups difference estimated indirectly from peak speed and grade of treadmill testing, associated with fewer clinical events. This increase is double the 0.5-MET pre-post CR exercise capacity quality indicator of the Canadian Cardiovascular Society (CCS-QI), which we treat here as a minimal clinically important difference (MCID). A critical question is whether such an individual's performance reflects truly improved exercise capacity, over and above test practice and measurement error.13, 14, 15, 16 Pre-/post-intervention change scores, including differences, percentages, and standard deviations (SDs) may be inadequate to answer this question, because they may reflect measurement error or prior test exposure/practice effects, in addition to clinically relevant change, such as that from treatment, recovery, or deterioration.

We address the question of individual CR participants’ change in exercise capacity and psychological symptoms, through an innovative application of reliable change methodology from psychometric theory. Reliable change indices (RCIs) can account for error and practice in difference scores, potentially facilitating measurement of true clinical change of individual CR patients, offering a direct clinical application.

Measuring Exercise Capacity, Anxiety, and Depression

Reliability (which estimates error) of exercise capacity measured with stress-testing is moderate to high, assuming consistent test-termination criteria and standardized protocols. Direct measurement of peak oxygen uptake capacity (peak VO2) is less sensitive to practice and error than functional capacity (FC) measures dependent on treadmill time, determined from peak speed and grade. Although FC can improve due to merely test practice,,,21, 22, 23 giving nonreproducible results, CR programs have often relied upon such “indirect” exercise-capacity measurement.

The Hospital Anxiety and Depression Scale (HADS) measures self-reported depression and anxiety symptoms, with good reliability and validity.24, 25, 26 Anxiety and depression subscales range from 0 to 21 (higher = worse), with case identification > 7. HADS depression scores predicted 1-year mortality after acute coronary syndrome, with a 20% risk increment/point. In the Ontario Cardiac Rehabilitation Pilot Project (OCRPP), the mean HADS depression score and HADS anxiety score each improved significantly (n = 1554; P < 0.0001), by 1 point over 6 months (Cardiac Care Network [CCN] 2002).

In the current set of 3 studies, we calculated RCIs for CR entry-to-discharge FC changes in peak METs, and HADS anxiety and depression scores. Specific objectives were to (i) estimate stability and test practice/exposure effects for FC, and depression and anxiety measures, for RCI calculations; (ii) illustrate clinical use of RCIs for individuals; (iii) in a large clinical cohort, compare RCI distributions to those of other change criteria based upon mean and percent changes, cut-off scores, and SDs, including the 0.5-MET CCS-QI. We used 1-week retest intervals to estimate practice/exposure effects relatively free of treatment-related, maturational, or other systematic influences, hypothesizing that distributions of RCIs would differ from those of other change criteria, thereby providing unique information in determining change in individual patients.

Methods

This research was approved by the Health Sciences Research Ethics Board, Western University, London, Ontario. Separate samples for studies 1 and 2 were recruited from participants in our Cardiac Rehabilitation & Secondary Prevention Program, which accepts referrals of cardiovascular patients according to standard recommendations.

Inclusion criteria: studies 1 and 2

The inclusion criteria were as follows: (i) referral to CR within 1 year of an acute coronary syndrome event, percutaneous coronary intervention, coronary artery bypass grafting, stable angina, or heart valve repair/replacement; (ii) age ≥19 years; (iii) ability to participate in stress-testing and exercise; (iv) written, informed consent to use data in research; (v) residence location within 1 hour of our CR program; (vi) no medical contraindications to CR participation. For studies 1 and 2, the 1-week interval was chosen to minimize spontaneous change of exercise capacity or anxiety/depression, which could impact practice/exposure effects and test–retest reliability estimates.

Study 3, in which the RCI formulae were applied retrospectively, used a sample with inclusion criteria similar to those in studies 1 and 2. Additionally, subjects (i) had consecutively entered our CR program from March 24, 2003 to February 23, 2012; (ii) had both entry and discharge measurements available for stress-testing, HADS anxiety score, or HADS depression score (at least 1 of the 3); (iii) did not have recurrent CR intakes; and (iv) participated in neither study 1 nor study 2.

Study 1: exercise

For time 1, we used results from usual-care stress-testing at CR entry, patients routinely undergo a peak symptom–limited exercise test using a modified Bruce protocol. Peak METs were derived algorithmically from maximum speed and grade attained. Patients completing an entry stress test were asked to undergo an identical test 1 week later (time 2). This interval was chosen to minimize spontaneous change in exercise capacity. Testing was physician-supervised, occurred before exercise program entry to avoid treatment effects, and was scheduled at similar times of day, at least 2 hours after eating, with room environment and equipment conforming to the American Heart Association Guidelines for Exercise Testing Laboratories. We analyzed peak METs. At our centre, CR stress-testing is supervised by physician CR specialists, using usual American Heart Association/American College of Sports Medicine stopping rules. To limit observer effects, physicians were asked not to review the first test results prior to the second test. We strove to maintain consistency of supervising physicians between tests, although we did not require that for technicians. Detailed, standardized instructions for administration of a modified Bruce protocol were provided for each stress test. This protocol had 2 preliminary 3-minute stages (stage 1: 1.7 mph, 0% grade; stage 2: 1.7 mph, 5% grade) added prior to the usual stage 1 of the Bruce protocol; thus, stage 3 was the usual stage 1 of the Bruce protocol. Participants were administered the Borg Rating of Perceived Exertion (RPE) Scale (range in integers: 6 = no exertion at all, to 20 = maximal exertion). A usual-care goal was to have patients exercise to peak exertion by achieving or exceeding a Borg RPE of 17 (very hard), with light handrail holding permitted for safety. Location, use of treadmill, and medication regime were permitted to vary between tests, as a function of scheduling or clinical contingencies.

Study 2: psychometrics

All subjects were involved in CR exercise programming, conducted by CR program kinesiologists at a local YMCA. To minimize potential effects of early post-event adjustment, program habituation, and imminent discharge, we planned to test patients within the middle third of 6-month CR. Participants completed the HADS (time 1), repeated at the same time of day 1 week later (time 2), well within the minimum recommended 2-week interval. Participants had typically completed the HADS previously at CR entry.

Sample sizes

For both studies 1 and 2, we aimed to recruit at least 30 men and 30 women, assuming the confidence interval around a reliability score of 0.60 would not include 0.

Study 3: CR clinical cohort

We categorized the CR clinical cohort of 2066 patients according to whether each individual had worsened, not changed, or improved from CR entry to discharge. For FC, the change criteria in either direction were: ≥0.5 MET, the absolute value of which corresponded with the CCS-QI; 1.0 MET, associated with morbidity and mortality,; 17.5%, the mean percentage change from the OCRPP (CCN, 2002); meeting or crossing the 7-MET mark, a functional cut-off point35; distribution-based changes of 0.5 SD or 1.0 SD; and the RCI. For HADS scores, criteria were as follows: change in either direction of ≥ 1 point, corresponding with the mean improvement in the OCRPP (CCN, 2002), associated with mortality; meeting or crossing 7 points, a clinical screening criterion; distribution-based changes of 0.5 SD or 1.0 SD; and the RCI.

Statistical analyses

For studies 1 and 2, given the relatively small sample sizes, we employed robust methods (bootstrapped, 2000 samples) to obviate concerns about potential deviations from normality. In study 1, to assess consistency and termination of the modified Bruce protocol stress-test administration, we compared time 1 and time 2 with respect to mean peak heart rate and RPE using dependent-measures t-tests, and calculated Pearson test-retest reliability. For METs and HADS scores, we expressed central tendency as means (SD), and mean changes at time 2 (M2) vs time 1 (M1) as effect size, Cohen's d = (M2 – M1)/SD1; we used dependent-measures t-tests to compare means at time 2 vs time 1; and we estimated test-retest stability with Pearson product-moment correlation coefficients.

RCIs

From cardiology, based on data from 14 male patients with exercise-induced angina and ST-segment depression, who were exercise stress-tested once on each of 2 consecutive days, Sullivan et al. recommended multiplying the standard error of the difference x 1.96, to determine the minimum statistically reliable change (P < 0.05, 2-tailed) in an individual patient. Jacobson and Truax calculated a reliable change index (RCI) to determine whether psychometric difference scores of individuals undergoing psychotherapy were statistically reliable, noting that the standard error of the difference estimates the probability of a change score occurring by chance. To correct for practice over repeated neuropsychological test administrations, Chelune et al. subtracted a constant from the expression of Jacobson and Truax, derived from the mean change observed in a “control” sample unexposed to treatment.

To the extent that a “control” sample has retest intervals and characteristics matching those of a patient or cohort receiving an intervention, the approach of Chelune et al. permits estimation of treatment-specific effects, apart from error and test practice, but also distinct from spontaneous recovery or deterioration; which are sources of true change. However, a treatment-free, no-CR “control” condition of cardiovascular patients with a retest interval comparable to the duration of usual-care CR might be affected by non-CR factors, including spontaneous recovery or deterioration, or by CR-independent medication changes or exercise, which would be challenging to exclude, and potentially of interest, as they contribute to “true” clinical change. In any case, withholding CR from eligible patients to form a control condition would be unethical.

We reasoned that clinicians would be interested primarily in gauging individual CR patients’ overall true improvements in exercise capacity and psychological well-being, whether caused specifically by CR, already known to be efficacious, or by other true-change factors such as spontaneous recovery or deterioration, CR-independent exercise, or medications. Therefore, we modified the approach of Chelune et al., by using a 1-week retest interval (in contrast to intervals corresponding with actual CR programming duration) with CR-referred patients, to minimize treatment-related, maturational, or other systematic influences, while correcting for measurement error and test practice/exposure. The RCI of Chelune et al. is calculated as

As applied here, X and X represent individuals’ test scores in the large clinical cohort at CR program discharge and entry, respectively (retest intervals: approximately 7.3-9.5 months). From the study 1 (exercise) or study 2 (psychometrics) research samples, r12 is the test-retest stability estimate (to estimate error; retest interval = 1 week); M1 and M2 are the mean test and retest scores (to estimate mean test exposure/practice), respectively; SD1 is the time-1 standard deviation. The Pearson product-moment correlation coefficient (r) was considered the most appropriate correlation coefficient to use to estimate stability, for 2 reasons: (i) r was used in the development of RCIs,,; and (ii) other possibilities, such as using the intraclass correlation coefficient in the RCI of Chelune et al., would result in 2 corrections for systematic differences such as practice effects, potentially resulting in a mathematical overcorrection.

For CR clinical cohort individuals (study 3), we calculated RCIs using (X2 – X1) for (i) indirectly measured peak MET change, corrected using variables (i.e., M2, M1, SD112) from study 1; and (ii) HADS anxiety score and HADS depression score changes, corrected with study 2 variables. We calculated RCI variables based on whole samples for both study 1 and study 2; where sample size permitted (n ≥30, study 2), we developed sex-specific RCI variables. Whole-sample, male-specific, and female-specific variables were subsequently applied to the CR clinical cohort and each of its male and female subsamples, respectively, for whom entry and discharge data were available. As an RCI is a z-score, reliable change for an individual was defined as an RCI ≤–1.96 or ≥1.96 (P < 0.05; 2-tailed).

For exercise and psychometric outcomes of the CR clinical cohort, we used analysis of variance with time (program discharge vs entry) and sex as within- and between-subjects factors, respectively, and t-tests (2-tailed, bootstrapped, 2000 samples) for dependent measures to compare time points within sexes. For each HADS subscale, we set 2-tailed significance at α = 0.025 to control type 1 error. Distributions of CR cohort patients categorized as worsened, unchanged, or improved were compared with nonparametric statistics using a Bonferroni correction to the alpha level for number of comparisons. Analyses were performed using SPSS v.24 or v.26 (IBM, Armonk, NY).

Results

A total of 43 subjects consented to participate in study 1. Two stress tests were completed by each of 35 participants (mean age 59.0 years, SD = 8.3, range = 47.9-81.4 years; 6 women = 17.1%; ethnicity Caucasian, n =28 [80.0%], 5 missing; modal education level = university/college completed, n = 12 [35.3%], 1 missing; see Table 1, clinical characteristics).

Table 1

Clinical characteristics of samples at cardiac rehabilitation entry

Characteristic	Study 1: exercise stress-testing(n = 35)	Study 2: psychometrics(n = 96)	Study 3: cardiac rehabilitation clinical cohort (n = 2066)
Risk factors
Body mass index ≥ 25, kg/m2*	33/34 (97.1)	74/96 (77.1)	1593/2066 (77.1)
Sedentary*,†	24/35 (68.6)	64/96 (66.7)	1264/2058 (61.4)
Dyslipidemia	20/35 (57.1)	70/96 (72.9)	1317/2059 (64.0)
Hypertension	23/35 (65.7)	64/96 (66.7)	1153/2062 (55.9)
Personal cardiovascular history	7/21 (33.3)	Not available	424/1035(41.0)
Family cardiovascular history	18/35 (51.4)	54/96 (56.3)	1083/2028 (53.4)
Diabetes	4/35 (11.4)	19/96 (19.8)	395/2065 (19.1)
Smoking
Current	1/34 (2.9)	13/96 (13.5)	294/2062 (14.3)
Past	20/34 (58.8)	45/96 (46.9)	910/2062 (44.1)
Index/referral events‡
Stable coronary artery disease	1/35 (2.9)	8/96 (8.3)	158/2066 (7.6)
Myocardial infarction	16/35 (45.7)	40/96 (41.7)	1027/2066 (49.7)
Unstable angina	5/35 (14.3)	6/96 (6.3)	175/2066 (8.5)
Acute coronary syndrome—unclassified	3/35 (8.6)	6/96 (6.3)	50/2066 (2.4)
Percutaneous coronary intervention	22/33 (66.7)	38/96 (39.6)	867/2066 (42.0)
Coronary artery bypass grafting	6/35 (17.1)	26/96 (27.1)	616/2066 (29.8)
Aortic valve surgery	2/35 (5.7)	6/96 (6.3)	84/2066 (4.1)
Mitral valve surgery	0	3/96 (3.1)	53/2066 (2.6)
Congestive heart failure	0	5/96 (5.2)	33/2066 (1.6)
Cardiomyopathy	0	2/96 (2.1)	10/2066 (0.5)
Transient ischemic attack or mild stroke	0	0	6/2066 (0.3)
Other	1/35 (2.8)	4/96 (4.2)	41/2066 (2.0)

Values are n/total n (%; in repeated measures sample for risk factors).

Ascertained from measurement (body mass index) or enquiry at cardiac rehabilitation entry (other variables ascertained from chart/referral documentation).

Sedentary was defined as < 30 min/d moderate physical activity 3 d/wk.

Patients may have >1.

Between test 1 and test 2, there was 100% consistency with respect to supervising physicians (n = 34; 1 missing). As shown in Table 2, mean peak heart rate and RPE did not differ significantly. For each, there was a significant correlation between test 1 and test 2 (P < 0.001; for RPE, n = 28, 7 missing).

Table 2

Study 1: exercise stress-testing variables

n = 35	Time 1, mean (SD)	Time 2, mean (SD)	Mean difference	Pdifference (2-tailed)	Percent difference	r12	Pr
Days from referral	77.2 (91.2)	79.2 (91.2)	7.0	n/a	n/a	n/a	n/a
Peak HR (bpm)	131.66 (25.63)	132.43 (26.51)	0.77	0.62	0.59	0.940	< 0.001
Peak RPE (n = 28)	17.36 (1.99)	17.68 (1.76)	0.32	0.10	n/a	0.858	< 0.001
Peak METs	9.78 (3.01)	10.13 (3.15)	0.35	0.09	3.60	0.934	< 0.001

bpm, beats per minute; HR, heart rate; METs, metabolic equivalents; n/a, not applicable; RPE, rate of perceived exertion; SD, standard deviation.

Mean peak METs did not change significantly (Table 2), showed a small effect size (dMETs = 0.12), and significant test-retest reliability (P < 0.001).

A total of 114 subjects consented to study 2 participation; 100 completed 2 administrations of at least one psychometric subscale; none participated in study 1. Of these, retest intervals in days (number of subjects) were 2 (2), 7 (88), 8 (2), 9 (4), 14 (3), and 37 (1). Post hoc, we excluded the 4 subjects who met/exceeded the minimum 14-day retest interval specified by the test developer. Characteristics of the remaining n = 96 (Table 1) were as follows: mean age = 64.4 (9.7) years, range = 35.9-84.9 years; 32 women (33.3%); 93 Caucasians (97%); modal education level = university/college completed, n = 23 (24.0%). The mean (SD) inter-test interval was 7.0 (0.85) days. The mean (SD) interval from CR referral to test 1 was 25.9 (6.8) weeks, and from test 2 to CR discharge was 12.3 (4.6) weeks. This sample included one subject who had been formally discharged from CR 9 weeks before his test 1.

The mean HADS anxiety (HADS-A) score (Table 3) decreased significantly overall and within sexes (P ≤ 0.013); with a whole-sample small effect size (dHADS-A = 0.17). The mean HADS depression (HADS-D) score showed no significant changes (Table 3), with a whole-sample trivial effect size (dHADS-D = 0.02) Test-retest reliability was significant for all measures (P < 0.001).

Table 3

Study 2: psychometric variables

Variable	Time 1, mean (SD)	Time 2, mean (SD)	Mean change	Pchange (2-tailed)	r12	Pr
Whole sample
HADS anxiety score (n = 93)	5.91 (3.80)	5.26 (3.81)	–0.66	< 0.001	0.912	< 0.001
HADS depression score (n = 93)	3.59 (3.34)	3.53 (3.43)	–0.065	0.679	0.900	<0.001
Women
HADS anxiety score (n = 32)	6.28 (3.84)	5.41 (4.03)	–0.88	0.003	0.922	< 0.001
HADS depression score (n = 31)	2.90 (2.66)	3.10 (3.22)	–0.19	0.508	0.876	< 0.001
Men
HADS anxiety score (n = 61)	5.72 (3.80)	5.18 (3.73)	–0.54	0.013	0.908	< 0.001
HADS depression score (n = 62)	3.94 (3.61)	3.74 (3.54)	–0.194	0.311	0.913	< 0.001

HADS, Hospital Anxiety and Depression Scale; SD, standard deviation.

Study 3: CR cohort

The clinical cohort (Table 1) was comprised of 2066 patients (mean age = 62.0 years, SD = 10.7, range = 23.0-89.7; 533 women = 25.8%; 1941 Caucasian = 94.0%, 2 missing; modal education level = university/college completed, n = 564 or 27.5%, 18 missing). Mean (SD) intervals (weeks), based on available data, were: referral to CR entry (n = 2058), 12.8 (6.8); CR entry to discharge, (n = 1969), 32.7 (5.5); stress-testing retest (n = 2024), 31.4 (6.3); psychometric retest (n = 1956), 40.6 (7.4).

For the whole cohort and each sex (Table 4), all variables changed significantly from CR entry to discharge (P < 0.001). The sex main effect was significant for all variables (P ≤ 0.002), with men showing higher mean exercise capacity, and lower mean anxiety and depression scores. The interaction of time with sex was significant for METs (P < 0.001) but not for the HADS anxiety score (P = 0.122) or the HADS depression score (P = 0.041; alphacrit = 0.025).

Table 4

Study 3: exercise stress-testing and psychometric outcomes among cardiac rehabilitation cohort of n = 2066

Measure	n	Entry, mean (SD)	Discharge, mean (SD)
Exercise: METs
Overall	2028	7.56 (3.30)	9.29 (3.62)
Men	1508	8.15 (3.28)	9.97 (3.56)
Women	520	5.83 (2.59)	7.31 (3.01)
HADS anxiety score
Overall	1722	5.94 (3.88)	4.99 (3.48)
Men	1272	5.58 (3.76)	4.67 (3.38)
Women	450	6.98 (4.04)	5.88 (3.59)
HADS depression score
Overall	1722	3.91 (3.30)	2.82 (2.84)
Men	1272	3.75 (3.21)	2.72 (2.83)
Women	450	4.36 (3.49)	3.10 (2.86)

HADS, Hospital Anxiety and Depression Scale; METs, metabolic equivalents; SD, standard deviation.

Figures 1 (exercise) and 2 (psychometrics) display frequency distributions of clinical cohort patients who worsened, did not change, or improved from CR program entry to discharge, according to the aforementioned change criteria. Most change criteria produced distributions differing significantly (P < 0.001) from each other, and in particular, from those of the RCIs. Among all distributions, only the HADS anxiety “meet/cross 7 points” vs 1.0 SD (P = 0.18) and; vs RCI (P = 0.019) did not differ significantly. HADS anxiety score (P = 0.58) and HADS depression score (P = 0.30) RCI distributions (not shown) did not differ significantly between sexes.

Figure 1

Percentages of patients in cardiac rehabilitation cohort (n = 2066) with worsened, unchanged, or improved functional capacity, by change criterion. METs, metabolic equivalents; RCIs, reliable change indices; SD, standard deviation.

Figure 2

Percentages of patients in cardiac rehabilitation cohort (n = 1722) with worsened, unchanged, or improved Hospital Anxiety and Depression Scale (HADS) anxiety or depression scores, by change criterion. RCIs, reliable change indices; SD, standard deviation.

Discussion

We propose a method for clinical application, correcting for measurement error and mean test practice/exposure effects, to determine whether an individual patient has manifested true change from CR entry to discharge. Our exploration of different approaches demonstrates that ascertainment of change depends strongly upon the change criteria applied to individual patients. Present findings suggest the 0.5-MET CCS-QI may be too small or liberal to ascertain true change over and above that from error and test practice in individuals, when using “indirect” FC measurement.

Stress-test protocols were administered very consistently between time 1 and time 2, including consistency of supervising physicians, very similar levels of mean peak heart rates and Borg scale ratings, the latter corresponding with “very hard” RPEs, and high FC test-retest reliability. We achieved uniformity of 7-day retest intervals, and both measurements occurred before exercise program entry. Consequently, any test-retest FC differences were not due to CR exercise training, and were unlikely to have been caused by spontaneous change in fitness, but instead probably reflected test practice. The FC improvement was not statistically significant, although this was likely due to type II error, given combined small effect and sample sizes. Nonetheless, it is noteworthy that the estimated practice effect of 0.35 METs equates to 70% of the 0.5-MET CCS-QI, assuming that they were measured indirectly as FC.

Peak METs demonstrated very high stability, providing a strong basis to calculate RCIs. For practical illustration, applying study 1 variables to equation 1 and requiring RCIMETS ≤ –1.96 or ≥ 1.96 (2-tailed) or RCIMETS ≥ 1.64 (1-tailed) for significance, an individual patient would have to change by at least ±2.50 METs, or gain 2.15 METs, respectively, to show true FC change, over and above that from error and test practice. Thus, hypothetically, individual Y, who gained 3.0 METs from CR entry to discharge, improved over and above the change from error and practice. However, for individual Z, whose performance increased by 1.0 MET, twice the CCS-QI of 0.5 METs, and matching the significant overall mean increase from the OCRPP, we are unable to conclude that FC changed reliably.

The mean retest interval was well within the recommended minimum retest interval of 2 weeks; tests 1 and 2 occurred well after referral and before CR-discharge, respectively. Therefore, the significant improvements obtained in mean anxiety scores were unlikely to reflect treatment, true spontaneous change, acute adjustment, or anticipation of discharge. This finding suggests the importance of considering test exposure even in cases in which responses do not depend on psychomotor performance. Scores from the HADS demonstrated high to very high stabilities, providing a strong basis to calculate RCIs.

Applying study 1 variables to equation 2 and requiring RCIHADS ≤ –1.96 or ≥ 1.96 (2-tailed), or RCIHADS ≥ 1.64 (1-tailed) for significance, an individual patient would have to change by at least ±2.42, or 1.91 HADS anxiety points, respectively, to evidence true change in psychometrically measured anxiety symptoms, over and above that from error and test exposure. Corresponding values for the HADS depression score are ±2.80 or 2.32 points, respectively. Recently, Lemay et al. reported an MCID of 1.7 points for each HADS subscale. As 1.7 < 1.91 points, this MCID is within the potential measurement error and test exposure effects we estimated for the HADS, and it therefore may be too liberal for application with individual patients.

The samples for studies 1 and 2 were recruited from the same CR population as the large clinical cohort. Consequently, variables determined from the study samples were generalizable to the large clinical cohort. All samples were referred according to standard criteria and thus were broadly similar to CR populations elsewhere, being composed substantially of patients with coronary heart disease with or without acute coronary syndrome, large proportions of whom had undergone revascularization procedures., Mean initial FC in study 1 and the CR cohort suggested relatively high exercise capacity for CR patients. However, indirect measurement with light handrail-holding, as conducted in our centre, probably overestimates direct peak VO2 by approximately 2 METs. Subtracting 2 METs from the CR cohort FC levels at intake would give VO2 estimates comparable to those in Alter et al.’s real-world CR outcomes study, as determined by direct peak VO2 measurement. The higher baseline level in study 1 could represent a selection bias in favour of healthier volunteers. In general, we believe that the external validity of our study is reasonable.

This cohort, overall and by sex, showed robustly significant mean improvements in exercise capacity, anxiety, and depression, which we used to illustrate a clinical problem: given statistically significant group changes and no control condition, how can a clinician infer what constitutes true change for individuals? Comparisons of the change distributions show that answering this question depends crucially on the criterion applied.

With respect to exercise, the criterion least sensitive to change (highest percentage of unchanged patients) was meeting or crossing the 7-MET mark. Although this level may be functionally important, we recommend against using it as a simple benchmark of individual change.

The criterion giving the most favourable ascertainment of change in exercise capacity (highest percentage improved) was ±0.5 MET, corresponding with the CCS-QI. Notably, this QI does not consider measurement error and test practice effects. Furthermore, it (intentionally) does not specify the method of measurement, whether through gas-exchange techniques, or indirect calculation from peak treadmill speed and grade, which is more subject to practice and error effects. These issues are crucial, given that the mean (nonsignificant) practice effect in study 1 alone could account for 70% of the 0.5-MET QI in a larger sample with more statistical power. Although the CCS-QI may meet criteria for an MCID, an FC increase of 0.5 METs by an individual would fall within the range of measurement error and practice effect, as illustrated above; in other words, that change would not be reliable. Therefore, we recommend against interpreting a gain of 0.5 METs measured indirectly, as meeting the CCS-QI, for individuals. We also recommend measuring exercise capacity (peak VO2) directly, with cardiopulmonary exercise testing.

When direct cardiopulmonary measurement of peak VO2 is not feasible, we suggest application of the critical RCI scores (±2.50 METs, 2-tailed; or 2.15 METs, 1-tailed) derived from RCI analyses reported here to ascertain reliable change by individuals.

We also suggest a review to determine whether the CCS-QI suffices as a reliable criterion of change for individual patients. As direct peak VO2 uptake measures are less sensitive to practice and error effects than indirect FC measures based on peak treadmill speed and grade,,,21, 22, 23 such a review should consider the “direct” and “indirect” exercise-capacity measurement situations separately.

With respect to HADS anxiety and depression scores, the change criterion of 1 point produced the most favourable ascertainment of improvement. Comparison with RCI distributions and critical RCI values suggests that using a 1-point change criterion may overestimate the frequency of reliable improvement for individuals.

There are depression-related CCS-QIs for programmatic processes but not outcome. Challenges to standardized QI implementation include the variety of psychometric instruments used to measure depression. These factors, and scientific challenges to defining clinically meaningful change, could complicate any depression-outcome QI. RCIs, which are standard (“z”) scores, could enable meaningful comparisons with respect to depression outcomes, across psychometric instruments and CR programs. We propose that any future outcome QI should at minimum be capable of measuring individual-level reliable change.

Practical implications of our findings include benchmarking for program evaluation or to inform quality improvement, as, for example, with respect to frequencies of participants who either show reliable decrements, do not change, or reliably improve over CR, with respect to key outcomes including exercise capacity and emotional functioning. In light of the potential impact on individual CR participants, direct feedback to them might be considered, after further validation of RCIs against clinical criteria. These RCIs can be used clinically, similar to how they are applied in other disciplines (for example, psychology) to help determine if a patient seen for at least 2 testing sessions has improved, declined, or stayed the same on a standardized measure. This determination is impossible to make by eyeballing raw score changes. In contrast, RCIs allow a clinician to determine if any changes in raw scores are beyond expected fluctuations in test scores based on measurement error and practice effects.

Strengths and limitations

Strengths of our research include the following: short retest intervals, minimization of potential contributions of “true” treatment or maturational change to our practice/exposure estimates; high stability estimates; and use of RCIs, which permits correction of individual change scores for measurement error and test practice/exposure.

The principal limitation is that we have not validated RCIs against clinical criteria such as future cardiovascular event risk. Using 1-week retest intervals may not accurately estimate error or practice/exposure over CR. If treadmill practice effects decayed, 1-week retest intervals might overcorrect for practice over longer intervals. Conversely, treadmill use in CR might preserve or enhance practice, independently of true exercise-capacity change. Short retest intervals might then underestimate practice in CR. Further studies should test the validity of generalizing practice estimates derived from short retest intervals to 6-month CR.

Further challenges to our estimation of error and practice effects are that we did not control for medication changes between time 1 and time 2, which could have occurred for clinical reasons after the first stress test; or for retest changes in technician, location, or treadmill. However, changes to medications would have been difficult to control, as they reflect normal clinical practice. Also, all testing was done at London Health Sciences Centre locations, reflecting commonality of equipment and maintenance. The high levels of inter-test consistency with respect to peak heart rate and RPE, and test-retest reliability of peak METs, suggest that these issues were not influential.

This method of assessing practice effects assumes uniform benefit from practice/exposure, but baseline fitness could interact with test repetition. This possibility could be explored in future work, by comparison of RCIs as calculated here to regression-based approaches for measuring individuals’ change., Our work was also limited by small sample sizes, which preclude sex-specific exercise estimates, and by low ethnic diversity, which limits its generalizability.

Conclusions

Although requiring further validation, reliable change methodology offers a promising and pragmatic approach to measuring true change in individuals in CR. RCIs can provide complementary information to external criteria of clinically important change. Ascertaining change from CR entry to discharge strongly depends upon the specific criteria applied, as RCI distributions differed significantly from those of other change criteria we tested. Our data suggest that improvement by 0.5 MET in exercise capacity, a CCS-QI, is too liberal a criterion, when indirect measurement is used, to ascertain true change in individual patients over and above error and practice effects. We recommend use of direct, cardiopulmonary measurement of peak VO2 to ascertain exercise capacity, when feasible. Although the 0.5 CCS-QI may meet criteria for an MCID, we recommend a review of its reliability and validity when it is applied to individuals, particularly in situations of indirect measurement. In any such review, we recommend separate consideration of direct VO2 measurement vs indirect algorithmic estimation of exercise capacity. We propose that RCI methodology may have utility in development of future outcome-related psychometric QIs. The RCI critical values reported here may be applied to frequencies of individual CR participants for program evaluation or quality assurance purposes.