The use of Systemic Lupus Erythematosus Disease Activity Index-2000 to define active disease and minimal clinically meaningful change based on data from a large cohort of systemic lupus erythematosus patients.

OBJECTIVES: To examine SLEDAI-2000 cut-off scores for definition of active SLE and to determine the sensitivity to change of SLEDAI-2000 for the assessment of SLE disease activity and minimal clinically meaningful changes in score.
METHODS: Data from two multi-centre studies were used in the analysis: in a cross-sectional and a longitudinal fashion. At every assessment, data were collected on SLEDAI-2000 and treatment. The cross-sectional analysis with receiver operating characteristic (ROC) curves was used to examine the appropriate SLEDAI-2000 score to define active disease and increase in therapy was the reference standard. In the longitudinal analysis, sensitivity to change of SLEDAI-2000 was assessed with multinomial logistic regression. ROC curves analysis was used to examine possible cut-points in score changes associated with change in therapy, and mean changes were estimated.
RESULTS: In the cross-sectional analysis, the most appropriate cut-off scores for active disease were 3 or 4. In the longitudinal analysis, the best model for predicting treatment increase was with the change in SLEDAI-2000 score and the score from the previous visit as continuous variables. The use of cut-points was less predictive of treatment change than the use of continuous score. The mean difference in the change in SLEDAI-2000 scores, adjusted for prior score, between patients with treatment increase and those without was 2.64 (95% CI 2.16, 3.14).
CONCLUSIONS: An appropriate SLEDAI-2000 score to define active disease is 3 or 4. SLEDAI-2000 index is sensitive to change. The use of SLEDAI-2000 as a continuous outcome is recommended for comparative purposes.

Introduction

The SLEDAI index is a global score index developed for the assessment of SLE disease activity [1]. The index has been shown to be reliable, has construct validity and is sensitive to change [2-11]. However, the index focused on new or recurrent manifestations and failed to capture on-going activity, which led to a revision (SLEDAI-2000 index) [12].

SLEDAI-2000 has not been formally validated for assessment of SLE disease activity. It has been shown to correlate well with the original SLEDAI, which is to be expected as SLEDAI-2000 is derived from SLEDAI and the majority of the items are identical. Appropriate definitions of active disease based on the score have not been clearly established. This is an important issue as cut-off scores are used in studies to stratify patients and to determine eligibility. Three studies have suggested a cut-off score of 4 or 6 for the original SLEDAI [13-15]. However, these studies were hampered by small sample size, use of abstracted case histories or the use of physician’s global assessment as gold standard for disease activity, which was not ideal as it had unsatisfactory performance and poor agreement between expert physicians, particularly in patients with manifestations in multiple systems [5, 8, 11, 16, 17]. Furthermore, minimal clinically meaningful changes in the score for SLEDAI-2000 have not been established. These may be relevant in studies either to classify whether the patient’s disease activity over time has improved, worsened or remained unchanged, or on a mean basis to be used for sample size calculations. A few studies have tried to address the former issue but again were limited by small numbers [10], use of simple descriptive statistics for analysis [14], use of abstracted case histories [16] or employing physician’s global assessment as the gold standard [10, 14, 16, 18]. As a result, the minimal increase in SLEDAI score associated with worsening of disease activity from these studies ranges from 3 to 8. This could lead to inconsistencies with classification of response and difficulties in the interpretation/analysis of results.

We have used data from two large multi-centre studies to address the following:

to examine SLEDAI-2000 cut-off scores that could define active SLE;

to determine if SLEDAI-2000 is sensitive to change; and

to consider the minimal clinically meaningful changes in SLEDAI-2000 score at individual level and population level.

Patients and methods

Data were available from two multi-centre studies in the UK. Both studies were designed primarily for the validation of the BILAG-2004 index and have been described in detail previously [19, 20]. All patients met the revised ACR criteria for classification of SLE [21, 22]. Patients were excluded if they were pregnant, under the age of 18 years or unable to give valid consent. Both studies received ethical approval from Hull and East Riding Research Ethics Committee, and were carried out in accordance with the Helsinki Declaration. Written consent was obtained from all patients. The majority of patients were involved in both studies that ran concurrently. It should be noted that treatment of the patient was based on physician’s clinical judgement, and not on the BILAG-2004 index or SLEDAI-2000 scores (which were not available to the physician when the treatment decision was made).

Cross-sectional analysis

This analysis was used to examine SLEDAI-2000 cut-off scores to define active disease. The data came from a study that commenced in March 2005 and was completed in August 2006. At every assessment, data (SLEDAI-2000 and treatment) were collected. This study is longitudinal in design as the majority of patients had repeated assessments. However, the analysis was cross-sectional in nature at the time of the assessment and statistical methods are used to allow for multiple assessments from the same patient (see below).

Change in therapy was used as the reference standard for disease activity. This was the change in therapy following the assessment and a robust definition was used as described previously [19] (see supplementary data available at Rheumatology Online). For this analysis, change in therapy was categorized into increase in therapy and no increase in therapy.

Longitudinal analysis

A longitudinal study was used to examine the sensitivity to change for SLEDAI-2000. This study commenced in March 2005 and was completed in April 2007. Patients were followed up prospectively and data (SLEDAI-2000 index and treatment) were collected for all consecutive visits/encounters the patients had with their physicians. This is conceptually different from the cross-sectional analysis above as the changes in disease activity and treatment between two consecutive visits are analysed in a longitudinal fashion. Therefore, each observation for the analysis was derived from two consecutive visits.

Change in therapy between consecutive visits was used as the external reference for change in disease activity. Change in therapy was the change in treatment between two consecutive visits. The definition for change in therapy (supplementary data are available at Rheumatology Online) was similar to the one used in the cross-sectional study and had been described [20]. Three categories of changes in therapy were defined: no change, increase in therapy and decrease in therapy. All statistical analyses were performed using Stata for Windows version 8 (Stata Corporation, College Station, TX, USA).

Cross-sectional statistical analysis

A receiver operating characteristic (ROC) curve was used to derive information on appropriate cut-off scores for active disease associated with increase in therapy [23]. Logistic regression was used to estimate the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) associated with various SLEDAI-2000 cut-off scores. Robust variance estimation was used in the analysis as this is a commonly used statistical method that accounts for multiple assessments from the same patient [24]. Increase in therapy was the outcome variable, and the classification of active disease according to SLEDAI-2000 score using various cut-off scores were the explanatory variables for the PPV and NPV estimates, and vice-versa for the sensitivity and specificity calculations. The Youden index (sensitivity + specificity − 1) was used to compare alternative cut-off scores [25]. This index is a measure of overall diagnostic effectiveness. It ranges between 0 and 1, with values close to 1 indicating very good diagnostic effectiveness and values near 0 indicating poor effectiveness.

Longitudinal statistical analysis

The sensitivity to change of the index was assessed using the external responsiveness method [26]. The extent to which changes in SLEDAI-2000 score between two consecutive visits relate to the corresponding changes in therapy (external reference) was studied. This analysis was performed using multinomial logistic regression (with robust variance estimation) with change in therapy as the three-level outcome variable and change in SLEDAI-2000 score and SLEDAI-2000 score of the previous visit as potential explanatory variables. Where appropriate, fractional polynomials were used to examine the best fitting function (powers) of the continuous variables (such as change in SLEDAI-2000 score and the SLEDAI-2000 score of the previous visit), which predict the outcome variables [27].

In addition, analyses were done using increase in therapy vs no increase in therapy as a binary outcome variable to generate ROC curves related to possible cut-off points, at individual level, for increases and decreases in SLEDAI-2000 score associated with increase in therapy and no increase in therapy, respectively. Estimated population-averaged changes were derived from linear regression analyses with change in SLEDAI-2000 score as the outcome variable.

In a model for change in treatment with two binary explanatory variables defining specified increase and decrease in SLEDAI-2000 score, the baseline comparator for change in score was minimal change in score (defined as neither specified increase nor decrease in score had been observed), while the baseline comparator for change in treatment was no change in therapy. The results were reported in odds ratio (OR) or coefficient with 95% CI.

Results

There were 369 patients who contributed 1510 assessments and the demographics are summarized in Table 1. Increase in therapy occurred in 22.6% of assessments, while in 21.2% there was reduction in therapy and no change in treatment occurred in 56.2%. The mean (s.d.) SLEDAI-2000 score was 2.9 (3.4) with a range from 0 to 26.

T

Demographics of recruited patients

Patient characteristics	Cross sectional analysis (n = 369)	Longitudinal analysis (n = 347)
Female sex, %	92.7	92.9
Age, mean (s.d.), years	41.6 (13.2)	40.9 (12.9)
Race, %
Caucasian	59.9	57.9
Afro-Caribbean	18.4	20.5
South Asian	18.4	19.0
Oriental	1.4	1.2
Others	1.9	1.4
Disease duration, mean (s.d.), years	8.8 (7.7)	8.2 (7.8)
Number of assessments, median (range)	4 (1–11)	4 (2–18)

ROC curves analysis for SLEDAI-2000 score as a predictor of increase in therapy is summarized in Table 2. The most appropriate cut-off scores for active disease appears to be 3 or 4, as both have similar performance characteristics and the best Youden index values. The performance of using the cut-off scores of 3 or 4 (ROC area under the curve 0.71) is comparable with that of using the total score as a continuous variable (ROC area under the curve 0.76) in predicting treatment increase.

T

ROC curves analysis of SLEDAI-2000 index as a predictor of increase in therapy

Cut-off score	Sensitivity % (95% CI)	Specificity % (95% CI)	PPV % (95% CI)	NPV % (95% CI)	Youden index
2	87.7 (83.2, 91.1)	43.0 (38.5, 47.6)	31.1 (27.9, 34.4)	92.3 (89.5, 94.3)	0.30
3	71.9 (65.8, 77.3)	70.7 (66.1, 75.0)	41.8 (37.5, 46.3)	89.6 (87.2, 91.6)	0.42
4	70.8 (64.6, 76.3)	72.2 (67.5, 76.4)	42.7 (38.2, 47.2)	89.3 (87.0, 91.4)	0.42
5	44.4 (38.0, 51.1)	87.5 (84.3, 90.1)	51.0 (45.0, 57.0)	84.3 (81.9, 86.5)	0.32
6	42.1 (35.9, 48.6)	88.1 (84.9, 90.7)	50.9 (44.7, 57.0)	83.9 (81.4, 86.0)	0.30
7	28.4 (22.4, 35.1)	93.3 (90.7, 95.2)	55.4 (48.0, 62.6)	81.6 (79.0, 84.0)	0.22
8	26.3 (20.6, 32.9)	93.8 (91.2, 95.7)	55.6 (47.6, 63.2)	81.3 (78.7, 83.6)	0.22
9	17.3 (13.1, 22.4)	96.7 (95.1, 97.9)	60.8 (52.0, 69.0)	80.0 (77.4, 82.3)	0.14
10	17.3 (13.1, 22.4)	97.0 (95.3, 98.1)	62.8 (53.5, 71.1)	80.0 (77.5, 82.3)	0.14

There were 1761 assessments from 347 SLE patients that contributed 1414 observations for analysis (demographics are summarized in Table 1). There was an increase in treatment in 22.7% of observations, whereas 37.3% had therapy decreased, and, in 40.0%, there was no change in treatment. An increase in score occurred in 344 observations (mean increase 3.5, range 1–22), whereas there was a decrease in score in 409 observations (mean decrease 3.8, range 1–14).

The increase in SLEDAI-2000 score was significantly associated with treatment increase (OR 1.24, 95% CI 1.18, 1.32) and inversely associated with treatment reduction (OR 0.93, 95% CI 0.90, 0.97). When the SLEDAI-2000 score of the previous visit was also included in the regression model, both the change in score and the previous visit score were significantly associated with increase in therapy (Table 3). Hence, the model with just change in score was insufficient to explain change in therapy (particularly increase in therapy).

T

Association between change in therapy with change in SLEDAI-2000 score and SLEDAI-2000 score of the previous visit

SLEDAI-2000 Variables	Increase in therapy ORa (95% CI)	Decrease in therapy ORa (95% CI)
Change in SLEDAI-2000 score	1.37 (1.28, 1.46)	0.93 (0.87, 0.99)
Previous visit SLEDAI-2000 score	1.24 (1.17, 1.31)	1.0 (0.9, 1.1)

aPer unit change in SLEDAI-2000 score.

ROC curves analysis of cut-off points for change in SLEDAI-2000 score as predictors of increase in therapy is summarized in Table 4. High Youden index value was not achieved. The highest values were associated with minimal cut-points for increase in scores of 1 and 2. However, for the best performance in predicting increase in therapy, an increase in score of at least 3 or 4 would be preferable as both had substantially better PPVs than the cut-points of 1 or 2. There was, however, a compromise in the sensitivity.

T

ROC curves analysis of SLEDAI-2000 index to determine the most appropriate minimal increase in score associated with increase in therapy

Increase in SLEDAI-2000 score (magnitude)	Sensitivity % (95% CI)	Specificity % (95% CI)	PPV % (95% CI)	NPV % (95% CI)	Youden index, %
≥1	47.4 (41.6, 53.2)	82.4 (80.0, 84.6)	44.2 (39.0, 49.5)	84.2 (81.6, 86.5)	0.30
≥2	44.9 (39.3, 50.5)	84.4 (82.2, 86.3)	45.7 (40.4, 51.2)	83.9 (81.4, 86.2)	0.29
≥3	28.7 (23.7, 34.2)	95.4 (94.0, 96.5)	64.8 (56.8, 72.0)	82.0 (79.5, 84.3)	0.24
≥4	26.2 (21.4, 31.6)	95.9 (94.5, 96.9)	65.1 (56.5, 72.8)	81.6 (79.1, 83.8)	0.22
≥5	14.6 (10.6, 19.9)	98.4 (97.4, 99.1)	73.4 (60.8, 83.1)	79.7 (77.1, 82.0)	0.13
≥6	13.7 (9.7, 19.0)	98.8 (97.9, 99.3)	77.2 (64.5, 86.3)	79.6 (77.0, 81.9)	0.13
≥7	9.0 (6.0, 13.3)	99.5 (98.7, 99.8)	85.3 (67.7, 94.1)	78.8 (76.3, 81.2)	0.09

ROC curves analysis was also performed to examine the minimal decrease in score that was associated with no increase in therapy (combination of decrease in therapy and no change in therapy) as summarized in Table 5. Very low Youden index values were achieved with the best values corresponding to the use of minimal decrease in score of 1 or 2. The results were similar when the analysis with decrease in therapy was used instead of no increase in treatment (data not shown). Interestingly, high PPVs could be achieved for the prediction of no increase in therapy, whereas it was high NPVs that were achieved for the prediction of an increase in therapy.

T

ROC curves analysis of SLEDAI-2000 index to determine the most appropriate minimal decrease in score associated with no increase in therapy

Decrease in SLEDAI-2000 score (magnitude)	Sensitivity % (95% CI)	Specificity % (95% CI)	PPV % (95% CI)	NPV % (95% CI)	Youden index
≥1	31.7 (28.9, 34.6)	80.7 (76.3, 84.4)	84.8 (81.1, 87.9)	25.7 (22.8, 29.0)	0.12
≥2	30.2 (27.5, 33.2)	81.9 (77.6, 85.6)	85.1 (81.2, 88.3)	25.7 (22.7, 28.8)	0.12
≥3	16.0 (13.8, 18.5)	91.6 (88.1, 94.1)	86.6 (81.4, 90.6)	24.3 (21.6, 27.1)	0.08
≥4	14.9 (12.7, 17.4)	93.1 (90.0, 95.4)	88.1 (82.8, 91.9)	24.3 (21.7, 27.2)	0.08
≥5	7.2 (5.7, 9.2)	96.6 (94.1, 98.0)	87.8 (80.1, 92.8)	23.4 (20.9, 26.1)	0.04
≥6	7.0 (5.5, 9.0)	97.2 (94.8, 98.5)	89.5 (81.8, 94.2)	23.5 (21.0, 26.2)	0.04
≥7	4.7 (3.4, 6.4)	98.1 (96.0, 99.1)	89.5 (79.7, 94.8)	23.2 (20.8, 25.8)	0.03

Combining the findings of the above analyses on cut-points for change in SLEDAI-2000 score in a multinomial logistic regression analysis (with minimal clinically meaningful change in score defined as an increase of ≥3 for worsening and decrease of ≥1 for improvement), worsening of score was significantly associated with increase in therapy, whereas improvement in score was associated with treatment reduction (Table 6). However, there was a significant association between worsening of the score and decrease in treatment as well. The results were similar with minimum increase in score of 4 or minimum decrease in score of 2 (data not shown).

T

Sensitivity to change analysis of SLEDAI-2000 index with minimal increase in score of 3 to indicate worsening and minimal decrease in score of 1 to indicate improvement (n = 1414)

Change in SLEDAI-2000 score	Number of observations	Increase in therapy OR (95% CI)	Decrease in therapy OR (95% CI)
Minimal change	863	1.00	1.00
Increase ≥3	142	10.57 (6.20, 18.01)	1.89 (1.03, 3.47)
Decrease ≥1	409	1.05 (0.74, 1.49)	1.97 (1.51, 2.56)

Several models of change in SLEDAI-2000 score were examined and compared (data not shown). The model with change in score as a continuous variable and the score of the previous visit included, had the best performance in explaining increase in therapy as compared with any model based on cut-off points. This relationship between increase in therapy with change in score and the score of the previous visit (both as continuous variables) was further examined using multivariable fractional polynomial regression. This analysis confirmed that the best fitting model for increase in therapy is the one with a linear function (power of 1) of change in SLEDAI-2000 score and the SLEDAI-2000 score of the previous visit.

The estimated mean change in score associated with treatment increase was 1.49 (95% CI 1.06, 1.92), whereas the mean decrease associated with no increase in therapy was 0.76 (95% CI 0.92, 0.60). The resultant difference, 2.25 (95% CI 1.70, 2.80), might be taken as the minimal clinically important difference for clinical trial design. However, a strong dependence of change in SLEDAI-2000 score on the score at the previous visit was found (P < 0.001). With adjustment for prior score, the estimated mean difference in the change in scores for patients with a treatment increase vs those without was 2.64 (95% CI 2.16, 3.14). This represents the effect associated with treatment increase compared with patients with no treatment increase and comparable prior scores. Moreover, the estimated residual variance associated with change in SLEDAI-2000 score after the adjustment for prior score is 6.64, whereas it is 9.94 without adjustment. Thus, a trial with the adjusted difference in change in SLEDAI-2000 score as the primary outcome would require a sample size approximately one-half that of a trial that aimed to demonstrate an unadjusted effect [(6.64/(2.64)2)/(9.94/(2.25)2) = 0.49].

Discussion

We have undertaken a comprehensive assessment of SLEDAI-2000 index with regards to definition of active disease, minimal clinically meaningful change in score and sensitivity to change of the index. These have involved large sample sizes in routine practice and the analyses were performed from both the cross-sectional and longitudinal perspectives.

This is the first study to assess the sensitivity to change of SLEDAI-2000, since it was revised from the SLEDAI index. Using the robust external responsiveness method, the results confirmed that SLEDAI-2000 index was sensitive to change as the changes in the score correlated well with the corresponding change in therapy. This result is consistent with previous studies on sensitivity to change of SLEDAI [7-11].

Our analysis demonstrated that the most appropriate SLEDAI-2000 cut-off score for definition of active disease linked to the need to increase therapy was 3 or 4. This was consistent with the results from our reliability study [15] and another study by Gladman et al. [14]. However, this is different from the cut-off score of 6 suggested by Abrahamowicz et al. [13]. This is most likely due to the difference in the study design. Our study was prospective and derived from a large number of patients within clinical practice, whereas the Abrahamowicz study was based on hypothetical situations derived from 30 abstracted case histories. As such, the result of our study is more applicable to clinical practice.

The longitudinal analysis to determine the minimal clinically meaningful changes in SLEDAI-2000 score (based on the need for treatment) showed that, in general, the performance of cut-points at individual level was not attractive. Although a minimal clinically meaningful increase in score of 1 or 2 provided the best results for sensitivity and specificity, a higher threshold for increase in score of 3 or 4 was more appropriate (and recommended) for prediction of increase in therapy (due to its superior PPV), but with a resultant decrease in the sensitivity. This is similar to the results of two previous studies with SLEDAI index [14, 18]. However, Liang et al. [16] suggested a higher cut-off (increase in score of ≥8 for flare and decrease in score of ≥6 for improvement). This discrepancy could be due to the fact that Liang et al. involved lupus experts reviewing abstracted case histories, whereas this study is based on clinical practice.

Our data suggest a minimal clinically meaningful decrease in score of 1 or 2, which is much smaller in magnitude than that used in the SLE Responder Index (SRI) (decrease in score of ≥4) [28]. Although SRI uses Safety of Estrogens in Lupus Erythematosus National Assessment (SELENA)-SLEDAI [29] instead of SLEDAI-2000, both the indices are almost identical and would be expected to have similar performance. The SELENA-SLEDAI cut-off point for the SRI was based on previous work by Gladman et al. [14] using the original SLEDAI, not through the analysis of the belimumab trial data. In the analysis of the belimumab trial data, there were significantly more patients with improvement in score of 1 or 2 in the belimumab group than in the placebo group, but there was no significant difference between the groups when the improvement in score of ≥4 was used. This is consistent with our findings and suggests that a lower threshold for improvement in SELENA-SLEDAI score might be considered for the SRI. However, it is acceptable to select a higher threshold that would require the drug to have bigger treatment effect as compared with placebo. For a higher threshold to be used, it would be desirable to have a higher SELENA-SLEDAI score of >4 (instead of ≥4) at study entry to allow the drug to achieve the required treatment effect (decrease in score of ≥4).

More importantly, our results indicate that the use of a single cut-off for change in SLEDAI-2000 score will sacrifice information and performance. In addition, change in SLEDAI-2000 score on its own is inadequate in explaining change in therapy. The score from which it has changed from is equally important. This is not surprising as the score from the previous assessment puts the change in score into context; for example, a change from a score of 18 to 15 may not constitute significant change as the patient continues to have very active disease, whereas a change of similar magnitude from 6 to 3 would indicate improvement in disease activity.

We did assess several alternative models of change in score in the analysis (data not shown), but, nevertheless, the best model to explain change in therapy is with the score of the previous visit and the change in score (as a continuous variable) included. Although the minimal clinically meaningful change at individual level is desirable for definition of response in clinical studies, our analysis highlights the drawback of using such cut-off points with change in score. Therefore, it has to be emphasized that SLEDAI-2000 score is designed as a continuous variable and performs best as such. The use of cut-off points will lead to loss of information and compromise its performance.

Further analysis revealed that the estimated population average difference in change in SLEDAI-2000 score between patients requiring an increase in therapy and those without treatment increase is ∼2.6, after adjustment for prior SLEDAI-2000 score. This difference could be recommended as a basis for defining minimal clinically important treatment effects for clinical studies.

We have not performed specific analysis with regards to the effect of differential system involvement on the cut-off values. This is because SLEDAI-2000 index was designed and intended to be used as a global score index. Furthermore, it is not uncommon for SLE disease activity to affect a few systems concomitantly (such as pleurisy with rash and arthritis). It is less common for disease activity to affect only one single system and the numbers would be too small to make meaningful interpretation. We would not recommend using global score index (such as SLEDAI-2000) as the primary outcome end-point for clinical trials assessing the differential effects of therapy on different systems: a system-based index would be more appropriate.