A web-based tool for personalized prediction of long-term disease course in patients with multiple sclerosis.

BACKGROUND AND
PURPOSE: The Evidence-Based Decision Support Tool in Multiple Sclerosis (EBDiMS) is the first web-based prognostic calculator in multiple sclerosis (MS) capable of delivering individualized estimates of disease progression. It has recently been extended to provide long-term predictions based on the data from a large natural history cohort.
METHODS: We compared the predictive accuracy and consistency of EBDiMS with that of 17 neurologists highly specialized in MS.
RESULTS: We show that whilst the predictive accuracy was similar, neurologists showed a significant intra-rater and inter-rater variability.
CONCLUSIONS: Because EBDiMS was consistent, it is of superior utility in a specialist setting. Further field testing of EBDiMS in non-specialist settings, and investigation of its usefulness for counselling patients in treatment decisions, is warranted.

Introduction

It is still very difficult to predict disease course in individual cases of multiple sclerosis (MS), despite recent accumulation of prognostic principles [1]. This uncertainty results in significant psychological morbidity [2]. It also compounds clinical management decisions because precise prognoses to inform risk/benefit ratios prior to starting treatment are not usually available [3].

The use of Web-based prognostic calculators is well established in cardiology (e.g. Framingham Risk Score provides 10-year cardiovascular risk), rheumatology (e.g. WHO-FRAX delivers 10-year fracture risk) and oncology (e.g. Adjuvant!, Numeracy and Predict estimate survival in cancer). They are increasingly being integrated into treatment algorithms to help clinicians and patients make shared decisions about lifestyle changes, introduce measures to reduce fracture risk and decide on adjuvant therapy in cancer.

The Sylvia Lawry Centre for MS Research has gathered data from placebo arms of randomized controlled trials, observational studies and natural history cohorts comprising >100 000 patient-years of data. This unique resource has been used to develop EBDiMS (Evidence-Based Decision Support Tool in Multiple Sclerosis) that is able to provide individual estimates of short-term [4] and now long-term prognosis. The user inputs individual patient characteristics such as disease course, age at first MS symptom, attack number in the first 2 years, first interattack interval and/or time to Expanded Disability Status Scale (EDSS) = 3. The software then searches for the best matching patients, and based on this optimized subsample that most closely matches the individual, it calculates prognoses such as time to conversion to secondary progression or time to reach EDSS = 6/8/10. An external validation of the short-term tool, using placebo data from clinical trials, confirmed usability and safety [5].

In this study, we compared the performance of EBDiMS with that of MS specialist neurologists, mostly from European centres.

Methods

A number of neurologists highly specialized in MS were offered participation in a Web-based study. Seventeen MS specialist neurologists from 10 European centres (UK, Denmark, Germany, France) and one US institution were presented with 40 real-life historical clinical vignettes online. Neurologists were asked to predict the time to conversion to secondary progressive MS or time to EDSS = 6/8/10, if at all, over the subsequent 30 years. They were also asked to estimate the accuracy/uncertainty of their prediction. Four cases were duplicated without the neurologists' knowledge in order to assess intra-rater variability.

In a typical example, the following patient characteristics were presented to the neurologists: sex, initial disease course (e.g. relapsing–remitting), age at first MS symptom, the number of attacks in the first 2 years, first interattack interval, time from first MS symptom to EDSS = 3. The neurologists were then asked (i) how many years after the first MS symptom will EDSS = 10 be reached, if at all, and (ii) the self-estimated precision of this forecast, in years.

The study was restricted to data from one single well-documented MS cohort [6]. In order to increase comparability, data were truncated after 30 years. Of 717 patients, 148 had a mean follow-up of 23 years ending with EDSS = 10, whilst 569 patients had a mean follow-up of 24 years. Table S1 shows the characteristics of the cases selected for prediction.

Predictive accuracy was measured using the integrated Brier Score [7], a well-accepted and standard method in prediction statistics (e.g. weather forecasting [8], oncology [9]). It quantifies in a single score the ability to predict a point estimate, weighted according to the self-estimated precision. A score of 0 reflects perfect accuracy; a score of 0.5 reflects chance.

Results

Whilst EBDiMS was 100% consistent, there was considerable inter-rater variability amongst the neurologists (range 0–27 years, mean standard deviation per patient 3.5 years) (Fig. 1a).

Figure 1

(a) Inter-rater variability. (b) Intra-rater variability. Two neurologists did not complete all the questions and hence had no duplicate patients.

The intra-rater variability was also considerable (Fig. 1b). In 4.7% of duplicated patients in the questionnaire, physicians switched from the statement ‘does not reach milestone’ to ‘does reach milestone’ (95.3% consistency). The mean difference between predictions of the same physician for the same patient was 3.3 years. Moreover, not included in the analysis were 13 cases (2%) in which the prognosis provided by neurologists was in logical conflict with the information provided (e.g. time to EDSS = 6 was estimated at 11 years in a patient whose time to EDSS = 3 was 13 years in the clinical vignette).

The integrated Brier Score of EBDiMS and the specialist neurologists was in the same range (0.1–0.2) (Fig. 2). Because the score was well below 0.5, both predictions were better than chance. EBDiMS did not show superiority in the prediction of particular subgroups compared with specialist neurologists (data not shown).

Figure 2

Integrated Brier Scores.

Discussion

Unlike other medical specialities, prognostic prediction algorithms are still in their infancy in neurology. EBDiMS is the first such tool in MS, delivering individual patient prognoses by drawing on a large natural history cohort of untreated MS patients. We show that EBDiMS parallels the predictive accuracy of MS specialist neurologists and has the advantage of being 100% consistent. Different neurologists offered disparate prognoses to the same patients, a situation that may result in practical difficulties and patient distress in multistaffed MS specialist clinics. This inter-rater variability may reflect uncertainty as well as different experiences between individual neurologists.

EBDiMS relies heavily on EDSS, and future tools will need to follow the establishment of better outcome measures. Because it is based on one cohort, its external validity will need to be tested in different populations. Inclusion of selected paraclinical information and treated cohorts will enhance the tool, increasing its utility in guiding clinical management.

Despite the lack of paraclinical information which may aid prognosis, the neurologists' predictive accuracy matched that of the tool; this may reflect the highly specialist knowledge of neurologists in this study. Therefore, it is possible that EBDiMS will have an even greater impact in non-specialist settings; this needs further investigation. Neurology trainees may benefit from EBDiMS as training material.

EBDiMS may also play a role in patients' psychological well-being. In cancer, there is preliminary evidence that discussion of prognosis leads to more patient satisfaction and less anxiety/depression [10]. However, statistical analysis may obscure individual patients in whom discussion of prognosis may lead to adverse psychological consequences, as highlighted by descriptive studies [11]. Therefore, the potential use of EBDiMS to discuss patient prognosis will need to be weighed in each case, using sensitivity to deliver the appropriate balance between honesty and hope.