Predicting patient-reported stroke outcomes: a validation of the six simple variable prognostic model.

BACKGROUND: Case-mix represents the range of disease severity and baseline characteristics that may be the cause of variation in outcomes between individuals and populations. Adjustment for case-mix is therefore important to allow meaningful comparison of healthcare outcomes. The best available case-mix adjustment model for stroke (the Six Simple Variable [SSV] model) was developed to adjust the hard endpoints of independent survival, survival and alive and living at home. There is increasing interest in the measurement of patient-reported outcomes through self-completed questionnaires, though there are currently no robust adjustment models for any such outcome. We aimed to determine whether the SSV prognostic model derived to predict 6-month post-stroke independent survival has wider utility in case-mix adjustment of a patient-reported functional outcome measure, the Subjective Index of Physical and Social Outcome (SIPSO), collected by post 6 months after stroke onset.
METHODS: We examined data from 176 patients admitted following an acute stroke and recruited into a prospective cohort study in three participating acute hospitals in Yorkshire, UK. Patients in receipt of palliative care or with transient ischaemic attack were excluded. Using the beta coefficients from the published SSV model to predict independent survival, individual probabilities of 'good' outcome as measured with the dichotomised SIPSO collected by post 6 months after stroke onset were calculated. The ability of the SSV case-mix adjustment model to discriminate patients with 'good' over 'poor' outcome was assessed through calculation of C statistics. Correct predictions were visualised with calibration plots.
RESULTS: The C statistics for the SSV model to predict the physical and social subscales of the SIPSO outcome measure were 0.73 (95% CI 0.65-0.79) and 0.66 (0.58-0.82), respectively. Inclusion of patients who died prior to follow-up and ascribing them a score of 0 improved the discrimination (0.76 [0.70-0.82] and 0.70 [0.64-0.76], respectively). Calibration plots demonstrated a tendency to over-optimistic predictions, although confidence limits were wide.
CONCLUSIONS: The SSV model predicts adequately the physical component of the SIPSO patient-reported outcome measure and may be useful to adjust this outcome for case-mix following stroke in survivors to follow-up. This could be of benefit in observational studies, stratified randomisation for trials, and in comparison of between-institution clinical trials. Further exploration of the generalizability of the model to adjust other patient-reported stroke outcomes may be warranted.

Introduction

Post-stroke outcomes require adjustment for baseline severity and patient characteristics (case-mix) to allow comparisons and to assess the quality of delivered care. Differences in observed outcomes between groups have been shown to be wholly or partly attributable to case-mix in a number of experimental stroke studies [1,2,3,4]. The Six Simple Variable (SSV) model is the best available case-mix adjustment model following stroke [5] and was developed to predict 6-month independent survival (measured with the Oxford Handicap Score [OHS] dichotomised at ≤2), 30-day survival or alive and living at home at 6 months [6]. The SSV model comprises the variable age, the pre-stroke variables of living alone and independence, and the post-stroke variables of able to lift both arms off bed, able to walk and normal verbal component of the Glasgow Coma Score [6].

Measurement and prediction of patient-reported post-stroke outcomes may be of greater relevance to patients, carers, clinicians and service developers than these hard endpoints of death and dependency. However, there are currently no reliable models to predict patient-reported functional stroke outcomes [5].

If the SSV case-mix adjustment model can be shown to adjust more complex patient-reported functional outcomes for case-mix, the model may have wider utility in adjusting these outcomes for the purposes of observational studies, stratified randomisation for trials, or in routine care. We used data from a prospective cohort study to ascertain whether the SSV model developed to predict 6-month independent survival could also discriminate patients with ‘good’ over those with ‘poor’ outcome, measured with a patient-completed functional outcome questionnaire.

Methods

We used data from a prospective observational cohort study conducted at three sites in Yorkshire, UK as part of the Leeds, York, Bradford Collaboration for Leadership in Applied Health Research and Care (CLAHRC). This study was designed to explore the relationships between routinely collected measures of stroke care process and patient-reported functional outcomes collected 6 months after recruitment, adjusted for case-mix with the SSV model. The sample size for the cohort study was based on a pragmatic consideration of the average number of patients admitted to each study site with stroke over the 8-month recruitment period. Using the case-mix and outcomes data from this study, we tested the ability of the SSV case-mix adjustment model to predict the primary endpoint of the patient-reported postal Subjective Index of Physical and Social Outcome (SIPSO) [7].

Approval for the cohort study was obtained from Bradford Regional Ethics Committee. Consent (or a consultee declaration) was sought from all patients admitted to the participating sites with a primary diagnosis of stroke within the 8-month recruitment period. Patients with transient ischaemic attack, subarachnoid haemorrhage or in receipt of palliative care were excluded. Case-mix variables were collected within 10 days of the stroke event.

The physical and social integration SIPSO subscales have been shown to fit the Rasch model, and were converted to continuous variables using the published coefficients [8].

Probabilities of good or poor outcome predicted with the SSV model (defined as OHS ≤2) were calculated for individual participants using the beta coefficients derived by the authors of the SSV instrument [6].

Discrimination, the ability of the SSV model to correctly predict and discriminate participants with ‘good’ over those with ‘poor’ observed SIPSO outcomes, was measured with the C statistic. A clinically appropriate cut point of 3 out of 4 for each SIPSO question was used to represent a mild residual deficit that does not interfere appreciably with daily living, comparable to an OHS ≤2. Each SIPSO subscore was therefore dichotomised at 15 to represent ‘good’ over ‘poor’ outcome. As a measure of sensitivity, post hoc analyses were performed to calculate C statistics for the SIPSO subscores at cut points of 14, 16, 17 and 18. Calculations of C statistics were performed first excluding, and then including and ascribing a score of 0 to patients who had died prior to 6-month follow-up.

Calibration, the proportion of occasions where predicted outcome matches the observed outcome, is presented graphically plotted per decile of predicted probability of poor outcome.

Results

312 patients were recruited to participate in the study. At 6-month follow-up, 44 had died and 13 had withdrawn. Complete SIPSO physical and social subscore questionnaires were returned 6 months post recruitment by 176 and 174 survivors, respectively. The characteristics of responders and non-responders to the SIPSO questionnaires are given in table 1. Responders were significantly older than non-responders and less disabled. Participants who died or withdrew from the study were older and more disabled than both responders and non-responders.

Table 1

Characteristics of responders and non-responders to the 6-month SIPSO questionnaires and of participants who died or withdrew from thestudy

	Responders (n = 176)	Non-responders (n = 79)	Died or withdrawn (n = 57)
Mean age, years (95% CI]	72.8 (71.2–74.5)	67.0 (63.7–70.3)	79.8 (77.5–82.0)
Females, n (%)	82 (47)	38 (48)	38 (67)
Median baseline Barthel Index [IQR]	17 [11–20]	13 [6–20]	4 [0–14]
Median probability of good outcome (SSV) [IQR]	0.57 [0.01–0.79]	0.35 [0.08–0.85]	0.018 [0.01–0.34]

Median 6-month physical and social SIPSO subscores were 14 (IQR 8.3-17.8) and 12 (IQR 8.1-15.8), respectively. Using a SIPSO cut point of 15, 77/176 (44%) participants had observed ‘good’ physical outcome, and 50/174 (29%) ‘good’ social outcome.

In the discrimination of good over poor outcome determined with the dichotomised SIPSO, the SSV model had a C statistic of 0.73 (95% CI 0.66-0.79) if patients who did not survive to follow-up were excluded, and 0.77 (0.71-0.82) if these patients were included but ascribed a score of 0. The equivalent C statistics for the discrimination of good over poor social subscore were 0.66 (0.58-0.82) and 0.70 (0.63-0.76). The C statistic for prediction of 6-month independent survival (OHS ≤2) was 0.77 (95% CI 0.71-0.83).

Post hoc examination did not reveal any significant differences in C statistics for cut-off values of 14, 15, 16, 17 and 18 for either SIPSO subscore.

Calibration plots for the individual predictions of physical and social subscores of the SIPSO with the SSV model are given in figure 1.

Fig. 1

Calibration plots for SSV prediction of ‘good’ outcome measured with physical (a) and social (b) SIPSO subscores (cut-off at 15 points; a score of 0 was ascribed to those who died).

Discussion

This study is the first to assess the prediction of a patient-reported functional outcome measure (the SIPSO) with the extensively externally validated SSV case-mix adjustment model (developed to predict 6-month independent survival). The SIPSO has been identified as valid and reliable for self- or proxy-completed postal survey following stroke [9,10]. The SIPSO comprises two subscales measuring physical and social reintegration. Both subscores consist of five questions each with five responses scored 0-4, giving a subscore total out of 20 [7].

Determining the discriminatory properties and calibration of the SSV model to adjust the SIPSO outcomes requires the SIPSO to be dichotomised at a value reflecting ‘good’ over ‘poor’ outcome. Cut points for the SIPSO subscales have not been determined in the literature. We used clinical reasoning rather than a data-driven approach to define a cut point for the purposes of this study. These cut points were defined a priori so as not to introduce bias. We felt it was important that cut points were clinically comparable to the dichotomised OHS that the SSV model was developed to predict (OHS ≤2), and chose the cut point of 15 accordingly (a score of 3 out of 4 on each SIPSO question). Post hoc examination suggested that using other cut points for both physical and social SIPSO subscores did not significantly improve discrimination.

Discrimination is measured with the C statistic and is a measure of the ability of a model to distinguish who, out of a pair of patients, will have a good over a poor outcome [11]. There is no cut-off to determine what constitutes an acceptable C statistic although, as the C statistic is identical to the area under a receiver operating curve for a dichotomous outcome, a value of 0.5 indicates the model is no better than chance [12].

We have found that, in our dataset, the SSV model predicts ‘good’ physical outcome measured with the dichotomised physical subscore of the SIPSO with a similar degree of accuracy (C statistic = 0.77 [0.71-0.82]) to the prediction of independent survival. Previous studies have shown C statistics for the prediction of 6-month post-stroke independent survival (OHS ≤2) with the SSV model of between 0.79 and 0.84 [6,12,13,14].

The C statistic for social outcome was less reliable (0.70 [0.63-0.76]). The variables of the SSV are conceptually linked to physical rather than social constructs, which may explain the superior performance of the SSV in the prediction of physical patient-reported outcome.

Discrimination for both physical and social outcomes was improved with the inclusion of patients who did not survive to follow-up. The original SSV model was developed to predict independent survival [6]; predicted probability of good outcome for patients who did not survive to follow-up with these models is therefore likely to be low, which is likely to explain the improved discrimination when these cases were included.

Perfect calibration is indicated by a line where observed and predicted probabilities are identical across all deciles of predicted probability (x = y). Deviation from this line suggests that the model tends to over- or under-predict outcome in the study population. Calibration plots (fig. 1) demonstrate that the SSV tended to make over-optimistic individual predictions of both physical and social SIPSO outcomes. However, due to the small sample size, the confidence limits around these probabilities are wide, and further calibration in a larger dataset is desirable.

Conclusion

We have demonstrated that the SSV case-mix adjustment model is able to discriminate good over poor patient-reported functional outcome measured with the physical subscore of the patient-reported SIPSO. This suggests that the model may be more widely generalizable to outcomes other than the hard endpoints for which it was developed. Wider utility of the model would have benefits for the adjustment of routinely collected datasets, patient-reported outcome measures and in future research studies.

Funding

We are pleased to acknowledge funding from the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) at Leeds, York and Bradford. The views expressed are those of the authors and not necessarily those of the National Health Service (NHS), the NIHR or the Department of Health.

Disclosure Statement

The authors declare no conflicts of interest.