Treatment success of internet-based vestibular rehabilitation in general practice: development and internal validation of a prediction model.

OBJECTIVES: To develop and internally validate prediction models to assess treatment success of both stand-alone and blended online vestibular rehabilitation (VR) in patients with chronic vestibular syndrome.
DESIGN: Secondary analysis of a randomised controlled trial.
SETTING: 59 general practices in The Netherlands. PARTICIPANTS: 202 adults, aged 50 years and older with a chronic vestibular syndrome who received either stand-alone VR (98) or blended VR (104). Stand-alone VR consisted of a 6-week, internet-based intervention with weekly online sessions and daily exercises. In blended VR, the same intervention was supplemented with physiotherapy support. MAIN OUTCOME MEASURES: Successful treatment was defined as: clinically relevant improvement of (1) vestibular symptoms (≥3 points improvement Vertigo Symptom Scale-Short Form); (2) vestibular-related disability (>11 points improvement Dizziness Handicap Inventory); and (3) both vestibular symptoms and vestibular-related disability. We assessed performance of the predictive models by applying calibration plots, Hosmer-Lemeshow statistics, area under the receiver operating characteristic curves (AUC) and applied internal validation.
RESULTS: Improvement of vestibular symptoms, vestibular-related disability or both was seen in 121, 81 and 64 participants, respectively. We generated predictive models for each outcome, resulting in different predictors in the final models. Calibration for all models was adequate with non-significant Hosmer-Lemeshow statistics, but the discriminative ability of the final predictive models was poor (AUC 0.54 to 0.61). None of the identified models are therefore suitable for use in daily general practice to predict treatment success of online VR.
CONCLUSION: It is difficult to predict treatment success of internet-based VR and it remains unclear who should be treated with stand-alone VR or blended VR. Because we were unable to develop a useful prediction model, the decision to offer stand-alone or blended VR should for now be based on availability, cost effectiveness and patient preference. TRIAL REGISTRATION NUMBER: The Netherlands Trial Register NTR5712. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

RCT Entities:   Population Interventions Outcomes

This is the first study to develop prediction models to assess treatment success of both stand-alone and blended online vestibular rehabilitation in patients with chronic vestibular syndrome.

Strengths of our study are consistent data collection in a randomised controlled trial setting, and high generalisability of results by only using predictors easily obtainable in daily general practice.

Limitations of the study are the relatively small number of participants, limited number of included potential predictors and the inability to include predictors that were not collected in the randomised controlled trial.

Introduction

Vestibular symptoms (ie, vertigo, dizziness, vestibulovisual and postural symptoms)1 are common symptoms in general practice.2 3 Nowadays, the recommended way to approach a vestibular problem is to assess timing and triggers of vestibular symptoms to classify patients as having an acute vestibular syndrome, an episodic vestibular syndrome or a chronic vestibular syndrome.4–6 A chronic vestibular syndrome consists of vestibular symptoms, lasting months to years. Symptoms in chronic vestibular syndromes can have a progressively deteriorating course; reflect a stable, yet incomplete recovery after an acute vestibular event; or represent persistent, lingering symptoms between episodic vestibular attacks. Vestibular rehabilitation (VR), an exercise-based treatment that gradually stimulates the vestibular system, can be used to treat chronic vestibular syndromes. There is moderate to strong evidence that VR is safe and effective in reducing vestibular symptoms in unilateral and bilateral peripheral vestibular dysfunction.7 8 Currently, VR is the recommended therapy in USA,9 10 UK11 and Dutch12 guidelines to treat chronic vestibular syndromes. Nevertheless, according to surveys in the UK13 and The Netherlands14 less than 10% of general practitioners (GPs) uses VR in daily practice. To increase implementation of VR, an internet-based form of VR was developed by the University of Southampton.15 16 In a randomised controlled trial, this automated, stand-alone online VR treatment was shown to be able to reduce vestibular symptoms in patients aged 50 years and older with a chronic vestibular syndrome at 6-month follow-up compared with usual care.17 Stand-alone internet-based interventions are easily accessible, but because they are also prone to non-adherence and attrition they are often combined with face-to-face therapy (blended treatment).18–20 We recently used a Dutch translation of the stand-alone online VR treatment in a three-armed randomised controlled trial in which we compared stand-alone VR and blended VR with physiotherapeutic support to usual care.14 Both stand-alone VR and blended VR were shown to be effective in reducing vestibular symptoms at 6 months compared with usual care.21 However, this does not mean that each eligible participant benefits equally from stand-alone VR and blended VR. Certain patient characteristics may have been associated with a different treatment effect. Exploring this heterogeneity may lead to identification of profiles of patients who benefit from stand-alone VR and/or blended VR. This information could be used to construct treatment guiding decision tools and provide patients with personalised medicine.22 This approach has already shown effectiveness in other health areas. In depression, prediction models provided better antidepressant therapy recommendations than the group average suggested by the RCT.23 In chronic stress, this method showed that an internet-based stress management intervention was most effective in patients with severe stress symptoms, even though the researchers were initially reluctant to offer this low-threshold treatment to severely burdened individuals.24 Results from a qualitative interview study with participants and physiotherapists from the blended VR group indicate that the value of physiotherapeutic support in addition to the stand-alone VR intervention may differ between patients.21 The aim of this study was to develop and internally validate prediction models to predict treatment success of online VR in order to decide which patients with chronic vestibular syndrome should be offered stand-alone VR and who should be offered blended VR.

Methods

Study design and participants

We used data that were prospectively collected in a recent pragmatic randomised controlled trial investigating the effectiveness of stand-alone and blended online VR compared with usual care in general practice. The study protocol14 and results of the effectiveness analyses21 can be found elsewhere. In short, 322 patients aged 50 years and older with a chronic vestibular syndrome were randomised to receive stand-alone VR (n=98), blended VR (n=104) or usual care (n=120). Stand-alone VR consisted of a 6-week, internet-based intervention with weekly online sessions and daily VR exercises (10–20 min/day). In the blended VR group, the same internet-based intervention was supplemented by face-to-face physiotherapy support (home visits in weeks 1 and 3). Participants in the usual care group received usual GP care without any restrictions. Measurements were collected at baseline, 3 months and 6 months follow-up. For this prediction study, we only used data from participants who were allocated to stand-alone VR (n=98) or blended VR (n=104). We published a study protocol before the start of the trial,14 and followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) recommendations to report on this prediction model study.25

Outcome variable

We assessed improvement 6 months after baseline with the Vertigo Symptom Scale-Short Form (VSS-SF),26 which assessed vestibular symptoms; and the Dizziness Handicap Inventory (DHI),27 28 which assessed vestibular-related impairment. In this prediction study, we defined the dependent dichotomous outcome variable ‘successful treatment’ in three different ways: (1) a decrease in vestibular symptoms by improvement in VSS-SF score with at least three points (clinically significant change)17 29–31; (2) a decrease in vestibular-related impairment by improvement in the DHI score with at least 11 points (clinically significant change)28 32; and (3) a decrease in vestibular symptoms and vestibular-related impairment defined as clinically significant improvement in both the VSS-SF and DHI score. Although we planned to include subjective improvement compared with baseline (yes/no) as a dichotomous outcome measure in the study protocol, we decided to only use VSS-SF and DHI because, unlike dichotomous subjective improvement, these questionnaires have been validated in general practice.14

Predictor variables

We chose to use candidate predictors that can be easily obtained by GPs in daily practice. The final selection was based on a combination of previous studies10 33–37 and clinical reasoning, which can be explained as the cognitive process of clinicians underlying diagnosis and management of a patient’s presenting problem.38 We identified the following six candidate predictors before conducting the analyses: (1) age (continuous)10 33 35–37; (2) level of education (dichotomous, ie, low vs middle/high)10 35 36; (3) time since onset (dichotomous, ie, less vs more than 2 years)10 34 37; (4) sex (dichotomous);36 (5) living status (dichotomous, that is, alone vs together with partner)34 36 37; and (6) the presence of a panic disorder, generalised anxiety disorder or major depressive disorder at baseline (dichotomous).10 34 37 For the continuous variable age, we used restricted cubic spline regression functions to test for non-linearity in each model.39

Statistical analyses

Development of prediction models

We developed multiple prediction models using multivariable logistic regression models. We used logistic regression models to develop the prediction models as these models have shown to perform similar as machine learning techniques. Moreover, machine learning techniques are developed for high dimensional data situations, ie, much more variables compared with subjects, which is not the case in our study.40 To deal with missing values, multiple imputation according Multivariate Imputation by Chained Equations was applied (m=5) and the models were created using the Rubin’s Rules pooled estimates. To develop the prediction models, we first assessed if successful treatment of stand-alone and blended online VR was associated with specific candidate predictors by adding predictor-treatment interactions to the model. Subsequently, we developed new models for each of the three different outcomes of ‘successful treatment’ and used the initial set of candidate predictors for all models. Because adding more than one interaction caused model instability, we only included the interaction with the lowest p-value from the first step in each model. Finally, we excluded candidate predictors with a p<0.157 from the multivariable logistic regression models through backwards selection, that is, the model with all potential predictor variables and the interaction term with the lowest p-value included. In addition, we also performed secondary analyses without interaction terms to assess predictors of successful treatment separately in stand-alone internet-based VR participants (n=98) and blended internet-based VR participants (n=104).

Internal validation and performance of prediction models

Prediction models perform better in development cohorts than in other similar populations due to overfitting.39 41 42 Therefore, we performed an internal validation bootstrapping procedure in order to validate the final models and to adjust (shrink) the estimated performance and regression coefficients. To assess the calibration of the final models, we calculated the Hosmer-Lemeshow goodness-of-fit statistic and constructed calibration plots. To assess the discriminative ability of the models, we calculated the area under the curve (AUC). We performed all statistical analyses with RStudio statistical software (V.1.1.463), the mice (V.3.7.0), the rms (V.5.1–3.1) and psfmi (V.0.2.0) package.

Patient and public involvement

Patients played an important role in the development of our internet-based VR interventions. Detailed feedback by patients with vestibular symptoms on the content, usability and Dutch translation in prototype versions led to several improvements. No patients advised on interpretation of the results, nor were they involved in writing the manuscript. A lay summary of the research findings will be distributed to all participants in the study and the results will be disseminated to the relevant patient community.

Results

We analysed 204 participants, 98 who received stand-alone VR and 104 who received blended VR. Follow-up data for outcomes at 6 months was complete for 176 out of 202 patients (87%). Clinically relevant improvement after 6 months in vestibular symptoms was seen in 121 participants, clinically relevant improvement in vestibular-related disability was seen in 81 participants, and 64 participants experienced both. The baseline characteristics of the predictors and outcomes for all participants are summarised in table 1.

Table 1

Baseline characteristics and treatment outcomes

	Stand-alone VR	Blended VR	Total sample
	(n=98)	(n=104)	(n=202)
Predictors
Mean age in years (SD)	66.7 (9.5)	67.4 (9.8)	67.1 (9.6)
Level of education
Low	33 (34)	37 (36)	70 (35)
Middle/high	65 (66)	67 (64)	132 (65)
Time since onset*
One month to 2 years	43 (44)	49 (47)	92 (46)
More than 2 years	54 (56)	55 (53)	109 (54)
Sex
Female	64 (65)	69 (66)	133 (66)
Male	34 (35)	35 (34)	69 (34)
Living situation
With partner	64 (65)	71 (68)	135 (67)
Alone	34 (35)	33 (32)	67 (33)
Panic disorder, generalised anxiety disorder or major depressive disorder at baseline according to PHQ
Absent	84 (86)	88 (85)	172 (85)
Present	14 (14)	16 (15)	30 (15)
Outcomes
Clinically relevant improvement in vestibular symptoms after 6 months†
No	23 (24)	34 (33)	57 (28)
Yes	62 (63)	59 (57)	121 (60)
Clinically relevant improvement in vestibular-related disability after 6 months‡
No	48 (49)	50 (48)	98 (49)
Yes	39 (40)	42 (40)	81 (40)
Clinically relevant improvement in both vestibular symptoms and vestibular-related disability after 6 months*§
No	50 (51)	62 (60)	112 (55)
Yes	34 (35)	30 (29)	64 (32)

Figures are numbers (percentages) unless stated otherwise.

*Data on variable missing for one participants (n=1 stand-alone VR).

†Data on variable missing for 24 participants (n=13 stand-alone VR, n=11 blended VR).

‡Data on variable missing for 23 participants (n=11 stand-alone VR, n=12 blended VR).

§Data on variable missing for 26 participants (n=14 stand-alone VR, n=12 blended VR).

VR, vestibular rehabilitation.

Developed predictive models

In the main analysis, where we assessed if successful treatment was associated with different candidate predictors for stand-alone and blended online VR (n=202), we found different predictors for the three defined outcomes of ‘successful treatment’ (table 2). For the first defined outcome of ‘successful treatment’, clinically significant improvement in vestibular symptoms, living alone decreased the chance of treatment success (OR 0.38; 95% CI 0.19 to 0.78) and having a psychiatric disorder increased the chance (OR 2.48; 0.86 to 7.16).

Table 2

Pooled estimates of the final prediction models for treatment success after internet-based VR at 6-month follow-up (stand-alone and blended vestibular rehabilitation (VR) participants; n=202)

	Predictors	Regression coefficient	ORs (95% CI)	P value
‘Successful treatment’ 1:Clinically relevant improvement in VSS-SF score†	Living alone	−0.97	0.38 (0.19 to 0.78)	0.01
	Presence of a psychiatric disorder	0.90	2.48 (0.86 to 7.16)	0.09
‘Successful treatment’ 2:Clinically relevant improvement in DHI score†	Interaction living alone * blended VR treatment	−1.10	0.33 (0.10 to 1.15)	0.08
	Living alone	0.95	2.69 (1.04 to 6.45)	0.04
	Blended VR treatment	0.45	1.57 (0.75 to 3.29)	0.23
	Age	−0.04	0.96 (0.93 to 1.00)	0.03
	Middle/high level of education	−0.61	0.54 (0.29 to 1.02)	0.06
‘Successful treatment’ 3:Clinically relevant improvement in both VSS-SF and DHI score§	Interaction more than 2 years since onset * blended VR treatment	−1.07	0.34 (0.09 to 1.25)	0.10
	More than 2 years since onset	0.19	1.21 (0.46 to 3.16)	0.69
	Blended VR treatment	1.34	3.82 (0.50 to 29.41)	0.45

*Age was analysed as a linear predictor in all models, spine regression modelling revealed no non-linear relationship of age with the outcome.

†Predictor-treatment interaction with ‘time since onset’ had the lowest p value of all interactions (p=0.19) and was included in the model.

‡Predictor-treatment interaction with ‘living status’ had the lowest p value of all interactions (p=0.09) and was included in the model.

§Predictor-treatment interaction with ‘time since onset’ had the lowest p value of all interactions (p=0.10) and was included in the model.

DHI, Dizziness Handicap Inventory; VSS-SF, Vertigo Symptom Scale—Short Form.

For the second outcome of ‘successful treatment’, clinically significant improvement in vestibular-related impairment, patients who both lived alone and received blended VR had a lower chance of treatment success (OR 0.33; 0.10 to 1.15). Being older (OR 0.96; 0.93 to 1.00) and having a middle or high level of education (OR 0.54; 0.29 to 1.02) also reduced the chance of successful treatment. For the third outcome of ‘successful treatment’, clinically significant improvement in both vestibular symptoms and vestibular-related impairment, patients who both experienced vestibular symptoms since more than 2 years and received blended VR had a decreased chance of treatment success (OR 0.34; 0.09 to 1.25).

In secondary analyses, where we looked at predictors of treatment success of internet-based VR separately in stand-alone (stand-alone VR participants; n=98) and blended VR (blended VR participants; n=104), the predictors in the final models also differed for the three outcomes of ‘successful treatment’. In stand-alone VR participants, living alone decreased the chance of improvement of vestibular symptoms (OR 0.39; 0.15 to 1.00) but increased the change of improvement of vestibular-related disability (OR 2.79; 1.07 to 7.27). In blended VR participants, the chance of combined improvement in both vestibular symptoms and vestibular-related impairment was reduced when participants had experienced vestibular symptoms for more than 2 years (OR 0.42; 0.18 to 0.96). The complete final models are shown in online supplemental table 1 (stand-alone VR participants; n=98) and online supplemental table 2 (blended VR participants; n=104).

Internal validation and performance of the models

For the first outcome, clinically significant improvement in vestibular symptoms, calibration of the prediction model was adequate based on the both the calibration plot (figure 1) and the non-significant Hosmer-Lemeshow statistic (p=0.77). Internal validation revealed substantial overfitting with a shrinkage factors of 0.70. The corrected median AUC was poor with 0.61. The second outcome of ‘successful treatment’, clinically significant improvement in vestibular-related impairment, also showed adequate calibration (figure 1; Hosmer-Lemeshow statistic p=0.94), substantial overfitting (shrinkage factor 0.60) and a poor performance of the model (corrected median AUC 0.56). The prediction model for the third outcome, both clinically significant improvement in vestibular symptoms and vestibular-related impairment, showed similar results on internal validation (calibration plot figure 1; Hosmer-Lemeshow statistic p=1.00; shrinkage factor 0.57; corrected median AUC 0.54). The ROC curves for the updated models for all outcomes of the main analysis are shown in figure 2. The results of the internal validation of the secondary analyses, in which we assessed predictors of ‘successful treatment’ separately in stand-alone (stand-alone VR participants; n=98) and blended VR (blended VR participants; n=104) are shown in online supplemental table 3.

Figure 1

Calibration plots for main analysis (stand-alone and blended vestibular rehabilitation (VR) participants; n=202). DHI, Dizziness Handicap Inventory; VSS-SF, Vertigo Symptom Scale—Short Form.

Figure 2

Receiver operating characteristic (ROC) curves for updated models of main analysis (stand-alone and blended vestibular rehabilitation participants; n=202). DHI, Dizziness Handicap Inventory; VSS-SF, Vertigo Symptom Scale—Short Form.

Discussion

Principal findings

In this study we developed and internally validated several models to predict clinically relevant improvement of vestibular symptoms and/or vestibular-related impairment after internet-based VR. Our final predictive models showed adequate calibration, but they all showed poor discriminative abilities, with AUCs ranging from 0.54 to 0.61. Our findings indicate that treatment success of internet-based VR is difficult to predict when using common and easy obtainable variables.

Comparison with existing literature

This is the first study to predict successful treatment with internet-based VR in patients with a chronic vestibular syndrome. However, the effects of personal characteristics on outcomes has been investigated in VR delivered by physiotherapists.10 According to the VR guideline of the American Physical Therapy Association age, sex and time since onset are stated not to influence treatment success of (not internet-based) VR based on, respectively, strong, weak and moderately strong evidence.10 Psychiatric disorders and the use of vestibular-suppressant medications were identified as negative factors in recovery following VR.10 In our study, age appeared to be a negative predictor for an improvement of vestibular-related disability and the presence of a psychiatric disorder surprisingly appeared to be a positive predictor for improvement of vestibular symptoms. However, the implications of these findings have to be interpreted with caution, considering the poor discriminative abilities of the final prediction models. We also attempted to identify favourable treatment profiles to guide a treatment decision for stand-alone VR or blended VR, by analysing interactions in the total dataset and developing prediction models for each group of VR participants separately. Participants who lived alone and received blended VR had a lower chance of treatment success on the second outcome, improvement of vestibular-related impairment. Similarly, a decreased chance of treatment success on the third outcome, combined improvement of vestibular symptoms and vestibular-related disability, was seen in participants with vestibular symptoms for more than 2 years who received blended VR. These findings indicate that patients who live alone and have experienced vestibular symptoms for more than 2 years may be less suited to receive blended VR. This is unexpected, since previous studies suggest that these factors are related to lower adherence, which would theoretically make blended VR more appropriate.20 43 However, as noted before, the discriminative abilities of the final predictive models are not good enough to conclusively state that stand-alone VR should be the treatment of choice for these patients. Unlike previously mentioned applications in conditions such as depression23 and chronic stress,24 predictive models in internet-based VR are not yet ready to help personalise treatment. By including more potential predictors, the discriminative ability of the predictive models might be increased. The use of vestibular-suppressant medication,10 comorbidities,34 44 the severity and type of vestibular symptoms,33 overall self-rated quality of life,34 previous falls45 and level of self-efficacy46 47 are all factors that may influence treatment success of stand-alone and blended VR. Unlike the predictors used in our study, collecting information on these personal characteristics will take more time because patients will have to fill out extra questionnaires. By investigating both the easily obtainable predictors we used in our study and these additional potential predictors in a larger dataset, we may be able to produce a prediction model suitable for clinical practice in the future.

Strengths and limitations

Our study has several strengths. Performing the study in a randomised controlled trial setting provided accurate, consistent data collection and a relatively low level of missing data. The generalisability of this prediction study is also relatively high because of the pragmatic nature of the randomised controlled trial and the choice we made to only include potential predictors that can be easily obtained in general practice. There were also some limitations. First, the relatively small number of 202 participants limited the number of predictors we could include in the study and the discriminative ability of the models. For the secondary analyses, we conducted in the separate stand-alone VR and blended VR populations the number of participants was even lower. However, a small sample size is common in prediction studies that use data from RCTs and would have been less of an issue if the chosen predictors had been stronger.48 Second, adding more than one predictor-treatment interaction caused the models to become unstable. We therefore only added the interaction with the lowest p value to the model. A result of this approach could be that viable interactions were missing in the final predictive model. Third, the selection of potential predictors was based on results from previous studies, clinical expertise, the availability in the study data and the possibility to easily collect the data in daily general practice. As a result of this premodelling selection strategy, important predictors may have been missed. For instance, we had no complete data on the daily use of vestibular-suppressant medications during the 6-month follow-up period and the number of comorbidities was thought to be too difficult to apply in daily general practice. When choosing the potential predictors before the analyses, we assumed that, based on previous studies and clinical reasoning, our selection reflected the strongest possible predictors.

Conclusions and implications for research and/or practice

Our study results suggest that treatment success with internet-based VR is difficult to predict. Although we were able to develop several prediction models, the discriminative abilities of these models are currently insufficient to endorse their use in daily general practice. Further research should first be aimed at improving the discriminative ability of the models. Unfortunately, it is still unclear who should be treated with stand-alone VR and who should be offered blended VR. At the moment, choosing between both effective forms of online VR should therefore be dependent on other factors such as availability, costs and personal patient preference.