Clinical assessment of balance using BBS and SARAbal in cerebellar ataxia: Synthesis of findings of a psychometric property analysis.

BACKGROUND: In the previous psychometric analysis paper in our series for identifying the core set of balance measures for the assessment of balance, we recommended the Berg Balance Scale (BBS) and balance sub-components of the Scale for the assessment and rating of ataxia (SARAbal) as psychometrically sound measures of balance for people with cerebellar ataxia (CA) secondary to multiple sclerosis.
OBJECTIVE: The present study further examined the suitability of BBS and SARAbal for the assessment of balance in CA with regard to psychometric property strength, appropriateness, interpretability, precision, acceptability and feasibility.
METHODS: Criteria to fulfill each factor was defined according to the framework of Fitzpatrick et al. (1998). Based on the findings of our previous psychometric analysis, each criterion was further analyzed.
RESULTS: The psychometric analysis reported good reliability and validity estimates for the BBS and SARAbal recommending them as psychometrically sound measures; they fulfilled both criteria for appropriateness and interpretability, the measures showed evidence for precision and acceptability, and they were found to be feasible in terms of the time and cost involved for the balance assessment.
CONCLUSION: We have provided evidence for the use of the BBS and SARAbal for the assessment of balance among people with CA.

Introduction

Poor balance and gait difficulties are hallmarks of health conditions that result in cerebellar ataxia (CA).[1] Assessment of balance and gait in CA is challenging as there are no standardized measures of balance available. Previously, a series of studies by our research group recommended a set of core measures. A systematic review[2] and a Delphi survey[3] reported the Berg Balance scale (BBS), the Timed Up and Go (TUG) test, posture and gait sub-component of the International Co-operative Ataxia Rating Scale (PG-ICARS) and the gait, stance and sit sub-components of the Scale for the Assessment and Rating of Ataxia (SARAbal) as appropriate measures of balance in CA.[2,3] Further, a psychometric property analysis was done to estimate constructs of reliability and validity of these four measures among people with CA secondary to multiple sclerosis in New Zealand and the United States of America. The study aimed at proposing the best outcome measures based on the findings of the psychometric analysis. The BBS and SARAbal were recommended as the optimal measures of balance in people with CA secondary to multiple sclerosis.[4]

Fitzpatrick et al. reported eight factors to be addressed while selecting an outcome measure for clinical trials.[5] In the process of choosing a standardized set of measures for balance in people with CA, these eight factors were considered. The present study therefore aimed to examine the psychometric properties, appropriateness, interpretability, precision, acceptability and feasibility of the BBS and SARAbal for people with CA based on the findings of the psychometric property analysis done by our research team earlier.[4]

Methods

This paper examined eight factors in light with Fitzpatrick’s framework of evaluating a suitable outcome measure for clinical trials and clinical practice.[5] The findings of the present study were based on the outcomes of a psychometric property analysis of four outcome measures of balance tested in people with CA secondary to multiple sclerosis.[4] For the present study, we grouped reliability, validity and responsiveness as psychometric properties.[4] The factors are analyzed and their definitions are listed in Table 1.

Table 1.

Descriptors of the factors analyzed.

Factor	Descriptor
Psychometric properties	Common term that includes reliability, validity and responsiveness of the outcome measures
Appropriateness	Described as how suitable the contents of the instrument are for use in people with CA[5]
Interpretability	Indicates how meaningful are the scores obtained from the outcome measures[5]
Precision	Defined as the accuracy of the instrument in categorizing sub-groups and distribution of numerical value
Acceptability	Defined as the level to which the outcome measure is tolerable for its use in people with CA[5]
Feasibility	Described as the ease of use of the outcome measure in terms of administering it, and the associated financial cost[5]

Each factor was analyzed based on the set criteria outlined as follows.

Key findings of the psychometric analysis of the BBS and SARAbal were summarized to report the reliability and validity of these measures in people with CA. The detailed methodology and results of this psychometric property analyses are published elsewhere.[4,6] The other reported factors including appropriateness, interpretability, precision, acceptability and feasibility we based on the experience gained during the data collection and interpretation of results of our previous psychometric analysis study. To summarize, 60 participants aged 18–65 years with CA secondary to multiple sclerosis were recruited. Data were collected at four outpatient units in New Zealand and the United States of America. All included participants underwent balance assessment using the BBS, TUG, SARAbal and PG-ICARS. The participants were assessed on a single occasion and during the assessment, a video recording was done. The video recording was later used to estimate the intra-rater and inter-rater reliabilities. The Barthel Index, the Expanded Disability Status Scale (EDSS), the full scales of the ICARS and the SARA were also assessed and disease duration was recorded. The EDSS was completed by a neurologist. To investigate the intra-rater and inter-rater reliabilities, a repeat assessment was performed by the same physiotherapist (intra-rater) or a second physiotherapist (inter-rater) from the video recording.

In this study, appropriateness was analyzed based on two criteria: (i) to judge whether the contents of the outcome measure suit the target population and (ii) if the recommended set of outcome measures has a combination of a condition-specific tool and a generic tool for the assessment of balance. Interpretability was analyzed based on two criteria: (i) to determine how meaningful the obtained scores were, using the BBS and the SARAbal and (ii) to determine if the outcome measures have established normative data. Precision was analyzed based on two criteria: (i) to determine if the instrument is able to discriminate between two known sub-groups within the collected samples and (ii) the accuracy of distribution of numerical values assessed using Rasch analysis or estimation of unidimensionality of the testing items using factorial analysis. Acceptability was determined by estimating the response rate of the participants to the items of the outcome measures. In general, the lesser the missing items, the better the acceptability.[5] Feasibility was assessed by observing the ease of use, cost involved for the assessment, time taken to complete and training required for the assessor to complete the balance assessment using the two outcome measures. The criteria were organized into a tabular column and the reviewer marked either “yes” if the criteria were met or “no” if the criteria were not met or “unclear” if the answer was ambiguous. Each of the criterion was independently reviewed by two authors (SW and CS) and discrepancies in findings were discussed. A third reviewer (LC) was involved for unresolved discrepancies in the findings between the first two reviewers or if the reviewers marked “unclear” for the criteria.

Balance measures

The BBS is a performance-based measure of balance[7] and has been reported to be the most commonly used balance tool by physiotherapists.[8] The BBS is a five-point ordinal scale scored between 0 and 4 for each task and has 14 tasks in total. The highest total score a participant may obtain is 56. This measure is interpreted as better balance with higher scores. Normative scores for the BBS have been established among community dwelling older adults.[9] This measure has good inter-rater () and test retest () reliabilities and low standard error of measurement (SEM).[10] The BBS is found to have acceptable concurrent validity in assessing balance and poor in discriminating between fallers and non-fallers in people with multiple sclerosis.[11]

The SARA is an ataxia severity rating measure.[12] It consists of eight items among which gait, sitting and the standing sub-components are related to balance. The full scale is scored out of 40. The three sub-components of balance are scored out of 18 (SARAbal). Scoring of the eight sub-components do not have equal weighting, with scores ranging between eight for the “gait” sub-component and four for the “heel-shin glide”. The higher the score obtained, the worse the condition. The SARA has high test re-test reliability (ICC 0.90), inter-rater reliability (ICC 0.97) and internal consistency ().[12] Structural validity has been reported,[13] satisfactory convergent validity when correlated with other ataxia rating scales[12] and adequate responsiveness has been demonstrated.[14] The testing has been done and conducted with both genetic and acquired forms of cerebellar disorders.

Results

The review of criteria for each factor resulted in 100% agreement between the reviewers and therefore the third reviewer was not approached. The reliability and validity of the measures were found to be strong and the responsiveness was not estimated. A summary of the findings on the psychometric properties of the BBS and the SARAbal are highlighted in Table 2. For appropriateness, the measures met both the criteria. With regards to interpretability, off the two required criteria, both the measures met the first criteria whereas the BBS met the second criteria and SARAbal did not. The first criteria for precision were met by both the measures however, the second criteria were not established as Rasch analysis and factor analysis were outside the scope of the psychometric analysis. Both the measures met the criteria for acceptability and in addition, they were found to be feasible.

Table 2.

Definition, accepted statistical analysis, interpretation and findings of the psychometric properties considered.

Psychometric property	Description	Statistical analysis	Interpretation	Results
Reliability
Internal consistency	Defined as the degree of interrelatedness between the test items within each outcome measures considered.[15]	Cronbach alpha	There are no universal guidelines for interpreting reliability, in general, higher the value towards 1, greater the reliability. We interpreted as follows: α>0.80: good, α between 0.5 and 0.79: moderate, α<0.50: poor	α=0.94 (BBS) α=0.72 (SARAbal)
Inter-rater reliability	Defined as the proportion of variation in the scores of the participant done by two different investigators.[16]	Continuous scores: ICC Dichotomous/nominal/ordinal scores: kappa (κ) or weighted kappa	ICC>0.80: good, ICC between 0.5 and 0.79: moderate, A<0.50: poor	ICC = 0.97 (BBS) ICC = 0.96 (SARAbal)
Intra-rater reliability	Defined as the proportion of variation in the scores of the participant done by the same investigator with an interval of 7–10 days.[15]	Same as inter-rater reliability		ICC = 0.99 (BBS) ICC =0.98 (SARAbal)
Validity
Criterion validity	Defined as the degree to which the scores of the measure under investigation are an adequate reflection of a “gold standard”.[16]	Spearman or Pearson correlation co-efficient. Since the outcome measures considered were ordinal, we used the Spearman correlation co-efficient (ρS). Since “‘gold standard” was not available for balance assessment, we correlated the measures of balance (BBS, TUG, PGICARS and SARAbal) against each other.	ρS>0.80: good, ρS between 0.5 and 0.79: moderate, ρS<0.50: poor	BBS versus TUG: −0.88 PGICARS: −0.80 SARAbal: −0.92 SARAbal versus BBS: −0.92 TUG: 0.72 PGICARS: 0.92
Hypothesis testing	Defined as the degree to which the scores of the measures under investigation are consistent with the hypotheses.[16] Convergent, divergent, external and construct validity are grouped under hypothesis testing.	Spearman correlation co-efficient (ρS).	Same as above
Convergent validity	Indicates that two measures examining similar underlying phenomenon will provide similar results. For example, high correlation can be anticipated between the results of two outcome measures assessing balance.	Spearman correlation co-efficient (ρS). The measures of balance were correlated with two ataxia rating scales (ICARS and SARA)	Same as above	BBS versus ICARS: −0.76 SARA: −0.82 SARAbal versus ICARS: 0.79 SARA: 0.85
External validity	Defined as the degree to which the outcome measure under investigation correlates with other instruments or other constructs, for example ADL, disease severity or disease duration.	Spearman correlation co-efficient (ρS). The measures of balance were correlated with ADL status, disease duration and disease severity. ADL was assessed using BI and disease severity using the EDSS.	Same as above	BBS versus EDSS: −0.78 BI: 0.55 Disease duration: −0.61 SARAbal versus EDSS: 0.76 BI: −0.44 Disease duration: 0.58
Discriminant validity	Defined as the ability of the outcome measures to differentiate between two-known groups within the study population.	Group differences of scores between users and non-users of assistive walking devices were considered for establishing discriminant validity. We used Mann–Whitney U test.	Statistically significant difference (p<0.05) in the scores between groups was considered evidence for discriminant validity.	Mean, SD and p value BBS ADU: 34.6 (11.8) ADNU: 52.19 (4.43) p<0.01 SARAbal ADU:7.0(2.8) ADNU: 1.71 (1.37) p<0.01
Cut-off score, sensitivity and specificity for assistive walking device use	Sensitivity is an indication that the outcome measure is capable of identifying certain trait that is really present in the given population. Specificity is an indication that the outcome measure is capable of identifying the lack of certain trait that is really absent in the given population.	Receiver operating characteristics (ROC) curve was constructed to determine the cut-off score, sensitivity and specificity of the measures to predict the users of an assistive walking device. In addition, to determine and quantify which measure had a better predictive ability, the “Area Under the Curve” (AUC) was used.	The examiner makes a logical decision based on the needs for the cut-off score. In this case, the score needs to precisely identify an assistive device user more than identifying a non-user. Thereof, the sensitivity was kept high and constant at 90% and the corresponding cut-off score and the highest specificity at 90% of sensitivity were derived.	BBS: cut-off < 44 out of 56 sensitivity 90% specificity 94% SARAbal: cut-off >5 out of 18 sensitivity 90% specificity 100%
Responsiveness
Responsiveness	Described as the ability of the outcome measure to detect changes over time.[16]	Can be determined using different approaches. Some commonly adopted analysis include ROC (distribution-based approach) or relating the change of score to “Global Rating of Change” score (anchor-based approach).[17]		Responsiveness was not estimated.

Notes:  — Cronbach’s alpha, ICC — intra class correlation co-efficient,  — Spearman’s Rho, BBS — Berg Balance Scale, SARAbal, gait, sit and stance sub-component of the SARA, PGICARS — Posture and gait sub-component of the ICARS, SARA — Scale for the Assessment and Rating of Ataxia, ICARS — International Co-operative Ataxia Rating Scale, ADl — activities of daily living, BI — Barthel Index, EDSS — Expanded Disability Status Scale, ADU — assistive device user, ADNU — assistive device non-user SD — standard deviation,  — level of significance.

Discussion

This study aimed at identifying the suitability of using the BBS and SARAbal for the clinical assessment of balance in people with CA. The framework of Fitzpatrick et al.[5] was used to address eight independent factors for this recommendation. We have provided evidence for most of the factors and in addition, recommendations for future research for strengthening the present findings have been provided.

Psychometric properties of the measures of balance

The BBS and SARAbal reported good intra-rater, inter-rater reliabilities and internal consistency.[4] The criterion validity was found to be good for both the measures (). The measures were correlated against disease duration, disease severity and functional independence to determine construct validity and correlation was moderate (). The measures were correlated against ataxia severity rating scales to estimate convergent validity which was found to be good. The study participants were sub-divided into assistive walking device users and non-users. The ability of the measures of balance to differentiate between users and non-users of assistive devices was studied to determine the discriminant validity. The balance scores showed a statistically significant difference between the scores of assistive device users and non-users showing evidence for discriminant validity. In summary, both the BBS and SARAbal have good reliability and acceptable validity for the assessment of balance among people with CA secondary to multiple sclerosis. The structural validity and responsiveness of the measures of balance were not determined.

Appropriateness of the measures of balance

A straightforward method to determine if the contents of the outcome measure suit the target population is to obtain feedback from end users, the clinicians. The psychometric analysis involved testing four balance measures of which three were endorsed by experts through the Delphi survey done earlier by our research team.[3] In the Delphi survey, neurologists and physiotherapists involved in research and clinical practice of CA were interviewed. They were asked to indicate the most appropriate measure of balance they might use to quantify balance deficits relating to CA. The internet-based survey went on for two rounds and the participants came to a consensus on the use of the BBS, TUG and SARAbal as the most appropriate choice of assessment tool. Two of the measures recommended as the core set were those endorsed by the clinical experts in the Delphi study providing evidence for appropriateness.[3]

Secondly, it is recommended that an appropriate set of patient outcome measures should have one condition-specific measure and a generic measure.[5] A condition-specific measure identifies changes that are in close relation or “proximal” to the disease such as difficulty in performing tandem walking in CA and the generic measure identifies changes that are slightly less proximal or “distal” to the health condition,[18] such as altered stepping secondary to coordination deficits in CA. Among the core set of measures, the SARAbal is condition-specific and the BBS is a generic measure of balance.[2]

Interpretability of the measures of balance

In order to identify a meaningful score, the most significant approach may be to relate the scores achieved to the minimal clinically important difference (MCID).[5] The MCID is described as the smallest difference in the score following an intervention that the patient perceives as beneficial.[19] Since the psychometric analysis did not involve a repeat assessment, where arguably a change in score could be expected, determining the MCID score was not possible. However, we established the minimal detectable change (MDC) for the BBS and the SARAbal.

The MDC is described as the smallest change that an outcome measure detects due to a notable change in the participants’ performance. The established MDC is a reflection of the SEM for the measures of balance and could be considered as a “proxy” for the MDC. The term “proxy” in statistics refers to a value that is probably not in itself of any great interest, but from which a variable of interest can be obtained. The MDC was estimated using a data-driven method proposed by Wyrwich et al.[20] The Cronbach alpha of the measures of balance was used to estimate the SEM that reflected the MDC. Therefore, the derived MDC provides meaningful information on the expected change in score that may be perceived to be clinically meaningful for the patient following intervention. Future studies may use the obtained MDC as reference scores for reporting their results.

The second method of assessing interpretability is to compare the scores with normative data in a way that the difference in the score reflects the magnitude of difference between the tested sample and an age-matched healthy peer. The BBS has established normative data among community dwelling healthy older adults.[9] Being condition-specific and relatively new, the SARA does not have established normative data. Future studies are recommended to establish the normative scores for the SARAbal among healthy older adults.

Acceptability of the measures of balance

The response rate to the outcome measures was high for the psychometric analysis and there were no missing items in our data providing evidence for acceptability.[5] Acceptability can also be demonstrated by determining the floor and ceiling effect of the tool. These estimates report on the level of ease to complete the items i.e., were the contents of the tool too easy or too difficult or tolerable for the tested population? Determining the acceptability was outside the scope of the psychometric analysis; however, based on the findings of the psychometric analysis, the answer to this question may be partially resolved. Of the 60 participants, only one (2%) had difficulty in completing all four assessments due to fatigue providing some evidence for acceptable floor effect. The participants were able to complete all four tests. Eight (13%) participants obtained full score for BBS and five (8%) for the SARAbal. However, we hesitate to comment on the question “were the contents too easy to complete?” Future studies are recommended to estimate the floor and ceiling effect for these measures of balance.

Precision of the measures of balance

The psychometric property analysis estimated discriminant validity by sub-dividing the participants into assistive device users and non-users. Mann–Whitney test was used to determine the group differences between the two known groups (assistive device users and non-users). The findings of this analysis revealed a statistically significant () difference between the two groups for both the measures of balance providing evidence for precision.[4] Secondly, it is recommended that the precision could be derived by estimating the unidimensionality of the measures under consideration. However, unidimensionality estimation was outside the scope of the psychometric analysis. Therefore, we recommend future studies to conduct Rasch analysis or factorial analysis to provide evidence for unidimensionality for these measures in future.

Feasibility of the measures of balance

Based on the experience gained during data collection, the two measures of balance took 15–20 min to complete. They did not require the use of sophisticated equipment and are available at free of cost. In addition, formal training is not needed to perform these tests (the measures include instruction). However, the examiners who conducted these tests were qualified physiotherapists and therefore, the feasibility of administration is limited to qualified physiotherapists. With regard to patient safety, it is recommended that the assessment room is well-lighted, surface is non-slippery, and adequate rest breaks are given between the assessment sessions. There were no adverse events documented during data collection providing evidence for feasibility of the measures.

Generalizability of the findings

The findings of the present study are based on the outcomes of a psychometric analysis conducted earlier. As reported, the psychometric property analysis recruited people with CA secondary to multiple sclerosis. The recruited sample was heterogeneous in terms of disease course of multiple sclerosis which enables the generalizability of findings to all types of multiple sclerosis. In addition, the sample was homogenous in terms of the type of lesion. The included participants with multiple sclerosis were restricted to primary cerebellar impairment. Therefore, these recommendations may be considered for people with other types of cerebellar ataxic lesions.

Conclusion

The findings of this study suggests that the BBS and SARAbal are psychometrically sound, appropriate, interpretable, precise, acceptable and feasible for the assessment of balance in people with CA and multiple sclerosis. Future studies are warranted to estimate the structural validity, responsiveness, MCID, plus floor and ceiling effect for these measures to strengthen the present findings.

Conflict of Interest

The authors declare that there is no conflict of interest relevant to the study.

Funding/Support

This research was funded by the University of Otago, New Zealand Doctoral Scholarship, School of Health and Rehabilitation Research, University of Pittsburgh, USA and Division of Health Science, University of Otago, New Zealand, travel grants, Maurice and Phyllis Paykel Trust (MPPT) fund Physiotherapy New Zealand, Otago Branch Educational Fund, Physiotherapy New Zealand’s Neurology Special Interest Group Grant, School of Physiotherapy, Research fund, and was partially and financially supported by King Saud University, through Vice Deanship of Research Chairs, Rehabilitation Research Chair.

Author Contributions

Dr. Stanley John Winser contributed to structuring study design, write up for funding, subject recruitment, data collection, data review, data analysis, data interpretation, project management, writing the manuscript and revising the manuscript. Dr. Catherine Smith contributed to write up for funding, data review, data interpretation, project management and writing the manuscript. Prof. Leigh A Hale contributed to write up for funding, data interpretation, project management and writing the manuscript. Dr. Leica S Claydon contributed to write up for funding, data review, data interpretation, project management and writing the manuscript. Prof. Susan L Whitney contributed to data interpretation, project management and writing the manuscript.