Reliability and validity of the Upper Limb Physician's Rating Scale in children with cerebral palsy.

PURPOSE: The Upper Limb Physician's Rating Scale (ULPRS) is a tool that assesses movement quality of the upper limbs. It is used as an outcome measure after botulinum toxin type A injection in children with cerebral palsy (CP). This study aimed to investigate the reliability and validity of the ULPRS in children with spastic CP.
MATERIALS AND METHODS: Thirty children with spastic CP (M:F=17:13) aged 5 to 13 years old were recruited. The ULPRS was scored based on recorded videotapes by four physicians on two separate occasions. The Melbourne Assessment of Unilateral Upper Limb Function (MUUL) was scored by an occupational therapist. Intraclass correlation coefficients (ICCs), 95% confidence intervals and weighted kappa statistics were calculated for the scores of ULPRS to obtain interrater and intrarater reliability. The relationship between ULPRS and MUUL was assessed using Pearson correlation coefficients.
RESULTS: The ICCs for the total ULPRS scores were 0.94 between raters and 0.99 to 1.00 within raters. The weighted kappa statistics for subitem scores for the ULPRS ranged from 0.67 to 1.00 within raters and from 0.46 to 0.86 between raters. The relationship between ULPRS and MUUL was strong (Pearson correlation coefficient=0.751; p<0.05).
CONCLUSION: The results demonstrated the high reliability of the total ULPRS score within and between raters. A significant concurrent validity between ULPRS and MUUL also supports the clinical utility of the ULPRS as an outcome measure of spastic upper limb in children with CP.

INTRODUCTION

Cerebral palsy (CP) is the most common neurologic disorder that causes chronic motor disability in children.1 The majority of CP cases are of the spastic type, and the upper limb is commonly involved. Spasticity has been considered a main contributor to both impairment of function and the development of deformity. Thus, reduction of spasticity plays an important role in managing children with spastic CP. Botulinum toxin type A (BoNT-A) injection into spastic upper limb muscles has been widely used as an adjunct to conventional therapeutic techniques as a means of reducing muscle spasticity and improving function in the affected arm.2 A reduction of muscle tone, changes in movement pattern, and also an improvement in upper limb function following BoNT-A injection has been described using various outcome measures.3,4,5,6,7

The Upper Limb Physician's Rating Scale (ULPRS) is a semi-quantitative modification of the Lower Limb Physician's Rating Scale. It was designed to assess changes in the movement pattern, focusing on all 3 levels of the arm (palm, forearm, and elbow) to determine whether there is an isolated functional impairment, such as thumb in palm, restricted forearm supination, or a total flexion pattern with thumb in palm, wrist in flexion, forearm supinated, and elbow flexed (Supplementary Table 1, only online).2,8,9

The ULPRS has been recommended by previous studies as an outcome measure to assess functional measures and changes in the movement pattern following BoNT-A injection.2,8,10 The merits of ULPRS are its ease of use in measuring spasticity and speed of assessment by a variety of health care providers without special training. Thus, it may be suitable as an outcome measure for periodic assessments in a busy office setting. However, the reliability and validity of ULPRS have not yet been investigated.

The Melbourne Assessment of Unilateral Upper Limb Function (MUUL) is a reliable and valid tool for measuring the quality of upper limb movement in children aged 5 to 15 years with cerebral palsy.11,12 It is widely used to examine the effectiveness of specific therapeutic interventions. In addition, it was considered the best measure of unimanual capacity in a systematic review article.13 However, to our knowledge, there is no information about the relationship between ULPRS and MUUL. We hypothesized that the ULPRS would be a reliable clinical test for assessing upper limb function in children with CP. Therefore, the aims of this study were: 1) to evaluate the intrarater and interrater reliability of the ULPRS in children with spastic cerebral palsy and 2) to examine the concurrent validity of the ULPRS by investigating its relationship with the MUUL.

MATERIALS AND METHODS

Participants

Thirty children with spastic CP (M:F=17:13) were recruited. The Gross Motor Function Classification System (GMFCS) is widely used to assess the gross motor function in children with CP, and the Manual Ability Classification System (MACS) was developed to evaluate a child's typical manual performance in daily life.14,15 GMFCS and MACS levels of the participants and the distributions of sex, age, and involvement are shown in Table 1.

Table 1

Characteristics of Participants

GMFCS, gross motor functional classification system; MACS, manual ability classification system.

Ethical approval was granted by the Institutional Review Board and Ethics Committee of Severance Hospital (#4-2012-0265). Because all of the children in this study were younger than 18 years, informed consent was obtained from the parents of the children for their participation in the study.

Measurements

All of the children were videotaped so that physicians could assess the ULPRS of the upper arm while specific tasks were performed. One physician gave instructions and demonstrated the movements while the other physician recorded the videotape. The scoring of the ULPRS from the videotape was completed by 4 physicians on 2 separate occasions with a time interval of at least one week. The score sheet includes 9 items, and 3-, 4-, and 5-point scales are used to score each component of movement tested.

The MUUL for the affected or more severely affected upper limb was scored by one occupational therapist.

Statistical analysis

Missing values on items scores were handled in the analysis as follows: 1) for single-item summaries of reliability, participants with missing values were omitted, and 2) for analysis based on total scores, mean substitution was used so that for each child, their total score was equal to the average on the non-missing items rescaled to the total number of items recorded.

The intrarater and interrater reliability of the total ULPRS scores were examined using intraclass correlation coefficients (ICCs, based on a two-way random effects model with absolute agreement). To calculate interrater and intrarater reliability of the subitems of the ULPRS (ordinal data), we used linearly weighted kappa statistics. ICCs and 95% confidence intervals (95% CI) were calculated using SPSS software 19.0 for Windows (SPSS Inc., Chicago, IL, USA). The goal of this analysis was to describe the relative homogeneity of the scores with data sets16 and to indicate the degree of agreement between ratings provided by multiple raters.17 Measures from trials 1 and 2 for each rater were compared to determine intrarater reliability. Trial 1 for each rater was used to assess interrater reliability. To further examine reliability, the correlation coefficients were used to calculate the standard error of measurement (SEM), which was calculated on the basis of inter-value differences and standard deviation. The SEMs were calculated for the ULPRS by using the following formula: SEM%=SD·√(1-ICC)×(1/mean)×100.17 The consistency of measurements between occasions for the total ULPRS score for each physician was examined using the method of Bland and Altman.18

Interpretation of the ICCs was as follows: values above 0.80 were considered very high, between 0.60 and 0.79 were moderately high, between 0.40 and 0.59 were moderate, and less than 0.40 were low.19 There were no absolute criteria for acceptable reliability; however, ICCs greater than 0.7 were considered reliable for sample-based research.20

According to the guidelines of Landis and Koch,21 kappa values of more than 0.80 were considered excellent, between 0.61 and 0.80 were substantial, between 0.40 and 0.60 were moderate, and less than 0.40 were poor to fair.

To examine the concurrent validity of ULPRS, the relationship between the total ULPRS and MUUL scores was tested using the Pearson correlation coefficient.

RESULTS

The mean ULPRS scores performed by four physicians on two different occasions, the ULPRS score at each reading and the mean score for the MUUL are listed in Table 2. There were no missing values among the data.

Table 2

ULPRS and MUUL Scores

SD, standard deviation; ULPRS, Upper Limb Physician's Rating Scale scoring; MUUL, Melbourne Assessment Unilateral Upper Limb Function; SEM, standard error of measurement.

The ICCs for the total ULPRS scores were 0.94 between raters and 0.99 to 1.00 within raters, indicating very high interrater and intrarater reliability (Table 3).

Table 3

Interrater and Intrarater Reliability as Assessed by Intra-Class Correlation Coefficients (ICC) with 95% CI for the Total Scores on the ULPRS

CI, confidence interval; ULPRS, Upper Limb Physician's Rating Scale scoring.

The weighted kappa statistics for ULPRS subitem scores ranged from 0.67 to 1.00 within raters. The k values for intrarater reliability were excellent except for subitems of finger opening, thumb in palm, and associated increase in muscle tone for Rater 3, and active supination in flexion and extension for Rater 4. On the other hand, weighted kappa statistics for subitems of ULPRS ranged from 0.46 to 0.86 between raters. The k values between raters were excellent for only 1 subitem and substantial for 4 ULPRS subitems (Table 4). Fig. 1 displays the differences in the total ULPRS scores between occasions for each rater. The means and 95% limits of agreement (LOA) between occasions were 0.03±1.54 for Rater 1, -0.03±3.37 for Rater 2, -0.13±2.79 for Rater 3, and 0.03±1.54 for Rater 4. As shown in Fig. 1, there were no obvious outliers.

Table 4

Intrarater Reliability as Assessed by Intra-Class Correlation Coefficients (ICC) with 95% Confidence Intervals (CI) and Intrarater Reliability as Assessed by Weighted Kappa Statistics for Individual Subitem Scores on the ULPRS

ULPRS, Upper Limb Physician's Rating Scale scoring.

Fig. 1

Repeat score differences for each rater. Mean: repeat score mean; difference: repeat-score difference; dotted line: median of repeat-score difference; middle solid line: mean of repeat-score difference; top and bottom solid line: 95% limits of agreement for the repeat-score difference.

The relationship between the ULPRS and MUUL was strong (Pearson correlation coefficient=0.751; p<0.05).

DISCUSSION

There are various measures to assess upper arm function in children with CP, such as the Quality of Upper Extremity Skills Test (QUEST),22 the MUUL,11 the Assisting Hand Assessment (AHA)23 and the Shriners Hospital Upper Extremity Evaluation (SHUEE),24 which differ in terms of which aspect of upper limb function and which age range they target. Therefore, clinicians may select an upper limb measurement tool to assess a specific aspect of upper limb function. The reliability and validity of an assessment tool is important in effectively measuring the outcomes of a specific therapeutic intervention. In addition, convenience and ease of administration are crucial for periodic assessments. The ULPRS is a simple tool for assessing the quality of movement and function that can be performed by a variety of health care providers, and its ease of application can be useful in a busy clinical setting. In our study, the ICCs of the total ULPRS score were over 0.9 for both intrarater and interrater reliability, which indicates strong reliability of the total ULPRS score. As far as we know, this is the first study to investigate the reliability of the ULPRS. The ICCs for interrater and intrarater reliability of the total ULPRS score in our study are compatible with those of QUEST and MUUL in the literature, which are commonly used to assess upper arm function for children with CP.11,22,25

As for the intrarater reliability of the ULPRS item scores, the weighted kappa statistics were overall excellent except for 3 subitems for Rater 3 and 2 subitems for Rater 4. On the other hand, the interrater reliability for ULPRS subitems varied from excellent to moderate agreement. In addition, there were no obvious outliers on the Bland-Altman plot, which was narrow. The LOA indicated that the agreement was strong, with the 95% LOA range within ±3.37 on a 47-point scale. These findings suggest that differences in the ULPRS of more than 3.37 after intervention when performed by the same rater may reflect a true change in function as a result of therapeutic intervention.

The ICCs for each item of the ULPRS are in accordance with the reliability of each score on the MUUL in previous studies, where the ICCs varied from low to high (ranging from 0.37 to 0.91).26,27 Some reports demonstrated that the reliability of the scores on each item of the MUUL were acceptable for all items except items 1 and 2.11,26 In contrast, only 9 of 16 items of the MUUL achieved ICCs over 0.7 in Spirtos, et al.'s27 report. Although the reason for this significant difference was not clear, differences in the severity of the subjects' cerebral palsy have been suggested as a possible reason in the Spirtos study. In our study, the weighted kappa statistics for only 4 of 9 items were greater than 0.7. The elbow flexors, pronators, and wrist flexors are commonly targeted for spastic upper limb management with BoNT-A injection,10 and thus, our results indicate that changes in movement patterns after BoNT-A injection into those muscles may be reliably measured with the ULPRS. However, ICCs for the five remaining items on the ULPRS were not adequate for clinical use; therefore, scoring guidelines for clarifying differences between individual scoring criteria are needed.

The MUUL is based on 16 items comprising of tasks that are representative of the most important components of unilateral upper limb function (reach, grasp, release, and manipulation).28 The test is administered by a therapist, with a video recording of the child's performance obtained for subsequent scoring. The reliability and validity of the MUUL has been well-established in previous studies.12,26,27 According to a recent systematic review article, MUUL is the best scale to measure unimanual functional changes as a result of spasticity management.13 Therefore, the validity of the ULPRS was assessed in relation to the MUUL. The significant positive correlation between total ULPRS and MUUL scores in our study indicates that the two scales may be comparable.

According to the International Classification of Functioning, Disability and Health (ICF), upper limb function can be measured based on body impairment, activity, and participation levels. The MUUL is a tool for assessing the activity level of ICF, whereas the ULPRS has components representing impairment levels of ICF in addition to activity levels. Differences in the way components of ICF were measured may have contributed to the weaker relationship between the ULPRS and MUUL.

Changes in movement pattern and quality after BoNT-A injection are of interest to clinicians involved in the management of spastic upper limbs. From this perspective, the ULPRS may have unique clinical value and thus is often used as an outcome measure following BoNT-A injection in the literature.2,8,10 The acceptable reliability, reproducibility, and also validity of the ULPRS demonstrated in our study support the clinical use of the ULPRS as an upper arm measure for children with spastic CP following BoNT-A injection into spastic upper limb muscles.

Videotaped evaluation of upper extremity function has several advantages as well as certain disadvantages. One advantage is the ability to observe the same performance several times to capture movement quality more accurately. On the other hand, the procedure is time-consuming, and scoring from video clips using the ULPRS may be difficult depending on the angle of the video. Therefore, intrarater and interrater reliability from videotaped performances of upper extremity function in children with CP may differ from reliability measured live. Comparison of reliability between live and video observation is needed.

In addition, the ULPRS originally aimed to measure post-intervention changes; however, therapeutic interventions were not involved in our study, the effectiveness of the ULPRS in this situation was not examined. Further study is needed to demonstrate the validity of measuring post-intervention changes.

Our study demonstrated high ICC values within and between raters for the total ULPRS score and also a strong relationship between the ULPRS and MUUL. These findings support the clinical utility of ULPRS as an outcome measure after spastic management of the upper limb. However, k values for subitem scores within and between raters varied from excellent to moderate agreements. Further efforts are required to provide scoring guidelines that clarify the differences between individual scoring criteria for the scores of each item.