Reproducibility and reliability of performance indicators to evaluate the therapeutic effectiveness of biofeedback therapy after elbow surgery: An observational case series.

Electromyographic biofeedback (EMG-BF) therapy provides information on the state of contraction of the targeted muscles and relaxation of their antagonists, which can facilitate early active range of motion (RoM) after elbow surgery. Our aim in this study was to calculate the minimum detectable change (MDC) during EMG-BF therapy, initiated in the early postoperative period after elbow surgery.This study is an observational case series. EMG-BF of muscle contraction and relaxation was provided during active elbow flexion and extension exercises. Patients completed 3 sets of 10 trials each of flexion and extension over 4 weeks. The total range of flexion-extension motion and scores on the Japanese Society for Surgery of the Hand version of the disability of the arm, shoulder, and hand questionnaire and the Japanese version of the Patient-Rated Elbow Evaluation were obtained at baseline and weekly during the 4-week intervention period. A prediction formula was developed from the time-series data obtained during the intervention period, using the least-squares method. The estimated value was calculated by removing the slope from the prediction formula and adding the initial scores to residuals between the measured scores and predicted scores individually. Systematic error, MDC at the 95th percentile cutoff (MDC95), repeatability of the measures, and the change from the baseline to each time-point of intervention were assessed.The MDC95 was obtained for all 3 outcome measures and the range of values was as follows: RoM, 8.3° to 22.5°; Japanese version of the Patient-Rated Elbow Evaluation score, 17.6 to 30.6 points; and disability of the arm, shoulder, and hand questionnaire subscale: disability and symptoms score, 14.2 to 22.9 points.The efficacy of EMG-BF after elbow surgery was reflected in earlier initiation of elbow RoM after surgery and improvement in patient-reported upper limb function scores. The calculated MDC95 cut-offs could be used as reference values to assess the therapeutic effects of EMG-BF in individuals.

Introduction

The elbow joint is particularly prone to contracture development after surgery due to the shortening of the peri-articular soft tissues during prolonged immobilization.[ Hence, early mobilization of the elbow joint after surgery is recommended to avoid this complication.[ Fear of moving the elbow immediately after surgery is, thus, a risk factor for postoperative contracture.[ Electromyographic biofeedback (EMG-BF) therapy provides information on the state of contraction of the targeted muscles and relaxation of their antagonists, which can facilitate early active range of motion (RoM) after elbow surgery, as well as reduce the severity of pain and, therefore, anxiety around moving the elbow.[ The ability to “self-regulate” the contraction and relaxation of the muscles during active elbow movement, based on visual feedback via the BF system, shows promise as an effective intervention to minimize joint immobility after surgery.[

In Japan, the therapeutic effectiveness of interventions to improve the elbow function after surgery is generally evaluated using both joint-specific metrics, such as RoM, and patient-reported outcomes (PRO), such as the Japanese Society for Surgery of the Hand version of the disability of the arm, shoulder, and hand questionnaire (DASH-JSSH) score[ and the Japanese version of the patient-rated elbow evaluation (PREE-J) score.[ However, clinical assessment of the therapeutic effectiveness using these measures is based only on the relative reliability of the measurement, which is principally expressed as correlation coefficients.[ Nonetheless, in the real-world setting, this measured change will include some measurement error and/or systematic bias, which would affect the interpretation of the score in clinical practice.[ It would be clinically relevant to consider the systematic bias of the measurement when evaluating the therapeutic effectiveness of an intervention, including EMG-BF therapy. Furthermore, the use of the PRO measures, such as the DASH score, to evaluate the change in the patient status or the therapeutic effectiveness of an intervention requires an understanding of how the different outcome measures relate to each other.[ The minimal detectable change (MDC)[ is the minimal amount of change that is not likely to be due to chance variation in measurement and is, thus, clinically meaningful.[ With respect to the PRO for the upper limb function, the MDC has previously been reported for the DASH score but not the PREE score. Generally, the intraclass correlation coefficient (ICC) is calculated for the steady state measurement,[ with the MDC during the acute phase and intervention periods not having been appropriately addressed, despite the clinical relevance.

The time-series data are generally used to evaluate the therapeutic effectiveness during periods of change, such as the acute phase and intervention periods, and interpreted using the trend analysis of change.[ This trend in the data must be eliminated to create a regression model of recovery.[ The elimination of this trend requires the calculation of an estimated value, using a prediction formula created by flexible discriminant analysis, unit root test, or least-square method, which is subtracted from each data point to detrend the data set.[ This study aimed to determine the MDC in EMG-BF therapy, initiated in the early postoperative period after elbow surgery, for 3 outcome measures (elbow RoM, DASH-JSSH score, and PREE-J score) typically used in practice. The calculated MDC can be used to assess the individual treatment effect of EMG-BF therapy after elbow surgery by comparing the changes in the 3 outcome measures.

Methods

Study design and setting

This was an observational case series. The protocol of this observational case series was approved by the Ethics Committee of Saitama Prefectural University on August 23, 2013 (approval no. 25513) and the Bioethics Committee of Dokkyo Medical University Saitama Medical Center on September 4, 2013 (reference number: 25015). Informed written consent was obtained from the participants.

Study population and recruitment

The patients who underwent elbow surgery at 1 of our 2 affiliated centers (First Department of Orthopedics, Dokkyo Medical University Saitama Medical Center; and Department of Orthopedics, Koshigaya Seiwa Hospital), between July 2013 and January 2017, were included. The eligible patients were diagnosed by trauma surgeons and physicians, using the AO Foundation/Orthopaedic Trauma Association fracture classification of bone fractures. The exclusion criteria included non-closure of the epiphysis, involvement of both the upper limbs, and inability to follow the instructions for EMG-BF therapy. Data for the patients’ characteristics (age, sex, dominant side, and diagnosis) and disease severity were provided using the medical records.

The sample size was determined to be sufficient through calculations using the G∗Power 3.1.1 computer program software.[ Power analysis indicated that a total of 21 participants were needed when α = 0.95 for a power of 0.95, using a change score of −0.15, as previously reported for the DASH score to be indicative of a clinically meaningful change.[ Therefore, the sample size was set at 30 patients approximately, with anticipation of a 30% dropout rate.

Postoperative rehabilitation program

All patients received standard care after elbow surgery at our medical centers and hospital, including physical therapy (with passive RoM, avoiding varus/valgus stress) and a home program intervention of active RoM within a pain-free range. Patients whose surgery included ligament repair used a functional brace, except during RoM exercises, for the first 6 weeks postoperatively. The brace included an external strut to prevent excessive valgus stress. Dynamic splinting was used after postoperative week 6 in patients who developed a severe contracture. Patients were permitted to perform minor tasks related to activities of daily living after postoperative week 6, with lifting activities being permitted after postoperative week 12 (Fig. 1).

Figure 1

Study protocol. The outcome measures of therapeutic effectiveness are the active elbow range of motion, expressed as the total sum of flexion and extension, the Japanese Society for Surgery of the Hand version of the Disability of the Arm, Shoulder, and Hand questionnaire (DASH-JSSH) disability/symptom score, and the Japanese version of the Patient-Rated Elbow Evaluation (PREE-J) score. Baseline measurements were obtained at the first biofeedback (EMG-BF) therapy session and were subsequently obtained at weekly intervals over the 4-wk period of EMG-BF intervention. In addition to EMG-BF, postoperative management included physical therapy and a home program of active range of motion. All restrictions in activities of daily living were lifted by 12 wk after surgery.

Biofeedback therapy

EMG-BF therapy was provided during the physical therapy sessions. All EMG-BF-assisted RoM exercises were performed with the patients seated in a chair with their feet on the ground. Surface EMG electrodes were secured on the skin overlying the biceps and triceps muscles, with the EMG signal recorded using the TeleMyo DTS system (Noraxon USA, Scottsdale, AZ) and provided as visual feedback on a monitor placed in front of patients[ (see Appendix for details on EMG-BF therapy).

Measured outcomes

The following outcomes were measured: total elbow RoM (sum of the range of flexion and extension), DASH-JSSH total score, and PREE-J total score. The baseline measurements were obtained during the first BF therapy session and were also obtained at the end of each of the 4 weeks of the EMG-BF therapy program (Fig. 1). The RoM was measured using a standard universal goniometer (SAKAI Medical Co., Ltd, Tokyo, Japan). Each measurement was performed 3 times, with the average value being used for analysis. The DASH-JSSH is the Japanese version of the DASH, a self-reported questionnaire developed by the American Academy of Orthopaedic Surgeons to specifically assess upper limb disability in individuals with musculoskeletal conditions.[ In this study, as we focused on the early postoperative period, we only included the disability and symptoms subscales of the DASH (DASH-DS). Each item is rated on a scale of 1 to 5, with higher scores being indicative of more severe disability and symptoms. The PREE-J is the Japanese version of the PREE, which is also a patient-reported measure developed to quantify upper limb disability and elbow-related pain.[ The PREE includes the following 2 subscales: pain (PREE-P) and function (PREE-F). The PREE-P subscale includes 5 items, rated on a 10-point scale ranging from 0 (no pain) to 10 (worst possible pain). The PREE-F includes 11 items to measure specific activities and 4 items regarding usual activities, with each item rated on a 10-point scale ranging from 0 (no difficulty) to 10 (completely impossible). All measurements were performed by a registered hand therapist.

Data for the characteristics and variables

Demographics and outcomes were measured at baseline. The following baseline factors were recorded as potential confounding variables: sex, age, affected side, dominant hand, days after surgery, diagnosis, clinical profile, and details of the surgery. Continuous variables were reported as the mean (and standard deviation)

Signal processing and analysis

We constructed a state-changing model, which includes a detrending process (using the initial y-intercept and slope of the RoM and DASH-DS and PREE-J scores) and a steady-state process (with a random variation for decomposing the slope of the RoM and DASH-DS and PREE-J scores) as follows:

where α is the initial RoM and DASH-DS and PREE-J scores; β, the slope of the RoM and DASH-DS and PREE-J scores; ε, the steady process (with random variation of the RoM and DASH-DS and PREE-J scores); and t, number of assessments. The data from each patient were fitted to the model using the least-square method, thus, eliminating the slope of the RoM and DASH-DS and PREE-J scores. The calculated α and ε values were used to evaluate the inherent random error in the RoM and DASH-DS and PREE-J scores.

Data analysis

Reproducibility was assessed by comparing the RoM and the DASH-DS and PREE-J scores across the time-points of the assessment, namely, values for the first (baseline) and second time-points (week 1), first and third time-points (week 2), first and fourth time-points (week 3), and first and fifth time-points (week 5). The ICC values were used to estimate the variance in the score between the time-points, with ICC values of 0.8 to 1.0 indicative of excellent repeatability, 0.6 to 0.8 indicative of good reliability, and <0.6 indicative of poor repeatability.[ The values are presented as the ICC, with the associated 95% confidence interval (Table 3).

Table 3

Reliability coefficient of measurements during EMG-BF therapy.

The Bland-Altman (BA) analysis was used to identify the systematic error in the measurements,[ with the difference between pairs of scores (d, x-axis) plotted against their mean (y-axis) for each outcome measure. In this way, the BA analysis identified the relationship between the measurement error and true value.

Absolute reliability was evaluated using the MDC at the 95th percentile cutoff (MDC95), which indicates the smallest change in measurement required to exceed the measurement error and indicate a true change that can be attributed to the intervention, which was EMG-BF therapy in our study.[ The MDC95 and SEM were calculated as follows[:

where SEM is the standard error of measurement,[ and 1.96 is the z-score at a 95% confidence interval for normal distribution. In this formula, the square root of 2 takes into account errors made in repeat measurements.

The change in the score between the first (baseline) and second time-points (week 1), first and third time-points (week 2), first and fourth time-points (week 3), and first and fifth time-points (week 5) was subsequently compared to the MDC95 value to determine if the change in the RoM and the DASH-DS and PREE-J scores exceeded the measurement error. In all cases, we defined statistical significance as P < .05. The participants with missing data were excluded from the analysis without compensation (Fig. 2). All statistical analyses were performed using R 3.4.2 software (R Foundation for Statistical Computing, Vienna, Austria).

Figure 2

Patient selection and inclusion criteria. Data acquisition and selection procedure for analysis.

Results

Figure 2 shows the flow chart of patient selection. The baseline characteristics of the 36 patients included in the final analysis are summarized in Table 1. The MDC95 values for all 3 outcome measures are presented in Table 2, with the range of values being as follows: RoM, 8.3° to 22.5°; PREE-J score, 17.6 to 30.6 points; and DASH-DS score, 14.2 to 22.9 points. For the detrended data (Eq. 1), the ICC values between pairs of time-points of measurement were excellent for RoM (0.80–0.97), good to excellent for the PREE-J score (0.75–0.92), and excellent for the DASH-DS score (0.86–0.95; Table 3).

Table 1

The baseline characteristics patients included in the analysis.

Table 2

MDC95 of measured outcomes during EMG-BF therapy.

Results of the BA analysis are presented in Figures 3–5. The mean change in scores were as follows: RoM, –6.1° to –0.3° (standard deviation [SD], 8.8°–20.3°); DASH-DS score, 2.2 to 4.2 points (SD, 14.4–22.6 points); and PREE-J score, 3.0 to 7.4 points (SD, 16.5–28.4 points). The BA plot confirms the absence of any systematic bias for all 3 outcome measures at each time-point of measurement, namely, baseline and week 1, baseline and week 2, baseline and week 3, and baseline and week 4.

Figure 3

Bland-Altman plots of the range of motion (RoM) measures between baseline and (A) week 1, (B) week 2, (C) week 3, and (D) week 4 of treatment. The dotted line denotes the mean difference in the scores between pairs of assessments, with the ±2 standard deviations of the mean boundaries identified.

Figure 5

Bland-Altman plots of the Japanese Society for Surgery of the Hand version of the Disability of the Arm, Shoulder, and Hand questionnaire disability/symptom (DASH-DS) scores between baseline and (A) week 1, (B) week 2, (C) week 3, and (D) week 4 of treatment. The dotted line denotes the mean difference in the scores between pairs of assessments, with the ±2 standard deviations of the mean boundaries identified.

Bland-Altman plots of the Japanese version of the patient-rated elbow evaluation (PREE-J) scores between baseline and (A) week 1, (B) week 2, (C) week 3, and (D) week 4 of treatment. The dotted line denotes the mean difference in the scores between pairs of assessments, with the ±2 standard deviations of the mean boundaries identified.

The time-series plots for RoM and for the PREE-J and DASH-DS scores for all 36 patients are shown in Figure 6. The DASH-DS and PREE-J scores decreased from baseline to the fifth time-point of measurement (week 4), with the total RoM at the elbow increasing from baseline to the fifth time-point of measurement (week 4). The values obtained after detrending are plotted in Figure 6. Compared to the estimated MDC95, the change in the RoM improved beyond the MDC95 cutoff in 8 (22%) out of 36 patients from baseline to week 1, in 20 patients (56%) from baseline to week 2, in 34 patients (94%) from baseline to week 3, and in 35 patients (97%) from baseline to week 4. A change in the PREE-J score above the estimated MDC95 was achieved in 5 patients (14%) from baseline to week 1, in 8 patients (22%) from baseline to week 2, in 20 patients (56%) from baseline to week 3, and in 22 patients (61%) from baseline to week 4. With respect to the DASH-JSSH score, a change above the MDC95 was achieved in 6 patients (17%) from baseline to week 1, in 5 patients (14%) from baseline to week 2, in 19 patients (53%) from baseline to week 3, and in 22 patients (61%) from baseline to week 4.

Figure 6

Time course of change in the (A) range of motion (RoM), (B) the Japanese version of the patient-rated elbow evaluation (PREE-J) score, and (C) the Japanese Society for Surgery of the Hand version of the Disability of the Arm, Shoulder, and Hand questionnaire disability/symptom (DASH-DS) score. The black squares represent the MDCs. The circles indicate the scores for the treatment periods subtracted from the patient's initial scores. The gray lines represent their transitions. The RoM increased beyond the MDC95 from baseline and at all time-points of assessment (from week 1 to week 4), with a concomitant decrease in the DASH-DS and PREE-J scores beyond the MDC95.

Discussion

The use of validated performance indicators improves the reliability of the assessment of the therapeutic effectiveness of interventions.[ Our results indicate that elbow RoM and the PREE-J and DASH-DS scores measured after elbow surgery are reproducible, providing a reliable measure of the change in the elbow and upper limb function to evaluate the effectiveness of an intervention (EMG-BF in our study). We evaluated the MDC95 values during the acute phase after surgery and early rehabilitation phase (4 weeks study to have estimated the MDC95 by using the time-series data from the time of surgery to the recovery period, correcting for the trend of change.

The MDC95 of the DASH-DS score calculated in this study was equivalent to that reported previously.[ In their case series of 104 patients evaluated using the DASH score after surgery, Dawson et al calculated the 95th percentile MDC90 value of 9.3 points for the pain and function subcomponents of the DASH.[ Franchignoni et al evaluated the test-retest reliability of the DASH score in a group of 255 patients with upper limb musculoskeletal disorders (including 13 elbow fractures) before and after physical therapy and reported an ICC (2, 1) value of 0.93 and an MDC90 value of 10.8 points.[ The interval between DASH score measurements in these studies ranged between 1 and 14 days; therefore, the MDC values did not reflect the recovery process, including therapeutic interventions. In our study, we included the values related to both the natural recovery after elbow surgery and the recovery related to EMG-BF. We controlled for the effects of early recovery and EMG-BF intervention on the MDC95 values by applying a detrending analysis.

The MDC95 values for the PREE-J scores have not been previously reported. In their systematic review, Vincent et al reported an ICC value for the inter-rater reliability of the PREE-J ≥0.90, but the MDC value was not calculated.[ It is possible that the MDC value for the PREE score might reflect the extent to which this PRO is used; specifically, the DASH has been translated in 47 languages, whereas the PREE has been translated in only 3 languages. PREE is a specific index for elbow joint disorders, being widely used in Japan, the United States, and Germany.[ The MDC95 value that we calculated for the PREE score in our study will serve as a clinical reference to evaluate the therapeutic effectiveness of an intervention, such as EMG-BF.

With respect to the RoM, Armstrong et al reported a measurement error of 5.9° for elbow flexion and 6.6° for elbow extension using a hand-held goniometer, based on measurements obtained in 38 patients after injury and surgery for various injuries to the elbow, forearm, or hand.[ With respect to the elbow RoM measured with a goniometer, the MDC for elbow flexion was approximately 7.0° to 9.6°.[ Our MDC95 value for the elbow RoM was equivalent to that reported previously, which ranged between 8.3° and 22.5°. These data suggested that a change of less than 10° may be considered clinically nonsignificant for the elbow RoM.

Reporting the proportion of patients who achieve a degree of improvement that is beyond the measurement error is more informative for describing the effects of the intervention than the overall mean change.[ In our study, we confirmed that changes in the RoM, PREE-J score, and DASH-DS score after the 4-week program of EMG-BF therapy exceeded the respective MDC95 estimates for each of the 3 outcome measures. The MDC can be used to determine the therapeutic effects on individuals. Furthermore, our valid method for MDC calculation by eliminating the slope from the state-changing model may be applied to calculate the MDC in the acute phase or early recovery phase. This is the first attempt of using this method for MDC calculation under these conditions; hence, the validity of this analysis method is not guaranteed and requires further confirmation.

However, the limitations of our study must be acknowledged when evaluating the application of our findings in clinical practice. First, because the control of disease severity, sex differences, and age differences is not adequate in this study, further stratification analysis is required in studies conducted in the future to clarify the effects of these factors on the measured outcomes. Second, we measured the RoM using a hand-held goniometer; hence, the inter-rater reliability of measurement was not the same as that for a smartphone or electronic goniometer.[ High reliability and validity of electric devices in measuring the active movements of the elbow joint were reported.[ Third, this study investigated data during the early treatment phase after elbow surgery. Consequently, there was a considerable difference in the test-retest interval and underlying conditions between our study and previously published studies on this topic. Although the sample size was small, it was equivalent to the number of cases in the study by Schmitt et al.[ Lastly, all measures were obtained by 1 examiner, and all patients were from the same institution. Therefore, the possibility of inherent selection bias cannot be denied, and multicenter studies are required to evaluate the reproducibility of our findings.

Conclusion

The efficacy of EMG-BF after elbow surgery was reflected in earlier initiation of elbow RoM after surgery and improvement in patient-reported upper limb function scores. The calculated MDC95 cut-offs could be used as reference values to assess the therapeutic effects of EMG-BF in individuals.

Acknowledgments

The authors thank the staff of Dokkyo Medical University Saitama Medical Center, at the Department of Rehabilitation Medicine for their contributions and operational approval to conduct the study. The authors would like to thank Editage (www.editage.jp) for English language editing.

Author contributions

Conceptualization: Rina Takahashi, Toshiyuki Ishioka, Satoru Ozeki, Toyohiro Hamaguchi.

Data curation: Rina Takahashi, Kazufumi Sano, Kazumasa Kimura, Satoru Ozeki.

Formal analysis: Rina Takahashi, Makoto Suzuki, Toyohiro Hamaguchi.

Funding acquisition: Rina Takahashi.

Investigation: Kazufumi Sano, Kazumasa Kimura.

Methodology: Rina Takahashi, Toshiyuki Ishioka.

Project administration: Toyohiro Hamaguchi.

Supervision: Kazumasa Kimura, Toshiyuki Ishioka, Naoki Nakaya, Satoru Ozeki, Toyohiro Hamaguchi.

Validation: Kazufumi Sano, Kazumasa Kimura, Naoki Nakaya, Toyohiro Hamaguchi.

Visualization: Rina Takahashi, Toyohiro Hamaguchi.

Writing – original draft: Rina Takahashi, Makoto Suzuki, Toyohiro Hamaguchi.

Writing – review & editing: Kazufumi Sano, Makoto Suzuki, Naoki Nakaya, Toyohiro Hamaguchi.