Functional Status Assessment of Patients With COPD: A Systematic Review of Performance-Based Measures and Patient-Reported Measures.

Presently, there is no recommendation on how to assess functional status of chronic obstructive pulmonary disease (COPD) patients. This study aimed to summarize and systematically evaluate these measures.Studies on measures of COPD patients' functional status published before the end of January 2015 were included using a search filters in PubMed and Web of Science, screening reference lists of all included studies, and cross-checking against some relevant reviews. After title, abstract, and main text screening, the remaining was appraised using the Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) 4-point checklist. All measures from these studies were rated according to best-evidence synthesis and the best-rated measures were selected.A total of 6447 records were found and 102 studies were reviewed, suggesting 44 performance-based measures and 14 patient-reported measures. The majority of the studies focused on internal consistency, reliability, and hypothesis testing, but only 21% of them employed good or excellent methodology. Their common weaknesses include lack of checks for unidimensionality, inadequate sample sizes, no prior hypotheses, and improper methods. On average, patient-reported measures perform better than performance-based measures. The best-rated patient-reported measures are functional performance inventory (FPI), functional performance inventory short form (FPI-SF), living with COPD questionnaire (LCOPD), COPD activity rating scale (CARS), University of Cincinnati dyspnea questionnaire (UCDQ), shortness of breath with daily activities (SOBDA), and short-form pulmonary functional status scale (PFSS-11), and the best-rated performance-based measures are exercise testing: 6-minute walk test (6MWT), endurance treadmill test, and usual 4-meter gait speed (usual 4MGS).Further research is needed to evaluate the reliability and validity of performance-based measures since present studies failed to provide convincing evidence. FPI, FPI-SF, LCOPD, CARS, UCDQ, SOBDA, PFSS-11, 6MWT, endurance treadmill test, and usual 4MGS performed well and are preferable to assess functional status of COPD patients.

INTRODUCTION

Chronic obstructive pulmonary disease (COPD), characterized by persistent airflow limitation, is usually progressive and associated with an enhanced chronic inflammatory response in the airways and the lung to noxious particles or gases.[1] When the disease becomes aggravated, patients suffer from deteriorated functional status and limitations to daily life. The impaired functional status is proven to be predictors of exacerbations, hospital admissions, and mortality.[2,3] The worsening functional status presents a tough challenge for patients and their families and causes an increasing burden for the society.[4] Therefore, assessing functional status accurately and systemically is one of demanding require of COPD treatments, as indicated in COPD guidelines.[1,5,6]

Measuring the type and magnitude of functional damage and evaluating treatment effect on functional improvement is a challenging work in clinical practice. Even in some large pulmonary rehabilitation programs, variables like activities of daily living (ADLs) and exercise tolerance were not adequately assessed.[7,8] Functional status contains multidimensional constructs and is often confused with other relevant constructs.[9-13] According to the Wilson-Cleary framework, functional status was broadly defined as the ability to perform particular defined tasks in multiple domains, including physical function, social function, role function, and psychological function.[14,15] Similarly, there are also many aspects of functional status in terms of intension, including functional capacity, functional performance, functional reserve, and functional capacity utilization. Correspondingly, many functional status instruments were proposed for different purposes, including performance-based measures and patient-reported measures.[12,16-19] Two limitations in present studies, undermining the development and validation of these instruments, are (1) lack of an assessment of the quality of methodology used, resulting in unconvincing conclusions of measures’ development and/or validation; (2) lack of a clear-defined, systematical, and quantifiable assessment standard, resulting in partial and ambiguous judgments on measures performance.[20-22]

Consensus-based standards for the selection of health measurement instruments (COSMIN) was proposed in 2006.[20-23] Besides evaluating the quality of studies on measurement property critically, COSMIN also includes the measurement properties systematically. It has been used in many systematic reviews to evaluate studies and instruments of various diseases, such as hip and knee osteoarthritis, geriatrics, non-small cell lung carcinoma, and neuro-rehabilitation patients.[24-27] In terms of COPD, COSMIN has been used to assess COPD assessment test (CAT) questionnaire, health-related quality of life (HRQOL) questionnaires, and arm exercise capacity.[28-30] In this article, we employed COSMIN to review both patient-reported measures and performance-based measures of COPD patients’ functional status.

The objectives of this review is threefold: (1) to appraise the quality of methodology in the studies on the measures of COPD patients’ functional status and to provide insights for future researches, (2) to summarize all candidate instruments and to make recommendations for instrument selection, and (3) to compare performance-based measures and patients-reported measures.

METHOD

Search Strategy

We searched PubMed, Web of Science using a search filter developed by Terwee to identify studies describing development or evaluation of measurement properties of instruments measuring functional status of COPD patients up to the end of January 2015. (See text, Supplemental Digital Content 1, which describes the detailed search strategy.)[31]

1# Construct search

2# Population search

3# Instrument search

4# #1 AND #2 AND #3 AND filter for measurement properties

5# #4 NOT exclusion filter

For supplement, we searched each instrument in the entire database and looked up the references of each included article. Finally, our review was cross-checked against some relevant reviews.[16,17,19,24]

Eligibility Criteria and Study Selection

By applying the inclusion and exclusion criteria (Table 1), 3 reviewers (YL, HL, and ND) independently screened titles and abstracts of the identified records and independently assessed full texts for eligibility. Discussion was conducted when there were differences concerning exclusion criteria. If consensus could not be reached, the final decision was made by the forth reviewer (NW).

TABLE 1

Inclusion Criteria and Exclusion Criteria for Eligible Studies

Evaluation of Methodological Quality of the Included Studies

Before the evaluation of methodological quality of the included studies, descriptive variables of these studies including authors/year, country, study sample, study design, sex (female, %), mean age years ± SD (range), mean percentage of the forced vital capacity (FEV1%) predicted ± SD, and patients status were collected. Then the methodological quality of included studies was evaluated according to the COSMIN 4-point checklist.[32] The COSMIN checklist consists of 9 boxes concerning methodological standards on how each measurement property should be assessed, including 5 to 18 items in each box. The overall score (i.e., poor, fair, good, or excellent) for each item was obtained by taking the lowest score for any question within the item.

Quality Assessment of Instruments

The quality of the instruments was determined according to the rating system provided by Terwee (Table 2). It contains criteria for content validity, internal consistency, criterion validity, construct validity, reproducibility (agreement and reliability), responsiveness, floor and ceiling effects, and interpretability. Each measurement property was reported by positive (+), intermediate (?), negative (–), or no information available (0). The version provided by Terwee was used in this review.

TABLE 2

Quality Criteria for Measurement Properties[7]

Data Synthesis and Quality Assessment

To synthesize the evidence, “best-evidence synthesis” was performed. As proposed by the Cochrane Back Review Group, the levels of evidence were “strong,” ““moderate,” “limited,” “conflicting,” or “unknown” (Table 3).[33,34] Methodological quality of the studies (COSMIN score), rating of quality assessment of instruments, consistency between different studies, and the number of studies were taken into consideration using the synthesis. We defined best rated instruments as those which had a “+++” (strong positive) in at least one measurement property or a “+” or “++” in at least three measurement properties according to the results of data synthesis.

TABLE 3

Levels of Evidence for the Quality of the Measurement Property[34]

Since this study merely reviewed the articles already published without involving any human participants directly, ethical approval is not necessary.

RESULTS

Electronic Literature Search Results

The selection process for all studies is shown in Figure 1. With the search filter, 6447 records were identified. After screening the title and abstract, 6225 records were excluded. The remaining 222 records were screened for full text, among which 145 records were excluded for various reasons shown in Figure 1. Twenty-five additional records were identified through screening of references lists and review articles and searching for each particular instrument in PubMed. A total of 102 articles were analyzed in the review.

FIGURE 1

Flow diagram of search results.

Description of the Included Studies and Included Instruments

A total of 95 of the 102 studies were published after 2000. These included cross-sectional studies, longitudinal studies, and randomized double-blind studies. Mean age of subjects include in these studies ranged from 51.0 to 74.7 years. Fifty of the 102 studies declared that subjects include in their studies were stable patients. In total, 58 instruments were identified, including 44 performance-based measures and 14 patient-reported measures. The 44 performance-based measures could be divided into 28 exercise tests and 16 activity monitors. (See table, Supplemental Digital Content 2, which describes the included studies.)

Quality of the Included Studies

The quality of included studies can be found in Tables 4 and 5   . The methodological quality of the existing studies ranged from poor to excellent, with good and excellent collectively taking 21%.

TABLE 4

Results of Quality of Design, Methods, and Reporting for Included Studies on Performance-Based Measures

TABLE 4 (Continued)

Results of Quality of Design, Methods, and Reporting for Included Studies on Performance-Based Measures

Studies on performance-based measures

There were 89 studies that analyzed the measurement properties of performance-based measures. Reliability and hypothesis testing were the most reported measurement properties of this type of instrument (reported in 52 studies and 35 studies, respectively). Criterion validity and responsiveness were reported in 20 studies and 24 studies, respectively. Unlike the patient-reported measures, performance-based measures had some evidence of measurement error from 7 studies.

Of the studies reporting on reliability, 2 were excellent, 12 were good, 12 were fair, and 26 were poor. Inadequate sample size and no intraclass correlation coefficient (ICC) or Pearson or Spearman correlations calculated were the main reasons contributing to poor quality of the study. For hypothesis testing, 2 studies were good, 16 fair, and 17 poor. The main weakness lies in inadequate sample sizes and a lack of adequate hypotheses. Evaluating the criterion validity, most studies chose the direct observation as the gold criterion. However, 12 studies were considered poor in criterion validity because of inadequate sample sizes. Among the 20 studies reporting responsiveness, 1 study had good quality, 7 studies had fair quality, and 16 studies had poor quality.

Studies on patient-reported measures

Among the 32 studies analyzing the measurement properties of patient-reported measures, internal consistency, reliability, and hypothesis testing were the measurement properties reported most frequently (reported in 21, 20, and 21 studies, respectively), whereas no study reported on the measurement error of the patient-reported measures. Cross-cultural validity and criterion validity were also only reported in 6 studies and 2 studies, respectively. Content validity and cross-cultural validity were more so evaluated in the development of the scales rather than in the final version.

The qualities of the studies analyzing the internal consistency of patients-reported measures were as follows: 1 excellent, 3 good, 4 fair, and 13 poor. Studies were deemed poor mostly because of the fact that unidimensionality was not properly checked. The quality of the studies analyzing the reliability was 1 excellent, 5 good, 9 fair, and 5 poor. Inadequate sample size was the decisive factor of lesser quality. The quality of the studies analyzing the hypothesis testing was 4 good and 17 fair. The quality of most studies stopped at fair because of that they did not formulate any hypotheses in their studies. Studies reporting responsiveness did not have high quality because of inadequate sample sizes or to the fact that inappropriate methods were used. Structural validity was analyzed in 8 studies and the qualities were mostly determined by the sample size.[111,113-115,119,124-126]

Quality of Psychometric Properties for Outcome Measures

A summary of best-evidence synthesis is provided in Table 6 . The summary was driven from the results of study qualities and the quality of psychometric properties for outcome measures (see table, Supplemental Digital Content 3, which describes quality of psychometric properties for outcome measures) using the criteria displayed in Table 3.

TABLE 4 (Continued)

Results of Quality of Design, Methods, and Reporting for Included Studies on Performance-Based Measures

Patient-reported measures performed better than performance-based measures. All positive evidence of patient-reported measures were evenly distributed in both reliability and validity. Most of the positive evidence of performance-based measures was confined to exercise testing and reliability measurement property.

Best rated instruments with a “+++” in one measurement property or “+”/“++” in at least three measurement properties among performance-based measures are 6-minute walk test (6MWT), endurance treadmill test, and usual 4-meter gait speed (usual 4MGS). The best rated patients-reported measures were functional performance inventory (FPI), functional performance inventory short form (FPI-SF), living with COPD questionnaire (LCOPD), COPD activity rating scale (CARS), University of Cincinnati dyspnea questionnaire (UCDQ), shortness of breath with daily activities (SOBDA), and short-form pulmonary functional status scale (PFSS-11).

DISCUSSION

The present review provides the first evidence on comparing all candidate instruments measuring functional status in COPD patients according to the COSMIN criteria. It highlighted some areas worthy of future researched, including the lack of adequate positive evidence on measurement properties of performance-based measures compared with patient-reported measures, the weakness limiting the quality of the existing studies, and the important measurement properties neglected by existing studies. Although none of the instruments was tested for all measurement properties, the existing evidence still confirms that some instruments performed better in terms of some measurement properties or some survey types. For clinical practice, this review recommends 10 out or 57 instruments assessing functional status of COPD patients. More importantly, it demonstrates how to choose suitable measures according to both the studies on elevating these measures and the requirements of clinical practice.

Comparing Performance-Based Measures with Patient-Reported Measures

According to the summary of best-evidence synthesis, performance-based measures did not have as much positive evidence on measurement property in comparison to patient-reported measures. The lack of adequate positive evidence contradicts their present importance in measuring functional status in COPD patients. Performance-based measures objectively measure what patients actually do by assessing indicators like timing, counting, and distance.[137] It was believed to be more likely to fully characterize a change in functional status than patient-reported measures alone.[138] Some of these performance-based measures have been widely approved and used for many years to evaluate treatment effect, to assess health status, and to explore etiology. For example, 6MWT is a widely used walking test in clinical practice, and it was often used as a standard for other instruments.[139-142] However, its positive evidence confined to reliability, which is also a common situation in all performance-based measures. Activity monitors capture the patients’ activities of daily living. They are an essential supplement to laboratory tests. Although there are numerous studies (27 studies), the qualities of these studies were poor (16 poor studies), leading to a weaker positive rating in evidence synthesis. More good quality studies need to be conducted in order to assess the measurement properties of these performance-based measures.

Weakness Limiting the Quality and Neglected Measurement Properties in the Existing Studies on Performance-Based Measures and Patient-Reported Measures

The methodological qualities of the studies included in this review ranged from poor to excellent. Good and excellent quality studies only took up ∼20% of all studies. In terms of performance-based measures, inadequate sample size was one major drawback, probably because performance-based measurements are more difficult to conduct. Some studies on performance-based measures had a sample size <10.[37,47,66,71,90,105] The sample size should be enlarged in future similar studies according to COSMIN criterion, which is ≥100 for excellent, 50–99 for good, 30–49 for fair, and <30 for poor. However, one thing to note is that the COSMIN checklist was originally developed to assess studies focusing on patient-reported measures. Considering the differences in instrument characteristics and study designs between studies on performance-based measures and studies on patient-reported measures, the sample size criteria may need some adjustment. Methodology on performance-based measures should be discussed in the future. Another obvious drawback affecting studies on performance-based measures was that methods did not meet the COSMIN criteria. For example, some studies measuring reliability tend to not calculate the ICC or Pearson or Spearman correlations, no correlation was calculated with other comparative instruments, whereas only P values were used when testing responsiveness, and no adequate hypotheses were formulated a priori. Qualities of studies on patient-reported measures were better than studies on performance-based measures (Table 5 ). However, checking for unidimensionality, enlarging sample sizes, and formulating hypotheses a priori may further improve all study quality.

According to the results, the included studies and positive evidence were confined to several measurement properties. Some important measurement properties, including content validity and responsiveness, were neglected or poorly reported. Content validity examines the extent to which the concepts of interest are comprehensively represented by the items of the questionnaire,[24,143] so it is especially important for studies on patient-reported measures. To measure content validity, a clear concept model is to be developed.[144] However, present PROs that aim to measure physical activity in chronic respiratory disease patients or similar populations (chronic heart disease patients or the elderly) are rarely based on a conceptual framework.[145] Additionally, a standard method to assess content validity should be applied. According to COSMIN, an appropriate method is to have experts and the target population to assess the relevance and comprehensiveness of the instrument (s) based on criteria set by COSMIN. The two studies on content validity measurement were determined to be poor because they did not meet the above-mentioned criterion. Responsiveness is another key issue for future studies on both performance-based measures and patient-reported measures. An important role of functional status measurement is the evaluation of the effect of rehabilitation or treatment. Therefore, it is important for measurement instruments to respond to change. In the present studies measuring responsiveness, the rating of poor was given because of inadequate sample size. Also, most fair studies used P values instead of showing correlation with comparative instruments or with AUC values. Further studies exploring the responsiveness of functional status instruments should be conducted by applying appropriate methods.

Choosing Measures According to the Present Evidence

Valid and systematical measures of COPD patients’ health status are the base of the accurate quantification of the therapy effects. Facing an impressive and increasing number of measures assessing functional status of COPD patients, clinicians might be confused and feel difficult to find one measure satisfying all of their demands. Another source of confusion is the inconsistent conclusions of reports which employed various measures to evaluate the effect of therapy. It is difficult for clinicians to choose best care for patients by comparing and combining results of these clinical trials.

According to the results, none of measures has been tested for all measurement properties. However, the existing evidence demonstrates that some instruments perform better: 6MWT, endurance treadmill test, and usual 4MGS; and FPI, FPI-SF, LCOPD, CARS, UCDQ, SOBDA, and PFSS-11. These instruments should be preferred in future studies and clinical practice (Table 7). 6MWT was proven to predict the survival in COPD patients well.[146-148] Usual 4MGS needs much shorter course than 6MWT, making it useful for frail patients and applicable in most healthcare settings (including home). MCID was reported to be 0.11 m/s.[77] It is worth to be considered as an instrument for health management of COPD patients. Endurance treadmill test can clearer reflect the physiological limitations.[67] FPI, FPI-SF, LCOPD, CARS, UCDQ, SOBDA, and PFSS-11 are different in terms of their measurement focus and the length of scales. In other words, each measure has its own advantage and most suitable domain. Thus, researchers and clinicians should employ those measures whose measurement properties alignment to their purposes. For example, if the instruments were to be used to measure the therapeutic effect of pulmonary rehabilitation or a respiratory medicine, then the responsiveness of the measurement instruments should be preferred. Finally, functional status measurement of COPD patients is complex, as it contains multidimensional constructs. Different types of instruments have their own strength. It was suggested that both types of measures—performance-based measures and patient-reported measures—are complementary rather than competing when assessing functional status of COPD patients.[149] Finding an optimal combination of measures from both types is worth for further research.

TABLE 4 (Continued)

Results of Quality of Design, Methods, and Reporting for Included Studies on Performance-Based Measures

LIMITATIONS

This study has several limitations. First, some comprehensive HRQOL instruments, including dimensions measuring functional status were excluded in our study. The reason is that their reliability and validity were calculated for the whole instrument rather than the dimension of interest, functional status. Second, some studies focusing on evaluating the therapy effect were excluded because they failed to provide enough information on measurement properties. Admittedly, reviewing measurement properties for a certain type of therapy is interesting and valuable for clinical practice, which should be implemented in the future.

In summary, further research is needed to evaluate the measurement properties of performance-based measures because there is a lack of available information and present studies lack in quality. Content validity and responsiveness should be fully assessed in all instruments, and sample size needs to be enlarged. As for choosing measurement tools for functional status in COPD patients, we recommend FPI, FPI-SF, LCOPD, CARS, UCDQ, SOBDA, PFSS-11, 6MWT, endurance treadmill test, and usual 4MGS. These instruments are different in their measurement content or administer requirement, which may tailor to different usage in clinical practice. We also recommend selecting instruments that perform well in certain measurement properties required for certain assessment purposes and combining instruments from both measurement types.

TABLE 5

Results of Quality of Design, Methods, and Reporting for Included Studies on Patient-Reported Measures

TABLE 5 (Continued)

Results of Quality of Design, Methods, and Reporting for Included Studies on Patient-Reported Measures

TABLE 6

A Summary of Best-Evidence Synthesis

TABLE 6 (Continued)

A Summary of Best-Evidence Synthesis

TABLE 7

Characteristics of Recommended Instruments