Development and Validation of the Modified Patient-Reported Outcome Scale for Chronic Obstructive Pulmonary Disease (mCOPD-PRO).

Purpose: The present study aimed to develop and validate the modified patient-reported outcome scale for chronic obstructive pulmonary disease (mCOPD-PRO) for measuring the health status in COPD using both classical test theory and item response theory.
Methods: A working group was initially established. The conceptual framework of COPD-PRO was modified. Subsequently, items related to COPD were gathered and selected through expert consultation, patient cognitive interviewing, classical test theory methods, as well as the item response theory method. Finally, the formed mCOPD-PRO was evaluated in terms of reliability, content validity, construct validity, criterion validity, known groups validity, and feasibility.
Results: A total of 155 items were gathered in the item bank, and two rounds of expert consultation, interviews with patients and field survey were conducted. The mCOPD-PRO included 27 items in the physiological, psychological, and environmental domains. The Cronbach's alpha of the instrument was 0.954. The correlation coefficients between the scores of each item and its domain scores ranged from 0.429 to 0.902. Confirmatory factor analysis showed that the comparative fit index, incremental fit index, non-normed fit index, standardized root-mean-square residual, and root-mean-square error of approximate were 0.91, 0.91, 0.90, 0.11, and 0.16, respectively. The correlation coefficient between mCOPD-PRO total scores and COPD assessment test scores and the modified Medical Research Council dyspnea scale scores was 0.771 and 0.651, respectively. The differences in mCOPD-PRO total scores and domain scores between the mild/moderate group and severe/extremely severe group of patients with COPD were both statistically significant (P<0.01). The acceptance and completion rates of mCOPD-PRO were both 99.5%, and the median completion time was 5 min (IQR, 4-11 min).
Conclusion: The 27-item mCOPD-PRO is well developed and has good reliability, validity, and feasibility. It may provide a scientific and effective instrument for the clinical evaluation of COPD.

Introduction

Chronic obstructive pulmonary disease (COPD) is a leading cause of morbidity and mortality worldwide with a substantial and increasing economic and social burden.1 Health status impairment is common in patients with COPD. A patient-reported outcome (PRO) is any report of the status of a patient’s health condition derived directly from the patient.2 In recent years, use of a PRO instrument for COPD has been increasingly recognized as a valuable measure for evaluating the effect of a medical intervention on the quality of life (QOL) of patients with COPD in clinical trials.3,4

Previously, our team developed and validated a PRO scale for COPD (COPD-PRO) consisting of amelioration of clinical symptoms, satisfaction of health condition, and satisfaction of treatment effect domains based on the classical test theory (CTT).5 The 17-item COPD-PRO has demonstrated good reliability, validity, and responsiveness, and has been used in clinical trials of COPD.5,6 It is well established that CTT is the most widely used measurement theory for the development and verification of instruments. However, with a growing number of measurement practices, the limitations of CTT are increasingly apparent.7 These include, but are not limited to the following features: (i) the accuracy of reliability estimation is not high; (ii) the error index is generally single and imprecise; (iii) the estimation of various parameters depends excessively on samples; and (iv) the matching of parameters is poor.7

However, item response theory (IRT), a commonly used modern measure theory for item analysis and selection, can overcome these shortcomings and offers the following advantages over CTT: (i) it connects the ability level of subjects with their behavior on items, and parameterizes and models this ability; (ii) the estimation of item parameters is independent of the sample; (iii) the difficulty parameter of items and the ability level of subjects are defined on the same scale; and (iv) the error of measurement of each subject can be estimated through the ability level measured by the model.7 Nevertheless, this theory has its own shortcomings.8 For example, it requires large sample sizes for model fitting. Considering that these two theories are applied based on different assumptions and statistical approaches, and both are characterized by shortcomings, it is necessary to combine the two theories to achieve complementary advantages.

Therefore, the present study aimed to use both CTT and IRT for the development and validation of the modified COPD-PRO (mCOPD-PRO) based on previous study by reference to the guidance for PRO measures. This approach may provide a scientific and effective PRO instrument for the clinical evaluation of COPD.

Materials and Methods

Establishing the Working Group

A working group (composed of researchers, clinicians, and postgraduate students) responsible for the conceptual framework, items, and data was initially established for the revision of COPD-PRO. In addition, an expert committee composed of experts from various fields, such as respiratory diseases, QOL, and health statistics, was established to provide guidance for this study. The working group and expert committee were independent of each other.

Predefining the Basic Characteristics

Basic concepts and terminology, such as QOL, PRO, domain, item, scale, questionnaire, CTT, IRT, and cognitive interviewing, were predefined according to the guidance.2

The basic characteristics involving intended population, instrument type, purpose, number and order of items, survey time, response options, scoring, recall period, and administration mode were represented as follows: (1) the intended population of the instrument was patients with COPD; (2) the instrument type was a disease-specific scale; (3) the purpose of the instrument was to measure the health status in COPD; (4) the number of items of the instrument was determined by qualitative and quantitative analysis; (5) the order of items of the instrument was sorted by domains; (6) the completion time of the instrument was ≤20 min; (7) a 5-point Likert scale was utilized for the response options of the instrument; (8) each item of the instrument was equally weighted; the total scores were calculated by adding the scores of all items of the instrument and dividing the sum by the number of items of the instrument, with lower scores indicating better health status; the domain scores were calculated by adding the scores of all items of the domain and dividing the sum by the number of items of the domain with lower scores indicating better health status; (9) the recall period of the instrument was “during the past 2 weeks”; and (10) the administration mode of the instrument was self-administration.

Modifying the Conceptual Framework

The conceptual framework of COPD-PRO was modified through literature research, expert consultation, panel discussion and interviews with patients.

Generating and Screening Items

Items related to COPD were gathered from the following sources: (1) published literature on QOL in COPD; (2) international, national, and industry guidelines for COPD; (3) medical records of outpatients and inpatients with COPD in the First Affiliated Hospital of Henan University of Chinese Medicine; and (4) interviews with patients.

The collected items were initially reviewed by expert consultation and interviews with patients in terms of importance, wording, response options of items, structure of instrument, necessity to add items, etc. The items were mainly determined by the importance score of experts with other supplementary comments. Subsequently, CTT and IRT methods were both used to conduct item selection based on the survey. Items meeting one of the following criteria were excluded: (1) the item was recommended for deletion through IRT analysis; or (2) the item was recommended simultaneously for deletion by >2 types of CTT methods. On this basis, the mCOPD-PRO was formed.

Validating the mCOPD-PRO

Another survey was conducted in six Grade III hospitals in China to evaluate the mCOPD-PRO. The hospitals were the First Affiliated Hospital of Henan University of Chinese Medicine (Zhengzhou, China), the Second Affiliated Hospital of Henan University of Chinese Medicine (Zhengzhou, China), the Third Affiliated Hospital of Henan University of Chinese Medicine (Zhengzhou, China), the Second Affiliated Hospital of Liaoning University of Traditional Chinese Medicine (Shenyang, China), Shaanxi Traditional Chinese Medicine Hospital (Xi’an, China), and Hebei Provincial Hospital of Traditional Chinese Medicine (Shijiazhuang, China). Patients with COPD completed the mCOPD-PRO. The COPD was diagnosed according to the Global strategy for the diagnosis, management, and prevention of COPD (updated 2019),1 and Guidelines for the diagnosis and treatment of COPD (revised 2013).9 The sample size was estimated using the experience and method described elsewhere.10 Written informed consent was provided by all participants.

The inclusion criteria for participants were: (1) a confirmed diagnosis of COPD; (2) age ≥18 years; (3) ability to communicate and read in Mandarin Chinese; and (4) willingness to provide written informed consent. The exclusion criteria for participants were: (1) the disease status or mental state of the participants affected their self-reporting; or (2) there was cognitive or other impairment that affected the self-reporting of participants.

The measurement properties of mCOPD-PRO assessed in the current study referred to internal consistency reliability, content validity, construct validity, criterion validity, known groups validity, and feasibility.

Statistical Analysis

Continuous data were expressed as the mean ± standard deviation (SD) or median (interquartile range), while categorical data were presented as frequencies (percentages). Confirmatory factor analysis (CFA) was conducted using LISREL version 8.70 (Scientific Software International, Inc., Chicago, IL, USA), while IRT analysis was performed using MULTILOG version 7.03 (Scientific Software International, Inc., Chicago, IL, USA). Other data analysis was conducted using SPSS version 22.0 (IBM Corporation, Armonk, NY, USA). The data of mCOPD-PRO were considered invalid if 20% of the item scores were missing. When the data of individual items were missing, the scores of this item were replaced by the mean.

The CTT method used to conduct item selection included the discrete trend, correlation coefficient, Cronbach’s alpha, and factor analysis methods. The discrete trend method used the SD of item scores to measure the discrete degree, and the item was retained if the SD was ≥0.85. The correlation coefficient between item scores and its domain scores should be greater than that between item scores and other domain scores; meanwhile, the correlation coefficient between item scores and its domain scores, as well as that between item scores and scale scores should be ≥0.50. Cronbach’s alpha method was adopted by analyzing whether the Cronbach’s alpha of the domain or the instrument was increased after deleting the current item. Exploratory factor analysis was employed to remove items with low factor loadings (<0.40) on given factors based on the modified conceptual framework. Regarding the IRT method, unidimensionality is a crucial assumption for IRT models. The graded response model for polytomous response data was employed to analyze the discrimination parameter (a), difficulty parameter (b), and information of the items if the unidimensionality was considered sufficient. A good item met the following criteria: (1) the discrimination parameter (a) should range from 0.3 to 3.0; (2) the difficulty parameter (b) should vary from −4.0 to 4.0 and increase monotonically; and (3) the mean information should be ≥16/n, where n represented the number of items.

Cronbach’s alpha was used to assess the internal consistency reliability of mCOPD-PRO and its domains. A Cronbach’s alpha ranging from 0.70 to 0.95 indicated good internal consistency.11 The content validity was evaluated by both qualitative and quantitative analyses. For the quantitative analysis, a correlation coefficient of ≥0.40 between item scores and its domain scores was acceptable.12 CFA was adopted to estimate the construct validity. Ideally, the fit indices met the following criteria: (1) the comparative fit index and non-normed fit index were close to 0.90; (2) the incremental fit index was close to 0.95; and (3) the standardized root-mean-square residual and root-mean-square error of approximate (RMSEA) were close to 0.08.13,14 The criterion validity was assessed by the correlation coefficient between mCOPD-PRO total scores and COPD assessment test (CAT) scores and the modified Medical Research Council dyspnea scale (mMRC) scores.15,16 The correlation coefficient of ≥0.40 was considered acceptable.12 The known groups validity was evaluated by comparing the differences of mCOPD-PRO total scores and domain scores between the mild/moderate group and severe/extremely severe group of patients with COPD. In addition, the acceptance rate, completion rate, and average completion time of mCOPD-PRO were employed to evaluate the feasibility. The acceptance rate and completion rate should be ≥85%, and the average completion time should be within 20 min.12

Results

Modification of the Conceptual Framework

The modified conceptual framework of COPD-PRO consisted of the physiological, psychological, environmental, social, and satisfaction domains, with eight, two, five, five and two facets, respectively ().

Item Generation and Selection

A total of 155 items were gathered in the item bank. There were 113, nine, 13, 16, and four items in the physiological, psychological, environmental, social, and satisfaction domains, respectively.

Subsequently, two rounds of expert consultation and interviews with patients were performed. In the first round, 20 experts were asked for advice, and 18 of them provided comments; meanwhile, interviews of 10 patients with COPD were conducted. After removing, merging, or rewording part of the items, 43, seven, 19, four, and four items were retained in the physiological, psychological, environmental, social, and satisfaction domains, respectively. In the second round, 18 experts were asked for advice, and 15 of them provided comments; meanwhile, interviews of 30 patients with COPD were conducted. After removing, merging, or rewording part of the items, 40, eight, five, three, and four items were retained in the physiological, psychological, environmental, social, and satisfaction domains, respectively.

Subsequently, two rounds of surveys were performed. In the first round, data from 364 patients with COPD were analyzed. The mean age of these patients was 66 years; 287 were males (78.8%) and 77 were females (21.2%). The number of items recommended for deletion by discrete trend, correlation coefficient, Cronbach’s alpha, factor analysis, and IRT methods were seven, 16, three, three, and 19, respectively. Thus, 19 items were recommended for deletion by either IRT analysis or >2 CTT methods. Given the administrator burden of the remaining 41 items, another round of survey was performed to further reduce the items. In the second round, data from 264 patients with COPD were analyzed. The mean age of these patients was 68 years; 197 were males (74.6%) and 67 were females (25.4%). In the current survey, the criteria of item selection were raised based on the first round of survey except Cronbach’s alpha method. For the discrete trend method, the item was retained if the SD was ≥0.90. The correlation coefficient between item scores and its domain scores as well as that between item scores and scale scores should be ≥0.60; other conditions remained unaltered versus those of the first round of survey. CFA was conducted in the current survey, where items with factor loadings of <0.60 were removed. The criteria of the difficulty parameter (b) for the IRT method were raised ranging from −3.0 to 3.0. Accordingly, the number of items recommended for deletion by the discrete trend, correlation coefficient, Cronbach’s alpha, factor analysis, and IRT methods were nine, five, one, five and 18, respectively. Finally, the 27-item mCOPD-PRO () composed of the physiological, psychological, and environmental domains was developed after merging two items and removing 13 items based on the CTT and IRT methods combined with panel discussion. The physiological, psychological, and environmental domains included four facets (17 items), two facets (seven items), and three facets (three items), respectively. Ten identical or similar items originated from the 17-item COPD-PRO were retained in different domains of the mCOPD-PRO.

Validation of the mCOPD-PRO

A total of 370 copies of the mCOPD-PRO were issued, and 368 copies were retrieved. Considering the missing data, data from 366 patients with COPD were eventually used for analysis. The mean age of the participants was 66 years; 279 were males and 87 were females.

Reliability

The Cronbach’s alpha of the mCOPD-PRO was 0.954, and that of the physiological, psychological, and environmental domains was 0.930, 0.929, and 0.673, respectively.

Content Validity

The patients with COPD, occasionally with experts, participated in the stages such as modification of the conceptual framework, item generation and selection, ensuring that the mCOPD-PRO had good content validity. The correlation coefficient between item scores and domain scores is described in Table 1. The correlation coefficients between the scores of each item from the physiological domain and its domain scores ranged from 0.429 to 0.775; that between the scores of each item from the psychological domain and its domain scores ranged from 0.670 to 0.902; and that between the scores of each item from the environmental domain and its domain scores ranged from 0.709 to 0.808.

Table 1

The Correlation Coefficient Between Item Scores and Domain Scores of mCOPD-PRO

Item No.	Physiological Domain	Psychological Domain	Environmental Domain
1	0.579	0.419	0.355
2	0.555	0.329	0.368
3	0.725	0.433	0.469
4	0.771	0.442	0.477
5	0.758	0.440	0.467
6	0.709	0.508	0.519
7	0.750	0.532	0.513
8	0.657	0.525	0.486
9	0.429	0.269	0.232
10	0.555	0.337	0.451
11	0.431	0.360	0.287
12	0.534	0.396	0.358
13	0.683	0.516	0.543
14.1	0.705	0.538	0.486
14.2	0.775	0.556	0.516
14.3	0.766	0.507	0.520
14.4	0.773	0.508	0.539
15	0.544	0.732	0.492
16	0.570	0.698	0.526
17	0.582	0.670	0.465
18.1	0.549	0.858	0.554
18.2	0.509	0.902	0.510
18.3	0.504	0.888	0.525
18.4	0.509	0.892	0.544
19	0.607	0.588	0.709
20	0.439	0.366	0.755
21	0.484	0.510	0.808

Notes: Data are expressed as correlation coefficient; all P-values<0.001.

Abbreviation: mCOPD-PRO, modified patient-reported outcome scale for chronic obstructive pulmonary disease.

Construct Validity

The CFA showed that comparative fit index, incremental fit index, non-normed fit index, standardized root-mean-square residual, and RMSEA were 0.91, 0.91, 0.90, 0.11, and 0.16, respectively. The factor loadings of each item ranged from 0.43 to 0.97 (Figure 1).

Figure 1

Confirmatory factor analysis model.

Criterion Validity

The Spearman’s rank correlation coefficient between mCOPD-PRO total scores and CAT scores and mMRC scores was 0.771 and 0.651, respectively; that between domain scores and CAT scores and mMRC scores is represented in Table 2.

Table 2

The Correlation Coefficient Between Domain Scores of mCOPD-PRO and CAT Scores and mMRC Scores

	Physiological Domain	Psychological Domain	Environmental Domain	mCOPD-PRO
CAT	0.792	0.564	0.571	0.771
mMRC	0.676	0.449	0.514	0.651

Notes: Data are expressed as correlation coefficient; all P-values <0.001.

Abbreviations: mCOPD-PRO, modified patient-reported outcome scale for chronic obstructive pulmonary disease; CAT, chronic obstructive pulmonary disease assessment test; mMRC, modified Medical Research Council dyspnea scale.

Known Groups Validity

The differences in mCOPD-PRO total scores and domain scores between the mild/moderate group and severe/extremely severe group of patients with COPD were statistically significant (P<0.01) (Table 3).

Table 3

Comparison in mCOPD-PRO Total Scores and Domain Scores Between the Mild/Moderate Group and Severe/Extremely Severe Group of Patients with COPD

Domain	Mild/Moderate Group	Severe/Extremely Severe Group	t	P-values
Physiological domain	1.48±0.62	1.94±0.76	−6.202	P<0.001
Psychological domain	1.19±0.72	1.44±1.00	−2.739	0.006
Environmental domain	1.59±0.71	1.97±0.87	−4.491	P<0.001
mCOPD-PRO	1.42±0.59	1.81±0.77	−5.398	P<0.001

Notes: Data are expressed as mean±SD; P-values are reported based on independent sample t-test.

Abbreviations: mCOPD-PRO, modified patient-reported outcome scale for chronic obstructive pulmonary disease; COPD, chronic obstructive pulmonary disease.

Feasibility

The acceptance and completion rates of the mCOPD-PRO were both 99.5%, and the median completion time was 5 min (IQR, 4 to 11 min).

Discussion

This study modified the COPD-PRO based on two psychometric theories, namely the CTT and IRT. As a result, the 27-item mCOPD-PRO, composed of the physiological, psychological, and environmental domains, was developed. Moreover, the mCOPD-PRO was validated with a good reliability, validity, and feasibility. Derived from the original version, the mCOPD-PRO performed better in some aspects. It may provide a scientific and effective instrument for the evaluation of COPD in clinical trials.

The conceptual framework in the current study was in line with international concepts. At the start, the modified conceptual framework of the COPD-PRO included the physiological, psychological, environmental, social, and satisfaction domains. Among them, the physiological, psychological, and environmental domains were derived from the amelioration of clinical symptoms domain of the COPD-PRO; the satisfaction domain was formed by merging the satisfaction of health condition domain and satisfaction of treatment effect domain of the COPD-PRO. Apart from the satisfaction domain, the remaining domains were in accord with the domains (physical health, psychological health, social relationships, and environment) shared by the generic, patient-completed measure of health-related QOL, the World Health Organization Quality of Life assessment instrument (WHOQOL-100) and the short version (WHOQOL-BREF).17 In addition, the physiological, psychological, and social domains were similar to the focus of the Patient-Reported Outcomes Measurement Information System. The latter is a set of person-centered measures that evaluate and monitor physical, mental, and social health in adults and children, and can be used with the general population and individuals living with chronic conditions.18 After changing the items, physiological, psychological, and environmental domains were eventually retained in the mCOPD-PRO, where the environmental domain broadly involved natural and social environment factors.

This study combined the CTT and IRT methods to achieve complementary advantages. It is established that item selection plays a vital role in the development of instruments. The Delphi method and interviews with patients were first performed to select items in term of importance, certainty, or suitability. For the CTT methods, discrete trend, correlation coefficient, Cronbach’s alpha, and factor analysis methods were employed from different perspectives, such as sensitivity, representativeness, independence, and internal consistency. In our study, the IRT method was adopted to analyze the discrimination parameter (a), difficulty parameter (b), and information of the items. The comprehensive application of these methods laid a good foundation for the screening of high-quality items.

In addition, the measurement properties of our instrument performed well. The Cronbach’s alpha of the instrument and the three domains indicated a good internal consistency reliability. In this study, the instrument purpose, the intended population, and the concepts measured have been stated above, and patients and experts were involved in multiple stages, for example, item selection. In addition, correlation analysis showed that the correlation coefficients between item scores and its domain scores were all >0.40. Both qualitative and quantitative analyses supported a good content validity. The RMSEA from the CFA was slightly large, and the other fit indices were close to the reference value. Furthermore, the factor loadings of each item were all >0.40. To some degree, the instrument demonstrated good construct validity. The correlation coefficients between the instrument total scores and the CAT scores and the mMRC scores both suggested a good criterion validity of the mCOPD-PRO. The results obtained from the independent sample t-test supported a good known groups validity of the mCOPD-PRO. Considering the high acceptance and completion rate together with the short completion time, use of the mCOPD-PRO is rather feasible.

In comparison with several instruments widely used at home and abroad, our instrument exhibited its own importance for the evaluation of COPD. The single-item mMRC is a simple measure of breathlessness, and the 8-item CAT is a comprehensive measure of symptoms.15,16 The Clinical COPD Questionnaire (CCQ) is a comprehensive disease-specific health status measure including symptom, functional state, and mental state domains.19 The 50-item St. George’s Respiratory Questionnaire (SGRQ) is a most comprehensive disease-specific health status measure, which includes symptom, activity, and impact domains.20 Indeed, our 27-item instrument has more items and higher administration burden than the mMRC, CAT, and CCQ. However, compared with the mMRC and CAT, our instrument is not just a measure of symptoms, but a comprehensive disease-specific health status measure, which includes physiological, psychological, and environmental domains. Compared with the CCQ, our instrument includes the environmental domain, which is an important factor focused on by Chinese individuals, especially those with respiratory diseases, for example, COPD. According to the Global Strategy for the Diagnosis, Management, and Prevention of COPD (updated 2019), the SGRQ is excessively complex for use in routine practice.1 Therefore, our instrument has fewer items and lower administration burden compared with the SGRQ. Moreover, the measurement properties of our instrument performed well, and the administration burden was acceptable with a median completion time of 5 min. In addition, a short version of our instrument may be considered in the future.

This study had some limitations. Firstly, although the known groups validity was analyzed, the ability of the mCOPD-PRO to detect change over time was not evaluated. Secondly, there is no consensus regarding the adoption of the most suitable item selection method and the criteria for each index. Moreover, given that IRT requires a large sample size for model fitting, further studies involving large samples are warranted. Future studies may also determine the minimal clinically important difference of the mCOPD-PRO.

Conclusions

The 27-item mCOPD-PRO, derived from the COPD-PRO and composed of physiological, psychological, and environmental domains, was successfully developed. The mCOPD-PRO demonstrated good internal consistency reliability, content, construct, criterion, and known groups validity, as well as feasibility. It may provide a scientific and effective instrument for the evaluation of COPD in clinical trials.