Reporting of Patient-Reported Outcome Measures in Randomized Controlled Trials on Shoulder Rotator Cuff Injuries Is Suboptimal and Requires Standardization.

Purpose: To evaluate the completeness of patient-reported outcomes (PROs) reporting using Consolidated Standards of Reporting Trials Patient-Reported Outcome (CONSORT-PRO) in randomized controlled trials (RCTs) involving rotator cuff injuries.
Methods: We performed a comprehensive search of MEDLINE, Embase, and Cochrane Central Register of Controlled Trials for published RCTs focused on rotator cuff injuries that included at least one PRO measure. We included RCTs published from 2006 to 2020. Investigators extracted data from RCTs using the CONSORT-PRO and evaluated each RCT using the Cochrane Risk of Bias 2.0 tool. Our primary objective was to evaluate the mean completion percentage of CONSORT-PRO. Our secondary objective used bivariate regression analyses to explore the relationship between trial characteristics and completeness of reporting.
Results: The initial search returned 467 results, with 33 published RCTs meeting the prespecified inclusion criteria. The mean CONSORT-PRO completeness across all included RCTs was 49.7% (standard deviation 15.43). An increase in sample size was associated with an increase in mean completeness of reporting (t = 2.31; P = .028). The Risk of Bias assessment found 29 (of 33, 87.88%) RCTs had "some concerns" for bias. We did not find any additional significant associations between completeness of reporting and trial characteristics. Conclusions: Randomized controlled trials involving rotator cuff injuries frequently use PRO measures as primary outcomes. Reporting of these PRO measures is suboptimal and may benefit from rigorous standardization. Clinical Relevance: PRO measures are increasingly incorporated as primary or secondary outcomes of RCTs. Appropriate reporting and use of state-of-the-art PRO measures may improve the dissemination of clinical knowledge from RCTs to guide treatment and determine intervention effectiveness. With increased adoption of Patient-Reported Outcome Measure Information System and adherence to CONSORT-PRO, orthopaedic literature may improve PRO reporting to optimize the interpretability of PROs and facilitate patient-centered care.

Rotator cuff injuries are the most common type of shoulder disability, leading to approximately 4.5 million physician visits each year. While rotator cuff injuries can be linked to trauma, they also can be considered a part of normal degenerative changes associated with aging., It is estimated that approximately 25% of adults in their 60s and 50% of adults in their 80s have a full thickness rotator cuff tear., Symptoms of rotator cuff injuries often include pain, decreased strength, and range of motion, which can significantly limit the ability to complete activities of daily living. In addition to direct cost to individuals for treatment, the loss of function can lead to the inability to work, loss of income, and depression., Furthermore, injury recurrence rates after surgical repair have been estimated at 57.8%, and several studies have failed to determine a significant difference in efficacy between surgical and nonsurgical management.8, 9, 10 Taken together, the prevalence, symptomatology, and rate of reinjury highlight the need to consider patient perspectives when evaluating management strategies for rotator cuff injuries.

An increased focus on patient-centered care has led to the desire to gather patient perspectives on treatment effects, which can be assessed through patient-reported outcomes (PROs). The Cochrane Collaboration defines PROs as a report directly from the patient pertaining to their treatment, health, and functional status without the interpretation by health care professionals. PROs play a vital role in diseases, such as cancer, and have been deemed useful in identifying a patient’s quality of life as they experience health deterioration, emotional stress, and the economic burden of treatment. In a study assessing physiotherapy for rotator cuff tears, the authors used the Oxford Shoulder Disability Questionnaire and showed that while patients’ pain and overall function improved, their outlook on general health had declined. Owing to the influence that a shoulder injury has on a person’s overall health, it is important that randomized controlled trials (RCTs)—which evaluate the efficacy of treatments—include PROs to assess quality of life following treatment.

The use of PROs in orthopaedics has been increasing in recent years; within shoulder literature, Mosher et al. found that the overall use of PRO measures had increased by 18% from 2007 to 2017. Considering this increase in patient-reported measures, the American Academy of Orthopaedic Surgeons (AAOS) has endorsed the Patient-Reported Outcome Measure Information System (PROMIS) to improve the convenience, validity, and high-quality reporting of PROs. To further enhance the reporting of PROs, the Consolidated Standards of Reporting Trials Patient-Reported Outcome (CONSORT-PRO) provides additional guidance for the promotion of high-quality reporting. The purpose of this study was to evaluate the completeness of PROs reporting using CONSORT-PRO in RCTs involving rotator cuff injuries. We hypothesized that RCTs with PROs as a primary outcome would more completely report PRO measures compared with studies with PROs as a secondary outcome.

Methods

Study Design

We conducted a meta-epidemiologic study of published RCTs regarding the management of rotator cuff injuries. Data were collected by extracting information from published RCTs. This study did not involve humans and, thus, did not meet the regulatory definition for human subjects research per the U.S. Code of Federal Regulations and was not subject to institutional review board oversight. Our study adhered to the reporting guidelines for meta-epidemiological studies.

Search Strategy

One researcher (R.O.), with the consultation of a medical research librarian, used Ovid interface to search MEDLINE, Embase, and Cochrane Central Register of Controlled Trials for published RCTs focused on the management of rotator cuff injuries. To identify RCTs, we used the Cochrane search strategy, a validated filter for OVID interfaces. The search string was uploaded to Open Science Framework.

Eligibility

RCTs that focused on the management of rotator cuff injuries with a PRO as a primary or secondary outcome were included. Studies published between 2006 and 2020 and written in English were included. We excluded the following: clinical trial protocols, clinical efficacy trials that did not have a PRO measure, observational studies, letters to the editor, animal studies, cost-effectiveness studies, secondary analyses, case reports, meta-analyses, systematic reviews, and other reviews.

Selection Process

The returns from our literature search were then combined and uploaded to a systematic review screening platform, Rayyan (https://rayyan.qcri.org/). Duplicates were removed and 2 authors (L.B and B.H.) screened the title and abstract in a masked, duplicate fashion. A third author (S.S.) was available to resolve disagreements but was not needed.

Data-Collection Process

Two authors (B.H. and T.M.) received training on how to appraise reporting completeness by applying CONSORT-PRO used in previous methodology to evaluate CONSORT-PRO adherence., Following training, 3 RCTs not included in our sample were extracted in a masked, duplicate fashion until consensus was achieved in an effort to reduce discrepancies. Authors (L.B. and D.N.) were trained for Risk of Bias (RoB) appraisal using Cochrane training modules. Both CONSORT-PRO and RoB were performed in a masked, duplicate fashion using pilot-tested Google forms. Upon completion of data extraction and the RoB assessment, the authors resolved all differences; a third author (S.S.) was available to resolve disagreements, but was not needed.

Data Items

We reported our primary objective as the mean percent completion (see scoring of CONSORT-PRO). Our secondary objective assessed relationships between the mean completeness of PRO reporting and trial characteristics. The characteristics analyzed were (1) year of publication (before or after 2014, a year following the publication of CONSORT-PRO); (2) intervention of RCT (e.g., drug or surgical technique); (3) conflict of interest statement; (4) journal endorsement of CONSORT-PRO; (5) citation of the CONSORT guideline within the publication; (6) whether an RCT used a PRO as a primary or secondary outcome; (7) RoB assessed by the Cochrane RoB 2.0 tool (see RoB assessment); (8) the length of PRO follow-up time; (9) sample size of the trial; and (10) whether the trial cited the use of PROMIS measures.

The publishing journal’s endorsement of CONSORT guidelines were coded as follows: not mentioned, recommended, or required. This data item was evaluated by screening the instructions to authors' pages for mention of EQUATOR, CONSORT, or CONSORT-PRO guidelines.

The following bias domains were evaluated: (1) bias arising from the randomization process, (2) bias due to deviations from intended interventions, (3) bias due to missing endpoint data, (4) bias in measurement of the endpoint, (5) bias in selection of the reported result, and (6) overall RoB assessment.

Scoring CONSORT-PRO

Our scoring framework follows published methodology from Mercieca-Bebber et al. 2017. Of note, items not part of the PRO extension or elaborations were not scored to better evaluate PRO-related reporting. Item 4a of CONSORT-PRO (the use of PROs in eligibility or stratification) was removed from scoring in our study as there is difficulty in the verification of this criteria. Instead, we recorded whether a study describes adherence to this item as a “yes” or “no.” We allocated a maximum value of 0.5 or 1 when information for an item was present. Items that received the maximum value (1 or 0.5 if the item is double-barreled) were considered “complete,” while items that failed to reach maximum value are reported as “not complete.” Item P1b was scored partially complete if an RCT reported the PRO measure used in the study, but failed to identify whether the PRO was a primary or secondary outcome. Therefore, Item P1b could be scored as 0, 0.5, or 1 based on the information available. Item 7a was dependent on the PRO measure being reported as a primary outcome. Because of this dependency, RCTs with a primary PRO had a maximum score of 15 versus RCTs with a secondary outcome had a maximum score of 14. A percent completeness of the checklist per RCT was calculated by adding the items and dividing by the total of possible items.

RoB Assessment

We used a decision algorithm developed by the Cochrane Collaboration to score RoB. If investigators had partially divergent assessments on bias domains (e.g., one investigator answered “yes” and another investigator answered “partial yes”), the overall RoB judgment for the trial outcome was not altered. The overall risk of bias domain was determined per the Excel tool provided by Cochrane as “high,” “some concerns,” or “low’ risk.”26, 27, 28

Data Analysis

First, we report our search return followed by the frequencies and percentages of trial characteristics among the RCTs in our sample. Additionally, we report the frequency of RCTs that mention PROMIS measures. We then report the mean completion percentage of CONSORT-PRO across all RCTs in our sample and by outcome category (i.e., RCTs with PROs as either primary or secondary outcomes). Next, we reported on the frequency and percentage of individual items on CONSORT-PRO for all RCTs and by outcome category. Finally, to address our secondary outcome, we used bivariate regression models to determine the association between mean completion percentage of CONSORT-PRO and the trial characteristics listed in Data Items. All analyses were performed using Stata 16.1 (StataCorp, LLC, College Station, TX).

Reproducibility

The study protocol, data sheets, analysis scripts, a data dictionary, and extraction forms were uploaded on Open Science Framework, to ensure the transparency and reproducibility of our study. This investigation was conducted in tandem with other studies addressing completeness of reporting in other fields of medicine using similar methodology.

Results

General Characteristics

Our literature search returned 467 records. After deduplication, 312 records remained and then were screened by title and abstract. After title and abstract screening, 86 RCTs were included for full-text screening and 33 published RCTs were included in our final sample. Rationales for exclusion can be found in Figure 1.

Fig 1

Rationale for exclusion.

The majority of RCTs were published after 2014 (24/33, 72.73%). The most common of the interventions studied were therapies as defined by our data dictionary (11/33, 33.33%). Fifteen (of 33, 45.45%) RCTs were published in journals that required CONSORT reporting. Among RCTs that included a conflict-of-interest statement, 18 (of 30, 60%) reported no conflicts. Thirty-one (of 33, 93.94%) RCTs did not mention the CONSORT guideline in the manuscript. When evaluating RCTs for risk of bias, we found that 1 RCT was rated as “high” (of 33, 3.03%), 29 RCTs were rated as “some concerns” (of 33, 87.88%), and 3 were rated as “low” (of 33, 9.09%). Twelve (of 33, 36.36%) RCTs had a follow-up time of 3 months or less. Of the 33 RCTs in our sample, no RCTs reported using PROMIS. Additional characteristics of RCTs are found in Table 1.

Table 1

Characteristics of Randomized Controlled Trails and Bivariate Associations With CONSORT-PRO Completion

Characteristic	Total 33 (100)	Coef. (SE)	t	P
Year of publication, No. (%)
<2014	9 (27.27)	1 (Ref)	–	–
≥2014	24 (72.73)	–3.97, (6.08)	–0.65	.519
Intervention of RCT, No. (%)
Combination	4 (12.12)	1 (Ref)	–	–
Device	1 (3.03)	–10.42, (17.38)	–0.6	.554
Drug	6 (18.18)	–18.47, (10.03)	–1.84	.077
Other	2 (6.06)	–13.75, (13.46)	–1.02	.316
Surgical	1 (3.03)	–33.75, (17.38)	–1.94	.063
Surgical technique	8 (24.24)	–15.83, (9.52)	–1.66	.108
Therapy	11 (33.33)	–14.03, (9.08)	–1.55	.134
Includes COI statement, No. (%)
No statement	3 (9.09)	1 (Ref)	–	–
Reports COI	12 (36.36)	–1.09, (10.14)	–0.11	.915
Reports No COI	18 (54.55)	–5.93, (9.8)	–0.6	.55
Journal requirement of reporting guidelines, No. (%)
Not mentioned	8 (24.24)	1 (Ref)	–	–
Recommended	10 (30.3)	0.14, (7.53)	0.02	.986
Required	15 (45.45)	2.86, (6.95)	0.41	.684
Mention of CONSORT or CONSORT-PRO within RCT, No. (%)
No	31 (93.94)	1 (Ref)	–	–
Yes	2 (6.06)	9.19, (11.32)	0.81	.423
PRO as a primary or secondary outcome, No. (%)
Primary	30 (90.91)	1 (Ref)	–	–
Secondary	3 (9.09)	1.63, (9.49)	0.17	.864
Overall ROB, No. (%)
High	1 (3.03)	1 (Ref)	–	–
Some concern	29 (87.88)	–11.14, (16)	–0.7	.492
Low	3 (9.09)	–5.56, (18.16)	–0.31	.762
Length of PRO follow-up
3 months or less	12 (36.36)	1 (Ref)	–	–
3+ to 6 months	2 (6.06)	–21.94, (11.41)	–1.92	.064
6+ months to 1 year	8 (24.24)	4.31, (6.82)	0.63	.533
1 years +	11 (33.33)	1.64, (6.24)	0.26	.795
Sample size
Mean (SD)	85.27 (56.98)	0.1, (0.04)	2.3	.028
Reported PROMIS measures
Not reported	33 (100.0)	–	–	–
Reported	0 (0.0)	–	–	–

COI, conflict of interest; CONSORT, Consolidated Standards of Reporting Trials; CONSORT-PROs; PROMIS, Patient-Reported Outcomes Measurement Information System; RCT, randomized controlled trial; SD, standard deviation.

A P-value less than or equal to 0.05 was considered statistically significant indicated in bold.

Completeness of Reporting According to CONSORT-PRO

The mean CONSORT-PRO complete percentage of included RCTs was 49.70% (standard deviation [SD] 15.42). RCTs with a PRO as a primary outcome had a mean completion percentage of 49.55% (SD 14.79) and RCTs with a PRO as a secondary outcome had a mean completion percentage of 51.19% (SD 25.08).

Throughout all the RCTs in our sample, we found that item 17aii—results include an estimate of precision—was the most completely reported item (31/33, 93.94%). Additionally, no RCTs completely reported Item 2a—rationale for PRO endpoint. Regarding RCTs with a primary PRO, the most completely reported items included: PRO identified as RCT endpoint in abstract (P1b), number of questionnaires at baseline (13ai), and results include an estimate of precision (17aii) (28/33, 93.33%). Item 2a—rationale for PRO endpoint—was not reported by any RCT with a primary PRO. All RCTs with a secondary PRO (3/3, 100%) completely reported the following items: number of questionnaires at baseline and at time of analysis (13ai and 13aii), results include an estimate of precision (17aii), and PRO study limitations (P20). No RCTs with a secondary PRO completely reported the following items: rationale for PRO endpoint (2a), PRO domains specified in the hypothesis (P2bi and P2bii), and completion of PRO questionnaire (P6aiii). We did not assess Item 7a—sample size determination—for RCTs with secondary PRO. Additional information related to completion of CONSORT PRO may be found in Table 2.

Table 2

Completion of CONSORT-PRO by Primary and Secondary Objective Designation

	Primary 30 (90.91)		Secondary Endpoint 3 (9.09)		Total 33 (100)
	Complete	Not Complete	Complete	Not Complete	Complete	Not Complete
CONSORT-PRO item	n (%)	n (%)	n (%)	n (%)	n (%)	n (%)
Introduction
P1b. Abstract—PRO as primary/secondary endpoint	28 (93.33)	2 (6.67)	2 (66.67)	1 (33.33)	30 (90.91)	3 (9.09)
2a. Rationale for including PRO endpoint	0 (0)	30 (100)	0 (0)	3 (100)	0 (0)	33 (100)
P2bi. PRO hypothesis present	6 (20)	24 (80)	0 (0)	3 (100)	6 (18.18)	27 (81.82)
P2bii. PRO domains in hypothesis	1 (3.33)	29 (96.67)	0 (0)	3 (100)	1 (3.03)	32 (96.97)
Methods
P6ai. Evidence of PRO instrument validity	26 (86.67)	4 (13.33)	2 (66.67)	1 (33.33)	28 (84.85)	5 (15.15)
P6aii. Statement of the person completing the questionnaire	15 (50)	15 (50)	2 (66.67)	1 (33.33)	17 (51.52)	16 (48.48)
P6aiii. Mode of administration (paper, e-PRO)	3 (10)	27 (90)	0 (0)	3 (100)	3 (9.09)	30 (90.91)
P7a. How sample size was determined (not required unless PRO is a primary endpoint)∗	25 (83.33)	5 (16.67)	–	–	25 (83.33)	5 (16.67)
P12a. Statistical approach for dealing with missing data (imputation, exclusion, other)	6 (20)	24 (80)	2 (66.67)	1 (33.33)	8 (24.24)	25 (75.76)
Results
13ai. Report no. questionnaires submitted/available for analysis at baseline	28 (93.33)	2 (6.67)	3 (100)	0 (0)	31 (93.94)	2 (6.06)
13aii. Report no. questionnaires submitted/available for analysis principal time point for analysis	27 (90)	3 (10)	3 (100)	0 (0)	30 (90.91)	3 (9.09)
15. Demographics table includes baseline PRO	12 (40)	18 (60)	1 (33.33)	2 (66.67)	13 (39.39)	20 (60.61)
16. Number of pts (denominator) included in each PRO analysis	14 (46.67)	16 (53.33)	2 (66.67)	1 (33.33)	16 (48.48)	17 (51.52)
17ai. PRO results reported for the hypothesized domains and time point specified in the hypothesis—OR—reported for each domain of the PRO questionnaire if no PRO hypothesis provided	6 (20)	24 (80)	2 (66.67)	1 (33.33)	8 (24.24)	25 (75.76)
17aii. Results include confidence interval, effect size or some other estimate of precision	28 (93.33)	2 (6.67)	3 (100)	0 (0)	31 (93.94)	2 (6.06)
18. Results of any subgroup/adjusted/exploratory analyses	10 (33.33)	20 (66.67)	1 (33.33)	2 (66.67)	11 (33.33)	22 (66.67)
Discussion
P20. PRO study limitations	27 (90)	3 (10)	3 (100)	0 (0)	30 (90.91)	3 (9.09)
P21. Implications of PRO results for generalizability, clinical practice	6 (20)	24 (80)	1 (33.33)	2 (66.67)	7 (21.21)	26 (78.79)
22. PROs interpreted in relation to clinical outcomes	12 (40)	18 (60)	1 (33.33)	2 (66.67)	13 (39.39)	20 (60.61)

CONSORT-PRO, Consolidated Standards of Reporting Trials Patient-Reported Outcomes.

Item P7a only applies to PROs identified as primary outcomes.

Associations Between Trial Characteristics and Completeness of Reporting

Our bivariate regression analyses revealed that an increase in trial sample size was significantly associated with increased completeness of reporting (t1 = 2.3; P = .028). We found no other significant associations between trial characteristics and completeness of reporting. Relationships are demonstrated in Table 1.

Discussion

Our study identified that PRO reporting was incomplete within RCTs focused on rotator cuff injuries, with an overall mean completeness of less than 50%. Studies with PROs as secondary outcomes were found to be associated with a greater level of completeness; this finding is ambiguous due to the contrast in number of studies with PROs as primary outcomes versus secondary. One commonly underreported item in CONSORT-PRO checklist was the description of statistical methods for handling missing data. Conversely, a well reported item within RCTs was the documentation of instrument validity. Additionally, our analysis revealed that none of the included RCTs mentioned the use of PROMIS. In the following discussion, we will elaborate on our findings within the context of existing literature, address the implications of the aforementioned checklist items, and recommend the implementation of PROMIS to improve PRO reporting within orthopaedics.

In accordance with our findings, studies have found that CONSORT-PRO completeness was suboptimal., Additionally, a systematic review assessing PRO reporting in multiple myeloma found similar findings with commonly omitted checklist items including the statistical plan for handling missing data (23.0% reported) and implications for generalizability and clinical practice (26.7% reported). In a study looking at PROs in 23 publications related to osteoporosis, the authors found that the mean CONSORT-PRO completeness was approximately two-thirds. Taken together, these findings suggest reporting deficits, which can be addressed through the application of the CONSORT-PRO checklist.

An inconsistently reported item on the CONSORT-PRO checklist was a lack of reporting of an approach to handling missing data. Nearly three-fourths of the RCTs in our sample failed to report this item. This finding is not unique to our study. An investigation of the Journal of the American Medical Association, The New England Journal of Medicine, the Lancet, and The BMJ found that 77% of RCTs published between 2013 and 2014 failed to report on appropriate handling of missing data within their publication. Missing data can introduce bias and decrease the power of the study when researchers fail to report their statistical approach. These deficits in reporting could impair interpretation of the quality of data collection and analysis.

While deficiencies in reporting were common in our sample, information regarding PRO instrument validity was well documented at 84.85% across rotator cuff injury RCTs. This finding contrasts with previous literature, suggesting that medical research may not report necessary information regarding instrument selection. In our previous example by LeBlanc et al., only 50% of studies in their systematic review of multiple myeloma literature cited validation studies for the included PROs. Additionally, a study analyzing PRO quality across head and neck cancer RCTs found that 34% of these RCTs reported evidence of PRO instrument validity. Without the use of a validated PRO instrument, clinicians are unable to assess the relevance of RCTs to patients and whether or not the PROs are appropriate for the trial design.

Despite the strengths in reporting that were found in orthopaedic literature, opportunities to further improve PRO reporting exist. The endorsement of PROMIS by the AAOS offers the opportunity to overcome historical barriers associated with PRO reporting within clinical trials. In a study assessing the use of PROMIS in upper extremity injuries—including rotator cuff injuries—PROMIS physical function computerized adaptive testing was found to be more efficient and have a strong correlation with American Shoulder and Elbow Surgeons score and the Simple Shoulder Test scores. Several elements which make PROMIS superior to legacy systems are: (1) computerized adaptive testing is a form of computer based questioning which is built on item-response theory to adapt to the ability level of the patient; (2) the ability to report baseline functional T-score using a standard set of outcomes across differing subspecialties in orthopaedics; (3) the multidomain capability which PROMIS offers with its physical function, pain, emotional distress, and fatigue domains would normally require multiple scoring systems to be used. Eighteen RCTs in our sample were published after the AAOS endorsement of PROMIS in 2016, and no trials in our sample incorporated PROMIS measures. Our findings are supported by a cross-sectional analysis among orthopaedic trauma literature from 2014 to 2018 looking at 319 RCTs. They found only 7 trials used PROMIS measures. Given these findings, there is still opportunity for the field of orthopaedics to consistently adopt PROMIS and become a leader in reporting patient outcomes.

Recommendations

To increase high-quality reporting in orthopaedics, we agree with Hussain et al. in their recommendation of emphasizing the use of the CONSORT checklist. To further emphasize the importance of RCTs abiding by CONSORT-PRO, Mercieca-Bebber et al. found that greater rates of complete reporting correlated with RCTs that cited CONSORT-PRO in their study. Despite the adoption of PROMIS by the AAOS in 2016, https://paperpile.com/c/WAvgO0/oThM use of this tool is lacking in recent studies. Following the guidance of the AAOS, we recommend the use of PROMIS to increase the quality in which PRO data is captured. Additionally, reporting standardization of PROs will allow for incorporation of convenient, value-based assessments in clinical practice. Finally, further investigation of the relationship between sample size and completeness of reporting is encouraged. Factors influencing the sample size of RCT s are numerous and may have influenced our results. These factors may include funding source, amount of funding, recruitment methods, and study design.

Limitations

This study is not without limitations. First, three RCTs in our sample used a PRO as a secondary outcome. This limited the assessment of our hypothesis. Second, although MEDLINE, Embase, and Cochrane Central Register of Controlled Trials database searches were performed, it is possible that all relevant RCTs were not included. Lastly, the results of our cross-sectional study should not be generalized to other qualifying studies, other journals, or studies published from other years.

Conclusions

Randomized controlled trials involving rotator cuff injuries frequently use PRO measures as primary outcomes. Reporting of these PRO measures is suboptimal and may benefit from rigorous standardization.