A Historical Analysis of Randomized Controlled Trials in Rotator Cuff Tears.

Background and objectives: Our research aimed to evaluate the quality of reporting of randomized controlled trials (RCTs) linked to rotator cuff (RC) tears. The present study analyzed factors connected to the quality of the RCTs and trends in the quality of reporting through time. Materials and
Methods: The online databases used to search all RCTs on the topic of RC surgery completed until March 2020 were PubMed and Ovid (MEDLINE). The quality of reporting was evaluated using the modified Coleman methodology score (MCMS) and the consolidated standards of reporting trials (CONSORT).
Results: The online search found 957 articles. Finally, 183 studies were included in the quantitative synthesis. A total of 97 (53%) of 183 studies had a level of evidence I and 86 (47%) of 183 studies had a level of evidence II, according to the Oxford Center of Evidence Based Medicine (EBM). A statistically significant difference in MCMS between articles written before 2010 and articles written after 2010 was found. Articles written after 2010 had, on average, the highest Coleman score. The average number of CONSORT checklist items for each article across all analyzed RCTs was 21.67. The 37 studies completed up to 2010 averaged a number of checklist items of 19.97 and the studies completed between 2011 and 2019 averaged a number of checklist items of 22.10. A statistically significant difference in the number of checklist items between articles written before 2010 and articles written after 2010 was found. Articles written after 2010 had on average more checklist items. However, low correlation (0.26) between the number of checklist items for each article and the respective Coleman score was found. On the other hand, articles with the CONSORT diagram had a significantly high Coleman score. Conclusions: An improvement in the quantity and quality of RCTs relating to RC surgery over the analyzed period was found.

1. Introduction

Rotator cuff (RC) tears are among the most frequent injuries of the tendon-bone junction in the human population [1]. RC repair is one of the most common surgical operations for shoulder disorders [2] or may be the most common of all shoulder surgeries [3].

Randomized controlled trials (RCTs) can be useful to improve the health of patients affected by RC tears. Randomization provides one single important advantage over all the other methodologies: it equipoises known and unknown prognostic elements among treatment groups. For this reason, it is very important to draft an accurate report of the results to make the analysis scientifically creditable [4].

Evidence based medicine (EBM) aims to optimize clinical care with the use of published literature, evaluating not only medicine studies, but also other disciplines such as nursing, dentistry, public health, and health policy [5].

RCTs must be conducted with careful attention because there are a lot of known and unknown confounding factors. It is useful to implement factors such as the allocation of randomization and to perform an intent-to-treat analysis of the patients minimizing the bias effects [6].

However, the quality of RCT reports was improved when CONSORT (consolidated standards of reporting trials) was introduced to reduce the problems arising from the inadequate non-pharmacologic treatments (NPTs) such as surgery, rehabilitation, or physiotherapy RCT reports [7].

Our systematic analysis aims to widely evaluate the quality of reporting of RCTs linked to RC tears. In particular, our research investigated potential factors linked to the quality of the RCTs and reports quality trends over time.

2. Materials and Methods

2.1. Search Strategy and Study Eligibility

A systematic analysis of the literature was made following the preferred reported items for systematic review and meta-analysis statement (PRISMA). RCTs and prospective cohort studies on the topic of RC surgery published until 1 March 2020 were included. The following combinations of keywords were used: “rotator cuff”; “rotator cuff tear”; “rotator cuff repair”; “pain after rotator cuff repair”; “rotator cuff physiotherapy”; and “rotator cuff surgery”. All articles were initially screened for relevance by title and abstract excluding articles without an abstract and obtaining the full-text article if the abstract did not allow the investigators to assess the defined inclusion and exclusion criteria. A cross-reference search of the selected articles was also performed to obtain other relevant articles for the study. Inclusion criteria were: level of evidence I or II according to the Oxford Center of EBM; studies on human patients; English-language; and focus on RC tears. Exclusion criteria were: review articles; animal and cadaveric studies; commentary reports; pilot studies; meta-analysis; and conference papers. The articles (titles, abstracts and full-text) were screened independently by two researchers: two investigators (G.S and C.D.N) separately performed a careful reading of all papers and extracted data. In this review, the CONSORT checklist was used independently to assess each trial by the two reviewers, with particular attention to those items that were not addressed in the Coleman score. The CONSORT checklist is a series of 25 items and 37 checklist items focused on reporting how the trial was designed, analyzed, and interpreted. For each item, a yes/no answer, depending on the completeness of the information reported in the study is given. The flow diagram just displays the progress of all participants through the trial and is one of the items in the CONSORT checklist.

2.2. Data Abstraction

The data extracted were recorded using Word and Excel (Microsoft). Full name of the first author, year of publication, number of patients enrolled, number of patients lost at follow-up, level of evidence of the study, location of the study, age of patients, sex of patients, financial support, number of centers involved, the presence of CONSORT flow diagram, sample size, trial type and topic, mean follow-up, type of rehabilitation, and journal of publication were recorded.

2.3. Methodological Quality Assessment

The methodological quality of the included articles was independently evaluated by two reviewers using the modified Coleman methodology score (MCMS). The MCMS is characterized by the presence of eleven criteria evaluated through a scoring system between 0 and 100, where 100 indicates a high-quality study without chances, bias, and other confounding factors [8]. The MCMS was used for the statistical analysis to evaluate the correlation between the variables involved.

2.4. Assessment of Agreement

The data from the full-text selected articles were extracted independently by two authors (G.S., C.D.N.) and checked by a third author (G.F.). To ascertain the validity of the included studies, two reviewers independently evaluated the methodological quality and reliability of the findings through MCMS.

2.5. Statistical Analysis

To compare the MCMS of articles written before 2010 and articles written after 2010 and to find statistically significant differences in terms of MCMS, between studies containing a CONSORT diagram and studies without the flow-chart of the independent samples t-test was used. The inter-rater reliability of grading the Coleman score and the association between the number of checklist items and the Coleman score for each article with Pearson’s correlation were evaluated. The inter-rater reliability of grading the CONSORT checklist with percent agreement between raters was evaluated.

3. Results

3.1. Study Characteristics

The selection process is illustrated in Figure 1. The literature search and cross-referencing resulted in a total of 957 articles. Finally, 183 studies were included in quantitative synthesis [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183].

Figure 1

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2009 Flow diagram.

A total of 14,138 patients were initially analyzed in the involved studies. One thousand one hundred fifty patients were lost at the final follow-up [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183]. A total of 97 (53%) of 183 studies had a level of evidence I and 86 (47%) of 183 studies had a level of evidence II, according to the Oxford Center of EBM. A total of 170 (93%) of the 183 studies were single-centered [3,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,32,33,34,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183]; nine (5%) were multi-centered [4,5,7,31,35,36,58,145,161], and in four (2%) of the articles the data were not indicated [1,2,6,8].

Most of the included studies were published in the following three major journals: Arthroscopy (34 studies; 18.6%), The American Journal of Sports Medicine (30 studies; 16.4% of the total), and J Shoulder Elbow Surg (23 studies; 12.6%).

3.2. Topic

Among the 183 identified articles, six were focused on acromioplasty, 27 on the use of platelet rich plasma, platelet-rich fibrin matrix, leucocyte and platelet-rich fibrin and augmentation, 13 on single-row versus double-row repair, 54 on pain, five on conservative versus surgical repair, one on trauma-related rotator cuff tears, 31 on physiotherapy, six on the treatment of the long head of the biceps tendon disorder, four on mini-open versus arthroscopic repair, five on subacromial decompression, 10 on surgical techniques, four on post-operative immobilization, and 17 on other items linked to rotator cuff pathology (miscellaneous) (Supplementary Materials Tables S1–S13).

3.3. Modified Coleman Methodology Score (MCMS)

The inter-rater reliability of grading the Coleman score was 98.6%. The average MCMS across all analyzed RCTs was 71.40. The 37 studies completed up to 2010 averaged a MCMS of 68.54 and the studies completed between 2011 and 2019 averaged a MCMS of 72.12. (Table 1).

Table 1

Mean Coleman score in articles written before 2010 and articles written after 2010.

Years	Mean Coleman Score	Number of Studies	Standard Deviation
1996–2010	68.54	37	8927
2011–2019	72.12	146	7869
Total	71.40	183	8196

A statistically significant difference in MCMS between articles written before 2010 and articles written after 2010 was found (Table 2).

Table 2

Difference in modified Coleman methodology score (MCMS) between articles written before 2010 and articles written after 2010 (using Independent Samples t-test).

t	df	Significance (2-tailed)	Mean Difference	Std. Error Difference	95% Confidence Interval of the DifferenceLower Upper
−2231	51,079	0.030	−3583	1606	−6806	−0.359

Articles with CONSORT diagram had the highest Coleman score (Table 3). Statistically significant differences in terms of MCMS between studies containing a CONSORT diagram or not were found (Table 4).

Table 3

Difference in mean Coleman score between articles with and without a CONSORT diagram.

CONSORT Flow Diagram	Mean Coleman Score	Number of Studies	Standard Deviation
NO	70.46	136	8409
YES	74.11	47	6941
Total	71.40	183	8196

Table 4

Differences in terms of MCM, between studies containing a CONSORT diagram or not (using the Independent Samples t-test).

t	df	Significance (2-tailed)	Mean Difference	Std. Error Difference	95% Confidence Interval of the DifferenceLower Upper
2931	96,069	0.004	3643	1243	1176	6111

3.4. Trends

The majority of the identified studies had been produced since 2014. From about 2012, the number of RCTs published has increased significantly when compared to previous years (Figure 2).

Figure 2

Coleman score analysis.

3.5. Other Methodological Factors on the Consolidated Standards of Reporting Trials (CONSORT) Checklist

The inter-rater reliability of grading the CONSORT checklist was 97.4%.

The rate of missed checklist items of the CONSORT checklist for each trial, calculated as the ratio of number of checklist items with incomplete information to total checklist items for each trial, is shown in Supplementary Materials Table S14.

Items and checklist items most frequently lost were analyzed. Items “3b. Important changes to methods after trial commencement (such as eligibility criteria), with reasons” (182 missed, Trial Design item), “6b. Any changes to trial outcomes after the trial commenced, with reasons” (180 missed, Outcomes item), “7b. When applicable, explanation of any interim analyses and stopping guidelines” (181 missed, Sample Size item), “14b. Why the trial ended or was stopped” (183 missed, Recruitment item), “17b. For binary outcomes, presentation of both absolute and relative effect sizes is recommended” (178 missed, outcomes, and estimation item), and “18. Results of any other analyses performed including subgroup analyses and adjusted analyses, distinguishing pre-specified from exploratory” (181 missed, Ancillary analyses item) were the most frequently lost.

Low correlation (0.26) between the number of checklist items for each article and the respective Coleman score was found.

The average number of checklist items for each article across all analyzed RCTs was 21.67. The 37 studies completed until 2010 averaged a number of checklist items of 19.97 and the studies completed between 2011 and 2019 averaged a number of checklist items of 22.10 (Table 5).

Table 5

Mean number of checklist items in articles written before 2010 and articles written after 2010.

Years	Mean Number of Checklist Items	Number of Studies	Standard Deviation
1996–2010	19.97	37	3594
2011–2019	22.10	146	2822
Total	21.67	183	3103

A statistically significant difference in the number of checklist items between articles written before 2010 and articles written after 2010 was found (Table 6). Articles written after 2010 have on average more checklist items.

Table 6

Difference in the number of checklist items between articles written before 2010 and articles written after 2010 (using the Independent Samples t-test).

t	df	Significance(2-tailed)	Mean Difference	Std. Error Difference	95% Confidence Interval of the DifferenceLower Upper
−3342	47,839	0.002	−2123	0.635	−3400	−0.845

A CONSORT flow diagram that outlines the inclusion and exclusion of patients for the trial as well as follow-up rate was included in 47 studies (25.6% of total studies analyzed). Taking into consideration only the studies completed between 1996 and 2010, three included a CONSORT flow diagram. Next, the remaining 44 studies in which a flow diagram was found were completed between the years 2011 and 2019. All the analyzed studies completed until 2000 did not include a CONSORT flow diagram.

4. Discussion

Rotator cuff disease is among the most common musculoskeletal disorders [184]. It is a disabling condition with high prevalence rate [185], causing high direct and indirect costs [186]. The appropriate treatment for rotator cuff disease is debated. Rotator cuff repair is an option for patients with chronic, symptomatic full-thickness rotator cuff tear, but the quality of evidence is unconvincing. There is also little compelling evidence for conservative treatment [187]. RCTs are considered to be the most reliable form of scientific evidence in the hierarchy of evidence that influences healthcare policy and practice because RCTs reduce spurious causality and bias. Well done RCTs are the only way to find strong evidence on rotator cuff disease and treatment. This systematic review aimed to analyze factors connected to the quality of the RCTs and trends in the quality of reporting through time. The use of a CONSORT flow diagram, CONSORT checklist, and the MCMS have been found to be the most important predictive factor for a good quality RCT.

Rotator cuff repair can be performed with several techniques: transosseous simple suture, single-row suture anchor, double-row suture anchor, and transosseous-equivalent techniques. Current evidence has failed to prove a significant difference between single-row and double-row techniques in terms of clinical outcomes. The ideal rotator cuff repair technique should provide a good restoration of the footprint contact area and compression of the tendon on the footprint. Moreover, rotator cuff repair is not the only procedure undertaken to manage rotator cuff tears. Acromioplasty, long head of biceps tendon tenotomy, or tenodesis are surgical procedures often performed with rotator cuff repair. Acromioplasty is an arthroscopic surgical procedure made to prevent the impingement of the rotator cuff and consists of the decompression of subacromial space. Stem cell therapy is an emerging treatment for tendon disorders and can be used during arthroscopic rotator cuff repair, after rotator cuff repair, or instead of rotator cuff repair. The real efficacy of stem cell therapy for rotator cuff tears is nowadays debated. The 183 studies analyzed focused on this major topic concerning rotator cuffs. Pain resulted in being the most macro-area studied with 54 articles (29.5% of total RCTs), followed by physiotherapy (31 articles, 17% of total RCTs). Another popular topic is the use of stem cell therapy (27 articles, 15%).

The number of RCTs has increased over the last few years thanks to higher attention to the process of randomization and the prospective calculation of sample size. On the other hand, factors such as the description of concealment of patient allocation and the intention-to-treat analysis did not receive the same attention.

The MCMS, CONSORT flow diagram, and CONSORT checklist have been used to assess the quality of RCTs previously identified. This review showed a significant improvement in the quality in RCTs in the long term. This was confirmed by the studies completed until the year 2010 that had an average MCMS of 68.54 compared to the studies completed between 2011 and 2020 with an average of MCMS of 72.12. A statistically significant difference in MCMS between articles written before 2010 and articles written after 2010 was found. The main limitation of MCMS is its capacity to evaluate only the quality of reporting and not the quality of the study itself; hence high-quality studies that are reported poorly would receive a low score.

On the other hand, the CONSORT checklist is a series of 25 items focused on reporting how the trial was designed, analyzed, and interpreted. The CONSORT checklist was introduced to reduce the problems arising from the inadequate NPTs and to improve the quality of RCT reports [7]. The CONSORT statement is an evidence-based minimum set of recommendations including a checklist and flow diagram for reporting RCTs to improve the complete reporting of trials [57]. In 2008, the CONSORT Group developed an extension to the original statement that addressed methodological issues specific to trials of NPTs such as surgery, rehabilitation, or psychotherapy [59]. Thus, a modified checklist, aimed specifically at trials with non-pharmacologic interventions, was created, with a detailed explanation published in 2008 [73]. The factors outlined in this extension included descriptions of how such trials should outline the randomization procedure used, calculate sample sizes, report the blinding status, and outline the flow of participants, with flow diagrams strongly recommended [79].

The average number of CONSORT checklist items for each article across all analyzed RCTs was 21.67. The 37 studies completed until the year 2010 averaged a number of checklist items of 19.97 and the studies completed between 2011 and 2019 averaged a number of checklist items of 22.10. A statistically significant difference in the number of checklist items between articles written before 2010 and articles written after 2010 was found. Articles written after 2010 had on average more checklist items. However, low correlation (0.26) between the number of checklist items for each article and the respective Coleman score was found. On the other hand, articles with a CONSORT diagram had a significant high Coleman score.

Taking into consideration only the studies completed between 1996 and 2010, three included a CONSORT flow diagram. The remaining 44 studies in which we found a flow diagram were completed between the years 2011 and 2020. All of the analyzed studies completed up to 2000 did not include a CONSORT flow diagram.

There are several limitations related to the use of MCMS to measure the quality of RCTs. This index assesses the quality of reporting of the trials rather than the actual quality of the trials themselves. In fact, some trials may have used higher methodological safeguards that were not reported in the study. Moreover, certain factors that compose the MCMS can be only addressed to surgical trials (e.g., appropriate description of surgical technique and description of postoperative protocol). Therefore, by definition, non-surgical studies could never reach the maximum score of 100.

This study included only English-language trial and those indexed in PubMed and Ovid (MEDLINE); thus, our analysis did not address RCTs on RC surgery in other languages or those that were not indexed in these databases. Moreover, RCTs focused on shoulder pain with intact rotator cuff tendons were excluded [188]. High-certainty evidence shows that subacromial decompression does not provide clinically important benefits over placebo in pain, function, or health-related quality of life [189].

In this context, despite the “new” updated checklist specifics to evaluate a report of a non-pharmacological trial, it is often impossible to perform sham interventions or to blind patients and care providers.

Future research might investigate the impact that RCTs have had on clinical practices in RC surgery over time including the effect of specific methodological factors. It is also important to evaluate the quality of reporting in other areas of orthopedics to compare areas that require improvements.

5. Conclusions

The last decade was characterized by a significant increase in the quantity of RCTs evaluating RC repairs. The quality of RCT reporting (assessed through MCMS and other quality indexes) has shown a steady increase in results after 2010 compared to previous years. In particular, significant improvements in the reporting of the randomization process as well as prospective sample size calculation were noted. Statistically significant differences, in terms of MCMS, between studies containing a CONSORT diagram or not were found. Articles with a CONSORT diagram had the highest Coleman score. A low correlation between the number of checklist items for each article and the respective Coleman score was found.