Reporting quality of randomized, controlled trials evaluating immunotherapy in lung cancer.

BACKGROUND: With the improvement of therapeutic strategies from cytotoxic chemotherapy to immunotherapy, the possibility of achieving timely intervention for lung cancer has dramatically increased. This study aimed to systematically evaluate the reporting quality of randomized controlled trials (RCT) on immunotherapy in lung cancer.
METHODS: The RCTs evaluating the efficacy of immunotherapy in lung cancer published up to 2021 were searched and collected from PUBMED and EMBASE by two investigators. The 2010 Consolidated Standards for Test Reports (CONSORT) statement-based 28-point overall quality score (OQS) and the 2001 CONSORT statement-based 19-point OQS was utilized for assessing the overall quality of each report.
RESULTS: One hundred and fifty-two related RCTs were retrieved in this study, including 81,931 patients. The average OQS in 2010 was 17.89 (range, 7.5-24.5). Overall, studies have sufficiently reported the eligibility criteria (143/152; 94.07%), described the scientific background (150/152; 98.7%) and discussed interventions (147/152; 96.7%). However, the RCTs did not consistently report the changes to trial after commencement (48/152; 31.6%), allocation, enrollment and assignment personnel (34/152; 22.4%), blinding (48/152; 31.6%), or randomization method (58/152; 38.2%).
CONCLUSIONS: The overall reporting quality of RCTs on immunotherapy in lung cancer was found to be unsatisfactory despite the fact that the CONSORT statement was issued more than a decade ago. Furthermore, there was virtual selectivity and heterogeneity in reporting some key issues in these trials. This is the first study to enlighten lung cancer researchers especially focusing on immunotherapy, and also to remind editors and peer reviewers to strengthen their due diligence.

INTRODUCTION

The gold standard of evidence‐based medicine is still randomized controlled trials (RCTs), and the reporting quality of RCTs is crucially important for guiding clinical practice. , The most significant and immediate resource for evaluating trial quality is the published RCTs. In order to ensure the transparency and clarity of the trial report, the Consolidated Standards of Reporting Trials (CONSORT) statement was formulated, which was subsequently updated in 2001 and 2010.

As one of the most lethal solid cancers in the world, lung cancer patients, 95% of which are classified as non‐small cell lung cancer (NSCLC) or small cell lung cancer (SCLC), , , , , exhibited poor clinical outcome when diagnosed with stage IV, making early intervention extremely crucial. Fortunately, with the deeper understanding of molecular genetics and the immune microenvironment, therapeutic strategies have advanced from pure cytotoxic chemotherapy to targeted therapy and, most recently, immunotherapy. , , , , Several antiprogrammed death receptor‐1 (PD‐1)/PD‐ligand‐1 (PD‐L1) antibodies have presented inspiring and long‐lasting benefit in driver gene‐negative NSCLCs in either setting of monotherapy or combined therapy, and also exhibited encouraging efficacy in some settings of SCLCs, many of which have already been included in the international and domestic guidelines. , , , , ,

Therefore, it is very important to evaluate the reporting quality of RCTs on immunotherapy in lung cancer, since those which are poorly reported will bring adverse effect to clinical practice. In this study, we aimed to evaluate the reporting quality of published RCTs on immunotherapy in lung cancer on the basis of the CONSORT statement.

METHODS

Study selection

This study was driven by a transcendental, written plan (see supplementary methods). To determine the human prospective randomized controlled trials, we searched the all‐sided English literature published in PUBMED and EMBASE, with the search conditions: (“lung cancer”, “randomized” and “immunotherapy”) OR (“lung cancer”, “randomized” and “PD‐1”) OR (“lung cancer”, “randomized” and “PD‐L1”) OR (“lung cancer”, “randomized” and “pembrolizumab”) OR (“lung cancer”, “randomized” and “nivolumab”)OR (“lung cancer”, “randomized” and “toripalimab”) OR (“lung cancer”, “randomized” and “tislelizumab”) OR (“lung cancer”, “randomized” and “sintilimab”).

OR (“lung cancer”, “randomized” and “camrelizumab”).

OR “lung cancer”, “randomized” and “atezolizumab” OR “lung cancer”, “randomized” and “durvalumab” as of January 2021 in Pubmed. We searched EMBASE by following:#1 durvalumab OR immunotherapy OR atezolizumab OR toripalimab OR tislelizumab OR sintilimab OR camrelizumab, #2 “lung cancer” AND immunotherapy, “#1 AND #2” was our final terms. This study excluded pediatric participants.

Reporting quality assessment

The data for each publication was reviewed and extracted by two well‐trained investigators. A standardized evaluation checklist was used to compare the results of each test. The 2010 and 2001 CONSORT standard were both applied for comparison experiments. What is more, detailed assessment of key methodological factors, endpoints, follow‐up, subgroup analyses and adverse events were also conducted. Each item in the standard list was divided into completed or incomplete. (Supplementary methods).

Data collection

The criteria of each study were evaluated and entered into a dedicated electronic research database by two observers. Including the first three CONSORT evaluations of each observer, a pilot study was carried out which arbitrated the results to guarantee that the interpretation of standards was agreed upon. The remainder of the studies that applied the CONSORT guidelines checklist were then arbitrated to correct any discrepancies between observers. An investigator analyzed the differences.

Analysis

In this study, the main purpose was to estimate the quality of cancer literature reports related to randomized controlled trials of lung cancer. Twenty‐five categories and multiple subcategories were contained in the CONSORT standard to help explain the quality of the evidence report for any given study. Each criterion was assigned 1 point. Indicators containing two, three, or four subindices were assigned one‐half, one‐third, or one‐quarter, respectively, to attain overall weight. This was consistent with previous studies that used similar indicators. We carried out an description of the CONSORT statement on the reporting of harms.

Results

In the literature search, 2038 studies were obtained. This study finally included 152 of them, as demonstrated by the flowchart in Figure 1. A total of 98 studies were excluded because they were nonlung cancer trials, and 1526 studies were excluded because they were not randomized controlled trials or randomized controlled trials assessing immunotherapy. The final 152 studies which were included were subsequently divided equally into two groups for further analysis. Most of the studies took place in Asia (n = 28, 18.4%), and 77.6% (n = 118) of them were from North America and Europe (Table 1).

FIGURE 1

Flowchart of the selection process of randomized, controlled trial (RCT) articles

TABLE 1

Characteristics of the trials

Characteristic	No. of studies (n = 152)	%	95% Lower limit	95% Upper limit
Year of publication
1976–2000	27	17.76315789	12%	24%
2001–2010	8	5.263157895	2%	9%
2011–2021	117	76.97368421	70%	84%
Region in which trials were conducted
Asia	28	18.42105263	12%	25%
Europe and North America	118	77.63157895	71%	84%
Others	6	3.947368421	1%	7%
Journal
Annals of Oncology	10	6.578947368	3%	11%
The Lancet	4	2.631578947	0%	5%
Lancet Oncol	19	12.5	7%	18%
J Clin Oncol	10	6.578947368	3%	11%
Other journals	99	65.13157895	58%	73%
Journal impact factor
<4	24	15.78947368	10%	22%
4–10	45	29.60526316	22%	37%
>10	83	54.60526316	47%	63%
Phase
2	9	5.921052632	2%	10%
3	106	69.73684211	62%	77%
Unclear	38	25	18%	32%
Intervention
Immunotherapy + radiotherapy	3	1.973684211	0%	4%
Immunochemotherapy	118	77.63157895	71%	84%
Comparison	33	21.71052632	15%	28%
Primary outcome
Positive	130	85.52631579	80%	91%
Negative	22	14.47368421	9%	20%
Sample size
<200	71	46.71052632	39%	55%
200–400	16	10.52631579	6%	15%
>400	65	42.76315789	35%	51%

As shown in Figure 2, the percentages of RCTs reporting scientific background or specific objectives/hypotheses were 98.7% and 99.3%, respectively. However, the percentages of RCTs reporting blinding, random allocation mechanism and allocation, or enrollment and assignment personnel were 31.6%, 29.6%, and 22.4%, respectively.

FIGURE 2

Percentage of studies meeting CONSORT criteria grouped by study section

The research characteristics and descriptive results based on the 2010 CONSORT statement are demonstrated in Table 2. The median CONSORT score was 17.89 (range, 7.5–24.5). Overall, studies sufficiently reported the eligibility criteria (143/152; 94.07%), described the scientific background (150/152; 98.7%) and discussed interventions (147/152; 96.7%). However, the RCTs did not consistently report the changes to trial after commencement (48/152; 31.6%), allocation, enrollment and assignment personnel (34/152; 22.4%), blinding (48/152; 31.6%), or randomization method (58/152; 38.2%). The evaluation based on the 2001 CONSORT statement is shown in the Table S1.

TABLE 2

Overall quality of reporting: Rating using items based on the 2010 CONSORT statement (n = 152)

Item	Criteria	Description	No. of positive trials	%	95% CI
1	Title	Identification as a randomized trial in the title	74	48.68421053	(0.407,0.566)
2	Abstract structure	Structured summary of trial design, methods, results and conclusions	109	71.71052632	(0.646,0.789)
3	Background	Structured summary of trial design, methods, results and conclusions	150	98.68421053	(0.969,1.005)
4	Objectives	Description of the specific objectives or the scientific hypotheses in the introduction	151	99.34210526	(0.981,1.006)
5	Trial design	Description of trial design, including allocation ratio	134	88.15789474	(0.830,0.933)
6	Participants	Description of the eligibility criteria for participants	143	94.07894737	(0.903,0.978)
7	Settings and location	Description of the settings and locations where the data were collected	97	63.81578947	(0.562,0.715)
8	Interventions	Details of the interventions intended for each group	147	96.71052632	(0.939,0.995)
9	Outcomes	Definition of primary and secondary outcome measures, including how and when they were assessed	146	96.05263158	(0.930,0.991)
10	Sample size	Description of sample size calculation	112	73.68421053	(0.667,0.807)
11	Randomization, sequence generation	Definition of the method used to generate the random allocation sequence	58	38.15789474	(0.304,0.459)
12	Randomization, restriction	Description of the type of randomization; details of any restriction	83	54.60526316	(0.467,0.625)
13	Allocation concealment	Description of the mechanism used to implement the random allocation sequence to assure concealment until interventions were assigned	45	29.60526316	(0.223,0.369)
14	Implementation	Description of who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions	34	22.36842105	(0.157,0.290)
15	Blinding	Whether or not participants, those administering the interventions, or those assessing the outcomes were blinded to group assignment; if relevant, description of the similarity of interventions	48	31.57894737	(0.242,0.390)
16	Statistical methods	Description of the statistical methods used to compare groups for primary and secondary outcomes	130	85.52631579	(0.799,0.911)
17	Ancillary analysis, method	Description of the methods for additional analyses, such as subgroup analyses and adjusted analyses	104	68.42105263	(0.610,0.758)
18	Diagram	A CONSORT diagram was presented to show the flow of participants	0	0
19	Participant flow	Details on the flow of participants through each stage of the trials (number of patients randomly assigned, receiving intended treatment, and were analyzed for the primary outcome)	143	94.07894737	(0.903,0.978)
20	Recruitment	Dates defining the periods of recruitment and follow‐up	123	80.92105263	(0.747,0.872)
21	Baseline data	A table showing baseline demographic and clinical characteristics for each group	130	85.52631579	(0.799,0.911)
22	Intent‐to‐treat analysis	Number of patients in each group included in each analysis and whether patients were analyzed according to the group to which they were randomly assigned	19	12.5	(0.072,0.178)
23	Outcomes measures	For each primary and secondary outcome, a summary of results for each group, the estimated effect size and its precision (e.g., 95% CI) are provided	124	81.57894737	(0.754,0.877)
24	Ancillary analyses	Results of subgroup analyses and adjusted analyses, distinguishing prespecified from exploratory	78	51.31578947	(0.434,0.593)
25	Adverse event classification	Description of all important adverse events in each group, with classification	144	94.73684211	(0.912,0.983)
26	Registration	Presentation of the registration number and name of trial registry	73	48.02631579	(0.401,0.560)
27	Protocol	Where the full trial protocol can be accessed	65	42.76315789	(0.349,0.506)
28	Funding	Sources of funding and other support	112	73.68421053	(0.667,0.807)

Additionally, adverse event reporting scores based on the CONSORT recommendations for harms are reported in Table S2. When clinical intervention trials are performed, it is generally unethical to “force” patients to accept a randomly assigned treatment. We would take the “advise” approach and allow patients to refuse. Then, during analysis, we analyze according to random assignment, regardless of whether we really accept the treatment plan or not, which is known as the intention to treat principle (ITT) in this study. ITT was defined as an analysis of all randomized patients in the treatment group assigned to it. If no information was provided, the RCT was considered not to have followed ITT principles (Table S3). According to our study, 41.45% of the trials followed the ITT principle. Descriptions of the use of endpoints in included trials were reported in Table S4.

DISCUSSION

Our study is the first quality evaluation related to immunotherapy for lung cancer. We found that although the CONSORT statement was published more than a decade ago, there are still many problems with the overall reporting quality of randomized controlled trials on immunotherapy in lung cancer. , , , , ,

Through data analysis, the OQS of all the included trials was 17.89 (range, 7.5–24.5), indicating that the quality of the trials was not ideal. Items added or redefined in the 2010 revision are generally poorly reported, compared with the 2001 CONSORT statement. In addition, some common OQS project reports in 2010/2001 were neither uniform nor complete (for example, blinding and randomization were implemented; Table 2). With the lack of information, the explanation would be harder and more complicated, thus some key methodological factors and other low‐quality reports may even potentially cause misguided consequences.

However, the reporting quality of key methodology projects is also not ideal. With the occasional neglect of some detailed elements and the word limit of one report, it is inevitable for readers to insufficiently understand the whole intention of the author and commentator. , , For published studies, enough descriptions of major methodological measures are usually lacking, such as the use of allocation concealment in human trials and blindness, in addition, the individual studies are small in scale, so they are vulnerable to a lack of motivation, impairing the effectiveness and usefulness of published studies. Therefore, it is difficult to establish the clinical practice and evidence‐based guidelines from these studies. Considering that many authors are clinicians, they might consider the clinical features of randomized controlled trials as more interesting indices instead of methodology. Actually, some methods that are not reported in publications are often sufficiently carried out. The insufficiency of the data for these items may present intentional focus rather than a deficient design. In addition, reports on other important issues are unsatisfactory. Under joint projects, future publications need to improve the quality of reporting.

Several methods can promote the quality of reports: increasing compliance with spouses, training, or improving the quality of reporting. Training should be focused on those areas that correspond to poor quality reporting of projects. In this study, CONSORT has not been adopted in some journals. More journal‐approved spouses should be encouraged to improve trial reports. It also helps to make the current CONSORT statement more user friendly.

In order to make the article more comprehensive, we adopted many quality evaluation articles to evaluate the index of RCT to complete our article. For example, we used the OQS scoring method used in some quality evaluation articles, as well as all the entries in the 28 CONSORT statements in some quality evaluation articles. Compared with other quality evaluation articles, our article is more comprehensive and advanced. , , , , , , , , , , , , , , , , , ,

In conclusion, the overall reporting quality of RCTs including adverse event reporting score, key methodological factors and endpoints on immunotherapy in lung cancer are not ideal. Investigators should pay more attention to CONSORT projects when designing and reporting trials. Additionally, editors and peer reviewers also need to be more scrutinizing and diligent.

CONFLICT OF INTEREST

All authors declare no conflict of interests.

Table S1 Overall Quality of Reporting: Rating Using Items Based on the 2001 CONSORT Statement (n = 152)

Table S2. Adverse Event Reporting Scores: Rating Using Items Based on the CONSORT Recommendations for Harms (n = 152)

Table S3. Descriptions of Key Methodologic Factors in Included Trials (n = 152)

Table S4. Descriptions of the Use of Endpoints in Included Trials (n = 152)

Click here for additional data file.

Appendix S1 Supporting Information

Click here for additional data file.