Characteristic Analysis of Complementary and Alternative Medicine in Randomized Controlled Trials of Oncology: A Comparison of Published Studies.

BACKGROUND: Complementary and alternative medicine (CAM) has been widely used by cancer patients and oncologists in the past decades. The present study aimed to examine and compare the characteristics and registration status of published studies in a sample of recently published CAM randomized controlled trial (RCT) reports of oncology in leading journals of 3 categories: general and internal medicine (GIM), clinical oncology (CO), and CAM.
METHODS: Articles published in the top 5 journals of the 3 categories from 2006 to 2015 were searched in PubMed. Basic characteristics, registration information, impact factor, and citations were identified and extracted from the included RCTs. Data were summarized by frequency, mean, and median and compared using χ2 test and Kruskal-Wallis H test.
RESULTS: A total of 59 RCTs were included; among them, 34 (58%) could be identified with a registration number. GIM journals (15) enjoyed the highest average number of citations per article, followed by CO (12) and CAM (3) journals ( P < .0001). ClinicalTrials.gov was the most popular registry for these RCTs. Of the RCTs registered in ClinicalTrials.gov , 24% (4/17) of the published studies in CO journals put their results in the registry; however, no study in GIM and CAM journals put the result in the registry ( P = .372).
CONCLUSION: The top GIM, CO, and CAM journals rarely published CAM RCTs of oncology from 2006 to 2015, and the CAM articles of oncology were less cited. However, there was a clear improvement in the trial registration rate over the past decades.

Introduction

Complementary and alternative medicine (CAM) refers to a group of diverse medical and health care interventions, practices, products, or disciplines that are not generally considered as part of conventional medicine. Clearly the boundaries between CAM and conventional medicine (also called Western or allopathic medicine) are not absolute.[1] CAM has been an increasing feature of health care practice in the past decade. In 2007, almost 40% of adults in the United States had used CAM therapy in the past 12 months.[2] The popularity of CAM use among cancer patients has been reported in various countries, including the United States (70.2%), Japan (44.6%), and China (83.0%).[3-5] Apart from the wide use of CAM in cancer treatments, anticancer research on CAM is also remarkable.[6] However, the characteristics and quality of these research articles in this field have not reported yet.

Besides, the need for clinical trial registration as well as the benefits it has brought for the transparency of clinical research has been recognized for years.[7,8] In 2005, the International Committee of Medical Journal Editors (ICMJE) instituted a policy that required investigators to deposit information about trial design into an accepted clinical trials registry before the beginning of patient enrollment. It aimed to foster a comprehensive, publicly available database of clinical trials and ultimately benefit the general public.[9] The main purpose of trial registration is to minimize potential bias in trial publication and result reporting and thus provide reliable evidence for the publics.[10]

The present study was conducted to examine and compare the characteristics and registration status of published studies in a sample of recently published CAM randomized controlled trial (RCT) reports of oncology in leading journals of 3 categories: general and internal medicine (GIM), clinical oncology (CO), and CAM.

Methods

Search Strategy

The article search was conducted in the PubMed database, which aimed to access systemic anticancer therapy and symptom management of cancer patients. Limits were set on journal type, publication time, and article type. We selected the top 5 GIM journals, the top 5 CO journals, and the top 5 CAM journals according to the Journal Citation Report (JCR) 2014 released by Thomson Reuters (Table 1). The time period was set between January 1, 2006, and December 31, 2015. For article type, only “RCT” was selected. Two authors (Zhang HQ and Yang GL) systematically reviewed the search result using Endnote X3.0 software. Articles were included if the study was identified as RCT, and fell into the category of CAM studies of oncology (2 authors, Yang GL and Zhang W, separately reviewed the title, abstract of each article, and full text if necessary; any disagreement was resolved by consulting Zhang HQ). Exclusion criteria included the following: review, meta-analysis, literature analysis, commentary, letter, retrospective studies, pilot studies, secondary analysis of completed studies, follow-up studies, or protocol studies.

Table 1.

Distribution of Included Trials in the Top 5 Journals of 3 Categories.

Rank	General and Internal Medicine Journal	n (%)	Clinical Oncology Journal	n (%)	CAM Journal	n (%)
1	New England Journal of Medicine	0	Lancet Oncology	0	Journal of Ethnopharmacology	0
2	Lancet	0	Journal of Clinical Oncology	33 (89)	American Journal of Chinese Medicine	2 (14)
3	Journal of the American Medical Association	4 (50)	Journal of the National Cancer Institute	4 (11)	Integrative Cancer Therapies	9 (64)
4	Annals of Internal Medicine	1 (13)	Leukemia	0	BMC CAM	3 (21)
5	British Medical Journal	3 (38)	Clinical Cancer Research	0	Evidence-based CAM	0

Abbreviation: CAM, complementary and alternative medicine.

Data Extraction

For each publication, 2 authors (Zhang HQ and Yang GL) independently retrieved the following elements: journal name, publication year, first author’s origin, study type (single center, which means patients were only enrolled in one site; multicenter, which means patients were enrolled in at least 2 sites; study group, which means the study was designed and conducted by a study group and published in the study group’s name), category of CAM (using the classification system employed by the National Center for Complementary and Integrative Health, CAM therapies were grouped into 5 broad categories for analytical purposes: alternative medical systems [acupuncture, homeopathic treatment, etc], biologically based therapies [chelation therapy, folk medicine, etc], manipulative and body-based therapies [massage, movement therapies, etc], mind-body therapies [yoga, tai chi, etc], and energy healing therapies [Reiki, etc]),[2] sample size, sample size calculation (yes or no), type of RCT’s result (positive or negative), type of blinding method (double-blind, single-blind, or open), type of control arm, and funding sources.

To compare the quality of the CAM research articles of oncology published in the 5 leading journals from GIM journals, CO journals, and CAM journals, the average impact factor (IF), 5-year IF, and the average number of citations per article of the published articles involved were extracted and calculated. The average IF of each of the 15 journals in 2015 and 5-year IF were generated using the JCR 2014, and the average number of citations per article of the published articles from GIM journals, CO journals, and CAM journals from 2006 to 2015 was calculated.

Registration information of each included article was sought by 2 authors separately. First, each article was carefully read by 2 authors to see if trial registration number was provided; if not, the authors searched the article in the registries that was accepted by the ICMJE.[8,11] Second, to ensure that the searched trial record matched the published article, we cross-checked the condition, intervention, principle investigator, phase of trial, and study start and end date. If no registration number was found, the published article was regarded as not registered.

For each registered trial, the trial registry was collected. In 2008, the ClinicalTrials.gov was expanded to include a database for reporting summary results.[12] We recorded the trial registry if the registered record showed result information in ClinicalTrials.gov.

Statistical Analysis

For the basic study characteristics of the included studies, we used descriptive variables such as mean and median for quantitative variables, and number of articles (percentage) for categorical variables. Differences in categorical variables were assessed by χ2 test and Kruskal-Wallis H test where appropriate. All tests were 2-sided, and P < .05 was considered statistically significant. All analyses were performed using SAS version 9.4 (SAS Institute, Cary, NC).

Results

Baseline Descriptive Information of Included Studies

Of the 6086 retrieved articles from 2006 to 2015, a total of 59 CAM RCTs of oncology were included in this analysis (Figure 1). The number of included trials of each journal and the percentage of the total are listed in Table 1. The number of published articles in the fields of CO was significantly larger than that of the articles published in GIM or CAM journals. Figure 2 showed the trend of the number of the published articles in the 3 categories. Most (56%) of the 59 articles were published in the Journal of Clinical Oncology. Each citation of the 59 articles were provided in supplementary material.

Figure 1.

Flowchart for study screening and selection of complementary and alternative medicine’s randomized controlled trials of oncology.

Figure 2.

The numbers of articles in the 3 categories of journals from 2006 to 2015.

Characteristics of these RCTs are presented in Table 2. For the studies published in GIM and CO journals, the authors from the United States were the most prolific (63% and 70%, respectively), while Asian authors (43%) published the most studies in CAM journals. Approximately 88% of the articles published in GIM journals were multicenter studies and 13% were single-center studies, compared with 43% and 57% of those published in CAM journals, respectively. Sixty percent, 38%, and 3% of the articles published in CO journals were multicenter studies, single-center studies, and published in the name of a study group, respectively. For category of CAM, manipulative and body-based therapies took up the highest percentage in the studies published both in GIM (63%) and CO (35%) journals, whereas mind-body therapies accounted the highest percentage in the studies published in CAM studies (57%). Mean and median of sample size of the articles published in GIM journals were larger than those of the articles published in CO journals and CAM journals (mean: 2055, 252, and 66, respectively; median: 206, 200, and 61, respectively). The majority of the trial results in the involved studies were positive (GIM journal: 75%; CO journal: 78%; CAM journal: 64%), P = .587. The majority type of blinding method in the involved trials was open, followed by double-blind and single-blind (GIM journal: 63%, 25%, and 13%; CO journal: 76%, 19%, and 5%; CAM journal: 71%, 21%, and 7%).

Table 2.

Characteristics of the Included Studies.

Variable	Articles
	General and Internal Medicine Journal (n = 8)	Clinical Oncology Journal (n = 37)	CAM Journal (n = 14)
First author origin, n (%)
America	5 (63)	26 (70)	3 (21)
Europe	3 (38)	8 (22)	4 (29)
Asia	0	2 (6)	6 (43)
Other	0	1 (3)	1 (7)
Type of study, n (%)
Single center	1 (13)	14 (38)	8 (57)
Multicenter	7 (88)	22 (60)	6 (43)
Study group	0	1 (3)	0
Category of CAM, n (%)[a]
Alternative medical systems	0	6 (16)	2 (14)
Biologically based therapies	3 (38)	10 (27)	3 (21)
Manipulative and body-based therapies	6 (63)	13 (35)	1 (7)
Mind-body therapies	0	10 (27)	8 (57)
Energy healing therapies	0	0	0
Sample size, mean (median)	2055 (206)	252 (200)	66 (61)
Sample size calculation, n (%)[b]	8 (100)	32 (87)	8 (57)
Type of results, n (%)[c]
Positive	6 (75)	29 (78)	9 (64)
Negative	2 (25)	8 (22)	5 (36)
Type of blinding method, n (%)
Double-blind	2 (25)	7 (19)	3 (21)
Single-blind	1 (13)	2 (5)	1 (7)
Open	5 (63)	28 (76)	10 (71)
Type of control arm, n (%)
Placebo/best supportive care/blank	7 (88)	21 (57)	6 (43)
Conventional treatment	1 (13)	16 (43)	8 (57)
Funding source, n (%)
Industry	0	1 (3)	0
University/hospital/government	4 (50)	25 (68)	7 (50)
Multiple source of funding	4 (50)	8 (22)	2 (14)
Other funding	0	2 (5)	2 (14)
Not reported	0	1 (3)	3 (21)

Abbreviations: CAM, complementary and alternative medicine.

Three articles have 2 categories, with 1 article in general and internal medicine journals and 2 articles in clinical oncology journals.

P = .019, obtained from χ2 test.

P = .587, obtained from χ2 test.

Impact Factor and Average Number of Citations of the Included Studies

Table 3 shows the average IF, 5-year IF, and average number of citations per article of the published articles in the 3 journal categories. According to JCR 2014, both average IF and 5-year IF of GIM, CO, and CAM journals showed great differences. The average IF of GIM journals (34.327) was much higher than that of CO (14.971) and CAM (2.403) journals. Similarly, the 5-year IF of GIM journals (32.485) was obviously higher than that of CO (14.896) and CAM (2.437) journals. Compared with the average IF and 5-year IF, the average number of citations per article of the published articles in the 3 categories of journals showed relatively small differences. From 2006 to 2015, GIM journals (15) enjoyed the highest average number of citations per article, followed by CO (12) and CAM (3) journals (P < .0001).

Table 3.

Impact Factor (IF) and Average Number of Citations per Article of the Included Studies.

Variables	Articles
	General and Internal Medicine Journal (n = 8)	Clinical Oncology Journal (n = 37)	CAM Journal (n = 14)
IF in 2015, mean (median)	34.327 (35.289)	14.971 (12.583)	2.403 (2.361)
5-Year IF, mean	32.485	14.896	2.437
Number of citations per article, mean[a]	15	12	3

P < .0001, obtained from Kruskal-Wallis H test.

Registration Information of the Included Studies

Table 4 shows the registration information of the included studies. A total of 34 studies were found to have a registration number, either identified by reading (GIM journal: 100%; CO journal: 86%; CAM journal: 100%) or searching (CO journal: 14%); all the included RCTs published in GIM journals provided the trial registration number in the published articles, compared with 57% of those published in CO journals and only 29% in CAM journals (P = .005). The ClinicalTrials.gov was the most popular registry for authors (GIM journal: 63%; CO journal: 81%; CAM journal: 50%). Of the RCTs registered in ClinicalTrials.gov, 24% (4/17) of the published studies in CO journals put their results in the registry; however, no study in GIM and CAM journals put the result in the registry (P = .372).

Table 4.

Registration Characteristics of the Included Studies.

Variables	Articles
	General and Internal Medicine Journal (n = 8)	Clinical Oncology Journal (n = 37)	CAM Journal (n = 14)
Registration, n (%)[a]	8 (100)	21 (57)	4 (29)
Registration number identified by, n (%)
Reading	8 (100)	18 (86)	4 (100)
Searching	0	3 (14)	0
Trial registry, n (%)
ClinicalTrials.gov	5 (63)	17 (81)	2 (50)
Current Controlled Trials	2 (25)	0	0
NTR	1 (13)	1 (5)	0
UMIN	0	0	0
ANZCTR	0	0	0
Other	0	3 (14)	2 (50)
Trial results put in ClinicalTrials.gov, n (%)[b]	0/5	4/17 (24)	0/2

Abbreviations: UMIN, University Hospital Medical Information Network Clinical Trial Registry; NTR, Netherlands Trial Register; ANZCTR, Australian New Zealand Clinical Trials Registry; CAM, complementary and alternative medicine.

P = .005, obtained from χ2 test.

P = .372, obtained from χ2 test.

Discussion

This is the first study to demonstrate the registration status and characteristics of CAM RCTs of oncology published in the top 5 GIM, CO, and CAM journals during 2006 to 2015. We identified a sample of 59 RCTs by searching PubMed, among which 34 (58%) could be identified with a registration number. In this study, we found that the top GIM, CO, and CAM journals rarely published CAM RCTs of oncology from 2006 to 2015, except the Journal of Clinical Oncology. In addition, the average number of citations per article of published articles from the 3 categories of journals revealed that the CAM articles of oncology were less cited, compared with the average IF and 5-year IF of the 3 categories of journals.

In this 10-year retrospective study, among the 59 included RCTs, only 14% (8/59) were published in the top 5 GIM journals and only 24% (14/59) were published in the top 5 CAM journals, which might be related to the fact that the methods developed for pharmaceutical or other intervention studies were not always suitable for trials of CAM interventions.[13] In order to be accepted by high-IF journals, more high-quality CAM RCTs should be conducted to provide more optional anticancer treatments so as to help cancer patients overcome cancer diseases and maintain the high quality of life except for conventional treatments.

For type of study, the majority of the articles published in GIM and CO journals were multicenter ones, compared with 57% of the articles published in CAM journals were single-center ones. In addition, mean and median of sample size of the articles published in CAM journals were much smaller than those of the articles published in GIM and CO journals. General speaking, RCTs represent the gold standard in evaluating the effectiveness of novel interventions and high-quality RCT research requires multicenter and large case numbers,[14] and factors such as single center and limited case numbers may restrict the level of CAM RCT research and influence the IF and citation of the articles. The majority type of blind in the involved studies was open, due to the fact that most studies focused on the manipulative and body-based therapies and mind-body therapies. In these fields, it is difficult to set a placebo group for control.

The Science Citation Index, an authoritative evaluating tool, is usually used for measuring research performance from an international perspective: international orientation of national research activities, visibility and impact at the international research front, and international scientific collaboration.[13] The IF just represents a rough estimation of the journal’s IF and fails to reflect the actual current IF of each article. Thus, in this study, the average IF, 5-year IF, and average number of citations per article of the published articles were applied to measure and compare the research performance. From the average number of citations per article of the published articles, we found that the citations of the CAM RCTs of oncology were obviously lower than the average IF and 5-year IF, which meant that fewer researchers and readers focused on the field of CAM therapies for cancer treatment.

The registration of clinical trials can improve public access to information about ongoing trials and help identify and reduce reporting bias. There was a clear improvement in the trial registration rate over the past decades, with 58% RCTs reported in this study being registered, compared with 43% in 2005.[15] Although all the 15 involved journals provided guidance on trial registration in their “instruction to authors,” registration data were found more often for trials published in journals with higher IF, especially the GIM journals more powerfully enforced the requirement on trial registration. CAM RCTs of oncology often take several years to complete, and this study was conducted by searching the literature from 2006 to 2015, so it is possible that some RCTs included in this study were in fact initiated before the 2005 ICMJE policy. Therefore, it is expected that the registration rate of RCTs initiated more recently will be even higher. This result was consistent with previous studies on other medical specialties.[15-18] It is essential for medical journals to enforce the trial registration policy. According to the ICMJE trial registration policy, medical editors and reviewers can keep to trials to be registered, and those unregistered or not properly registered should be absolutely rejected.[19] In 2007, the Food and Drug Administration Amendments Act of 2007 mandated public registration and disclosure of results of “applicable” clinical trials of drugs, biologics, and devices on www.clinicaltrials.gov and required that results for trials of Food and Drug Administration–approved products must be posted within 12 months of trial completion. However, this requirement raised a debate over the past years on when and how to issue the trial results.[20,21] It would be a challenge for the medical journals to accept that a study fully disclosed its results to the public before submission.[16]

This study has some limitations. The CAM RCTs of oncology included for analysis were only from 15 high-impact medical journals of 3 different categories but some other journals concerning CAM oncology were not included. Such a sample may not represent the full view of the characteristics and registration status of CAM RCTs of oncology published in the past decade. Consequently, our results might underestimate the true extent of the problems described.

Conclusion

This study suggested that the top GIM, CO, and CAM journals rarely published CAM RCTs of oncology from 2006 to 2015, and the CAM articles of oncology were less cited. There was a clear improvement in the trial registration rate over the past decades. With an increasing prevalence of CAM use, great efforts should be made to promote the development of CAM RCTs of oncology, improving major components of the registry and study design, such as type of study, type of blind, sample size, and plans for statistical analysis.