Randomized controlled trials in pediatric complementary and alternative medicine: where can they be found?

BACKGROUND: The safety and effectiveness of CAM interventions are of great relevance to pediatric health care providers. The objective of this study is to identify sources of reported randomized controlled trials (RCTs) in the field of pediatric complementary and alternative medicine (CAM).
METHODS: Reports of RCTs were identified by searching Medline and 12 additional bibliographic databases and by reviewing the reference lists of previously identified pediatric CAM systematic reviews.
RESULTS: We identified 908 reports of RCTs that included children under 18 and investigated a CAM therapy. Since 1965, there has been a steady growth in the number of these trials that are being published. The four journals that published the most reported RCTs are The American Journal of Clinical Nutrition, Pediatrics, Journal of Pediatrics, and Lancet. Medline, CAB Health, and Embase were the best database sources for identifying these studies; they indexed 93.2%, 58.4% and 42.2 % respectively of the journals publishing reports of pediatric CAM RCTs.
CONCLUSIONS: Those working or interested in the field of pediatric CAM should routinely search Medline, CAB Health and Embase for literature in the field. The four core journals identified above should be included in their collection.

Background

Complementary and alternative medicine (CAM) is defined as a broad domain of healing resources that encompasses all health systems, modalities and practices and their accompanying theories and beliefs, other than those intrinsic to the politically dominant health system of a particular society or culture in a given historical period [1].

The safety and effectiveness of CAM interventions are of great relevance to pediatric health care providers. Two recent surveys indicate a use of pediatrics CAM in the United States, finding a 12% and 21% overall use, respectively [2,3]. Factors such as ongoing medical problems and parents' use of CAM are associated with pediatric use [3,4]. Ideally, evidence-based practitioners can approach children and their guardians regarding their use of CAM and on a case-by-case basis to help identify relevant evidence regarding therapies that patients use or wish to consider.

Such discussions are likely to be more rewarding when practitioners and families can advocate for the use of CAM therapies for which there is stronger evidence, such as from the results of randomized trials of effective herbal therapies [5]. Similarly, interventions for which there is little of no evidence might be best avoided [6].

We undertook a program of research to discover whether such evidence, in the form of published systematic reviews and randomized controlled trials involving children, existed [7,8]. Based on the large number of randomized controlled trials we did find, we identified a need for a guide for evidence-based practitioners who wish to determine whether sound evidence exists.

The objective of the research presented here was to identify where health care practitioners can locate reported RCTs of pediatric CAM; specifically, in which journals these reports are published and through which databases those reports may be identified.

Methods

Identification of Candidate Studies

We searched for reports of RCTs in 13 bibliographic databases, using either a detailed search strategy based on the main subject search of the Cochrane Collaboration's Complementary Medicine Field subject search, [9] or a modification of that search tailored to the capabilities of particular databases [10]. The databases searched are listed in Table 1, and the search strategies are available from the authors upon request. We also identified reports of RCTs from the cited references of 49 pediatric CAM systematic reviews. The reviews were previously identified by the Chalmers Research Group in a study that complements this project [7].

Table 1

Databases, date ranges, and strategies used.

Database	Host	Date Range Searched	Strategy
Medline & Pre-Medline	Ovid Online	1966-Feb 2001	1*
Embase	Ovid Online	1988-June 2000	1
CINAHL	Ovid Online	1982-July 2000	1
Dissertation Abstracts	Ovid Online	1990-Feb 2000	1
CAB Health	Dialog	1973-March 2001	1
Allied and Complementary Medicine (AMED)	Dialog	1985-March 2001	1
ExtraMED	Dialog	1992-March 2001	1
Manual, Alternative and Natural Therapy (MANTIS)	Dialog	1880-March 2001	1
The Cochrane Library CCTR	Update Software (CD-ROM)	Issue 1, 2001	2**
AGRICOLA	Ovid Online	1975-Jan 2001	3***
NCCAM (now incorporated into PubMed)		1963–2000	3
BioMed Central		1991-Feb 2001	3
IBIDS – NIH Office of Dietary Supplements		Feb 2001	3

*Strategy 1: The main subject search of this strategy was based Cochrane Collaboration's Complementary Medicine Field subject search [9]. **Strategy 2: ((SR-COMPMED and CHILD*:ME) not (ADULT*:ME and ADOLESCENCE:ME)) ***Strategy 3: A combination of the following truncated or non-truncated terms with the appropriate syntax were used: randomized, controlled, child, pediatric, complementary medicine, and alternative medicine.

Bibliographic records were imported into a Reference Manager 9.0™ database and duplicate items were consolidated. Bibliographic records were screened by KC to determine if: the study included children under 18; a CAM therapy was investigated (following the definition of CAM adopted by the Cochrane Complementary Medicine Field Registry of Randomized Controlled Trials and presented in their Registry Guidelines) [11]; and the article was a report of an RCT. Where a final determination of eligibility could not be made based on information contained in the bibliographic record, the complete article was retrieved and reviewed. After about the first 100 records had been assessed, every tenth record was reviewed by MS as a quality measure. Inter-observer agreement was almost perfect, with Kappa = 0.97 [12] without the reviewers undertaking any discussion designed to achieve consensus. A total of 908 reports of RCTs met the criteria for inclusion in the study (see Figure 1).

Figure 1

Study flowchart. Note: For some reports, more than one exclusion criteria was noted, therefore numbers do not add up to 2061.

Studies published in languages other than English were included if: (1) the article was written in a language we were able to read (German, French, and Spanish); or (2) we were able to ascertain, from the English abstract, whether the study met the inclusion criteria. Only journal-published articles were considered. Theses, conference proceedings, book chapters, and other grey literature were not considered.

A post hoc analysis was undertaken to assess whether the apparent decline in the number of reports of trials in complementary and alternative medicine in the final years of this study was due to a delay in the indexing of these articles. For this analysis, the number of Medline records retrieved by search strategy 1 was broken down by year, ending with September Week 3 2002. Simple counts were taken – the individual items were not evaluated against the study inclusion criteria.

Data Extraction

For each eligible publication, the date of publication, authors, language of publication, and journal name were extracted from the Reference Manager database or from inspection of the article. We consulted the reference material for each of the 13 databases searched to determine whether or not the database indexed each of the journals that had published one or more eligible reports of RCTs.

Results

A total of 908 reports of RCTs met the criteria for inclusion in the study.

Year of publication

The earliest reports of RCTs we found were published in 1965, and there was little growth in this literature until 1975. From 1975, there was significant steady growth for over twenty years, with a peak of 89 articles published in 1997 (see Figure 2). There is the beginning of a decline in the next two years, followed by a sudden drop to 45 reports of RCTs in 2000 and 2 in 2001. This decline is probably an artifact of indexing delay, as this research was conducted in the winter of 2001. Post hoc analysis conducted in September 2002 suggests that growth in Medline of records meeting the criteria of search strategy 1 (but not formally screened according to study inclusion criteria) continued to grow through 2000, and again shows a dip for the previous and current year (2001 and 2002).

Figure 2

Growth rate in publication of Pediatric CAM RCT reports, 1965–1999. Note: A few journal articles published in 2000 and 2001 were indexed at the time our study. We have elected not to show them in this portrayal of the growth of the literature. The rate of publication seems to be dropping even in the final years shown here. We believe this to represent a lag between the time of publication and the time the journal articles are indexed in the bibliographic databases, rather than a true decline in the rate of publication of reports of randomized controlled trials in pediatric complementary and alternative medicine.

Authorship

92.2% of reports of RCTs published in this field are written by more than one author. The largest number of authors for a single paper was 25 [13], while almost half of all papers (49.3%) have four or fewer authors. The five most prolific authors have published at least 15 reports of RCTs each. The vast majority (57.8%) of all authors cited in the 908 studies authored a single paper and 70% of authors published four or fewer CAM trials.

Language of Publication

Most reports of RCTs were published in English (93.2%), although 12 other languages were represented. The second most common language of publication was Chinese, with 18 reports of RCTs (2%) and a concentration of studies in herbal medicine. There were more than five reports of RCTs in each of German, French, Spanish and Italian, each with trials in a variety of topics. Seven languages were represented by one or two reports of RCTs (Table 2). The language restrictions resulted in the exclusion of 41 publications as we were unable to confirm that they were reports of RCTs in pediatric CAM; 23 of these were published in Russian, 6 written in Chinese, 4 in Ukrainian, 3 in Japanese, 2 in Italian, and one each in Hungarian, Czech and Swedish.

Table 2

Language of Publication

Language	N of RCT reports	%
English	846	93.2
Chinese	18	2.0
German	13	1.4
French	10	1.1
Italian	6	0.7
Spanish	6	0.7
Russian, Turkish	2 each	0.2 each
Danish, Hungarian, Norwegian, Slovak, Swedish	1 each	0.1 each

Journal

The reports of RCTs originated from 308 different journals. These journals were ranked by the number of reports of RCTs published. As a method to examine the dispersion of the literature, we divided the journals into four zones, each representing approximately one quarter of the articles published.

Of the total number of reports of RCTs, 217 (23.9%) were identified as having been published in the top four journals (1.3% of the total of 308 journals that published at least one relevant reported RCT). In Zone 2, 24 (7.8%) journals published 27.1% of the total number of reports of RCTs identified. Zone 3 encompasses 96 (31.2%) journals and contains 28.7% of the total reports of RCTs identified. Zone 4 contains 184 journals (59.7%), and 20.3% of the total reports of RCTs identified. The top journals, located in Zones 1 and 2, are displayed in Table 3. The names and rankings of journals in Zones 3 and 4 are available from the authors on request.

Table 3

Journals in Zones 1 & 2 and the corresponding number of pediatric CAM RCT reports identified.

	Journal Name	No. of RCT reports	No. of Journals
Zone 1		217 (23.9 %)	4 (1.3 %)
	American Journal of Clinical Nutrition	73
	Pediatrics	55
	Journal of Pediatrics	51
	Lancet	38
Zone 2		246 (27.1 %)	24 (7.8 %)
	Journal of Pediatric Gastroenterology and Nutrition	27
	Journal of Nutrition	25
	European Journal of Clinical Nutrition	18
	Archives of Disease in Childhood	17
	Acta Paediatrica	14
	New England Journal of Medicine	12
	BMJ	10
	Chung-Kuo Chung Hsi i Chieh Ho Tsa Chih	10
	Pediatric Research	10
	British Journal of Nutrition	9
	Journal of Tropical Pediatrics	9
	Transactions of the Royal Society of Tropical Medicine & Hygiene	9
	Archives of Disease in Childhood (Fetal and Neonatal edition)	7
	Developmental Medicine & child Neurology	7
	European Journal of Pediatrics	7
	Journal of Pediatric Psychology	7
	Br Med J (Clin Res Ed)	6
	Indian Pediatrics	6
	Journal of Manipulative & Physiological Therapeutics	6
	Journal of the American College of Nutrition	6
	Lipids	6
	Nursing Research	6
	Nutrition Research	6
	Pediatric Infectious Disease Journal	6
Zone 3	(individual journal names not listed)	261 (28.7 %)	96 (31.2 %)
Zone 4	(individual journal names not listed)	184 (20.3 %)	184 (59.7 %)

For the 308 journals that published reports of pediatric CAM RCTs, the number of journals indexed by each of the databases is shown in Table 4, along with the unique contribution of coverage provided by each database.

Table 4

Total number & percentage of journals indexed by database

Database	N of Journals Indexed	% of total	Incremental n of journals indexed1	Cumulative % of journals indexed	N of zone 1 & 2 journals indexed (of 28)	%	Cumulative % of trials accounted for2
Medline	287	93.2	287	93.2	28	100.0	97.7
Embase	130	42.2	10	96.4	24	85.7	98.8
CAB Health	180	58.4	6	98.4	26	92.3	99.4
CINAHL	36	11.7	3	99.4	7	25.0	99.8
AMED	14	4.5	2	100.0	4	14.3	100.0
CCTR	112	36.4	0	-	18	64.3	-
IBIDS	31	10.1	0	-	16	57.1	-
NCCAM	13	4.2	0	-	5	17.9	-
MANTIS	11	3.6	0	-	3	10.7	-
AGRICOLA	1	0.3	0	-	1	3.6	-
Total			308	100			100

1 The incremental percentage column shows the increase in number of additional journals identified, with each additional database search. For example, searching the CAB Health database, given that the Medline and Embase databases had already been searched, would yield relevant reports of RCTs from six additional journals. Databases are listed in order of the number of journals indexed that were not indexed by databases already considered, thus providing a clearer picture of the cumulative coverage if databases were searched in the order listed, which is the most efficient order for this collection. The cumulative % column shows the percent of journals that published reports of pediatric CAM RCTs that would be covered by searching the databases in the order listed. 2This figure assumes that all trials published in that journal were indexed in the database. The actual indexing rate may be less than 100%, and the indexing status of each report of an RCT in each database was not verified.

Discussion

We identified over 900 reports of RCTs using a CAM intervention in children. We found a steady growth in the literature. We found fewer articles in most recent years, likely due to a lag in indexing. There was an apparent decline in the frequency of reporting from 1998 to 2001. We expected that the indexing of reports of RCTs published during this period would not have been completed by the time we conducted our research in 2001, but post hoc examination of the year by year number of items retrieved from Medline by search strategy 1 shows that some articles in this area can take more than one year to be indexed in bibliographic databases. Those wishing to obtain all journal articles describing randomized controlled trials of pediatric complementary and alternative medicine may wish to consider hand-searching the most recent years of key journals unless it can be determined that recent issues have been indexed by the bibliographic databases.

Our results indicate that most reports (93%) of pediatric CAM RCTs are published in English. This finding has been reported elsewhere [14], although not specifically for pediatrics. Although we had to exclude relatively few trials (n = 41) on the basis of language, and the databases we used provide content for journals published in many languages besides English, it is possible that we missed important clusters of non-English reports. In particular, Tang et al. [15] describe a large number of RCTs of traditional Chinese medicine conducted in China, (approximately 7500 published prior to 1997) as being for the most part inaccessible to Western doctors. However, the journal cited by Tang as having published ten times as many RCTs as any other they examined (Zhongguo Zhong Xi Yi Jie He Za/ Chinese Journal of Integrated Traditional and Western Medicine) is indexed in Medline with entries dating back to 1992. Of 1568 articles from that journal indexed, 284 are tagged as randomized controlled trials in Medline. Of these, only 87 relate to childhood age groups, and all of these have abstracts, thus would be have been picked up by our search and eligible for inclusion in our study. Still, Tang's review underscores the risks of over-reliance on database searching to identify all trials in a given area. For those wishing to search exhaustively, there is potentially great value to involving subject experts who can identify the locations of specialty literatures, such as the literature reporting trials of traditional Chinese medicine described by Tang et al.

The 908 reports were identified in more than 300 journals, yet almost a quarter (24%) of them were found in just four journals (American Journal of Clinical Nutrition, Pediatrics, Journal of Pediatrics, and The Lancet). By expanding the core journal list to 28 journals, it is possible to account for slightly more than half of all reports of pediatric CAM RCTs we found. These results suggest that most reports of randomized controlled trials of CAM are 'mainstream' medical journals. This finding is similar to that reported by Pittler and colleagues [16] who investigated the relationship between the statistical directions of CAM RCT results and where they were likely to be published. They found that mainstream medical journals with an impact factor greater or equal to 1 published an equal number of CAM trials with positive and negative results, while lower impact factor journals published more CAM trials with positive findings.

In terms of database coverage, Medline provided the best coverage of the Zone 1 and 2 journals, indexing all 28. Embase and CAB Health indexed 85.7% and 92.3% respectively. CINAHL and AMED indexed relatively few of the journals in Zones 1 and 2, (25% and 14% respectively) but they did provide unique material not available from the other databases. Together, these five databases gave complete coverage of all journals and reports of RCTs identified in this study. Two other databases, CCTR and IBIDS, provided relatively good coverage of these first two zones (65.3% and 57.1%), but because of overlapping coverage with other databases, provided no unique coverage.

In terms of database selection, a search of Medline alone could potentially identify 97.7% of these reports and a search combining Medline, Embase and CAB Health could achieve 99.4% retrieval, assuming a comprehensive search strategy is used. Given the recent development of the CCTR we were impressed with its coverage although its unique contribution is only marginal in terms of the cumulative increase of identified reports of RCTs resulting from searching this source. In an effort to further enhance the CCTR's coverage we have forwarded the database from this study to the Cochrane Collaboration's CAM field so that the information can be added to the CCTR.

Searching Medline, along with any two of Embase, CCTR, or CAB Health, would appear to constitute a reasonable approach for those wishing to search this topic comprehensively. More exhaustive searching, of the type undertaken for a systematic review, would ideally be supplemented by CINAHL and AMED, as well as selective hand searching and other traditional bibliographic techniques [17].

Those wanting to keep abreast of the latest pediatric CAM reports of RCTs by scanning journals as they are published, may be able to maximize their time by focusing their attention on four journals. Information specialists are likely to be able to track most new evidence by searching three databases.

Shortly after we completed the retrieval work for this project, the National Library of Medicine introduced a complementary medicine subset to its Medline database [18]. Searches can be limited to this subset through PubMed (available at , and this feature is likely to become available through database vendors. This may prove to be a useful tool for clinicians wishing quick access to some of the best literature in this area. PubMed includes Medline, is available at no cost to those with Internet access and can be searched without complex syntax. As a case study, we entered the word random* in the search box, applied the limits of age "All Child: 0–18 years" and subset "Complementary Medicine" and publication type of "Randomized Controlled Trial" retrieved 1858 citations. A clinician replacing the search string random* with a subject term massage would find 56 citations as a starting point, the term chiropract* with the same limits would yield 29, st johns wort or st john's wort would yield 8 reports and herbal would yield 140 citations.

A limitation of this study is that our figures on the cumulative percent of trials accounted for (Table 4) assume that all trials published in that journal were indexed in the database. The actual indexing rate may be less than 100%, and the indexing status of each report of an RCT in each database was not verified in this study. In addition, researchers should be aware that indexing practices and search capabilities available with particular databases vary, thus there is some advantage to building redundancy in coverage into a search strategy that has high recall as an objective.

It should be noted that the Cochrane Controlled Trials Registry is a special case among the databases we searched. As its name suggests, it is a database of controlled trials, including randomized controlled trials, and is constructed from trials found in other databases and identified through hand searching of journals [19,20]. At the time our searches were conducted, it was searchable only through its own proprietary interfaces. It has subsequently become available through at least one database aggregator, Ovid. Had such an interface been available at the time of our searching, we would have searched it using search strategy 1. This could potentially have increased its unique contribution beyond what is seen in Table 4. In a recent study in which known item searching techniques were used to determine if reports of randomized controlled were indexed in various databases, the CCTR performed extremely well, and was identified by the authors as a valuable but underused source for meta-analysts [21].

We used a strict definition of CAM, following the Registry Guidelines adopted by the Cochrane Complementary Medicine Field Registry of Randomized Controlled Trials, thereby excluding some psychological interventions that might be considered CAM by some (such as cognitive therapy) while including others (such as biofeedback). Our view is that some psychological interventions, such as cognitive behavior therapy, might well be considered a 'standard of care' in many Canadian and American settings.

We did not specifically follow a mega-dose definition for classifying nutritional therapies as CAM or conventional medicine. This may have overstated the contribution of the American Journal of Clinical Nutrition. We attempted to follow the Registry Guidelines for the Cochrane Complementary Medicine Field Registry of Randomized Controlled Trials but erred on the side of over-inclusion of therapies as CAM if the author's self-classification was not clear from the title, abstract and indexing information.

It is clear from this study that there is a large body of evidence from randomized controlled trials in pediatric complementary and alternative medicine. The quality of that literature is also relevant [22-24], although a recent paper suggests that quality measures may be less important than previously reported in obtaining unbiased estimates of intervention effectiveness in meta-analysis [25]. Schulz and colleagues reviewed 250 reports of RCTs and found that those with inadequate allocation concealment, compared to reports in which this information was adequately reported, exaggerated the estimates of an intervention's effectiveness by 30%, on average. Recently, Moss and colleagues [26] reported that in a review of 134 pediatric surgical RCTs, less than half of the trials (46%) reported any details of the method of randomization used. Tang et al. notes that although improving, methodological problems remain in the large body of trials of traditional Chinese medicine [15]. We previously reported on the quality characteristics of a subset of the trials found in this project [8].

Conclusions

RCT evidence in pediatric CAM is being published in a substantial number of journals. This study identifies the primary sources of reports of RCTs in the field of pediatric CAM-those journals found in Zone 1. These journals are a necessary core for any collection in this field. The journals in Zone 2 are also important sources of evidence, and are also recommended resources to be included in a pediatric CAM collection. Journals in Zone 3 would be a useful addition for those with larger collection budgets, and are good resources for those needing a comprehensive collection. The journals within Zone 4 contain only one report of an RCT each identified during the time period studied. While not prime candidates for a collection focusing on complementary and alternative medicine, they should be considered by systematic reviewers, who may identify potentially relevant studies from them through the relevant bibliographic databases. This study also indicates that those seeking pediatric CAM evidence require access to at least three indexing databases. These are Medline, Embase, and CAB Health.

List of abbreviations used

RCT – Randomized Controlled Trial, CAM – Complementary and Alternative Medicine

Abbreviations

CINAHL – Cumulative Index to Nursing and Allied Health

AMED – Allied and Complementary Medicine

CCTR – The Cochrane Controlled Trials Register

IBIDS – International Bibliographic Information on Dietary Supplements

MANTIS – Manual, Alternative and Natural Therapy

NCCAM – The National Center for Complementary and Alternative Medicine

AGRICOLA – AGRICultural OnLine Access

Competing interests

None declared.

Authors' contributions

Margaret Sampson participated in the design of study, development of search strategy, data extraction, statistical analysis, and writing of the manuscript. Kaitryn Campbell executed the search strategy, evaluated all trials for eligibility, performed data extraction, statistical analysis and writing of the manuscript. Isola Ajiferuke participated in the design of the study, supervised the statistical analysis and participated in the preparation of manuscript. David Moher conceived of the study, secured funding, participated in the design of the study, advised in the statistical analysis and participated in the preparation of the manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here: