Assessment of the quality of reporting in randomised controlled trials of acupuncture in the Korean literature using the CONSORT statement and STRICTA guidelines.

OBJECTIVES: This study aims to assess the completeness of reporting of randomised controlled trials (RCTs) of acupuncture in the Korean literature.
DESIGN: Systematic review.
METHODS: We searched 12 Korean databases and 7 Korean journals to identify eligible RCTs of acupuncture published from 1996 to July 2011. We used the Consolidated Standards of Reporting Trials (CONSORT) checklist for parallel RCTs and the revised STandards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA) to assess the quality of reporting in Korean RCTs. We compared the completeness of reporting of CONSORT and STRICTA items in RCTs published in two time periods (1996-2004 referred to as the early period and 2005-2011 referred to as the late period).
RESULTS: We analysed 146 eligible RCTs using the CONSORT statement concerning RCTs of both needling and non-needling acupuncture and the STRICTA guidelines for 90 trials of needling acupuncture. Among the 103 RCTs in the late period, the proportion of RCTs that completely reported the CONSORT items of outcome definition (15.5%), sample size calculation (2.9%), randomisation (56%), allocation concealment (5.8%), implementation of allocation (11.7%), outcome assessor blinding (20.3%), flow of participants (25.2%), number of participants analysed (19.4%), ancillary analyses (0.0%), adverse events (24.3%), generalisability of findings (1.9%) and overall evidence (32.0%) remained small. Among the 61 RCTs of needling acupuncture in the late period, the STRICTA items of setting/context (24.6%) and practitioner background (27.9%) showed incomplete reporting. The completeness of reporting improved over time in several CONSORT and STRICTA items.
CONCLUSIONS: The completeness of reporting of Korean RCTs of acupuncture was suboptimal according to the CONSORT and revised STRICTA statements. Trial authors and journal editors should use the CONSORT statement and STRICTA guidelines for transparent reporting in Korean RCTs of acupuncture. The endorsement of the CONSORT and revised STRICTA statements in author instructions is also required. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

This study is the first to investigate the completeness of reporting in Korean RCTs of acupuncture with regard to the CONSORT and STRICTA recommendations and their changes over time.

Our findings indicate that the majority of the core components of trials remained substantially under-reported in Korean RCTs of acupuncture.

The assessment criteria for the completeness of reporting in each item may differ from other relevant reviews. Future periodical updates of the results are warranted.

Background

Rigorous randomised controlled trials (RCTs) can reduce bias and thus contribute to the establishment of gold-standard evidence for medical interventions. A complete, accurate and transparent report of RCTs facilitates dissemination, interpretation, translation and replicability, whereas incomplete reporting of RCTs impedes the reliability of evidence.1 An international group of clinical trialists, statisticians, epidemiologists and biomedical journal editors developed the Consolidated Standards of Reporting Trials (CONSORT) statement to improve the reporting of RCTs, thus enabling readers to understand the study design, conduct, analysis and interpretation through complete transparency.2 3 The STandards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA), which were developed by international experts of acupuncture and trialists in 2001 and revised in 2010, serves as an official extension of the statement for descriptions of acupuncture treatments.1 Although these statements aimed to improve the reporting of RCTs, a recent Cochrane review suggested that the completeness of reporting remained suboptimal.3 There was also the incompleteness of reporting of treatment details based on STRICTA checklist items in RCTs of acupuncture published in English, which suggests that future research should investigate the completeness of reporting in acupuncture trials in languages other than English.4 To the best of our knowledge, no study has investigated the completeness of reporting of Korean RCTs based on CONSORT and STRICTA checklist items in acupuncture research fields, although acupuncture is regularly practised in Korea and many clinical trials assessing the effects of acupuncture have been published in the Korean language.5 Hence, we aimed to evaluate the completeness of reporting of RCTs of acupuncture indexed in the Korean literature based on the CONSORT and revised STRICTA statements. We anticipate that this study will reveal the current status of the completeness of reporting in RCTs of acupuncture indexed in the Korean literature. This study will thus provide information for facilitating transparent and more complete reporting in RCTs of acupuncture.

Methods

Study design

The primary aim of this study was to identify the current weakest components of reporting based on the CONSORT statement and STRICTA guidelines. Another aim was to investigate whether the completeness of reporting of RCTs had improved over time. We stratified the RCTs based on the publication year. Trials that were published before 2005 and from 2005 were grouped as those published in the ‘early’ period and the ‘late’ period, respectively. We set the cut-off year as 2005 because we expected that it would take at least 4 years for trial authors to be aware of and use the revised CONSORT statement and the first STRICTA guidelines that were published in 2001. The RCTs published in the ‘late’ period were used to identify the current status of reporting. We compared the RCTs in the ‘early’ period with those in the ‘late’ period to assess the changes of completeness of reporting over time.

Type of studies

Parallel group RCTs of acupuncture listed in Korean databases and published either in Korean or in English languages were eligible for inclusion in the present study. The Korean RCTs listed in English databases were not eligible because the primary aim of this study was to assess adherence to the CONSORT statement among trials in the Korean literature that might be unknown due to language restriction or inaccessibility to the databases. We excluded crossover or cluster RCTs because we employed the CONSORT guidelines for parallel RCTs.

Type of participants

We used RCTs that comprised patients who had any type of health problems or diseases. We excluded RCTs that comprised healthy individuals.

Types of interventions and comparisons

For the CONSORT analyses, we defined acupuncture as a stimulation of the body or auricular points regardless of the type of stimulation. We included studies using acupuncture-related interventions that stimulate acupuncture points (ie, acupuncture point injection or acupressure) when classified and reported as a type of acupuncture in an RCT. We included such studies because there is a diverse range of methods for acupuncture point stimulation that is classified as a subtype of acupuncture in Korea.6 Studies that combined acupuncture with moxibustion were eligible when they used moxibustion as one of the cointerventions of acupuncture. Studies testing moxibustion as a primary intervention were excluded. Any type of control group intervention was eligible.

For the STRICTA analyses, acupuncture was defined as needle penetration of the body or auricular points using manual and electrical stimulation because the STRICTA guidelines were originally developed to report the components of needling acupuncture. RCTs comparing acupuncture as a control group intervention with other types of treatments were also eligible; only acupuncture-related information was extracted for the STRICTA analysis. Studies that compared different types of needle-penetration acupuncture interventions were also eligible; the most comprehensively described acupuncture intervention was extracted for the STRICTA analysis.

Search methods

The studies included were selected from the data set of Korean RCTs previously described.7 In that published study, 12 Korean databases (ie, NANET, RISS, KISS, DBpia, KMbase, KoreaMed, KISTI, NDSL, OASIS, Dlibrary, KoreanTK and RICHIS) were searched from their inception to July 2011. Simple search terms and strategies were used (table 1). Theses and dissertations that were accessible from four databases (NANET, RISS, Dlibrary and RICHIS) were included if they met the eligibility criteria.

Table 1

Search terms used in titles and abstracts

	English search term
#1 Acupuncture related	acupuncture OR acupressure OR acupoint OR meridian OR acup*
#2 Design related	Random OR control OR group OR divide
#3	#1 AND #2
	Korean search term
#1	침 OR 경혈 OR 경락
#2	대조군 OR 무작위
#3	# 1 AND # 2

Data extraction

One author (KHK) extracted general characteristics of the included RCTs, such as publication year, type of acupuncture and control intervention, type and scope of journals, number of arms and sample size. We used the CONSORT statement for parallel RCTs revised in 2001 to assess the completeness of reporting of RCTs. Although the CONSORT initiatives recommended the use of the most recently released version of CONSORT when reporting and analysing RCTs,2 we did not use the revised CONSORT published in 2010 to avoid potential systematic disadvantages for RCTs that were published before 2010.4 We used the revised version of STRICTA guidelines published in 2010 to analyse the completeness of reporting of treatment components of acupuncture in RCTs. We believe there was sufficient consistency among the STRICTA guidelines between the original and revised versions, thereby justifying the use of the latest version.8 We did not extract the data regarding non-acupuncture interventions of a control group, because our primary interest was to assess the completeness of reporting of acupuncture treatment. Two authors (KHK and JWK) converted the CONSORT statement and STRICTA guidelines into 22 and 15 checklist items for data extraction and assessment, respectively. The checklists of each statement provided in the CONSORT and STRICTA web pages served as the primary sources of data extraction and assessment sheets.2 9 Each item had equal weight. Two authors (KHK and JWK) independently assessed the completeness of reporting in each item. Any disagreements were resolved with discussion.

Two authors (KHK and JWK) rated each item as ‘yes’ or ‘no’ based on whether it was reported in the study. For an item that contained multiple subitems, the reporting of the item was considered to be complete when at least one subitem was completely reported. For example, item #8 in the CONSORT statement assesses the random sequence generation with two subitems (ie, the method used for generating the random allocation sequence, including details of any restrictions). The reporting of the item was considered to be complete when the sequence generation method was reported regardless of the information of restriction method provided in a given RCT. We used the explanation and elaboration documents of the CONSORT10 and the revised STRICTA1 as assessment references. For item #4 in the CONSORT statement (ie, details of the intervention intended for each group and how and when they were actually administered) we adopted and slightly modified the rating criteria of Hoffmann et al.11 We selected four items (ie, procedure, materials, intensity and schedule) from the checklist11 that was developed to assess the reports of non-pharmacological interventions in RCTs because we considered these items to be most relevant for the replication of acupuncture interventions. If all of the four items were assessed as ‘yes’, we rated item #4 as ‘yes’. Table 2 presents the detailed assessment criteria for item #4. Item #11 in the CONSORT statement (ie, blinding of participants, intervention providers and outcome assessors) was modified to include only the outcome assessor blinding. Participant and intervention provider blinding is often not feasible in complex interventions, such as acupuncture,12 whereas outcome assessors can be blinded without interfering with the acupuncture treatment process. We rated item #18 (ie, reporting of ancillary analyses) as ‘yes’ only if a RCT reported the results of ancillary analyses (eg, subgroup analyses) with the notion whether those analyses were prespecified, based on the elaboration document of the CONSORT statement.10 Otherwise, the item was rated as ‘no’.

Table 2

Assessment criteria of the item #4 of the CONSORT 2001 statement

Criteria item	Components of respective STRICTA items
Procedure	At least one item should be reported for the ‘yes’ assessment
	De-qi response sought (item 2d)
	Needle stimulation methods (item 2e)
	Names of points (item 2b)
Materials	Needle specification (item 2g)
Intensity	Needle retention time (item 2f)
Schedules	At least one item should be reported for the ‘yes’ assessment
	Duration (item 3b)
	Frequency (item 3b)
	Total or average number of sessions (3a)

The STRICTA guidelines recommended reporting acupuncture treatments that were actually provided because what was in fact administered may likely have differed from the predefined treatment protocol.1 We assumed the reporting of acupuncture interventions in RCTs as performed because the distinctions were not clear in most cases.8 We calculated the CONSORT and STRICTA index scores to summarise the overall completeness of reporting in one item by summing the scores of 22 items of the CONSORT checklist and 15 items of the STRICTA.13

Statistical analysis

For each time period (ie, early and late), general publication details (eg, sample size, type of journals, acupuncture/non-acupuncture trials) were compared using t tests or χ2 tests. For each CONSORT and STRICTA item, the number and percentage of trials that completely reported the item and mean differences of percentages between two time periods with binomial 95% CI were reported. The STATA V.13.0 (Stata-Corp, College Station, Texas, USA) was used for statistical analyses.

Results

Search results and characteristics of included RCTs

In total, 146 of 227 screened RCTs were included (figure 1). Forty-three and 103 RCTs were published in the early period (1996–2004) and the late period (2005–2011), respectively. The primary interventions were needle acupuncture and non-needle acupuncture point stimulation in 90 and 56 RCTs, respectively. The types of non-needle acupuncture point stimulation were pharmacopuncture (ie, the injection of herbal medicine), bee venom acupuncture (ie, the injection of diluted bee venom), and acupressure (by hand or device) or non-penetrating electrostimulation to acupuncture points. General characteristics of the included RCTs are shown in table 3.

Figure 1

Flow chart of the trial selection process (RCT, randomised controlled trial).

Table 3

General characteristics of included RCTs

	Early period (1996–2004) (n=43)	Late period (2005–2013) (n=103)
Total number of included studies
CONSORT analysis	43	103
STRICTA analysis	29	61
Type of journals
Peer-review journals	43 (100.0%)	97 (94.2%)
Unpublished (master dissertation or PhD thesis)	0 (0.0%)	6 (5.8%)
Scope of journals
Acupuncture-related	34 (79.1%)	93 (90.3%)
Others*	9 (20.9%)	10 (9.7%)
Sample size†	45.6 (49.5)	41.3 (19.4)
Publication year‡	2002 (1996–2004)	2007 (2005–2011)
Type of intervention
Needle acupuncture	29 (67.4%)	61 (59.2%)
Non-needling acupuncture§	14 (32.6%)	42 (40.8%)
Type of control
Active treatment	35 (81.4%)	70 (68.0%)
Sham or placebo	5 (11.6%)	31 (30.1%)
Waitlist	3 (7.0%)	2 (1.9%)
Number of arms
2 arms	39 (90.7%)	92 (89.3%)
3 or 4 arms	4 (9.3%)	11 (10.7%)

*Others refer to journals that are not related with acupuncture and unpublished dissertation/thesis.

†Values are presented as mean (SD).

‡Values are presented as median (range).

§Non-needling acupuncture includes pharmacopuncture, bee venom acupuncture and other acupuncture point stimulation using non-penetrating techniques.

Completeness of reporting for the CONSORT items

Of 103 RCTs published in the late period, there was considerable incompleteness of reporting in items related to the study design, implementation, reporting and interpretation (figure 2). Items with markedly incomplete reporting were allocation concealment (item #9; 5.8%), implementation of allocation process (item #10; 11.7%), definition of primary/secondary outcomes (item #6; 15.5%), methods of sample size calculation (item #7; 2.9%), blinding of outcome assessors (item #11; 20.3%), participant flow (item #13; 25.2%), number of participants analysed (item #16; 19.4%), ancillary analyses (item #18; 0%), adverse events (item #19; 24.3%), generalisability of the study findings (item #21; 1.9%) and overall evidence (item #22; 32.0%).

Figure 2

Percentage of RCTs with complete reporting of the CONSORT items. # and number in parentheses refer to the item number. Green dash in the figure indicates a reference line of 50%. Blue and red bar represent the percentages of randomised controlled trials (RCTs) with complete reporting in the early and late periods, respectively.

Item #8 (random sequence generation) showed the most salient improvement over time (mean difference 42.7%; 95% CI 29.2% to 56.3%), although the completeness of reporting remained modest in the late period (54.4%). The CONSORT index score was significantly increased in the late period RCTs compared with the early period RCTs (p=0.0082) (table 4). The detailed information regarding the reporting of each item is shown in online supplementary appendix 1.

Table 4

Comparison of the CONSORT/STRICTA index between early and late periods

	N	Mean	95% CI	p Value
CONSORT
Early period (1996–2004)	43	9.5	8.9 to 10.2
Late period (2005–2011)	103	10.6	10.2 to 11.1
Difference		1.1	0.2 to 1.9	0.0082
STRICTA
Early period (1996–2004)	29	8.8	7.9 to 9.7
Late period (2005–2011)	61	11.2	10.5 to 11.8
Difference		2.3	1.2 to 3.4	<0.0001

Completeness of reporting for the STRICTA items

We found that 61 RCTs of needle acupuncture interventions published in the late period completely reported 9 of 15 items, with more than 70% of reporting rates (figure 3). The reporting of items for setting/context (24.6%) and practitioner background (27.9%), however, remained incomplete in the late period. In all of the items, the completeness of reporting had improved over time; the most prominent improvements of the completeness of reporting were evident in items related to depth of needle insertion (item #6; mean difference 36.2%; 95% CI 15.6% to 56.7%), response to needle stimulation (item #7; mean difference 26.7%; 95% CI 6.7% to 46.7%) and the methods of acupuncture stimulation (item #8; mean difference 23.5%; 95% CI 2.7% to 44.3%). The STRICTA index score was significantly increased in the late period RCTs compared with the early period RCTs (p<0.0001) (table 4). The detailed information regarding the reporting of each item is provided in online supplementary appendix 2.

Figure 3

Percentage of RCTs with complete reporting of STandards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA) items. # and number in parentheses refer to the item number. Green dash in the figure indicates a reference line of 50%. Blue and red bar represent the percentages of randomised controlled trial (RCTs) with complete reporting in early and late period, respectively.

Discussion

We identified a considerable number of items that were incompletely reported, which may limit the assessment of internal validity and applicability of the trial results. Our findings are consistent with those of previous studies that investigated the adherence of RCTs to CONSORT and STRICTA statements in various clinical fields, including acupuncture.13–18 The reporting of the CONSORT items that are core components for assessing the risk of bias, such as random sequence generation (item #8), concealment of allocation (item #9), outcome assessor blinding (item #11) and flow of participants through each stage of trials (item #13), were particularly incomplete. None of the Korean RCTs reported the critical criteria of subgroup analyses (item #18) including the use of subgroup variables measured at baseline, the prespecification of subgroup hypotheses, and the statistical significance of interaction tests. The side effects of study interventions (item # 19) were reported in less than 25% of the included RCTs, which does not permit the investigation of acupuncture safety in the context of trials. Discussions regarding the generalisability of trial findings (item #21) were lacking in the majority of the Korean RCTs, which may interfere with the application of trial results to actual clinical situations. Overall, the completeness of reporting of Korean RCTs of acupuncture was suboptimal, which could represent a significant obstacle to the establishment of a sound evidence base.

Regarding the completeness of reporting of intervention details based on the revised STRICTA statements, the theoretical background (items #1–3) and several details of the needling processes (items #4, #5 and #9–12) showed relative completeness of reporting, whereas the items related to contextual factors (items #13 and #14) were markedly under-reported. The inconsistent completeness of reporting among items may imply that certain items are perceived to be less important by researchers or journal editors,19 although there is no evidence to justify such inference in the Korean context. Practitioner qualifications were also incompletely reported (item #15), which may increase the uncertainty with regard to treatment quality and safe implementation of interventions. Collectively, the completeness of reporting of acupuncture details was inconsistent, which may be problematic for replicating acupuncture treatments in other contexts.

Our findings indicate that the STRICTA items were generally more completely reported than the CONSORT items because the subject of included studies was acupuncture. Another likely explanation refers to the advantages from the translated Korean version of STRICTA guidelines,20 21 whereas no official Korean translation of the CONSORT statement exists. Currently, the official translated version of the CONSORT statement is available in 11 different languages22 to assist local authors in the comprehensive reporting of the recommended trial components according to international standards.23 24 Consistent with these international efforts, the official Korean translation of the CONSORT statement is expected to be a useful resource for Korean authors and journal editors.

In a study that is being prepared separately, only 1 of 36 traditional Korean medical journals (ie, Journal of Acupuncture and Moxibustion Society) endorsed the CONSORT statement and STRICTA guidelines as a component of the author instructions (unpublished). A recent Cochrane review found that four CONSORT items (ie, allocation concealment, introduction, sample size and random sequence generation) and a total sum score of 22 CONSORT items were significantly more completely reported in RCTs favouring CONSORT-endorsing journals over the non-endorsers.3 The results are consistent with our findings that the most incomplete reporting was in items related to allocation concealment, sample size and random sequence generation. The absence of endorsement of the CONSORT statement and STRICTA guidelines in the majority of Korean journals may be a potential factor of incompleteness of reporting in Korean RCTs of acupuncture.

Strengths and limitations

The present study is the first systematic investigation to assess the completeness of reporting of Korean RCTs of acupuncture based on the CONSORT statements. We attempted to evaluate the best available data set of Korean RCTs of acupuncture by employing extensive search strategies that targeted 17 Korean databases; however, the likelihood of undetected studies could not be completely excluded. Two recent articles analysed study characteristics and bibliographic information using updated search results and database information.25 26 Our finding needs periodic updates to monitor whether the items of the CONSORT statement and STRICTA guidelines are transparently and completely reported in Korean RCTs of acupuncture. The weaknesses in our study should be mentioned. First, the cut-off year of the CONSORT and STRICTA analysis was arbitrarily defined because there are no standard criteria regarding a sufficient period for disseminating and implementing the CONSORT and STRICTA recommendations. Although we assumed that local authors would require at least 4 years to implement the CONSORT and STRICTA recommendations into the design and writing of a trial, no concrete evidence justifies our assumption. Second, we considered the partial reporting of CONSORT and STRICTA items that had multiple subitems as complete despite the fact that the recent Cochrane review considered only full descriptions of the required content for a given item as complete.3 Consequently, our assessment may have reported inflated scores and our findings should be considered as results under the best-case scenario. Third, time may serve as a potential confounder for the completeness of reporting, which may have naturally changed over time regardless of the use of the CONSORT statement and STRICTA guidelines by trial authors.3 Whether trial authors referred to the CONSORT statement and STRICTA guidelines was not investigated in this study; thus, we do not know that more complete reporting in several items and the higher aggregate scores might be related either to the natural improvement over time or to the influences of these resources on the trial reporting by Korean researchers.

Implications for future research

Collective efforts for increasing adherence to the CONSORT statement and STRICTA guidelines during design, implementation and reporting of clinical trials are needed to improve the completeness of reporting in Korean RCTs of acupuncture. In particular, the items related to randomisation, concealment of allocation, blinding of outcome assessors, flow of participants, subgroup analyses and harms of acupuncture should be more completely reported in future Korean RCTs of acupuncture. There should be improved reporting of practitioner-related and context-related STRICTA items. Barriers related to the incompleteness of reporting by Korean authors of RCTs should also be investigated. The endorsement of the CONSORT statement and STRICTA guidelines in relevant Korean medical journals is urgently required. All of the stakeholders (editors, peer-reviewers and authors) should promote the use of these resources during the manuscript preparation, submission and peer-review processes. Relevant education is necessary. The official version of the Korean translation of the CONSORT statement is required to increase the accessibility of international trial reporting guidelines by Korean researchers.

Conclusions

The completeness of reporting in Korean RCTs of acupuncture has remained suboptimal over time. Trial authors and journal editors should use the CONSORT statement and STRICTA guidelines for transparent reporting of Korean RCTs of acupuncture.