Literature DB >> 29272315

Randomized clinical trials in dentistry: Risks of bias, risks of random errors, reporting quality, and methodologic quality over the years 1955-2013.

Humam Saltaji¹, Susan Armijo-Olivo², Greta G Cummings³, Maryam Amin¹, Carlos Flores-Mir¹.

Abstract

OBJECTIVES: To examine the risks of bias, risks of random errors, reporting quality, and methodological quality of randomized clinical trials of oral health interventions and the development of these aspects over time.
METHODS: We included 540 randomized clinical trials from 64 selected systematic reviews. We extracted, in duplicate, details from each of the selected randomized clinical trials with respect to publication and trial characteristics, reporting and methodologic characteristics, and Cochrane risk of bias domains. We analyzed data using logistic regression and Chi-square statistics.
RESULTS: Sequence generation was assessed to be inadequate (at unclear or high risk of bias) in 68% (n = 367) of the trials, while allocation concealment was inadequate in the majority of trials (n = 464; 85.9%). Blinding of participants and blinding of the outcome assessment were judged to be inadequate in 28.5% (n = 154) and 40.5% (n = 219) of the trials, respectively. A sample size calculation before the initiation of the study was not performed/reported in 79.1% (n = 427) of the trials, while the sample size was assessed as adequate in only 17.6% (n = 95) of the trials. Two thirds of the trials were not described as double blinded (n = 358; 66.3%), while the method of blinding was appropriate in 53% (n = 286) of the trials. We identified a significant decrease over time (1955-2013) in the proportion of trials assessed as having inadequately addressed methodological quality items (P < 0.05) in 30 out of the 40 quality criteria, or as being inadequate (at high or unclear risk of bias) in five domains of the Cochrane risk of bias tool: sequence generation, allocation concealment, incomplete outcome data, other sources of bias, and overall risk of bias.
CONCLUSIONS: The risks of bias, risks of random errors, reporting quality, and methodological quality of randomized clinical trials of oral health interventions have improved over time; however, further efforts that contribute to the development of more stringent methodology and detailed reporting of trials are still needed.

Entities: Disease Gene Species

Mesh：

Year: 2017 PMID： 29272315 PMCID： PMC5741237 DOI： 10.1371/journal.pone.0190089

Source DB: PubMed Journal: PLoS One ISSN： 1932-6203 Impact factor: 3.240

Introduction

Randomized clinical trials are the ideal type of clinical research to examine the effectiveness of treatment interventions in health sciences [1]. The value and significance of a randomized clinical trial depends on the control of potential biases, how rigorously it was conducted, and how thoroughly the results were reported. The SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) Statement [2], the Consolidated Standards of Reporting Trials (CONSORT) Statement [3], and recent initiatives such as the International Committee of Medical Journal Editors (ICMJE) statement on clinical trial registration [4], have led to improvements in both the methodological and reporting quality of medical randomized clinical trials [5-7]. Adhering to these initiatives is critical to oral health research and practice, as high quality randomized clinical trials contribute largely to the body of evidence measured in systematic reviews and meta-analyses, especially when assessing therapeutic interventions. Currently, nearly 50 clinical trials of oral health interventions are estimated to be published every month, and this number is expected to increase over time [8]. Emerging evidence from methodological reports published in various dentistry-related specialties over the last decade (periodontics [9], prosthodontics [10], implantology [11], orthodontics [12], restorative dentistry [13], and six dental specialties [14]) report methodological quality of oral heath randomized clinical trials that is below acceptable levels to adequately lead clinical decision making. Moreover, there is evidence that some trials are biased and, due to weaknesses in their methodological characteristics, they tend to exaggerate the magnitude of the treatment effect [15]. This emerging evidence raises questions about the validity of trial results for oral health interventions, which dental practitioners use when making day-to-day clinical decisions in dental practice, and which policy makers use more generally when developing clinical practice guidelines. In the context of medical research methodology, the internal validity of a trial is dependent on “risk of bias” (which concerns the internal validity of a trial) [16] and “risks of random errors” (i.e., risks of play of chance). Moreover, while “reporting quality” refers to the reporting of the conduct and design of a trial [17, 18], “methodological quality” is related to the internal validity of a trial and determined by the extent to which the conduct and design of a trial are precisely and rigorously performed to generally acceptable standards so that biases are minimized. In the field of oral health, to our knowledge, no study has assessed changes over time regarding quality of reporting, methodological characteristics, and risks of bias in randomized clinical trials of oral health interventions. A recent report by Reveiz et al. [19] described the results of an examination of different risk of bias domains in a sample of medical randomized clinical trials (identified from a cohort of Cochrane reviews). Reveiz and colleagues’ report stated that the rate of trials found at low risk of bias consistently increased with time. However, since the randomized clinical trials in Reveiz and colleagues’ sample were performed in the field of medicine and were dependent on the risk of bias assessment performed by investigators presenting published reviews (rather than by conducting standardized data extraction from each trial), the findings cannot be compared to findings from trials in the field of dentistry which tend to have different design characteristics, such as difficulty in applying blinding and common use of the split-mouth design. Consequently, it is unclear if the increase in number of published randomized clinical trials of oral health interventions over time has been associated with changes in the conduct and reporting of the trials. In this study, we set out to assess these aspects in randomized clinical trials of oral health interventions, and if these aspects have improved over time. Our objectives were to (1) examine the risks of bias, risks of random errors, reporting quality, and methodological quality, and the general trial characteristics of randomized clinical trials of oral health interventions; and (2) determine whether (and to what extent) risks of bias, risks of random errors, reporting quality, and methodological quality have improved over time.

Methods

Study sample

We used the Oral Health Database of Systematic Reviews [20, 21] which includes a comprehensive selection of oral health systematic reviews published in the field of oral health research between 1991 and 2014. From this database, we selected a sample of systematic reviews with meta-analyses and their associated randomized clinical trials that met the following criteria: the meta-analysis was (1) published in any language and (2) conducted in an oral health field that examined an intervention concerning craniofacial, oral, or dental diseases (as defined by the American Dental Association [ADA] scope of practice) [22]. A randomized clinical trial was defined as “an experiment in which two or more interventions (possibly including a control intervention or no intervention) are compared, by being randomly allocated to participants” [16]. Further details regarding the study selection included in the final database of systematic reviews have been published [20, 21] (see ). Briefly, two reviewers (dentists with oral health research backgrounds) independently selected relevant reports and determined the final eligibility of the full texts (any disagreements were resolved through consensus). Ultimately, 540 randomized clinical trials that met the predefined eligibility criteria were selected and utilized in this study. The list of included randomized clinical trials is provided in . The study protocol was not registered or published in advance.

Data extraction

A panel of five reviewers from diverse health research areas (dentistry, pediatrics, and physical therapy) extracted the data. To ensure consistency during data extraction, two team members (H.S., S.A.) facilitated a reviewer training process, similar to the process followed in our team’s previous investigations [23, 24]. In this process, the review panel evaluated and discussed 10 randomized clinical trials not included in the final set of trials. Once agreement on data extraction protocols and interpretation was achieved, the review panel performed data extraction in duplicate. Two assessors independently carried out data extraction for each included randomized clinical trial (consensus meetings were employed to resolve any disagreements). One assessor (H.S.) who has a background in oral health research performed a complete data extraction (n = 540, 100%), while other members of the review panel (C.H., J.S., J.F.) who have medical (non-oral health) research backgrounds, acted as secondary assessors. If two assessors could not reach an agreement, then a third assessor (S.A.) assisted with consensus. Only data that received consensus were used for data analyses. We used a structured and pilot-tested data extraction template, designed using Microsoft Office Access, for data extraction. We extracted details from each of the selected randomized clinical trials with respect to publication and trial characteristics, risks of bias, risks of random errors, reporting quality, and methodological quality, as described below.

Publication and trial characteristics

Data elements related to publication and trial characteristics included the following information: publication year, dental speciality as classified by the American Dental Association (ADA) (e.g., periodontics, dental public health, prosthodontics and restorative dentistry, oral medicine and oral pathology, implantology, oral and maxillofacial surgery, orthodontics and dentofacial orthopedics, pediatric dentistry, endodontics [22]), country and continent of first author, number of authors, funding source (e.g., foundation, government, industry, academic), type of journal (e.g., specialty oral health, general oral health, nonoral health), type of intervention (e.g., surgical, drug, dental material, device, psychological, educational, policy), age of participants, number of centres (e.g., multicentres, single centre), design (e.g., parallel, crossover, split-mouth, cluster), type of outcome (e.g., subjective, objective), and sample size.

Risks of bias

We employed the Cochrane risk of bias tool introduced in 2008 [1], which contains six domains and seven items, namely, “sequence generation,” “allocation concealment,” “blinding of outcome assessors,” “blinding of participants,” “incomplete outcome data,” “selective outcome reporting,” and “other sources of bias.” We used the Cochrane Collaboration guidelines to score domains (e.g., high, low, unclear). However, we developed specific rules to make final decisions (see and . For “other sources of bias,” we examined baseline comparability, control for cointerventions, whether treatment compliance was acceptable, and funding [25]. For the overall assessment of risk of bias, if one domain was assessed as having a high risk, the overall risk of bias assessment was labelled “high risk.” A randomized trial was considered to be at low risk of bias if it was assessed as “low risk” in all individual domains. If the assessment was “unclear” in at least one domain (and other domains were unclear or low) the overall risk of bias assessment was designated “unclear” [26, 27].When reporting the results of the risk of bias assessment, we combined “high” and “unclear” risk of bias assessments into one type of trials, considering that trials that these types of trials are at risks of overestimating benefits and at risk of underestimating harms [1, 14, 25].

Risks of random error

Risks of random errors involves two criteria: (1) sample size calculation and (2) adequate sample size (see ).

Reporting quality and methodologic quality

Reporting quality and methodological quality are difficult to distinguish and often overlap to some extent. Methodological quality is defined as “the confidence that the trial design, conduct, and analysis has minimized or avoided biases in its treatment comparisons” [17, 18] (e.g., the sequence generation was appropriate). Reporting quality involves the provision of “information about the design, conduct, and analysis of the trial” [17, 18] (e.g., this was a randomized trial). Accordingly, based on preliminary work performed by the research team [28, 29], we obtained 40 quality assessment criteria and their classifications (“reporting” vs. “conduct”) from the most commonly used quality assessment tools in health care research [30-37]. Of the 40 quality criteria selected, 15 criteria assess “reporting” quality, 21 criteria assess “methodological” quality, and four quality criteria assess both reporting quality and methodological quality. We classified the items that evaluated methodological quality according to type of bias as follows [28, 29] (see ): selection bias (6 criteria), performance and detection bias (4 criteria), performance bias (9 criteria), performance and compliance bias (2 criteria), information bias (3 criteria), reporting bias (3 criteria), attrition bias (5 criteria), detection bias (2 criteria), statistical bias (1 criterion), threats to precision (1 criterion), and multiple biases (2 criteria). We also grouped the selected quality criteria according to the following categories [16]: patient selection (inclusion and exclusion criteria, description of participants); assignment, randomization, and allocation concealment; blinding; interventions; attrition, follow up and protocol deviation; outcomes; and statistical analysis. Using original tools as guidelines, the definitions and methods were derived for each criterion, using a three-part answering scheme (yes, no, unclear) for each item. We established decision rules and guidelines to ensure consistency (see and ).

Data analysis

We conducted descriptive analyses for each trial characteristic, quality assessment item, and risk of bias domain (using means and standard deviations [SD] or median and interquartile range [IQR] for continuous outcomes, and proportions and percentages for categorical outcomes, where appropriate). To evaluate whether the quality of randomized clinical trials has improved over years, trials were grouped according to four periods of publication year: before 1990, 1990–1999, 2000–2006, 2007–2013. We used Chi-square statistics and two-tailed Fisher exact tests to examine the difference in proportion with respect to time periods for all quality assessment items and risk of bias domains. Furthermore, we used a logistic regression to explore the relationship between each criterion and time; we entered time into the logistic regression model as a continuous variable (publication year) and a categorical variable (time period of publication year; < 1990 was used as a reference category). The outcome of each analysis was each methodological criterion dichotomized in low risk vs. others (unclear, high risk of bias), or yes vs. others (no, unclear, not-reported). We reported odds ratios (OR) with 95% confidence intervals (CIs) and we set the statistical significance at P < .05. We performed statistical analyses using Stata Version 14.0 (StataCorp) and the Statistical Package for Social Sciences for Windows (SPSS) Version 18.0 (IBM, Armonk, NY).

Results

Trial publication and characteristics

From the selected 540 trials published between 1955 and 2013 (median year of publication: 2000; IQR: 1990, 2007) (see ), the majority of trials were published either in periodontics (n = 233; 43.1%), dental public health (n = 124; 23.0%), or prosthodontic and restorative dentistry (n = 54; 10.0%). More than half of the trials were published in journals that specialized in oral health (n = 304; 56.3%) (see ). The trials’ first authors were most frequently from Europe (n = 239; 44.3%) followed by North America (n = 202; 37.4%). Three countries (the United Kingdom, Italy, and the United States) accounted for almost half of all trials (n = 280; 51.9%). Approximately one fifth of the trials were multicenter trials, nearly half of the trials involved four to six authors (n = 249; 46.1%), and one third included two to three authors (n = 169; 31.3%). In approximately half of the trials, the authors did not declare whether they received a source of funding (n = 256; 47.4%), while nearly one third of trials received funding from industry (n = 171; 31.7%). Approximately one third of the trials were placebo-clinical (n = 204; 37.8%) and two thirds examined nondrug (n = 359; 66.5%) or nonsurgical (n = 370; 68.5%) interventions. One quarter of the trials examined pediatric patients (n = 136; 25.2%), while the majority examined adults (n = 398; 73.7%). The majority of trials used parallel design (n = 372; 68.9%), while almost one quarter used the split-mouth design (n = 126; 23.3%).

Changes in risks of bias, risks of random errors, reporting quality, and methodological characteristics over time

Sequence generation was assessed to be adequate (low risk of bias) in 32% (n = 173) of the trials, while it was inadequate (unclear or high risk of bias) in 68% (n = 367) of the trials. Allocation concealment was inadequate (unclear or high risk of bias) in the majority of trials (n = 464; 85.9%). Blinding of participants was judged to be adequate (at low risk of bias) in 71.5% (n = 386) of the trials, and blinding of the outcome assessment was judged to be adequate (at low risk of bias) in 59.4% of the trials. Other sources of bias—baseline comparability, similarity of cointerventions, and compliance to treatment—were judged at low risk of bias in 77.8%, 40.2%, and 53.5% of the trials, respectively. The influence of the trial sponsor was assessed as being inadequate (unclear or high risk of bias) in 83.3% (n = 450) of the trials, while it was assessed as appropriate in 16.7% (n = 90) of the trials. The overall risk of bias was inadequate (unclear or high risk of bias) in 94.6% (n = 511) of the trials, and an overall low risk of bias was assessed in only 5.4% (n = 29) of the trials (see ). † Does not equal 100% for overall, as the item was not applicable in 310 trials. §Does not equal 100% for overall, as the item was not applicable in 36 trials. Regarding risks of random errors, a sample size calculation before the initiation of the study was not performed or reported in 79.1% (n = 427) of the trials, while the sample size was assessed as adequate in only 17.6% (n = 95) of the trials. Inclusion and exclusion criteria were clearly defined in the majority of trials, while baseline comparability was adequate in 72.8% of the trials. Two thirds of the trials were not described as double blinded (n = 358; 66.3%), while the method of blinding was appropriate in 53% (n = 286) of the trials. Blinding of the principal investigator and statistician was unclear/not reported in 92% and 97.8% of the trials, respectively, that is, in the vast majority of the trials. The treatment protocol was adequately described for treatment and control groups in the vast majority of trials, with 73.1% (n = 395) having a designated control group, and 38.1% (n = 206) using a placebo group. Whether cointerventions were avoided/comparable was assessed as unclear/not reported in 60.7% (n = 328) of the trials, while 84.1% (n = 424) of the trials did not report cointerventions for each group. Participants compliance to treatment protocol was tested in 61.1% (n = 330) of the trials, with compliance being acceptable (more than or equal to 80%) in 50.9% (n = 275) of the trials. Withdrawals/dropouts were reported in the vast majority (89.4%, n = 483) of trials, with withdrawal/dropout rates being acceptable (less than or equal to 20%) in 73.1% (n = 395) of the trials, and reasons for withdrawals/dropouts reported in 71.1% (n = 384) of the trials. Adverse effects were not described in nearly half (n = 264; 48.9%) of the trials. Outcome measures were described in the majority of trials, while psychometric properties of main outcome measures—validity, reliability, and responsiveness—were not reported in 96.7%, 93.1%, and 96.7% of the trials, respectively. The statistical analysis was appropriate in 85.7% (n = 463) of the trials, with descriptive measures being reported in the majority of the trials. The clinical significance was not reported in 70.9% (n = 382) of the trials, while the intention to treat analysis was not used/reported in 59.6% (n = 264) of the trials (see ). We identified a significant decrease (P < 0.001) in the proportion of trials judged as being inadequate (having high or unclear risk of bias) over time in five domains of the Cochrane risk of bias tool: sequence generation, allocation concealment, incomplete outcome data, other sources of bias (including: baseline comparability, similarity of cointerventions, and compliance to treatment), and overall risk of bias (see ). The proportion of trials assessed as being inadequate (having high or unclear risk of bias), with respect to patient blinding, decreased significantly (P < 0.031) in the sample, while change in blinding of outcome assessment and selective outcome reporting were not statistically significant. The proportion of trials assessed as having inadequately addressed methodological quality items decreased significantly over time in 30 out of the 40 quality criteria (23 quality criteria at P < 0.001, seven quality criteria at P < 0.05). This was not statistically significant in the following items: study described as randomized, method of blinding appropriate, blinding of principle investigator, blinding of assessor, treatment protocol adequately described for the treatment group and for the comparison group, report of withdrawals/dropouts, outcome measures described, validity and responsiveness for main outcome measures reported, descriptive measures reported, and early cessation of a trial (see ). The results of the logistic regression analyses showed that a significant change over time was evident in 29 out of the 36 quality criteria (that is, 10 risk of bias domains and 26 quality items) of which 26 quality criteria improved over time, while 3 criteria (study described as double blind, blinding of care-provider, and presence of placebo group) worsened over time. Conversely, 8 quality criteria (selective outcome reporting, report of withdraws and dropouts, and 6 blinding-based criteria) did not show a significant change over time (see ). Abbreviations: CI, confidence interval; OR, odds ratio. Low risk vs. others (both Unclear and High risk of bias); or Yes vs. others (both No and Unclear/Not-reported). Time was entered in the logistic regression model as a continuous variable. Time was entered in the logistic regression model as a categorical variable. The following criteria were not considered in the analysis because of either having a small number of trials judged as being adequate: The following criteria were not considered in the analysis because of having a small number of trials judged as being inadequate/unclear: treatment protocol adequately described for treatment, control, or comparison group; short or long follow-up measurement performed; outcome measures described; descriptive measures identified and reported. The factor which the odds of the quality criteria, being adequate, increased by every year.

Discussion

Bias is a threat to the quality of trials [38, 39]; it may impact the reported treatment effect estimates in randomized clinical trials [40-44]. The degree of bias in randomized clinical trials of oral health intervention has decreased over time according to our study. We used the Cochrane Collaboration’s risk of bias tool, in addition to a comprehensive set of reporting and methodological characteristics (selected from seven quality assessment tools reported to be valid), to assess the methodological quality of randomized clinical trials of oral health intervention. Thus, this study provides an in-depth analysis of the methodological characteristics and risks of bias present in dental literature from 1955–2013. In the majority of the quality items and risk of bias domains, our study showed that the proportion of trials having inadequate quality (or having high or unclear risk of bias) decreased significantly over time. This encouraging trend is similar to what was identified in a recently published report by Reveiz et al. [19]. However, rather than conducting standardized data extraction from each trial, Reveiz used a risks of bias assessment reported by the investigators of reviews; this might be problematic, especially given the documented low reliability of the Cochrane Collaboration’s risk of bias tool [26, 45]. The trend in our study is comparable to that found in a cohort of child related trials [46] and medical randomized clinical trials [5]. A similar trend was identified when the methodological quality of trials of physical therapy interventions was assessed, where an improvement of nearly 0.6 points each decade was found in the total Physiotherapy Evidence Database (PEDro) score [47]. The trend in our report is also similar to a recently published study [48] which analyzed 20,920 randomized clinical trials included in Cochrane reviews; the study concluded that the proportion of trials at unclear risk of bias decreased over time, especially for sequence generation (fell from 69.1% to 31.2%) and for allocation concealment (fell from 70.1% to 44.6%). Although an improvement over time was identified in our study with respect to randomized clinical trials of oral health intervention, results of risks of bias, risks of random error, and reported methodological quality assessments were still unpropitious, indicating substandard quality and a high potential for bias. We believe that sizable improvements in the conduct and reporting of oral health randomized clinical trials is possible. The fact that the proportion of trials having low risk of bias did not exceed 60% in the majority of risk of bias domains is a significant concern. Because inadequate design and unrigorous conduct of a trial can bias the estimation of the treatment effect size, decisions made in dental practice might not be based on valid findings. For example, allocation concealment (a process of concealing information about which patients are to be assigned to a new treatment versus those to be given a conventional therapy) and sequence randomization (allocation is carried out using a chance mechanism so that neither the participant nor the investigator will know in advance which will be assigned to an intervention) [49-51] were unclear in 84.8% and 66.7% of the trials, respectively, although these factors improved significantly over time. It should be noted that an “unclear” risk of bias results in a trial that may not mirror the actual design and conduct of the trial. Because journals have a word limit that may restrict authors in reporting detailed methods, all of the methodological characteristics used might not be reported [52], thus restricting the accuracy of quality assessment tools. In the field of dentistry, treatment effect size estimates were found to be larger in trials with inadequate trial’s design characteristics which can can affect overestimation of beneficial or underestimation of harmful treatment effects within trials [15, 44]. Our study showed that sample size calculation was not performed or reported in 79.1% of the trials; this finding is concerning as it raises the risks of random errors leading to the risk of having increased numbers of false positive conclusions (type I errors) and false negative conclusions (type II errors) [53] in randomized clinical trials of oral health interventions. It has recently been estimated that randomized clinical trials have to include at least 1000 participants in order for simple randomisation to serve its purpose: allocation a sufficient sample to allow a fair comparison of intervention effects [54]. Moreover, having an appropriate influence of trial sponsor was found to be inadequate or not reported in 83.3% of the trials. This is concerning because sponsorship bias in oral heath randomized clinical trials can potentially benefit the sponsoring company and might lead to inappropriate clinical decisions. Industry sponsorship of trials may lead to favorable findings than sponsorship by other sources [43]. For example, a recently published report examined the influence of industry sponsorship in 41 randomized clinical trials of dental implants and found that the probability of implant failure in sponsored randomized clinical trials was much lower than the probability of implant failure in nonsponsored randomized clinical trials. Also, there is a debate whether the Cochrane risk of bias tool should include funding source as an individual item or not [55-57]. Accordingly, there is a need for large meta-epidemiological studies to quantify the extent of bias associated with influence of funders on the magnitude of treatment effect size in oral heath randomized clinical trials. Our study revealed that more than half of the trials were published in specialist journals, and that nearly half of the trials were from the United Kingdom and the United States. These trends are similar for medical trials [52]. Possibly the interest of government and public sectors in the aforementioned countries is responsible for facilitating the financial support for such randomized clinical trials [58]. The improvement observed in risks of bias and reported methodological quality of randomized clinical trials over time, could be attributed to efforts made by editors and reviewers of oral health journals, through endorsement of the CONSORT Statement [59, 60], and by the mandatory implementation of trial registration, as recommended by the ICMJE [6, 61]. The CONSORT Statement is an accepted and widely used approach in medical and dental research to assess the reporting quality of randomized clinical trials. This approach covers the fundamental aspects of a trial’s reporting quality; the CONSORT Statement aims to advance the transparency and quality of medical and dental trial reporting through the creation of reporting criteria [62, 63]. It has been endorsed during the last 10 years by several medical journals worldwide, including the majority of high impact oral health journals [59, 63]. Although the CONSORT Statement applies only to reporting quality, it is used commonly and erroneously by many dentistry researchers as a methodological quality assessment tool. Moreover, the SPIRIT Statement has recently become a widely used tool that aims to advance and improve the quality of protocols of randomized clinical trials and involve a set of criteria to address in a trial protocol [64, 65]. In the dental literature, the concepts “reporting quality” and “methodological quality” are often used interchangeably, contributing to conceptual ambiguity. Methodological quality depends mainly on the degree to which the design, conduct, and analysis of a trial follows the highest possible standards (to reduce multiple potential biases) and, hence, suggests that the findings can be based on the used intervention [1, 16, 17]. While the internal validity of a trial (which is closely connected to the risks of bias [16] and the methodological quality) should be the core of quality assessment, “reporting quality” is mistakenly used by researchers as an alternative for methodological quality; this has induced a conceptual ambiguity in the definition of trial “quality” [28]. In the context of medical research, a risk of bias assessment will benefit from an explicit and unambiguous definition of “methodological quality”. Although endorsement of the CONSORT Statement by dental journal editors and reviewers results in improvement in the reporting quality of trials, it does not guarantee compliance by trialists [60]. Furthermore, reliance on the CONSORT Statement only, may give reviewers, authors, and readers a false sense of security. Transparent reporting is desirable, but it does not necessarily raise methodological quality or lower the risks of bias [39]. For example, good reporting fails to prevent publication bias (i.e., trials of methods that have beneficial and large effects are published rapidly in journals with high impact), and selective outcome reporting (i.e., beneficial findings get publishing priority) [16]. These reported biases can exaggerate the magnitude of treatment effects in clinical trials, and can distort findings in meta-analysis [66, 67]. Implementation of the mandatory trial registration policy [4] could contribute to the improvement of trial quality and lower the risks of bias identified in this study. Implementation of the mandatory trial registration policy started over 10 years ago by 11 leading medical journals, and is currently applied by over 300 medical journals [6], including many leading dental journals [68]. However, recently only 23% of dental randomized clinical trials, published in 15 high impact dental journals, were registered [61, 69]. The results of this study have several implications. Dental trialists need to explicitly report their trials’ results and adhere to published guidelines. Dental journal editors and reviewers should continue to be committed to international initiatives and statements developed to ensure adequate and appropriate conduct and reporting of randomized clinical trials. Adherence to the above guidelines can reduce the risk that inaccurate conclusions will be drawn from the research and, accordingly, will reduce inappropriate recommendations regarding treatment interventions in dental practice. To adequately apply trial findings to care of their patients, clinicians should be aware of the design, conduct, and reporting of a clinical trial. This knowledge will enable a clinician to deliver the best possible results in his or her own dental practice. Our findings call for oral health policy makers, methodologists, clinicians, and researchers to develop initiatives for improving clinical trials, which would spread such actions within the oral health community. The formation of a global oral health initiative that aims to improve the conduct and reporting of oral health trials, and that prioritizes methodological criteria in oral health research, would be an example of a potentially needed measure to raise standards of randomized clinical trials.

Strengths and limitations

This cross-sectional observational study provided a comprehensive assessment of oral health randomized clinical trials with respect to trial characteristics, reporting quality, methodological characteristics, and risks of bias, and attempts to identify the variation of these factors over time. The range and size of our sample provided a comprehensive evaluation of oral health trials over the 58 year-period of 1955–2013. One of the strengths of our research was the data extraction method, which was performed in duplicate by two independent assessors to ensure high accuracy and avoid potential biases during the data extraction process. We performed a standardized data extraction rather than relying on the risks of bias assessment reported in systematic reviews, which was the case in recent reports by Reveiz et al. where the risks of bias in medical randomized clinical trials were assessed (19) and by Dechartres et al. using data from 20,920 randomized clinical trials included in Cochrane reviews within all disease areas [48]. A potential limitation of our research is that the choice of sample trials is not strictly random. Our sample of trials originated from 64 dental, oral, and craniofacial meta-analyses and was designed to cover the overall spectrum of clinical oral health research during 1955–2013, therefore, we believe that it represents a realistic cohort for that period. However, as the topics of meta-analyses are often focused on topics of high importance to researches and societies, we cannot exclude that the quality of the trials may be better than the average dentistry randomized clinical trial. Also, it is a drawback that we did not publish our protocol for this observational study. Another potential limitation is that we did not contact the authors of the studied trials for missing data. A large proportion of the trials were published before the year 2000 when an author’s correspondence information was sometimes not current and not always provided in the publication. Moreover, because we extracted data based on the data reported in the published trials, the actual risk of bias potential was not visible in the majority of risk of bias domains studied due to the poor quality of the reporting identified in the studied trials. As our study did not look at factors that contributed to methodological quality improvement over time, these factors must be considered in future research. We applied an educated judgement to assign each trial to a primary dental specialty (e.g., dental public health), although the trial could be classified under more than one specialty (e.g., both pediatric dentistry and dental public health).

Conclusions

Our study showed a significant increase over time (1955–2013) in the proportion of trials judged to be adequate in risks of bias, risks of random errors, reporting quality, and methodological quality. However, the proportion of trials judged as having low risk of bias did not exceed 60% in the majority of the risk of bias domains. We found the risks of bias and the quality assessment in the studied trials to be generally unfavorable. That is, in the trials of oral health interventions the methodology and reporting quality were substandard, resulting in a high potential for bias. We believe that a commitment to international initiatives by researchers, journal editors, and manuscript reviewers can contribute to the development of more stringent methodology and more detailed reporting of randomized clinical trials of oral health interventions.

Tools and items to assess quality of randomized trials.