Reporting quality and spin in abstracts of randomized clinical trials of periodontal therapy and cardiovascular disease outcomes.

OBJECTIVE: Poor reporting in randomized clinical trial (RCT) abstracts reduces quality and misinforms readers. Spin, a biased presentation of findings, could frequently mislead clinicians to accept a clinical intervention despite non-significant primary outcome. Therefore, good reporting practices and absence of spin enhances research quality. We aim to assess the reporting quality and spin in abstracts of RCTs evaluating the effect of periodontal therapy on cardiovascular (CVD) outcomes.
METHODS: PubMed, Scopus, the Cochrane Central Register of Controlled Trials (CENTRAL), and 17 trial registration platforms were searched. Cohort, non-randomized, non-English studies, and pediatric studies were excluded. RCT abstracts were reviewed by 2 authors using the CONSORT for abstracts and spin checklists for data extraction. Cohen's Kappa statistic was used to assess inter-rater agreement. Data on the selected RCT publication metrics were collected. Descriptive analysis was performed with non-parametric methods. Correlation analysis between quality, spin and bibliometric parameters was conducted.
RESULTS: 24 RCTs were selected for CONSORT analysis and 14 fulfilled the criteria for spin analysis. Several important RCT elements per CONSORT were neglected in the abstract including description of the study population (100%), explicitly stated primary outcome (87%), methods of randomization and blinding (100%), trial registration (87%). No RCT examined true outcomes (CVD events). A significant fraction of the abstracts appeared with at least one form of spin in the results and conclusions (86%) and claimed some treatment benefit in spite of non-significant primary outcome (64%). High-quality reporting had a significant positive correlation with reporting of trial registration (p = 0.04) and funding (p = 0.009). Spinning showed marginal negative correlation with reporting quality (p = 0.059).
CONCLUSION: Poor adherence to the CONSORT guidelines and high levels of data spin were found in abstracts of RCTs exploring the effects of periodontal therapy on CVD outcomes. Our findings indicate that journal editors and reviewers should consider strict adherence to proper reporting guidelines to improve reporting quality and reduce waste.

Introduction

The abstract of randomized clinical trials (RCT) provides the reader with the first account of the trial objectives, methodology and results. Therefore, reporting accuracy, clarity and quality have a critical role during the initial assessment of the trial and affects the decision to read the full text [1]. Furthermore, in many geographic locations, RCT abstracts are often the only section of an RCT freely accessible to clinicians [2].

In recognition of the importance of RCT abstracts, the Consolidated Standards of Reporting Trials (CONSORT) for abstracts guidelines [3] were developed as an extension to the original CONSORT, addressing clarity, completeness and transparency and ensuring that key trial elements are properly reported. Hence, poor reporting refers to omitting important information in abstracts as required by the well-defined CONSORT items [2].

Furthermore, spin is defined as failure to accurately and faithfully report the findings of a scientific study in a manner that would affect the reader’s perception of the outcomes [4]. The tool for spin assessment in publication abstracts [4] identifies reporting practices that constitute an intentional or unintentional attempt to spin the results and/or conclusions leading to misreporting and bias. Despite the development of reporting and spin guidelines, abstracts in biomedical literature are often characterized by poor reporting quality and biased finding interpretation [5-12].

The impact of poor reporting and spin on the public and professional perception of research findings is discernible. In fact, abstracts with high levels of spin were found to be more frequently read compared to abstracts of the same trial after being edited to omit spin, and were also more likely to mislead clinicians to accept a clinical intervention as being beneficial despite a non-significant primary outcome [1]. Moreover, spin in abstracts percolates into media coverage and press releases, which in turn generates greater public attention [13] Paradoxically, articles that received greater media attention showed improved citation metrics in subsequent publications [14], creating what resembles of a vicious circle of public and scientific misinformation.

Ever since the publication of the earliest studies indicating a correlation between cardiovascular (CVD) disease and periodontitis [15, 16], the findings have received considerable professional and public interest. To test causal relationships, several RCTs explored the effect of periodontal therapy on CVD outcomes. Subsequently, the topic sparked intense debates between researchers, caused wide-scale media coverage and public interest, and prompted involved professional organizations to issue official statements [17, 18].

Although multiple periodontal-CVD RCTs have been published, the adherence to the CONSORT guidelines and the incidence of spin has not been studied. Therefore, the aim of this study was to evaluate the reporting quality and the incidence of spin in abstracts of RCTs investigating the effect of periodontal therapy on CVD disease outcomes.

Materials and methods

Search methods and study selection

Studies were retrieved from PubMed, Scopus based on search strategy shown below. In addition, we crosschecked 17 trial registration platforms included in the World Health Organization International Clinical Trials Registry Platform [19] to confirm trial registration status and information (S1 Table). The search was conducted for all registers on 01/01/2018.

Search keywords and limitations or filters for each database were as follows:

Pubmed: ("Lipids"[Mesh] OR "Acute-Phase Proteins"[Mesh] OR "Blood Pressure"[Mesh] OR "Arterial Pressure"[Mesh] OR "Hypertension"[Mesh] OR "Hypotension"[Mesh] OR "Cholesterol"[Mesh] OR "Cholesterol, LDL"[Mesh] OR "Cholesterol, HDL"[Mesh] OR "Cholesterol Esters"[Mesh] OR "Embolism, Cholesterol"[Mesh] OR "Cholesterol, VLDL"[Mesh] OR "Cardiovascular System"[Mesh] OR "Cardiovascular Infections"[Mesh] OR "Cardiovascular Abnormalities"[Mesh] OR "Cardiovascular Diseases"[Mesh] OR "Cardiovascular Physiological Phenomena"[Mesh] OR "Endothelium"[Mesh] OR "Endothelial Cells"[Mesh]) AND ("Periodontal Debridement"[Mesh] OR "Periodontal Diseases"[Mesh] OR "Periodontal Pocket"[Mesh] OR "Alveolar Bone Loss"[Mesh] OR "Dental Scaling"[Mesh] OR "Periodontitis"[Mesh] OR "Dental Prophylaxis"[Mesh] OR "Periodontal Attachment Loss"[Mesh]) filter: clinical trial

Scopus: TITLE-ABS-KEY (("Lipids" OR " Proteins*" OR "Pressure*" OR "Hypertension" OR "Hypotension" OR "Cholesterol" OR "Cardiovascular*" OR "Endothelium" OR "Endothelial*") AND ("Periodontal*" OR "Alveolar Bone Loss" OR "Dental*" OR "Periodontitis") AND "clinical trial") AND (LIMIT-TO (DOCTYPE, "ar"))

17 trial registration platforms (S1 Table) were searched. Since these platforms were limited to one or two keywords, “periodont*” was used as a main keyword, then records were scanned for eligible studies.

The retrieved articles were hand-screened for identification of additional RCT reports (Fig 1), and then duplicates were excluded.

Fig 1

Search Strategy with flow chart representing the identification, screening, eligibility and inclusion stages.

Study Design:

Only publications of periodontal-CVD RCTs were included. Cohort, non-randomized trials or observational trials were excluded. RCT publications in languages other than English were excluded.

Participants:

Targeted populations included adult patients with no systemic diseases other than CVD diseases. Studies were included if the participants were diagnosed with chronic periodontitis only. Studies with participants diagnosed with aggressive periodontitis, gingivitis, or peri-implantitis were excluded.

Intervention:

The tested intervention included subgingival scaling and root planing (SRP) or SRP with adjunctive therapy. Interventional studies employing adjuncts alone, supragingival scaling alone or surgical therapy were excluded.

Outcomes:

True or surrogate CVD outcomes were included. For descriptive purposes, outcomes were segregated into two groups [20]:

CVD true events; such as angina, myocardial infarction, stroke, and CVD end points (CVD related-death).

Surrogate outcomes; such as blood pressure, lipids, blood tests, ugh sensitivity C-reactive protein (hs-CRP), lipoproteins, and blood cell count.

RCT report inclusion criteria

Additional selection criteria for spin assessment

Only studies with a clearly defined primary outcome were included in the spin analysis. To fulfill this condition, the primary outcome should be either explicitly stated in the abstract or the full text. In cases were the primary outcome was not explicitly reported, the outcome stated in the sample size calculations was considered as the primary outcome. If no outcome was stated in the sample size calculation, we deduced a primary outcome based on the stated objectives of the study. If no primary outcome could be identified, the study was excluded. In addition, studies with multiple primary outcomes were excluded.

Selection of studies was carried out according to the inclusion criteria. Titles and abstracts of the search results were initially screened for identification of other potential eligible studies. Then, full texts were retrieved and assessed to further assess eligibility. In cases of multiple published reports associated with the same trial registration number, the primary publication on the RCT results was selected to avoid duplication.

Data extraction: Two authors (MS, KA) independently reviewed the abstracts -and the full texts, when needed- of the included RCT reports, and applied the CONSORT for abstracts [3] and the SPIN checklist [1]. Disagreements were resolved by a third author (EI). Two items of the CONSORT abstracts guidelines were excluded because they only apply to unpublished studies or conference abstracts.

Characteristics of each RCT abstract and the respective publishing journal were extracted:

Abstract word count

Number of citations as shown on Scopus [21].

Trial registration number, trial registration date was determined based on the information provided by the trial registry.

Trial funding source, number of authors, geographic location,

Trial sample size, intervention and outcomes.

Journal metrics such as 5 years-impact factor, impact factor without self-citation, influence factor as reported on Thomson-Reuters/Clarivate Analytics 2018 [22].

A decision-making guide was implemented to assist with the calibration and review process [23]. An Overall CONSORT Score for reporting quality and Overall Spin Score were calculated for each RCT publication based on the CONSORT and Spin checklist.

Statistical analysis

Cohen’s Kappa statistic was used to assess inter-rater agreement. For the descriptive analysis categorical variables were expressed as proportion percent. For the exploratory bivariate association between Overall CONSORT Score and related variables, we applied a Spearman correlation model. The limited sample size did not allow for a further multivariate regression model to assess predictors of reporting quality. Statistical analysis was performed with SPSS software (SPSS Inc).

Results

General findings

Twenty-four RCT reports were deemed eligible and entered the analysis (Fig 1). Among them, one study was a secondary analysis publication [24]. The PAVE study had multiple publications, and according to our inclusion criteria we included only the results publication [25]

For each trial, journal and article metrics are presented in Table 1.

Table 1

Characteristics of included publications and publishing journals.

RCT	Year	Journal metrics					Article metrics
		Journal	5 years impact	Impact factor without self-citation	Eigenfactor	Influence factor	Abstract Word count	Authors #	Citations	Registration	Geographic location	Funding
[24]	2017	JCP	4.62	3.34	0.011	1.15	208	5	3	Yes	Netherlands	Yes
[25]	2009	JP	3.52	3.02	0.011	0.86	241	17	13	Yes	USA	Yes
[26]	2012	JCP	4.62	3.34	0.011	1.15	193	7	20	Yes	Pakistan	Yes
[27]	2015	Medicine	2.19	1.89	0.05	0.59	237	11	132	No	Italy	No
[28]	2014	JCP	4.62	3.34	0.011	1.15	174	4	198	No	Brazil	Yes
[29]	2015	JP	3.52	3.02	0.011	0.86	266	5	113	Yes	Brazil	Yes
[30]	2005	J Dent Res	5.72	5.07	0.02	1.55	160	5	40	No	UK	Yes
[31]	2006	Am Heart J	4.63	4.08	0.04	2.11	346	6	27	No	UK	Yes
[32]	2016	Clin Oral Invest	2.55	2.23	0.011	0.68	240	5	682	No	China	Yes
[33]	2007	S Med J	0.96	0.81	0.002	0.32	145	7	36	No	Turkey	No
[34]	2011	JP	3.52	3.02	0.011	0.86	239	3	12	No	India	No
[35]	2015	JP	3.52	3.02	0.011	0.86	277	4	1	Yes	India	No
[36]	2003	JCP	4.62	3.34	0.011	1.15	174	6	27	No	UK	Yes
[37]	2016	Lasers Surg Med	2.74	2.49	0.004	0.59	249	8	16	No	Saudi Arabia	Yes
[38]	2011	J Perio Res	2.71	2.70	0.004	0.63	253	5	1	No	Jordan	Yes
[39]	2014	Hypertension	6.74	6.16	0.04	2.19	254	13	40	Yes	Australia	Yes
[40]	2011	JCP	4.62	3.34	0.011	1.15	192	5	1	Yes	China	Yes
[41]	2012	JP	3.52	3.02	0.011	0.86	310	6	88	No	Chile	Yes
[42]	2010	Eur J Oral Sci	1.91	1.57	0.003	0.56	177	11	24	No	Australia	Yes
[43]	2007	NEJM	67.51	78.54	0.70	29.45	255	10	8	No	UK	Yes
[44]	2007	JCP	4.62	3.34	0.011	1.15	201	12	240	No	Turkey	Yes
[45]	2008	JCP	4.62	3.34	0.011	1.15	189	9	16	No	Japan	Yes
[46]	2009	JP	3.52	3.02	0.011	0.86	228	4	5	No	Brazil	Yes
[47]	2017	JP	3.52	3.02	0.011	0.86	252	10	25	Yes	China	Yes

JCP: Journal of Clinical Periodontology, JP: Journal of Periodontology, J Dent Res: Journal of Dental Research, Am Heart J: American Heart Journal, Clin Oral Invest: Clinical Oral Investigations, J Perio Res: Journal of Periodontal Research, Eur J Oral Sci: European Journal Oral Science, S Med J: Southern Medical Journal, Lasers: Surg Med: Lasers in Surgery and Medicine, NEJM: New England Journal of Medicine.

An overview of other outcomes of the included articles is shown Table 2.

Table 2

Descriptive data on primary outcome and interventions.

	CHARACTERISTIC	N = 24	%
1	Nature of primary outcome
	True outcome (CV event)	0	0
	Surrogate outcome	24	100
2	Primary outcome source
	Explicitly stated in abstracts	3	13%
	Explicitly stated in full texts	3	13%
	Based on power analysis	7	29%
	Implied in objectives	11*	33%
3	Intervention type
	SRP alone	14	58%
	SRP + adjunct	10	42%

* the outcomes of 11 papers were implied by objectives and were more than one primary outcome, they weren’t included in the spin analysis

Generally, all RCTs explored surrogate outcomes and none examined CVD events (Table 2). Only 3 abstracts had explicitly stated primary outcome, while for the remaining RCTs, we identified and extracted the primary outcome from the full text (Table 2). RCTs with more than one outcome were excluded from spin analysis, as primary outcome identification was impossible.

In terms of trial interventions, Table 2 shows that SRP alone was used in 58.3% of the included RCTs, while SRP and adjuncts were used in 41.7%. The majority of the RCTs reported some type of funding (83.3%). Specifically, 58% were funded intramurally, 46% by foundations, 29% by federal agencies, and 25% by industry. 60% of the funded RCTs reported multiple funding sources.

Abstract word count for each trial is presented in Table 1, and was categorized into 3 groups according to CONSORT findings [3], which is presented in Table 2. 66.7% of the abstracts had <250-word count, 25% had word count from 250–300, and 8.3% had word count >300 [3].

CONSORT checklist findings

Following the RCT abstract assessment, Overall CONSORT Score ranged from 2 to 9 out of 15. Table 3 presents the frequency of each CONSORT item fulfillment.

Table 3

Fulfillment (%) of CONSORT items stratified by CONSORT sections.

CONSORT FOR ABSTRACT CHECK LIST	(NUMBER) PERCENTAGE
TITLE	12 (50%)
TRIAL DESIGN	3 (13%)
METHODS
PARTICIPANTS	0.0%
INTERVENTIONS	13 (54%)
OBJECTIVE	22 (92%)
OUTCOME	3 (13%)
RANDOMIZATION (METHOD)	0 (0%)
BLINDING (MASKING)	0 (0%)
RESULTS
NUMBERS RANDOMIZED	20 (83%)
NUMBERS ANALYZED	4 (17%)
OUTCOME	3 (13%)
HARMS	1 (4%)
CONCLUSIONS	3 (13%)
TRIAL REGISTRATION	3 (13%)
FUNDING	8 (33%)

Specifically, only 50% of the included RCTs acknowledged the term “randomized clinical trial” in the title. Only 3 of the studies (13%) included the specific study design in the abstract (i.e. parallel group, crossover, superiority, etc.).

Assessment of methods reporting

Three items in the methods section lacked reporting in all RCTs, including the item “participants”, which lacked information about the location of the study and the detailed description of the participants that were included, and the item “randomization”, were all of the studies did not report the randomization method that was used, and the item “blinding”, were studies did not report the level blinding. The item “intervention” mostly lacked the necessary detailed description of the intervention; therefore, only 54% of the RCTs fulfilled this item.

Most of the study abstracts (92%) included the objectives of the study. Interestingly, only 3 abstracts (18.5%) explicitly stated the primary outcome.

Assessment of results reporting

Although 83% of the abstracts included the numbers of randomized populations, only 17% included the numbers of analyzed populations as part of the abstract materials and methods rather than the results section violating the CONSORT recommendations. Only 1 RCT indicated the harms in the abstracts (4%).

Assessment of conclusion, registration and funding reporting

Only 13% discussed the results and conclusion of the primary outcome. Trial registration information was reported in only 3 abstracts (13%). When the public trial registration records were examined using the registration number included in the study, we found that 4 RCTs registered following the study initation and the first subject recruitment.

Spin analysis findings

After applying the exclusion criteria as outlined in the methodology, 14 out of the 26 RCT reports were included in the spin analysis (S2 Fig). The prevalence and type of spin for the included articles is outlined in Table 4.

Table 4

Fulfillment (%) of SPIN items in result and conclusion sections.

Type of spin	(number) Percentage
1) spin in the result
Focus on statistically significant within-group comparison	3 (21%)
Focus on statistically significant secondary outcomes	9 (64%)
Focus on statistically significant subgroup analyses	2 (14%)
Focus on statistically significant modified population of analyses (eg, per-protocol analyses)	4 (29%)
Focus on statistically significant within- and between-group comparisons for secondary outcomes	9 (64%)
OTHER SPIN: NO DEFINITION OF PRIMARY OR SECONDARY OUTCOMES	11 (79%)
2) spin in the conclusions
Focus only on treatment effectiveness:
1. Claiming equivalence for statistically nonsignificant results	0 (0%)
2. Claiming efficacy with no consideration of the statistically nonsignificant primary outcome	9 (64%)
3. Focusing only on statistically significant results	6 (43%)
Acknowledge statistically nonsignificant results for the primary outcome but emphasize the beneficial effect of treatment	9 (64%)
Acknowledge statistically nonsignificant results for the primary outcome but emphasize other statistically significant results	9 (64%)
Other spin in Conclusions section:
1. Conclusion ruling out an adverse event on statistically nonsignificant results	0 (0%)
2. Conclusion focusing on within-group assessment (both treatments are effective/treatment administered in both groups is effective (eg, add-on studies)	2 (14%)
3. Recommendation to use the treatment	1 (7%)
4. Focus on another objective	1 (7%)
5. Comparison with placebo group of another trial	0 (0%)
6. Statistically nonsignificant subgroup results reported as beneficial	0 (0%)
Others: Inadequate extrapolation to larger population, intervention or outcome	12 (86%)
misleading statements designed to exaggerate or falsely claim efficacy	7 (50%)
3) Spin in both results and conclusions	12 (86%)

Some form of spin in both of the results and conclusions sections was detected in the majority of the RCTs (86%). Given that 79% of the included studies failed indicate the primary or secondary outcomes in the abstracts, we considered that these studies employed diverse strategies of spin.

In the results section, at least one checklist item showed a form of spin. 64% of the studies focused on statistically significant secondary outcomes, and on statistically significant within- and between- group comparisons of secondary outcomes.

In the conclusion sections, half of the included RCTs (50%) made statements that were misleading and designed to exaggerate or falsely claim efficacy. 43% of the conclusions were focusing only on significant results regardless if they corresponded to the primary outcome, 7% focusing on another objective, and 7% making treatment recommendations. 64% acknowledged statistically non-significant results for the primary outcome yet emphasized the beneficial effect of treatment, and emphasized other statistically significant results.

Bivariate correlation analyses

The Overall CONSORT Score ranged between 2–9 out of 15 with some articles fulfilling only had 2 items of the CONSORT checklist. The maximum number of fulfilled items by a single publication was 9 out of 15 (S1 Fig). The Overall Spin Score ranged between 1–13. Some publications included only 1 item with some form of spin, while some articles had 13 items that were spun (S2 Fig).

Within the limitations of the study, there was a positive and significant bivariate correlation between the Overall CONSORT Score and funding source (correlation coefficient: 0.416, P-value: 0.043). In addition, we observed a significant correlation between Overall CONSORT Score and registration reporting in the abstract (correlation coefficient: 0.518, P-value: 0.009). In summary, we observed that abstracts that included trial registration and funding information were characterized by higher reporting quality.

Overall Spin Score showed a marginal negative correlation with Overall CONSORT Score (correlation coefficient: -0.517, P-value: 0.059), which signified that the higher the reporting quality in the abstract, the lower the spin. Overall Spin Score showed negative correlation, although not significant, with funding and registration. Overall Spin Score showed marginal negative correlation with the number of publication authors (correlation coefficient of -0.509, P-value of 0.063).

Discussion

Our study evaluated the reporting quality and incidence of spin in the abstracts of 24 RCT publications assessing the impact of periodontal interventions on CVD outcomes. To our knowledge, this is the first paper that evaluated both reporting quality and spin in abstracts of such publications. The overall reporting quality of the included abstracts was deemed to be poor. Overall, we found that the RCT objectives and numbers of randomized subjects were the only most adequately reported items (92% and 83% respectively). All other CONSORT items were adequately reported in less than 50% of the abstracts. Notably, we found limited RCT abstracts with adequate reporting on the exact trial design (17%), method of randomization (0%), blinding (0%), number of subjects analyzed (17%), harms (4%), outcomes in both trial arms (13%), as well as the interpretation of the results in the conclusions (13%). Our findings were in agreement with other studies in the medical and dental literature confirming inadequate reporting according to CONSORT guidelines [9, 48–50]. Surprisingly, even after dental journals adhered to the CONSORT for abstracts guidelines [50], those guidelines were not systematically reinforced. Therefore, RCT abstracts were still characterized by inadequate reporting quality.

Consistent with other reports, the CONSORT items most adequately reported in the RCT abstracts were related to objectives and numbers randomized [49, 50]. Journal and article metrics including impact factor or citation metrics were unreliable in predicting reporting quality, as confirmed in other studies [51]. The lack of significance in this correlation could be also related to the small number of included publications.

In the spin analysis, it is noteworthy, that 10 of 24 of the included RCT publications were excluded due to the lack of an explicitly defined single primary outcome. The use of multiple primary outcomes in RCTs combined with the lack of adequate power analysis for multiple outcomes might significantly increase the risk of bias.

The spin analysis according to established criteria [4] showed that various strategies of spin were adapted in the included abstracts (n = 14). Specifically, spin phenomena in either the result or conclusion sections of the abstracts were detected in the majority of the publications. Half of the abstracts presented a tendency for conclusions that had misleading statements designed to exaggerate or falsely claim efficacy. One third of the RCT abstracts presented the trial results in a before-after therapy manner focusing on within group analysis, highlighting statistical significance and ignoring the between group comparisons as directed by the study objectives. More than half of the RCT abstracts emphasized significance in secondary outcomes, a commonly used spin strategy, when the primary outcome results were not significant.

Our results agreed with other studies in the medical literature that investigated spin strategies and misrepresentation of RCT results with various methodologies [4, 5, 51–54]. Austin et al [5] reported some form of spin to exist in 47% of the included RCT abstracts while Cooper et al [53] reported spin to be as high as 70% of the included articles. Interestingly, Pitkin et al [8] compared data reported in the abstract of a random sample of RCTs published in 6 major medical journals to the data presented in the full-text manuscripts and found that inconsistencies at variable levels (18%-68%) existed between data reported in the abstract compared to the full text.

The present study has several strengths. We applied strict inclusion criteria as directed by the research hypothesis and only included RCT publications examining the impact of periodontal intervention on CVD outcomes [55]. We standardized the data extraction methodology utilizing well-defined decision guide and calibration between assessors. Therefore, we have demonstrated a high level of inter-rater agreement with any differences resolved by a third evaluator to ensure greatest accuracy in our analysis. While the present study focused on RCT abstracts alone and not the full text of the included publications, these considerable reporting shortages and/or misrepresentations were a cause for concern, given the wide attention abstracts receive within the healthcare and media communities.

Our study also has some inherent limitations. Although the spin assessment is characterized by subjectivity, two independent and calibrated reviewers per abstract conducted the data extraction and determined the spin strategies. With this method, we aimed to control the magnitude of subjectivity. We employed spin analysis previously used by other groups [1, 4, 6, 53]. Therefore, our analysis was focused on abstract sections and might have missed additional spin strategies present in the full text.

It is important to emphasize that poor reporting quality does not translate into poor study design [56]. It does, however, indicate lack of transparency and prevent the replication of the given experiment [57]. Therefore, quality reporting is necessary for the advancement of science [58]. It is also even more important to emphasize that the identification of spin in an abstract (according to criteria of Boutron et al 2014) should not be taken as a verdict that a research report is fraudulent or fake. The spin strategies examined don’t all carry the same weight in terms of the impact on a reader’s perception of the abstract and it does not assess if these spin strategies were applied in the full text of the manuscript. Nevertheless, low quality reporting and introduction of spin might be contributing to continued controversy in this field of research [59], flawed professional and public perception of research findings [1, 13] and continued ill-advised expenditure of valuable time and resources [60]. The responsibility to improve reporting of RCTs and avoidance of misrepresentations falls on multiple parties. Journal editors and peer reviewers as gatekeepers could reinforce strict practices to ensure adherence to CONSORT or other reporting guidelines, and to require trial registration prior to the commencement of the trial as recommended by the ICMJE [61]. An additional effort by academic institutions, professional organizations, and scientific communities should be exerted to raise awareness among the general scientific audience on proper reporting practices and spin strategies. The scientific community should embrace post-publication appraisal and critique with a goal to improve reporting quality and minimize the incidence of spin.

Conclusions: Poor adherence to the CONSORT for abstracts guidelines and high levels of data “spin” were found in the abstracts of RCTs examining the effect of periodontal therapy on CVD outcomes. Our findings indicate that journal editors and reviewers should reinforce strict adherence to proper reporting guidelines by researchers and article authors to improve reporting quality and reduce spin.

List of the trials register that were used to identify eligible studies.

(DOCX)

Click here for additional data file.

Overall Consort score for each article.

Articles were de-identified. Y-axis represents the fulfilled CONSORT items per article.

(TIF)

Click here for additional data file.

Overall Spin Score for each article.

The articles were de-identified. Y-axis represents the numbers of fulfilled Spin checklist items.

(TIF)

Click here for additional data file.

(XLSX)

Click here for additional data file.

(XLSX)

Click here for additional data file.

(XLSX)

Click here for additional data file.

9 Jan 2020

PONE-D-19-31524

Reporting quality and spin in abstracts of randomized clinical trials of periodontal therapy and cardiovascular disease outcomes

PLOS ONE

Dear Dr Ioannidou,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Both reviewers found this to be an interesting paper and worthy of publication. However, they did have a few issues that they would like addressed, in particular, the relationship between spin and propaganda.

We would appreciate receiving your revised manuscript by Feb 23 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Dermot Cox

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please upload a copy of Figure 1, to which you refer in your text on page 7 and 10. If the figure is no longer to be included as part of the submission please remove all reference to it within the text.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is an interesting topic, and one that is gathering pace recently – probably belatedly! I have very few comments, mainly to do with phrasing and updating the literature.

1. Introduction

1.1 Page 4: Spin is probably not the same as propaganda, since spin is the biased reporting of factual information in such a way as to mislead the reader, while propaganda can (and usually does) include lies.

1.2 The introduction is well written, but needs a little updating to reflect recent publications. The authors might want to check these :

1. Alharbi F, Almuzian M. The quality of reporting RCT abstracts in four major orthodontics journals for the period 2012-2017. J Orthod. 2019;46(3):225-34.

2. Austin J, Smith C, Natarajan K, Som M, Wayant C, Vassar M. Evaluation of spin within abstracts in obesity randomized clinical trials: A cross-sectional review. Clin Obes. 2019;9(2):e12292.

3. Boutron I, Haneef R, Yavchitz A, Baron G, Novack J, Oransky I, et al. Three randomized controlled trials evaluating the impact of "spin" in health news stories reporting studies of pharmacologic treatments on patients'/caregivers' interpretation of treatment benefit. BMC Med. 2019;17(1):105.

4. Hua F, Sun Q, Zhao T, Chen X, He H. Reporting quality of randomised controlled trial abstracts presented at the SLEEP Annual Meetings: a cross-sectional study. BMJ Open. 2019;9(7):e029270.

5. Roszhart JI, Kumar SS, Allareddy V, Childs CA, Elangovan S. Spin in abstracts of randomized controlled trials in dentistry: A cross-sectional analysis. J Am Dent Assoc. 2019.

2. Methods

Clearly described.

2.1 Page 10 : Cohen's kappa is a statistic, not a test.

2.2 Page 10 : 'non parametric' should be omitted. Statistical procedures are parametric or nonparametric, but data are just data.

3. Results

3.1 The results are interesting, but of course with a small sample they come with wide confidence intervals. I would include these in table 4, as these constitute the focal findings of the study. For instance, claiming efficacy with no consideration of non-significant primary outcome, which is 9 of 14 studies is 64% with a CI of 39% to 84%. So even if the lower figure were the true prevalence, it would still be pretty damning. I don't know if SPSS can be persuaded to do reasonable confidence intervals for small sample sizes (I haven't used it for some years) but the authors should use either Wilson's or Jeffrey's confidence interval. See

Brown, L., Cai, T., DasGupta, A. (2001). Interval estimation for a binomial proportion Statistical Science 16(2), 101 - 117. https://dx.doi.org/10.1214/ss/1009213286

for a discussion of the various formulas for confidence intervals.

3.2 I'd like an explanation of "inadequate extrapolation to larger population" – does this mean overgeneralising the results, or failing to provide adequate confidence intervals? Both, I think, are sins.

3.3 Page 20 : "made hyped statements" hype is deception carried out in publicity. But it can also mean exaggerated claims. I think that "misleading statements designed to exaggerate or falsely claim efficacy" is clearer, and makes explicit what is going in.

3.4 The abbreviations for reporting quality score and spin score make the paper less readable, and reduce the impact. Abbreviations don't save paper, so the authors should confine themselves to ones that are used in everyday language (eg everyone says RCT nowadays) or ones that are accepted acronyms (eg CONSORT)

3.5 The graphs are not helpful because they display the data in no sort of order, leaving the reader unable to form an overall impression. I suggest boxplots with data shown, as in the uploaded picture (produced by jamovi, a free, open-source package that is going to eat SPSS!)

4. Discussion

4.1 "deemed to be poor" = poor.

4.2 "in regard to the spin analysis" = "in the analysis of spin"

4.3 I would prefer "intentionally misleading conclusions" to "hyped conclusions"

Reviewer #2: The take of spin is of course exiting and novel, at least in the dental field. The authors start off with declaring that Spin is just another word for propaganda and then present a list of 20 points that need to be included in an abstract to not be considered as propaganda. In the age of fake news this can easily be misinterpreted as all research paper abstracts failing one of the points should be considered to be propaganda.

The results section needs to be more focused on Spin and its criteria. The abstract is more spin oriented than the results section, which is a bit ironic.

In the discussion section the authors need to add a section in which they discuss the concept of spin – to use the same verdict for insinuating that non-significant data are significant as for putting some spotlight on a secondary outcome that turned out to be interesting than the authors thought when designing the study is something that should be discussed. It needs to be made extremely clear (also in the abstract) that an abstract failing one of the 20 point (in table 4) is not the same thing as fraudulent research. The paper and the research its presenting could still be excellent.

Finally the authors need to compare their data to similar Spin analysis in other medical fields. Right now that’s just one sentence, it need to be elaborated.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Ronán M Conroy, DSc

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

18 Feb 2020

Reviewer #1: This is an interesting topic, and one that is gathering pace recently – probably belatedly! I have very few comments, mainly to do with phrasing and updating the literature.

We thank the review for the kind words.

1. Introduction

1.1 Page 4: Spin is probably not the same as propaganda, since spin is the biased reporting of factual information in such a way as to mislead the reader, while propaganda can (and usually does) include lies.

The reviewer raises a valid concern. Therefore, the term ”propaganda” was removed from the text.

1.2 The introduction is well written, but needs a little updating to reflect recent publications. The authors might want to check these:

1. Alharbi F, Almuzian M. The quality of reporting RCT abstracts in four major orthodontics journals for the period 2012-2017. J Orthod. 2019;46(3):225-34.

2. Austin J, Smith C, Natarajan K, Som M, Wayant C, Vassar M. Evaluation of spin within abstracts in obesity randomized clinical trials: A cross-sectional review. Clin Obes. 2019;9(2):e12292.

3. Boutron I, Haneef R, Yavchitz A, Baron G, Novack J, Oransky I, et al. Three randomized controlled trials evaluating the impact of "spin" in health news stories reporting studies of pharmacologic treatments on patients'/caregivers' interpretation of treatment benefit. BMC Med. 2019;17(1):105.

4. Hua F, Sun Q, Zhao T, Chen X, He H. Reporting quality of randomised controlled trial abstracts presented at the SLEEP Annual Meetings: a cross-sectional study. BMJ Open. 2019;9(7):e029270.

5. Roszhart JI, Kumar SS, Allareddy V, Childs CA, Elangovan S. Spin in abstracts of randomized controlled trials in dentistry: A cross-sectional analysis. J Am Dent Assoc. 2019.

We appreciate highlighting these references to us. The papers as recommended by the reviewer were cited appropriately.

2. Methods

Clearly described.

2.1 Page 10: Cohen's kappa is a statistic, not a test.

We revised the manuscript accordingly and the word “test” was changed into “statistic”.

2.2 Page 10: 'non parametric' should be omitted. Statistical procedures are parametric or nonparametric, but data are just data.

The phrase “non parametric” was removed in page 10.

3. Results

3.1 The results are interesting, but of course with a small sample they come with wide confidence intervals. I would include these in table 4, as these constitute the focal findings of the study. For instance, claiming efficacy with no consideration of non-significant primary outcome, which is 9 of 14 studies is 64% with a CI of 39% to 84%. So even if the lower figure were the true prevalence, it would still be pretty damning. I don't know if SPSS can be persuaded to do reasonable confidence intervals for small sample sizes (I haven't used it for some years) but the authors should use either Wilson's or Jeffrey's confidence interval. See

Brown, L., Cai, T., DasGupta, A. (2001). Interval estimation for a binomial proportion Statistical Science 16(2), 101 - 117. https://dx.doi.org/10.1214/ss/1009213286

for a discussion of the various formulas for confidence intervals.

Although the reviewer raises a good point, the limited sample size prevented us from using inferential statistics to avoid misleading conclusions.

3.2 I'd like an explanation of "inadequate extrapolation to larger population" – does this mean over-generalizing the results, or failing to provide adequate confidence intervals? Both, I think, are sins.

Boutron et al. define this item as “over generalizing the results”. Therefore, when we applied the checklist, we followed their definition.

3.3 Page 20: "made hyped statements" hype is deception carried out in publicity. But it can also mean exaggerated claims. I think that "misleading statements designed to exaggerate or falsely claim efficacy" is clearer, and makes explicit what is going in.

We agreed with the recommendation. Therefore, the item title was changed into “misleading statements designed to exaggerate or falsely claim efficacy”.

3.4 The abbreviations for reporting quality score and spin score make the paper less readable, and reduce the impact. Abbreviations don't save paper, so the authors should confine themselves to ones that are used in everyday language (e.g. everyone says RCT nowadays) or ones that are accepted acronyms (e.g. CONSORT)

We agreed with the recommendation and we revised the manuscript accordingly.

3.5 The graphs are not helpful because they display the data in no sort of order, leaving the reader unable to form an overall impression. I suggest boxplots with data shown, as in the uploaded picture (produced by jamovi, a free, open-source package that is going to eat SPSS!)

We agreed with the recommendation and we revised the supplement figures into box plots.

4. Discussion

4.1 "deemed to be poor" = poor.

4.2 "in regard to the spin analysis" = "in the analysis of spin"

4.3 I would prefer "intentionally misleading conclusions" to "hyped conclusions"

We agreed with the recommendation and these rephrasing suggestions were adopted.

Reviewer #2:

Comment 1: The take of spin is of course exiting and novel, at least in the dental field. The authors start off with declaring that Spin is just another word for propaganda and then present a list of 20 points that need to be included in an abstract to not be considered as propaganda. In the age of fake news this can easily be misinterpreted as all research paper abstracts failing one of the points should be considered to be propaganda.

The reviewer raises a valid concern. We strived to be sensitive to this aspect both during the preparation of the manuscript and during this review process. Accordingly, and to avoid sending out a wrong message, the term “propaganda” was removed. Also, some changes in the wording of the abstract were made to reflect our findings, as much as an abstract allows, with least possible misinterpretation. Moreover, the distinction between poor reporting and poor research design/ execution, and between presence of spin and fraud was highlighted and emphasized further in our discussion section.

Comment 2: The results section needs to be more focused on Spin and its criteria. The abstract is more spin oriented than the results section, which is a bit ironic.

The criteria of spin according to Boutron et al are outlined/elaborated in table 4. In addition, our findings regarding each spin criterion were presented. Our reporting of results highlights these findings. We thank the reviewer for the comment and we made an effort to balance CONSORT and Spin in in both abstract and result sections.

Comment 3: In the discussion section the authors need to add a section in which they discuss the concept of spin – to use the same verdict for insinuating that non-significant data are significant as for putting some spotlight on a secondary outcome that turned out to be interesting than the authors thought when designing the study is something that should be discussed. It needs to be made extremely clear (also in the abstract) that an abstract failing one of the 20 point (in table 4) is not the same thing as fraudulent research. The paper and the research its presenting could still be excellent.

The reviewer raises a valid and concern and we refer him to our response to his first comment/ recommendation. Specifically, changes were made in the abstract and we included a dedicated paragraph in the discussion section to highlight this point

Comment 4: Finally, the authors need to compare their data to similar Spin analysis in other medical fields. Right now, that’s just one sentence, it needs to be elaborated.

We agree with the recommendation and we elaborated on our discussion of relevant medical literature. We emphasized similarities to our findings but also an interesting finding of discordance between abstracts and the full text of a sample of studied RCT’s.

In summary, we hope that the revisions are satisfactory and the manuscript is now accepted for publication.

Sincerely

Effie Ioannidou

UCONN Health

Ioannidou@uchc.edu

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

11 Mar 2020

Reporting quality and spin in abstracts of randomized clinical trials of periodontal therapy and cardiovascular disease outcomes

PONE-D-19-31524R1

Dear Dr. Ioannidou,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Dermot Cox

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors have made a good job of the revision. I tend to agree with them that for small numbers like this, and given the lack of clarity around what population could be generalised to, confidence intervals are not appropriate.

Reviewer #2: Well done! Accepted as it stands. The paper has actually made an impression on me and I'm looking forward to citing it.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Ronán M Conroy

Reviewer #2: No

23 Mar 2020

PONE-D-19-31524R1

Reporting quality and spin in abstracts of randomized clinical trials of periodontal therapy and cardiovascular disease outcomes

Dear Dr. Ioannidou:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Dermot Cox

Academic Editor

PLOS ONE