The use of the phrase "data not shown" in dental research.

OBJECTIVE: The use of phrases such as "data/results not shown" is deemed an obscure way to represent scientific findings. Our aim was to investigate how frequently papers published in dental journals use the phrases and what kind of results the authors referred to with these phrases in 2021.
METHODS: We searched the Europe PubMed Central (PMC) database for open-access articles available from studies published in PubMed-indexed dental journals until December 31st, 2021. We searched for "data/results not shown" phrases from the full texts and then calculated the proportion of articles with the phrases in all the available articles. From studies published in 2021, we evaluated whether the phrases referred to confirmatory results, negative results, peripheral results, sensitivity analysis results, future results, or other/unclear results. Journal- and publisher-related differences in publishing studies with the phrases in 2021 were tested with Fisher's exact test using the R v4.1.1 software.
RESULTS: The percentage of studies with the relevant phrases from the total number of studies in the database decreased from 13% to 3% between 2010 and 2020. In 2021, out of 2,434 studies published in 73 different journals by eight publishers, 67 (2.8%) used the phrases. Potential journal- and publisher-related differences in publishing studies with the phrases were detected in 2021 (p = 0.001 and p = 0.005, respectively). Most commonly, the phrases referred to negative (n = 16, 24%), peripheral (n = 22, 33%) or confirmatory (n = 11, 16%) results. The significance of unpublished results to which the phrases referred considerably varied across studies.
CONCLUSION: Over the last decade, there has been a marked decrease in the use of the phrases "data/results not shown" in dental journals. However, the phrases were still notably in use in dental studies in 2021, despite the good availability of accessible free online supplements and repositories.

Introduction

The foundation of science and research is sustainable, valid and reliable when the results are available to be tested, replicated and reproduced [1-4]. Open access articles, data and code sharing, funding and conflicts of interest disclosures and detailed descriptions of materials, methods and results are great facilitators to open science [4-6]. However, studies suggest that frequently published results may be non-reproducible, which means the findings are difficult or impossible to reproduce [5, 7–9]. Fortunately, open science practices have also been adopted in biomedical research over the last decades [5, 10]. For instance, some journals have adopted compulsory data and code availability statements and abandoned strict word, table and figure limits. Additionally, online repositories and scientific publishers’ online supplements to articles have facilitated easy and free data as well as code and document sharing. Open Science Framework (OSF, www.osf.io), figshare (www.figshare.com) and GitHub (www.github.com) are some examples of online platforms where one can manage project information and data/code sharing and archive for free.

Frequently, phrases such as “data not shown” or “results not shown” are used to refer to unpublished results. It has been assumed that that results may be related to confirmatory analyses (similar results published elsewhere), negative results, peripheral results (not directly related to the topic), sensitivity analyses or future results (e.g., results related to the manuscript in preparation) [11]. However, we are unaware of any systematically conducted study that investigated what kind of unpublished results the phrases actually refer to. Using such phrases to refer to results can be seen problematic for multiple reasons. If researchers report a considerable amount of results or important results with such phrases, and without sharing results, data or code, free interpretation and verification of results are doubly harder or even impossible [4, 5]. In other words, the use of phrases obscures transparency, reproducibility and weakens the peer review process [12]. Focusing on statistically significant results and neglecting the negative ones is a major reason behind publication bias, but may also threaten the reproducibility of scientific results [13]. In addition, proper interpretation of sensitivity analyses require that modelling modifications, parameters and sensitivity results are adequately reported [14, 15]. Results not considered important for the purposes of the current study may be crucial to conduct a systematic review or meta-analysis on the closely related topic.

However, it remains unknown how frequently published studies include such phrases to refer to unpublished results in the current era of open access online journals and free online repositories and supplements to articles. Therefore, we aimed to investigate how frequently papers published in dental journals used the phrases “data/results not shown”. Accordingly, to describe current practices, we examined what kind of results the authors referred to with these phrases from the studies with the phrases published in 2021. Further, we also examined the data sharing statements, data, supplement and code availability for the studies that used these phrases.

Materials and methods

Protocol registration

We shared the protocol for this study on OSF on 26 September 2021 (osf.io/5zryu). All codes and data are also available on osf.io/5zryu. Deviations from the protocol are available in S1 Text.

Bibliographic search

We conducted searches in the Europe PubMed Central (PMC) database, which contains over seven million full-text articles at the moment. First, we searched all PMC open access articles (PMCOA) published in the PubMed-indexed dental journals in the database until 31 December 2021. Then, we searched for the phrases “data not shown” and “results not shown” from the PMCOA articles published in the same journals until the same date.

Selection of 2021 subsample

From those searches, we selected studies published in 2021. From studies for which the search indicated that they included the phrases, we manually confirmed whether the phrase referred to unreported/unpublished results. If they did so, we included them for further analysis.

Data extraction from the 2021 subsample

From studies published in 2021 with the “data/results not shown” phrases, we documented whether the study shared data or code or online supplementary materials/appendices within the journal website or via other platforms. Then, we categorised the studies based on whether the “data/results not shown” phrases referred to confirmatory results, negative results, peripheral results, sensitivity analysis results, future results or other/unclear categories (Table 1) [11]. We also searched for information about publishing free online supplementary materials from the websites of all journals which had published at least one paper in the subsample (yes unlimited, yes limited, unclear, no). We searched the name of the publisher of each journal from Publons (and for sensitivity analysis also from SCImago and National Library of Medicine, NLM, Catalog). All data extractions from full texts were performed first by one of the authors, and all extractions were confirmed/checked by another author; discrepancies were solved through discussion.

Table 1

Definitions of phrase types with examples, adapted from [11].

Type	Definition	Example from the sample
Confirmatory results	Demonstrating the reproducibility of previous findings or demonstrating validity methods (e.g., negative controls).	“As elaborated in a preliminary experiment (data not shown), the amount of biofilm on the outer surface of the experimental abutments in both treatment groups was very variable.” [16]
Negative results	Results that did not show statistically significant effect/association on some threshold, e.g., p > 0.05.	“The average loss of lingual ridge height between the four groups was 0.6–1.0 mm, with no significant difference among the groups (p > 0.05; data not shown).” [17]
Sensitivity analysis results	Sensitivity analyses assess the robustness of results, for instance, the impact of including or excluding some variables from the analysis.	“In addition, sensitivity analyses conducted, limiting the data to those with known values of isoprostanes and plasma carbonyls, yielded similar results (data not shown).” [18]
Peripheral results	Referring to results not directly relevant to the main topic of article, often mentioned in the discussion section.	“Results of different studies revealed moderate correlations between the MDAS [Modified Dental Anxiety Scale] and dentists’ observations (0.4 to 0.66) [9, 34, 37]. In our study, the strength of the correlation between the dentist’s observations and MDAS scoring was also moderate (results not shown).” [19]
Future results	Results which are going to be represented in other subsequent papers, often mentioned in the discussion.	Not in the sample
Other/unclear	For instance, related to statistical procedure selection (e.g., normality tests), studies with multiple phrases and varying purposes, and phrases with vague purposes.	“Faced with the intensification of these already existing barriers, the population would have a greater tendency to search the Internet for ways of self‐resolution of toothache, especially in developing countries (United Nations, 2020), which presented significantly higher RSV values than developed ones (data not shown).” [20]

Data synthesis and analyses

For each publication year, we calculated the percentage of studies with the “data/results not shown” phrases from the total number of PMCOA articles published in the same dental journals during the year. We also searched the phrases from all PubMed-indexed PMCOA articles and wrote down the returned number of hits for each year to make a comparison with PMCOA articles from dental journals. We reported our findings with simple descriptive tables and figures, as well as provided some examples of how the phrases were used. Journal- and publisher-related differences in the number of studies with the phrase, from the total number of PMCOA articles from each journal or publisher in 2021, were tested using Fisher’s exact test with Monte Carlo simulations, with 100 000 replications. We used the R v4.1.1 software (2021-08-10, R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org) for statistical analysis.

Results

Overall perspective

The search from the Europe PMC database identified 21 217 unique PMCOA articles from 116 different dental journals until 31 December 2021. Of these, the search for “data/results not shown” phrases produced 1 474 unique records from 70 various dental journals.

As the total number of PMCOA dental articles was low before 2010, there was considerable fluctuation in the percentage of studies with the phrases. As the number of available dental articles increased, the percentage of studies with the phrases stabilised to around 13% by 2010. From 2010, the percentage decreased to approximately 3% in dental journals from 2010 to 2020 (Fig 1A).

Fig 1

The number of available PubMed Central open access articles with “data/results not shown” phrase from dental (A) and all PubMed-indexed PMCOA (B) journals by publication year (bars), and percentage of articles with at least one “data/results not shown” phrase within each year (line).

Amongst all PubMed-indexed PMCOA articles the percentages were clearly higher in all study years than in dental journals. The percentage of articles with the phrases decreased from 1990 (35%) to 2020 (6%) (Fig 1B). In both samples, these trends showed a steeper downtrend from the late 2000s onwards.

2021 subsample

In 2021, the search identified 2 434 unique PMCOA articles from 73 dental journals and 22 publishers. Of these, the search identified 67 PMCOA articles (from 22 different journals and eight publishers), with at least one of either of the “data/results not shown” phrases. In the full-text review, all 67 were confirmed as including the phrase(s). Thus, 2.8% of the full texts in 2021 referred to unpublished results with the phrase. Fisher’s exact test showed a p-value of <0.001 for the journal-related, and 0.002 for the publisher-related differences in the percentages of studies with the phrase(s), from the total number of available PMCOA articles, from each journal or publisher in 2021 (Tables 2 and 3).

Table 2

The number of available open access articles from Europe PubMed Central and the number of articles with “data/results not shown” phrases in each journal.

Journal	Total number of studies	Studies with “data/results not shown”	Proportion (%) of studies with “data/results not shown”
BMC Oral Health	672	24	3.5
J Appl Oral Sci	64	6	9.4
Int J Oral Sci	45	4	8.9
Clin Exp Dent Res	81	3	3.7
Clin Oral Investig	144	3	2.1
J Am Dent Assoc	41	3	7.3
J Clin Periodontol	13	3	23.1
J Dent Res	35	3	8.6
J Periodontol	4	3	75.0
Head Face Med	50	2	4.0
Int J Implant Dent	118	2	1.7
Angle Orthod	116	1	0.9
Br J Oral Maxillofac Surg	13	1	7.7
Int Endod J	2	1	50.0
J Oral Pathol Med	2	1	50.0
J Periodontal Res	6	1	16.7
Mol Oral Microbiol	2	1	50.0
Odontology	13	1	7.7
Oral Dis	33	1	3.0
Pediatr Dent	4	1	25.0
Others (51 journals)	865	0	0
Total	2434	67	2.8

Table 3

The number of available open access articles from Europe PubMed Central and the number of articles with “data/results not shown” phrases in journals of each publisher.

Publisher	Total number of studies	Studies with “data/results not shown”	Proportion (%) of studies with “data/results not shown”
Springer Nature	1100	36	3.3
John Wiley & Sons	294	15	5.1
Faculdade de Odontologia de Bauru	64	6	9.4
American Dental Association	41	3	7.3
SAGE Publishing	56	3	5.4
Allen Press	140	1	0.7
Churchill Livingstone	15	1	6.7
Others (13 publishers)	544	0	0
Total	2434	67	2.8

Of those 67 studies with the phrases, the phrases related most often to peripheral (n = 22, 33%), negative (n = 16, 24%) or confirmatory (n = 11, 16%) results. Few referred to the sensitivity analysis results (n = 4, 6%). However, in some cases, it was difficult to evaluate their meaning or what results the phrase referred to (n = 14, 21%). Nineteen studies used the phrase(s) multiple times (twelve studies two times, six studies three times, one study four times).

Some authors seemed to use the phrase just to indicate that certain numbers were not represented in the tables or figures but while providing data or results in the same sentence with words; for example: “The ratio of patients showing a history of head and neck cancer (19/47 vs. 14/97, P   =  0.0007, data not shown)…” [21].

However, in some articles, notable conclusions were made based on data not shown. For instance, in a study investigating the association of preventive dental care to healthcare outcomes, tooth extractions and endodontic treatments were given considerable attention in terms of methodological decisions, results and their interpretation, but for restorative treatments, it was just stated in the discussion that, “Furthermore, we examined the effect of receiving restorative dental care on health outcomes, but no associations were seen (data not shown)” [22]. Other examples of how the phrases were used in the studies are provided in Table 1.

Thirty-six of the studies (54%) included a data sharing statement. Three studies shared data, and no study shared code, while 27 studies (40%) included supplementary material. The search from the journal websites of 22 journals that published at least one study with the phrase(s) showed that 20 would publish online supplementary materials attached to the research articles for free, without limits. From two journals, we were unable to detect the information from their websites (The Angle Orthodontist and Medicina Oral, Patologia Oral, Cirugia Bucal). Sensitivity analysis for publisher-related differences is available in S2 Text.

Discussion

In agreement with promising trends in open science practices in biomedical studies over the last decades [5], we found that the proportion of PMCOA articles in dental journals that used the “data/results not shown” phrases had decreased significantly, from over 10% to approximately 3%, during the last decade.

We also investigated the use of phrases in 2021 in more detail. These years researchers’ have had plenty of possibilities of sharing all kinds of data via numerous free and accessible platforms. Findings showed that from all PMCOA articles published in dental journals in 2021, 67 (2.8%) studies from 22 different journals used the “data/results not shown” phrases to refer to unpublished results. We found that there were differences in the use of the phrases between dental journals and publishers. Most commonly the phrases referred to negative, peripheral or confirmatory results. The significance of unpublished results to which the phrases referred varied considerably across studies. From the 67 studies, three studies shared raw data, and no study shared code.

Our findings showed a decreasing trend of PubMed-indexed PMCOA articles with the phrase “data/results not shown” from 1999 onwards. This trend, however, showed a steeper downtrend from 2008. This occurred after the publication of data availability editorials in Nature journals, starting from 2006 [23-25]. Thereafter, several pieces of evidence tried to elaborate and express the concerns about data availability and reproducibility of results [26-29]. In 2016, these concerns were translated into a policy of a mandatory statement on including information on whether and how others can access the underlying data for all research papers accepted for publication in Nature [30].

Some reasons for the use of the phrases to refer to unpublished results can be postulated. First, pressure to publish articles and minimising the amount of work may be one reason, that is also seen to be behind other poor scientific practices [31, 32]. In short, the results or data is not seen as worthy of publishing. Secondly, some of the results or data may be hard or impossible to share. Thirdly, as we showed, some authors just used the phrase to indicate that the results were not given in table or graphical format but were given only in text and so the results or data were not actually unpublished. However, what we see as an important reason, is that the transparency of science has not given the value in scientific practice it deserves. On the positive side, at least authors using such phrases make it honestly clear that the data or results are not shared.

Many of the studied PMCOA articles were from open access dental journals, indexed in PubMed (like BMC Oral Health and Clinical and Experimental Dental Research). Nieminen and Uribe [33] showed that in non-predatory (legitimate and indexed by established databases) open access dental journals, the presentation of results (particularly in tables and figures) was poorer than in more visible subscription-based dental journals but still better than in predatory (non-indexed) dental journals (from predatory publishers). Since referring to “data not shown” evidently is an obscure way of presenting results, it may be related to how results are presented in these studies in general.

Our findings showed that the proportion of all PubMed-indexed PMCOA articles with “data/results not shown” phrases was considerably higher than in PubMed-indexed dental journals. Whereas the decreasing trend was evident in both, all PubMed-indexed PMCOA articles had a two-fold proportion in 2021 compared with dental journals. This potentially implies subject-related differences in the use of the phrases. So, investigating these differences in a further study could provide a better picture of the current situation.

Implications for research policy

Solutions to enhance the movement towards open science through abandoning “data/results not shown” can be postulated. The strictest solution could be banning the use of these phrases, accompanied by editorial requests for providing the data or results not shown, as some journals and publishers have done [12]. However, a more sustainable solution could be wider adoption of open science practices, as particularly free data and code sharing have remained rare in biomedical literature over the last decades [5]. In open access journals advocating for more open science [34, 35], the obscure representation of results as well as not sharing data or code should not be overlooked by the publishers or editorial teams. For instance, during the study process, we noted a study which used the raw data availability statement template without any changes: “The data that support the findings of this study are openly available in [repository name e.g., “figshare”] at http://doi.org/[doi], reference number [reference number]” [36]. Thus it seems that despite data availability statements being mandatory in some journals, the actual content of the statement does not always receive careful consideration. Evidently, we need more commitment to open science principles from all stakeholders in science.

Limitations

First, it is evident that the use of these phrases is not the only way to refer to unpublished results or data. Secondly, it is unknown how well PMCOA articles from dental journals represent the wider dental literature because subscription-based journals are underrepresented in the database of open access articles. However, at least in terms of transparency indicators, the differences between PubMed-indexed and PMCOA articles might be small [37]. In addition, it is worth noting that the composition of the PMCOA database varies over time due to changes in open access practices and differences in how soon after publication, the journal’s articles are made available to the database (ncbi.nlm.nih.gov/pmc/journals). Further, the total number of articles included all types of papers (commentaries, letters, etc.); hence, the proportion of studies with the phrase may be higher than what was found if solely research articles had been considered. Although the analysis of PMCOA articles published in 2021 showed that the search from the database retrieved no false positives (studies without the phrases), we cannot be sure about the actual false-positive rates before 2021 or about the false-negative rate (missed studies with the phrases) of our identification strategy. Additionally, due to a large and heterogeneous sample of studies, we were unable to detect how the use of the phrases was related to other critical characteristics of studies, e.g., risks of bias. Finally, it must be noted that we do not know whether reviewers or editors had seen results to which “data not shown” referred during the peer-reviewing process which could thus justify the use of the phrase to some extent. However, editor experiences and studies have shown that (raw) data to support the findings of a study may not be shared despite reasonable requests, and sometimes given data doesn’t support the conclusions made from it [38, 39].

Conclusions

We showed that a great decrease in the use of the selected phrases occurred in PMCOA articles published in PubMed-indexed dental and other journals over the last decades. However, dental or other researchers have not completely abandoned the outdated caveat of “data/results not shown”, and it was still seen to be in use in 2021. Researchers, reviewers, editorial teams and publishers are responsible for further promoting and adopting open science practices, including providing all results, data and code, whenever possible, in a freely accessible online format, one way or another; fortunately, it is possible today.

Deviations from the protocol.

(DOCX)

Click here for additional data file.

Sensitivity analysis for publisher-related differences.

(DOCX)

Click here for additional data file.

24 Jan 2022

PONE-D-21-34697

Movement towards open science: Have dental researchers abandoned the outdated caveat “data not shown”?

PLOS ONE

Dear Dr. Sofi-Mahmudi,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we have decided that your manuscript does not meet our criteria for publication and must therefore be rejected.

I am sorry that we cannot be more positive on this occasion, but hope that you appreciate the reasons for this decision.

Yours sincerely,

Peter Eickholz

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The reviewer's concerns were expressed in the attached marked-up PDF document. The authors are required to better justify the necessity of this research, given the actual possibilities of a reviewer to require all available data/results, when needed.

Reviewer #2: Dear authors,

The Manuscript ID PONE-D-21-34697 entitled " Movement towards open science: Have dental researchers abandoned the outdated caveat “data not shown”?" analyses the evolution in time of the use of the phrases “data/results not shown” and to what kind of results the authors referred to with these phrases in 2021.

Although the paper is beautifully written and has few changes to be made, I am afraid I don´t find the interest in the topic. The conclusions reached add little innovation and, although I think editors should really handle this information, I don´t know if readers will get much from it.

Specifically, there are some mistakes in the text that should be corrected:

- What do authors mean by “open access”? Some of the indexed journals included in the sample are not available in open access, only subscription-based; I am afraid I don´t really understand how you made the selection of journals and editors.

- Page 4 line 44: There is a missing point after the (1-4)

- Page 6 line 98: the citation is badly spelled “m [11)”.

- The authors establish a comparison between their results from 2021 and previous data. Nevertheless, I don´t get to find in the text the original source from where you achieved previous figures. Allusions to the trends in the past are made and cited in the Discussion, but not in the Results, where the authors first make the comparison.

- Page 11 line 160: the use of the adverb “Interestingly” reads sarcastic and complicates the understanding of the statement.

- The quality of the figures is quite poor; I don´t know if it would be possible to upload them in higher resolution.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

- - - - -

For journal use only: PONEDEC3

Submitted filename: marked-upPONE-D-21-34697_reviewer.pdf

Click here for additional data file.

26 Mar 2022

Authors’ response: We thank the reviewers for their constructive and insightful comments. We see that the main reason for the rejection was that the reviewers didn’t consider the study worth conducting or publishing. We definitely agree with the reviewers that the importance of the study wasn’t well communicated in the submitted version.

First, what our study shows is what kinds of results the phrase “data not shown” refer to in the studies these days. We are unaware of any systematically conducted study that investigated what kind of unpublished results the phrases actually refer to. We see the investigation we made important for four reasons:

1) If researchers report a considerable amount of results or important results with such phrases, and without sharing results, data or code, free interpretation and verification of results are doubly harder or even impossible. In other words, the use of phrases obscures the transparency, reproducibility and weakens the peer review process.

2) As we showed, these phrases referred many times to non-statistically significant (negative) results. Focusing on statistically significant results and neglecting the negative ones is a major reason behind publication bias, but may also threaten the reproducibility of scientific results as argued by Amrhein et al (2017).

Amrhein V, Korner-Nievergelt F, Roth T. The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research. PeerJ. 2017 Jul 7;5:e3544.

3) Proper interpretation of sensitivity analyses require that modelling modifications, parameters and sensitivity results are adequately reported. As we showed, few papers referred to these kinds of results with the phrases.

Lash TL, Fox MP, MacLehose RF, Maldonado G, McCandless LC, Greenland S. Good practices for quantitative bias analysis. Int J Epidemiol. 2014 Dec;43(6):1969-85.

4) Results not considered important for the purposes of the current study may be crucial to conduct a systematic review or meta-analyses on the closely related topic. As we showed, many papers referred to peripheral (not directly related to the topic of study) or negative results with the phrases.

We have revised the introduction and discussion accordingly. We also made most of the revisions the reviewers suggested. We also made our analyses to cover the whole 2021 instead of including only part of 2021.

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: I Don't Know

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

Authors’ response: All data and code produced and analyzed during the study are shared via Open Science Framework https://osf.io/5zryu/

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The reviewer's concerns were expressed in the attached marked-up PDF document. The authors are required to better justify the necessity of this research, given the actual possibilities of a reviewer to require all available data/results, when needed.

[We extracted the comments from the PDF:]

Although originating in the text of an authorless note (editorial ?) in Nat Chem Biol, the title seems pretentious and bombastic. This is a study of scientometrics, not a chapter of a mystery novel. Besides, the meaning is not clear. As far as the reviewer knows, a "caveat" means "warning", "caution". As such, a caveat cannot be characterized neither as "fashionable", nor as "outdated". Even more, the results of the research show clearly that the mention "data not shown" seems not to be outdated, but rather reduced as frequency of use. Please try to modify the title as to reflect the content of the research.

Authors’ response: We edited the title.

The phrase is unclear. Please reformulate.

Authors’ response: The sentence was reformulated.

This source-phrase sounds pretentious, and, at a closer look, is trivial. Hidden data always affect the transparency of a research and weakens the peer review.

Authors’ response: We reformulated the sentence.

Please justify the necessity of such a research. At what extent so far has actually the phrase "data/results not shown" produced damages in terms of interpreting the results and drawing sound conclusions? It si still not clear why this research was needed.

Authors’ response: We agree with the reviewer that the importance of the study wasn’t elaborated clearly enough. Please see the beginning of the response,

OSF means...?

Authors’ response: Open Science Framework, it was stated in the introduction.

Please explain why 2021 was chosen as a year of reference?

Authors’ response: Most recent year. Current use of the phrase is most interesting because every day sharing data and other material becomes easier and easier.

How does apply this statement to the present research?

Authors’ response: The sentence was removed.

In the Discussion section, authors should investigate the significance of the mention "data not shown" in the context of an undeterrable request for data from the reviewers. In other words, how effective could be the mention "data/results not shown", knowing that these data/results can be always requested by the reviewers?

Authors’ response: This is now mentioned in the limitations section.

Based on the distribution of the hidden data/results, in the Discussion section, authors should further speculate what could be the real reason(s) for not showing the data/results.

Authors’ response: We added a paragraph about the reasons for the use of the phrase.

Since discussing the reasons: the reviewer's experience shows that in many cases less relevant data are poorly presented/not shown, and this is not even stated as such. Given this reality, could the mention "data/results not shown" be seen as a sign of ultimate, twisted honesty from the part of the authors?

Authors’ response: This is now stated in the paragraph we added to the discussion.

It is not clear what "false-positive" or "false-negative" means in this context. Please explain.

Authors’ response: This was clarified.

Reviewer #2: Dear authors,

The Manuscript ID PONE-D-21-34697 entitled " Movement towards open science: Have dental researchers abandoned the outdated caveat “data not shown”?" analyses the evolution in time of the use of the phrases “data/results not shown” and to what kind of results the authors referred to with these phrases in 2021.

Although the paper is beautifully written and has few changes to be made, I am afraid I don´t find the interest in the topic. The conclusions reached add little innovation and, although I think editors should really handle this information, I don´t know if readers will get much from it.

Authors’ response: We agree with the reviewer that the importance of the study wasn’t elaborated clearly enough. Please see the beginning of the response,

Specifically, there are some mistakes in the text that should be corrected:

- What do authors mean by “open access”? Some of the indexed journals included in the sample are not available in open access, only subscription-based; I am afraid I don´t really understand how you made the selection of journals and editors.

Authors’ response: All PubMed-indexed dental journals were selected based on a list provided by National Library of Medicine (NLM). Open access in this context means that the article is freely accessible via Europe PubMed Central database. Therefore, the study sample also includes studies from subscription-based journals.

- Page 4 line 44: There is a missing point after the (1-4)

- Page 6 line 98: the citation is badly spelled “m [11)”.

- The authors establish a comparison between their results from 2021 and previous data. Nevertheless, I don´t get to find in the text the original source from where you achieved previous figures. Allusions to the trends in the past are made and cited in the Discussion, but not in the Results, where the authors first make the comparison.

Authors’ response: These were corrected to manuscript.

- Page 11 line 160: the use of the adverb “Interestingly” reads sarcastic and complicates the understanding of the statement.

Authors’ response: This was removed.

- The quality of the figures is quite poor; I don´t know if it would be possible to upload them in higher resolution.

Authors’ response: Figures were improved.

Submitted filename: Response to reviewers - PLOS ONE.docx

Click here for additional data file.

6 Jun 2022

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Rafael Sarkis-Onofre

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

Additional Editor Comments (if provided):

Academic Editor:

The comments of reviewer 3 should be considered to improve the article.

My main concern is about including only studies published in 2021. Authors should highlight in the discussion why this decision and what is the impact of that.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #3: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #3: This is a well-written paper that addressed how frequently papers published in dental journals use the phrases “data/results not shown" and what kind of results the authors referred to with these phrases in 2021. Some study limitations cannot be disregarded, especially the underrepresentation of subscription-based journals and the lack of information on the true reasons for using such phrases. Despite that, I acknowledge that the exploratory nature of these findings can give us insightful information regarding open access practices in dental journals.

The authors have substantially improved the first version of the paper. However, I strongly suggest authors change the title. Surely the authors made a systematic search. However, this was not a systematic review, but a scientometric analysis. Also, the authors should explain why the protocol for this study was not prospectively registered.

Another comments:

Introduction, last paragraph: “Accordingly, from relevant studies in 2021, we examined….” Please consider replacing the word relevant with eligible.

Material and methods: the topic “study selection” should be rewritten and divided into three topics: “protocol registration”, “bibliographic search”, and “eligibility criteria” (or similar topics). Information regarding these three topics is all mixed in “study selection”, and the reader may find it difficult to understand.

Results: “The percentage of studies with the phrases from all the PMCOA articles decreased from 13% to approximately 3% in dental journals from 2010 to 2020 (Figure 1A)”. It is not clear why the years 2010 and 2020 were chosen.

In S4 Text (Sensitivity analysis for publisher-related differences), please provide the raw numbers and not only the p values.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

11 Jun 2022

We thank the editor and reviewers for the opportunity to submit our revised manuscript.

The comments of reviewer 3 should be considered to improve the article.

My main concern is about including only studies published in 2021. Authors should highlight in the discussion why this decision and what is the impact of that.

Response: Our aim was to investigate reasons for current use of the phrases, because nowadays researchers have many options to share data and results freely and easily. We didn’t see it worthwhile investigating the use of phrases in more detail from the whole sample, because to some extent the use of phrases has been acceptable (for instance before widespread use of the internet).

Reviewer #3: This is a well-written paper that addressed how frequently papers published in dental journals use the phrases “data/results not shown" and what kind of results the authors referred to with these phrases in 2021. Some study limitations cannot be disregarded, especially the underrepresentation of subscription-based journals and the lack of information on the true reasons for using such phrases. Despite that, I acknowledge that the exploratory nature of these findings can give us insightful information regarding open access practices in dental journals.

Response: We thank the reviewer for his/her insightful comments.

The authors have substantially improved the first version of the paper. However, I strongly suggest authors change the title. Surely the authors made a systematic search. However, this was not a systematic review, but a scientometric analysis.

Response: We thank the reviewer for this good comment. We removed the type of study from the title completely: The use of the phrase “data not shown” in dental research

Also, the authors should explain why the protocol for this study was not prospectively registered.

Response: The reviewer is right that the protocol was not registered beforehand, it was just shared publicly beforehand. That was our fault, we deemed it acceptable to just publish it beforehand instead of using registers such as those available via Open Science Framework. However, this doesn’t change the fact that we have worked closely following the open science principles when conducting this study.

Another comments:

Introduction, last paragraph: “Accordingly, from relevant studies in 2021, we examined….” Please consider replacing the word relevant with eligible.

Response: We edited the sentence in this regard, but also to highlight why we conducted in more detail analysis of those articles that used the phrases in 2021.

Material and methods: the topic “study selection” should be rewritten and divided into three topics: “protocol registration”, “bibliographic search”, and “eligibility criteria” (or similar topics). Information regarding these three topics is all mixed in “study selection”, and the reader may find it difficult to understand.

Response: We thank the reviewer for pointing this out. We edited and added more subsections with more detail subheaders: “protocol registration”, “bibliographic search”, “Selection of 2021 subsample”, “Data extraction from the 2021 subsample” and “Data synthesis and analyses“)

Results: “The percentage of studies with the phrases from all the PMCOA articles decreased from 13% to approximately 3% in dental journals from 2010 to 2020 (Figure 1A)”. It is not clear why the years 2010 and 2020 were chosen.

Response: We thank the reviewer for pointing this out. We improved the whole paragraph and added a more detailed description of our findings:

“As the total number of PMCOA dental articles was low before 2010, there was considerable fluctuation in the percentage of studies with the phrases. As the number of available dental articles increased, the percentage of studies with the phrases stabilised to around 13% by 2010. From 2010, the percentage decreased to approximately 3% in dental journals from 2010 to 2020 (Figure 1A).

Amongst all PubMed-indexed PMCOA articles the percentages were clearly higher in all study years than in dental journals. The percentage of articles with the phrases decreased from 1990 (35%) to 2020 (6%) (Figure 1B). In both samples, these trends showed a steeper downtrend from the late 2000s onwards.”

In S4 Text (Sensitivity analysis for publisher-related differences), please provide the raw numbers and not only the p values.

Response: We thank the reviewer for pointing this out. Raw numbers for journal- and publisher-related differences are now provided in supplementary material available at https://osf.io/5zryu/.

Submitted filename: Response to reviewers_rev2.docx

Click here for additional data file.

26 Jul 2022

The use of the phrase “data not shown” in dental research

PONE-D-21-34697R2

Dear Dr. Sofi-Mahmudi,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Rafael Sarkis-Onofre

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #3: The authors have adequately addressed all the issues previoulsy raised and therefore I recommend publication.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #3: No

**********

28 Jul 2022

PONE-D-21-34697R2

The use of the phrase “data not shown” in dental research

Dear Dr. Sofi-Mahmudi:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Rafael Sarkis-Onofre

Academic Editor

PLOS ONE