A systematic analysis of preprints in Trauma & Orthopaedic surgery.

AIMS: Preprint servers allow authors to publish full-text manuscripts or interim findings prior to undergoing peer review. Several preprint servers have extended their services to biological sciences, clinical research, and medicine. The purpose of this study was to systematically identify and analyze all articles related to Trauma & Orthopaedic (T&O) surgery published in five medical preprint servers, and to investigate the factors that influence the subsequent rate of publication in a peer-reviewed journal.
METHODS: All preprints covering T&O surgery were systematically searched in five medical preprint servers (medRxiv, OSF Preprints, Preprints.org, PeerJ, and Research Square) and subsequently identified after a minimum of 12 months by searching for the title, keywords, and corresponding author in Google Scholar, PubMed, Scopus, Embase, Cochrane, and the Web of Science. Subsequent publication of a work was defined as publication in a peer-reviewed indexed journal. The rate of publication and time to peer-reviewed publication were assessed. Differences in definitive publication rates of preprints according to geographical origin and level of evidence were analyzed.
RESULTS: The number of preprints increased from 2014 to 2020 (p < 0.001). A total of 38.6% of the identified preprints (n = 331) were published in a peer-reviewed indexed journal after a mean time of 8.7 months (SD 5.4 (1 to 27)). The highest proportion of missing subsequent publications was in the preprints originating from Africa, Asia/Middle East, and South America, or in those that covered clinical research with a lower level of evidence (p < 0.001).
CONCLUSION: Preprints are being published in increasing numbers in T&O surgery. Depending on the geographical origin and level of evidence, almost two-thirds of preprints are not subsequently published in a peer-reviewed indexed journal after one year. This raises major concerns regarding the dissemination and persistence of potentially wrong scientific work that bypasses peer review, and the orthopaedic community should discuss appropriate preventive measures.Cite this article: Bone Jt Open 2022;3(7):582-588.

Introduction

Preprint servers allow authors to publish full-text manuscripts or interim findings prior to undergoing a peer-review process. After the initial use of preprint servers in basic research, including mathematics and physics, several preprint servers extended their services to biological sciences, clinical research, and medicine. The potential advantages include rapid publication, allowing immediate public dissemination of preliminary or novel research findings, and the possibility to provide and receive feedback on the work from peers, all free of charge. Supporters of preprint servers emphasize that preprint servers improve the prompt spread of scientific information.

However, with the use of preprint servers, sometimes two possibly conflicting versions of the same content exist, both of which could potentially be considered evidence. Most importantly, distinguishing a non-peer-reviewed preprint from a seminal article in a leading journal is increasingly difficult for readers. Thus, preprint servers artificially inflate the body of knowledge and make it even harder to reliably find and apply usable knowledge, as the important selection process through peer review is omitted. This is the very reason why several high-ranked orthopaedic journals do not accept manuscripts published previously on preprint servers due to concerns about publication and dissemination of false results or scientific fraud without undergoing a meticulous peer review process, whereas others, such as the British Medical Journal (BMJ) and The Lancet, promote the use of preprints.

Overall, opinions about the role of preprints in orthopaedics remain conflicted and will most likely gain importance with a growing number of publications. The increase in online journals, open-access online publications, and promotion on social media platforms adds to the flood of information. With a growing quantity of research and facilitated publication options, including preprint servers, it is increasingly difficult for the orthopaedic clinician to cut out the noise and identify correct and meaningful research that will benefit the patient. To date, the extent to which published preprints are subsequently published in peer-reviewed journals and the factors that favour successful publication remain unclear.

To gain insights into the future role of preprints in the orthopaedic field, the purpose of this study was to systematically identify and analyze all articles published in five medical preprint servers related to Trauma & Orthopaedic (T&O) surgery. We hypothesized that the publication of preprints has increased over time, and that the rate of successful subsequent publication in a peer-reviewed indexed journal is dependent on the geographical origin and level of evidence covered.

Methods

Search strategy

Five medical preprint servers (medRxiv, OSF Preprints, Preprints.org, PeerJ, and Research Square) were searched from server launch to 1 August 2020, to allow a minimum 12-month period for definitive publication of the index preprint. The search was conducted from 5 August 2021 to 22 August 22 2021 using the following terms: “orthopedic,” “orthopedic surgery,” “orthopaedic,” “orthopaedic surgery,” “trauma surgery,” “traumatology,” “hand,” “shoulder,” “elbow,” “spine,” “hip,” “knee,” “foot,” and “ankle.” As the possibilities to search the databases differed among the preprint servers, the detailed search strategy is summarized in Table I. Three reviewers (FS, SM, SMM) independently screened the titles, abstracts, and full texts according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. The inclusion and exclusion criteria are summarized in Figure 1. Disagreements about inclusions or exclusions were resolved by consensus between the reviewers.

Table I.

Overview of preprint servers and search strategy.

Server	Description	Launch year	Search strategy applied
medRxiv	Free online archive for preprints in the medical, clinical, and related health sciences	2019	Keywords in full text, abstract, or title in all medical categories
OSF Preprints	Free online archive for preprints covering life, biology, medicine, and environmental sciences	2017	Keywords in category “Medicine and health sciences”
Preprints.org	Free online archive covering life, biology, medicine, and environmental sciences	2016	Keywords in category “Medicine and Pharmacology”
PeerJ*	Free online archive covering life, biology, medicine, and environmental sciences	2013	Keywords in “orthopaedics», «surgery and surgical-specialties», «anatomy and physiology”
Research Square	Free online archive covering life, biology, medicine, mathematics and environmental sciences	2018	Keywords in categories “Orthopaedic” and “orthopaedic surgery”

PeerJ stopped accepting preprints on 30 September 2019.

Fig. 1

Flowchart and eligibility of preprints.

Assessment of definitive peer-reviewed publications and journals

The title, keywords, and corresponding author of all the included preprints (n = 858) were searched for a subsequent peer-reviewed publication in the following databases: Google Scholar, PubMed, Scopus, Embase, Cochrane, and the Web of Science. This search was conducted from 22 October 2021 to 1 December 2021. Successful publication of a work was defined as publication in a peer-reviewed journal indexed in one of the aforementioned databases. The peer-reviewed publications were then assessed for time from preprint to peer-reviewed publication, geographical origin, type of research, anatomical region covered, and time available online (defined as the interval between the preprint publication date and the start date of the search of the preprint servers). The geographical origin was defined as the location of the institution listed for the corresponding author.

The type of research was categorized as either basic/animal or clinical. For all clinical studies, the level of evidence was defined according to guidelines published by The Journal of Bone & Joint Surgery. As preprints have not undergone peer review, the exact levels of evidence were not stated in all works. To overcome this limitation and to enable a distinct classification in all preprints, the levels of evidence were grouped and defined as follows: Level V: case report, technical note, expert opinion, or narrative review; Level III/IV: retrospective cohort studies or case series; and Level II/I: prospective studies, randomized controlled trials, systematic reviews, or meta-analyses. All journals that published definitive articles were identified and analyzed for the type of peer review (not blinded, single-blinded, or double-blinded), and the journal impact factor was based on the journal citation report of 2020 retrieved from the Web of Science (Supplementary Table i).

Statistical analysis

Numeric data are reported as mean and standard deviation (SD) and range, and categorical data are reported as number (percentage). Normal distribution was assessed using the Kolmogorov-Smirnov test. The increase in publication of preprints in 2020 compared to previous years was assessed using a binomial test. To compare differences in the subsequent definitive publication rate according to preprint server, geographical origin, type of research, and anatomical region, a chi-squared test was performed. Difference in time available online between subsequently published and unpublished preprints was assessed using the Mann-Whitney U test. The significance was set at p < 0.05. The data were analyzed with SPSS v. 26 (SPSS, USA).

Results

After applying the inclusion and exclusion criteria, a total of 858 preprints were included in the final analysis (Figure 1). Overall, 38.6% of the preprints (n = 331) were published in a peer-reviewed indexed journal after a mean time of 8.7 monhts (SD 5.4; 1 to 27). For the period 2014 to 2020, 178 preprints were identified; this number increased to 680 published preprints in 2020 (p < 0.001, binomial test) (Figure 2).

Fig. 2

Publication of preprints over time. The bar chart depicts the number of published preprints for each year. *Depicts a significant increase of preprints in 2020 compared to the previous years (p < 0.001, binomial test).

Most articles (n = 650; 75.8%) originated from Asia/Middle East and covered spine, hip, or knee surgery (n = 557; 64.9%) (Table II; Figure 3). The preprint server, geographical origin, level of evidence, and anatomical region all demonstrated significant differences in the proportion of successful subsequent publications (Table II). The highest proportion of missing subsequent peer-reviewed publications was found in the preprints originating from Africa, Asia/Middle East, and South America (p < 0.001, chi-squared test) or covering clinical research with a lower level of evidence (p < 0.001, chi-squared test) (Table II; Figure 3 and Figure 4).

Table II.

Subsequent definitive publication according to geographical origin, type of research, and anatomical region.

Variable	Preprints identified, n (%)	Subsequent peer-reviewed publication, n (%)*	p-value†
Total	858 (100)
Preprint server			< 0.001
medRxiv	141 (16.4)	71 (50.4)
OSF	7 (0.8)	4 (57.1)
Preprints.org	7 (0.8)	7 (100)
PeerJ	20 (2.3)	15 (75.0)
Research Square	683 (79.6)	234 (34.3)
Geographical origin			< 0.001
Europe	126 (14.7)	59 (46.8)
North America	48 (5.6)	32 (66.7)
South America	8 (0.9)	2 (25.0)
Asia/Middle East	650 (75.8)	227 (34.9)
Africa	7 (0.8)	1 (14.3)
Oceania	19 (2.2)	10 (52.6)
Type of research			< 0.001
Basic/animal	138 (16.1)	76 (55.1)
Level V	51 (5.9)	6 (11.8)
Case report	21	1 (4.8)
Technical note	25	3 (12.0)
Expert opinion/narrative review	5	2 (40)
Level IV/III	506 (59.0)	177 (35.0)
Retrospective cohort study or case series	506	177 (35.0
Level II/I	163 (19.0)	72 (44.2)*
Prospective/RCT	113	52 (46.0)
Meta-analysis/systematic review	50	20 (40)
Anatomical region			0.007
Hand	26 (3.0)	12 (46.2)
Shoulder & elbow	75 (8.7)	30 (40.0)
Spine	229 (26.7)	72 (31.4)
Hip	148 (17.2)	46 (31.1)
Knee	180 (21.0)	83 (46.1)
Foot & ankle	64 (7.5)	24 (37.5)
Not specified	136 (15.9)	64 (47.1)

Percentages derived from the number of intial preprints.

Chi-squared test.

RCT, randomized controlled trial.

Fig. 3

Subsequent definitive publication according to geographical origin. The bar chart depicts the number (%) of subsequently published preprints (blue) and unpublished preprints (red) according to geographical origin.

Fig. 4

Subsequent definitive publication of clinical research according to level of evidence. The bar chart depicts the number (%) of subsequently published preprints (blue) and unpublished preprints (red) covering clinical research according to the level of evidence (I to V).

The preprints that were definitively published had been available online longer compared to the unpublished preprints: 18.4 months (SD 9.4; 12 to 88) versus 16.5 months (SD 6.5; 12 to 89), respectively (p = 0.004, Mann-Whitney U test).

We identified 153 journals that subsequently published preprints. The peer review type was reported as not blinded (n = 22; 14.4%), single-blinded (n = 69; 45.1%), double-blinded (n = 55; 35.9%), or not applicable (n = 7; 4.6%) (Supplementary Table i).

Discussion

As controversy persists regarding the role of preprints in the medical field, this study is the first to provide a systematic analysis of preprints published in T&O surgery. Both our hypotheses were confirmed. As assumed, preprints have increased dramatically in T&O surgery in recent years. This is of concern, as there are no scientific quality control mechanisms for such publications, and exposure to potentially invalid data or studies with invalid methodologies is unavoidable. The exponential growth of preprints described over the years is in accordance with the overall growing amount of orthopaedic literature. However, almost two-thirds of all preprints have not undergone subsequent publication in a peer-reviewed indexed journal after a minimum of one year, yet are still available for download.

In this study, the rate of later peer-reviewed publication was dependent on the geographical origin and the level of evidence covered. Several reasons for this could be identified. First, the rate of subsequent successful publication is dependent on the geographical origin. Europe, North America, and Oceania demonstrated higher publication rates than Africa, South America, and Asia/Middle East. Conceptions about the role of preprints might differ among countries, authors, and healthcare systems, and most likely influenced these findings. Three out of four preprints originated from Asia/Middle East, which represented most of the investigated preprints and demonstrated a missing publication rate of 65%. This is a considerable proportion, and geographical origins should be accounted for when discussing the role of preprints. The geographical differences could also have occurred due to article processing fees requested by some journals, as preprint servers allow the dissemination of work free of charge. If no funding is available, publication in certain peer-reviewed journals may not be affordable for some researchers, and they may favour publishing their work on a preprint server instead. Furthermore, native English speakers might be more accurate in scientific writing and therefore more successful in publishing their research. Broader global discussion and education about the role of preprints and the funding of meaningful research could resolve some of the cultural and geographical variability.

Second, the rate of definitive publication differed significantly between the levels of evidence covered in clinical research. Preprints covering a higher level of evidence were more likely to be published subsequently. This is not surprising, as a trend toward publishing level I/II studies has been demonstrated in different orthopaedic subspecialties. Overall, most of the analyzed preprints (n = 557; 64.1%) covered lower levels of evidence (≤ III), including simple retrospective designs, technical notes, or case reports.

The high proportion of research papers with lower levels of evidence, and higher rates of missing later peer-reviewed publication, could be due to the temptation to disseminate research results primarily online without going through a peer review process and risking rejection of the paper, or the need to pay publication fees. Conversely, these preprints could also include work that had already been rejected by a previous peer review.

Additionally, the identified preprint servers and the anatomical region covered demonstrated significant differences in the definitive publication rate. The variability among the preprint servers might have occurred due to the variable inception times of the servers during the investigated periods. Research Square and MedRxiv were launched later than the other servers and demonstrated the highest number of missing definitive publications. Furthermore, cooperation between preprint servers and journals most likely contributed to the reported variability in definitive publication rates among journals. For example, Springer Nature is an investor in Research Square. These findings encourage transparent disclosure of ties between preprint servers and scientific journals.

Additionally, published preprints have been available online longer than unpublished ones, albeit only averaging two months longer. The time available online represents a potential bias regarding the definitive publication rate. Nevertheless, 85.4% of the subsequent peer-reviewed preprints were published within 12 months. Therefore, the defined period to allow definitive publication of the preprint within 12 months seems justified.

It is most likely that publishing in multiple preprint servers and the applied search strategy both contributed to the identified duplicates. The question of whether publishing in multiple preprint servers increases the chance of successful subsequent publication cannot be answered by our study and warrants further research. Moreover, it would be of interest if authors publish on preprint servers after submission to the journal. This was not answered by our study, as these data were not available for all the investigated journals.

The substantial variability of missing definitive publications of preprints raises serious concerns about the potential misuse of preprints for fast distribution by evading the rigorous, time-consuming, and sometimes costly peer review process. Even worse, potentially false results can be disseminated and persist online without peer review. Moreover, a published preprint does not allow a double-blinded peer review process, which is considered gold-standard for scientific literature. One-third of all identified journals described a double-blinded peer review process on their homepage; however, it remains unclear how this can be assured when preprints are online prior to publication. Special caution and education are needed in interpreting these unfiltered data.

The rapid growth of online publications and promotion of research via social media will most likely encourage this trend over the next few years. This will make it increasingly difficult for the orthopaedic clinician to distinguish meaningful research from potentially faulty preliminary results. To tackle these challenges, preprints should either be clearly identifiable and labelled, or even prohibited, as stated by renowned orthopaedic journals. At the very least, a transparent statement including the journal’s procedures regarding the use of preprints would be helpful for contributing authors and readers. However, defining the future role of preprints in orthopaedic surgery is beyond the scope of this systematic analysis. The systematic approach presented provides a baseline, informing the orthopaedic community about the increase in preprints and their geographical locations, topics, and levels of evidence. Conclusions on the merits and drawbacks of preprints cannot be made based on our results, and further analysis is warranted. However, the reported results underline the need for the orthopaedic community and journals to address the future roles of preprints.

Several limitations must be considered when interpreting our findings. First, the selection of the searched preprint servers and the applied search strategy will not have identified all manuscripts in orthopaedic surgery, as new preprint servers continue to be launched. Nevertheless, we identified five main preprint servers in the medical field and conducted systematic literature research, resulting in a representative analysis covering all anatomical regions and levels of evidence of the orthopaedic literature. Second, the search criteria for definitive publication most likely influenced our findings, as titles and authors could have changed between the preprint and the submission to a journal, resulting in an overestimation of missing definitive publication. However, in our opinion, it is reasonable to assume that the corresponding author and main keywords of the title did not change over time in the vast majority.

Preprints are being published in increasing numbers, including in T&O surgery. Depending on the geographical origin and level of evidence, almost two-thirds of preprints have not been published in a peer-reviewed indexed journal after one year. This raises major concerns regarding the spread and persistence of potential wrongful scientific work by bypassing peer review, and the orthopaedic community should discuss appropriate preventive measures.

Take home message

- Preprints are being published in increasing numbers in orthopaedic surgery.

- Almost two-thirds of preprints have not been published in a peer-reviewed indexed journal after one year.