Reproducible and transparent research practices in published neurology research.

BACKGROUND: The objective of this study was to evaluate the nature and extent of reproducible and transparent research practices in neurology publications.
METHODS: The NLM catalog was used to identify MEDLINE-indexed neurology journals. A PubMed search of these journals was conducted to retrieve publications over a 5-year period from 2014 to 2018. A random sample of publications was extracted. Two authors conducted data extraction in a blinded, duplicate fashion using a pilot-tested Google form. This form prompted data extractors to determine whether publications provided access to items such as study materials, raw data, analysis scripts, and protocols. In addition, we determined if the publication was included in a replication study or systematic review, was preregistered, had a conflict of interest declaration, specified funding sources, and was open access.
RESULTS: Our search identified 223,932 publications meeting the inclusion criteria, from which 400 were randomly sampled. Only 389 articles were accessible, yielding 271 publications with empirical data for analysis. Our results indicate that 9.4% provided access to materials, 9.2% provided access to raw data, 0.7% provided access to the analysis scripts, 0.7% linked the protocol, and 3.7% were preregistered. A third of sampled publications lacked funding or conflict of interest statements. No publications from our sample were included in replication studies, but a fifth were cited in a systematic review or meta-analysis.
CONCLUSIONS: Currently, published neurology research does not consistently provide information needed for reproducibility. The implications of poor research reporting can both affect patient care and increase research waste. Collaborative intervention by authors, peer reviewers, journals, and funding sources is needed to mitigate this problem.

Background

Scientific advancement is hampered by potential research flaws, such as the lack of replication; poor reporting; selective reporting bias; low statistical power; and inadequate access to materials, protocols, analysis scripts, and experimental data [1-3]. These factors may undermine the rigor and reproducibility of published research. Substantial evidence suggests that a large proportion of scientific evidence may be false, unreliable, or irreproducible [4-8]. Estimates of irreproducible research range from 50 to 90% in preclinical sciences [9] and substantiated in a recent survey of scientists. Prior survey studies reported that roughly 70% of scientists were unable to replicate another scientist’s experiment, and 90% agreed that scientific research is currently experiencing a “reproducibility crisis” [7].

Reproducibility is vital for scientific advancement as it aids in enhancing the credibility of novel scientific discoveries and mitigates erroneous findings. One review discussed potential pitfalls in fMRI reproducibility, such as scanner settings, consistency of cognitive tasks, and analysis methods [10]. Boekel et al. replicated five fMRI studies measuring a total of 17 structural brain-behavior correlations. After reanalysis, only one of the 17 was successfully replicated [11]. Thus, practices related to transparency and reproducibility can be improved within fMRI and other neurology research.

Adopting open science in neurology would help mitigate irreproducible research, such as studies on brain-behavior correlation. Open science practices—such as data sharing, open access articles, sharing protocols and methods, and study preregistration—promote transparency and reproducibility [12]. For example, preregistering a study helps guard against selective outcome reporting [13]. Selective outcome reporting occurs when discrepancies exist between outcome measures prespecified in trial registries or research protocols and the outcomes listed in the published report [14]. In neurology, an audit of randomized clinical trials published in neurology journals found 180 outcome inconsistencies across 180 trials, with most inconsistencies favoring changes in accordance with statistically significant results. Additionally, only 55% of neurology trials were prospectively registered [15], providing indications that neurology researchers are not adhering to transparency and reproducibility practices early in research planning. Reproducible research and open science practices are widely endorsed by a large proportion of authors. Despite this support, evidence suggests that authors infrequently implement them [16-18].

Given the recent attention to the reproducibility crisis in science, further investigation is warranted to ensure the existence of reproducible and transparent research in the field of neurology. Here, we examine key transparency- and reproducibility-related research practices in the published neurology literature. Our findings from this investigation may serve as a baseline to measure future progress regarding transparency and reproducibility-related practices.

Methods

This observational, cross-sectional study used the methodology proposed by Hardwicke et al [3], with modifications. We reported this study in accordance with the guidelines for meta-epidemiological methodology research [19] and, when pertinent, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [20]. Our study did not use any human subjects or patient data and, as such, was not required to be approved by an institutional review board prior to initiation. We have used The Open Science Framework to host our protocol, materials, training video, and study data in a publically available database (https://osf.io/n4yh5/). This study was part of a comprehensive investigation on reproducibility across multiple clinical specialties.

Journal and publication selection

On June 25, 2019, one investigator (D.T.) searched the National Library of Medicine (NLM) catalog for all journals using the subject terms tag “Neurology [ST].” The inclusion criteria required that all journals publish English, full-text manuscripts and be indexed in the MEDLINE database. The final list of included journals was created by extracting the electronic international standard serial number (ISSN) or the linking ISSN, if necessary. PubMed was searched with the list of journal ISSNs on June 25, 2019, to identify all publications. We then limited our publication sample to those between January 1, 2014, and December 31, 2018. Four hundred publications within the time period were randomly sampled for data extraction. The rest were available, but not needed (https://osf.io/wvkgc/).

To estimate the required sample size for our study, we used Open Epi 3.0 (openepi.com). We selected data availability as our primary outcome based on its importance for study [3]. Our estimated parameters included a population size of 223,932 publications; a hypothesized % frequency of 18.5% for the data availability factor in the population (which was based upon data obtained by Hardwicke et al.); a confidence limit of 5%; and a design factor of 1, which is used in random sampling. Based upon these considerations, a 95% confidence level would require a sample size of 232. From our previous studies [21, 22], we estimated that approximately 40% of studies would be excluded following screening. Thus, a random sample of 400 publications with a hypothesized attrition rate of 40% would yield a final, minimum sample of 240 for analysis. Previous investigations, upon which this study is based, have included random samples of 250 publications in the social sciences and 150 publications in the biomedical sciences. Thus, our sample size exceeds those used in previous investigations.

Extraction training

Prior to data extraction, two investigators (S.R. and J.P.) completed in-person training designed and led by another investigator (D.T.). The training sessions included reviewing the protocol, study design, data extraction form, and likely locations of necessary information within example publications. The two authors being trained received two sample publications to extract data from. This example data extraction was performed in the same duplicate and blinded fashion used for data acquisition for this study. The two investigators then met to reconcile any discrepancies. After the two sample publications were completed, the investigators extracted data and reconciled differences from the first 10 of the included 400 neurology publications. This process insured interrater reliability prior to analyzing the remaining 390 publications. A final reconciliation meeting was conducted, with a third investigator (D.T.) available for disputes but not needed.

Data extraction

After completing the training, the same two investigators extracted the data from the included list of randomly sampled publications between June 3, 2019, and June 10, 2019, using a pilot-tested Google form. This Google form was based on the one used by Hardwicke et al., but including modifications [3]. We specified the 5-year impact factor and that for the most recent year as opposed to the impact factor of a specific year. The available types of study designs were expanded to include case series, cohort studies, secondary analyses, chart reviews, and cross-sectional analyses. Last, we specified funding sources, such as hospital, private/industry, non-profit, university, or mixed, instead of restricting the criteria to public or private.

Assessment of reproducibility and transparency characteristics

This study used the methodology by Hardwicke et al. [3] for analyses of transparency and reproducibility of research, with modifications. Full publications were examined for funding disclosures, conflicts of interest, available materials, data, protocols, and analysis scripts. Publications were coded to fit two criteria: those with and those without empirical data. Publications without empirical data (e.g., editorials, reviews, news, simulations, or commentaries without reanalysis) were analyzed for conflict of interest statements, open access, and funding. Given that protocols, data sets, and reproducibility were not relevant, these were omitted. Case studies and case series were listed as empirical studies; however, questions pertaining to the availability of materials, data, protocol, and registration were excluded due to previous study recommendations [18]. Data extraction criteria for each study design are outlined in Table 1.

Table 1

Reproducibility-related characteristics. Variable numbers (N) are dependent upon study design. Full detailed protocol pertaining to our measured variables is available online (https://osf.io/x24n3/)

Indicators of reproducibility included in present study		Significance of measure variable for transparency and reproducibility
Publications
All (N = 400)	Publication accessibility (Is the publication open access to the general public or accessible through a paywall?)	The general public’s ability to access scientific research may increase transparency of results and improve the ability for others to critically assess studies, potentially resulting in more replication studies
Funding
Included studies (N = 389)	Funding statement (Does the publication state their funding sources?)	Explicitly providing source of funding may help mitigate bias and potential conflicts of interest
Conflict of interest
Included studies (N = 271)	Conflict of interest statement (Does the publication state whether or not the authors had a conflict of interest?)	Explicitly providing conflicts of interest may allow for full disclosure of factors that may promote bias in the study design or outcomes
Publication citations
Empirical studiesa (N = 271)	Citations by a systematic review/meta-analysis (Has the publication been cited by any type of data synthesis publication, and if so, was it explicitly excluded?)	Systematic reviews and meta-analyses evaluate and compare existing literature to assess for patterns, strengths, and weaknesses of studies regarding a particular field or topic
Analysis scripts
Empirical studiesb (N = 271)	Availability statement (Does the publication state whether or not the analysis scripts are available?)	Providing access to the analysis script helps improve credibility by providing the replicators the opportunity to analyze raw data with the same analysis procedure
	Method of availability (Ex: Are the analysis scripts available upon request or in a supplement?)
	Accessibility (Can you view, download, or otherwise access the analysis scripts?)
Materials
Empirical studiesc (N = 255)	Availability statement (Does the publication state whether or not the materials are available?)	Providing the materials list allows replicators to reproduce study using the same materials, promoting
	Method of availability (Ex: Are the materials available upon request or in a supplement?)
	Accessibility (Can you view, download, or otherwise access the materials?)
Pre-registration
Empirical studiesb (N = 271)	Availability statement (Does the publication state whether or not it was pre-registered?)	Pre-registering studies may help mitigate potential bias and increase the overall validity and reliability of a study
	Method of availability (Where was the publication pre-registered?)
	Accessibility (Can you view or otherwise access the registration?)
	Components (What components of the publication were pre-registered?)
Protocols
Empirical studiesb (N = 271)	Availability statement (Does the publication state whether or not a protocol is available?)	Providing replicators access to protocols allows for a more accurate replication of the study, promoting credibility
	Components (What components are available in the protocol?)
Raw data
Empirical studiesb (N = 271)	Availability statement (Does the publication state whether or not the raw data are available?)	Providing replicators with access to raw data can help reduce potential bias and increase validity and reliability
	Method of availability (Ex: Are the raw data available upon request or in a supplement?)
	Accessibility (Can you view, download, or otherwise access the raw data?)
	Components (Are all the necessary raw data to reproduce the study available?)
	Clarity (Are the raw data documented clearly?)

a“Empirical studies” are publications that include empirical data such as clinical trial, cohort, case series, case reports, case-control, secondary analysis, chart review, commentaries (with data analysis), laboratory, surveys, and cross-sectional designs

bEmpirical studies determined to be case reports or case series were excluded in regard to reproducibility related questions (materials, data, protocol, and registration were excluded) as recommended by Wallach et al.

cEmpirical studies determined to be either case reports, case series, commentaries with analysis, meta-analysis, or systematic review were excluded as they did not provide materials to fit the category

Publication citations included in research synthesis and replication

For both empirical and nonempirical studies, we measured the impact factor of each journal by searching for the publication title on the Web of Science (https://webofknowledge.com). For empirical studies, we used the Web of Science to determine whether our sample of studies was cited in either a meta-analysis, systematic review, or a replication study. The Web of Science provided access to studies that cited the queried publication and provided the title, abstract, and link to the full-text article. This permitted the evaluation of the inclusion of the queried article in data synthesis. Extraction was performed by both investigators in a duplicate, blinded fashion.

Assessment of open access

Important core components of publications necessary for reproducibility are only available within the full text of a manuscript. To determine the public’s access to each publication’s full text, we systematically searched the Open Access Button (https://openaccessbutton.org), Google, and PubMed. First, we searched the title and DOI using the Open Access Button to determine if the publication was available for public access. If this search returned no results or had an error, then we searched the publication title on Google or PubMed and reviewed the journal website to determine if the publication was available without a paywall.

Statistical analysis

Microsoft Excel was used to report statistics for each category of our analysis. In particular, we used Excel functions to calculate our study characteristics, results, and 95% confidence intervals.

Results

After searching the National Library of Medicine catalog, 490 neurology journals were eligible for analysis. After screening for inclusion criteria, 299 journals remained for analysis, yielding 223,932 publications. Of the 223,932 publications, we randomly sampled 400 (https://osf.io/qfy7u/). Eleven publications were inaccessible, which left 389 publications for analysis. Of the 389 eligible publications, 291 provided analyzable empirical data, and 118 articles were excluded because they did not contain characteristics measurable for reproducibility. Of the 291 publications eligible for analysis, an additional 20 case studies and case series were excluded, as they are irreproducible. Our final analysis was based on 271 publications with measurable reproducibility characteristics (Fig. 1 and Table 1).

Fig. 1

Flow diagram of included and excluded studies for the reproducibility analysis

Sample characteristics

Of the eligible publications, the median 5-year impact factor was 3.5 (interquartile range (IQR) 2.6–4.9), although 17 publications had inaccessible impact factors. The USA was the location of most of the primary authors (32.6%, 127/389) and the country of most publications (56.6%, 220/389). Of the 389 publications that were accessible, 32.1% (125/389) did not report a funding source, and 25.7% (100/389) reported funding from mixed sources (Table 2).

Table 2

Characteristics of included publications

Characteristics		Variables
		N (%)	95% CI
Funding, N = 389	University	12 (3.1)	1.4–4.8
	Hospital	2 (0.5)	0–1.2
	Public	79 (20.3)	16.4–24.3
	Private/industry	17 (4.4)	2.4–6.4
	Non-profit	15 (3.9)	2.0–5.7
	Mixed	100 (25.7)	21.4–30.0
	No statement listed	125 (32.1)	27.6–36.7
	No funding received	39 (10.0)	7.1–13.0
Type of study, N = 389	No empirical data	98 (25.2)	20.9–29.4
	Meta-analysis	15 (3.9)	2.0–5.7
	Commentary with analysis	1 (0.3)	0–0.8
	Cost-effectiveness	2 (0.5)	0–1.2
	Clinical trial	29 (7.5)	4.9–10.0
	Case study	13 (3.3)	1.6–5.1
	Case series	7 (1.8)	0.5–3.1
	Cohort	54 (13.9)	10.5–17.3
	Chart review	7 (1.8)	0.5–3.1
	Case control	19 (4.9)	2.8–7.0
	Survey	5 (1.3)	0.2–2.4
	Cross-sectional	43 (11.1)	8.0–14.1
	Secondary analysis	3 (0.8)	0–1.6
	Laboratory	92 (23.7)	19.5–17.8
	Multiple study types	1 (0.3)	0–0.8
5-year impact factor, N = 372	Median	3.5	–
	1st quartile	2.6	–
	3rd quartile	4.9	–
	Interquartile range	2.6–4.9	–

Of the randomly sampled 400 publications, 77.2% were behind a paywall (227/400), and only 57.1% were available to the public via the Open Access Button (168/400). Approximately half of analyzed publications stated that they did not have any conflicts of interest (55.5, 216/389), and 32.4% did not report whether or not conflicts of interest existed (126/389). Humans were the focus of 51.2% of the analyzed publications (199/389). Additional sample characteristics are viewable in Supplemental Tables 1, 2, and 3.

Reproducibility-related characteristics

Among the 271 publications with empirical data that were analyzed, a mere 3.7% provided preregistration statements or claimed to be preregistered (10/271). Of the 271 publications, just 0.7% provided access to the protocol (2/271). Only 9.4% provided access to the materials list (24/255), 9.2% provided access to the raw data (25/271), and just 2 articles provided the analysis script (0.7%, 2/2271). Not a single publication claimed to be a replication study. Additional characteristics are viewable in Supplemental Tables 1, 2, and 3.

Discussion

Our analysis demonstrates inadequate reproducibility practices within published neurology and neuroscience research. We found that few publications contained data or materials availability statements and even fewer contained a preregistration statement, made the protocol available, or included an analysis script. Our overall finding—that a majority of neurology publications lack the information necessary to be reproduced and transparent—is comparable to findings in the social and preclinical sciences [3, 5, 23–26]. Here, we present a discussion on prominent reproducibility and transparency indicators that were lacking in our study while presenting recommendations and practices to help improve neurology research.

First, data and materials availability is essential for reproducing research. Without source data, corroborating the results is nearly impossible. Without a detailed description of materials, conducting the experiment becomes a guessing game. Less than 10% of publications in our sample reported either a data or a materials availability statement. Efforts toward data sharing in neurological research originated with brain mapping and neuroimaging, but have spread to other areas within the specialty to improve reproducibility, transparency, and data aggregation [27]. Although data sharing poses challenges, steps have been taken in fMRI studies [28, 29]. fMRI data are complex and cumbersome to handle, but can be managed with software, such as Automatic Analysis [30], C-BRAIN [31], and NeuroImaging Analysis Kit [32]. Furthermore, these data can be hosted on online repositories, such as The National Institute of Mental Health Data Archive [33], Figshare [34], and other National Institutes of Health repositories [35]. Although researchers may take these steps voluntarily, journals—the final arbiters of research publications—can require such practices. Our study found that less than half of the sampled journals had a data availability policies, with approximately 20% of articles from these journals reporting source data [36]. Another study in PLOS ONE found that only 20% of nearly 50,000 publications included a data sharing statement and found that once a data sharing policy was enacted, open access to raw data increased [37]. Based on this evidence, journals and funders should consider implementing and enforcing data sharing policies that, at a minimum, require a statement detailing whether data are available and where data are located. For example, the journal Neurology has endorsed the International Committee of Medical Journal Editors policy of requiring a data sharing statement and encourages open access [38-40]. If other neurology journals follow suit, an environment of transparency and reproducibility may be established.

Second, preregistration practices were uncommon among neurology researchers. Preregistration prior to conducting an experiment safeguards against selective outcome reporting. This form of bias affects the quality of research in neurology. For example, when a randomized controlled trial (RCT) contains an outcome deemed “not significant” and is selectively removed from a trial, the validity of the RCT may be questioned. Previous studies have already established outcome reporting bias as an issue within neurology, noting that only 40% of analyzed RCTs were preregistered and, therefore, prespecified their analysis [15]. This same study found outcome reporting inconsistencies that often favored statistically significant results [15]. JAMA Neurology, The Lancet Neurology, and Neurology all require the preregistration of clinical trials prior to study commencement in accordance with the International Committee of Medical Journal Editors (ICJME) [41]. Only The Lancet Neurology mentions registration of other study designs, such as observational studies, and only “encourages the registration of all observational studies on a WHO-compliant registry” [42-44]. The ICJME notes that although non-trial study designs lack a researcher prespecified intervention, it is recommended to preregister all study types to discourage selective reporting and selective publication of results [41]. On ClinicalTrials.gov alone, almost 65,000 observational study designs have been preregistered, comprising 21% of all registered studies [45]. Encouraging the preregistration of clinical trials and observational studies, alike, will increase transparency, increase the evidence available for systematic reviews and meta-analyses, and improve reproducibility [46, 47].

Moving forward

We propose the following solutions to promote reproducible and transparent research practices in neurology. With regard to journals, we recommend requiring open data sharing upon submission, or, at least, a statement from the authors signifying why open data sharing does not apply to their study. There are many open data repositories available, including the Open Science Framework (https://osf.io/), opendatarepository.org, and others listed at re3data.org. Second, we recommend journals and funding providers to consider incentivizing reproducible research practices. For example, the Open Science Framework awards “badges” for open research practices, such as open data sharing, materials availability, and preregistration [48]. If one or more of these reproducible research practices do not apply to a particular study, a statement as to such should still qualify for the award. One Neuroscience journal, Journal of Neurochemistry, has already implemented open science badges with considerable success [49].

With regard to researchers, better awareness and education is necessary to encourage transparent and reproducible practices. Organizations, such as the Global Biological Standards Institute, have committed to improving the reproducibility of life sciences research through multiple methods, including training and educating researchers in effective trial design [50, 51]. The institute’s president has called for and implemented training programs aimed at teaching students, postdoctoral fellows, and principal investigators the importance of robust study design [50]. Additionally, we propose that medical schools and residency programs incorporate classes and didactic programs detailing proper experimental design with an emphasis on reproducible scientific practices. Research education should be a pillar of medical education, as physicians play an important role in guiding evidence-based healthcare. We anticipate that these recommendations, if implemented, will improve reproducibility within neurology and, as a result, the quality of research produced within this specialty.

Strengths and limitations

We feel that our methodology is robust and has many strengths, including blind and duplicate data extraction. Additionally, our protocol and data are available online to encourage reproducibility and transparency. However, we acknowledge a few limitations. First, we recognize that not all publications (clinical trials and protected patient data) are readily able to share their data and materials, although we feel a statement should still be reported, as justification was not always provided in each publication. Second, we did not contact the authors to obtain data, materials, or analysis scripts and only used published materials for extraction. Had we contacted the authors, then source data, materials, and protocols may have been available, but the goal of this publication was to examine readily available, published indicators of reproducibility. Finally, the scope of this study is limited to PubMed-indexed journals in neurology, and the results of this cross-sectional study may not be generalizable beyond this reach.

Conclusions

In summary, improvement is needed to incorporate reproducibility factors in neurology publications. Such necessary improvement is attainable. Authors, journals, and peer-reviewers all have a part to play in developing an improved community of patient-centered neurology researchers. Reproducibility is paramount in evidence-based medicine to corroborate findings and ensure physicians have the highest quality evidence upon which to base patient care.

Additional file 1: Table S1. Additional Characteristics of Reproducibility in Neurology Studies.

Additional file 2: Table S2. Additional Characteristics of Reproducibility in Neurology Studies.

Additional file 3: Table S3. Additional Characteristics of Reproducibility in Neurology Studies.