Developmental cognitive neuroscience initiatives for advancements in methodological approaches: Registered Reports and Next-Generation Tools.

Developmental Cognitive Neuroscience is excited to introduce two new article types at the journal, effective May 15, 2020: Next-Gen Tools and Registered Reports. Next-Gen Tools provide an outlet for methodological advancements and best practices in developmental cognitive neuroscience. Registered Reports provide the opportunity for peer review of study protocol, including hypotheses and methods, prior to data collection or analysis. As many articles have been written about this important tool for open science, we focus on topics we believe are of particular relevance to developmental cognitive neuroscientists. One emphasis is on what are called registered reports of secondary data analysis, that is, proposals to analyze data that have already been collected. Given the wide-spread use of longitudinal methods and pre-existing data in the field, we believe this type of registered report will be common, perhaps the majority of this submission type at DCN. We also present potential challenges for registered reports in developmental cognitive neuroscience and suggest possible solutions, in an effort to facilitate the adoption of this approach. Both article types will also appear as ongoing virtual special sections.

Why Registered Reports?

The reproducibility crisis in psychology (Open Science Collaboration, 2015; Pashler and Wagenmakers, 2012) and cognitive neuroscience (Gilmore et al., 2017; Huber et al., 2018) has led many to the recognition that practices and incentives related to conduct and publication of research need to evolve in ways that renew confidence in our findings (see also Pfeifer et al., 2019). The Registered Reports (RRs) process is one tool that was developed to address this pressing need (Chambers, 2013; Chambers et al., 2015). This format is now offered in over 200 journals across a wide variety of disciplines. The goal of RRs is to identify high quality protocols that adhere to best practices and then further refine them through a process of rigorous peer review prior to conducting the study. RRs are designed to minimize problems such as low power, analytical flexibility, p-hacking, hypothesizing after the results are known (HARKing), and publication bias (Munafò et al., 2017; Simmons et al., 2011). The sections below provide a brief overview of the process and contrast types of RRs; to more fully explore the details and workflow of RRs, see https://cos.io/rr/.

Overview and types of registered reports

In Stage 1, manuscripts consisting of an introduction, methods, and any pilot data are submitted. The editors and reviewers assess the manuscript and suggest changes to the proposed protocol, if needed. If the original or a revised version of the manuscript meet reviewer standards, an inprinciple acceptance is offered. This inprinciple acceptance provides reassurance that the study will be published regardless of the statistical result and conclusion, as long as the authors follow exactly the agreed upon protocol, which is placed into a repository such as the Open Science Framework. Therein lies one key weapon against questionable research practices and other forms of bias that are typically incentivized when publication hinges upon findings being deemed novel or groundbreaking: publication decisions are independent of results. In Stage 2, authors submit a revised manuscript that includes the original introduction and methods, plus the final results from the registered protocol, and discussion. Exploratory analyses (including application of novel techniques) may still be conducted and serendipitous findings may still be reported, but it becomes easier to clearly distinguish between a priori hypotheses and post hoc exploration, because each must be reported as such with reference to the approved protocol. Additional relevant findings that emerge in the literature after in principle acceptance can be integrated into the discussion.

There are several types of RRs. Primary RRs provide peer review in advance of study inception, and Stage 1 RRs should be submitted for review (and inprinciple acceptance) prior to the collection of any data. This format of RR has been adopted in psychological journals, and also in an increasing number of cognitive neuroscience journals. However, two other types of RRs are worth covering in greater detail - RRs of secondary data analysis, and registered replication reports (RRRs).

Secondary RRs introduce the process of peer review after data has already been collected but prior to data analysis and are an essential format given the importance and prevalence of existing datasets in developmental cognitive neuroscience. In the case of RRs on secondary data analysis, Stage 1 reports should be submitted prior to data analysis. Reviewers will then comment on the appropriateness of the analytic strategy for the research question, the appropriateness of the data for both the statistical methods and research question, and the inference criteria in the context of potential prior knowledge. This approach has been adopted successfully in other journals, including Cortex and The Journal of Research in Personality (citation forthcoming). Moreover, greater attention has been paid to improving research using pre-existing data more broadly, including the development of recommendations for robust and transparent analyses (Weston et al., 2019) and tools for reporting researchers’ prior knowledge (see Journal of Gerontology: Series B special issue on preregistration). As described in DCN’s Guide for Authors, in the case of secondary RRs, care should be taken to (1) specify any prior knowledge of the dataset(s) used, including but not limited to prior work published by the research team, unpublished analyses by the research team, and work published or disseminated by others, and (2) design analyses to avoid overfitting and bias, for example, through the use of cross-validation or specification curves.

A third type of RRs, Registered Replication Reports (RRRs), can rely on either new data collection or secondary data analysis. However, proposing to use secondary data analysis for this purpose should typically happen only under the conditions that i) new data collection would be unnecessarily onerous (e.g., diverse samples including underrepresented groups or longitudinal data analysis) and ii) the original report did not include the potential source of replication data. For example, data collected in a later wave of a longitudinal cohort study, such as the Adolescent Brain and Cognitive Development (ABCD) Study, cannot be used to replicate findings from an earlier wave. This includes conceptual replications, such as when using different instruments (e.g., different questionnaires or paradigms) to measure the same underlying construct.

Registered reports and big, open data

The large-scale, longitudinal Adolescent Brain and Cognitive Development (ABCD) Study is an example of the kind of dataset which is well-suited to secondary RRs. Because data collection is already underway and baseline descriptives across all domains of assessment have been released, key information is already and will be increasingly widely known about the sample, and therefore analysis of most ABCD data will always be considered secondary data analysis. Exceptions may be for measures that do not appear in previous waves, but that is expected to be a minority of the manuscripts.

In 2018, Cortex issued a call for registered reports timed to coincide with a planned ABCD data release. Specifically, a baseline release of 4524 9-10 year-olds provided a sample for hypothesis generation, descriptive modeling, and pipeline development. Stage 1 RRs were submitted prior to the data release in 2019, from the remaining 7354 participants. This provided an ideal way to confirm that analysis decisions were independent from any knowledge of the data. Of course, this situation was unique in the lifecycle of the ABCD study, even though there are scheduled dates for subsequent waves of data to be released from ABCD, and there are many other large-scale developmental cognitive neuroscience studies being conducted around the globe (such as the IMAGEN study [Schumann et al., 2010] or the “Lifebrain” study [Walhovd et al., 2018]), for many of which some or all data are already accessible.

The degree to which the data are openly available and/or have already been analyzed produces a potential dilemma in maximizing the value of RRs built upon secondary data analysis, which we expect many developmental cognitive neuroscience RRs to be. That is, most secondary data analyses will be designed with some knowledge of the data, which decreases the utility of hypothesis testing to guide inferences from statistical analysis (see Pfeifer et al., 2019). For example, imagine one wants to characterize the association between pubertal development and white matter trajectories. Initial descriptives of how white matter volume metrics vary across pubertal stage or by pubertal timing may soon become available. As subsequent waves of imaging data are acquired and released, these initial associations will be known, and only incremental new information will be available. This reveals a few imperatives for researchers and reviewers engaged in the RR process. First, manuscripts need to be abundantly clear in the Stage 1 submission about what data authors have already accessed and what analyses they may have already conducted. Second, authors need to be especially diligent when searching the published literature to ensure they are aware of any publications using their dataset(s) of choice that address their research question(s), as lack of awareness about any already published findings could create challenges for reviewers — ignorance should not be taken as evidence for independence. One tool for the ABCD study that may support keeping abreast of what analyses have already been run is the Data Exploration and Analysis Portal https://github.com/ABCD-STUDY/DEAP, where hypotheses can be registered and viewed. However, the DEAP also provides advanced, automated statistical analysis of many ABCD variables, which would amplify problems associated with undisclosed analytical flexibility.

Given these considerations for secondary data analysis, we therefore do not plan to issue a call for RRs at Developmental Cognitive Neuroscience timed with each and every neuroimaging data release for ABCD. However, there are some key occasions where a data release might produce qualitatively unique analysis opportunities. Returning to our example of puberty and white matter development, researchers may also be interested in how pubertal tempo is associated with white matter development, and tempo cannot be assessed with fewer than three data points. In fact, to avoid overfitting trajectories, linear slopes require a minimum of three data points and nonlinear forms require a minimum of four data points. As such, Developmental Cognitive Neuroscience may release calls for special issues associated with the third release of neuroimaging data, and either the fourth or fifth release; and for this and other big data sets, a virtual special collection may be implemented.

Other considerations for registered reports

In the preceding section, we described some challenges with big, open datasets and secondary RRs. Next we outline a few additional RRs scenarios that may generate interest in this as a publication format, as well as provide some general guidelines for further consideration. We note that while researchers in developmental cognitive neuroscience learn how to best engage in the RR process, pre-submission consultation may be useful.

One issue is the degree to which proposed analyses in a secondary RR or RRR constitute an independent test. For example, researchers may publish on the relationship between cortical thickness and marijuana use in the ABCD dataset. Subsequently, these same researchers or different researchers may want to test the relationship between cortical thickness and alcohol use in the same dataset. In some ways, these two sets of analyses may test the same research question — to what degree is cortical thickness associated with substance use — and fall under the domain of “sensitivity analyses” whereby a researcher tests the boundaries of when an effect occurs (Condon et al., 2017). Testing the cortical thickness-marijuana use relationship in a new dataset would be a replication; and examining cortical thickness-alcohol use in a new dataset would best be described as “generalizability.” The size of ABCD and other big datasets may allow the sample to be split to create independent datasets to ask such questions. In general, considering the independence of tests carefully is a must in RRs of secondary data analysis as well as RRRs.

Another set of issues illustrates circumstances under which coordination among research groups may be beneficial. For example, multiple teams may have the same general research question, but different opinions about which analytical approach(es) would be most appropriate. In addition, research groups who ask similar questions may want to combine datasets, allowing a model to be specified in one dataset, tested in another, expanded to look at moderators that aren’t well sampled in a given dataset, and so on.

Why Next-Generation Tools?

Next-Generation Tools (NGT) reflect the need for our methods to continually improve and the recognition that persisting with outdated analytical techniques can actually do harm to the quality of our inferences. For example, when it was discovered that much more stringent correction for motion was required in resting-state functional connectivity MRI (Power et al., 2012; Satterthwaite et al., 2012), this led to an overhaul of protocols and revisitation of existing findings. Some of the insights using the previous methods needed to be corrected (Fair et al., 2012; Satterthwaite et al., 2013). On occasion, some methodological advancements in cognitive neuroscience are not fully adopted by developmental cognitive neuroscientists, which may be a cause for concern. Some such “advancements” are relatively basic insights, such as the presence of software bugs (Pfeifer et al., 2019) or use of statistically incorrect modules in older versions of imaging analysis software packages (Madhyastha et al., 2018; Telzer et al., 2018). Researchers may wish to publicize such best practices via the NGT article type, perhaps including demonstrations of the consequences of using (or not using) them, on real or simulated data. For assistance in determining whether a practice, tool, or insight is suitable for the NGT article type, pre-submission consultation may again be useful.

In addition, developmental cognitive neuroscientists may be at the forefront of developing other methodological tools that are of particular relevance to our field. For example, a recent publication in the special issue of Developmental Cognitive Neuroscience described a new software program, neuropointillist (Madhyastha et al., 2018). Neuropointillist was designed to meet the needs particularly of developmental cognitive neuroscientists who wished to run more sophisticated whole-brain analyses than those afforded by the major software packages such as SPM, AFNI, and FSL — particularly those well-suited for longitudinal questions. Describing and demonstrating the utility of new software packages or tools, and providing access to code or apps to use these methods, is another ideal fit for NGT articles. This may provide a venue for early career researchers to disseminate methods that they expend much time and effort in developing.

Conclusion

This is a very exciting time for developmental cognitive neuroscience. The level of investment in big, open, and/or deep datasets — and the rapid pace of methodological advancement to deal with these potential goldmines — hold the promise of transforming our field. But to take full advantage of these opportunities, we cannot continue to use many of our old scientific strategies. We need new publication formats and new analytical tools, to align our priorities and ensure they are well communicated. The adoption of these two new article types by Developmental Cognitive Neuroscience represents our strong desire to promote these forward-looking solutions. We also recognize this is an evolving process — as a field, we are co-creating how to do our best, most rigorous science. It is our hope that initiatives like these inspire others to contribute and, in so doing, launch a new decade of impactful scientific discovery about brain development.