The ENIGMA Stroke Recovery Working Group: Big data neuroimaging to study brain-behavior relationships after stroke.

The goal of the Enhancing Neuroimaging Genetics through Meta-Analysis (ENIGMA) Stroke Recovery working group is to understand brain and behavior relationships using well-powered meta- and mega-analytic approaches. ENIGMA Stroke Recovery has data from over 2,100 stroke patients collected across 39 research studies and 10 countries around the world, comprising the largest multisite retrospective stroke data collaboration to date. This article outlines the efforts taken by the ENIGMA Stroke Recovery working group to develop neuroinformatics protocols and methods to manage multisite stroke brain magnetic resonance imaging, behavioral and demographics data. Specifically, the processes for scalable data intake and preprocessing, multisite data harmonization, and large-scale stroke lesion analysis are described, and challenges unique to this type of big data collaboration in stroke research are discussed. Finally, future directions and limitations, as well as recommendations for improved data harmonization through prospective data collection and data management, are provided.

INTRODUCTION

Stroke results in neural damage to the brain and subsequent physical, cognitive, and emotional deficits (Dimyan & Cohen, 2011; Meyer et al., 2015; Ward, 2017). Globally, there are over 13.7 million new strokes each year, and stroke is a leading cause of adult long‐term disability (Benjamin et al., 2018; Johnson et al., 2019; Lindsay et al., 2019). Since stroke is a vascular disease, there is vast heterogeneity in both brain and behavioral changes across patients, posing a significant challenge to the development of effective stroke neurorehabilitation strategies (Bernhardt et al., 2016). The recovery process also contributes to additional interindividual variability in the time course and extent of the resolution of neural damage and behavioral deficits.

Given the heterogeneity in both injury and recovery, large samples of diverse patients are needed to accurately study processes related to and supporting stroke recovery. However, a recent systematic review suggested that most studies seeking to determine the best predictors of stroke recovery are underpowered (Kim & Winstein, 2017). This is in part because acquiring large datasets of this nature is logistically and financially prohibitive for a single stroke research site. Recognizing these challenges, the Stroke Recovery and Rehabilitation Roundtable (SRRR) task force developed consensus statements on how to move stroke recovery research forward (Bernhardt et al., 2016; Bernhardt et al., 2019). Two key priority areas include: (a) a better understanding of the neurobiology of spontaneous and treatment‐dependent recovery in humans; and (b) characterizing different stroke recovery phenotypes (Bernhardt et al., 2017), with a long‐term goal of identifying potential biomarkers that predict stroke recovery (Boyd et al., 2017). Here, we describe the steps that the Enhancing Neuroimaging Genetics through Meta‐Analysis (ENIGMA) Stroke Recovery working group has taken to address these priorities using a unique, big data approach with an emphasis on reliable and reproducible methods.

The ENIGMA worldwide consortium consists of over 1,400 researchers from 45 countries who have come together to form 26 different disease working groups as well as 24 additional working groups to study healthy human variation and develop novel, robust methods for neuroimaging and genetic meta‐analyses (Thompson, 2019; Thompson et al., 2019; Thompson et al., 2020). Formed in 2009, ENIGMA has resulted in significant, large‐scale contributions on the neural correlates of major depression (Schmaal et al., 2016), schizophrenia (Van Erp et al., 2018), bipolar disorder (Hibar et al., 2016), obsessive–compulsive disorder (Boedhoe et al., 2016), and epilepsy (Whelan et al., 2018). ENIGMA methods, discussed in this article, have also resulted in some of the largest studies of genetic correlates of specific brain characteristics, such as genetic variations determining subcortical, hippocampal, and intracranial brain volumes (Grasby et al., 2020; Hibar et al., 2015; Stein et al., 2012). In order to analyze multisite data in these large studies, ENIGMA typically uses a meta‐analytic approach in which analysis methods, which are tested for reliability across diverse research sites, are openly shared. Participating research sites can analyze their brain magnetic resonance imaging (MRI) and genetic data locally and share either their site results (meta‐analysis) or individual participant results (mega‐analysis) with the rest of the working group (Zugman et al., 2020). In this way, many research sites can be brought together to meta‐ and mega‐analyze retrospective MRI and genetic data collected across different research sites around the world. The developed methods perform consistently despite high variability in MRI scanner manufacturers, scanner strengths, and pulse sequences, and the same methods can be applied across different disease working groups, allowing for cross‐disorder comparisons despite using different acquisition protocols (Boedhoe et al., 2019).

The ENIGMA Stroke Recovery working group is one of the 26 disease working groups within the ENIGMA consortium. Initially developed by a small group of researchers in 2015, ENIGMA Stroke Recovery has grown to over 100 members and has amassed a dataset of 2,137 MRI scans and counting from 39 research studies across 10 countries (see Table 1 for a summary of the number of scans by institution and geographical region at the time of publication). A primary goal of ENIGMA Stroke Recovery is to provide a reliable infrastructure for the collection and analysis of large, diverse datasets of poststroke brain MRI and behavioral data across research laboratories worldwide. The focus of the ENIGMA Stroke Recovery working group was initially on understanding the neural correlations of poststroke sensorimotor performance, which is affected in up to 80% of individuals after stroke (Rathore, Hinn, Cooper, Tyroler, & Rosamond, 2002). However, this focus has diversified as new members have joined. ENIGMA Stroke Recovery currently includes measures of sensorimotor, cognitive, emotional, and psychosocial behavior. Both ischemic and hemorrhagic stroke are included in the database, along with demographic, clinical and bedside diagnostic information. ENIGMA Stroke Recovery uses pipelines developed by several of the ENIGMA methods working groups (for a recent review, see Thompson et al., 2020, as well as other widely used software packages (Fischl et al., 2002; Fischl & Dale, 2000; Smith et al., 2004; Smith et al., 2007). Given the diversity and the unique challenges associated with stroke‐specific brain imaging data, discussed further in the article, the ENIGMA Stroke Recovery working group also requests raw MRI data when available. The raw data are processed by ENIGMA Stroke Recovery research staffs who perform additional quality control and lesion segmentation steps. Reliable and robust systems for data collection and maintenance are thus critical for the success of this collaborative approach.

TABLE 1

Number of T1‐weighted MRI scans by geographical region/institute

Country	City	Institute	Number of scans
Australia	Melbourne	The Florey Institute of Neuroscience and Mental Health	317
Brazil	Sao Paolo	University of Sao Paolo	28
Brazil	Sao Paolo	Albert Einstein Israeli Hospital	15
Canada	Toronto	University of Toronto/Sunnybrook Research Institute	29
China	Tianjin	Tianjin Medical University General Hospital	65
Germany	Griefswald	University of Griefswald	68
Germany	Tübingen	University of Tübingen	175
Italy	Rome	IRCCS Santa Lucia Foundation	192
New Zealand	Auckland	University of Auckland	104
Norway	Oslo	University of Oslo	265
UK	London	University College London	50
USA	Atlanta	Emory University	110
USA	Charleston	Medical University of South Carolina	174
USA	College Park	University of Maryland	128
USA	Irvine	University of California, Irvine	191
USA	Los Angeles	University of Southern California	189
USA	Philadelphia	University of the Sciences	37
Total scans			2,137

Note: The total number of T1‐weighted MRI scans (N = 2,137) includes data from both individuals with stroke (n = 1,918, or 89.8% of the total dataset) and healthy individuals (n = 219, or 10.2% of the total dataset). A subset of the scans also includes repeated MRIs from the same individual (e.g., longitudinal data; n = 672 scans, or 31.4% of the total dataset).

The ENIGMA Stroke Recovery database can be used for many different purposes. First, the data can be used to test whether brain–behavior relationships identified in smaller, more homogeneous samples also exist in larger, more diverse samples. A specific example might be examining whether individuals with left hemisphere stroke show better or worse outcomes than those with right hemisphere stroke, as reports from the literature on this topic are inconsistent (Beuscher et al., 2017; Macciocchi, Diamond, Alves, & Mertz, 1998; Ween, Alexander, D'Esposito, & Roberts, 1996; Wu et al., 2015). Second, these data can be used to identify characteristics, such as specific lesion locations that affect stroke outcomes of interest. For example, the data could be used to identify whether depression is more common following a stroke that impacts the left frontal cortex, or whether injury to specific cortical regions, such as the right dorsal premotor region, is associated with differences in functional outcomes. The data could also be used in a voxel‐wise analysis to identify voxels (i.e., three‐dimensional pixels) in the poststroke brain that are related to a specific behavior, such as spasticity, mild cognitive impairment or poststroke fatigue. This voxel‐wise whole‐brain approach provides a granular way to examine entire vascular territories, not limited by specific anatomical boundaries. A third use of the data is in generating new data‐driven hypotheses using machine‐learning methods. The large size of the dataset makes it uniquely powered for machine‐learning methods, which require big datasets to train and test computer algorithms to identify patterns. In particular, this dataset is well suited for unsupervised machine‐learning techniques, which can be used to identify clusters, or subgroups, of people who show similar patterns of recovery, and then examine what brain traits are common within those specific subgroups. Notably, as all of the data provided in ENIGMA Stroke Recovery is collected for specific research studies, the secondary use of these data in ENIGMA Stroke Recovery reduces research waste and improves the efficiency and speed with which we can test the reproducibility of existing research findings, identify novel brain–behavior relationships, and generate data‐driven hypotheses.

In this article, we provide a comprehensive description of the ENIGMA Stroke Recovery working group's approach to large‐scale analyses of multisite retrospective poststroke brain MRI and behavioral data for the broad neuroscience community. Specifically, this article will discuss methods and protocols developed for: (a) data intake, (b) data harmonization, and (c) lesion analysis. In each section, we also highlight future directions and recommendations to facilitate collaborative data sharing (see also Box 1).

Add language in ethics protocols and informed consent forms for data sharing (see Appendix 2).

Use standardized protocols for MRI data collection, following either the Human Connectome Project's Lifespan Aging protocol (https://www.humanconnectome.org/study/hcp‐lifespan‐aging/project‐protocol/imaging‐protocols‐hcp‐aging) or the Alzheimer's Disease Neuroimaging Initiative protocol (http://adni.loni.usc.edu/methods/mri‐tool/mri‐analysis/) where possible.

Collect Stroke Recovery and Rehabilitation Roundtable (SRRR) task force consensus‐based demographic variables and behavioral measures as prescribed (e.g., do not modify the measure to exclude certain items or include extra items). See Kwakkel et al. (2017) for a complete list of sensorimotor measures and McDonald et al. (2019) for recommendations for cognitive measures (McDonald et al., 2019).

Use the Brain Imaging Data Structure (BIDS) format to name, describe, organize and store all data (https://bids.neuroimaging.io/ ; Gorgolewski et al., 2016).

Share full research protocols on an open platform like Open Science Framework to improve the replicability of experiments (https://osf.io/).

Consider openly sharing data to increase the reach and impact of any collected data (Nichols et al., 2017).

DATA INTAKE

Data components

Key components for the initial ENIGMA Stroke Recovery analyses rely on a T1‐weighted (T1w) anatomical brain MR image and at least one poststroke behavioral outcome measure. These two elements comprise the minimum data required to join ENIGMA Stroke Recovery. Demographic data are also strongly requested, although not required. A summary of all data components is included in Table 2. Each component is described in more depth below.

TABLE 2

Data elements collected by the ENIGMA Stroke Recovery working group

	MRI	Behavior	Demographics
Required	T1‐weighted structural MRI Scanner strength, brand, and model Alternative if unable to share raw T1 MRIs: A spreadsheet with FreeSurfer cortical and subcortical measurements, quality control 2D image slices, and lesion masks (registered to a standardized template)	At least 1 behavioral outcome measure Most common measures at time of publication: Fugl‐Meyer Assessment (72%) NIH Stroke Scale (19%) Motor Activity Log (16%) Modified Ashworth (12%) Action Research Arm Test (11%) Wolf Motor Function Test (9%)	None required Most common demographics at time of publication: Age Sex Time since stroke/last known well (in days) Lesioned hemisphere
Recommended	FLAIR Diffusion MRI Resting‐state fMRI Lesion masks Longitudinal scans EEG	We suggest collecting measures recommended by the Stroke Recovery and Rehabilitation Roundtable (Bernhardt et al., 2017) Current recommendations for sensorimotor outcomes can be found in Kwakkel et al. (2017) Current recommendations for cognitive outcomes can be found in McDonald et al. (2019)	First stroke or multiple strokes Race/ethnicity Hand dominance prior to stroke Therapy received (hours per week) Risk factors for cardiovascular disease (e.g., hypertension, obesity, diabetes smoking) Dementia status Comorbidities

Note: The data elements are divided into three main components: MRI, behavioral measures, and demographic data, and further separated into required versus recommended elements.

Magnetic resonance imaging

The high‐resolution T1w MRI is used to quantify anatomical variation in regions across the brain, which is then related to the behavioral measures. We use a T1w MRI instead of other MR sequences, which might be more sensitive to the stroke lesion acutely, because high‐resolution T1w MRIs are routinely collected during research studies and can be used with FreeSurfer, a software used by all ENIGMA working groups for structural MRI analysis. FreeSurfer performs automated cortical and subcortical brain segmentation (Fischl et al., 2002; Fischl & Dale, 2000) and has reliable performance across different scanner characteristics and T1w acquisition parameters (Han et al., 2006). However, we acknowledge that the T1w MRI is not the best choice for identifying the lesion volume in acute and subacute patients. Therefore, we also collect additional MRI data that can be received and combined for analysis, such as diffusion MRI, T2‐weighted fluid‐attenuated inversion recovery (FLAIR), as well as other modalities of interest, such as resting‐state functional MRI (fMRI).

If raw MRI data cannot be shared, research sites can choose to analyze the data locally using processing scripts found on the ENIGMA website (http://enigma.ini.usc.edu/protocols/); results can then be sent for subsequent analyses (outputs include a spreadsheet with measures of cortical thickness, cortical surface area, and subcortical volumes following FreeSurfer segmentation). However, the ENIGMA Stroke Recovery group has the complicating issue of lesions within the data (see Section 4). Therefore, more stringent quality control, as well as delineation of stroke lesion boundaries, is required for more fine‐grained analyses of the lesion's impact on subsequent brain structure and behavior. For this reason, sites not sending raw data are requested to send the FreeSurfer segmentation measures, along with quality control images generated by our in‐house code (see Section 2.2 for more detail), and lesion masks normalized to a standard template brain when possible.

Behavioral data

ENIGMA Stroke Recovery collects many different types of behavioral data. As mentioned previously, the focus of the initial research question is on the neural correlates of poststroke sensorimotor behavior. To this end, over 80% of the dataset includes measures of sensorimotor performance. However, the growth and diversification of the ENIGMA Stroke Recovery group has led to the collection of additional types of behavioral data, including measures of cognition, mood, dysphagia, and psychosocial well‐being (see Section 3 for more information).

Demographic data

Demographic information on the participants is also collected, such as age, sex, time from last known well to imaging, type of stroke, number of previous strokes, stroke location, risk factors, and comorbidities (see Section 3 for more information). There are no exclusionary demographic criteria for entry into the database. There is growing interest in characterizing rehabilitation history, although this is an extremely complex variable to harmonize across research sites and encode. In general, the reported demographic variables can be included as covariates or inclusion/exclusion criteria for specific analyses.

Data intake workflow

When a research site joins ENIGMA Stroke Recovery, the site securely transfers de‐identified MRI data as well as a comma‐separated values (CSV) spreadsheet with all demographic and behavioral outcomes. If the raw MRI data are not available or cannot be shared, the site can run analyses scripts locally (https://github.com/npnl/ENIGMA-Wrapper-Scripts) and send FreeSurfer results in a CSV file along with the behavioral/demographic CSV file. Transfer can be accomplished via a secure file transfer protocol to an ENIGMA Stroke Recovery dedicated Linux server, or by the research site's preferred secure transfer method (e.g., using Box). The complete process, from data intake to data analysis, is displayed in Figure 1 and described in brief below. The scripts and code developed for these processes are freely and publicly available on Github (https://www.github.com/npnl).

FIGURE 1

ENIGMA Stroke Recovery workflow. Workflow for ENIGMA Stroke Recovery from data intake to data analysis

First, the received data are manually inspected to ensure: (a) good MRI data quality, and (b) appropriate values for the behavioral data. For MRI data quality, we visually inspect the images to ensure there are not large motion artifacts or other visible sources of noise in the data. Data that are of poor quality are excluded from the database. More recently, to standardize this process, we have begun to use the University of Southern California's Laboratory of Neuroimaging (LONI) Quality Control system, which is a freely available, semiautomated, web‐based system for quantitatively evaluating MRI image quality (Kim et al., 2019). We ensure that the behavioral data are appropriately coded according to our ENIGMA Stroke Recovery database conventions. For instance, we convert and record time since stroke in units of days and denote the lesioned hemisphere by an integer value where left = 1, right = 2, both = 3, and other = 4. The behavioral data are also inspected for values within the normal range for each measure. For the initial analysis of sensorimotor outcomes, a “primary” sensorimotor outcome that most closely aligns with the existing ENIGMA dataset is selected (see Section 3 for details). Altogether, these steps ensure that the incoming data are consistent with the existing database for subsequent analysis.

Next, we write a site‐specific script to reformat the data to conform to the Brain Imaging Data Structure (BIDS; https://bids.neuroimaging.io/) (Gorgolewski et al., 2016). BIDS provides a standardized way to organize and describe neuroimaging and behavioral data. If the data are not consistently named and formatted within each site, we manually reformat the data to conform to the BIDS standard. Having all of the data in BIDS format allows us to quickly and easily analyze ENIGMA Stroke Recovery data using a variety of software tools, many of which are written with the assumption that the data are in BIDS format (Gorgolewski et al., 2017). It also allows all of the code and scripts created for ENIGMA Stroke Recovery analyses to be easily used by others who use BIDS format, thereby increasing the transparency, reproducibility, and impact of this work.

Third, we run FreeSurfer, a brain imaging software package developed to analyze MRI scans, which segments the brain into anatomical regions for morphometric analysis (Fischl et al., 2002). We run two versions of FreeSurfer (version 5.3 and version 6.0) on the MRI data using scripts that automate the processing of all participants in parallel on the LONI computing cluster. FreeSurfer version 5.3 is currently the primary analysis version, which is used to maintain consistency with recent ENIGMA working group analyses. We also analyze all raw data using FreeSurfer version 6.0 because this more recent version has been shown to have improved performance and more accurate brain segmentations compared to FreeSurfer version 5.3. In addition, FreeSurfer version 6.0 also provides improved hippocampal segmentations and analysis of hippocampal subfields in stroke (Khlif et al., 2018), which is an area of research interest given recent findings from our ENIGMA Stroke Recovery working group showing a relationship between poststroke sensorimotor behavior and hippocampal volumes (Zavaliangos‐Petropulu et al., 2019).

We then use the ENIGMA scripts mentioned previously to extract standard measures of subcortical volume (from eight subcortical regions of interest, bilaterally), and cortical thickness and cortical surface area (from 34 cortical regions of interest, bilaterally; https://github.com/npnl/ENIGMA-Wrapper-Scripts). All regions are from the standard Desikan‐Killany atlas implemented in FreeSurfer (Desikan et al., 2006). These measures provide insight into brain morphometry for both subcortical and cortical regions. ENIGMA scripts are used to generate images of the overlap between each segmented region of interest and the underlying brain and put these into a webpage that can be visually inspected for segmentation accuracy. This is done for each brain. Although FreeSurfer has shown acceptable poststroke brain segmentation performance (Li et al., 2015), it may occasionally fail to create accurate segmentations on lower quality data (e.g., MRIs with lower spatial resolution or greater noise from movement) or on data with lesions that disrupt its surface‐based algorithms. In our ENIGMA Stroke Recovery dataset, FreeSurfer fails to produce accurate segmentations in 10–20% of the data. To address this, trained researchers manually inspect each region of interest for each brain MRI. Any regions that are not properly segmented by FreeSurfer are marked as “failed” and excluded from the analyses (see Appendix 1 for additional information about the ENIGMA Stroke Recovery FreeSurfer quality control protocol).

Finally, we use an in‐house script that pulls all of each site's FreeSurfer outputs and behavioral data and enters them into a relational database using Structured Query Language (SQL). Data are entered into a SQLite database (https://sqlite.org), a free, relational database engine that allows for simple and intuitive data storage using the SQL language (Owens, 2006). Key benefits of a SQL‐based environment—compared to other database options, such as REDCap (Harris et al., 2009) (https://www.project-redcap.org/)—are that it is freely available (not institutionally constrained) and widely used across both research and industry, leading to many options for integrating SQL analyses with popular programming environments such as Python, and statistical packages such as R. SQLite runs quickly and does not require specialized computing resources, and SQLite databases can be easily shared. SQL‐based databases are also being used in other large‐scale data sharing projects, such as the NIH‐funded All of Us research initiative, which aims to gather health‐related data about one million or more people living in the United States (Klann, Joss, Embree, & Murphy, 2019). Given these factors, SQLite provides an accessible option for scalable data analysis and for future data sharing and is aligned with existing big data initiatives. Within the SQLite database, there are seven tables that hold information about demographics, behavioral measures, brain measures (two tables, one for each of the FreeSurfer analysis versions), MR scanner information, research site information, and metadata about the database itself. This database is queried to extract the specific data used in each analysis.

Future directions for improving data intake

A key future priority for updating the data intake process is using artificial intelligence to improve the manual quality control process, capitalizing on the efforts from other ENIGMA working groups (Petrov et al., 2017; Petrov et al., 2018). Currently, performing visual quality control of each segmented FreeSurfer region is a significant bottleneck in our data intake pipeline. Each individual brain has a total of 84 subcortical and cortical regions that should be inspected by a trained investigator, which takes on average about 20 min per brain; performing quality control on so many regions manually is not easy to scale up. In addition, visual quality control is an inherently subjective process. Investigators are intensively trained to ensure good inter‐rater reliability, but there is still the potential for human error. Machine‐learning algorithms to reliably perform quality control of the segmentations would remove an enormous barrier to performing this work. To facilitate the development of an automated quality control process, we have manually generated ratings on the subcortical volumes of over 1,000 poststroke brain MRIs to date. We anticipate that these data can be used to train and test deep learning algorithms (such as convolutional neural networks) to identify failures in segmentation.

DATA HARMONIZATION

A second key challenge that ENIGMA Stroke Recovery faces is harmonizing data collected using different MRI scanners and scanning protocols, as well as using different behavioral outcome measures. Data harmonization refers to all efforts to combine data collected across different sources and with different formats, naming conventions, and measures into one cohesive dataset. Optimizing data harmonization is important because MRI data collected using different scanners and/or scanning protocols can significantly affect analyses and results [e.g., (Zavaliangos‐Petropulu et al., 2019)]. For behavioral data, different measures may focus on measuring specific aspects of sensorimotor impairment, function, or quality, and maybe more or less sensitive to a certain range of performance values. Importantly, the harmonization of methods also provides an opportunity to explore the effects of inter‐site differences on both MRI and behavioral data analysis, which is difficult to examine otherwise. Here, we describe efforts to address each of these challenges so that we can combine imaging scans and behavioral data obtained at different centers.

MRI harmonization

ENIGMA working groups have developed robust methods for analyzing T1w anatomical MRIs, diffusion MRI, and resting‐state fMRI in a way that is reproducible and reliable across multiple research sites (Acheson et al., 2017; Jahanshad et al., 2013; Pizzagalli et al., 2019). While the scans themselves are not directly combined, a number of specific metrics are extracted from each type of imaging sequence so that data across centers can be used in mega‐ and meta‐analyses. The key measures extracted for each sequence are described below. In addition, we provide guidelines for quality control and harmonization in order to ensure that all segmentations are representative of the desired anatomy. In this way, even if the scans themselves are not combined, statistical approaches for combining these extracted features in a meta‐analytical or mega‐analytical manner can be taken. Detailed instructions and code for analysis and quality control for each method may be found on the ENIGMA website ( http://enigma.ini.usc.edu/protocols/imaging-protocols/).

T1‐weighted MRI

As noted previously, a T1w MRI is a key component of the initial ENIGMA Stroke Recovery analyses. T1w MRIs are processed using ENIGMA's structural image processing protocols (publicly available in the link above), which include several methods for analyzing T1w MRIs including cortical and subcortical volume and surface area, sulcal geometry analysis, and vertex‐wise subcortical shape analysis. The ENIGMA structural protocols all utilize outputs from FreeSurfer, a brain imaging software package developed to analyze MRI scans of brain tissue, which segments and labels neuroanatomical structures in the data (Fischl et al., 2002). Importantly, FreeSurfer is reliable across research sites and demonstrates good test–retest reliability across scanner manufacturers and field strengths (Han et al., 2006). In addition, as mentioned above in Data Intake (Section 2.2), we perform manual quality control on all FreeSurfer segmentations, so that subsequent structural analyses use only quality‐controlled inputs. Trained researchers manually inspect each region of interest for each structural brain MRI. Regions that are not properly segmented by FreeSurfer are marked as “failed” and excluded from further analysis (see Appendix 1 for the full ENIGMA Stroke Recovery FreeSurfer quality control protocol).

Diffusion MRI

The ENIGMA‐DTI protocol is used to study fractional anisotropy and diffusion tensor imaging‐derived diffusivity measures of whole brain and atlas‐defined regions of interest in the white matter, based on the tract‐based spatial statistics method implemented in FMRIB Software Library (Smith et al., 2007). While preprocessing diffusion MRI can vary by data quality, the ENIGMA‐DTI protocol provides suggestions including steps for motion correction, echo‐planar imaging distortion correction, and tensor fitting. The ENIGMA‐DTI protocol can be found on the ENIGMA website ( http://enigma.ini.usc.edu/protocols/dti-protocols/) and is detailed elsewhere (Jahanshad et al., 2013). This protocol has been shown to have excellent reproducibility between scanners for the analysis of white matter microstructure (Acheson et al., 2017). The ENIGMA‐DTI working group also works on methods for harmonizing multisite diffusion MRI (Zhu, Moyer, Nir, Thompson, & Jahanshad, 2018), and the ENIGMA Stroke Recovery group will use recently recommended mega‐analytic methods for diffusion MRI analyses (Boedhoe et al., 2019).

Resting‐state fMRI

Resting‐state fMRI (rs‐fMRI) offers an approach to understand patterns of synchronized brain activity in the resting state, which can further be decomposed into networks with known functions (e.g., default mode, salience, attention networks) (Biswal et al., 2010; Biswal, Yetkin, Haughton, & Hyde, 1995). Harmonized processing of rs‐fMRI in ENIGMA has used one of two pipelines: (a) an echo‐planar imaging‐based pipeline, based on the Analysis of Functional NeuroImages software, which does not require the use of a co‐registered anatomical MRI dataset (Adhikari et al., 2018; Adhikari et al., 2019; Adhikari et al., 2019); and (b) a pipeline known as fMRIprep+, based on the fMRIprep approach (Esteban et al., 2019), which can also be used for the analysis of multisite task‐based fMRI (Veer, Waller, Lett, Erk, & Walter, 2019).

Future Directions in Brain Imaging Harmonization

Future directions include the harmonization of stroke electroencephalography (EEG) data, as the ENIGMA EEG working group is currently developing methods for analyzing resting‐state EEG (Smit, 2020; Smit et al., 2016). EEG has garnered growing attention in stroke rehabilitation over the years due to its portability, safety, and lower cost, compared to MRI or fMRI. Advancing our capability to use EEG was also identified as a developmental priority in the SRRR taskforce on biomarkers (Boyd et al., 2017). EEG measures, such as hemispheric asymmetry or frontoparietal coherence, have also been related to and/or predictive of stroke outcomes (Ramos‐Murguialday et al., 2013; Zhou et al., 2018).

Finally, although all our existing MRI pipelines offer methods for handling previously collected data, the ideal scenario is a harmonized prospective data collection. We recommend that individual research groups consider using MRI sequences that match the publicly available sequences from the NIH‐funded Lifespan Human Connectome Project Aging group https://www.humanconnectome.org/study/hcp-lifespan-aging/project-protocol/imaging-protocols-hcp-aging) or the NIH‐funded Alzheimer's Disease Neuroimaging Initiative (ADNI; (http://adni.loni.usc.edu/methods/mri-tool/mri-analysis/). Doing so would expand the reach of one's individual research data by allowing for comparison with large, existing datasets, and help align prospective scanning efforts in ENIGMA Stroke Recovery with that of other groups. Future work may also try to align the ENIGMA diffusion MRI and rs‐fMRI data processing protocols with the UK Biobank imaging efforts, which aim to provide multimodal brain imaging data on 100,000 individuals living in the United Kingdom (Alfaro‐Almagro et al., 2018).

Behavioral and demographic data harmonization

Stroke researchers test a broad range of hypotheses, and different behavioral outcomes are selected to address specific hypotheses. In addition, the same behavior may be assessed using multiple measures. Thus, although many researchers in the ENIGMA Stroke Recovery working group study poststroke arm performance, the current ENIGMA Stroke Recovery database has more than 75 unique behavioral measures. This large number of behavioral measures is in line with a study reporting that 144 different outcome measures were used to study poststroke arm rehabilitation across 243 clinical trials (Duncan Millar, van Wijck, Pollock, & Ali, 2019).

In order to organize the many different types of behavioral data, ENIGMA Stroke Recovery relies on the International Classification of Functioning, Disability and Health (ICF) (World Health Organization, 2001), which provides a framework that can be used to categorize different assessments of poststroke outcomes. The ICF model has three levels that can be used to conceptualize behavioral measurements about a person: (a) body functions and structures (measuring the person's impairment), (b) activities (measuring function at the level of the person), and (c) participation (measuring function of the person as a member of society). The ICF framework is used by the SRRR task force to categorize core measures for all stroke recovery trials to collect (Kwakkel et al., 2017). Specifically, the SRRR recommends collecting the Fugl‐Meyer Assessment (FMA) (Fugl‐Meyer, Jaasko, Leyman, Olsson, & Steglind, 1975) to measure body function and structure for the upper and lower limbs, and the Action Research Arm Test (ARAT) (Lyle, 1981) and a 10‐min walk test to measure activity limitations in the upper and lower limbs. Although there was no consensus recommendation on participation, there were recommendations to collect the EuroQoL 5 dimension scale (EQ‐5D) as a measure of quality of life (Brooks & Group, 1996) and the modified Rankin Scale as a measure of global disability (Van Swieten, Koudstaal, Visser, Schouten, & Van Gijn, 1988). Finally, the National Institutes of Health Stroke Scale (NIHSS) (Brott et al., 1989) was recommended to measure stroke severity (i.e., global impairment across multiple domains), but not as an outcome measure.

These ICF‐based and SRRR‐recommended measures are well represented in the ENIGMA Stroke Recovery database. The most common measures include (listed in order of frequency in the database with percent frequency in parentheses): (a) FMA‐UE (72%), (b) NIHSS (19%), (c) Motor Activity Log (MAL; 16%,) (Uswatte, Taub, Morris, Vignolo, & McCulloch, 2005), (d) modified Ashworth measure of spasticity (12%) (Bohannon & Smith, 1987), (e) ARAT (11%), and (f) Wolf Motor Function Test (WMFT; 9%) (Wolf et al., 2001). Notably, the most common measures focus on sensorimotor performance of the upper extremity after stroke, which reflects the research focus of the many ENIGMA Stroke Recovery working group members who study this topic. As the working group grows more diverse, so too do the behavioral measures. At present, the database also includes measures of gait and balance, such as the 6‐min walk test (Butland, Pang, Gross, Woodcock, & Geddes, 1982), cardiovascular fitness, such as VO2 max (Shephard et al., 1968), cognition (e.g., Mini‐Mental State Examination (Folstein, Robins, & Helzer, 1983), Montreal Cognitive Assessment (Nasreddine et al., 2005), and mood [e.g., the Hospital Anxiety and Depression Scale (Zigmond & Snaith, 1983)].

Prospectively, we recommend that stroke researchers collect all of the measures recommended by the SRRR task force (Kwakkel et al., 2017), with a particular emphasis on the FMA, which is contained in 72% of entries in our database and thus the most well‐represented measure. However, given that ENIGMA Stroke Recovery currently contains only retrospective data, there are several ways that we maximize this data, despite the different behavioral measures.

For our initial study of sensorimotor outcomes, for each site, we define a primary sensorimotor score, which is the measure from the site that is the most widely reported measure in the database. For instance, if a site collected the ARAT, nine‐hole pegboard test, and grip strength, we would use the ARAT as the primary sensorimotor score, since, of the three collected by that site, it is the best represented within the ENIGMA Stroke Recovery database across all sites. Using this measure, we then take a percentage of the maximum possible score for that measure, where 0% is the worst sensorimotor outcome and 100% is the best. For example, if someone received a 33 on the ARAT (where the maximum score is 57), their primary sensorimotor score would be 58%. Doing this also addresses the variability in how people collect even standardized measures. For instance, some research sites do not collect the full FMA‐UE because they do not agree with the reflex measures. The maximum possible score for the site might then be 60 instead of the full FMA‐UE scale of 66. Calculating a percentage of the maximum possible sensorimotor score allows for the normalization of different behavioral scores, enabling a comparison across participants from different research studies. A limitation of our approach is that it mixes measures of sensorimotor impairment, function, and participation, examiner‐ versus patient‐reported outcomes, and different times after stroke that may be more or less ideal for a specific behavioral measure. We attempt to overcome this limitation by also grouping measures by the level of measurement for analyses when possible: impairment (e.g., FMA‐UE, grip strength), function (e.g., ARAT, WMFT), and participation (e.g., MAL, Stroke Impact Scale). Lastly, we analyze the most commonly reported measures separately. Although this last approach limits the sample size for these analyses, it allows for a more rigorous examination of the neural correlates of specific measures (e.g., to study neural correlates of impairment versus function or to examine differences between examiner‐ versus patient‐reported outcomes).

Demographic variables

Many different demographic variables, such as age, sex, hemiparetic side, and time since stroke, have been shown to strongly influence poststroke behavioral outcomes (Appelros, Stegmayr, & Terént, 2009; Chang, Chang, Cragg, & Cramer, 2013; Jongbloed, 1986; Stewart, Gordon, & Winstein, 2014; Stinear et al., 2006; Stinear et al., 2017). Similar to the behavioral outcome measures, the demographic variables collected across research sites and studies are also wide‐ranging. Within ENIGMA Stroke Recovery, the most consistently reported variables are age, race, sex, time since stroke, and lesioned hemisphere. Not surprisingly, these variables are reported with different levels of granularity across research sites. For instance, time since stroke is often reported in units of days, weeks, months, or even years. To maximize precision in at least a subset of the data, we convert time since stroke to days for all (e.g., 2 months after stroke would be converted to 61 days, calculated as 2 × 30.5 days on average per month). Notably, this reduces the precision of the data, especially if the time since stroke is reported in years for some research studies, in which case the value “2 years” could actually represent anywhere from 730 to 1,094 days. To this end, we strongly recommend collecting the smallest unit of measurement possible. Additional variables are highly useful as well, although less frequently reported, such as hand dominance prior to the stroke, therapy received, risk factors for cardiovascular disease such as hypertension, obesity, diabetes and smoking, and comorbidities (including several of the aforementioned risk factors). The latter variables have been shown to affect the brain beyond the lesion and may result in decreased white matter integrity and increased structural atrophy.

Future directions in behavioral measure and demographic variable harmonization

We recommend that researchers prospectively conducting research follow the SRRR task force recommendations regarding both behavioral measurements and demographic variables, as well as the precision with which to collect them (Kwakkel et al., 2017). As data collection of these measures can be subjective, we also highly recommend standardized training for the collection of behavioral measures. For example, previous research has shown that training on how to administer and measure the FMA‐UE (See et al., 2013) can greatly improve measurement reliability across different research sites. ENIGMA Stroke Recovery would also benefit from increased reporting of behavioral measures beyond sensorimotor outcomes, such as measures of gait, balance, cognition, and psychosocial health, as well as neuropsychiatric outcomes. Future directions for this aspect of the dataset include harmonizing and modeling longitudinal data aggregated across research sites. This important analysis would allow us to not only discuss sensorimotor behavior at a single, cross‐sectional timepoint but also sensorimotor recovery, which by definition requires an examination of longitudinal changes over multiple timepoints.

LESIONS

Finally, in addition to harmonizing the measurement of brain volumes and behavior across research sites, ENIGMA Stroke Recovery also has to account for different types of brain lesions. There are two primary types of brain lesions encountered in stroke MRI data: those that are a direct result of the stroke, and white matter lesions, also known as white matter hyperintensities, which represent small vessel disease. Both types of lesions are important to capture in stroke analyses, as both have previously been associated with stroke outcomes across cognitive and motor domains (Arsava et al., 2009; Auriat et al., 2019; Boyd et al., 2017; Feng et al., 2015; Riley et al., 2011). While white matter hyperintensities can be measured using automated software (Ramirez et al., 2011), software to accurately and automatically identify stroke lesions still poses a significant challenge. This is because the size, shape, and location of acute stroke lesions are more variable, and thus less predictable, than white matter lesions, which typically occur in the periventricular or deep white matter and are of a characteristic shape and size. There is a strong scientific interest in understanding how stroke lesions both directly and indirectly (e.g., through secondary degeneration) affect brain structure and, subsequently, behavior. Therefore, the ENIGMA Stroke Recovery working group has developed methods for accurately capturing the stroke lesion to characterize both direct versus indirect effects of the lesion on the brain. This section will detail the steps we have taken to develop a complete neuroinformatics pipeline for large‐scale, semiautomated stroke lesion segmentation.

An open‐source stroke lesion dataset

The current gold standard for lesion segmentation using T1w MRIs is manual segmentation (Ito, Kim, & Liew, 2019). When ENIGMA Stroke Recovery started in 2015, there were few publicly available automated lesion segmentation methods for T1w MRIs. A key reason for the small number of T1w lesion segmentation methods was the limited availability of open‐source stroke MRI data with lesion masks that could be used to develop, train, and test algorithms. Of the available methods, most were developed on small datasets of individuals with stroke (e.g., 8–60 patients) collected at one or two local research sites (Griffis, Allendorfer, & Szaflarski, 2016; Pustina et al., 2016; Seghier, Ramlackhansingh, Crinion, Leff, & Price, 2008). This lack of a large public stroke lesion dataset has not only resulted in limited methods for lesion segmentation but also resulted in methods that may have limited generalizability to more diverse stroke datasets.

To address this problem, we developed a large, manually segmented dataset of N = 304 stroke lesion masks and T1w brain MRIs collected at eight different research sites around the world, which we publicly released (Liew et al., 2018). The Anatomical Tracings of Lesions After Stroke (ATLAS) dataset is the largest open‐source dataset of stroke anatomical MRIs and manually segmented lesion masks (see Box 2). It is shared on two public repositories (ICPSR: https://www.icpsr.umich.edu/icpsrweb/ADDEP/studies/36684; and FCP/INDI: http://fcon_1000.projects.nitrc.org/indi/retro/atlas.html). The procedure and training protocols for lesion segmentation are publicly available (Liew et al., 2018). The purpose of releasing this dataset was to provide a large, standardized test dataset, or benchmark, for researchers to compare the performance of their lesion segmentation algorithms with those of other research groups. Releasing this dataset has now achieved the goal of improving lesion segmentation algorithms and much more. It has been downloaded over 650 times by individuals across more than 30 countries around the world and has been used for educational, research, and industry projects. Toward the primary goal, it has been published in papers reporting improved automated lesion segmentation performance of more than 90% accuracy (Sharique, Pundarikaksha, Sridar, Krishnan, & Krishnakumar, 2019). However, it has also been used in creative and unexpected ways. It has been used as an example dataset in numerous university classes on machine learning and computer vision and as a dataset for masters and PhD theses. It has also led to the creation of new stroke tools, such as probabilistic stroke atlas (Wang, Juliano, Liew, McKinney, & Payabvash, 2019) and an in silico head model for stroke simulation (Bing, Garcia‐Gonzalez, Voets, & Jérusalem, 2020). These unexpected uses highlight a key benefit of data sharing, which is that sharing data can not only move research forward in ways that are intended but can also lead to novel methods, data discovery and powerful educational opportunities for trainees.

ATLAS (Anatomical Tracings of Lesions After Stroke) dataset—the largest open‐source dataset of stroke anatomical MRIs and manually segmented lesion masks (Liew et al., 2018). It can be downloaded from two public repositories:

ICPSR (raw data): https://www.icpsr.umich.edu/icpsrweb/ADDEP/studies/36684

INDI (preprocessed data): http://fcon_1000.projects.nitrc.org/indi/retro/atlas.html

PALS (Pipeline for Analyzing Lesions after Stroke) Toolbox—an open‐source toolbox for lesion analysis (Pipeline for Analyzing Lesions after Stroke; PALS) (Ito et al., 2018).

The PALS toolbox can be downloaded from Github: https://www.github.com/npnl/PALS

Braindrles—a web‐based platform for crowd‐sourcing manual quality control of lesion segmentations (Liew et al., 2019).

Braindrles can be played at https://braindrles.us/

Source code for Braindrles can be found at https://github.com/npnl/braindrles

Comparing automated lesion segmentation methods

Using ATLAS, we systematically compared the existing, publicly available methods for lesion segmentation using T1w MRIs (Ito et al., 2019). Although there are new methods constantly emerging for lesion segmentation, most of these are not publicly available. Of the existing methods available, we determined that Lesion Identification with Neighborhood Data Analysis (LINDA) performed the best (Pustina et al., 2016), with a median value on the Dice coefficient, or measure of similarity between the LINDA segmentations and the gold standard manual segmentations, of 0.5. LINDA performs extremely well on larger lesions, which are the most time‐consuming lesions to manually segment. In ENIGMA Stroke Recovery, we therefore use LINDA to generate initial lesion masks on all the data, and then perform manual quality control and manual correction on the resulting lesion masks (see Box 2 for more details).

A key question regarding the adoption of automated lesion segmentation techniques is: When is the automated lesion segmentation method good enough to use? The answer to this question is determined by the research topic of interest and the research group. For ENIGMA Stroke Recovery, a primary research focus is on understanding how lesion overlap with different brain structures relates to specific behaviors. This can be performed by calculating a lesion load (e.g., lesion overlap) with specific regions of interest, or at the voxel‐level, using methods like voxel‐lesion symptom mapping to examine whole‐brain correlates of lesioned tissue within a stroke cohort. Both of these approaches require precise lesion boundaries, and we are therefore aiming for an automated lesion segmentation method that performs with over 90% accuracy in the ENIGMA Stroke Recovery dataset. As of December 2019, no publicly available automated lesion segmentation method has been able to meet this threshold. When available, we request working group members to submit their expert‐drawn lesion masks with their MRI data. When lesion masks do not exist, ENIGMA Stroke Recovery research staff run LINDA on the MRIs and manually correct the resulting segmentations. In all cases, we perform a careful visual inspection to validate the quality of the segmentations.

Finally, as many recent methods have reported improved performance (e.g., >90% accuracy on the ATLAS dataset) but have not openly shared their methods, we are creating a web‐based challenge for research groups to evaluate their novel lesion segmentation methods on a test dataset, for which the labeled data are hidden from the contestants, similar to previous lesion challenges (Maier et al., 2017). This challenge website is being built in collaboration with researchers at the Paris‐Saclay Center for Data Science using the Rapid Analytics and Model Prototyping framework (https://ramp.studio/), which will require the open sharing of the lesion segmentation method for entry, in order to encourage collaboration, transparency, and reproducibility (Kégl et al., 2018).

Quality control and analysis of automated lesion segmentation outputs

In order to manage the large volume of stroke lesion masks, we also created an open‐source toolbox for lesion analysis (Pipeline for Analyzing Lesions after Stroke; PALS) (Ito, Kumar, Zavaliangos‐Petropulu, Cramer, & Liew, 2018). The PALS toolbox can be downloaded from Github (https://www.github.com/npnl/PALS) and installed locally (see also Box 2). It has a function that creates a local quality control webpage that shows each brain's lesion mask overlaid on the T1w MRI. The webpage is interactive, and each lesion mask can be rated as “good,” “maybe,” or “fail.” This sorts the lesions into the respective folders, and any lesions that fall into the “maybe” or “fail” category are manually corrected. In this way, a person can review hundreds of lesion masks quickly and efficiently, without needing to open and overlay each mask on each brain individually.

We also created a crowd‐sourcing web‐based platform called Braindrles to engage citizen scientists in performing manual quality control of automated lesion segmentation outputs (https://braindrles.us/) (Liew et al., 2019; Box 2). Braindrles is built on the same web‐based platform as its predecessor, Braindr (Keshavan, Yeatman, & Rokem, 2018). Braindrles presents dynamic gifs of a lesion mask overlaid on a stroke brain, and users swipe right if they believe the lesion mask is correct, and left if they believe it is incorrect or inaccurate. There is also an option to “chat” about the lesion mask and ask questions. To encourage player engagement, Braindrles is gamified with a leaderboard and levels based on the number of swipes. Anyone in the public is welcome to play, and all users are given a tutorial as well as feedback about their accuracy on a subset of 100 stroke lesion masks. Because not all users are expected to perform similarly (e.g., a lay person with no brain anatomy knowledge versus an experienced neuroradiologist), a machine learning boosted tree‐based algorithm called XGboost (Chen & Guestrin, 2016) is used to identify and heavily weight raters who are the most accurate. XGboost is also used to generate an aggregate probability score from 0 (fail) to 1 (pass) for each lesion mask, based on the weighted rater scores. To date, Braindrles contains 1,464 lesion masks generated by a mix of both automated and manual segmentations and has garnered over 18,000 votes from over 100 users. Once fine‐tuned, this platform will allow for a higher volume of quality control decisions on lesion segmentation outputs and should provide a scalable solution as the project grows.

Future directions for stroke lesion segmentation

In the future, we aim to create an automated lesion segmentation pipeline with embedded, automated quality control (see Figure 2). This is critical because in addition to FreeSurfer segmentation quality control (mentioned previously), manual lesion segmentation and lesion quality control are the other major bottlenecks in the analysis of large‐scale stroke data. Manual lesion segmentation and quality control is tedious, slow, and requires extensive expertise, and it cannot be easily scaled up for thousands of stroke brains. Thus, an automated pipeline in which manual input is only needed to correct failed lesion masks would greatly accelerate the pace of stroke research and could be used for small and large studies alike.

FIGURE 2

Optimized lesion segmentation pipeline. Example of a future neuroinformatics system for lesion segmentation, with only one point of manual input (manual segmentation of failed lesion masks, indicated in bold)

DISCUSSION AND FUTURE DIRECTIONS

In this article, we have described the approaches and methods that the ENIGMA Stroke Recovery working group has taken to facilitate the establishment of the first large‐scale collaborative multisite stroke brain MRI and behavioral data analyses. We hope that these efforts are useful in addressing key priorities in stroke research, including better characterizing the neurobiology of stroke recovery in humans, and identifying different stroke recovery phenotypes (Bernhardt et al., 2017), with an ultimate goal of discovering biomarkers that predict stroke recovery (Boyd et al., 2017).

The work presented here is designed to ultimately improve our ability to: (a) test whether brain–behavior relationships identified in smaller, more homogeneous samples also exist in larger, more diverse samples and (b) identify characteristics, such as specific lesion locations or specific demographics, which affect stroke outcomes of interest. Toward these two aims, the data generated in ENIGMA Stroke Recovery provide a valuable, standardized sample for researchers to test findings reported in smaller studies on a larger, heterogeneous dataset. Doing so may inform researchers of specific demographics or eligibility criteria that may be critical for specific brain–behavior relationships—a hypothetical example is finding that lesion overlap with the parietal cortex may be significantly related to motor behavior but primarily in people with chronic, right‐hemisphere stroke. In this way, ENIGMA Stroke Recovery's large data approach allows researchers to ask not only “Is this true?” but also “For whom is this true?” The importance of the latter question is that it accepts that what is true for one person may not be true for another and strives to identify individual differences in poststroke neural relationships underlying sensorimotor performance. This approach aligns well with the NIH's Precision Medicine Initiative, which recognizes that there are individual differences in health and recovery and places importance on the personalization of healthcare wherever possible (Collins & Varmus, 2015).

In addition, the exceptionally large size of this database affords the statistical power to begin using supervised machine‐learning techniques to test and train models to predict sensorimotor outcomes and to develop data‐driven hypotheses. Unsupervised machine learning can also be developed to identify clusters, or subgroups, of individuals who show similar behavioral outcomes. These subgroups can then be examined for similar traits that may relate to these outcomes. Although the current size of the ENIGMA Stroke Recovery database is relatively small compared to existing machine‐learning datasets, such as those used in Kaggle competitions (https://www.kaggle.com/), it provides a beginning point for developing these methods for stroke data, and will become more powerful as it grows.

Finally, while we have initially applied this framework toward understanding the structural brain correlates of sensorimotor performance, the infrastructure and tools created can now support many additional questions and analyses. As noted throughout this article, we have begun to expand the types of imaging sequences and modalities collected across sites, as well as the behavioral measures, including cognitive, affective, psychosocial, and interpersonal questionnaires. Scientists who are participating members of ENIGMA Stroke Recovery can propose secondary analyses of the data and partner with other members to ask questions that may require a certain type of data or a specific behavioral outcome measure. ENIGMA Stroke Recovery provides a dynamic and growing platform for international collaborations across diverse topics. Furthermore, these collaborations are bound to extend beyond the datasets with prespecified MRI characteristics as additional methodologies emerge in the future to reconcile the variability across sequences and to extract the data relevant to stroke recovery. For instance, clinical MRI scans obtained during emergency hospitalization for acute ischemic stroke provide an abundance of data related to poststroke outcomes when analyzed using novel machine‐learning methods for stroke lesion segmentation on diffusion‐weighted imaging (Wu et al., 2019). Future artificial intelligence‐powered methodologies will allow researchers to optimize the analysis of these clinical scans and reconcile different types of data and approaches for lesion segmentation. This will further enable the growth and expansion of the stroke recovery database capabilities and data utilization, ranging from “real‐life” clinical scans to research‐protocol driven scans.

There are several current limitations of ENIGMA Stroke Recovery's methods, which we are working to address. First, there are still several manual steps that create a bottleneck in the processing pipelines and limit the speed at which we can perform analyses, specifically regarding the quality control of brain segmentations (from both FreeSurfer and the lesion masks). For instance, performing quality control of just the FreeSurfer subcortical regions requires the manual review of 16 regions per individual—or 16,000 regions for a dataset of N = 1,000. Performing quality control of the FreeSurfer cortical regions will require the review of an additional 68 regions per individual. We anticipate that with enough initial effort, we will be able to generate sufficient data to effectively train machine‐learning algorithms to perform these tasks; however, at present, we have limited the analyses to subcortical regions of interest and lesion analyses. Second, the current methods (e.g., FreeSurfer) work best on high‐resolution scans (e.g., dimensions of isotropic 1 mm3 voxels) with minimal distortions or noise. However, we have access to over 14,000 clinically acquired stroke MRI scans, which typically have much lower resolution (e.g., dimensions of 1 × 1 × 5 mm voxels); we thus need to develop robust methods to organize and analyze these scans. Although FreeSurfer does not work well on this type of data, other methods, such as those examining white matter hyperintensities, ventricular asymmetries, and total brain volume (Etherton et al., 2017; Rost et al., 2018), as well as lesion overlap with standardized template structures, may provide insights into the relationship between specific disease states indicated by these measures and mortality and recovery. Finally, the ENIGMA Stroke Recovery measures are limited to primarily cross‐sectional MRI and behavioral data, with a limited subset of longitudinal data. Questions regarding neural recovery, as well as the neural effects of specific interventions, are more difficult to address because the duration between two time points and the nature of the interventions administered varies widely across research sites. Even within one intervention category, such as brain–computer interfaces or robotic interventions, there are different eligibility criteria for entry into each study. Studies also vary in the total length of time and number of treatment sessions during the study, as well as the specific content of treatment (e.g., specific type of neurofeedback or robotic intervention provided).

While there are significant challenges to harmonizing multisite brain MRI and behavioral data after stroke, there are also many new opportunities for the development of novel methods related to automated quality control and lesion segmentation, robust low‐resolution data analysis, and harmonized longitudinal analysis that may ultimately improve our understanding of neural recovery after stroke. By openly sharing our protocols and methods, we hope to provide useful tools not only for ENIGMA Stroke Recovery but also for any stroke researchers wishing to collaborate with others in the field. Overall, we hope to improve the reproducibility of stroke research by reducing barriers to collaboration and to accelerate the innovation and discovery of more effective, personalized rehabilitation strategies for individuals after stroke.

CONFLICT OF INTEREST

Neda Jahanshad and Paul Thompson are MPIs of a research‐related grant from Biogen, Inc. for work unrelated to the contents of this manuscript. Steven C. Cramer is a consultant for Abbvie, Constant Therapeutics, MicroTransponder, Neurolutions, Regenera, SanBio, Stemedica, Fujifilm Toyama Chemical Co., Biogen, and TRCare.