Leveraging real-world data to investigate multiple sclerosis disease behavior, prognosis, and treatment.

Randomized controlled clinical trials and real-world observational studies provide complementary information but with different validity. Some clinical questions (disease behavior, prognosis, validation of outcome measures, comparative effectiveness, and long-term safety of therapies) are often better addressed using real-world data reflecting larger, more representative populations. Integration of disease history, clinician-reported outcomes, performance tests, and patient-reported outcome measures during patient encounters; imaging and biospecimen analyses; and data from wearable devices increase dataset utility. However, observational studies utilizing these data are susceptible to many potential sources of bias, creating barriers to acceptance by regulatory agencies and the medical community. Therefore, data standardization and validation within datasets, harmonization across datasets, and application of appropriate analysis methods are important considerations. We review approaches to improve the scope, quality, and analyses of real-world data to advance understanding of multiple sclerosis and its treatment, as an example of opportunities to better support patient care and research.

Introduction

Disease course and severity of multiple sclerosis (MS) vary between individuals and over time, and factors influencing these outcomes are poorly understood. The availability of an increasing number of disease-modifying therapies (DMTs) has made treatment decisions more complicated.[1,2] Randomized controlled clinical trials (RCTs; Table 1 provides a glossary of key terms) have been invaluable in evaluating DMTs, since their design reduces the risk of alternative explanations for results that can affect non-randomized studies.[6,7] However, MS treatment has not advanced to evidence-informed personalized medicine. Major questions remain concerning risk stratification, treatment sequencing, outcome definition and monitoring, and biology-based selection of treatment for individual patients, for which RCTs are not feasible.[8] Moreover, results from RCTs might not generalize to clinical populations due to trial restrictions on age, disease type, comorbidities, treatment environment, and other factors.[8] Finally, typical, relatively short RCTs fail to provide information about the long-term risks and benefits of therapies.[8]

Table 1.

Glossary of terms.

Term	Definition
Big data	Extremely large or complex collections of data, which can be mined using non-traditional data processing approaches
Causal inference	The process of reaching a conclusion about whether a causal relationship exists among variables
Comorbidity	Total burden of illness other than the specific (index) disease of interest
Data	Set of values for one or more variables, either quantitative or qualitative
Database	An organized collection of information which is stored electronically for ease of storage, management, and retrieval
Dataset	Data stored in tabular form, for example, rows and columns, often to enable analysis
Disease behavior	Clinical, imaging, and non-biomarker manifestations of MS; their interrelatedness; and how they evolve over time
Effectiveness	Therapeutic benefit under usual use in clinical practice
Efficacy	Therapeutic benefit under optimal circumstances in restricted populations, for example, in a traditional randomized clinical trial
Electronic medical record	An electronic version of a patient’s medical history maintained by the provider over time. May include all of the key administrative clinical data relevant to that person’s care under a particular provider, including demographics, progress notes, problems, medications, vital signs, past medical history, immunizations, laboratory data, and radiology reports[3]
Explanatory clinical trial	A clinical trial primarily designed to determine the safety and efficacy of an intervention under optimal circumstances
Harmonization	The process of combining data from different sources to make measurements, methods, data format, and terminologies compatible. Prospective harmonization is conducted a priori, that is, before data collection. Retrospective harmonization is conducted using data that are already collected. Harmonization can be stringent, the most stringent form being standardization (e.g. using identical methods), or flexible (e.g. using differing data collection methods)
Observational study	A study that draws inferences from an identified sample (such as clinical records or administrative data) to a larger population, where the investigator does not assign treatment but simply notes and analyzes outcomes
Patient-reported outcome measure	A report of a patient’s health status that comes directly from the patient without any interpretation of the patient’s response by a clinician or anyone else (US Food and Drug Administration); an outcome evaluated directly by the patient himself or herself and based on the patient’s perception of the disease and its treatment (European Medicines Agency)
Pragmatic clinical trial	Clinical trial embedded in clinical practice primarily designed to determine the effects of an intervention under the usual conditions of use
Randomized controlled trial	A formal experiment testing the efficacy and safety of an intervention on human subjects, in which the investigator attempts to control variability, confounding factors, and bias by defining the population studied, randomized assignment and application of the treatments, ascertainment and measurement of the outcomes, and analysis of the results;[4] clinical trials generally exist on the spectrum of explanatory or pragmatic design
Real-world data	Data generated in routine clinical practice that capture an individual’s health status or health services use
Real-world evidence	Evidence concerning the risks and benefits of an intervention (or interventions) from analysis of real-world data
Registry	An organized system that uses observational study methods to collect uniform data to evaluate specified outcomes for a particular disease, condition, or exposure and that serves one or more predetermined scientific, clinical, or policy purposes[5]
Standardization	Creation and adoption of uniform technical specifications, criteria, methods, and processes to measure an item, which can be regarded as the most stringent form of (prospective) harmonization

Real-world data (RWD) are used increasingly to address clinical questions related to MS disease behavior, prognosis, and treatment (Table 2). MS registries including RWD have existed for 70 years,[9] but have proliferated recently. Over 500,000 persons with MS have participated in registries and related efforts worldwide,[10] leading to treatment recommendations that guide daily practice.[8] Experience with RWD emphasizes the need for collaboration to address data quality and harmonization, sharing of data across jurisdictions with differing legal and ethical regulations, and sustainability of data sources.[8,11] This review identifies key clinical questions best addressed using RWD; critically evaluates the current uses of available registries, databases, and other MS RWD sources; discusses challenges and barriers to gathering, using, sharing, and pooling such data; and provides conceptual, structural, and analytical solutions to address them.

Table 2.

Multiple sclerosis–related questions addressed using real-world data.

Disease behavior

• Defining disease course, phenotype, and prognosis, including in specific populations

• Predictors of long-term disability

• Impact of comorbidities, health behaviors, genetics and epigenetics, microbiome, and environment

• Pre-diagnosis manifestations

Diagnostic criteria

• Evaluation of the performance of updated criteria compared to previous versions

• Evaluation of the performance of diagnostic criteria in diverse populations

Disease outcomes

• Measurement properties, validity, and usefulness of outcome measures, including clinician-assessed outcomes, patient-reported outcomes, imaging outcome measures, non-imaging paraclinical markers, and health-care utilization

• Interrelationships between outcome measures

• Definition of clinically meaningful change (within an individual) of outcomes

Therapeutic effectiveness of disease therapies

• Long-term effectiveness

• Definitions and predictors of treatment response/non-response

• Comparative effectiveness of treatments in different scenarios, for example, relapsing-remitting disease, progressive disease, as initial therapy, or upon treatment switching

• Comparative effectiveness of treatment strategies, for example, treat-to-target, escalation versus early high-efficacy treatment

• Economics and cost–benefit evaluation

• Utilization in clinical practice

Treatment safety and tolerability of disease therapies

• Safety and risk/benefit assessments, including less common adverse events, delayed adverse events, and cumulative risk of adverse events

• Comparative safety and tolerability between treatments

• Comparative safety in treated subjects versus untreated patients or healthy individuals

• Factors that determine or predict safety and tolerability, including comorbidities, health behaviors, concomitant medications, genetics, race and ethnicity, and therapy adherence and persistence

• Reproductive safety, including to the mother (e.g. ovulation, fertility, pregnancy, breastfeeding), to the father (e.g. sperm production and quality, fertility), to the fetus (e.g. developmental abnormalities), newborn (e.g. short-term impact on developmental milestones), and child (e.g. longterm impact)

Therapeutic effectiveness of non-medication approaches and effects on disease course of other factors

• Benefit of rehabilitation strategies

• Benefit of management of comorbidities or health behaviors

• Effects of pregnancy and breast feeding

RWD

RWD related to an individual’s health status or health services use are collected from various sources—product and disease registries, electronic medical records (EMRs), and administrative (health claims) databases. Other data types include clinician-assessed and performance measures, patient-reported outcome measures (PROMs), imaging studies and biospecimens, and data from wearable devices. The sources of data included in current registries were recently summarized.[11] No single data source can address all study questions. While each source presents specific opportunities and challenges (Supplementary Material, Supplementary Table 1), certain issues are common to all RWD sources. Data quality control is critical to the quality of related research.[12] Data quality control begins with developing a data quality framework which includes a list of data to be collected, groups the data according to data sources, lists the common sources of error for each data source, and identifies potential strategies to prevent or mitigate them. The framework should identify the data quality dimensions (e.g. accuracy, completeness, internal consistency) which are important given the intended uses of the data and list the metrics and operational definitions for each data quality dimension. Finally, the framework should describe data monitoring frequency and how data quality will be reported.[13]

Registries

Registries use observational study methods to collect uniform health status data and are defined by their focus on a particular disease or exposure.[5] Their purpose is defined a priori, and some aspects of data collection are active (solely for registry purposes). Data collection is focused on the registry’s goals, since stakeholder participation may depend on data collection burden. In multicenter disease registries, ensuring high data quality is a major challenge.[14] All participating centers must follow a plan to collect the same minimum dataset using consistent tools, standardized definitions, and trained assessors. Data quality should be routinely evaluated using systematic quality control procedures.[15,16]

EMR data

EMRs are increasingly used in MS care,[17] facilitating clinical information capture, though such data are typically unstructured. Available natural language processing or other techniques that extract information from unstructured data are of limited use,[18] but utilization of structured data fields makes the information more useful for research. Systematic capture of health status data can inform the care of individual patients contributing the data and improve disease management generally, a so-called “learning health system.”[19] This model may be more scalable than passive data collection or active data collection added to clinical care, increasing the range of data elements collected and improving the representativeness of the patients (Table 3 presents an example).

Table 3.

Example of a learning health system in multiple sclerosis.

What is Multiple Sclerosis Partners Advancing Technology and Health Solutions (MS-PATHS)?

• MS-PATHS is a multi-institutional observational study sponsored by Biogen that uses technology to create and capture standardized data from people with MS during routine office visits.[20] By reducing heterogeneity in the way clinical information is captured and reported, the goal is to enable and accelerate research in a large sample of people with MS while providing information to monitor outcomes and support clinical decision-making. The project focuses on data collected as part of clinical care that are comparable, sharable, and usable for research studies. Biogen received feedback from the FDA, so these data potentially could be used for regulatory decision-making.

How are the data collected?

• Patients arrive before clinical visits to complete data gathering, using a device (MSPT).[20] The MSPT includes a battery of neuroperformance tests modeled after the MSFC[21] and PROMs (disease history, patient-reported relapses, PDDS, Neuro-QoL, PHQ-9, and PROMIS physical health).

• Routine clinical MRIs are acquired according to a standardized protocol and shared with the research platform.

• Structured EMR data are also shared with the research platform.

• Data are uploaded immediately to the EMR to be available at the clinical encounter, and anonymized data are simultaneously uploaded to a cloud-based multi-institutional pooled database.

What are the implementation issues?

• Clinicians need to change workflow.

• Some information relevant to clinical care and research, such as neurologist-determined relapse occurrence and EDSS, is not currently captured.

• Some standardized information captured (e.g. symptoms of depression) is more familiar to clinicians than others (e.g. quantification of emotional/behavioral dyscontrol).

• Acquisition and reporting of imaging studies need to be standardized, and radiologists need to change the workflow.

• Pooling of data across institutions and secondary use of clinical data for research require oversight by institutional review boards.

• Information technology teams are needed to design the approach to push data to the EMR and outside of the institutional firewall to the pooled database.

MS: multiple sclerosis; FDA: US Food and Drug Administration; MSPT: Multiple Sclerosis Performance Test; MSFC: multiple sclerosis functional composite; PROM: patient-reported outcome measure; PDDS: Patient-Determined Disease Steps; Neuro-QoL: Quality of Life in Neurological Disorders; PHQ-9: Patient Health Questionnaire–9; PROMIS: Patient-Reported Outcomes Measurement Information System; MRI: magnetic resonance imaging; EMR: electronic medical record; EDSS: Expanded Disability Status Scale.

Administrative (health claims) databases

Administrative data result from the delivery of health care services, reimbursement for health care services, or enrollment into health insurance plans.[22] They may be generated by government or private (commercial insurance) organizations. Information captured may include demographics, diagnoses, procedures, medications, and health services used.

Clinician-reported data and performance measures

In MS studies, the most common clinician-reported data comprise DMT utilization, reasons for changing therapy, relapses, and disability measures (e.g. rating scales based on the neurological examination such as the Expanded Disability Status Scale (EDSS)).[23] Performance-based measures, such as the MS Functional Composite,[23] quantify patients’ ability to perform functional tasks. They assess what the person can do, rather than what the clinician reports (e.g. EDSS) or what the person perceives he or she can do (e.g. PROMs).

Patient-reported data sources

While some domains relevant to MS can be assessed by clinicians (e.g. disability), others can only be assessed using PROMs (e.g. subjective symptoms or quality of life). PROMs are collected from the patient without interpretation or influence of a clinician or other party and are intended to capture a comprehensive picture of the impact of MS and its treatments.[24] Regulatory agencies strongly encourage their use in RCTs.[25] PROMs are increasingly incorporated in patient care and registries.[26]

Several factors should be considered when selecting PROMs, such as validity, reliability, and responsiveness; relation to the research question; relevance to patient experience; and feasibility of use. PROMs are either generic, allowing comparisons across diseases, or symptom- or disease-specific, which might be more sensitive to changes in a particular domain or population. Challenges exist in harmonizing the results of different PROMs assessing the same domain (e.g. fatigue) and accounting for cross-cultural and environmental differences when using a PROM in different populations.[27]

Imaging

Harmonization of magnetic resonance imaging (MRI) data acquired through patient care to allow secondary use for research presents challenges related to standardizing image acquisition, analysis, and reporting.[28,29] Structured radiology report templates that quantify active lesions, lesion burden, and brain volume facilitate capture of MRI data into registries.[30] However, differences in image interpretation, scanners (field strength, vendors), acquisition protocols, and image quality introduce variability, making longitudinal assessment difficult. Importantly, MS-related lesion burden and brain atrophy require segmentation-based image analysis to be accurate and reliable, but use of different methods further increases variability. A partial solution is linkage of clinical registries to MR image repositories, allowing centralized analysis using standardized image-processing software.

Optical coherence tomography (OCT) provides non-invasive quantitative imaging of the retinal structures and is increasingly used as an MS outcome measure.[31] Obstacles to incorporation of OCT data into registries are analogous to those of MRI: optimization and standardization of image acquisition, analysis, quality control, and reporting.[32,33]

Biospecimens

Numerous non-imaging biomarkers for diagnosis, monitoring disease status, and measuring treatment response are under investigation.[34] Analogous to imaging data repositories, biobanks can be linked to registries to augment the information available by including the results of biospecimen analyses. Biospecimens can include blood components (e.g. serum or peripheral blood mononuclear cells), other body fluids, genetic material, microbiota samples, and others. Reuse of specimens for multiple studies reduces costs and time. Standardized operating procedures for sample collection, processing, transportation, storage, and assays reduce variability. Recording meta-data, which may account for variation in the samples collected (e.g. demographics, health behaviors, and time of day of collection), is critical.

Data from wearable biosensors

Traditionally, MS disease status is assessed during office visits. There is substantial interest in utilizing wearable biosensors to measure physiologic and kinetic parameters to assess activity, ambulation, falls, sleep, vital signs, and other parameters during routine activities of daily living, thus reflecting functional status more accurately.[35] How best to analyze the large quantity of data provided by continuous monitoring and integrate it with clinical, patient-reported, imaging, and biomarker data are active areas of research.

RWD challenges common to all data sources

Important differences exist in completeness and quality between data captured through prospectively defined research (whether RCT or observational studies) and those routinely collected. RWD offer the advantage of large samples but typically provide less detailed information compared with the smaller numbers of highly selected and intensely studied participants in RCTs. Information quality from RWD is a multidimensional concept, encompassing the degree to which data are appropriate for the desired purpose (i.e. the research question), syntactic (structure) and semantic (meaning) data accuracy, completeness, depth of information available, consistency, and timeliness. For example, the MSBase database contains information on >60,000 unique individuals, but only approximately 80% have an MRI time point captured and approximately 40% have routine MRI follow-up.[16] Also, information about some data elements such as relapses (e.g. symptoms, severity) are often incomplete, and data quality varies across centers.[16] Increased emphasis on capturing MRI data and implementation of data quality processes will help mitigate these limitations.[16]

Quality of RWD is also influenced by the clinical setting in which such data are gathered and reasons for collection. Important information regarding exposures, outcomes, and confounders such as health behaviors may be lacking, although linkage of different data sources for identified subjects can address this concern. For example, administrative databases include large cohorts but lack information about MS-related clinical characteristics. Linkage to clinical databases and registries can address this gap, allowing more complex research questions, such as the impact of comorbidities on disability progression[36] and identifying biases in tertiary care populations.[37] Similarly, linkage to non-clinical databases can support analyses of the effects of environmental exposures.[38]

Real-world evidence

The US Food and Drug Administration (FDA)[39] defines real-world evidence (RWE) as “clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of RWD.” Observational and interventional studies relying on routinely collected health data from clinical practice constitute forms of RWE. High-quality RWE can support treatment decisions and provide information about quality of care.

Regulatory agencies sometimes use RWE to inform post-approval evaluation of drug utilization, particularly long-term safety and generalizability of safety results from registration RCTs to a clinical setting where patients may differ in age or comorbidity burden from trial populations.[40] Other uses might include identification of drug–drug interactions, assessment of long-term effectiveness, evaluation of alternative dosing regimens, and support of product label revisions or extensions for already-approved agents.[41] While approval of new therapeutics by regulatory agencies usually requires robust RCTs to demonstrate safety and efficacy,[42] alternative study designs might accelerate approval of treatments for rare and serious diseases where placebo-controlled trials are not feasible or ethical, or when sufficient sample sizes or active treatment controls are not available.[43] In such situations, RWD might include historical controls against which a new therapeutic can be compared. The use of RWE generated from RWD in these ways requires a priori engagement of and guidance from regulators, who will usually require information about data structure, content, and standardization; quality assurance measures; and data quality to determine the acceptability of the proposed study design. Regulatory agencies are actively exploring the value of disease registries and other RWD sources for evaluating drugs and devices,[39,44] including in MS.[45]

Considerations in study designs to generate RWE

Use of RWD to generate high-quality RWE requires a clearly articulated research question, high data quality, and a sound study design and analysis plan. A spectrum of study designs can be used to generate RWE, including observational studies (e.g. cross-sectional studies, case–control studies, cohort studies), intervention studies (e.g. registry-based pragmatic RCTs), and meta-analyses synthesizing reported data.

Observational studies

Associations identified in observational studies might be altered due to random error, bias (systematic error), or confounders. These issues and related mitigation strategies are reviewed in the MS context elsewhere.[8] An important obstacle to causal inference is time-varying confounding (i.e. affected by previous exposure or treatment and affecting future exposures). For example, in observational studies assessing the effect of DMT on MS-related disability, individuals with more frequent relapses may be more likely to initiate DMT and, because of this relapse activity, are also more likely to experience short-term disability worsening. Conversely, those who initiate DMT may experience a reduction in relapse rate, which reduces risk of short-term disability worsening. Marginal structural modeling is a common approach that accounts for this issue if the relevant confounding factors are known.[46,47]

Analogous to the Cochrane Collaboration tool to assess risk of bias in randomized trials,[48] several tools have been proposed to assess the risk of bias in observational studies of exposures (Risk of Bias in Observational Studies of Exposures (ROBINS-E)),[49] interventions (Risk of Bias in Non-randomized Studies of Interventions (ROBINS-I)),[50,51] and diagnostic tests (Quality Assessment of Studies of Diagnostic Accuracy included in Systemic reviews (QUADAS and QUADAS-II)).[52-54] The Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework was developed to assess risk of bias, quality of evidence, and recommendations based on observational studies.[55,56]

Pragmatic registry-based randomized controlled trials

Clinical trial designs can be categorized generally as explanatory or pragmatic,[57,58] but exist on a continuum when all of the trials’ design features are considered.[59] Pivotal trials of DMTs in MS have used explanatory designs, evaluating DMT efficacy under optimized conditions. These trials seek to enroll relatively homogeneous populations based on rigorous inclusion and exclusion criteria, and employ randomization and tight control on how the interventions are delivered and outcomes are assessed. However, such efforts to increase internal validity reduce generalizability to clinical practice.

Pragmatic trials can bridge the gap between observational studies and RCTs by evaluating the effectiveness of an intervention in real-world clinical settings. The design and setting can enhance generalizability and lower costs compared to explanatory RCTs, since many costs may be covered through the health system. Randomization in a pragmatic design addresses the major weaknesses of observational studies[60]—the potential lack of comparability of the treatment groups and other biases—and minimizes the influence of known and unknown confounders. Pragmatic trials sometimes require larger sample sizes and longer follow-ups than explanatory trials, as increased heterogeneity may lead to reduced effect sizes.

A pragmatic registry-based RCT uses a RWD source (e.g. a registry) as the platform for recruitment, recording intervention delivery, and collecting follow-up data.[57] This approach can facilitate enrollment by using existing data to determine eligibility and provide information regarding the population not enrolled.[61] Registry-based trials face potential challenges related to data quality, ethical issues, and methodological considerations.[57,60] The data collected may be of lower quality than from an RCT due to missing data; failure to capture relevant prognostic factors; and uncertainty regarding variable definitions, accuracy, and collection methods. Ethical issues may arise related to screening participants for inclusion if they have not consented to the use of their data for this purpose. Consenting participants to receive treatment(s) routinely used in clinical practice, participant withdrawal from a trial using a registry platform, data management, and privacy concerns are also issues. Methodologically, the information a registry captures determines the research questions it can address and study designs it can accommodate.

Data storage, sharing, harmonization, and analysis

Data storage and sharing

Increasingly, funders require data sharing, and scientists seek opportunities for secondary analyses of previously collected data and for joint analyses. Data sharing is a complex topic, which includes research data management plans, open access, licensing, and so on.[62] Timely and effective use of existing data is impeded when existence of the data is unknown, the data are not accessible, or structured documentation about the data is lacking. The FAIR Guiding Principles (findability, accessibility, interoperability, and reusability) propose ways to improve discoverability, access, and reuse of data.[63] Data should be findable with a globally unique and persistent identifier and be registered in a searchable resource. Similarly, meta-data should be registered in a searchable resource and be explicitly linked to the data source. To be interoperable, data should use a formal, shared language (syntax and vocabulary), include references to other (meta-)data, and use vocabularies that follow FAIR principles. Meta-data catalogs are emerging to improve findability, accessibility, and description of data.[64] The European Medical Information Framework, for example, developed a data catalog in Alzheimer’s disease and information technology to support data reuse.[65]

Effective data sharing requires a community of trust and transparency between those who provide and who use the data, and clarity about credit for data creation and ownership, particularly when data derive from clinical practice. Data scientists using information technology solutions are critical to address these concerns.[66,67] Regulations concerning confidentiality and consent are additional barriers, depending on the data source and jurisdiction (see below).[66]

Harmonization

The analysis of combined data from different sources allows comparison and replication of findings, supporting generalizability, and enhancing statistical power in a collaborative and cost-effective manner.[68] Data harmonization can improve comparability of similar measures from different studies to facilitate joint analysis. The analysis of individual-level data following data harmonization offers multiple advantages compared with meta-analysis of published studies, including enhanced comparability across studies; minimization of biases; and the ability to study rarer outcomes, explore sources of heterogeneity, and conduct sensitivity analyses.[68]

Data harmonization can be achieved prospectively (Table 4) or retrospectively. Standardization, the most stringent form of prospective harmonization and the best way to ensure that data from multiple sources are comparable, involves a priori adoption of uniform technical specifications of measurement procedures and of data structure, format, and terminology.[73] This approach improves data quality and synthesis, but agreement between multiple parties and implementation can be difficult. Standardization limits the studies that can be included and takes time to develop given its prospective nature. Flexible harmonization of data across studies using different data collection tools and methods allows inclusion of more studies, but data synthesis is more difficult, and the amount of data that can be harmonized is influenced by the degree of heterogeneity across studies. Common data models (CDMs) are a mechanism by which raw data are standardized to a common structure, format, and terminology independent from any particular study.

Table 4.

Selected examples of prospective data harmonization initiatives in multiple sclerosis.

Pediatric MS Tool-Kit[69,70]

• Measurement framework designed to improve exposure measurement in future pediatric MS etiologic studies and facilitate harmonization

• Includes six core variables each to measure environmental tobacco exposure, sun exposure, and vitamin D intake

The National Institute of Neurological Disorders and Stroke Common Data Elements (CDE) project: MS[71]

• Aimed at facilitating future data sharing including comparisons across studies and data aggregation for meta-analysis, improving data standardization and quality, and reducing the time and costs involved in developing data collection tools

• Provides data standards for clinical research in neurology, including common definitions, core variables (essential for all studies), supplemental variables and exploratory variables, and case report forms

• CDE for MS include demographics, general health history, disease characteristics, paraclinical tests, and outcome measures (patient-reported, clinician-assessed, and performance-based). Some elements are stringent (e.g. core variables), and some are flexible (e.g. multiple choices of outcome measures proposed for fatigue)

Clinical Data Interchange Standards Consortium (CDISC)[72]

• CDISC is a common data model for clinical trials, which is flexible and scalable. It is an organization that develops data standards that “enable information system interoperability to improve medical research and related areas of healthcare”

• Therapeutic area data standards specify how to structure data, but not how to collect the data or what should be collected. Data structure is standardized using a tabulation model that organizes domains and variables within those domains

• Data content is structured using controlled terminology (vocabulary) because it is difficult to combine data when different terms or codes are used for similar or identical concepts

• CDISC standards are required by the US Food and Drug Administration and Japanese Pharmaceuticals and Medical Devices Agency for regulatory submissions

Prospective data harmonization might not always be desirable for technical or scientific reasons. Future data comparability requirements cannot always be foreseen at the time of study design. Moreover, the strength of epidemiologic evidence is enhanced by demonstrating similar findings across studies conducted at varied times with different designs and measurement approaches. Therefore, retrospective data harmonization remains important. A high degree of variability exists in the methods, terminology, and technology used to support retrospective data harmonization initiatives.[74] Potential challenges include timeliness of data access; variable restrictions and procedures surrounding access to individual-level participant data; recognizing contributions of study-specific investigators; diverse authorship rules; time and resources required to conduct the project; the need to gather comprehensive information regarding the aspects of the studies such as design, standard operating procedures, data collection tools, data format, and data content and quality; defining acceptable levels of heterogeneity across databases; data processing and documentation thereof; and sustainability.[74] The Maelstrom Research guidelines proposed a stepwise approach to achieving retrospective data harmonization.[74] The Big MS Data Network has undertaken an examination of the feasibility of retrospective harmonization preparatory to sharing data across five MS registries.[75]

Analysis of data from multiple sources

Several approaches exist for analyzing data from multiple studies. The choice depends on ethical and regulatory limitations on data sharing. These approaches include study-specific analysis followed by meta-analysis, federated analysis, and pooled analysis. In a study-specific analysis, each study is analyzed separately using common variable definitions and protocol[76] to produce an effect estimate. Meta-analysis then assigns a weight to each study using a fixed-effects or random-effects models, producing a summary estimate.

In a federated information governance model,[77] data custodianship remains at the level of the individual contributing data sources (e.g. registries). Person-level data remain on local servers under the direction of local data custodians. Analysis is centralized such that the datasets undergo simultaneous analysis in parallel, and then the results are combined.[78] This approach enables a high level of data security and protection without needing to address international laws and ethics approaches. To succeed, standard operating procedures must be clearly defined because local queries require local harmonization procedures to allow maximal interoperability and efficient exchange of results. Information technology solutions,[79] such as DataSHIELD,[80] have been developed to support such analyses, producing findings similar to individual-level meta-analysis. Several initiatives in MS have used this approach, including the Big MS Data Network.[75]

In a pooled analysis, all data are gathered centrally under a single custodian and analyzed as a whole. This approach requires the greatest data sharing and is the most complex from a governance perspective. It also ignores any heterogeneity in the characteristics of the individual studies being combined, which may lead to paradoxical findings.

Pragmatic issues related to maintaining and using RWD

In addition to methodological challenges related to collection, pooling, and analyzing RWD, several other pragmatic challenges threaten the ability to optimally share data and to maintain data sources.

Data governance and ethics

Legal and ethical frameworks, and data governance structures differ substantially across jurisdictions.[81] In 2018, the European Commission implemented the General Data Protection Regulation (GDPR), which aims to harmonize the rules for all the European Union (EU) member states to reduce the legal fragmentation, complexities, and uncertainties that existed between member states and to strengthen the data subject’s rights.[82] Essential steps needed for data transfer outside the EU include consent adequacy, standard contractual clauses, binding corporate rules, and codes of conduct.[83] Importantly, the GDPR specifies consent language that allows maintenance of personal sensitive data for unanticipated future scientific uses. In 1996, the Health Insurance Portability and Accountability Act (HIPAA) created a legal framework to protect health data derived from US sources.[84] Further regulations are mandated on a state-by-state basis.[85,86] Canadian regulations differ from those in the EU and US and may differ between Canadian provinces.[87] This complexity creates additional barriers to data transfer and sharing between countries and across continents.

Registry governance and sustainability

The feasibility and sustainability of a registry require that issues related to privacy and confidentiality, human and financial resources, governance, and data quality be addressed.[87] Important initial steps in registry development include a precise definition of its goals that reflect the needs of its users and early identification of key stakeholders, which influence whether the registry development can proceed, how the registry will be conducted, and the type and scope of data collected.[5,87]

Most registries have a governance model relying on principles and constraints based on their mandate, operating procedures, legal environment, and funding sources.[88] The governance structure usually includes an Executive or Steering Committee with financial, administrative, legal/ethical, and organizational responsibilities to ensure that objectives are clear and that the interests of the general community are met; a Scientific Committee composed by clinicians, methodologists, and representatives of participating centers which oversees scientific initiatives, promotes specific strategic projects, and approves requests for access to centralized data for research projects; and a variable number of stakeholder representatives.[10] All aspects of governance should be clearly codified in a format that can be revised over time according to the legal or regulatory context and requirements of different stakeholders.

Sustainable funding is a major challenge for maintaining registry infrastructure and for adapting the registry to changes in the legal environment, technology, and key scientific questions over time. Different sources of funding, sometimes temporary or transitional, often coexist[10] and may include patient advocacy organizations, public health or regulatory authorities, product manufacturers, health care service providers, payer or commissioning authorities, academic institutions or consortia, and professional societies. A registry’s funding often evolves over time.

The future of real-world studies

This review focuses on approaches to improve the scope, quality, and analyses of RWD to advance understanding of MS and its treatment. Although substantial progress has occurred in understanding the natural history of MS and developing therapies, many questions persist concerning disease behavior and therapeutics. RCTs and real-world studies, including observational studies and pragmatic trials, are complementary approaches to inform clinical practice, representing a continuum that provides different information with different validity and insights. Some questions are better answered using RWD (Table 2) than data from traditional RCTs. However, RWD present methodological challenges, and care must be taken to address data quality and ensure that study design accounts for potential sources of bias. Despite these challenges, observational studies and registries have informed our understanding and treatment of the disease.[8,11]

Several approaches could improve the quality of RWE generated from existing RWD sources and facilitate its application in MS (Table 5). Creation of a meta-data catalog of existing cohorts and registries would help improve use of existing RWD sources. This meta-data catalog could include a tool kit with standard procedures for exchanging data and guidelines to facilitate data harmonization across datasets, as well as resources and guidance to help investigators navigate increasingly complex privacy and data protection regulations, their evolution over time, and their heterogeneity across jurisdictions.

Table 5.

Approaches to improve the quality of real-world evidence to advance understanding of the MS disease process and treatment.

• Develop a detailed meta-data catalog of existing cohorts and registries

• Develop standard procedures to facilitate exchanging data

• Develop guidelines to facilitate harmonization of data in the MS field

• Develop a tool kit, which would be housed with the meta-data catalog and include information on common ethical and privacy issues (e.g. HIPAA, GPDR), standard informed consent language and process, reporting guidelines, and standard operating procedures for imaging and biospecimen repositories

• Further develop methods to facilitate efficient, standardized, and comprehensive data capture as part of patient care

• Develop the methodology and infrastructure to incorporate data from PROMs, imaging and non-imaging biomarkers, and environmental monitoring into existing and future registries

• Further research into methods of combining and synthesizing real-world data, and methods for handling bias in the context of large linked, longitudinal observational datasets

• Standardize and improve reporting of studies describing real-world data. Although several relevant guidelines already exist, these guidelines need to be implemented with MS-specific journals

• Funding and sustainability of real-world data sources is a key issue, which needs to be addressed

• Develop/retain the workforce of biostatisticians, data scientists, and epidemiologists to support this work

MS: multiple sclerosis; HIPAA: Health Insurance Portability and Accountability Act; GPDR: General Data Protection Regulation; PROM: patient-reported outcome measure.

Currently, patient care and clinical investigation are largely separate activities in most institutions.[89] Data acquired during patient care are sometimes entered into registries and utilized for research, but the process often is inefficient, and the data may not directly inform the care of individual patients from whom the data are generated. The widespread implementation of EMRs offers the opportunity for more efficient collection of standardized comprehensive data during patient encounters for use in both patient management and clinical research.

Many current RWD sources have the advantage of scale but are largely limited to disease and treatment history and clinician-reported measures of disease activity and disability. The utility of RWD would be improved by inclusion of data from performance measures, PROMs, imaging studies, biomarkers, and wearable biosensors—by adding these existing databases or by linkage to other data sources.

Reporting of RWD studies needs to be improved to promote their acceptance by regulatory agencies and the scientific community. MS-specific journals need to implement existing reporting guidelines.[90] Funding and sustainability of RWD sources is also critical. Technical and personnel costs require support from project inception onward, and the need for ongoing support of infrastructure remains a major obstacle for many RWD projects. Finally, the development and retention of the necessary workforce of biostatisticians, data scientists, and epidemiologists with interest and expertise in MS is critical to the field.

Click here for additional data file.

Supplemental material, MSJ892555_supplemental_material for Leveraging real-world data to investigate multiple sclerosis disease behavior, prognosis, and treatment by Jeffrey A Cohen, Maria Trojano, Ellen M Mowry, Bernard MJ Uitdehaag, Stephen C Reingold and Ruth Ann Marrie in Multiple Sclerosis Journal