Making science computable: Developing code systems for statistics, study design, and risk of bias.

The COVID-19 crisis led a group of scientific and informatics experts to accelerate development of an infrastructure for electronic data exchange for the identification, processing, and reporting of scientific findings. The Fast Healthcare Interoperability Resources (FHIR®) standard which is overcoming the interoperability problems in health information exchange was extended to evidence-based medicine (EBM) knowledge with the EBMonFHIR project. A 13-step Code System Development Protocol was created in September 2020 to support global development of terminologies for exchange of scientific evidence. For Step 1, we assembled expert working groups with 55 people from 26 countries by October 2020. For Step 2, we identified 23 commonly used tools and systems for which the first version of code systems will be developed. For Step 3, a total of 368 non-redundant concepts were drafted to become display terms for four code systems (Statistic Type, Statistic Model, Study Design, Risk of Bias). Steps 4 through 13 will guide ongoing development and maintenance of these terminologies for scientific exchange. When completed, the code systems will facilitate identifying, processing, and reporting research results and the reliability of those results. More efficient and detailed scientific communication will reduce cost and burden and improve health outcomes, quality of life, and patient, caregiver, and healthcare professional satisfaction. We hope the achievements reached thus far will outlive COVID-19 and provide an infrastructure to make science computable for future generations. Anyone may join the effort at https://www.gps.health/covid19_knowledge_accelerator.html.

Introduction

Crisis leads to innovations. The COVID-19 crisis stimulated collaborative efforts resulting in a breakthrough in the communication of evidence in scientific literature. Today the evidence is not reported in a form that computers can understand. Evidence is not yet expressed in precise, unambiguous format (i.e., computable formats). The near-infinite variations in how evidence can be expressed using natural language means that it requires substantial expertise and contextual awareness for people to determine if the evidence matters, to interpret what the evidence means, and to determine the certainty of these interpretations. To make scientific evidence shareable, interoperable, and computable, it is essential to use standardized concepts from controlled terminologies and vocabularies. This article introduces early efforts to develop an infrastructure for electronic data exchange for the identification, processing, and reporting of scientific findings, and presents a 13-step Code System Development Protocol created to support global development of terminologies for exchange of scientific evidence

Background

Introduction to Fast Healthcare Interoperability Resources

Fast Healthcare Interoperability Resources (FHIR®) is rapidly overcoming the seemingly intractable interoperability problem in the sharing and exchange of health information [1]. FHIR solves the interoperability problems by breaking down key units of data exchange into resources. Each FHIR resource instance describes a distinct identifiable entity, and each FHIR resource has a FHIR StructureDefinition Resource instance that describes the set of data element definitions and their rules of use that define the FHIR specification itself. Rather than forcing all health-related knowledge to fit one organizational pattern for a common structural model, FHIR enables resource-specific structure definitions to enable the most efficient and flexible approach. Health Level 7 International (HL7®), the standards developing organization that created and maintains FHIR, addresses the human problem in universal agreement to a technical standard by supporting open, transparent, logical processes and systems for people from all perspectives to participate [2].

Extension of FHIR to evidence-based medicine

There is currently no widely implemented standard that overcomes the seemingly intractable interoperability problem of sharing and exchange of computable representations of scientific knowledge. Facing such challenges with the communication of scientific knowledge to inform healthcare decision making, communities within and across researchers, systematic reviewers, guideline developers, and healthcare professionals have advanced human-interpretable expectations for trustworthy interpretation and application of scientific knowledge [3]. This area is often labeled evidence-based medicine (EBM), evidence-based practice, or evidence-based healthcare [4], [5].

HL7 approved a project in 2018 to develop FHIR Resources for Evidence-Based Medicine Knowledge Assets (EBMonFHIR) [6]. In the following 18 months via weekly web meetings and five Connectathons, the EBMonFHIR project created FHIR StructureDefinition Resources for Evidence, EvidenceVariable, Statistic, and OrderedDistribution FHIR resources.

The EvidenceVariable Resource is used to describe a variable used in statistical expressions, with one or more of defining characteristics expressed using standardized concept codes (i.e., codable concepts [7]).

The Statistic Resource supports the expression of a statistic, including the numerical values, the related attributes which are also statistics, and the type of statistic as a codable concept [8].

The OrderedDistribution Resource supports expression of a statistical array [9].

The Evidence Resource supports expression of the statistics for a distinct combination of variables and the certainty of the interpretation of the statistics [10].

Extension of EBMonFHIR to COVID-19 Knowledge Accelerator

Multiple groups in the EBM community sought to use EBMonFHIR resources to support efforts related to global collaboration, cooperation and coordination for identification, evaluation, and reporting of COVID-19 evidence. Participating and related efforts include Agency for Healthcare Research and Quality (AHRQ) evidence-based Care Transformation Support initiative (ACTS) [11], ACTS COVID-19 Guidance-to-Action Collaborative [12], Australian National Clinical Evidence Taskforce [13], Centers for Disease Control and Prevention's Adapting Clinical Guidelines for the Digital Age [14], COVID-19 Advanced Literature Classifier (CALC) [15], COVID-19 DistillerSR Access [16], COVID-19 Evidence Alerts from McMaster Plus [17], COVID-19 Evidence Network to support Decision making (COVID-END) [18], COVID-19 Open Research Dataset (CORD-19) [19], HL7 Biomedical and Research Regulation (BRR) Work Group [20], HL7 Clinical Decision Support Work (CDS) Group [21], Librarian Reserve Corps [22], the LIVING Project [23], Mobilizing Computable Biomedical Knowledge (MCBK) [24], and Systematic Review Data Repository (SRDR) [25].

On March 30, 2020, we started the COVID-19 Knowledge Accelerator (COKA) and by July had more than 150 working meetings with more than 40 active participants from more than 25 organizations from academia, industry, government, and nonprofits in 7 countries [26]. The COKA developed 10 active working groups meeting virtually 12 times per week. COKA efforts revised the FHIR Statistic Resource to include expressions of the statistical model. COKA efforts also created two more FHIR StructureDefinition resources: Citation Resource to support exchange of about 100 elements used to identify articles referenced for scientific reporting [27], and EvidenceReport Resource to support compositions of all the other resources in many combinations [28].

Across the six FHIR resources maintained by the EBMonFHIR/COKA efforts, there were more than 30 elements that would benefit from the use of standardized encoded concepts. Some concepts can be expressed with commonly used code systems such as SNOMED CT® [29], RxNorm [30], and LOINC® [31]. However, we discovered many situations where we could not find a comprehensive code system that was functionally applicable for the concepts commonly communicated.

Methods

Development of code system development protocol

We initially developed code systems [32] with pragmatic approaches by using codable concepts found in other code systems where available (such as the STATistics Ontology [STATO] [33] and National Cancer Institute thesaurus [NCIt] [34]) and developing mnemonic codes for terms commonly used by the EBMonFHIR and COKA participants. Though functional for the growing but small community, the desire for interoperability with many related communities included those represented in the HL7 CDS, Clinical Quality Information, BRR, and Vocabulary Work Groups. This demanded development of methods to support open, multinational, multidisciplinary input; comprehensive attention to existing ontologies; global consensus development; and sustainability planning.

Through multiple open virtual web meetings and shared documents, we developed a Code System Development Protocol (full protocol in Appendix A, related image in Fig. 1 ) which includes 13 steps [35]:

Fig. 1

Code System Development Protocol Flow Diagram.

Assemble an expert working group.

Identify tools or systems commonly used today to express relevant concepts.

Map out a single list of non-redundant concepts to support common uses.

Identify existing ontologies that are openly available without restrictions.

Map related terms and definitions across the ontologies.

Define preferred terms, alternative terms, and definitions for the new code system.

Identify code system entries with universal agreement by the expert working group.

Deliberate suggested changes and reach universal agreement for code system entries where possible.

Deliberate unresolved disagreements and reach at least 80% agreement for code system entries where possible.

Determine the relative contribution of ontologies to the code system and seek further collaboration for heavily used ontologies.

Publish the initial version of the new code system.

Evaluate implementation of the code system and refine the system as needed.

Maintain continued support to adjust the code system based on changes in the prior 12 steps.

Scope setting

We selected four domains for initial application of the Code System Development Protocol and defined them as [35]:

will be used to precisely classify univariate statistics (such as mean, median, and proportion), comparative statistics (such as relative risk, mean difference, and odds ratio), and statistic attribute estimates (such as confidence interval, p value, and measures of heterogeneity). Consistent reporting across systems will facilitate interoperability for science communication.

will precisely communicate characteristics that define the model used for a statistic. Science reports often do not convey complete information about statistical models. Model characteristics may include concepts such as fixed-effects analysis, linear regression, and Mantel-Haenszel method for pooling. Consistent reporting of statistical models will facilitate interoperability for science communication.

will be used to precisely describe methodology characteristics of scientific observations including exposure introduction (such as interventional or observational), cohort definition (such as parallel, crossover or case-control), and group assignments (such as block randomization, every-other quasi-randomization, or non-randomized). Consistent reporting of research study design across systems will facilitate interoperability for science communication.

will be used to precisely describe concerns with methods or reporting of scientific observations including selection bias (such as gaps in randomization or allocation concealment), performance bias (such as gaps in blinding), and analysis bias (such as gaps in intention to treat analysis or selective analysis reporting). Consistent reporting of risk of bias across systems will facilitate interoperability for science communication.”

Step 1: Assemble an expert working group

For Step 1, we developed an Invitation to Join an Expert Working Group for any of the four code systems (Statistic Type, Statistic Model, Study Design, Risk of Bias). Joining the group was open to anyone and group members could self-identify their expertise. Relevant expertise for a code system could include without limitation experience evaluating or expressing the concepts to be included in the code system, either for human interpretation or for machine interpretation.

We shared the invitations through multiple communities (mostly via email distribution lists) including the COKA Initiative, COVID-END, the evidence-based healthcare (EBH) listserv, Grading of Recommendations, Assessment, Development and Evaluation (GRADE) Working Group, the Developing and Evaluation Communication strategies to support Informed Decisions and practice based on Evidence (DECIDE) project participants, the AHRQ evidence-based practice centers (EPCs), the HL7 CDS and BRR work groups, the Society for Clinical Trials, the Society for Participatory Medicine, International Society for Clinical Biostatistics, and Patient-Centered Outcomes Research Institute (PCORI).

Step 2: Identify commonly used tools and systems

For Step 2, we asked Expert Working Group members to identify sources to signal the scope of (or common need for) a code system, namely tools or systems in common current use for reporting concepts relevant to the code system.

Step 3: Create lists of non-redundant concepts

For Step 3, we started with one of the common tools or systems, identified a series of non-redundant concepts for expression to support it, and provided a categorical classification. We then mapped the next identified tool or system, matched concepts where possible, added more concepts where needed, and adjusted the categorical classification. The process was shared openly during weekly Steering Group web meetings and summarized for the Expert Working Group by email distribution lists with open links to the Step 3 mapping spreadsheets.

Time course for initial development

The COVID-19 Knowledge Accelerator consists of 10 active working groups meeting a total of 12 times weekly in open web meetings. Several working groups were developing code systems and the discussions about a common approach started on August 24, 2020. The first draft of a Code System Development Protocol with 11 steps was created on August 28. The protocol was finalized on September 17. Initial efforts were started ahead of wider dissemination of invitations. Invitations to join the expert working groups were sent widely during the week of September 21. All participants were asked to comment by an October 14 cutoff date for communicating the degree of contribution to Step 3 for version 1.0.0 of the code systems.

We report here the results of Steps 1–3 of this effort as of October 14, 2020. These results are not complete in terms of code system development as they do not include definitions or codes and may change through the remaining steps. These remaining steps, and the overall protocol, share and build upon principles and practices in existing ontology development methods [36], [37], [38]. Key aspects such as reusing existing ontologies, enumerating important terms (ie, concepts) across ontologies, and the overall iterative and agile nature of ontology development are well represented in our code system development protocol. We presented our protocol and preliminary findings in an October 30 Workshop on COVID-19 Ontologies (https://github.com/CIDO-ontology/WCO). In November of 2020, we met with ontology developers of the STATO and Ontology of Biological and Clinical Statistics (OBCS), both of which are Open Biological and Biomedical Ontologies (OBO) Foundary recognized ontologies. The ontology developers found our work valuable for identifying gaps, alignments, new terms, and other improvements for existing ontologies and potentially for creating an application ontology.

Results

Expert working groups

As of October 10, 2020, a total of 55 people from 26 countries in 6 continents joined an Expert Working Group for up to four code system development efforts (see Table 1 and Appendix B).

Table 1

Demographics of 55 Members of Expert Working Groups.

Country (total 26)	Australia (1), Bangladesh (2), Brazil (2), Canada (5), Costa Rica (1), Czech Republic (1), Egypt (1), Finland (2), France (1), Ghana (1), Greece (2), India (2), Ireland (1), Italy (2), Japan (1), Lebanon (1), Malaysia (1), Nigeria (4), Peru (1), Romania (2), South Africa (1), South Korea (1), Sri Lanka (1), Switzerland (2), United Kingdom (2), United States (14)
Type of expertise*, n (%)
Researcher	42 (76%)
Evaluate scientific concepts	34 (62%)
Systematic Reviewer	32 (58%)
Statistician	23 (42%)
Guideline developer	14 (25%)
Developer of reporting systems	12 (22%)
Learner	10 (18%)
Software engineer/Informatics specialist	10 (18%)
Write-in responses	Librarian (3), Teacher of medical literature evaluation (2), Clinician/health professional, Terminologist, Standards developer, Qualitative researcher, Book author
Age, n (%)
18–25 years	2 (4%)
26–40 years	16 (29%)
41–55 years	21 (38%)
56–69 years	13 (24%)
70+ years	1 (2%)
Not shared	2 (4%)
Sex, n (%)
Female	18 (33%)
Male	36 (65%)
Not shared	2 (2%)
Race/ethnicity*, n (%)
Asian	11 (20%)
Black	6 (11%)
Hispanic/Latino	6 (7%)
Indigenous	2 (4%)
White	27 (49%)
Not stated	7 (13%)

*More than one selection may apply to each person.

Initial results (Step 2 and Step 3)

Twenty-three commonly used tools and systems were applied across the four code systems, ranging from 2 to 12 per code system (Table 2 ). There were 368 non-redundant concepts (draft display terms for a code system) identified across the four code systems, ranging from 53 to 170 per code system (Table 2, Appendices C, D, E and F).

Table 2

Step 2 and Step 3 Results to Inform Code System Development.

Code System	Tools and Systems Considered	# draft codable concepts
Statistic Type	− StatisticType code system defined by the FHIR project [39] − StatisticAttributeEstimateType code system defined by the FHIR project [40] − ObservationMethodAggregate value set from HL7 V3 ObservationMethod code system [41] − Cochrane Review Manager (RevMan) [42]	88
Statistic Model	− StatisticModelCode code system defined by the FHIR project [43] − StatisticModelMethod code system defined by the FHIR project [44]	53
Study Design	− StudyType code system defined by the FHIR project [45] − ResearchStudyPhase code system defined by the FHIR project [46] − MEDLINE MeSH Headings for Study Characteristics [Publication Type][47] − ClinicalTrials.gov study type classifiers [48] − ResearchStudy-StudyDesign code system used in the database of Genotypes and Phenotypes (dbGaP) [49]	57
Risk of Bias	− StatisticCertaintySubcomponentType code system defined by the FHIR project [50] − StatisticCertaintySubcomponentRating code system defined by the FHIR project [51] − Cochrane Collaboration’s Risk of Bias tool (ROB-1) [52] − Revised Risk of Bias Tool (ROB-2) [53] − Risk Of Bias In Non-randomised Studies - of Interventions (ROBINS-I) [54] − Newcastle-Ottawa Scale for non-randomized studies [55] − Risk of Bias in Systematic Reviews (ROBIS) [56] − Prediction Model Risk of Bias Assessment Tool (PROBAST) [57] − Quality in Prognosis Studies (QUIPS) [58] − Quality Assessment of Diagnostic Accuracy Studies (QUADAS) [59] − Mixed Methods Assessment Tool (MMAT) [60] − Cochrane Handbook Chapter 9 (reporting styles for risk of bias tables) [61]	170

Discussion

Progress toward code system development

Coordinating 55 experts from 26 countries to identify 198 concepts for the development of code systems for scientific methodology (statistics and study design) and 170 concepts for the assessment of quality of evidence (risk of bias) is an early step in what is needed to support interoperable data exchange for scientific communication.

Next steps include mapping concepts across ontologies, reaching universal or near-universal agreement for common code systems for data exchange, and continuous adaptation to meet needs discovered in implementation.

The COKA effort will benefit from the newly crafted HL7 Unified Terminology Governance (UTG) process wherein terminology artifacts, such as the code systems and mappings we are creating, are published by HL7 [62]. The UTG approach aligns with our protocol by subjecting the artifacts created to an open comment and review process. The UTG process starts with transforming the code system and concept map terminology content into FHIR code system and concept map artifacts, typically represented in FHIR JSON or XML [63]. Once the content is entered into the UTG environment, it exists as a set of proposed changes to the core HL7 terminology. Those proposals are made available for review and comment within the UTG environment, consistent with steps 12 and 13 of our protocol. Once comments on the proposed artifacts are resolved and voting requirements are met, if approved, the terminology additions are merged into the HL7 terminology environment at terminology.hl7.org, which is updated and made available through a continuous integration process [63]. In this way, updates and improvements for any content can be developed, proposed, reviewed, improved, voted on and released within a documented environment aligned with the American National Standards Institute (ANSI)-sectioned HL7 ballot process, and ultimately published as part of the official HL7 terminology content.

Our protocol (step 6) includes entering data into an ontology web editor which by design would include top-level ontology concepts (classes, hierarchy, attributes) such as those represented in the Basic Formal Ontology [64] to help refine the classes and hierarchy. The consideration of the FHIR CodeSystem Resource StructureDefinition [65] in preparation for the UTG approach helped us realize we can represent these top-level ontology concepts as property elements within the CodeSystem Resource and we are currently considering modifying step 6c of our protocol to use FHIR tooling directly instead of a web ontology editor.

Strengths and limitations

Strengths of our approach include a substantial spirit of comradery across many diverse people facing a common challenge, multidisciplinary engagement, and coordination with global systems for standards development. In addition, use of FHIR as the underlying standard provides support from a method demonstrated to meet the interoperability needs for a similarly complex global community.

Limitations include the rapid timeline for development, having processed the initial listing of hundreds of concepts in just a month or so. There will undoubtedly be multiple revisions. The current list does not include outcome-specific statistic types (such as mortality for observed proportion or incidence related to death) or application-specific statistic types (such as recall instead of sensitivity for the application to information retrieval). This approach was purposefully taken to maximize simplicity and flexibility. Also, it is not yet established what resources will be needed to complete and maintain the code systems. For the initial effort, the degree of volunteerism and availability was influenced substantially by COVID-19 and we hope the spirit will continue for application across other domains.

Example for computable evidence

We demonstrate a computable expression of evidence [66] with the results (summary effect estimate) of a meta-analysis of three randomized trials [67], [68], [69] for the effect of remdesivir on 14-day mortality in patients with COVID-19 pneumonia. This example includes 43 instances of a “coding” element to express codable concepts with a “system” element to denote the code system, a “code” element to denote the specific code, and a “display” element for human-readable interpretation of the code. For example, the JSON includes (see Table 3 ):

Table 3

Example of coding element.

Element name	Value
System	http://build.fhir.org/codesystem-study-type.html
Code	RCT
Display	randomized trial

This example of computable evidence uses existing codes in published code systems where available, and these may differ from the code systems in development. Where not available, we use “system”: “not yet published” and “code”: “not yet defined” and this shows the need for creation or extension of code systems. One can search the JSON in this example to find 1 code related to study design (“display”: “randomized trial”), 8 codes related to statistic type (“display” values of “Relative Risk”, “Confidence Interval”, “Z-score”, “P-value”, “I-squared”, “Cochran’s Q statistic”, “degrees of freedom”, and “Tau squared”), 4 codes related to statistic model (“display” values of “Meta-analysis”, “Fixed-effects”, “Random-effects”, and “Dersimonian-Laird method”), and 1 code related to risk of bias (“display”: “Lack of blinding”). In this example, the effect estimate is statistically significant using a fixed-effect model and not statistically significant using a random-effects model for the meta-analysis, a situation for which explicit representation of the statistic model is necessary for proper interpretation.

Benefits of code system development

When completed, the code systems will make finding knowledge easier. For example, systematic reviewers may specify study design concepts to facilitate identification of articles meeting their inclusion criteria. The code systems will facilitate re-use of scientific results. For example, clinical trial reporters who express their results for regulatory purposes could re-use the data to express their results for publication, and the systematic reviewers could directly re-use these results without the need for manual data extraction. All of these code systems will expedite recognition of the trustability of scientific knowledge whether seeking the data parameters (as expressed with statistic type codes), the methods for data creation (as expressed with study design and statistic model codes), or the assessments of others (as expressed with risk of bias codes). Someday, via explicit encoded study results, data within published papers can integrate with clinical decision support systems, particularly when reporting meta-analysis results.

We hope the processes, systems, and accomplishments we have produced so far in response to the COVID-19 crisis are sufficient to provide an infrastructure that will endure to make scientific communication accessible for a long time.

Conclusion

We started with efforts to support each other to accelerate knowledge transfer for COVID-19, and then developed solutions with expansive potential. We identified non-redundant concepts to support computable expression of scientific methods. Mapping these concepts to existing ontologies, selecting preferred terms and definitions by the global community, evaluating the implementation of the code systems, and supporting continued development of the systems will support an extensive ecosystem for communicating scientific evidence. More efficient scientific communication will reduce cost and burden and improve health outcomes, quality of life, and patient, caregiver and healthcare professional satisfaction. Anyone who is communicating these concepts may join the effort at https://www.gps.health/covid19_knowledge_accelerator.html [70].

CRediT authorship contribution statement

Brian S. Alper: Conceptualization, Methodology, Investigation, Data curation, Writing - original draft, Writing - review & editing, Supervision, Project administration. Joanne Dehnbostel: Methodology, Investigation, Data curation, Writing - review & editing, Writing - original draft, Project administration. Muhammad Afzal: Methodology, Investigation, Data curation, Writing - review & editing, Writing - original draft. Vignesh Subbian: Methodology, Data curation, Writing - review & editing. Andrey Soares: Methodology, Data curation, Writing - review & editing, Writing - original draft. Ilkka Kunnamo: Methodology, Investigation, Data curation, Writing - review & editing. Khalid Shahin: Methodology, Data curation. Robert C. McClure: Methodology, Writing - review & editing, Visualization.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: All authors are members of the COVID-19 Knowledge Accelerator (COKA) Initiative. The COKA Initiative is a volunteer virtual organization with no funding or contractual relations. The non-software content created by the COKA Initiative (including the data shared in this manuscript) is openly and freely available by Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license. BSA is the owner of Computable Publishing LLC which may commercialize software services related to this content. JD and KS are employed by Computable Publishing LLC. MA, VS, AS, IK, and RCM have no conflicts to report.

Table B1

Expert Working Group Contributors to the Code System Development Concept Lists.

Name	Country	Code System Development Expert Working Group (number of participants)1 = signed up for continued participation2 = AND approved Step 3 results3 = AND actively contributed to Step 3 results
		Study Design (44)	Statistic Type (34)	Statistic Model (32)	Risk of Bias (31)
Gaelen P. Adam	United States	1			1
Muhammad Afzal	South Korea		3	3
Tanvir Ahammed	Bangladesh	1	1	1	1
Brian S. Alper	United States	3	3	3	3
Eric H. Au	Australia	1	1	1	1
Phillip O. Awodutire	Nigeria		2	2
Sébastien Bailly	France	1	1	1	1
Yusentha Balakrishna	South Africa	1	1	1
Sorana D. Bolboacă	Romania	1	3	3
Marek Brabec	Czech Republic		1	1
Stacy B. Brody	United States	3
Comes Calin-Adrian	Romania		1	1
Rachel Couban	Canada	2
Keitty Regina C. de Andrade	Brazil	1	1	1	1
Joanne Dehnbostel	United States	3	3	3	3
Sandra Dimitri	Egypt	1
Marc L. Duteau	Canada		2		2
Zbys Fedorowicz	United Kingdom	1			1
Emilia J. Flores	United States	1		1
Isaac Fwemba	Ghana	1	1	1	1
Abhay M. Gaidhane	India	1
Eric M. Harvey	United States	3			2
Danielle Johnson	United Kingdom	1	1
Samer A. Kharroubi	Lebanon		1	1
Bhagvan Kommadi	India	3	3	3	3
Polychronis Kostoulas	Greece			1
Evangelos Kritsotakis	Greece	3	3
Ilkka Kunnamo	Finland	1			1
Louis E. Leff	United States		2	2
Harold Lehmann	United States	1	1	1	1
Jesus Lopez-Alcalde	Switzerland	1			1
Robert C. McClure	United States	2	2	2	2
Matthew D. Mitchell	United States	1			1
Tamara Navarro-Ruan	Canada	3			1
Pentti Nieminen	Finland		1	1
Akaninyene Patrick Obot	Nigeria	1	1	1
Aloysius Odii	Nigeria	1	1	1
Cheow Peng Ooi	Malaysia	1
Alejandro Piscoya	Peru	1	1		1
Vivek Podder	Bangladesh	1			1
K.M. Saif-UR- Rahman	Japan	1			1
Karen A. Robinson	United States	1	1	1	1
Paola Rosati	Italy	1			1
Carolyn M. Rutter	United States	1	1	1	1
Khalid S. Shahin	United States	3	3	3	3
Roshini Sooriyarachchi	Sri Lanka	1	1
Vignesh Subbian	United States	3	2	1	1
Lehana Thabane	Canada	3	3	3	3
Mario Tristan	Costa Rica	1	1	1	1
Chidi Ugwu	Nigeria	1			1
Linlu Zhao	Canada				1
Name Withheld	Brazil	1			1
Name Withheld	Ireland	1		1
Name Withheld	Italy	1	1	1	1
Name Withheld	Switzerland		1	1

Table B2

Participants in the COVID-19 Knowledge Accelerator (COKA) Initiative.

Name	Organization	A	B	C	D	E	F	G	H	I	J
Gaelen P. Adam	Brown University				o
Muhammad Afzal	Sejong University	o	o							o	o
Eitan Agai	PICO Portal						o
Brian S. Alper	Computable Publishing LLC	o	o	o	o	o	o	o	o	o	o
Ray Alsheikh	Johns Hopkins University		o	o
Stacy B. Brody	George Washington University, Librarian Reserve Corps				o
Mary Butler	University of Minnesota									o
Comes Calin-Adrian	George Emil Palade University of Medicine, Pharmacy, Science and Technology of Targu Mures										o
Rachel Couban	McMaster University				o
Joanne Dehnbostel	Computable Publishing LLC	o	o	o	o	o	o	o	o	o	o
Marc L. Duteau	Duteau Design	o	o						o
Zbys Fedorowicz	Veritas Health Sciences										o
Gilbert, Mike	Evidence Partners Inc.	o					o
Eric M. Harvey	Swedish Health Services, University of Washington				o
Sharon Hibay	Advanced Health Outcomes							o
Alfonso Iorio	McMaster University				o
Jens Jap	SRDR, Brown University							o
Bhagvan Kommadi	Value Momentum		o	o	o	o					o
Ilkka Kunnamo	Duodecim Medical Publications Ltd.	o					o	o
Pawel Kunstman	Evidence Prime Inc.							o
Eddy Lang	Alberta Health Services				o	o					o
Harold Lehmann	Johns Hopkins University	o	o	o	o				o
Sara Loree	Librarian Reserve Corps								o	o
Martin Mayer	EBSCO		o	o	o	o
Robert C. McClure	MD Partners	o							o		o
Tamara Navarro-Ruan	McMaster University				o
Jerry Osheroff	TMIT Consulting									o
Amy Price	Stanford University				o			o
Joshua Richardson	RTI International										o
Karen A. Robinson	Johns Hopkins University	o	o	o	o	o	o	o	o	o	o
Lisa Schilling	University of Colorado						o
Birol Senturk	Brown University EPC, SRDR							o
Khalid S. Shahin	Computable Publishing LLC	o	o	o	o	o	o	o	o	o	o
Andrey Soares	University of Colorado				o			o			o
Ian Saldanha	Brown University EPC, SRDR							o
Vignesh Subbian	University of Arizona				o						o
Jennifer Tetzlaff	Evidence Partners Inc.		o	o	o	o					o
Lehana Thabane	McMaster University			o	o
Mario Tristan	IHCAI Institute-Cochrane Centroamerica and DIME				o		o
Danny van Leeuwen	Health Hats				o	o
Jody Wachs	Vizient		o	o

Bold type used for organization-level participation.

A Project Management Group

B Statistic Type Code System Development Steering Group

C Statistic Model Code System Development Steering Group

D Study Design Code System Development Steering Group

E Risk of Bias Code System Development Steering Group

F Content Citation and Classification Tools Development Work Group

G Evidence Evaluation and Reporting Tools Development Work Group

H Systematic Meta-Review Project Group

I Knowledge Ecosystem Liaison Work Group

J Communications Work Group