The PhenX Toolkit: Establishing Standard Measures for COVID-19 Research.

The PhenX (consensus measures for Phenotypes and eXposures) Toolkit (https://www.phenxtoolkit.org/) is a publicly available, web-based catalog of recommended, well-established measurement protocols of phenotypes and exposures. The goal of PhenX is to facilitate the use of standard measures, enhance data interoperability, and promote collaborative and translational research. PhenX is driven by the scientific community and historically has depended on working groups of experts to recommend measures for release in the PhenX Toolkit. The urgent need for recommended, standard measures for COVID-19 research triggered the development of a "rapid release" process for releasing new content in the PhenX Toolkit. Initially, PhenX collaborated with the National Institutes of Health (NIH) Office of Behavioral and Social Sciences Research, the National Human Genome Research Institute, and the NIH Disaster Research Response (DR2) program to create a library of COVID-19 measurement protocols. With additional support from NIH, PhenX adapted crowdsourcing techniques to accelerate prioritization and recommendation of protocols for release in the PhenX Toolkit. Prioritized COVID-19-specific protocols were used to anchor and define specialty collections of protocols that were subject to review and approval by the PhenX Steering Committee. In addition to the COVID-19-specific protocols, the specialty collections include existing, well-established PhenX protocols, use of which will further enhance data interoperability and cross-study analysis. The COVID-19 specialty collections are Behaviors and Risks; Ethnicity, Race and Demographics; History, Treatment and Outcomes; Information Resources; Psychosocial and Mental Health; and Socioeconomic. The development and usage of PhenX COVID-19 specialty collections are described in this article.

INTRODUCTION

The NIH‐Wide Strategic Plan for COVID‐19 Research states that the National Institutes of Health (NIH) “will facilitate the use of COVID‐19 data to the greatest extent possible, both by those generating the data and by other researchers” (HHS/NIH, 2020, p. 29) and has responded with key behavioral‐ and social science−related efforts as part of the NIH coronavirus disease 2019 (COVID‐19) research response (Riley et al., 2020). The NIH engaged PhenX (consensus measures for Phenotypes and eXposures) to help make COVID‐19 data collection more consistent and results more readily comparable. Historically, the PhenX consensus process has relied on the contributions of working groups of experts, feedback from the scientific community, and overarching guidance from the PhenX Steering Committee (SC) to recommend well‐established, standard measures for release in the PhenX Toolkit (https://www.phenxtoolkit.org/). The urgent need for recommended, standard measures for COVID‐19 research triggered the development of a “rapid release” process for new Toolkit content (for terminology, see Table 1 and Table 2). Prioritized COVID‐19‐specific protocols were used to anchor and define specialty collections of protocols that were subject to review and approval by the PhenX Steering Committee. In addition to the COVID‐19‐specific protocols, the specialty collections include existing, well‐established PhenX protocols, use of which will improve the consistency of data collection and enhance cross‐study analysis. Initially, PhenX collaborated with the NIH Office of Behavioral and Social Sciences Research, the National Human Genome Research Institute, and the NIH Disaster Research Response (DR2) program to create a library of COVID‐19 measurement protocols (referred to as protocols; see Table 1). With additional support from the NIH, PhenX adapted crowdsourcing techniques to accelerate prioritization and recommendation of COVID‐19 protocols for release in the PhenX Toolkit, established COVID‐19 collections of protocols, and made it easier to find COVID‐19 protocols in the PhenX COVID‐19 library.

Table 1

PhenX Concepts and Terminology

Concept	Definition
Domain	A PhenX (consensus measures for Phenotypes and eXposures) domain is a field of research with a unifying theme and easily enumerated quantitative and qualitative measures (e.g., demographics, anthropometrics, organ systems, complex diseases, and lifestyle factors).
Collection	A collection of measures with a shared characteristic, target population, or topic. The measures included in a collection may cut across research domains.
Measure	A measure refers broadly to a standardized way of capturing data on a certain characteristic of, or relating to, a study subject.
Protocol	A protocol is a standard procedure recommended by a Working Group for investigators to collect and record a measure.

Table 2

PhenX Release Details

Release type	Description
Standard release	Uses an established PhenX (consensus measures for Phenotypes and eXposures) consensus process to release protocols in the PhenX Toolkit. Working Groups recommend measures for inclusion based on these criteria: a Clearly defined Well‐established Broadly applicable and generally accepted Low burden to participants and investigators Broadly validated and demonstrated utility Reproducible Specific Reliable Standard measurement protocols exist
Rapid release	An evolved PhenX method to make these cutting‐edge data‐collection protocols quickly available to the research community. The PhenX Steering Committee recommends measures for inclusion based on these criteria: Reputable source Broadly applicable to both research and clinical assessments Input from crowdsourcing results

Recommendations are reviewed and approved by the PhenX Steering Committee before measures are released in the PhenX Toolkit.

Accordingly, the PhenX Toolkit provides two different resources for investigators interested in COVID‐19 research: COVID‐19 collections of protocols that are recommended as standard measures or common data elements, and a COVID‐19 library of protocols currently in use. The COVID‐19 collections of protocols are fully integrated into the PhenX Toolkit and include a combination of COVID‐19‐specific rapid release protocols and well‐established PhenX protocols. Toolkit features to select and implement these collections of protocols are described. The COVID‐19 library (of pdf files) is provided as a reference so that investigators can see what other COVID‐19 protocols are in use. Search and sort features help visitors explore more than 100 measurement protocols and 400 modules (components of measurement protocols). The COVID‐19 collections of protocols (https://www.phenxtoolkit.org/collections/view/8) can be reached by clicking the COVID‐19 Research button on the homepage (Fig. 1). The PhenX SC also recommends use of the Social Determinants of Health (SDOH) Core collection of protocols for COVID‐19 research. The SDOH collections (https://www.phenxtoolkit.org/collections/view/6) can be reached directly from the homepage or from the COVID‐19 Research Collections page.

Figure 1

The PhenX Toolkit homepage.

Strategic Planning

PhenX Toolkit COVID‐19 library

In a collaboration with NIH DR2, PhenX created a COVID‐19 library (Fig. 2). It is important for users to understand that this a resource separate from the typical PhenX infrastructure, for protocols that are processed through the PhenX consensus process and bioinformatics pipeline. The purpose of this resource is to provide current protocols focused on COVID‐19 research; it was intended to help reduce the proliferation of new COVID‐19 measurement protocols (related terminology found in Table 3). The PhenX COVID‐19 library provides currently in use, publicly available COVID‐19‐related measurement protocols (case report forms, data clarification forms, surveys, questionnaires), and consists of 127 full instruments (or “measurement protocols”, Table 3) and 506 modules in PDF format. These protocols were submitted to NIH and then sent to PhenX for further processing. There are 14 main topics, which come from NIH DR2, and over 100 subtopics, which are PhenX‐created. The full protocols are broken into modules, which are subject‐specific. The protocols listed cover a range of topics, from COVID‐19 symptoms and treatments to mental health and socioeconomic changes due to the pandemic. Protocols are categorized both by their source organization and by main topics and subtopics. Each module is assigned from 1 to 3 subtopics. The goal of this page is to provide a one‐stop directory for investigators interested in incorporating existing protocols for COVID‐19 research—rather than creating new ones—in their studies. This page is publicly available for use at no cost to the scientific community in a web‐based resource.

Figure 2

PhenX COVID‐19 library.

Table 3

Key PhenX COVID‐19 Concepts

Concept	Definition
Full Instrument	The full‐length protocol or questionnaire submitted by the original authors.
Module	A section or portion of a full instrument. These subject‐related sections are suggested by the authors and the National Institutes of Health (NIH). PhenX (consensus measures for Phenotypes and eXposures) creates modules from the full instruments and assigns sub‐topics.
Main Topic	NIH Disaster Research Response (DR2) categories.
Subtopic	PhenX‐created categories to describe the content of the modules.

Creating COVID‐19 collections

Based on PhenX SC recommendations, crowdsourcing was implemented to rapidly filter and recommend a subset of the COVID‐19 protocols that had been submitted to NIH. PhenX used crowdsourcing to help inform the subject focus of the collections and the content of the collections. The PhenX COVID crowdsourcing campaign took place in three distinct phases. The phases were distinguished based on their intended purpose rather than their touchpoint with participants. The crowdsourcing effort's full breadth is a novel approach, but is grounded in these preexisting methodologies to ensure functional outcomes.

A web‐based platform called OptimalSort (https://www.optimalworkshop.com/optimalsort/) was employed to create and distribute all phases of crowdsourcing. Questionnaires appropriate for each phase were programmed in this platform and distributed over e‐mail listservs. Phase 1's intention was to discover the protocol areas that researchers, and the general public, felt were important in COVID‐19 research. This step used individual “cards” to sort; participants were presented these cards with a 5‐min time limit to perform this task. Phase 2's intention was to group the results of Phase 1 into possible collections, using a Dendrogram Study methodology, and using the top 75% of topics from Phase 1 to form the basis of the next card‐sort methodology. This step of the process used an “open card sort,” where individuals produced buckets of their design to create groupings without the bias introduced by the study hosts. Phase 2.5 verified the results from Phase 2 by having the participants pick a group for each topic area and place topics into the buckets in which they believed the topics belonged. These categories also should have received their highest‐rated naming by the results of Phase 1 and Phase 2 and informed the COVID‐19 collection names and study areas. Topics slotted into the categories in Phases 1‐2.5 also provided a foundation used to inform which PhenX protocols and new COVID‐19 protocols would be a part of these collections. Phase 3 was intended to pick the measurement protocols that would end up in the final collections. Since this step was very time intensive for the user, standard triage measures were used to trim the number of options for each task to a manageable level for participants. Individuals read through the possible protocol options and selected the one they believed was the best option. These were then ranked by the overall vote each one received.

PhenX then utilized the results to inform which existing PhenX protocols and which new COVID‐19 protocols would be in these collections. The PhenX steering committee (SC) conducted a final review, and protocols were designated as “rapid release” as part of the evolved PhenX process. Overall, a typical start‐to‐finish process for a collection takes around 1 year; the process to release the COVID‐19 collections took about 3 months. The collections were finalized and released in the PhenX Toolkit on October 30, 2020.

COVID‐19 collections

The PhenX Toolkit created six COVID‐19 Research specialty collections (Fig. 3, Table 4) to promote use of common data elements. The protocols were released with a “rapid release” status, indicating our effort to make these cutting‐edge data collection protocols quickly available to the research community. Primary considerations for selection of the recommended COVID‐19 protocols included their being from a reputable source, that they were broadly applicable to both research and clinical assessment, and that that they were based on input from crowdsourcing results. Other considerations included availability in other languages, applicability to people of various life stages, being available free of charge to the public, and willingness to address health disparities.

Figure 3

The PhenX COVID‐19 Research Collections page.

Table 4

PhenX COVID‐19 Research Collections

Specialty collection	Protocols
Behaviors and risks	Alcohol‐Treatment for alcohol usage Body Mass Index COVID‐19 Knowledge, Attitudes, and Avoidant Behaviors a COVID‐19 Related Social Distancing Experiences a COVID‐19 Related Social Distancing Experiences (Spanish) a COVID‐19 Related Social Interactions a Injection Drug Use Substance Abuse and Dependence–Past Year–Drugs Substances‐Lifetime Use Treatment Availability and Level of Care Use of Personal Protective Equipment (PPE) by Healthcare Workers a Use of Tobacco Products
Ethnicity, race and demographics	Birthplace Current Employment Status Current Marital Status Cyberbullying Due to Racial/Ethnic Bias a Discrimination Disparate Health Care Quality Educational Impact Due to COVID‐19 a Ethnicity and Race Race/Ethnic Residential Segregation–American Community Survey Race/Ethnic Residential Segregation–Separation (S) Index, Unbiased Race/Ethnic Residential Segregation–U.S. Census Years Living in the U.S.
History, treatment and outcomes	Access to Health Services COVID‐19 Related Health Questions a Health Conditions, Medications and Health Care During COVID‐19 Pandemic a Medical History Medication Inventory Personal History of Allergies, Infectious Diseases, and Immunizations–Adult Personal Medical History of Allergies, Infectious Diseases, and Immunizations–Child Vulnerability and Exposure to COVID‐19 a
Information resources	Access to Health Technology COVID‐19 Related Information and Trust in News and Government Institutions a Media Use Media Use During COVID‐19 a
Psychosocial and mental health	Broad Psychopathology–Adult Coping with COVID‐19 a Coronavirus Anxiety Scale a COVID‐19 and Mental Health Impacts a COVID‐19 Related Impact on Sleep a Effects of COVID‐19 Outbreak–Adult a Effects of COVID‐19 Outbreak–Adult (Spanish) a Effects of COVID‐19 Outbreak–Child Parent‐Report a Effects of COVID‐19 Outbreak–Child Parent‐Report (Spanish) a Effects of COVID‐19 Outbreak–Child Self‐Report a Effects of COVID‐19 Outbreak–Child Self‐Report (Spanish) a Perceived Stress Quality of Life‐Adult Quality of Life‐Pediatric
Socioeconomic	Annual Family Income COVID‐19 Related Employment Status a COVID‐19 Related Household Finances a Current Employment Status Educational Attainment–Individual Family Interpersonal Relationships Food Insecurity Health Insurance Coverage Job Insecurity School Social Environment Social Networks Wealth

Indicates “rapid release” COVID‐19 protocols. Note that all other protocols are PhenX protocols that were released using the traditional PhenX process.

COVID‐19 Knowledge, Attitudes, and Avoidant Behaviors

COVID‐19 Related Social Distancing Experiences

COVID‐19 Related Social Distancing Experiences (Spanish)

COVID‐19 Related Social Interactions

Use of Personal Protective Equipment (PPE) by Healthcare Workers

Cyberbullying Due to Racial/Ethnic Bias

Educational Impact Due to COVID‐19

COVID‐19 Related Health Questions

Health Conditions, Medications and Health Care During COVID‐19 Pandemic

Vulnerability and Exposure to COVID‐19

COVID‐19 Related Information and Trust in News and Government Institutions

Media Use During COVID‐19

Coping with COVID‐19

Coronavirus Anxiety Scale

COVID‐19 and Mental Health Impacts

COVID‐19 Related Impact on Sleep

Effects of COVID‐19 Outbreak–Adult

Effects of COVID‐19 Outbreak–Adult (Spanish)

Effects of COVID‐19 Outbreak–Child Parent‐Report

Effects of COVID‐19 Outbreak–Child Parent‐Report (Spanish)

Effects of COVID‐19 Outbreak–Child Self‐Report

Effects of COVID‐19 Outbreak–Child Self‐Report (Spanish)

COVID‐19 Related Employment Status

COVID‐19 Related Household Finances

The creation of COVID‐19 Research collections fully integrates the rapid release COVID‐19 protocols into the PhenX Toolkit. That is, the Toolkit provides a brief description of each measurement protocol, the purpose and rationale for the measurement protocol's inclusion, standard protocols, references, and requirements (e.g., training, personnel, equipment, applicable licensing fees) for data or specimen collection (Figs. 4, 5, 6, 7, 8). Toolkit users can search or browse for these COVID‐19‐related protocols and save selected protocols in My Toolkit. Users can also download a custom Data Collection Worksheet (DCW) that details their selections and allows them to easily “cut and paste” protocols to incorporate them into their data collection study design documentation. Finally, users can also download a REDCap (Research Electronic Data Capture) instance for data collection.

Figure 4

A measurement protocol information tab.

Figure 5

A measurement protocol administration information tab.

Figure 6

A measurement protocol detailed information tab.

Figure 7

A measurement protocol source information tab.

Figure 8

A measurement protocol general information tab.

SELECTING COVID‐19 PROTOCOLS

The following provides a step‐by‐step process in selecting COVID‐19 protocols to include in a new study. In this scenario, an investigator has designed an epidemiological study of the effects of the COVID‐19 pandemic on minority populations. The investigator plans to assess a variety of phenotypes and exposures. The investigator is interested in how minority populations may be disproportionately affected by COVID‐19 infections and by impacts due to shutdown mandates. The investigator wants to include high‐quality measures, ensure consistent data collection, and be well positioned for data sharing and cross‐study analysis when the study is complete.

The investigator has designed the study to include collection of basic demographic information and changes that may have occurred since the start of the COVID‐19 pandemic. The investigator recognizes that using PhenX measurement protocols in the study will facilitate combining their data with other studies that have also used PhenX protocols and will allow them to more readily replicate findings from other studies. The investigator visits the PhenX Toolkit to identify recommended protocols of phenotypes and exposures that could be included in their COVID‐19 study.

Materials

Computer or laptop

Internet browser

Navigate to https://www.phenxtoolkit.org and browse protocols in the COVID‐19 Research Specialty Collections (see Fig. 3).

Add the entire Ethnicity, Race and Demographics collection to your toolkit. This will display the contents of the toolkit (Fig. 9).

Figure 9

My Toolkit contents after selecting measurement protocols.

Scroll to the bottom of the page and select “Add all Essential Protocols to your Toolkit” (Fig. 10).

Figure 10

My Toolkit showing need to add Essential Protocols.

After protocols are added to My Toolkit, the Toolkit presents “Essential Protocols” identified by domain experts. For example, “Current Age” is essential to interpreting the results of two protocols: “COVID‐19 Related Employment Status,” and “COVID‐19 Related Household Finances.” As a result, the investigator clicks “Add all Essential Protocols to your Toolkit,” and these protocols are now added to My Toolkit.

Browse Social Determinants of Health: Core (Fig. 11).

Figure 11

The Social Determinants of Health: Core collection (protocols not seen in the figure: Health Insurance Coverage, Health Literacy, Occupational Prestige, Sexual Orientation).

Based upon research needs, investigators can customize their toolkit by returning to the COVID‐19 Research Collections page (Fig. 3) and adding additional collections that might be of interest.

Search the entire Toolkit for additional measurement protocols (Fig. 12)

Figure 12

Results of search using keyword “nutrition.”

Optional: If investigators do not identify a collection of interest by name alone, they can utilize the Search bar at the top of every page to look for specific measurement protocols.

To see additional COVID‐19‐related measurement protocols already in use, investigators can visit https://www.phenxtoolkit.org/covid19 to browse the COVID‐19 Protocol Library.

COVID‐19 measurement protocols and modules can be sorted either by subtopic (Fig. 13) or source organization. Source organization and source contact e‐mail are also provided. The library is also searchable by keywords.

Figure 13

PhenX COVID‐19 page sorted by subtopic.

Once all protocols have been added to “My Toolkit,” click “Select All” or individually select protocols to highlight the protocols of interest.

By selecting “Download,” investigators can choose to download a basic report, a detailed report, a data collection worksheet (multiple language options), or a data dictionary (multiple formats).

Basic Reports include a listing of all of the protocols in “My Toolkit” and the essential measures associated with them. For each protocol, it lists protocol ID, protocol description, specific instructions, and the protocol text.

Detailed reports include a listing of all of the protocols in “My Toolkit” and the essential measures associated with them. For each protocol, it lists protocol ID, protocol description, specific instructions, protocol text, selection rationale, language, participants, personnel and training required to perform the protocol, equipment needs, standards, references, protocol type, derived variables, and requirements.

Data Collection Worksheet(s) include questions to collect information from research participants.

Data Dictionaries provides definitions of variables and other relevant parameters for a study's data management system. It also provides detailed information about each variable as well as skip patterns that are implemented throughout.

Downloading in REDCap format allows investigators to upload directly to their pre‐existing REDCap data collections

COMMENTARY

Background Information

The PhenX Toolkit is a web‐based resource that makes it easy for researchers to add high‐quality, well‐established standard measures to their studies. The value of standard measures is widely recognized (Fortier et al., 2010, 2011; Hendershot et al., 2015; Windle, Bennett, & Noyes, 2011) because they facilitate cross‐study analysis and are relevant for genome‐wide association studies and other biomedical studies. Without standard measures, investigators must infer correspondence between data that are conceptually related but were collected using different methodologies, and attempt to harmonize disparate data across multiple studies. This can be a time‐consuming and difficult process that often reduces the number of variables that can be used for data analysis.

This article highlights the features of the PhenX Toolkit that allow researchers to access and incorporate recommended COVID‐19 protocols into their studies. These COVID‐19 collections of protocols cover key subject areas related to the COVID‐19 pandemic. In addition, researchers can utilize the COVID‐19 library to access additional COVID‐19‐related protocols currently in use by a wide array of sources, such as the World Health Organization and the Environmental Influences on Child Health Outcomes Program. This overview is intended to introduce the PhenX Toolkit COVID‐19 specialty collections and to promote the use of these COVID‐19 measures and common data elements.

Critical Parameters and Troubleshooting

The PhenX Toolkit is intended to provide support for a variety of study designs. Because not all protocols are suitable for all study designs, decisions on which protocols to select are the responsibility of the investigator. For the protocols included in the COVID‐19 Research collections and the entire PhenX Toolkit, the Toolkit provides the following:

Related Protocols are other protocols in the Toolkit that may be of interest (related protocols are suggestions only);

Essential Protocols that are required to correctly interpret selected protocols;

DCW to improve consistency of data collection; and

Data Dictionary (DD) to enhance data sharing and compatibility with REDCap.

For example, Current Age is considered essential to interpreting COVID‐19 Related Employment Status. When a user selects COVID‐19 Related Employment Status, they are encouraged to also select Current Age. The Toolkit also offers a DCW and REDCap DD. The DCW helps investigators integrate PhenX protocols into their existing research and also helps ensure that investigators collect the data needed for their measures. The DD describes the variables associated with the selected measures.

Understanding Results

PhenX provides a resource of recommended and standard measurement protocols for use by the research community. By incorporating standard protocols into their studies, investigators can amplify their results by comparing or combining their data with other studies using the same standard protocols. NIH is supporting and coordinating multiple data science and research efforts to better understand, treat, and mitigate COVID‐19. Specifically, NIH supports data sharing and data interoperability via the PhenX Toolkit, the NIH Public Health Emergency and Disaster Research Response (DR2), and the Systemic Harmonization and Interoperability Enhancement for Laboratory Data (SHIELD). NIH also supports COVID‐19 research initiatives and development of new technologies, as evidenced by the newly established Collaborating Network of Networks for Evaluating COVID‐19 and Therapeutic Strategies (CONNECTS), the National COVID Cohort Collaborative (N3C), and the Rapid Acceleration of Diagnostics (RADx) initiative.

NIH has posted 11 requests for application, 9 notices, and 1 program announcement related to COVID that mention PhenX in 2020 from the Office of the Director, National Institute on Drug Abuse, National Institute on Minority Health and Health Disparities (NIMHD), and National Institute of Mental Health (NIMH). These funding opportunities from the Office of the Director, NIMHD, and NIMH encourage the use of both the PhenX COVID‐19 library and collections.

For the PhenX COVID‐19 collections, the process is an evolution of the PhenX consensus process that allows for more rapid identification of high‐quality measures suitable for release in the Toolkit. PhenX will add a variable compare tool that is envisioned to assist investigators in finding protocols best suited for their research needs.

The COVID‐19 collections will allow researchers to combine and compare studies with increasing ease. Studies need to be compared in order to replicate results. In addition, combining studies increases the sample size, thus providing the statistical power needed. Widespread adoption of PhenX measures will have a dramatic impact on biomedical research and ultimately on the health and well‐being of the population. Using the NIH‐recommended COVID‐19 efforts alongside PhenX will help increase understanding of the COVID‐19 pandemic and its related socioeconomic effects.

Time Considerations

The time required to collect data on a particular measure is addressed in the Requirements Table (see Fig. 5) for each specific protocol. Time as it pertains to use of the PhenX Toolkit is difficult to estimate. It could take as little as an hour for an investigator to select a few measures and request a Report, DCW, and DD. However, it is more likely that an investigator would spend an hour or two becoming familiar with the Toolkit, select a few measures, and then return to the Toolkit later to select additional measures. The PhenX Toolkit should be used as an integral part of the development of a new study.

Author Contributions

Michelle C. Krzyzanowski: Conceptualization, Data curation, Funding acquisition, Methodology, Software, Supervision, Writing‐original draft, Writing‐review & editing, Ian Terry: Data curation, Formal analysis, Methodology, Software, David Williams: Data curation, Software, Writing‐review & editing, Pat West: Writing‐review & editing, Lauren N. Gridley: Data curation, Resources, Carol M. Hamilton: Conceptualization, Funding acquisition, Investigation, Project administration, Supervision, Writing‐original draft, Writing‐review & editing

Conflict of Interest

The authors declare no conflict of interest.