A Regulatory Science Initiative to Harmonize and Standardize Digital Pathology and Machine Learning Processes to Speed up Clinical Innovation to Patients.

Unlocking the full potential of pathology data by gaining computational access to histological pixel data and metadata (digital pathology) is one of the key promises of computational pathology. Despite scientific progress and several regulatory approvals for primary diagnosis using whole-slide imaging, true clinical adoption at scale is slower than anticipated. In the U.S., advances in digital pathology are often siloed pursuits by individual stakeholders, and to our knowledge, there has not been a systematic approach to advance the field through a regulatory science initiative. The Alliance for Digital Pathology (the Alliance) is a recently established, volunteer, collaborative, regulatory science initiative to standardize digital pathology processes to speed up innovation to patients. The purpose is: (1) to account for the patient perspective by including patient advocacy; (2) to investigate and develop methods and tools for the evaluation of effectiveness, safety, and quality to specify risks and benefits in the precompetitive phase; (3) to help strategize the sequence of clinically meaningful deliverables; (4) to encourage and streamline the development of ground-truth data sets for machine learning model development and validation; and (5) to clarify regulatory pathways by investigating relevant regulatory science questions. The Alliance accepts participation from all stakeholders, and we solicit clinically relevant proposals that will benefit the field at large. The initiative will dissolve once a clinical, interoperable, modularized, integrated solution (from tissue acquisition to diagnostic algorithm) has been implemented. In times of rapidly evolving discoveries, scientific input from subject-matter experts is one essential element to inform regulatory guidance and decision-making. The Alliance aims to establish and promote synergistic regulatory science efforts that will leverage diverse inputs to move digital pathology forward and ultimately improve patient care. Copyright:

“The scientist and science provide the means, the politician and politics decide the ends.”

INTRODUCTION

Regulatory science is an established discipline that entails the application of the scientific method to support regulatory and other policy objectives.[2] Simply put, when medical research provides a novel solution to a health need, regulatory science applies the scientific method to assess benefits and risks before marketing for clinical use. To assess benefits and risks, regulatory scientists develop new tools, standards, and approaches to evaluate the effectiveness, safety, and quality of medical products. A primary challenge in the field of digital pathology is the lack of understanding that strong relationships between regulatory, basic, and translational scientists can substantially improve clinical innovation.[3456] For example, regulatory science is not restricted to regulatory agencies.[2456] As a scientific discipline, regulatory science challenges current concepts of benefit and risk assessments, submission and approval strategies, patient involvement, and various ethical aspects. Regulatory science includes the creation of a scientific dialog for launching new ideas – not only derived from industry and regulatory authorities but also by, for example, academics, clinicians, and patients.[7] It has been recognized that regulatory science can have a significant impact in bringing new devices to patients in need.[7]

Here, we outline a recently established, volunteer, collaborative regulatory science initiative termed the Alliance for Digital Pathology (the Alliance). To prevent confusion, our intent is to familiarize the community with the aims, scope, and rationale of the Alliance. The Alliance aims to move the field of digital pathology forward by systematically assessing relevant aspects and providing publicly available resources (e.g., data, tools, and methods) to inform and improve the relevant regulatory guidance landscape.[8] Our premise (thesis) is that the Alliance promotes regulatory science as a bridge between digital pathology (the means) and moving the field of diagnostic pathology forward (the ends). By promoting regulatory science, the Alliance helps to unlock the potential of new technologies and thereby overcomes the dichotomy illustrated in the epigraph by Dr. Weinberg.[1]

TOWARD AN OPERATIONAL DEFINITION OF A CLINICAL, INTEROPERABLE, INTEGRATED SOLUTION FOR DIGITAL PATHOLOGY

The key aim of the Alliance is to help convert the existing (traditional) pathology technologies and workflows into interoperable, digitally enhanced solutions by contributing regulatory science deliverables that can be used to inform and improve the applicable regulatory guidance landscape. Numerous groups have attempted to specify the relevant components of digital pathology solutions;[9101112131415161718] however, given the modularized nature of diagnostic pathology, defining the specific scope of a digital pathology solution is highly context dependent. For example, the variability of a stain (e.g., hematoxylin and eosin across or within laboratories) may influence the performance of a downstream mutation prediction algorithm.[192021] In this example, one may consider drawing an arbitrary boundary before the staining step; however, the fixation and processing method (e.g., formalin fixed, paraffin embedded) or even the tissue acquisition, handling, or image acquisition[22] may influence the performance of the predictor as well. Thus, for the purpose of the Alliance, we considered three descriptors for the solution. First, we aim toward a clinical (as opposed to a research-based) solution. Second, due to the modularized nature of the various subprocesses within the main workflows in pathology, we aim for interoperability of systems. Third, to account for the various and arbitrary boundaries of workflow steps (modules) and technologies relevant for a given task (intended use), we consider every step, from the medical procedure acquiring the cell or tissue sample all the way to the fully integrated diagnostic output (e.g., report or model output), as relevant. As opposed to an end-to-end solution, where the supplier of an application or system will provide all the hardware and/or software to meet specific requirements, we are aiming for modularized solutions within the main workflow. We refer to these three solution descriptors (clinical, interoperable, and modularized) as an “integrated solution” for digital pathology. We acknowledge that this definition is operational and arguably incomplete yet represents a technique that enables flexible modeling to solve challenging problems.[23242526]

THE MULTIFACETED NATURE OF DIGITAL PATHOLOGY NEEDS INCREASED REGULATORY CLARITY

Digital pathology has grown into a multimillion-dollar vendor landscape,[27] and the application of machine learning algorithms holds big promise for improving diagnostics in numerous ways.[282930] Despite this active and promising research, the Food and Drug Administration (FDA) has only recently authorized two digital pathology whole-slide imaging (WSI) systems for primary diagnosis.[39113132] Even with the authorization of two WSI systems and numerous use cases,[12131418333435363738] in the U.S., we see few hospitals changing their daily clinical operations to integrate WSI for primary diagnosis.[3940414243] Clinical laboratories face additional challenges when implementing high complexity and/or high-risk medical devices coupled with software solutions as laboratory-developed tests (LDTs).[444546] For example, even when using an FDA-authorized whole-slide imaging device, the approval or clearance does not eliminate the need for an individual laboratory to verify the performance of these systems for the specific intended diagnostic purpose. Specifically, Clinical Laboratory Improvement Amendments of 1988 or CLIA '88 in the US requires at least verification[47] and substantial adaptation to implement.[4849505152]

One value proposition for digital pathology is to take advantage of the digital nature of WSI and use artificial intelligence/machine learning (AI/ML) algorithms to support clinical decisions.[1153] In fact, several groups have proposed that AI/ML will unlock the full potential of digital pathology.[5354]

To examine the current regulatory guidance landscape related to digital pathology and AI, four authors (HDM, RH, EA, and JKL) performed a review of pertinent documents from the FDA. We noted the official release dates and assigned each document to one of five dimensions [Figure 1 and Supplemental Table 1]. By plotting these documents and dimensions over time, we show how the regulatory guidance landscape evolves. A novice in the field may look for one comprehensive guidance document for digital pathology and may be discouraged by the initial complexity; however, we hope that Figure 1 provides a reasonable starting point for learning the current regulatory guidance landscape. As we show Figure 1, arrows], the regulatory guidance landscape adapts over time as technologies and the associated regulatory science matures. One key element in the multistep process to improve the regulatory guidance landscape is critical scientific input from subject-matter experts.[34510111553] We strongly believe that “watching and waiting” will not help the case of digital pathology. Similarly, workarounds[848586878889] turn into long and winding roads that ultimately end at the FDA and within the FDA's regulatory framework.[83] The Alliance intends to organize subject-matter experts and provide scientific input.

Figure 1

Overview of selected FDA guidance documents. Four of the authors (HM, RH, EA, and JKL) performed a meta-review of selected FDA guidance documents relevant to the scope and aims of the Alliance. The figure shows grouping of these guidance documents across five dimensions over time. Please note: the numbers refer to the order of review during the meta-review process; Supplemental Table 1 provides the original release dates, the official FDA guidance title, and the issuer. AI/ML: Artificial intelligence/machine learning; CMS: Centers for Medicare and Medicaid Services; FDA: Food and Drug Administration; IMDRF: International Medical Device Regulators Forum; MDDT: Medical Device Development Tools; SaMD: Software as a Medical Device; QMS: Quality management system; WSI: Whole-slide imaging

Supplemental Table 1

Meta-review of pertinent Food and Drug Administration documents

Date	n*	Title	Issuer
January 11, 2002	16	General Principles of Software Validation https://www.fda.gov/media/73141/download	CDRH and OPEQ
January 14, 2005	10	Cybersecurity for Networked Medical Devices Containing Off-the-Shelf (OTS) Software https://www.fda.gov/media/72154/download	CDRH and OPEQ
August 17, 2011	1	Advancing Regulatory Science at FDA https://www.fda.gov/media/81109/download	FDA
July 02, 2012	12	Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data - Premarket Notification [510(k)] Submissions https://www.fda.gov/media/77635/download	CDRH, OSEL, and OPEQ
July 02, 2012	13	Clinical Performance Assessment: Considerations for Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data - Premarket Approval (PMA) and Premarket Notification [510(k)] Submissions https://www.fda.gov/media/77642/download	CDRH, OSEL, and OPEQ
December 09, 2013	17	Software as a Medical Device (SaMD): Key Definitions http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-131209-samd-key-definitions-140901.pdf	IMDRF and SaMD WG
September 18, 2014	18	Software as a Medical Device: Possible Framework for Risk Categorization and Corresponding Considerations http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-140918-samd-framework-risk-categorization-141013.pdf	IMDRF and SaMD WG
February 09, 2015	27a	Medical Device Data Systems, Medical Image Storage Devices, and Medical Image Communications Devices https://www.fda.gov/media/88572/download	CDRH and CBER
October 02, 2015	19	Software as a Medical Device (SaMD): Application of Quality Management System http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-151002-samd-qms.pdf	IMDRF and SaMD WG
April 20, 2016	6	Technical Performance Assessment of Digital Pathology Whole Slide Imaging Devices https://www.fda.gov/media/90791/download	CDRH, OPEQ, OHT7, and DMGP
August 24, 2016	2	Patient Preference Information - Voluntary Submission, Review in Premarket Approval Applications, Humanitarian Device Exemption Applications, and De Novo Requests, and Inclusion in Decision Summaries and Device Labeling https://www.fda.gov/media/92593/download	CDRH and OCD
October 24, 2016	3	Parallel Review with Centers for Medicare and Medicaid Services (CMS) https://www.federalregister.gov/documents/2016/10/24/2016-25659/program-for-parallel-review-of-medical-devices	FDA and CMS
August 10, 2017	4	Qualification of Medical Device Development Tools https://www.fda.gov/media/87134/download	CDRH
August 31, 2017	7	Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices https://www.fda.gov/media/99447/download	CDRH and OPEQ
September 06, 2017	14	Design Considerations and Premarket Submission Recommendations for Interoperable Medical Devices https://www.fda.gov/media/95636/download	CDRH, OSPTI, DDH,
September 21, 2017	20	Software as a Medical Device (SaMD): Clinical Evaluation http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-170921-samd-n41-clinical-evaluation_1.pdf	IMDRF, and SaMD WG
October 25, 2017	8	Deciding When to Submit a 510(k) for a Change to an Existing Device https://www.fda.gov/media/99812/download	CDRH and OPEQ
October 25, 2017	21	Deciding When to Submit a 510(k) for a Software Change to an Existing Device https://www.fda.gov/media/99785/download	CDRH and OPEQ
December 08, 2017	22	Software as a Medical Device (SAMD): Clinical Evaluation https://www.fda.gov/media/100714/download	CDRH, OSPTI, and DDH
October 18, 2018	11	Content of Premarket Submissions for Management of Cybersecurity in Medical Devices https://www.fda.gov/media/119933/download	CDRH and OCD
January 08, 2019	23	Developing a Software Precertification Program, A Working Model (v1.0 January 2019) https://www.fda.gov/media/119722/download	CDRH, OSPTI, and DDH
April 02, 2019	24a	Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) - Discussion Paper and Request for Feedback https://www.fda.gov/files/medical%20devices/published/US-FDA-Artificial-Intelligence-and-Machine-Learning-Discussion-Paper.pdf	CDRH, OSPTI, and DDH
April 19, 2019	9	Technical Performance Assessment of Quantitative Imaging in Device Premarket Submissions https://www.fda.gov/media/123271/download	CDRH and OPEQ
May 07, 2019	5	Requests for Feedback and Meetings for Medical Device Submission: The Q-Submission Program https://www.fda.gov/media/114034/download	CDRH, OPEQ, ORP, and DRP1
September 27, 2019	25	Off-The-Shelf Software Use in Medical Devices https://www.fda.gov/media/71794/download	CDRH, OSPTI, and DDH
September 27, 2019	15	Clinical Decision Support Software https://www.fda.gov/media/109618/download	CDRH, OSPTI, and DDH
September 27, 2019	26	Changes to Existing Medical Software Policies Resulting from Section 3060 of the 21st Century Cures Act https://www.fda.gov/media/109622/download	CDRH and CBER
February 09, 2019	27b	Medical Device Data Systems, Medical Image Storage Devices, and Medical Image Communications Devices https://www.fda.gov/media/88572/download	CDRH and CBER
January 28, 2020	24b	Artificial Intelligence and Machine Learning in Software as a Medical Device - update to: Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) - Discussion Paper and Request for Feedback https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device	CDRH and CBER
April 24, 2020	28	Enforcement Policy for Remote Digital Pathology Devices During the Coronavirus Disease 2019 (COVID-19) Public Health Emergency https://www.fda.gov/regulatory-information/search-fda-guidance-documents/enforcement-policy-remote-digital-pathology-devices-during-coronavirus-disease-2019-covid-19-public	CDRH and OPEQ

No* refers to numbering in main Figure 1; a,bRefers to updated guidance documents. CBER: Center for Biologics Evaluation and Research; CDRH: Center for Devices and Radiological Health; CMS: Centers for Medicare and Medicaid Services; DDH: Division of Digital Health; DMGP: Division of Molecular Genetics and Pathology; DRP1: Division of Submission Support; FDA: Food and Drug Administration; IMDRF: International Medical Device Regulators Forum; OCD: Office of the Center Director; OHT7: Office of Health Technology 7; OPEQ: Office of Product Evaluation and Quality; ORP: Office of Regulatory Programs; OSEL: Office of Science and Engineering Laboratories; OSPTI: Office of Strategic Partnerships and Technology Innovation; SaMD WG: Software as a Medical Device Working Group

Simply put, the practical dilemma in digital pathology is that developers are challenged to create an FDA submission following the evolving and complex regulatory guidance landscape, and the adoption of WSI by pathologists is slowed because they cannot realize the full potential and utility of digital pathology and AI/ML without full clinical integration. The field of digital pathology is looking for broader guidance, practical advice, and streamlined regulatory pathways to help navigate this uncharted and exciting territory.

REGULATORY SCIENCE, THE PRECOMPETITIVE SPACE, AND REAL-WORLD EVIDENCE

FDA clearance of a medical device offers a vendor market access. Once introduced, market forces tend not to encourage the vendor to make the device or its subsystems interoperable.[55565758596061] We like to emphasize that routine diagnostic pathology is highly modularized and the practice does not lend itself easily to nonmodular, locked down solutions.[3910112750515462] The Alliance believes that it can promote interoperability and innovation by launching initiatives and creating deliverables (data, standards, tools, and methods) in the precompetitive space. Organizing industry to work collaboratively in the precompetitive space will eliminate unnecessary or duplicative (proprietary) efforts and thereby save all parties' time, money, and resources when pursuing device authorizations.[63] The Alliance initiatives and deliverables will speed clinical integration and carry mutual benefit to all stakeholders, including regulators, clinicians, manufacturers, and most importantly, patients.

Real-world evidence (RWE) comes from the competitive, postmarket space. RWE can identify trends in adverse events, summarize where resources are being spent, and track the impact of a new diagnostic device or therapy in terms of patient outcomes. RWE can support clinical practice guidelines and decisions about reimbursement and policy. Furthermore, RWE can inform regulatory decision making, as effectively demonstrated by the Medical Device Innovation Consortium,[6465] the National Evaluation System for health Technology Coordinating Center,[66] the Patient-Centered Outcomes Research Institute,[6768] Friends of Cancer Research,[6970] and others.[356971727374]

FROM KEY MISSION ELEMENTS TO A DELIVERY PROCESS

Accomplishing mutual benefit to multiple stakeholders is a daunting value proposition that requires a unique regulatory science approach and stakeholder involvement for selection and prioritization of deliverables. The approach of the Alliance [Figure 2a] is to deliver tools by harnessing existing, precompetitive FDA programs and use the gained experience to inform effective regulation. The approach thereby aims to streamline precompetitive and eventually competitive submissions that enable faster time to market to improve patient care. Regulatory science deliverables, including tools and the experience from precompetitive submissions, will be shared, and when one integrated solution has been enabled, the Alliance can dissolve [Figure 2a]. The key mission elements of the Alliance are summarized in Table 1.[75]

Figure 2

Concept, process, role, and proposed benefits of the Alliance. (a) The approach of the Alliance is to deliver tools via precompetitive FDA programs and use the gained experience to support effective FDA review. The concept also includes a predetermined exit strategy (i.e., one fully integrated solution for digital pathology). (b) The process of moving Alliance projects forward is essentially a two-step, multidisciplinary peer review by subject-matter experts. First, projects are reviewed, and after a multidisciplinary selection process that emphasizes the patient perspective and relevance for patient care, the steering committee (jointly with relevant partners) attempts to allocate resources. (c) Role and proposed benefits of the Alliance exemplified using the high-throughput truthing project for tumor-infiltrating lymphocytes as a biomarker in breast cancer. AMCs: Academic medical centers; MDDT: Medical Device Development Tools (precompetitive FDA submission program); Mock: mock submission program (precompetitive FDA submission program); OIR: Office of In vitro Diagnostics and Radiological Health; OPEQ: Office of Product Evaluation and Quality; OSEL: Office of Science and Engineering Laboratories; FDA: Food and Drug Administration

Table 1

Key mission elements of the Alliance

Definition	Explanation
Aim	To move the field of digital pathology, AI/ML and computational pathology, forward
Focus	Key emphasis on regulatory science (“how to get to the next step”); inform regulatory guidance and decision-making; explore new regulatory programs
Deliverables	The Alliance focuses on concrete practical deliverables, such as projects or practical guidelines, that can be used to inform and improve the regulatory guidance landscape (regulatory science)
Collaboration	We seek participation from all stakeholders
Participatory	We aim to sustain and expand the existing collaborative infrastructure of the Alliance
Market strategy	Focus on the precompetitive space with an emphasis on clinical deliverables towards financial sustainability for all stakeholders
Patient perspective	Make the patient perspective and clinical relevance an integral part of the deliverables
Temporary	Exit strategy: Once an end-to-end solution has been clinically integrated, the Alliance ends
Free	No membership fees

AI: Artificial intelligence; ML: Machine learning

To align stakeholder interests, initiatives and deliverables need to be prioritized and prioritization requires a process. We conceptualized an approach that is composed of synergistic review, project components, and resource allocation [Figure 2b]. The process starts with synergizing various stakeholder interests into concise individual projects. An Alliance project may consist of a clinically relevant intended use case, a data set (e.g., pixel and metadata), and an applicable regulatory science pathway [e.g., Figure 2b, triangle]. The Alliance membership, composed of subject-matter experts from various domains, will have the opportunity to review, contribute, and potentially modify these projects through free and voluntary feedback to the project owner. Over time, individual effort and maturation of ideas will result in optimized projects (“big ideas”). To help realize the proposed deliverables and/or allocate additional resources, we established the Alliance Steering Committee, a flexible organizational structure, and a code of conduct [Supplemental Table 2].

Supplemental Table 2

The Alliance Steering Committee and Membership by Sector

Founders	Affiliation	Sector
Jochen K. Lennerz, MD, PhD	Medical Director, center for Integrated Diagnostics, Massachusetts General Hospital/Harvard Medical School	Academia
Esther Abels, MSc	Vice President of Regulatory Affairs, Clinical Affairs and Strategic Business Development, PathAI	Industry
Brandon D. Gallas, PhD	Mathematician, FDA/CDRH/OSEL/Division of Imaging, Diagnostics, and Software Reliability	Government
Steering Committee	Affiliation	Sector
Alain C. Borczuk, MD	Professor of Pathology and Laboratory Medicine, Weill Cornell Medicine	Academia
Amanda Lowe	Managing Director of Americas, Visiopharm Corporation	Industry
Ashish Sharma, PhD	Associate Professor, Department of Biomedical Informatics, Emory University School of Medicine	Academia
Clive R. Taylor, MD, DPhil	Professor Emeritus, University Southern California	Academia
David A. Clunie, MBBS	Owner, PixelMed Publishing, LLC	Industry
Frank R. Dookie, MBA	CEO and President, Sales Management Operations Consulting, Inc.; Strategic Consultant, JAV Advisors Corp.	Industry
Gina Giannini, MS	Manager of Regulatory Affairs, Digital Pathology, Roche Tissue Diagnostics	Industry
Hetal D. Marble, PhD	Program Manage of Biomarker Development and CDx, Left for Integrated Diagnostics, Massachusetts General Hospital/Harvard Medical School	Academia
Jithesh Veetil, PhD	Program Director of Data Science and Technology, Medical Device Innovation Consortium	Nonprofit
Joachim H. Schmid, PhD	Vice President of Research and Development, Digital Pathology, Roche Tissue Diagnostics	Industry
Jon Hunt, PhD	Vice President of Clinical Science and Technology, Medical Device Innovation Consortium	Nonprofit
Keyvan Farahani, PhD	Program Director, National Cancer Institute	Government
Lakshman Ramamurthy, PhD	Head of Regulatory Affairs, Precision Medicine and Digital Health, GlaxoSmithKline Inc.	Industry
Laura Lasiter, PhD	Director of Health Policy, Friends Of Cancer Research	Nonprofit
Mark D. Zarella, PhD	Deputy Director of Informatics, Department of Pathology, Johns Hopkins University	Academia
Markus D. Herrmann, MD, PhD	Director of Computational Pathology, Massachusetts General Hospital/Harvard Medical School	Academia
Matthew G. Hanna, MD	Director of Digital Pathology Informatics, Assistant Attending Pathologist, Memorial Sloan Kettering Cancer Left	Academia
Matthew O. Leavitt, MD	Chairman, Founder, and Chief Medical Officer, LUMEA	Industry
Mike Bonham, MD, PhD	Chief Medical Officer, Proscia Inc.	Industry
Michael Isaacs	Director of Clinical Informatics and Business Development, Washington University School of Medicine	Academia
Pamela W. Goldberg, MBA	President and Chief Executive Officer, Medical Device Innovation Consortium	Nonprofit
Richard Huang, MD	Clinical Informatics Fellow, Massachusetts General Hospital/Harvard Medical School	Academia
S. Joseph Sirintrapun, MD	Director of Pathology Informatics, Associate Attending Pathologist, Memorial Sloan Kettering Cancer Left	Academia
Sarah N. Dudgeon, MPH	Research Fellow, FDA/CDRH/OSEL/Division of Imaging, Diagnostics, and Software Reliability	Government
Scott M. Blakely	Business Development Manager of Whole Slide Imaging and Digital Pathology, Hamamatsu Corporation USA	Industry
Steven Barbee	President, JAV Advisors Corp	Industry
Overall Membership By Sector (Total: 320)	Academia: 102 Members Industry: 128 Members
	Government: 76 Members Nonprofit: 14 Members

CDRH: Center for Devices and Radiological Health; OSEL: Office of Science and Engineering Laboratories; FDA: Food and Drug Administration

An example project is illustrated in Figure 2c. A subset of members in the Alliance are studying the relevance of tumor-infiltrating lymphocytes (TILs) as a prognostic and predictive biomarker.[7677] The interest in this clinical use case led to a collaborative project that includes members from the FDA, academic medical centers (AMCs), and industry. The project, referred to as the high-throughput truthing (HTT) project, aims to demonstrate the collection and use of pathologist annotations for the purpose of evaluating AI/ML algorithms and other digital pathology initiatives. The project also aims to qualify the glass slides, whole-slide images, and pathologist annotations for evaluating AI/ML algorithms through the precompetitive FDA's Medical Device Development Tools (MDDT) program.[78] If qualified, the “ground-truth” materials can serve as a publicly available, standardized evaluation “tool” for algorithm evaluation that can be used in submissions to the FDA.

In relation to the Alliance, the HTT project was submitted to the Alliance and discussed in November 2019. The Alliance can contribute in multiple ways to accelerate the realization of this and similar projects. First, the Alliance confirmed that the aims of the project could benefit many stakeholders.

The discussions provided useful feedback from subject-matter experts regarding the clinical use case, sourcing slides from multiple sites, agreements for sharing materials within the project, and issues related to sharing materials publicly. The discussions also identified future work that could build on the lessons, methods, infrastructure, and relationships created while pursuing the current aims. Important future work identified in the discussions included scaling the effort to address generalizability across sites and generalizability across use cases.

The Alliance has since provided help with the project [Figure 2b, triangle 01, relevant intended use case; Figure 2c, 01] by disseminating the project needs. This networking through the Alliance has yielded volunteers for sourcing and scanning slides, pathologists to annotate slides and images, and opportunities to collect data. Connections have been created that are expected to help in the development of the statistical analyses and the future hosting of slides, images, and annotations. Currently, the project is developing the strategy and materials for the FDA's MDDT program [Figure 2b, triangle, MDDT; Figure 2c, 03]. The development is a learning experience for all involved, with contributions from project and Alliance subject-matter and regulatory affairs experts. The learning experience is expected to continue through official interactions with the FDA related to the MDDT submission. Thus, aside from helping to create the ground-truth data set, the Alliance aims to understand regulatory issues and processes for future streamlining of other projects and submissions. As demonstrated here, a qualified data set may result in time-savings when preparing submissions, generating additional tools, and streamlining regulatory review, resulting in faster time to market and improved patient care.

WHO IS THE ALLIANCE?

The Alliance is composed of a diverse and interdisciplinary group of stakeholders who contribute to various aspects of diagnostic pathology, from tissue acquisition to reporting and data analytics. When deconstructing the clinical digital pathology and AI/ML pipeline into its component parts, numerous workflow steps have to function in unison [Figure 3a]. Aside from the modular nature and operational complexity, these components emphasize the importance of involving various stakeholders with each module. Given the novelty of pursuing a collaborative regulatory science effort to solve the challenge of clinical adoption of digital pathology, we noted a lack of concrete data on interested stakeholders and their priorities. In September 2019, we conducted an internal survey [n = 42; Supplemental Table 3]. At that time, the survey respondents stated that the top 3 deliverables/workflow steps to focus on should be the DICOM standard, AI/ML test validation, and pixel and metadata capture [Figure 3b]. By self-reported primary affiliation, the Alliance encompasses representation from academia (32%), industry (50%), government regulators and nongovernment organizations (12%), and patient advocacy groups (6%) [Figure 3c].

Figure 3

Workflow steps and Alliance survey results. (a) Digital pathology workflows include preanalytical, retrieval, scan (image acquisition), clinical data, metadata, machine learning algorithm development, clinical integration, clinical utility, and financial sustainability considerations; all dependent on the specific use case/application. These workflow steps correspond to the axis labels in b. (b) The Alliance conducted a survey among the members in September 2019. Bar graphs show the workflow steps that survey respondents felt the Alliance should focus on. These steps are reflected in a workflow diagram in a. (c) Survey results from September 2019. DICOM: Digital Imaging and Communications in Medicine (here referring to an interoperable file format for digital pathology); EHR: Electronic health record; H&E: Hematoxylin and eosin stain; IHC: Immunohistochemistry; LIMS: Laboratory information management system; MDIC: Medical Device Innovation Consortium

Supplemental Table 3

Survey questions and answer choices sent to the Alliance for Digital Pathology membership

Question number	Question	Answer choices
1	How long have you been involved with digital pathology?	<1 year
		1-5 years
		5-10 years
		>10 years
2	How many papers have you published about digital pathology?	Open ended
3	What sector do you represent?	Academia
		Industry
		Government
		Nongovernmental organization
		Other
4	Are you familiar with the MDIC?	Yes
		No
5	Should patient advocacy groups be a part of the Alliance?	Yes
		No
6	FDA regulatory oversight of digital pathology is:	Too simple
		Adequate
		Too complex
7	Should the Alliance focus on slide generation as a preanalytical factor?	Yes
		No
8	Should the Alliance focus on metadata capture?	Yes
		No
9	Which workflow steps should the Alliance focus on?	Archive retrieval
		Preanalytics
		Slide scan
		Pixel data
		Electronic health record
		Laboratory inventory management system
		Metadata
		DICOM
		Storage
		Computation
		Modeling
		Test validation
		Deployment
		Utilization

DICOM: Digital Imaging and Communications in Medicine (here referring to an interoperable file format for digital pathology); FDA: Food and Drug Administration; MDIC: Medical Device Innovation Consortium

MEETINGS, GROWTH, AND WORKING GROUPS

Since its inception in May 2019, the Alliance hosted numerous teleconferences, web meetings, and three, in-person, national meetings [Figure 4a]. Over this period (May 2019–January 2020), the Alliance membership grew from an initial n = 37 (July 2019) to n = 322 individuals [May 2020; Figure 4a]. Each of these in-person meetings solicited collaborative input from stakeholders toward execution of concrete regulatory science deliverables. Figure 4a also includes the number of participants and frequency of steering committee web meetings. By July 2019, it became clear that various stakeholders worked on or had interest in distinct topics that the Alliance subsequently organized into 8 working groups by autumn 2019 [Figure 4b]. These group topics are intended to align stakeholders with subject-matter expertise and interest. Clearly, some functional requirements are relevant for multiple groups. However, we hope to minimize such redundancies by providing clear documentation of projects through appropriate project management and frequent content updates. The names of the founding and current working group leaders are provided in Figure 4b. One example of a regulatory science deliverable is also provided per group [Figure 4b]. For further updates or details on the various topics, please visit the Alliance website[8] or to become a member and get involved.

Figure 4

Roadmap and working groups. (a) Roadmap of in-person events (status May 2020). In addition to the date, the roadmap shows hosting organization, key developments, and location of the meetings. The graph shows the membership number over time along with the number and frequency of the steering committee meetings as well as the high-throughput truthing working group. (b) The Alliance proposed to tackle regulatory science deliverables in digital pathology by splitting up the topic into eight distinct working groups. Each workgroup is provided with the steering committee member (s) and at least one key regulatory science deliverable. The steering committee is also responsible for minimizing redundancy between the workgroups. AI: Artificial intelligence; DPA: Digital Pathology Association; FDA: Food and Drug Administration; HTT: High-throughput truthing (an independent workgroup); MDIC: Medical Device Innovation Consortium; ML: Machine learning; USCAP: USCAP stands for United States and Canadian Academy of Pathology

THE A FACILITATES REGULATORY SUBMISSIONS

As a first key regulatory science deliverable, in late 2019, members of the Alliance submitted an MDDT proposal to the FDA for review (HTT project described above). The experience gained through this submission will create a starting point and testing ground for the proposed approach of the Alliance. In contrast to the largely confidential submission owned by the submitting entity (typically represented through a consulting firm and/or a regulatory affairs division), gaining and sharing the submission experience may inform subsequent submissions, and Alliance members can draw from the experience of these submissions. This particular concept is new to digital pathology. Similarly, we consider several precompetitive submission programs by the FDA[7879] a paradigm shift that enables different ways to engage with regulatory entities. Importantly, the Alliance intends to create a repository of submission documents as a resource to bolster subsequent submissions with the collective experience of previous submitters. We propose that the field, and in particular patients,[80] will ultimately benefit from sharing the experiences of Alliance members who have submitted to regulatory agencies.

CONCLUSION

In the current environment of sparse and dispersed regulatory guidance for digital pathology and AI/ML, with siloed pursuits by diverse stakeholders, the Alliance saw an opportunity to establish an important missing element: a precompetitive regulatory science collaboration. We believe that for patients to benefit from highly complex new technologies, benefit and risk assessments are essential.[8182] The Alliance helps tackle this daunting task (i. e., benefit and risk assessment for digital pathology and AI/ML) through regulatory sciences with the hope of successful clinical integration and improved patient care. That said, there are numerous issues that we need to address. For example, we want to investigate and develop protocols and definitions for continuous performance assessments of continuously learning ML algorithms. Similarly, approaching financial sustainability will require clear demonstration of clinical utility. However, the fact that numerous unanswered questions persist represents an opportunity for other agencies, regulatory entities, professional groups, and collaborative movements (like the Alliance) to step up and drive developments toward comprehensive risk and safety assessments. It is important to emphasize the crucial importance of funding for regulatory and implementation science projects, in particular those that aim to inform technically appropriate and efficient science-based regulatory decision-making processes. Such funding is needed to advance cutting-edge innovations into clinical practice. In summary, the Alliance aims to advance the field of digital pathology and we hope that synergistic efforts between various stakeholders and regulatory scientists will ultimately speed the improvement of patient care. This begs the question: Who, if not us?

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.