Digital Pathology and Artificial Intelligence Applications in Pathology.

Digital pathology is revolutionizing pathology. The introduction of digital pathology made it possible to comprehensively change the pathology diagnosis workflow, apply and develop pathological artificial intelligence (AI) models, generate pathological big data, and perform telepathology. AI algorithms, including machine learning and deep learning, are used for the detection, segmentation, registration, processing, and classification of digitized pathological images. Pathological AI algorithms can be helpfully utilized for diagnostic screening, morphometric analysis of biomarkers, the discovery of new meanings of prognosis and therapeutic response in pathological images, and improvement of diagnostic efficiency. In order to develop a successful pathological AI model, it is necessary to consider the selection of a suitable type of image for a subject, utilization of big data repositories, the setting of an effective annotation strategy, image standardization, and color normalization. This review will elaborate on the advantages and perspectives of digital pathology, AI-based approaches, the applications in pathology, and considerations and challenges in the development of pathological AI models.

INTRODUCTION

Nowadays, pathology has undergone dramatic transformations due to digitization and the introduction of new computing technologies, in line with other medical fields. Digital pathology is ending the era of optical microscopy, which has lasted over 100 years, and is ushering in a new era in which digitized whole slide images (WSIs) are utilized for pathology diagnoses. The advent of digital pathology systems has facilitated expedited clinical pathological workflows, a new model of education, and the development of practically actionable image analysis tools and investigation techniques [1].

Pathological data analysis using artificial intelligence (AI) is considered an essential approach in the age of precision oncology [2]. In cancer, the complexity of genomic and proteomic alterations and cellular characteristics and interactions between tumor cells and their microenvironment can influence the progression of the disease and affect therapeutic response. Pathological assessment of these alterations requires the simultaneous interrogation of multiple features in a highly sensitive and accurate approach. In addition, well-validated and reproducible assessment tools are needed for clinical decision-making. Although the extraction of multiple visual and morphometric AI-based features from the pathological WSI is still limited due to slide quality problems and tumor heterogeneity, it can help to overcome the limitations of the pathologist’s subjective visual and semi-quantitative evaluation and comprehensively analyze complex tissue architectures.

This review will elaborate on the advantages and perspectives of digital pathology, AI-based approaches, the applications in pathology, and considerations and challenges in the development of pathological AI models.

PATHOLOGY IN THE DIGITAL ERA

Digital pathology was initially referred to as the process of digitizing WSIs using advanced slide scanning technology, and now is a generic term that includes AI-based approaches for detection, segmentation, diagnosis, and analysis of digitalized images [2]. Digital pathology drives changes across pathology, including pathological diagnostic workflows, telepathology, the adoption of various AI-based diagnostic algorithms in routine diagnosis, pathology education, and big data generation and management [3]. A digital pathology system allows a pathologist to observe digitized images on a computer monitor rather than examining tissue glass slides using a microscope in the diagnostic process (Fig. 1) [4]. In detail, a patient tissue specimens obtained through biopsy or excision is fixed in formalin to prepare a paraffin tissue block. The paraffin block is sliced into thin sections off 3–4 µm thickness, and the thin section are placed onto a glass slide. These processes are the same as the existing slide preparing method. In a digital pathology system, the tissue glass slides are digitized using a slide scanner after the tissue slide preparation is completed. The slide scanner, equipped with a lens with a magnification of ×20 or ×40, scans in a line or patch method, stitches scanned images together, and then makes WSIs. As with examining tissue slides under a microscope, pathologists can observe all areas of the WSI at various magnifications. A WSI is a gigapixel, ultrahigh-resolution image and is approximately 1 GB in size when a 1.5 cm×1.5 cm tissue section is scanned. A digital pathology system should be equipped with a slide scanner and an image viewer capable of viewing WSI. To operate the system properly in a hospital, very fast communication network, advanced slide compression technology, the latest streaming technology, sufficient storage server, and full integration with the other medical systems such as electronic medical systems (EMR) or picture archiving and communication systems for radiology are essential.

Fig. 1

Workflow of digital pathology.

Recently, primary diagnosis in digital pathology systems has been approved and put into practical use in Korea as well as the United States, the European Union, and Japan [3]. Many studies have compared the diagnostic accuracy of primary diagnosis in digital pathology systems with conventional microscopic diagnostics [567]. Most of the validation studies have demonstrated that routine diagnosis with a digital pathology system is not inferior to conventional diagnostics, and a recent meta-analysis study demonstrated equivalent performance of digital pathology in comparison with light microscopy for routine diagnosis [8]. Mukhopadhyay et al. [5] evaluated the diagnostic performance of digitalized images compared to microscopic images on specimens from 1,992 patients with different tumor types diagnosed by 16 surgical pathologists. Routine diagnostic performance with digitalized WSIs was not inferior to that achieved with light microscopy-based approaches (with a major discordance rate from the reference standard of 4.9% for WSI and 4.6% for microscopy). Through internet networks, digital pathology enables real-time mutual consultation about the case with many pathological experts at other institutions and in different countries [9]. In addition, pathological diagnosis can be made without time and space limitations in the digital pathology system. Pathological diagnosis requires a detailed review of all medical data and past pathological data of the patient, and digital pathology enables the integrated analysis of data-based medical data. One of the important advantages of digital pathology is that pathological AI-based models can be easily used during the diagnostic process and collection and management of pathologic big data [10]. Pathological big data can be used for learning various AI diagnostic models needed for pathological diagnosis and can be used as various educational materials [11].

Digital pathology is still evolving and has many challenges to overcome. Currently, digital pathology is being introduced in several hospitals around the world, but there are only a few so far. There are relatively a few reports based on digital pathology experience to completely replace light microscopy for routine diagnostic purposes [1213]. The digital pathology system is supplied by several commercial companies, but the technical compatibility between them is an issue. The initial investment for having a digital pathology system is high because a slide scanner, viewer, and storage server are required. A UK pathology laboratory focus group published that they identified benefits (e.g., collaboration, teaching, and cost-savings) of the implementation of digital pathology [14]. It is important to integrate digital pathology systems to make smooth use of digital pathology systems. This requires a lot of time, manpower, cost, and collaboration with existing system operators. In addition, the level of operation of the digital pathology system may be limited depending on the level of computerization of the pathology order communication system or EMR system [3].

PATHOLOGICAL AI-BASED APPROACHES AND APPLICATIONS

As qualified pathological big data have been accumulated due to the introduction of digital pathology to medical institutions with the development of scanning technology, pathologists can explore various unresolved problems in pathology by collaborating with technical experts, including data scientists, computational engineers, and image physicists [15]. Computer vision tasks include methods for acquiring, processing, analyzing, and understanding digital images, and extraction of high-dimensional data from the real world to produce numerical or symbolic information [16]. AI and machine learning (ML) technologies are ideal for examining complex data sets in an exhaustive manner and have been deployed to extract features, patterns, and information from histopathological images [4]. Deep learning (DL) is a concept belonging to ML and consists of computer algorithms that learn by themselves as they perform tasks and has strengths in extracting features from medical images, speech recognition, and natural language processing [17]. In pathology, DL provides an opportunity to interrogate pathological images at a deeper level, and DL algorithms are used for detection, segmentation, registration, processing, and classification of digitized WSI [18].

Pathological AI models can be used for various purposes, including automatic detection, localization, quantification of histological parameters and structural changes, and disease diagnosis [18]. These models support pathologists in diagnosing disease by analyzing the characteristics of tumor cells, structural changes of lesions, and expression patterns of biomarkers. In addition to histopathological image analysis, AI models can comprehensively investigate various medical metadata (e.g., clinical, genetic, radiological, laboratory data, etc.) coupled with histopathological image data [4]. Through “omics” analysis, AI models may predict disease aggressiveness, patient outcomes, and therapeutic response. These new capabilities of AI models can extract useful information in ways that were not possible in conventional microscopic diagnostics in the context of direct human cognitive and visual assessment.

The increase in quantitative and qualitative demands for pathological diagnosis for the realization of precision medicine is further promoting the application of AI models in pathological diagnosis. While applying DL approaches to pathology in the early days, AI models that mimic pathologists were mainly developed with the goal of reducing the burden of pathologists or increasing the accuracy of pathological diagnosis [19]. For example, prostate cancer detection and localization models for needle biopsy specimen have been approved for clinical practice [20]. Using DL algorithm, biomarker morphometric analysis models to determine patients eligible for targeted therapy, for Ki67, c-erbB2, estrogen receptor, and progesterone receptor expressions, have been developed by the Asan Medical Center in collaboration with Vuno Inc. and approved by the Ministry of Food and Drug Safety in Korea. Prediction models of microsatellite instability in colon cancer and non-small cell lung cancer that have EGFR or KRAS gene mutations are also being actively developed and validated [2122]. These predictive models are expected to be used for screening purposes in the pathological diagnosis process, and they aim to minimize unnecessary molecular genetic testing and reduce the waste of time and medical expenses. With the accumulation of pathological big data, AI models for rare diseases or differential diagnosis, which are one of the most unmet needs, have recently been proposed [23].

IMPORTANCE OF PATHOLOGICAL AI MODELS IN NEUROONCOLOGY

Pathological diagnosis is a confirmative diagnosis of a disease, and pathological diagnosis plays an absolutely important role in the determination of the treatment plan of patients in neurooncology. The recent trend in WHO central nervous system (CNS) tumors classification has highly emphasized the importance of genetic mutations in addition to the conventional morphological and immunophenotypical characteristics [24]. Many types of targeted therapeutic and immunotherapeutic agents are being developed for CNS tumors. In line with the development of therapeutic agents, various biomarkers are being excavated to select the most effective therapeutic agents and to predict and evaluate the responsiveness of treatment [25]. For these reasons, currently, the evaluation of morphological, immunophenotypical, and genetic biomarkers associated with an accurate diagnosis and precision medicine using tumor tissue in pathological diagnosis is essential. To evaluate the expression of biomarkers and the aberration of genes, many kinds of ancillary tests, such as immunohistochemical staining, fluorescent in situ hybridization analysis, Sanger’s sequencing, next-generation sequencing, and PCR-based DNA methylation analysis, are required. To perform these tests, it takes a lot of money and a long time, and a considerable amount of tissue is consumed. In order to manage the patient at the appropriate time, it may be very difficult to carry out all the tests. Studies have reported that AI-based algorithms can predict the expression of biomarkers and genetic mutations of tumors by evaluating only the morphological characteristics of hematoxylin and eosin (H&E)-stained slides that are basically produced for pathological diagnosis [2122262728]. Studies have also shown that novel biomarkers can be discovered by utilizing AI models in various types of cancers [293031]. Therefore, for the patient’s appropriate care, it is important to use AI in neurooncology to screen for pathological classification or to predict the characteristics of cancer.

PATHOLOGICAL AI MODELS IN NEUROONCOLOGY

Recently, pathological AI models are being actively developed in the field of neurooncology, and they mainly target gliomas. In detail, pathological AI models being developed focus on the classification of gliomas, the mutation status of cancer-related genes, predicting patients’ prognosis, discovering new histological implicating features of tumors affecting cancer behaviors, and analyzing tumor microenvironment [2728323334]. Most of the research uses H&E-stained slides, immunohistochemical stained slides, and, in some cases, genetic analysis information. Jin et al. [27] developed a convolutional neural network model for glioma classification. This model classified five major histological subtypes of glioma, which were oligodendroglioma, anaplastic oligodendroglioma, astrocytoma, anaplastic astrocytoma, glioblastoma, and nontumor, with high patient-level accuracy of 87.5% using H&E-stained slide, IDH mutation, and 1p/19q deletion. Pathological AI models were proposed for glioma grading and for estimating IDH mutation status in low-grade glioma and for predicting the patient’s prognosis on H&E-stained slides [3233]. Zadeh Shirazi et al. [34] defined the tumor cell-perivascular niche and spatial associated genetic signatures using a semantic segmentation model and demonstrated that they were associated with poor survival in glioblastoma.

DEVELOPMENTAL OF AI MODEL IN PATHOLOGY AND CHALLENGES

Pathological AI algorithms can be helpfully utilized for diagnostic screening, morphometric analysis of biomarkers, the discovery of new meanings of prognosis and therapeutic response in pathological images, and improvement of diagnostic efficiency. In the case of determining a subject for a pathological AI model, it is necessary to collect relevant data. Histopathological images are basically stained with H&E and most of all surgical pathology diagnoses are made based on H&E-stained slides. Now, the majority of pathological AI models have been developed using H&E-stained WSI. Most of the pathological data is image data, but various types of pathological images are created depending on the staining methods and the examination methods. Pathology produces various types of images to observe macroscopic, histological, and cellular-level morphological changes, increases or decreases in copy number, translocation, and mutation of DNA, protein expression patterns, mRNA expression patterns, microorganisms, and ultrastructural morphological changes (Fig. 2). Since WSI shows its own histological characteristics according to the applied staining methods and is used for different purposes for pathological diagnosis, a pathological AI model can be developed using one type of image data but can also be developed using several different types of images depending on the subject. Therefore, it is important to determine and collect the kind of pathological image data that are best suited to the topic of the AI model to be developed.

Fig. 2

Various types of pathological images. A: Gross image. B: Hematoxylin and eosin (H&E) stain. C: 3,3'-Diaminobenzidine (DAB) staining immunohistochemistry. D: In situ hybridization. E: Fluorescent in situ hybridization. F: Periodic acid methenamine silver stain. G: Masson's trichrome stain. H: Transmission electron microscopic image. I: BIOMED-2. PCR assay for T-cell monoclonality image.

Because big data is essential for AI model development, data repositories are being actively built by medical organizations/associations and medical institutions. TCGA is one of the most representative data repositories in the field of cancer, and it contains clinical data and genetic data along with pathological WSIs, which have been widely used for learning and validation of pathological AI model development [263536]. There are currently several data repositories in the field of renal pathology, such as NEPTUNE, CureGN, and FOrNE [4]. Many medical institutions, led by developed countries, have been able to actively collect pathological big data by implementing digital pathology systems. Until recently, one of the biggest difficulties in developing pathological AI models was the lack of pathological big data, but it is expected that many of these difficulties will be resolved within the next few years. Moreover, as transfer learning and synthetic data-related technologies have recently started to be applied to pathology, it is expected that the problem of data shortage can be overcome in a shorter period [1037].

A pathological examination is performed very rarely compared to other medical examinations such as radiologic and laboratory examinations. Currently, pathological big data is being actively generated, but there is very little data available to develop a pathological AI algorithm that can handle the complex characteristics of pathological image data and subjects that are very difficult to solve but, at the same time, require accuracy of output. So, annotations are indispensable for pathological AI model development. Pathological image annotation is too difficult for anyone other than pathologists to perform. In addition, even when performed by a pathologist, it can be time-consuming, taking up to several hours per case, and in some cases, it must be performed by a highly specialized pathologist. Therefore, when developing a pathological AI algorithm, generating annotation data rather than collecting digitized images often becomes the bottleneck of development. In the case of using fully supervised learning to develop a pathological AI model, a strategic approach that can minimize annotation burden is required, and when a relatively large amount of data can be collected, it is necessary to consider weakly supervised learning methods. In the case of a cancer diagnosis model for prostate cancer needle biopsy diagnosis, multiple instance learning, one of the weakly supervised learning methods, was applied to develop an AI model with high accuracy through simple labeling without detailed annotation of images [38]. Active learning is a kind of semi-supervised learning in which a learning algorithm is allowed to interactively ask a user to provide new labels for samples [39]. Active learning suggests that a learning algorithm can perform better with less data than traditional methods if it can choose the data it wants to learn from. In pathology, active learning has been successfully applied to classifiers to support quality control in nucleus segmentation [40]. The main hypothesis in transfer learning is that if a previously well-trained algorithm is applied to a different but related problem, it can perform better than traditional methods with substantially less data for training, and transfer learning frameworks are also actively used in pathology. Annotations are available on platforms that can handle pathological images. Annotation of WSIs in various formats can be performed through open platforms such as Automated Slide Analysis Platform (ASAP) (https://computationalpathologygroup.github.io/ASAP/) or Quantitative Pathology & Bioimage Analysis (QuPath) (https://qupath.github.io/) in addition to commercial software. Of note, these platforms can perform image overlays or use open-source pathological image analysis tools.

WSIs are not standardized yet, unlike radiological data that are standardized as digital image and communication model (DICOM, www.dicomstandard.org) format. Slide scanner manufacturers use the most appropriate image format for WSIs, such as .tif, .svs, .ndpi, .mrxs, bif., and .isyntax, produced by their scanners. The lack of a standard image format is a major obstacle not only to the routine diagnosis of digital pathology but also to the development of pathological AI models. For routine diagnosis in digital pathology, each medical institution must be equipped with an image viewer that can operate all image formats or a digital pathology system that can convert to an image format suitable for their own viewer. Even when developing an AI model, it is necessary to be able to read the collected WSIs. A library that can read various image formats can be provided by open sources such as OpenSlide (www.openslide.org). Efforts to establish a pathological DICOM standard format have been made in recent years [41]. WG-26 is a US- and European-led pathology working group within DICOM. The scope of WG-26 is to support and develop the DICOM standard, allowing the pathology domain to process WSIs and macros to produce, store, and communicate. They try to open up those systems for interoperability using global standards to create workflows as a part of the overall healthcare process. WSIs show different image characteristics, even on H&E-stained slides, depending on the difference that occurs in the slide production process and the brand or model of a slide scanner. Differences in image characteristics affect the learning of AI algorithms and may cause performance degradation when the developed AI model is applied. Color normalization approaches have been made to compensate for these variations [42]. Stain-color deconvolution and template color matching have been used for color normalization of histopathological images. The stain-color deconvolution method considers prior knowledge of the reference staining vector for all dyes and splits an input RGB image into three staining channels, each representing the actual staining color [43]. Template color matching achieves color correction by choosing an appropriate source image and applying its characteristics to another image [44]. Generative adversarial networks are also used for histopathological images [45].

CONCLUSIONS

Digital pathology is currently leading a comprehensive shift in pathology. The implementation of digital pathology impacts pathological diagnostic workflows, telepathology, the adoption of various AI-based diagnostic algorithms in routine diagnosis, pathology education, and big data generation and management. Pathological AI algorithms can be helpfully utilized for diagnostic screening, morphometric analysis of biomarkers, the discovery of new meanings of prognosis and therapeutic response in pathological images, and improvement of diagnostic efficiency. In order to develop a successful pathological AI model, it is necessary to consider the selection of a suitable type of image for a subject, the utilization of big data repositories, the setting of an effective annotation strategy, image standardization, and color normalization.