Automated Computational Detection, Quantitation, and Mapping of Mitosis in Whole-Slide Images for Clinically Actionable Surgical Pathology Decision Support.

BACKGROUND: Determining mitotic index by counting mitotic figures (MFs) microscopically from tumor areas with most abundant MF (hotspots [HS]) produces a prognostically useful tumor grading biomarker. However, interobserver concordance identifying MF and HS can be poorly reproducible. Immunolabeling MF, coupled with computer-automated counting by image analysis, can improve reproducibility. A computational system for obtaining MF values across digitized whole-slide images (WSIs) was sought that would minimize impact of artifacts, generate values clinically relatable to counting ten high-power microscopic fields of view typical in conventional microscopy, and that would reproducibly map HS topography.
MATERIALS AND METHODS: Relatively low-resolution WSI scans (0.50 μm/pixel) were imported in grid-tile format for feature-based MF segmentation, from naturally occurring canine melanomas providing a wide range of proliferative activity. MF feature extraction conformed to anti-phospho-histone H3-immunolabeled mitotic (M) phase cells. Computer vision image processing was established to subtract key artifacts, obtain MF counts, and employ rotationally invariant feature extraction to map MF topography.
RESULTS: The automated topometric HS (TMHS) algorithm identified mitotic HS and mapped select tissue tiles with greatest MF counts back onto WSI thumbnail images to plot HS topographically. Influence of dye, pigment, and extraneous structure artifacts was minimized. TMHS diagnostic decision support included image overlay graphics of HS topography, as well as a spreadsheet and plot of tile-based MF count values. TMHS performance was validated examining both mitotic HS counting and mapping functions. Significantly correlated TMHS MF mapping and metrics were demonstrated using repeat analysis with WSI in different orientation (R 2 = 0.9916) and by agreement with a pathologist (R 2 = 0.8605) as well as through assessment of counting function using an independently tuned object counting algorithm (OCA) (R 2 = 0.9482). Limits of agreement analysis support method interchangeability. MF counts obtained led to accurate patient survival prediction in all (n = 30) except one case. By contrast, more variable performance was documented when several pathologists examined similar cases using microscopy (pair-wise correlations, rho range = 0.7597-0.9286).
CONCLUSIONS: Automated TMHS MF segmentation and feature engineering performance were interchangeable with both observer and OCA in digital mode. Moreover, enhanced HS location accuracy and superior method reproducibility were achieved using the automated TMHS algorithm compared to the current practice employing clinical microscopy.

INTRODUCTION

The proliferative capacity of tumors contributes to their growth and metastasis. Identification and enumeration of mitotic figures (MFs) in cancer biopsy tissue sections microscopically is a surrogate for proliferative activity for many tumor types.[12345678910] Enumeration of MF can be an independent clinically prognostic biomarker relevant for tumor grading. Counting MF on hematoxylin and eosin (H&E)-stained tissue sections, which is not a mitotic-specific stain, is a challenging exercise in MF recognition that requires experience. The strategy can involve selecting tumor areas with the greatest number of MFs, known as mitotic hotspots (HSs). The tedious and time-consuming nature of this error-prone task has been documented in studies examining the reproducibility of MF counting.[7111213141516] The variation in obtaining tumor mitotic count values can be attributed to issues such as varying recognition accuracy. For example, agreement about what is a MF and its differentiation from other nuclear structures is not unanimous, and choice of tumor tissue area for evaluation can also play a role in designation of grade.[712] Other limitations such as cellular level variations in color, intensity, and morphological shape/size can contribute to counting errors. These circumstances can result in poor reproducibility and discordant inter- and intra-reader mitotic count values.

Approaches such as antibody immunolabeling of MF and the automation of MF counting through computer-assisted image analysis offer the opportunity to minimize variability and increase reproducibility in assessing tumor proliferative activity in tissue biopsies at the time of diagnosis.[111317] Molecular markers of mitosis, such as phosphorylated histone H3 (pHH3) and MKI67 (Ki-67), improve prognostic utility of counting MF compared to routine H&E tissue stains.[2715171819] Active development of computer-assisted image analysis indicates promise for improving reproducibility and accuracy.[1113141517202122232425] The ability to specifically identify and segment the proliferating cells in digitized whole-slide images (WSIs) using computer-assisted diagnosis (CAD) can be complex, however. The potential to miscategorize MF based on morphological characteristics can become amplified once biopsies are optically scanned for image analysis.

Certain challenges in assessing proliferative activity using computer-assisted decision support have been described, and a limited number of automated approaches for detecting MF in tissue sections have been published.[141518192021] Some shortcomings in image processing performance persist, nevertheless. Chief among these includes insufficient locating clinically meaningful mitotic HS regions topographically, substantial minimization of common confounding tissue artifacts, generation of MF counts that are clinically relatable to the MF count values currently produced using conventional microscopy, and the ability for pathologists to quality assure algorithm performance on a patient by patient basis. Development of an automated system improving upon these features to benefit tumor grading based on mitotic activity was a primary objective.

In this study, the automated topometric HS (TMHS) algorithm addresses these needed analytics, providing topographic information on mitotic HS location in the tissue context, while yielding MF metrics as surrogate for the tumor proliferative activity. We developed a feature-based computational approach built upon processing WSI as rendered image grid tiles for extracting immunolabeled MF and incorporated filters capable of subtracting several common tissue artifacts. Performance of the automated image processing characteristics was validated in 30 naturally occurring canine mucosal melanoma tissue biopsies. Naturally occurring canine mucosal melanoma serves as a valuable preclinical model for human mucosal and other triple wild-type melanomas.[222324] Repeat analysis of these clinical specimens using the automated TMHS mapping algorithm provided significant evidence of method interchangeability with a pathologist or a second algorithm, based on counting MF at the automated TMHS identified HS. Automated TMHS provides needed enhancements for clinically actionable mitotic HS mapping, the prognostic utility of which was significantly correlated with the standard of care.

MATERIALS AND METHODS

Specimens

Formalin-fixed and paraffin-embedded biopsy tissue samples from 30 dogs with naturally occurring mucosal melanoma (manuscript in preparation) were used to evaluate the automated TMHS mitotic mapping application. The biopsy specimens were surgically obtained from dogs in the course of veterinary patient care. The pathological characteristics of human melanoma were shared among these canine melanomas.[22] Patients selected for analysis had follow-up survival data available.

Immunohistochemistry

Five-micrometer-thick paraffin-embedded tissue sections that ranged in area from 30.32 to 396.83 mm2 were placed on silanated slides for immunohistochemistry (IHC). Slides were deparaffinized and rehydrated. Following rehydration, slides were bleached in 3% hydrogen peroxide in phosphate-buffered saline (pH 7.2) at 60°C for 30 min to remove melanin. Heat-induced epitope retrieval was performed using citrate solution (S1699, pH 6.0, Agilent Dako, Santa Clara, CA) for 15 min in a steam bath (Black & Decker, Beachwood, OH, USA) and allowed to cool an additional 15 min on the bench at room temperature. Subsequent immunolabeling steps using anti-pHH3, an antibody targeting a mediator of chromatin condensation during cell replication, were performed using an autostainer (Agilent Dako, Santa Clara, CA, USA) at room temperature with an alkaline phosphatase immunoenzyme polymer-linked detection system and permanent red chromogen (K535511-2, Agilent Dako, Santa Clara, CA, USA). Incubation steps included endogenous enzyme block for 10 min, protein block for 20 min, primary antibody (rabbit monoclonal anti-pHH3, ab32107 (clone E173), Abcam, Cambridge, MA, USA) diluted at 1:50 (Dako Antibody Diluent) and incubated for 1 h, followed by polymer-conjugated secondary antibodies as prepared in the kit for 30 min, alkaline phosphatase enzyme with enhancer for 30 min, and permanent red chromogen for 20 min. Slides were then removed from the autostainer, counterstained with hematoxylin, dehydrated, cleared, and coverslipped. Melanomas were processed along with tissue sections of canine intestines, including crypt epithelia, as a positive control tissue. Tissue areas of tumor-adjacent normal and intestinal strata superficial to glandular mucosae served as negative control tissue.

Whole-slide image scanning

Anti-pHH3-immunolabeled slides were scanned at ×20 using an Aperio AT2 digital slide scanner (Leica Biosystems, Buffalo Grove, IL, USA) to create WSI data files at 0.50 μm/pixel resolution. Image files were stored in spectrum image management system and viewed and annotated using Aperio ImageScope Software (Leica Biosystems, Buffalo Grove, IL, USA).

Computational framework for topometric mapping of mitotic hotspots

The computational pipeline is devised and developed in MATLAB image processing and computer vision toolbox (MathWorks, version R2017a, Natick, MA, USA) for WSI of tissue sections. Detection and mapping of MF in cells undergoing mitotic (M) phase is broadly based on color, morphology (shape), and texture (surface roughness) features. Image processing algorithms, including color-based segmentation, color-to-gray conversion, gray-to-binary conversion, counting binary dots, and mapping local features, are used to implement the developed workflow [Figure 1]. Unique coding loops for batch processing image grid tiles and for integrating different building blocks contribute operational sequences. Methods include design of a custom graphical user interface (GUI) to enable downstream unsupervised execution [Supplemental Figure S1].

Figure 1

Three phases of the analytic workflow for detecting, quantifying, and mapping proliferative cell nuclei (mitotic figure) in tissue whole-slide images. Initial phase involves creating grid tile regions of interest covering the entire tissue and extracting the necessary features to detect and segment phosphorylated histone H3-immunolabeled nuclei, a marker of proliferating cells in mitosis (M phase). In Phase II, the mitotic figure objects of interest are counted, and each grid tile tally is exported to a spreadsheet and rank ordered according to total counts. In the final conceptual Phase III, selected tiles (with greatest counts) are analyzed to obtain local features which can be mapped back to the whole slide image for display

Functional execution is made possible through a graphical user interface developed in MATLAB to implement the workflow. WSI: Whole-slide image

Click here for additional data file.

Image analysis execution encompassed the entire image region using a rendered grid subdivided into 3000 × 3000 pixel tiles, which coincided to image analysis at approximately ×4 magnification, relative to an image viewed using standard light microscopy [Figure 2a]. Each grid tile corresponded to 1.514 mm × 1.514 mm (2.292 mm2) tissue area/tile, which were then segregated into tiles containing tissue images and those lacking tissue. To segment and extract MF in images, a red, green, and blue (RGB) color space was established for pHH3-immunolabeled MF, which enabled detection of the red chromogenic signal developed for the IHC. During MF segmentation, additional color-based segmentation filters, as well as morphology and thresholding filters, were processed to segment MF from the tissue image and to digitally subtract common individual or combinations of shape- and color-based tissue and stain-related artifacts that could confound MF object counting function [Supplemental Figure S2]. Approximately 7% of the 3356 image tiles eventually analyzed were used in training.

Figure 2

Pictorial depiction of computational workflow for topometric proliferative hotspot detection and mapping for tumor tissue. (a) Input whole-slide image is overlain with a grid of tiles corresponding to 3000 × 3000 pixels each or approximately ×4 magnification. The processing includes a series of steps to reduce the confounding effects of tissue and staining artifacts [Supplemental Figure S2]. Specific color and shape object features corresponding to red chromogen-immunolabeled proliferating cell nuclei within the image are segmented and quantified according to filters established. (b) Mitotic figure extracted features are binarized, counted, and exported to a spreadsheet, as a rank order set of tiles, based on count magnitude. These mitotic figure values are represented in a slope plot for the specimen for up to the first 100 tiles containing the most mitotic figure. (c) Local tissue invariant feature extraction methods are used to obtain the proper orientation of selected identifying features within grid tiles of interest, for those tiles with the largest counts, tiles h1–h5. These are then marked by a bounding box and displayed on the thumbnail image of the input tissue image

Example artifact filter series illustrates combination of color, shape, size, and thresholding filters for subtracting common tissue and stain-related artifacts. Artifacts removed included: off-target red-chromogen labeling, green and black ink, hemosiderin, melanin, hair, tissue folds and tears, and tissue edge effects. Images, original capture magnification ×4 or ×5

Click here for additional data file.

Extracted MFs were converted to gray values based on their respective features. RGB-to-gray function parameters were established to process the true RGB color image to the grayscale intensity image. Segmented MFs were converted to a binary image mask using clustering-based Otsu's method to choose the threshold value for minimizing intraclass variance of the black and white pixels.[25] In succeeding steps, counting algorithms were used to count MF per tile of grid-based image partitioning. MF count data were autoexported to a spreadsheet along with corresponding image tile spatial information. To topographically localize the greatest MF activity areas from the counts generated back onto the tissue, an algorithm (speeded-up robust features [SURF]) was employed to invariantly match local features of maximum count tiles (top five HS regions of interest (ROIs), designated h1–h5) to the corresponding location of those tiles in WSI tissue area.[26] The uppermost five HSs (based on MF counts) were selected to display a survey of mitotic activity heterogeneity; however, a different number of tiles could be chosen for exhibition. The novel workflow and steps for the automated TMHS are illustrated [Figure 1].

Validation studies

Independent of TMHS algorithm image analysis, the 30 cases were evaluated by pathologists. A pathologist (CHH) examined and annotated digital images using Aperio ImageScope and digital display monitor to demarcate areas of the greatest mitotic activity within specimens. The annotation consisted of a circular ROI corresponding to 2.37 mm2 area (R1) or approximately ten high-power fields of view (FOVs) in standard clinical microscopes with ocular lens eyepiece field number 22 (manuscript in preparation). A pathologist (HAA) counted immunolabeled MF visually and recorded counts electronically (manuscript in preparation). In addition to producing MF counts in the independently designated R1 HS, this pathologist also counted MF in the TMHS-designated h1 HS. The MF count values generated within h1 and R1 HS by the TMHS algorithm and the pathologist, respectively, were compared to counts obtained independently using a second machine algorithm established for cell-object counting algorithm (OCA) (HALO v2.1.1637, Cytonuclear Algorithm v1.4, Indica Labs, Corrales, NM, USA). The OCA algorithm does not include an automated topographic tissue mapping function. The degree to which MF R1 HS identified by the pathologist coregistered with the TMHS algorithm-designated HS (h1–h5) was assessed visually in overlay images.

Further performance appraisal was provided by a group of six pathologists who obtained MF counts as part of a larger concurrent study devoted to comparing pathologist performance agreement between conventional microscopy and digital display of WSI (in preparation). For this comparison, 35 anti-pHH3-immunolabeled canine melanomas were read by pathologists in their individual diagnostic sign-out environments using conventional microscopy. The pathologists independently identified their preferred mitotic HS for evaluation and used the ×40 objective lens for counting MF in ten contiguous nonoverlapping microscopic FOVs. Pair-wise interobserver correlation analyses among pathologists provided contrast with automated TMHS method reproducibility.

Statistical analysis

MF count values acquired from TMHS, independent pathologist, and OCA were analyzed for methods comparisons by Passing and Bablok linear regression agreement analysis using log10-transformed data.[27] Bland and Altman analysis of MF values derived by the various modalities was performed to gauge the limits of agreement between methods using log10-transformed data.[28] These difference versus average plots included analyses of the following three comparisons:

Pathologist R1 – TMHS h1 versus (TMHS h1 + pathologist R1)/2

Pathologist h1 – TMHS h1 versus (TMHS h1 + pathologist h1)/2

OCA h1 – TMHS h1 versus (TMHS h1 + OCA h1)/2.

Analyses were performed to understand the correlation of MF counts with melanoma patient survival data using Kaplan–Meier survival analysis and the log-rank test. Pair-wise correlation of MF counts among pathologists obtained by conventional microscopy was analyzed using Spearman's (rho) correlation analysis of log10-transformed data. Statistical significance of tests is claimed for P < 0.05.

RESULTS

Automated quantification and mapping of proliferative activity

The automated TMHS computational process quantified and mapped mitotic activity HS in digital image files of tumor tissue. During the initial steps (Phase I) [Figure 1], image tiles were segregated into groups that either included tissue or tiles that lacked tissue (glass only). Tiles that contained both tissue and glass (tissue edges) were treated as tissue tiles. Each tissue-containing tile was automatically assigned a unique identification number and exported for feature extraction. MF feature extraction employed a combination of color, size, and shape filters to detect MF features corresponding to the red chromogen of pHH3-immunolabeled mitotically active cells developed during IHC [Figure 2a]. This step also compensated for a range of confounding artifacts commonly associated with tissue processing and staining, based on color, size, and shape to permit subtraction of elements such as pigments, dyes, and extraneous objects [Supplemental Figure S2]. Segmentation and extraction filters employed in Phase I were able to identify anti-pHH3-labeled mitotic cells with notable specificity while eliminating background noise due to common artifacts occurring during slide preparation, labeling, and staining. This was confirmed by visually comparing the postprocessed h1 HS tile binary images with the corresponding bright-field micrograph tile images for all cases [Figure 2]. With background subtraction leaving only MF objects in the segmentation stage, image tiles were recorded in separate folders and moved to the second phase [Figure 1].

In Phase II [Figure 1], tiles were converted to grayscale to enable subsequent object binarization, and thresholding filters permitted MF counting as binary objects, per tile area. MF count data were exported to a spreadsheet, and values were rank-ordered according to the highest to lowest tile count [Figure 2b]. Analysis of this series of canine melanoma tumor tissues included a wide range of proliferative activity, from 2 to 793 MF in the h1 HS tile, and assessment of up to 100 tissue-containing image grid tiles generally included the scope of MF counts. A total of 3356 image tiles (mean tissue-containing tiles per specimen = 111.9 [range 29–210]) were processed.

Phase III of the automated TMHS tool function mapped the five tiles with greatest MF counts back to the WSI thumbnail, to plot mitotic activity topographically [Figure 1]. These five tiles, designated HS h1–h5 corresponding to the ordered magnitude of the top five MF ROIs, were matched by tile address back to the RGB color tile tissue folder. Mapping employed SURF, a local tissue rotation invariant feature extraction computer vision algorithm [Figure 2c]. Local features extracted for this function included distinctive structures, objects, and edges occurring in images. Once maximum confidence in the strongest features matching to the WSI for the maximum count tiles was achieved, a bounding box corresponding to the magnitude of count HS tiles h1–h5 was displayed automatically [Figures 2c and 3a].

Figure 3

Diagnostic decision support graphics provide topography and count metrics for tissue hotspots. Automated mapping of the tissue grid tiles having the most numerous mitotic figure counts across the entire tissue section image is illustrated in the context of the image processing tile grid. Each tile area (2.29 mm2) is commensurate with ten high-power (×40) fields of view on standard clinical microscopes (2.37 mm2 area). (a) Square tiles h1–h5, assigned by the automated topometric mitotic hot spot tool, represent the five tiles with the highest mitotic figure count. In context of the grid regions of interest, the hot spot tiles reveal the topographic heterogeneity of the proliferative activity in this example tumor. The square (red box) depicts the h1 tile with the highest mitotic figure count. (b) Mitotic figure count values are plotted for up to 100 tiles having the greatest counts, providing an indication of the proliferative characteristics across a tissue section. The graphic depicts plots for 30 biopsies. The inset graphic depicts the relative differences in proliferative characteristics within the 50 highest count tiles of 27 tumor specimens. The three most proliferatively active melanoma cases were excluded for illustration purposes

The automated TMHS algorithm provides diagnostic decision support and tumor grading functions for review by the pathologist. These outputs include graphics of tumor image and overlay tiles depicting the MF HS locations topographically [Figure 3a]; a MF counts slope plot for up to 100 grid tile count values across each tissue image [Figure 3b], as well as a corresponding spreadsheet of MF count values. Due to the established grid tile metrics, the MF values are scaled based on a standard area in diagnostic surgical pathology equal to ten high-power FOVs, for each tissue-containing tile [Supplemental Table 1]. The automated TMHS diagnostic support algorithm function was executed through a GUI controller [Supplemental Figure S1].

Supplemental Table 1

Quantitative mitotic figure values for automated topometric mitotic hot spot mapping compared to counts by a pathologist expert and a second counting algorithm on 30 mucosal melanoma biopsies

Patient	Analysis Region	TMHS counts	OCA counts	Pathologist Counts	Co-registration of R1 to h1-h5 HS	90° Rotated Image h1 HS (Counts) Compared to Original h1 TMHS
1284	h1	16	19	34	Substantial overlap with h1	Complete overlap with original h1 (17)
	h2	12	12
	h3	10	13
	h4	10	13
	h5	10	11
	R1		35	49
1619	h1	42	53	29	Complete overlap with h3 minimal with h1 (adjacent)	Partial overlap with original h1 substantial overlap with original h4 (h1 and h4 are contiguous) (40)
	h2	30	32
	h3	30	34
	h4	28	29
	h5	27	42
	R1		37	26
1964	h1	49	50	52	Substantial overlap with h1 partial with h2	Minimal overlap with original h1 substantial overlap with original h2 (h1 and h2 are contiguous) (42)
	h2	34	36
	h3	32	40
	h4	28	32
	h5	25	26
	R1		80	79
2501	h1	113	127	105	No overlap separate tissue from h1	Substantial overlap with original h1 (100)
	h2	108	113
	h3	99	112
	h4	85	95
	h5	83	87
	R1		78	59
2594	h1	3	4	8	Partial overlap with h5	Complete overlap with original h1 (3)
	h2	3	3
	h3	2	3
	h4	2	2
	h5	2	3
	R1		4	2
2633	h1	22	22	27	Substantial overlap with h2 minimal with h1 (h1,2 are contiguous)	Substantial overlap with original h1 substantial overlap with original h5 (h1 and h5 are contiguous) (25)
	h2	21	20
	h3	21	22
	h4	19	20
	h5	17	18
	R1		26	20
2641	h1	22	22	24	No overlap separate tissue	Substantial overlap with original h1 substantial overlap with original h5 (h1 and h5 are contiguous) (25)
	h2	19	19
	h3	16	16
	h4	12	12
	h5	10	10
	R1		2	12
2676	h1	75	92	93	Minimal overlap with h2	Complete overlap with original h1 (68)
	h2	64	92
	h3	61	70
	h4	53	75
	h5	49	79
	R1		86	63
3043	h1	19	5	45	Substantial overlap with h2 minimal with h1, h5	Substantial overlap with original h1 substantial overlap with original h2 (h1 and h2 are contiguous) (16)
	h2	8	5
	h3	5	4
	h4	4	4
	h5	3	2
	R1		4	37
3109	h1	414	445	457	Partial overlap with h5 (h5 is contiguous with h4)	Complete overlap with original h1 (389)
	h2	354	359
	h3	313	362
	h4	295	309
	h5	275	289
	R1		392	391
3676	h1	25	31	28	Partial overlap with h3	No overlap with original h1 but adjacent to original h2 (27)
	h2	24	43
	h3	24	27
	h4	23	26
	h5	22	24
	R1		28	25
4479	h1	17	21	17	No overlap	Complete overlap with original h1 (16)
	h2	14	16
	h3	14	13
	h4	11	16
	h5	11	12
	R1		9	8
4583	h1	5	10	14	Substantial overlap with h1 minimal with h2 (h1 and h2 are contiguous)	Substantial overlap with original h1 partial overlap with original h2 (h1 and h2 are contiguous) (6)
	h2	4	6
	h3	3	4
	h4	3	3
	h5	3	7
	R1		11	15
5223	h1	59	61	57	No overlap separate tissue	Complete overlap with original h1 (59)
	h2	52	57
	h3	47	55
	h4	44	54
	h5	41	48
	R1		39	38
5579	h1	136	151	112	Substantial overlap with h1 minimal with h5 (h1 and h5 are contiguous)	Substantial overlap with original h1 partial overlap with original h5 (h1 and h5 are contiguous) (139)
	h2	120	133
	h3	106	123
	h4	92	102
	h5	87	97
	R1		123	93
5632	h1	19	15	18	Complete overlap with h4	No overlap with original h1 substantial overlap with original h2 (16)
	h2	17	18
	h3	12	11
	h4	10	10
	h5	9	7
	R1		10	14
5812	h1	100	102	109	Substantial overlap with h1 minimal with h2, h5 (h1-h5 are contiguous)	Substantial overlap with original h1 partial overlap with original h5 (h1 and h5 are contiguous) (90)
	h2	71	83
	h3	71	76
	h4	66	74
	h5	39	43
	R1		107	104
6534	h1	33	34	38	Substantial overlap with h3	No overlap with original h1 substantial overlap with original h3 partial overlap with original h4 (h3 and h4 are contiguous) (26)
	h2	26	27
	h3	25	26
	h4	19	22
	h5	19	21
	R1		27	29
6589	h1	21	17	20	Substantial overlap with h1	Substantial overlap with original h1 (19)
	h2	20	15
	h3	20	19
	h4	13	12
	h5	9	9
	R1		17	22
6612	h1	47	55	38	Partial overlap with h2	Substantial overlap with original h1 partial overlap with original h4 (h1 and h4 are contiguous) (42)
	h2	43	55
	h3	42	55
	h4	39	52
	h5	38	46
	R1		48	49
7142	h1	2	2	2	Partial overlap with h2, h4 (all tiles have same count value)	Substantial overlap with original h1 minimal overlap with original h2 (h1 and h2 are contiguous) (all tiles have same count value) (2)
	h2	2	2
	h3	2	2
	h4	2	2
	h5	2	2
	R1		3	2
7204	h1	165	182	260	Substantial overlap with h1 partial h2 , minimal h3 (h1-h4 contiguous)	Substantial overlap with original h1 (156)
	h2	117	131
	h3	104	128
	h4	71	82
	h5	70	77
	R1		265	309
7870	h1	75	79	89	Partial overlap with h5	No overlap with original h1, separate tissue substantial overlap with original h2 (70)
	h2	65	68
	h3	64	68
	h4	57	59
	h5	56	58
	R1		48	44
8032	h1	793	867	678	No overlap separate tissue	Substantial overlap with original h1 (747)
	h2	761	826
	h3	644	713
	h4	621	676
	h5	550	603
	R1		320	348
8478	h1	280	315	285	Substantial overlap with h4 minimal with h5	Substantial overlap with original h1 (313)
	h2	185	208
	h3	174	199
	h4	167	185
	h5	163	181
	R1		249	196
8489	h1	53	63	30	No overlap	No overlap with original h1 substantial overlap with original h2 (64)
	h2	52	55
	h3	49	62
	h4	48	58
	h5	48	55
	R1		50	37
8615	h1	36	40	40	Partial overlap with h4 minimal with h1 (h1 and h4 are contiguous)	Complete overlap with original h1
	h2	35	40			(39)
	h3	32	35
	h4	27	25
	h5	24	49
	R1		39	40
9352	h1	35	37	46	Substantial overlap with h2 minimal with h1	Substantial overlap with original h1 (26)
	h2	17	21
	h3	12	21
	h4	7	13
	h5	6	7
	R1		33	28
9623	h1	99	113	105	Partial overlap with h5 (h5 adjacent to h1)	Partial overlap with original h1 (86)
	h2	92	170
	h3	89	101
	h4	87	102
	h5	85	113
	R1		144	124
9830	h1	81	96	56	No overlap	Partial overlap with original h1 (84)
	h2	77	93
	h3	73	84
	h4	69	90
	h5	66	73
	R1		79	73

h1 – h5 analysis regions=mitotic hot spots assigned by TMHS algorithm. R1 analysis region=mitotic hot spot assigned by pathologist. OCA=Object Counting Algorithm where counts were obtained by a second algorithm established to count MF in previously assigned regions, h1 - h5 and R1. Pathologist counts reflect a pathologist visually counting MF either in previously assigned regions (h1 - h5) or in pathologist designated R1 region. HS co-registration agreement: Complete overlap ≥90% , Substantial overlap ≥50% and <90% , Partial overlap ≥10% and<50% ; Minimal overlap ≥1% and <10% ; No overlap; ROI located on separate piece of tissue. Rotated Image Analysis=Repeat Analysis by TMHS tool (see methods).

Automated topometric hotspot performance validation

Automated TMHS functionality was validated further in a representative clinical application using the series of 30 canine melanoma biopsies immunolabeled with anti-pHH3 IHC for proliferating cells. Tumor mitotic activity was ascertained, and counts were obtained and mapped. Several measures were compared for evidence of correlation, reproducibility, and method interchangeability. For these comparisons, the automated TMHS machine performance output was correlated with the pathologists who independently identified, mapped, and counted tumor mitotic HS (R1) in the case series. Additional appraisal of the TMHS performance was made by comparing the automated TMHS-generated h1 MF counts with counts made visually by the pathologist from the identical TMHS-designated h1 HS. A third comparison was made between the TMHS algorithm count values and those obtained by quantifying the immunolabeled MF within the TMHS-designated h1 HS using a second machine OCA. This OCA was independently established using the same immunolabeled MF objects of interest in the WSI; however, the OCA algorithm does not include a mapping function. In addition, mitotic counts generated by the pathologist and from the TMHS algorithm were tested for prognostic utility by attempting to stratify patient survival intervals.

Mitotic counts are a prognostic indicator of survival/prognosis in canine melanoma,[29] similar to the clinical utility of determining MF frequency in several human tumor types, including melanoma.[5678910] Therefore, an examination predicting longer versus shorter survival time risk, based on MF values from HS, was undertaken analogous to how the automated TMHS tool would be applied clinically. A MF cut score value of 110 immunolabeled MF was determined for dogs in this study through a process of serially testing candidate MF count values to segregate patient populations based on survival postdiagnosis. Kaplan–Meier plot and log-rank analysis demonstrate a difference in survival, with a cut point of 110 pHH3-positive cells [Supplemental Figure S3, for TMHS, P = 0.0034; and for manual evaluation, P = 0.0013]. The median survival of the poor prognostic group was 45 or 38 days and for the more favorable prognostic group, 197.5 or 197 days, based on TMHS and pathologist, respectively.

Kaplan–Meier survival analysis curve based on mitotic figure counts for melanoma patients assessed by automated TMHS algorithm (a) (P = 0.0034) and by pathologist (b) (P = 0.0013). Plots include subtle differences. Patients were treated by surgery with intent to cure and survival days were tracked from time of diagnosis (n = 30)

Click here for additional data file.

The extent of topographic agreement between the automated TMHS algorithm-mapped HSs (h1–h5) and the R1 HS manually identified by the pathologist was evaluated using image overlays of these two independent approaches to HS identification [Supplemental Figure S4]. Topographically, the manual primary HS (R1) partially or more substantially coincided with one or more of the automated mitotic HS (h1–h5) in the majority of cases (23 of 30 [76.7%]) [Supplemental Table 1]. In one of the seven cases that lacked co-registration, the survival prognosis based on mitotic activity was misjudged by the pathologist. Another discrepancy revealed a distinct HS focus that the pathologist identified, which shared a MF count similar to the TMHS-designated h5 HS. This latter discrepancy may not be clinically meaningful, whereas the former one in which the mitotic HS is not aligned with the representative tumor mitotic activity represents a variance that risks leading to diagnostic error.

Automated TMHS hotspots (h1–h5) mapped to whole-slide images thumbnail image including overlay image of the single annotated regions of interest independently identified and assessed by the pathologist (a 2.37 mm2 blue circular area, R1). In this representative sample, the R1 hotspot substantially corresponded to portions of h1 and h2 hotspot, the area of maximum mitotic activity identified by the automated topometric method

Click here for additional data file.

The agreement among MF count values obtained by the automated TMHS mapping algorithm, OCA machine analyses, and by pathologist visual inspection was also considered through nonparametric Passing and Bablok linear regression for method comparison. Examining MF count correlation in this manner allows for measurement error (imprecision) in the approaches compared, does not require the measurement error to be normally distributed, and is insensitive to outliers.[27] This analysis provided strong evidence of agreement among the various approaches [Figure 4 a, b and c, P < 0.0001]. The degree of correlation was greater when precisely the same area was assessed (h1 compared to h1, in contrast to R1 compared to h1), with stronger parallel observed between OCA and TMHS in most of the cases, compared to visual counts of MF by the pathologist. Two data points deviating most from the regression line were noted [Figure 4c]. Both cases were influenced by a varying and regionally weak IHC chromogenic signal development [lower point, ID 3043, upper point, ID 4583, Supplemental Table 1]. Both algorithms were impacted, resulting in undercounts, while the pathologist appears to have compensated in these instances [Supplemental Table 1].

Figure 4

Performance fit comparisons between the automated topometric mitotic figure hotspot tool (TMHS) and mitotic figure quantification by expert visual assessment or a second machine object counting algorithm (OCA). Regression scatter plots of paired mitotic figure count observations (log10) from the TMHS h1 hot spot, with (a) counts independently obtained by pathologist (R1 hotspot), (b) counts of the automated h1-selected topometric hotspot obtained by the pathologist, or (c) counts from the automated h1-selected topometric hotspot obtained by object counting algorithm counting. Two data points deviating from the regression line were noted in particular. One of these cases displayed in c was influenced by a weaker immunohistochemistry signal development (lower point, ID 3043). In the second case (upper point, ID 4583), repeat counts obtained by the pathologist ranged from 14 to 16 for both h1 and R1 hotspots, which were regions of interest completely overlapping topographically. Blue-shaded areas represent 95% confidence intervals. R2 values and regression equations are indicated on the figure. All comparisons represented significant correlations

Similarly, paired assessments of MF HS were depicted in a series of difference versus average plots [Figure 5]. Bland and Altman analysis estimates the degree to which two methods differ in quantitative measurement (limits of agreement) and can thereby aid in deciding if one method can be substituted for another.[28] The limits of agreement analyses indicate that methods were interchangeable [Figure 5].

Figure 5

Limits of agreement. The difference in measurements made in paired analyses was plotted against the average of two methods compared (all data log10 transformed). General agreement between methods is evident with minimal variance (limits of agreement, shaded area). Bland–Altman scatter plots of paired count observations (a) obtained by pathologist-independent visual assessment of R1 minus automated h1-selected topometric hotspot, (b) pathologist counts by visual inspection from the h1-selected topometric hotspot minus counts generated from the automated h1-selected topometric hotspot, or (c) mitotic figure counts obtained from the h1-selected hotspot using a second machine object counting algorithm (OCA) minus the automated h1-selected topometric hotspot counts

The reproducibility of the automated TMHS algorithm HS mapping and counting functions was further assessed on the identical image files following 90° in-plane image rotation, to create an alternative input orientation. The resulting h1 HS topographic location and the respective MF counts substantially reproduced the initial output when the tissue image was reprocessed by the automated TMHS algorithm in a different orientation [Supplemental Table 1] and Figure 6a, b]. The automated TMHS h1 HS was registered to within a few hundred microns of its original tissue location in all but five specimens [Supplemental Table 1]. In four of the five cases, the new h1 HS in the rotated image overlapped or was adjacent to the original h2 HS, the ROI with the second highest mitotic count. In two of these four cases for which the newly mapped h1 HS was aligned with the original h2 HS after image rotation, the counts were within 3 MF of the initial h1 mitotic count values and not >9 MF among the remaining two [Supplemental Table 1]. Repeating the automated TMHS MF count values using images in a second orientation provided evidence of excellent correlation for the mapping and counting functions (R2 = 0.9916) [Figure 6c]. This degree of reproducibility contrasts with a range of relatively less well-correlated MF values evident when assessing MF agreement among a group of pathologists [Figure 6d]. The latter data were obtained evaluating anti-pHH3-labeled mitotic activity in a series of melanomas by microscopy, under typical case sign-out circumstances. This operational discordance when pathologists perform this task remains a concern, and the functional reproducibility of the automated TMHS method is a strong characteristic of the tool and represents a distinct advantage.

Figure 6

Performance fit comparisons between repeated automated TMHS analyses and among a group of pathologists examining anti-phosphorylated histone H3-immunolabeled melanomas by conventional microscopy. (a) For TMHS, an original input image, with grid tile overlay, includes the computed TMHS h1 hotspot indicated by red box tile, which is illustrated in relation to the R1 hotspot selected by the pathologist (circle). (b) Identical image as in a, after 90° in-plane rotation, is processed in TMHS to demonstrate the computed h1 hotspot within the same tissue region as the original. The R1 hotspot, selected by the pathologist and shown as an image overlay, has the same tissue coordinates as in a. Bar = 1.514 mm. (c) Regression scatter plot of paired mitotic figure count values in the two image orientations among the 30 cases, from automated TMHS mapping tool (log10 transformed). Blue-shaded area represents 95% confidence interval. R2 value and regression equation are indicated on the figure. Measurements from the two image orientations were highly significantly correlated (P < 0.0001). (d) Pair-wise, interobserver agreement of mitotic figure assessment by each pathologist using conventional microscopy on anti-phosphorylated histone H3-immunolabeled melanomas (n = 35). Each cell in the heat map represents the Spearman correlation coefficient (rho values, bar graph) between each pair of pathologists (A, B, F, J, K, and L), using the total mitotic figure count in ten high-power microscope fields of view

DISCUSSION

Assessment of proliferative capacity is an important component of tumor grading and treatment planning for several types of malignancies, including breast, adrenal gland, meningiomas, neuroendocrine carcinomas, soft-tissue sarcomas, and melanoma.[1245910121430] Yet, during routine visual inspection of H&E-stained biopsies, pathologists frequently fail to agree on which structures represent MF. In a recent study, only 21 of 92 MF in H&E-stained tumor biopsies were unanimously identified as MF by all five participating pathologists examining the same 40 FOVs (https://nciphub.org/groups/eedapstudies/wiki/Presentation: AReaderStudyona14headMicroscope December 3, 2018). Selection bias, sample size, and tumor heterogeneity can all contribute to disparities.[2] The use of immunohistochemical methods to label cell proliferation markers has been shown to improve MF detection and quantitation for prognostic purposes.[213173132] The approach demonstrated here takes advantage of the differential contrast and target probe enhancement of mitotic (M) phase cells, through pHH3 immunolabeling. These characteristics are commonly shared among tumor types and even across species. While prognostic grading can be accomplished using MF counts from H&E-stained tissues and mitotic immunolabeled tissues, the absolute values for the prognostic cut points differ between cancer types and mitotic counts obtained by evaluation of H&E stains and pHH3 IHC.

Furthermore, quantifying MF from tumor HS using automated computer assistance stands to improve the interobserver variability inherent in clinical MF assessment. The automated TMHS algorithm uses an unsupervised tile-based ROI approach, executed through a GUI, fully processing each image pixel in subdivided static images at approximately ×4 magnification (size ≈ KB). Computationally, this helps to simplify processing requirements manifest in some other systems. For example, the automated selection of hotspot (ASH) tool, applied in the detection of HS in melanomas and adrenocortical tumors, differs from TMHS by the block size, and ASH image tiles number in the thousands (size ≈ GB), which together would appear to add substantially to processing requirements.[19] By contrast, the automated TMHS approach permitted HS detection from the first 100 tissue tiles. TMHS analysis indicated that MFs were concentrated within ≤80–100 tiles, approximating a tissue surface area of ≤240 mm2. In addition, TMHS operates with lower resolution input images from ×20 magnification optical scans, which can be more rapidly acquired, providing further advantageous image processing distinctions from other methods.[182130] Consequently, these TMHS design characteristics afford relative computational efficiency for handling the large size of WSI data files.

Clinical practice typically includes assessment of MF density for areas of tumor mitotic HS. Customarily, counts are made beginning in an area of the tumor with the most plentiful MF, i.e., the mitotic HS. Many MF cut points used for tumor grading are based on this approach.[1212] Prognostic assessment of proliferative activity by microscopy usually involves examination of ten high-power magnification FOVs (×40 objective lens).[12123233] Significant differences in the predictive value of MF counts assessed from mitotic HS, in contrast to MF counts obtained from either randomly selected FOV, or the tumor area more globally, have been documented.[1633] While counts obtained from mitotic HS are clearly the preferred practice, the visual determination of HS location is subjective, time-consuming, error prone, and thereby influences the prognostic values represented.[1434] In general, results obtained can be poorly reproducible, which can be attributed to factors such as tissue areas chosen, structures identified, and counts acquired by different pathologists. In our study, TMHS-designated HS locations co-registered topographically with mitotic HS manually selected by the pathologist, overlapping in approximately 77% of cases. This level of intermodality agreement between pathologist and TMHS machine was principally reinforced by the analysis of survival prognosis of the 30 dogs with melanoma. MF count values produced with the automated TMHS method, and independently by the pathologist, both resulted in the ability to predict patient survival differences, although intermodal discrepancy resulted in different prognostic grading for one case. By contrast, analysis of cutaneous melanoma revealed that pathologist manual versus automated HS topographies were less commonly co-registered (67%) than for the automated TMHS analysis (77%), even though skin biopsy tissue areas (range: 0.020–50 mm2) were significantly smaller than mucosal melanomas overall.[15] Furthermore, in cutaneous melanoma,[15] the clinical utility of the automated HS-acquired pHH3-immunolabeled MF counts was judged prognostically inferior to the manually derived values, a less optimal distinction compared to the TMHS algorithm performance shown here. In part, this is likely due to improvements in MF object segmentation and feature extraction made using the automated TMHS tool, contributing therefore to enhanced precision.

Low MF counts can also be prognostically informative and the automated TMHS global analysis approach provides such insight when limited proliferative capacity is manifest in biopsies. The quantifying and mapping features of some MF detection systems appear to be more constrained than TMHS. For example, testing of all tissue locations is not always completely inclusive,[21] and sparse small clusters of MF may be excluded, or counts are not yielded in each case, as further improvements in nuclear segmentation are required.[20]

The image analysis process must reproducibly distinguish mitotic cells from a variety of potentially confounding structures and artifacts in tissues. Addressing tissue artifacts, while assuring the quality of MF recognition, segmentation, and counting, remain key challenges. The automated TMHS tool confronts these issues, and essential functional reproducibility is demonstrated. Feature extraction and quantitation of pHH3-immunolabeled MF using automated TMHS were highly correlated with a second independently established OCA although it is noteworthy that the OCA does not map HS localization topography. Likewise, automated TMHS function was significantly correlated with the pathologist, an outcome requiring both MF segmentation and HS mapping precision. Together, these findings provide substantial confidence for the nuclear segmentation function and the avoidance of noise in the form of artifacts. Additional assurance was provided by qualitative review of the MF feature segmentation markup by the pathologist for all algorithms on all specimens, using the computer display. This important decision support feature is not a universal element in other HS detection methods.[14]

Tissue- and stain-related artifacts inherent in biopsy tissue sections must be addressed for proper performance. We encountered a range of artifacts that could not be filtered out by selecting a single approach; therefore, a series of filters were designed to deal with a number of common artifacts. TMHS development addressed typical artifacts hampering previous algorithms primarily incorporating commercially available CAD or other open source tools.[1519] For example, TMHS was able to circumvent melanin, hemosiderin, and other pigments, in contrast to previous tools.[1519] Red chromogenic signal development likely contributed to this. In addition, automated TMHS processing otherwise eliminated a need for supplemental postprocessing required to address false-positive HS due to sporadic off-target labeling, and the necessity to exclude tissue folds and necrosis by employing independent preprocessing steps was averted.[1935] In distinction to other systems,[1830] particular tissue artifacts that necessitated the exclusion of 8%–20% of cases altogether were not similarly problematic for TMHS mapping. Previously, artifact complications were reduced using a combination of Unser features and local binary pattern.[18] However, the tools were combined in a series of support vector machine applications of Gaussian kernel function that required machine learning. By contrast, automated TMHS proliferation quantification and HS mapping programming is a feature-based algorithm that does not comprise machine learning.

A feature-based approach requires specimens preserved and processed consistently. Tissue processing factors have the potential to impact the success of the automated TMHS algorithm and remain perennial issues across all types of tissue image analysis.[18] In the present study, there were two cases for which the analyses appeared to deviate and for which the level of agreement was less representative of the remaining samples [Figure 4c]. Anti-pHH3-immunolabeling signal appeared to be faint in some tissue areas in these two cases, which was apparent on the quality assurance review process. However, this circumstance resulted in undercount of some MFs, which were excluded by TMHS algorithm threshold settings established to subtract background off-target noise. The efficiency of the automated TMHS system rests in part on proper probe recognition of MF and its signal generation. The clinical implementation of automated TMHS will necessitate adherence to standardized preanalytic control procedures to maximize specimen integrity in preparation for tissue processing, image acquisition, and computation. Successfully accomplishing unsupervised image processing for MF HS detection is reliant upon uniform reproducible tissue labeling and signal contrast generation for efficient computer vision function.

CONCLUSIONS

HS detection and mitotic counting plays a crucial role in proliferation assessment and tumor grading.[12345678910] In this paper, we established a method of HS location and mapping that does not rely upon machine learning but by contrast is tile-based and computationally less complex in comparison to other methods which utilize algorithm processing power across an entire WSI. Identification of tissue HS, assessments of mitotic activity, and the utility of the automated TMHS functions were all shown to be clinically relevant, prognostically predictive, and highly correlated with the pathologist tasked to directly challenge the machine. Limits of agreement correlation analyses indicated that the automated TMHS method was clinically interchangeable with the pathologist using digital display of WSI. Technical quality of MF immunolabeling and IHC chromogenic signal development could be a factor in our automated computational approaches, whereas pathologist compensation for varying and regionally weak IHC was apparent in at least two cases. Notwithstanding, the overarching goal of providing improved clinically actionable prognostic decision support free from the widely recognized interobserver variability prone to traditional MF assessments[7111213141516] was achieved. The reproducibility of the automated TMHS algorithm proved superior compared to replicates among six pathologists using routine microscopy, which included identifying their own mitotic HSs as in a clinical practice setting. Utilizing the reproducible precision of the computer-assisted TMHS diagnostic mapping function will enhance pathologist tumor prognostic grading efficiency across a range of malignancies with varying clinical MF count cut points.

Financial support and sponsorship

This research was supported by the Intramural Research Program, Center for Cancer Research, National Cancer Institute, Bethesda, Maryland. This work utilized the computational resources of the NIH High Performance Computing Biowulf cluster (http://hpc.nih.gov). Commercial and academic licensing inquiries can be addressed to Dr. Eric Chen, NCI Technology Transfer Center, eric.cheng2@nih.gov; NIH reference E-038-2019-0.

Conflicts of interest

There are no conflicts of interest.