Real-time whole slide mosaicing for non-automated microscopes in histopathology analysis.

CONTEXT: Mosaics of Whole Slides (WS) are a valuable resource for pathologists to have the whole sample available at high resolution. The WS mosaic provides pathologists with an overview of the whole sample at a glance, helping them to make a reliable diagnosis. Despite recent solutions exist for creating WS mosaics based, for instance, on automated microscopes with motorized stages or WS scanner, most of the histopathology analysis are still performed in laboratories endowed with standard manual stage microscopes. Nowadays, there are lots of dedicated devices and hardware to achieve WS automatically and in batch, but only few of them are conceived to work tightly connected with a microscope and none of them is capable of working in real-time with common light microscopes. However, there is a need of having low-cost yet effective mosaicing applications even in small laboratories to improve routine histopathological analyses or to perform remote diagnoses. AIMS: The purpose of this work is to study and develop a real-time mosaicing algorithm working even using non-automated microscopes, to enable pathologists to achieve WS while moving the holder manually, without exploiting any dedicated device. This choice enables pathologists to build WS in real-time, while browsing the sample as they are accustomed to, helping them to identify, locate, and digitally annotate lesions fast.
MATERIALS AND METHODS: Our method exploits fast feature tracker and frame to frame registration that we implemented on common graphics processing unit cards. The system work with common light microscopes endowed with a digital camera and connected to a commodity personal computer. RESULT AND
CONCLUSION: The system has been tested on several histological samples to test the effectiveness of the algorithm to work with mosaicing having different appearances as far as brightness, contrast, texture, and detail levels are concerned, attaining sub-pixel registration accuracy at real-time interactive rates.

INTRODUCTION

The possibility to have a large overview of the relevant histological features of tissue sections without losing fine resolution details is a key feature of modern histopathology analysis. Most today's automated microscopes allow increasing the Field of View (FOV) of the specimen under analysis by building a mosaic of assembled images, called also Virtual Slide (VS) or Digital Slide (DS), by exploiting a motorized stage or a dedicated external slide scanner.[1] Here, the motorized XY- tables can be used to create mosaics even from thousands tiles of the specimen captured separately. Usually, these tiles are processed in batch and the final mosaic is built at the end of the working session. Nevertheless, often the whole mosaic still requires post-processing to compensate artefacts due to uneven illumination as well as stitching errors caused by the finite precision of the motorized stage.

Recent advances in mosaicing techniques, commodity personal computer's (PC) processing power and data storage availability have given rise to a Virtual Microscope (VM) approach.[23] The VM system allows a large storage of Mega or even Giga-pixel microscopy data to be accessed on-line using visualization systems that simulate a real microscope by positioning the Region of Interest (ROI) within the Whole Slide (WS) as well as setting a virtual objective magnification according to the recorded resolution and quality. The availability of a VM opened doors not just to telepathology,[4] but it has relevant applications in double blind studies and second opinion diagnoses, education, and teaching.[5]

Since the whole sample acquisition at a high resolution could require a long time, automated microscopes can be programmed for night acquisition, making the whole histological sample available the day after for histopathological analysis.

However, even though automated microscopes and DS scanners are performing faster due to technology improvements, nowadays most of the histopathology analyses still rely on non-automated microscopes. In case of standard microscopes, the common practice is to add a motorized stage.[6] Nevertheless, camera alignment and stage calibration are difficult tasks to be performed without using dedicated pattern matching algorithms. Moreover, stage calibration tends to drift with time and requires periodic adjustments in order to prevent misalignments in the sequence of the acquired tiles. Some authors proposed a semi-automated approach to extend the FOV in non-automated microscopes through image mosaicing.[7] Authors build an on-line mosaic by first letting the user stitch the images in a coarse slide, then refining the whole mosaic through an automatic technique by means of least squares model fitting. Although the method has sub-pixel accuracy, it does not perform in real-time and requires specialized user intervention. Some authors proposed an image mosaicing method based on sparse feature point extraction, projection profile alignment, and wavelet-based blending to deal with microscopy images.[8] The system automatically mosaics all the images acquired from a specimen, also correcting shading artifacts, but it requires an automated XY-table to achieve the final mosaic. Other authors propose an image mosaicing method for non automated microscopes relying on KLT (Kanade-Lucas-Tomasi) feature tracker to infer the transformation between each pair of images.[910] This approach is necessary when external information regarding the stage position is unavailable: The overlapped region between two consecutive images could occur at any point. However, such methods alone do not offer sufficient robustness in image microscopy, due to the presence of self similar regions in low-contrast images with few corner points, especially at a high magnification. Despite the simple transformation model adopted, further processing steps are required to increase the confidence in the final transformation, such as a Phase Correlation approach, which on the other hand is time consuming.

It is worth pointing out that the techniques used to achieve mosaics are strictly application-dependent. For instance, mosaicing is used to build whole slides in microscopy as well as panoramas with today's smart phones. In this latter case, color and details can be missed without losing visual quality (this is also the basic principle of lossy compression), small deformations can be allowed, since the purpose is just building a visually pleasant panorama. Accordingly, low accuracy image registration and blending techniques can be employed, and they are so simple to allow implementation even on power bounded central processing units (CPU) for mobile devices. Nevertheless, resolution and quality constraints required for mosaics in digital pathology are far from these simplistic cases (this is also the reason why biomedical image formats require lossless compression by default). Besides, usually images are much bigger and with a lack of “structures” that makes the registration task much harder to be performed, yet more when real-time performance is required. Moreover, small laboratories often cannot afford a DS scanner or even a motorized stage.

MATERIALS AND METHODS

To enable the use of VM techniques for histopathology analysis even in case of a manual stage, we have developed a fully automatic image mosaicing method which is able to build the DS in real time, while the user is exploring the sample by manually moving the holder of the specimen, thus offering crucial improvements in routine tasks. First, the specimen can be examined (even remotely) by other pathologists as soon as only interesting regions have been constructed. This enables quick second opinion diagnoses and double blind studies. Besides, it permits to efficiently build a reliable ground truth, which is as necessary as almost difficult to achieve when developing Computer Assisted Diagnosis (CAD) system. Second, there is no need to achieve the whole slide; rather, pathologists are free of spotting the relevant sample regions without waiting for a night scheduled automatic acquisition. This allows widening the FOV in the ROIs only, which would be impracticable with an on-line or automated WS acquisition. Finally, this method speeds up the diagnosis, keeping track of the regions observed either for recording purposes or just to keep tracks of visited regions, so that they cannot be mistaken as unexplored. Contrarily to what has been reported so far, our method is not limited in the output mosaic dimensions, neither for the need to process the input images off-line. Nevertheless, the size of the histological images being registered on a DS requires intensive computational demands for processing, storage, and viewers’ capabilities. To achieve real-time mosaicing, our algorithm has been conceived to use sparse features and frame-to-frame registration, which can be implemented in graphics processing unit (GPU). A correction of the illumination field is performed during the frame stitching, thus producing a mosaic with even illumination. Results show that our dead reckoning approach yields good registration accuracy even for looping paths, a common and well-known problem in on-line frame-to-frame image registration approaches, where registration errors tend to accumulate.

Our method is compliant with the real-time acquisition of images from a digital camera attached to the microscope where the stage holder is positioned manually. Since the stage moves almost perpendicularly to the camera's optical axis, any two consecutive images can be related by a translative model. Moreover, the large aperture of the microscope objectives limits the geometric distortions, which can be considered negligible. On the other hand, artifacts due to an uneven lighting condition have to be faced with a proper method.

Image Registration

The mosaic is composed through real-time image registration techniques, exploiting a certain overlap between the captured images which are aligned both from geometric and photometric points of view, thus achieving a seamless stitching. The image registration steps can be summarized as follows:

Feature Detection

We adopted a feature-based approach relying on keypoint descriptor matching. To reduce the computational burden, we use the Features from Accelerated Segment Test (FAST) corner detector[11] for its high repeatability and speed in corner extraction. We make use of Binary Robust Independent Elementary Features (BRIEF) binary local descriptor,[12] representing comparisons of pixels inside a patch, that are fast to compute, fast to match and memory efficient. We choose 64 bytes as the descriptor length, which has shown to outperform other descriptor-based features. The similarity based matching is efficiently computed using the Hamming distance, exploiting exclusive or (XOR) and Single instruction, multiple data (SIMD) on GPU or CPU.

Frame to Frame Registration

Each frame is registered with the subsequent one through matching their local descriptors. A robust approach like RANdom SAmple Consensus (RANSAC) is used to achieve the final transformation by fitting the geometric model whilst removing outliers.[13] For large images (i.e., greater than 1200 × 1000 pixels), a multi resolution approach has been devised: Down-sampled images are used to achieve the guess transformation, which is then refined using only limited patches of the original images.

The acquisition frame rate influences the user interactivity of the application: A too low frame rate (i.e., lower than 5 fps) would cause couples of frames to overlap only partially when the user moves the stage position knob. This would possibly lower the registration accuracy, which could degrade the final mosaic. On the other hand, a high acquisition frame rate (>25 fps) would stress too much the registration algorithm, which would simply register frames with a minimum displacement (few pixels). The latter case would not be feasible in practice, since acquisition and processing frame rates are related and they compensate each other: A high frame rate would need more CPU resources, thus lowering the overall frame rate. Based on our experience, we found a minimum overlap of 20% to be necessary to register consecutive images and we observed a 10-25 fps range for the whole system operations (acquisition and processing)–also depending on the shutter time and imaging conditions–this being enough in terms of user interactivity.

As far as the algorithm's parameters are concerned, they do not need tuning for most of cases, since their values have been determined through our experience gathered after extended studies with a huge number of different cases. However, one parameter remains that is quite sensitive, that is the threshold of the consensus for the RANSAC algorithm. In fact, too a low value would likely lead to accept false transformations, which could invalidate the mosaic. Contrarily, too a high value would prevent most frames from being registered. For this reason, we chose to keep this threshold quite high (above 80%), using appropriate fail recovery strategies when needed.

Fail Recovery

As a matter of fact, not all the frames can be registered, because either the transformation has a weak consensus or simply no transformation exists at all. The former case could be due to the tracker being misled by a set of corner points belonging to similar image regions, whereas the latter occurs when subsequent frames do not overlap, for example, due to the user moving the stage too fast compared with the acquisition frame rate. In the first case, we perform a re-projection of newly extracted corners from the previous image and a re-extraction of the descriptors in the current one by using the weak consensus inliers. This filters out most of the outliers and if the transformation is correct, a stronger consensus is achieved. Otherwise, the current frame is discarded to prevent inconsistency in the mosaic and a visual feedback is given to the user to help him/her realigning the image in the FOV with the mosaic being constructed.

Mosaic Composition

Once the transformation has been successfully recovered, each frame is warped with bilinear interpolation into a common reference frame. Usually, interpolation and blending are employed to render the images in a pleasant form, trying to compensate for lens distortions, uneven light field or registration inaccuracy.

Several approaches are available to correct the vignetting effects. First of all, in a previous work, we have shown that using empty field is the best method with histological samples.[14] Therefore, the first choice could be to acquire the glass slide beforehand, but this would reduce the operational simplicity we aim to provide to the user. Actually, this could not even be necessary. In fact, to compute the illumination field, we build an empty field by collecting a sequence of images[15] at start-up to collect the background parts of the image free of tissue, according to what explained in a previous work.[16] Once the background is achieved, it is used to estimate the vignetting function, V.[17] Finally to correct the vignetting effect and remove stitching artifacts, we recomputed each input image I, thus achieving a new image I’ = (I/V)×μ, where μ controls the white equalization, being set to the luminance value.

To avoid an upper limit on the final mosaic size, we devised an optimized tile-based image stitching algorithm, which builds the mosaic using a limited amount of memory and stores the mosaic rendering buffer in tiles to disk when it is no longer needed. The mosaic can be explored through a Graphical User Interface with interactive pan and zoom capabilities, by exploiting the stored tiles and their mipmaps.

RESULTS

Firstly, the accuracy of the algorithm is assessed. In Figure 1a, a sequence of 175 histological images (640 × 512 pixels) has been acquired by manually moving the stage to build a mosaic whose final size is of 7800 × 5570 pixels. The presence of the looping path enables us to assess the accuracy of the registration algorithm on the common region when the path closes [Figure 1b]: As one can see, the stitching is seamless. By registering also the first frame at the end of the sequence, the error drift accumulated during the registration can be assessed by concatenating all the transformation matrices.

Figure 1

(a) Mosaic of a histological sample composed of 175 images in a looping path to test dead-reckoning cumulative error; (b) A detail in the closing path region

Since the model is assumed to be translative, the result is the sum of each recovered offset along the x-y plane. In the ideal case, this sums up to zero. Our algorithm achieves a dead reckoning error of (0.64; 0.71) pixel along x and y, respectively.

Therefore, sub pixel accuracy is attained even considering such a long path. As far as time performance is concerned, the frame registration works at 23.6 frames per second (fps) on an Intel i3 PC with a common GPU card. No frames have been discarded during the images registration process. In Figures 2a and b, two images of 1600 × 1200 pixels, acquired by a Polaroid MC2 digital camera with a 20× objective, are shown. These have been manually annotated by the pathologist to provide the ground truth to the training stage of a CAD. Since images share a common region, this results in a double annotation. We can see that the rightmost region in (a) is segmented differently from the same on the left in (b). Although the difference could seem limited, this could provide different segmentation results and mislead the CAD's classifier.

Figure 2

(a, b) Two subsequent original images of a histological sample; (c, d) The mosaics, with channels equalization, annotated by the pathologists

The whole sequence of images, corrected for illumination and registered, is shown in the mosaic of Figure 2c. The total size is of 15,842 × 13,926 pixels, covering about 4 × 3.6 mm2. Here, the pathologist made coherent segmentations. It is worth noticing how generating this mosaic into a single bitmap on a consumer PC would not have been feasible. Rather, our tile-based mosaic warping method generated 58 tiles, of 4,096 × 2,048 pixels each.

In Figure 2d, another histological sample, made of 28 images acquired with the MC2 camera, is presented. The mosaic, annotated by the pathologist, is of 5,492 × 6,262 pixels and covers a FOV of 1.4 × 1.6 mm2. Here, pathological regions that could be even larger than the single frame have been consistently marked. To emphasize the difference between single image and whole mosaic annotations, Figure 3a shows a composition of original annotated images for a bladder sample, built by registering the single annotated images through the same transformation matrices used to perform the mosaic. Lighter regions represent the overlapping areas between subsequent frames. Here, the mismatch shown in the (light yellow) lines drawn by the pathologist to bound the regions of cancerous cells (see the red squares for details) disappears when using the mosaic [Figure 3b], without channels equalization μ). This proves the need of a mosaic in order to eliminate multiple annotations of same regions and intra-observer variability, accordingly.

Figure 3

Composition of images of a stained bladder tissue annotated by the pathologist one at a time (a) with some mismatches highlighted in the red squares (1-3). The mosaic viewed through our software (b) ready to be annotated by the pathologist

CONCLUSIONS

We presented the first feature-based method conceived for non automated microscopes, which permits real-time mosaicing during sample exploration. Contrarily to WS automated acquisition systems, which require dedicated hardware and, in some cases, a night scheduled acquisition, our approach can be used on conventional light microscope systems, as those employed routinely in today's histopathology analysis. Experiments carried out on several histological specimens prove that the method is fast, accurate and reliable, allowing histopathologists to identify and locate lesions or relevant regions in a consistent way, also providing a recorded track of the analysis. Unlike on-line whole slide approaches, our real time mosaicing can be even performed on selected regions only, thus enabling prompt automated analyses performed at interactive rates.

Our systems are actually being tested as an assisted tool for pathology analyses in two Italian Centres for cancer research and treatment, which will give us a valuable feedback as well as more challenging cases to be considered in future work to extend the capability of our system.