Testing of Actual Scanner Performance in a High-loaded UNIM Laboratory Environment.

BACKGROUND: Scanners are the main tool in digital pathology. The technical abilities of scanners determine the workflow logic in the pathology laboratory. Its performance can be restricted by the divergence between the scanning time presented by the manufacturer and the actual scanning time. This could lead to critical deviations from the established business processes in a 24/7 laboratory. AIM: Our investigation is focused in exploring the performance of three main models of high-performance scanners available on the Russian market: 3DHistech, Hamamatsu и Leica.
OBJECTIVES: We compared the performance of the scanners on the samples of a given size with the manufacturer's stated specifications and evaluated the speed of the scanners on the reference and routine laboratory material. SUBJECTS AND METHODS: We examined 3DHistech Pannoramic 1000, Hamamatsu NanoZoomer s360 and Leica AT2 with default settings and automatic mode. Two sets of glasses were used (glass slide): Group 1 included 120 slides with 15 mm × 15 mm slices, Group 2 included 120 workflow slides.
RESULTS: The average slide scan times in Groups 1 and 2 for the C13220 (156 ± 1.25 s and 117 ± 4.17 s) and Pannoramic 1000 (210 ± 1.64 s and 183 ± 3.78 s) differ statistically significantly (P < 0.0001). Total scanning time including rack reloading was shorter for the workflow slide set group for the modern C13220 and Pannoramic 1000 scanner models.
CONCLUSIONS: The scanner specifications provided by manufacturers are not sufficient to evaluate the performance. The guidelines and regulations concerning scanner selection should be consented by the digital pathology community. We suggest discussing criteria for evaluating scanner performance. Copyright:

INTRODUCTION

Successful implementation of whole slide imaging for routine clinical practice and COVID-19 epidemic crisis demonstrates the need to introduce digital technology into pathology for rapid integration into the chain of telemedicine consultations. The area of digital pathology was introduced in the clinical practice quite recently. As a result, the choice the right technical solution in this area is even more urgent. Most guidelines address the topic generally and suggestively, and it needs further investigation.[12345] General problem of scanner performance in the workflow was explored by Zarell et al.[2] and García-Rojo.[5]

The UNIM laboratory, founded in 2014, works as a fully digital pathology laboratory. The lab is ISO 15189:2012 certified. During the work process, it was found that the throughput of the scanners differed from the manufacturer's stated capacity. Therefore, it was crucial to evaluate the real capabilities of the equipment. We have had two Aperio AT2 scanners in service 24/7 since 2018. They are serviced once a year. The average equipment utilization rate in our laboratory is 80%. For this reason, we are constantly looking for relevant solutions on the market and regularly request equipment from all manufacturers represented in Russia for testing.

SUBJECTS AND METHODS

Three models of scanners participated in the study: Aperio AT2, 3DHistech Pannoramic 1000 and Hamamatsu NanoZoomer s360 C13220. It should be noted that these scanners have been on the market for different number of years. To compare them as identical models would be rather inappropriate. This was the reason for simplifying the comparison method.

Scanner settings

Manufacturers equip their scanners with a wide range of settings. In this study, we only used the default settings offered by the manufacturer. We are primarily interested in the performance of the equipment with preconfigured settings to minimize startup and maintenance times during the scanning process.

To equalize the working conditions of the scanners, we chose automatic mode without prescan checks. We were interested not only in the speed of scanning one slide or one rack, but also in the time it takes to rack reloading. All scanners had different rack capacities. We chose the smallest multiple of 120 slides for all scanners. General technical parameters are presented in Table 1.

Table 1

Basic characteristics of the scan modes

Parameters		Scanners
Vendor	Leica	Hamamatsu	3DHistech
Model	Aperio AT2	S360 C13220	Pannoramic 1000
Software version	102.0.7.5	1.2.2.21	2.0.3.105654
Batch type	Automatic	Automatic	Automatic
Z-stack	Single layer	Single layer	Single layer
Objective lens	x20 0.75 NA Plan Apo	x20 0.75 NA Plan Apo	x20 0.95 NA Plan Apo
Image encoding	JPEG compressed image 70% + slide information	JPEG compressed image 80% + slide information	JPEG compressed image 80% + slide information
Image depth (bit)	8	8	8
Allocation of 120 slides	4 racks of 30 slides	3 racks of 40 slides	6 racks of 20 slides

Advanced settings (threshold, focus point, etc) were used from the default presets

Samples for reference scanning

In the characteristics of scanners manufacturers indicate the scanning speed of one sample of a given size, 15 mm × 15 mm.[678] As a rule, pure scanning time of a tissue sample of this size is indicated. That is, without taking into account focusing, recording time, moving slides in the magazine, changing magazines, etc. These are justified minimum parameters to estimate the scanning speed. However, it is not enough to understand the throughput of high-performance scanners for 24/7 operation. We decided to expand the testing conditions.

Manufacturers do not specify the histological affiliation of the tissue. In our study, we used myocardial tissue. A 15 mm × 15 mm × 5 mm tissue sample from autopsy without cardiac pathology was taken to make Group 1 preparations. This is a sufficiently homogeneous tissue, which is minimally prone to artifacts when making slides. The myocardial sample was fixed in 10% buffered formalin, and the blocks and slides were made according to the standard laboratory process chain. The technological chain included devices Tissue-Tek VIP Jr, Excelsior AS, Leica R2235, Leica BOND, Tissue-Tek Prisma, and Tissue-Tek Film.

For group 1, 120 slides with a slice thickness of 2–3 μ were made. In routine work, there are usually several slices on a single slide. Based on this, we decided to simulate the maximum load when scanning surgical biopsies. For this purpose, we placed two 15 mm × 15 mm slices on each slide. Hematoxylin and eosin staining was used. For the study, we needed native format files of scanner manufacturers. The slides were numbered manually from 1 to 120, because in the technological chain of the laboratory slides with barcode are loaded immediately after scanning to the server with LIS UNIM. Saving the scans in LIS UNIM would have affected the purity of the experiment. The slides were moved between scanners with strict sequencing.

Scanning a 15 mm × 15 mm reference sample

To make a comparison with the manufacturers' stated characteristics, we took one slide with reference slices of 15 mm × 15 mm. We manually selected a 15 mm × 15 mm area [Figure 1]. Afterward, we measured the time between pressing the start button and getting the finished virtual slide for the AT2. For other scanners, we have set the full scanning time according to the manufacturer's software.

Figure 1

Scanner software interface in single area 15 mm × 15 mm scanning mode ([a] - AT2, [b] - C13220, [c] - Panoramic 1000)

Scanning 120 reference slides

In the next step, we studied the scanning of 120 slides loaded at a time with two 15 mm × 15 mm slices. The slides were distributed in racks. Racks were placed sequentially in the scanner. Afterward, the scanning process was started [Figure 2].

Figure 2

Software interface in the automatic scanning mode with two sections 15x15 mm ([a] - AT2, [b] - C13220, [c] - Panoramic 1000). The scanning area tends to the cover glass area.

To calculate the average scanning time of a slide in each group, we took the scanning time from each file for the C13220 and Panoramic 1000 models using the manufacturers' software. Since the AT2 manufacturer's software does not provide this metric, this scanner was excluded from this step. Statistical analysis of the data obtained was performed with Statistica 10.0 software (StatSoft, Tulsa, OK, USA). Data were presented as M ± standard deviation (SD). Significance of differences was assessed by the Student's t-test. The results were considered significant at 95.5% probability (P < 0.05).

The formula was used to calculate the total scanning time including the rack reloading: Tt = (T120-T1) +Td, where Tt is the total scan time in format (HH: MM: SS), T120 is the start time of the last slide scan in format (DD: MM: YYYY) (HH: MM: SS), T1 is the start time of the first slide scan in format (DD: MM: YYY) (HH: MM: SS) and Td is the duration of the last slide scan time in format (HH: MM: SS). Last slide scan time was measured with a stopwatch for AT2. The results are presented in Table 2.

Table 2

Comparison of scanning speed of 15×15 mm object with manufacturer’s data

AT2 (s)	C13220 (s)	Pannoramic 1000 (s)	Comment
60*	30	30	Scanning speed 20x for 1 slide by vendor information
71	51	51	Total scanning time 20x in research for 1 slide

*There is no marking of scanning speed for this value on the manufacturer’s site (explanation in the text)

Scanning of workflow slide set

For group 2, we selected glasses from the workflow of our laboratory, which had already been scanned without errors on AT2. The characteristics of the materials are presented in Table 3.

Table 3

Structure of Group 2, the flow material for the study

Tissue type	H and E (n)	IHC/HC (n)
Fine-needle biopsy	2	10
Gastric biopsy	14	8
Surgical biopsy	84	2
Total	120

IHC: Immunohistochemistry, H and E: Hematoxylin and eosin, HC: histochemistry (PAS, trichrome stain, Giemsa stain and etc)

Selected workflow slides were used for the examination on all three scanners. Each workflow slide stained with H and E contained at least two slices (two or more tissue samples for surgical biopsy, at least four slices for gastric biopsy, and at least two slices for fine-needle biopsy). For cases stained with ICH/HC, two slices were placed on the slide, the specimen to be examined and the control. This is an important point for estimating the scanning area of workflow slides.

Scanning errors

Accounting for scan failures is definitely an important metric for workflow scans. In this study, we eliminated the possibility of errors in the scanning process as much as possible. We selected glasses for Group 2 (workflow) that had already been scanned without errors on AT2. There were no scanning failures in Group 1 (reference).

We assume that a large study with a representative sample or a long scan cycle (24–72 h) is needed to estimate the errors. In addition, the homogeneity of the histological material (surgical, trepan, gastro, etc.) and the technical conditions of the preparation must be considered. The design of such a study requires careful preparation.

RESULTS

Scanning a reference slide 15 mm × 15 mm

Of course, the area of one 15 mm × 15 mm slice is much larger than one gastric or fine-needle biopsy. However, in workflow conditions, it is very rare to have one slice of small samples on the same slide. Thus, the scanning area in the automatic mode with default settings tends to the area of the coverslip and when scanning such slides. Therefore, we allow a scanning speed metric of one slice of 15 mm × 15 mm as the minimum necessary but insufficient.

The results presented in Table 2 show the scanning speed of one area 15 mm × 15 mm for one slide compared to the manufacturer's data.

The manufacturers specify in the technical specifications the scanning time without the focus and recording time. The official websites of 3DHistech and Hamamatsu indicate the parameter-scanning speed. Leica for the AT2 scanner says “.... scans and moves the images in record time with eSlides ready in 60 s or less for remote pathologist review....based on Aperio AT2 scanning a 15 mm × 15 mm area at ×20.”[7] The “scan speed” parameter for the AT2 is listed for the comparison with other models on the manufacturer's website. With the ×40 objective lens, the scan speed for this model is stated as “~155 s.”[9] In our study, we used only ×20 lenses. For comparison, we left the value of 60 s from the official page of the AT2 model.

Scanning 120 reference and workflow slides

For the C13220, the mean scan time (M ± SD) of the slide in Group 1 was 156 ± 1.25 s and 117 ± 4.17 s in Group 2. For the Pannoramic 1000, the average scan time (M ± SD) of a slide in Group 1 was 210 ± 1.64 s and 183 ± 3.78 s in Group 2. The average scan time per slide with two 15 × 15 slices (Group 1) and a workflow slide set (Group 2) differed statistically significantly (P < 0.0001) for both scanners.

AT2 was excluded from this comparison because the manufacturer's software does not provide scan time data for each slide.

The average scan time data does not take into account the time to reload the rack. Our main goal was to compare the performance of the scanners with regard to rack reloading. The total scan times calculated by the formula are shown in Table 4.

Table 4

Results of calculated total scanning time with rack reloading of 120 slides with two 15×15 mm slides (Group 1) and workflow slide set (Group 2)

Scanner model	Group 1 (HH: MM: SS)	Group 2 (HH: MM: SS)	Differences between groups (HH: MM: SS)
C13220	05:29:52	04:20:05	01:09:47
P1000	07:35:02	06:35:24	00:59:38
AT2	06:50:34	07:01:47	00:11:13

DISCUSSION

We assume that the methodology for scanner testing should be approximated to real-world usage scenarios in heavily loaded digital laboratories. The “scan speed” and “throughput” characteristics without detailing the methodology can lead to the errors in business process planning.

The average scan time in Group 1 is longer than in Group 2 for the C13220 and Pannoramic 1000 models. The significant difference between Groups 1 and 2 can be explained by the heterogeneity of the slice sizes on all 120 slides in Group 2 [Table 3].

The difference between the groups in Table 5 for the C13220 and Pannoramic 1000 models was approximately 1 h. For AT2, it was approximately, 11 min. This discrepancy is most likely due to the different age of these models and the features of the software settings. The new models use improved algorithms to adapt the scanning process in the automatic mode.

Table 5

Comparison of scanner throughput data

AT2 (slides/h)	C13220 (slides/h)	Pannoramic 1000 (slides/h)	Comment
~50	>82	Up to 100	Sustained throughput 20x by vendor site information
18	22	16	Group 1 (including rack reloading)
17	28	18	Group 2 (including rack reloading)

Information from the manufacturer’s website (first line) and calculated throughput for 120 reference and workflow slide sets are given

Scanning speed depends on many factors: Sample size, number of focus points, threshold values, etc. If the laboratory has the ability to rank the material and apply presets for different types of material, it can significantly reduce the scanning time. This scheme can be implemented if several scanners are available in the laboratory, or if there is a scheduled inflow of the same type of material. Unfortunately, such conditions are rare. Thus, users are forced to allocate a specialist to scan and experiment with the settings.

Scanner throughput

Based on the measurements from Table 4, we calculated the estimated throughput of the scanners, taking into account the time to rack reloading. The formula of the proportion x = n × 3600/t, where × is the number of slides per hour, n is the number of scanned slides, 3600 is 1 h converted to seconds and t is the total scanning time for n in seconds. All calculated values were rounded to a whole value.

One can disregard such notations on the manufacturers' website as “~”, “more than”, “up to,”[678] because a spread of 10 slides could be considered insignificant. However, if we take the average time of scanning one slide 2–3 min in real circumstances, then in the workflow condition, the shift in the time of readiness of scans for viewing by a pathologist will be significant. These nuances must be taken into account if the entire laboratory material is digitized in 24/7 mode and there are routine consultations with specialists working in different time zones.

It is generally known that this model of operation (full-scale digitization) is currently affordable to a limited number of laboratories. However, we would like to share our vision of potential problems in the process chain. They can lead to errors in business processes and consequently cause reputation and/or economic loss.

On the one hand, each laboratory needs to carefully study scanner settings and adjust them to their own needs. This will help significantly reduce scanning time and the amount of information stored. In this case, a repository of settings that other laboratories have tested under their own conditions can be a useful solution for the community. On the other hand, digital-only laboratories should not have to spend time searching for such settings on their own. The challenge for manufacturers is to provide ready-made set of settings for different scanning scenarios.

CONCLUSIONS

With our article, we want to start discussing the methodology for testing scanners in a real workflow, using metrics that will be helpful for choosing a scanner for laboratories of different levels. Our goal is not to identify the absolute leader and focus on it in the future, but to evaluate the features of different models as fully as possible under load testing, close to the real work of a digital laboratory, working in the continuous mode.

In the absence of a uniform standard and mass experience with 24/7 scanning microscopes, it makes sense to consider a multivendor approach. This would increase flexibility and reduce dependence on one vendor.

Hopefully that a consensus will emerge from the digital pathology community to evaluate the scanner performance. Moreover, these characteristics will be specified by scanner manufacturers in the official documentation.

Our suggestions for a basic testing methodology:

Use a single 15 mm × 15 mm tissue type as the minimum reference set (similar to the manufacturer)

To use a reference set with two 15 mm × 15 mm slices on a single slide as the basic set

To use the smallest total multiple of slides for one series of slides. At the moment, 120 is suitable for most high-performance scanners

To use an additional set that simulates workflow slides (10 trepanobiopsy slides, 10 gastric biopsy slides, 10 skin slides, etc.)

To specify the number of slices per slide to simulate workflow slide set

To evaluate two scanning modes: Full automatic and manual. The automatic mode is designed for fully digital laboratories with minimal involvement of an employee in the scanning process (“walk away” scanning). Manual mode is designed for small laboratories or for material, requiring attention during scanning. It helps to increase scanning accuracy and reduce slide size

In both modes to take into account, the total time taken to get one virtual slide or a series of slides

To specify file recording characteristics (hdd/ssd, internal/external, lan/thunderbolt, raid massive). We believe that the next generation of scanners will use NAS and cloud technologies as storage.

To conclude this part of our study of scanner characteristics, we want to emphasize that our goal was not to identify a good or bad scanner. We suggest that the digital pathology community discuss objective criteria for scanner specifications that affect laboratory performance.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.