How innovations in methodology offer new prospects for volume electron microscopy.

Detailed knowledge of biological structure has been key in understanding biology at several levels of organisation, from organs to cells and proteins. Volume electron microscopy (volume EM) provides high resolution 3D structural information about tissues on the nanometre scale. However, the throughput rate of conventional electron microscopes has limited the volume size and number of samples that can be imaged. Recent improvements in methodology are currently driving a revolution in volume EM, making possible the structural imaging of whole organs and small organisms. In turn, these recent developments in image acquisition have created or stressed bottlenecks in other parts of the pipeline, like sample preparation, image analysis and data management. While the progress in image analysis is stunning due to the advent of automatic segmentation and server-based annotation tools, several challenges remain. Here we discuss recent trends in volume EM, emerging methods for increasing throughput and implications for sample preparation, image analysis and data management.

INTRODUCTION

Method development is a key factor in accelerating biological discovery. Advances in imaging techniques have fulfilled the desire of biologists to unravel the structure and function of biological systems across a wide spectrum of spatial scales. Electron microscopy (EM) is especially suited for this goal. With its high resolving power, the structure of tissues can be revealed down to the nanoscale. This makes it a useful tool for determining the wiring patterns of neurons, but also the detailed investigation of cell organelles, such as microtubules, mitochondria, ER and extracellular vesicles.

The imaging of tissues with EM has a long history of development. The protocols and techniques used to prepare the specimen, initially intended for transmission electron microscopy (TEM), were developed in the early 1940s. Serial section transmission electron microscopy (ssTEM) was introduced in the 1950s to provide a three‐dimensional context of the tissue. , More than half a century later, resin‐embedded tissue samples are still cut into thin sections (albeit much thinner than before) and subsequently imaged with TEM. , Until the introduction of computer‐assisted methods in the 1970s, 3D reconstructions of tissue had to be done entirely by hand. For this reason, and because of the extensive manual labour involved in cutting and handling sections, ssTEM applications remained quite limited.

Innovations in the 2000s led to more automated and routine EM techniques for 3D reconstructions of tissue (Table 1), thereby establishing a new research field: volume electron microscopy (volume EM). As an alternative to ssTEM, serial section electron tomography (ET) was introduced, , , in which a tomographic reconstruction of each serial section is made. Scanning electron microscopes (SEM) allowed for the cutting device to be integrated into the microscope, leading to serial blockface scanning electron microscopy (SBF‐SEM) and focused‐ion beam scanning electron microscopy (FIB‐SEM). , While both offer better axial resolution than ssTEM, they lack the high lateral resolution of TEM and destroy the sample during acquisition. Combining serial sectioning with SEM led to the development of serial section SEM, also known as array tomography (AT). , Additionally, Automated Tape‐collecting Lathe UltraMicrotome (ATLUM, later combined into ATUM‐SEM) allows consistent collection and handling of thousands of serial sections. ,

TABLE 1

Volume EM methodology

Methodology	Description	Typical x, y resolution (nm)	Typical z resolution (nm)	Typical volume (μm3)	Reference
Serial section transmission electron microscopy (ssTEM)	Serial sections are cut with a diamond knife and collected in a water bath, (manually) transferred to a support grid and imaged with transmission electron microscopy. Relatively inexpensive but requires high skill and labour.	4	50	103–104	10 , 11
Electron tomography (ET)	Tomographic reconstruction of 200–1000 nm thin sections by recording at multiple tilt angles. Limited to small volumes.	2–10	2–10	102–103	13 , 14 , 15
Serial block‐face scanning electron microscopy (SBF‐SEM)	Automated method with microtome inside vacuum chamber. Iteratively a thin layer of material (down to 20 nm) is removed from tissue the block, after which the surface of the block is imaged and the scattered electrons are recorded.	10	30	105–107	16
Focused ion beam scanning electron microscopy (FIB‐SEM)	A very thin (2–5 nm) layer of material is iteratively removed by a focused gallium ion beam positioned 45° with respect to the sample, after which the top of the block is imaged. Offers high isotropic resolution.	5	5	102–105	17 , 18
Automated tape‐collecting ultramicrotome SEM (ATUM‐SEM)	Serial sections are automatically cut by a microtome and collected from a water bath on Kapton tape by a computer‐controlled reel‐to‐reel conveyer belt mechanism. Sections are consecutively imaged with SEM.	Flexible	60	107–1010	21 , 22
Array tomography (AT)	Ribbons of serial sections are collected on solid surface (silicon wafer, glass) and imaged consecutively with SEM.	4	30, 60	104–106	19 , 20

Note: The development and application of these methods is reviewed in Refs. (12), (38), (175) and (176).

Volume EM techniques have been successfully applied in various fields, such as connectomics research (i.e. mapping the connections between neurons), virology , and cell biology. However, when considering the imaging of ‘large’ volumes (>106 μm3), the aforementioned EM approaches quickly run into their limits as the throughput of modern electron microscopes remains low. The imaging and reconstruction of larger volumes can take up to several months or years in some cases. , , , Additional challenges include the increased risk of acquisition errors and loss of material during long acquisitions, generation of very large data sets (hundreds of terabytes) and enormous manual annotation efforts. , As a consequence, every volume EM study is a tradeoff between resolution, acquisition speed, long‐term system stability and the effort needed in annotation.

Despite these challenges, new light is shining on the volume EM field. Powerful electron microscopes with unrivalled acquisition speeds have recently made their entrance. , At the same time, the throughput of existing methods has increased significantly by advancements in software and hardware. , , , , , Years of imaging with conventional systems could now in principle be reduced to a few weeks. In this review, we analyse trends in volume EM and focus on the specific improvements in methodology that relieve the bottleneck in throughput of electron microscopes. We then discuss the implications of these developments for sample preparation, image analysis and data management respectively.

TRENDS IN VOLUME ELECTRON MICROSCOPY

To distil general trends in volume EM, we summarised relevant statistics from a pool of over 200 EM volumes from 115 unique studies (Figure 1) conducted between 2009–2021, including those covered in earlier reviews. , , It is inevitable that certain statistics are missing from a number of studies as certain data set parameters such as volume size, voxel resolution and data set size are not consistently reported and have yet to be standardised (connectomics studies being an exception ).

FIGURE 1

Overview of VEM studies from 2009–2021 reporting voxel resolution, method, volume and data set size. (A) Voxel size (x vs. z) and (B) volume size for all data sets. Ellipses indicate application regimes. (C) Cumulative sum of studies per method. (d) Increase in data set size per year. All data points represent a single data set, except those marked by * and +, which are targeted re‐acquisitions from Kasthuri et al. and Hildebrand et al. respectively

We searched and grouped studies based on the used techniques: serial section transmission electron microscopy (ssTEM), serial blockface scanning electron microscopy (SBF‐SEM), focused ion beam scanning electron microscopy (FIB‐SEM) or automated tape‐collecting ultramicrotome SEM (ATUM‐SEM). The latter can be considered as a subset of array tomography, but whereas array tomography is also frequently associated with light microscopy, ATUM‐SEM is a more dedicated volume EM technique. Certain application regimes can be distinguished (Figure 1A and B). FIB‐SEM is clearly in the high resolution but low volume regime, whereas ssTEM studies typically target large volumes with high lateral resolution – suitable for use in connectomics. ATUM‐SEM and ssTEM show great flexibility in the volume size and resolution, because they allow re‐imaging of parts of the sample with different settings. SBF‐SEM is a ‘mid‐range’ method, covering a volume range from roughly 104 to 107 μm3. Additionally, the number of volume EM studies is increasing at a steady rate. The majority of studies has been conducted using SBF‐SEM or FIB‐SEM (Figure 1C). Although the number of studies per year varies quite a bit, there is a clear trend towards bigger data sets (Figure 1D).

The push towards larger volumes can be explained by connectomics research. Scientists have fully reconstructed the nervous systems and determined the connectomes of small organisms, and partially in bigger organisms (Table 2). From the smallest (C. elegans larval brain ) to the largest volume (mouse visual cortex ) at full resolution, the size difference is more than five orders of magnitude. While connectomics research can be considered a driver for innovation in the field, the application of volume EM is linked to several other research fields. , , , , , We found over 110 distinct applications in 23 different organisms, including animals (and their larval stages), plants, bacteria and cell lines (Supplementary Table S1). Some studies feature reconstructions of single‐cell organisms and small organisms such as the budding yeast (S. cerevisiae), parasite Trypanosoma brucei or the ringed worm (P. dumerilii) at 6 days post‐fertilisation. Some of the larger samples are intersegmental vessels and dorsal‐lateral lanastomotic vessels in zebrafish embyos (FIB‐SEM), root tips of the barrelclover (M. truncatula), (SBF‐SEM), human and mouse fibrous connective tissue (SBF‐SEM) and mouse liver tissue (FIB‐SEM).

TABLE 2

Relevant imaged volumes in connectomics research

Organism/species	Stage	Tissue	Method	Connectome determined	Volume size (μm3)	Data set size (TB)	Reference(s)
C. elegans (Roundworm)	Larva L3	Hermaphrodite full brain	SSTEM	Fully	15,707	–	40
	Larva L2	Hermaphrodite full brain	ATUM‐SEM	Fully	15,351	–	40
	Larva L1	Hermaphrodite full brain	ATUM‐SEM	Fully	5148	–	40
	Larva L1	Hermaphrodite full brain	SSTEM	Fully	4251	–	40
	Larva L1	Hermaphrodite full brain	ATUM‐SEM	Fully	2989	–	40
	Larva L1	Hermaphrodite full brain	ATUM‐SEM	Fully	4536	–	40
	Adult	Hermaphrodite CNS	SSTEM	Fully	–	–	177
	Adult	Hermaphrodite full brain	SSTEM	Fully	52,382	–	40
	Adult	Hermaphrodite full brain	ATUM‐SEM	Fully	73,850	–	40
	Adult	Male CNS	SSTEM	Fully	724,776	–	178
C. intestinalis (Sea squirt)	Larva	Central nervous system	SSTEM	Fully	–	–	179
D. melanogaster (Fruit fly)	Larva	Central nervous system	SSTEM	Partly	1,755,837	–	180
	Adult	Central brain	FIB‐SEM	Fully	1,562,5000	26	28
	Adult	Full brain	SSTEM	Partly	79,150,500	106	27 , 132
	Adult	Mushroom body	FIB‐SEM	Fully	240,000	3.8	181
	Adult	Olfactory system	SSTEM	Partly	250,000,000	50	75
	Adult	Ventral nerve cord	SSTEM	Partly	21,000,000	172.6	37
D. rerio (Zebrafish)	Larva	Entire brain	ATUM‐SEM	Partly	10,200,000,000	4.4	26
	Larva	Hindbrain	ATUM‐SEM	Partly	1,404,480	–	182
	Adult	Spinal segment	SBF‐SEM	Fully	11,716,726	–	183
H. Sapiens (Human)	Adult	Cerebral cortex	ATUM‐SEM	Partly	1,000,000,000	1400	41
M. musculus (Mouse)	Adult	Cochlea (inner hair cells)	SBF‐SEM	Partly	6,841,300	0.194	184
	Adult	Cochlea (inner hair cells)	SBF‐SEM	Partly	7,796,872	–	184
	Adult	Cortex layer 4	SBF‐SEM	Fully	542,510	–	31
	Adult	Neo cortex	SSTEM	–	1,000,000,000	2000	29
	Adult	Neo cortex	ATUM‐SEM	Partly	130,000,000	0.3	72
	Adult	Somatosensory cortex	ATUM‐SEM	Partly	30,500,000	0.14	185
	Adult	Visual cortex	SSTEM	partly	30,375,000	100	65
	Adult	Visual cortex	SSTEM	Partly	8,190,000	–	35
	Adult	Visual cortex	SSTEM	Partly	1,049,022,720	2000	45
	Adult	Visual Thalamus	ATUM‐SEM	Partly	67,200,000	100	73

The number of volumes with high isotropic resolution is also increasing. Abnormalities in cell organelle structure and function are implicated in the development of diseases, which can be studied in detail with FIB‐SEM. Volume EM studies with FIB‐SEM have resulted in high resolution 3D reconstructions including (but not limited to) HeLa cells, T cells and macrophages, , , cancer cells, mouse primary beta cells and COS‐7 cells. Moreover, studies have been performed on human cardiac telocytes, mouse liver tissue and lung alveolar epithelium. Studies of abnormal ultrastructure are emerging, including breast carcinoma and pancreatic adenocarcinoma.

In short, volume EM applications have expanded well beyond the scope of connectomics, and the various techniques can be demarcated into distinct application regimes. Data sets are increasing in size and becoming more diverse. We will show later that some trends can be attributed to specific developments in hardware (Section 3), while others may be a result of the general increase in popularity of volume EM and access to better equipment. We will now layout the new developments in methodology which have contributed to some of the trends that are described here.

IMAGING OF LARGER BIOLOGICAL VOLUMES

A major feat in volume EM would be to routinely image volumes larger than 1 mm3 at nanometre resolution in a few months. Achieving this is not only a matter of improving speed; instrumentation must be able to robustly image thousands of tissue sections or slices for extended periods with minimal intervention. Therefore, instrumentation development has focused not only on increasing imaging speed, but also robustness and automation. New developments can roughly be divided into four groups: (1) parallelisation by multiple beams, (2) parallelisation by multiple cameras, (3) parallel processing in block‐face imaging and (4) re‐imaging of volumes at different resolution scales (Figure 2).

FIGURE 2

Four different methods for scaling up volume EM studies: (A) Multibeam electron microscopy. Throughput is increased by using multiple beams in parallel. (B) TEMCA and AutoTEM principle. Throughput increase by multiple parallel cameras to enlarge the field of view of the microscope. (C) ‘Enhanced FIB‐SEM’: ultrathick sectioning is applied to a sample that is too thick to be handled by a single FIB‐SEM. Throughput increase is achieved by higher system stability and using multiple FIB‐SEMs in parallel. (D) Single‐beam multiscale EM. ‘Increased’ throughput by scanning a large area at low magnification followed by multiple rounds of targeted acquisition at higher resolution

Multiple scanning beams in parallel

The imaging speed in (volume) EM is limited by the minimum signal‐to‐noise ratio (SNR) needed to make biological features sufficiently visible against a noisy background. The SNR is influenced by the exposure time, beam current, sample contrast and detection efficiency. In order to achieve faster imaging, it seems straightforward to increase the beam current. However, this leads to lower resolution due to increased coulomb interactions and can go at the cost of sample charging, inducing sample drift and artefacts. A workaround would be to use multiple beams in parallel. This idea has led to the development of multibeam scanning electron microscopy (MB‐SEM). , An MB‐SEM scans the sample simultaneously with an array of beamlets produced by a single electron source (Figure 2A), increasing the acquisition speed proportionally to the number of beamlets with theoretically no compromise on resolution compared to single‐beam SEM. Multiple concepts have been developed for multibeam electron microscopy with different source and column configurations, beam array sizes and detector systems. , , ,

The first commercially available MB‐SEM (MultiSEM) was released in 2015, , producing 61 or 91 beams in a hexagonal pattern (Figure 3A). The primary beams originate from a single source and go through a single column, where they are separated from secondary electron signals by a magnetic beam splitter. Each secondary electron signal is lead to a dedicated secondary electron detector. The number of beams can be increased without changing the primary design; a 331 beam version has subsequently been developed, though it is not yet commercially available.

FIGURE 3

Different multibeam systems. (A) Zeiss MultiSEM, which makes use of secondary electron detection by a multidetector. The primary beams and detected electrons are separated by a magnetic beam splitter. (B) Delft Multibeam (FAST‐EM). , Instead of secondary electron detection, this multibeam system detects transmitted electrons via conversion to a light signal by a scintillator located directly under the sample

A single‐source 196‐beam MB‐SEM was developed at Delft University of Technology. This model employs transmission electron detection instead of direct secondary electron detection. The sample is placed on a luminescent material (scintillator) coated with a thin conductive layer which converts the electrons to photons. The light beams are then imaged onto a detector array , (Figure 3B). A dedicated 64‐beam MB‐SEM system (FAST‐EM) using this technology has recently been commercialised.

While MB‐SEM is not yet widely applied in volume EM, the first study results are impressive. A large‐scale (2D) study of mouse and marmoset brain tissue was performed. Another study revealed for the first time the complex structure of the chicken retina. The latest result is a 1.4 petabyte data set of human cerebral cortex acquired in 326 days, which was fully segmented using automated methods discussed later (Section 5). These pioneering studies indicate great potential.

Multiple cameras: TEMCA and AutoTEM

Transmission electron microscopy is inherently parallel compared to scanning electron microscopy. However, it is slowed down significantly by sample stage movement, detector readout time and sample grid replacement. TEM camera array (TEMCA, Figure 2B) was developed to improve the throughput of transmission electron microscopes. , The field of view of the TEM is increased by using a 2 × 2 array of high‐speed CCD (charge‐coupled device) cameras coupled to lenses, connected to an extended vacuum column. To further improve throughput, Zheng et al. built two second‐generation TEMCA systems (TEMCA2), equipped with four CMOS cameras, a custom piezo‐driven fast stage and an automated transport and positioning system, which allow unsupervised sample loading and imaging for extended periods. Together, these innovations allow 40× faster imaging than conventional TEM. The TEMCA2 design was used to image a full adult fruit fly brain in 16 months.

The TEMCA2 has inspired the development of a further automated TEM system, autoTEM. AutoTEM consists of 6 parallel TEMCA‐inspired systems with a summed burst acquisition rate of 3 Gpixel/s and a net rate of 600 Mpixel/s. A new nano‐positioning sample stage offers fast montaging of large areas. , The sections are loaded onto a new aluminium‐coated polyimide tape with regularly spaced TEM‐grid‐resembling holes (GridTape ), which enables section collection with ATUM. The implementation of a new reel‐to‐reel tape translation system allows loading and selection of 5500 sections per vacuum cycle. With AutoTEM, two 1 mm3 volumes of mouse neocortex and primary visual cortex were imaged in about 6 months, resulting in two petabyte data sets. , Additionally, a TEMCA system upgraded with GridTape was used to reconstruct the ventral nerve cord of a female fruit fly, resulting in a 172.6 terabyte data set.

Parallel processing in block‐face imaging

While serial block‐face methods are used in the majority of volume EM studies (Figure 1C), increasing their throughput is not trivial. So far, block‐face methods remain incompatible with MB‐SEM. While acquisition in SBF‐SEM is highly automated, the samples are prone to charging and sensitive to beam dose. Solutions to these problems are described later (Section 4). An even bigger challenge is increasing the low throughput of FIB‐SEM, which is a result of slow FIB‐milling speeds and limited robustness of FIB‐SEM systems.

Parallel ‘Enhanced’ FIB‐SEM

To make FIB‐SEM systems more suitable for volume EM, Xu et al. developed ‘Enhanced FIB‐SEM’ (Figure 2C). Enhanced FIB‐SEM expands the scope of FIB‐SEM from 1000 μm3 to 3 × 107 μm3 – four orders of magnitude. FIB‐milling limits the sample thickness to about 100 m in the milling direction because it introduces streaks and waves of thickness variation. A solution was found in smooth ‘ultrathick’ partitioning of tissue volumes. Resin embedded tissue is cut into multiple chunks of 20 μm with a hot ultrasonic vibrating diamond knife to reduce distortions and slips. The chunks can then be imaged separately and stitched together. Additionally, signal detection is improved by a small positive stage bias that filters out secondary electrons, allowing efficient backscatter detection by an in‐column detector. The working distance is reduced by repositioning the FIB column to be 90° from the SEM column. Lastly, a special closed loop control system is used to maintain ion beam stability and allow seamless restarts. Two ‘enhanced’ FIB‐SEMs were employed in a study that reconstructed the connectome of the fruit fly central brain.

Alternative milling approaches

Other milling approaches could potentially offer higher throughput than conventional gallium ion FIB, including gas cluster ion beam (GCIB), broad ion beam (BIB) and plasma focused ion beam (PFIB) milling. In GCIB‐SEM, 500 nm to 1 μm thick sections are collected from the sample. These sections are pre‐irradiated with the SEM to reduce charging, followed by milling at 30° with clusters of low energy argon ions. Volumes with 10 nm isotropic resolution were acquired, but full integration with high‐throughput SEM (i.e. MB‐SEM) has yet to be demonstrated. With BIB, large areas (up to several mm) can be milled while simultaneously offering a sputter rate up to five times higher than in gallium ion FIB. Milling and imaging of liver and mouse brain tissue has been demonstrated with an integrated BIB‐SEM system, although not with high isotropic resolution as in FIB‐SEM. Xenon ion PFIB offers low damage milling compared to gallium FIB with 20–60× faster rates, but has not been widely adopted for biological samples. Oxygen has also been proposed as an alternative ion species with greater resin compatibility and similar potential gains in throughput.

Targeted reimaging with multiscale EM

Unlike block‐face methods, serial sectioning methods like ssTEM and ATUM‐SEM allow re‐imaging of tissue sections. This has inspired some researchers to use a multiresolution approach when imaging large volumes with ATUM‐SEM , (Figure 2D), to limit acquisition time. After recording the complete volume at low magnification, targeted regions of interest can be re‐imaged at higher magnification to reveal smaller features. In connectomics, this is convenient because most neuronal branches can be traced at lower resolutions while only some parts are needed in high resolution for completion. Moreover, the different data sets can be registered and combined into a multiresolution data set.

Another multiresolution approach combines ATUM with targeted high isotropic resolution FIB‐SEM, a new method called ‘multiscale ATUM‐FIB microscopy’. In ATUM‐FIB, serial sectioning of tissue into ‘semithick’ 2–10 μm sections is done first to create a library by attaching them onto glass slides that can be imaged with light microscopy. Then, they are remounted onto silicon wafers for serial section SEM to identify regions of interest to target with high resolution FIB‐SEM.

SAMPLE PREPARATION FOR LARGE VOLUMES

The success of a volume EM study is ultimately determined by the quality of sample preparation. It is inherently difficult to prepare biological samples for electron microscopy; they should be compatible with staining and residing in vacuum, have sufficient and homogeneous contrast and be resistant to sectioning and beam irradiation. While sample preparation protocols are typically designed for a specific target species or tissue type, they follow roughly the same steps: (1) fixation with aldehydes, (2) staining with heavy metals such as osmium, uranium and lead, (3) tissue dehydration and (4) resin embedding, followed by sectioning. The whole procedure, including sectioning, can take up to a few weeks per sample. , With acquisition times of large volumes being significantly reduced by emerging new methods (Section 3), further optimisation of sample preparation protocols with respect to throughput becomes increasingly important. We describe next the implications of the throughput increase in acquisition on sample preparation.

Approaches in fixation and staining

Sample preparation protocols have been modified to allow for higher throughput acquisition methods (Figure 4) as well as for homogeneity of fixation and staining for larger than before sample volumes. To provide homogeneous preservation of the tissue, it is either dissected before fixation, , , or perfused with a fixative solution before dissection. , , , , , , To further promote diffusion of the fixative into the sample, the surrounding skin can be removed. Fixation is typically followed by en bloc staining, in which the sample is submerged into one or more solutions of (different) heavy metal compounds to increase electron scattering. In traditional serial section TEM, samples are typically treated twice: first en bloc, then by post staining the ultrathin sections to enhance the contrast. However, post‐staining is laborious, prone to contamination and incompatible with block‐face techniques.

FIGURE 4

Sample preparation steps for high‐throughput volume EM. Next to improvements in en bloc staining, ultrathick partitioning was introduced in FIB‐SEM and tape collection in ssTEM. Figure not to scale

Hence, volume EM sample preparation protocols have been designed to optimise en bloc staining. The traditional osmium‐thiocarbohydrazide‐osmium (OTO) protocol, in which thiocarbohydrazide acts as a bridging agent for osmium tetraoxide to crosslink and stain cell membranes, typically leads to inhomogeneous staining for larger volumes. By addition of potassium ferri‐ or ferrocyanide, the osmium can be reduced to make it more reactive (reduced OTO or rOTO ). While this improves contrast and thereby allows for lower dwell time, it still has a limited penetration depth (∼200 μm) and weakens large tissue samples due to the formation of nitrogen bubbles. The OTO protocol was therefore modified further by separating the osmium and ferrocyanide treatment steps. This allows the osmium to penetrate deeply into the tissue, after which it is reduced to allow for deeper staining. A variant on this protocol adds formamide during the reducing osmium step and replaces thiocarbohydrazide by pyrogallol, which prevents nitrogen bubble formation. This protocol was further optimised to reduce the long incubation times, thus allowing both homogeneous, strong fixation and staining as well as faster sample preparation.

Sectioning of large volumes

Three out of four emerging volume EM techniques discussed earlier rely on serial sectioning, motivating the need for reliable sectioning approaches. Cutting and collecting (thousands of) ultrathin serial sections is a delicate process; many factors affect the consistency and continuity. An inherent issue is that interruptions are needed to resharpen or replace the knife, which impair sectioning quality as the knife needs to be repositioned. A closed‐loop repositioning system as introduced in FIB‐SEM may offer a solution. Another issue is section collection. Multiple tools have been developed that simplify the handling and collection of moderate amounts of sections, , , but for larger amounts automated collection (ATUM) is currently the only viable option. The collection tape of ATUM has a low packing density (∼200 sections per metre) and needs to be carbon‐coated for conductivity. Intrinsically conductive alternatives such as carbon‐nanotube tape, on the contrary, need plasma treatment for hydrophilisation and manual grounding. In order to increase the packing density of sections, Templier introduced MagC, in which the tissue block is glued to a magnetic resin, which allows magnetic collection of the sections directly onto wafers.

Charge‐compensation and artefact reduction

Artefacts created during sample preparation and acquisition increase the difficulty of reconstructing volumes with automated image processing methods. , , There are several ways in which these artefacts can be reduced. One way is focal charge compensation, in which surface charges are neutralised by local injection and ionisation of nitrogen gas onto the sample. Additionally, the conductivity can be increased by coating the sample with a thin metallic film prior to each cycle of imaging. The embedding material can also be made more conductive, either with a metallic particle filler or adding carbon powder. , New types of resins can also offer higher contrast with low stain concentrations, offering a way to reduce artefacts introduced by staining. Lastly, the sample can be embedded within another biological sample to improve stability of the tissue block and prevent shrinkage and deformation.

IMAGE PROCESSING AND ANALYSIS IN VOLUME EM

Image processing and analysis of volume EM data sets are nontrivial tasks due to their size and complex nature. Roughly speaking, the steps in image processing are intensity normalisation, 2D stitching and 3D alignment, while image analysis concerns the annotation (labelling individual biological features in the data set) and segmentation (assigning every pixel or voxel to a class) of the reconstructed volume to extract biological information (Figure 5). The computer algorithms that handle these tasks have to overcome difficulties such as variable intensity and contrast, sample drift, missing or low‐quality data, and imaging artefacts introduced by sample preparation, sectioning (shear, distortion), pickup, inhomogeneous staining and beam damage. The throughput increase also poses additional challenges for image analysis. Manual segmentation of volumes, already a time‐consuming process for small data sets, becomes impractical for large data sets. We will illustrate the steps in image processing and analysis while discussing the state of the art approaches and methods.

FIGURE 5

Image processing and analysis workflow. First, the images are normalised. Next, images that belong to one section are stitched with help of point‐pair matches and transformations. Similarly, the composite images of the sections are 3D aligned. When the 3D volume has been solved, the segmentation can be performed (automatically). Segmentation is followed by annotation and visualisation. The data can then be interpreted and analysed

Stitching of large FOVs and 3D alignment

When a region of interest (ROI) is larger than the field of view (FOV) of the microscope at the desired magnification, multiple FOVs are acquired with a small overlap and digitally stitched together to reconstruct the whole ROI (commonly referred to as a montage or mosaic). There are multiple algorithms for stitching, which generally work for all EM techniques considered here. The simplest in terms of computational complexity is phase correlation, which computes the translation between two overlapping image tiles based on the normalised cross‐correlation. However, phase correlation does not take into account affine transformations and only allows for local optimisation. A more robust approach is to find local point‐pair correspondences between images with a feature detection algorithm, such as the scale‐invariant feature transform (SIFT) or speeded up robust features (SURF). Both algorithms use scale‐space representations – consisting of increasingly downsampled versions of the images – to find scale‐invariant features. Corresponding point‐pair matches are selected using robust sampling methods (RANSAC) and from these the affine transformations are determined to generate a globally optimised alignment.

The same algorithms can be used to align individual mosaics in 3D. First, each mosaic is downsampled and roughly aligned to its neighbouring layers. This is then refined by extracting and matching point‐pair correspondences between neighbouring tiles in different layers. In serial block‐face methods (FIB‐SEM and SBF‐SEM), only subtle refinement may be needed as the FOV is inherently highly similar between adjacent slices. Alignment of ssTEM and ATUM‐SEM is more complicated as it requires significant corrections for rotation and non‐linear distortion compared to block‐face data sets.

There are several dedicated software packages for stitching and 3D alignment, including AlignTK, , NCR tools, StackReg and Big Feature Aligner (BigFeta). Popular tools that can perform EM image registration are TrakEM2 – implemented in Fiji (ImageJ), popular among bioimage analysts – IMOD and Microscopy Image Browser (MIB). Lastly, a novel approach was developed for multiscale EM alignment, known as signal whitening Fourier transform (SWiFT‐IR), in which modulated Fourier transform amplitudes produce more robust image matching.

Manual annotation and segmentation

After the volume is reconstructed, the next step is to extract biological information from the data. To quantify the morphology of tissues, cells and cell organelles, their 3D structure has to be annotated and segmented. Successful interpretation of biological EM images is time consuming and requires training in anatomy. It was recently estimated that it would take up to 60 years to manually segment each organelle in a single cell by hand. Nevertheless, volume EM studies still rely on manual or semi‐automatic segmentation and annotation. Usually, only a fraction of the entire volume is annotated by hand (sparse annotation) to reduce the workload. These annotations can be used as training data for machine learning algorithms to process the whole volume in an automated fashion.

Voxel painting and neuron tracing

The most straightforward method is inspecting and labelling voxels with the help of (web‐based) software. These applications allow browsing through a volume and facilitate tracing of cells, membranes, cell organelles or other features of interest. Groups of voxels can be assigned a label with brush or bucket tools. Neurons are frequently annotated by a centre‐line tracing (skeleton). Software tools often support multiple approaches. To get better accuracy, tracings can be proofread by an additional annotator. In large connectomics studies, typically a team of multiple annotators performs the tracing and proofreading, with assistance of anatomy experts. , , , The exact segmentation approach depends on the complexity of the tissue and which type of annotation is desired (sparse or dense).

Tracing, annotation and segmentation software

Annotation tools combine segmentation, annotation and visualisation into one interface. A distinction can be made between commercial and open‐source software. Examples of commercial software are Amira (Thermo Fisher Scientific), Imaris (Oxford Instruments) and Vision4D (Arivis), whereas often used open‐source programs are the Collaborative Annotation Toolkit for Massive Amounts of Image Data (CATMAID), KNOSSOS and its web version WebKnossos, Volume Annotation and Segmentation Tool (VAST) and earlier mentioned tools, TrakEM2, IMOD and MIB. Another distinction can be made between offline and web‐based tools. VAST, TrakEM2, IMOD and MIB are offline tools, while CATMAID and webKnossos retrieve image data and annotations hosted on a remote server and work with databases to manage annotations. It can be accessed anywhere (with an internet connection) and multiple annotators can simultaneously work on different parts of the volume. A comprehensive list comparing various features of all tools has been published elsewhere.

Automated segmentation

The last 10 years have seen an increased usage and improvement of automated segmentation, made possible by developments in machine and deep learning. The choice for machine and/or deep learning seems obvious. Volume EM data sets are significantly growing in size, rendering complete manual segmentation impossible. Traditional segmentation methods most often fail or generalise poorly, because EM images are often noisy and characterised by variations in contrast and texture as well as artefacts introduced during sample preparation or imaging. Additionally, tissue structure can be very complex, such as the dense wiring patterns found in neural tissue. It has been shown that data driven models can cope with complex segmentation problems—convolutional neural networks (CNNs), from the domain of deep learning, outperformed traditional segmentation methods more than a decade ago. CNNs are very popular for image segmentation because they efficiently extract and combine information from different hierarchical levels in the image. Automated segmentation using deep learning has become the predominant strategy in two different domains: connectomics and cell biology.

Automated segmentation in connectomics

In connectomics, the interest lies mainly in cell boundary segmentation and synapse detection. Many of the new developments can be attributed to several crowd‐sourcing competitions for automated segmentation in 2D and 3D: the International Symposium on Biomedical Imaging (ISBI), 3D segmentation of neurites in EM images (SNEMI3D)i and circuit reconstruction from EM images (CREMI)ii challenges. State‐of‐the‐art cell boundary segmentation approaches are typically either based on the popular U‐Net CNN architecture , or variants thereof. Variants based on U‐Net have achieved near‐human or even super‐human segmentation performance on neural EM data. , , , , , Alternatively, flood‐filling networks have been employed to increase segmentation accuracy at the expense of higher computational costs. , A comprehensive overview of these approaches and their implementation has been described elsewhere.

Synaptic relations can be used to infer connectivity between neurons. Machine learning algorithms are therefore employed to find synaptic relations between neurons by classifying each voxel as ‘synaptic’ or ‘non‐synaptic’. Classical machine learning algorithms such as the random forest classifier are used, , but also here CNNs are gaining popularity. , , , , , , , In short, these methods try to predict candidate synapses and their directionality, while some also distinguish the pre‐ and postsynaptic neurons. Recent efforts in automated synapse detection resulted in a reliable connectivity graph in the whole brain fruit fly data set.

Cell organelle segmentation

In cell biology, the interest lies in segmentation of cell organelles to enable quantification of their morphology, distribution and size. A clear motive for this work is evidence that links alterations of organelle structure to neurodegenerative diseases and cancer. , , , The high axial resolution of SBF‐SEM and especially FIB‐SEM data allows for accurate segmentation of cell organelles. Due to the diversity of organelles and cell types as well as a lack of publicly available training data, automated organelle segmentation has not experienced the same surge as in connectomics, which has benefited from years of substantial manual annotation effort. , , , , , Nonetheless, there has been successful pioneering work within different types of volume EM data (Table 3). Similar to dense reconstructions of neural tissue, several studies have now demonstrated (fully) automated multiclass segmentation of organelles in single cells. ,

TABLE 3

Summary of studies on automatic organelle segmentation with machine learning and/or deep learning

Application	Organelle(s)	Technique	Network architecture	Reference
HeLa cells	NE	SBF‐SEM	U‐Net	133
HeLa cells, Jurkat cells, macrophages	Chromatin, ER, endosomal membranes, lysosome, MTs, Mito, NE, PM, vesicle membrane	FIB‐SEM	U‐Net	2
Rat hippocampus, mouse cortex	Mitochondria, ER	ATUM‐SEM, FIB‐SEM	ResNet, region proposal Net, Recursive Net, Mask R‐CNN	5
Rat hippocampus, mouse cortex	Mitochondria	ATUM‐SEM, FIB‐SEM	ResNet	4
Mouse primary beta cells	MTs, Golgi, centrioles, insulin granules	FIB‐SEM	U‐Net, random‐forest classifier	3
HeLa cells	ER, mitochondria, PM	FIB‐SEM	U‐Net, EfficientUnet	186
Mouse hippocampus	Vesicles, nuclei, mitochondria, membranes	SBF‐SEM	DeepEM3D	154
Human breast carcinoma	Nuclei, nucleoli	FIB‐SEM	ResNet, U‐Net	46
Mouse hippocampus	Mitochondria	FIB‐SEM	‘Conventional’ CNN	139
Mouse urinary bladder urothelial cells	Mitochondria, endolysosomes	FIB‐SEM	HighRes3DNet	140
Rat and human cortex	Mitochondria	ATUM‐MBSEM	U‐Net	141
Pancreatic beta cells	Insulin granules	FIB‐SEM	Multibranch FCN	144
Ovarian cancer cells	Extracellular vesicles	ssTEM	Fully residual U‐Net	6

Note: The organelles, EM technique and neural network architecture are indicated.

NE: nuclear envelope; ER: endoplasmic reticulum; MTs: microtubules; PM: plasma membrane.

Examples of important yet difficult segmentation problems in EM data include mitochondria, nuclei and vesicles. Mitochondria vary greatly in shape and size. This variation is not well represented in commonly used training data sets. , Nuclei segmentation is a common segmentation problem, also in light microscopy. Vesicles come in many forms and sizes. Automated mitochondria segmentation has been successfully applied to FIB‐SEM , , , and ATUM‐SEM data , , (despite its lower axial resolution). While it is possible to segment plasma and nuclear membranes with traditional segmentation algorithms, , two different groups approached nuclear envelope and nuclei segmentation with U‐Net variants. , To deal with the limited availability of expert manual annotations, the authors either aggregated multiple volunteer annotations or utilised sparse labelling techniques. Automated vesicle segmentation was developed for insulin granules and small extracellular vesicles.

Challenges with convolutional nets

There are several problems associated with CNN‐based segmentation. Generally, the performance is best on data sets with high isotropic resolution and proper alignment (SBF‐SEM, FIB‐SEM). Performance on serial section EM data, which is characterised by anisotropic resolution and slight defects in the alignment, can be improved by encouraging topologically correct segmentations obtained from the affinity graphs. , A second problem is that several methods do not generalise well outside of their particular source and tissues. To cope with this, domain adaption techniques can be used that transform the image content of different data sets to make them more similar to training data set. , On the other hand, training on data from various types of tissues may improve robustness. Lastly, problems arise due to artefacts introduced during sample preparation and imaging, which are rare in commonly used training data sets (e.g. CREMI, SMEMI3D). Solutions include increasing the occurrence artificially using data augmentation, locally realigning image sub volumes before region agglomeration or by supplementing these public data sets with manually segmented data from a portion of the imaging volume.

Deep learning for the masses

Although automatic segmentation methods are becoming more powerful, they are often difficult to adopt by those with limited programming skills. To leverage the power of automatic segmentation in a more user‐friendly way, several state‐of‐the‐art algorithms and architectures have been integrated into popular image analysis tools. Fiji contains plugins such as ‘Trainable WEKA segmentation’ for interactive training of machine learning algorithms and ‘DeepImageJ for straightforward importing and deployment of deep learning models. Similarly, ilastik also supports simple and interactive training of machine learning algorithms and currently offers limited support for pre‐trained CNNs. Microscopy Image Browser (MIB) has been extended with a user‐friendly U‐Net. UNI‐EM is yet another user‐friendly tool that integrates multiple top‐performing 2D and 3D network architectures. Some tools work with cloud deployment to circumvent the need for local computational resources and software installation, such as DeepEM3D and ZeroCostDL4Mic.

While these applications have reduced the barrier to entry for AI‐based analysis, there are several potential drawbacks. These include the limited number of implemented models and the verification of performance, as users generally look at visual segmentation quality without employing quantitative performance statistics. Furthermore, computational expertise and resources remain necessary for the documentation and maintenance of these tools. Lastly, different implementations require a varying level of knowledge about machine learning.

CHALLENGES IN DATA STORAGE, MANAGEMENT AND VISUALISATION

The output of volume EM is very information dense, but there are several hurdles in maximising its potential use. Currently, volume EM data set sizes range from several gigabytes to hundreds of terabytes, with several studies already having reached the petabyte scale. , , This has big implications for data storage and management. Data formats should be clear, accessible and complete to make sure data can be reused and revisited. Visualisation tools should offer fast terabyte scale data inspection. We will discuss the implications of the throughput increase and widened scope of volume EM methods on data management and visualisation.

Data storage

Where to store large volume EM data sets and their annotations? Small data sets can be managed on individual PCs or workstations, but with the current trend (Figure 1D) it is likely that storage on institutional servers or in the cloud will become the standard.

Repositories and metadata

Systematic archiving of data and metadata in online repositories is not yet routine practice in the field of volume EM, though several dedicated repositories have emerged. These include the Electron Microscopy Public Image Archive (EMPIAR), Image Data Resource (IDR) and the EMBL BioImage Archiveiii (BIA). EMPIAR is EM specific, whereas IDR and BIA are more broad. Currently, these repositories allow the download and upload of whole data sets, but it may be easier to interact with (a subset of) the data via application programming interfaces (APIs) or viewers. How data should be formatted and stored is an ongoing discussion in the EM community. While it is generally accepted that EM data should follow the FAIR format to maximise reuse, it is difficult to standardise metadata because the needs vary greatly based on the application or imaging modality. Nonetheless, recently a set of guidelines for Recommended Metadata for Biological Images (REMBI) was published. This will be incorporated as a standard for submission into IDR.

Data formats

How is volume EM data stored? Different file formats are used depending on the application and storage location. During acquisition, data are often saved in proprietary microscopy data formats, which are optimised for writing. For visualisation purposes, however, the optimal format is entirely different. Data with high lateral but low axial resolution (i.e. ssTEM, ATUM‐SEM) are made possible by making use of pyramids of increasingly downsampled flat images (tiles), either remotely (CATMAID) or locally (TrakEM2). This is convenient because these images are usually viewed in 2D. Data with high axial resolution (FIB‐SEM, SBF‐SEM) is instead saved in a cube format (employed by KNOSSOS) which makes browsing or reslicing in the z direction faster and easier. In data archiving, flexible file extensions such as TIFF and HDF5 are used, which can store multidimensional pyramidal data with associated metadata. However, data from TIFFs can only be read as individual 2D tiles, while HDF5 and other ‘next‐generation file formats’ such as N5iv and Zarrv allow reading and writing of three‐dimensional chunks of images to separate, smaller files, which is much faster and better suited for cloud storage.

Data management

Client‐server applications are becoming a popular tool to interact with volume EM data. , , , Plugins for processing, visualisation and annotation can be remotely installed, and there is no need to download data or install software locally other than a web browser. Moreover, research data can be more easily shared as data can be made accessible to multiple users from different locations simultaneously. The data and metadata are stored in remote servers, while the user retrieves the data via a client (Figure 6).

FIGURE 6

Example of data management structure. The image data are acquired by an electron microscope and stored within a central database. The database is connected to a post‐processing service which performs image processing and computes the transformations between the images, which are contained in a separate database. The annotations are contained in a separate database (here, CATMAID ). A client can send requests to the image, transformation and annotation database to view certain image data and corresponding annotations. The image data is tiled before it is sent. The arrows indicate the flow of information.

‘Local’ data management

An open, flexible and scalable data management platform suitable for electron microscopy data is Open Microscopy Environment Remote Objects (OMERO). It was created with the idea of standardising data access. Data can be imported using Bio‐Formats, which converts proprietary microscopy data formats into a common data model (currently, OME XML with OME‐TIFF).

Cloud data management

Apart from the costs, hosting EM data in the cloud offers several advantages in terms of convenience and accessibility. One such example is Neurodata.io,vi a community‐developed and maintained software ecosystem for neuroscience data deployed in the commercial cloud (AWS), containing TrakEM2 for registration, NeuroGlancervii for visualisation and BossDB for data management. Multiple data sets of different formats and imaging modalities can be combined. In addition, the hosted data can be accessed via annotation software (VAST/KNOSSOS). Another example is OpenOrganelle, a repository for cell biology data, created at the Janelia Research Campus and also hosted in the cloud. The platform hosts 10 FIB‐SEM data sets of various cell lines and tissues for online visualisation and offline data mining, while at the same time providing the code and tutorials for all available tools. Separately, the entire EM volume of Platynereis dumerilii is hosted in N5 format in cloud object storage at EMBL and can be accessed using an N5 API. The data can be browsed using a specifically designed Fiji plugin ‘PlatyBrowser’ in MoBIE. These projects demonstrate the potential and conveniences of cloud data management of volume EM data.

Developing scalable architectures

Apart from developing new tools, there is the challenge to make scalable architectures out of existing tools. Employing these tools on high performance computing (HPC) clusters allows large data sets to be processed in parallel. Vescovi et al. developed a scalable and modular pipeline which integrates multiple software modalities such as TrakEM2, NeuroGlancer and Flood‐Filling networks to perform several tasks from registration to annotation and visualisation. These tools are made HPC deployable and wrapped in an operational database which can be used to create custom pipelines for image processing and annotation.

Terabyte data viewers

Some annotation tools have been designed with large data sets in mind. Examples are VAST and NeuTu. VAST is mainly a segmentation and annotation tool, and is able to handle very large data sets, which can be imported from a server or locally. Manual segmentation at different zoom levels is also supported, although simultaneous editing by multiple users is not. In contrast, NeuTu allows collective proofreading and correcting errors created by automated segmentation by manually merging or splitting segments. It is part of DVID, a distributed, versioned, image‐orientated data service in which NeuTu acts as the data client. DVID works with 2D and 3D data and has a version control feature to manage different annotation states. Lastly, BigDataViewer and Multimodal Big Image Data Exploration (MoBIE), both Fiji plugins, make use of the convenient HDF5 format to interactively navigate and visualise large image sequences. MoBIE is additionally equipped with an object storage backend to load data from remote sources.

Collective annotation

The last 10 years have seen the emergence of projects in which researchers can collectively work on annotating volume EM data. Some of these endeavours actively encourage non‐scientists to participate through ‘citizen science’ initiatives. Non‐scientists can help with proofreading annotations produced by automated segmentation algorithms (for example in the game Eyewire ) or they can assist in generating training data for deep learning applications (e.g. ‘Etch a Cell ). The fruit fly community has developed FlyWire, with the goal of collectively mapping the fruit fly connectome from whole brain data sets. , It resembles EyeWire but is currently limited to researchers only.

CONCLUSION AND OUTLOOK

Almost 10 years after the review of Briggman and Bock, there has not yet emerged a volume EM technique that makes others obsolete. FIB‐SEM and to some extent SBF‐SEM remain the methods of choice in studies where high isotropic resolution is favoured over throughput, for example in cell biology. The high isotropic resolution also provides the advantage of more precise automated segmentation. ‘Enhanced FIB‐SEM’ allows the imaging of larger samples. The implementation of new milling approaches will hopefully speed up FIB‐SEM even further. ssTEM has seen several innovations (TEMCA, GridTape, AutoTEM) that greatly improve the throughput. It offers the highest lateral resolution, but its limited axial resolution and artefacts hamper automated image analysis. For all techniques, a considerable amount of manual proofreading remains necessary after automated segmentation. This results in many studies still heavily relying on manual annotation efforts. Nevertheless, the performance of automated segmentation algorithms will likely improve further given their recent introduction in the field and the general interest in AI research.

Multibeam SEM should not be ignored. The possibility to increase the throughput of a single microscope by orders of magnitude is very cost‐effective. However, both approaches are (for the moment) incompatible with block‐face approaches, making them dependent on serial sectioning. Nevertheless, the speed of MB‐SEM could be key in rapid diagnostics (digital pathology) but also in studying brain development in multiple specimens. , , The MultiSEM has demonstrated compatibility with ATUM. FAST‐EM still has to demonstrate compatibility with a high‐throughput sectioning approach.

Currently, the expertise in large volume EM is limited to several research groups. Centralised imaging facilities could take a leading role in investing in high‐throughput electron microscopes and the elaborate data infrastructure required for these machines. Will it be possible to image and annotate a full adult zebrafish in the near future, maybe even a mouse brain? Will the imaging and annotation of a small animal brain follow a path similar to, for example, the human genome project? An average adult mouse brain has a volume of 485–530 mm3. It is clear that more automation is needed. Advancements in methodology will probably again play a key role, but it will also require extensive collaboration and sharing of resources.

Information from high‐throughput studies, such as connectomes of animals or atlases of healthy and diseased tissue, will presumably give critical insights. It can lead to direct discoveries or provide a starting point to test hypotheses on the relation between structural changes and disease onset with functional studies. Novel data mining approaches and meta‐analyses of various data sets could give new insights in ultrastructural differences between data sets of various tissues and animals, similar to microarray studies.

In conclusion, methodological improvements are making volume EM more accessible and are alleviating the burden on throughput. Automated segmentation methods are reducing the workload of manual annotation, but considerable human effort remains necessary. In the field of data management, there is a need for a joint approach on how to manage large volume EM data. The adoption of a common file format could improve collaboration and simplify training of automated segmentation methods. We see an opportunity for open hosting of data sets with corresponding annotations to maximise the profit to the community. The potential of volume EM may be greater than ever before.

AUTHOR CONTRIBUTIONS

ECC initiated the project and suggested the topic. AJK conducted the literature survey, drafted the original manuscript and produced the figures. RL revised the manuscript and contributed to Sections 4, 5 and 6. JPH and ECC supervised the writing process and offered critical remarks on the manuscript and figures.

FUNDING

AJK and JPH acknowledge support from the ECSEL Joint Under‐taking (JU) under grant agreement No. 826589. The JU receives support from the European Union's Horizon 2020 research and innovation programme and Netherlands, Belgium, Germany, France, Italy, Austria, Hungary, Romania, Sweden and Israel. RL, ECC, JPH further acknowledge support from the EU‐React G4P4 ‘Kansen voor West’ project IMDAP.

COMPETING INTERESTS

AJK, RL and ECC declare no competing interests. JPH is co‐founder of and shareholder in Delmic BV, a company selling integrated microscopes, including FAST‐EM.

Supplementary Material

Click here for additional data file.