Deep learning enables automated volumetric assessments of cardiac function in zebrafish.

Although the zebrafish embryo is a powerful animal model of human heart failure, the methods routinely employed to monitor cardiac function produce rough approximations that are susceptible to bias and inaccuracies. We developed and validated a deep learning-based image-analysis platform for automated extraction of volumetric parameters of cardiac function from dynamic light-sheet fluorescence microscopy (LSFM) images of embryonic zebrafish hearts. This platform, the Cardiac Functional Imaging Network (CFIN), automatically delivers rapid and accurate assessments of cardiac performance with greater sensitivity than current approaches.This article has an associated First Person interview with the first author of the paper.

INTRODUCTION

Despite the advantages of utilizing zebrafish embryos for modeling human heart failure (Bakkers, 2011; Becker et al., 2012; Chen et al., 1996; Liu et al., 2014; Shih et al., 2015; Stainier et al., 1996), the metrics employed to monitor cardiac function remain limited and unrefined. The most common metric, ‘fractional shortening’ (FS), is a proxy for volumetric output calculated from the cross-sectional diameters of the ventricle during diastole (chamber relaxation) and systole (chamber contraction), which are measured from two-dimensional (2D) images (Hoage et al., 2012; Yalcin et al., 2017). Despite its ease of use, FS suffers from inconsistencies caused by variable imaging angles, and subjective identification of systole, diastole and cross-sectional diameter. Furthermore, the accuracy of this method relies upon the assumption that chamber diameter is uniformly correlated with chamber volume, which is not always the case. Regional abnormalities in wall movement can over- or underestimate functional deficits depending on where the abnormalities reside relative to the diameters being measured. In short, the most widely used indicator of cardiac performance in zebrafish embryos relies on indirect, non-volumetric measurements that are subject to confounding factors and inaccuracies.

By contrast, functional indices derived from direct measurements of chamber volume, such as ejection fraction (EF) and cardiac output (CO), do not suffer from the same weaknesses. The imaging modalities required to obtain these measurements in larger subjects, such as three-dimensional (3D) echocardiography and magnetic resonance imaging (MRI), are not adaptable to the small scale of zebrafish embryos. Recently, four-dimensional (4D) reconstructions of the embryonic zebrafish heart from dynamic light-sheet fluorescence microscopy (LSFM) images have been shown to offer unparalleled spatial and temporal resolution with which to characterize cardiac morphology and chamber wall dynamics (Lee et al., 2016; Mickoleit et al., 2014). However, quantifying dynamic chamber volumes from these datasets, especially from multiple subjects, remains impractical because it requires the tedious manual segmentation of atrial and ventricular chambers in thousands of component static images.

While various traditional computational approaches exist to automate image segmentation (Sharma and Aggarwal, 2010), many of these methods still require some level of human input and can be limited in their ability to segment and classify distinct, yet similar, structures within biomedical images (Fei et al., 2016; Krämer et al., 2009; Packard et al., 2017). Several of these algorithms, including watershed segmentation, were previously evaluated for their ability to segment fluorescent images of embryonic zebrafish hearts at 48 hours post-fertilization (hpf) (Krämer et al., 2009). While these methods were capable of segmenting whole hearts from background, they failed to consistently delineate individual cardiac chambers. Even if this shortcoming could be overcome, these approaches would only incrementally reduce user workload because the manual annotation of ‘atrium’ and ‘ventricle’ would still be required for each image to calculate chamber volumes.

Convolutional neural networks (CNNs), a class of deep learning neural network, can overcome many of these limitations by applying unsupervised, higher-level and adaptable segmentation criteria to image analysis, while concurrently increasing speed and reducing the amount of processing power required (Litjens et al., 2017). When compared to other approaches, CNNs have been shown to best match the performance of human experts when analyzing biomedical images from numerous sources, including computed tomography (CT), MRI and fluorescence microscopy (Bai et al., 2018; Ciompi et al., 2015; Dou et al., 2017; Hay and Parthasarathy, 2018; Zhang et al., 2019). Therefore, we utilized a deep learning CNN to implement a novel image-analysis platform we termed Cardiac Functional Imaging Network (CFIN). CFIN is designed to: (1) automatically and rapidly segment atrial and ventricular boundaries within LSFM datasets of live embryonic zebrafish hearts, and (2) utilize these data to calculate volumetric indices of cardiac function.

RESULTS

To image the beating hearts of embryonic zebrafish, we used LSFM to acquire serial dynamic images of 48- to 52-hpf transgenic animals expressing the myocardial-specific Tg(myl7:GFP) fluorescent reporter [formerly Tg(cmlc2:GFP)] (Burns et al., 2005) (Fig. 1A). Embryos were immobilized in a column of low-melt agarose and submerged in embryo media (EM) within the temperature-controlled incubation chamber of a Zeiss Lightsheet Z.1 microscope (Fig. 1B). We captured dynamic images spanning at least four cardiac cycles at 25.6 frames per second (fps) in 1 μm z-depth intervals until the entire heart had been imaged (Fig. 1C). These imaging parameters produced approximately 4500 frames per heart.

Fig. 1.

microscopy (LSFM). (A) Fluorescent image of a 48 hpf Tg(myl7:GFP) zebrafish embryo with bright-field overlay. Scale bar: 100 μm. (B) Schematic depicting the imaging of a live zebrafish embryo immobilized in agarose within the excitation and detection axis of a light-sheet microscope. (C) Schematic of dynamic image acquisition through z-depths.

To efficiently process the vast amount of data generated by LSFM, we utilized a deep learning convolutional neural network to automatically identify and annotate the atrial and/or ventricular boundaries in each frame of 2D LSFM movies. This aspect of CFIN was built in MATLAB using the underlying architecture of the deep convolutional encoder-decoder SegNet (Badrinarayanan et al., 2017) (Fig. 2A). To train the network to distinguish atrium, ventricle and background in each frame, we created a ground-truth reference dataset by manually segmenting systole and diastole for both chambers, at each z-depth, across four animals (Fig. 2B). This manually annotated dataset amounted to 13% of the total images collected for each heart (≈2400 images total from four hearts). A cross-validation dataset was composed of 230 images from a fifth manually segmented heart, to which the network was naïve. Training was conducted at a variable learning rate beginning at 0.1, which was reduced by a factor of five every two epochs. Using these parameters, we observed an increase in accuracy and a decrease in the loss function for both the training and cross-validation datasets (Fig. 2C,D), which were consistent with successful training. We terminated training when the cross-validation loss function was minimized (Fig. 2D).

Fig. 2.

Neural network design and training. (A) Conceptual depiction of CFIN's image segmentation workflow: using the SegNet architecture, LSFM images are deconstructed by the encoder to extract local features at varying resolutions, which are then upsampled by the decoder to map each feature weight back to their correct positions relative to the input image. Then, the final decoder layer data is fed to a multi-class Softmax classifier, which assigns each pixel to their respective label of atrium, ventricle or background. (B) Illustration representing the generation of training data via manual segmentation of systole and diastole for each chamber within a dynamic image. Cyan frames and labels indicate ventricle; blue frames and labels indicate atrium. (C) CFIN training and validation accuracy. (D) CFIN training and validation loss. Each epoch represents 30 iterations.

Following training, we visually assessed the network's ability to correctly segment atrial and ventricular chambers in successive frames of the cross-validation dataset. At every z-depth inspected (n=10 z-depths containing 30 frames each), CFIN successfully segmented and identified each chamber throughout the entire cardiac cycle, including intermediate phases between systole and diastole that were not included in the training dataset (Fig. 3A, Fig. S1). Chamber segmentation also appeared consistent across cardiac phases over multiple cardiac cycles (Movie 1). This can be further appreciated by observing CFIN's chamber area measurements over time, which revealed the expected mirror-image contraction and relaxation patterns of the atrium and ventricle (Fig. 3B). Notably, CFIN's ability to segment both cardiac chambers far surpassed those achieved when non-machine learning-based algorithms were applied to images of fluorescent zebrafish hearts at the same developmental stage (Krämer et al., 2009).

Fig. 3.

Validation of CFIN image segmentation. (A) Sequential frames from a representative dynamic image annotated by CFIN. Ventricle is shown in cyan and atrium in purple. Scale bar: 50 μm. (B) CFIN's quantification of chamber areas from a single dynamic image over time. Atrium (purple dashed); ventricle (blue). (C) Dice similarity coefficient values for 230 validation images. (D-I) Single-frame comparisons of CFIN and human chamber segmentation from z-depths containing the ventricle (D,E), the atrium (F,G), or both the atrium and ventricle (H,I). Scale bars: 50 μm.

To gauge CFIN's segmentation accuracy within each frame, we generated Dice similarity coefficients comparing its labels to those that were manually annotated for 230 frames within the cross-validation dataset (Zou et al., 2004). Across all images, the average Dice score was 0.9 with a standard deviation (s.d.) of 0.03 (Fig. 3C), demonstrating a high degree of similarity between CFIN and ground-truth labels. In fact, visual comparisons at multiple z-depths revealed instances where CFIN's chamber labels were more closely associated with the chamber wall than those that were segmented manually (Fig. 3D-G). Surprisingly, we also found that our network appropriately assigned mutually exclusive regions within the atrioventricular canal to either ventricle or atrium even though this region was deliberately avoided during manual segmentation because absolute boundaries between chambers in this region do not exist (Fig. 3H,I). This demonstrates CFIN's remarkable ability to correctly infer chamber boundaries with a high degree of accuracy while simultaneously overcoming the inherent limitations of our human-computer interface.

To extract volumetric parameters of cardiac function after image segmentation, CFIN identifies the maximal and minimal chamber areas at each z-depth. These numbers are multiplied by the z-depth interval (1 μm) and summed within each category [maximal (diastole) or minimal (systole); atrium or ventricle] to calculate chamber-specific end diastolic volumes (EDVs) and end systolic volumes (ESVs). From these values we calculate EF, the percentage of blood expelled per beat, for the atrium and ventricle, defined by EF=(EDV−ESV)/EDV. Heart rate (HR) is determined from the periodicity of area measurements and used to calculate CO for each chamber according to CO=HR×(EDV−ESV). In the human population, the left-ventricular EF and CO are the most commonly monitored clinical parameters of heart function (Zipes et al., 2018). In most cases, measuring these parameters for the single zebrafish ventricle would be the most accurate measurement of cardiac performance. However, zebrafish embryos can survive without ventricular function (Auman et al., 2007), making atrial measurements more appropriate in special cases.

Next, we tested the hypothesis that volumetric assessments of cardiac function by CFIN could be more sensitive than those based on FS. A previous study demonstrated that zebrafish embryos exposed to doxorubicin (Dox) beginning at 24 hpf exhibit reductions in FS at 72 hpf (Liu et al., 2014). We repeated this experiment but analyzed embryos at 48 hpf, when the functional deficits are likely to be milder. Embryos were imaged using LSFM as described and cardiac function was assessed using FS (Fig. 4A) followed by volumetric assessment by CFIN. Although the FS index detected a small reduction in ventricular function in Dox-treated animals, it was unable to resolve any significant differences in atrial contraction between experimental groups (Fig. 4B). Similarly, calculating EF using estimates of chamber volume, derived from measurements of chamber radii and the assumption that the chambers are spherical, produced similar results (Fig. S2A). Conversely, the EF for both chambers, HR and the CO as measured by CFIN were all clearly reduced following Dox exposure, and presented a much greater statistical significance than FS (Fig. 4C-E). Furthermore, CFIN's performance was unaffected by the presence of Dox-induced autofluorescence in the yolk (Fig. 4A, Fig. S2B,C). These data demonstrate that volumetric measurements made by CFIN can identify functional deficits that may otherwise go unnoticed using FS.

Fig. 4.

Experimental validation of CFIN functional assessments of the heart. (A) Methodology for determining fractional shortening (FS) using EDD (vertical line in A) and ESD (vertical line in A′). Scale bars: 100 μm. Red asterisk denotes Dox autofluorescence. V-, ventricle. (B) FS measurements of doxorubicin (DOX)-treated embryos and control siblings. (C-E) CFIN’s assessment of ejection fraction (EF; C), heart rate (HR; D) and cardiac output (CO; E) within the same experimental cohort. (F-H) CFIN's assessment of EF (F), HR (G) and CO (H) in 48 hpf embryos treated with isoproterenol. Significance is denoted as *P<0.05, **P<0.005, ***P<0.0005, ****P<0.00005, or ns (not significant) as determined by the Student's two-tailed t-test.

Next, we determined whether CFIN could detect increases in CO. Previous reports have demonstrated that embryonic zebrafish exposed to the β-adrenergic agonist isoproterenol exhibit elevated FS. However, this effect was only detectable when animals 72 hpf or older were exposed to the drug and was not apparent in 48 hpf animals (Kossack et al., 2017; Steele et al., 2011). To determine whether our platform could again detect functional changes that traditional methods were unable to resolve, we treated 48 hpf embryos with isoproterenol for 2 h prior to LSFM imaging. Image analysis with CFIN revealed significant elevations in EF, HR and CO when compared to control animals (Fig. 4F-H). Collectively, these data demonstrate that CFIN is capable of detecting elevations in CO and provides another example of increased sensitivity over FS.

Importantly, the time it takes for CFIN to segment every frame and derive functional parameters from dynamic LSFM images is approximately 3-4 min per heart. This is compared to the ∼2 h required for an expert to manually segment only systole and diastole for each chamber and z-depth. Removing the human component in this fashion also offers absolute consistency and removes the potential for errors during manual segmentation, such as poorly drawn labels and incorrectly identified cardiac phases. Because imaging the hearts with LSFM remains the rate-limiting step of our platform, we next explored whether increasing the z-depth step size, thus decreasing the time required for imaging, would affect CFINs measurements in experimental conditions. Using our Dox experiment as a model, our data displayed the expected inverse correlation between the step size and accuracy of EF measurements (Fig. S3A,B). Despite the decrease in statistical power that accompanies larger step sizes, CFIN was still able to detect significant differences within each chamber using either a 2 μm or 4 μm step size. These data further demonstrate the robustness of the CFIN platform and allow for the optimization of imaging protocols based on desired functional resolution.

Although CFIN does not require image synchronization or 4D reconstruction to calculate indices of cardiac function such as EF and CO, segmented images can be efficiently used for these purposes, therefore expanding the number of downstream applications. To accomplish this, we synchronized optical sections to the end-diastolic filling period of the atrium as defined by CFIN's measurements of area. In cases where slices did not contain the atrium, we instead used the ventricular period closely corresponding to the atrial end-diastolic filling period identified in the slices with both chambers. Once synchronized in this fashion, images can then be used to observe continuous chamber volume over time (Fig. 5A), which has the potential to reveal complex functional deficits and/or arrhythmias in one or both chambers. Further, graphical 3D and 4D reconstructions built from synchronized data (Fig. 5B,C, Movie 2) have the potential to reveal regional abnormalities in wall movement that may otherwise go unnoticed.

Fig. 5.

Volumetric reconstructions using CFIN. (A) Continuous chamber volumes over three cardiac cycles in a wild-type 48 hpf zebrafish embryo. Volumes were calculated using synchronized images processed by CFIN. Ventricle (cyan); atrium (purple dashed). (B,C) Graphical 3D reconstruction of ventricular EDV (B) and ESV (C) as measured by CFIN. Ventricle (V); atrium (A).

DISCUSSION

Our platform represents a novel, rapid and robust method for making volumetric assessments of cardiac function in embryonic zebrafish. CFIN greatly increases experimental rigor by not relying on approximations or the inherent assumptions of traditional methods, thereby providing a higher standard of accuracy and consistency. It should be noted that CFIN was specifically developed and validated for use with embryos of approximately 48-55 hpf. Preliminary attempts to apply the platform, as currently configured, to later-stage embryos (72 hpf) resulted in suboptimal chamber segmentation, presumably because the gross morphology of the heart is sufficiently different, and this stage was not included in the training dataset. Nonetheless, later-stage analysis will likely be feasible with additional stage-specific training (i.e. transfer learning).

Given the recent commercialization and increased popularity of light-sheet fluorescence microscopes, we anticipate that our method, as presented here, will be widely accessible to other researchers. Contemporary advances in spinning-disk and resonant scanning confocal microscopes suggest that these imaging modalities will be compatible with CFIN, provided that the image quality and frame rate are sufficiently similar to those parameters in our training dataset. Importantly, our network does not require expensive software, hardware or cloud-based subscriptions and is operated in its entirety via the widely used MATLAB computing environment. Collectively, these attributes make CFIN an accessible and powerful new tool to accurately measure heart function in an influential vertebrate model organism, which will ultimately advance our understanding of heart failure.

MATERIALS AND METHODS

Zebrafish

Zebrafish were bred and maintained following protocols approved by the Massachusetts General Hospital Institutional Animal Care and Use Committee (IACUC), and adhere to the recommendations outlined by the Guide for the Care and Use of Laboratory Animals (National Academic Press). The following zebrafish (Danio rerio) strain was used in this study: Tg(myl7:GFP) (Burns et al., 2005).

Microscopy

For compound microscopy, zebrafish embryos were anesthetized in 0.16% tricaine (Western Chemical) and immobilized in 0.9% low-melt agarose dissolved in EM. Embryos were harvested at stages ≥48 hpf and imaged on depression slides using epifluorescence or bright-field optics on a Nikon Eclipse 80i compound microscope.

For light-sheet microscopy, embryos were treated with 0.003% phenylthiourea (PTU) at 24 hpf to prevent pigmentation. At 48 hpf, treated embryos were briefly anesthetized and immobilized in 0.9% low-melt agarose within a glass capillary (Zeiss). The capillary was then submerged in temperature-controlled EM with 0.016% tricaine and imaged with the ZEISS Lightsheet Z.1 microscope. Live-mounted Tg(myl7:GFP) zebrafish were imaged through the entire ventricle by collecting dynamic plane images at each GPF-positive z-depth. Dynamic images were taken every 1 μm and spanned at least four cycles of the beating heart. Images were taken at each z-depth until the entire heart had been recorded.

Training the network

Training datasets were generated from five animals across two separate crosses that were imaged on different days. Image brightness was normalized prior to training and/or analysis. To minimize the potential training bias caused by a large number of background pixels (primarily coming from yolk autofluorescence), pixel classification weights were set as the reciprocal of the frequency of each classification (‘atrium’=38.2, ‘ventricle’=64.9, ‘background’=1). CFIN utilizes the underlying architecture of SegNet, with an encoder depth of 4 and an input image matrix of 160×160 pixels to reduce the time required for training and analysis. Training data was created via manual segmentation of each chamber in 2400 static images using MATLAB.

Synchronization and volumetric reconstruction

After running CFIN to initially annotate each 2D image of the entire 4D dataset, we performed synchronization by first identifying the end-diastolic atrial cardiac phase, t_EDA, that maximizes the cost parameter, phi(t), for each 2D temporal image at each z-stack:where A(t), V(t) and t_N represent the area of labeled atrium pixels, area of labeled ventricle pixels and maximum number of temporal images acquired, respectively. Once the t_EDA is determined for each z-stack, we re-index the 2D temporal images at each z-stack to start at the t_EDA. Then, one period of the cardiac cycle is identified by searching forward for the next t_EDA maximizing again phi(t) for each z-stack. Finally, the single period of cardiac cycle is temporally cropped and interpolated to 24.6 fps. This ensures that any slight arrhythmias that may occur between z-stacks are accounted. At each cardiac time frame, 3D reconstruction surface models were constructed using the segmented atrial and ventricular binary masks generated from CFIN. Then, the 3D reconstructions were played dynamically at the acquired temporal resolution of the LSFM.

Doxorubicin treatment

Stock Dox (Sigma Aldrich) was reconstituted in Milli-Q water and stored at 4°C in the dark. At 24 hpf, embryos from the same clutch were randomly selected to be treated with 100 μM Dox or an equivalent volume of Milli-Q (vehicle) diluted in EM. Embryos were kept in their respective solution until imaging at 48 hpf, prior to which they were briefly rinsed in EM to reduce autofluorescence caused by excess Dox. Data were collected over two separate experiments performed on different days and with different animals.

Isoproterenol treatment

Isoproterenol hydrochloride (Sigma Aldrich) was reconstituted in EM fresh before each use. At 48 hpf, embryos were randomly selected within each clutch to be treated with 20 μM isoproterenol in EM for 2 h prior to imaging. Embryos were then anesthetized and imbedded in agarose as described above, and re-submerged in EM containing 20 μM isoproterenol for the duration of imaging. Sibling controls were processed and imaged as described above. Data were collected over two separate experiments performed on different days and with different animals.

Functional analysis

FS was measured in ImageJ using end-diastolic dimension (EDD) and end-systolic dimension (ESD) of LSFM optical sections. FS was then defined as (EDD–ESD)/EDD×100. Optical sections through the middle of each chamber were chosen for measurements. Chamber diameters were measured within each frame of the optical sections to ensure an accurate description of both diastole and systole. Estimated EF was calculated similarly to previously described methods (Shin et al., 2010), which were modified to assume a spherical, rather than prolate spheroid, chamber morphology because the relative angle of the light sheet precludes measurement of the long axis at a single z-depth. FS and EF/CO measurements were derived using the same datasets in order to eliminate variability between cohorts. Each point within Fig. 4B-H represents data from distinct hearts within its respective analysis. The data presented in Fig. S3 represents the same Dox experiment with increased step size simulated through data subtraction. Significance in all functional analyses was determined using the Student's two-tailed t-test.