Deep learning as a predictive tool for fetal heart pregnancy following time-lapse incubation and blastocyst transfer.

STUDY QUESTION: Can a deep learning model predict the probability of pregnancy with fetal heart (FH) from time-lapse videos? SUMMARY ANSWER: We created a deep learning model named IVY, which was an objective and fully automated system that predicts the probability of FH pregnancy directly from raw time-lapse videos without the need for any manual morphokinetic annotation or blastocyst morphology assessment. WHAT IS KNOWN ALREADY: The contribution of time-lapse imaging in effective embryo selection is promising. Existing algorithms for the analysis of time-lapse imaging are based on morphology and morphokinetic parameters that require subjective human annotation and thus have intrinsic inter-reader and intra-reader variability. Deep learning offers promise for the automation and standardization of embryo selection. STUDY DESIGN, SIZE, DURATION: A retrospective analysis of time-lapse videos and clinical outcomes of 10 638 embryos from eight different IVF clinics, across four different countries, between January 2014 and December 2018. PARTICIPANTS/MATERIALS, SETTING,
METHODS: The deep learning model was trained using time-lapse videos with known FH pregnancy outcome to perform a binary classification task of predicting the probability of pregnancy with FH given time-lapse video sequence. The predictive power of the model was measured using the average area under the curve (AUC) of the receiver operating characteristic curve over 5-fold stratified cross-validation. MAIN RESULTS AND THE ROLE OF CHANCE: The deep learning model was able to predict FH pregnancy from time-lapse videos with an AUC of 0.93 [95% CI 0.92-0.94] in 5-fold stratified cross-validation. A hold-out validation test across eight laboratories showed that the AUC was reproducible, ranging from 0.95 to 0.90 across different laboratories with different culture and laboratory processes. LIMITATIONS, REASONS FOR CAUTION: This study is a retrospective analysis demonstrating that the deep learning model has a high level of predictability of the likelihood that an embryo will implant. The clinical impacts of these findings are still uncertain. Further studies, including prospective randomized controlled trials, are required to evaluate the clinical significance of this deep learning model. The time-lapse videos collected for training and validation are Day 5 embryos; hence, additional adjustment would need to be made for the model to be used in the context of Day 3 transfer. WIDER IMPLICATIONS OF THE
FINDINGS: The high predictive value for embryo implantation obtained by the deep learning model may improve the effectiveness of previous approaches used for time-lapse imaging in embryo selection. This may improve the prioritization of the most viable embryo for a single embryo transfer. The deep learning model may also prove to be useful in providing the optimal order for subsequent transfers of cryopreserved embryos. STUDY FUNDING/COMPETING INTEREST(S): D.T. is the co-owner of Harrison AI that has patented this methodology in association with Virtus Health. P.I. is a shareholder in Virtus Health. S.C., P.I. and D.G. are all either employees or contracted with Virtus Health. D.G. has received grant support from Vitrolife, the manufacturer of the Embryoscope time-lapse imaging used in this study. The equipment and time for this study have been jointly provided by Harrison AI and Virtus Health.

Introduction

The advent of more physiological culture conditions for the human embryo, conceived through IVF, has led to the routine culture and transfer of embryos at the blastocyst stage (Gardner ; Biggers and Racowsky, 2002). Transfer of a single blastocyst can avoid the many adverse medical conditions for mother and baby associated with multiple gestations (Adashi ; Sullivan ).

However, our ability to select the optimum embryo for transfer has changed very little since the birth of Louise Brown about 40 years ago (Steptoe and Edwards, 1978). From the beginnings of IVF, it was noted that the rate of embryo development was associated with transfer outcome (Edwards ). Subsequently, elegant grading systems have been developed for each successive stage of preimplantation human embryo development (Alpha Scientists in Reproductive Medicine and ESHRE Special Interest Group of Embryology, 2011; Gardner and Balaban, 2016), and each one has been able to relate aspects of discrete stages of development at specific times to viability. In the case of the blastocyst, an alpha numeric system has been in place since the turn of the century, which takes into account both inner cell mass development, as well as the trophectoderm, and its biological activity (measured indirectly through the expansion of the blastocoel cavity itself) (Schoolcraft ; Gardner ). Although such an approach has proved particularly effective in selecting embryos for transfer, data on the metabolic activity of the human blastocyst indicate that Day 5 morphology alone is not the sole predictor of embryonic viability (Gardner ), with glucose consumption being positively correlated with the ability of a blastocyst to give rise to a pregnancy (Gardner )

Prior to the introduction of time-lapse technologies, all embryo assessments were restricted to specific time points during the first 5 days of life. Clearly, the majority of developmental events were not being captured, and the use of time-lapse to image the developing embryo every few minutes has confirmed that several key features of human embryo development were being missed, such as direct cleavage (Rubio ), or were simply not being quantitated, such as the time taken to initiate and complete cavitation. This latter point is of great physiological interest, as it appears tied to both metabolic activity and embryonic ploidy (Desai ). In an attempt to glean more predictive potential from time-lapse images, several algorithms have been created (Motato ; Petersen ; Fishel ), which take into account the annotated time at which key morphological events occur. Data to date have suggested that this approach may have value in embryo selection, and its efficacy has been studied in randomized trials. Results have been variable (Rubio ; Goodman ), but accumulating data infer that key kinetic events are associated with important biological information (Desai ).

However, a current limitation of the analysis of time-lapse images through these algorithms was that it was unfeasible to capture the full temporal and spatial richness of time-lapse video within a few morphokinetic parameters. As such, time-lapse data are currently under-utilized in making predictions about clinical outcomes. Such morphokinetic algorithms also rely on the embryologist to manually annotate the morphological features and morphokinetic timing data. These parameters can then be used as input for statistical or machine-learned scoring tools. Unfortunately, time-lapse annotation and grading is a subjective process with intrinsic inter-reader and intra-reader variability (Venetis ) that will impact the performance of the downstream scoring tools. What determines fetal heart (FH) pregnancy was likely not a simple correlation between a few known features, but rather a complex interaction between many factors across both the temporal and spatial dimension (information acquired through time-lapse microscopy), which may or may not have been identified yet.

Deep learning is a subfield of machine learning that is based on learning hierarchical knowledge from data rather than rule-based programming. Deep learning models are inspired by the biological nervous system in the way that information is processed through interconnecting neurons that are many layers deep (Kim, 2016). Recent advancement in learning algorithms, together with the explosion of electronic medical data and the increase in computational processing power, has seen an explosion of application of artificial intelligence in several aspects of human ART (Bui ; Blank ; Curchoe and Bormann, 2019). A deep learning algorithm can directly analyse the entire raw time-lapse video without the need for annotated parameters, making use of every data point collected from time-lapse to predict the probability of FH pregnancy. The objective of the study was to investigate the hypothesis that a deep learning model named IVY is a valid tool for the prediction of the implantation potential of human preimplantation embryos.

Materials and Methods

Data collection

This study was carried out across eight IVF laboratories in four countries, IVFAustralia (Sydney, Australia), IVFAustralia (Canberra, Australia), Hunter IVF (Newcastle, Australia), Melbourne IVF (Melbourne, Australia), Queensland Fertility Group (Brisbane, Australia) SIMS IVF (Dublin, Ireland), Complete Fertility Centre (Southampton, UK) and Aagard Fertility (Aarhus, Denmark). Each clinic used its own approach to superovulation, egg collection and embryo transfer as shown in Table I. All embryos were cultured in the Embryoscope™ or Embryoscope Plus™ (Vitrolife, Copenhagen, Denmark). The study was a retrospective analysis of the videos obtained from fresh embryos that were fertilized and cultured in a time-lapse incubator in these laboratories from January 2014 to December 2018.

Table I

Number of embryos, patient ages and culture media used in each laboratory.

Laboratory number	Laboratory	Number of embryos studied	Mean age	Age range	Media used in the laboratory
1	IVFAustralia (Sydney, Australia)	1264	36.8	23–46	Vitrolife, Sequential; Vitrolife, Single Stage (G-TL™)
2	IVFAustralia (Canberra, Australia)	150	34.2	24–44	Sage, Sequential
3	Hunter IVF (Newcastle, Australia)	632	34.8	25–43	Vitrolife, Single Stage (G-TL™)
4	Melbourne IVF (Melbourne, Australia)	758	36.6	30–45	Vitrolife, Single Stage (G-TL™)
5	Queensland Fertility Group (Brisbane, Australia)	3827	35.6	22–50	Sage, Sequential; COOK, Sequential; Vitrolife, Single Stage (G-TL™)
6	SIMS IVF (Dublin, Ireland)	1454	35.9	25–46	Vitrolife, Single Stage (G-TL™)
7	Complete Fertility Centre (Southampton, UK)	915	34.7	24–44	Vitrolife, Sequential; Vitrolife, Single Stage (G-TL™)
8	Aagard Fertility (Aarhus, Denmark)	1683	34.2	24–44	Sage 1-step

All embryos cultured to the blastocyst stage in a time-lapse incubator in these laboratories during the time period were studied, regardless of their stage, grade, ploidy, fertilization status and culture method. No embryos cultured in a time-lapse incubator were excluded from the analysis. As a result, the study group included embryos that resulted from fresh oocyte retrieval, vitrified oocyte warming and oocyte donation. Embryos that underwent embryo biopsy for preimplantation genetic testing were included. Embryos that were cryopreserved for later use were studied and the later outcome from the thawed embryo transfer was included in the analysis. Patients from all demographic groups and medical history were included. The mean age was 35.6 years (age range, 22–50 years). No patients were excluded.

The study included 1835 unique treatment cycles from 1648 individual patients. On average, there were 7.9 embryos per treatment. Twenty-eight percent of the embryos transferred were part of a multiple transfer cycle. Twenty-nine percent of the embryos transferred were frozen embryos, and the remainder were fresh embryo transfers. The resulting outcomes for the embryos are shown in Fig. 1.

Figure 1

The outcomes of the embryos being studied. FH, Fetal heart.

Deidentified videos, as well as patient age and the outcome for each embryo, were provided to the researchers by each participating laboratory. All other patient information was held confidentially by the participating laboratory, and no identifying information was made available to the researchers. The outcomes of the analysis were not reported back to the participating laboratories.

The clinical outcome for each embryo was classified as per Table II. The principal endpoint was an FH pregnancy that was defined by the observed presence of an FH on ultrasound at or beyond 7 weeks gestation. The entire dataset includes the 8836 embryos coded as positive or negative, which were used for training and testing of the deep learning model.

Table II

Classification of the outcome of each embryo for training of the deep learning system.

Classification	Outcome
POSITIVE for each embryo involved	FH observed on ultrasound after 7 weeks gestation following a single embryo transfer or multiple FH observed equal to the number of embryos transferred
NEGATIVE for each embryo involved	No pregnancy occurred or no FH was observed on ultrasound after 7 weeks gestation following transfer or embryo discarded because of a failed or abnormal fertilization, grossly abnormal morphology or aneuploidy from preimplantation genetic testing
UNKNOWN for each embryo involved	Multiple embryos transferred and FH(s) seen but the number is fewer than the number transferred
PENDING	Embryo in storage and not yet used

FH, Fetal heart.

Training of the deep learning model

The training process began by randomly partitioning the entire dataset into an 80% training dataset and a 20% testing dataset. The deep learning model named IVY was only trained on the embryos in the training set and subsequently used to make predictions on the held-out testing dataset to estimate its predictive power.

IVY is a feed forward deep learning model that takes nothing but the raw time-lapse video sequence as input and produces a confidence score ranging from 0 to 1 as output that represents the model’s confidence that the input embryo video will lead to an FH pregnancy. The behaviour characteristic of this deep learning model was dictated by a large set of weights that were randomly initialized at the beginning of training; hence, the model starts out making random predictions.

During the training process, batches of time-lapse videos were randomly sampled from the training dataset. The deep learning model then attempted to make predictions on these time-lapse videos and produce confidence scores for each embryo. These predictions were compared with the known target outcomes (0 for negative and 1 for positive) to compute the difference known as the ‘loss’, which represents how far off the predictions was compared to the known outcomes.

The loss value was then used to compute the incremental updates on all of the model’s weights to improve its prediction, a process known as backpropagation. This process was repeated for many thousands of updates to drive down the loss value. As the training progressed, the loss value was continuously reduced closer to 0, which translate to predictions that were closer to the actual FH pregnancy outcome. The training set was looped over randomly for 20 epochs. Each epoch represents a single pass over the dataset. Once the loss value had plateaued, and no more improvement could be made, the training was stopped. The trained model and all its weights were then saved to make predictions against the unseen embryos in the testing dataset. The predictions were compared with the known FH pregnancy outcome from the testing dataset to obtain the performance characteristic of the trained model.

ROC

The performance characteristic of IVY was calculated using the receiver operating characteristic (ROC) curve generated by plotting the true positive rate (sensitivity) against the false positive rate (1-specificity) across all possible thresholding values using the predicted confidence score compared to the actual FH pregnancy outcome. Sensitivity and specificity values can be calculated by selecting a specific thresholding value. A lower threshold value will yield a higher sensitivity and lower specificity while a higher threshold value will yield a lower sensitivity and higher specificity. The nature of this trade-off can be summarized by calculating the area under the curve (AUC) of the ROC curve.

AUC ranged from 0.5 to 1.0, which represents the predictive power of a binary classifier. An AUC of 0.5 represents completely random choices while an AUC of 1 represents perfect discrimination. The higher the AUC, the more favourable the trade-off between sensitivity and specificity. The quantitative value of AUC can also simply be interpreted as the probability that the binary classifier will score a randomly selected positive embryo higher than a randomly selected negative embryo. As a result, AUC was the most appropriate benchmark for a binary classifier’s ability to rank embryos according to their likelihood of creating an FH pregnancy.

5-fold cross-validation

To increase the robustness of the performance estimation we performed 5-fold stratified cross-validation (Kuhn and Johnson, 2013). During this process, the entire dataset was randomly partitioned into five subsets of equal size such that each subset maintained the same prevalence of positive embryos. We subsequently trained five separate deep learning models from scratch on four out of the five subsets and performed validation on the fifth hold-out subset. For each of the training-testing runs, we calculated the AUC on the testing dataset as described above. The final AUC was reported as the mean AUC over five separate training-testing runs. Cross-validation enables the estimate a more reliable performance metric. This methodology lowers the risk of overestimating or underestimating the model true performance characteristic by sampling a uniquely favourable or challenging testing dataset by chance.

Eight laboratories hold-out validation

To investigate the transferability of the deep learning model across different laboratory environments and patient demographics, we also performed an eight-laboratory hold-out validation test (Kuhn and Johnson, 2013). In this approach, the entire dataset was partitioned into eight cohorts according to their laboratory of origin. We subsequently trained another eight deep learning models from scratch by holding out embryos from each of the eight laboratories for validation and performed training on the other seven laboratories’ videos. We then used each of the eight trained models to calculate the AUC on the embryos from the corresponding hold-out laboratories to estimate its predictive power on the untrained lab.

Ethical approval

Under the Australian National Statement on Ethical Conduct in Human Research (National Health and Medical Research Council, 2015), this project was classified as negligible risk and was exempted from ethical review.

Results

Analysis of the ROC is shown in Fig. 2. The resulting AUC of IVY to predict FH pregnancy on the testing dataset was 0.93 (95% CI 0.92–0.94).

Figure 2

ROC curve for prediction of FH pregnancy on the testing dataset by IVY. ROC, Receiver operating characteristic; AUC, area under the curve.

5-fold stratified cross-validation

The results of the 5-fold cross-validation are shown in Table III. The mean AUC over 5-fold stratified cross-validation was 0.93. The AUC was reproducible across five separate training-validation runs.

Table III

Results of the 5-fold cross-validation analysis.

	Fold 1 (n = 1767)	Fold 2 (n = 1767)	Fold 3 (n = 1767)	Fold 4 (n = 1767)	Fold 5 (n = 1768)	AUC
1	Test	Train	Train	Train	Train	0.93
2	Train	Test	Train	Train	Train	0.93
3	Train	Train	Test	Train	Train	0.92
4	Train	Train	Train	Test	Train	0.94
5	Train	Train	Train	Train	Test	0.93
					Mean AUC	0.93

Mean AUC, The mean area under the curve across 5 cross-validation steps.

The results of the eight laboratories hold-out validation are shown in Supplementary Table S1. The validation AUC from each laboratory was very similar indicating excellent transferability across the different laboratory process and clinical environments.

Discussion

The data presented demonstrate that deep learning can predict, with a high degree of probability and reproducibility, the likelihood of a transferred embryo implanting and developing as far as FH. The AUC finding presented here categorizes the diagnostic test quality of IVY as excellent (Šimundić, 2009). Our findings suggest that the combination of time-lapse imaging and deep learning provides an assessment of the embryo viability that is likely to be significantly better than previous algorithm-based approaches.

Previous approaches to morphokinetic analysis (Meseguer ; Conaghan ; Basile ; Milewski ; Fishel ; Liu ) have been based on algorithms that have studied known events in embryological development, such as rapid early cleavage and blastulation, and applied scores to these known events. Deep learning takes an entirely different approach, where time-lapse videos are objectively reviewed with no assumptions at all being made about the significance, or otherwise, of different events in early embryo development.

A number of studies have examined the predictive power of morphokinetic algorithms. A comparison of six previously published embryo selection algorithms to a single set of blastocysts (Barrie ) found that none of the algorithms had an AUC of greater than 0.65. A study from our own group (Storr ) carried out a logistic regression of four different approaches. They found the AUC on ROCs curve varied from 0.585 to 0.748. A further study (Liu ) observed a wide range of predictive powers from 0.509 to 0.762, depending on the algorithm tested.

Previous studies of the predictive capacity of morphokinetics on Day 2 embryos (Ahlstrom ; Milewski ) give comparable results and have suggested that an embryologist approach gives an AUC of 0.74 (Ahlstrom ) while morphokinetics gives an AUC of 0.67 (Ahlstrom ) and 0.70 (Milewski ).

The inbuilt software in the Embryoscope system is the KIDscore (Petersen ) whereby embryos are allocated to one of five categories (KID1–KID5) based on early cleavage events. This approach has previously been studied and found to surpass the predictive value of 0.745 for morphology based on accepted criteria (Alpha Scientists in Reproductive Medicine and ESHRE Special Interest Group of Embryology, 2011).

Two recent reports have added understanding to this field. Blank ) has studied the use of machine learning to combine known morphokinetic algorithms, similar to the above, with clinical predictors of IVF outcome such as parental age, Anti-Mullerian hormone concentration and past pregnancies. Although it is also an application of machine learning to the problem of embryo selection, the overall approach is quite different from the deep learning methodology of the present study with a correspondingly lower AUC (0.74) on ROC analysis.

Khosravi ) have used deep learning to analyse blastocyst images derived from time-lapse imaging and were able to demonstrate a very high ROC (0.98) in being able to predict the embryologist assessment of the embryo. However, unlike our model, this group was unable to demonstrate any direct predictive value for pregnancy. This model differs from our model in being based on only a limited number of images. Our deep learning model studies the whole video and it is likely to be this that gives it the capacity to identify the likelihood of pregnancy with a high degree of predictability.

Clearly, the clinically significant endpoint for any fertility intervention is live birth per cycle started. In this study, we have used FH as the assessment of clinical pregnancy per embryo transferred, as the approach being evaluated is a prediction of the implantation rate for each individual embryo. We acknowledge that the FH rate is clearly only a proxy for live birth, and future clinical studies will need to evaluate the impact on live birth rate. However, at this stage of assessing a rapidly developing technology, the use of live birth as an endpoint would be impractical.

One further limitation of this study that should be noted is that the deep learning system was trained on data from two specific incubator systems (Embryoscope™ and Embryoscope Plus™, Vitrolife) and the applicability to other time-lapse incubator systems remains unclear. However, the different laboratories involved in this study have heterogeneous patient characteristics, apply a wide range of clinical IVF stimulation regimes and use a variety of different culture media. The use of the eight laboratories hold-out validation demonstrates that the predictive value of the deep learning approach is robust despite a wide range of clinical settings of IVF in different parts of the world.

Previous randomized controlled trials of morphokinetics applied to time-lapse images have not provided convincing evidence of any clinical benefit from such approaches. Some studies have not found any benefit of time-lapse images in improving success rates (Ahlstrom ; Goodman ), while others have suggested a benefit in success rate (Rubio ). Further, it is difficult to separate the different effects of the morphokinetic analysis from varying laboratory conditions. Given the relatively low predictive value with the algorithm-based approaches, a paucity of clinical efficacy may not be entirely surprising. The significantly improved predictive value of the deep learning approach such as IVY suggests that further clinical prospective studies based on a deep learning approach are needed to investigate the possibility of a clinical benefit from this approach.

As in all of IVF, the cumulative likelihood of a successful pregnancy for one cycle started is a consequence of a number of factors including the clinical context, the stimulation approach, the culture system used and the clinic’s own inherent success rates. Given modern cryopreservation capacity, the cumulative success rate would be very unlikely to be affected by the ability to predict the implantation rate for a particular embryo. However, the capacity to predict the likelihood of an embryo transfer successfully will result in the embryo with the highest developmental potential being selected first. This would not necessarily affect the cumulative likelihood of obtaining a pregnancy from a batch of embryos cultured from an IVF cycle, but will shorten the time to pregnancy.

The development of deep learning systems may also offer other benefits in the IVF laboratory. Deep learning may increase laboratory efficiency, particularly compared to the time-consuming work involved in annotating morphokinetic parameters as well as eliminating inter-observer variation between embryologists. These potential benefits merit investigation through cost analysis studies.

The data presented here open up the exciting area of investigation of the relationship between the critical features of time-lapse imaging that the deep learning identifies as predictive of implantation and the cellular and physiological events of preimplantation embryo development. Considering that the model operates at raw pixel level across time, macroscopic features such as embryo/cell shape, size and finer features such as texture and movement pattern can all be learned through the training data. The model was trained tabula rasa and decides for itself the significance or otherwise of all these features (or the complex interaction between these features) and assigns predictive weighting accordingly. However, the exact logic utilized by the model, in making its decisions, remains an active area of investigation and will be a topic for future study.

Further, the model presented utilizes the entire video to inform its prediction of the study endpoint in one single pass, which is distinctly different to how embryologist analyse time-lapse video from frame to frame. There may be time intervals in the 5 days of development that are more important than others but this will require further exploration. Future studies to correlate IVY’s prediction with other parameters of embryo physiology, such as metabolic activity and chromosomal constitution, will be of value in advancing our understanding of preimplantation embryo development.

One major question is how these predictive calculations relate to other clinical factors that are known to predict embryo outcomes such as female age and previous reproductive history. The findings presented in this study, quite deliberately, result from the study of each individual embryo from a heterogeneous group of patients and the predictive value that has been derived is independent of age, or any other clinical factors. Further studies investigating the prognostic relationship between known clinical factors and deep learning prediction will therefore be of great value.

Conclusion

In summary, these data demonstrate the potential for deep learning to contribute to clinical IVF in the same way as it has contributed to other areas of human health (Patel ). This study is a retrospective analysis that has demonstrated an effective means of IVY in predicting implantation rate. Further detailed prospective clinical studies are now underway to investigate the clinical impact and cost-effectiveness of this development.

Click here for additional data file.