Predicting Blastocyst Formation in IVF: Integrating DINOv2 and Attention-Based LSTM on Time-Lapse Embryo Images
Authors: Zahra Asghari Varzaneh, Niclas Wölner-Hanssen, Reza Khoshkangini, Thomas Ebner, Magnus Johnsson
Organizations: Sustainable Digitalisation Research Center, Department of Computer Science and Media Technology, Malmö University, Malmö, Sweden · School of Information Technology, Halmstad University, Halmstad, Sweden · Kepler Universitätsklinikum, Linz, Austria · Department of Computer Science, Kristianstad University, Kristianstad, Sweden
The selection of the optimal embryo for transfer is a critical yet challenging step in in vitro fertilization (IVF), primarily due to its reliance on the manual inspection of extensive time-lapse imaging data. A key obstacle in this process is predicting blastocyst formation from the limited number of daily images available. Many clinics also lack complete time-lapse systems, so full videos are often unavailable. In this study, we aimed to predict which embryos will develop into blastocysts using limited daily images from time-lapse recordings. We propose a novel hybrid model that combines DINOv2, a transformer-based vision model, with an enhanced long short-term memory (LSTM) network featuring a multi-head attention layer. DINOv2 extracts meaningful features from embryo images, and the LSTM model then uses these features to analyze embryo development over time and generate final predictions. We tested our model on a real dataset of 704 embryo videos. The model achieved 96.4% accuracy, surpassing existing methods. It also performs well with missing frames, making it valuable for many IVF laboratories with limited imaging systems. Our approach can assist embryologists in selecting better embryos more efficiently and with greater confidence.
Figures & tables
Figure 1: Time-lapse images of the 16 stages of embryonic development [ 10 ] .
Stage
Definition
Stage
Definition
tPB2
second polar body appearance
t7
seventh cell appears
tPNa
pronuclei appearance
t8
eight cell appears
tPNf
pronuclei fading
t9
ninth cell appears
t2
second cell appears
tM
morula formation
t3
third cell appears
tSB
blastulation start
t4
fourth cell appears
tB
blastocyst formation
Table 1: Definitions of development stage labels assigned to individual images
Figure 2: Proportion of blastocyst formation over time. The blue line tracks the proportion of blastocyst-formatted embryos out of the total 704 embryos. The grey-dashed line follows the Active Embryos over time (Notice that some of the embryos are not annotated until +24 h).
Figure 3: An overview of our proposed hybrid model:The process begins with DINOv2 extracting features from embryo frames, followed by temporal analysis using an LSTM with Multi-Head Attention and hyperparameter tuning. The model then classifies sequences into Blasto or Non-Blasto categories.
Figure 4: A framework of DINOv2 architecture
Figure 5: A framework of LSTM-Multi-Head Attention fusion. An architecture combining stacked LSTMs for temporal feature extraction with Multi-Head Attention to capture long-range dependencies, followed by normalization and classification layers.
Figure 7
Figure 8: The training history for loss and accuracy metrics. An early stopping patience of 30 epochs was set; however, as the curves show, the model converged and showed no significant improvement after about epoch 20, confirming the early termination of training to prevent overfitting.
Model
Class
Precision
Recall
F1-Score
LSTM-MHA Fusion
Blastocyst
0.971±0.016
0.971±0.018
0.971±0.014
Non-Blastocyst
0.918±0.015
0.918±0.015
0.918±0.009
Kalyani et al. [ 13 ]
Blastocyst
0.93
0.98
0.95
Non-Blastocyst
0.91
0.77
0.83
Table 2: Performance comparison of different models for each class
Model
Classification Type
Accuracy
CNN
Binary
0.64
CNN with GA
Binary / Multiclass
0.72 / 0.35
DenseNet201–LSTM
Binary
0.76
ResNet50–LSTM (Kalyani et al. [ 13 ] )
Binary
0.85
ResNet50–GRU (Kalyani et al. [ 13 ] )
Binary
0.93
LSTM-MHA Fusion
Binary
0.964±0.017
Table 3: Accuracy and classification types of different models
Dataset
Class
Precision
Recall
F1-Score
Accuracy
Financial Distress
Class 0
0.96
0.92
0.94
0.92
Class 1
0.84
0.93
0.88
Occupancy Detection
Class 0
1.00
0.99
1.00
0.99
Class 1
0.98
1.00
0.99
EEG Eye State
Class 0
0.82
0.84
0.83
0.81
Class 1
0.79
0.78
0.79
Table 4: Performance evaluation of the proposed model on various time-series datasets.
⋆Department of Computer Science and Media Technology, Malmö University, Malmö, Sweden · ◦Faculty of Electrical and Computer Engineering, Qom University of Technology, Iran · †Kepler Universitätsklinikum, Linz, Austria
Department of Computer Science and Media Technology, Malmö University, Malmö, Sweden · Research Environment of Computer Science, Kristianstad University, Sweden · Kepler University Hospital, Krankenhausstr, Linz, Austria +1