From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification
Organizations: Faculty of IT, Technical University of Communication and Informatics, Moscow, Russia, 111024 · AI lab, Avito, Moscow, Russia, 125196
Abstract
Automated animal identification is a practical task for reuniting lost pets with their owners, yet current systems often struggle due to limited dataset scale and reliance on unimodal visual cues. This study introduces a multimodal verification framework that enhances visual features with semantic identity priors derived from synthetic textual descriptions. We constructed a massive training corpus of 1.9 million photographs covering 695,091~unique animals to support this investigation. Through systematic ablation studies, we identified SigLIP2-Giant and E5-Small-v2 as the optimal vision and text backbones. We further evaluated fusion strategies ranging from simple concatenation to adaptive gating to determine the best method for integrating these modalities. Our proposed approach utilizes a gated fusion mechanism and achieved a Top-1 accuracy of 84.28% and an Equal Error Rate of 0.0422 on a comprehensive test protocol. These results represent an 11% improvement over leading unimodal baselines and demonstrate that integrating synthesized semantic descriptions significantly refines decision boundaries in large-scale pet re-identification.\\ Conference update. This work was accepted to the FGVC13 Workshop at CVPR 2026. We thank Vadim Ezhov and Nikolai Shcheglov for their help with the poster presentation and demonstration at the conference.
Figures & tables
| Gap | Representative Works | How This Work Addresses It |
| No unified evaluation protocol | ( Mougeot et al., 2019 ; Zhang et al., 2024 ; Kang and Oh, 2025 ; Vidal et al., 2021 ) | We combine five large training datasets into a unified corpus and define a single standardized training and evaluation protocol used for all backbones and configurations. |
| Unimodality | ( Mougeot et al., 2019 ; Cermak et al., 2025 ; Kang and Oh, 2025 ; Vidal et al., 2021 ) | We introduce a multimodal pipeline that augments visual embeddings with identity-level textual descriptions. |
| No systematic ablation studies | ( Mougeot et al., 2019 ; Zhang et al., 2024 ; Cermak et al., 2025 ; Kang and Oh, 2025 ; Vidal et al., 2021 ) | We perform controlled ablations over multiple vision encoders, text encoders, and training configurations, isolating their contributions to verification performance on a common benchmark suite. |
| No controlled comparison of multimodal fusion strategies | ( Mougeot et al., 2019 ; Zhang et al., 2024 ; Kang and Oh, 2025 ; Li et al., 2025 ; Vidal et al., 2021 ) | We implement and compare several fusion mechanisms under identical settings, identifying effective designs for animal verification. |
| Dataset | Identities | Photos | Usage | Min | Max | Mean | Med | Std |
| Pet911 (ours) | 22,050 | 65,961 | Training | 2 | 10 | 2.99 | 3 | 1.34 |
| Telegram (ours) | 73,101 | 131,698 | Training | 2 | 10 | 2.73 | 2 | 1.05 |
| Dogs-World ( Toumbourou, 2022 ) | 200,458 | 301,342 | Training | 2 | 17 | 2.36 | 2 | 0.63 |
| LCW ( DSEIDLI, 2023 ) | 140,732 | 381,267 | Training | 2 | 6 | 3.71 | 2 | 1.45 |
| PetFace ( Shinoda and Shiohara, 2024 ) | 257,349 | 1,001,532 | Training | 3 | 11 | 4.58 | 4 | 1.29 |
| Cat Individual Images ( Lin, 2019 ) | 518 | 13,542 | Evaluation | 6 | 144 | 25.58 | 20 | 10.06 |
| Configuration | Parameters | Mult-Adds | VRAM | Epoch Time | Throughput |
| CLIP-ViT-Base | 151,277,313 | 201,094,656 | 375.24 | 7641 | 211 |
| DINOv2-Small | 22,056,576 | 79,195,392 | 201.66 | 2497 | 224 |
| SigLIP-Base | 203,155,970 | 205,686,528 | 500.96 | 4354 | 151 |
| SigLIP2-Base | 375,187,970 | 205,686,528 | 500.96 | 4271 | 160 |
| Zer0int CLIP-L | 427,616,513 | 457,503,744 | 1696.21 | 10,041 | 66 |
| SigLIP2-Giant | 1,871,885,426 | 1,833,395,712 | 7394.99 | 64,641 | 10 |
| Configuration | Parameters | Mult-Adds | VRAM | Epoch Time | Throughput |
| BERT | 109,482,240 | 109,482,240 | 519.65 | 669 | 990 |
| E5-Base | 109,482,240 | 109,482,240 | 519.65 | 688 | 985 |
| E5-Base-v2 | 109,482,240 | 109,482,240 | 519.65 | 670 | 963 |
| E5-Small | 33,360,000 | 33,360,000 | 178.26 | 657 | 986 |
| E5-Small-v2 | 33,360,000 | 33,360,000 | 178.26 | 661 | 997 |
| Configuration | Parameters | Mult-Adds | VRAM | Epoch Time | Throughput |
| CLIP-ViT-Base + E5-Base-v2 | 261,415,937 | 311,233,280 | 897.41 | 2813 | 203 |
| CLIP-ViT-Base + E5-Small-v2 | 185,097,089 | 234,914,432 | 555.27 | 2788 | 204 |
| CLIP-ViT-Base + E5-Small-v2 + cross-attention | 187,988,865 | 235,704,960 | 588.3 | 2952 | 195 |
| CLIP-ViT-Base + E5-Small-v2 + weighted text | 185,097,090 | 234,914,432 | 555.27 | 2905 | 194 |
| SigLIP2-Giant + E5-Small-v2 | 1,907,803,890 | 1,869,314,176 | 7557.53 | 63,959 | 10 |
| SigLIP2-Giant + E5-Small-v2 + cross-attention | 1,907,805,426 | 1,868,265,088 | 7556.02 | 63,296 | 10 |
| Configuration | Parameters | Mult-Adds | VRAM | Throughput |
| MiewID-msv3 | 51,109,277 | 24,307,036,098 | 891.97 | 159 |
| MD-T-CNN-288 | 12,233,232 | 1,589,219,176 | 211.22 | 353 |
| MD-CLIP-336 | 303,507,456 | 649,129,984 | 2360.4 | 36 |
| MD-L-384 | 195,198,516 | 445,579,776 | 1960.17 | 46 |
| BioCLIP | 86,192,640 | 172,314,624 | 391.57 | 267 |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| Train data | 0.9557 | 0.1042 | 0.6392 | 0.8088 | 0.8535 |
| Train data + YOLO Preprocessing | 0.9626 | 0.0978 | 0.6282 | 0.7826 | 0.8256 |
| Train data + PetFace(train split) | 0.9598 | 0.0982 | 0.6460 | 0.8093 | 0.8504 |
| Train data + Telegram | 0.9716 | 0.0818 | 0.6791 | 0.8329 | 0.8713 |
| Train data + Pet911 + Telegram | 0.9729 | 0.0772 | 0.6527 | 0.8139 | 0.8568 |
| Train data + PetFace (full) + Pet911 + Telegram | 0.9752 | 0.0729 | 0.6511 | 0.8122 | 0.8555 |
| Configuration | Train Data | Train Data + YOLO Preprocessing | Train Data + PetFace (Train Split) | Train Data + Telegram | Train Data + Pet911 + Telegram |
| Train data + YOLO Preprocessing | 0.0 (↑) | - | - | - | - |
| Train data + PetFace (train split) | 0.000012 (↑) | 0.593305 (↑) | - | - | - |
| Train data + Telegram | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | - | - |
| Train data + Pet911 + Telegram | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | 0.000152 (↑) | - |
| Train data + PetFace (full) + Pet911 + Telegram | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | 0.001651 (↑) |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| Train data | 0.9753 | 0.0761 | 0.8082 | 0.9475 | 0.9655 |
| Train data + YOLO Preprocessing | 0.9783 | 0.0708 | 0.8411 | 0.9601 | 0.9721 |
| Train data + PetFace(train split) | 0.9772 | 0.0713 | 0.8491 | 0.9677 | 0.9790 |
| Train data + Telegram | 0.9857 | 0.0528 | 0.8575 | 0.9737 | 0.9832 |
| Train data + Pet911 + Telegram | 0.9822 | 0.0593 | 0.8296 | 0.9561 | 0.9712 |
| Train data + PetFace (full) + Pet911 + Telegram | 0.9821 | 0.0604 | 0.8359 | 0.9579 | 0.9711 |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| Train data | 0.9653 | 0.0920 | 0.4408 | 0.6477 | 0.7241 |
| Train data + YOLO Preprocessing | 0.9494 | 0.1207 | 0.3802 | 0.5772 | 0.6575 |
| Train data + PetFace(train split) | 0.9735 | 0.0774 | 0.4098 | 0.6264 | 0.7029 |
| Train data + Telegram | 0.9715 | 0.0866 | 0.4705 | 0.6688 | 0.7415 |
| Train data + Pet911 + Telegram | 0.9727 | 0.0778 | 0.4460 | 0.6479 | 0.7239 |
| Train data + PetFace (full) + Pet911 + Telegram | 0.9739 | 0.0772 | 0.4350 | 0.6417 | 0.7204 |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| CLIP-ViT-Base | 0.9752 | 0.0729 | 0.6511 | 0.8122 | 0.8555 |
| DINOv2-Small | 0.9848 | 0.0546 | 0.7180 | 0.8678 | 0.9009 |
| SigLIP-Base | 0.9811 | 0.0572 | 0.7359 | 0.8831 | 0.9140 |
| SigLIP2-Base | 0.9793 | 0.0631 | 0.7400 | 0.8889 | 0.9197 |
| Zer0int CLIP-L | 0.9842 | 0.0565 | 0.7626 | 0.8994 | 0.9267 |
| SigLIP2-Giant | 0.9912 | 0.0378 | 0.8243 | 0.9471 | 0.9641 |
| Configuration | CLIP-ViT-Base | DINOv2-Small | SigLIP-Base | SigLIP2-Base | Zer0int CLIP-L |
| DINOv2-Small | 0.0 (↑) | - | - | - | - |
| SigLIP-Base | 0.0 (↑) | 0.005463 (↓) | - | - | - |
| SigLIP2-Base | 0.0 (↑) | 0.0 (↓) | 0.0 (↓) | - | - |
| Zer0int CLIP-L | 0.0 (↑) | 0.134735 (↓) | 0.30085 (↓) | 0.0 (↓) | - |
| SigLIP2-Giant | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| CLIP-ViT-Base | 0.9821 | 0.0604 | 0.8359 | 0.9579 | 0.9711 |
| DINOv2-Small | 0.9904 | 0.0422 | 0.8547 | 0.9660 | 0.9764 |
| SigLIP-Base | 0.9899 | 0.0390 | 0.8649 | 0.9757 | 0.9842 |
| SigLIP2-Base | 0.9894 | 0.0388 | 0.8660 | 0.9772 | 0.9863 |
| Zer0int CLIP-L | 0.9881 | 0.0509 | 0.8768 | 0.9767 | 0.9845 |
| SigLIP2-Giant | 0.9940 | 0.0344 | 0.8899 | 0.9868 | 0.9921 |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| CLIP-ViT-Base | 0.9739 | 0.0772 | 0.4350 | 0.6417 | 0.7204 |
| DINOv2-Small | 0.9829 | 0.0571 | 0.5581 | 0.7540 | 0.8139 |
| SigLIP-Base | 0.9792 | 0.0606 | 0.5848 | 0.7746 | 0.8319 |
| SigLIP2-Base | 0.9776 | 0.0672 | 0.5925 | 0.7856 | 0.8422 |
| Zer0int CLIP-L | 0.9814 | 0.0625 | 0.6289 | 0.8092 | 0.8597 |
| SigLIP2-Giant | 0.9926 | 0.0326 | 0.7475 | 0.9009 | 0.9316 |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| BERT | 0.9698 | 0.0736 | 0.3675 | 0.5717 | 0.6554 |
| E5-Base | 0.9703 | 0.0740 | 0.3733 | 0.5748 | 0.6595 |
| E5-Base-v2 | 0.9697 | 0.0755 | 0.3699 | 0.5743 | 0.6575 |
| E5-Small | 0.9693 | 0.0764 | 0.3795 | 0.5783 | 0.6615 |
| E5-Small-v2 | 0.9700 | 0.0757 | 0.3736 | 0.5731 | 0.6597 |
| Configuration | BERT | E5-Base | E5-Base-v2 | E5-Small |
| E5-Base | 0.696150 (↓) | - | - | - |
| E5-Base-v2 | 0.013254 (↓) | 0.029675 (↓) | - | - |
| E5-Small | 0.044146 (↑) | 0.093004 (↓) | 0.771638 (↑) | - |
| E5-Small-v2 | 0.016518 (↑) | 0.041364 (↓) | 1.000000 (↓) | 0.731556 (↓) |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| BERT | 0.9707 | 0.0726 | 0.5301 | 0.7397 | 0.8060 |
| E5-Base | 0.9706 | 0.0739 | 0.5396 | 0.7464 | 0.8157 |
| E5-Base-v2 | 0.9708 | 0.0751 | 0.5386 | 0.7484 | 0.8131 |
| E5-Small | 0.9693 | 0.0764 | 0.5510 | 0.7504 | 0.8188 |
| E5-Small-v2 | 0.9700 | 0.0754 | 0.5373 | 0.7439 | 0.8146 |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| BERT | 0.9756 | 0.0674 | 0.1776 | 0.3745 | 0.4791 |
| E5-Base | 0.9758 | 0.0674 | 0.1789 | 0.3731 | 0.4765 |
| E5-Base-v2 | 0.9759 | 0.0676 | 0.1721 | 0.3713 | 0.4751 |
| E5-Small | 0.9745 | 0.0685 | 0.1804 | 0.3761 | 0.4772 |
| E5-Small-v2 | 0.9753 | 0.0692 | 0.1821 | 0.3743 | 0.4788 |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| CLIP-ViT-Base | 0.9752 | 0.0729 | 0.6511 | 0.8122 | 0.8555 |
| CLIP-ViT-Base + E5-Base-v2 | 0.9711 | 0.0768 | 0.6331 | 0.8130 | 0.8561 |
| CLIP-ViT-Base + E5-Small-v2 | 0.9707 | 0.0800 | 0.6454 | 0.8193 | 0.8617 |
| CLIP-ViT-Base + E5-Small-v2 + cross-attention | 0.9807 | 0.0627 | 0.6409 | 0.8149 | 0.8589 |
| CLIP-ViT-Base + E5-Small-v2 + weighted text | 0.9703 | 0.0793 | 0.6696 | 0.8371 | 0.8806 |
| SigLIP2-Giant | 0.9912 | 0.0378 | 0.8243 | 0.9471 | 0.9641 |
| Configuration | CLIP-ViT-Base + E5-Base-v2 | CLIP-ViT-Base + E5-Small-v2 | CLIP-ViT-Base + E5-Small-v2 + Cross-Attention | CLIP-ViT-Base + E5-Small-v2 + Weighted Text | SigLIP2-Giant + E5-Small-v2 | SigLIP2-Giant + E5-Small-v2 + Cross-Attention | SigLIP2-Giant + E5-Small-v2 + Weighted Text |
| CLIP-ViT-Base + E5-Small-v2 | 0.004174 (↓) | - | - | - | - | - | - |
| CLIP-ViT-Base + E5-Small-v2 + cross-attention | 0.0 (↓) | 0.0 (↓) | - | - | - | - | - |
| CLIP-ViT-Base + E5-Small-v2 + weighted text | 0.044237 (↓) | 0.524457 (↑) | 0.0 (↑) | - | - | - | - |
| SigLIP2-Giant + E5-Small-v2 | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | - | - | - |
| SigLIP2-Giant + E5-Small-v2 + cross-attention | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | 0.002496 (↑) | - | - |
| SigLIP2-Giant + E5-Small-v2 + weighted text | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | - |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| CLIP-ViT-Base | 0.9821 | 0.0604 | 0.8359 | 0.9579 | 0.9711 |
| CLIP-ViT-Base + E5-Base-v2 | 0.9824 | 0.0617 | 0.8004 | 0.9445 | 0.9634 |
| CLIP-ViT-Base + E5-Small-v2 | 0.9839 | 0.0596 | 0.8141 | 0.9532 | 0.9694 |
| CLIP-ViT-Base + E5-Small-v2 + cross-attention | 0.9807 | 0.0627 | 0.8062 | 0.9477 | 0.9669 |
| CLIP-ViT-Base + E5-Small-v2 + weighted text | 0.9843 | 0.0587 | 0.8176 | 0.9497 | 0.9693 |
| SigLIP2-Giant | 0.9940 | 0.0344 | 0.8899 | 0.9868 | 0.9921 |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| CLIP-ViT-Base | 0.9739 | 0.0772 | 0.4350 | 0.6417 | 0.7204 |
| CLIP-ViT-Base + E5-Base-v2 | 0.9739 | 0.0744 | 0.4371 | 0.6589 | 0.7309 |
| CLIP-ViT-Base + E5-Small-v2 | 0.9717 | 0.0817 | 0.4479 | 0.6628 | 0.7358 |
| CLIP-ViT-Base + E5-Small-v2 + cross-attention | 0.9715 | 0.0836 | 0.4469 | 0.6593 | 0.7326 |
| CLIP-ViT-Base + E5-Small-v2 + weighted text | 0.9786 | 0.0676 | 0.4960 | 0.7054 | 0.7768 |
| SigLIP2-Giant | 0.9926 | 0.0326 | 0.7475 | 0.9009 | 0.9316 |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| MiewID-msv3 | 0.8484 | 0.2442 | 0.7270 | 0.8468 | 0.8685 |
| MiewID-msv3 | 0.9227 | 0.1508 | 0.5569 | 0.7057 | 0.7447 |
| MD-T-CNN-288 | 0.7592 | 0.3082 | 0.1599 | 0.2515 | 0.2955 |
| MD-CLIP-336 | 0.7767 | 0.2998 | 0.3410 | 0.4648 | 0.5138 |
| MD-L-384 | 0.9070 | 0.1673 | 0.4607 | 0.6170 | 0.6695 |
| BioCLIP | 0.8383 | 0.2465 | 0.3720 | 0.5047 | 0.5539 |
| Configuration | MiewID-msv3 | MD-T-CNN-288 | MD-CLIP-336 | MD-L-384 | BioCLIP |
| MD-T-CNN-288 | 0.0 (↓) | - | - | - | - |
| MD-CLIP-336 | 0.0 (↓) | 0.0 (↑) | - | - | - |
| MD-L-384 | 0.0 (↓) | 0.0 (↓) | 0.0 (↑) | - | - |
| BioCLIP | 0.0 (↓) | 0.0 (↓) | 0.0 (↓) | 0.0 (↓) | - |
| SigLIP2-Giant + E5-Small-v2 + gating | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) | 0.0 (↑) |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| MiewID-msv3 | 0.9383 | 0.1392 | 0.8723 | 0.9695 | 0.9760 |
| MD-T-CNN-288 | 0.8695 | 0.2111 | 0.7347 | 0.8546 | 0.8743 |
| MD-CLIP-336 | 0.8953 | 0.1848 | 0.8182 | 0.9158 | 0.9277 |
| MD-L-384 | 0.9364 | 0.1401 | 0.8421 | 0.9464 | 0.9567 |
| BioCLIP | 0.8754 | 0.2141 | 0.8473 | 0.9478 | 0.9585 |
| SigLIP2-Giant + E5-Small-v2 + gating | 0.9929 | 0.0344 | 0.8952 | 0.9872 | 0.9932 |
| Configuration | ROC AUC | EER | Top-1 | Top-5 | Top-10 |
| MiewID-msv3 | 0.9227 | 0.1508 | 0.5569 | 0.7057 | 0.7447 |
| MD-T-CNN-288 | 0.7592 | 0.3082 | 0.1599 | 0.2515 | 0.2955 |
| MD-CLIP-336 | 0.7767 | 0.2998 | 0.3410 | 0.4648 | 0.5138 |
| MD-L-384 | 0.9070 | 0.1673 | 0.4607 | 0.6170 | 0.6695 |
| BioCLIP | 0.8383 | 0.2465 | 0.3720 | 0.5047 | 0.5539 |
| SigLIP2-Giant + E5-Small-v2 + gating | 0.9920 | 0.0314 | 0.7818 | 0.9233 | 0.9482 |
Animal Re-Identification
Animal re-identification matches observations of the same individual across time, cameras, viewpoints, and changing conditions.
See all 33 papers