CounterFactual Retrieval for Evidence-Aware Individual Animal Identification
Authors: Ankit K. Upadhyay, Aleksandr Algasov, Ekaterina Nepovinnykh, Fedor Zolotarev, Tuomas Eerola, Heikki Kälviäinen, Charles V. Stewart, Lukas Picek
Organizations: Rensselaer Polytechnic Institute · LUT University · Brno University of Technology · University of Helsinki · MIT · University of West Bohemia
Automated identification of individual animals is moving from small curated studies toward shared benchmarks and reusable foundation models. This progress is useful, but current evaluation leaves the evidence behind a match unresolved, i.e., high retrieval accuracy does not show whether a model used animal features or identity-correlated shortcuts. We address this with Counterfactual Foreground Retrieval (CFR), an evidence-aware evaluation protocol. CFR evaluates the same query-gallery task under different views: full image, foreground-only, and shortcut-only. The full-image measures standard retrieval performance; the foreground-only measures foreground-supported retrieval; and shortcut-only view measure how much identity signal is recoverable from background, or silhouette. This turns a retrieval score into an evidence profile: how well the model performs on the original images, how much of that performance remains when only the animal is visible, and how much identity signal remains recoverable when the animal is removed. We release around 0.5M masks for 34 datasets and code for mask generation and CFR evaluation.
Figures & tables
Figure 1: Strong retrieval scores depend on different evidence. Standard retrieval reports whether the correct individual is retrieved, but not which visual evidence supports it. We therefore introduce the Shortcut Signal Indicator (SSI), which measures how much of the full-image retrieval signal can be recovered from shortcut-only inputs. Among otherwise strong results, SSI ranges from almost zero to almost one, showing why retrieval performance should be evaluated along with the evidence behind it.
Dataset
Model
Random
Full image
Foreg.
Shortcut
WhaleSharkID [ 19 ]
DINOv3
0.5
35.3
30.3
19.1
MegaD C
0.5
30.5
24.8
16.3
LeopardID2022 [ 47 ]
DINOv3
1.1
57.4
52.2
46.3
MegaD C
1.1
51.6
44.0
40.5
Saimaa seal [ 41 ]
DINOv3
0.7
46.6
24.4
39.1
MegaD C
0.7
36.7
28.3
28.9
Table 1: Standard evaluation hides shortcut signal. Even in the zero-shot setting, the shortcut-only view can retrieve correct identities above random chance. This indicates that full-image Retrieval@ k can reflect not only animal-visible evidence, but also identity-correlated background or silhouette cues. C –not trained on any datasets in this table.
Figure 2: CounterFactual views. For each image, the provided mask allows us to generate matched full-image, foreground-only, and shortcut-only inputs. Such views alter the available visual evidence and enable evidence-aware evaluation.
Dataset
Model
Random
Full
Frg.
SC
SSI
CFR
High shortcut recoverability
OpenCows2020 [ 4 ]
MiewID
3.7
99.1
98.4
73.3
73.0
98.4
MegaDes.
3.7
100.0
99.0
70.9
69.9
99.0
AAUZebraFish [ 9 ]
MiewID
16.7
98.3
97.6
90.2
90.1
97.1
MegaDes.
16.7
100.0
99.3
89.6
87.5
99.1
ATRW [ 25 ]
MiewID
1.2
94.1
91.0
86.0
91.4
90.9
Table 2: Counterfactual foreground retrieval performance. Re-evaluating MiewID and MegaDescriptor on selected representative datasets shows that full-image Retrieval@1 often hides qualitatively different evidence profiles. Some datasets are cleanly foreground-supported, some are overdetermined by both animal and shortcut evidence, and others show weak foreground support despite high shortcut recoverability. See full results in Tab. S4 .
Figure 3: CFR vs. SSI . Seemingly similarly performing models can use different types of evidence (colored samples). Numbered points correspond to the dataset indices in Tab. S4 .
Figure 4: Evidence-profile changes under foreground-mixed training. For each of the 34 datasets, × marks the full-image baseline and its arrow ends at the corresponding foreground-mixed checkpoint. ◯ denotes the 22 represented datasets and △ the 12 unseen datasets. Darker arrows and outlined endpoints mark transitions across the descriptive 0.5 boundaries.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Prompt(s)
Masks
[%]
AAUZebraFish [ 9 ]
zebrafish, fish
6,671
100
ATRW [ 25 ]
tiger
5,398
99.7
AerialCattle2017 [ 5 ]
cow, cattle
30,010
64.8
Amvrakikos turtles [ 3 ]
sea turtle head, sea turtle, turtle
195
97.5
BelugaID [ 45 ]
beluga whale, whale
4,916
83.3
BirdIndividualID [ 15 ]
bird
52,076
98.7
Appendix
Table S1 : SAM3-assisted mask extraction details. Prompts used for each dataset, together with the number of accepted masks.
Median IoU
Mean IoU
IoU >90%
Dataset
(%)
(%)
(%)
BrownBearHeads [ 34 ]
99.3
98.9
100.0
CzechLynx [ 33 ]
98.3
97.6
94.0
HyenaID2022 [ 46 ]
97.9
97.2
96.0
LeopardID2022 [ 47 ]
96.2
95.1
92.0
NyalaData [ 14 ]
97.0
87.3
90.0
Appendix
Table S2: Manual mask-quality audit of SAM masks. IoU >90% reports the percentage of manual reference masks whose SAM mask exceeds 90% IoU.
Figure S1 : Robustness of CFR and SSI to controlled mask and image perturbations. CFR and SSI remain stable under moderate perturbations and change primarily when discriminative animal evidence is removed, consistent with their intended interpretation. Each point represents one paired evaluation on 34 datasets using two models (e.g., MegaDescriptor and MiewID), comparing the clean metric with its perturbed counterpart. Rows correspond to increasing perturbation severity (low, medium, high), and columns show Clear CFR and SSI. ◯ indicates mask perturbations, □ generic image degradations, and △ targeted evidence removal (e.g., foreground / background CutOut). The orange curve shows the median perturbed value in ten equal-width bins of the clean metric, with interquartile-range error bars.
Dataset
Species or group
Image conditions
CFR unit
Intended biometric
Silhouette role
AAUZebraFish [ 9 ]
Zebrafish (teleost fish)
Controlled tank video
Image/crop
Stripe pattern, color, morphology
Ambiguous
ATRW [ 25 ]
Amur tiger (felid)
Zoo video, varied pose
Image/crop
Coat stripe pattern
Shortcut
AerialCattle2017 [ 5 ]
Holstein-Friesian cattle (bovid)
Outdoor drone video
Image/crop
Coat pattern
Ambiguous
Amvrakikos turtles [ 3 ]
Loggerhead turtle (reptile)
Boat-based head photographs
Head bbox
Facial scale pattern
Shortcut
BelugaID [ 45 ]
Beluga whale (cetacean)
Aerial/field top-view images
Image/crop
Scars, pigmentation, dorsal ridge
Ambiguous
BirdIndividualID [ 15 ]
Three bird species
Feeder cameras, captive images
Image/crop
Plumage and facial appearance
Ambiguous
Appendix
Table S3 : Characteristics of the CFR benchmark datasets. A systematic summary of the evaluation datasets, complementing the mask-coverage details in Table S1 . CFR unit is the localized region actually embedded (whole image, body/head crop, or bounding box), not every annotation released with the source dataset. Intended biometric is the animal-intrinsic evidence each dataset targets; the foreground-only view is the input, and identity is carried by this evidence inside it. Silhouette role records whether body outline is a shortcut (outline is not the intended biometric), identity (outline is the biometric, e.g., a fin or fluke contour), or ambiguous (both readings are plausible); it sets how silhouette enters Sshortcut and SSI (Sec. 3.5 ). Descriptions follow the WildlifeDatasets overview [ 12 ] .