cs.CVApr 6, 2026
SaveID-Sim: An Identity-Focused Similarity Metric
Organizations: MIT CSAIL · Adobe Research
Abstract
Humans have remarkable selective sensitivity to identities -- easily distinguishing between highly similar identities, even across significantly different contexts such as diverse viewpoints or lighting. Vision models have struggled to match this capability, and progress toward identity-focused tasks such as personalized image generation is slowed by a lack of identity-focused evaluation metrics. To help facilitate progress, we propose ID-Sim, a feed-forward metric designed to faithfully reflect human selective sensitivity. To build ID-Sim, we curate a high-quality training set of images spanning diverse real-world domains, augmented with generative synthetic data that provides controlled, fine-grained identity and contextual variations. We evaluate our metric on a new unified evaluation benchmark for assessing consistency with human annotations across identity-focused recognition, retrieval, and generative tasks.
Figures & tables
Figure 2: Dataset curation pipeline. We highlight the different real and synthetic data subsets that enable ID-Sim training. Together, they provide high context, domain, and visual identity diversities.
| Dataset | Type | Objects | #Cat | Included in | #Inst |
| ILIAS [ 36 ] | Img | General | N/A | S1 | 281 |
| FORB [ 92 ] | Img | Flat Obj | 7 | S1 | 761 |
| MET [ 100 ] | Img | Artworks | 1 | S1 | 226 |
| GLDv2 [ 90 ] | Img | Landmarks | 12 | S1 | 769 |
| Dogs [ 45 ] | Img | Animal | 1 | S1 | 494 |
| Cats [ 6 ] | Img | Animal | 1 | S1 | 140 |
Table 1: Overview of datasets used for training set curation. # Cat and # Inst refer to number of categories and instances respectively. Colored text indicates different subsets that the dataset images appear in: S1 , S2a , S2b , which can be matched to Figure 2
Figure 3: ID-Sim training pipeline. We train our metric with dual contrastive supervision. At the global level, CLS-token projections for anchor–positive pairs are contrasted against one hard negative and additional batch negatives using InfoNCE. At the patch level, projected patch tokens are compared using Sinkhorn distance for the same instance pairs.
Figure 4: Performance of ID-Sim vs. baseline models . We compare ID-Sim against standard perceptual metrics, large-scale vision foundation models, and a supervised “Universal Embedding” model (the top entry in Google’s universal embedding challenge). Across tasks – instance retrieval, concept preservation, and re-identification – ID-Sim consistently outperforms all baselines, including the instance-retrieval-focused model, despite using over 100× less labeled data and a smaller backbone (our ViT-L vs. ViT-H ). Full results and seed variance are reported in the Supplemental E.2 .
Table 2: Overview of results. (Left): comparison with MLLMs on concept preservation. (Middle): performance across recognition and retrieval datasets. (Right): transfer to personalized segmentation with PerSAM.
Figure 5: Newly annotated Subjects2k. We release a 2k high-quality human annotations with a subset of Subjects200k to serve as a new challenging concept preservation eval benchmark.
| Dataset | Bal. | Pos. | Neg. | Ratio | Val |
| Group | Edit | Edit | Score | ||
| All datasets | ✗ | ✗ | ✗ | – | 0.693 |
| All datasets | ✓ | ✗ | ✗ | – | 0.752 |
| Filtered datasets | ✓ | ✗ | ✗ | – | 0.890 |
| Filtered datasets | ✓ | ✓ | ✗ | 1:1 | 0.937 |
| Filtered datasets | ✓ | ✓ | ✓ | 1:1:1 | 0.965 |
Table 3: Ablation of dataset composition and editing strategies. Balancing and targeted editing of positive and negative samples improve performance.
Figure 6: Selective sensitivity analysis. We evaluate model sensitivity across four axes of visual change: identity, background, viewpoint, and lighting. For 100 anchor instances, we generate controlled variations and compute both sensitivity scores and similarity trends. (Top row.) Compared with baseline methods, our model is notably more sensitive to identity differences while remaining stable under background, viewpoint, and lighting changes. (Bottom row.) When systematically increasing variations across each dimension, we see that, as desired, only identity changes significantly reduce similarity measured by ID-Sim .
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Inst. | Imgs | Cats | Img/Inst | Domain |
| MET [ 100 ] | 734 | 3,429 | – | 2–17 | Art |
| ILIAS [ 36 ] | 900 | 5,326 | – | 2–35 | Everyday Obj |
| FORB [ 92 ] | 4,050 | 16,781 | 7 | 2–22 | Flat Obj |
| GLDv2 [ 90 ] | 4,503 | 81,964 | – | 2–6,272 | Landmarks |
| WildlifeReID10k [ 1 ] | 9,756 | 126,302 | 22 | 1–411 | Animals |
| SOP [ 74 ] | 11,318 | 59,551 | 12 | 2–12 | Products |
Table 1: Initial real instance-level dataset pool. MVImgNet2* is a subset of MVImgNet2 composed of the first two released parts, as the full released dataset contains 180k+ videos. We also use the train_clean split of GLDv2.
| Dataset | Unbalanced | Balanced |
| MET | 105 | 663 |
| ILIAS | 111 | 804 |
| FORB | 592 | 1428 |
| GLDv2 | 625 | 1419 |
| WildlifeReID10k | 1450 | 1418 |
| StanfordOnlineProducts | 1566 | 1435 |
Table 2: Instance counts in the unbalanced vs. balanced 10k training mixtures.
Figure 7: Filtered out FORB logo category. We observe consistent appearance inconsistencies between the same ”instance” category in FORB’s ”logo” class.
Figure 8: Filtered GLDv2 categories. Many GLDv2 classes cover broad geographic areas rather than a single localized site, building or an object, making it difficult for a class to correspond to a consistent visual identity.
| Dataset | Filtered |
| MET | 671 |
| ILIAS | 826 |
| WildlifeReID10k (Dogs and Cats) | 1501 |
| FORB (Filtered) | 2346 |
| GLDv2 (Filtered) | 2315 |
| DeepFashion2 | 2341 |
Table 3: Instance counts for the filtered and balanced 10k training mixture.
| Dataset | Frames per Second (FPS) | Frame Stride | Number of Frames ( ) | Window Size | Number of Sampled Parts |
| LaSOT | 30 | 5 | 6 | 30 (1s) | 6 |
| GOT-10k | 10 | 5 | 2 | 10 (1s) | 5 |
| YouTubeVIS | 6 | 5 | 2 | 10 (1.7s) | 5 |
| UCO3D | 30 | 1 | 30 | 30 (1s) | 5 |
Table 4: Dataset-specific parameters for video frame sampling. About 5-6 frames are sampled from each instance sequence, at intervals that are at 1 second apart.
Figure 9: Generative Contextual Edited Images
Figure 10: Generative Edited Hard-Negatives
| Dataset Scale | Real Instance Validation (Acc) | Identity Validation (ROC AUC) |
| 5k even split | 0.860 | 0.97 |
| 10k even split | 0.870 | 0.97 |
| 20k even split | 0.880 | 0.98 |
| 30k | 0.8825 | 0.91 |
Table 5: Effect of training dataset scale on validation performance.
| Dataset Composition | Real Instance Validation (Acc) | Identity Validation (ROC AUC) |
| Even Split | 0.8715 | 0.97 |
| Subset 1 dominant | 0.8905 | 0.95 |
| Subset 2a dominant | 0.8715 | 0.95 |
| Subset 2b dominant | 0.8685 | 0.96 |
Table 6: Effect of dataset composition on validation performance (fixed 10k scale).
| Backbone / Resolution | Real Instance Validation (Acc) | Identity Validation (ROC AUC) |
| DINOv3-L/16 @ 448 (Baseline) | 0.8715 | 0.965 |
| DINOv3-L/16 @ 224 | 0.8715 | 0.965 |
| DINOv3-B/16 @ 448 | 0.834 | 0.921 |
| DINOv2-L/16 @ 448 | 0.8355 | 0.895 |
| DINOv2-L/16 @ 224 | 0.8135 | 0.819 |
Table 7: Backbone and resolution ablation.
| Feature Loss Setting | Real Instance Validation (Acc) | Identity Validation (ROC AUC) |
| CLS + Patch (Baseline) | 0.8715 | 0.965 |
| Patch Loss Only | 0.8665 | 0.908 |
| CLS Loss Only | 0.8065 | 0.893 |
Table 8: Ablation on CLS vs. patch vs. joint training.
| Objective / Patch Metric | Real Instance Validation (Acc) | Identity Validation (ROC AUC) |
| InfoNCE + Sinkhorn (Baseline) | 0.8715 | 0.965 |
| InfoNCE + Cosine Patch Metric | 0.8655 | 0.940 |
| Hinge + Sinkhorn | 0.8705 | 0.945 |
| BCE + Sinkhorn | 0.8395 | 0.923 |
Table 9: Loss and patch similarity ablation.
Figure 11: PODS Dataset. The dataset is composed of household objects occurring under different distribution shifts, with varying backgrounds, distractor objects, and poses.
Figure 12: DeepFashion2. The DeepFashion2 dataset features query / gallery images of the same clothing item in-shop as well as worn by consumers.
Figure 13: AerialCattle2017. This dataset is composed of aerial imagery of various cows on fields, and the task is to retrieve the same individuals based on a query image.
Figure 14: Petface. Evaluation benchmark of 13 unseen animals. Red depicts different individual and green depicts same individual.
Figure 15: CUTE triplets. Examples of triplets selected for ”Hard” mode. The positive and negative examples are drawn from the same scene type, which may be different from the scene type of the anchor, forcing a match across extrinsic characteristics.
Figure 16: Subjects2k Pairs. Newly annotated 2k subset of Subjects200k [ 81 ] . Green depicts same instance, red is different.
Figure 17: DreamBench Pairs. DreamBench images are accompanied by human annotations out of 4.
Figure 18: Introduction Page for Subjects2k Annotation Server. We provide a clear definition of an instance to all participants prior to starting their annotations.
Figure 19: Subjects2k Annotation Server Task Page. Example of a task page for our annotators.
| Label range | # pairs |
| 788 | |
| 94 | |
| 111 | |
| 96 | |
| 141 | |
| 70 |
Table 10: Subjects2k continuous label histogram. Counts of image pairs falling into each bin of the average human “same” vote fraction .
Figure 20: Human-labeled identity pairs. Left: Examples annotated as “Same.” Right: Examples annotated as “Different.” Human annotators reliably pick up subtle, fine-grained cues—such as texture, pattern, and small structural differences.
Figure 22: Qualitative Results for Per-SAM. We show predicted segmentation masks and corresponding predicted confidence scores, ordered in highest to lowest with respect to a reference object. First, we observe that when combined with PerSAM, both ID-Sim and DINOv3 are able to produce reliable segmentation mask predictions (mask drawn in red around the instance). However, we observe that ID-Sim is significantly better at recognizing instances across distribution shifts and discriminating fine-grained neighbors compared to DINOv3, as shown by the predicted mask scores and the ordering.
Figure 23: Dense masks can resolve ambiguities in multi-object Scenes. Given the test image with 2 shirts (left), ID-Sim features are sensitive to the identity of the query image (right 2 images), evidenced by the patch-level similarity heatmaps (2nd to left).
Figure 26: Background vs. Identity Variation Grid. Rows vary the foreground identity through Qwen-Edit inpainting at increasing edit strengths, while columns vary the scene background using inpainting prompts. Each cell shows the similarity of the edited image to the original anchor. This grid isolates how models respond jointly to identity changes and background shifts.
Figure 27: Viewpoint Variation Grid. Rows vary identity strength and columns sweep natural viewpoint changes using the multi-view MVImgNet sequence. This grid evaluates how well each model maintains invariance to viewpoint while still detecting identity-altering edits.
Figure 28: Lighting Variation Grid. Rows correspond to increasing levels of identity change, while columns apply eight different lighting edits using Qwen-Edit. This grid tests whether models remain stable under illumination changes while remaining sensitive to small identity perturbations.
More in this topic
Image Generation Evaluation
Image generation evaluation measures the quality, fidelity, diversity, and instruction-following ability of AI-generated images.
See all 113 papers