Humans have remarkable selective sensitivity to identities -- easily distinguishing between highly similar identities, even across significantly different contexts such as diverse viewpoints or lighting. Vision models have struggled to match this capability, and progress toward identity-focused tasks such as personalized image generation is slowed by a lack of identity-focused evaluation metrics. To help facilitate progress, we propose ID-Sim, a feed-forward metric designed to faithfully reflect human selective sensitivity. To build ID-Sim, we curate a high-quality training set of images spanning diverse real-world domains, augmented with generative synthetic data that provides controlled, fine-grained identity and contextual variations. We evaluate our metric on a new unified evaluation benchmark for assessing consistency with human annotations across identity-focused recognition, retrieval, and generative tasks.
Figures & tables
Figure 2: Dataset curation pipeline. We highlight the different real and synthetic data subsets that enable ID-Sim training. Together, they provide high context, domain, and visual identity diversities.
Dataset
Type
Objects
#Cat
Included in
#Inst
ILIAS [ 36 ]
Img
General
N/A
S1
281
FORB [ 92 ]
Img
Flat Obj
7
S1
761
MET [ 100 ]
Img
Artworks
1
S1
226
GLDv2 [ 90 ]
Img
Landmarks
12
S1
769
Dogs [ 45 ]
Img
Animal
1
S1
494
Cats [ 6 ]
Img
Animal
1
S1
140
Table 1: Overview of datasets used for training set curation. # Cat and # Inst refer to number of categories and instances respectively. Colored text indicates different subsets that the dataset images appear in: S1 , S2a , S2b , which can be matched to Figure 2
Figure 3: ID-Sim training pipeline. We train our metric with dual contrastive supervision. At the global level, CLS-token projections for anchor–positive pairs are contrasted against one hard negative and additional batch negatives using InfoNCE. At the patch level, projected patch tokens are compared using Sinkhorn distance for the same instance pairs.
Figure 4: Performance of ID-Sim vs. baseline models . We compare ID-Sim against standard perceptual metrics, large-scale vision foundation models, and a supervised “Universal Embedding” model (the top entry in Google’s universal embedding challenge). Across tasks – instance retrieval, concept preservation, and re-identification – ID-Sim consistently outperforms all baselines, including the instance-retrieval-focused model, despite using over 100× less labeled data and a smaller backbone (our ViT-L vs. ViT-H ). Full results and seed variance are reported in the Supplemental E.2 .
Table 2: Overview of results. (Left): comparison with MLLMs on concept preservation. (Middle): performance across recognition and retrieval datasets. (Right): transfer to personalized segmentation with PerSAM.
Figure 5: Newly annotated Subjects2k. We release a 2k high-quality human annotations with a subset of Subjects200k to serve as a new challenging concept preservation eval benchmark.
Dataset
Bal.
Pos.
Neg.
Ratio
Val
Group
Edit
Edit
Score
All datasets
✗
✗
✗
–
0.693
All datasets
✓
✗
✗
–
0.752
Filtered datasets
✓
✗
✗
–
0.890
Filtered datasets
✓
✓
✗
1:1
0.937
Filtered datasets
✓
✓
✓
1:1:1
0.965
Table 3: Ablation of dataset composition and editing strategies. Balancing and targeted editing of positive and negative samples improve performance.
Figure 6: Selective sensitivity analysis. We evaluate model sensitivity across four axes of visual change: identity, background, viewpoint, and lighting. For 100 anchor instances, we generate controlled variations and compute both sensitivity scores and similarity trends. (Top row.) Compared with baseline methods, our model is notably more sensitive to identity differences while remaining stable under background, viewpoint, and lighting changes. (Bottom row.) When systematically increasing variations across each dimension, we see that, as desired, only identity changes significantly reduce similarity measured by ID-Sim .
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Inst.
Imgs
Cats
Img/Inst
Domain
MET [ 100 ]
734
3,429
–
2–17
Art
ILIAS [ 36 ]
900
5,326
–
2–35
Everyday Obj
FORB [ 92 ]
4,050
16,781
7
2–22
Flat Obj
GLDv2 [ 90 ]
4,503
81,964
–
2–6,272
Landmarks
WildlifeReID10k [ 1 ]
9,756
126,302
22
1–411
Animals
SOP [ 74 ]
11,318
59,551
12
2–12
Products
Appendix
Table 1: Initial real instance-level dataset pool. MVImgNet2* is a subset of MVImgNet2 composed of the first two released parts, as the full released dataset contains 180k+ videos. We also use the train_clean split of GLDv2.
Dataset
Unbalanced
Balanced
MET
105
663
ILIAS
111
804
FORB
592
1428
GLDv2
625
1419
WildlifeReID10k
1450
1418
StanfordOnlineProducts
1566
1435
Appendix
Table 2: Instance counts in the unbalanced vs. balanced 10k training mixtures.
Figure 7: Filtered out FORB logo category. We observe consistent appearance inconsistencies between the same ”instance” category in FORB’s ”logo” class.
Figure 8: Filtered GLDv2 categories. Many GLDv2 classes cover broad geographic areas rather than a single localized site, building or an object, making it difficult for a class to correspond to a consistent visual identity.
Dataset
Filtered
MET
671
ILIAS
826
WildlifeReID10k (Dogs and Cats)
1501
FORB (Filtered)
2346
GLDv2 (Filtered)
2315
DeepFashion2
2341
Appendix
Table 3: Instance counts for the filtered and balanced 10k training mixture.
Dataset
Frames per Second (FPS)
Frame Stride
Number of Frames ( k )
Window Size
Number of Sampled Parts
LaSOT
30
5
6
30 (1s)
6
GOT-10k
10
5
2
10 (1s)
5
YouTubeVIS
6
5
2
10 (1.7s)
5
UCO3D
30
1
30
30 (1s)
5
Appendix
Table 4: Dataset-specific parameters for video frame sampling. About 5-6 frames are sampled from each instance sequence, at intervals that are at ∼ 1 second apart.
Figure 9: Generative Contextual Edited Images
Figure 10: Generative Edited Hard-Negatives
Dataset Scale
Real Instance Validation (Acc)
Identity Validation (ROC AUC)
5k even split
0.860
0.97
10k even split
0.870
0.97
20k even split
0.880
0.98
30k
0.8825
0.91
Appendix
Table 5: Effect of training dataset scale on validation performance.
Dataset Composition
Real Instance Validation (Acc)
Identity Validation (ROC AUC)
Even Split
0.8715
0.97
Subset 1 dominant
0.8905
0.95
Subset 2a dominant
0.8715
0.95
Subset 2b dominant
0.8685
0.96
Appendix
Table 6: Effect of dataset composition on validation performance (fixed 10k scale).
Backbone / Resolution
Real Instance Validation (Acc)
Identity Validation (ROC AUC)
DINOv3-L/16 @ 448 (Baseline)
0.8715
0.965
DINOv3-L/16 @ 224
0.8715
0.965
DINOv3-B/16 @ 448
0.834
0.921
DINOv2-L/16 @ 448
0.8355
0.895
DINOv2-L/16 @ 224
0.8135
0.819
Appendix
Table 7: Backbone and resolution ablation.
Feature Loss Setting
Real Instance Validation (Acc)
Identity Validation (ROC AUC)
CLS + Patch (Baseline)
0.8715
0.965
Patch Loss Only
0.8665
0.908
CLS Loss Only
0.8065
0.893
Appendix
Table 8: Ablation on CLS vs. patch vs. joint training.
Objective / Patch Metric
Real Instance Validation (Acc)
Identity Validation (ROC AUC)
InfoNCE + Sinkhorn (Baseline)
0.8715
0.965
InfoNCE + Cosine Patch Metric
0.8655
0.940
Hinge + Sinkhorn
0.8705
0.945
BCE + Sinkhorn
0.8395
0.923
Appendix
Table 9: Loss and patch similarity ablation.
Figure 11: PODS Dataset. The dataset is composed of household objects occurring under different distribution shifts, with varying backgrounds, distractor objects, and poses.
Figure 12: DeepFashion2. The DeepFashion2 dataset features query / gallery images of the same clothing item in-shop as well as worn by consumers.
Figure 13: AerialCattle2017. This dataset is composed of aerial imagery of various cows on fields, and the task is to retrieve the same individuals based on a query image.
Figure 14: Petface. Evaluation benchmark of 13 unseen animals. Red depicts different individual and green depicts same individual.
Figure 15: CUTE triplets. Examples of triplets selected for ”Hard” mode. The positive and negative examples are drawn from the same scene type, which may be different from the scene type of the anchor, forcing a match across extrinsic characteristics.
Figure 16: Subjects2k Pairs. Newly annotated 2k subset of Subjects200k [ 81 ] . Green depicts same instance, red is different.
Figure 17: DreamBench Pairs. DreamBench images are accompanied by human annotations out of 4.
Figure 18: Introduction Page for Subjects2k Annotation Server. We provide a clear definition of an instance to all participants prior to starting their annotations.
Figure 19: Subjects2k Annotation Server Task Page. Example of a task page for our annotators.
Label range
# pairs
[0.00,0.09)
788
[0.09,0.18)
94
[0.18,0.27)
111
[0.27,0.36)
96
[0.36,0.45)
141
[0.45,0.55)
70
Appendix
Table 10: Subjects2k continuous label histogram. Counts of image pairs falling into each bin of the average human “same” vote fraction ℓ .
Figure 20: Human-labeled identity pairs. Left: Examples annotated as “Same.” Right: Examples annotated as “Different.” Human annotators reliably pick up subtle, fine-grained cues—such as texture, pattern, and small structural differences.
Figure 22: Qualitative Results for Per-SAM. We show predicted segmentation masks and corresponding predicted confidence scores, ordered in highest to lowest with respect to a reference object. First, we observe that when combined with PerSAM, both ID-Sim and DINOv3 are able to produce reliable segmentation mask predictions (mask drawn in red around the instance). However, we observe that ID-Sim is significantly better at recognizing instances across distribution shifts and discriminating fine-grained neighbors compared to DINOv3, as shown by the predicted mask scores and the ordering.
Figure 23: Dense masks can resolve ambiguities in multi-object Scenes. Given the test image with 2 shirts (left), ID-Sim features are sensitive to the identity of the query image (right 2 images), evidenced by the patch-level similarity heatmaps (2nd to left).
Figure 35
Figure 26: Background vs. Identity Variation Grid. Rows vary the foreground identity through Qwen-Edit inpainting at increasing edit strengths, while columns vary the scene background using inpainting prompts. Each cell shows the similarity of the edited image to the original anchor. This grid isolates how models respond jointly to identity changes and background shifts.
Figure 27: Viewpoint Variation Grid. Rows vary identity strength and columns sweep natural viewpoint changes using the multi-view MVImgNet sequence. This grid evaluates how well each model maintains invariance to viewpoint while still detecting identity-altering edits.
Figure 28: Lighting Variation Grid. Rows correspond to increasing levels of identity change, while columns apply eight different lighting edits using Qwen-Edit. This grid tests whether models remain stable under illumination changes while remaining sensitive to small identity perturbations.
When evaluating identity-focused tasks such as personalized generation and image editing, existing vision encoders entangle object identity with background context, leading to unreliable representations and metrics. We introduce the first principled framework to address this vulnerability using Near-identity (NearID) distractors, where semantically similar but distinct instances are placed on the exact same background as a reference image, eliminating contextual shortcuts and isolating identity as the sole discriminative signal. Based on this principle, we present the NearID dataset (19K identities, 316K matched-context distractors) together with a strict margin-based evaluation protocol. Under this setting, pre-trained encoders perform poorly, achieving Sample Success Rates (SSR), a strict margin-based identity discrimination metric, as low as 30.7% and often ranking distractors above true cross-view matches. We address this by learning identity-aware representations on a frozen backbone using a two-tier contrastive objective enforcing the hierarchy: same identity > NearID distractor > random negative. This improves SSR to 99.2%, enhances part-level discrimination by 28.0%, and yields stronger alignment with human judgments on DreamBench++, a human-aligned benchmark for personalization. Project page: https://gorluxor.github.io/NearID/
Aleksandar Cvejic, Rameen Abdal, Abdelrahman Eldesokey +2
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Snap Research, Palo Alto, CA, USA
Identity-preserving image generation becomes increasingly unreliable when a scene must contain many specified people. Beyond retaining each identity, the model must bind every reference to a distinct person and location, while training-time identity losses must establish correspondence among several noisy predicted faces. We introduce WithEveryone, a unified framework for generating group images up to ten reference identities. WithEveryone injects each selected identity as an addressed token, predicts a structured identity--layout plan, and renders the plan as a visual condition. Its key objective, Layout-Grounded ID Loss, uses annotated face regions to supervise the intended identities directly, avoiding unstable embedding-based face matching; ID Representation Forcing additionally trains a prediction for each identity before image synthesis. On an identity-disjoint benchmark, WithEveryone achieves the highest target-context identity similarity, improving face similarity from 0.462 for GPT-Image-2 to 0.499, while reducing copy-paste artifacts from 0.169 to 0.055. It further covers 97.3% of the requested identities with a duplicate rate of only 2.8%. These results show that explicit identity--layout grounding enables identity-preserving generation to scale to larger groups without relying on direct reference-face copying.
Hengyuan Xu, Qixun Wang, Yiji Cheng +5
1Fudan University · 2Hunyuan, Tencent · 3The University of Hong Kong
Modern generative models can produce visually plausible images while violating exact identity constraints. Evaluating such failures requires distinguishing violations of the requested identity from permissible variation in appearance. We introduce Low-Entropy Visual Identity (LEVI), which characterizes visual identities determined by compact facts and explicit rules. The low-entropy requirement concerns identity uncertainty, while appearance may remain diverse. We formulate identity fidelity in terms of the probability that a conditional image distribution assigns to the identities permitted by the prompt. Visual Identity Spaces (VIS) formalize this distinction by specifying admissible identity values and which different realizations express the same identity. Building on this formulation, we develop Verifiable Visual Grammar (VVG), a declarative language whose typed primitives and compositional constraints describe identity-bearing structures. Using this grammar, we build a benchmark for image generation. It assesses each output against the task's identity requirements, drawing evidence from the final pixels. The curated benchmark contains 1,375 tasks spanning 120 base identity spaces across six domains. Among ten evaluated model configurations, the highest whole-task pass rate is 58.69%. These results reveal a persistent gap between visually plausible images and faithful realization of their specified identities. Project page: https://maxiao0234.github.io/LEVI/.