NearID: Identity Representation Learning via Near-identity Distractors

Aleksandar Cvejic; Rameen Abdal; Abdelrahman Eldesokey; Bernard Ghanem; Peter Wonka

cs.CVApr 2, 2026

NearID: Identity Representation Learning via Near-identity Distractors

Authors: Aleksandar Cvejic, Rameen Abdal, Abdelrahman Eldesokey, Bernard Ghanem, Peter Wonka

Organizations: King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Snap Research, Palo Alto, CA, USA

Abstract

When evaluating identity-focused tasks such as personalized generation and image editing, existing vision encoders entangle object identity with background context, leading to unreliable representations and metrics. We introduce the first principled framework to address this vulnerability using Near-identity (NearID) distractors, where semantically similar but distinct instances are placed on the exact same background as a reference image, eliminating contextual shortcuts and isolating identity as the sole discriminative signal. Based on this principle, we present the NearID dataset (19K identities, 316K matched-context distractors) together with a strict margin-based evaluation protocol. Under this setting, pre-trained encoders perform poorly, achieving Sample Success Rates (SSR), a strict margin-based identity discrimination metric, as low as 30.7% and often ranking distractors above true cross-view matches. We address this by learning identity-aware representations on a frozen backbone using a two-tier contrastive objective enforcing the hierarchy: same identity > NearID distractor > random negative. This improves SSR to 99.2%, enhances part-level discrimination by 28.0%, and yields stronger alignment with human judgments on DreamBench++, a human-aligned benchmark for personalization. Project page: https://gorluxor.github.io/NearID/

Figures & tables

Figure 1: NearID: From Context to Identity. (Left) Traditional representations entangle object identity with background context. (Middle) Synthetic Data lacks explicit control over visually similar distractors. (Right) NearID introduces matched-context distractors to remove contextual shortcuts and isolate intrinsic identity signals.

Figure 2: NearID overview. Left: In the pretrained image-embedding manifold (e.g., SigLIP2 [ 66 ] ), identity-consistent positives can be misaligned: edited or re-rendered views of the same instance may lie farther from the anchor than visually confusable negatives (NearID distractors illustrated by $d_{i}<d_{j}$ ), which degrades retrieval and scoring reliability. Right: We keep the SigLIP2 image encoder frozen and train only a lightweight MAP [ 82 , 25 ] projection head to reshape the similarity geometry for the target task. Compared to the frozen baseline, the trained head increases similarity for true positives (green) while pushing NearID distractors and unrelated gallery items lower (red), improving the desired ordering of similarity scores with respect to a gallery.

Scoring Model	NearID		MTG				DB++
Scoring Model	SSR $\uparrow$	PA $\uparrow$	M–O $\uparrow$	M–O pair $\uparrow$	SSR $\uparrow$	PA $\uparrow$	M–H $\uparrow$
Qwen3VL30B [ 3 ]	49.73	69.20	0.219	0.329	17.0	26.0	–
CLIP [ 51 ]	10.31	20.92	0.239	0.484	0.0	0.0	0.493
DINOv2 [ 45 ]	20.43	34.55	0.324	0.519	0.0	0.0	0.492
VSM* [ 14 ]	32.13	46.70	0.394	0.445	7.0	24.5	0.190
SigLIP2 [ 66 ]	30.74	48.81	0.180	0.366	0.0	0.0	0.516

Table 1: Near-identity discrimination and alignment. We evaluate matched-context Near-identity discrimination at the object level (NearID) and part level (MTG [ 14 ] ), and measure correlation with MTG oracle scores and human concept-preservation judgments on DreamBench++ (DB++) [ 48 ] , verifying alignment with both oracle-defined edit severity and human perception. Top: Existing embedding and VLM methods under the Near-identity protocol. Bottom: Training in our setting, showing improvements on NearID and gains on MTG and DB++. MTG reports metric-to-oracle alignment ( M–O / M–O pair ); DB++ reports metric-to-human alignment ( M–H ). Deltas are relative to the frozen SigLIP2 baseline. * denotes a method trained directly on MTG. SSR and PA are averaged across seven inpainting settings (three excluded from training; see Sec. 3.4 ).

Panel of Figure 3 — Figure 3: Left: Attention map comparison vs baseline on positives and negatives. Right: Summary of properties evaluated in this paper. NearID improves matched-context Near-ID rejection on NearID-bench and transfers to part-level identity evaluation on MTG, while remaining mask-free at inference.

Figure 4: Contrastive training paradigms. Unlike (A) SimCLR [ 9 ] (augmented views of a single image) and (B) StableRep [ 65 ] (same-caption generations), (C) NearID uses multi-view positives of the same instance and adds near-identity distractors , visually similar but distinct instances inpainted into the anchor’s background, a structured negative signal absent from both. See the supplement for a full discussion.

Figure 5: Per-category human alignment (

M-H

) on DreamBench++ [ 48 ] . Each radar shows the Pearson correlation between a metric and human concept-preservation judgments across four DB++ categories; NearID (Ours) is repeated as a dashed reference in each subplot. Despite training exclusively on rigid objects, NearID improves over the frozen SigLIP2 baseline on Animal (

+0.105

) and Human (

+0.065

), indicating that disentangling identity from context transfers across semantic domains. The expected decrease on Style (

-0.092

), a category entirely absent from training, confirms that the gains reflect genuine identity learning rather than general score inflation.

Method	T2I Model	Concept Preservation (Metric–Human Alignment)
Method	T2I Model	H–H	G–H	D–H	C–H	S–H	NearID–H
Textual Inversion [ 18 ]	SD v1.5	0.684	0.625 $\pm$ 0.032	0.611 $\pm$ 0.016	0.610 $\pm$ 0.001	0.660 $\pm$ 0.003	0.675 $\pm$ 0.007
DreamBooth [ 53 ]	SD v1.5	0.731	0.722 $\pm$ 0.011	0.632 $\pm$ 0.010	0.623 $\pm$ 0.015	0.652 $\pm$ 0.012	0.655 $\pm$ 0.023
DreamBooth LoRA [ 53 , 23 ]	SDXL v1.0	0.651	0.637 $\pm$ 0.004	0.446 $\pm$ 0.013	0.481 $\pm$ 0.007	0.480 $\pm$ 0.006	0.546 $\pm$ 0.006
BLIP-Diffusion [ 36 ]	SD v1.5	0.630	0.466 $\pm$ 0.019	0.311 $\pm$ 0.024	0.290 $\pm$ 0.004	0.363 $\pm$ 0.016	0.382 $\pm$ 0.002
Emu2 [ 59 ]	SDXL v1.0	0.763	0.777 $\pm$ 0.004	0.774 $\pm$ 0.018	0.744 $\pm$ 0.021	0.766 $\pm$ 0.021	0.768 $\pm$ 0.028

Table 2: DreamBench++ concept-preservation human alignment. Pearson correlations between human (H) and metric scores: G = GPT, D = DINO, C = CLIP, S = SigLIP2, NearID (Ours). Bolded numbers indicate improvement over SigLIP2.

Training Loss	NearID		MTG				DB++
Training Loss	SSR $\uparrow$	PA $\uparrow$	M–O $\uparrow$	M–O pair $\uparrow$	SSR $\uparrow$	PA $\uparrow$	M–H $\uparrow$
None (frozen)	30.74	48.81	0.180	0.366	0.0	0.0	0.516
InfoNCE (sym., 1-pos)	60.97	75.26	0.267	0.418	8.0	12.0	0.555
+ $\mathcal{R}$ neg	99.57	99.79	0.236	0.267	64.0	74.0	0.251
+ Oracle Ranking †	86.34	92.25	0.299 †	0.444 †	7.0 †	11.0 †	0.167 †
+ $\mathcal{R}$ neg + Oracle Ranking	99.60	99.89	0.247	0.277	65.0	74.5	0.227

Table 3: Training objective ablation. All variants share a frozen SigLIP2 backbone with a trainable MAP head, trained on NearID + MTG data for 11 epochs with identical hyperparameters. More details in the supplementary material about the specifics of each loss and the extension. † Representation collapse: high discrimination but degraded general-purpose alignment (see text). We observe that our configurations pose the best balance between the different adaptations to the NearID setting.

Figure 6: KernelPCA visualization of Near-Identity separation. We project embeddings for

n{=}12

identities (S1–S12) into 2D using identical KernelPCA [ 54 ] settings in both panels. Circles denote positives (same identity) and crosses denote matched-context NearID distractors (negatives). Compared to the frozen SigLIP2 baseline, NearID increases separation by pushing the distractors away from the corresponding positive clusters, providing visual evidence of improved near-identity discrimination.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: NearID qualitative results. Each row shows one identity: Anchor — Positive 1 — Positive 2 — Distractor 1 — Distractor 2. Positives depict the same instance in a different background; near-identity distractors show a different but visually similar instance inpainted into the anchor’s original background. NearID (N) consistently assigns higher scores to positives than to distractors across diverse object categories (storage crates, conch shells, basketballs, coffee cups, spacecraft). In contrast, frozen SigLIP2 ( S ) frequently assigns distractor scores on par with or exceeding positives; for instance, in the seashell row S rates Distractor 2 at $0.97$ , actually exceeding Positive 1 ( $0.79$ ), while N correctly suppresses the distractor to $0.71$ . The VLM judge ( VL ) is inconsistent: near-zero scores for the basketball row ( $0.10$ ) yet $1.00$ for a spacecraft distractor, reflecting the VLM’s tendency toward category-level rather than instance-level reasoning. NearID successfully resolves these confounds, maintaining a clear margin between positives and near-identity distractors in all five categories shown.

Figure 8: MTG qualitative results. Each row shows one identity: Anchor — Positive — Distractor 1 — Distractor 2. Distractors are generated by inpainting a localized part edit (e.g., the central motif of a brooch, the toe-cap design of a boot) while preserving the overall object shape and background; Oracle ( O ) measures the fractional area of the edited region. The red contour marks the edited part boundary; the green contour traces the full object extent. In the ornate brooch row, frozen SigLIP2 ( S $=0.93$ ) actually ranks Distractor 1 higher than the genuine Positive ( $0.89$ ), a failure caused by overall appearance similarity. NearID (N) correctly ranks Positive ( $0.96$ ) above both distractors ( $0.79$ , $0.84$ ), detecting the fine-grained motif difference despite near-identical backgrounds. In the boot row, VSM ( V ) assigns Distractor 2 a score of $0.97$ , nearly identical to the Positive ( $0.98$ ), whereas NearID cleanly separates them ( $0.95$ vs. $0.66$ ) even though the Oracle score ( $0.47$ ) confirms a moderate but genuine edit. These examples illustrate that NearID captures part-level identity differences beyond what holistic embedding models detect.

Training Loss	$\mathcal{L}_{\text{disc}}$	$\mathcal{L}_{\text{rank}}$	$\mathcal{R}$ usage	Margin	Oracle
InfoNCE (sym.) [ 51 ]	Symmetric softmax CE over global pool $\mathcal{G}$ ; single positive per anchor	—	—	✗	✗
$+\,\mathcal{R}$ neg	(same); distractors appended as extra negatives in softmax over $\mathcal{G}$	—	D	✗	✗
$+$ Oracle † [ 4 ]	(same as InfoNCE)	RankNet on oracle-ordered distractor pairs: $-\!\log\sigma(\ell_{r_{i}}{-}\ell_{r_{j}}{-}m)$	R	✓	✓
$+\,\mathcal{R}$ neg $+$ Oracle	(same as $+\,\mathcal{R}$ neg)	(same as $+$ Oracle)	D+R	✓	✓
SigLIP $+$ Rank [ 83 ]	Pairwise sigmoid BCE over $\mathcal{G}$ ; $\text{BCE}(\ell_{r},0)$ treats distractors as negatives	Log-sigmoid: $-\!\log\sigma(\ell_{r_{k}}{-}\text{LSE}_{b})$	S	✗	✗
Circle $+$ Rank † [ 60 ]	Circle Loss with adaptive weighting; $\mathcal{R}$ pooled with batch negatives	Hinge: $\tfrac{1}{K}\sum_{k}[\ell_{b}^{\max}{-}\ell_{r_{k}}{+}m]_{+}$	S	✓	✗

Appendix

Table 4: Training objective decomposition (Table 3 in the main paper). Each objective combines a discrimination term

\mathcal{L}_{\text{disc}}

that separates positives from all negatives, and an optional hierarchy term

\mathcal{L}_{\text{rank}}

that enforces ordering among negative types, yielding

\mathcal{L}{=}\mathcal{L}_{\text{disc}}{+}\alpha\,\mathcal{L}_{\text{rank}}

. Notation (cf. Sec. 3.1):

\ell_{u,v}{=}\cos(\mathbf{u},\mathbf{v})/\tau

;

\ell_{p}

\ell_{r}

\ell_{b}

denote logits to a positive, a NearID distractor, and a batch negative, respectively;

\text{LSE}_{b}{=}\log\!\sum_{g\in\mathcal{B}}e^{\ell_{a,g}}

(log-sum-exp over batch negatives);

[\cdot]_{+}{=}\max(\cdot,0)

;

\text{sp}(\cdot){=}\log(1{+}e^{(\cdot)})

\mathcal{R}

usage indicates how the NearID distractor set

\mathcal{R}

enters each loss: D = in softmax denominator (competes directly with positives); S = separate pairwise or hinge term; R = pairwise ranking only; — = batch-implicit negatives only (no explicit

\mathcal{R}

mechanism). † Representation collapse (Table 3). Margin = the loss includes an explicit numeric margin

m

between similarity scores. Oracle = the ranking term is supervised by ground-truth edit severity (the ratio of edited area to object area); when ✗ , ranking is unsupervised and purely structural (distractor

{>}

batch negative), requiring no per-sample annotations.

Panel of Figure 9 — Figure 9: MTG score distribution alignment with the Oracle. Empirical CDF of per-edit identity-preservation scores on the MTG benchmark [ 14 ] . The Oracle score is defined as $1-(|\Omega_{p}|/|\Omega_{o}|)$ , where $\Omega_{p}$ and $\Omega_{o}$ are the part and object masks respectively, reflecting the true magnitude of each edit. (a) Frozen embeddings (SigLIP2, CLIP, DINO) cluster near $1.0$ regardless of edit severity, indicating insensitivity to part-level changes. (b) Both NearID configurations produce score distributions that track the Oracle far more closely than VLM or VSM baselines, demonstrating improved sensitivity to fine-grained identity-altering edits.

			SigLIP2		Qwen3-VL		VSM		NearID (Ours)
Generator	Res.	$n$	SSR	PA	SSR	PA	SSR	PA	SSR	PA
Training sources
FLUX.1-Fill †	512	500	35.8	55.7	48.3	68.3	23.7	38.7	99.4	99.8
FLUX.1-Fill †	1024	500	27.4	44.8	38.9	60.0	15.7	25.6	99.4	99.7
Qwen-Image †	512	500	33.8	51.4	48.1	67.7	27.9	42.2	99.2	99.7
Qwen-Image †	1328	500	16.2	32.1	43.3	64.7	26.7	39.3	98.2	99.5

Appendix

Table 5: Per-source discrimination breakdown (full image). SSR (%) and PA (%) for each inpainting source used to generate NearID distractors. The main paper (Table 1) reports results averaged over seven fill-based sources; here we additionally include two Canny-guided sources (FLUX.1-Canny) that are excluded from the main evaluation. Training sources (

\dagger

) supply distractors during NearID training; the remaining sources are unseen , testing cross-generator generalization. NearID achieves

{\geq}\,97.8\%

SSR across all nine configurations, with only

0.1\%

difference between training and unseen fill-based averages.

Training Data	NearID src	MTG	NearID		MTG
Training Data	NearID src	MTG	SSR $\uparrow$	PA $\uparrow$	M–O $\uparrow$	M–O pair $\uparrow$	SSR $\uparrow$	PA $\uparrow$
None (frozen)	—	—	30.74	48.81	0.180	0.366	0.0	0.0
NearID (Ours)	4	✓	99.17	99.71	0.465	0.486	35.0	46.5
NearID only	4	✗	98.37	99.47	0.318	0.457	4.0	6.0
MTG only †	0	✓	58.34	75.64	0.349	0.395	49.0	62.0
NearID 1 only	1	✗	86.86	93.77	0.326	0.456	3.0	5.5

Appendix

Table 6: Training data ablation. All rows use

\mathcal{L}_{\text{NearID}}

(Table 3) with identical hyperparameters and a frozen SigLIP2 backbone with trainable MAP head. Data scale is accounted for; in the MTG setup, we use x8 to match the number of steps in our main setting. NearID src = number of inpainting sources providing near-identity distractors during training; MTG = whether the MTG part-level dataset is included. † Object-level discrimination failure: despite the highest MTG SSR/PA, NearID SSR drops to

58.3\%

, indicating that part-level data alone cannot teach background-invariant identity separation.

Method	NearID SSR $\uparrow$		M–O $\uparrow$		MTG SSR $\uparrow$
Method	Full	FG	Full	FG	Full	FG
SigLIP2 (frozen)	30.7	64.7	0.179	0.365	0.0	6.0
NearID (Ours)	99.2	95.3	0.465	0.641	35.0	32.0
( $-$ masking)	99.8	95.4	0.512	0.391	43.0	21.0

Appendix

Table 7: Role-aware foreground masking ablation. Both trained rows use

\mathcal{L}_{\text{NearID}}

(Table 3) with identical hyperparameters and a frozen SigLIP2 backbone with trainable MAP head; only the role-specific masking probabilities differ. Full = standard evaluation on the complete image; FG = foreground-only evaluation with the background removed via the oracle segmentation mask. NearID SSR = near-identity discrimination; M–O = Pearson correlation with the MTG part-level oracle; MTG SSR = part-level discrimination. Both trained variants reach

{\sim}99\%

full-image NearID SSR, so the diagnostic signal lies in the FG columns: without masking M–O drops and MTG SSR roughly halves, revealing reliance on background context.

Method	Full Image			FG Only			$\boldsymbol{\Delta}$	$\boldsymbol{\Delta}$
Method	SSR $\uparrow$	PA $\uparrow$	M–O $\uparrow$	SSR $\uparrow$	PA $\uparrow$	M–O $\uparrow$	SSR	M–O
CLIP ViT-L/14	10.3	20.9	0.239	51.3	71.6	0.297	+41.0	+0.06
DINOv2 ViT-L/14	20.4	34.6	0.324	63.7	77.4	0.301	+43.3	$-$ 0.02
SigLIP2-SO400M	30.7	48.8	0.179	64.7	80.5	0.365	+33.9	+0.19
VSM [ 14 ]	32.1	46.7	0.394	32.0	46.7	0.394 †	$-$ 0.1	$\approx$ 0
Qwen3-VL 4B [ 3 ]	13.4	32.9	0.012	24.4	44.3	$-$ 0.046	+11.0	$-$ 0.06

Appendix

Table 8: Effect of foreground masking and VLM scaling on NearID discrimination and MTG oracle alignment. Full : evaluation on unmodified images. FG : evaluation on foreground-only crops (background removed via oracle mask).

\Delta

: absolute change from masking (FG

-

Full). Frozen embeddings gain

+34

–

43\%

SSR from background removal, confirming strong background dependence. VLMs also benefit (

+5

+14\%

), with diminishing gains at larger scale. NearID SSR is nearly unaffected (

-3.8\%

), validating inherent background invariance. Notably, NearID oracle alignment ( M–O ) improves substantially under FG masking (

+0.18

), suggesting that background context adds noise to part-level correlation even when object-level discrimination is already robust. † VSM uses foreground masks internally, so external masking has no additional effect. ‡ Qwen3-VL M–H on DB++ is omitted from Table 1 as the autoregressive VLM evaluation on DB++ was skipped due to computational cost (

{\sim}324\times

embedding-based evaluation).

$\alpha$	NearID		MTG				DB++
$\alpha$	SSR $\uparrow$	PA $\uparrow$	M–O $\uparrow$	M–O pair $\uparrow$	SSR $\uparrow$	PA $\uparrow$	M–H $\uparrow$
$\alpha{=}0$ (disc. only)	99.57	99.85	0.306	0.336	63.0	73.0	0.229
$\alpha{=}0.1$	99.57	99.84	0.362	0.422	58.0	69.0	0.521
$\alpha{=}0.25$	99.20	99.73	0.466	0.518	49.0	62.0	0.555
$\alpha{=}0.5$ (default)	99.17	99.71	0.465	0.486	35.0	46.5	0.545
$\alpha{=}1.0$	98.80	99.61	0.542	0.583	35.0	49.5	0.571

Appendix

Table 9: Effect of the ranking weight

\alpha

in the NearID loss

\mathcal{L}_{\text{NearID}}=\mathcal{L}_{\text{disc}}+\alpha\,\mathcal{L}_{\text{rank}}

. All variants use a frozen SigLIP2 backbone with a trainable MAP head, trained on NearID + MTG data.

\alpha{=}0

ablates the ranking term entirely, retaining only the discrimination loss with near-identity distractors in the softmax denominator. Oracle alignment ( M–O ) generally increases with

\alpha

up to

1.0

, while discrimination (SSR) degrades only mildly until

\alpha{=}1.0

and drops at

\alpha{=}2.0

. Human-judgment alignment ( M–H ) rises sharply at

\alpha{=}0.1

and remains broadly stable (

0.54

–

0.57

) for

\alpha{\geq}0.25

, confirming that the ranking term is essential for perceptually meaningful embeddings.

$\beta$ (cohesion)	NearID		MTG				DB++
$\beta$ (cohesion)	SSR $\uparrow$	PA $\uparrow$	M–O $\uparrow$	M–O pair $\uparrow$	SSR $\uparrow$	PA $\uparrow$	M–H $\uparrow$
$\beta{=}0$ (no cohesion, $\alpha{=}1.0$ )	98.80	99.61	0.542	0.583	35.0	49.5	0.571
$\beta{=}0.1$	98.94	99.67	0.442	0.478	30.0	40.0	0.555
$\beta{=}0.2$	98.74	99.59	0.444	0.482	29.0	37.5	0.550
$\beta{=}0.3$	98.94	99.68	0.430	0.468	29.0	36.5	0.553

Appendix

Table 10: Effect of positive cohesion weight

\beta

in the combined objective

\mathcal{L}_{\text{NearID}}+\beta\,\mathcal{L}_{\text{coh}}

( Section 0.E.7 ), evaluated at

\alpha{=}1.0

. All variants use a frozen SigLIP2 backbone with a trainable MAP head, trained on NearID + MTG data.

\beta{=}0

(no cohesion) achieves the best oracle alignment ( M–O ) and human alignment ( M–H DB++); increasing

\beta

degrades all alignment metrics without improving discrimination (SSR).

Variant	NearID		MTG		DB++
Variant	SSR $\uparrow$	PA $\uparrow$	M–O $\uparrow$	M–O pair $\uparrow$	M–H $\uparrow$	$\Delta$ M–H
None (frozen)	30.74	48.81	0.180	0.366	0.516	—
InfoNCE (sym., 1-pos)	60.97	75.26	0.267	0.418	0.555	(ref.)
$+\,\mathcal{R}$ neg	99.57	99.79	0.236	0.267	0.251	$-0.304$
$+$ Oracle Ranking †	86.34	92.25	0.299 †	0.444 †	0.167 †	$-0.388$
$+\,\mathcal{R}$ neg $+$ Oracle	99.60	99.89	0.247	0.277	0.227	$-0.328$

Appendix

Table 11: InfoNCE component ablation. Rows extracted from Table 3 with DB++ M–H now reported for all variants. “

\mathcal{R}

neg” adds near-identity distractors to the softmax denominator; “Oracle Ranking” adds a pairwise RankNet term supervised by oracle similarity.

\mathcal{L}_{\text{NearID}}

differs in three ways: symmetric multi-positive softmax, distractors in the denominator, and an unsupervised softplus ranking regulariser (see Table 4 ). † Representation collapse.

Panel of Figure 10 — Figure 10: VLM judge prompt template used for the Qwen3-VL 30B baseline. The prompt enforces instance-level (not category-level) identity judgment with an explicit evidence hierarchy and structured JSON output.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 799 papers

Apr 6, 2026

ID-Sim: An Identity-Focused Similarity Metric

Humans have remarkable selective sensitivity to identities -- easily distinguishing between highly similar identities, even across significantly different contexts such as diverse viewpoints or lighting. Vision models have struggled to match this capability, and progress toward identity-focused tasks such as personalized image generation is slowed by a lack of identity-focused evaluation metrics. To help facilitate progress, we propose ID-Sim, a feed-forward metric designed to faithfully reflect human selective sensitivity. To build ID-Sim, we curate a high-quality training set of images spanning diverse real-world domains, augmented with generative synthetic data that provides controlled, fine-grained identity and contextual variations. We evaluate our metric on a new unified evaluation benchmark for assessing consistency with human annotations across identity-focused recognition, retrieval, and generative tasks.

Julia Chae, Nicholas Kolkin, Jui-Hsien Wang +3

MIT CSAIL · Adobe Research

Image Generation Evaluation

Ask PDF

Mar 16, 2026

AnyPhoto: Multi-Person Identity Preserving Image Generation with ID Adaptive Modulation on Location Canvas

Multi-person identity-preserving generation requires binding multiple reference faces to specified locations under a text prompt. Strong identity/layout conditions often trigger copy-paste shortcuts and weaken prompt-driven controllability. We present AnyPhoto, a diffusion-transformer finetuning framework with (i) a RoPE-aligned location canvas plus location-aligned token pruning for spatial grounding, (ii) AdaLN-style identity-adaptive modulation from face-recognition embeddings for persistent identity injection, and (iii) identity-isolated attention to prevent cross-identity interference. Training combines conditional flow matching with an embedding-space face similarity loss, together with reference-face replacement and location-canvas degradations to discourage shortcuts. On MultiID-Bench, AnyPhoto improves identity similarity while reducing copy-paste tendency, with gains increasing as the number of identities grows. AnyPhoto also supports prompt-driven stylization with accurate placement, showing great potential application value.

Longhui Yuan

Identity-Preserving Image Generation Controllable Image Generation Diffusion-Based Image Generation

Ask PDF

Oct 1, 2026

Anti-Persona: Disrupting Unauthorized Identity Binding and Recognition in Personalized Vision--Language Models

Few-shot personalization enables large vision--language models (LVLMs) to learn user-specific visual concepts for applications such as personalized retrieval and subject-aware querying. However, it also creates a privacy risk: an adversary can bind a target identity from a few reference images and subsequently detect that identity in new images through natural-language queries. We introduce Anti-Persona, an image-level defense against unauthorized identity binding and recognition in personalized LVLMs. Our key insight is that identity personalization relies on visual features shared across multiple reference images. We aggregate these features into an identity prototype and optimize visually subtle perturbations that disrupt prototype alignment in the vision-encoder space. Spatial smoothing and low-frequency preservation further promote visual fidelity and practical resilience to image compression. The resulting protection does not depend on a specific prompt and supports both proactive anti-personalization and reactive image protection. Experiments on two representative personalized LVLMs demonstrate protection rates of up to

95.0\%

while preserving visual fidelity. The method remains stable across prompt variations and evaluated identity-query tasks, and improves black-box transfer under encoder mismatch.

Abhishek Basu, Fahad Shamshad, Karthik Nandakumar

MBZUAI, UAE · Michigan State University, USA

VLM Robustness Vision-Language Models Adversarial Examples Privacy Protection in VLMs

Ask PDF

Abstract

Representation Learning

Explore similar work

ID-Sim: An Identity-Focused Similarity Metric

AnyPhoto: Multi-Person Identity Preserving Image Generation with ID Adaptive Modulation on Location Canvas

Anti-Persona: Disrupting Unauthorized Identity Binding and Recognition in Personalized Vision--Language Models