Localizing Concepts in Visual Autoregressive Models

Nanxiang Jiang; Yang Liu; Yuanhao Wang; Min Xu

cs.CVOct 29, 2026

Localizing Concepts in Visual Autoregressive Models

Authors: Nanxiang Jiang, Yang Liu, Yuanhao Wang, Min Xu

Organizations: University of California, Los Angeles · Carnegie Mellon University

Abstract

Understanding the internal routing of visual concepts is crucial for the safe and controllable adaptation of generative models. While concept localization has been widely studied in diffusion models, the emerging paradigm of next-scale visual autoregressive models remains largely unexplored. In this paper, we introduce LoCo, the first model-agnostic framework to localize where and when specific concepts emerge within autoregressive models. Driven by the native coarse-to-fine nature of next-scale generation, our method precisely maps conceptual knowledge across three distinct dimensions: Layer, Scale, and Position. To systematically localize and evaluate concept routing without context bias, we propose LoCoBench, a comprehensive dataset spanning 10 diverse categories. Extensive probing on image and video autoregressive models like Infinity, HunyuanImage-3.0, and InfinityStar shows that the localized positions are both interpretable and causally related to concept emergence. Building on these insights, we apply our localization method to three key applications: concept erasure, model personalization, and adversarial concept injection. Experiments demonstrate that our targeted intervention achieves state-of-the-art performance, substantially reducing computational overhead while preserving benign utility. Overall, our findings offer insights into how conceptual knowledge is routed during autoregressive generation, introducing a practical pathway for more interpretable, efficient, and secure adaptation.

Figures & tables

Figure 1: Localization and applications in VAR models. Left: Heatmaps indicate the frequency of specific Layer-Scale-Position triplets being selected as dominant carriers of target concepts based on attention responses. Right: This precise localization enables highly efficient targeted interventions for concept erasure, personalization, and adversarial injection.

Figure 2: Overview of our concept localization method. (a) For a target concept $c$ , we build diverse prompts $\{p_{i}^{c}\}$ and trace how the concept tokens influence current-scale queries through cross-attention. (b) Averaging these scores across prompts gives a layer-scale-position localization map, from which we rank the top- $K$ localized positions. (c) Replacing their inputs with concept-agnostic prompts $\{p_{i}^{c\text{-neutral}}\}$ suppresses $c$ while the rest of the scene is largely preserved.

Figure 3: Qualitative comparison of concept erasure. Our method successfully removes diverse concrete and abstract concepts while preserving benign generation capabilities. EAR Fan et al. (2025) frequently fails to erase the target, and S-VARE Zhong et al. (2026) introduces noticeable visual artifacts. CAT Chrabaszcz et al. (2026) suffers from severe over-purification, collapsing into generic outputs that fail to align with the user prompt.

	Nudity Erasure Task						Violence Erasure Task
	Detected Nudity				Utility (COCO)			Utility (COCO)
Method	Common	Female	Male	Total $\downarrow$	FID $\downarrow$	CLIP $\uparrow$	Detected Violence $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
Base Model: Infinity-8B
ESD Gandikota et al. (2023a) (adapted)	15	78	51	144	32.14	29.85	845	32.56	29.50
MACE Lu et al. (2024a) (adapted)	9	56	38	103	31.05	30.12	612	31.45	30.05
EraseAnything Gao et al. (2025b) (adapted)	12	64	42	118	30.85	30.45	684	31.12	30.25

Table 1: Evaluation on NSFW erasure. We evaluate nudity and violence erasure on 4,703 prompts from the I2P dataset, and report their respective FID and CLIP scores on MS-COCO to test utility preservation. Results of the original base models are presented for reference.

Figure 4: Qualitative comparison of model personalization. Left: Fine-tuning with LoCo better adheres to prompt fidelity. Right: Fine-tuning with LoCo largely preserves irrelevant visual identities (e.g., husky, golden retriever), effectively eliminating concept bleed.

Figure 5: Quantitative comparison of model personalization. Localized fine-tuning outperforms full-tuning across all metrics, achieving higher prompt alignment and better identity preservation.

Method	Poisoning Ratio	Attack Effectiveness		Benign Utility Preservation
Method	Poisoning Ratio	LIR (%) $\uparrow$	FAE $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
Full-Model (Baseline)	10%	4.2	48.5	32.14	30.12
Ours (LoCo-Targeted)	10%	51.5	7.5	29.45	31.35
Full-Model (Baseline)	25%	10.5	28.0	35.80	28.50
Ours (LoCo-Targeted)	25%	78.5	3.5	29.50	31.22

Table 2: Quantitative evaluation of trigger-free data poisoning. Standard full-model poisoning struggles at low data ratios and degrades benign utility. By restricting updates to localized concept routes, LoCo achieves significantly higher LIR and faster convergence (lower FAE) while preserving original image quality.

Figure 6: Ablation study on LoCoBench. Left: The trade-off between concept alignment (CLIP score) and the Top- $K$ intervention ratio. Right: The localized heatmap and per-scale generation process, demonstrating our coarse-to-fine concept routing.

Dimension Sweep	I2P	MS-COCO		Efficiency
Dimension Sweep	Total $\downarrow$	FID $\downarrow$	CLIP $\uparrow$	Time	VRAM
Infinity-8B	211	29.32	31.40	–	–
Full-Model FT	82	30.14	30.98	35	65
Layer-only	96	33.21	28.16	10	48
Scale-only	159	48.24	26.85	12	52
Layer + Scale	87	31.87	30.04	8	45

Table 3: Ablation study on I2P. Left: Dimensional ablation demonstrates that combining Layer, Scale, and Position is strictly necessary to balance erasure and image quality, while being highly parameter-efficient. Right: Sweeping the Top-

K

reveals the trade-off between concept suppression, utility preservation, and training cost (measured on a single H20 GPU).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Category	# Entities	# Train Prompts	# Eval Prompts	Total Size
Animals	150	3,000	4,500	7,500
Artists	516	10,320	15,480	25,800
Brands	100	2,100	2,000	4,100
Celebrities	120	2,400	3,000	5,400
Copyrights	120	2,400	3,600	6,000
Places	120	1,200	2,400	3,600

Appendix

Table 4: Dataset statistics across the ten knowledge categories in LoCoBench.

Category: Artists Anchor = “a painting”
“ a.y. jackson , a red double-decker bus on a wet street”
“ ad reinhardt , a windmill beside a tulip field at sunrise”
Category: Style Anchor = “a scene”
“ origami style , a narrow alley with hanging laundry”
“ voxel art style , a rooftop garden with potted herbs”
Category: Copyrights Anchor = “a character”

Appendix

Table 5: Example prompts from each knowledge category in LoCoBench.

Figure 7: Comparison of concept erasure on VAR models. Naive fine-tuning causes severe error accumulation. Conditional alignment stabilizes generation but introduces visual shifts when applied to the full model. Our localized fine-tuning restricts updates to the top- $K$ routes, successfully erasing the concept while perfectly preserving the background.

Figure 8: Qualitative video personalization results.

Figure 9: Qualitative video personalization results.

Figure 10: Qualitative video personalization results.

Intervention Strategy	Infinity (VAR)		HunyuanImage-3.0
Intervention Strategy	Target CLIP $\downarrow$	Background CLIP $\uparrow$	Target CLIP $\downarrow$	Background CLIP $\uparrow$
Base Model (No Erasure)	0.284	0.312	0.291	0.320
Bottom-5% Intervention	0.282	0.311	0.289	0.319
Random-5% Intervention	0.252	0.264	0.258	0.271
Targeted Top-5% (Ours)	0.185	0.311	0.190	0.318

Appendix

Table 6: Causal Verification via Counterfactual Interventions. Metrics are averaged over 500 prompts for concept erasure from LoCoBench. Targeted Top-5% intervention is strictly necessary to achieve significant concept erasure (Target CLIP

\downarrow

) without compromising background utility (Background CLIP

\uparrow

Concept Category	Cross-Seed IoU $\uparrow$	Cross-Prompt IoU $\uparrow$	Cross-Concept IoU $\downarrow$
Van Gogh (Style)	89.4	84.2	4.1
The Batman (Copyright)	91.2	85.7	3.8
Nudity (Safety)	87.5	81.4	5.2
Eiffel Tower (Place)	90.1	83.6	4.5
Average	89.6	83.7	4.4

Appendix

Table 7: Structural Stability and Specificity. We report the Jaccard Similarity (IoU %) of localized Top-5% triplets. High Cross-Seed and Cross-Prompt stability, combined with low Cross-Concept overlap, confirms that our method identifies unique and invariant neural bottlenecks.

Visual Autoregressive Models

Visual autoregressive models generate or analyze images and videos by predicting visual tokens in sequence, often from coarse scales to fine detail.

See all 58 papers

Obliviate: Erasing Concepts from Autoregressive Image Generation Models
Jun 26, 2026Hossein Shakibania, Jonas Henry Grebe, Tobias Braun +4Image Generation Safe T2I Generation
TU Darmstadt, Germany · Multimodal AI Lab, TU Darmstadt, Germany · Zuse School ELIZA +1
ScaleErasure: Inference-Time Minimal Intervention for Precise Concept Erasure in Next-Scale Autoregressive Image Generation
Jun 28, 2026Cong Wang, Haiyu Wu, Zhiwei Jiang +4Image Generation Safe T2I Generation
State Key Laboratory for Novel Software Technology, Nanjing University · National University of Singapore
Visual Autoregressive Generation with Next Token Imagination
Oct 27, 2026Ruihang Jiang, Zongyao Zhao, Zhaolin Wang +2Visual Autoregressive Models Autoregressive Image Generation
University of Hong Kong · University of Surrey

Jun 26, 2026J/K move · Enter open · S save

Obliviate: Erasing Concepts from Autoregressive Image Generation Models

Hossein Shakibania, Jonas Henry Grebe, Tobias Braun, Ege Aktemur+3

TU Darmstadt, Germany · Multimodal AI Lab, TU Darmstadt, Germany · Zuse School ELIZA+1

Image Generation Safe T2I Generation Autoregressive Image Generation Concept Erasure

Open paper AskPDF arXiv

The widespread adoption of generative AI models has intensified concerns about misuse, including the creation of unsafe or disturbing imagery. To mitigate such issues, several concept erasure approaches have been proposed to remove harmful content from multimodal generative models. Yet concept erasure for autoregressive image generation remains largely unexplored, despite the growing relevance of these models in recent trends toward unified multimodal architectures. In this work, we fill this gap by introducing Obliviate, a guidance-based concept erasure method for autoregressive image generation. Our method builds on three key design choices: KL-based supervision over visual token distributions, trajectory-level updates over full autoregressive rollouts, and aligned visual prefixes for stable target construction. We evaluate Obliviate on three state-of-the-art autoregressive text-to-image models, Liquid, Emu3-Gen, and Janus-Pro, covering the erasure of explicit content, graphic violence, and branded imagery. Obliviate consistently outperforms current alternatives, reducing nudity on the defensive RAB benchmark from 91.58 to 3.15 while preserving overall model utility.

Abstract

Visual Autoregressive Models

Explore similar work

Obliviate: Erasing Concepts from Autoregressive Image Generation Models

ScaleErasure: Inference-Time Minimal Intervention for Precise Concept Erasure in Next-Scale Autoregressive Image Generation

Visual Autoregressive Generation with Next Token Imagination