Understanding the internal routing of visual concepts is crucial for the safe and controllable adaptation of generative models. While concept localization has been widely studied in diffusion models, the emerging paradigm of next-scale visual autoregressive models remains largely unexplored. In this paper, we introduce LoCo, the first model-agnostic framework to localize where and when specific concepts emerge within autoregressive models. Driven by the native coarse-to-fine nature of next-scale generation, our method precisely maps conceptual knowledge across three distinct dimensions: Layer, Scale, and Position. To systematically localize and evaluate concept routing without context bias, we propose LoCoBench, a comprehensive dataset spanning 10 diverse categories. Extensive probing on image and video autoregressive models like Infinity, HunyuanImage-3.0, and InfinityStar shows that the localized positions are both interpretable and causally related to concept emergence. Building on these insights, we apply our localization method to three key applications: concept erasure, model personalization, and adversarial concept injection. Experiments demonstrate that our targeted intervention achieves state-of-the-art performance, substantially reducing computational overhead while preserving benign utility. Overall, our findings offer insights into how conceptual knowledge is routed during autoregressive generation, introducing a practical pathway for more interpretable, efficient, and secure adaptation.
Figures & tables
Figure 1: Localization and applications in VAR models. Left: Heatmaps indicate the frequency of specific Layer-Scale-Position triplets being selected as dominant carriers of target concepts based on attention responses. Right: This precise localization enables highly efficient targeted interventions for concept erasure, personalization, and adversarial injection.
Figure 2: Overview of our concept localization method. (a) For a target concept c , we build diverse prompts {pic} and trace how the concept tokens influence current-scale queries through cross-attention. (b) Averaging these scores across prompts gives a layer-scale-position localization map, from which we rank the top- K localized positions. (c) Replacing their inputs with concept-agnostic prompts {pic-neutral} suppresses c while the rest of the scene is largely preserved.
Figure 3: Qualitative comparison of concept erasure. Our method successfully removes diverse concrete and abstract concepts while preserving benign generation capabilities. EAR Fan et al. (2025) frequently fails to erase the target, and S-VARE Zhong et al. (2026) introduces noticeable visual artifacts. CAT Chrabaszcz et al. (2026) suffers from severe over-purification, collapsing into generic outputs that fail to align with the user prompt.
Nudity Erasure Task
Violence Erasure Task
Detected Nudity
Utility (COCO)
Utility (COCO)
Method
Common
Female
Male
Total ↓
FID ↓
CLIP ↑
Detected Violence ↓
FID ↓
CLIP ↑
Base Model: Infinity-8B
ESD Gandikota et al. (2023a) (adapted)
15
78
51
144
32.14
29.85
845
32.56
29.50
MACE Lu et al. (2024a) (adapted)
9
56
38
103
31.05
30.12
612
31.45
30.05
EraseAnything Gao et al. (2025b) (adapted)
12
64
42
118
30.85
30.45
684
31.12
30.25
Table 1: Evaluation on NSFW erasure. We evaluate nudity and violence erasure on 4,703 prompts from the I2P dataset, and report their respective FID and CLIP scores on MS-COCO to test utility preservation. Results of the original base models are presented for reference.
Figure 4: Qualitative comparison of model personalization. Left: Fine-tuning with LoCo better adheres to prompt fidelity. Right: Fine-tuning with LoCo largely preserves irrelevant visual identities (e.g., husky, golden retriever), effectively eliminating concept bleed.
Figure 5: Quantitative comparison of model personalization. Localized fine-tuning outperforms full-tuning across all metrics, achieving higher prompt alignment and better identity preservation.
Method
Poisoning Ratio
Attack Effectiveness
Benign Utility Preservation
LIR (%) ↑
FAE ↓
FID ↓
CLIP ↑
Full-Model (Baseline)
10%
4.2
48.5
32.14
30.12
Ours (LoCo-Targeted)
10%
51.5
7.5
29.45
31.35
Full-Model (Baseline)
25%
10.5
28.0
35.80
28.50
Ours (LoCo-Targeted)
25%
78.5
3.5
29.50
31.22
Table 2: Quantitative evaluation of trigger-free data poisoning. Standard full-model poisoning struggles at low data ratios and degrades benign utility. By restricting updates to localized concept routes, LoCo achieves significantly higher LIR and faster convergence (lower FAE) while preserving original image quality.
Figure 6: Ablation study on LoCoBench. Left: The trade-off between concept alignment (CLIP score) and the Top- K intervention ratio. Right: The localized heatmap and per-scale generation process, demonstrating our coarse-to-fine concept routing.
Dimension Sweep
I2P
MS-COCO
Efficiency
Total ↓
FID ↓
CLIP ↑
Time
VRAM
Infinity-8B
211
29.32
31.40
–
–
Full-Model FT
82
30.14
30.98
35
65
Layer-only
96
33.21
28.16
10
48
Scale-only
159
48.24
26.85
12
52
Layer + Scale
87
31.87
30.04
8
45
Table 3: Ablation study on I2P. Left: Dimensional ablation demonstrates that combining Layer, Scale, and Position is strictly necessary to balance erasure and image quality, while being highly parameter-efficient. Right: Sweeping the Top- K reveals the trade-off between concept suppression, utility preservation, and training cost (measured on a single H20 GPU).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Category
# Entities
# Train Prompts
# Eval Prompts
Total Size
Animals
150
3,000
4,500
7,500
Artists
516
10,320
15,480
25,800
Brands
100
2,100
2,000
4,100
Celebrities
120
2,400
3,000
5,400
Copyrights
120
2,400
3,600
6,000
Places
120
1,200
2,400
3,600
Appendix
Table 4: Dataset statistics across the ten knowledge categories in LoCoBench.
Category: Artists Anchor = “a painting”
“ a.y. jackson , a red double-decker bus on a wet street”
“ ad reinhardt , a windmill beside a tulip field at sunrise”
Category: Style Anchor = “a scene”
“ origami style , a narrow alley with hanging laundry”
“ voxel art style , a rooftop garden with potted herbs”
Category: Copyrights Anchor = “a character”
Appendix
Table 5: Example prompts from each knowledge category in LoCoBench.
Figure 7: Comparison of concept erasure on VAR models. Naive fine-tuning causes severe error accumulation. Conditional alignment stabilizes generation but introduces visual shifts when applied to the full model. Our localized fine-tuning restricts updates to the top- K routes, successfully erasing the concept while perfectly preserving the background.
Figure 8: Qualitative video personalization results.
Figure 9: Qualitative video personalization results.
Figure 10: Qualitative video personalization results.
Intervention Strategy
Infinity (VAR)
HunyuanImage-3.0
Target CLIP ↓
Background CLIP ↑
Target CLIP ↓
Background CLIP ↑
Base Model (No Erasure)
0.284
0.312
0.291
0.320
Bottom-5% Intervention
0.282
0.311
0.289
0.319
Random-5% Intervention
0.252
0.264
0.258
0.271
Targeted Top-5% (Ours)
0.185
0.311
0.190
0.318
Appendix
Table 6: Causal Verification via Counterfactual Interventions. Metrics are averaged over 500 prompts for concept erasure from LoCoBench. Targeted Top-5% intervention is strictly necessary to achieve significant concept erasure (Target CLIP ↓ ) without compromising background utility (Background CLIP ↑ ).
Concept Category
Cross-Seed IoU ↑
Cross-Prompt IoU ↑
Cross-Concept IoU ↓
Van Gogh (Style)
89.4
84.2
4.1
The Batman (Copyright)
91.2
85.7
3.8
Nudity (Safety)
87.5
81.4
5.2
Eiffel Tower (Place)
90.1
83.6
4.5
Average
89.6
83.7
4.4
Appendix
Table 7: Structural Stability and Specificity. We report the Jaccard Similarity (IoU %) of localized Top-5% triplets. High Cross-Seed and Cross-Prompt stability, combined with low Cross-Concept overlap, confirms that our method identifies unique and invariant neural bottlenecks.
The widespread adoption of generative AI models has intensified concerns about misuse, including the creation of unsafe or disturbing imagery. To mitigate such issues, several concept erasure approaches have been proposed to remove harmful content from multimodal generative models. Yet concept erasure for autoregressive image generation remains largely unexplored, despite the growing relevance of these models in recent trends toward unified multimodal architectures. In this work, we fill this gap by introducing Obliviate, a guidance-based concept erasure method for autoregressive image generation. Our method builds on three key design choices: KL-based supervision over visual token distributions, trajectory-level updates over full autoregressive rollouts, and aligned visual prefixes for stable target construction. We evaluate Obliviate on three state-of-the-art autoregressive text-to-image models, Liquid, Emu3-Gen, and Janus-Pro, covering the erasure of explicit content, graphic violence, and branded imagery. Obliviate consistently outperforms current alternatives, reducing nudity on the defensive RAB benchmark from 91.58 to 3.15 while preserving overall model utility.
Hossein Shakibania, Jonas Henry Grebe, Tobias Braun +4
TU Darmstadt, Germany · Multimodal AI Lab, TU Darmstadt, Germany · Zuse School ELIZA +1
Concept erasure aims to prevent image generative models from producing unsafe content while preserving their general generative capability. Meanwhile, next-scale autoregressive (AR) image generation has recently emerged as a new generative paradigm characterized by next-scale prediction, for which concept erasure remains largely unexplored. In this paradigm, semantic information is highly compressed at early scales, leading to severe entanglement between unsafe and unrelated semantics. In this paper, we propose ScaleErasure, an inference-time concept erasure method that performs minimal intervention. ScaleErasure precisely selects and guides predicted logits that are most relevant to the unsafe concept, thereby enabling effective erasure under severe semantic entanglement. Specifically, ScaleErasure performs two additional forward passes conditioned on the unsafe concept and the corresponding safe concept, and leverages their outputs to guide the target logits away from unsafe concepts toward safe concepts. To enable precise and minimal intervention, logits selection and guidance are conducted across three dimensions: scales, tokens, and bit channels. Experiments demonstrate that ScaleErasure outperforms adapted baselines in the next-scale AR paradigm, achieving more precise concept erasure while largely preserving general generative capability. The code is available at https://github.com/coziiizz/ScaleErasure.
Cong Wang, Haiyu Wu, Zhiwei Jiang +4
State Key Laboratory for Novel Software Technology, Nanjing University · National University of Singapore
The space of possible visual content is enormous, and arguably unbounded. Any training set, however large, therefore covers only a tiny fraction of it. Yet generative models are typically trained to fit only the images they are given. In visual autoregressive (AR) generation, for example, each token is predicted from the preceding ground-truth (GT) tokens and supervised solely by the GT next token. Confining learning to observed data in this way limits how much of the visual distribution a model can capture. In this work, we hypothesize that letting a model imagine beyond its training data, by exploring plausible visual realizations it has never observed, can ease this limitation. We use visual AR generation as a proof of concept, since its fixed, discrete token space makes imagination easier to steer and constrain. We propose Next Token Imagination (NTI), a method that allows the model to explore beyond the GT next token, subject to a carefully designed criterion that keeps this imagination bounded by the underlying visual distribution. Extensive experiments on ImageNet show that NTI substantially and consistently improves the fidelity of generated images, approaching state-of-the-art diffusion and AR models while using a smaller model and lower training cost.