Attribute Token Arithmetic: Disentangled and Continuous Semantic Control for Visual Autoregressive Models

Xindi Yang; Yicheng Wu; Cheng Zhang; Jianfei Cai; Tien-Tsin Wong

cs.CVAug 28, 2026

Attribute Token Arithmetic: Disentangled and Continuous Semantic Control for Visual Autoregressive Models

Authors: Xindi Yang, Yicheng Wu, Cheng Zhang, Jianfei Cai, Tien-Tsin Wong

Organizations: Faculty of IT, Monash University

Abstract

Autoregressive text-to-image generation has recently achieved remarkable progress, offering high-fidelity synthesis via a unified generative framework. However, fine-grained semantic control remains challenging due to the attribute entanglement and the misalignment between textual and fine-grained visual representations. In this paper, we introduce Attribute Token Arithmetic (ATA), a method that enables disentangled and continuous attribute control in visual autoregressive modelling. Inspired by the vector arithmetic property observed in word embeddings, ATA identifies semantic directions corresponding to visual attributes (e.g., aging, fatness, emotion) directly within the pretrained autoregressive latent space. These directions are learned from a single reference image, without model retraining or large-scale supervision. During generation, attributes can be continuously adjusted and compositionally combined through simple arithmetic operations with other attribute tokens. Extensive experiments demonstrate that ATA achieves identity-preserving, fine-grained, and multi-attribute adjustment, outperforming existing autoregressive editing baselines in controllability, generality, and computational efficiency. Our code will be available at https://github.com/Madaoer/ATA.

Figures & tables

Figure 1: Exemplar edited images with disentangled and continuous semantic control . Given a reference and paired text prompts, our ATA extracts semantic attribute directions in the VAR-aligned text space. These directions are derived from the image concepts that correspond to the semantic differences between paired prompts. Specifically, ATA modifies source images to gradually increase the strength of two attributes. The first two results increase with the first attribute, while in the last two, the second attribute is introduced. (Left) ATA enables disentangled and continuous semantic control of “ angry ” and “ old ”, while preserving the identity. (Right) Moreover, ATA can transfer the general disentangled attribute concepts across object categories. Although “ fat ”-ness is not a native attribute of chairs, ATA can still transfer the “ fat ” & “ furry ” concepts and enable continuous control to yield creative chair designs, while both VAREdit and Qwen-Image-Edit fail to provide continuous control.

Figure 2: Preliminary image editing results . By manipulating semantic directions to the corresponding text tokens, (a) we achieve the localized editing and the preservation of object identities; (b) we can also edit non-localized attributes, e.g . , background.

Model	Strength Control	Cross-Category Transfer	Object Composition
Concept Slider	✓	✓	✗
Tokenverse	✗	✗	✓
VAREdit, Qwen-Image-Edit	✓	✓	✗
Ours	✓	✓	✓

Table 1: Comparison among existing image editing methods.

Figure 3: Pipeline of our ATA framework . (a) A pretrained VAR model processes image and text tokens with a causal transformer. At each next-scale prediction step, VAR applies Self-Attention, Cross-Attention, and Feed-Forward blocks. Text and image tokens interact through cross-attention. Our ATA adds the learned semantic directions to the corresponding VAR-aligned text tokens in the Cross-Attention Block. (b) Given a reference image and a pair of semantically different prompts, our ATA learns two semantic directions, $\vec{d}_{\rm general}$ and $\vec{d}_{\rm specific}$ , for each target concept, via a lightweight attention block, Delta-Mod. These directions capture the general attribute concept and image-specific personalized traits. They are learned with the original VAR training loss. (c) During inference, the learned directions are applied to the corresponding text tokens. Our ATA enables disentangled and continuous control in VAR.

Figure 4: Concept illustration of ATA . Given a reference image $I$ and a pair of semantically different prompts ( e.g . , “ A fat cat. ” and “ A cat. ”), our method learns two semantic directions with respect to $I$ : $Fat_{\rm specific}$ , the semantic direction from the mean of the distribution “ fat cat ” to $I$ , and $Fat_{\rm general}$ , the semantic direction from the mean of the distribution “ cat ” to $I$ , both in the VAR-aligned text space. We obtain the general attribute direction $Fat_{\rm attribute}$ by simple vector arithmetic. The learned attribute transfers across object categories ( e.g . transfer "fat" to chair).

Figure 5: General attribute transfer and cross-category semantic editing . We present more examples of general attribute transfer. (Left) We apply the attribute “ Apple-shaped ” to the concept “ cake ” and perform continuous control. Our method gradually transforms the cake into an apple-shaped cake, whereas baseline methods turn the cake into an apple. (Right) We transfer the attribute concept “ Sphynx ”, originally defined for Sphynx cats, to sheep. Our method achieves a gradual transition to hairlessness with pronounced skin folds, whereas the baselines fail to do so.

Model	Intensity	Control Dataset		GEdit
Model	Intensity	$\Delta$ VQA( $\uparrow$ )	I-LPIPS( $\uparrow$ )	S.C.( $\uparrow$ )	P.Q.( $\uparrow$ )	Disen.( $\uparrow$ )
Concept-slider	Slight	0.421	0.891	4.992	7.612	5.172
	Moderate	0.513	0.847	4.826	7.261	5.018
	Extreme	0.572	0.812	4.622	7.027	4.920
Qwen-Image	Slight	+3.82%	+4.12%	+7.13%	+4.27%	+5.02%
	Moderate	+5.31%	+5.44%	+6.20%	+5.11%	+3.86%

Table 2: Quantitative comparison of semantic alignment and content preservation . We use

\Delta

VQA and Semantic Consistency (S.C.) to measure the semantic shift toward the target attribute. Image preservation is evaluated via the inverted LPIPS metric (I-LPIPS) and Perceptual Quality (P.Q.). Moreover, we leverage Disen. for the disentanglement performance in multi-attribute editing.

Model	I.P.( $\uparrow$ )	P.A.( $\uparrow$ )
Concept-slider	10%	6%
Qwen-Image	30%	23%
VAREdit	23%	27%
Ours	37%	43%

Table 3: User study. Preference rates for Image Preservation (I.P.) and Prompt Adherence (P.A.).

Method	Semantic Consistency ( $\uparrow$ )	Perceptual Quality ( $\uparrow$ )	Disentanglement ( $\uparrow$ )
$\mathrm{mean}(T_{\rm attr}^{\rm tar})-T_{\rm attr}^{\rm src}$	5.048	7.447	5.296
ATA (Infinity-2B)	5.374	7.584	5.455

Table 4: Comparison with a training-free text-token baseline. We report mean scores under the same GEdit protocol. The baseline directly uses the token difference in Eq. ( 1 ) as the semantic direction.

Analysis	Method / Setting	Mean $\Delta$ VQA ( $\uparrow$ )	Mean I-LPIPS ( $\uparrow$ )
Same-backbone comparison	Infinity-2B with Concept Slider	0.532	0.879
Same-backbone comparison	ATA (Infinity-2B, original prompts)	0.567	0.895
Prompt sensitivity	ATA (Infinity-2B, perturbed prompts)	0.541	0.864

Table 5: Ablations on the Control Dataset. ATA with original prompts serves as the shared reference for comparison with Infinity-2B using Concept Slider and for the prompt sensitivity study.

Figure 6: Ablation study . (a) Optimizing the attribute direction on the identity token ‘cat’ instead of the attribute token ‘A’ changes the subject identity in the generated image. (b) Adding the fat direction into a non-attribute token ‘ground’ does not produce a fatter cat. (c) Our method can make a fatter cat while preserving identity.

Figure 7: Identity–attribute compositionality . We extract identity directions from three reference images and recombine them with various attribute directions. Our method preserves the identity of the chosen reference while expressing attributes such as glasses , hat , necklace , or fog , illustrating clean semantic disentanglement.

Figure 8: Advanced Editing Capabilities . (a) Our model can both intensify (+ fat) and diminish (– fat) the same attribute, while preserving the subject’s identity and overall visual coherence. (b) Beyond simple word-level attributes, ATA enables the transfer of complex, composite attributes across distinct object categories. For example, the attribute “ wearing a shirt with the text AI ” can be consistently applied to a human, a teddy bear, and a bronze statue.

Figure 9: Failure cases . (a) There is a bound on attribute strength; beyond this bound, artifacts appear in the generated images; (b) The method is less effective for certain particular attributes ( e.g . , color); (c) ATA is limited by the pre-trained generative prior of the backbone.

Explore similar work

OmniGen-AR: AutoRegressive Any-to-Image Generation
Jun 8, 2026Junke Wang, Xun Wang, Qiushan Guo +4Conditional Image Generation Image Generation
Institute of Trustworthy Embodied AI, Fudan University · 2Shanghai Collaborative Innovation Center of Intelligent Visual Computing · 3Bytedance Seed +1
End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer
May 1, 2026Wenda Chu, Bingliang Zhang, Jiaqi Han +4Image Tokenization Visual Tokenization
ByteDance Seed · California Institute of Technology · Stanford University
Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Rendering
Jun 1, 2026Dongxing Mao, Jinpeng Wang, Jiahao Tang +6Image Tokenization Visual Tokenization
Central South University · University of Oxford · Microsoft Research

Jun 8, 2026J/K move · Enter open · S save

OmniGen-AR: AutoRegressive Any-to-Image Generation

Junke Wang, Xun Wang, Qiushan Guo, Peize Sun+3

Institute of Trustworthy Embodied AI, Fudan University · 2Shanghai Collaborative Innovation Center of Intelligent Visual Computing · 3Bytedance Seed+1

Conditional Image Generation Image Generation Autoregressive Image Generation Attention Mechanisms Autoregressive Video Generation

Open paper AskPDF arXiv

Autoregressive (AR) models have demonstrated strong potential in visual generation, offering superior performance with simple architectures and optimization objectives. However, existing methods are typically limited to single-modality conditions, e.g., text, restricting their applicability in real-world scenarios that demand image synthesis from diverse controls. In this work, we present OmniGen-AR, a unified autoregressive framework for Any-to-Image generation. By discretizing various visual conditions through a shared visual tokenizer and text prompts with a text tokenizer, OmniGen-AR supports a broad spectrum of conditional inputs within a single model, including text (text-to-image generation), spatial signals (segmentation-to-image and depth-to-image), and visual context (image editing, frame prediction, and text-to-video generation). To mitigate the risk of information leakage from condition tokens to content tokens, we introduce Disentangled Causal Attention (DCA), which separates the full-sequence causal mask into condition causal attention and content causal attention. It serves as a training-time regularizer without affecting the standard next-token prediction during inference. With this design, OmniGen-AR achieves new state-of-the-art or at least competitive results across a range of benchmark, e.g., 0.63 on GenEval and 80.02 on VBench, demonstrating its effectiveness in flexible and high-fidelity visual generation.

Abstract

Explore similar work

OmniGen-AR: AutoRegressive Any-to-Image Generation

End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer

Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Rendering