Rethinking Structure Preservation in Text-Guided Image Editing with Visual Autoregressive Models
Authors: Tao Xia, Jiawei Liu, Yukun Zhang, Ting Liu, Wei Wang, Lei Zhang
Organizations: Beijing Institute of Technology, Beijing, China · Shenyang Institute of Automation, CAS, Shenyang, China · Meitu Inc, MTLab, Beijing, China
Visual autoregressive (VAR) models have recently emerged as a promising family of generative models, enabling a wide range of downstream vision tasks such as text-guided image editing. By shifting the editing paradigm from noise manipulation in diffusion-based methods to token-level operations, VAR-based approaches achieve better background preservation and significantly faster inference. However, existing VAR-based editing methods still face two key challenges: accurately localizing editable tokens and maintaining structural consistency in the edited results. In this work, we propose a novel text-guided image editing framework rooted in an analysis of intermediate feature distributions within VAR models. First, we introduce a coarse-to-fine token localization strategy that can refine editable regions, balancing editing fidelity and background preservation. Second, we analyze the intermediate representations of VAR models and identify structure-related features, by which we design a simple yet effective feature injection mechanism to enhance structural consistency between the edited and source images. Third, we develop a reinforcement learning-based adaptive feature injection scheme that automatically learns scale- and layer-specific injection ratios to jointly optimize editing fidelity and structure preservation. Extensive experiments demonstrate that our method achieves superior structural consistency and editing quality compared with state-of-the-art approaches, across both local and global editing scenarios.
Figures & tables
Figure 2: Method overview. Given a source image Isrc and its corresponding text description Tsrc , our framework generates the edited image Iedit according to the target text Ttgt . The tokenizer E encodes the input Isrc into multi-scale residuals {R1,…,RK} . For clarity, we illustrate the inference pipeline at the k -th scale. The framework adopts a dual-branch architecture, where the source branch takes F~k−1src and the target branch takes F~k−1tgt , producing probability distributions Pksrc and Pktgt . The intermediate feature maps—illustrated in (b)—are selectively injected from the source branch into the target branch ( Sec. 3.3 ) with learnable weights ( Sec. 3.4 ). The predicted distributions Pktgt are fed into the CFTL module ( Sec. 3.2 ) to obtain a refined editing mask M^k , which is then used by token reassembly (TR) to produce the prediction Rktgt at the k -th scale. Finally, multi-scale predictions {R1,…,Rγ,Rγ+1tgt,…,RKtgt} are jointly decoded to produce the final edited image Iedit . Here, γ denotes the number of source scales reused in the target branch.
Figure 3: Effect of CFG on background preservation and editing fidelity. A higher CFG yields better editing fidelity but weaker background preservation, whereas a lower CFG produces the opposite effect. Our hybrid method achieves a good balance. The number in each grid indicates the amount of tokens that need to be replaced at each position.
Figure 4: VAR features and attention maps. PCA visualization of intermediate representations and attention maps in a VAR block. Cross-attention (CA) offers rough object localization akin to diffusion models, but VAR self-attention (SA) does not display the spatial affinity structure typically observed in diffusion-based methods.
Figure 5: Dependency analysis for feature injection. Top: Experimental results of injecting features at different scales and layers, where the numbers denote layer and scale indices. Injecting features at the 0-th layer and at scales 5–8 leads to markedly better structural preservation. Bottom: Feature injection ratios after genetic algorithm optimization.
Figure 6: Qualitative comparison with baselines. Visual results of text-guided image editing from P2P [ 11 ] , MasaCtrl [ 3 ] , RFInversion [ 35 ] , AREdit [ 49 ] and SAVAREdit (Ours). The original images and corresponding source/target prompts are provided. Our method achieves superior detail preservation in non-edited regions for local edits, and better structural consistency in global editing scenarios.
Method
Base Model
Structure
Background Preservation
CLIP Similarity ↑
Distance ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Whole
Edited
P2P [ 11 ]
diffusion
0.0694
17.87
0.7114
0.2088
25.01
22.44
Pix2Pix-Zero [ 30 ]
diffusion
0.0617
20.44
0.7467
0.1722
22.80
20.54
MasaCtrl [ 3 ]
diffusion
0.0284
22.17
0.7967
0.1066
23.96
21.16
PnP [ 48 ]
diffusion
0.0282
22.28
0.7905
0.1134
25.41
22.55
PnP-DirInv. [ 20 ]
diffusion
0.0243
22.46
0.7968
0.1061
25.41
22.62
Table 1: Quantitative comparison with baselines. Our method achieves comparable CLIP similarity to other state-of-the-art methods while demonstrating superior structure preservation.
Figure 7: Ablation study on CFTL. CFTL enables more precise localization of editable regions, thereby achieving better background preservation while maintaining high editing fidelity.
Figure 8: Ablation study on FI. FI exhibits strong capability in preserving image structures even without reusing any tokens ( γ=0 ). Moreover, compared with the token-reuse strategy ( γ>0 ), FI provides greater flexibility and achieves a better balance between structural consistency and editing fidelity.
Figure 9: Ablation study on AFI. AFI achieves a better balance between structural consistency and editing fidelity.
Text-guided video editing modifies target content while preserving the appearance and temporal coherence of unedited regions. Training-based approaches provide strong control but demand substantial data and computation. Training-free methods fall into inversion-free and inversion-based paradigms. Inversion-free approaches avoid trajectory recovery, but their source-preserving guidance can limit editing strength and leave semantic changes incomplete. Inversion-based approaches recover a latent trajectory before regeneration, where approximation errors can accumulate and cause source-content drift and temporal inconsistency. We introduce Edit-VAR, the first training-free and inversion-free framework for text-guided video editing with a pretrained visual autoregressive video model. Edit-VAR directly encodes the source video into multi-scale discrete tokens and performs probability-guided conditional token replacement for source preservation. Attention-guided token-wise and scale-aware modulation selectively relaxes source constraints over edit-relevant positions and generation stages. Scale-Decoupled Generation, implemented as late-scale constraint release, regenerates motion-consistent details and reduces texture fragmentation. Residual-guided token pruning further exploits redundancy at the final two high-resolution scales to reduce inference cost. Extensive experiments and a blind user study demonstrate that Edit-VAR outperforms existing training-free video editing methods overall in editing fidelity, source preservation, temporal coherence, and inference efficiency.
Chongbo Zhao, Jiangming Wang, Xilai Wang +5
Sun Yat-sen University · South China University of Technology · Tsinghua University +3
Next-scale visual autoregressive models (VARs) have emerged as a powerful generative paradigm, producing high-quality images through efficient coarse-to-fine prediction. However, their potential for text-guided image editing remains largely underexplored. Existing training-free VAR editing approaches often formulate editing as target-conditioned regeneration guided or constrained by the source image, and may rely on inversion, test-time optimization, attention control, or user-provided masks. This generation-centric formulation does not fully exploit the multiscale source representations provided by VARs and may introduce additional computation or intervention. We instead take a source-centric perspective on VAR editing, in which the encoded source image tokens serve as the primary visual state and the editing process focuses on condition-induced changes. Based on this perspective, we propose EditMod, which compares source- and target-conditioned predictions under a shared autoregressive context, treats their difference as a scale-wise editing direction, and applies it as a residual update to source tokens at selected scales. Experiments show that EditMod achieves leading source-image fidelity while maintaining strong text alignment, and completes end-to-end editing of a 1K image in only 1.57 seconds on a single A100 GPU without per-image preparation.
Hongyi Fang, Chuwen Xie, Benjia Zhou +6
Beijing Institute of Technology, Zhuhai · Zhejiang University · Tencent +1
We address the problem of prompt-guided image editing in visual autoregressive models. Given a source image and a target text prompt, we aim to modify the source image according to the target prompt, while preserving all regions which are unrelated to the requested edit. To this end, we present Masked Logit Nudging, which uses the source image token maps to introduce a guidance step that aligns the model's predictions under the target prompt with these source token maps. Specifically, we convert the fixed source encodings into logits using the VAR encoding, nudging the model's predicted logits towards the targets along a semantic trajectory defined by the source-target prompts. Edits are applied only within spatial masks obtained through a dedicated masking scheme that leverages cross-attention differences between the source and edited prompts. Then, we introduce a refinement to correct quantization errors and improve reconstruction quality. Our approach achieves the best image editing performance on the PIE benchmark at 512px and 1024px resolutions. Beyond editing, our method delivers faithful reconstructions and outperforms previous methods on COCO at 512px and OpenImages at 1024px. Overall, our method outperforms VAR-related approaches and achieves comparable or even better performance than diffusion models, while being much faster. Code is available at 'https://github.com/AmirMaEl/MLN'.
Amir El-Ghoussani, Marc Hölle, Gustavo Carneiro +1
Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany · University of Surrey, United Kingdom