When to Lock Attention: Training-Free KV Control in Video Diffusion

Tianyi Zeng; Jincheng Gao; Tianyi Wang; Zijie Meng; Miao Zhang; Jun Yin; Haoyuan Sun; Junfeng Jiao

cs.CVMar 10, 2026

When to Lock Attention: Training-Free KV Control in Video Diffusion

Authors: Tianyi Zeng, Jincheng Gao, Tianyi Wang, Zijie Meng, Miao Zhang, Jun Yin, Haoyuan Sun, Junfeng Jiao,

Organizations: Shanghai Jiao Tong University, Shanghai, China · Tongji University, Shanghai, China · The University of Texas at Austin, Austin, TX, USA · Peking University, Beijing, China ·

Abstract

Maintaining background consistency while enhancing foreground quality remains a core challenge in video editing. Injecting full-image information often leads to background artifacts, whereas rigid background locking severely constrains the model's capacity for foreground generation. To address this issue, we propose KV-Lock, a training-free framework tailored for DiT-based video diffusion models. Our core insight is that the hallucination metric (variance of denoising prediction) directly quantifies generation diversity, which is inherently linked to the classifier-free guidance (CFG) scale. Building upon this, KV-Lock leverages diffusion hallucination detection to dynamically schedule two key components: the fusion ratio between cached background key-values (KVs) and newly generated KVs, and the CFG scale. When hallucination risk is detected, KV-Lock strengthens background KV locking and simultaneously amplifies conditional guidance for foreground generation, thereby mitigating artifacts and improving generation fidelity. As a training-free, plug-and-play module, KV-Lock can be easily integrated into any pre-trained DiT-based models. Extensive experiments validate that our method outperforms existing approaches in improved foreground quality with high background fidelity across various video editing tasks.

Figures & tables

Figure 1: We propose KV-Lock , which dynamically schedules key-value (KV) cache and classifier-free guidance (CFG) based on diffusion model hallucination detection. It enhances foreground quality while ensuring background consistency. Experiments demonstrate that our KV-Lock outperforms VACE ( Jiang et al., 2025 ) in both reference-based and reference-free video editing tasks.

Figure 2: Overview of the KV-Lock framework. The encoder first encodes the inputs. Subsequently, during the inversion process, the KV pairs of source tokens are cached. Then, in the denoising process, a hallucination-detection-based scheduler enables the dynamic fusion of newly generated KV and cached KV to ensure background consistency, while dynamic scheduling of CFG enhances foreground quality. Finally, decoding is performed by the decoder.

Method

VBench Metrics

Background

User Study

Inf. Time

\downarrow

\uparrow

\uparrow

\uparrow

\uparrow

\uparrow

Ave.

\uparrow

SSIM

\uparrow

PSNR

\uparrow

\uparrow

\uparrow

\uparrow

Ave.

\uparrow

FateZero ( Qi et al., 2023 )

87.17\%

92.89\%

94.72\%

53.84\%

57.53\%

77.23\%

0.7151

17.57

2.04

1.78

1.41

1.74

3.98

FLATTEN ( Cong et al., 2023 )

92.90\%

95.54\%

97.48\%

53.24\%

59.41\%

79.71\%

0.7716

19.30

2.41

3.15

2.26

2.60

\boldsymbol{1.14}

TokenFlow ( Geyer et al., 2023 )

{93.64\%}

96.17\%

98.46\%

57.22\%

69.67\%

83.03\%

0.8050

20.07

2.00

2.78

2.74

2.51

11.92

CFG-Zero* ( Fan et al., 2025 )

93.80\%

95.99\%

{98.75\%}

61.22\%

71.04\%

84.16\%

0.9107

26.65

4.30

3.93

3.81

4.01

5.58

APG ( Sadat et al., 2024 )

93.39\%

96.25\%

\boldsymbol{98.83\%}

60.09\%

71.53\%

84.02\%

0.9211

26.04

4.22

3.85

3.78

3.95

5.80

Table 1: Quantitative results.

Method	VBench Metrics						Background
Method	SC $\uparrow$	BC $\uparrow$	MS $\uparrow$	AQ $\uparrow$	IQ $\uparrow$	Ave. $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$
Variance KV schedule only	$93.01\%$	$95.89\%$	$98.10\%$	$60.28\%$	$71.19\%$	$83.69\%$	$0.9129$	$31.01$
CFG $\omega$ schedule only	$93.32\%$	$93.89\%$	$97.72\%$	$61.84\%$	$70.53\%$	$83.46\%$	$0.9217$	$29.84$
CFG $s^{*}$ schedule only	$91.76\%$	$92.18\%$	$96.92\%$	$60.16\%$	$70.17\%$	$82.24\%$	$0.9141$	$29.59$
CFG $s^{*}$ and $\omega$ schedule	$93.28\%$	$95.71\%$	$\boldsymbol{98.63\%}$	$61.67\%$	$70.95\%$	$84.05\%$	$0.9130$	$30.55$
Variance KV and CFG $\omega$ schedule	$93.26\%$	$94.79\%$	$97.88\%$	$61.70\%$	$71.25\%$	$83.78\%$	$0.9261$	$30.53$

Table 2: Results of ablation study.

Figure 3: Two comparison experimental samples. It can be observed that FLATTEN ( Cong et al., 2023 ) and TokenFlow ( Geyer et al., 2023 ) achieve an almost complete failure, with extensive distortions and artifacts. Specifically, in the first sample, the two eyes of the fox generated by VACE ( Jiang et al., 2025 ) and CFG-Zero* ( Fan et al., 2025 ) are asymmetrical and unnatural, APG ( Sadat et al., 2024 ) causes obvious distortion to the fox. while the fur texture rendered by KV-Lock is more refined than that of ProEdit ( Ouyang et al., 2025 ) . In the second sample, the distant dust generated by VACE ( Jiang et al., 2025 ) is excessively bright and unrealistic, and the road surface is an asphalt road with soil instead of a dirt road, also appearing in CFG-Zero* ( Fan et al., 2025 ) and APG ( Sadat et al., 2024 ) ; In CFG-Zero* ( Fan et al., 2025 ) , the dust raised by the front vehicle exhibits an obvious unnatural boundary. For APG ( Sadat et al., 2024 ) , the rear vehicle generates no dust at all, and ProEdit ( Ouyang et al., 2025 ) produces more prominent distant dust than near dust, which are contrary to common sense.

Figure 4: More samples of KV-Lock, including changing, removing and adding tasks.

Figure 5: In the early diffusion stage, the variance of in-support and hallucination samples are both high. After hallucination is detected, KV-Lock control the variance under threshold $\tau$ through dynamic scheduling.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value	Note
$\tau$	$0.01$	Hallucination threshold
Total diffusion timestep	$50$	-
$W$	$10$	Hallucination detection window size
$\kappa$	$20$	Hallucination detection final step ( Aithal et al., 2024 )
$\omega_{0}$	$5$	Origin CFG scale ( Jiang et al., 2025 )
$b$	$2$	Clamp boundar

Appendix

Table 3: Hyperparameter Settings.

Figure 6: KV-cache memory footprint for each sampling timestep.

Figure 7: Video editing task without reference image. The task requires converting the steam locomotive in the original video into a technologically advanced train. As can be observed, the trains generated by CFG-Zero*, APG, and VACE suffer from obvious derailment. The train lights in the APG result distort the surrounding environment slightly. Moreover, the original long train is reduced to only two carriages in the outputs of CFG-Zero* and VACE. ProEdit produces unnatural connections between carriages and the distant background, as well as unrealistic ground dust. Only our method generates a properly scaled train without derailment, yielding natural and stable videos.

Figure 8: Video editing task with reference image. This task requires replacing the tablet held by the vlogger in the video with a piece of bread. Notably, the subject raises her right hand while presenting, leaving only her left hand to hold the object. As observed, CFG-Zero*, APG, and VACE directly replace the tablet with bread. However, physically, the bread cannot maintain such a posture when the right hand is raised; it would fall or at least tilt, making these results unrealistic. In contrast, ProEdit and our method introduce a tray under the bread to improve physical plausibility. Furthermore, the bread generated by ProEdit exhibits inconsistent brightness and saturation with the surrounding environment, while our method achieves both physical realism and environmental consistency.

Figure 9: Video editing task with reference image. This task aims to replace the content on the canvas with Monet’s painting from the reference image. Due to interference from the surrounding context and the color information on the palette, none of the methods can directly and precisely transfer Monet’s painting onto the canvas. However, the results of CFG-Zero*, ProEdit, and VACE exhibit excessively high saturation and deviate completely from the style of Monet’s work, as the model directly colors the output based on the surrounding environment. The result of APG is overly dark in tone; although it attempts to approximate the target style, it still fails to capture the characteristics of the reference image. Only our method preserves the composition of Monet’s painting in the reference image. The waves, clouds, distant trees, and houses are clearly visible, and the overall brushwork is the closest to the target. The slight tonal difference is a reasonable adjustment made by the model to adapt to the ambient lighting.

Video Diffusion Models

Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.

See all 732 papers

ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformer
Mar 16, 2026Ruonan Yu, Zhenxiong Tan, Zigeng Chen +2Diffusion-Based Video Editing Video Editing
National University of Singapore · Shanghai Jiao Tong University
Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion
May 18, 2026Peiliang Cai, Evelyn Zhang, Jiacheng Liu +8Video Diffusion Models Autoregressive Diffusion
1SJTU · 2SDU · 3HUST +4
Diff-VF: Training-free High-quality Long Video Generation via Diffusion Model
Aug 6, 2026Haoning Yang, Xinyuan Chen, Yaohui Wang +1Diffusion Model Guidance Video Diffusion Models
Shanghai Jiao Tong University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China

Mar 16, 2026J/K move · Enter open · S save

ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformer

Ruonan Yu, Zhenxiong Tan, Zigeng Chen, Songhua Liu+1

National University of Singapore · Shanghai Jiao Tong University

Diffusion-Based Video Editing Video Editing Video Diffusion Models Controllable Video Generation Diffusion Transformer Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Diffusion Transformers (DiTs) have demonstrated remarkable scalability and quality in image and video generation, prompting growing interest in extending them to controllable generation and editing tasks. However, compared to the image counterparts, progress in video control and editing remains limited, mainly due to the scarcity of paired video data and the high computational cost of training video diffusion models. To address this issue, in this paper, we propose a video-free tuning framework termed ViFeEdit for video diffusion transformers. Without requiring any forms of video training data, ViFeEdit achieves versatile video generation and editing, adapted solely with 2D images. At the core of our approach is an architectural reparameterization that decouples spatial independence from the full 3D attention in modern video diffusion transformers, which enables visually faithful editing while maintaining temporal consistency with only minimal additional parameters. Moreover, this design operates in a dual-path pipeline with separate timestep embeddings for noise scheduling, exhibiting strong adaptability to diverse conditioning signals. Extensive experiments demonstrate that our method delivers promising results of controllable video generation and editing with only minimal training on 2D image data. Codes are available https://github.com/Lexie-YU/ViFeEdit.

Abstract

Video Diffusion Models

Explore similar work

ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformer

Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion

Diff-VF: Training-free High-quality Long Video Generation via Diffusion Model