Diffusion-based audio-visual generation methods typically perform joint denoising of audio and video latents throughout the entire sampling trajectory, repeatedly executing modality-specific denoising branches and cross-modal attention at every timestep. However, we observe that such uniform joint computation is not always necessary. The key is a modality-asymmetric recoverability: audio-side drift introduced by mid-stage caching can be substantially reduced by only a few terminal joint denoising steps, whereas video-side degradation caused by caching leads to persistent loss of motion and temporal details. This asymmetry suggests that inference acceleration should selectively cache recoverable audio-side computation while preserving full video refinement. Motivated by this observation, we propose Convergence-Adaptive Audio-Visual Cache (CAVC), a training-free modality-asymmetric caching framework. CAVC adopts a sandwich-structured computation allocation: after detecting that audio denoiser outputs enter a cache-tolerant regime, it caches and reuses the audio denoiser output to bypass audio denoising and cross-modal attention in the middle stage, while the video branch continues independent spatio-temporal refinement; a few joint steps are then restored at the end to reduce accumulated cross-modal drift and improve synchronization. Experiments across multiple public benchmarks demonstrate that CAVC achieves 1.10--1.35x wall-clock acceleration, improves several visual dynamics and temporal statistics, and maintains competitive audio fidelity and audio-visual alignment.
Figures & tables
Figure 1: A: CAVC generates high-fidelity video frames synchronized with complex audio waveforms. B: On General-80, training-free CAVC achieves a favorable quality-efficiency trade-off across generation quality, cross-modal alignment, and measured wall-clock inference speed. C: CAVC exploits modality-asymmetric recoverability: mid-stage audio-side drift induced by caching can be substantially reduced by a short terminal joint refinement, whereas video-side degradation is difficult to recover. CAVC preserves full video refinement while bypassing recoverable audio-side computation in the middle stage.
Figure 2: Overview of the CAVC framework. The denoising trajectory ( t=T→0 ) is dynamically partitioned based on modality convergence. (1A) Full AV Warm-up: During early high-noise stages, audio/video branches and AV cross-attention fully execute to establish semantic binding. (Trigger τ ): An online monitor tracks audio prediction changes; once stabilized, the audio cache activates. (1B) Cached Continuation: Cached audio predictions are reused. Bypassing the expensive audio branch and AV attention frees the video branch for independent spatio-temporal refinement without being over-constrained by static audio. (2) Final Joint Refinement: Full joint computation is terminally restored for a few steps to reduce accumulated cross-modal drift and refine audio-visual synchronization before decoding.
Figure 3: Empirical approximation residuals over the forward execution index k . Top: The instantaneous audio cache residual grows after cache triggering (dashed lines) but stays within a stable empirical range; a tighter threshold T ( θ in text; e.g., 0.3) delays the trigger and lowers the observed residual ceiling. Bottom: The video-only residual stabilizes and decays, suggesting that the dominant cross-modal dependencies are established during Phase A warm-up.
Figure 4: Qualitative comparison of audio-visual joint generation. Given the text prompt “a cat playing with a ball in the yard”, the generic Cache-DiT and TeaCache baselines exhibit visible artifacts and weaker prompt following. For this 121-frame 1K example, their cache-state estimation and update operations also introduce net overhead (see Table 1 ). In contrast, CAVC achieves a 1.15 × inference speedup while preserving stronger visual quality, spatio-temporal consistency, and dynamics.
Benchmark
Method
Speedup ↑
Visual Quality Huang et al. (2025)
Low-Level Statistics
Subj.Con ↑
Bg.Con ↑
Mot.Smth ↑
Dyn.Deg ↑
Aes.Qual ↑
Natural ↑
Temporal ↑
Flicker ↓
AVGen-Bench
TeaCache Liu et al. (2024a)
0.45 ×
0.864
0.920
0.989
0.272
0.452
0.473
0.619
0.522
Cache-DiT DefTruth (2025)
0.24 ×
0.869
0.920
0.989
0.298
0.462
0.463
0.612
0.517
Full AV HaCohen et al. (2026)
1.00 ×
0.7895
0.8622
0.9200
0.655
0.524
0.5272
0.6534
0.3346
CAVC (Ours)
1.10 × (+10.0%)
0.874 (+10.7%)
0.925 (+7.3%)
0.991 (+7.7%)
0.6701 (+2.3%)
0.522
0.5540 (+5.1%)
0.6600
0.2800 (-16.3%)
VABench T2AV
TeaCache Liu et al. (2024a)
0.64 ×
0.936
0.956
0.993
0.099
0.482
0.409
0.668
0.180
Table 1: Evaluation of Visual Quality and Low-Level Statistics across three benchmarks. Low-level statistics are no-reference diagnostic metrics; definitions are provided in the supplement. ↑ indicates higher is better, and ↓ indicates lower is better. Bold indicates the best performance. CAVC provides a favorable quality-efficiency trade-off while achieving inference speedups.
Benchmark
Method
Speedup ↑
Automated Audio & MLLM Quality
Audio-Visual & Physics
DNSMOS ↑
CLAP ↑
Aud.Real ↑
Expr. ↑
Vis.Real ↑
AV-F1 ↑
AV Sync ↑
Lo-Phy ↑
AVGen-Bench
TeaCache Liu et al. (2024a)
0.45 ×
1.607
0.366
4.055
4.264
4.609
0.601
0.429
3.936
Cache-DiT DefTruth (2025)
0.24 ×
1.676
0.418
4.068
4.319
4.566
0.614
0.249
3.911
Full AV HaCohen et al. (2026)
1.00 ×
1.706
0.435
3.925
4.070
4.475
0.5381
0.561
3.847
CAVC (Ours)
1.10 × (+10.0%)
1.688 (-1.1%)
0.438
4.077 (+3.9%)
4.294 (+5.5%)
4.723 (+5.5%)
0.5650 (+5.0%)
0.525 (-6.4%)
3.872
VABench T2AV
TeaCache Liu et al. (2024a)
0.64 ×
1.324
0.294
3.912
4.392
4.468
0.659
0.549
4.503
Table 2: Evaluation of Automated Audio & MLLM-Based Quality and Audio-Visual Alignment & Physics across three benchmarks. AV-F1 is an onset-based diagnostic metric. ↑ indicates higher is better; bold indicates best performance.
Table 3: Comprehensive ablation studies. (a) Branch caching/freezing strategy. (b) Alternative caching strategies. (c) Phase processing components and terminal refinement. (d) Hyperparameter analysis on threshold ( θ ) and window size ( W ). The configuration θ=0.30,W=3 is selected as the default setting, balancing acceleration and recoverability. Audio@ denotes the average forward execution index kτ at which audio caching is triggered, not the decreasing diffusion timestep τ . ↑ indicates higher is better, and ↓ indicates lower is better.
Cached Branch
Phase C
Dyn. ↑
Aes.Qual ↑
AV Sync ↑
CLAP ↑
None (Full AV)
3 steps
0.600
0.602
0.511
0.271
Audio (CAVC)
0 steps
0.560
0.563
0.495
-0.005 (0.0% Rec.)
Audio (CAVC)
1 step
0.640
0.603
0.679
0.220 (81.5% Rec.)
Audio (CAVC)
3 steps
0.620
0.596
0.502
0.235 (87.0% Rec.)
Audio (CAVC)
5 steps
0.620
0.596
0.566
0.247 (91.3% Rec.)
Video
0 steps
0.360 (0.0% Rec.)
0.286
0.290
0.300
Table 4: Empirical validation of modality-asymmetric capability recovery. Percentages denote gap closure from the corresponding 0-step cached condition toward Full AV, measured by CLAP for audio capability and Dynamics for video capability. Terminal recoupling repairs most audio capability lost to caching, whereas video dynamics remain unrecovered.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Setting
Base model
LTX-2.3-22B-dev
Distilled LoRA
ltx-2.3-22b-distilled-lora-384-1.1 , strength 0.8
Spatial upscaler
ltx-2.3-spatial-upscaler-x2-1.1
Denoising steps / generation seed
30 / 100
AVGen-Bench and VABench
1280×704 , 241 frames, 24 FPS
General-80
1536×1024 , 121 frames, 24 FPS
Appendix
Table 5: Generation configuration shared by Full AV, CAVC, Cache-DiT, and TeaCache.
Method
Parameter
Value
CAVC
convergence threshold θ / window W
0.30 / 3
Cache-DiT
warm-up steps
8
Cache-DiT
front/back compute blocks
8 / 0
Cache-DiT
residual-difference threshold
0.08
Cache-DiT
max cached / continuous cached steps
−1 / −1
Cache-DiT
TaylorSeer / transformer compilation
disabled / disabled
Appendix
Table 6: Method-specific cache configuration. A value of −1 means that no explicit maximum is imposed.
Split or AV-hard category
Prompts
General
40
Speech and lip synchronization
5
Music and pitch
5
Environmental sound
5
Physical-event synchronization
5
Animal vocalization
5
Appendix
Table 7: Composition of General-80.
Metric
Model
Definition
Subject consistency
DINO ViT-B/16
Mean non-negative feature similarity to the first and preceding frames
Background consistency
CLIP ViT-B/32
Analogous temporal similarity using CLIP image features
Motion smoothness
AMT-S
Agreement between interpolated and observed intermediate frames
Dynamic degree
RAFT
Fraction of videos exceeding the resolution-adjusted optical-flow criterion
Aesthetic quality
CLIP ViT-L/14 + LAION predictor
Mean predicted aesthetic score over sampled frames
Imaging quality
MUSIQ-SPAQ
No-reference per-frame quality, scaled by VBench to [0,1]
Appendix
Table 8: VBench visual metrics.
Metric
Definition
Direction
Sharpness
Variance of the grayscale Laplacian
Higher
Colorfulness
Hasler–Suesstrunk opponent-color statistic
Higher
Naturalness
Luminance/gradient kurtosis proximity to reference values
Higher
Temporal consistency
1/(1+std(d)/(mean(d)+ε)) for frame differences d
Higher
Flicker
Mean second difference of frame-average brightness changes
Lower
Motion smoothness
Consecutive Farneback-flow directional agreement, mapped to [0,1]
Higher
Appendix
Table 9: Pure-pixel no-reference diagnostics.
Figure 5: Empirical trigger curves over the forward execution index k for threshold values 0.30, 0.50, and 0.80. Solid lines show the mean instantaneous audio cache residual (top) and video-only residual (bottom), shaded regions show dispersion, and vertical dashed lines mark the corresponding mean trigger indices kτ . The reported CAVC configuration uses θ=0.30 and W=3 .
Benchmark
Median kτ
Mean kτ
Cached ratio
CAVC time (s)
Speedup
AVGen-Bench
4
5.29
82.4%
329.69
1.10 ×
VABench T2AV
4
4.91
83.6%
392.73
1.35 ×
Appendix
Table 10: Observed trigger statistics for the default configuration θ=0.30,W=3 . Trigger locations are reported using the forward execution index kτ .
Audio-driven video generation (A2V) has achieved promising progress in synthesizing temporally coherent and audio-visually aligned videos, yet its inference remains expensive due to the iterative denoising process of diffusion models. Existing caching methods mainly exploit temporal redundancy in visual features while overlooking the cross-modal alignment of A2V, where audio drives visual generation with highly non-uniform temporal importance. In this paper, we identify two levels of misalignment in existing A2V caching methods: temporal-semantic and computation-storage misalignment. To address them, we propose EchoCache, an energy-guided cross-modal caching framework for efficient A2V generation. EchoCache leverages audio time-frequency energy as a saliency anchor to guide latent-level cache updates and further introduces a dynamic timestep-latent caching mechanism with quantized cache management for joint efficiency and memory optimization. Extensive experiments on mainstream A2V models show that EchoCache consistently improves the latency-quality trade-off while preserving generation quality and audio-visual consistency. In particular, on Wan2.2-S2V over the EMTD benchmark, EchoCache achieves a 2.46x speedup with the best overall performance. Code is available at https://github.com/IF-LAB-PKU/EchoCache.
Jiayu Chen, Xiaoyu Wu, Rongshan Gao +6
Peking University Beijing, China · Taiyuan University of Technology Taiyuan, China
Recent audio-visual generation models can synthesize synchronized video and sound in a unified diffusion process, but their inference cost remains high because long video token sequences require repeated attention computation across denoising steps. A variety of acceleration techniques have been developed for video generation models, including low-bit quantization, attention sparsification, and feature caching. However, since these methods are originally designed for video generation, directly applying them to audio-visual models overlooks the interactions between the audio and video branches and may therefore disrupt audio-video synchronization. We present a synchronization-aware acceleration framework for efficient audio-visual generation. Our key observation is that bidirectional audio-video cross-attention reveals structured interactions between the two branches, with high responses often concentrated on a few sound-related visual and temporal regions. Guided by this interaction pattern, we introduce a protected sparse attention strategy that preserves high-fidelity computation for synchronization-critical tokens while sparsifying redundant attention interactions. By explicitly accounting for cross-modal dependence during acceleration, our method improves inference efficiency while keeping video quality, audio quality, and audio-video synchronization.
Shengchuan Gao, Teng Hu, Bohao Feng +4
Shanghai Jiao Tong University · Alibaba Token Hub, Alibaba Group · Alibaba Cloud Computing
Feature caching has become a mainstream training-free strategy for accelerating video diffusion models by reusing intermediate outputs across denoising steps. However, existing adaptive methods rely on heuristic difference measures misaligned with actual generation quality, employ a single global threshold that ignores heterogeneous phase dynamics, and never validate their budget estimates against true approximation error, allowing systematic bias to propagate uncorrected. We propose AlignCache, a closed-loop caching framework that directly aligns scheduling decisions with their true influence on the final generated video through three tightly coupled components: (1) phase-adaptive budget routing that partitions the denoising trajectory into structurally distinct phases, each governed by an independent threshold and a staleness-aware guard; (2) closed-loop threshold calibration that exploits counterfactual error measurement at the classifier-free guided output level to adjust decision boundaries online via an EMA feedback loop; and (3) momentum-augmented draft synthesis combining direction-magnitude gated residual extrapolation with anchor-guided unconditional correction. By transforming scheduling from a static risk accumulator into a feedback-driven, self-correcting controller, AlignCache consistently achieves superior acceleration-quality trade-offs across state-of-the-art video generation models. Our code is available at https://github.com/jiangjyjy/AlignCache.
Jiyue Jiang, Zhiyu Huo, Yuhan Chen +3
Department of Computer Science and Engineering, The Chinese University of Hong Kong · Theory Lab, 2012 Labs, Huawei Technology Co. Ltd