Carnator: Fast Text-to-Video Generation with Generation-Native Compatibility-Guided Cross-Request Reuse
Authors: Xingkun Yin, Xuebin Tang, Mingkun Xu, Hongyang Du
Organizations: Department of Electrical and Computer Engineering, University of Hong Kong, Hong Kong SAR, China · Guangdong Institute of Intelligence Science and Technology, Zhuhai, China
Video diffusion transformers produce high-quality videos, yet iterative denoising incurs substantial inference latency, limiting interactive and large-scale serving. Most existing acceleration methods focus on individual requests, thereby restricting efficiency gains to redundancy within a single generation trajectory. Recent cross-request reuse offers an additional source of savings, but existing approaches often infer reusability from coarse semantic similarity. This conflates semantic relatedness with generation-level computational compatibility, so aggressive reuse may accept incompatible historical computation while conservative reuse leaves substantial acceleration unrealized. We present Carnator, a cross-request acceleration framework that addresses this challenge by extracting and using generation-native compatibility evidence directly from the model's evolving internal states. Specifically, Carnator performs a lightweight early probe to construct an Early Signature from internal diffusion states, assessing reuse validity through risk-aware compatibility decisions. The same evidence characterizes reuse scope by localizing target-specific computation and guiding joint reuse of historical latent trajectories and sparse attention connectivity. Across three text-to-video backbones, Carnator consistently achieves higher cache-hit end-to-end acceleration than the evaluated cross-request baselines despite more selective cache acceptance, reaching up to 2.17× speedup while maintaining competitive generation quality.
Figures & tables
Figure 1: Failure of semantic-similarity-based reuse. A semantically similar historical request yields competitive CLIP-T and VBench-Q scores, yet naive reuse preserves the incompatible non-snowy background and violates the target request. Carnator detects the incompatibility and falls back to dense generation.
Figure 2: Overview of Carnator . Historical requests are first retrieved through text-level recall and filtered using generation-native Early Signatures. For an accepted cache, Carnator jointly reuses historical latent trajectories and sparse attention connectivity while preserving target-specific computation.
Figure 3: Early generation states provide informative signals for cross-request compatibility. Similar request pairs exhibit stronger alignment in early denoising representations, while easy and hard negative pairs show increasingly distinct prompt-conditioned residuals and latent dynamics.
Figure 4: Joint latent-attention reuse in Carnator .
Method
CLIP-T ↑
VBench-Q ↑
DiT Hit Latency (s) ↓
E2E Hit Latency (s) ↓
DiT Speedup (Hit) ↑
E2E Speedup (Hit) ↑
E2E Speedup ↑
Hit Rate
Wan2.2-TI2V-5B
Dense
0.3372
0.6331
265.23
293.88
1.00 ×
1.00 ×
1.00 ×
N/A
NIRVANA-VID
0.2955
0.6457
204.21
256.09
1.30 ×
1.15 ×
1.12 ×
80.67%
Chorus
0.3005
0.6392
175.48
264.47
1.51 ×
1.11 ×
1.09 ×
80.67%
Chorus II
0.2986
0.6400
201.58
222.44
1.32 ×
1.32 ×
1.24 ×
80.67%
CARNATOR
0.2913
0.6382
159.65
187.30
1.66 ×
1.57 ×
1.36 ×
72.67%
Table 1: End-to-end performance on the VidProM workload across three video diffusion models. Within each model, all methods process the same request sequence under the same generation configuration. For cross-request methods, cache misses fall back to dense generation.
Retrieval / Gate
Hit Rate
CLIP-T ↑
VBench-Q ↑
E2E Hit Latency (s) ↓
Speedup ↑
Text-Level Retrieval
77.33%
0.2725
0.6301
185.6
1.58 ×
+ Early Signature
72.67%
0.2913
0.6382
187.3
1.57 ×
Table 2: Ablation of reuse-validity assessment on the VidProM workload. Speedup is calculated with respect to dense generation.
Variant
Latency (s) ↓
Speedup ↑
CLIP-T ↑
VBench-Q ↑
Full Computation
266.26
1.00 ×
0.3391
0.6432
Latent Reuse Only
206.06
1.29 ×
0.3237
0.6436
Attention Reuse Only
160.91
1.65 ×
0.3291
0.6291
Joint Latent-Attention Reuse
155.85
1.71 ×
0.3281
0.6433
Table 3: Ablation of latent and attention reuse in CARNATOR .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Latent-state discrepancy for similar, easy-negative, and hard-negative request pairs across video frames.
Figure 6: Blockwise attention mass and sparse connectivity for two related text-to-video requests. Similar requests preserve substantial structural similarity in their attention patterns despite changes in object identity.
Figure 7: Qualitative result of Carnator
Figure 8: Heat Map of Evidence Region Revidence under the first ten denoising steps.
Figure 9: Qualitative comparison of CHAI across video diffusion architectures. From left to right, we show the OpenSora 1.2 baseline, CHAI on OpenSora 1.2, the Wan2.2 baseline, and a direct adaptation of CHAI to Wan2.2. CHAI preserves generation quality on its native OpenSora architecture, while the direct Wan2.2 adaptation exhibits noticeable visual degradation.
Region-update layers
Window
Latency (s) ↓
Speedup ↑
CLIP-T ↑
VBench-Q ↑
None (dense)
—
273.61
1.00 ×
0.3330
0.6699
5
13–17
160.67
1.70 ×
0.2681
0.6447
11
10–20
156.67
1.75 ×
0.2676
0.6415
15
8–22
150.58
1.82 ×
0.2700
0.6440
20
6–25
150.97
1.81 ×
0.2803
0.6497
25
3–27
148.97
1.84 ×
0.2749
0.6268
Appendix
Table 4: Layer-window ablation on Wan2.2-TI2V-5B. All reuse variants use the same accepted cache. Latency includes the dense probe, and speedup is relative to dense inference.
τ⋆
Coverage
Accepted n
Unsafe u
Empirical Risk
Uα(u,n)
(1.62,2.17,1.00,7.58)
71.43%
50
0
0.00%
4.50%
Appendix
Table 5: Risk-aware threshold selection and certification results. The target unsafe-reuse rate is δ=5% at a one-sided confidence level of 90% .
Text-to-video diffusion models deliver impressive results but remain slow because of the sequential denoising of 3D latents. Existing approaches to speed up inference either require expensive model retraining or use heuristic-based step skipping, which struggles to maintain video quality as the number of denoising steps decreases. Our work, CHAI, aims to use cross-inference caching to reduce latency while maintaining video quality. We introduce Cache Attention as an effective method for attending to shared objects/scenes across cross-inference latents. This selective attention mechanism enables effective reuse of cached latents across semantically related prompts, yielding high cache hit rates. We show that it is possible to generate high-quality videos using Cache Attention with as few as 8 denoising steps. When integrated into the overall system, CHAI is 1.65x - 3.35x faster than baseline OpenSora 1.2 while maintaining video quality.
Joel Mathew Cherian, Ashutosh Muralidhara Bharadwaj, Vima Gupta +1
Serving diffusion models for image-to-video generation is computationally expensive, posing significant challenges for large-scale deployment. Real I2V workloads often contain similar requests, such as repeated effect templates, related subjects, and recurring shot layouts. Existing cross-request acceleration methods mainly exploit this redundancy through feature reuse. We observe that similar I2V requests also share highly consistent sparse attention patterns, enabling historical sparse masks to serve as request-conditioned priors with almost no online mask-prediction overhead. We propose a cross-request reuse framework centered on sparsity reuse, with feature reuse as an optional extension safeguarded by a lightweight guidance enhancement. Our sparsity reuse is implemented as shared sparse mask reuse, which reuses high-quality sparse masks from similar historical requests to avoid per-request online mask prediction. Optional feature reuse applies downsampled computation to highly redundant spatiotemporal regions, mitigating boundary artifacts while preserving efficiency gains. Guidance enhancement reinforces image/text conditioning after reuse, mitigating semantic drift and condition-adherence issues. Experiments show that default sparsity reuse configuration preserves generation quality with a \textbf{2.16×} speedup.
Hao Liu, Chenghuan Huang, Hao Liu +6
Sun Yat-sen University · WeChat HPC, Tencent Inc. · WeChat Vision, Tencent Inc.
Diffusion Transformers (DiTs) achieve remarkable performance in video synthesis, but their iterative denoising process suffers from high inference latency. To address this, caching has emerged as an effective acceleration strategy by capitalizing on inter-step redundancy during denoising. Existing dynamic caching methods typically estimate the error that cache reuse would introduce at each denoising step (step error) to guide cache decisions, whereas our concern is how much quality loss cache reuse would cause in the final generated video (terminal error). We show that step error does not directly correspond to terminal error and that latent information helps capture their relationship, thereby informing cache decisions. Moreover, existing threshold-based methods cannot provide precise speedup control, making it difficult to meet practical requirements for user-specified acceleration targets. To address these limitations, we introduce MORCA, a cache scheduling framework trained through offline-to-online reinforcement learning to make latent-aware reuse/recompute decisions under user-specified acceleration targets. Extensive experiments on different video generation models across multiple target acceleration ratios demonstrate that MORCA achieves better generation fidelity than state-of-the-art caching methods under comparable computational budgets. Code is available at https://github.com/x10ngyx/MORCA.
Yuxiang Xiong, Ruiyan Wang, Wenqiang Wang +5
Shanghai Jiao Tong University · Alibaba Cloud Computing · Alibaba Token Hub, Alibaba Group