Organizations: School of Software, Northwestern Polytechnical University, Xi’an 710129, China · School of Computer Science, Northwestern Polytechnical University, Xi’an 710129, China · Beijing University Of Technology, Beijing 100124, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China
Current video captioning methods usually use an encoder-decoder structure to generate text autoregressively. However, autoregressive methods have inherent limitations such as slow generation speed and large cumulative error. Furthermore, the few non-autoregressive counterparts suffer from deficiencies in generation quality due to the lack of sufficient multimodal interaction modeling. Therefore, we propose a non-autoregressive framework based on Diffusion model for Video Captioning (DiffVC) to address these issues. Its parallel decoding can effectively solve the problems of generation speed and cumulative error. At the same time, our proposed discriminative conditional Diffusion Model can generate higher-quality textual descriptions. Specifically, we first encode the video into a visual representation. During training, Gaussian noise is added to the textual representation of the ground-truth caption. Then, a new textual representation is generated via the discriminative denoiser with the visual representation as a conditional constraint. Finally, we input the new textual representation into a non-autoregressive language model to generate captions. During inference, we directly sample noise from the Gaussian distribution for generation. Experiments on MSVD, MSR-VTT, and VATEX show that our method can outperform previous non-autoregressive methods and achieve comparable performance to autoregressive methods, e.g., it achieved a maximum improvement of 9.9 on the CIDEr and improvement of 2.6 on the B@4, while having faster generation speed. The source code will be available soon.
Figures & tables
Fig. 1: The sample comparison between (a) Previous diffusion-based video captioning approach and (b) Our proposed DiffVC.
Fig. 2: Comparison on generation speed between non-autoregressive and autoregressive video captioning. Autoregressive (AR) Captioner is KG-VCN [ 6 ] .
Fig. 3: Comparison on generation quality between non-autoregressive and autoregressive video captioning, Autoregressive (AR) Captioner is KG-VCN [ 6 ] .
Fig. 4: The overall architecture and key modules of our proposed DiffVC. (a) is the overall architecture of DiffVC, the string in italics denotes the generated caption of a video. (b) is the architecture of the discriminative denoiser. (c) is the architecture of the non-autoregressive language model.
Method
Venue
MSR-VTT
MSVD
VATEX
B@4
M
R
C
B@4
M
R
C
B@4
M
R
C
Autoregressive methods
Two-stream [ 22 ]
TPAMI’20
39.7
27.0
-
42.1
54.3
33.5
-
72.8
-
-
-
-
STAT [ 23 ]
TMM’20
39.3
27.1
-
43.8
52.0
33.3
-
73.8
-
-
-
-
VideoTRM [ 2 ]
ACM MM’20
38.8
27.0
-
44.7
-
-
-
-
-
-
-
-
STGCN [ 24 ]
CVPR’20
40.5
28.3
60.9
47.1
52.2
36.9
73.9
93.0
-
-
-
-
TABLE I: Video captioning model performance on MSR-VTT and MSVD datasets. - denotes that the data is not given in the corresponding literature. The bold number denotes the best results among all non-autoregressive methods.
Fig. 5: Case study on MSR-VTT for video captioning. ‘GT’ denotes the ground-truth caption, and ‘NACF’ denotes the caption generated by NACF [ 12 ] .
Method
MSR-VTT
MSVD
B@4
M
R
C
B@4
M
R
C
DiffVC
44.5
31.1
63.9
56.7
53.5
37.1
72.5
95.6
w/o Discriminative Denoiser
43.1
29
60.5
52.3
52.0
34.3
71.4
90.2
w/o NAR Language Model
42.5
28.2
58.5
51.7
50.4
33.9
70.6
88.7
TABLE II: Ablation study for video captioning on MSR-VTT and MSVD datasets. The bold number denotes the best results among all methods.
N
MSR-VTT
MSVD
B@4
M
R
C
B-4
M
R
C
=10
44
30.5
61.4
52.1
53.2
34.3
70.8
90.2
=12
44.5
31.1
63.9
56.7
53.5
37.1
72.5
95.6
=14
44.2
31.1
63.2
55.8
52.0
36.1
72.2
93.7
TABLE III: Ablation study on the number of denoiser blocks. The bold number denotes the best results among all methods.
N
MSR-VTT
MSVD
B@4
M
R
C
B-4
M
R
C
=4
40
22.6
57.4
41.3
49.2
30.8
63.3
79.6
=6
44.5
31.1
63.9
56.7
53.5
37.1
72.5
95.6
=8
44.2
29.9
63.0
50.7
52.8
36.1
68.2
90.4
=10
43.7
27.4
60.8
48.5
50.9
32.7
68.4
87.7
TABLE IV: Ablation study on the number of LM blocks. The bold number denotes the best results among all methods.
N
MSR-VTT
MSVD
B@4
M
R
C
B-4
M
R
C
=5
41.5
22.6
57.3
47.7
51.0
32.4
67.1
83.5
=20
44.5
31.1
63.9
56.7
53.5
37.1
72.5
95.6
=50
45.0
32.3
64.1
61.8
53.8
37.2
73.8
97. 9
TABLE V: Ablation study on the inference step. The bold number denotes the best results among all methods.
Causal video generators must predict from the past, but they need not learn only from it. In streaming autoregressive video diffusion, each emitted segment becomes a commitment that future segments must preserve. Standard training, however, only asks each causal state to explain the present. This creates what we call a representation-level planning gap: states that fit the current segment may discard identity, layout, and motion information needed for a consistent future. We introduce Video-Mirai, a training-only method that closes this gap without changing causal inference: the generator rolls out causally, a frozen foresight encoder reads the completed rollout non-causally, and a lightweight predictor distills the resulting stopped-gradient targets into causal states. Future frames supervise representations, never generator inputs. At inference, the encoder and predictor are discarded, leaving the original architecture, per-step FLOPs, and KV-cache behavior unchanged. Video-Mirai improves a strong Causal-Forcing baseline on 5-second VBench from 83.8 to 84.6 in terms of Total Score. On 30-second rollouts beyond the training horizon, subject consistency improves from 84.9 to 88.5 and background consistency from 90.2 to 91.9. Ablations identify future-conditioned targets as the key ingredient, and probes show that future frames become more decodable from current features. Causality should constrain inference, not representation supervision. Our study highlights that visual autoregressive models need foresight. Project page: https://y0uroy.github.io/Video-Mirai.
Yonghao Yu, Lang Huang, Runyi Li +2
The University of Tokyo · National Institute of Informatics · Peking University
We describe an adaptation of VACE (Video All-in-one Creation and Editing) for real-time autoregressive video generation. VACE provides unified video control (reference guidance, structural conditioning, inpainting, and temporal extension) but assumes bidirectional attention over full sequences, making it incompatible with streaming pipelines that require fixed chunk sizes and causal attention. The key modification moves reference frames from the diffusion latent space into a parallel conditioning pathway, preserving the fixed chunk sizes and KV caching that autoregressive models require. This adaptation reuses existing pretrained VACE weights without additional training. Across 1.3B and 14B model scales, VACE adds 20-30% latency overhead for structural control and inpainting, with negligible VRAM cost relative to the base model. Reference-to-video fidelity is severely degraded compared to batch VACE due to causal attention constraints. A reference implementation is available at https://github.com/daydreamlive/scope.
Recent progress in large-scale generative models has substantially advanced video generation, yet existing methods remain constrained by a rigid inference paradigm. Bidirectional diffusion models excel at global coherence and visual fidelity but suffer from slow inference, while autoregressive models offer efficient and streaming generation at the cost of long-range consistency and exposure bias. We introduce Flex-Forcing, a unified training and inference framework that enables a video diffusion model to seamlessly operate under both bidirectional and autoregressive generation regimes. The core idea is a flexible chunking mechanism jointly defined over the temporal axis and denoising steps. This design allows the model to (1) perform flexible chunking according to different device budgets, (2) perform bidirectional inference across chunks for global structure planning, while generating frames autoregressively within each chunk for efficient and fine-grained synthesis, and (3) perform any-order, any-timestep autoregressive generation without the strict causal constraint. Extensive experiments on multiple video generation benchmarks demonstrate that Flex-Forcing achieves consistently better video quality, long-video stability than strong baselines with a rigid inference schedule, while offering faster inference.
Xinyin Ma, Julius Berner, Chao Liu +3
Electrical and Computer Engineering, National University of Singapore, Singapore · NVIDIA Research, California, USA