ProactGen: Learning Proactive Clarification for Video Generation

Zhixiao Wang; Yang Chen; Yifan Xu; Chenbin Li; Yue Chang; Kai Long; Fengming Dong; Limin Wang

cs.CVOct 15, 2026

ProactGen: Learning Proactive Clarification for Video Generation

Authors: Zhixiao Wang, Yang Chen, Yifan Xu, Chenbin Li, Yue Chang, Kai Long, Fengming Dong, Limin Wang

Organizations: Nanjing University · University of Science and Technology of China · The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Jiao Tong University

Abstract

Recent advances in video generation let users create high-quality videos from natural language descriptions, yet obtaining results that match user intent remains time-consuming: individual generations are computationally expensive, and underspecified prompts force repeated cycles of generation, inspection, and revision. Existing video prompt optimization methods improve generation through automatic rewriting, but cannot supply the ideas and preferences that were not in the initial description. We propose ProactGen, a video generation framework that learns proactive clarification, eliciting user intent through questions grounded in generated video drafts. Specifically, we train a vision-language model as a questioning policy with reinforcement learning: observing the initial prompt and a video draft, it learns to ask the questions most valuable to downstream generation, rewarded by the alignment between the final generated video and a target reference. We further find that effective questioning does not require drafts of final-output quality, and exploit this property by generating drafts with fewer denoising steps, keeping the added cost of clarification low. Experiments show that ProactGen improves target alignment over direct generation by at least 1.8x the gain of the strongest baselines. Moreover, a single round of ProactGen already exceeds the alignment that the strongest baseline reaches after four refinement rounds, in 29% of its time.

Figures & tables

Figure 1: A user’s initial prompt often omits preferences the user already has in mind (here, the hand gesture, the boy’s posture, the table’s material, and the framing). (a) Automatic prompt optimization rewrites the prompt from the prompt alone, so missing preferences can only be guessed. (b) Uncertainty-based clarification asks the user, but derives its questions from the prompt text without knowing what the generator will actually produce. (c) ProactGen first observes what the generator produces for the prompt through a low-cost draft, asks the user targeted questions grounded in it, and learns what to ask from the alignment of the final video.

Figure 2: Overview of ProactGen. The questioning policy asks scoped questions conditioned on the initial prompt and a low-cost video draft; a simulated user answers from the reference video; the answers are merged into a refined prompt from which the final video is generated at full budget. During training, the alignment between the final video and the reference is the reward for updating the questioning policy; all other components are frozen.

Method	R	DS-DTW $\uparrow$	3D-VAE $\downarrow$	DnS $\uparrow$	Obj. $\uparrow$	Bkg. $\uparrow$	Style $\uparrow$	Cam. $\uparrow$	Overall $\uparrow$
		InstanceCap-500			MiraBench-500 ∗
Direct	–	0.658	144.03	$-0.689$	0.175	0.135	0.144	0.141	0.198
SCMAPR	–	0.657	142.94	$-0.685$	0.178	0.141	0.149	0.143	0.206
SSPO	1	0.665	143.00	$-0.676$	0.182	0.133	0.148	0.144	0.205
SSPO	4	0.669	142.53	$-0.672$	0.182	0.133	0.146	0.143	0.204
Proactive T2I	5	0.662	143.55	$-0.685$	0.195	0.152	0.154	0.146	0.221

Table 1: Target alignment. Left: 500 InstanceCap samples, measured as similarity between the final video and the reference video. Right: MiraBench-500, measured as ViCLIP alignment between the final video and the structured captions of the reference (Obj., Bkg., Style, Cam.) and its dense caption (Overall). R gives refinement rounds for SSPO, question turns for Proactive T2I, and interaction rounds for ProactGen. ∗ The official MiraBench evaluation set is not released; we sample 500 MiraData videos disjoint from our training set and use its metrics on them.

Setting	Trained	Draft	DS-DTW $\uparrow$	3D-VAE $\downarrow$	DnS $\uparrow$
Direct	–	–	0.658	144.03	$-0.689$
Base, text	$\times$	$\times$	0.672	142.63	$-0.669$
Base, draft	$\times$	✓	0.668	142.93	$-0.672$
RL, text	✓	$\times$	0.675	142.56	$-0.665$
ProactGen	✓	✓	0.678	142.34	$\mathbf{-0.659}$

Table 2: Ablation of RL training and draft observation. Trained: the policy is trained with RL; Draft: the policy observes an 8-step draft.

Figure 3: Per-sample time versus target alignment (InstanceCap, DS-DTW). The horizontal axis is linear up to 50 s and logarithmic beyond.

Figure 4: Qualitative examples. Each row shows the reference video, the initial prompt, the draft, three question–answer pairs, and the final video, with the refined prompt below and answers highlighted. In both cases the questions land where the prompt was silent and the draft decided on the user’s behalf.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Comparison	DreamSim-DTW	3D-VAE	DnS
ProactGen vs. Direct	$+0.020$ $[+0.015,+0.024]$	$+1.68$ $[+1.23,+2.15]$	$+0.030$ $[+0.020,+0.041]$
ProactGen vs. SCMAPR	$+0.021$ $[+0.016,+0.025]$	$+0.60$ $[+0.12,+1.10]$	$+0.026$ $[+0.015,+0.038]$
ProactGen vs. RAPO++-SSPO (1)	$+0.013$ $[+0.008,+0.017]$	$+0.66$ $[+0.15,+1.17]$	$+0.017$ $[+0.005,+0.029]$
ProactGen vs. RAPO++-SSPO (4)	$+0.009$ $[+0.004,+0.013]$	$+0.19$ $[-0.32,+0.70]$	$+0.013$ $[+0.001,+0.026]$
ProactGen vs. Proactive T2I (5)	$+0.016$ $[+0.012,+0.020]$	$+1.21$ $[+0.76,+1.66]$	$+0.026$ $[+0.015,+0.037]$
ProactGen vs. Proactive T2I (15)	$+0.010$ $[+0.006,+0.014]$	$+1.03$ $[+0.56,+1.48]$	$+0.019$ $[+0.008,+0.031]$

Appendix

Table 3: Paired differences with 95% confidence intervals for the comparisons in the main text (

n=500

). DreamSim-DTW and DnS are first minus second; 3D-VAE is second minus first, so positive values in all three columns favor the first method; parentheses give refinement rounds or question turns.

Method	Draft	Q&A + merge	Ref. I/O	Rewrite + feedback	Final gen.	Total
Direct	–	–	–	–	7.09	7.09
SCMAPR	–	–	–	97.18	7.08	104.26
RAPO++-SSPO (1)	–	–	–	14.18	7.07	21.24
RAPO++-SSPO (4)	–	–	–	48.07	7.07	55.14
Proactive T2I (5)	–	90.70	1.45	–	7.09	99.25
Proactive T2I (15)	–	233.88	1.45	–	7.09	242.43

Appendix

Table 4: Per-sample stage times (seconds, mean over 100 samples) of each method in the single-round setting. “Questioning and merging” covers policy questioning, simulated-user answering, prompt merging, and observation packing; “reference I/O” is the media processing for the simulated user’s reference frames.

Video Generation

Video generation studies models that create and edit moving images from text, images, audio, and structured controls.

See all 350 papers

VideoGen-Agent: Reinforcing Video Generation Agents
Sep 21, 2026Binxu Li, Haoyi Duan, Yuhui Zhang +9RL for Video Generation Agentic Video Generation
Princeton University · Stanford University · UC Davis +4
Knowledge-Intensive Video Generation
May 31, 2026Chenxu Wang, Mingda ChenT2V Generation Video Generation
Fudan University · Shanghai Jiao Tong University
VQQA: An Agentic Approach for Video Evaluation and Quality Improvement
Mar 12, 2026Yiwen Song, Tomas Pfister, Yale SongPrompt Optimization for T2V Generation Video Generation
Google

Sep 21, 2026J/K move · Enter open · S save

VideoGen-Agent: Reinforcing Video Generation Agents

Binxu Li, Haoyi Duan, Yuhui Zhang, Yaohui Zhang+8

Princeton University · Stanford University · UC Davis+4

RL for Video Generation Agentic Video Generation Video Generation Multi-Shot Video Generation

Open paper AskPDF arXiv

Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, specific identities, physical consistency, or ordered events. In this paper, we present VideoGen-Agent, a multimodal agent trained through multitask agentic reinforcement learning to use external tools for video generation. The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the prompt and intermediate observations to guide its decisions. We train a shared policy on a category-balanced dataset spanning six tasks. Supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior, which is then refined through reinforcement learning. A category-aware hybrid reward evaluates tool-call validity, task-appropriate tool use, and generated video quality. We further introduce VABench, a held-out benchmark of 600 prompts covering procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, and multi-shot temporal structure. On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points, from 56.5 to 75.6. Upgrading the generation tools further raises the score to 86.1 without additional agent training. Human raters prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons. These results support learning tool use across video-generation tasks and show that the trained agent can benefit from subsequent advances in generation tools. Project page: https://andyca111.github.io/VideoGen_Agent/

Abstract

Video Generation

Explore similar work

VideoGen-Agent: Reinforcing Video Generation Agents

Knowledge-Intensive Video Generation

VQQA: An Agentic Approach for Video Evaluation and Quality Improvement