Meta Inverse Prompting for Video Generative Models

Yerin Jung; Jongheon Jeong

cs.CVOct 18, 2026

Meta Inverse Prompting for Video Generative Models

Authors: Yerin Jung, Jongheon Jeong

Organizations: Korea University

Abstract

Recent advances in video generative models have enabled flexible conditional generation and editing pipelines, such as text-to-video, image-to-video, and reference-guided synthesis. However, these workflows are inherently one-directional, mapping conditions to videos without providing a mechanism to invert this process. In complex media production settings, the ability to recover controllable textual conditions from a given video is highly desirable, yet remains underexplored. While inverse prompting has been studied for image generation, directly extending these approaches to video is challenging due to the need to capture temporal dynamics beyond static appearance, as well as the substantial computational cost of per-video optimization. In this work, we propose the first practical framework for efficient video inverse prompting. Specifically, we formulate inverse prompting as a meta-learning problem and introduce Meta Video Inverse Prompting (MVIP), which learns a meta-prompt optimized over temporal dimensions to enable fast and effective prompt inference. Our approach amortizes the inversion process, eliminating the need for costly iterative optimization at test time. Experimental results demonstrate that MVIP recovers higher-quality inverse prompts with significantly improved reconstruction fidelity compared to image-based inversion and iterative captioning baselines, while achieving substantial gains in efficiency. Code and video results are available at https://ueriniuno.github.io/mvip-page/.

Figures & tables

Figure 1: Illustration of MVIP and downstream editing. MVIP recovers a prompt from a target video through inverse prompting, enabling a video generator conditioned on the prompt to reproduce the video. Using the recovered prompt, video generation can be controlled by editing the prompt, enabling controllable video editing.

Figure 2: Overview of MVIP. MVIP meta-learns a shared soft-prompt initialization for video inverse prompting across video tasks. During inner-loop adaptation, a frozen VLM generates candidate prompts, and the candidate with the highest reconstruction score from a frozen video generator is selected as a pseudo-target for updating the soft prompt. The task-adapted prompts are then used to update the shared initialization, which can be directly applied to unseen videos.

Figure 3: Qualitative comparison of video reconstruction over time. Compared with the baselines, which begin to drift from the reference video at early time steps such as $f=4$ or $f=8$ , MVIP preserves the reference content and temporal progression throughout the sequence, remaining consistent even at $f=60$ . Additional discussion is provided in Appendix F.1 .

Target Video
Sampled Frame	$f_{1}$	$f_{2}$	$f_{3}$	$f_{4}$	$f_{5}$	$f_{6}$
Qwen2.5-VL [ 3 ]	A female athlete is running down the track, preparing to jump into a sand pit for long jump competition. She leaps off her feet and lands gracefully on one foot before landing fully onto both feet inside of the sandpit. The camera then cuts away from the action as she walks back towards where she started.
PEZ [ 65 ]	1. ipl lick minogue javelin athletics hurdle musicvideo … 2. cracking gopher crumbs ibm dden euro decathathletics … 3. hewadrenaline sprinter jumper latvia serbia … 4. robyn gwynecapped athletics athletics equalsardinia …
MVIP (Ours)	The video captures a female athlete in the midst of a long jump event at an indoor stadium. … She is running towards the sand pit, her body leaning forward as she approaches the takeoff board. As she reaches the board, she leaps into the air. … Her body is in mid-air when the video cuts to a close-up of her face. She lands in the sand pit. … The video then cuts to a scoreboard displaying her name, "Ineta Radevica," and her jump distance of 6.97 meters.

Table 2: Image-text qualitative comparison. Given sampled frames from the target video, our prompt inversion produces more temporally informative and semantically coherent descriptions compared to captioning and image-based inverse prompting baselines.

Table 5

Method	ViCLIP V-V $\uparrow$	DINOv2 $\uparrow$	CLIP-I $\uparrow$
Captioning (Qwen2.5-VL-7B) [ 3 ]	0.8021	0.6214	0.7872
Captioning (Qwen3.5-9B) [ 48 ]	0.8312	0.6207	0.8063
+ Detailed instruction	0.8123	0.6206	0.7932
MVIP (Ours)	0.8434	0.6516	0.8091

Table 5: Comparison with stronger captioning baselines on MSR-VTT. Bold and underline denote the best and runner-up results, respectively.

Dimension	Metric	RF-Edit	Wan-Edit	MVIP (Ours)
Compliance	Instruction Satisfaction	3.09	2.09	3.25
	Phrase Semantic Consistency	0.18	0.19	0.19
	Overall Semantic Consistency	0.23	0.25	0.25
Quality	Subject Consistency	0.90	0.93	0.92
	Background Consistency	0.94	0.95	0.95
	Temporal Flickering	0.98	0.97	0.97

Table 6: Quantitative comparison of video editing performance. We compare MVIP with RF-Edit [ 61 ] and Wan-Edit [ 30 ] under the same edit instructions in terms of instruction compliance, video quality, and content fidelity. Best results are marked in bold .

Figure 4: Qualitative comparison on prompt-based video editing. All methods use the same source video and edit instruction. More qualitative results can be found in Appendix F.3 .

Method	Before editing Source / Recovered	After editing Target / Edited
IVEBench	A woman is enjoying a sunny day on a yacht, smiling.	A woman is waving her hand and enjoying a sunny day on a yacht, smiling.
MVIP (Ours)	The, the woman is standing on a boat, holding a small object in her hand and examining it closely . She appears to be focused, …	The, the woman is standing on a boat, holding a small object in one hand and waving her other hand . She appears to be focused, …

Table 7: Prompt transformations for downstream video editing. For an IVEBench [ 12 ] example, we show the prompts recovered and edited by MVIP, with the benchmark’s source and target prompts for reference. Phrases modified to reflect the edit instruction are highlighted in bold .

Method	Meta	BoN	ViCLIP V-V $\uparrow$	DINOv2 $\uparrow$	CLIP-I $\uparrow$
MVIP (Ours)	$\checkmark$	$\checkmark$	0.8434	0.6516	0.8091
MVIP (Ours)	$\checkmark$	$\times$	0.8389	0.6174	0.8081
MVIP (Ours)	$\times$	$\checkmark$	0.8314	0.6303	0.8057
Captioning	–	–	0.8021	0.6214	0.7872

Table 9: Ablation on meta-prompt initialization and configuration. Bold and underline denote the best and runner-up results, respectively.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Additional qualitative reconstruction results.

Panel of Figure 6 — Figure 6: Additional qualitative reconstruction results.

Method	It.	CLIP-I $\uparrow$	DINOv2 $\uparrow$	VideoMAE $\uparrow$	DreamSim $\downarrow$	RAFT $\downarrow$
Captioning	0	0.858	0.733	0.969	0.191	0.471
Captioning	1	0.862	0.736	0.972	0.179	0.460
Captioning	2	0.862	0.735	0.972	0.180	0.458
Captioning	3	0.861	0.733	0.972	0.180	0.459
Ours	0	0.870	0.755	0.974	0.168	0.446

Appendix

Table 10: Comparison with iterative caption refinement. We extend the ARPO-style iterative refinement strategy to the video setting by refining the captioning baseline for three iterations. Although refinement improves the captioning baseline, it remains below MVIP across all reported reconstruction metrics.

Method	ViCLIP V-V $\uparrow$	DINOv2 $\uparrow$	CLIP-I $\uparrow$
Captioning	0.9281 $\pm$ 0.0018	0.8448 $\pm$ 0.0052	0.9224 $\pm$ 0.0008
MVIP	0.9441 $\pm$ 0.0017	0.8728 $\pm$ 0.0050	0.9309 $\pm$ 0.0024

Appendix

Table 11: Robustness to random seeds on the synthetic video benchmark. Results are reported as mean

\pm

standard deviation over three random seeds (42, 43, and 44).

Table 12: Results by editing subcategory. Results on the two predefined Subject Motion Editing subcategories of IVEBench [ 12 ] .

Dimension	Metric	RF-Edit [ 61 ]	Wan-Edit [ 30 ]	MVIP (Default)	MVIP (Random)
Compliance	Instruction Satisfaction	3.09	2.09	3.25	3.302 $\pm$ 0.039
	Phrase Semantic Consistency	0.18	0.19	0.19	0.191 $\pm$ 0.001
	Overall Semantic Consistency	0.23	0.25	0.25	0.247 $\pm$ 0.001
Quality	Subject Consistency	0.90	0.93	0.92	0.921 $\pm$ 0.002
	Background Consistency	0.94	0.95	0.95	0.951 $\pm$ 0.001
	Temporal Flickering	0.98	0.97	0.97	0.976 $\pm$ 0.001

Appendix

Table 13: Robustness to random seeds in downstream video editing. Random-seed results are reported as mean

\pm

standard deviation over three independently sampled seeds per video.

Panel of Figure 7 — Figure 7: Qualitative comparison on downstream video editing.

Table 14: Prompt transformations for downstream video editing. Relevant prompt changes are highlighted in bold.

Real videos
Method	Subject $\uparrow$	Background $\uparrow$	Flicker $\uparrow$	Motion Smooth. $\uparrow$	Dynamic $\uparrow$
PEZ [ 65 ]	0.7832	0.8897	0.9774	0.9855	0.8448
Captioning (Qwen2.5-VL) [ 3 ]	0.7706	0.8892	0.9788	0.9871	0.8793
Captioning + Refine [ 50 ]	0.7716	0.8907	0.9786	0.9880	0.8966
MVIP (Ours)	0.7777	0.8919	0.9816	0.9888	0.8621
Synthetic videos

Appendix

Table 15: VBench results on real and synthetic videos.

Split	Method	ViCLIP V-T $\uparrow$	SigLIP $\uparrow$	BLIP2-ITM $\uparrow$
Synthetic	FLF-Only	N/A	N/A	N/A
	PEZ [ 65 ]	0.1820	0.0711	0.5050
	Captioning (Qwen2.5-VL) [ 3 ]	0.2687	0.1408	0.5213
	Captioning + Refine [ 50 ]	0.2644	0.1419	0.5172
	MVIP (Ours)	0.2814	0.1550	0.7986
Real	FLF-Only	N/A	N/A	N/A

Appendix

Table 16: Text-alignment evaluation.

Figure 8: Qualitative comparison of prompts generated without and with the learned soft prompt $\Phi$ . The learned soft prompt yields a more detailed reconstruction-oriented prompt, capturing the subject, scene context, and action progression, while the prompt without $\Phi$ only describes the coarse action.

Training step	Decoded output (InstructZero) [ 10 ]
30	“It seems like you’ve provided a series of unrelated video clips and descriptions. Could you please clarify what specific information you’re looking for?”
60	“The prompt you provided seems to be a series of unrelated video clips with different scenes and themes. It appears to be asking for a description or analysis of the content shown in each clip.”
100	“It seems like you’ve provided a series of video clips with different scenes and themes. It looks like the task was to describe what each video clip shows.”

Appendix

Table 17: Natural-language interpretation of the learned soft prompt during meta-training. We decode the continuous soft prompt at different training steps using the procedure of InstructZero [ 10 ] .

Inner steps $K$	Time / outer update $\downarrow$	Relative cost
1	10.65 min	1.00 $\times$
3	33.01 min	3.10 $\times$

Appendix

Table 18: Runtime analysis of inner-loop adaptation depth. We report the average wall-clock time per outer update over three matched outer updates under the same training video schedule. Increasing the number of inner-loop adaptation steps from

K=1

K=3

increases runtime by 3.10

\times

Controllable Video Generation

Controllable video generation steers how videos look, move, and change over time using user-provided instructions and structured controls.

See all 321 papers

VI-Bench: Benchmarking Prompt Inversion from AIGC Videos
Sep 8, 2026Wulin Xie, Rui Zhao, Kecen Li +5VLM Evaluation Video Generation Evaluation
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · University of Virginia +3
InverseCrafter: Efficient Video ReCapture as a Latent Domain Inverse Problem
Dec 5, 2025Yeobin Hong, Suhyeon Lee, Hyungjin Chung +1Camera-Controlled Video Generation Video Diffusion Models
Graduate School of AI, KAIST, South Korea · EverEx, South Korea · Korea University, South Korea
ProactGen: Learning Proactive Clarification for Video Generation
Oct 15, 2026Zhixiao Wang, Yang Chen, Yifan Xu +5Video Generation Multimodal Video Generation
Nanjing University · University of Science and Technology of China · The Hong Kong University of Science and Technology (Guangzhou) +1

Sep 8, 2026J/K move · Enter open · S save

VI-Bench: Benchmarking Prompt Inversion from AIGC Videos

Wulin Xie, Rui Zhao, Kecen Li, Xiujin Liu+4

Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · University of Virginia+3

VLM Evaluation Video Generation Evaluation

Open paper AskPDF arXiv

Recent advances in video generation have made prompt-based control increasingly central to AIGC video generation. Prompts specify what a video should depict and how it should be represented, controlling factors such as visual style or camera behavior. Understanding this recoverability is important both for creative reuse and editing, and for assessing prompt leakage risks. However, existing video understanding benchmarks do not measure this capability: a caption may describe what is visible, but a replayable prompt must recover the generation-relevant controls needed to reproduce the video. To address this gap, we introduce VI-Bench, a benchmark built from 16.1 million real-user prompts and 900 human-verified AIGC videos. VI-Bench spans three progressively harder settings, namely single-shot semantic grounding, control over style and camera behavior, and multi-shot compositional inversion, and evaluates five generation-critical dimensions: subject, action, scene, style, and camera. We evaluate 18 representative VLMs, including 2 proprietary and 16 open-source models on VI-Bench, using an Inversion Score that measures prompt-level alignment with the original prompt and video-level fidelity of the regenerated video. The results reveal substantial limitations: even the strongest model achieves only 0.632 on Inversion Score, performance degrades sharply as samples require richer control and multi-shot reasoning, and models often produce plausible prompts whose regenerated videos deviate from the reference. These findings show that video prompt inversion is a distinct and under-evaluated capability requiring models to transform visual understanding into replay-stable generative control.

Abstract

Controllable Video Generation

Explore similar work

VI-Bench: Benchmarking Prompt Inversion from AIGC Videos

InverseCrafter: Efficient Video ReCapture as a Latent Domain Inverse Problem

ProactGen: Learning Proactive Clarification for Video Generation