Controllable Video Generation

Overview

The field studies how to make generated videos follow user intent, from preserving a subject or style to specifying poses, object trajectories, camera moves, events, and physical properties. Common approaches condition diffusion models with text, images, tracks, masks, or 3D geometry, adapt pretrained models with ControlNet or LoRA modules, or steer sampling at inference; newer world models add explicit state, planning, and memory for interactive, long-horizon generation. Evaluation spans prompt and control adherence, identity and scene consistency, motion accuracy, physical plausibility, and video quality, including tests on driving, human-object interaction, and cinematic scenes. A central tension remains: stronger or denser controls can improve precision but may reduce naturalness or visual quality, while long rollouts can drift or fail to realize intended state changes even when individual frames look convincing.

Latest papers

CardsList

342 papers34 papers in the last four weeks, down 26% on the four weeks before. 0.3% of all new papers.

  1. SolarRenderer: Few-Step Generative Reference Rendering

    Oct 20, 2026Guian Fang, Shengju Qian, Dongyue Lu +6Controllable Video GenerationVideo Diffusion Models

  2. Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds

    Oct 20, 2026Jihai Zhang, Runjia Qian, Kaiqin Kong +15Real-Time Interactive Video GenerationAutoregressive Video Generation

  3. State World Models: Execute the Rules, Generate the Pixels

    Oct 19, 2026Chan Yan Ki, Yanran Zhang, Wenzhao Zheng +5World ModelsAction-Conditioned World Models

  4. Mise-en-Video: Fine-Grained 3D Control for Multi-Shot Video Generation

    Oct 19, 2026Yingshuang Zou, Xiaoyu Shi, Yawen Luo +8Multi-Shot Video GenerationControllable Video Generation

  5. Wan-Alpha++: Animating a Single RGBA Asset into Controllable Transparent Video Effects

    Oct 19, 2026Haotian Dong, Xin Wang, Wenjing Wang +7Controllable Video GenerationVideo Diffusion Models

  6. UniState: Unifying Memory and Intervention for Controllable World Simulation

    Oct 18, 2026Jaeseok Jeong, Kyungmook Choi, Sohyun Chung +2Controllable Video GenerationVideo World Models

  7. Painting Change Map as guidance for generating painting processes

    Oct 18, 2026Markus Pobitzer, Chang Liu, Niccolò Biondi +1Controllable Video GenerationVideo Generation

  8. Map2Vid: Embodied Navigation Video Generation with Human-Friendly Motion Control

    Oct 18, 2026Yulin Dai, Yuqian Fu, Tianwen Qian +3Controllable Video GenerationCamera-Controlled Video Generation

  9. Meta Inverse Prompting for Video Generative Models

    Oct 18, 2026Yerin Jung, Jongheon JeongControllable Video Generation

  10. SpriteStage: Long Horizon Animation Generation via Progressive Scene Programming and Replay

    Oct 17, 2026Zhaoqi Wang, Zijian Zhang, Jiamou LiuLong-Horizon Video GenerationControllable Video Generation

  11. Video Diffusion Avatars: Taming Video Diffusion Models into Controllable 3D Avatars

    Oct 16, 2026Simon Giebenhain, Tobias Kirschstein, Liam Schoneveld +33D Avatar GenerationHuman Avatar Animation

  12. Temporal Specification Guidance for Text-to-Video Generation

    Oct 14, 2026Avishree Khare, Hideki Okamoto, Bardh Hoxha +2T2V GenerationControllable Video Generation

  13. SonoWorld: Towards Interactive Pose-Controllable Ultrasound World Models

    Oct 14, 2026Zefan Yang, Zheyuan Zhang, Boris Maihe +7World ModelsAction-Conditioned World Models

  14. EmoTrace-Bench: Diagnosing Emotional Expression in Text-to-Video Generation through Controlled Cues and Intensity

    Oct 11, 2026Yeheng Wang, Liudan Shi, Yi Zheng +5T2V GenerationVideo Generation Evaluation

  15. CatPortrait: Taming Video Generative Models for Portrait Animation via In-context Conditioning

    Oct 10, 2026Ziqi Ni, Zhou Fang, Shijian Wang +2Controllable Video GenerationVideo Generation

  16. Hybrid Cinematography: Previsualizing and Managing Hallucination Risk in Generative Video Reshooting

    Oct 8, 2026Nhan Tran, Neal Wadhwa, Abe Davis +1Controllable Video Generation

  17. VIS-Ground: Video Interactive Storytelling with Contextual Grounding

    Oct 7, 2026Bingxuan Li, Yiwen Song, Xueqing Wu +10Video GenerationInteractive Storytelling

  18. RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer

    Oct 7, 2026Huang Huang, Wensi Ai, Ziyu Chen +6Sim-to-Real TransferSynthetic Data Generation for Robotics

  19. Diverse Motion Customization via Control-based Dynamic Optimization

    Oct 6, 2026Youngyoon Choi, Kihyun Kim, Jeongwoo Shin +1Video Diffusion ModelsControllable Video Generation

  20. Kepler4D: Controllable Future Video Generation via 4D Scene State Evolution

    Oct 2, 2026Feiran Wang, Bin Duan, Junyi Wu +2Video GenerationControllable Video Generation

  21. ProAR: Learning Prospective Reasoning with Autoregressive Video Models

    Oct 2, 2026Linghui Shen, Tinghui Zhu, Sheng Zhang +1Video ReasoningAutoregressive Video Generation

  22. VDOT++: Unified Few-Step Video Generation via Unbalanced Optimal Transport Distillation

    Oct 2, 2026Yutong Wang, Xingtong Ge, Enhuai Liu +6Diffusion Model DistillationFew-Step Diffusion Sampling

  23. 4Director: Controlling Video World Models with Rigid 3D Geometry

    Oct 1, 2026Wei Cao, Hao Zhang, Vikram Voleti +5Camera-Conditioned Video GenerationCamera-Controlled Video Generation