Controllable Video Generation

Overview

The field studies how to make generated videos follow user intent, from preserving a subject or style to specifying poses, object trajectories, camera moves, events, and physical properties. Common approaches condition diffusion models with text, images, tracks, masks, or 3D geometry, adapt pretrained models with ControlNet or LoRA modules, or steer sampling at inference; newer world models add explicit state, planning, and memory for interactive, long-horizon generation. Evaluation spans prompt and control adherence, identity and scene consistency, motion accuracy, physical plausibility, and video quality, including tests on driving, human-object interaction, and cinematic scenes. A central tension remains: stronger or denser controls can improve precision but may reduce naturalness or visual quality, while long rollouts can drift or fail to realize intended state changes even when individual frames look convincing.

Latest papers

CardsList

341 papers34 papers in the last four weeks, down 26% on the four weeks before. 0.3% of all new papers.

  1. TokenStream: Multi-Subject Personalized Video Generation with Token-wise LoRA and Guided Attention

    Nov 12, 2026Hung Nguyen, Quang Qui-Vinh Nguyen, Khoi NguyenIdentity-Preserving Video GenerationSubject-Driven Video Generation

  2. PhysLoop: Harnessing LLMs' Physics Awareness for Self-Improving Video Generation Models

    Nov 12, 2026Yuxuan Chen, Xiaojie Li, Hao Chen +11Video GenerationPhysical Consistency in Video Generation

  3. CoCode-World: Shared 3D Proxy Representation for Multi-Player World Models

    Nov 11, 2026Qirui Li, Xin Zhang, Yabo Chen +7Video World ModelsMulti-View Video Generation

  4. PhysOPSD: Code as Privilege for Physically Plausible Video Generation

    Nov 9, 2026Harold Haodong Chen, Rui Peng, Lingteng Qiu +3Physical Consistency in Video GenerationControllable Video Generation

  5. Generator-Aligned Reward Modeling with Target-Aware Bounded Optimization for Visual Generation

    Nov 5, 2026Ziqiao Weng, Yuanzhi Liang, Haibin Huang +2Reward ModelingControllable Video Generation

  6. Designing with Cinematic Language: An Intent-Driven Approach to Human-AI Video Clip Co-Creation

    Nov 5, 2026Yi He, Zhengxi Yu, Feilin Han +3Human-AI Co-CreationControllable Video Generation

  7. Householder Motion Embedding: Noise Modulation for Motion-Controllable Video Generation

    Nov 5, 2026Tao Jun Lin, Jinhong Ni, Ruyi Zha +4Controllable Video GenerationVideo Diffusion Models

  8. ReactWorld: Structuring Interaction Responses for Embodied Video Generation

    Nov 4, 2026Hanwei Liu, Huijia Zhang, Hanzhang Wang +6Embodied World ModelsAction-Conditioned Video Generation

  9. TouchéWorld tech report : Towards Online Unified Camera--Hand Exploration in Generative Worlds

    Nov 4, 2026Jiaao Yu, Yanjian Zhang, Haoyang Jiang +3Controllable Video GenerationReal-Time Interactive Video Generation

  10. Chronos: Sequential Identity-Preserving Video Generation by Temporal Anchoring

    Nov 4, 2026Mengqi Huang, Wenchuan Wang, Zhendong Mao +8Sparse Keyframe-Conditioned Video GenerationIdentity-Preserving Video Generation

  11. DualCast: Dual-Image Conditioned Diffusion Transformers for Digital Human E-Commerce Video Generation

    Nov 4, 2026Lifan Jiang, Yibo Zhao, Senlin Cheng +6Video GenerationImage-to-Video Generation

  12. IMPACT: Image-Conditioned Motion with Physics-Aware Collision and Trajectory Guidance for Video Generation

    Nov 1, 2026Shiwei Yang, Yunlong Yu, Zhao Wang +1Physical Consistency in Video GenerationImage-to-Video Generation

  13. UniCaMo: Unified Camera and Motion Control for Video Diffusion Models

    Oct 31, 2026Long Vu, Tan Ngo, Animesh Karnewar +5Controllable Video GenerationCamera-Controlled Video Generation

  14. Fight World Model: Customizable Combat through Visual Interaction Proxies

    Oct 29, 2026Jiahua Li, Zixu John, Tom Zhong +5Game World ModelsInteractive World Models

  15. Regional Anchor Routing for Multi-Object Motion Transfer

    Oct 29, 2026Yuze Li, Fuchen Long, Xinlong Sun +3Controllable Video GenerationVideo Diffusion Models

  16. Edited Endpoints as Partial Goals for Image-to-Video Generation

    Oct 29, 2026Sihyun Kim, Sanghyun Jo, Kyungsu KimImage-to-Video GenerationControllable Video Generation

  17. ReferStream: Interactive Subject Injection for Streaming Video Generation

    Oct 27, 2026Shuolin Xu, Guanyuan Li, Boyu Li +5Subject-Driven Video GenerationStreaming Video Generation

  18. Hy-Nomas: Towards Free-form Interactive World Generation

    Oct 27, 2026Qiaoling Chen, Jiaxiang Cheng, Xueqing Deng +35Interactive World ModelsVideo Generation

  19. BoxWorld: Let World Engines Control and Video Generators Render

    Oct 26, 2026Zike Wu, Ahmed Hatem, Chintan Trivedi +4Real-Time Interactive Video GenerationInteractive World Models

  20. UniWorld: A Unified World Model for Real-Time Visual World Simulation

    Oct 23, 2026Xin Zhang, Yabo Chen, Shuaiwei Wang +4Video World ModelsVideo Generation

  21. Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

    Oct 22, 2026Songlin Yang, Zhe Wang, Xiaotong Zhao +8Multimodal Video GenerationVideo Generation

  22. Latent Space Jam: Layout Guided Video Synthesis

    Oct 21, 2026Asaf Joseph, Moshe Mandel, Ohad FriedControllable Video GenerationTrajectory-Guided Video Generation

  23. JointLight: Human-Centric Video Relighting with Asynchronous RGB-Irradiance Joint Modeling

    Oct 21, 2026Henglei Lv, Bailin Deng, Xiaoqiang Liu +4Video RelightingVideo Diffusion Models

  24. Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs

    Oct 20, 2026Noam Rotstein, Yoad Tewel, Chen Tessler +53D-Aware Video GenerationControllable Video Generation

  25. SolarRenderer: Few-Step Generative Reference Rendering

    Oct 20, 2026Guian Fang, Shengju Qian, Dongyue Lu +6Controllable Video GenerationVideo Diffusion Models