3D Scene Prediction

Latest papers

20 papers8 papers in the last four weeks, with none the four weeks before. 0.1% of all new papers.

Hybrid Particle-Voxel Dynamics for Multimodal 3D World Simulation
Nov 3, 2026Jimmy Wang, Yifan Chou, Kaifeng Zhang +5World Models for Robotics 3D Scene Prediction
Columbia University · UC Berkeley · University of Michigan +2
WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
Oct 29, 2026Yuhao Lu, Jiongxu Chen, Hexiang Wei +9Video World Models Multimodal Trajectory Prediction
RoboScience · National University of Singapore
E0.5: From Reconstructive to Future-Predictive 3D Foundations for Robotic World-Action Modeling
Oct 26, 2026X-Era AI Lab Team, :, Jianhao Lai +25World Models for Robotics Robot Foundation Models
DeXplicit: Making Physical State Explicit in Dexterous Manipulation World Models
Oct 22, 2026Adam Lee, Mutian Xu, Yukang Cao +2World Models for Robotics 3D-Aware Video Generation
S-Lab, Nanyang Technological University · University of California, Berkeley
Learning the 4D World through Point Trajectory Forecasting
Oct 20, 2026Byungwoo Jeon, Sunghwan Hong, Dohyeon Kim +103D Scene Prediction Multi-View Video Generation
KAIST · ETH Zürich · Seoul National University +3
From Predictive Geometry to Action: Learning Complementary Predictive Representations for Embodied Manipulation
Oct 20, 2026Xiaoxu Xu, Hao Li, Hongtao Zhang +6Robotic Manipulation Predictive Representation Learning
Beihang University · Shanghai Artificial Intelligence Laboratory
Persistent Planar Memory for Video World Models
Oct 16, 2026Yuze He, Bin Tan, Zelin Gao +4Video World Models Memory-Augmented Video Generation
Tsinghua University · Ant Group · HKUST
GeoState: Learning View-Decoupled Geometry States for 3D World Modeling
Oct 15, 2026Hongchang Liu, Jian Liu, Yujuan Ding +5World Model Learning 3D Scene Prediction
Tongji University · Hong Kong Polytechnic University · Tencent +1
VGGTWorld-VLA: Intent-Conditioned 3D World Evolution for Autonomous Driving
Oct 8, 2026Zhaoyang Liu, Kun Jiang, Ziying Song +13D Scene Prediction Action-Conditioned World Models
Tsinghua University · Nanyang Technological University
GeoWAM: Visual Geometry World Action Models for Autonomous Driving
Aug 24, 2026Yiren Lu, Xin Ye, Jiaming Liu +103D Scene Prediction Autonomous Driving
Uber AV Labs · Case Western Reserve University
SR-JEPA: Learning Predictive Latent State in 3D Scenes
Aug 6, 2026Zihan Zhou, Qifu Wen, Xi ZengJoint-Embedding Predictive Architecture Object-Centric Representation Learning
Boston University, Boston, MA, USA · Shanghai Jiao Tong University, Shanghai, China
MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction
Jul 17, 2026Homanga Bharadhwaj, Yash JangirScene Flow Estimation Trajectory Prediction
Department of Computer Science, Johns Hopkins University
Perceptual 3D Simulation With Physical World Modeling
Jun 25, 2026Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh +23D Scene Prediction World Model Learning
1Stanford University
Future Dynamic 3D Reconstruction: A 3D World Model with Disentangled Ego-Motion
Jun 16, 2026Nils Morbitzer, Jonathan Evers, Artem Savkin +4World Model Learning Monocular 3D Reconstruction
Technical University of Munich (TUM) · Munich Center for Machine Learning (MCML) · BMW Group +1
Coarse Semantic Injection for LLM-Conditioned Structured Indoor Prediction
May 16, 2026Shuliang Zhu, Tomiwa Adey, Jinjia Zhou3D Scene Prediction Point Cloud Learning
Forecast-aware Gaussian Splatting for Predictive 3D Representation in Language-Guided Pick-and-Place Manipulation
May 11, 2026Kaixin Jia, Jiacheng Xu3D Gaussian Splatting 3D Scene Prediction
KTH Royal Institute of Technology, Stockholm, Sweden.
VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model
Mar 13, 2026Xiangyu Sun, Shijie Wang, Fengyi Zhang +53D Scene Prediction World Model Learning
UQMM Lab, The University of Queensland · Beijing Jiaotong University
StemVLA:An Open-Source Vision-Language-Action Model with Future 3D Spatial Geometry Knowledge and 4D Historical Representation
Feb 27, 2026Jiasong Xiao, Yutao She, Kai Li +23D Scene Prediction Long-Horizon Robotic Manipulation
Image Processing Ricoh Software Research Center (Beijing) Co., Ltd. Beijing, China · Guanghua School of Management Peking University Beijing, China · Image Processing Beijing YZH Engineering Technology Co., Ltd Beijing, China +2
Schrödinger's Navigator: Imagining an Ensemble of Futures for Zero-Shot Object Navigation
Dec 24, 2025Yu He, Da Huang, Zhenyang Liu +53D Scene Prediction Robot Navigation
Fudan University · Shanghai Innovation Institute · ActiMind +3
Aligned Delta-Triplane Transformers as Occupancy World Models
Mar 10, 2025Haoran Xu, Peixi Peng, Shijun Guo +53D Scene Prediction 3D Occupancy Prediction
School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China. · Peng Cheng Laboratory, Shenzhen 518108, China. · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, 518066, China. +2

Nov 3, 2026J/K move · Enter open · S save

Hybrid Particle-Voxel Dynamics for Multimodal 3D World Simulation

Jimmy Wang, Yifan Chou, Kaifeng Zhang, Yixuan Wang+4

Columbia University · UC Berkeley · University of Michigan+2

World Models for Robotics 3D Scene Prediction 3D Scene Representation Tactile World Models Synthetic Data Generation for Robotics

Open paper AskPDF arXiv

Predicting how the world evolves in response to robot actions is central to policy learning, evaluation, and planning. However, many action-conditioned world models predict future observations in 2D image or video space without explicitly modeling the underlying 3D scene, making spatial consistency across viewpoints and over long-horizon interactions challenging. We introduce 3D World Simulator, an action-conditioned multimodal world modeling framework that jointly predicts appearance, geometry, and tactile observations from a unified 3D representation. Our approach combines a hybrid particle-voxel autoencoder that compresses point clouds into compact 3D voxel latents with an action-conditioned latent dynamics model that predicts their evolution. The shared 3D representation supports geometrically consistent rendering from arbitrary viewpoints and flexible integration of spatially aligned tactile signals. Our experiments demonstrate interactive, long-horizon, multimodal prediction at 50 Hz. We then show how our model enables scalable data generation for policy training and evaluation using tactile and multi-view visual observations, including wrist-mounted and novel camera views. Furthermore, we scale our method to large datasets containing diverse objects and demonstrate model-based planning with unseen objects and robot configurations. Together, these results highlight the potential of hybrid particle-voxel dynamics as a foundation for multimodal 3D world simulation.