Synthetic Data Generation for Robotics

Follow this topic Ask about this topic

Momentum

40 papers in the last four weeks, up 567% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 196

Feb 12, 2026cs.RO

ReaDy-Go: Real-to-Sim Dynamic 3D Gaussian Splatting Simulation for Environment-Specific Visual Navigation with Moving Obstacles

Visual navigation models often struggle in real-world dynamic environments due to limited robustness to the sim-to-real gap and the difficulty of training policies tailored to target deployment environments (e.g., households, restaurants, and factories). Although real-to-sim navigation simulation using 3D Gaussian Splatting (GS) can mitigate these challenges, prior GS-based works have considered only static scenes or non-photorealistic human obstacles built from simulator assets, despite the importance of safe navigation in dynamic environments. To address these issues, we propose ReaDy-Go, a novel real-to-sim simulation pipeline that synthesizes photorealistic dynamic scenarios in target environments by augmenting a reconstructed static GS scene with dynamic human GS obstacles, and trains navigation policies using the generated datasets. The pipeline provides three key contributions: (1) a dynamic GS simulator that integrates static scene GS with a human animation module, enabling the insertion of animatable human GS avatars and the synthesis of plausible human motions from 2D trajectories, (2) a navigation dataset generation framework that leverages the simulator along with a robot expert planner designed for dynamic GS representations and a human planner, and (3) robust navigation policies to both the sim-to-real gap and moving obstacles. The proposed simulator generates thousands of photorealistic navigation scenarios with animatable human GS avatars from arbitrary viewpoints. ReaDy-Go outperforms baselines across target environments in both simulation and real-world experiments, demonstrating improved navigation performance even after sim-to-real transfer and in the presence of moving obstacles. Moreover, zero-shot sim-to-real deployment in an unseen environment indicates its generalization potential. Project page: https://syeon-yoo.github.io/ready-go-site/.

Seungyeon Yoo, Youngseok Jang, Dabin Kim +3

Department of Aerospace Engineering, Seoul National University, Seoul, Republic of Korea · InnoCORE AI-Transformed Aerospace Research Center, KAIST, Daejeon, Republic of Korea

Robotics Simulation Robot Navigation Synthetic Data Generation for Robotics Robot Policy Generalization Dynamic Obstacle Avoidance

Feb 5, 2026cs.RO

MobileManiBench: Simplifying Model Verification for Mobile Manipulation

Vision-language-action models have advanced robotic manipulation but remain constrained by reliance on the large, teleoperation-collected datasets dominated by the static, tabletop scenes. We propose a simulation-first framework to verify VLA architectures before real-world deployment and introduce MobileManiBench, a large-scale benchmark for mobile-based robotic manipulation. Built on NVIDIA Isaac Sim and powered by reinforcement learning, our pipeline autonomously generates diverse manipulation trajectories with rich annotations (language instructions, multi-view RGB-depth-segmentation images, synchronized object/robot states and actions). MobileManiBench features 2 mobile platforms (parallel-gripper and dexterous-hand robots), 2 synchronized cameras (head and right wrist), 630 objects in 20 categories, 5 skills (open, close, pull, push, pick) with over 100 tasks performed in 100 realistic scenes, yielding 300K trajectories. This design enables controlled, scalable studies of robot embodiments, sensing modalities, and policy architectures, accelerating research on data efficiency and generalization. We benchmark representative VLA models and report insights into perception, reasoning, and control in complex simulated environments, with all code, datasets, and models publicly released.

Wenbo Wang, Fangyun Wei, QiXiu Li +5

University of Sydney · 1Microsoft Research Asia · 3Tsinghua University

Mobile Manipulation Robotics Simulation Synthetic Data Generation for Robotics Vision-Language-Action Models Robot Manipulation Benchmarks

Jan 12, 2026eess.SY

Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directions

Video world models have emerged as promising candidates for high-fidelity world models, offering the potential to synthesize high-quality videos capturing fine-grained interactions between agents and their environments conditioned on multi-modal user inputs. Their impressive capabilities address many of the long-standing challenges faced by physics-based simulators, driving broad adoption in many problem domains, e.g., robotics. For example, video models can generate photorealistic, physically consistent deformable-body simulation without making prohibitive simplifying assumptions, which is a major bottleneck in physics-based simulation. Moreover, video models can serve as foundation world models that capture the dynamics of the world in a fine-grained and expressive way. They thus overcome the limited expressiveness of language-only abstractions in describing intricate physical interactions. However, despite their potential, video models still generate physics-violating future predictions, often manifesting as hallucinations. In this survey, we provide a review of video models and their applications as embodied world models in robotics, including efficient data generation and policy learning, dynamics and rewards modeling in reinforcement learning, policy evaluation, and visual planning. Further, we highlight important challenges hindering the trustworthy integration of video models, such as poor instruction following, hallucinations like violations of physics, unsafe content generation, in addition to significant data and compute overhead. We present potential future directions to address these open research challenges to motivate research and ultimately facilitate broader applications, especially in safety-critical settings. We provide a curated bibliography at https://github.com/irom-princeton/awesome-robotics-video-world-model-papers .

Zhiting Mei, Tenny Yin, Ola Shorinwa +9

Princeton University · Temple University

Physical Consistency in Video Generation World Models for Robotics Synthetic Data Generation for Robotics Embodied World Models Video World Models

Jan 5, 2026cs.RO

Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot

The development of robust and generalizable robot learning models is critically contingent upon the availability of large-scale, diverse training data and reliable evaluation benchmarks. Collecting data in the physical world poses prohibitive costs and scalability challenges, and prevailing simulation benchmarks frequently suffer from fragmentation, narrow scope, or insufficient fidelity to enable effective sim-to-real transfer. To address these challenges, we introduce Genie Sim 3.0, a unified simulation platform for robotic manipulation. We present Genie Sim Generator, a large language model (LLM)-powered tool that constructs high-fidelity scenes from natural language instructions. Its principal strength resides in rapid and multi-dimensional generalization, facilitating the synthesis of diverse environments to support scalable data collection and robust policy evaluation. We introduce the first benchmark that pioneers the application of LLM for automated evaluation. It leverages LLM to mass-generate evaluation scenarios and employs Vision-Language Model (VLM) to establish an automated assessment pipeline. We also release an open-source dataset comprising more than 10,000 hours of synthetic data across over 200 tasks. Through systematic experimentation, we validate the robust zero-shot sim-to-real transfer capability of our open-source dataset, demonstrating that synthetic data can server as an effective substitute for real-world data under controlled conditions for scalable policy training. For code and dataset details, please refer to: https://github.com/AgibotTech/genie_sim.

Chenghao Yin, Da Huang, Di Yang +16

Robotic Data Collection Robotics Simulation Synthetic Data Generation for Robotics Synthetic Benchmark Generation Robot Manipulation Benchmarks

Dec 12, 2025cs.RO

AnchorDream: Repurposing Video Diffusion for Embodiment-Aware Robot Data Synthesis

The collection of large-scale and diverse robot demonstrations remains a major bottleneck for imitation learning, as real-world data acquisition is costly and simulators offer limited diversity and fidelity with pronounced sim-to-real gaps. While generative models present an attractive solution, existing methods often alter only visual appearances without creating new behaviors, or suffer from embodiment inconsistencies that yield implausible motions. To address these limitations, we introduce AnchorDream, an embodiment-aware world model that repurposes pretrained video diffusion models for robot data synthesis. AnchorDream conditions the diffusion process on robot motion renderings, anchoring the embodiment to prevent hallucination while synthesizing objects and environments consistent with the robot's kinematics. Starting from only a handful of human teleoperation demonstrations, our method scales them into large, diverse, high-quality datasets without requiring explicit environment modeling. Experiments show that the generated data leads to consistent improvements in downstream policy learning, with relative gains of 36.4% in simulator benchmarks and nearly double performance in real-world studies. These results suggest that grounding generative world models in robot motion provides a practical path toward scaling imitation learning.

Junjie Ye, Rong Xue, Basile Van Hoorick +4

Toyota Research Institute · USC Physical Superintelligence (PSI) Lab

Video Diffusion Models Synthetic Data Generation for Robotics Robot Imitation Learning Embodied World Models

Dec 7, 2025cs.RO

MIND-V: Hierarchical World Model for Long-Horizon Robotic Manipulation with RL-based Physical Alignment

Scalable embodied intelligence is constrained by the scarcity of diverse, long-horizon robotic manipulation data. Existing video world models in this domain are limited to synthesizing short clips of simple actions and often rely on manually defined trajectories. To this end, we introduce MIND-V, a cognitive hierarchical world model designed to synthesize physically plausible and logically coherent videos of long-horizon robotic manipulation. Inspired by cognitive science, MIND-V bridges high-level reasoning with pixel-level synthesis through three core components: a Semantic Reasoning Hub (SRH) that leverages a pre-trained vision-language model for task planning; a Behavioral Semantic Bridge (BSB) that translates abstract instructions into domain-invariant representations; and a Motor Video Generator (MVG) for conditional video rendering. MIND-V employs Staged Visual Future Rollouts, a test-time optimization strategy to enhance long-horizon robustness. To enforce adherence to physical laws, we introduce a GRPO reinforcement learning post-training phase guided by a novel Physical Foresight Coherence (PFC) reward. PFC leverages the V-JEPA2 world model as a physics referee to penalize implausible dynamics in the latent feature space. Experiments confirm MIND-V's SOTA performance in long-horizon simulation and its significant value for policy learning, introducing a scalable and fully autonomous framework for embodied data synthesis.

Ruicheng Zhang, Mingyang Zhang, Jun Zhou +6

Tsinghua University · Sun Yat-sen University · X Square Robot +1

Physical Consistency in Video Generation Long-Horizon Robotic Manipulation RL for Video Generation Synthetic Data Generation for Robotics RL for Generative Models Embodied World Models

Dec 1, 2025cs.CV

TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulation

Simulation provides a low-cost, scalable pathway to large-scale robotic manipulation data collection. However, existing 3D scene generation methods can rarely be applied directly to manipulation data synthesis, as their generated scenes often lack instance-level interactivity and physical plausibility. Focusing on tabletop manipulation, we propose TabletopGen, a training-free and automated tabletop scene generation and interactive simulation engine. Starting from text or a single image, we first obtain independent 3D object models via generative instance extraction. Second, we introduce a novel pose and scale alignment approach that recovers a collision-free scene layout using a Differentiable Rotation Optimizer and a Top-View Spatial Alignment mechanism. Finally, we assemble the generated scene in a physics simulator with collision geometry, yielding a stable, interactable environment for synthesizing multimodal manipulation data. Extensive experiments and user studies demonstrate that TabletopGen achieves state-of-the-art performance in visual fidelity, layout accuracy, and physical plausibility. Furthermore, we validate the executability of the collected trajectories on a real robotic arm via zero-shot real-to-sim-to-real policy transfer, indicating that TabletopGen can serve as a reliable data engine for robotic manipulation data synthesis.

Ziqian Wang, Yonghao He, Licheng Yang +6

School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences, Beijing, China · D-Robotics, Beijing, China +2

Robotic Data Collection Robotics Simulation Synthetic Data Generation for Robotics 3D Scene Generation Robotic Manipulation 3D Scene Layout Generation

Oct 2, 2025cs.RO

FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models

Recent advances in robotic manipulation have integrated low-level robotic control into Vision-Language Models (VLMs), extending them into Vision-Language-Action (VLA) models. Although state-of-the-art VLAs achieve strong performance in downstream robotic applications, supported by large-scale crowd-sourced robot training data, they still inevitably encounter failures during execution. Enabling robots to reason and recover from unpredictable and abrupt failures remains a critical challenge. Existing robotic manipulation datasets, collected in either simulation or the real world, primarily provide only ground-truth trajectories, leaving robots unable to recover once failures occur. Moreover, the few datasets that address failure detection typically offer only textual explanations, which are difficult to utilize directly in VLA models. To address this gap, we introduce FailSafe, a novel failure generation and recovery system that automatically produces diverse failure cases paired with executable recovery actions. FailSafe can be easily adapted to a wide range of manipulation tasks in simulators with motion planning support, enabling scalable creation of failure-action data. To demonstrate its effectiveness, we fine-tune LLaVA-OneVision-7B (LLaVA-OV-7B) to build FailSafe-VLM. Experimental results show that FailSafe-VLM successfully helps robotic arms detect and recover from potential failures, improving the performance of three state-of-the-art VLA models (Pi-0-FAST, OpenVLA, OpenVLA-OFT) by up to 22.6% on average across several tasks in ManiSkill. Furthermore, FailSafe-VLM could generalize across different spatial configurations, camera viewpoints, object and robotic embodiments.

Zijun Lin, Jiafei Duan, Haoquan Fang +4

1Nanyang Technological University · Centre for Frontier AI Research, A*STAR · 3Allen Institute for AI +1

Robot Failure Recovery Synthetic Data Generation for Robotics Robotics Vision-Language-Action Models Robustness of VLA Models

Sep 30, 2025cs.RO

Learning from Hallucinating Critical Points for Navigation in Dynamic Environments

Generating large and diverse obstacle datasets to learn motion planning in environments with dynamic obstacles is challenging due to the vast space of possible obstacle trajectories. Inspired by hallucination-based data synthesis approaches, we propose Learning from Hallucinating Critical Points (LfH-CP), a self-supervised framework for creating rich dynamic obstacle datasets based on existing optimal motion plans without requiring expensive expert demonstrations or trial-and-error exploration. LfH-CP factorizes hallucination into two stages: first identifying when and where obstacles must appear in order to result in a near-optimal motion plan, i.e., the critical points, and then procedurally generating diverse trajectories that pass through these points while avoiding collisions. This factorization avoids generative failures such as mode collapse and ensures coverage of diverse dynamic behaviors. We further introduce a diversity metric to quantify dataset richness and show that LfH-CP produces substantially more varied training data than existing baseline. Experiments in simulation demonstrate that planners trained on a LfH-CP generated dataset achieves higher success rates compared to a prior hallucination method.

Saad Abdul Ghani, Kameron Lee, Xuesu Xiao

George Mason University · Thomas Jefferson High school intern affiliated with the RobotiXX Lab, George Mason University

Trajectory Generation Synthetic Data Generation for Robotics Motion Planning Dynamic Obstacle Avoidance

Sep 25, 2025cs.RO

Self-CriTeach: LLM Self-Teaching and Self-Critiquing for Improving Robotic Planning via Automated Domain Generation

Large Language Models (LLMs) have recently shown strong promise for robotic task planning, particularly through automatic planning domain generation. However, prior approaches largely treat generated planning domains as planning utilities, which are brittle under imperfect logical states and perception noise, overlooking their potential as scalable sources of reasoning supervision and structured reward signals. At the same time, reasoning LLMs depend on chain-of-thought (CoT) supervision that is expensive to collect for robotic tasks, and reinforcement learning (RL) faces challenges in reward engineering. We propose Self-CriTeach, an LLM self-teaching and self-critiquing framework in which an LLM autonomously generates symbolic planning domains that serve a dual role: (1) enabling large-scale generation of robotic planning problem-plan pairs, and (2) providing structured reward functions. First, the self-written domains enable large-scale generation of symbolic task plans, which are automatically transformed into extended CoT trajectories for supervised fine-tuning. Second, the self-written domains are reused as structured reward functions, providing dense feedback for reinforcement learning without manual reward engineering. This unified training pipeline yields a planning-enhanced LLM with higher planning success rates, stronger cross-task generalization, reduced inference cost, and resistance to imperfect logical states. GitHub Page: https://markli1hoshipu.github.io/Plan_LLM/

Jinbang Huang, Zhiyuan Li, Yuanzhao Hu +4

Huawei Noah’s Ark Lab · University of Toronto · University of British Columbia +1

Large Language Model-Based Robot Planning Synthetic Data Generation for Robotics LLM-Guided RL Robot Task Planning

Sep 18, 2025cs.RO

A Scalable Whole-body Motion Transfer via Implicit Kinodynamic Motion Retargeting

Human-to-humanoid imitation learning presents a promising pathway to address the severe data scarcity bottleneck in robotics by utilizing abundant, large-scale human motion collections. However, scaling this paradigm requires addressing two key challenges. First, human motion data acquired from videos, motion capture systems, or generative models often contains spatial noise, jitter, and frame-level flickering, which can be amplified during retargeting and lead to unsafe or physically infeasible robot motions. Second, existing motion retargeting methods typically rely on frame-by-frame numerical optimization, making them too computationally expensive for large-scale dataset synthesis. To overcome these limitations, we introduce Implicit Kinodynamic Motion Retargeting (IKMR), a highly scalable, neural-based data transformation pipeline. IKMR leverages a skeleton-based graph convolutional dual autoencoder to map cross-structural human and humanoid kinematic configurations into a shared topological latent space. To guarantee the physical viability of the generated data, the framework incorporates a physics-informed refinement phase that utilizes simulated physical tracking feedback to learn a robust motion prior. This implicit formulation fundamentally resolves both challenges. By shifting the computational burden from online optimization to offline inference, IKMR achieves an unprecedented data conversion throughput exceeding 5000 frames per second. Furthermore, leveraging the learned motion prior, it functions as an intrinsic data curation mechanism and naturally filters out high-frequency noise and spatial jitters from source data, yielding smooth trajectories that ensure physical hardware safety. Extensive evaluations, including real-world whole-body control deployments on humanoid robot, confirm that IKMR bridges the gap between human motion and robotic data.

Xingyu Chen, Hanyu Wu, Sikai Wu +7

1HKUST(GZ) · 4Xiaomi Robotics Lab · 2ETH Zurich +1

Humanoid Robot Control Synthetic Data Generation for Robotics Motion Retargeting Robot Imitation Learning

Jun 13, 2025cs.CV

Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale

Affordance grounding aims to localize where to interact with an object, a fundamental capability for embodied agents. Yet progress is bottlenecked by data: manual annotation is prohibitively expensive and confines existing datasets to a narrow set of predefined object and affordance categories. We introduce Affogato, a framework for open-vocabulary affordance grounding centered on Affogato-750K, a large-scale dataset of 750K 3D affordance heatmaps paired with natural language queries. We build it with a fully automated pipeline that orchestrates foundation models to generate them at scale without human labeling. It covers significantly more diverse categories than any existing dataset. For reliable evaluation, we further provide 5K human-verified test pairs. We also present Espresso-3D and Espresso-2D, simple yet effective models with a unified architecture across both modalities. Pretraining on Affogato-750K improves both Espresso and prior methods and yields the largest gains on unseen object and affordance categories, showing that it provides broadly transferable supervision across architectures.

Junha Lee, Eunha Park, Chunghyun Park +2

Pohang University of Science and Engineering (POSTECH) · SqueezeBits · RLWLRD

Synthetic Data Generation for Robotics Embodied AI 3D Visual Grounding

May 26, 2025cs.CV

ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation

Vision-based tactile sensing is widely used in perception, reconstruction, and robotic manipulation, yet collecting large-scale tactile data remains costly due to diverse sensor-object interactions and inconsistencies across sensor instances. Existing approaches to scaling tactile data---simulation and free-form tactile generation---often yield unrealistically rendered signals with poor transfer to highly dynamic real-world tasks. We propose \name, a two-stage controllable tactile image generation framework that generates realistic tactile images conditioned on a single reference tactile image, contact force, and contact pose. By grounding generation in these important physical priors, \name synthesizes realistic samples across different sensors while effectively capturing task-relevant variations. Across a series of downstream tasks and real-world experiments, such as object insertion, imitation learning, and object weighting, the augmented datasets using our approach consistently improve performance and demonstrate practical utility in dynamic real-world settings. Project page: https://dongyuluo.github.io/controltac.

Dongyu Luo, Kelin Yu, Amir-Hossein Shahidzadeh +3

The University of Hong Kong · University of Maryland · University of Maryland, College Park

Vision-Based Tactile Sensing Synthetic Data Generation for Robotics Tactile Sensing for Robotic Manipulation Controllable Image Generation

Mar 11, 2025cs.CV

Simulating Automotive Radar with Lidar and Camera Inputs

Low-cost millimeter automotive radar has received more and more attention due to its ability to handle adverse weather and lighting conditions in autonomous driving. However, the lack of quality datasets hinders research and development. We report a new method that is able to simulate 4D millimeter wave radar signals including pitch, yaw, range, and Doppler velocity along with radar signal strength (RSS) using camera image, light detection and ranging (lidar) point cloud, and ego-velocity. The method is based on two new neural networks: 1) DIS-Net, which estimates the spatial distribution and number of radar signals, and 2) RSS-Net, which predicts the RSS of the signal based on appearance and geometric information. We have implemented and tested our method using open datasets from 3 different models of commercial automotive radar. The experimental results show that our method can successfully generate high-fidelity radar signals. Moreover, we have trained a popular object detection neural network with data augmented by our synthesized radar. The network outperforms the counterpart trained only on raw radar data, a promising result to facilitate future radar-based research and development.

Peili Song, Dezhen Song, Yifan Yang +2

Institute of Robotics and Automatic Information System, Nankai University, Tianjin 300350, China; Tianjin Key Laboratory of Intelligent Robotics, Tianjin 300350, China; and also with TBI center, Nankai University, Tianjin 300350, China · Department of Robotics, Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates

Synthetic Data Augmentation Millimeter-Wave Radar mmWave Radar Sensing Autonomous Driving Perception Synthetic Data Generation for Robotics Autonomous Driving Simulation

Date pendingcs.CV

EVIS: Real-Time Event Camera Simulation with Multimodal Supervision in NVIDIA Isaac Sim

Event cameras are increasingly adopted in embodied perception for their microsecond temporal resolution, high dynamic range, and resilience to motion blur. However, training event-based models for robotics requires large-scale, action-conditioned data with dense physical annotations that are difficult to collect in the real world. We introduce EVIS, an open-source physics-grounded event simulator integrated into NVIDIA Isaac Sim that generates events from linear-HDR radiance from a closed-loop robot training episode. Rather than relying on learning-based video interpolation or expensive dense rendering, EVIS exploits renderer-provided motion vectors and depth maps through bi-directional warping and depth-based splatting. This enables high throughput, real-time, and high-fidelity event generation. We evaluate EVIS across runtime efficiency, sim-to-real transfer, and zero-shot model compatibility. A rotation-speed estimator trained solely on EVIS events achieves 2.75 rad/s MAE on real sensor data. Pretrained models for reconstruction, matching, and tracking perform competitively on EVIS events without any fine-tuning. EVIS sustains real-time generation across GPU-parallel environments on a single GPU. Code repository: https://github.com/spikelab-jhu/isaac-sim-event-camera-plugin.

Linli Shi, Ruijun Zhang, Ziyun Wang

Johns Hopkins University

Event-Based Vision Robotics Simulation Synthetic Data Generation for Robotics

Date pendingcs.RO

GIF: Agentic Generation of Interactive and Functional Object Compositions for Robot Learning

Robot manipulation foundation models require scalable evaluation and data generation across diverse scenarios, with simulation providing an environment for both. Automated scene generation offers a promising path, yet prior work has largely emphasized coarse-grained scene layouts rather than fine-grained functional object compositions. Motivated by this gap, we present GIF, an agentic Generation framework for Interactive and Functional object compositions. In this framework, we recast this problem as disentangled reconstruction followed by relative pose recovery. CoGen produces instance-disentangled meshes with coarse initial poses leveraging complementary strengths of 2D and 3D generative models. GPRM refines the relative pose under joint geometric and physical guidance, and a VLM verifier selects the candidate that best matches the structured specification. We further construct a benchmark spanning eight representative contact-geometry classes and compare with state-of-the-art generators; GIF improves both asset quality and relation matching, while reducing collision rate to below 1%. Finally, we synthesize data for policy learning, revealing diversity scaling in both simulation and real-world deployment.

Long Xu, Zhiqi Zhang, Mi Yan +8

Galbot · Zhejiang University · Peking University +3

Synthetic Data Generation for Robotics 3D Scene Generation Robotic Manipulation Functional 3D Scene Generation Robot Learning