Splat2Act: From Captured Scenes to Personalized VLA Manipulation

Xiaomeng Ye; Krishna Jaganathan; Gyubeum Lim; Gunjan Chhablani; Muhammad Zubair Irshad; Zsolt Kira

cs.ROOct 9, 2026

Splat2Act: From Captured Scenes to Personalized VLA Manipulation

Authors: Xiaomeng Ye, Krishna Jaganathan, Gyubeum Lim, Gunjan Chhablani, Muhammad Zubair Irshad, Zsolt Kira

Abstract

A user who brings a generalist manipulation policy into their home or office wants it to pick up their objects from their shelves, counters, and tables. Yet even a strong generalist like $π_{0.5}$ achieves 0% pickup success in our captured deployment scenes without adaptation, and collecting real demonstrations in every user's environment is prohibitively expensive. We present Splat2Act, an automated real-to-sim-to-real pipeline that personalizes a pretrained vision-language-action (VLA) policy to a specific scene and object set using only a handheld phone scan and images of the user's objects. Unlike prior real-to-sim work that reconstructs individual tabletops, Splat2Act operates on building-scale indoor environments: it builds a photorealistic digital twin with 3D Gaussian Splatting and collision geometry, automatically discovers tabletop and non-tabletop receptacles, constructs feasible pickup tasks, and generates successful demonstrations with a privileged simulation oracle, with only scene and object capture required from a human. Applied to 101 scanned campus environments with a median navigable area of 104.87 m $^2$ and 7 receptacles, the pipeline yields 852 receptacles and over 68,000 pickup demonstrations. We fine-tune $π_{0.5}$ in two stages: Stage 1 adapts to hundreds of public object assets, drawn either from the deployment scene alone or from all 101 scenes; Stage 2 personalizes to the user's own objects on the deployment scene's receptacles. Our strongest personalized policy reaches 62.5% aggregate pickup success in the target scene. Deployed on a real Stretch 3, the same policy achieves 63.9% success on a 12-task subset, closely tracking its 66.7% simulated success on those tasks. These results suggest that automated real-to-sim-to-real training can personalize VLA manipulation policies using only simulation-oracle data.

Figures & tables

Fig. 1 : An overview of Splat2Act. From a handheld scan of a deployment scene and images of user-selected objects, Splat2Act builds a simulation-ready digital twin, generates pickup demonstrations without any teleoperation, and fine-tunes a VLA for deployment in the captured scene.

Feature	Splat2Act	VLK [ 35 ]	LEGS [ 7 ]	DREAM [ 9 ]	PRISM [ 30 ]	SimFoundry [ 31 ]	RE 3 SIM [ 27 ]	RialTo [ 22 ]	ARSTAG [ 34 ]
Scene scale	Building- scale	Room	Room	Table- top	Work- space	Table- top	Table- top	Work- space	Table- top
Fully automated after capture ∗	✓	✗	✗	✗	✗	✗	✗	✗	✗
Scene-wide receptacle discovery	✓	✗	✗	✗	✗	✗	✗	✗	✗
Feasibility-aware task assembly †	✓	✓	✗	✗	✓	✓	✓	✗	✓
Teleop-free demonstrations	✓	✓	✓	✓	✓	✗	✓	✗	✓
User-object reconstruction	✓	✗	✓	✓	✗	✓	✓	✓	✓

TABLE I : Comparison with capture-conditioned manipulation pipelines

Fig. 2 : The end-to-end pipeline of Splat2Act. Scene captures are reconstructed as a 3DGS with a mesh collision geometry, while simulation-ready 3D assets are generated from object captures. Receptacle discovery and reachability-aware task planning construct feasible pickup tasks, and a privileged oracle generates demonstrations for broad Stage 1 adaptation and Stage 2 personalization before real-world deployment.

Source	Successful	Assigned	Oracle SR
Broad Office	32,052	40,478	79.18%
Broad multi-scene	35,969	55,067	65.32%
Personalized Office	3,781	4,086	92.54%
Personalized Conference	3,469	4,097	84.67%

TABLE II : Oracle demonstration generation success rates

Rec.	Tasks/obj.	Trajs	Success Rate
1	8	32	25.0%
1	16	64	75.0%
1	24	96	75.0%
2	8	64	43.8%
2	16	128	75.0%
2	24	192	87.5%

TABLE III : In-distribution pickup success as receptacle count and augmentation depth scale, with four rendered 3D objects used for training.

Fig. 3 : Simulation evaluation on the personalized task banks. Office and Conference each show pretrained $\pi_{0.5}$ base, Office1k Stage 1, Multi1k Stage 1, and Stage 2 policies fine-tuned on them. Lighter bars show base/Stage 1 policies; darker bars show scene-matched personalized policies. Labels show SR (%).

Fig. 4 : Office sim-to-real comparison over 12 tasks. (a) Aggregate simulation and real SR. Each policy has 12 simulated episodes and 36 real attempts over the same object-receptacle-difficulty cells. (b) Policy-level scatter of those aggregate SRs; each point is one policy, and the dashed line indicates equal simulation and real SR. Labels show SR (%).

Scene	Stage 2 from	Easy	Hard	Easy $-$ Hard
Office ( $n$ =21/19)	$\pi_{0.5}$ base	57.1	31.6	25.6
	Office1k	71.4	52.6	18.8
	Multi1k	52.4	42.1	10.3
Conference ( $n$ =18/17)	$\pi_{0.5}$ base	55.6	29.4	26.1
	Office1k	44.4	29.4	15.0
	Multi1k	61.1	41.2	19.9

TABLE IV : Stage 2 pickup success by difficulty. Row headers give the number of evaluated configurations per scene.

Fig. 5 : Simulation cross-scene personalized-policy performance. The horizontal labels specify personalization scene $\rightarrow$ evaluation scene; each group compares Stage 2 policies fine-tuned from $\pi_{0.5}$ base, Office1k Stage 1 policy, and Multi1k Stage 1 policy. Labels show SR (%).

Fig. 6 : Transfer to the unseen lab. Black table and Yellow table each include three attempts for each of three targets under both difficulties (18 trials per policy). Labels show SR (%).

Explore similar work

Oct 8, 2026cs.RO

SimVLA: Zero-Shot Sim-to-Real VLA Learning for Mobile Manipulation

Large-scale, diverse datasets have driven the success of LLMs and VLMs. But VLAs for robotics remain limited by the cost and complexity of real-world data collection. While simulation offers a scalable alternative, its potential for sim-to-real VLA learning in mobile manipulation remains largely underexplored. We introduce SimVLA, an end-to-end framework that trains VLAs entirely on synthetic simulation data without teleoperation for mobile manipulation. SimVLA is first pre-trained on two complementary simulation-derived datasets: SimAction, a large-scale robot action dataset spanning 35 diverse mobile manipulation tasks, generated by composing atomic skills, and SimVQA, which leverages privileged simulator state to provide spatial, geometric, and subtask-level visual-language supervision. We further post-train SimVLA on a mixture of SimAction and SimDeploy, a dataset collected from policy rollouts across diverse simulated environments. We evaluate SimVLA on tasks including restocking, pouring, and cleaning, and show zero-shot transfer to real-world mobile manipulation, including real home environments. SimVLA outperforms policies trained on 50 in-domain real-world demonstrations, suggesting that simulation can enable scalable sim-to-real mobile manipulation. We further demonstrate the value of multiple complementary forms of supervision for effectively leveraging simulation in VLA training.

Kyoungin Baik, Youngwoon Lee

Yonsei University · Seoul National University

Sim-to-Real Transfer Vision-Language-Action Models Zero-Shot Robotic Manipulation Mobile Manipulation Robotics Simulation Synthetic Data Generation for Robotics

Ask PDF

Sep 18, 2026cs.RO

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions. Most state-of-the-art models predict actions in the end-effector pose space as sequences of action chunks. Training and evaluating these models requires large-scale collections of real-world demonstrations, pairing robot actions with the corresponding visual and proprioceptive observations. Collecting such data on real hardware typically relies on human teleoperation, making the process costly, time-consuming, and difficult to scale. We present an open-source sim-to-real experimental protocol that addresses this bottleneck: expert trajectories generated in simulation are replayed open-loop on a real Franka FR3 setup, where the corresponding real visual and proprioceptive observations are recorded and converted into a format compatible with VLA training. The same deployment stack is then reused, in closed-loop, to evaluate a trained policy on that setup, so that data collection and evaluation share an identical hardware configuration. Because each real recording is paired with the simulated trajectory that produced it, the protocol also yields a direct measurement of the sim-to-real gap. We release the collected datasets on Hugging Face together with the pipeline source code https://gitlab.isir.upmc.fr/kappel/sim2real_public_chunk_control.

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani +5

Institute of Intelligent Systems and Robotics (ISIR), CNRS, Sorbonne University, Paris, F-75005, France

Sim-to-Real Transfer Robotic Manipulation Vision-Language-Action Models Robotics Simulation Robot Imitation Learning

Ask PDF

Jun 23, 2026cs.RO

InSight: Self-Guided Skill Acquisition via Steerable VLAs

Vision-language-action (VLA) models excel at robot manipulation via imitation learning, but adapting them to new tasks often requires additional human demonstrations, which can be costly or infeasible. Meanwhile, vision-language models (VLMs) offer semantic task understanding but lack the physical grounding required for execution. To bridge this gap, we present InSight, a framework for self-guided skill acquisition that uses a VLM to identify primitives missing from a VLA's repertoire, grounds the VLM's proposals through robot execution, and distills new primitives from successful rollouts into the VLA. Primitive steerability, the ability to execute and terminate primitives on command, enables the robot to reuse known primitives while collecting training data for missing primitives without requiring full-task human demonstrations for each new task. InSight has two stages: (1) a VLM automatically segments existing demonstrations into primitive-labeled trajectories to fine-tune a primitive-steerable VLA, and (2) the VLM plans a sequence of known primitives executed by the VLA and new primitives attempted by VLM-parameterized low-level controllers. New-primitive segments from successful task rollouts are added to the training data, and the VLA is retrained. The adapted VLA can then reliably execute new skills using the acquired primitives, without per-primitive VLM calls. We evaluate InSight on six simulated and real-world tasks with no human demonstrations of target skills, including block flipping, drawer closing, sweeping, twisting, and pouring. On hardware, acquired twisting and pouring skills achieve 92% and 96% success, versus 32% and 16% for a zero-shot CaP-X baseline. Composing both skills into a 14-primitive task achieves 80% success with no combined-task demonstrations. Project website: https://insight-vla.github.io/ .

Maggie Wang, Lars Osterberg, Stephen Tian +5

Stanford University · Princeton University

Robot Skill Learning Large Language Model-Based Robot Planning Generalization in Robotic Manipulation Robotic Manipulation Vision-Language-Action Models Robot Imitation Learning+1

Ask PDF