SCULPT-VLA: Learning Structured Control through Staged Action Grounding

Wenbo Li; Yiteng Chen; Wei Zhang; Wenhao Li; Jun Yang; Qingyao Wu

cs.ROSep 20, 2026

SCULPT-VLA: Learning Structured Control through Staged Action Grounding

Authors: Wenbo Li, Yiteng Chen, Wei Zhang, Wenhao Li, Jun Yang, Qingyao Wu

Organizations: School of Software Engineering, South China University of Technology Guangzhou, China · Yuanwu Technology, Shenzhen, China

Abstract

Vision-language-action (VLA) policies increasingly incorporate structured intermediate supervision beyond action labels. Yet specifying what an intermediate representation should encode leaves open how action prediction learns to depend on it. We introduce SCULPT-VLA, a policy that learns structured control through staged action grounding. Its action-conditioning state comprises complementary factors for task progression, scene dynamics, and spatial grounding. Training first forms these factors with teacher scaffolds, then grounds coarse action prediction through their composition as scaffold inputs are withdrawn. Direct perceptual access is subsequently restored for continuous refinement, combining the learned state with perceptual detail. The curriculum separates learning to condition actions on structure from refining continuous control. Deployment requires neither teachers nor discrete-action autoregression. SCULPT-VLA achieves higher average success than shared-backbone baselines on LIBERO, SimplerEnv-WidowX, and RoboTwin 2.0 Full. On SimplerEnv-WidowX, final success is 83.5%, versus 71.3% when Stage-II action learning directly accesses vision and language. Across four physical robot tasks, average success under the tested distribution shifts reaches 58.1%, compared with 45.6% for $π_{0.5}$ . Training ablations and factor-wise interventions support the staged design and show that the learned state continues to contribute to control after direct perceptual access is restored.

Figures & tables

Figure 1: SCULPT-VLA overview. Stage I forms teacher-aligned factors; Stage II withdraws scaffold inputs while grounding FAST prediction through interacting factors; Stage III restores direct perceptual access for continuous refinement. The continuous head jointly uses factor states, fixed action readout, and DINOv2 features.

Figure 2: Stage-specific attention masks and objectives. Rows are query groups; columns are key groups. Dark gray permits attention, light gray blocks it, and hatching marks excluded groups. The discrete Action group uses causal attention over fixed readout followed by FAST positions. The continuous Action group contains only fixed readout.

Published results
Method	LIBERO					SimplerEnv-WidowX
Method	Spatial	Object	Goal	Long	Avg.	Spoon	Carrot	Stack	Eggplant	Avg.
OpenVLA ( Kim et al., 2024 )	84.7	88.4	79.2	53.7	76.5	0.0	0.0	0.0	4.1	1.0
OpenVLA-OFT ( Kim et al., 2024 ; Kim et al., 2025 )	97.6	98.4	97.9	94.5	97.1	12.5	4.2	8.3	37.5	15.6
Octo-Small ( Team et al., 2024 )	78.9	85.7	84.6	51.1	75.1	47.2	9.7	4.2	56.9	29.5
$\pi_{0}$ ( Black et al., 2024 )	96.8	98.8	95.8	85.2	94.2	29.1	0.0	16.7	62.5	27.1

Table 1: Main results on LIBERO and SimplerEnv-WidowX (success, %). Published methods retain their reported protocols. Shared-backbone variants use Qwen3-VL in StarVLA ( Community, 2026 ) . SCULPT-VLA entries are three-seed means (Appendix B.3 ). Dashes indicate no reported result for the listed setting. Shades rank the five highest distinct scores per column, from green to yellow. For

\pi_{0.5}

, LIBERO results follow the official openpi release ; SimplerEnv results follow Lian et al. (2026) .

Published results
Method	LIBERO					RoboTwin 2.0 Full
Method	Spatial	Object	Goal	Long	Avg.	Clean	Random	Avg.
$\pi_{0}$ ( Black et al., 2024 )	96.8	98.8	95.8	85.2	94.2	65.9	58.4	62.2
LingBot-VLA ( Wu et al., 2026 )	81.8 †	95.0 †	86.6 †	67.8 †	82.8 †	88.6	86.7	87.7
Motus ( Bi et al., 2025 )	96.8	99.8	96.6	97.6	97.7	88.7	87.0	87.9
Fast-WAM ( Yuan et al., 2026 )	98.2	100.0	97.0	95.2	97.6	91.9	91.8	91.9

Table 2: Comparison on LIBERO and RoboTwin 2.0 Full (success, %). Published results retain their source protocols (Appendix B.1 ). Shared-backbone variants use Qwen3-VL in StarVLA ( Community, 2026 ) . RoboTwin Avg. averages Clean and Random.

\dagger

: LIBERO reproduction by Sun et al. (2026a) ;

\ddagger

: LIBERO episode limits of 600 steps (Spatial/Object/Goal) and 700 (Long). Motus’s LIBERO and

\pi_{0}

’s RoboTwin results follow Yuan et al. (2026) . Darker and lighter green mark the best and second-best distinct scores per column, respectively.

Figure 3: Real-world evaluation on Fruit Collecting, Paper Cup Stacking, Toothbrush Insertion, and Cloth Folding. Top: representative Cloth Folding success and failure. Middle: four task setups. Bottom: task-specific OOD configurations. Task definitions and perturbations are detailed in Table 12 .

	Fruit Collecting		Cup Stacking		Toothbrush Insertion		Cloth Folding
Method	ID	OOD	ID	OOD	ID	OOD	ID	OOD
OpenVLA-OFT	$50.0{\pm}3.3$	$24.4{\pm}5.1$	$61.1{\pm}1.9$	$36.7{\pm}6.7$	$44.4{\pm}1.9$	$17.8{\pm}5.1$	$25.6{\pm}1.9$	$2.2{\pm}1.9$
$\pi_{0}$	$65.6{\pm}1.9$	$45.6{\pm}6.9$	$75.6{\pm}1.9$	$56.7{\pm}3.3$	$67.8{\pm}3.8$	$44.4{\pm}5.1$	$35.6{\pm}1.9$	$14.4{\pm}3.8$
$\pi_{0.5}$	$74.4{\pm}1.9$	$52.2{\pm}5.1$	$81.1{\pm}1.9$	$60.0{\pm}6.7$	$73.3{\pm}3.3$	$47.8{\pm}3.8$	$41.1{\pm}1.9$	$22.2{\pm}5.1$
SCULPT-VLA	$80.0{\pm}3.3$	$60.0{\pm}3.3$	$85.6{\pm}1.9$	$70.0{\pm}6.7$	$84.4{\pm}5.1$	$64.4{\pm}5.1$	$56.7{\pm}3.3$	$37.8{\pm}5.1$

Table 3: Real-world task success (%). Each method uses 100 ID demonstrations and one trained checkpoint per task. Entries are mean

\pm

sample SD across three independent evaluation batches of 30 physical rollouts per condition. OOD settings are defined in Table 12 ; all open-loop scores are in Table 13 .

Variant	FAST	Continuous	mOLS $\uparrow$
Full SCULPT-VLA	$74.7{\pm}1.0$	$83.5{\pm}0.0$	0.369
shared factor FFN	$65.7{\pm}1.0$	$76.8{\pm}1.0$	0.329
generic latent pool	$52.8{\pm}1.2$	$63.9{\pm}1.1$	0.271
w/o subtask factor	$67.3{\pm}1.3$	$78.3{\pm}1.3$	0.342
w/o spatial factor	$64.8{\pm}0.8$	$72.2{\pm}0.8$	0.312
w/o dynamics factor	$61.6{\pm}1.0$	$73.3{\pm}0.8$	0.322

Table 4: Complete ablations on SimplerEnv-WidowX. FAST and Continuous denote the Stage-II discrete and Stage-III continuous policies. Success (%) is reported as mean

\pm

sample SD over evaluation runs; variants are trained independently. mOLS is defined in Appendix B.2 ; per-threshold scores and variant definitions are in Appendix B.4 .

Replaced states	ID	OOD
Full SCULPT-VLA	$80.0{\pm}3.3$	$60.0{\pm}3.3$
Replace subtask factor	$54.4{\pm}5.1$	$28.9{\pm}10.2$
Replace spatial factor	$73.3{\pm}3.3$	$50.0{\pm}6.7$
Replace dynamics factor	$65.6{\pm}5.1$	$42.2{\pm}6.9$
Replace all factors	$15.6{\pm}5.1$	$5.6{\pm}5.1$

Table 5: Factor-wise intervention on Fruit Collecting. Selected final factor states are replaced with their training-set means before continuous decoding. Success (%): mean

\pm

sample SD across three evaluation batches of 30 rollouts per condition. Raw counts: Table 16 .

Figure 4: Factor probes on turn_on_the_stove . Rows show observations, depth reconstructed from spatial slots, and depth-change predictions from dynamics slots. These probes visualize decodable information; Appendix C.4.1 describes the diagnostic setup.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Example training formats across the three stages of SCULPT-VLA. <slot_subtask> , <slot_dynamic> , <slot_spatial> , and <slot_action> denote placeholder tokens reserved in the VLM sequence for subtask, dynamics, spatial, and action-side latent slots, respectively. Stage II additionally appends teacher-forced FAST action-token positions, assigned type act , for coarse next-token supervision. The Stage III continuous sequence excludes FAST tokens; weak FAST supervision uses a separate autoregressive path. The continuous head reads final factor and action-readout states, together with DINOv2 detail features. Boundary tokens such as <SOST> / <EOST> mark the start and end of each slot block for type assignment, masking, and stage-specific loss computation.

Stage	Teacher Encoders	VLM Attn LoRA	Base- FFN	Sub./Dyn./Spa. FFNs	Action FFN	Slot Proj./ Adapters	FAST Action LM Head	DiT Action Head
Stage I	$\times$	$\times$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$
Stage II	$\times$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$
Stage III	$\times$	$\triangle$	$\times$	$\triangle$	$\triangle$	$\checkmark$	$\triangle$	$\checkmark$

Appendix

Table 6: Stage-wise module trainability in SCULPT-VLA.

\checkmark

denotes trainable modules,

\times

denotes frozen modules, and

\triangle

denotes modules co-trained with a reduced learning rate. For the VLM backbone, “trainable” denotes LoRA-updated linear projections rather than full-rank backbone training.

Learning Rates
Hyperparameters	LIBERO / RoboTwin 2.0			SimplerEnv			Real Robot
Hyperparameters	Stage I	Stage II	Stage III	Stage I	Stage II	Stage III	Stage I	Stage II	Stage III
VLM LR	—	$1\times 10^{-5}$	$5\times 10^{-6}$	—	$1\times 10^{-5}$	$5\times 10^{-6}$	—	$1\times 10^{-5}$	$5\times 10^{-6}$
Teacher Encoder LR	—	—	—	—	—	—	—	—	—
Slot Projector LR	$1\times 10^{-4}$	$1\times 10^{-4}$	$5\times 10^{-5}$	$1\times 10^{-4}$	$1\times 10^{-4}$	$5\times 10^{-5}$	$1\times 10^{-4}$	$1\times 10^{-4}$	$5\times 10^{-5}$
FAST Action Head LR	—	$1\times 10^{-4}$	$5\times 10^{-5}$	—	$1\times 10^{-4}$	$5\times 10^{-5}$	—	$1\times 10^{-4}$	$5\times 10^{-5}$

Appendix

Table 7: Hyperparameter settings of SCULPT-VLA across LIBERO, RoboTwin 2.0, SimplerEnv, and real-robot experiments. Scheduled indicates scaffold-input annealing or coarse-loss weighting following the schedule in Sec. 3.2 ; Weak denotes retained at a small weight as auxiliary regularization.

Figure 6: Automated long-video segmentation pipeline. An untrimmed demonstration video is decomposed into overlapping temporal windows. Within each window, sampled visual evidence and semantic anchors are provided to Qwen3-VL-32B-Instruct for object-centric transition prediction. The resulting transition proposals are mapped to the global timeline and merged through cross-window aggregation to remove redundant detections from overlapping windows. The final consolidated boundaries partition the original demonstration into atomic task segments with aligned language instructions.

Dataset	JSON valid	Discarded	Conf. $>80\%$
LIBERO-SUBTASK	$98.1\%$	$3.0\%$	$94.8\%$
BRIDGE-SUBTASK	$96.2\%$	$5.8\%$	$91.5\%$

Appendix

Table 8: Processing statistics of the derived subtask annotations. Validity and discard rates are computed after one regeneration attempt; confidence denotes the percentage of valid annotations whose VLM-reported boundary confidence exceeds

80\%

Method	Clean	Random	Avg.
X-VLA ( Zheng et al., 2025 )	72.8	72.8	72.8
$\pi_{0.5}$ ( Physical Intelligence et al., 2025 )	82.7	76.8	79.8

Appendix

Table 9: Additional published RoboTwin 2.0 Full results (success, %). X-VLA results follow Bi et al. (2025) , and

\pi_{0.5}

results follow Yuan et al. (2026) . Avg. is the arithmetic mean of Clean and Random.

Benchmark / Task	Seed 1	Seed 2	Seed 3	Success Rate (%)
LIBERO-Spatial	496/500	494/500	497/500	$99.1{\pm}0.3$
LIBERO-Object	498/500	496/500	499/500	$99.5{\pm}0.3$
LIBERO-Goal	492/500	493/500	489/500	$98.3{\pm}0.4$
LIBERO-Long	485/500	482/500	487/500	$96.9{\pm}0.5$
SimplerEnv-Spoon	45/50	45/50	46/50	$90.7{\pm}1.2$
SimplerEnv-Carrot	35/50	36/50	35/50	$70.7{\pm}1.2$

Appendix

Table 10: Seed-level closed-loop success counts of SCULPT-VLA on LIBERO and SimplerEnv-WidowX. LIBERO rows report successful trials out of 500 per seed, corresponding to 10 tasks with 50 trials each. SimplerEnv-WidowX rows report successful trials out of 50 per seed for each displayed task. Success Rate is mean

\pm

sample SD over three seeds.

Variant	Success Rate (%)		OLS $\uparrow$				mOLS $\uparrow$
	FAST	Cont.	@0.1	@0.05	@0.03	@0.01
Full SCULPT-VLA	$74.7{\pm}1.0$	$83.5{\pm}0.0$	0.738	0.431	0.248	0.057	0.369
shared factor FFN	$65.7{\pm}1.0$	$76.8{\pm}1.0$	0.697	0.379	0.201	0.039	0.329
generic latent pool	$52.8{\pm}1.2$	$63.9{\pm}1.1$	0.632	0.301	0.132	0.020	0.271
w/o subtask factor	$67.3{\pm}1.3$	$78.3{\pm}1.3$	0.713	0.397	0.216	0.044	0.342
w/o spatial factor	$64.8{\pm}0.8$	$72.2{\pm}0.8$	0.680	0.356	0.183	0.031	0.312

Appendix

Table 11: Ablation results on SimplerEnv-WidowX Avg. We report closed-loop success under Stage-II FAST decoding and Stage-III continuous control, together with open-loop OLS and mOLS. Success rates are mean

\pm

sample SD over evaluation runs; variants are trained independently. OLS and mOLS are defined in Appendix B.2 . Dashes indicate variants without a supervised Stage-II FAST policy.

Task	Success criterion	ID setting	OOD perturbations
Fruit Collecting	All target fruits are picked and placed into the basket without dropping, collision-induced failure, or premature termination.	Training fruit instances and canonical basket layout.	Unseen fruit instances, changed fruit order, and altered initial object positions.
Paper Cup Stacking	The target cup is lifted, aligned, and stably stacked on the base cup.	Training cup colors/textures and nominal relative placement.	Novel cup colors/textures, shifted cup positions, and altered approach directions.
Toothbrush Insertion	The toothbrush is grasped, transported, and inserted into the target container with stable final placement.	Training toothbrush/container pair and nominal insertion pose.	Unseen toothbrush or container appearance, shifted container position, and altered insertion orientation.
Cloth Folding	The cloth is folded along the intended direction and remains approximately aligned after release.	Training cloth appearance and canonical tabletop placement.	Changed cloth position/orientation, visual texture variation, and perturbed initial spread.

Appendix

Table 12: Real-world task definitions and OOD perturbations.

		ID			OOD
Method	Task	[email protected]	mOLS	SC	[email protected]	mOLS	SC
OpenVLA-OFT	Fruit Collecting	$0.648$	$0.271$	$50.0{\pm}3.3$	$0.529$	$0.203$	$24.4{\pm}5.1$
	Paper Cup Stacking	$0.689$	$0.298$	$61.1{\pm}1.9$	$0.596$	$0.239$	$36.7{\pm}6.7$
	Toothbrush Insertion	$0.629$	$0.263$	$44.4{\pm}1.9$	$0.485$	$0.184$	$17.8{\pm}5.1$
	Cloth Folding	$0.556$	$0.224$	$25.6{\pm}1.9$	$0.427$	$0.129$	$2.2{\pm}1.9$
$\pi_{0}$	Fruit Collecting	$0.704$	$0.319$	$65.6{\pm}1.9$	$0.637$	$0.276$	$45.6{\pm}6.9$

Appendix

Table 13: Complete real-world open-loop and closed-loop results. Each task has ID and OOD columns for [email protected], mOLS, and success rate (SC, %). Open-loop scores use held-out expert trajectories. SC is mean

\pm

sample SD across three independent evaluation batches of 30 physical rollouts per condition, using one checkpoint per method/task.

Method	Fruit ID	Fruit OOD	Cup ID	Cup OOD	Toothbrush ID	Toothbrush OOD	Cloth ID	Cloth OOD
OpenVLA-OFT	15/14/16	7/9/6	19/18/18	11/9/13	13/14/13	5/7/4	8/8/7	1/1/0
$\pi_{0}$	20/19/20	13/16/12	23/22/23	17/16/18	21/19/21	13/15/12	11/11/10	5/5/3
$\pi_{0.5}$	22/22/23	16/14/17	24/25/24	18/20/16	22/23/21	15/15/13	13/12/12	8/7/5
SCULPT-VLA	24/25/23	18/17/19	25/26/26	21/23/19	25/24/27	19/21/18	18/17/16	13/11/10

Appendix

Table 14: Real-world SC counts across evaluation batches. Each entry reports successful rollouts out of

30

for Batch 1/Batch 2/Batch 3, using one trained checkpoint per method and task.

Panel of Figure 7 — Figure 7: Group-level representation diagnostics. UMAP uses frame-level group means. Silhouette scores are calculated in the original hidden space.

Figure 8: All-stage group-level UMAP in a shared PCA+UMAP space. Points are frame-level means for the active token groups; the action group appears from Stage II onward. All stages share the same fitted projection.

Figure 9: Factor- and action-group UMAP across training stages. The three factor groups occupy distinct regions from Stage I onward; the action group is included in Stages II and III.

Figure 10: Stage-wise final-layer attention mass. Panels show Stages I, II, and III. Left nodes are key groups and right nodes are query groups. Band widths represent attention mass between groups.

Variant	Stage-II Mask	Stage-III Mask	LIBERO-Spatial Success (%)
Standard SCULPT-VLA	$\mathbf{M}^{(2)}$	$\mathbf{M}^{(3)}$	$99.1$
w/o Stage-II visibility restriction	$\mathbf{M}^{(3)}$	$\mathbf{M}^{(3)}$	$98.7$

Appendix

Table 15: Stage-II perceptual-access diagnostic on LIBERO-Spatial. Success is the three-seed mean under the main evaluation protocol. The w/o Stage-II visibility restriction variant uses

\mathbf{M}^{(3)}

in both Stages II and III, allowing act-typed positions to read vision and language throughout action grounding.

Figure 11: Spatial-slot probe against the depth target. Top: target depth map of the current observation; bottom: depth map reconstructed from spatial slots. The qualitative agreement in overall geometry is consistent with spatial slots preserving dense scene structure and spatial priors.

Figure 12: PCA of frame-level subtask-slot means from $100$ trajectories of turn_on_the_stove . Colors indicate five post-hoc, analysis-only execution phases. The 2D and 3D views show phase-associated geometry, including connected neighboring regions and a compact terminal region.

Figure 13: Dynamics-branch slot-to-visual attention on turn_on_the_stove . The high-response region shifts with the evolving end-effector–knob interaction, consistent with sensitivity to motion-dependent state change.

Figure 14: Spatial-branch slot-to-visual attention on turn_on_the_stove . Attention remains concentrated around the target knob and its geometric neighborhood, consistent with a role in target localization and local geometric grounding.

Figure 15: Subtask-branch slot-to-visual attention on turn_on_the_stove . Attention is broader and temporally more stable, covering the stove and surrounding semantic context, consistent with stage-level task progression modeling.

Variant	Fruit Collecting ID				Fruit Collecting OOD
Variant	Batch 1	Batch 2	Batch 3	SC (%)	Batch 1	Batch 2	Batch 3	SC (%)
Full SCULPT-VLA	24/30	25/30	23/30	$80.0{\pm}3.3$	18/30	17/30	19/30	$60.0{\pm}3.3$
Replace subtask factor	16/30	15/30	18/30	$54.4{\pm}5.1$	8/30	12/30	6/30	$28.9{\pm}10.2$
Replace spatial factor	22/30	23/30	21/30	$73.3{\pm}3.3$	15/30	13/30	17/30	$50.0{\pm}6.7$
Replace dynamics factor	20/30	21/30	18/30	$65.6{\pm}5.1$	12/30	15/30	11/30	$42.2{\pm}6.9$
Replace all factors	5/30	3/30	6/30	$15.6{\pm}5.1$	2/30	0/30	3/30	$5.6{\pm}5.1$

Appendix

Table 16: Factor-wise functional intervention on the real-world Fruit Collecting task. Each evaluation batch reports successful rollouts out of

30

under the same Stage-III checkpoint. SC is mean

\pm

sample SD over three independent evaluation batches per condition.

Figure 16: LIBERO turn_on_the_stove : approach, end-effector alignment with the knob, and contact-sensitive knob turning.

Figure 17: LIBERO open_the_middle_drawer_of_the_cabinet : handle approach and sustained drawer pulling while maintaining contact with the handle.

Figure 18: LIBERO open_the_top_drawer_and_put_the_bowl_inside : a two-phase task (drawer opening, bowl transport and placement) under changing scene geometry.

Figure 19: SimplerEnv-WidowX carrot : stable grasp with consistent target alignment during transport.

Figure 20: SimplerEnv-WidowX stack : relative-pose alignment prior to final placement, illustrating spatial grounding in contact-sensitive stacking.

Figure 21: SimplerEnv-WidowX spoon : stable grasp and placement of an elongated tool-shaped object under shape-dependent pose constraints.

Figure 22: Representative Fruit Collecting rollout showing repeated grasp–transport–release cycles as the basket fills.

Figure 23: Representative Paper Cup Stacking rollout showing cup lifting, alignment, and placement.

Figure 24: Representative Toothbrush Insertion rollout showing grasp, transport, pose adjustment, and insertion.

Figure 25: Representative Cloth Folding rollout showing pulling and folding as the cloth configuration changes.

Visuomotor Policy Learning

Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.

See all 286 papers

ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models
Oct 6, 2026Yuan Xu, Yixiang Chen, Qisen Ma +10Latent Action Learning Visuomotor Policy Learning
CASIA · UCAS · Alibaba Group +1
VLA- $Δ$ : Learning Physical State Transition for Vision-Language-Action Models
Oct 15, 2026Qi Lu, Jiarun Li, Ce Zhang +5VLA Model Training Vision-Language-Action Models
Zhejiang University (zju.edu.cn) · Central South University (csu.edu.cn)
Coarse-to-Control: Action-Token Planning for Vision-Language-Action Models
Jun 5, 2026Jinhao Wu, Shiduo Zhang, Yicheng Liu +9Visuomotor Policy Learning AI Control
1Nanjing University · 2Shanghai Innovation Institute · 3Fudan University +1

Oct 6, 2026J/K move · Enter open · S save

ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models

Yuan Xu, Yixiang Chen, Qisen Ma, Jiabing Yang+9

CASIA · UCAS · Alibaba Group+1

Latent Action Learning Visuomotor Policy Learning Robotic Manipulation Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have become a central paradigm for robot policy learning, which predict actions in three forms: raw action chunks, discrete action tokens, or continuous action latents. However, existing action representations primarily model action trajectories, with limited consideration of the visual dynamics induced by these actions. We introduce ViDAL, a Visual Dynamics-grounded Action Latent Space that anchors continuous action latents in the future visual dynamics of the scene. Specifically, ViDAL learns action latent space by training an Action Variational Autoencoder (Action VAE) to reconstruct action chunks while aligning its latent with future scene dynamics. When integrated into downstream robot policies, the proposed Action VAE serves as a plug-in action interface compatible with multiple VLA architectures and enables optional future-video prediction as an additional capability. Empirically, ViDAL outperforms competitive baselines on LIBERO with 98.1% average success, improves a multi-task

π_{0.5}

policy on RoboTwin 2.0 from 54.3% to 65.5% (Clean) and from 33.2% to 43.1% (Random) success rates over 50 dual-arm tasks, and yields 20.0% and 23.4% absolute success-rate gains on real-world single-arm Franka and dual-arm ARX robot platforms.

Abstract

Visuomotor Policy Learning

Explore similar work

ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models

VLA-ΔΔΔ: Learning Physical State Transition for Vision-Language-Action Models

Coarse-to-Control: Action-Token Planning for Vision-Language-Action Models

VLA- $Δ$ : Learning Physical State Transition for Vision-Language-Action Models