cs.RONov 3, 2026

Should the Action Expert Train the VLM? Action-Token Supervision and Gradient Routing in Mixture-of-Transformer VLAs

Authors: Asher James Hancock, Boyi Li, Yuxiao Chen, Shuhan Tan, Wenhao Ding, Thomas Tian, Yan Wang, Boris Ivanovic, 

Organizations: NVIDIA · Princeton University · Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-language-action (VLA) models increasingly pair a pre-trained vision-language model (VLM) with a flow-matching expert that conditions its action predictions on the VLM's key-value representations. Those representations must carry information useful for control, yet the VLM was never trained for the action expert's flow-matching objective. Two signals can address this: gradients backpropagated from the expert's loss (coupling), or VLM supervision to predict discrete tokens encoding the same target actions. Prominent training recipes forgo the first, blocking expert-to-VLM gradients (insulation) to protect the VLM's existing capabilities, while also supervising it on vision-language data and action tokens. Because vision-language co-training also aims to preserve those capabilities, the additional protection provided by insulation is unclear. At the same time, insulation removes a direct signal about which features continuous action prediction requires. On the other hand, action-token supervision offers an alternative way to learn action-relevant representations, which may explain why insulation succeeds in practice. We disentangle these factors in autonomous driving and manipulation by independently varying action-token supervision and whether the expert's loss updates the VLM. In open-loop driving, insulation is competitive with coupling when action tokens are supervised; without them, coupling reduces trajectory-prediction error by roughly 25%. When both components are trained jointly from the outset, coupling increases closed-loop driving route progress by approximately 17% with action-token supervision and 40% without it, and similarly improves robot manipulation performance under both settings. These findings show that the value of gradient insulation depends on the VLM's action supervision and the evaluation setting, challenging insulation as a default VLA training recipe.

Figures & tables

Figure 1: Action-token supervision shapes the effect of expert-to-VLM gradient flow. Left: the VLM is supervised to predict discrete action tokens while a flow-matching expert decodes continuous actions from the VLM’s K/V cache. We independently vary whether action tokens are supervised and whether the expert’s gradient ( ∇Lflow\nabla\mathcal{L}_{\mathrm{flow}} ) updates the VLM. Right: open-loop driving evaluation error during training. With action-token supervision, insulated and coupled training differ little; without it, coupling is substantially better. We test this interaction in closed-loop autonomous driving and robot manipulation.
PolicyVQA PredictedDiscrete Actions Predicted
Autonomous Driving
Alpamayo-R1 [ 32 ]✓\checkmark✓\checkmark
Qwen-Drive-1.0 [ 43 ]✓\checkmark–
SpanVLA [ 46 ]–✓\checkmark
UniDriveVLA [ 22 ]✓\checkmark–
Manipulation
Table 1: Driving and manipulation VLAs that block action-expert gradients from the VLM through freezing or knowledge insulation [ 6 ] . Checkmarks indicate VQA or discrete action-token supervision.
Figure 2: Action-token supervision, gradient routing, and training schedules. Left : we cross future-action-token supervision with coupled or insulated expert-to-VLM gradients. Right : bars show when the VLM (blue) and action expert (black) are trained. Concurrent training starts both modules together; VLM-first and expert-first training start one module after 100k steps of training the other module. All schedules run for 300k total steps in autonomous driving experiments.
Figure 3: Open-loop driving accuracy across training histories. First-sample mean ADE on the Public Golden (1,434 examples) and OOD (273 examples) evaluation splits. Bars show the mean across scenes; error bars show 95% clip-cluster bootstrap confidence intervals. Lower is better.
   Public Golden ( N=1,434N=1{,}434 )OOD ( N=273N=273 )
ScheduleRoutingAction CETop-1 [95% CI]Top- kkAvg- kkTop-1 [95% CI]Top- kkAvg- kk
ConcurrentInsulatedOn0.647 [0.621,0.673]0.8080.6300.414 [0.355,0.474]0.6590.466
ConcurrentCoupledOn0.649 [0.623,0.674]0.8230.6290.425 [0.364,0.487]0.6890.471
ConcurrentInsulatedOff0.566 [0.539,0.593]0.7670.5730.363 [0.305,0.423]0.6520.435
ConcurrentCoupledOff0.635 [0.610,0.661]0.8110.6220.348 [0.292,0.406]0.6850.450
VLM-FirstInsulatedOn0.639 [0.614,0.665]0.8070.6260.392 [0.335,0.452]0.6520.462
Table 2: Generated reasoning-trace quality. First-sample Lingo-Judge pass rates for each model. In each example, the frozen Lingo-Judge classifier compares the reasoning-trace with a reference rationale [ 27 ] ; an example passes when the judge score exceeds 0.50.5 . Brackets give 95% clip-cluster bootstrap confidence intervals. Top- kk is the fraction for which any of six sampled rationales passes, and Avg- kk is the mean judge score across the six samples. Results are evaluated on Public Golden and OOD. Within each training schedule, the best metric is bolded. Higher is better.
 InsulatedCoupled
HistoryTokens onTokens offTokens onTokens off
Concurrent0.6000.4250.7030.601
VLM-First0.6480.5550.6460.587
Expert-First0.5970.5380.6250.604
Table 3: Closed-loop route progress across training histories. Mean AlpaSim route progress on the same 827 scenes, with five stochastic rollouts per scene averaged within each scene. The best value in each row is bolded and the second-best is underlined. Higher is better.
Conditionπ0.5\boldsymbol{\pi_{0.5}}MolmoAct2
Insulated / FAST on19.5%22.0%
Coupled / FAST on30.7%24.3%
Insulated / FAST off10.3%14.5%
Coupled / FAST off32.9%36.5%
Insulated / FAST on40.0%32.9%
Table 4: Closed-loop manipulation progress. Equal-weight mean normalized task progress across three representative PolaRiS tasks. Bold indicates the best result among our four conditions within each model family. The final row provides additional references: the published π0.5\pi_{0.5} result is from CoVeR [ 20 ] and uses its evaluation protocol; the MolmoAct2 result uses its released checkpoint with our own fine-tuning and evaluation scheme. Higher is better.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

SettingValue
OptimizerFused AdamW
(β1,β2)(\beta_{1},\beta_{2})(0.9,0.95)(0.9,0.95)
Adam ϵ\epsilon10−810^{-8}
Weight decay0.10.1
Vision LR10−510^{-5}
VLM text/head LR10−410^{-4}
Appendix
Table 5: Driving optimization settings. Shared across conditions within each training history.
HistoryScheduleVLM updatesExpert updates
Concurrent300k joint steps300k≈145.5\approx 145.5 k
Expert-First100k expert-only + 200k joint200k≈197\approx 197 k
VLM-First100k VLM-only + 200k joint300k≈97\approx 97 k
Appendix
Table 6: Expected component updates. Endpoints have 300k total steps
ComponentBackboneLayersWidthMLPHeads (Q/KV)
π0.5\pi_{0.5}
VLM BackboneGemma 2B182,04816,3848/1
Action expertGemma 300M expert (random)181,0244,0968/1
MolmoAct2
VLM BackboneMolmo2-ER362,56019,45632/8
Action expertMolmoAct2 expert (random)367683,0728/8
Appendix
Table 7: Manipulation model dimensions. Attention heads are query/key-value counts.
SettingValue
OptimizerAdamW
(β1,β2)(\beta_{1},\beta_{2})(0.9,0.95)(0.9,0.95)
Adam ϵ\epsilon10−810^{-8}
Weight decay10−1010^{-10}
Peak/terminal LR5×10−55\times 10^{-5}
Warm-up1,000 steps
Appendix
Table 8: π0.5\pi_{0.5} manipulation optimization settings. Shared across conditions through 400k steps.
ComponentPeak LR
Language model1×10−51\times 10^{-5}
Vision encoder5×10−65\times 10^{-6}
Connector5×10−65\times 10^{-6}
Action expert5×10−55\times 10^{-5}
Appendix
Table 9: MolmoAct2 learning rates. Shared across conditions and used for both training and adaptation.
Figure 4: Closed-loop manipulation examples. Rollouts of the four concurrently trained policies in PanClean (left), MoveLatteCup (center), and BlockStackKitchen (right). MoveLatteCup is an additional qualitative example and is excluded from the three-task averages. Check marks indicate task success and crosses indicate failure. Among these examples, only the coupled policies succeed.
#CriterionConditionRequires
PanClean
1Reach spongeEnd effector within 0.2 m0.2\,\mathrm{m} of sponge—
2Lift spongeSponge height >0.09 m>0.09\,\mathrm{m}1
3Sponge on panAt least 80% overlap with pan; gripper open2
FoodBussing
1Reach ice creamEnd effector within 0.2 m0.2\,\mathrm{m}—
Appendix
Table 10: PolaRiS rubric criteria. Overlap is measured between horizontal bounding boxes. Placement requires the gripper to be open.
Figure 5: Open-loop driving across training histories. Selected-trajectory minADE on the Public Golden dataset and OOD evaluation dataset. Bars show condition means, and error bars show 95% clip-cluster bootstrap confidence intervals for those means. Lower is better.
Figure 6: Concurrent-training AlpaSim case study. Four conditions in the same traffic scene. The insulated rollout without action-token supervision has an at-fault collision. The displayed rollouts for the other three conditions succeed. This example illustrates a failure mode behind the aggregate results in Table 11 .
More in this topic

VLA Model Training

See all 53 papers

Explore similar work

CardsList
May 6, 2026

From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models

Latent actions serve as an intermediate representation that enables consistent modeling of vision-language-action (VLA) models across heterogeneous datasets. However, approaches to supervising VLAs with latent actions are fragmented and lack a systematic comparison. This work structures the study of latent action supervision from two perspectives: (i) regularizing the trajectory via image-based latent actions, and (ii) unifying the target space with action-based latent actions. Under a unified VLA baseline, we instantiate and compare four representative integration strategies. Our results reveal a formulation-task correspondence: image-based latent actions benefit long-horizon reasoning and scene-level generalization, whereas action-based latent actions excel at complex motor coordination. Furthermore, we find that directly supervising the VLM with discrete latent action tokens yields the most effective performance. Finally, our experiments offer initial insights into the benefits of latent action supervision in mixed-data, suggesting a promising direction for VLA training. Code is available at https://github.com/RUCKBReasoning/From_Pixels_to_Tokens.

Yihan Lin, Haoyang Li, Yang Li +4

School of Information, Renmin University of China, Beijing, China · Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China · Engineering Research Center of Database and Business Intelligence, Beijing, China

Latent Action LearningVision-Language-Action ModelsVLMs for Robotics
AskPDF
Jun 10, 2026

APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies

Vision-Language-Action (VLA) models that couple pretrained Vision-Language Models (VLMs) with continuous action experts have achieved strong manipulation performance, yet generalization to out-of-distribution (OOD) language instructions remains poor. A known challenge is the structural imbalance in VLA data, where language is far less diverse than visual and action content, making policies prone to visual shortcuts. While discrete-action methods mitigate this through vision-language co-training, continuous action experts lack such protection: they start from random initialization and learn entirely from imbalanced data, producing noisy gradients that corrupt the VLM and fail to exploit its language capability. We address this from a Bayesian perspective, factorizing the policy into a language-agnostic Vision-Action (VA) prior and a language-conditioned VLA likelihood, and propose APT, a two-stage training method emphasizing Action expert PreTraining. In Stage 1, the action expert is pretrained as a VA prior on vision-action pairs from a frozen VLM, bypassing the language imbalance. In Stage 2, language tokens are injected through a gated fusion mechanism that integrates VLM features while preserving the learned visuomotor prior. APT applies to mainstream VLA architectures, including the ππ and GR00T-style architectures. Comprehensive experiments validate that APT achieves consistent gains on unseen instructions and compositional tasks. Project Page: https://xukechun.github.io/papers/APT/

Kechun Xu, Zhenjie Zhu, Anzhe Chen +2

Zhejiang University · Zhejiang Humanoid Robot Innovation Center

Visuomotor Policy LearningVision-Language-Action ModelsInstruction Generalization in VLA ModelsVLA Model Training
AskPDF
Jul 2, 2026

VLAFlow: A Unified Training Framework for Vision-Language-Action Models via Co-training and Future Latent Alignment

Vision-language-action models (VLAs) have recently advanced robotic manipulation, yet the effects of different robot-data pre-training paradigms remain difficult to compare because existing models often differ in architecture, data, action space, and evaluation protocol. We present VLAFlow (Vision-Language-Action Flow), a unified flow-matching framework for controlled comparison of VLA training objectives. Using a heterogeneous robot corpus, OXEMix, containing approximately 5,000 hours of data from DROID, OpenX-Embodiment, OpenX-Augmented, and RoboCOIN, we evaluate four paradigms under the same pi0-style architecture, shared VLM backbone, action expert, and 14-dimensional action space: action-only modeling (MindPI), language-supervised co-training (MindLPI), future latent alignment (MindWPI), and their combination (MindLWPI). Experiments on LIBERO, LIBERO-Plus, and SimplerEnv show that action-only pre-training is sensitive to heterogeneous data. In contrast, language supervision helps preserve vision-language generalization, while future latent alignment improves state-transition and action-outcome modeling. By combining both signals, MindLWPI achieves the most stable overall transfer performance across benchmarks. These results suggest a meta-action space view: language and future latent representations provide complementary intermediate constraints that make heterogeneous action supervision smoother and more transferable.

Guoyang Xia, Fengfa Li, Hongjin Ji +4

Li Auto Inc. · School of Artificial Intelligence, Beijing University of Posts and Telecommunications · The Chinese University of Hong Kong, Shenzhen

VLA Model TrainingVision-Language-Action ModelsVLMs for Robotics
AskPDF