Instruction Generalization in VLA Models

VLA: Vision-Language-Action

Vision-language-action models must connect a command to what they see and then choose actions, even when wording, objects, locations, or task combinations differ from training. Researchers test this with paraphrase and language-variation benchmarks such as LIBERO-Para, as well as held-out object–location pairings and real-robot tasks. Approaches include synthetic instruction augmentation, counterfactual training, explicit grounding and subgoal planning, and test-time guidance. These methods can improve robustness, but models still show brittle language grounding and visual shortcuts; greater responsiveness to instructions does not always mean better task success or transfer to harder commands.

Latest papers

40 papers18 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Cartender: Verifiable Language-Conditioned Driving by Learning the Drives Not Taken
Oct 19, 2026Zhiwen Chen, Benjin Zhu, Yan Xie +1Instruction Generalization in VLA Models Trajectory Generation
The Hong Kong University of Science and Technology (hkust-gz.edu.cn) · Li Auto Inc. (lixiang.com) · The Hong Kong University of Science and Technology (ust.hk)
Robotic Photography System based on Vision-Language-Action Model
Oct 19, 2026Song-Li Wu, Chubin Zhang, Wenbo Ding +1Vision-Language-Action Models Instruction Generalization in VLA Models
Shenzhen International Graduate School, Tsinghua University
Do VLAs Really Follow Instructions? Diagnosing How-Level Instruction Following in Vision-Language-Action Models
Oct 15, 2026Sangwu Park, Wonjoong Kim, Yeonjun In +1Vision-Language-Action Models VLA Model Training
KAIST
Paraphrase Consensus Is Not Self-Consistency: Instruction Reliance and Robustness in VLA Policies
Oct 14, 2026Hanlong Li, Sihan Tan, Takeshi Ashizawa +2Robustness of VLA Models Instruction Generalization in VLA Models
Institute of Science Tokyo, Tokyo, Japan
Same Goal, Different Words: Spatial Instruction Sensitivity in World-Action Models
Oct 12, 2026Ali-Adeeb Abbas, Esmaeil Seraj, Behrad Toghi +1World Model Evaluation Vision-Language-Action Models
RIVeR Laboratory, Northeastern University, Boston, MA, USA. · General Motors Robotics, Mountain View, CA, USA.
Axiom-0: Hierarchical Self-Refinement for Vision-Language-Action Policies
Oct 9, 2026Renbin Shu, Jing Yang, Xiangyuan You +7Vision-Language-Action Models VLA Model Training
4Axiom Robotics Team
VioLA: Learning Generalist Humanoid Control Policies from Human Data
Oct 8, 2026Mert Albaba, Jens Beißwenger, Anna Manasyan +6Humanoid Robot Control Robot Imitation Learning
Vesoma · ETH Zürich · MPI-IS +2
Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models
Oct 7, 2026Mikey Watts, Yuchen CuiRobustness of VLA Models Instruction Generalization in VLA Models
Independent Researcher · University of California, Los Angeles
Speaking the Navigator's Language: Trajectory-Grounded Instruction Translation for Frozen Aerial VLN Agents
Oct 7, 2026Xi Chen, Zhe Liu, Xiaogang Xu +8Aerial VLN Vision-Language Navigation
Zhejiang University · Zhejiang Lab
Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization
Oct 7, 2026Haoru Li, Jinmei Liu, Zhiyong Wang +5Robot Policy Generalization RL Fine-Tuning
Nanjing University · Harbin Institute of Technology (Shenzhen) · Australian National University +1
TempoBridge: Language-Guided Tempo Control for Vision-Language-Action Policies
Oct 7, 2026Yeonseo Lee, Hyosup Shin, Guebin Hwang +1Language-Conditioned Robot Control Vision-Language-Action Models
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea.
OGAM: Connecting Systematic Testing to Runtime Assurance through Object-Grounded Attention Monitoring for VLA Policies
Oct 5, 2026Haki Darwish, Xiangyu Yin, Changwen Li +3Failure Detection for VLA Models Instruction Generalization in VLA Models
Carl von Ossietzky University of Oldenburg, Germany · Huadian Electric Power Research Institute, China · Institute of Software, Chinese Academy of Science, China +1
MixVLA: Adaptive Mixing of Non-Invariant Information for Generalizable Vision-Language-Action Models
Oct 2, 2026Pingrui Zhang, Yu Zhang, Pengyuan Wu +8Vision-Language-Action Models Robustness of VLA Models
Fudan University · Shanghai AI Laboratory · Harbin Institute of Technology, Shenzhen, China +2
Same Scene, Different Task: Skill Alignment for Compositional Generalization in VLAs
Sep 30, 2026Taegeun Yang, Youngju Na, Yoonki Cho +1Vision-Language-Action Models Counterfactual Data Augmentation
Korea Advanced Institute of Science and Technology (KAIST)
When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models
Sep 30, 2026Hung-Jen Chen, Yu-Hsun Hou, Yan-Hong Chen +4Robotic Manipulation Vision-Language-Action Models
National Tsing Hua University · National Taiwan University
Correcting WHERE, Preserving HOW: Compositional Generalization for Vision-Language-Action Models via Referential Guidance
Sep 29, 2026Yanyan Zhang, Disheng Liu, Xinpeng Li +8Vision-Language-Action Models Test-Time Adaptation of VLA Models
Case Western Reserve University Cleveland, OH, USA
Disentangling Spurious Correlations in Vision-Language-Action Models via Predicting Domain-Invariant Latent Lookahead
Sep 29, 2026Junghyun Kim, Ngseo Kim, ChungWoo Lee +7Domain Generalization Vision-Language-Action Models
OpenMind, San Francisco, CA, USA · Seoul National University, Seoul, Korea · Hyundai Motors, Korea +2
LexiconVLA: Learning Reusable Atomic Action Codebooks for Unseen Tasks
Sep 29, 2026Zeming Wei, Jianheng Ye, Xinshuai Song +3Visuomotor Policy Learning Robotic Manipulation
Sun Yat-sen University · X-Era AI Lab · Pengcheng Laboratory
Reasoning Without Inference Cost: Latent Semantic Scaffolding for Robot VLA Policies
Sep 4, 2026Andrew Ting Yan Li, Zhuo Li, Zhelin Yang +3Reasoning for VLA Models Vision-Language-Action Models
Collaborative and Versatile Robots (CLOVER) Laboratory, T-Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong
DeicticVLA: Unifying Instruction Modes Based on Language and Deictic Gestures in a Single VLA
Aug 28, 2026Kango Yanagida, Tatsuya Aoki, Yuichiro Yoshikawa +1Vision-Language-Action Models Instruction Generalization in VLA Models
Dept. of Systems Innovation, Graduate School of Engineering Science, The University of Osaka, Japan. · International Research Center for Neurointelligence, The University of Tokyo, Japan.
MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalization
Aug 26, 2026Zaibin Zhang, Junlan Xiao, Zhongbo Zhang +11Vision-Language-Action Models Instruction Generalization in VLA Models
Dalian University of Technology · Shanghai Jiao Tong University · The Chinese University of Hong Kong, Shenzhen +4
Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation
Aug 19, 2026Prachi Garg, Steve Xing, Prahit Yaugand +2Robot Learning Vision-Language-Action Models
University of Illinois Urbana-Champaign
GigaBrain-0.7: Scaling Embodied Foundation Models to Emergent Capabilities with a Three-System Architecture
Aug 16, 2026GigaBrain Team, Angen Ye, Axiang Sun +56Robot Foundation Models Vision-Language-Action Models
AWM-VLA: AlignedWorld Modeling for Efficient and Explainable Vision-Language-Action Policies
Aug 15, 2026An Lanji, Dawei Liu, Jin Li +3World Models for VLA Models Vision-Language-Action Models
University of Electronic Science and Technology of China, Chengdu, China.
StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models
Aug 12, 2026Siyu Xu, Yunke Wang, Zijian Wang +6Reasoning for VLA Models Continual Learning for VLA Models
StellarEdge AI Technical Team
Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models
Aug 3, 2026Zhaokai Yin, Zhipeng ZhangVision-Language Grounding Vision-Language-Action Models
1AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · 2Research Lab, Anyverse Dynamics
Pelican-VLA 0.5: Attending Before Acting Benefits Generalization
Jul 7, 2026Zeyuan Ding, Wenhai Liu, Yang Xu +6Next-Action Prediction Generalization in Robotic Manipulation
Beijing Innovation Center of Humanoid Robotics (X-Humanoid)
APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
Jun 10, 2026Kechun Xu, Zhenjie Zhu, Anzhe Chen +2Visuomotor Policy Learning Vision-Language-Action Models
Zhejiang University · Zhejiang Humanoid Robot Innovation Center
When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models
Jun 10, 2026Xuan Dong, Zhe Han, Tianhao Niu +2VLM Robustness Multilingual VLM Evaluation
Harbin Institute of Technology
See Less, Specify More: Visual Evidence Budgets for Generalizable VLAs
Jun 1, 2026Yueh-Hua Wu, Tatsuya Matsushima, Kei OtaVisuomotor Policy Learning Vision-Language-Action Models
AIRoA

Oct 19, 2026J/K move · Enter open · S save

Cartender: Verifiable Language-Conditioned Driving by Learning the Drives Not Taken

Zhiwen Chen, Benjin Zhu, Yan Xie, Jun Ma

The Hong Kong University of Science and Technology (hkust-gz.edu.cn) · Li Auto Inc. (lixiang.com) · The Hong Kong University of Science and Technology (ust.hk)

Instruction Generalization in VLA Models Trajectory Generation VLMs for Autonomous Driving Counterfactual Data Augmentation

Open paper AskPDF arXiv

Every driving scene is logged once, but driving is one-to-many: the scene determines which behaviors are feasible, while an instruction selects among them. A log records only the action taken, leaving no direct supervision for alternative feasible actions or instructions the scene cannot support. Consequently, a policy that ignores language and one that follows it without regard to scene constraints can fit the same data. We present Cartender, which learns from the drives not taken. Counterfactual Branching constructs matched and mismatched instruction-action pairs from real recordings without a simulator: a vision-language model proposes alternative futures and their selecting instructions, while unsupported instructions provide negative labels. A learned scene-language compatibility score regulates language conditioning in trajectory generation, and a reward-trained language residual re-ranks generated candidates over a scene-based score. The branches are used for training; instruction following is evaluated by swapping instructions under identical recorded inputs and measuring behavior rather than agreement with generated targets. On WOD-E2E, the best of four proposals achieves an RFS of 8.642 across 3,603 feasible instruction conditions. Across 8,974 admitted proposals, mean RFS is 8.203 versus 7.906 for matched logged trajectories, and 67% score higher than their logged counterpart. Compared with matched-only imitation using the same backbone, Cartender improves same-scene paired instruction success from 23.67% to 57.00% and RFS from 6.32 to 7.10; its gate reduces infeasible-instruction execution from 53.36% to 14.18%.