cs.ROMar 13, 2026

Language-Grounded Decoupled Action Representation for Robotic Manipulation

Authors: Wuding Weng, Tongshu Wu, Liucheng Chen, Siyu Xie, Zheng Wang, Xing Xu, Jingkuan Song, Heng Tao Shen

Organizations: School of Computer Science and Technology, Tongji University, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The heterogeneity between high-level vision-language understanding and low-level action control remains a fundamental challenge in robotic manipulation. Although recent methods have advanced task-specific action alignment, they often struggle to generate robust and accurate actions for novel or semantically related tasks. To address this, we propose the Language-Grounded Decoupled Action Representation (LaDA) framework, which leverages natural language as a semantic bridge to connect perception and control. LaDA introduces a fine-grained intermediate layer of three interpretable action primitives--translation, rotation, and gripper control--providing explicit semantic structure for low-level actions. It further employs a semantic-guided soft-label contrastive learning objective to align similar action primitives across tasks, enhancing generalization and motion consistency. An adaptive weighting strategy, inspired by curriculum learning, dynamically balances contrastive and imitation objectives for stable and effective training. Extensive experiments on simulated benchmarks (LIBERO and MimicGen) and real-world demonstrations validate that LaDA achieves strong performance and generalizes effectively to unseen or related tasks.

Figures & tables

Figure 1: Comparison of representative paradigms in vision-language-action learning. Existing approaches either entangle perception and control in end-to-end VLAs, rely on latent action embeddings without explicit semantics, or use discrete language-conditioned primitives that lack fine-grained motion grounding. Our LaDA framework bridges this gap by leveraging language as a semantic bridge to decouple and align vision, language, and action representations through soft-label contrastive learning, enabling semantically grounded and generalizable manipulation.
Figure 2: Overview of the proposed LaDA framework. LaDA leverages language as a semantic bridge to connect high-level vision–language understanding with low-level control. It decomposes continuous 7-DoF end-effector actions into interpretable primitives—translation, rotation, and gripper control—and encodes them within a shared semantic embedding space. Semantic-guided soft-label contrastive learning aligns multimodal representations across tasks, while an adaptive weighting strategy dynamically balances imitation and contrastive objectives, enabling efficient cross-task transfer and robust generalization.
Figure 3: Example tasks from the simulation environments. ( Top ) Sample tasks from LIBERO benchmark [ 30 ] , illustrating language-conditioned manipulation scenarios. ( Bottom ) Example tasks from MimicGen [ 34 ] , demonstrating contact-rich manipulation skills
ModelParamsLIBERO Success Rate (%)
SpatialObjectGoalLongAvg
UniACT [ 58 ]0.5B65.078.068.047.064.5
LAPA [ 55 ]7B73.874.658.855.465.7
DP [ 8 ]147M78.392.568.350.572.4
Octo [ 13 ]93M78.985.784.651.175.1
OpenVLA [ 23 ]7.5B84.788.479.253.776.5
Table 1: Comparison of success rates on LIBERO. LaDA achieves the highest overall performance and excels on long-horizon tasks, highlighting the benefit of language-grounded soft-label contrastive learning.
 MimicGen Simulation Success Rates
ModelC_D0C_D1S_D0S_D1ST_D0ST_D1T_D0TPA_D0TPA_D1Avg.
OpenVLA [ 23 ]42%18%84%86%36%20%20%28%8%38%
Task-conditioned66%24%88%68%30%6%74%20%0%42%
Subgoal-conditioned76%26%88%74%24%6%78%20%2%44%
Motion-conditioned68%32%92%84%38%16%58%30%4%47%
Subgoal self-reflection80%32%88%78%32%6%80%34%2%48%
Table 2: Simulation success rates of LaDA and baseline methods on the nine MimicGen tasks. We use the following abbreviations: C (Coffee), S (Stack), ST (StackThree), T (Threading), and TPA (ThreePieceAssembly), with D0 and D1 indicating different demonstration subsets. LaDA achieves the highest average performance, with notable gains on multi-step and long-horizon tasks.
Figure 4: Generalization evaluation of LaDA on novel and semantically related tasks.
MethodsSpatialObjectGoalLongAverage
w/o SCL79.282.876.663.475.5
w/o AW93.694.487.274.487.4
LaDA95.299.293.686.493.6
Table 3: Ablation study of LaDA components on the LIBERO benchmark. Removing the proposed soft-label contrastive learning (SCL) or the adaptive weighting mechanism (AW) leads to clear performance degradation, verifying the importance of fine-grained semantic alignment and balanced optimization in LaDA.
Figure 5: Comparison of average success rates on MimicGen tasks (Stack, StackThree, Threading) under single-task and multi-task training. LaDA consistently outperforms CLIP-RT, with larger gains in the multi-task setting.
Figure 6: t-SNE visualization of learned action embeddings. (a,b) Embedding distributions without and with LaDA, where LaDA yields more compact and semantically structured clusters. (c,d) t-SNE projections of translation and rotation primitives, showing that actions from different tasks exhibit overlapping patterns, indicating consistent cross-task motion semantics.
Figure 7: Real-world setup with a 7-DoF Franka Panda and a third-person static RGB camera (left). Representative rollout snapshots (right) show successful pick-and-place execution.

Explore similar work

CardsList

  1. DAM-VLA: A Dynamic Action Model-Based Vision-Language-Action Framework for Robot Manipulation

    Mar 1, 2026Xiongfeng Peng, Jiaqian Yu, Dingzhe Li +8Robotic ManipulationVision-Language-Action Models

    Advanced Research Lab, Samsung R&D Institute China-Beijing (SRCB), China · Samsung AI Center, DS Division, South Korea · Hanyang University ERICA, South Korea

  2. AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding

    Jun 4, 2026Qize Yu, Jiadi You, Yuran Wang +10Language-Conditioned Robot ManipulationRobotic Manipulation

    Peking University · Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong +1

  3. Semantic Anchoring for Robotic Action Representations

    Jul 15, 2026Yuan Xu, Youheng Shi, Chengyang Li +2Generalization in Robotic ManipulationVision-Language-Action Models

    Peking University · Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University