Language-Conditioned Robot Manipulation

Follow this topic Ask about this topic

The field studies how robots can interpret instructions such as which object to move and how, then carry them out through visual perception and motor control. Common approaches include vision-language-action policies trained on demonstrations, language-model planners paired with reusable skills, and methods that add geometric grounding, affordances, or predictive world models. Researchers evaluate these systems in simulation benchmarks such as LIBERO and RoboTwin as well as on physical robots, including long-horizon and cluttered tasks. A central challenge is ensuring that success reflects genuine instruction grounding and transfers reliably to new objects, scenes, and robot bodies while remaining safe under uncertainty.

Latest papers

239 papers41 papers in the last four weeks, up 5% on the four weeks before. 0.2% of all new papers.

ManiSoft: Towards Vision-Language Manipulation for Soft Continuum Robotics
May 18, 2026Ziyu Wei, Luting Wang, Chen Gao +2Language-Conditioned Robot Manipulation Robotic RL
Beihang University · Hangzhou Innovation Institute, Beihang University · National University of Singapore
Key-Gram: Extensible World Knowledge for Embodied Manipulation
May 18, 2026Jingjing Fan, Siyuan Li, Botao Ren +1Memory-Augmented VLMs Language-Conditioned Robot Manipulation
Department of Computer Science and Technology Tsinghua University · Department of Automation Tsinghua University
From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data
May 18, 2026Zhiyuan Feng, Qixiu Li, Huizhi Liang +12Language-Conditioned Robot Manipulation Cross-Embodiment Robot Learning
1Tsinghua University · 5Microsoft Research Asia · 2HKUST +4
RoboFlow4D: A Lightweight Flow World Model Toward Real-Time Flow-Guided Robotic Manipulation
May 17, 2026Sixu Lin, Junliang Chen, Huaiyuan Xu +8World Models for Robotics Language-Conditioned Robot Manipulation
School of Data Science, The Chinese University of Hong Kong (Shenzhen) · The Hong Kong Polytechnic University · Shenzhen Loop Area Institute +1
AffordVLA: Injecting Affordance Representations into Vision-Language-Action Models via Implicit Feature Alignment
May 17, 2026Weijie Kong, Zhian Su, Wei Yu +1Language-Conditioned Robot Manipulation Vision-Language-Action Models
Grasp Lab, School of Mechanical Engineering of Zhejiang University, Hangzhou 310027, China · Torch Kernel Co., Ltd., Hangzhou, 310000, P. R. China
How to Instruct Your Robot: Dense Language Annotations Power Robot Policy Learning
May 16, 2026Bosung Kim, Ruiyi Wang, David Acuna +5Robot Policy Learning Robot Skill Learning
University of California, San Diego · 2NVIDIA
Health-Conditioned Vision-Language-Action Models for Malfunction-Aware Robot Control
May 15, 2026Hüseyin Arslan, Özgür ErkentRobotic Control Language-Conditioned Robot Manipulation
*Computer Engineering, Hacettepe University, Ankara, Turkey
Evo-Depth: A Lightweight Depth-Enhanced Vision-Language-Action Model
May 14, 2026Tao Lin, Yuxin Du, Jiting Liu +14Language-Conditioned Robot Manipulation Efficient VLA Models
School of AI, Shanghai Jiao Tong University · King Abdullah University of Science and Technology · SJTU-Quic Robot Joint Lab +1
IntentVLA: Short-Horizon Intent Modeling for Aliased Robot Manipulation
May 14, 2026Shijie Lian, Bin Yu, Xiaopeng Lin +8Robot Policy Learning Language-Conditioned Robot Manipulation
1HUST · 2ZGCA · 4HIT +7
SECOND-Grasp: Semantic Contact-guided Dexterous Grasping
May 13, 2026Han Yi Shin, Heeju Ko, Jaewon Mun +6Robotic Grasping Language-Conditioned Robot Manipulation
1Korea University · University of Texas at Austin · University of Michigan +1
Beyond World-Frame Action Heads: Motion-Centric Action Frames for Vision-Language-Action Models
May 12, 2026Huoren Yang, Jianchao Zhao, Hu Yusong +7Language-Conditioned Robot Manipulation Visuomotor Policy Learning
Xi’an Jiaotong University, Xi’an, Shaanxi, CN · One Robotics, Shenzhen, Guangdong, China · Shenzhen University of Advanced Technology, Shenzhen, Guangdong, CN
Forecast-aware Gaussian Splatting for Predictive 3D Representation in Language-Guided Pick-and-Place Manipulation
May 11, 2026Kaixin Jia, Jiacheng Xu3D Gaussian Splatting 3D Scene Prediction
KTH Royal Institute of Technology, Stockholm, Sweden.
ElasticFlow: One-Step Physics-Consistent Policy with Elastic Time Horizons for Language-Guided Manipulation
May 9, 2026Kewei Chen, Yayu Long, Shuai Li +1Flow Matching Long-Horizon Robotic Manipulation
Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences · Chongqing School, University of Chinese Academy of Sciences · Faculty of Information Technology and Electrical Engineering, University of Oulu, Finland
TriRelVLA: Triadic Relational Structure for Generalizable Embodied Manipulation
May 7, 2026Hanyu Zhou, Chuanhao Ma, Gim Hee LeeLanguage-Conditioned Robot Manipulation Robotic Manipulation
School of Computing, National University of Singapore · School of Artificial Intelligence and Automation, Huazhong University of Science and Technology
Ablation Study of Multimodal Perception, Language Grounding, and Control for Human-Robot Interaction in an Object Detection and Grasping Task
May 1, 2026Zi Tian, Guanting ShenMultimodal Grounding Robotic Grasping
Dalian University of Technology, Dalian, China
Thinking in Text and Images: Interleaved Vision--Language Reasoning Traces for Long-Horizon Robot Manipulation
May 1, 2026Jinkun Liu, Haohan Chi, Lingfeng Zhang +6Long-Horizon Robotic Manipulation Language-Conditioned Robot Manipulation
Tsinghua University · Beijing Institute of Technology · Xiaomi Group
PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations
Apr 30, 2026Yang Zhang, Jiangyuan Zhao, Chenyou Fan +11Reinforcement Learning Long-Horizon Robotic Manipulation
Institute of Artificial Intelligence (TeleAI), China Telecom · Tsinghua University · Shanghai Jiao Tong University +1
Libra-VLA: Achieving Learning Equilibrium via Asynchronous Coarse-to-Fine Dual-System
Apr 27, 2026Yifei Wei, Linqing Zhong, Yi Liu +4Robotic Control Language-Conditioned Robot Manipulation
1Beihang University · 2AgiBot
$M^2$ -VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills
Apr 27, 2026Siyao Xiao, Yuhong Zhang, Zhifang Liu +9Language-Conditioned Robot Manipulation VLM Adaptation
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · PengCheng Laboratory, Shenzhen, China. · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China. +1
Long-Horizon Manipulation via Trace-Conditioned VLA Planning
Apr 23, 2026Isabella Liu, An-Chieh Cheng, Rui Yan +7Long-Horizon Robotic Manipulation Language-Conditioned Robot Manipulation
University of California, San Diego · NVIDIA
ST- $π$ : Structured SpatioTemporal VLA for Robotic Manipulation
Apr 20, 2026Chuanhao Ma, Hanyu Zhou, Shihan Peng +3Language-Conditioned Robot Manipulation Robotics
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · School of Computing, National University of Singapore · School of Computing, Macquarie University
AnchorRefine: Synergy-Manipulation Based on Trajectory Anchor and Residual Refinement for Vision-Language-Action Models
Apr 20, 2026Tingzheng Jia, Kan Guo, Lanping Qian +6Robot Policy Learning Language-Conditioned Robot Manipulation
Beijing University of Technology Beijing, China · Beihang University Beijing, China · Nanjing University of Science and Technology Nanjing, China
$π_{0.7}$ : a Steerable Generalist Robotic Foundation Model with Emergent Capabilities
Apr 16, 2026Physical Intelligence, Bo Ai, Ali Amin +85Robot Policy Learning Robot Skill Learning
AnySlot: Goal-Conditioned Vision-Language-Action Policies for Zero-Shot Slot-Level Placement
Apr 12, 2026Zhaofeng Hu, Sifan Zhou, Qinbo Zhang +4Goal-Conditioned Robotic Manipulation Robotic Manipulation
Stony Brook University, Stony Brook, USA · Carnegie Mellon University, Pittsburgh, USA · MBZUAI, Abu Dhabi, UAE +1
Text-Guided 6D Object Pose Rearrangement via Closed-Loop VLM Agents
Apr 10, 2026Sangwon Baik, Gunhee Kim, Mingi Choi +1Language-Conditioned Robot Manipulation Object Pose Estimation
Seoul National University · RLWRLD
BLaDA: Bridging Language to Functional Dexterous Actions within 3DGS Fields
Apr 9, 2026Fan Yang, Wenrui Chen, Guorun Yan +7Robotic Manipulation Robotic Grasping
School of Artificial Intelligence and Robotics, Hunan University, Changsha 410012, China. · National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410082, China. · Key Laboratory of Advanced Manufacturing Technology of the Ministry of Education, Guizhou University, Guiyang 550025, China.
A Physical Agentic Loop for Language-Guided Grasping with Execution-State Monitoring
Apr 8, 2026Wenze Wang, Mehdi Hosseinzadeh, Feras DayoubRobotic Manipulation Language-Conditioned Robot Manipulation
Australian Institute for Machine Learning (AIML), Adelaide University, Australia.
E-VLA: Event-Augmented Vision-Language-Action Model for Dark and Blurred Scenes
Apr 6, 2026Jiajun Zhai, Hao Shi, Shangwei Guo +2Event-Based Vision VLM Robustness
Zhejiang University · Ant Group · Hunan University
Veo-Act: Enhancing VLA Policies with Frontier Video Models
Apr 6, 2026Zhongru Zhang, Chenghan Yang, Qingzhou Lu +4Robotic Control Language-Conditioned Robot Manipulation
Tsinghua University
Hybrid Framework for Robotic Manipulation: Integrating Reinforcement Learning and Large Language Models
Mar 31, 2026Md Saad, Sajjad Hussain, Mohd SuhaibRobotic Manipulation Language-Conditioned Robot Manipulation
Department of Mechanical Engineering, Jamia Millia Islamia, New Delhi, India · School of Architecture, Technology and Engineering, University of Brighton, Brighton, UK

May 18, 2026J/K move · Enter open · S save

ManiSoft: Towards Vision-Language Manipulation for Soft Continuum Robotics

Ziyu Wei, Luting Wang, Chen Gao, Li Wen+1

Beihang University · Hangzhou Innovation Institute, Beihang University · National University of Singapore

Language-Conditioned Robot Manipulation Robotic RL Robot Manipulation Benchmarks Continuum Robot Control VLMs for Robotics Soft Robotics

Open paper AskPDF arXiv

Most existing vision-language manipulation research targets rigid robotic arms, whose fixed morphology limits adaptability in cluttered or confined spaces. Soft robotic arms offer an appealing alternative due to their deformability, but confront challenges such as unreliable proprioception and distributed low-level actuation. To investigate these challenges, we introduce \ManiSoft, a benchmark for vision-language manipulation with soft arms. ManiSoft features a tailored simulator that couples realistic soft-body dynamics with contact-rich interactions via an elastic force constraint. On this basis, ManiSoft defines four tasks, each highlighting distinct aspects of deformable control, from basic end-effector coordination to obstacle avoidance. To support policy training and evaluation, \ManiSoft{} includes an automated pipeline that generates

6{,}300

diverse scenes and corresponding expert trajectories. To produce high-quality trajectories at scale, we first employ a high-level planner to decompose each task into a sequence of waypoints, followed by a low-level reinforcement learning policy that generates torque commands to track waypoints. Benchmarking three representative policy models shows relatively promising results in clean scenes but substantial performance drop under randomization. Visualization analysis indicates that failures stem primarily from inaccurate visual estimation of proprioceptive state and limited exploitation of deformability for adaptive obstacle avoiding. We anticipate ManiSoft to serve as a valuable testbed, bridging the gap between rigid and soft arms in the context of vision-language manipulation. Out codes and datasets are released at https://buaa-colalab.github.io/ManiSoft.