Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
Organizations: Technical University of Munich, Munich, Germany · Corporate Research, Robert Bosch GmbH, Renningen, Germany · Institute for Artificial Intelligence, University of Stuttgart, Stuttgart, Germany · University of California Berkeley, USA · Microsoft, Switzerland · Google DeepMind, USA & UK · Carnegie Mellon University, USA · Imperial College London, UK · Princeton University, USA · University of Southern California, USA · Sun Yat-sen University, Guangzhou, China · University of Michigan, USA · The State Key Laboratory for Novel Software Technology, Nanjing University, Suzhou, China
Abstract
Language-conditioned robot manipulation is an emerging field aimed at enabling seamless communication and cooperation between humans and robotic agents by teaching robots to comprehend and execute instructions conveyed in natural language. This interdisciplinary area integrates scene understanding, language processing, and policy learning to bridge the gap between human instructions and robot actions. In this comprehensive survey, we systematically explore recent advancements in language-conditioned robot manipulation. We categorize existing methods based on the primary ways language is integrated into the robot system, namely language for state evaluation, language as a policy condition, language for cognitive planning and reasoning, and language in unified vision-language-action models. Specifically, we further analyze state-of-the-art techniques from five axes of action granularity, data and supervision regimes, system cost and latency, environments and evaluations, and task specification. Additionally, we highlight the key debates in the field. Finally, we discuss open challenges and future research directions, focusing on potentially enhancing generalization capabilities and addressing safety issues in language-conditioned robot manipulators.