One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning

Yiwei Liu; Luwei Yang; Shunbo Lei

cs.ROSep 4, 2026

One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning

Authors: Yiwei Liu, Luwei Yang, Shunbo Lei

Organizations: School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China · Shenzhen Research Institute of Big Data (SRIBD), Shenzhen 518172, China

Abstract

Changes in natural-language instructions can directly alter the behavior ultimately executed by a robot, but such changes do not always imply that the task itself has changed. We propose One Word, Different Action, a language-conditioned executable decision benchmark based on real-robot physical decision anchors. It evaluates robot behavioral responses under task-preserving and task-changing conditions and examines joint reasoning over multiple task constraints. The benchmark uses decision invariance and decision sensitivity to measure action stability when task semantics are preserved and correct decision updates when task requirements change. Matched atomic constituents directly relate correct handling of individual constraints to the corresponding joint executable decision. Experiments show that atomic-task correctness does not sufficiently represent compositional reasoning ability: even when every constituent constraint is handled correctly, a model may still produce an erroneous final action decision on the corresponding compositional task. These results indicate that the key challenge in language-conditioned robot reasoning lies not only in understanding individual task requirements, but also in integrating multiple correctly handled constraints into consistent executable behavior.

Figures & tables

Panel of Figure 1 — Fig. 2: Atomic operators and their composition over a shared executable action space. $N$ , $E$ , $Q$ , and $T$ denote action inhibition, set exclusion, cardinality, and temporal ordering, respectively. The current object-centric compositional subset jointly applies $N$ , $E$ , and $Q$ as $E+N$ , $E+Q$ , $N+Q$ , and $E+N+Q$ , while $T$ is evaluated on atomic action-sequence tasks.

Type	Representative instruction
$E{+}N$	“Inspect all listed objects excluding object_2; do not inspect object_3.”
$E{+}Q$	“Inspect exactly the following 2 objects: object_3 and object_4, excluding object_2.”
$N{+}Q$	“Inspect exactly the following two objects: object_2 and object_3, but do not inspect object_4.”
$E{+}N{+}Q$	“Choose exactly 1 object: object_4; exclude object_2 and never inspect object_3.”

TABLE I: Representative compositional instructions sampled from the evaluated benchmark.

Model	Checkpoint identifier	Modality	Interface	Max new tokens	Sampling
GLM-5-FP8	GLM-5-FP8	Text	Local inference API	512	Disabled
Qwen2.5-7B-Instruct	Qwen/Qwen2.5-7B-Instruct	Text	Native instruct	512	Disabled
Mistral-7B-Instruct-v0.3	mistralai/Mistral-7B-Instruct-v0.3	Text	Native instruct	512	Disabled
OLMo-2-1124-7B-Instruct	allenai/OLMo-2-1124-7B-Instruct	Text	Native instruct	512	Disabled

TABLE II: Model and decoding configuration.

Model	Atomic	Composite	Cond. Comp.	Invariance	Sensitivity
GLM-5-FP8	0.994	0.949	0.818	0.961	0.843
Qwen2.5-7B-Instruct	0.844	0.645	1.000	0.657	0.471
Mistral-7B-Instruct-v0.3	0.978	0.836	0.610	0.755	0.696
OLMo-2-1124-7B-Instruct	0.878	0.828	0.929	0.814	0.667

TABLE III: Atomic and compositional accuracy, conditional composition success, decision invariance, and decision sensitivity.

Model	K=2	K=3	Cond. Comp.
GLM-5-FP8	0.936	1.000	0.818
Qwen2.5-7B	0.720	0.338	1.000
Mistral-7B	0.890	0.613	0.610
OLMo-2-7B	0.948	0.338	0.929

TABLE IV: Composition-depth performance and conditional composition success.

Model	E+N	E+Q	N+Q	E+N+Q
GLM-5-FP8	0.925	1.000	0.911	1.000
Qwen2.5-7B	0.050	1.000	0.905	0.338
Mistral-7B	0.763	0.963	0.917	0.613
OLMo-2-7B	0.963	0.975	0.929	0.338

TABLE V: Accuracy by compositional constraint type.

Language-Conditioned Robot Control

Language-conditioned robot control connects natural-language instructions to grounded, safe actions across tasks and robot embodiments.

See all 73 papers

A Brain-inspired Hierarchical Framework for Zero-Shot Robot Task Reasoning and Execution
Sep 5, 2026Guangming Wang, Pengfei Ye, Qizhen Ying +7Robotics Zero-Shot Robotic Manipulation
Department of Engineering, University of Cambridge, Cambridge, U.K. · Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA · Department of Mechanical and Aerospace Engineering, Hong Kong University of Science and Technology, Hong Kong, China +2
$R^3$ : Training Robots to Reason in Natural Language via Reinforcement Learning
Aug 26, 2026Lehong Wu, Yuxiao Qu, Zheyuan Hu +4VLM Reasoning Language-Conditioned Robot Manipulation
Carnegie Mellon University
MIND: Instruction-Conditioned Incremental Grounding for Robots
Oct 13, 2026Mohamed Abouagour, Junhyun Nam, Wonse Jo +1Language-Conditioned Robot Planning Robot Task Planning
Indiana University Bloomington, USA. · Incheon National University, Republic of Korea.

Sep 5, 2026J/K move · Enter open · S save

A Brain-inspired Hierarchical Framework for Zero-Shot Robot Task Reasoning and Execution

Guangming Wang, Pengfei Ye, Qizhen Ying, Yixiong Jing+6

Department of Engineering, University of Cambridge, Cambridge, U.K. · Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA · Department of Mechanical and Aerospace Engineering, Hong Kong University of Science and Technology, Hong Kong, China+2

Robotics Zero-Shot Robotic Manipulation Robot Task Planning Language-Conditioned Robot Manipulation

Open paper AskPDF arXiv

Robots that follow open-ended language instructions need to connect semantic intent to visual scene understanding, geometric feasibility, object states, and physical interaction conditions. End-to-end Vision-Language-Action policies have improved cross-task generalization, but they typically map visual and language inputs directly to robot actions, leaving limited explicit structure for long-horizon decomposition, physical verification, and recovery. We present \method, a zero-shot hierarchical framework functionally inspired by the division of roles in the human brain, comprising visual perception and state inference, language grounding and action-sequence generation from a shared atomic action library, cost-based plan selection, and real-robot execution and verification. The framework grounds commands in explicit object states, composes reusable atomic actions into task-conditioned sequences, ranks alternative sequences by execution cost, and verifies intermediate physical outcomes from refreshed observations. In the evaluation, \method{} completes 10/10 clean board trials, 10/10 pick-and-place trials, and 4/5 pyramid stacking trials for both the flat and irregular initial-layout conditions; the corresponding mean task progress is

99.03\%

100.00\%

, and

96.67\%

respectively. Across all evaluated conditions, \method{} achieves higher success rates than ReKep, Dream2Flow, and

π_{0.5}

benchmarks, demonstrating the effectiveness of combining explicit object-state reasoning, compositional atomic actions, cost-based plan selection, and closed-loop execution verification.

Abstract

Language-Conditioned Robot Control

Explore similar work

A Brain-inspired Hierarchical Framework for Zero-Shot Robot Task Reasoning and Execution

R3R^3R3: Training Robots to Reason in Natural Language via Reinforcement Learning

MIND: Instruction-Conditioned Incremental Grounding for Robots

$R^3$ : Training Robots to Reason in Natural Language via Reinforcement Learning