Robots following persistent language instructions must decide which cached room judgments remain valid as their environment changes. We present MIND (Maintenance via INstruction-conditioned Dependencies), a framework for maintaining target-room decisions without re-evaluating every room after each event. A language model compiles each instruction into dependencies for semantic qualification and target selection. The controller intersects semantic dependencies with event footprints, refreshes affected judgments, and updates reachability and selection separately. For fixed deterministic evaluation, planning, and selection, conservative dependency and footprint coverage preserves the decisions of full recomputation. On 240 five-event ProcTHOR sequences with an exact benchmark predicate, MIND reduces mean semantic evaluations from 30.6 to 11.2 while preserving all 1,200 post-event decisions. Frozen-evaluator experiments with Phi-4 and Qwen3-14B also preserve all 1,200 decisions, without improving the evaluators' underlying task accuracy. In 32 closed-loop AI2-THOR episodes using exact-predicate qualification and metadata-derived footprints, MIND matches the final target decision of full recomputation in every episode. Unitree G1 experiments demonstrate target reuse and reselection across room transitions without additional semantic evaluations under MIND, and abstention after a barrier and an object removal at 4 evaluations against 15 to 18. Additional materials are available at https://mind-icra27.github.io/.
Figures & tables
Fig. 1: An overview of “Bring me the nearest mug.” At t0 every room is judged and cached. Moving ( t1 ) changes no room: judgments are reused, only the target is re-chosen. Removing the kitchen mug ( t2 ) refreshes one judgment. Full recomputation rejudges all four each time.
Fig. 2: Architecture for “bring me the nearest mug.” (A) Once per instruction, the compiler separates variables needed to judge each room from those needed only to choose among qualifying rooms. (B) After each event, the detector reports what changed and which rooms may be affected. (C) Their intersection fixes which room judgments to recheck; position and topology changes also update reachability. (D) The controller reuses untouched judgments and reselects a reachable target whenever the feasible set is recomputed or a reported change intersects either compiled dependency slot.
policy
episode acc.
sem. evals
inv. recall
full recomputation
1.000 [.984,1]
30.6±9.7
1.000
static inventory-only
0.479 [.417,.542]
8.1±1.6
0.703
universal semantic signature
1.000 [.984,1]
23.4±6.4
1.000
reference variables, no stage split
1.000 [.984,1]
14.1±6.6
1.000
reference stage-specific
1.000 [.984,1]
11.2±3.6
1.000
MIND, Phi-4 ( K=2 )
1.000 [.984,1]
11.2±3.6
1.000
TABLE I: Maintenance ablation on 240 five-event robot episodes with the exact benchmark predicate. Episode accuracy with 95% Wilson intervals; semantic evaluations per sequence, mean ± SD.
compiler, K
exact sig.
containment
inflation
non-containing (total)
gpt-oss-120b, K=1
0.935
1.000
1.00
0
Phi-4, K=1
0.727
1.000
1.04
0
Phi-4, K=2 (deployed)
0.667
1.000
1.06
0
Qwen3-14B, K=1
0.454
1.000
1.08
0
Qwen3-14B, K=2
0.353
1.000
1.11
0
Qwen3-8B, K=1
0.372
0.988
1.12
5
TABLE II: Dependency-compiler evaluation on the frozen 69-instruction held-out set.
compiler signature
interface
agreement
episode acc.
evals
Qwen3-8B, one sample
reference
0.992 ( 1190/1200 )
0.329
10.4
Qwen3-8B, one sample
compiled
1.000
0.329
10.4
Qwen3-8B, K=2 union
reference
1.000
0.342
11.5
Phi-4, K=2 (main)
both
1.000
0.342
11.2
TABLE III: Interface ablation: a missed dependency and its conservative repair (Phi-4 judge, 1200 decisions).
Fig. 3: Removal localization in ProcTHOR validation house 571. (A, B) Matched views before and after mug removal. (C) The candidate footprint contains the affected bedroom, the only room refreshed here. (D) Reselection identifies the kitchen mug; the living room remains unreachable.
Fig. 4: Unitree G1 hardware validation. Left: the robot, with the 3D lidar and the RGB-D camera used for localization and detection. Centre: ceiling view of the four-zone laboratory, chairs in A and C and a backpack in the start zone D, with the path taken in one run of scenario 2, the nearest thing to sit on . Insets: the robot’s camera in each zone; the three moments of the run carry the target and the evaluation count from the module log. Three initial evaluations select C, one hop away, and the robot walks there and arrives. When the operator then drives it into A, the target re-ranks to A on position alone; position is a selection dependency for this request, so no room is re-judged.
Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim
Mengao Zhao, Ziang Li, Chaodong Huang +15
Horizon Robotics · Southeast University · Huazhong University of Science and Technology +1
Large Reasoning Language Models (LRLMs) enable multi-step reasoning for robotic task planning, but continued reasoning can overwrite valid intermediate plans or leave constraint violations unresolved, reducing planning reliability and wasting inference-time computation. We develop an inference-time monitor that exposes and verifies intermediate plans without disrupting the original decoding trajectory. Building on this monitor, we propose SafeInferCom, a formal verifier-guided framework that preserves valid intermediate plans and directs error correction during generation. Experiments across multiple LRLMs and planning domains reveal reasoning-response inconsistency and limited self-correction under one-shot inference. SafeInferCom improves planning success and accelerates error correction relative to one-shot inference. When combined with iterative refinement, it further improves success while reducing token usage compared with refinement alone. We additionally evaluate SafeInferCom in VirtualHome and provide a real-world robotic-arm demonstration.
Weizhe Xu, Jialiang Fan, Mengyu Liu +1
University of Notre Dame · Washington State University
Changes in natural-language instructions can directly alter the behavior ultimately executed by a robot, but such changes do not always imply that the task itself has changed. We propose One Word, Different Action, a language-conditioned executable decision benchmark based on real-robot physical decision anchors. It evaluates robot behavioral responses under task-preserving and task-changing conditions and examines joint reasoning over multiple task constraints. The benchmark uses decision invariance and decision sensitivity to measure action stability when task semantics are preserved and correct decision updates when task requirements change. Matched atomic constituents directly relate correct handling of individual constraints to the corresponding joint executable decision. Experiments show that atomic-task correctness does not sufficiently represent compositional reasoning ability: even when every constituent constraint is handled correctly, a model may still produce an erroneous final action decision on the corresponding compositional task. These results indicate that the key challenge in language-conditioned robot reasoning lies not only in understanding individual task requirements, but also in integrating multiple correctly handled constraints into consistent executable behavior.
Yiwei Liu, Luwei Yang, Shunbo Lei
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China · Shenzhen Research Institute of Big Data (SRIBD), Shenzhen 518172, China