A robot asked to "place the cup near the red plate or the blue plate" may reach the centroid between them and appear geometrically successful, while satisfying neither disjunct of the instruction. This silent semantic failure exposes a structural limitation of language-conditioned robot policies: representations that collapse a disjunctive instruction into a single connected set cannot preserve all feasible modes, and planners that commit to one action degrade under run-time mode uncertainty. We address this limitation with CompCPZ, a sound algebraic layer that language-conditioned learning systems wrap to recover multi-modal disjunctive representation, recursively composing per-primitive constrained polynomial zonotope enclosures along the language parse tree with distribution-free conformal coverage and sub-millisecond runtime. On a closed-loop ManiSkill3 tabletop-manipulation benchmark, CompCPZ outperforms convex set baselines, multi-peak decoders, and a zero-shot vision-language-action model (1,900/1,918 paired wins, p << 10^(-30)); the same compiler also transfers without retuning to planar real-robot trials on a Unitree Go2 quadruped under motion capture. These results suggest that compositional language grounding should be evaluated not only by reaching a decoded target, but by whether the represented feasibility set preserves the connected-component structure of the user's intent.
Figures & tables
Figure 1: CompCPZ compiles a natural-language instruction into a sound k -component feasibility set via bottom-up Boolean composition of two-sided per-primitive enclosures, with run-time mode switching for handling occlusion of the nearest mode.
Figure 3: Sound convex enclosures structurally fail on disjunctive instructions ( Corollary 12 ). Four ManiSkill3 task families, each pairing a SAPIEN render with a top-down inset. The CompCPZ target (green halo) lands inside a true mode; the convex AABB target (red halo) lands in the inter-mode gap.
Method
In-GT-mode
CompCPZ
1.000
AABB-union (bounding)
0.928
AABB-union (inscribed)
0.715
Table 1: Synthetic ellipsoid-avoid (in-GT-mode, n=459 ): box unions cannot tile an ellipsoid complement tightly at any box count.
Figure 4: Closed-loop Franka EE servo on 18 ManiSkill3 task families ( 200 seeds each). Both methods reach the target at ≥0.99 geometric success; only CompCPZ lands the EE inside an analytic GT mode. Convex silently fails on 10/18 multi-modal families (red “silent failure” annotations). Pooled paired sign test: 1,900/1,918 wins, p≪10−30 ( Appendix E ).
Task
Instruction
k
CompCPZ
Convex
PickCube
near cube
1
0.99
0.99
StackCube
red ∨ blue
2
1.00
0.00
PokeCube
cube ∨ peg ∨ goal
3
1.00
0.00
PokeCube
(cube ∨ peg) ∧¬ goal
2
1.00
0.00
AssemblingKits
slot_0 ∨ slot_1 ∨ slot_2
3
1.00
0.00
PickClutterYCB
apple ∨ softball ∨ cups
3
1.00
0.98
Table 2: Closed-loop in-GT-mode rate on ManiSkill3 ( 200 seeds per row). Both methods reach ≥0.99 geometric success; only CompCPZ lands the EE inside a GT mode. Pooled paired sign test: 1,900/1,918 wins, p≪10−30 ( Appendix E ). PushT: Franka kinematic edge.
Method
S1
S2
S3 ( ¬ rug)
S4 ( ¬ m 0 )
Total
CompCPZ
3/3
3/3
3/3
3/3
12/12
AABB convex
0/3
0/3
0/3 (on rug)
0/3
0/12
Single clause
2/3
2/3
3/3
0/3 (blocked)
7/12
Table 3: Real-robot Go2 in-GT-mode rate ( 36 trials, 3 seeds per cell). All methods reach their planner goal in 12/12 trials; the gap is whether that goal is inside a GT mode. S3 excludes the rug; S4 blocks mode 0 (CompCPZ informed). Time/travel: Section H.5 .
Figure 5: Real-robot Go2 trials ( 36 trials total). Each panel: end-state photo with top-down MoCap trajectory inset (CompCPZ blue, AABB red). (a-b) CompCPZ reaches a GT mode. (c) AABB lands the dog on the forbidden rug (negation violation). (d) AABB stops in the inter-mode gap. Full per-trial table: Table 3 .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Per-task in-GT-mode rate of CompCPZ vs. the AABB and the tighter MVEE convex baselines on the 6 -task multi-modal subset ( ×20 seeds each). MVEE outperforms AABB only on AssemblingKits-v1 ( 0.75 vs. 0.00 ) but still trails CompCPZ; on the other five task families both convex baselines collapse to 0.00 , consistent with the geometric prediction of Corollary 12 .
Figure 7: Composition cost as a function of parse-tree depth, matching the O(2d) prediction of Proposition 16 . Left: compose wall-clock (sub-millisecond at depth ≤5 ; sub-second at depth ≤7 ). Middle: number of generators of C^+ . Right: number of CompCPZ modes returned by the composition. All three quantities scale as predicted under disjunction; conjunction stays approximately linear.
Figure 8: Empirical κ≤1 over N=6,828 internal nodes (random parse trees of depth ≤6 ). Left: per-node Hausdorff expansion ratio, 95 th-percentile κ^=1.000 . Right: worst-case per-tree expansion vs depth; depth-independent and holds empirically on rotated-box primitives outside the axis-aligned hypothesis.
Figure 9: Conformal coverage under four adversarial noise regimes (Gaussian σ=4 cm; Student- t df =3 ; anisotropic σx=6,σy=2 cm; 80/20 Gaussian + 8 cm bias mixture). All track 1−α within ±1% , supporting the distribution-free claim.
Task
Instruction
k
n
CompCPZ
AABB
Heatmap
Sampling
2 -mode disjunctions ( 1−1/k=0.50 )
StackCube
red ∨ blue
2
200
1.00
0.00
0.50
0.50
PullCube
cube ∨ goal_region
2
200
1.00
0.00
0.50
0.50
PlaceSphere
sphere ∨ bin
2
200
1.00
0.00
0.50
0.50
PullCubeTool
cube ∨ tool
2
200
1.00
0.00
0.50
0.50
PullCubeTool
(cube ∨ tool) ∧ table
2
200
1.00
0.00
0.50
0.50
Appendix
Table 4: Per-mode adversarial blocking on 13 multi-modal task families ( 100 seeds, every mode blocked in turn). CompCPZ recovers via mode switching; AABB lands in the inter-mode gap; non-sound multi-peak baselines hit the 1−1/k floor of Proposition 13 . ‡ : kinematic-edge tasks where both methods physically fail.
Instruction
n
CompCPZ
AABB-union (no ¬ )
near cube ∨ peg, ¬ goal_region
200
1.00
1.00
near cube ∨ peg ∨ goal_region, ¬ peg
200
1.00
0.71
Pooled
400
1.00
0.85
Appendix
Table 5: Direct negation ablation on the two PokeCube negation instructions ( 200 scenes each). CompCPZ uses exact axis-aligned complement; the AABB-union baseline has no complement primitive and silently drops the negation clause. Both then pick the nearest surviving mode.
Table 6: Perception-envelope ablations. (a) Centroid-noise sweep ( 30 seeds/cell). (b) Off-the-shelf open-vocab detectors on ManiSkill3 RGB. ‡ : human-render camera (different intrinsic, fixable); † : many edge detections project to NaN.
frames/task
total
median (cm)
mean (cm)
miss
train (s)
12
48
2.79
2.60
0.64
166
25
100
1.71
3.62
0.01
174
50
200
0.85
1.99
0.00
189
100
400
0.74
1.33
0.00
212
200
800
0.73
1.41
0.00
∼600
Appendix
Table 7: In-domain fine-tune data-efficiency ablation (YOLOv8n, 4 tasks, z -aware projection, 25 held-out seeds/task evaluated on disjoint seed range). The fine-tune clears CompCPZ’s σ≤10 mm working envelope at just 200 training frames and matches the 800 -frame production model at 400 . Training wall-clock is on a single GPU.
Task
Instruction
Wins
Loss
p
StackCube-v1
red ∨ blue
200
0
6.2×10−61
PullCube-v1
cube ∨ goal_region
200
0
6.2×10−61
PlaceSphere-v1
sphere ∨ bin
200
0
6.2×10−61
PushT-v1
tee ∨ goal_tee
97
18
1.4×10−14
PullCubeTool-v1
cube ∨ tool
200
0
6.2×10−61
PegInsertionSide-v1
peg ∨ box_with_hole
200
0
6.2×10−61
Appendix
Table 8: Per-task paired sign test on the 200 -seed closed-loop eval, multi-modal cases only. Wins/Loss/ nd as in Table 9 .
vs. AABB convex
vs. MVEE convex
Task
Instruction
Wins
Loss
p
Wins
Loss
p
StackCube-v1
red ∨ blue
20
0
9.5×10−7
20
0
9.5×10−7
PullCube-v1
cube ∨ goal_region
20
0
9.5×10−7
20
0
9.5×10−7
PlaceSphere-v1
sphere ∨ bin
20
0
9.5×10−7
20
0
9.5×10−7
PullCubeTool-v1
cube ∨ tool
20
0
9.5×10−7
20
0
9.5×10−7
PokeCube-v1
cube ∨ peg ∨ goal
20
0
9.5×10−7
20
0
9.5×10−7
Appendix
Table 9: Per-task statistical tests. Wins/Loss counted on decisive trials only (the two methods disagree on in-GT-mode); p -value is one-sided exact binomial under H0:pwin=0.5 . Pooled: 120/120 wins vs AABB ( p=7.5×10−37 ), 105/105 vs MVEE ( p=2.5×10−32 ).
Depth
n
tree-exact
mode-match
latency (ms)
0
9
09/09=1.00
09/09=1.00
1,369
1
19
19/19=1.00
19/19=1.00
2
12
04/12=0.33
12/12=1.00
3
8
01/08=0.12
08/08=1.00
4
1
00/01=0.00
01/01=1.00
Total
49
33/49=0.67
49/49=1.00
Appendix
Table 10: gpt-4o parser accuracy on the 49 -instruction benchmark by depth. Tree match is exact structural equality (modulo commutative-operator child order); mode match is post-compile mode count. The 16 tree mismatches are commutativity/associativity rewrites that Equation 2 is invariant to.
Role
Make / model
Software stack
Robot
Unitree Go2 EDU (MyBotShop-customised)
Ubuntu 20.04, ROS 2 Foxy, Sport API as a ROS 2 topic
Table 12: Per-scenario timing tˉ (s) and traversal travelˉ (m) for the Go2 trials, averaged over 3 seeds. All methods reach their planner-selected goal in 3/3 trials per cell.
Changes in natural-language instructions can directly alter the behavior ultimately executed by a robot, but such changes do not always imply that the task itself has changed. We propose One Word, Different Action, a language-conditioned executable decision benchmark based on real-robot physical decision anchors. It evaluates robot behavioral responses under task-preserving and task-changing conditions and examines joint reasoning over multiple task constraints. The benchmark uses decision invariance and decision sensitivity to measure action stability when task semantics are preserved and correct decision updates when task requirements change. Matched atomic constituents directly relate correct handling of individual constraints to the corresponding joint executable decision. Experiments show that atomic-task correctness does not sufficiently represent compositional reasoning ability: even when every constituent constraint is handled correctly, a model may still produce an erroneous final action decision on the corresponding compositional task. These results indicate that the key challenge in language-conditioned robot reasoning lies not only in understanding individual task requirements, but also in integrating multiple correctly handled constraints into consistent executable behavior.
Yiwei Liu, Luwei Yang, Shunbo Lei
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China · Shenzhen Research Institute of Big Data (SRIBD), Shenzhen 518172, China
When told to "cut the cake," a robot must choose the knife over nearby scissors, despite both objects affording the same cutting function. In real-world scenes, multiple objects may share identical affordances, yet only one is appropriate under the given task context. We call such cases confusing pairs. However, existing 3D affordance methods largely sidestep this challenge by evaluating isolated single objects, often with explicit category names provided in the query. We formalize Intent-Driven Confusable Affordance Grounding, a new 3D affordance setting that requires predicting a per-point affordance mask on the correct object within a multi-object point cloud, conditioned on implicit natural language intent. To study this problem, we construct CompassAD, the first benchmark centered on implicit intent in confusing multi-object compositions. It comprises 30 confusing object pairs spanning 16 affordance types, 6,422 compositions, and 88K+ query-answer pairs. Furthermore, we propose CompassNet, a framework that incorporates two dedicated modules tailored to this task. Instance-bounded Cross Injection (ICI) constrains language-geometry alignment within object boundaries to prevent cross-object semantic leakage. Bi-level Contrastive Refinement (BCR) enforces discrimination at both geometric-group and point levels, sharpening distinctions between target and confusable surfaces. Extensive experiments demonstrate state-of-the-art results on both seen and unseen queries, and deployment on a robotic manipulator confirms effective transfer to real-world grasping in confusing multi-object compositions.
Jingliang Li, Jindou Jia, Tuo An +7
MARS Lab, Nanyang Technological University, Singapore
Multi-task manipulation policies differ in architecture, scale, and pretrained priors all at once, so published comparisons cannot attribute performance to any single component. We argue that most of it comes from the visual representation, and that parameter scale and generative priors are largely incidental. To test this, we build CoRP (Compressed Representation Policy), a deliberately compact policy (48.9M parameters, no vision-language model and no video-generative prior) that factorizes into a representation extractor and a flow-matching action generator. It reaches 97.0% on LIBERO and 75.78%/73.36% on RoboTwin 2.0 Clean/Randomized, matching systems 40.9-163.6x larger. Holding the action generator fixed, we then vary one extractor property at a time. Pretrained initialization is decisive: a random ViT-S/14 drops to 78.1% and an ImageNet ResNet-34 to 74.5% on LIBERO. Pretraining alone is not enough, as freezing the encoder costs 19.8 points. Compression matters as much: resampling each view to 48 tokens beats passing all patch tokens (97.0% vs 83.2%), and a variational information bottleneck over those tokens is worse than a hard token budget, cutting LIBERO-Goal from 95.8% to 33.0% by suppressing the instruction-dependent token selection the policy relies on. Language conditioning contributes only where the observation leaves the goal ambiguous (LIBERO-Goal: 9.2% to 95.8%), while on RoboTwin 2.0, where observations are unambiguous, removing it slightly improves success. Therefore, we argue that a compact policy works when its representation is pretrained, task-adapted, and compressed. Project page: https://corp-policy.github.io/