CompCPZ: Preserving Multi-Modal Intent in Language-Guided Robot Manipulation

Zhen Zhang; Ahmad Hafez; Peng Xie; Yanliang Huang; Wenyuan Wu; Amr Alanwar

cs.ROAug 18, 2026

CompCPZ: Preserving Multi-Modal Intent in Language-Guided Robot Manipulation

Authors: Zhen Zhang, Ahmad Hafez, Peng Xie, Yanliang Huang, Wenyuan Wu, Amr Alanwar

Organizations: School of Computation, Information and Technology Technical University of Munich, Germany

Abstract

A robot asked to "place the cup near the red plate or the blue plate" may reach the centroid between them and appear geometrically successful, while satisfying neither disjunct of the instruction. This silent semantic failure exposes a structural limitation of language-conditioned robot policies: representations that collapse a disjunctive instruction into a single connected set cannot preserve all feasible modes, and planners that commit to one action degrade under run-time mode uncertainty. We address this limitation with CompCPZ, a sound algebraic layer that language-conditioned learning systems wrap to recover multi-modal disjunctive representation, recursively composing per-primitive constrained polynomial zonotope enclosures along the language parse tree with distribution-free conformal coverage and sub-millisecond runtime. On a closed-loop ManiSkill3 tabletop-manipulation benchmark, CompCPZ outperforms convex set baselines, multi-peak decoders, and a zero-shot vision-language-action model (1,900/1,918 paired wins, p << 10^(-30)); the same compiler also transfers without retuning to planar real-robot trials on a Unitree Go2 quadruped under motion capture. These results suggest that compositional language grounding should be evaluated not only by reaching a decoded target, but by whether the represented feasibility set preserves the connected-component structure of the user's intent.

Figures & tables

Figure 1: CompCPZ compiles a natural-language instruction into a sound $k$ -component feasibility set via bottom-up Boolean composition of two-sided per-primitive enclosures, with run-time mode switching for handling occlusion of the nearest mode.

Panel of Figure 2 — Figure 2: Tightness hierarchy: axis-aligned $\text{AABB}\supseteq$ centrally symmetric $\text{Zonotope}\supseteq$ possibly non-convex CPZ .

Panel of Figure 3 — Figure 3: Sound convex enclosures structurally fail on disjunctive instructions ( Corollary 12 ). Four ManiSkill3 task families, each pairing a SAPIEN render with a top-down inset. The CompCPZ target (green halo) lands inside a true mode; the convex AABB target (red halo) lands in the inter-mode gap.

Method	In-GT-mode
CompCPZ	1.000
AABB-union (bounding)	0.928
AABB-union (inscribed)	0.715

Table 1: Synthetic ellipsoid-avoid (in-GT-mode,

n{=}459

): box unions cannot tile an ellipsoid complement tightly at any box count.

Figure 4: Closed-loop Franka EE servo on

18

ManiSkill3 task families (

200

seeds each). Both methods reach the target at

\geq\!0.99

geometric success; only CompCPZ lands the EE inside an analytic GT mode. Convex silently fails on

10/18

multi-modal families (red “silent failure” annotations). Pooled paired sign test:

1{,}900/1{,}918

wins,

p\ll 10^{-30}

( Appendix E ).

Task	Instruction	$k$	CompCPZ	Convex
PickCube	near cube	1	0.99	0.99
StackCube	red $\vee$ blue	2	1.00	0.00
PokeCube	cube $\vee$ peg $\vee$ goal	3	1.00	0.00
PokeCube	(cube $\vee$ peg) $\wedge\neg$ goal	2	1.00	0.00
AssemblingKits	slot_0 $\vee$ slot_1 $\vee$ slot_2	3	1.00	0.00
PickClutterYCB	apple $\vee$ softball $\vee$ cups	3	1.00	0.98

Table 2: Closed-loop in-GT-mode rate on ManiSkill3 (

200

seeds per row). Both methods reach

\geq\!0.99

geometric success; only CompCPZ lands the EE inside a GT mode. Pooled paired sign test:

1{,}900/1{,}918

wins,

p\ll 10^{-30}

( Appendix E ). PushT: Franka kinematic edge.

Method	S1	S2	S3 ( $\neg$ rug)	S4 ( $\neg$ m 0 )	Total
CompCPZ	3/3	3/3	3/3	3/3	12/12
AABB convex	0/3	0/3	0/3 (on rug)	0/3	0/12
Single clause	2/3	2/3	3/3	0/3 (blocked)	7/12

Table 3: Real-robot Go2 in-GT-mode rate (

36

trials,

3

seeds per cell). All methods reach their planner goal in

12/12

trials; the gap is whether that goal is inside a GT mode. S3 excludes the rug; S4 blocks mode

0

(CompCPZ informed). Time/travel: Section H.5 .

Panel of Figure 5 — Figure 5: Real-robot Go2 trials ( $36$ trials total). Each panel: end-state photo with top-down MoCap trajectory inset (CompCPZ blue, AABB red). (a-b) CompCPZ reaches a GT mode. (c) AABB lands the dog on the forbidden rug (negation violation). (d) AABB stops in the inter-mode gap. Full per-trial table: Table 3 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Per-task in-GT-mode rate of CompCPZ vs. the AABB and the tighter MVEE convex baselines on the

6

-task multi-modal subset (

\times 20

seeds each). MVEE outperforms AABB only on AssemblingKits-v1 (

0.75

vs.

0.00

) but still trails CompCPZ; on the other five task families both convex baselines collapse to

0.00

, consistent with the geometric prediction of Corollary 12 .

Figure 7: Composition cost as a function of parse-tree depth, matching the

O(2^{d})

prediction of Proposition 16 . Left: compose wall-clock (sub-millisecond at depth

\leq 5

; sub-second at depth

\leq 7

). Middle: number of generators of

\hat{\mathcal{C}}^{+}

. Right: number of CompCPZ modes returned by the composition. All three quantities scale as predicted under disjunction; conjunction stays approximately linear.

Figure 8: Empirical

\kappa\leq 1

over

N=6{,}828

internal nodes (random parse trees of depth

\leq 6

). Left: per-node Hausdorff expansion ratio,

95

th-percentile

\hat{\kappa}=1.000

. Right: worst-case per-tree expansion vs depth; depth-independent and holds empirically on rotated-box primitives outside the axis-aligned hypothesis.

Figure 9: Conformal coverage under four adversarial noise regimes (Gaussian

\sigma{=}4

cm; Student-

t

{=}3

; anisotropic

\sigma_{x}{=}6,\sigma_{y}{=}2

cm;

80/20

Gaussian +

8

cm bias mixture). All track

1-\alpha

within

\pm 1\%

, supporting the distribution-free claim.

Task	Instruction	$k$	$n$	CompCPZ	AABB	Heatmap	Sampling
$2$ -mode disjunctions ( $1-1/k=0.50$ )
StackCube	red $\vee$ blue	2	200	1.00	0.00	0.50	0.50
PullCube	cube $\vee$ goal_region	2	200	1.00	0.00	0.50	0.50
PlaceSphere	sphere $\vee$ bin	2	200	1.00	0.00	0.50	0.50
PullCubeTool	cube $\vee$ tool	2	200	1.00	0.00	0.50	0.50
PullCubeTool	(cube $\vee$ tool) $\wedge$ table	2	200	1.00	0.00	0.50	0.50

Appendix

Table 4: Per-mode adversarial blocking on

13

multi-modal task families (

100

seeds, every mode blocked in turn). CompCPZ recovers via mode switching; AABB lands in the inter-mode gap; non-sound multi-peak baselines hit the

1{-}1/k

floor of Proposition 13 . ‡ : kinematic-edge tasks where both methods physically fail.

Instruction	$n$	CompCPZ	AABB-union (no $\neg$ )
near cube $\vee$ peg, $\neg$ goal_region	200	1.00	1.00
near cube $\vee$ peg $\vee$ goal_region, $\neg$ peg	200	1.00	0.71
Pooled	400	1.00	0.85

Appendix

Table 5: Direct negation ablation on the two PokeCube negation instructions (

200

scenes each). CompCPZ uses exact axis-aligned complement; the AABB-union baseline has no complement primitive and silently drops the negation clause. Both then pick the nearest surviving mode.

Table 6: Perception-envelope ablations. (a) Centroid-noise sweep (

30

seeds/cell). (b) Off-the-shelf open-vocab detectors on ManiSkill3 RGB.

\ddagger

: human-render camera (different intrinsic, fixable);

\dagger

: many edge detections project to NaN.

frames/task	total	median (cm)	mean (cm)	miss	train (s)
12	48	2.79	2.60	0.64	166
25	100	1.71	3.62	0.01	174
50	200	0.85	1.99	0.00	189
100	400	0.74	1.33	0.00	212
200	800	0.73	1.41	0.00	$\sim\!600$

Appendix

Table 7: In-domain fine-tune data-efficiency ablation (YOLOv8n,

4

tasks,

z

-aware projection,

25

held-out seeds/task evaluated on disjoint seed range). The fine-tune clears CompCPZ’s

\sigma\!\leq\!10

mm working envelope at just

200

training frames and matches the

800

-frame production model at

400

. Training wall-clock is on a single GPU.

Task	Instruction	Wins	Loss	$p$
StackCube-v1	red $\vee$ blue	200	0	$6.2{\times}10^{-61}$
PullCube-v1	cube $\vee$ goal_region	200	0	$6.2{\times}10^{-61}$
PlaceSphere-v1	sphere $\vee$ bin	200	0	$6.2{\times}10^{-61}$
PushT-v1	tee $\vee$ goal_tee	97	18	$1.4{\times}10^{-14}$
PullCubeTool-v1	cube $\vee$ tool	200	0	$6.2{\times}10^{-61}$
PegInsertionSide-v1	peg $\vee$ box_with_hole	200	0	$6.2{\times}10^{-61}$

Appendix

Table 8: Per-task paired sign test on the

200

-seed closed-loop eval, multi-modal cases only. Wins/Loss/

n_{d}

as in Table 9 .

		vs. AABB convex			vs. MVEE convex
Task	Instruction	Wins	Loss	$p$	Wins	Loss	$p$
StackCube-v1	red $\vee$ blue	20	0	$9.5{\times}10^{-7}$	20	0	$9.5{\times}10^{-7}$
PullCube-v1	cube $\vee$ goal_region	20	0	$9.5{\times}10^{-7}$	20	0	$9.5{\times}10^{-7}$
PlaceSphere-v1	sphere $\vee$ bin	20	0	$9.5{\times}10^{-7}$	20	0	$9.5{\times}10^{-7}$
PullCubeTool-v1	cube $\vee$ tool	20	0	$9.5{\times}10^{-7}$	20	0	$9.5{\times}10^{-7}$
PokeCube-v1	cube $\vee$ peg $\vee$ goal	20	0	$9.5{\times}10^{-7}$	20	0	$9.5{\times}10^{-7}$

Appendix

Table 9: Per-task statistical tests. Wins/Loss counted on decisive trials only (the two methods disagree on in-GT-mode);

p

-value is one-sided exact binomial under

H_{0}:p_{\mathrm{win}}=0.5

. Pooled:

120/120

wins vs AABB (

p=7.5\times 10^{-37}

105/105

vs MVEE (

p=2.5\times 10^{-32}

Depth	$n$	tree-exact	mode-match	latency (ms)
0	9	$\phantom{0}9/\phantom{0}9=1.00$	$\phantom{0}9/\phantom{0}9=1.00$	$1{,}369$
1	19	$19/19=1.00$	$19/19=1.00$
2	12	$\phantom{0}4/12=0.33$	$12/12=1.00$
3	8	$\phantom{0}1/\phantom{0}8=0.12$	$\phantom{0}8/\phantom{0}8=1.00$
4	1	$\phantom{0}0/\phantom{0}1=0.00$	$\phantom{0}1/\phantom{0}1=1.00$
Total	49	$\mathbf{33/49=0.67}$	$\mathbf{49/49=1.00}$

Appendix

Table 10: gpt-4o parser accuracy on the

49

-instruction benchmark by depth. Tree match is exact structural equality (modulo commutative-operator child order); mode match is post-compile mode count. The

16

tree mismatches are commutativity/associativity rewrites that Equation 2 is invariant to.

Role	Make / model	Software stack
Robot	Unitree Go2 EDU (MyBotShop-customised)	Ubuntu 20.04, ROS 2 Foxy, Sport API as a ROS 2 topic
Robot compute	NVIDIA Jetson Orin NX 8 GB	Python 3.8, ROS 2 client, our compositional grounding library, NOKOV’s Python SDK
Motion capture	NOKOV Mars (multi-camera passive IR)	$\sim\!100$ Hz, sub-mm; rigid bodies $\{$ Dog, A, B, C, Rug $\}$ ; world origin = Go2 StandUp pose, $+x$ = head
MoCap host	Windows workstation (NOKOV-supplied)	SDK + our broadcaster (newline-delimited stream over a network socket)
Field laptop	Windows + WSL 2 Ubuntu	Hosts the secure reverse tunnel from MoCap host to Jetson and runs the trial driver

Appendix

Table 11: Real-robot hardware (essential rows only).

	S1		S2		S3		S4
Method	$\bar{t}$	travel	$\bar{t}$	travel	$\bar{t}$	travel	$\bar{t}$	travel
CompCPZ	11.1	1.83	7.7	1.31	11.1	1.85	10.1	1.78
AABB convex	7.1	1.46	9.7	2.18	6.7	1.49	6.0	1.40
Single clause	13.1	2.13	9.4	1.79	10.8	2.00	9.1	1.85

Appendix

Table 12: Per-scenario timing

\bar{t}

(s) and traversal

\bar{\text{travel}}

(m) for the Go2 trials, averaged over

3

seeds. All methods reach their planner-selected goal in

3/3

trials per cell.

Language-Conditioned Robot Manipulation

Language-conditioned robot manipulation connects natural-language instructions to robot perception, planning, and physical actions.

See all 230 papers

One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning
Sep 4, 2026Yiwei Liu, Luwei Yang, Shunbo LeiLanguage-Conditioned Robot Control Embodied Reasoning
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China · Shenzhen Research Institute of Big Data (SRIBD), Shenzhen 518172, China
CompassAD: Intent-Driven 3D Affordance Grounding in Functionally Competing Objects
Apr 2, 2026Jingliang Li, Jindou Jia, Tuo An +73D Affordance Grounding Robotic Affordance Inference
MARS Lab, Nanyang Technological University, Singapore
Compact Robot Policies Need Fine-Grained Visual Representations
Oct 6, 2026Nanhe Chen, Runqiu Yang, Jiawei Tang +2Robot Policy Learning Visual Representation Learning
X Square Robot · KTH

Sep 4, 2026J/K move · Enter open · S save

One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning

Yiwei Liu, Luwei Yang, Shunbo Lei

School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China · Shenzhen Research Institute of Big Data (SRIBD), Shenzhen 518172, China

Language-Conditioned Robot Control Embodied Reasoning Robot Manipulation Benchmarks Robotic Control

Open paper AskPDF arXiv

Changes in natural-language instructions can directly alter the behavior ultimately executed by a robot, but such changes do not always imply that the task itself has changed. We propose One Word, Different Action, a language-conditioned executable decision benchmark based on real-robot physical decision anchors. It evaluates robot behavioral responses under task-preserving and task-changing conditions and examines joint reasoning over multiple task constraints. The benchmark uses decision invariance and decision sensitivity to measure action stability when task semantics are preserved and correct decision updates when task requirements change. Matched atomic constituents directly relate correct handling of individual constraints to the corresponding joint executable decision. Experiments show that atomic-task correctness does not sufficiently represent compositional reasoning ability: even when every constituent constraint is handled correctly, a model may still produce an erroneous final action decision on the corresponding compositional task. These results indicate that the key challenge in language-conditioned robot reasoning lies not only in understanding individual task requirements, but also in integrating multiple correctly handled constraints into consistent executable behavior.

Abstract

Language-Conditioned Robot Manipulation

Explore similar work

One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning

CompassAD: Intent-Driven 3D Affordance Grounding in Functionally Competing Objects

Compact Robot Policies Need Fine-Grained Visual Representations