From Language to Task Maps: Compiling Semantic Relations While Preserving Task-Relevant Freedom
Authors: Jaegyun Park, Jingwang Lee, Jungsoo Lee, Soonwoong Hwang, Wansoo Kim
Organizations: Department of Robotics, Hanyang University, Seoul, Republic of Korea. · Department of Robotics, Hanyang University ERICA, Ansan, Republic of Korea.
Natural-language manipulation instructions specify qualitative relations, whereas continuous controllers require state-evaluable task quantities, differentials, and completion conditions. Because a qualitative relation generally leaves part of the relative configuration unspecified, expanding it into a complete pose can introduce unintended constraints. We present a typed semantic-to-geometric interface in which language specifies entities, relations, and phases, while each relation indexes a registered specification of its task-relevant distinctions and preserved freedoms. A robot-side compiler grounds these specifications, constructs relation-specific task maps and consistent differentials using conformal geometric algebra, and composes the resulting policies through RMPflow. To evaluate the division of responsibility between the language model and the compiler, we compared a Semantic Topology interface with one that additionally requires relation-specific geometric specifications over 60 instructions. Both produced correct shared semantic content in 41/60 cases, but critical errors under their respective interface requirements occurred in 19/60 and 58/60 cases. Across 64 grounded evaluations spanning eight geometric relation forms, the task maps preserved registered null directions and responded to relation-relevant perturbations; analytic directional derivatives agreed with finite differences, and Jacobian ranks matched the registered dimensions. In three closed-loop ablations using a simulated Franka Emika Panda in MuJoCo, fixing a relation-preserved coordinate increased median terminal progress error by 20.24--71.00~mm while the retained relation errors remained within their evaluation bounds. These results support compiling relation-visible geometry and preserved freedom together into composable continuous objectives.
Figures & tables
Fig. 1: Overview of the semantic-to-geometric compilation pipeline. The language planner specifies entities, relations, and phases; the robot-side compiler selects and grounds relation-visible geometry before the resulting task maps are composed for continuous control.
Typed semantic relation
Meaning
REACH_AFFORDANCE (RA)
Bring the subject to a reference affordance
ALIGN_AFFORDANCES (ALN)
Align paired features under the selected condition
MAINTAIN_DIRECTION (MD)
Keep a body axis aligned with a reference direction
ESTABLISH_CONTACT (EC)
Bring the subject into contact with its support
MAINTAIN_AFFORDANCE_CONTACT (MAC)
Maintain a grasp or designated surface contact
CLEAR_SUPPORT (CS)
Move the subject clear of its support
TABLE I: Typed semantic relations and their meanings.
Geometric relation form
Primitive inputs
Task quantity
Preserved freedom
dimBγ
Line parallelism
Line–Line
Direction residual
All translations and rotation about the Line axis
2
Line coincidence
Line–Line
Direction and transverse-offset residual
Translation along the common axis and axial spin
4
Plane-normal alignment
Plane–Plane
Normal-alignment residual
Normal gap and in-plane pose
2
Plane coincidence
Plane–Plane, U
Normal and signed-gap residual (target gap d∗∈U , default 0)
TABLE III: Paired relation-interface results ( N=60 per condition); — denotes a quantity delegated to the compiler.
Geometric relation form
Imax
Nmax
EFD,max
rank(Jγ)
Line parallelism
1.12e−16
0†
1.72e−11
2
Line coincidence
1.30e−16
1.16e−17
1.11e−11
4
Plane-normal alignment
1.13e−16
0†
1.76e−11
2
Plane coincidence
1.26e−16
3.47e−17
1.11e−11
3
Directional alignment
1.11e−16
0†
1.67e−11
2
Signed gap/contact
0†
0†
1.00e−12
1
TABLE IV: Task-map invariance and differential consistency. † : structural zero (coordinate absent from the task map).
Composition (Lock)
Rank/null
Relation
Lock
Align–insert (mating axis)
4/2
2.91 [2.90, 17.68]
65.67 [59.74, 87.68]
Align–traverse (path tangent)
4/2
4.57 [4.44, 4.63]
75.58 [75.38, 75.75]
Directional alignment–progress (vertical)
3/3
0.95 [0.92, 0.96]
21.19 [21.19, 21.20]
TABLE V: Local 6D rank/nullity and terminal progress errors (mm; median [min, max], five matched seeds).
Fig. 2: Representative executions of seven tasks with active primitives overlaid. Table VI lists their primitive types and semantic relations. These executions illustrate reuse of registered relation specifications across different entities and primitives.
Task
Primitive types
Typed semantic relations
Stacking
Plane
RA , CS , MRP , ALN , EC
Nut Insertion
Line
RA , CS , MRP , ALN , EC
Surface Wiping
Plane
RA , CS , MRP , ALN , EC , MAC , TA
Cutting
Plane
RA , CS , MRP , ALN , EC , MAC , TA
Circular Orbit
Circle, Sphere
ALN , TA
Door Articulation
Line, Circle
RA , MAC , ACT
TABLE VI: Primitives and typed semantic relations in the executions of Fig. 2 . Abbreviations are defined in Table I .
Fig. 3: Prompt-conditioned relation selection in Nut insertion. N0 (left) releases after entry alignment; N1 (right) keeps grasp and alignment during descent and releases only after seating.
Robot manipulation policies often struggle to generalize beyond their demonstrations, even when new instructions involve familiar objects and behaviors. When language and scenes are strongly correlated during training, a policy can learn a fixed visual-action mapping rather than respond to the requested behavior. We investigate compositional reuse at two levels: within a subtask, combining familiar entities, action types, and action modifiers; and across subtasks, reusing learned subtasks in unseen long-horizon tasks. We introduce CoMani, a benchmark with controlled splits for evaluating both capabilities. Matched initial scenes and controlled changes to a single semantic factor encourage reliance on language rather than visual shortcuts. We further propose GraphPoint, which connects semantic entity graphs to geometric control by predicting future gripper point trajectories and converting them into actions using robot geometry. The framework organizes the gripper and objects by semantic roles and conditions their interactions on action types and modifiers, while predicted progress guides transitions during execution. Experiments and ablations on CoMani validate the effectiveness of our method for instruction-dependent generalization at both levels. Code will be released at GraphPoint.
Kang Luo, Hesheng Wang
IRMV Lab, the Department of Automation, Shanghai Jiao Tong University
We study how physical-state inputs affect a 0.8B hybrid language model adapted for manipulation with 6.2M trainable parameters. Six conditions are trained on three LIBERO-Spatial tasks and evaluated over three seeds and 540 held-out rollouts. Conditioning recurrent decay gates on geometric increments yields 28.9% success, compared with 36.7% when those increments are shuffled during training and 24.4% without explicit object/goal geometry. Both geometry policies receive correct inputs at evaluation. A token adapter using the same increments scores 27.8%; differences vary across seeds and remain inconclusive. Token-clock conditioning scores 11.1%, including one seed that fails to converge. In separate robustness tests, a state-only relative-coordinate policy retains 7/10 success under frame relabeling, whereas all four tested visual policies fall to at most 3/20 after a 5 cm object displacement. These results show no reliable advantage from training-time geometric alignment under this recipe and illustrate the gap between coordinate invariance and physical-layout generalization. Episode records, seed-level analyses, and figure-generation code accompany the paper.
Bridging abstract semantics and precise physical control remains a fundamental challenge in open-world robotic manipulation. While recent data-driven policies show promise, their reliance on isolated contact points or latent affordance embeddings lacks the rigorous kinematic constraints necessary for complex articulated objects.To overcome the limitation, we introduce RelAfford6D, a novel training-free framework centered on a Relational 6D Affordance Graph. Given a free-form instruction, our system deduces a semantic topology linking a primary interacting part to its physical anchor. By elevating these topological nodes into precise metric SE(3) poses via vision foundation models, we analytically formulate downstream execution as a kinematic constraint satisfaction problem. The robot synthesizes continuous trajectories by tracking strictly defined physical manifolds (e.g., revolute or prismatic orbits). Coupled with a closed-loop tracking mechanism for dynamic replanning against disturbances, our physically grounded approach achieves superior zero-shot success rates, cross-category generalization and execution robustness in both simulation and the real world environments, outperforming existing data-driven baselines.
Guodong Zhang, Qichen He, Wenyuan Xie +6
School of Computer Science, Shanghai Jiaotong University, Shanghai, P. R. China