From Language to Task Maps: Compiling Semantic Relations While Preserving Task-Relevant Freedom
Authors: Jaegyun Park, Jingwang Lee, Jungsoo Lee, Soonwoong Hwang, Wansoo Kim
Organizations: Department of Robotics, Hanyang University, Seoul, Republic of Korea. · Department of Robotics, Hanyang University ERICA, Ansan, Republic of Korea.
Natural-language manipulation instructions specify qualitative relations, whereas continuous controllers require state-evaluable task quantities, differentials, and completion conditions. Because a qualitative relation generally leaves part of the relative configuration unspecified, expanding it into a complete pose can introduce unintended constraints. We present a typed semantic-to-geometric interface in which language specifies entities, relations, and phases, while each relation indexes a registered specification of its task-relevant distinctions and preserved freedoms. A robot-side compiler grounds these specifications, constructs relation-specific task maps and consistent differentials using conformal geometric algebra, and composes the resulting policies through RMPflow. To evaluate the division of responsibility between the language model and the compiler, we compared a Semantic Topology interface with one that additionally requires relation-specific geometric specifications over 60 instructions. Both produced correct shared semantic content in 41/60 cases, but critical errors under their respective interface requirements occurred in 19/60 and 58/60 cases. Across 64 grounded evaluations spanning eight geometric relation forms, the task maps preserved registered null directions and responded to relation-relevant perturbations; analytic directional derivatives agreed with finite differences, and Jacobian ranks matched the registered dimensions. In three closed-loop ablations using a simulated Franka Emika Panda in MuJoCo, fixing a relation-preserved coordinate increased median terminal progress error by 20.24--71.00~mm while the retained relation errors remained within their evaluation bounds. These results support compiling relation-visible geometry and preserved freedom together into composable continuous objectives.
Figures & tables
Fig. 1: Overview of the semantic-to-geometric compilation pipeline. The language planner specifies entities, relations, and phases; the robot-side compiler selects and grounds relation-visible geometry before the resulting task maps are composed for continuous control.
Typed semantic relation
Meaning
REACH_AFFORDANCE (RA)
Bring the subject to a reference affordance
ALIGN_AFFORDANCES (ALN)
Align paired features under the selected condition
MAINTAIN_DIRECTION (MD)
Keep a body axis aligned with a reference direction
ESTABLISH_CONTACT (EC)
Bring the subject into contact with its support
MAINTAIN_AFFORDANCE_CONTACT (MAC)
Maintain a grasp or designated surface contact
CLEAR_SUPPORT (CS)
Move the subject clear of its support
TABLE I: Typed semantic relations and their meanings.
Geometric relation form
Primitive inputs
Task quantity
Preserved freedom
dimBγ
Line parallelism
Line–Line
Direction residual
All translations and rotation about the Line axis
2
Line coincidence
Line–Line
Direction and transverse-offset residual
Translation along the common axis and axial spin
4
Plane-normal alignment
Plane–Plane
Normal-alignment residual
Normal gap and in-plane pose
2
Plane coincidence
Plane–Plane, U
Normal and signed-gap residual (target gap d∗∈U , default 0)
TABLE III: Paired relation-interface results ( N=60 per condition); — denotes a quantity delegated to the compiler.
Geometric relation form
Imax
Nmax
EFD,max
rank(Jγ)
Line parallelism
1.12e−16
0†
1.72e−11
2
Line coincidence
1.30e−16
1.16e−17
1.11e−11
4
Plane-normal alignment
1.13e−16
0†
1.76e−11
2
Plane coincidence
1.26e−16
3.47e−17
1.11e−11
3
Directional alignment
1.11e−16
0†
1.67e−11
2
Signed gap/contact
0†
0†
1.00e−12
1
TABLE IV: Task-map invariance and differential consistency. † : structural zero (coordinate absent from the task map).
Composition (Lock)
Rank/null
Relation
Lock
Align–insert (mating axis)
4/2
2.91 [2.90, 17.68]
65.67 [59.74, 87.68]
Align–traverse (path tangent)
4/2
4.57 [4.44, 4.63]
75.58 [75.38, 75.75]
Directional alignment–progress (vertical)
3/3
0.95 [0.92, 0.96]
21.19 [21.19, 21.20]
TABLE V: Local 6D rank/nullity and terminal progress errors (mm; median [min, max], five matched seeds).
Fig. 2: Representative executions of seven tasks with active primitives overlaid. Table VI lists their primitive types and semantic relations. These executions illustrate reuse of registered relation specifications across different entities and primitives.
Task
Primitive types
Typed semantic relations
Stacking
Plane
RA , CS , MRP , ALN , EC
Nut Insertion
Line
RA , CS , MRP , ALN , EC
Surface Wiping
Plane
RA , CS , MRP , ALN , EC , MAC , TA
Cutting
Plane
RA , CS , MRP , ALN , EC , MAC , TA
Circular Orbit
Circle, Sphere
ALN , TA
Door Articulation
Line, Circle
RA , MAC , ACT
TABLE VI: Primitives and typed semantic relations in the executions of Fig. 2 . Abbreviations are defined in Table I .
Fig. 3: Prompt-conditioned relation selection in Nut insertion. N0 (left) releases after entry alignment; N1 (right) keeps grasp and alignment during descent and releases only after seating.