Making Task Abstractions Executable: Control-Aware Layout Repair for a Fixed Controller

Chiyoung Kim; Seungyeon Back; Sumin Shim; Doyoung Heo; Rita Singh

cs.ROOct 7, 2026

Making Task Abstractions Executable: Control-Aware Layout Repair for a Fixed Controller

Authors: Chiyoung Kim, Seungyeon Back, Sumin Shim, Doyoung Heo, Rita Singh

Organizations: Chung-Ang University · Hanyang University · Carnegie Mellon University · KAIST

Abstract

A task abstraction can specify the intended events while its spatial layout prevents a fixed agent and controller from completing them. Starting from a supplied structured task record, we compile whole-task tracking, clearance, and actuation requirements into auditable affine layout constraints. We repair only declared continuous coordinates, preserving event order, timing, topology, and the controller. A most-violated-row update admits conditional finite-certification and net-displacement bounds; a same-compiler quadratic projection separates the representation from the optimizer. On three researcher-authored task abstractions, both backends certify all three layouts and complete all 300 fresh paired rollouts per backend. A risk-target sweep also exposes fixed event tests that the chosen certificate cannot satisfy through layout edits alone.

Figures & tables

Task	Fixed event sequence	Isolates	Editable geometry
P1	Switch $\rightarrow$ Door $\rightarrow$ Goal	Door clearance	Door half-width
P2	Checkpoint 1 $\rightarrow$ Checkpoint 2 $\rightarrow$ Goal	Checkpoint clearance	Event $y$ -coordinates
P3	Switch 1 $\rightarrow$ Switch 2 $\rightarrow$ Goal	Fixed-time travel	Event $x$ -coordinates

Table 1: Supplied task abstractions. Event identities/order, gate tolerances, and timing remain fixed; only the listed geometry may move.

Figure 1: Matched P1 layouts and logged trajectories. Panels share the task record, controller, and paired disturbance. Dashed/solid curves show reference/actual motion; colored points mark ordered events, and the magenta X marks simulator contact. Raw and Geometry-only collide; both full-compiler backends widen the door half-width to about

0.168

m and succeed.

Method	Cert.	P1	P2	P3	Overall	Sat.	Edit
		Successes (of 100)
Raw	0/3	0	0	0	0/300	100	0
Geometry-only	0/3	1	12	99	112/300	100	0.272
Ours	3/3	100	100	100	300/300	0	0.935
QP-Projection	3/3	100	100	100	300/300	0	0.933

Table 2: Fresh headless evaluation (

\delta=0.5

). Successes are out of 100 per task. Cert.: layouts passing the full compiler; Sat.: rollouts with actuator saturation (all on P3); Edit: mean

\lVert\Delta\theta\rVert_{2}

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Task description	Fixed task abstraction	Editable spatial realization
P1	Switch $\rightarrow$ Door $\rightarrow$ Goal.	Three ordered monitored events; fixed event endpoints, timing, and event-controlled door logic.	Door half-width; the initial opening is insufficient.
P2	Ordered checkpoints $\rightarrow$ Goal.	Checkpoint 1, checkpoint 2, and goal, in that order; fixed horizontal coordinates, corridor, timing, and gate tolerances.	Three event heights; the initial targets lie too close to the walls.
P3	Switch 1 $\rightarrow$ Switch 2 $\rightarrow$ Goal.	Three ordered events; fixed start, transverse coordinates, minimum ordering distance, timing, and gate tolerances.	Three horizontal event positions; initial travel demands are large relative to time and input limits.

Appendix

Table 3: Task descriptions and the corresponding supplied task abstractions. The event records are provided inputs.

Figure 2: All tasks and methods under one preserved paired stress condition. Rows are P1–P3; columns are Raw, Geometry-only, Ours, and QP-Projection. Within each row, axes, controller, initial error, and disturbance array are identical. Each row has its own

x

range: the tick labels under the bottom row apply to P3 only, while P1 ticks are at

-1,-0.5,0,0.5,1

m and P2 ticks at

-1,0,1

m. Dashed gray and solid blue curves are reference and actual trajectories; the black square is the start; orange, purple, and green markers are ordered events; magenta X and purple plus mark logged collision and deadline miss. Walls and markers are reconstructed from the exact compiled layout, while curves and failure symbols come from simulator logs.

Task	Method	Succ.	Coll.	Track
P1	Raw	0/3	3	3
P1	Geometry-only	0/3	3	2
P1	Ours	3/3	0	0
P1	QP-Projection	3/3	0	0
P2	Raw	0/3	3	3
P2	Geometry-only	1/3	2	0

Appendix

Table 4: Legacy outcomes after adding QP-Projection. Each task–method cell has three archived paired realizations. Track counts episodes with

\max_{t}e_{t}^{\top}Pe_{t}/\rho_{\delta}>1

Task	Method	Succ.	95% CI	Coll.	Track
P1	Raw	0/100	[0,3.7]	100	89
P1	Geometry-only	1/100	[0.177,5.45]	99	77
P1	Ours	100/100	[96.3,100]	0	0
P1	QP-Projection	100/100	[96.3,100]	0	0
P2	Raw	0/100	[0,3.7]	100	86
P2	Geometry-only	12/100	[7,19.8]	88	3

Appendix

Table 5: Fresh results by task and method. Success confidence intervals are Wilson 95% intervals in percentage points; Coll., Deadl., Sat., and Track count collision, deadline-miss, saturation, and tracking-threshold episodes.

Task	Method	Full cert.	$\lVert\Delta\theta\rVert_{2}$	Full margin	Original margin	Physical edit [m]
P1	Raw	no	0	-0.106	-0.0528	0
P1	Geometry-only	no	0.011	-0.0947	-0.0473	0.0055
P1	Ours	yes	0.107	0.001	5.00e-04	0.0533
P1	QP-Projection	yes	0.106	0	5.00e-13	0.0528
P2	Raw	no	0	-0.126	-0.0628	0, 0, 0
P2	Geometry-only	no	0.0537	-0.0947	-0.0473	-0.0155, 0.0155, -0.0155

Appendix

Table 6: All 12 fixed method-layouts. Full margin is the minimum normalized full-compiler slack; original margin independently reevaluates unnormalized full rows. Values are rounded to at most three significant digits; full precision remains in the logs.

Task	Iter.	$\max(C\theta-b)$	Original res.	Stationarity	Complementarity	Signed gap
P1	1	0	-5.00e-13	2.78e-17	0	0
P2	2	0	-5.00e-13	1.11e-16	0	6.94e-18
P3	2	9.99e-16	-9.99e-13	1.78e-15	4.16e-15	-2.22e-15

Appendix

Table 7: QP-Projection diagnostics. Iter. is the SLSQP iteration count; residuals use the constraint matrix

C\theta\leq b

. Every solver status was successful, every original-row audit passed, and all displayed KKT diagnostics passed the frozen threshold.

Figure 3: Risk-target sweep. Left: recomputed position support $r_{p}$ , speed support $r_{v}$ , and feedback reserve $\eta$ , divided by their fixed event/input gates; the dashed line is equality with a gate. Right: compiler outcome for each task. Green means both constant and affine tests admit certification; red records a false constant test before either repair backend is called.

$\delta$	$\rho_{\delta}$	$r_{p}$ [m]	$r_{v}$ [m/s]	$\eta$ [m/s 2 ]	Outcome (P1–P3)
0.50	0.0429	0.0421	0.115	0.462	certified
0.20	0.107	0.0666	0.181	0.73	false constant test
0.10	0.215	0.0942	0.256	1.03	false constant test
0.05	0.429	0.133	0.362	1.46	false constant test

Appendix

Table 8: Certificate values and aggregate compiler outcome.

\eta

is one axis’ feedback-input reserve (the configured controller is axis-symmetric).

Figure 4: P3/Ours coefficient trace. Left: blue markers are logged best-so-far margins; the black line is zero-margin certification and the orange curve is the conservative progress quantity using

\bar{D}

. Right: logged net displacement and the

2\bar{D}

upper bound. Certification occurs after 72 updates with margin

1.64\times 10^{-4}

and displacement

2.48

Figure 5: Coefficient-only headroom sensitivity. Solid circles are observed first-certification counts or returned displacements for predeclared

\kappa

multipliers; dashed crosses are conservative bounds recomputed with a feasible comparison for each

\kappa

. All 12 repairs certified within budget; no additional simulator episodes were run.

Figure 6: Representative row margins for Ours. Curves are named normalized compiler rows; colored crosses mark the row selected as the current worst violation, and the black zero line is certification. P1 shows one width correction, while P2/P3 show interactions in which correcting one row can lower another.

Figure 7: Tracking energy in the preserved 27-episode, three-method study. Panels are P1 (Switch-Door-Goal), P2 (Ordered-Checkpoints-Goal), and P3 (Switch1-Switch2-Goal); the legend entry “Geometry” is Geometry-only. Each faint line is a saved paired realization; red, orange, and blue denote Raw, Geometry-only, and Ours. The horizontal dashed line is

e_{t}^{\top}Pe_{t}/\rho_{\delta}=1

, and vertical dotted lines are the fixed 2 and 4 s stage boundaries. Terminal post-step states are retained, including collision exceedances.

Workers	Jobs	Time [s]	Episodes/s	Peak RSS [GiB]	Stable
1	9	0.255	35.3	0.115	True
2	9	0.284	31.6	0.117	True
3	9	0.225	40	0.118	True

Appendix

Table 9: Measured Windows worker benchmark. Peak RSS is the Python process; selection follows the frozen low-memory policy.

Figure 8: Fresh outcomes and recorded backend costs. Left: success over 100 paired realizations per task. Right: recorded backend medians and interquartile ranges after 20 warm-ups and over 200 measurements. QP includes HiGHS feasibility, SLSQP, and audit; neither timing includes compilation or simulation. The operations differ, so this is an implementation-specific timing comparison.

Task	Operation	Median [ms]	IQR [ms]	Repeats
P1	Full compiler	0.0393	0.00395	200
P1	Geometry-only	0.0095	5.00e-04	200
P1	Ours	0.0096	0.00193	200
P1	QP + audit	0.987	0.322	200
P2	Full compiler	0.0715	0.00377	200
P2	Geometry-only	0.0183	0.00835	200

Appendix

Table 10: Separated compiler and backend timings after 20 warm-ups, with 200 measured calls per task/operation. QP includes HiGHS feasibility, SLSQP, and the original-row audit; simulator time is excluded.

Explore similar work

From Language to Task Maps: Compiling Semantic Relations While Preserving Task-Relevant Freedom
Sep 28, 2026Jaegyun Park, Jingwang Lee, Jungsoo Lee +2Geometric Constraints Goal-Conditioned Dynamic Manipulation
Department of Robotics, Hanyang University, Seoul, Republic of Korea. · Department of Robotics, Hanyang University ERICA, Ansan, Republic of Korea.
Roomer: Reflective Object-Grounded Model Editing and Repair for 3D Indoor Layout Synthesis
Aug 3, 2026Lingwei Dang, Ziyan Qiu, Jiajia Cheng +93D Layout Generation Repair
School of Software Engineering, South China University of Technology · School of Information Science and Technology, ShanghaiTech University
Layout-independent actuation allocator for fin-actuated marine robots
Jul 3, 2026Yuya Hamamatsu, Maarja Kruusmaa, Asko RistolainenActuator Robot Systems
Department of Computer Systems, Tallinn University of Technology, Tallinn, Estonia

Sep 28, 2026, cs.ROJ/K move · Enter open · S save

From Language to Task Maps: Compiling Semantic Relations While Preserving Task-Relevant Freedom

Jaegyun Park, Jingwang Lee, Jungsoo Lee, Soonwoong Hwang+1

Department of Robotics, Hanyang University, Seoul, Republic of Korea. · Department of Robotics, Hanyang University ERICA, Ansan, Republic of Korea.

Geometric Constraints Goal-Conditioned Dynamic Manipulation Semantic Relationships

Open paper AskPDF arXiv

Natural-language manipulation instructions specify qualitative relations, whereas continuous controllers require state-evaluable task quantities, differentials, and completion conditions. Because a qualitative relation generally leaves part of the relative configuration unspecified, expanding it into a complete pose can introduce unintended constraints. We present a typed semantic-to-geometric interface in which language specifies entities, relations, and phases, while each relation indexes a registered specification of its task-relevant distinctions and preserved freedoms. A robot-side compiler grounds these specifications, constructs relation-specific task maps and consistent differentials using conformal geometric algebra, and composes the resulting policies through RMPflow. To evaluate the division of responsibility between the language model and the compiler, we compared a Semantic Topology interface with one that additionally requires relation-specific geometric specifications over 60 instructions. Both produced correct shared semantic content in 41/60 cases, but critical errors under their respective interface requirements occurred in 19/60 and 58/60 cases. Across 64 grounded evaluations spanning eight geometric relation forms, the task maps preserved registered null directions and responded to relation-relevant perturbations; analytic directional derivatives agreed with finite differences, and Jacobian ranks matched the registered dimensions. In three closed-loop ablations using a simulated Franka Emika Panda in MuJoCo, fixing a relation-preserved coordinate increased median terminal progress error by 20.24--71.00~mm while the retained relation errors remained within their evaluation bounds. These results support compiling relation-visible geometry and preserved freedom together into composable continuous objectives.

Abstract

Explore similar work

From Language to Task Maps: Compiling Semantic Relations While Preserving Task-Relevant Freedom

Roomer: Reflective Object-Grounded Model Editing and Repair for 3D Indoor Layout Synthesis

Layout-independent actuation allocator for fin-actuated marine robots