Learning Unknown Constraints without Unsafe Data via Optimality and Counterfactual Regularization

Zhouyu Zhang; Chih-Yuan Chiu; Glen Chou

cs.ROOct 7, 2026

Learning Unknown Constraints without Unsafe Data via Optimality and Counterfactual Regularization

Authors: Zhouyu Zhang, Chih-Yuan Chiu, Glen Chou

Organizations: School of Electrical and Computer Engineering Georgia Institute of Technology Atlanta, GA 30309 · School of Cybersecurity and Privacy School of Aerospace Engineering Georgia Institute of Technology

Abstract

Learning from demonstrations (LfD) provides a framework for inferring unknown constraints from locally optimal, constraint-satisfying expert behavior. Existing approaches largely fall into two paradigms, constrained inverse optimal control (CIOC) and inverse constrained reinforcement learning (ICRL). CIOC exploits optimality conditions such as the Karush--Kuhn--Tucker (KKT) conditions but typically assumes known dynamics and structured constraint representations. Meanwhile, ICRL accommodates complex unknown constraints and unknown transition dynamics but often requires extensive online exploration, during which unsafe constraint violations may occur. In this work, we introduce Counterfactual KKT (CF-KKT), a constraint learning framework that leverages learned dynamics and locally optimal demonstrations to recover unknown constraints without requiring known dynamics or additional risky exploration, thereby combining the data efficiency and safety advantages of CIOC with the flexibility of ICRL. First, we use a locally learned differentiable dynamics model to impose KKT-inspired optimality conditions directly on the demonstrations. Second, we use the learned dynamics to generate reward-improving counterfactual behaviors near the demonstrations, revealing behaviors that would be preferable in the absence of the unknown constraint and thus providing synthetic infeasible data. When the constraint parameterization is known, the same learned-dynamics framework enables direct CIOC-based parameter recovery, and we characterize its sensitivity to dynamics misspecification. Across high-dimensional robotic control tasks, our approach learns neural constraint representations with improved safety and data efficiency relative to state-of-the-art offline ICRL baselines.

Figures & tables

Env.	$\theta^{*}$	MIP $\hat{\theta}$	ICSDICE $\hat{\theta}$	MIP error
Swimmer	$0.04$	$0.0484$	–	$+0.0084$
Quadcopter	$64$	$64$	–	$0$
HalfCheetah	$-3.0$	$-2.98$	$-1.23$	$+0.02$
BlockedAnt	$-3.0$	$-2.05$	$-3.99$	$+0.95$

Table 1: Parametric recovery performance

Env.	SMODICE	ICSDICE	CF-KKT	$\Delta$
Obstacle	$0.275\!\pm\!0.405$	$0.276\!\pm\!0.193$	$\mathbf{0.590\!\pm\!0.153}$	$+0.314$
Wide	$0.547\!\pm\!0.113$	$0.600\!\pm\!0.140$	$\mathbf{0.724\!\pm\!0.083}$	$+0.124$
Disk	$0.363\!\pm\!0.133$	$0.428\!\pm\!0.218$	$\mathbf{0.815\!\pm\!0.057}$	$+0.387$
Zigzag	$0.330\!\pm\!0.054$	$0.394\!\pm\!0.172$	$\mathbf{0.698\!\pm\!0.079}$	$+0.304$
Mean	$0.379$	$0.425$	$\mathbf{0.707}$	$+0.282$

Table 2: Maze constraint IoU (mean

\pm

std over seeds);

\Delta

is relative to ICSDICE.

Figure 1: Ground-truth and learned constraint maps for 4 maze environments across ICSDICE (top), SMODICE (middle), and CF-KKT (bottom). Red, blue, and green denote predicted unsafe states, predicted safe states, and the ground-truth constraint boundary.

Env.	Metric	Oracle	IQ-Learn	SMODICE	ICSDICE	CF-KKT
Obstacle	Goal	$20.0/20$	$10.0/20$	$18.3/20$	$18.7/20$	$20.0/20$
	Viol.	$0.018$	$0.124\pm 0.203$	$0.175\pm 0.053$	$0.174\pm 0.038$	$\mathbf{0.042\pm 0.068}$
	Return	$-58.2$	$-89.7\pm 5.8$	$-47.1\pm 7.4$	$-45.8\pm 1.6$	$-55.6\pm 6.8$
Wide	Goal	$19.0/20$	$5.0/20$	$17.7/20$	$18.3/20$	$17.7/20$
	Viol.	$0.032$	$0.001\pm 0.002$	$0.226\pm 0.111$	$0.200\pm 0.028$	$\mathbf{0.085\pm 0.121}$
	Return	$-70.2$	$-106.5\pm 11.3$	$-58.2\pm 7.3$	$-63.4\pm 7.0$	$-68.5\pm 11.4$

Table 3: Maze downstream planning over 20 starts.“Viol.” is the true violating-timestep fraction.

	ICSDICE		IQ-Learn		SMODICE		CF-KKT
Env.	Ret.	Viol.	Ret.	Viol.	Ret.	Viol.	Ret.	Viol.
Ant_ls	$1257\pm 123$	$87.5\pm 117.0$	$65\pm 155^{\dagger}$	$6.3\pm 1.8$	$970\pm 186$	$86.8\pm 113.5$	$1130\pm 91$	$\mathbf{2.8\pm 0.9}$
BlockedAnt	$2265\pm 400$	$82.3\pm 56.1$	$258\pm 316^{\dagger}$	$145.3\pm 135.2$	$1323\pm 537^{\dagger}$	$124.2\pm 84.7$	$2449\pm 237$	$\mathbf{15.3\pm 4.2}$
BlockedHopper	$420\pm 112$	$110.6\pm 26.6$	$269\pm 274^{\dagger}$	$134.1\pm 260.6$	$325\pm 275$	$82.9\pm 63.4$	$454\pm 252$	$\mathbf{17.6\pm 21.2}$
HalfCheetah	$2657\pm 318$	$83.0\pm 58.9$	$-270\pm 447^{\dagger}$	$0.0\pm 0.0$	$1915\pm 977$	$313.1\pm 349.0$	$2338\pm 788$	$\mathbf{54.3\pm 48.9}$
Walker_ls	$682\pm 392$	$170.1\pm 106.9$	$361\pm 233$	$39.0\pm 16.8$	$140\pm 224$	$\mathbf{37.6\pm 56.5}$	$410\pm 283$	$74.3\pm 56.3$

Table 4: MuJoCo downstream return and violations (

N=10

). Bold marks the lowest mean violation among explicit-mask methods; † denotes severe return collapse.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Quadcopter (top) and Swimmer (bottom) goal-reaching trajectories with the learned collision-avoidance constraint; red denotes the unsafe set.

Task	Base environment	$\dim(\mathcal{S})$	$\dim(\mathcal{A})$	Constraint: infeasible when
Ant_ls	Ant-v4	27	8	$v_{x}>0.5$
BlockedAnt	Ant-v4	27	8	$\sum_{j\in\mathcal{J}}\|\omega_{j}\|>1$
BlockedHopper	Hopper-v4	11	3	$\|\phi_{\mathrm{foot}}\|>0.3$
HalfCheetah	HalfCheetah-v4	18	6	$x_{\mathrm{pos}}<-3$
Walker_ls	Walker2d-v4	17	6	$v_{x}>1$

Appendix

Table 5: MuJoCo benchmark specifications. Here

v_{x}

denotes forward velocity,

\omega_{j}

the angular velocity of joint

j

with

\mathcal{J}

denoting the four constrained joints,

\phi_{\mathrm{foot}}

the foot-joint angle, and

x_{\mathrm{pos}}

the absolute horizontal torso position.

Task	Reward	Constrained quantity
Ant_ls	$v_{x}+1-0.5\\|u\\|^{2}$	forward velocity (rewarded)
BlockedAnt	$v_{x}+1-0.5\\|u\\|^{2}$	joint angular velocities
BlockedHopper	$v_{x}+1-10^{-3}\\|u\\|^{2}$	foot-joint angle
HalfCheetah	$\|v_{x}\|-0.1\\|u\\|^{2}$	horizontal position
Walker_ls	$v_{x}+1-10^{-3}\\|u\\|^{2}$	forward velocity (rewarded)

Appendix

Table 6: Task rewards and constrained physical quantities. The constant

+1

is the survival bonus in the corresponding locomotion task. HalfCheetah rewards

|v_{x}|

, so motion in either horizontal direction contributes positively to the task reward.

Figure 3: Representative maze plans. The dashed contour is the predicted unsafe boundary, green the true unsafe region, purple the oracle route, and yellow the executed timesteps inside the true unsafe region. The goal is indicated by blue star.

	Violations $\downarrow$		Return $\uparrow$
Environment	MLP	MLP + reg.	MLP	MLP + reg.
Ant_ls	$7.1\pm 13.8$	$2.3\pm 0.5$	$1177\pm 98$	$1177\pm 68$
BlockedAnt	$89.4\pm 40.2$	$67.0\pm 19.4$	$2142\pm 621$	$2165\pm 552$
BlockedHopper	$65.1\pm 26.9$	$37.3\pm 30.3$	$342\pm 239$	$277\pm 102$
HalfCheetah (obst.)	$133.5\pm 32.7$	$51.1\pm 33.9$	$3126\pm 407$	$2938\pm 408$
Walker_ls	$138.0\pm 76.4$	$100.2\pm 76.5$	$676\pm 420$	$447\pm 309$

Appendix

Table 7: Learned-reward robustness (mean

\pm

std,

N=10

); MLP + reg. also matches analytical reward gradients.

Figure 4: Constraint violation rate (left) and feasible cumulative reward (right) for the auxiliary online ICRL comparison.

	$H=12$		$H=48$		CF-KKT ( $H=24$ )
Env.	Viol. $\downarrow$	Ret. $\uparrow$	Viol. $\downarrow$	Ret. $\uparrow$	Viol. $\downarrow$	Ret. $\uparrow$
Ant_ls	$3.4\!\pm\!1.5$	$1157\!\pm\!57$	$3.3\!\pm\!1.6$	$\mathbf{1167\!\pm\!84}$	$\mathbf{2.8\!\pm\!0.9}$	$1130\!\pm\!91$
BlockedAnt	$16.5\!\pm\!3.3$	$2136\!\pm\!635$	$18.5\!\pm\!4.9$	$1968\!\pm\!733$	$\mathbf{15.3\!\pm\!4.2}$	$\mathbf{2449\!\pm\!237}$
BlockedHopper	$64.9\!\pm\!83.5$	$317\!\pm\!139$	$28.6\!\pm\!32.7$	$\mathbf{510\!\pm\!433}$	$\mathbf{17.6\!\pm\!21.2}$	$454\!\pm\!252$
HalfCheetah	$93.9\!\pm\!67.5$	$2196\!\pm\!583$	$80.2\!\pm\!78.7$	$\mathbf{2605\!\pm\!621}$	$\mathbf{54.3\!\pm\!48.9}$	$2338\!\pm\!788$
Walker_ls	$85.4\!\pm\!43.1$	$\mathbf{413\!\pm\!213}$	$89.6\!\pm\!65.6$	$406\!\pm\!279$	$\mathbf{74.3\!\pm\!56.3}$	$410\!\pm\!283$

Appendix

Table 8: Effect of counterfactual rollout horizon with the positive-advantage gate fixed. CF-KKT uses the default

H=24

	$q=0.25$		CF-KKT
Env.	Viol. $\downarrow$	Ret. $\uparrow$	Viol. $\downarrow$	Ret. $\uparrow$
Ant_ls	$3.3\!\pm\!1.7$	$1117\!\pm\!102$	$\mathbf{2.8\!\pm\!0.9}$	$\mathbf{1130\!\pm\!91}$
BlockedAnt	$21.8\!\pm\!14.4$	$2096\!\pm\!604$	$\mathbf{15.3\!\pm\!4.2}$	$\mathbf{2449\!\pm\!237}$
BlockedHopper	$27.3\!\pm\!32.3$	$\mathbf{624\!\pm\!261}$	$\mathbf{17.6\!\pm\!21.2}$	$454\!\pm\!252$
HalfCheetah	$84.6\!\pm\!58.2$	$\mathbf{2735\!\pm\!494}$	$\mathbf{54.3\!\pm\!48.9}$	$2338\!\pm\!788$
Walker_ls	$97.3\!\pm\!92.3$	$\mathbf{422\!\pm\!393}$	$\mathbf{74.3\!\pm\!56.3}$	$410\!\pm\!283$

Appendix

Table 9: Effect of the counterfactual acceptance rule at

H=24

. The quantile rule retains the top

25\%

of candidates by

\hat{A}_{V}^{(k)}

, whereas CF-KKT retains all candidates with

\hat{A}_{V}^{(k)}>0

Environment	Bag prec.	State prec.	Enrich.	$h{=}1$	$h{=}H$
Ant_ls	1.000	0.848	3.75	0.87	0.81
BlockedAnt	1.000	1.000	2.72	1.00	1.00
BlockedHopper	0.922	0.749	1.37	0.08	0.72
Walker_ls	1.000	0.969	1.31	1.00	0.95
HalfCheetah	0.953	0.867	1.78	0.00	0.83

Appendix

Table 10: Ground-truth diagnostics of selected CF-KKT counterfactuals over

N=10

seeds. Metrics are defined in the text.

Environment	$\rho^{\star}$	$A$	$B$	$A^{-\mathrm{act}}$	$A^{-\mathrm{cf}}$	KKT-only	CF-KKT
Ant_ls	0.774	0.023	0.014	0.095	0.426	0.992	0.681
BlockedAnt	0.632	0.151	0.149	0.281	0.480	0.890	0.528
BlockedHopper	0.453	0.088	0.080	0.445	0.471	0.586	0.241
HalfCheetah	0.512	0.009	0.003	0.064	0.431	0.394	0.360
Walker_ls	0.259	0.043	0.051	0.208	0.460	0.692	0.203

Appendix

Table 11: Mask admission rates across the seven ablation conditions.

Environment	$A$	$B$	$A^{-\mathrm{act}}$	$A^{-\mathrm{cf}}$	CF-only	KKT-only	CF-KKT
Ant_ls	0.768	0.746	0.804	0.570	0.907	0.458	0.831
BlockedAnt	0.444	0.438	0.446	0.449	0.904	0.522	0.616
BlockedHopper	0.614	0.617	0.574	0.510	0.802	0.481	0.602
HalfCheetah	0.635	0.619	0.636	0.494	0.655	0.976	0.993
Walker_ls	0.779	0.787	0.783	0.524	0.624	0.382	0.705

Appendix

Table 12: ROC AUC of the learned constraint score on held-out transitions.

	CF-only		CF-KKT
Environment	Viol. $\downarrow$	Return $\uparrow$	Viol. $\downarrow$	Return $\uparrow$
Ant_ls	$16.6\pm 10.4$	$246\pm 176^{\dagger}$	$\mathbf{2.8\pm 0.9}$	$\mathbf{1130\pm 91}$
BlockedAnt	$163.4\pm 137.9$	$276\pm 182^{\dagger}$	$\mathbf{15.3\pm 4.2}$	$\mathbf{2449\pm 237}$
BlockedHopper	$2.1\pm 6.1^{\dagger}$	$112\pm 298^{\dagger}$	$17.6\pm 21.2$	$\mathbf{454\pm 252}$
HalfCheetah	$0.0\pm 0.0^{\dagger}$	$-3\pm 3^{\dagger}$	$54.3\pm 48.9$	$\mathbf{2338\pm 788}$
Walker_ls	$0.0\pm 0.0^{\dagger}$	$-6\pm 9^{\dagger}$	$74.3\pm 56.3$	$\mathbf{410\pm 283}$

Appendix

Table 13: Downstream comparison of CF-only and CF-KKT. Mean

\pm

standard deviation over

N=10

seeds; † denotes severe return collapse.

Explore similar work

Learning from Demonstration via Spatiotemporal Tubes for Unknown Euler-Lagrange Systems
Jul 1, 2026Ratnangshu Das, Puneeth Shankar, Varuni Buereddy +2Robotic Control Robot Skill Learning
Robert Bosch Centre for Cyber-Physical Systems IISc, Bengaluru, India
Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Data
May 2, 2026Ruiqi Xue, Lei Yuan, Kainuo Cheng +2LLM-Guided RL Offline RL
National Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Polixir Technologies +2
CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning
Apr 26, 2026Rahul Narava, Siddharth Verma, Ojas Jain +2Control Barrier Functions Constrained RL
Indian Institute of Technology Ropar · CRAN, Université de Lorraine

Jul 1, 2026, cs.ROJ/K move · Enter open · S save

Learning from Demonstration via Spatiotemporal Tubes for Unknown Euler-Lagrange Systems

Ratnangshu Das, Puneeth Shankar, Varuni Buereddy, Ravi Prakash+1

Robert Bosch Centre for Cyber-Physical Systems IISc, Bengaluru, India

Robotic Control Robot Skill Learning Robust Control Learning from Demonstration

Open paper AskPDF arXiv

We present STT-LfD, a unified Learning from Demonstration (LfD) framework that integrates motion learning with control for unknown Euler-Lagrange systems. Unlike traditional decoupled approaches that track a fixed reference, the proposed method treats demonstrations as a data-driven safety specification. Using heteroscedastic Gaussian Processes, STT-LfD learns Spatiotemporal Tubes (STTs) as an intent envelope that capture time-varying precision requirements of a task. A closed-form feedback controller then enforces these learned constraints while respecting actuator limits, without requiring explicit system identification. The approach preserves the temporal structure of demonstrations, remains computationally efficient, and avoids explicit system identification. Hardware experiments on a mobile robot and a 7-DOF manipulator show that it outperforms baselines in robustness to disturbances and computational speed.

Abstract

Explore similar work

Learning from Demonstration via Spatiotemporal Tubes for Unknown Euler-Lagrange Systems

Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Data

CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning