Learning Unknown Constraints without Unsafe Data via Optimality and Counterfactual Regularization

Zhouyu Zhang; Chih-Yuan Chiu; Glen Chou

cs.ROOct 7, 2026

Learning Unknown Constraints without Unsafe Data via Optimality and Counterfactual Regularization

Authors: Zhouyu Zhang, Chih-Yuan Chiu, Glen Chou

Organizations: School of Electrical and Computer Engineering Georgia Institute of Technology Atlanta, GA 30309 · School of Cybersecurity and Privacy School of Aerospace Engineering Georgia Institute of Technology

Abstract

Learning from demonstrations (LfD) provides a framework for inferring unknown constraints from locally optimal, constraint-satisfying expert behavior. Existing approaches largely fall into two paradigms, constrained inverse optimal control (CIOC) and inverse constrained reinforcement learning (ICRL). CIOC exploits optimality conditions such as the Karush--Kuhn--Tucker (KKT) conditions but typically assumes known dynamics and structured constraint representations. Meanwhile, ICRL accommodates complex unknown constraints and unknown transition dynamics but often requires extensive online exploration, during which unsafe constraint violations may occur. In this work, we introduce Counterfactual KKT (CF-KKT), a constraint learning framework that leverages learned dynamics and locally optimal demonstrations to recover unknown constraints without requiring known dynamics or additional risky exploration, thereby combining the data efficiency and safety advantages of CIOC with the flexibility of ICRL. First, we use a locally learned differentiable dynamics model to impose KKT-inspired optimality conditions directly on the demonstrations. Second, we use the learned dynamics to generate reward-improving counterfactual behaviors near the demonstrations, revealing behaviors that would be preferable in the absence of the unknown constraint and thus providing synthetic infeasible data. When the constraint parameterization is known, the same learned-dynamics framework enables direct CIOC-based parameter recovery, and we characterize its sensitivity to dynamics misspecification. Across high-dimensional robotic control tasks, our approach learns neural constraint representations with improved safety and data efficiency relative to state-of-the-art offline ICRL baselines.

Figures & tables

Env.	$\theta^{*}$	MIP $\hat{\theta}$	ICSDICE $\hat{\theta}$	MIP error
Swimmer	$0.04$	$0.0484$	–	$+0.0084$
Quadcopter	$64$	$64$	–	$0$
HalfCheetah	$-3.0$	$-2.98$	$-1.23$	$+0.02$
BlockedAnt	$-3.0$	$-2.05$	$-3.99$	$+0.95$

Table 1: Parametric recovery performance

Env.	SMODICE	ICSDICE	CF-KKT	$\Delta$
Obstacle	$0.275\!\pm\!0.405$	$0.276\!\pm\!0.193$	$\mathbf{0.590\!\pm\!0.153}$	$+0.314$
Wide	$0.547\!\pm\!0.113$	$0.600\!\pm\!0.140$	$\mathbf{0.724\!\pm\!0.083}$	$+0.124$
Disk	$0.363\!\pm\!0.133$	$0.428\!\pm\!0.218$	$\mathbf{0.815\!\pm\!0.057}$	$+0.387$
Zigzag	$0.330\!\pm\!0.054$	$0.394\!\pm\!0.172$	$\mathbf{0.698\!\pm\!0.079}$	$+0.304$
Mean	$0.379$	$0.425$	$\mathbf{0.707}$	$+0.282$

Table 2: Maze constraint IoU (mean

\pm

std over seeds);

\Delta

is relative to ICSDICE.

Figure 1: Ground-truth and learned constraint maps for 4 maze environments across ICSDICE (top), SMODICE (middle), and CF-KKT (bottom). Red, blue, and green denote predicted unsafe states, predicted safe states, and the ground-truth constraint boundary.

Env.	Metric	Oracle	IQ-Learn	SMODICE	ICSDICE	CF-KKT
Obstacle	Goal	$20.0/20$	$10.0/20$	$18.3/20$	$18.7/20$	$20.0/20$
	Viol.	$0.018$	$0.124\pm 0.203$	$0.175\pm 0.053$	$0.174\pm 0.038$	$\mathbf{0.042\pm 0.068}$
	Return	$-58.2$	$-89.7\pm 5.8$	$-47.1\pm 7.4$	$-45.8\pm 1.6$	$-55.6\pm 6.8$
Wide	Goal	$19.0/20$	$5.0/20$	$17.7/20$	$18.3/20$	$17.7/20$
	Viol.	$0.032$	$0.001\pm 0.002$	$0.226\pm 0.111$	$0.200\pm 0.028$	$\mathbf{0.085\pm 0.121}$
	Return	$-70.2$	$-106.5\pm 11.3$	$-58.2\pm 7.3$	$-63.4\pm 7.0$	$-68.5\pm 11.4$

Table 3: Maze downstream planning over 20 starts.“Viol.” is the true violating-timestep fraction.

	ICSDICE		IQ-Learn		SMODICE		CF-KKT
Env.	Ret.	Viol.	Ret.	Viol.	Ret.	Viol.	Ret.	Viol.
Ant_ls	$1257\pm 123$	$87.5\pm 117.0$	$65\pm 155^{\dagger}$	$6.3\pm 1.8$	$970\pm 186$	$86.8\pm 113.5$	$1130\pm 91$	$\mathbf{2.8\pm 0.9}$
BlockedAnt	$2265\pm 400$	$82.3\pm 56.1$	$258\pm 316^{\dagger}$	$145.3\pm 135.2$	$1323\pm 537^{\dagger}$	$124.2\pm 84.7$	$2449\pm 237$	$\mathbf{15.3\pm 4.2}$
BlockedHopper	$420\pm 112$	$110.6\pm 26.6$	$269\pm 274^{\dagger}$	$134.1\pm 260.6$	$325\pm 275$	$82.9\pm 63.4$	$454\pm 252$	$\mathbf{17.6\pm 21.2}$
HalfCheetah	$2657\pm 318$	$83.0\pm 58.9$	$-270\pm 447^{\dagger}$	$0.0\pm 0.0$	$1915\pm 977$	$313.1\pm 349.0$	$2338\pm 788$	$\mathbf{54.3\pm 48.9}$
Walker_ls	$682\pm 392$	$170.1\pm 106.9$	$361\pm 233$	$39.0\pm 16.8$	$140\pm 224$	$\mathbf{37.6\pm 56.5}$	$410\pm 283$	$74.3\pm 56.3$

Table 4: MuJoCo downstream return and violations (

N=10

). Bold marks the lowest mean violation among explicit-mask methods; † denotes severe return collapse.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Quadcopter (top) and Swimmer (bottom) goal-reaching trajectories with the learned collision-avoidance constraint; red denotes the unsafe set.

Task	Base environment	$\dim(\mathcal{S})$	$\dim(\mathcal{A})$	Constraint: infeasible when
Ant_ls	Ant-v4	27	8	$v_{x}>0.5$
BlockedAnt	Ant-v4	27	8	$\sum_{j\in\mathcal{J}}\|\omega_{j}\|>1$
BlockedHopper	Hopper-v4	11	3	$\|\phi_{\mathrm{foot}}\|>0.3$
HalfCheetah	HalfCheetah-v4	18	6	$x_{\mathrm{pos}}<-3$
Walker_ls	Walker2d-v4	17	6	$v_{x}>1$

Appendix

Table 5: MuJoCo benchmark specifications. Here

v_{x}

denotes forward velocity,

\omega_{j}

the angular velocity of joint

j

with

\mathcal{J}

denoting the four constrained joints,

\phi_{\mathrm{foot}}

the foot-joint angle, and

x_{\mathrm{pos}}

the absolute horizontal torso position.

Task	Reward	Constrained quantity
Ant_ls	$v_{x}+1-0.5\\|u\\|^{2}$	forward velocity (rewarded)
BlockedAnt	$v_{x}+1-0.5\\|u\\|^{2}$	joint angular velocities
BlockedHopper	$v_{x}+1-10^{-3}\\|u\\|^{2}$	foot-joint angle
HalfCheetah	$\|v_{x}\|-0.1\\|u\\|^{2}$	horizontal position
Walker_ls	$v_{x}+1-10^{-3}\\|u\\|^{2}$	forward velocity (rewarded)

Appendix

Table 6: Task rewards and constrained physical quantities. The constant

+1

is the survival bonus in the corresponding locomotion task. HalfCheetah rewards

|v_{x}|

, so motion in either horizontal direction contributes positively to the task reward.

Figure 3: Representative maze plans. The dashed contour is the predicted unsafe boundary, green the true unsafe region, purple the oracle route, and yellow the executed timesteps inside the true unsafe region. The goal is indicated by blue star.

	Violations $\downarrow$		Return $\uparrow$
Environment	MLP	MLP + reg.	MLP	MLP + reg.
Ant_ls	$7.1\pm 13.8$	$2.3\pm 0.5$	$1177\pm 98$	$1177\pm 68$
BlockedAnt	$89.4\pm 40.2$	$67.0\pm 19.4$	$2142\pm 621$	$2165\pm 552$
BlockedHopper	$65.1\pm 26.9$	$37.3\pm 30.3$	$342\pm 239$	$277\pm 102$
HalfCheetah (obst.)	$133.5\pm 32.7$	$51.1\pm 33.9$	$3126\pm 407$	$2938\pm 408$
Walker_ls	$138.0\pm 76.4$	$100.2\pm 76.5$	$676\pm 420$	$447\pm 309$

Appendix

Table 7: Learned-reward robustness (mean

\pm

std,

N=10

); MLP + reg. also matches analytical reward gradients.

Figure 4: Constraint violation rate (left) and feasible cumulative reward (right) for the auxiliary online ICRL comparison.

	$H=12$		$H=48$		CF-KKT ( $H=24$ )
Env.	Viol. $\downarrow$	Ret. $\uparrow$	Viol. $\downarrow$	Ret. $\uparrow$	Viol. $\downarrow$	Ret. $\uparrow$
Ant_ls	$3.4\!\pm\!1.5$	$1157\!\pm\!57$	$3.3\!\pm\!1.6$	$\mathbf{1167\!\pm\!84}$	$\mathbf{2.8\!\pm\!0.9}$	$1130\!\pm\!91$
BlockedAnt	$16.5\!\pm\!3.3$	$2136\!\pm\!635$	$18.5\!\pm\!4.9$	$1968\!\pm\!733$	$\mathbf{15.3\!\pm\!4.2}$	$\mathbf{2449\!\pm\!237}$
BlockedHopper	$64.9\!\pm\!83.5$	$317\!\pm\!139$	$28.6\!\pm\!32.7$	$\mathbf{510\!\pm\!433}$	$\mathbf{17.6\!\pm\!21.2}$	$454\!\pm\!252$
HalfCheetah	$93.9\!\pm\!67.5$	$2196\!\pm\!583$	$80.2\!\pm\!78.7$	$\mathbf{2605\!\pm\!621}$	$\mathbf{54.3\!\pm\!48.9}$	$2338\!\pm\!788$
Walker_ls	$85.4\!\pm\!43.1$	$\mathbf{413\!\pm\!213}$	$89.6\!\pm\!65.6$	$406\!\pm\!279$	$\mathbf{74.3\!\pm\!56.3}$	$410\!\pm\!283$

Appendix

Table 8: Effect of counterfactual rollout horizon with the positive-advantage gate fixed. CF-KKT uses the default

H=24

	$q=0.25$		CF-KKT
Env.	Viol. $\downarrow$	Ret. $\uparrow$	Viol. $\downarrow$	Ret. $\uparrow$
Ant_ls	$3.3\!\pm\!1.7$	$1117\!\pm\!102$	$\mathbf{2.8\!\pm\!0.9}$	$\mathbf{1130\!\pm\!91}$
BlockedAnt	$21.8\!\pm\!14.4$	$2096\!\pm\!604$	$\mathbf{15.3\!\pm\!4.2}$	$\mathbf{2449\!\pm\!237}$
BlockedHopper	$27.3\!\pm\!32.3$	$\mathbf{624\!\pm\!261}$	$\mathbf{17.6\!\pm\!21.2}$	$454\!\pm\!252$
HalfCheetah	$84.6\!\pm\!58.2$	$\mathbf{2735\!\pm\!494}$	$\mathbf{54.3\!\pm\!48.9}$	$2338\!\pm\!788$
Walker_ls	$97.3\!\pm\!92.3$	$\mathbf{422\!\pm\!393}$	$\mathbf{74.3\!\pm\!56.3}$	$410\!\pm\!283$

Appendix

Table 9: Effect of the counterfactual acceptance rule at

H=24

. The quantile rule retains the top

25\%

of candidates by

\hat{A}_{V}^{(k)}

, whereas CF-KKT retains all candidates with

\hat{A}_{V}^{(k)}>0

Environment	Bag prec.	State prec.	Enrich.	$h{=}1$	$h{=}H$
Ant_ls	1.000	0.848	3.75	0.87	0.81
BlockedAnt	1.000	1.000	2.72	1.00	1.00
BlockedHopper	0.922	0.749	1.37	0.08	0.72
Walker_ls	1.000	0.969	1.31	1.00	0.95
HalfCheetah	0.953	0.867	1.78	0.00	0.83

Appendix

Table 10: Ground-truth diagnostics of selected CF-KKT counterfactuals over

N=10

seeds. Metrics are defined in the text.

Environment	$\rho^{\star}$	$A$	$B$	$A^{-\mathrm{act}}$	$A^{-\mathrm{cf}}$	KKT-only	CF-KKT
Ant_ls	0.774	0.023	0.014	0.095	0.426	0.992	0.681
BlockedAnt	0.632	0.151	0.149	0.281	0.480	0.890	0.528
BlockedHopper	0.453	0.088	0.080	0.445	0.471	0.586	0.241
HalfCheetah	0.512	0.009	0.003	0.064	0.431	0.394	0.360
Walker_ls	0.259	0.043	0.051	0.208	0.460	0.692	0.203

Appendix

Table 11: Mask admission rates across the seven ablation conditions.

Environment	$A$	$B$	$A^{-\mathrm{act}}$	$A^{-\mathrm{cf}}$	CF-only	KKT-only	CF-KKT
Ant_ls	0.768	0.746	0.804	0.570	0.907	0.458	0.831
BlockedAnt	0.444	0.438	0.446	0.449	0.904	0.522	0.616
BlockedHopper	0.614	0.617	0.574	0.510	0.802	0.481	0.602
HalfCheetah	0.635	0.619	0.636	0.494	0.655	0.976	0.993
Walker_ls	0.779	0.787	0.783	0.524	0.624	0.382	0.705

Appendix

Table 12: ROC AUC of the learned constraint score on held-out transitions.

	CF-only		CF-KKT
Environment	Viol. $\downarrow$	Return $\uparrow$	Viol. $\downarrow$	Return $\uparrow$
Ant_ls	$16.6\pm 10.4$	$246\pm 176^{\dagger}$	$\mathbf{2.8\pm 0.9}$	$\mathbf{1130\pm 91}$
BlockedAnt	$163.4\pm 137.9$	$276\pm 182^{\dagger}$	$\mathbf{15.3\pm 4.2}$	$\mathbf{2449\pm 237}$
BlockedHopper	$2.1\pm 6.1^{\dagger}$	$112\pm 298^{\dagger}$	$17.6\pm 21.2$	$\mathbf{454\pm 252}$
HalfCheetah	$0.0\pm 0.0^{\dagger}$	$-3\pm 3^{\dagger}$	$54.3\pm 48.9$	$\mathbf{2338\pm 788}$
Walker_ls	$0.0\pm 0.0^{\dagger}$	$-6\pm 9^{\dagger}$	$74.3\pm 56.3$	$\mathbf{410\pm 283}$

Appendix

Table 13: Downstream comparison of CF-only and CF-KKT. Mean

\pm

standard deviation over

N=10

seeds; † denotes severe return collapse.

Explore similar work

Jul 1, 2026cs.RO

Learning from Demonstration via Spatiotemporal Tubes for Unknown Euler-Lagrange Systems

We present STT-LfD, a unified Learning from Demonstration (LfD) framework that integrates motion learning with control for unknown Euler-Lagrange systems. Unlike traditional decoupled approaches that track a fixed reference, the proposed method treats demonstrations as a data-driven safety specification. Using heteroscedastic Gaussian Processes, STT-LfD learns Spatiotemporal Tubes (STTs) as an intent envelope that capture time-varying precision requirements of a task. A closed-form feedback controller then enforces these learned constraints while respecting actuator limits, without requiring explicit system identification. The approach preserves the temporal structure of demonstrations, remains computationally efficient, and avoids explicit system identification. Hardware experiments on a mobile robot and a 7-DOF manipulator show that it outperforms baselines in robustness to disturbances and computational speed.

Ratnangshu Das, Puneeth Shankar, Varuni Buereddy +2

Robert Bosch Centre for Cyber-Physical Systems IISc, Bengaluru, India

Robotic Control Robot Skill Learning Robust Control Learning from Demonstration

Ask PDF

May 2, 2026cs.LG

Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Data

Learning constraint-satisfying policies from offline data without risky online interaction is crucial for safety-critical decision making. Conventional methods typically learn cost value functions from abundant unsafe samples to define safety boundaries and penalize violations. However, in high-stakes scenarios, risky trial-and-error is infeasible, yielding datasets with few or no unsafe samples. Under this limitation, existing approaches often treat all data as uniformly safe, overlooking safe-but-infeasible states - states that currently satisfy constraints but inevitably violate them within a few steps - leading to deployment failures. Drawing inspiration from the concept of knowledge-data integration, we leverage large language models (LLMs) to incorporate natural language knowledge into the policy to address this challenge. Specifically, we propose PROCO, a model-based offline safe reinforcement learning (RL) framework tailored to datasets largely free of violations. PROCO first learns a dynamics model from offline data and constructs a conservative cost function by grounding natural-language knowledge of unsafe states in LLMs, enabling risk estimation even without observed violations. Using the cost function and learned model, PROCO performs model-based rollouts to synthesize diverse counterfactual unsafe samples, supporting reliable feasibility identification and feasibility-guided policy learning. Across a range of Safety-Gymnasium tasks with exclusively safe or minimally risky training data, PROCO integrates seamlessly with a variety of offline safe RL algorithms and consistently demonstrates reduced constraint violations and improved safety performance compared to both the original methods and other behavior cloning baselines.

Ruiqi Xue, Lei Yuan, Kainuo Cheng +2

National Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Polixir Technologies +2

LLM-Guided RL Offline RL Safe RL Model-Based RL

Ask PDF

Apr 26, 2026cs.LG

CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning

Ensuring safe exploration in high-dimensional systems with unknown dynamics remains a significant challenge. Existing safe reinforcement learning methods often provide safety guarantees only in expectation, which can still lead to safety violations. Control-theoretic approaches, in contrast, offer hard constraint-based safety guarantees but typically assume access to known system dynamics or require accurate estimation of control-affine models. In this paper, we propose a safe reinforcement learning framework that learns a probabilistic control-affine dynamics model in an offline setting. The learned model is leveraged to explicitly construct control barrier functions (CBFs) that incorporate model uncertainty to provide conservative safety constraints. These CBF constraints are enforced through an online constraint-based action correction mechanism, enabling safe exploration without overly restricting task performance. Empirical evaluations on nonlinear, complex continuous-control benchmarks demonstrate that our approach achieves returns comparable to those of existing baselines while significantly reducing safety violations.

Rahul Narava, Siddharth Verma, Ojas Jain +2

Indian Institute of Technology Ropar · CRAN, Université de Lorraine

Control Barrier Functions Constrained RL Safe RL Model-Based RL

Ask PDF