Discovering What You Can Control: Interventional Boundary Discovery for Reinforcement Learning

Jiaxin Liu; Anzhe Cheng; Paul Bogdan

cs.LGMar 18, 2026

Discovering What You Can Control: Interventional Boundary Discovery for Reinforcement Learning

Authors: Jiaxin Liu, Anzhe Cheng, Paul Bogdan

Organizations: University of Illinois Urbana-Champaign · University of Southern California

Abstract

When an RL agent's observations contain distractors driven by the same confounders as its true state, observational data alone cannot identify which dimensions the agent controls. In our benchmarks, even state-conditioned observational selectors can collapse when distractors mimic controllable state variables. We propose Interventional Boundary Discovery (IBD), which treats the agent's own action channel as a source of randomized interventions: randomizing actions implements an interventional contrast, and per-dimension two-sample tests with FDR correction produce a binary mask over observation dimensions. Across 12 continuous-control settings with up to 100 distractors, IBD matches oracle return in 11 of 12 settings, while observational baselines including mutual information, state-conditioned forward models, and gradient-based sensitivity often underperform simply passing the full observation to SAC. Code is available at https://github.com/jiaxin26/IBD-RL

Figures & tables

Figure 1: Overview of Interventional Boundary Discovery (IBD). From the same reset distribution, IBD collects two independent rollouts: a baseline branch under a probe policy $\pi_{\mathrm{probe}}$ , and an intervention branch in which the action mechanism is replaced by a confounder-independent randomized policy, $\tilde{a}_{t}\sim\mathrm{Unif}(\mathcal{A})$ . For each observation dimension and horizon, IBD compares trajectory statistics across the two branches using a two-sample test. Benjamini–Hochberg FDR correction then yields a binary SoI mask, which filters observations before downstream RL training.

Environment	Distr.	Full State	IBD (ours)	Oracle	MI Select	Var Select	Cond. MI
walker_walk	easy	857 $\pm$ 139	854 $\pm$ 127	785 $\pm$ 128	705 $\pm$ 98	750 $\pm$ 103	884 $\pm$ 70
walker_walk	medium	902 $\pm$ 26	792 $\pm$ 112	785 $\pm$ 128	129 $\pm$ 79	84 $\pm$ 6	220 $\pm$ 135
walker_walk	hard	414 $\pm$ 144	842 $\pm$ 121	785 $\pm$ 128	87 $\pm$ 34	46 $\pm$ 10	81 $\pm$ 23
cheetah_run	easy	508 $\pm$ 39	429 $\pm$ 80	472 $\pm$ 35	161 $\pm$ 42	85 $\pm$ 19	270 $\pm$ 112
cheetah_run	medium	113 $\pm$ 34	479 $\pm$ 95	472 $\pm$ 35	38 $\pm$ 11	62 $\pm$ 25	70 $\pm$ 37
cheetah_run	hard	63 $\pm$ 12	436 $\pm$ 70	472 $\pm$ 35	33 $\pm$ 6	39 $\pm$ 3	62 $\pm$ 41

Table 1: Main results (episode return, mean

\pm

std over 5 seeds). Bold : best non-oracle method (within 1 std).

Environment	Full State	Cond. MI	Grad. Attr.	IBD (ours)	Oracle
cartpole_swingup	725 $\pm$ 48	161 $\pm$ 60	56 $\pm$ 32	834 $\pm$ 26	821 $\pm$ 35
finger_spin	365 $\pm$ 13	4 $\pm$ 6	62 $\pm$ 113	587 $\pm$ 178	775 $\pm$ 181
hopper_hop	0 $\pm$ 1	0 $\pm$ 0	0 $\pm$ 0	6 $\pm$ 12	6 $\pm$ 8
reacher_hard	12 $\pm$ 7	7 $\pm$ 6	5 $\pm$ 10	929 $\pm$ 76	925 $\pm$ 75

Table 2: Gradient attribution baseline (episode return, mean

\pm

std over 5 seeds, medium distractors). Grad. Attr. performs comparably to or worse than Cond. MI, confirming that the failure of observational methods under spurious correlation is not resolved by gradient-based sensitivity analysis.

Figure 2: Distractor scaling curve. Episode return as a function of distractor dimensionality (6, 50, 100) for walker_walk (left), cheetah_run (center), and reacher_hard (right). Full State (red) degrades as distractors increase; IBD (blue) tracks oracle performance across all distractor counts. Shaded regions: $\pm$ 1 std over 5 seeds.

Environment	Distr.	Full State	Return	P / R / F1	Return	P / R / F1	Oracle
			Inverse Dyn.		IBD (ours)
reacher_hard	medium	12 $\pm$ 7	28 $\pm$ 53	0.67 / 0.67 / 0.67	929 $\pm$ 76	0.97 / 1.00 / 0.98	925 $\pm$ 75
walker_walk	hard	414 $\pm$ 144	624 $\pm$ 26	0.88 / 0.88 / 0.88	842 $\pm$ 121	0.97 / 0.85 / 0.91	785 $\pm$ 128
cheetah_run	medium	113 $\pm$ 34	285 $\pm$ 74	0.95 / 0.95 / 0.95	479 $\pm$ 95	0.94 / 1.00 / 0.97	472 $\pm$ 35
finger_spin	medium	365 $\pm$ 13	21 $\pm$ 26	0.44 / 0.44 / 0.44	587 $\pm$ 178	0.96 / 1.00 / 0.98	775 $\pm$ 181

Table 3: Multistep inverse dynamics baseline: episode return (mean

\pm

std over 5 seeds, medium/hard distractors) and boundary discovery accuracy (precision / recall / F1). Bold: best non-oracle method on return.

Method	$\kappa{=}0,\lambda{=}0$	$\kappa{=}0.5,\lambda{=}0$	$\kappa{=}0,\lambda{=}1$	$\kappa{=}0.5,\lambda{=}1$
IBD (ours)	0.934	0.934	0.949	0.949
Inverse Dyn.	0.870	0.833	0.872	0.733
MI Select	0.919	0.724	0.919	0.403
Grad. Attr.	0.563	0.531	0.559	0.457
Cond. MI	0.498	0.456	0.499	0.455
Var Select	0.329	0.356	0.359	0.368

Table 4: Confounded distractor regime (boundary F1, mean over walker_walk , cheetah_run , and reacher_hard ,

5

seeds each). A latent AR(1) process

C_{t}

enters the action channel with strength

\kappa

and drives

12

distractor dimensions with strength

\lambda

; the backdoor path is open only when both edges are active (

\kappa{=}0.5,\lambda{=}1

). Per-task results and false-positive breakdowns are in Appendix I .

Figure 3: Where does feature selection help? Episode return across 8 representative settings. Three regimes emerge. (1) walker_walk (easy): all methods perform similarly, so no selection is needed. (2) Most medium/hard settings: Oracle $\gg$ Full State but IBD $\approx$ Oracle, so distractors are the bottleneck and IBD resolves them. (3) hopper_hop : all methods near zero, so the bottleneck is exploration, not feature selection.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Partial controllability detection. Recall on partially controllable dimensions as a function of mixing coefficient $\alpha$ , for cheetah_run and walker_walk . At $\alpha=0$ (purely exogenous), recall is correctly zero; by $\alpha\approx 0.05$ ( ${\sim}5\%$ causal variance), recall reaches 1.0. The dashed line shows overall F1 on cheetah_run , confirming that boundary quality remains high. Precision $\geq 0.92$ at all $\alpha$ values. Shaded regions: $\pm$ 1 std over 3 seeds.

Task	Scout Budget	Precision	Recall	F1
cheetah_run	0	0.944	1.000	0.971
	80K	0.944	1.000	0.971
reacher_hard	0	0.952	1.000	0.974
	80K	0.952	1.000	0.974
walker_walk	0	0.910	0.972	0.940
	80K	0.902	0.889	0.895

Appendix

Table 5: Scout ablation (P/R/F1, mean over 3 seeds). Budget

=0

: structured random probe;

\geq

10K: SAC scout. Boundary accuracy is identical or better without RL pre-training.

Environment	Distr.	Precision	Recall	F1
walker_ walk	easy	0.98	0.92	0.95
walker_ walk	medium	0.95	0.86	0.90
walker_ walk	hard	0.97	0.85	0.91
cheetah_ run	easy	0.98	1.00	0.99
cheetah_ run	medium	0.94	1.00	0.97
cheetah_ run	hard	0.96	1.00	0.98

Appendix

Table 6: IBD boundary discovery accuracy (P/R/F1, mean over 5 seeds). High precision = few distractor dims leaked; high recall = few true dims missed.

Component	Parameter	Value
SAC / TD3	LR / batch / buffer / arch	$3{\times}10^{-4}$ / 256 / 300K / MLP [256, 256]
TD3 only	Action noise $\sigma$	0.1
IBD Probe	Trajectories $N$ / length $T$ / horizons $\mathcal{H}$ / $\alpha$	80 / 200 / {1,5,10} / 0.05
Training	Steps / eval freq / eval eps / seeds	300K / 50K / 10 / {42,142,242,342,442}
Cond. MI	Arch / epochs / LR / batch / data	MLP [64,64] / 50 / $10^{-3}$ / 2048 / 200 $\times$ 200
Grad. Attr.	Arch / epochs / LR / batch / data	MLP [128,128] / 80 / $10^{-3}$ / 2048 / 200 $\times$ 200

Appendix

Table 7: Hyperparameters used in all experiments.

Figure 5: Distractor scaling curves (dense sweep). Episode return vs. distractor count

d

for walker_walk (

d_{c}\!=\!24

, left), cheetah_run (

d_{c}\!=\!17

, center), and reacher_hard (

d_{c}\!=\!6

, right). Full State (red) collapses earlier on tasks with smaller

d_{c}

; IBD (blue) tracks Oracle across all distractor counts. Shaded regions:

\pm 1

std over

3

seeds.

Figure 6: Distractor scaling on the ratio axis. Normalized return

R/R_{\mathrm{oracle}}

vs. distractor-to-signal ratio

d/d_{c}

on log–log axes. Filled markers: Full State; open markers: IBD ; colors distinguish tasks. Solid lines are descriptive power-law fits restricted to

d\geq d_{c}

(vertical dotted line). Pooled across tasks, Full State follows

R/R_{\mathrm{oracle}}\propto(d/d_{c})^{-1.71}

(

R^{2}=0.73

); IBD stays at the oracle level with

\alpha\approx 0

Task	Method	F1	FP conf	FP exo
walker_walk	IBD (ours)	0.917 $\pm$ 0.035	0.2	2.0
	Inverse Dyn.	0.650 $\pm$ 0.020	8.4	0.0
	MI Select	0.442 $\pm$ 0.020	11.8	1.6
	Grad. Attr.	0.308 $\pm$ 0.020	8.4	8.2
	Cond. MI	0.350 $\pm$ 0.020	2.6	13.0
	Var Select	0.358 $\pm$ 0.020	0.0	15.4

Appendix

Table 8: Per-task results in the fully confounded cell (

\kappa=0.5

\lambda=1

; mean

\pm

std over

5

seeds). FP conf : confounded dimensions admitted (out of

12

). FP exo : exogenous distractors admitted. Bold: best F1 per task.

Setting	Statistic	Level recall	Precision	Recall	F1
cheetah_run	delta	0.067 $\pm$ 0.082	0.956 $\pm$ 0.022	0.757 $\pm$ 0.021	0.845 $\pm$ 0.020
	level	1.000 $\pm$ 0.000	0.958 $\pm$ 0.055	0.548 $\pm$ 0.044	0.695 $\pm$ 0.039
	both	1.000 $\pm$ 0.000	0.936 $\pm$ 0.030	1.000 $\pm$ 0.000	0.967 $\pm$ 0.016
reacher_hard	delta	0.000 $\pm$ 0.000	0.893 $\pm$ 0.096	0.500 $\pm$ 0.000	0.639 $\pm$ 0.025
	level	1.000 $\pm$ 0.000	0.931 $\pm$ 0.090	0.583 $\pm$ 0.000	0.715 $\pm$ 0.028
	both	1.000 $\pm$ 0.000	0.905 $\pm$ 0.090	1.000 $\pm$ 0.000	0.948 $\pm$ 0.050

Appendix

Table 9: Detection of level-shift dimensions (mean

\pm

std over

5

seeds, medium distractors,

6

constructed dimensions per task). Level recall is the detection rate on the constructed dimensions; precision, recall, and F1 are over the full boundary.

Variant	Precision	Recall	F1	Realized FDR	FP conf
Welch + BH (default)	0.913	0.975	0.942	0.087	0.23
Welch + BY	1.000	0.954	0.975	0.000	0.00
KS + BH	0.946	0.969	0.956	0.054	0.07
KS + BY	0.993	0.938	0.963	0.007	0.00

Appendix

Table 10: Test-statistic and multiplicity variants, pooled over the four confounding cells,

3

tasks, and

5

seeds (

n=60

). FDR is the realized false discovery proportion at nominal level

\alpha=0.05

RL Control

Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.

See all 272 papers

Sep 11, 2026

MInTRL: Off-policy Intervention can boost On-policy RL

Reinforcement learning with verifiable rewards is typically performed on-policy, keeping training data close to the current policy but limiting learning to trajectories that the policy can discover itself. Off-policy methods such as supervised fine-tuning, on the other hand, can leverage external knowledge beyond the base model's capabilities, but may suffer from large distribution shift. The key challenge is thus to expand exploration without sacrificing learnability. In this work, we introduce Minimal Intervention Reinforcement Learning (MInTRL), which expands the exploration frontier through sparse, local interventions in otherwise on-policy rollouts. During generation, a judge-intervention policy periodically reviews the current policy's output, replaces erroneous suffixes with short corrections, and immediately returns control to the policy. During training, MInTRL adopts a sequence-level advantage-regression objective that eliminates the need for importance sampling. We show that sparse, local interventions can substantially improve coverage beyond finite-budget on-policy sampling while preserving the overall on-policy nature of the resulting trajectories. Across math and code benchmarks, MInTRL consistently outperforms standard on-policy and off-policy baselines. Ablations show that MInTRL remains effective with self-intervention and across different judge policies, while performance peaks at moderate intervention intensity, highlighting the importance of intervening minimally. These results establish minimal intervention as an effective paradigm for enhancing on-policy RL.

Mingyu Chen, Yefan Tao, Gerald Friedland +2

Amazon Web Services · Boston University

Reinforcement Learning Reinforcement Learning with Verifiable Rewards RL Post-Training Policy Learning Off-Policy RL

Ask PDF

Jul 8, 2026

ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies

Explainability remains a key issue in reinforcement learning (RL). Distilling an interpretable policy from an agent trained in a complex environment is particularly challenging when the action space is continuous. We introduce ORCAID, a novel method for extracting interpretable rule-based policies from RL agents operating in mixed continuous-discrete environments with continuous action spaces. Our main contribution is an efficient oblique decision tree training algorithm that partitions the state space by hyperplanes and fits local linear models. The key idea lies in a three-stage split search: efficient random initialization, local refinement, and backward elimination. Finally, adjacent leaves are merged to yield a concise set of interpretable rules describing a given deep RL policy. We evaluate ORCAID across multiple RL environments, demonstrating that the extracted rule-based policies maintain strong performance with a low number of parameters and can even be used to improve the performance of the original deep RL policy.

Ignacio D. Lopez-Miguel, Ezio Bartocci, Thomas Eiter +1

TU Wien Vienna, Austria

Reinforcement Learning Interpretable ML Interpretability in RL

Ask PDF

Oct 2, 2026

UniIntervene++: An Adaptive Intervention Agent for Efficient Real-World Reinforcement Learning

Online reinforcement learning (RL) enables robot policies to improve through physical interaction, but the assistance they require changes as their competence evolves. Existing intervention strategies based on offline estimates or fixed decision rules can therefore become mismatched to the current policy. To address this, we propose UniIntervene++, an adaptive intervention agent that learns to allocate control between autonomous execution and heterogeneous assisted behaviors during online RL. Specifically, UniIntervene++ first formulates the evolving RL policy, trajectory correction, and a task-structured CodePolicy as Options in a unified semi-Markov decision process and learns their relative values online. Building on this, competence-adaptive intervention periodically probes the RL policy through unassisted execution, keeping control allocation responsive to its evolving capability. Finally, coupled experience learning allows assisted behaviors to improve the RL policy, whose evolving outcomes in turn reshape future intervention decisions. In this way, UniIntervene++ jointly determines when to intervene, how to intervene, and when to return control as the RL policy improves. Across five real-world manipulation tasks, UniIntervene++ achieves an average success rate of 89.67%, outperforming all baselines by at least 6 percentage points, while reducing human intervention to 0.77%, a relative reduction of at least 94.6% from the best baseline. Code is available in our GitHub repository.

Yudong Lin, Haoyuan Deng, Zhuoxuan Yuan +3

School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore

Reinforcement Learning Robotic RL Robot Policy Adaptation

Ask PDF

Abstract

RL Control

Explore similar work

MInTRL: Off-policy Intervention can boost On-policy RL

ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies

UniIntervene++: An Adaptive Intervention Agent for Efficient Real-World Reinforcement Learning