GSBR: Geometric Support Behavior Regularization for Online RL

Charles Xu; Jacob Berg; Liyiming Ke; Sergey Levine

cs.RONov 3, 2026

GSBR: Geometric Support Behavior Regularization for Online RL

Authors: Charles Xu, Jacob Berg, Liyiming Ke, Sergey Levine

Organizations: UC Berkeley · Physical Intelligence

Abstract

Regularizing online reinforcement learning (RL) toward a pretrained base policy can improve sample efficiency and learning stability, but regularization that is too restrictive limits how much the policy can improve. Base policies -- typically diffusion or flow-matching models trained by imitation learning on large, diverse datasets -- capture many behaviors that are reasonable but suboptimal. Existing regularizers using such base policies are too restrictive in one of two ways: behavioral cloning regularizers pull the actor toward the full base-policy distribution, penalizing it for discarding suboptimal modes, while hard support constraints forbid improvements that lie just outside the base policy's support. We propose Geometric Support Behavior Regularization (GSBR), which penalizes actions by their distance from the base policy's support. Actions within the support incur no penalty, so the actor can concentrate on high-value modes; outside the support, where value estimates are less reliable, the penalty grows smoothly with distance while still permitting improvement when the value gained outweighs the penalty. GSBR estimates this distance from a finite set of base-policy samples, so it requires only sample access to the base policy. Across 19 tasks on four robotic benchmarks, GSBR raises the average success rate from 33% for the base policy to 95%, achieving the highest average success rate and throughput among all compared methods, with about 23% higher normalized throughput than the next-best RL method. Project website and open-source code at https://charlesxu0124.github.io/gsbr/ .

Figures & tables

Figure 1: Left: (a) Behavior cloning regularizer matches the base policy distribution, not allowing the RL actor to drop poor modes. (b) Support constraint allows the actor to select the best behavior in the base policy distribution, but excludes improvements beyond it. (c) GSBR applies distance-based regularization on deviation outside the base policy support, allowing the actor to discard suboptimal behaviors and guide exploration beyond the prior. Right: GSBR achieves 23% higher normalized throughput across 19 tasks and 4 benchmarks than previous methods.

Figure 2: GSBR . To implement the geometric support regularizer on a generative robot policy, draw samples from $\pi_{\beta}(\cdot\mid s)$ and compute the nearest-neighbor distance between them. Set $r_{\mathcal{B}}$ to a quantile over these distances and construct the support region $C$ as the union of balls of radius $r_{\mathcal{B}}$ around each sample. The behavior regularizer loss is computed as the Euclidean distance to the closest point in $C$ .

Figure 3: Benchmarks . We evaluate our method across the four benchmarks above. From left to right, the tasks are RoboMimic tool-hang , OGBench cube-triple-task4 , OCBench chamber-lite-easy-task3 , and LIBERO task29 .

Figure 4: Normalized throughput across benchmarks . GSBR achieves the highest average normalized throughput across all 19 tasks and leads all baselines on RoboMimic , OGBench , and OCBench . Throughput (success rate / mean episode length) is normalized by the best method’s seed-average value per task, then averaged across tasks. Error bars show SEM across seeds.

Method	RoboMimic	OGBench	OCBench	LIBERO	All tasks
	(4 tasks)	(5 tasks)	(5 tasks)	(5 tasks)	(19 tasks)
Base Policy	$44$	$0$	$41$	$49$	$33$
TD3+BC	$96\pm 2$	$73\pm 1$	$98\pm 0$	$54\pm 2$	$79\pm 1$
RLT	$96\pm 2$	$90\pm 5$	$98\pm 0$	$\mathbf{86}\pm 2$	$92\pm 1$
Residual Policy	$94\pm 2$	$68\pm 10$	$95\pm 0$	$78\pm 1$	$83\pm 2$
BFN	$86\pm 2$	$69\pm 5$	$88\pm 2$	$66\pm 1$	$77\pm 1$

Table 1: Average success rate (%) across benchmarks . Uncertainty is SEM across three seed-wise task averages. Values are rounded to the nearest integer; bold marks the highest unrounded mean in each column. Base Policy SEM is omitted because complete per-seed evidence is unavailable.

Reference count $N$	Normalized TP
8 (Ours)	$\textbf{1.00}\pm\textbf{0.05}$
16	$0.91\pm 0.02$
32	$0.93\pm 0.02$

Table 2: Reference-count ablation. On cube-triple , the number of base policy samples,

N

, has little effect on performance.

Figure 5: Visualization of the true and estimated support region on cube-triple . The estimated support region (in bronze) closely matches the true support using only 8 samples. The RL actor learned an action just outside the support region (marked with star).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Absolute throughput (successes/h) over training steps

RoboMimic
Task	BC	TD3+BC	RLT	Residual policy	BFN	DSRL	EXPO-FT	DICE-RL	GSBR (ours)
can-mh	$73{\pm}1$	$\mathbf{99}{\pm}0$	$\mathbf{99}{\pm}0$	$97{\pm}1$	$94{\pm}2$	$\mathbf{99}{\pm}0$	$\mathbf{99}{\pm}0$	$\mathbf{99}{\pm}0$	$98{\pm}1$
square-mh	$37{\pm}1$	$96{\pm}1$	$96{\pm}1$	$98{\pm}0$	$78{\pm}3$	$98{\pm}0$	$98{\pm}0$	$96{\pm}2$	$\mathbf{99}{\pm}0$
transport-mh	$15{\pm}2$	$88{\pm}7$	$90{\pm}6$	$83{\pm}7$	$81{\pm}4$	$92{\pm}2$	$93{\pm}1$	$84{\pm}3$	$\mathbf{96}{\pm}1$
tool-hang-ph	$50{\pm}2$	$97{\pm}0$	$97{\pm}0$	$97{\pm}2$	$90{\pm}2$	$96{\pm}2$	$95{\pm}1$	$\mathbf{98}{\pm}0$	$98{\pm}0$
OGBench cube-triple

Appendix

Table 4: Final success rate (%) across all tasks . Values are mean

\pm

SEM.

RoboMimic
Task	BC	TD3+BC	RLT	Residual policy	BFN	DSRL	EXPO-FT	DICE-RL	GSBR (ours)
can-mh	$122{\pm}1$	$329{\pm}6$	$339{\pm}8$	$389{\pm}21$	$198{\pm}10$	$308{\pm}7$	$317{\pm}5$	$289{\pm}6$	$\mathbf{416}{\pm}16$
square-mh	$40{\pm}1$	$247{\pm}6$	$273{\pm}13$	$325{\pm}6$	$113{\pm}7$	$297{\pm}6$	$273{\pm}3$	$226{\pm}20$	$\mathbf{348}{\pm}8$
transport-mh	$7{\pm}1$	$80{\pm}11$	$86{\pm}11$	$65{\pm}11$	$63{\pm}5$	$104{\pm}5$	$90{\pm}2$	$52{\pm}2$	$\mathbf{105}{\pm}2$
tool-hang-ph	$24{\pm}2$	$133{\pm}3$	$127{\pm}5$	$112{\pm}10$	$80{\pm}7$	$121{\pm}12$	$143{\pm}10$	$128{\pm}6$	$\mathbf{165}{\pm}3$
OGBench cube-triple

Appendix

Table 5: Absolute throughput (successes/hour) across all tasks and methods assuming 10 Hz for RoboMimic, OGBench, and OCBench, and 20 Hz for LIBERO. Values are mean

\pm

SEM.

Benchmark	Tasks	Online Env Steps	UTD	$H$	$\gamma$	TD clip
RoboMimic	can , square	1M	2	5	$0.998$	$[0,1]$
RoboMimic	transport , tool-hang	2M	2	5	$0.998$	$[0,1]$
OGBench	cube-triple-T[1-5]	2M	2	5	$0.99$	None
OCBench Lite	block-double-T[1-2] chamber-easy-T[1-3]	1M	2	3	$0.99$	$[0,1]$
LIBERO	Task-{14, 29, 54, 60, 64}	1,000 episodes	2	10	$0.975$ per chunk	None

Appendix

Table 6: Training settings.

H

is the action chunk length. UTD is per environment step; the discount

\gamma

is per environment step except for LIBERO, where it is per chunk.

RoboMimic
Task	BC weight $\alpha$	Quantile $q$
can-mh	$\{\mathbf{0.1},0.3,1,3,10\}$	$\{0.1,0.3,\mathbf{0.5},0.7,0.9\}$
square-mh	$\{\mathbf{3},10,30\}$	$\{0.7,\mathbf{0.9}\}$
transport-mh	$\{0.1,0.3,1,\mathbf{3},10,30\}$	$\{\mathbf{0.1},0.3,0.7,0.9\}$
tool-hang-ph	$\{3,\mathbf{10},30\}$	$\{0.1,0.3,0.5,0.7,\mathbf{0.9}\}$
OGBench

Appendix

Table 7: GSBR Hyperparameter sweep range.

Method	Tasks	Hyperparameter sets
DICE-RL	can-mh	$\alpha\in\{\mathbf{3},\allowbreak 10,\allowbreak 30,\allowbreak 100,\allowbreak 300\}$ ; $u\in\{\mathbf{0.3125},\allowbreak 2\}$
	square-mh	$\alpha\in\{3,\allowbreak\mathbf{10},\allowbreak 30,\allowbreak 100,\allowbreak 300\}$ ; $u\in\{0.3125,\allowbreak\mathbf{2}\}$
	transport-mh	$\alpha\in\{3,\allowbreak 10,\allowbreak 30,\allowbreak 100,\allowbreak\mathbf{300}\}$ ; $u\in\{\mathbf{0.3125},\allowbreak 2\}$
	tool-hang-ph	$\alpha\in\{\mathbf{3},\allowbreak 10,\allowbreak 30,\allowbreak 100,\allowbreak 300\}$ ; $u\in\{0.3125,\allowbreak\mathbf{2}\}$
	cube 1	$\alpha\in\{1,\allowbreak 3,\allowbreak\mathbf{10},\allowbreak 30,\allowbreak 100,\allowbreak 300\}$ ; $u\in\{0.3125,\allowbreak\mathbf{2}\}$
	cube 2	$\alpha\in\{1,\allowbreak 3,\allowbreak 10,\allowbreak\mathbf{30},\allowbreak 100,\allowbreak 300\}$ ; $u\in\{\mathbf{0.3125},\allowbreak 2\}$

Appendix

Table 8: Baseline search grids. Bold values are selected. Here

s

is residual scale,

b

is the DSRL bound/noise scale, and

u

is UTD.

Figure 7: This figure demonstrates how the success rate, and throughput vary across different hyper parameter configs. Our method performs well across multiple magnitudes of BC weights.

Regularized RL

See all 28 papers

Aug 2, 2026

ReBRAC-v2: The Return of the King

Recent offline reinforcement learning methods increasingly rely on expressive generative policies and specialized value guidance. We ask how far systematic modernization of a conventional behavior-regularized actor-critic can go while preserving algorithmic simplicity. ReBRAC-v2 directly trains an exact-likelihood normalizing-flow actor with likelihood, MSE, and MAE behavior regularization, classification-based residual critics, staged optimization, and multi-sample inference. We develop one shared configuration through roughly 600 Bayesian proposals on six OGBench tasks, then adapt only two behavior-regularization coefficients using a 16-point grid for the main comparisons. Across ten common OGBench categories, ReBRAC-v2 averages 74.8 versus 52.3 for the next-best method and ranks first in eight categories. Without structural changes, it also obtains the strongest compared averages on D4RL AntMaze (90.2) and Adroit (33.6). Motivated by these results, we extend evaluation to all 235 state-based single-task problems across the 47 dataset variants in the original OGBench suite. To our knowledge, this is the first complete evaluation in this setting; no compared baseline reports results across the full suite. Per-task scores and configurations establish ReBRAC-v2 as a reference baseline for future work beyond the common task subset. The complete cloning objective retains the highest average after coefficient retuning of its alternatives, while further ablations characterize training, capacity, and inference choices. These results demonstrate the strength of systematic engineering within a minimalist offline RL foundation.

Denis Tarasov, Robert K. Katzschmann

Soft Robotics Lab, D-MAVT ETH Zurich, Switzerland

Reinforcement Learning Offline RL Actor-Critic Methods Policy Learning Regularized RL

Ask PDF

Oct 28, 2026

Variational Consequence-Driven Offline Reinforcement Learning

Offline reinforcement learning (RL) typically mitigates distribution shift by imposing divergence constraints on the action distributions of the learned and behavior policies. However, this paradigm cannot ensure the underlying dynamic consistency structure: even minor action deviations can lead to drastically different transitions, inadvertently driving the agent away from the offline dataset. To address this, we explore policy constraints through their induced state-consequence distributions rather than relying on pure action-distribution matching. Nevertheless, directly enforcing such consequence-aware constraints is computationally intractable without access to the true underlying dynamics model. To bypass this issue, we propose Structural Behavior Regularization (StBR), which replaces the intractable objective with a closed-form latent divergence. Structuring this latent space to reflect transition dynamics renders the latent divergence a theoretically bounded surrogate for the true consequence-distribution divergence. By regularizing the learned policy with this latent divergence, StBR adaptively tightens the behavior constraint in dynamics-sensitive regions, effectively preventing deviations outside the offline dataset. Empirically, we demonstrate that StBR achieves superior performance across diverse offline RL benchmarks.

Wen Jiang, Ke Jiang, Xiaoyang Tan

College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence · Graduate School of Information Science and Technology, The University of Osaka

Offline RL Constrained RL

Ask PDF

May 27, 2026

SPAR: Support-Preserving Action Rectification

Offline policy improvement faces an inherent conflict between maximizing value and fitting the data distribution. While in-sample weighted regression is stable, it suffers from over-conservatism that suppresses high-value actions in the distribution tail; conversely, gradient-based approaches often exhibit a fitting-optimization conflict of gradients, which drives the policy off the data manifold. To address this, we propose Support-Preserving Action Rectification (SPAR), which reframes global learning as a local residual rectification anchored to a frozen pure behavior cloning policy. This framework performs fine-grained fitting and local policy improvement in the residual space, thereby contracting the search space. We further introduce Latent Self-Imitation, utilizing a latent-sampling weighted-regression mechanism to address fitting-improvement gradient conflict in the residual space. Theoretically, we prove this mechanism eliminates the manifold-normal drift of standard value gradients, while extensive D4RL experiments show SPAR extracts significant gains from suboptimal baselines to achieve state-of-the-art performance.

Jiaxin Zhao, Weihang Pan, Xun Liang +1

Zhejiang University.

Imitation Learning Offline RL Residual Policy Learning Policy Learning

Ask PDF

Abstract

Regularized RL

Explore similar work

ReBRAC-v2: The Return of the King

Variational Consequence-Driven Offline Reinforcement Learning

SPAR: Support-Preserving Action Rectification