GSBR: Geometric Support Behavior Regularization for Online RL

Charles Xu; Jacob Berg; Liyiming Ke; Sergey Levine

cs.RONov 3, 2026

GSBR: Geometric Support Behavior Regularization for Online RL

Authors: Charles Xu, Jacob Berg, Liyiming Ke, Sergey Levine

Organizations: UC Berkeley · Physical Intelligence

Abstract

Regularizing online reinforcement learning (RL) toward a pretrained base policy can improve sample efficiency and learning stability, but regularization that is too restrictive limits how much the policy can improve. Base policies -- typically diffusion or flow-matching models trained by imitation learning on large, diverse datasets -- capture many behaviors that are reasonable but suboptimal. Existing regularizers using such base policies are too restrictive in one of two ways: behavioral cloning regularizers pull the actor toward the full base-policy distribution, penalizing it for discarding suboptimal modes, while hard support constraints forbid improvements that lie just outside the base policy's support. We propose Geometric Support Behavior Regularization (GSBR), which penalizes actions by their distance from the base policy's support. Actions within the support incur no penalty, so the actor can concentrate on high-value modes; outside the support, where value estimates are less reliable, the penalty grows smoothly with distance while still permitting improvement when the value gained outweighs the penalty. GSBR estimates this distance from a finite set of base-policy samples, so it requires only sample access to the base policy. Across 19 tasks on four robotic benchmarks, GSBR raises the average success rate from 33% for the base policy to 95%, achieving the highest average success rate and throughput among all compared methods, with about 23% higher normalized throughput than the next-best RL method. Project website and open-source code at https://charlesxu0124.github.io/gsbr/ .

Figures & tables

Figure 1: Left: (a) Behavior cloning regularizer matches the base policy distribution, not allowing the RL actor to drop poor modes. (b) Support constraint allows the actor to select the best behavior in the base policy distribution, but excludes improvements beyond it. (c) GSBR applies distance-based regularization on deviation outside the base policy support, allowing the actor to discard suboptimal behaviors and guide exploration beyond the prior. Right: GSBR achieves 23% higher normalized throughput across 19 tasks and 4 benchmarks than previous methods.

Figure 2: GSBR . To implement the geometric support regularizer on a generative robot policy, draw samples from $\pi_{\beta}(\cdot\mid s)$ and compute the nearest-neighbor distance between them. Set $r_{\mathcal{B}}$ to a quantile over these distances and construct the support region $C$ as the union of balls of radius $r_{\mathcal{B}}$ around each sample. The behavior regularizer loss is computed as the Euclidean distance to the closest point in $C$ .

Figure 3: Benchmarks . We evaluate our method across the four benchmarks above. From left to right, the tasks are RoboMimic tool-hang , OGBench cube-triple-task4 , OCBench chamber-lite-easy-task3 , and LIBERO task29 .

Figure 4: Normalized throughput across benchmarks . GSBR achieves the highest average normalized throughput across all 19 tasks and leads all baselines on RoboMimic , OGBench , and OCBench . Throughput (success rate / mean episode length) is normalized by the best method’s seed-average value per task, then averaged across tasks. Error bars show SEM across seeds.

Method	RoboMimic	OGBench	OCBench	LIBERO	All tasks
	(4 tasks)	(5 tasks)	(5 tasks)	(5 tasks)	(19 tasks)
Base Policy	$44$	$0$	$41$	$49$	$33$
TD3+BC	$96\pm 2$	$73\pm 1$	$98\pm 0$	$54\pm 2$	$79\pm 1$
RLT	$96\pm 2$	$90\pm 5$	$98\pm 0$	$\mathbf{86}\pm 2$	$92\pm 1$
Residual Policy	$94\pm 2$	$68\pm 10$	$95\pm 0$	$78\pm 1$	$83\pm 2$
BFN	$86\pm 2$	$69\pm 5$	$88\pm 2$	$66\pm 1$	$77\pm 1$

Table 1: Average success rate (%) across benchmarks . Uncertainty is SEM across three seed-wise task averages. Values are rounded to the nearest integer; bold marks the highest unrounded mean in each column. Base Policy SEM is omitted because complete per-seed evidence is unavailable.

Reference count $N$	Normalized TP
8 (Ours)	$\textbf{1.00}\pm\textbf{0.05}$
16	$0.91\pm 0.02$
32	$0.93\pm 0.02$

Table 2: Reference-count ablation. On cube-triple , the number of base policy samples,

N

, has little effect on performance.

Figure 5: Visualization of the true and estimated support region on cube-triple . The estimated support region (in bronze) closely matches the true support using only 8 samples. The RL actor learned an action just outside the support region (marked with star).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Absolute throughput (successes/h) over training steps

RoboMimic
Task	BC	TD3+BC	RLT	Residual policy	BFN	DSRL	EXPO-FT	DICE-RL	GSBR (ours)
can-mh	$73{\pm}1$	$\mathbf{99}{\pm}0$	$\mathbf{99}{\pm}0$	$97{\pm}1$	$94{\pm}2$	$\mathbf{99}{\pm}0$	$\mathbf{99}{\pm}0$	$\mathbf{99}{\pm}0$	$98{\pm}1$
square-mh	$37{\pm}1$	$96{\pm}1$	$96{\pm}1$	$98{\pm}0$	$78{\pm}3$	$98{\pm}0$	$98{\pm}0$	$96{\pm}2$	$\mathbf{99}{\pm}0$
transport-mh	$15{\pm}2$	$88{\pm}7$	$90{\pm}6$	$83{\pm}7$	$81{\pm}4$	$92{\pm}2$	$93{\pm}1$	$84{\pm}3$	$\mathbf{96}{\pm}1$
tool-hang-ph	$50{\pm}2$	$97{\pm}0$	$97{\pm}0$	$97{\pm}2$	$90{\pm}2$	$96{\pm}2$	$95{\pm}1$	$\mathbf{98}{\pm}0$	$98{\pm}0$
OGBench cube-triple

Appendix

Table 4: Final success rate (%) across all tasks . Values are mean

\pm

SEM.

RoboMimic
Task	BC	TD3+BC	RLT	Residual policy	BFN	DSRL	EXPO-FT	DICE-RL	GSBR (ours)
can-mh	$122{\pm}1$	$329{\pm}6$	$339{\pm}8$	$389{\pm}21$	$198{\pm}10$	$308{\pm}7$	$317{\pm}5$	$289{\pm}6$	$\mathbf{416}{\pm}16$
square-mh	$40{\pm}1$	$247{\pm}6$	$273{\pm}13$	$325{\pm}6$	$113{\pm}7$	$297{\pm}6$	$273{\pm}3$	$226{\pm}20$	$\mathbf{348}{\pm}8$
transport-mh	$7{\pm}1$	$80{\pm}11$	$86{\pm}11$	$65{\pm}11$	$63{\pm}5$	$104{\pm}5$	$90{\pm}2$	$52{\pm}2$	$\mathbf{105}{\pm}2$
tool-hang-ph	$24{\pm}2$	$133{\pm}3$	$127{\pm}5$	$112{\pm}10$	$80{\pm}7$	$121{\pm}12$	$143{\pm}10$	$128{\pm}6$	$\mathbf{165}{\pm}3$
OGBench cube-triple

Appendix

Table 5: Absolute throughput (successes/hour) across all tasks and methods assuming 10 Hz for RoboMimic, OGBench, and OCBench, and 20 Hz for LIBERO. Values are mean

\pm

SEM.

Benchmark	Tasks	Online Env Steps	UTD	$H$	$\gamma$	TD clip
RoboMimic	can , square	1M	2	5	$0.998$	$[0,1]$
RoboMimic	transport , tool-hang	2M	2	5	$0.998$	$[0,1]$
OGBench	cube-triple-T[1-5]	2M	2	5	$0.99$	None
OCBench Lite	block-double-T[1-2] chamber-easy-T[1-3]	1M	2	3	$0.99$	$[0,1]$
LIBERO	Task-{14, 29, 54, 60, 64}	1,000 episodes	2	10	$0.975$ per chunk	None

Appendix

Table 6: Training settings.

H

is the action chunk length. UTD is per environment step; the discount

\gamma

is per environment step except for LIBERO, where it is per chunk.

RoboMimic
Task	BC weight $\alpha$	Quantile $q$
can-mh	$\{\mathbf{0.1},0.3,1,3,10\}$	$\{0.1,0.3,\mathbf{0.5},0.7,0.9\}$
square-mh	$\{\mathbf{3},10,30\}$	$\{0.7,\mathbf{0.9}\}$
transport-mh	$\{0.1,0.3,1,\mathbf{3},10,30\}$	$\{\mathbf{0.1},0.3,0.7,0.9\}$
tool-hang-ph	$\{3,\mathbf{10},30\}$	$\{0.1,0.3,0.5,0.7,\mathbf{0.9}\}$
OGBench

Appendix

Table 7: GSBR Hyperparameter sweep range.

Method	Tasks	Hyperparameter sets
DICE-RL	can-mh	$\alpha\in\{\mathbf{3},\allowbreak 10,\allowbreak 30,\allowbreak 100,\allowbreak 300\}$ ; $u\in\{\mathbf{0.3125},\allowbreak 2\}$
	square-mh	$\alpha\in\{3,\allowbreak\mathbf{10},\allowbreak 30,\allowbreak 100,\allowbreak 300\}$ ; $u\in\{0.3125,\allowbreak\mathbf{2}\}$
	transport-mh	$\alpha\in\{3,\allowbreak 10,\allowbreak 30,\allowbreak 100,\allowbreak\mathbf{300}\}$ ; $u\in\{\mathbf{0.3125},\allowbreak 2\}$
	tool-hang-ph	$\alpha\in\{\mathbf{3},\allowbreak 10,\allowbreak 30,\allowbreak 100,\allowbreak 300\}$ ; $u\in\{0.3125,\allowbreak\mathbf{2}\}$
	cube 1	$\alpha\in\{1,\allowbreak 3,\allowbreak\mathbf{10},\allowbreak 30,\allowbreak 100,\allowbreak 300\}$ ; $u\in\{0.3125,\allowbreak\mathbf{2}\}$
	cube 2	$\alpha\in\{1,\allowbreak 3,\allowbreak 10,\allowbreak\mathbf{30},\allowbreak 100,\allowbreak 300\}$ ; $u\in\{\mathbf{0.3125},\allowbreak 2\}$

Appendix

Table 8: Baseline search grids. Bold values are selected. Here

s

is residual scale,

b

is the DSRL bound/noise scale, and

u

is UTD.

Figure 7: This figure demonstrates how the success rate, and throughput vary across different hyper parameter configs. Our method performs well across multiple magnitudes of BC weights.

Regularized RL

See all 28 papers

ReBRAC-v2: The Return of the King
Aug 2, 2026Denis Tarasov, Robert K. KatzschmannReinforcement Learning Offline RL
Soft Robotics Lab, D-MAVT ETH Zurich, Switzerland
Variational Consequence-Driven Offline Reinforcement Learning
Oct 28, 2026Wen Jiang, Ke Jiang, Xiaoyang TanOffline RL Constrained RL
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence · Graduate School of Information Science and Technology, The University of Osaka
SPAR: Support-Preserving Action Rectification
May 27, 2026Jiaxin Zhao, Weihang Pan, Xun Liang +1Imitation Learning Offline RL
Zhejiang University.

Aug 2, 2026J/K move · Enter open · S save

ReBRAC-v2: The Return of the King

Denis Tarasov, Robert K. Katzschmann

Soft Robotics Lab, D-MAVT ETH Zurich, Switzerland

Reinforcement Learning Offline RL Actor-Critic Methods Policy Learning Regularized RL

Open paper AskPDF arXiv

Recent offline reinforcement learning methods increasingly rely on expressive generative policies and specialized value guidance. We ask how far systematic modernization of a conventional behavior-regularized actor-critic can go while preserving algorithmic simplicity. ReBRAC-v2 directly trains an exact-likelihood normalizing-flow actor with likelihood, MSE, and MAE behavior regularization, classification-based residual critics, staged optimization, and multi-sample inference. We develop one shared configuration through roughly 600 Bayesian proposals on six OGBench tasks, then adapt only two behavior-regularization coefficients using a 16-point grid for the main comparisons. Across ten common OGBench categories, ReBRAC-v2 averages 74.8 versus 52.3 for the next-best method and ranks first in eight categories. Without structural changes, it also obtains the strongest compared averages on D4RL AntMaze (90.2) and Adroit (33.6). Motivated by these results, we extend evaluation to all 235 state-based single-task problems across the 47 dataset variants in the original OGBench suite. To our knowledge, this is the first complete evaluation in this setting; no compared baseline reports results across the full suite. Per-task scores and configurations establish ReBRAC-v2 as a reference baseline for future work beyond the common task subset. The complete cloning objective retains the highest average after coefficient retuning of its alternatives, while further ablations characterize training, capacity, and inference choices. These results demonstrate the strength of systematic engineering within a minimalist offline RL foundation.

Abstract

Regularized RL

Explore similar work

ReBRAC-v2: The Return of the King

Variational Consequence-Driven Offline Reinforcement Learning

SPAR: Support-Preserving Action Rectification