Conservative Offline Robot Policy Learning via Posterior-Transition Reweighting

Wanpeng Zhang; Hao Luo; Sipeng Zheng; Yicheng Feng; Haiweng Xu; Ziheng Xi; Chaoyi Xu; Haoqi Yuan

cs.ROMar 17, 2026

Conservative Offline Robot Policy Learning via Posterior-Transition Reweighting

Authors: Wanpeng Zhang, Hao Luo, Sipeng Zheng, Yicheng Feng, Haiweng Xu, Ziheng Xi, Chaoyi Xu, Haoqi Yuan,

Organizations: Peking University · BeingBeyond · Tsinghua University

Abstract

Offline post-training adapts a pretrained robot policy to a target dataset by supervised regression on recorded actions. In practice, robot datasets are heterogeneous: they mix embodiments, camera setups, and demonstrations of varying quality, so many trajectories reflect recovery behavior, inconsistent operator skill, or weakly informative supervision. Uniform post-training gives equal credit to all samples and can therefore average over conflicting or low-attribution data. We propose Posterior-Transition Reweighting (PTR), a reward-free and conservative post-training method that decides how much each training sample should influence the supervised update. For each sample, PTR encodes the observed post-action consequence as a latent target, inserts it into a candidate pool of mismatched targets, and uses a separate transition scorer to estimate a softmax identification posterior over target indices. The posterior-to-uniform ratio defines the PTR score, which is converted into a clipped-and-mixed weight and applied to the original action objective through self-normalized weighted regression. This construction requires no tractable policy likelihood and is compatible with both diffusion and flow-matching action heads. Rather than uniformly trusting all recorded supervision, PTR reallocates credit according to how attributable each sample's post-action consequence is under the current representation, improving conservative offline adaptation to heterogeneous robot data.

Figures & tables

Figure 1: Overview of PTR. Left: the standard policy stack (backbone + action expert) is augmented with a lightweight scorer and a BeliefTokenizer. Right: for each training chunk, the scorer identifies the matched post-action target among mismatched candidates; the resulting identification posterior is converted into a conservative weight that rescales the supervised action loss. No reward labels or policy likelihoods are needed.

	LIBERO					RoboCasa
Method	Spatial	Object	Goal	Long	Avg	Pick&Pl.	Door/Dr.	Others	Avg
SFT	98.8	99.0	98.6	96.8	98.3	36.0	71.3	55.3	54.2
SFT+Belief	98.2	98.4	98.0	95.6	97.6	36.7	71.5	55.0	54.4
PTR	98.0	99.2	97.6	97.0	97.8	38.3	73.0	55.5	55.6

Table 1: Standard simulation results (success rate %) on LIBERO and RoboCasa. LIBERO reports per-suite averages over

500

episodes each; RoboCasa reports category averages over

1200

total trials. Bold : best; Underlined : second best.

Figure 2: Robustness under corrupted training data (success rate %). Colored badges show PTR

-

SFT deltas. PTR maintains higher performance across all corruption types.

Figure 3: Real-robot platforms used for evaluation. (a) Unitree G1 with LinkerHand O6 dexterous hands. (b) PND Adam-U with bimanual dexterous manipulation and a movable head. (c) FR3 single-arm with Inspire dexterous hand.

Platform	DoF	DoF breakdown	Morphology	Camera	View	Hz
Unitree G1 + LinkerHand O6	$26$	$7{\times}2$ arms $+$ $6{\times}2$ hands	Bimanual, dexterous	D435	Fixed ego	$20$
PND Adam-U	$31$	$3$ waist $+$ $2$ head $+$ $7{\times}2$ arms $+$ $6{\times}2$ hands	Bimanual + head + waist	ZED Mini	Movable ego (dual)	$20$
FR3 + Inspire Hand	$13$	$7$ arm $+$ $6$ hand	Single-arm, dexterous	$2{\times}$ D435	Fixed 3rd-person	$50$

Table 2: Robot platform specifications for the three real-robot embodiments.

Figure 4: Representative real-robot tasks across three platforms and four capability suites.

	Simulation		Real Robot
Method	LIBERO	RoboCasa	Bimanual	Long-Hor.	Spatial	Robust	Overall
SFT-Specialist	98.3	54.2	55.0	63.3	75.0	50.0	71.1
PTR-Specialist	97.8	55.6	66.7	61.7	78.3	61.7	73.5
SFT-Generalist	96.2	50.8	45.0	51.7	63.3	40.0	65.7
PTR-Generalist	97.4	55.0	60.0	65.0	73.3	56.7	72.1

Table 3: Specialist and generalist training results (success rate %). Generalist trains a single checkpoint on all five sources (three real-robot sources plus LIBERO and RoboCasa); Specialist trains one checkpoint per source. “Overall” is the mean of the LIBERO average, the RoboCasa average, and the real-robot overall average. Bold : best; Underlined : second best per column.

Figure 5: Cross-embodiment task correspondence. Different robot platforms (Adam-U & FR3) execute semantically similar manipulation tasks, illustrating the shared post-action structure that enables PTR to selectively transfer useful knowledge.

Suite / Task	Platform	SFT-Spec	PTR-Spec	SFT-Gen	PTR-Gen
Bimanual
Pour Water	Adam-U	$60.0$	$70.0$	$50.0$	$65.0$
Handover	Adam-U	$50.0$	$65.0$	$40.0$	$55.0$
Put & Close Box	G1	$55.0$	$65.0$	$45.0$	$60.0$
Suite average		$55.0$	$66.7$	$45.0$	$60.0$
Long-Horizon

Table 4: Per-task real-robot success rates (%).

Figure 6: PTR rollouts on all $12$ real-robot tasks.

Figure 7: Illustrative hard out-of-distribution (OOD) evaluation with PTR. The scene contains unseen objects, a novel background, flickering overhead lighting, and a transparent bottle that challenges depth perception.

	LIBERO					RoboCasa
Configuration	Spatial	Object	Goal	Long	Avg	Pick&Pl.	Door/Dr.	Others	Avg
PTR (full)	98.0	98.8	97.4	97.0	97.8	38.3	73.0	55.5	55.6
w/o belief tokens	97.6	98.4	97.6	95.8	97.4	36.8	71.0	54.2	54.0
w/o cross-rank gather	97.2	98.2	97.0	95.4	97.0	35.7	71.7	55.5	54.3
w/o EMA (frozen enc.)	95.8	96.6	95.2	93.4	95.3	33.0	67.3	50.5	50.2
w/o refiner	97.8	97.8	96.4	96.2	97.1	34.7	69.3	52.0	51.9

Table 5: Ablation study on standard benchmarks (success %). Bold : best; Underlined : second best.

Figure 8: PTR training dynamics across six embodiment settings (

60

k steps). (a) Identification accuracy shows two convergence groups; the legend applies to all three subplots. (b) PTR score

T_{t}

stabilizes at embodiment-dependent levels. (c) Action loss

\ell_{\mathrm{act}}

decreases steadily.

Figure 9: Belief token and identification margin dynamics. (a) Belief token entropy

H_{\mathrm{tok}}

converges to near-zero across all settings. (b) Identification margin grows steadily, with cross-embodiment training achieving the highest discriminability.

Figure 10: Hyperparameter sensitivity on cross-embodiment LIBERO. Each subplot sweeps one parameter with others fixed at defaults, tracking three normalized diagnostics: success rate, final PTR score

T_{t}

, and weight stability

1/\mathrm{Std}(w)

. Dashed lines mark the default values. Legend is shared across all three subplots.

Figure 11: PTR weight evolution across benchmarks. (a)–(c) Sample weight distribution at early stage (

3

k steps, grey) vs. late stage (

60

k steps, colored) for LIBERO, RoboCasa, and real-robot generalist training. (d) Relative loss reduction

(\ell_{\mathrm{SFT}}-\ell_{\mathrm{PTR}})/\ell_{\mathrm{SFT}}

over training.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Parameter	Symbol	Value
Optimization
Optimizer	–	AdamW
Learning rate	$\eta$	$10^{-4}$
Weight decay	–	$0.01$
Schedule	–	Cosine with $2000$ -step linear warmup
Total steps	–	$60$ k

Appendix

Table 6: Complete hyperparameter listing for PTR post-training.

Instance Reweighting

See all 15 papers

When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning
Apr 23, 2026Elias Hossain, Mohammad Jahid Ibna Basher, Ivan Garibay +2Offline RL RL Post-Training
College of Engineering and Computer Science University of Central Florida, Orlando, FL, USA
Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning
May 19, 2026Dongjie Yu, Kun Lei, Zhennan Jiang +2Robot Policy Adaptation Reinforcement Learning
School of Computing and Data Science, The University of Hong Kong, Hong Kong SAR. · Shanghai Qizhi Institute, Shanghai, China. · Shanghai Jiao Tong University, Shanghai, China. +2
Proxy Policy Steering
Sep 8, 2026Chuanruo Ning, Tianrui Wang, Wei-Chiu Ma +1Robot Policy Adaptation Robot Foundation Models
Cornell University

Apr 23, 2026J/K move · Enter open · S save

When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning

Elias Hossain, Mohammad Jahid Ibna Basher, Ivan Garibay, Ozlem Garibay+1

College of Engineering and Computer Science University of Central Florida, Orlando, FL, USA

Offline RL RL Post-Training

Open paper AskPDF arXiv

Offline reinforcement learning (RL) can learn effective policies from fixed datasets, but deployment objectives may change after training, and in many applications the trained actor cannot be retrained because of data, cost, or governance constraints. We study deployment-time adaptation for frozen offline actors using Product-of-Experts (PoE) composition with a goal-conditioned prior. Our main practical finding is graceful degradation rather than universal performance gain: under degraded or random priors, precision-weighted composition remains anchored to the frozen actor, while additive and prior-only adaptation collapse, and a KL-budget selector often recovers a near-oracle operating point. We also make explicit a closed-form identity in the frozen-actor setting: for diagonal-Gaussian actors and priors, PoE with coefficient alpha yields the same deterministic policy as KL-regularized adaptation with beta = alpha / (1 - alpha), with posterior covariances differing only by a global scalar factor. Empirically, across four D4RL environments (3,900 MuJoCo episodes), we observe a 4/5/3 HELP/FROZEN/HURT split. Extending the analysis to six harder cells and two AntMaze diagnostics reveals an actor-competence ceiling: medium-expert remains HURT in all 9 cells at every tested alpha, while AntMaze with a behavior-cloned frozen actor yields zero success for all composition rules. Overall, PoE and KL-regularized adaptation are best viewed as a single actor-anchored safety mechanism for deployment-time steering.

Abstract

Instance Reweighting

Explore similar work

When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning

Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning

Proxy Policy Steering