Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting

Zhongjian Qiao; Jiafei Lyu; Boxiang Lyu; Yao Shu; Siyang Gao; Shuang Qiu

cs.LGMar 9, 2026

Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting

Authors: Zhongjian Qiao, Jiafei Lyu, Boxiang Lyu, Yao Shu, Siyang Gao, Shuang Qiu

Organizations: CityUHK · Tencent · UChicago · HKUST(GZ)

Abstract

Model-based offline reinforcement learning (RL) aims to enhance offline RL with a dynamics model that facilitates policy exploration. However, model exploitation could occur due to inevitable model errors, degrading algorithm performance. Adversarial model learning offers a theoretical framework to mitigate model exploitation by solving a maximin formulation. Within such a paradigm, RAMBO~\citep{rigter2022rambo} has emerged as a representative and most popular method that provides a practical implementation with model gradient. However, we empirically reveal that severe Q-value underestimation and gradient explosion can occur in RAMBO with only slight hyperparameter tuning, suggesting that it tends to be overly conservative and suffers from unstable model updates. To address these issues, we propose RObust value-aware Model learning with Implicitly differentiable adaptive weighting (ROMI). Instead of updating the dynamics model with model gradient, ROMI introduces a novel robust value-aware model learning approach. This approach requires the dynamics model to predict future states with values close to the minimum Q-value within a scale-adjustable state uncertainty set, enabling controllable conservatism and stable model updates. To further improve out-of-distribution (OOD) generalization during multi-step rollouts, we propose implicitly differentiable adaptive weighting, a bi-level optimization scheme that adaptively achieves dynamics- and value-aware model learning. Empirical results on D4RL and NeoRL datasets show that ROMI significantly outperforms RAMBO and achieves competitive or superior performance compared to other state-of-the-art methods on datasets where RAMBO typically underperforms. Code is available at https://github.com/zq2r/ROMI.git.

Figures & tables

Figure 1: Left: Q-value estimation for different values of

\lambda

. The slash-shaded bars indicate that the Q-value has diverged. Middle: Gradient norm of the adversarial loss recorded during training. Right: Q-value estimation recorded during training.

Task Name	CQL	IQL	MOPO	RAMBO	COUNT	MOBILE 1 1 1 3M steps are used in MOBILE paper, and we only report the results at 1M steps for a fair comparison.	ROMI (Ours)
hopper-r	7.9	7.6	27.8	23.9	30.0	28.5	31.7 $\pm$ 0.3
halfcheetah-r	17.5	13.1	36.7	29.5	37.3	39.4	38.4 $\pm$ 4.1
walker2d-r	5.1	5.4	0.0	0.0	19.9	23.6	22.4 $\pm$ 2.6
hopper-m	53.0	66.2	72.2	95.2	100.2	77.6	105.0 $\pm$ 3.3
halfcheetah-m	47.0	47.9	64.0	76.2	75.9	74.6	76.2 $\pm$ 0.6
walker2d-m	73.4	67.2	82.1	83.9	88.4	90.7	93.6 $\pm$ 1.3

Table 1: Normalized score comparison between different offline RL algorithms and ROMI on 12 D4RL MuJoCo datasets. We abbreviate ”random” as ”r”, ”medium” as ”m”, ”medium-replay” as ”mr”, ”medium-expert” as ”me”. Each method is run with 1M steps.

Task Name	CQL	IQL	EDAC	MOPO	RAMBO	MOBILE	ROMI (Ours)
Hopper-L	16.8	14.3	12.2	12.0	15.1	18.1	22.4 $\pm$ 0.5
HalfCheetah-L	35.0	34.1	31.3	34.2	30.1	38.9	35.8 $\pm 3.9$
Walker2d-L	42.4	40.1	43.1	15.6	24.3	39.0	36.4 $\pm 2.3$
Hopper-M	58.1	56.8	37.2	0.0	37.5	47.3	46.6 $\pm$ 6.8
HalfCheetah-M	50.0	52.0	54.6	53.1	51.2	55.6	57.7 $\pm$ 1.4
Walker2d-M	52.5	51.2	53.9	35.7	36.2	51.4	54.9 $\pm$ 2.0

Table 2: Normalized score comparison between different offline RL algorithms and our method ROMI on NeoRL tasks. Each method is run with 1M steps over 5 seeds.

Figure 2: Comparison of the prediction error and performance w/ and w/o adaptive weighting.

Figure 3: Comparison for different

\xi

in terms of Q-value, normalized score, inner loss gradient norm, outer loss gradient norm.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: D4RL MuJoCo tasks. Left: halfcheetah, Middle: hopper, Right: walker2d.

Panel of Figure 5 — Figure 5: D4RL Antmaze tasks. Left: umaze, Middle: medium, Right: large.

	Hyperparameter	Value
Dynamics Model	Ensemble size	7
	Architecture	(input, 200, 200, 200, 200, output)
	Real data ratio $(f)$	0.5
	Model learning batch size	256
	Model learning rate	$3\times 10^{-4}$
	Model pretraining epochs	50

Appendix

Table 3: Base hyperparameter setup for ROMI shared across all runs.

Domain Name	Task Name	$\xi$	$N$	$H$
	halfcheetah-random	0.01	10	5
	hopper-random	0.1	10	5
	walker2d-random	0.01	10	5
	halfcheetah-medium	0.1	10	2
	hopper-medium	1.0	10	2
D4RL MuJoCo	walker2d-medium	0.01	10	5

Appendix

Table 4: Hyperparameters of ROMI.

Task Name	Size of Parameters (MB)		GPU Memory (GB)		Runtime (s/epoch)
Task Name	ROMI	RAMBO	ROMI	RAMBO	ROMI	RAMBO
hopper-random	7.93	7.66	3.30	3.28	28.20	23.76
halfcheetah-random	8.19	7.91	3.33	3.32	29.41	22.24
walker2d-random	8.19	7.91	3.33	3.32	30.06	23.89
hopper-medium	7.93	7.66	3.30	3.28	28.43	23.31
halfcheetah-medium	8.19	7.91	3.33	3.32	29.91	22.63

Appendix

Table 5: Comparison of ROMI and RAMBO on computing resource consumption

Task Name	MOPO	COMBO	RAMBO	MOBILE	ROMI (Ours)
antmaze-umaze	0.0	80.3	42.6 $\pm$ 11.9	77.0	70.3 $\pm$ 11.4
antmaze-umaze-diverse	0.0	57.3	12.9 $\pm$ 5.6	20.4	32.8 $\pm$ 13.0
antmaze-medium-play	0.0	0.0	18.7 $\pm$ 9.2	64.6	51.3 $\pm$ 19.7
antmaze-medium-diverse	0.0	0.0	20.4 $\pm$ 3.7	1.6	30.1 $\pm$ 10.0
antmaze-large-play	0.0	0.0	0.0 $\pm$ 0.0	2.6	0.0 $\pm$ 0.0
antmaze-large-diverse	0.0	0.0	0.0 $\pm$ 0.0	7.2	6.4 $\pm$ 4.2

Appendix

Table 6: Successful Success rate comparison on Antmaze tasks. Results are averaged over 5 seeds. The top two scores for each task are bold .

Figure 6: Gradient norm curves on 9 D4RL datasets for RAMBO when

\lambda=0.05

and

\lambda=0.1

Figure 7: Normalized score comparison on 9 D4RL datasets across different

\lambda

for RAMBO.

Figure 8: Q-value estimation on 9 D4RL datasets across different

\lambda

for RAMBO.

Figure 9: Comparison for different

\xi

in terms of Q-value, normalized score, inner loss gradient norm, outer loss gradient norm.

Datasets	$N=10$	$N=20$	$N=30$
half-mr	70.7 $\pm$ 4.8	71.8 $\pm$ 4.2	72.2 $\pm$ 5.1
hopper-mr	102 $\pm$ 2.4	100.0 $\pm$ 1.5	103.3 $\pm$ 2.8
walker2d-mr	85.2 $\pm$ 2.6	87.4 $\pm$ 4.6	81.5 $\pm$ 3.8
half-me	108.5 $\pm$ 4.2	110.4 $\pm$ 5.3	111.0 $\pm$ 3.4
hopper-me	111.5 $\pm$ 3.8	110.0 $\pm$ 1.4	113.9 $\pm$ 4.7
walker2d-me	113.3 $\pm$ 1.9	115.6 $\pm$ 2.8	111.7 $\pm$ 0.5

Appendix

Table 7: Performance comparison for different

N

on 6 D4RL datasets.

Figure 10: Learning curves of RAMBO and ROMI on 12 D4RL datasets. Averaged over 5 seeds.

Figure 11: Learning curves of MOBILE on 12 D4RL datasets. Averaged over 5 seeds.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 297 papers

Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
Jul 10, 2026Wenke Xia, Pei Ren, Wenbo Yu +10Value Function Estimation Robotic Manipulation
1Gaoling School of Artificial Intelligence, Renmin University of China · 2Beijing Innovation Center of Humanoid Robotics · 3Beijing Forestry University +4
Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective
Feb 17, 2025Yu-Wei Yang, Wei Hung, Yun-Ming Chan +2Model Selection Offline RL
Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Applied Machine Learning, Meta AI, Menlo Park, CA, USA
Offline Policy Optimization with Posterior Sampling
May 8, 2026Hongqiang Lin, Dongxu Zhang, Yiding Sun +3OOD Generalization Posterior Sampling
Zhejiang University · Xi’an Jiaotong University · Institute of Automation, Chinese Academy of Sciences +1

Jul 10, 2026J/K move · Enter open · S save

Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning

Wenke Xia, Pei Ren, Wenbo Yu, Yizhuo Zhang+9

1Gaoling School of Artificial Intelligence, Renmin University of China · 2Beijing Innovation Center of Humanoid Robotics · 3Beijing Forestry University+4

Value Function Estimation Robotic Manipulation Offline-To-Online RL

Open paper AskPDF arXiv

Offline-to-online reinforcement learning is promising for generalizable robotic manipulation, yet its full-stack complexity obscures reproduction and diagnosis. Within such systems, value estimation plays a central role in prioritizing heterogeneous data for policy improvement. Despite its importance, the central question remains underexplored: how value-function reliability shapes policy optimization in offline-to-online reinforcement learning. To answer this question, we propose Robo-ValueRL, a unified framework that enables reliable value estimation and systematically traces its downstream effects on policy pretraining and online improvement. Concretely, Robo-ValueRL learns a history-conditioned value estimator and evaluates its reliability through global-progress and local-preference metrics. These resulting value estimates are propagated into quality-conditioned consistency-policy pretraining and a residual adaptation module on online rollouts, providing a unified testbed for analyzing how value reliability shapes downstream policy performance. Across 240 hours of offline demonstrations and over 3,000 online rollout trajectories, our extensive experiments show that downstream performance is strongly associated with value reliability. Reliable value functions provide better action-quality estimates, allowing value-guided offline RL to scale more effectively than quality-agnostic behavior cloning, and stabilize online improvement by prioritizing high-quality rollout data. Integrating reliable value guidance through offline pretraining with online improvement, our system achieves 86% success on millimeter-level precise chip insertion and 84% on generalizable block disassembly. We hope these findings highlight the importance of value-guided data utilization for effective policy improvement from heterogeneous robotic experience.

Abstract

Offline RL

Explore similar work

Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning

Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective

Offline Policy Optimization with Posterior Sampling