CoDrift: Compositional Drifting for Offline Reinforcement Learning

Xiewei Ni; Ruofeng Mei; Xiangyu Xu

cs.LGAug 25, 2026

CoDrift: Compositional Drifting for Offline Reinforcement Learning

Authors: Xiewei Ni, Ruofeng Mei, Xiangyu Xu

Organizations: Xi’an Jiaotong University

Abstract

Offline reinforcement learning is intrinsically multi-objective: a policy must remain compatible with the behavioral support of a fixed dataset while preferentially selecting high-value actions. We recast these objectives in a common form by viewing each as an action-space motion field that specifies how generated actions should move. This perspective enables heterogeneous learning objectives to be combined directly through field composition. Inspired by drifting models, we propose CoDrift, a compositional framework for one-step generative policy learning. CoDrift combines three objective-level fields into a unified policy field. The conditional field preserves state-dependent behavioral structure, while the marginal field pools actions across states to provide a more stable generative signal in the single-positive-sample regime of continuous-control offline RL. The value field moves generated actions toward higher-value regions. The composed field is absorbed into a stochastic generator that produces an action with a single forward pass at deployment. We evaluate CoDrift on 73 tasks from OGBench and D4RL in both offline and offline-to-online settings. CoDrift compares favorably with state-of-the-art methods and achieves the best average rank in both settings.

Figures & tables

Figure 1: Overview of CoDrift. Conditional and marginal behavioral fields keep generated actions within the support of the offline data at two statistical scales, while the value field moves them toward higher-return regions. The three are composed additively into a single policy field $V_{\mathrm{CoDrift}}$ .

	Gaussian Policies			Diffusion Policies			Flow Policies					Drift Policy
Task Category	BC	IQL	ReBRAC	IDQL	SRPO	CAC	FAWAC	FBRAC	IFQL	FQL	DeFlow	CoDrift
OGBench antmaze-large	$11{\scriptscriptstyle\,\pm 1}$	$53{\scriptscriptstyle\,\pm 3}$	$\mathbf{81}{\scriptscriptstyle\,\pm 5}$	$21{\scriptscriptstyle\,\pm 5}$	$11{\scriptscriptstyle\,\pm 4}$	$33{\scriptscriptstyle\,\pm 4}$	$6{\scriptscriptstyle\,\pm 1}$	$60{\scriptscriptstyle\,\pm 6}$	$28{\scriptscriptstyle\,\pm 5}$	$79{\scriptscriptstyle\,\pm 3}$	$\mathbf{81}{\scriptscriptstyle\,\pm 3}$	$\mathbf{84}{\scriptscriptstyle\,\pm 3}$
OGBench antmaze-giant	$0{\scriptscriptstyle\,\pm 0}$	$4{\scriptscriptstyle\,\pm 1}$	$26{\scriptscriptstyle\,\pm 8}$	$0{\scriptscriptstyle\,\pm 0}$	$0{\scriptscriptstyle\,\pm 0}$	$0{\scriptscriptstyle\,\pm 0}$	$0{\scriptscriptstyle\,\pm 0}$	$4{\scriptscriptstyle\,\pm 4}$	$3{\scriptscriptstyle\,\pm 2}$	$9{\scriptscriptstyle\,\pm 6}$	$12{\scriptscriptstyle\,\pm 5}$	$\mathbf{46}{\scriptscriptstyle\,\pm 22}$
OGBench humanoidmaze-medium	$2{\scriptscriptstyle\,\pm 1}$	$33{\scriptscriptstyle\,\pm 2}$	$22{\scriptscriptstyle\,\pm 8}$	$1{\scriptscriptstyle\,\pm 0}$	$1{\scriptscriptstyle\,\pm 1}$	$53{\scriptscriptstyle\,\pm 8}$	$19{\scriptscriptstyle\,\pm 1}$	$38{\scriptscriptstyle\,\pm 5}$	$60{\scriptscriptstyle\,\pm 14}$	$58{\scriptscriptstyle\,\pm 5}$	$48{\scriptscriptstyle\,\pm 4}$	$\mathbf{83}{\scriptscriptstyle\,\pm 5}$
OGBench humanoidmaze-large	$1{\scriptscriptstyle\,\pm 0}$	$2{\scriptscriptstyle\,\pm 1}$	$2{\scriptscriptstyle\,\pm 1}$	$1{\scriptscriptstyle\,\pm 0}$	$0{\scriptscriptstyle\,\pm 0}$	$0{\scriptscriptstyle\,\pm 0}$	$0{\scriptscriptstyle\,\pm 0}$	$2{\scriptscriptstyle\,\pm 0}$	$11{\scriptscriptstyle\,\pm 2}$	$4{\scriptscriptstyle\,\pm 2}$	$5{\scriptscriptstyle\,\pm 2}$	$\mathbf{15}{\scriptscriptstyle\,\pm 5}$
OGBench antsoccer-arena	$1{\scriptscriptstyle\,\pm 0}$	$8{\scriptscriptstyle\,\pm 2}$	$0{\scriptscriptstyle\,\pm 0}$	$12{\scriptscriptstyle\,\pm 4}$	$1{\scriptscriptstyle\,\pm 0}$	$2{\scriptscriptstyle\,\pm 4}$	$12{\scriptscriptstyle\,\pm 0}$	$16{\scriptscriptstyle\,\pm 1}$	$33{\scriptscriptstyle\,\pm 6}$	$60{\scriptscriptstyle\,\pm 2}$	$\mathbf{67}{\scriptscriptstyle\,\pm 3}$	$\mathbf{66}{\scriptscriptstyle\,\pm 5}$

Table 1: Offline RL results. CoDrift is evaluated against representative Gaussian-, diffusion-, and flow-based offline RL methods across OGBench and D4RL. Results for CoDrift are averaged over 8 seeds (4 for pixel-based tasks), and baseline values are taken from prior work ( Tarasov et al., 2023 ; Hansen-Estruch et al., 2023 ; Chen et al., 2024 ; Park et al., 2025b ; Mu, 2026 ) . The pixel-based row is excluded from ranking as several baselines are unavailable. Lower average rank is better.

Task	IQL	ReBRAC	Cal-QL	RLPD	IFQL	FQL	DeFlow	CoDrift
humanoidmaze-medium-navigate-singletask-v0	$21{\scriptscriptstyle\,\pm 13}\,\rightarrow\,16{\scriptscriptstyle\,\pm 8}$	$16{\scriptscriptstyle\,\pm 20}\,\rightarrow\,1{\scriptscriptstyle\,\pm 1}$	$0{\scriptscriptstyle\,\pm 0}\,\rightarrow\,0{\scriptscriptstyle\,\pm 0}$	$0{\scriptscriptstyle\,\pm 0}\,\rightarrow\,8{\scriptscriptstyle\,\pm 10}$	$56{\scriptscriptstyle\,\pm 35}\,\rightarrow\,82{\scriptscriptstyle\,\pm 20}$	$12{\scriptscriptstyle\,\pm 7}\,\rightarrow\,22{\scriptscriptstyle\,\pm 12}$	$13{\scriptscriptstyle\,\pm 5}\,\rightarrow\,65{\scriptscriptstyle\,\pm 12}$	$94{\scriptscriptstyle\,\pm 5}\,\rightarrow\,\mathbf{99}{\scriptscriptstyle\,\pm 1}$
antsoccer-arena-navigate-singletask-v0	$2{\scriptscriptstyle\,\pm 1}\,\rightarrow\,0{\scriptscriptstyle\,\pm 0}$	$0{\scriptscriptstyle\,\pm 0}\,\rightarrow\,0{\scriptscriptstyle\,\pm 0}$	$0{\scriptscriptstyle\,\pm 0}\,\rightarrow\,0{\scriptscriptstyle\,\pm 0}$	$0{\scriptscriptstyle\,\pm 0}\,\rightarrow\,0{\scriptscriptstyle\,\pm 0}$	$26{\scriptscriptstyle\,\pm 15}\,\rightarrow\,39{\scriptscriptstyle\,\pm 10}$	$28{\scriptscriptstyle\,\pm 8}\,\rightarrow\,\mathbf{86}{\scriptscriptstyle\,\pm 5}$	$44{\scriptscriptstyle\,\pm 8}\,\rightarrow\,\mathbf{86}{\scriptscriptstyle\,\pm 3}$	$52{\scriptscriptstyle\,\pm 7}\,\rightarrow\,\mathbf{86}{\scriptscriptstyle\,\pm 5}$
cube-double-play-singletask-v0	$0{\scriptscriptstyle\,\pm 1}\,\rightarrow\,0{\scriptscriptstyle\,\pm 0}$	$6{\scriptscriptstyle\,\pm 5}\,\rightarrow\,28{\scriptscriptstyle\,\pm 28}$	$0{\scriptscriptstyle\,\pm 0}\,\rightarrow\,0{\scriptscriptstyle\,\pm 0}$	$0{\scriptscriptstyle\,\pm 0}\,\rightarrow\,0{\scriptscriptstyle\,\pm 0}$	$12{\scriptscriptstyle\,\pm 9}\,\rightarrow\,40{\scriptscriptstyle\,\pm 5}$	$40{\scriptscriptstyle\,\pm 11}\,\rightarrow\,92{\scriptscriptstyle\,\pm 3}$	$48{\scriptscriptstyle\,\pm 13}\,\rightarrow\,93{\scriptscriptstyle\,\pm 4}$	$65{\scriptscriptstyle\,\pm 10}\,\rightarrow\,\mathbf{100}{\scriptscriptstyle\,\pm 0}$
scene-play-singletask-v0	$14{\scriptscriptstyle\,\pm 11}\,\rightarrow\,10{\scriptscriptstyle\,\pm 9}$	$55{\scriptscriptstyle\,\pm 10}\,\rightarrow\,\mathbf{100}{\scriptscriptstyle\,\pm 0}$	$1{\scriptscriptstyle\,\pm 2}\,\rightarrow\,50{\scriptscriptstyle\,\pm 53}$	$0{\scriptscriptstyle\,\pm 0}\,\rightarrow\,\mathbf{100}{\scriptscriptstyle\,\pm 0}$	$0{\scriptscriptstyle\,\pm 1}\,\rightarrow\,60{\scriptscriptstyle\,\pm 39}$	$82{\scriptscriptstyle\,\pm 11}\,\rightarrow\,\mathbf{100}{\scriptscriptstyle\,\pm 1}$	$58{\scriptscriptstyle\,\pm 14}\,\rightarrow\,\mathbf{100}{\scriptscriptstyle\,\pm 1}$	$92{\scriptscriptstyle\,\pm 6}\,\rightarrow\,\mathbf{100}{\scriptscriptstyle\,\pm 0}$
puzzle-4x4-play-singletask-v0	$5{\scriptscriptstyle\,\pm 2}\,\rightarrow\,1{\scriptscriptstyle\,\pm 1}$	$8{\scriptscriptstyle\,\pm 4}\,\rightarrow\,14{\scriptscriptstyle\,\pm 35}$	$0{\scriptscriptstyle\,\pm 0}\,\rightarrow\,0{\scriptscriptstyle\,\pm 0}$	$0{\scriptscriptstyle\,\pm 0}\,\rightarrow\,\mathbf{100}{\scriptscriptstyle\,\pm 1}$	$23{\scriptscriptstyle\,\pm 6}\,\rightarrow\,19{\scriptscriptstyle\,\pm 33}$	$8{\scriptscriptstyle\,\pm 3}\,\rightarrow\,38{\scriptscriptstyle\,\pm 52}$	$4{\scriptscriptstyle\,\pm 4}\,\rightarrow\,\mathbf{100}{\scriptscriptstyle\,\pm 0}$	$12{\scriptscriptstyle\,\pm 5}\,\rightarrow\,88{\scriptscriptstyle\,\pm 32}$
antmaze-umaze-v2	$77\,\rightarrow\,\mathbf{96}$	$98\,\rightarrow\,75$	$77\,\rightarrow\,\mathbf{100}$	$0{\scriptscriptstyle\,\pm 0}\,\rightarrow\,\mathbf{98}{\scriptscriptstyle\,\pm 3}$	$94{\scriptscriptstyle\,\pm 5}\,\rightarrow\,\mathbf{96}{\scriptscriptstyle\,\pm 2}$	$97{\scriptscriptstyle\,\pm 2}\,\rightarrow\,\mathbf{99}{\scriptscriptstyle\,\pm 1}$	$96{\scriptscriptstyle\,\pm 3}\,\rightarrow\,\mathbf{99}{\scriptscriptstyle\,\pm 2}$	$90{\scriptscriptstyle\,\pm 5}\,\rightarrow\,\mathbf{99}{\scriptscriptstyle\,\pm 1}$

Table 2: Offline-to-online RL results. Performance before and after online fine-tuning on the 15-task benchmark suite. CoDrift results are averaged over 8 seeds, and baseline values are taken from prior work. Lower average rank is better.

Task	CoDrift	w/o Marg	$\Delta$
antmaze-large	$\mathbf{84}{\scriptscriptstyle\,\pm 3}$	$77{\scriptscriptstyle\,\pm 6}$	$+7$
humanoidmaze-medium	$\mathbf{83}{\scriptscriptstyle\,\pm 5}$	$77{\scriptscriptstyle\,\pm 5}$	$+6$
cube-single	$\mathbf{92}{\scriptscriptstyle\,\pm 3}$	$86{\scriptscriptstyle\,\pm 3}$	$+6$
puzzle-3x3	$\mathbf{82}{\scriptscriptstyle\,\pm 13}$	$74{\scriptscriptstyle\,\pm 4}$	$+8$
Average	$\mathbf{85.3}$	$78.5$	$+6.8$

Table 3: Effectiveness of marginal drifting. Success rates on four representative tasks, averaged over 8 seeds (

\pm

standard deviation). w/o Marg removes the marginal behavioral field.

\Delta

denotes the resulting gain from marginal drifting.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 298 papers

Drift Q-Learning
May 29, 2026Anas Houssaini, Mohamad H. Danesh, Amin Abyaneh +3Reinforcement Learning Q-Learning
McGill University · Mila - Quebec AI Institute
Drifting Field Policy: A One-Step Generative Policy via Wasserstein Gradient Flow
May 8, 2026Juil Koo, Mingue Park, Jiwon Choi +2Robot Policy Learning Wasserstein Gradient Flows
KAIST
Discrete Flow Matching for Offline-to-Online Reinforcement Learning
May 12, 2026Fairoz Nower Khan, Nabuat Zaman Nahim, Peizhong JuFlow Matching Reinforcement Learning
Department of Computer Science, University of Kentucky

May 29, 2026J/K move · Enter open · S save

Drift Q-Learning

Anas Houssaini, Mohamad H. Danesh, Amin Abyaneh, Scott Fujimoto+2

McGill University · Mila - Quebec AI Institute

Reinforcement Learning Q-Learning Offline RL Policy Learning

Open paper AskPDF arXiv

Offline reinforcement learning requires improving a policy from fixed data while avoiding out-of-distribution actions with unreliable value estimates. Diffusion and flow policies handle this trade-off by modeling the behavior distribution to regularize the RL objective, but they require iterative denoising, solver integrations, and in more efficient variants, distillation or other approximations at inference. We propose DriftQL, which combines a drift-based behavioral regularizer with critic-driven policy improvement. The value signal biases the policy toward high-value regions of the data support, while attraction and repulsion together keep generated actions near the data and prevent collapse onto a single mode. DriftQL is implemented as a single network with a unified training objective and generates actions in a single forward pass. On D4RL and OGBench, DriftQL consistently outperforms diffusion and flow methods, advancing the state of the art. Under degraded data quality, where the baselines visibly struggle, DriftQL remains close to its clean-data performance, positioning it as a promising alternative to diffusion and flow-based methods while maintaining the simplicity and efficiency of deterministic approaches. Project page: https://driftql.github.io/

Abstract

Offline RL

Explore similar work

Drift Q-Learning

Drifting Field Policy: A One-Step Generative Policy via Wasserstein Gradient Flow

Discrete Flow Matching for Offline-to-Online Reinforcement Learning