Q-Learning with Scalar Adjoint Matching

Yonghoon Dong; Minsung Yoon; Jaehyuk Kim; Jungwoo Park; Changyeon Kim; Jinwoo Shin

cs.LGOct 7, 2026

Q-Learning with Scalar Adjoint Matching

Authors: Yonghoon Dong, Minsung Yoon, Jaehyuk Kim, Jungwoo Park, Changyeon Kim, Jinwoo Shin

Organizations: KAIST · RLWRLD

Abstract

Flow policies capture rich and diverse action distributions, and fine-tuning them with off-policy RL to improve beyond the demonstrations has drawn growing interest. However, fine-tuning a flow policy against a learned value function is not trivial, because the policy generates its action over many flow steps. Adjoint matching offers a principled way to update the flow model itself by propagating value information from the final action back to each flow step, but it requires a vector--Jacobian product through the policy at every step, a cost that grows with the number of flow steps and the policy size. We observe that the batch-averaged velocity Jacobian of pretrained flow policies concentrates on its diagonal. Motivated by this finding, we derive a closed-form scalar adjoint that scales the value gradient at the final action by the flow time, eliminating the per-step vector--Jacobian products. We further find that controlling the critic's value at policy-generated actions is particularly important under the scalar adjoint. Based on these findings, we propose Q-learning with Scalar Adjoint Matching (SQAM), which combines the scalar adjoint with a value penalty at those actions. SQAM's gains concentrate on the four hardest OGBench domains, where its success rate exceeds that of the strongest baseline in each domain by 18 to 35 percentage points. To test whether SQAM extends to large pretrained policies, we also fine-tune a vision-language-action policy on a real bimanual robot. SQAM improves over supervised fine-tuning on all three tasks.

Figures & tables

Figure 1 : Adjoint matching propagates the critic’s gradient at the final action back to the flow steps that produced it, and the propagated gradient updates the flow model. Left: QAM and TRQAM integrate the lean adjoint ODE backward along the sampling chain, which takes a vector–Jacobian product

J_{i}^{\top}

through the policy at every flow step, so the cost grows with the number of flow steps and the policy size. Right: SQAM instead scales the gradient at the final action,

\nabla_{a}Q(s,a)

, by the flow time

\tau

, with no vector–Jacobian product at any flow step.

Figure 2 : Velocity Jacobian of the pretrained flow on four domains at two flow times, averaged over a batch of states and normalized per panel by the mean diagonal. The batch-averaged Jacobian is dominated by its diagonal.

Figure 3 : Simplified versions of the TRQAM and SQAM algorithms. The parts SQAM changes are marked in blue , and Appendix A gives the full algorithm.

scene

p33

p44

all

5 tasks

50 tasks

Backprop

FQL

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

Guidance

CGQL-L

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

Post Processing

DSRL

\pm

\pm

\pm

\pm

\pm

100

\pm

\pm

\pm

\pm

\pm

IFQL

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

Adjoint Matching

QAM

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

Table 1 : Offline RL on 50 OGBench ( Park et al., 2025a ) tasks at 1M training steps (8 seeds). Mean success rate (%)

\pm

one standard deviation. Domain abbreviations are listed in Section D.1 , and the per-task breakdown across all 50 tasks is in Table 8 .

Figure 4 : Exact and scalar adjoints computed on the same rollout states. Left: the norm of each adjoint relative to

\|\nabla_{a}Q\|

, where the scalar adjoint is the dotted line

\tau

. Right: the angle between the two adjoints. The two adjoints contract similarly in norm and stay positively aligned. The angle is larger on the manipulation domains, where the scalar adjoint alone underperforms. Three seeds per domain at 0.3M steps, without the value penalty.

Figure 5 : Value learning schemes compared on the two domains where the scalar adjoint alone fails. Each scheme is shown at its best coefficient on the grid we swept, and the full sweeps on all ten domains are in Section F.6 . The value penalty lifts both domains and the gradient-norm penalty follows it on cube-quadruple , while ReBRAC and the in-sample critic stay close to the scalar adjoint without regularization. Eight seeds per curve, and bands are one standard deviation.

Figure 6 : Sweep over value penalty

c

on four of the ten swept domains. Eight seeds per curve, and bands are one standard deviation. The full analysis on all ten domains is in Section F.5 .

Figure 7 : Sweep over KL budget

\epsilon_{\text{KL}}

on four of the ten swept domains. Eight seeds per curve, and bands are one standard deviation. The full analysis on all ten domains is in Section F.5 .

Figure 8 : Overview of the three real-world manipulation tasks. From top to bottom: flipping a plastic bag, placing a straw in a cup, and placing fruit in a pot and closing the lid. Each row illustrates the task progression from left to right.

Method	Flip plastic bag	Place the straw	Put the fruit and close the lid
SFT	$15/30$	$7/30$	$4/30$
EXPO-FT (offline)	$8/30$	$0/30$	$5/30$
EXPO-FT (offline2online)	$15/30$	$0/30$	$3/30$
SQAM (offline)	$20/30$	$13/30$	$8/30$
SQAM (offline2online)	$\mathbf{25/30}$	$\mathbf{15/30}$	$\mathbf{13/30}$

Table 2 : Success rate on three real-robot manipulation tasks with a vision-language-action policy. The offline rows evaluate the checkpoint after 15K offline steps, and the offline-to-online rows the checkpoint after a further 10K online steps. Each cell counts successes over

30

held-out episodes.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

{\color[rgb]{0.2578,0.5234,0.957}X_{\tau+h}=X_{\tau}+h\,v^{\mathrm{ft}}_{\theta}(s,X_{\tau},\tau),\quad X_{0}\sim\mathcal{N}(0,I),}

Appendix

Algorithm 3 Q-learning with Scalar Adjoint Matching (SQAM)

Domain	Category	Data	Horizon	Act. dim.
cube-double-*	Manipulation	1M	500	5
cube-triple-10M-*	Manipulation	10M	1000	5
cube-quadruple-100M-*	Manipulation	100M	1000	5
antmaze-large-*	Locomotion	1M	1000	8
antmaze-giant-10M-*	Locomotion	10M	1000	8
humanoidmaze-medium-*	Locomotion	1M	2000	21

Appendix

Table 3: Domain metadata. Following OGBench, we group the domains into locomotion and manipulation. Action dimensions are per control step, and the manipulation domains use action chunks of size

5

Parameter	Value
Batch size	256
Discount factor ( $\gamma$ )	0.995 (default), 0.999 ( humanoidmaze )
Optimizer	Adam
Learning rate	$3\times 10^{-4}$
Target network update rate	$5\times 10^{-3}$
Critic ensemble size ( $K$ )	10

Appendix

Table 4: Common hyperparameters.

Domain	FQL	DSRL	IFQL	CGQL-L	QAM	QAM-E	TRQAM	SQAM
	$\alpha$	$\sigma_{z}$	$\kappa$	$(\vartheta,\varrho,\tau)$	$\beta$	$(\beta,\sigma_{a})$	$\varepsilon_{\mathrm{KL}}$	$(\varepsilon_{\mathrm{KL}},c)$
scene-*	300	0.4	0.9	$(10,0.1,0.1)$	1	$(1,0)$	0.5	$(0.5,0.3)$
puzzle-3x3-*	300	1.0	0.95	$(10,0.001,0.1)$	3	$(1,0.1)$	2.0	$(2.0,0.3)$
puzzle-4x4-10M-*	1	1.0	0.9	$(10,0.001,1)$	30	$(0.1,0.9)$	4.0	$(4.0,0.3)$
cube-double-*	300	1.0	0.9	$(10,0.001,0.01)$	1	$(1,0)$	0.5	$(0.5,0)$
cube-triple-10M-*	30	1.4	0.95	$(10,0.001,0.1)$	3	$(3,0.1)$	0.5	$(0.5,0.3)$

Appendix

Table 5: Domain-specific hyperparameters.

Figure 9 : Hardware setup: RB-Y1 humanoid with a Wuji Hand 2 on each wrist and a head-mounted ZED Mini stereo camera.

Task	Language instruction	Demos	$\gamma$
flip-plastic-bag	“flip the plastic bag”	50	0.999
place-straw	“place the straw in the cup”	50	0.999
put-and-close	“put the fruit in the pot and close the lid”	50	0.999

Appendix

Table 6: Task specification. For each task we list the language instruction, the number of teleoperated demonstrations, and the discount factor

\gamma

Parameter	Value
Shared
Offline RL steps	15,000
Online RL steps	10,000
Batch size	128 offline, 192 online
Policy delay	5
UTD ratio	1

Appendix

Table 7: Hyperparameters for the real-robot runs. The shared block applies to both methods, and the blocks below give each method’s own values.

Figure 10 : Success rate of the training rollouts during the online RL phase, over a trailing window of 20 episodes, against minutes of robot data collected. The dashed line is the SFT policy’s held-out success rate. SQAM works at the scale of a vision-language-action policy, improving over SFT on all three tasks.

Figure 11 : Evaluation episodes on flip-plastic-bag . Left: superimposed final frames of all episodes. Right: the main failure mode of each method.

Figure 12 : Evaluation episodes on place-straw . Left: superimposed final frames of all episodes. Right: the main failure mode of each method.

Figure 13 : Evaluation episodes on put-and-close . Left: superimposed final frames of all episodes. Right: the main failure mode of each method.

Figure 14 : Success episode durations normalized by each task’s SFT median (100%). Bars show medians, error bars indicate the 25th–75th percentiles, and

n

denotes the number of success episodes.

Figure 15 : Wall-clock time of one update with the exact and the scalar adjoint. Left: actor width, with the critic fixed at

256\times 4

. Right: number of flow steps

F

. Percentages give the extra time of the exact adjoint.

Figure 16 : TRQAM and SQAM with and without the value penalty (

c=0.3

and

c=0

) during offline training on task 2 of cube-triple and cube-quadruple . Four seeds per curve, with the trust region at the KL budget of each method.

Figure 17 : SQAM with a constant

\lambda

, for five values, against SQAM with the trust region (dashed) on one representative task from each of the ten domains. Eight seeds per curve, and bands are one standard deviation.

Figure 18 : The batch-averaged velocity Jacobian on the locomotion domains and scene at four flow times, each panel normalized by its mean diagonal.

Figure 19 : The batch-averaged velocity Jacobian on the cube and puzzle domains at four flow times, with the same normalization as Figure 18 .

Figure 20 : The KL budget

\varepsilon_{\mathrm{KL}}

swept from

0.5

4

on all ten domains. Eight seeds per budget, and bands are one standard deviation.

Figure 21 : The value penalty coefficient

c

swept over

\{0,0.1,0.2,0.3\}

on all ten domains. Eight seeds per coefficient, and bands are one standard deviation.

Figure 22 : The gradient-norm penalty coefficient

c_{g}

swept over

\{0.002,0.01,0.05,0.25\}

on all ten domains, with the unregularized critic ( none ) as the dashed reference. Eight seeds per coefficient, and bands are one standard deviation.

Figure 23 : ReBRAC’s bootstrap penalty swept over

\beta\in\{0.01,0.1,1,10\}

on all ten domains, with the unregularized critic ( none ) as the dashed reference. Eight seeds per coefficient, and bands are one standard deviation.

Figure 24 : The in-sample critic swept over the expectile

\kappa\in\{0.7,0.9,0.95,0.99\}

on all ten domains, with the unregularized critic ( none ) as the dashed reference. Eight seeds per expectile, and bands are one standard deviation.

Figure 25 : Success rate curves on the locomotion domains and scene , with rows for domains and columns for their five tasks. Offline training runs to 1M steps and online training to 1.5M, with the dashed line at the transition. Eight seeds per curve, and bands are one standard deviation.

Figure 26 : Success rate curves on the cube and puzzle domains, with rows for domains and columns for their five tasks. Offline training runs to 1M steps and online training to 1.5M, with the dashed line at the transition. Eight seeds per curve, and bands are one standard deviation.

Explore similar work

May 26, 2026cs.LG

Trust Region Q Adjoint Matching

Off-policy reinforcement learning of pretrained flow policies remains challenging due to the instability of optimization arising from the multi-step sampling process. Recently, Q-learning with Adjoint Matching (QAM) addressed this issue by reformulating into a memoryless stochastic optimal control (SOC) problem with a learned critic. However, QAM inherits a fundamental fragility of critic-guided improvement: small critic errors are amplified when critics are ill-conditioned, often leading to model collapse. This paper introduces Trust Region Q-Adjoint Matching (TRQAM), a stable off-policy fine-tuning algorithm that adaptively controls the path-space KL with pretrained flow policies through projected dual descent. Specifically, we optimize the trust-region parameter

λ

in SOC dynamics, and theoretically show that the path-space KL can be represented by a closed-form function of

λ

. As a result, our method can precisely control the exact deviation from pretrained flow policies, achieving stable off-policy RL. Through experiments on 50 OGBench tasks, TRQAM consistently outperforms prior arts in both offline RL and offline-to-online RL. In particular, TRQAM achieves an overall success rate of 68% in offline RL, substantially improves the strongest baseline at 46%.

Yonghoon Dong, Kyungmin Lee, Changyeon Kim +2

1KAIST AI · 2Seoul National University · 3RLWRLD

Flow Policies Q-Learning Adjoint Matching Descent AI Trustworthiness

Ask PDF

Sep 28, 2026cs.LG

QAMM: Adjoint MeanFlow Matching for Few-Step Offline Reinforcement Learning

Flow policies can model rich action distributions, but their iterative sampling limits decision speed. Adjoint matching uses the critic's action gradient to improve a flow policy without backpropagating through its sampling trajectory, yet its supervision is defined for instantaneous velocities. We propose QAMM, a method that turns the critic-derived adjoint signal into supervision for MeanFlow's average velocity. The resulting policy learns finite-interval transport directly and generates actions with few network evaluations. We derive the adjoint MeanFlow target, specify its gradient boundaries, and train it with an offline actor-critic. On ten HumanoidMaze tasks, QAMM produces effective two-call policies and achieves competitive performance against strong flow-policy baselines. These results show that adjoint-based Q optimization can be combined with average-velocity learning to obtain expressive offline policies with few-step action generation.

Yuehu Gong, Shutong Ding, Mokai Pan +4

School of Data Science, Fudan University · ShanghaiTech University · Shanghai Innovation Institute

Flow Policies Meanflow Multimodal Action Distributions Action Generation

Ask PDF

Jun 6, 2026cs.RO

Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policies

We propose Q-Guided Value-Gradient Matching (Q-VGM), an offline-to-online RL method for fine-tuning flow-matching vision-language-action policies with a learned critic. Directly applying critic gradients to flow policies requires backpropagation through the multi-step denoising process (BPTT), which is costly and unstable at VLA scale. Motivated by an optimal-control view of denoising, we derive a local approximation connecting clean-action value gradients to local velocity corrections at late denoising steps. Q-VGM uses this connection to construct critic-guided targets for fine-tuning the pretrained action expert through velocity matching. Gradients pass only through local velocity predictions, avoiding backpropagation through the denoising trajectory. An action-sensitive chunk critic is trained with IQL offline and TD learning online. On LIBERO, Q-VGM improves the few-shot-SFT four-suite average from 84.6% to 90.7% offline using 150 policy rollouts per suite, and reaches 98.4% after online training. Compared with on-policy RL, Q-VGM achieves 5.3x higher sample efficiency on average across four LIBERO suites, measured by the episode budget for 95% rollout success rate. On three real-world bimanual tasks, offline Q-VGM improves average success from 66.7% to 98.3%.

Ziqian Wang, Rui Zhang, Yitian Liu +3

Shanghai Jiao Tong University · University of Michigan, Ann Arbor

Flow-Matching Vision-Language-Action Flow Policies Optical Flow

Ask PDF