cs.LGOct 7, 2026
SaveQ-Learning with Scalar Adjoint Matching
Organizations: KAIST · RLWRLD
Abstract
Flow policies capture rich and diverse action distributions, and fine-tuning them with off-policy RL to improve beyond the demonstrations has drawn growing interest. However, fine-tuning a flow policy against a learned value function is not trivial, because the policy generates its action over many flow steps. Adjoint matching offers a principled way to update the flow model itself by propagating value information from the final action back to each flow step, but it requires a vector--Jacobian product through the policy at every step, a cost that grows with the number of flow steps and the policy size. We observe that the batch-averaged velocity Jacobian of pretrained flow policies concentrates on its diagonal. Motivated by this finding, we derive a closed-form scalar adjoint that scales the value gradient at the final action by the flow time, eliminating the per-step vector--Jacobian products. We further find that controlling the critic's value at policy-generated actions is particularly important under the scalar adjoint. Based on these findings, we propose Q-learning with Scalar Adjoint Matching (SQAM), which combines the scalar adjoint with a value penalty at those actions. SQAM's gains concentrate on the four hardest OGBench domains, where its success rate exceeds that of the strongest baseline in each domain by 18 to 35 percentage points. To test whether SQAM extends to large pretrained policies, we also fine-tune a vision-language-action policy on a real bimanual robot. SQAM improves over supervised fine-tuning on all three tasks.
Figures & tables
Figure 1 : Adjoint matching propagates the critic’s gradient at the final action back to the flow steps that produced it, and the propagated gradient updates the flow model. Left: QAM and TRQAM integrate the lean adjoint ODE backward along the sampling chain, which takes a vector–Jacobian product through the policy at every flow step, so the cost grows with the number of flow steps and the policy size. Right: SQAM instead scales the gradient at the final action, , by the flow time , with no vector–Jacobian product at any flow step.
Figure 2 : Velocity Jacobian of the pretrained flow on four domains at two flow times, averaged over a batch of states and normalized per panel by the mean diagonal. The batch-averaged Jacobian is dominated by its diagonal.
Figure 3 : Simplified versions of the TRQAM and SQAM algorithms. The parts SQAM changes are marked in blue , and Appendix A gives the full algorithm.
| al | ag | hm | hl | scene | p33 | p44 | c2 | c3 | c4 | all | ||
| 5 tasks | 5 tasks | 5 tasks | 5 tasks | 5 tasks | 5 tasks | 5 tasks | 5 tasks | 5 tasks | 5 tasks | 50 tasks | ||
| Backprop | FQL | 38 9 | 2 6 | 74 5 | 2 1 | 70 5 | 25 10 | 9 7 | 44 4 | 7 5 | 9 5 | 28 |
| Guidance | CGQL-L | 48 7 | 7 5 | 57 2 | 6 3 | 58 1 | 0 0 | 0 0 | 55 2 | 0 1 | 1 1 | 23 |
| Post Processing | DSRL | 53 2 | 1 1 | 53 10 | 1 1 | 80 0 | 100 0 | 61 8 | 72 4 | 34 6 | 9 3 | 46 |
| IFQL | 29 8 | 12 3 | 93 2 | 30 7 | 36 1 | 64 4 | 42 4 | 9 2 | 24 7 | 6 3 | 35 | |
| Adjoint Matching | QAM | 62 9 | 29 4 | 64 7 | 4 3 | 64 4 | 15 3 | 1 1 | 71 2 | 19 6 | 18 3 | 35 |
Table 1 : Offline RL on 50 OGBench ( Park et al., 2025a ) tasks at 1M training steps (8 seeds). Mean success rate (%) one standard deviation. Domain abbreviations are listed in Section D.1 , and the per-task breakdown across all 50 tasks is in Table 8 .
Figure 4 : Exact and scalar adjoints computed on the same rollout states. Left: the norm of each adjoint relative to , where the scalar adjoint is the dotted line . Right: the angle between the two adjoints. The two adjoints contract similarly in norm and stay positively aligned. The angle is larger on the manipulation domains, where the scalar adjoint alone underperforms. Three seeds per domain at 0.3M steps, without the value penalty.
Figure 5 : Value learning schemes compared on the two domains where the scalar adjoint alone fails. Each scheme is shown at its best coefficient on the grid we swept, and the full sweeps on all ten domains are in Section F.6 . The value penalty lifts both domains and the gradient-norm penalty follows it on cube-quadruple , while ReBRAC and the in-sample critic stay close to the scalar adjoint without regularization. Eight seeds per curve, and bands are one standard deviation.
Figure 6 : Sweep over value penalty on four of the ten swept domains. Eight seeds per curve, and bands are one standard deviation. The full analysis on all ten domains is in Section F.5 .
Figure 7 : Sweep over KL budget on four of the ten swept domains. Eight seeds per curve, and bands are one standard deviation. The full analysis on all ten domains is in Section F.5 .
Figure 8 : Overview of the three real-world manipulation tasks. From top to bottom: flipping a plastic bag, placing a straw in a cup, and placing fruit in a pot and closing the lid. Each row illustrates the task progression from left to right.
| Method | Flip plastic bag | Place the straw | Put the fruit and close the lid |
| SFT | |||
| EXPO-FT (offline) | |||
| EXPO-FT (offline2online) | |||
| SQAM (offline) | |||
| SQAM (offline2online) |
Table 2 : Success rate on three real-robot manipulation tasks with a vision-language-action policy. The offline rows evaluate the checkpoint after 15K offline steps, and the offline-to-online rows the checkpoint after a further 10K online steps. Each cell counts successes over held-out episodes.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| {\color[rgb]{0.2578,0.5234,0.957}X_{\tau+h}=X_{\tau}+h\,v^{\mathrm{ft}}_{\theta}(s,X_{\tau},\tau),\quad X_{0}\sim\mathcal{N}(0,I),} |
Algorithm 3 Q-learning with Scalar Adjoint Matching (SQAM)
| Domain | Category | Data | Horizon | Act. dim. |
| cube-double-* | Manipulation | 1M | 500 | 5 |
| cube-triple-10M-* | Manipulation | 10M | 1000 | 5 |
| cube-quadruple-100M-* | Manipulation | 100M | 1000 | 5 |
| antmaze-large-* | Locomotion | 1M | 1000 | 8 |
| antmaze-giant-10M-* | Locomotion | 10M | 1000 | 8 |
| humanoidmaze-medium-* | Locomotion | 1M | 2000 | 21 |
Table 3: Domain metadata. Following OGBench, we group the domains into locomotion and manipulation. Action dimensions are per control step, and the manipulation domains use action chunks of size .
| Parameter | Value |
| Batch size | 256 |
| Discount factor ( ) | 0.995 (default), 0.999 ( humanoidmaze ) |
| Optimizer | Adam |
| Learning rate | |
| Target network update rate | |
| Critic ensemble size ( ) | 10 |
Table 4: Common hyperparameters.
| Domain | FQL | DSRL | IFQL | CGQL-L | QAM | QAM-E | TRQAM | SQAM |
| scene-* | 300 | 0.4 | 0.9 | 1 | 0.5 | |||
| puzzle-3x3-* | 300 | 1.0 | 0.95 | 3 | 2.0 | |||
| puzzle-4x4-10M-* | 1 | 1.0 | 0.9 | 30 | 4.0 | |||
| cube-double-* | 300 | 1.0 | 0.9 | 1 | 0.5 | |||
| cube-triple-10M-* | 30 | 1.4 | 0.95 | 3 | 0.5 |
Table 5: Domain-specific hyperparameters.
Figure 9 : Hardware setup: RB-Y1 humanoid with a Wuji Hand 2 on each wrist and a head-mounted ZED Mini stereo camera.
| Task | Language instruction | Demos | |
| flip-plastic-bag | “flip the plastic bag” | 50 | 0.999 |
| place-straw | “place the straw in the cup” | 50 | 0.999 |
| put-and-close | “put the fruit in the pot and close the lid” | 50 | 0.999 |
Table 6: Task specification. For each task we list the language instruction, the number of teleoperated demonstrations, and the discount factor .
| Parameter | Value |
| Shared | |
| Offline RL steps | 15,000 |
| Online RL steps | 10,000 |
| Batch size | 128 offline, 192 online |
| Policy delay | 5 |
| UTD ratio | 1 |
Table 7: Hyperparameters for the real-robot runs. The shared block applies to both methods, and the blocks below give each method’s own values.
Figure 10 : Success rate of the training rollouts during the online RL phase, over a trailing window of 20 episodes, against minutes of robot data collected. The dashed line is the SFT policy’s held-out success rate. SQAM works at the scale of a vision-language-action policy, improving over SFT on all three tasks.
Figure 11 : Evaluation episodes on flip-plastic-bag . Left: superimposed final frames of all episodes. Right: the main failure mode of each method.
Figure 12 : Evaluation episodes on place-straw . Left: superimposed final frames of all episodes. Right: the main failure mode of each method.
Figure 13 : Evaluation episodes on put-and-close . Left: superimposed final frames of all episodes. Right: the main failure mode of each method.
Figure 14 : Success episode durations normalized by each task’s SFT median (100%). Bars show medians, error bars indicate the 25th–75th percentiles, and denotes the number of success episodes.
Figure 15 : Wall-clock time of one update with the exact and the scalar adjoint. Left: actor width, with the critic fixed at . Right: number of flow steps . Percentages give the extra time of the exact adjoint.
Figure 16 : TRQAM and SQAM with and without the value penalty ( and ) during offline training on task 2 of cube-triple and cube-quadruple . Four seeds per curve, with the trust region at the KL budget of each method.
Figure 17 : SQAM with a constant , for five values, against SQAM with the trust region (dashed) on one representative task from each of the ten domains. Eight seeds per curve, and bands are one standard deviation.
Figure 18 : The batch-averaged velocity Jacobian on the locomotion domains and scene at four flow times, each panel normalized by its mean diagonal.
Figure 19 : The batch-averaged velocity Jacobian on the cube and puzzle domains at four flow times, with the same normalization as Figure 18 .
Figure 20 : The KL budget swept from to on all ten domains. Eight seeds per budget, and bands are one standard deviation.
Figure 21 : The value penalty coefficient swept over on all ten domains. Eight seeds per coefficient, and bands are one standard deviation.
Figure 22 : The gradient-norm penalty coefficient swept over on all ten domains, with the unregularized critic ( none ) as the dashed reference. Eight seeds per coefficient, and bands are one standard deviation.
Figure 23 : ReBRAC’s bootstrap penalty swept over on all ten domains, with the unregularized critic ( none ) as the dashed reference. Eight seeds per coefficient, and bands are one standard deviation.
Figure 24 : The in-sample critic swept over the expectile on all ten domains, with the unregularized critic ( none ) as the dashed reference. Eight seeds per expectile, and bands are one standard deviation.
Figure 25 : Success rate curves on the locomotion domains and scene , with rows for domains and columns for their five tasks. Offline training runs to 1M steps and online training to 1.5M, with the dashed line at the transition. Eight seeds per curve, and bands are one standard deviation.
Figure 26 : Success rate curves on the cube and puzzle domains, with rows for domains and columns for their five tasks. Offline training runs to 1M steps and online training to 1.5M, with the dashed line at the transition. Eight seeds per curve, and bands are one standard deviation.