SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer

Nathan Samuel de Lara; Florian Shkurti

cs.LGFeb 19, 2026

SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer

Authors: Nathan Samuel de Lara, Florian Shkurti

Organizations: Department of Computer Science, University of Toronto, Toronto, Canada · Vector Institute

Abstract

Modern offline Reinforcement Learning (RL) methods find performant actor-critics, however, fine-tuning these actor-critics online with value-based RL algorithms typically causes immediate drops in performance. We provide evidence consistent with the hypothesis that, in the loss landscape, offline maxima for prior algorithms and online maxima are separated by low-performance valleys that gradient-based fine-tuning traverses. Following this, we present Score Matched Actor-Critic (SMAC), an offline RL method designed to learn actor-critics that transition to online value-based RL algorithms with no drop in performance. SMAC avoids valleys between offline and online maxima by regularizing the Q-function during the offline phase to respect a first-order derivative equality between the score of the policy and action-gradient of the Q-function. We experimentally demonstrate that SMAC converges to offline maxima that are connected to better online maxima via paths with monotonically increasing reward found by first-order optimization. SMAC achieves smooth transfer to Soft Actor-Critic and TD3 in 6/6 D4RL tasks. In 4/6 environments, it reduces regret by 34-58% over the best baseline.

Figures & tables

Figure 1: Past offline RL methods converge to maxima separated from online optima by low-reward valleys . Top: reward landscapes on the Kitchen task for CalQL (left) and SMAC (right). Blue and checkered flags being the real locations of the pre-trained and fine-tuned checkpoints on the landscape respectively. The paths and red/yellow flag are illustrative annotations showing the hypothesized trajectory during transfer. Paths demonstrate the existence of a low reward valley between pre-trained and fine-tuned checkpoints when using CalQL. Our method SMAC has no such valleys and is on a unified hill with the fine-tuning checkpoint. Bottom: SMAC vs. CalQL performance in the Kitchen task. See Section 5 for analysis.

Figure 2: Increasing dataset size and coverage does not bridge offline-to-online gap. We generate rollouts in two environments with a policy that has a 0.7 success rate and plot the offline-to-online performance as we increase the dataset size. We observe that even when the dataset is so large that it is sufficient for learning optimal policies, the actor-critics found are still quickly unlearned by online fine-tuning.

Figure 3: Reward visualized along a plane in parameter space reveals difference in maxima found by different pre-training and fine-tuning methods on Kitchen task . We see that the SAC maxima are wider and not connected to the pre-trained checkpoint along monotonically improving line across all baselines. Conversely, SAC maxima and SMAC maxima are linearly connected. Subplot titles denote offline algorithm used

Figure 4: Reward valleys when linearly interpolating between pre-training and fine-tuning checkpoints for all baselines in tasks show linearly disconnected maxima consistent with offline-to-online transfer performance in later plots . We plot the performance along the line between the pre-trained checkpoint and final fine-tuning checkpoint for methods in kitchen-partial (left), door-binary (centre), and hopper-medium-replay (right).

0

is the pre-trained checkpoint, and

1

is the SAC fine-tuned checkpoint. Lines show mean over 4 seeds with shading being standard error.

Figure 5: t-SNE projections of training trajectories show linearly disconnected maxima . We take the pre-training checkpoints, SAC fine-tuning checkpoints, and TD3+BC fine-tuning checkpoints and plot their T-SNE projections with lines and arrows signifying the training trajectory/ordering of the checkpoints. We observe that the projected checkpoints (i) travel in straight lines, and (ii) cross a valley of low reward when fine-tuned with SAC but not when fine-tuned with TD3+BC, providing evidence consistent with the reward valley hypothesis.

Figure 6: SMAC achieves smooth offline-to-online transfer to SAC . Plot shows offline-to-online transfer results for SMAC and baselines when fine-tuned with SAC. Environments are only named with the first word since that uniquely identifies them among the six. The dotted line shows where the agent transfers from offline learning to online learning. The shaded region shows the final offline training window before transfer; the first 200k offline gradient steps are omitted from the plot.

Figure 7: SMAC achieves smooth offline-to-online transfer to TD3 . Plot shows offline-to-online transfer results for SMAC and baselines when fine-tuned with TD3. The format follows Figure 6 , including the omission of the first 200k offline gradient steps.

Figure 8: Fine-tuning with TD3+BC stabilizes offline-to-online . Plot shows offline-to-online transfer results for SMAC and baselines when fine-tuned with TD3+BC. The format follows Figure 6 , including the omission of the first 200k offline gradient steps.

	Online Algorithm
Offline Algorithm	AWR	SAC	TD3	TD3+BC
IQL	0.508	0.471	0.653	0.494
SMAC	0.380	0.031	0.090	0.226
TD3+BC	0.654	0.962	0.545	0.562
CalQL/CQL	0.482	0.448	0.442	0.614

Table 1: Normalized Regret averaged over all six environments (

\downarrow

lower is better). Values are min-max normalized per environment then averaged. SMAC achieves the lowest regret across all four online algorithms, with particularly strong gains when paired with SAC.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Offline Batch Size has negligible effect on failing baselines, but impacts SMAC

Figure 10: Larger batch sizes lead to faster online adaptation

Figure 11: Removing RvS hurts transfer in 2/3 environments but has mixed impact on online performance. SMAC still dominates baselines without using RvS for training the diffusion model

Figure 12: Q-function and score of policy are proportional throughout training . We plot the Q-function and policy for a SAC agent after 50, 100, 250, and 400 episodes of training in the Reacher-v2 environment.

	Offline Algorithm
Environment	CalQL/CQL	IQL	SMAC	TD3+BC
door	134.5 $\pm$ 1.6	120.0 $\pm$ 0.8	50.3 $\pm$ 2.7	129.7 $\pm$ 2.4
hopper	293.7 $\pm$ 20.4	798.0 $\pm$ 126.5	386.3 $\pm$ 10.8	3213.3 $\pm$ 16.3
kitchen	467.1 $\pm$ 38.1	492.9 $\pm$ 13.7	131.4 $\pm$ 13.0	762.5 $\pm$ 11.4
pen	8.0 $\pm$ 0.7	10.3 $\pm$ 0.7	5.3 $\pm$ 0.7	55.2 $\pm$ 1.9
relocate	98.1 $\pm$ 0.4	97.4 $\pm$ 0.4	62.8 $\pm$ 2.1	99.2 $\pm$ 0.0

Appendix

Table 2: Offline-to-online regret when fine-tuning with SAC (

\downarrow

lower is better). Entries are mean

\pm

std over seeds; “-” indicates not evaluated. Cell colors are assigned per environment across all four tables : best (gold), second (silver), third (bronze).

	Offline Algorithm
Environment	CalQL/CQL	IQL	SMAC	TD3+BC
door	131.1 $\pm$ 3.0	126.1 $\pm$ 0.4	65.5 $\pm$ 2.1	137.7 $\pm$ 1.4
hopper	311.4 $\pm$ 46.5	1834.3 $\pm$ 150.3	297.3 $\pm$ 6.5	329.3 $\pm$ 39.0
kitchen	526.6 $\pm$ 32.3	445.9 $\pm$ 13.2	259.3 $\pm$ 22.8	529.1 $\pm$ 21.6
pen	8.4 $\pm$ 0.7	16.1 $\pm$ 0.5	8.5 $\pm$ 0.8	41.6 $\pm$ 2.9
relocate	97.8 $\pm$ 0.5	95.2 $\pm$ 0.3	58.3 $\pm$ 1.3	99.1 $\pm$ 0.0

Appendix

Table 3: Offline-to-online regret when fine-tuning with TD3 (

\downarrow

lower is better). Entries are mean

\pm

std over seeds; “-” indicates not evaluated. Cell colors are assigned per environment across all four tables : best (gold), second (silver), third (bronze).

	Offline Algorithm
Environment	CalQL/CQL	IQL	SMAC	TD3+BC
door	129.9 $\pm$ 0.7	127.9 $\pm$ 0.4	122.8 $\pm$ 0.8	136.0 $\pm$ 0.7
hopper	533.7 $\pm$ 43.0	958.0 $\pm$ 89.9	353.0 $\pm$ 2.6	552.4 $\pm$ 26.8
kitchen	343.7 $\pm$ 7.3	446.4 $\pm$ 10.0	340.6 $\pm$ 15.8	837.0 $\pm$ 3.7
pen	28.8 $\pm$ 2.6	18.0 $\pm$ 0.5	13.3 $\pm$ 0.7	32.9 $\pm$ 3.0
relocate	99.0 $\pm$ 0.1	95.8 $\pm$ 0.3	98.3 $\pm$ 0.2	98.1 $\pm$ 0.2

Appendix

Table 4: Offline-to-online regret when fine-tuning with AWR (

\downarrow

lower is better). Entries are mean

\pm

std over seeds; “-” indicates not evaluated. Cell colors are assigned per environment across all four tables : best (gold), second (silver), third (bronze).

	Offline Algorithm
Environment	CalQL/CQL	IQL	SMAC	TD3+BC
door	140.8 $\pm$ 0.1	127.0 $\pm$ 0.3	72.3 $\pm$ 1.8	140.3 $\pm$ 0.2
hopper	2469.9 $\pm$ 42.3	1392.6 $\pm$ 118.4	425.5 $\pm$ 44.3	1295.4 $\pm$ 62.3
kitchen	256.6 $\pm$ 4.0	385.2 $\pm$ 7.6	262.2 $\pm$ 14.6	373.4 $\pm$ 11.5
pen	17.8 $\pm$ 2.9	16.1 $\pm$ 0.6	7.7 $\pm$ 0.6	31.8 $\pm$ 1.0
relocate	99.1 $\pm$ 0.0	95.3 $\pm$ 0.3	84.9 $\pm$ 2.0	98.2 $\pm$ 0.2

Appendix

Table 5: Offline-to-online regret when fine-tuning with TD3+BC (

\downarrow

lower is better). Entries are mean

\pm

std over seeds; “-” indicates not evaluated. Cell colors are assigned per environment across all four tables : best (gold), second (silver), third (bronze).

Figure 13: Fine-tuning with AWR is stable but less efficient . Offline-to-online curves when fine-tuning with AWR. SMAC achieves smooth transfer in 4/6 environments and dominates the baselines in 5/6 environments, but AWR fine-tuning generally produces lower reward curves than the main online fine-tuning algorithms.

Figure 14: Same-objective fine-tuning is stable but slower in Kitchen . We compare same-objective fine-tuning for IQL and CalQL against SAC fine-tuning from the same offline checkpoint families. Continuing the offline objective avoids the sharp initial SAC drop, but SMAC fine-tuned with SAC improves more quickly and reaches stronger final performance.

Figure 15: Same-objective Kitchen fine-tuning produces linearly connected endpoints . We plot reward along the line between the offline checkpoint and the same-objective fine-tuned checkpoint. Both IQL

\to

IQL and CalQL

\to

CalQL avoid a low-reward interpolation valley, matching their stable online learning curves in Figure 14 .

Figure 16: Large high-coverage Door data does not remove transfer instability . We fine-tune actor-critics trained on a Door dataset with over 10 million samples. Despite strong offline performance, the baselines still experience sharp online drops, while SMAC improves immediately after transfer.

Figure 17: Compute matching does not close the Kitchen transfer gap . We compare SMAC to IQL and CalQL checkpoints trained with additional offline gradient steps to roughly match SMAC’s offline wall-clock cost. The compute-matched baselines still transfer substantially worse than SMAC fine-tuned with SAC.

Figure 18: Additional Hopper and Cube transfer results . We evaluate offline-to-online transfer in Hopper-medium and OGBench Cube. SMAC remains stable in both settings, including Cube where the behavior data is generated by preset motions rather than by an actor-critic policy.

Figure 19: SMAC is robust to varying

\kappa

once

\kappa

exceeds a certain threshold . SMAC shows good performance across a wide range of values for

\kappa

, with performance improving as

\kappa

increases before eventually levelling off. Once

\kappa>5

performance begins to look the same between runs.

Figure 20: Optimizing the baselines with Muon yields no offline-to-online improvement

Figure 21: Muon critical for offline-to-online success of SMAC we plot the offline-to-online curves for SMAC when using Muon vs. Adam optimizers. We observe that using Adam leads to a drop in performance during transfer in 3/6 environments whereas using Muon experiences no drop across all environments.

Hyper-parameter	Locomotion	Adroit Tasks	Kitchen
Obs Dimension / Emb Dimension	64 / 64	64 / 64	64 / 64
Hidden Dimension	512	512	512
# of blocks	3	5	5
Condition Network architecture	MLP	MLP	MLP
Condition Network Hidden Dimensions	[512, 512, 512]	[256, 256, 256]	[256, 256, 256]
Condition out dim	64	64	64

Appendix

Table 6: Diffusion-model hyper-parameters used by SMAC (diffusion BC) across task families. Each column corresponds to the diffusion configuration used for that environment family.

Hyper-parameter	Locomotion	Adroit Tasks	Kitchen
Critic Learning rate	$0.0003$	$0.0003$	$0.0003$
Critic hidden dims	[512, 512, 512]	[512, 512, 512, 512]	[512, 512, 512, 512]
Critic activations	tanh	tanh	tanh
Critic ensemble size	10	10	10
Critic Target Update Ratio	0.005	0.005	0.005
Policy Learning rate	$0.0001$	$0.0001$	$0.0001$

Appendix

Table 7: SMAC (non-diffusion) hyper-parameters across task families. Diffusion-related fields are excluded and in section above.

Hyper-parameter	Locomotion	Door (Adroit)	Kitchen
Critic learning rate	$0.0003$	$0.0003$	$0.0003$
Critic hidden dims	[256, 256]	[256, 256, 256]	[512, 512, 512]
Critic activations	relu	relu	relu
Critic ensemble size	2	2	2
Target update rate	$0.005$	$0.005$	$0.005$
Value net learning rate	$0.0003$	$0.0003$	$0.0003$

Appendix

Table 8: IQL hyper-parameters across task families (non-diffusion). Door column uses adroit_iql . Locomotion/Kitchen columns are filled using the IQL default config in get_config and then overridden where specified in the task config.

Hyper-parameter	Locomotion	Adroit	Kitchen
Critic learning rate	$0.0003$	$0.0003$	$0.0003$
Critic hidden dims	[512, 512, 512]	[512, 512, 512, 512]	[512, 512, 512]
Critic activations	relu	relu	relu
Critic ensemble size	10	10	10
Target update rate	$0.005$	$0.005$	$0.005$
Policy tanh-squash dist.	True	True	True

Appendix

Table 9: CQL / Cal-QL hyper-parameters across task families

Hyper-parameter	Locomotion	Adroit	Kitchen
Critic learning rate	$0.0003$	$0.0003$	$0.0003$
Critic hidden dims	[512, 512, 512]	[512, 512]	[512, 512, 512]
Critic activations	relu	relu	relu
Critic ensemble size	$10$	$10$	$10$
Critic target update ratio	$0.005$	$0.005$	$0.005$
Policy learning rate	$0.0001$	$0.0001$	$0.0001$

Appendix

Table 10: TD3+BC hyper-parameters across task.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 298 papers

Aug 27, 2026

Simple Actors and Deep Critics for Scalable Reinforcement Learning

Recent progress in offline reinforcement learning (RL) has been driven by expressive generative actors such as diffusion and flow-matching policies, which capture multimodal behavior in offline datasets. However, these actors require multiple denoising or integration steps per action and thus incur substantial overhead at every decision in deployment. In this work, we revisit where capacity should be invested in an offline actor--critic method. Since the critic is used only during training and is discarded at deployment while the actor runs at every decision step, allocating capacity to the critic rather than the actor is more favorable for inference-time efficiency. However, scaling MLP critics in offline RL is known to introduce several distinct instabilities that have, in practice, kept critics shallow. We identify three distinct failure modes that arise when critics are deepened in offline RL---optimization, bootstrap-noise amplification, and value-range drift---and address each with a corresponding ingredient: a residual MLP backbone, n-step bootstrap targets, and a categorical cross-entropy loss. Combining these ingredients with a lightweight deterministic actor, we propose LAC (Light Actor, deep Critic). On OGBench, LAC matches the strongest diffusion- and flow-matching baselines while achieving up to 4x lower inference latency, comparable to one-step distilled policies without distillation. Its critic recipe also transfers across actor parametrizations.

Guhyeon Kang, Jaehwi Lee, Minhae Kwon

Sungkyunkwan University Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea · Soongsil University Republic of Korea

Offline RL Actor-Critic Methods

Ask PDF

Feb 20, 2026

Flow Actor-Critic for Offline Reinforcement Learning

The dataset distributions in offline reinforcement learning (RL) often exhibit complex and multi-modal distributions, necessitating expressive policies to capture such distributions beyond widely-used Gaussian policies. To handle such complex and multi-modal datasets, in this paper, we propose Flow Actor-Critic, a new actor-critic method for offline RL, based on recent flow policies. The proposed method not only uses the flow model for actor as in previous flow policies but also exploits the expressive flow model for conservative critic acquisition to prevent Q-value explosion in out-of-data regions. To this end, we propose a new form of critic regularizer based on the flow behavior proxy model obtained as a byproduct of flow-based actor design. Leveraging the flow model in this joint way, we achieve new state-of-the-art performance for test datasets of offline RL including the D4RL and recent OGBench benchmarks.

Jongseong Chae, Jongeui Park, Yongjae Shin +3

KAIST · UNIST

Flow-Based Policy Learning Conservative RL Reinforcement Learning Offline RL Actor-Critic Methods

Ask PDF

Aug 2, 2026

ReBRAC-v2: The Return of the King

Recent offline reinforcement learning methods increasingly rely on expressive generative policies and specialized value-guidance mechanisms. We ask whether comparable progress can instead come from systematically modernizing a conventional behavior-regularized actor-critic while preserving its algorithmic simplicity. We introduce ReBRAC-v2, which directly trains an exact-likelihood normalizing flow as the RL actor, combines likelihood, MSE, and MAE behavior regularization, and integrates a classification-based residual critic, staged optimization, and multi-sample test-time action selection. Rather than tuning this recipe separately for every task, we develop a single shared configuration via roughly 600 Bayesian proposals on six challenging OGBench tasks, freeze all structural and optimization choices, and adapt only two behavior-regularization coefficients over a 16-point grid. Across ten common state-based OGBench categories, ReBRAC-v2 averages 74.8 compared to 52.3 for the next-best aggregate result and ranks first in eight categories. The same recipe, without structural changes, obtains the strongest averages in our comparisons on D4RL AntMaze (90.2) and Adroit (33.6). Fixed-recipe ablations show the largest sensitivity to the selected mixed cloning objective, staged training, sufficient flow capacity, and multi-sample inference, while showing that several smaller choices depend on the values of other hyperparameters. These results show that disciplined, transferable engineering can achieve state-of-the-art aggregate performance without abandoning a minimalist offline RL foundation.

Denis Tarasov, Robert K. Katzschmann

Soft Robotics Lab, D-MAVT ETH Zurich, Switzerland

Reinforcement Learning Offline RL Actor-Critic Methods Policy Learning Regularized RL

Ask PDF

Abstract

Offline RL

Explore similar work

Simple Actors and Deep Critics for Scalable Reinforcement Learning

Flow Actor-Critic for Offline Reinforcement Learning

ReBRAC-v2: The Return of the King