SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer

Nathan Samuel de Lara; Florian Shkurti

cs.LGFeb 19, 2026

SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer

Authors: Nathan Samuel de Lara, Florian Shkurti

Organizations: Department of Computer Science, University of Toronto, Toronto, Canada · Vector Institute

Abstract

Modern offline Reinforcement Learning (RL) methods find performant actor-critics, however, fine-tuning these actor-critics online with value-based RL algorithms typically causes immediate drops in performance. We provide evidence consistent with the hypothesis that, in the loss landscape, offline maxima for prior algorithms and online maxima are separated by low-performance valleys that gradient-based fine-tuning traverses. Following this, we present Score Matched Actor-Critic (SMAC), an offline RL method designed to learn actor-critics that transition to online value-based RL algorithms with no drop in performance. SMAC avoids valleys between offline and online maxima by regularizing the Q-function during the offline phase to respect a first-order derivative equality between the score of the policy and action-gradient of the Q-function. We experimentally demonstrate that SMAC converges to offline maxima that are connected to better online maxima via paths with monotonically increasing reward found by first-order optimization. SMAC achieves smooth transfer to Soft Actor-Critic and TD3 in 6/6 D4RL tasks. In 4/6 environments, it reduces regret by 34-58% over the best baseline.

Figures & tables

Figure 1: Past offline RL methods converge to maxima separated from online optima by low-reward valleys . Top: reward landscapes on the Kitchen task for CalQL (left) and SMAC (right). Blue and checkered flags being the real locations of the pre-trained and fine-tuned checkpoints on the landscape respectively. The paths and red/yellow flag are illustrative annotations showing the hypothesized trajectory during transfer. Paths demonstrate the existence of a low reward valley between pre-trained and fine-tuned checkpoints when using CalQL. Our method SMAC has no such valleys and is on a unified hill with the fine-tuning checkpoint. Bottom: SMAC vs. CalQL performance in the Kitchen task. See Section 5 for analysis.

Figure 2: Increasing dataset size and coverage does not bridge offline-to-online gap. We generate rollouts in two environments with a policy that has a 0.7 success rate and plot the offline-to-online performance as we increase the dataset size. We observe that even when the dataset is so large that it is sufficient for learning optimal policies, the actor-critics found are still quickly unlearned by online fine-tuning.

Figure 3: Reward visualized along a plane in parameter space reveals difference in maxima found by different pre-training and fine-tuning methods on Kitchen task . We see that the SAC maxima are wider and not connected to the pre-trained checkpoint along monotonically improving line across all baselines. Conversely, SAC maxima and SMAC maxima are linearly connected. Subplot titles denote offline algorithm used

Figure 4: Reward valleys when linearly interpolating between pre-training and fine-tuning checkpoints for all baselines in tasks show linearly disconnected maxima consistent with offline-to-online transfer performance in later plots . We plot the performance along the line between the pre-trained checkpoint and final fine-tuning checkpoint for methods in kitchen-partial (left), door-binary (centre), and hopper-medium-replay (right).

0

is the pre-trained checkpoint, and

1

is the SAC fine-tuned checkpoint. Lines show mean over 4 seeds with shading being standard error.

Figure 5: t-SNE projections of training trajectories show linearly disconnected maxima . We take the pre-training checkpoints, SAC fine-tuning checkpoints, and TD3+BC fine-tuning checkpoints and plot their T-SNE projections with lines and arrows signifying the training trajectory/ordering of the checkpoints. We observe that the projected checkpoints (i) travel in straight lines, and (ii) cross a valley of low reward when fine-tuned with SAC but not when fine-tuned with TD3+BC, providing evidence consistent with the reward valley hypothesis.

Figure 6: SMAC achieves smooth offline-to-online transfer to SAC . Plot shows offline-to-online transfer results for SMAC and baselines when fine-tuned with SAC. Environments are only named with the first word since that uniquely identifies them among the six. The dotted line shows where the agent transfers from offline learning to online learning. The shaded region shows the final offline training window before transfer; the first 200k offline gradient steps are omitted from the plot.

Figure 7: SMAC achieves smooth offline-to-online transfer to TD3 . Plot shows offline-to-online transfer results for SMAC and baselines when fine-tuned with TD3. The format follows Figure 6 , including the omission of the first 200k offline gradient steps.

Figure 8: Fine-tuning with TD3+BC stabilizes offline-to-online . Plot shows offline-to-online transfer results for SMAC and baselines when fine-tuned with TD3+BC. The format follows Figure 6 , including the omission of the first 200k offline gradient steps.

	Online Algorithm
Offline Algorithm	AWR	SAC	TD3	TD3+BC
IQL	0.508	0.471	0.653	0.494
SMAC	0.380	0.031	0.090	0.226
TD3+BC	0.654	0.962	0.545	0.562
CalQL/CQL	0.482	0.448	0.442	0.614

Table 1: Normalized Regret averaged over all six environments (

\downarrow

lower is better). Values are min-max normalized per environment then averaged. SMAC achieves the lowest regret across all four online algorithms, with particularly strong gains when paired with SAC.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Offline Batch Size has negligible effect on failing baselines, but impacts SMAC

Figure 10: Larger batch sizes lead to faster online adaptation

Figure 11: Removing RvS hurts transfer in 2/3 environments but has mixed impact on online performance. SMAC still dominates baselines without using RvS for training the diffusion model

Figure 12: Q-function and score of policy are proportional throughout training . We plot the Q-function and policy for a SAC agent after 50, 100, 250, and 400 episodes of training in the Reacher-v2 environment.

	Offline Algorithm
Environment	CalQL/CQL	IQL	SMAC	TD3+BC
door	134.5 $\pm$ 1.6	120.0 $\pm$ 0.8	50.3 $\pm$ 2.7	129.7 $\pm$ 2.4
hopper	293.7 $\pm$ 20.4	798.0 $\pm$ 126.5	386.3 $\pm$ 10.8	3213.3 $\pm$ 16.3
kitchen	467.1 $\pm$ 38.1	492.9 $\pm$ 13.7	131.4 $\pm$ 13.0	762.5 $\pm$ 11.4
pen	8.0 $\pm$ 0.7	10.3 $\pm$ 0.7	5.3 $\pm$ 0.7	55.2 $\pm$ 1.9
relocate	98.1 $\pm$ 0.4	97.4 $\pm$ 0.4	62.8 $\pm$ 2.1	99.2 $\pm$ 0.0

Appendix

Table 2: Offline-to-online regret when fine-tuning with SAC (

\downarrow

lower is better). Entries are mean

\pm

std over seeds; “-” indicates not evaluated. Cell colors are assigned per environment across all four tables : best (gold), second (silver), third (bronze).

	Offline Algorithm
Environment	CalQL/CQL	IQL	SMAC	TD3+BC
door	131.1 $\pm$ 3.0	126.1 $\pm$ 0.4	65.5 $\pm$ 2.1	137.7 $\pm$ 1.4
hopper	311.4 $\pm$ 46.5	1834.3 $\pm$ 150.3	297.3 $\pm$ 6.5	329.3 $\pm$ 39.0
kitchen	526.6 $\pm$ 32.3	445.9 $\pm$ 13.2	259.3 $\pm$ 22.8	529.1 $\pm$ 21.6
pen	8.4 $\pm$ 0.7	16.1 $\pm$ 0.5	8.5 $\pm$ 0.8	41.6 $\pm$ 2.9
relocate	97.8 $\pm$ 0.5	95.2 $\pm$ 0.3	58.3 $\pm$ 1.3	99.1 $\pm$ 0.0

Appendix

Table 3: Offline-to-online regret when fine-tuning with TD3 (

\downarrow

lower is better). Entries are mean

\pm

std over seeds; “-” indicates not evaluated. Cell colors are assigned per environment across all four tables : best (gold), second (silver), third (bronze).

	Offline Algorithm
Environment	CalQL/CQL	IQL	SMAC	TD3+BC
door	129.9 $\pm$ 0.7	127.9 $\pm$ 0.4	122.8 $\pm$ 0.8	136.0 $\pm$ 0.7
hopper	533.7 $\pm$ 43.0	958.0 $\pm$ 89.9	353.0 $\pm$ 2.6	552.4 $\pm$ 26.8
kitchen	343.7 $\pm$ 7.3	446.4 $\pm$ 10.0	340.6 $\pm$ 15.8	837.0 $\pm$ 3.7
pen	28.8 $\pm$ 2.6	18.0 $\pm$ 0.5	13.3 $\pm$ 0.7	32.9 $\pm$ 3.0
relocate	99.0 $\pm$ 0.1	95.8 $\pm$ 0.3	98.3 $\pm$ 0.2	98.1 $\pm$ 0.2

Appendix

Table 4: Offline-to-online regret when fine-tuning with AWR (

\downarrow

lower is better). Entries are mean

\pm

std over seeds; “-” indicates not evaluated. Cell colors are assigned per environment across all four tables : best (gold), second (silver), third (bronze).

	Offline Algorithm
Environment	CalQL/CQL	IQL	SMAC	TD3+BC
door	140.8 $\pm$ 0.1	127.0 $\pm$ 0.3	72.3 $\pm$ 1.8	140.3 $\pm$ 0.2
hopper	2469.9 $\pm$ 42.3	1392.6 $\pm$ 118.4	425.5 $\pm$ 44.3	1295.4 $\pm$ 62.3
kitchen	256.6 $\pm$ 4.0	385.2 $\pm$ 7.6	262.2 $\pm$ 14.6	373.4 $\pm$ 11.5
pen	17.8 $\pm$ 2.9	16.1 $\pm$ 0.6	7.7 $\pm$ 0.6	31.8 $\pm$ 1.0
relocate	99.1 $\pm$ 0.0	95.3 $\pm$ 0.3	84.9 $\pm$ 2.0	98.2 $\pm$ 0.2

Appendix

Table 5: Offline-to-online regret when fine-tuning with TD3+BC (

\downarrow

lower is better). Entries are mean

\pm

std over seeds; “-” indicates not evaluated. Cell colors are assigned per environment across all four tables : best (gold), second (silver), third (bronze).

Figure 13: Fine-tuning with AWR is stable but less efficient . Offline-to-online curves when fine-tuning with AWR. SMAC achieves smooth transfer in 4/6 environments and dominates the baselines in 5/6 environments, but AWR fine-tuning generally produces lower reward curves than the main online fine-tuning algorithms.

Figure 14: Same-objective fine-tuning is stable but slower in Kitchen . We compare same-objective fine-tuning for IQL and CalQL against SAC fine-tuning from the same offline checkpoint families. Continuing the offline objective avoids the sharp initial SAC drop, but SMAC fine-tuned with SAC improves more quickly and reaches stronger final performance.

Figure 15: Same-objective Kitchen fine-tuning produces linearly connected endpoints . We plot reward along the line between the offline checkpoint and the same-objective fine-tuned checkpoint. Both IQL

\to

IQL and CalQL

\to

CalQL avoid a low-reward interpolation valley, matching their stable online learning curves in Figure 14 .

Figure 16: Large high-coverage Door data does not remove transfer instability . We fine-tune actor-critics trained on a Door dataset with over 10 million samples. Despite strong offline performance, the baselines still experience sharp online drops, while SMAC improves immediately after transfer.

Figure 17: Compute matching does not close the Kitchen transfer gap . We compare SMAC to IQL and CalQL checkpoints trained with additional offline gradient steps to roughly match SMAC’s offline wall-clock cost. The compute-matched baselines still transfer substantially worse than SMAC fine-tuned with SAC.

Figure 18: Additional Hopper and Cube transfer results . We evaluate offline-to-online transfer in Hopper-medium and OGBench Cube. SMAC remains stable in both settings, including Cube where the behavior data is generated by preset motions rather than by an actor-critic policy.

Figure 19: SMAC is robust to varying

\kappa

once

\kappa

exceeds a certain threshold . SMAC shows good performance across a wide range of values for

\kappa

, with performance improving as

\kappa

increases before eventually levelling off. Once

\kappa>5

performance begins to look the same between runs.

Figure 20: Optimizing the baselines with Muon yields no offline-to-online improvement

Figure 21: Muon critical for offline-to-online success of SMAC we plot the offline-to-online curves for SMAC when using Muon vs. Adam optimizers. We observe that using Adam leads to a drop in performance during transfer in 3/6 environments whereas using Muon experiences no drop across all environments.

Hyper-parameter	Locomotion	Adroit Tasks	Kitchen
Obs Dimension / Emb Dimension	64 / 64	64 / 64	64 / 64
Hidden Dimension	512	512	512
# of blocks	3	5	5
Condition Network architecture	MLP	MLP	MLP
Condition Network Hidden Dimensions	[512, 512, 512]	[256, 256, 256]	[256, 256, 256]
Condition out dim	64	64	64

Appendix

Table 6: Diffusion-model hyper-parameters used by SMAC (diffusion BC) across task families. Each column corresponds to the diffusion configuration used for that environment family.

Hyper-parameter	Locomotion	Adroit Tasks	Kitchen
Critic Learning rate	$0.0003$	$0.0003$	$0.0003$
Critic hidden dims	[512, 512, 512]	[512, 512, 512, 512]	[512, 512, 512, 512]
Critic activations	tanh	tanh	tanh
Critic ensemble size	10	10	10
Critic Target Update Ratio	0.005	0.005	0.005
Policy Learning rate	$0.0001$	$0.0001$	$0.0001$

Appendix

Table 7: SMAC (non-diffusion) hyper-parameters across task families. Diffusion-related fields are excluded and in section above.

Hyper-parameter	Locomotion	Door (Adroit)	Kitchen
Critic learning rate	$0.0003$	$0.0003$	$0.0003$
Critic hidden dims	[256, 256]	[256, 256, 256]	[512, 512, 512]
Critic activations	relu	relu	relu
Critic ensemble size	2	2	2
Target update rate	$0.005$	$0.005$	$0.005$
Value net learning rate	$0.0003$	$0.0003$	$0.0003$

Appendix

Table 8: IQL hyper-parameters across task families (non-diffusion). Door column uses adroit_iql . Locomotion/Kitchen columns are filled using the IQL default config in get_config and then overridden where specified in the task config.

Hyper-parameter	Locomotion	Adroit	Kitchen
Critic learning rate	$0.0003$	$0.0003$	$0.0003$
Critic hidden dims	[512, 512, 512]	[512, 512, 512, 512]	[512, 512, 512]
Critic activations	relu	relu	relu
Critic ensemble size	10	10	10
Target update rate	$0.005$	$0.005$	$0.005$
Policy tanh-squash dist.	True	True	True

Appendix

Table 9: CQL / Cal-QL hyper-parameters across task families

Hyper-parameter	Locomotion	Adroit	Kitchen
Critic learning rate	$0.0003$	$0.0003$	$0.0003$
Critic hidden dims	[512, 512, 512]	[512, 512]	[512, 512, 512]
Critic activations	relu	relu	relu
Critic ensemble size	$10$	$10$	$10$
Critic target update ratio	$0.005$	$0.005$	$0.005$
Policy learning rate	$0.0001$	$0.0001$	$0.0001$

Appendix

Table 10: TD3+BC hyper-parameters across task.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 298 papers

Simple Actors and Deep Critics for Scalable Reinforcement Learning
Aug 27, 2026Guhyeon Kang, Jaehwi Lee, Minhae KwonOffline RL Actor-Critic Methods
Sungkyunkwan University Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea · Soongsil University Republic of Korea
Flow Actor-Critic for Offline Reinforcement Learning
Feb 20, 2026Jongseong Chae, Jongeui Park, Yongjae Shin +3Flow-Based Policy Learning Conservative RL
KAIST · UNIST
ReBRAC-v2: The Return of the King
Aug 2, 2026Denis Tarasov, Robert K. KatzschmannReinforcement Learning Offline RL
Soft Robotics Lab, D-MAVT ETH Zurich, Switzerland

Aug 27, 2026J/K move · Enter open · S save

Simple Actors and Deep Critics for Scalable Reinforcement Learning

Guhyeon Kang, Jaehwi Lee, Minhae Kwon

Sungkyunkwan University Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea · Soongsil University Republic of Korea

Offline RL Actor-Critic Methods

Open paper AskPDF arXiv

Recent progress in offline reinforcement learning (RL) has been driven by expressive generative actors such as diffusion and flow-matching policies, which capture multimodal behavior in offline datasets. However, these actors require multiple denoising or integration steps per action and thus incur substantial overhead at every decision in deployment. In this work, we revisit where capacity should be invested in an offline actor--critic method. Since the critic is used only during training and is discarded at deployment while the actor runs at every decision step, allocating capacity to the critic rather than the actor is more favorable for inference-time efficiency. However, scaling MLP critics in offline RL is known to introduce several distinct instabilities that have, in practice, kept critics shallow. We identify three distinct failure modes that arise when critics are deepened in offline RL---optimization, bootstrap-noise amplification, and value-range drift---and address each with a corresponding ingredient: a residual MLP backbone, n-step bootstrap targets, and a categorical cross-entropy loss. Combining these ingredients with a lightweight deterministic actor, we propose LAC (Light Actor, deep Critic). On OGBench, LAC matches the strongest diffusion- and flow-matching baselines while achieving up to 4x lower inference latency, comparable to one-step distilled policies without distillation. Its critic recipe also transfers across actor parametrizations.

Abstract

Offline RL

Explore similar work

Simple Actors and Deep Critics for Scalable Reinforcement Learning

Flow Actor-Critic for Offline Reinforcement Learning

ReBRAC-v2: The Return of the King