Simple Actors and Deep Critics for Scalable Reinforcement Learning
Authors: Guhyeon Kang, Jaehwi Lee, Minhae Kwon
Organizations: Sungkyunkwan University Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea · Soongsil University Republic of Korea
Recent progress in offline reinforcement learning (RL) has been driven by expressive generative actors such as diffusion and flow-matching policies, which capture multimodal behavior in offline datasets. However, these actors require multiple denoising or integration steps per action and thus incur substantial overhead at every decision in deployment. In this work, we revisit where capacity should be invested in an offline actor--critic method. Since the critic is used only during training and is discarded at deployment while the actor runs at every decision step, allocating capacity to the critic rather than the actor is more favorable for inference-time efficiency. However, scaling MLP critics in offline RL is known to introduce several distinct instabilities that have, in practice, kept critics shallow. We identify three distinct failure modes that arise when critics are deepened in offline RL---optimization, bootstrap-noise amplification, and value-range drift---and address each with a corresponding ingredient: a residual MLP backbone, n-step bootstrap targets, and a categorical cross-entropy loss. Combining these ingredients with a lightweight deterministic actor, we propose LAC (Light Actor, deep Critic). On OGBench, LAC matches the strongest diffusion- and flow-matching baselines while achieving up to 4x lower inference latency, comparable to one-step distilled policies without distillation. Its critic recipe also transfers across actor parametrizations.
Figures & tables
Figure 1. Performance vs. inference latency on humanoidmaze-medium . Iterative generative actors ( IDQL , IFQL , FBRAC , FAWAC ) require multiple denoising or flow-integration steps per action. 2 2 footnotemark: 2 LAC (star) acts in a single forward pass, reaching higher success rate at lower latency.
Gaussian / Det.
Diffusion
Flow
Ours
Environment
BC
TD3+BC
IQL
ReBRAC
IDQL
SRPO
CAC
FAWAC
FBRAC
IFQL
FQL
LAC-S
LAC-L
ant-large
11 ± 1
82 ± 6
53 ± 3
81 ± 5
21 ± 5
11 ± 4
33 ± 4
6 ± 1
60 ± 6
28 ± 5
79 ± 3
92 ± 5
96 ± 1
ant-giant
0 ± 0
3 ± 4
4 ± 1
26 ± 8
0 ± 0
0 ± 0
0 ± 0
0 ± 0
4 ± 4
3 ± 2
9 ± 6
35 ± 23
79 ± 13
hum-medium
2 ± 1
12 ± 9
33 ± 2
22 ± 8
1 ± 0
1 ± 1
53 ± 8
19 ± 1
38 ± 5
60 ± 14
58 ± 5
79 ± 10
76 ± 38
hum-large
1 ± 0
8 ± 7
2 ± 1
2 ± 1
1 ± 0
0 ± 0
0 ± 0
0 ± 0
2 ± 0
11 ± 2
4 ± 2
50 ± 18
69 ± 15
scene
5 ± 1
0 ± 0
28 ± 1
41 ± 3
46 ± 3
20 ± 1
40 ± 7
30 ± 3
45 ± 5
30 ± 3
56 ± 2
29 ± 37
41 ± 39
Table 1. Success rates on 7 OGBench environments ( 5 task variants averaged). The best result in each row is shown in bold , and the second-best is underlined .
Figure 2. Critic depth scaling on hum-large . LAC scales stably with depth while baselines plateau or fail to learn.
Figure 3. Success rate over critic depth ×n -step horizon on hum-large , with residual backbone and categorical loss fixed. All horizons fail at shallow depth; beyond depth 16 , only n=1 degrades with further depth, while n≥2 remains stable and n=4 is consistently strongest.
Figure 4. Training curves on hum-large with a 32 -layer residual critic and n=4 , varying only the critic loss. The MSE-regression critic collapses mid-training; the categorical critic of LAC remains stable. Shaded: ±1 std over 4 seeds.
Figure 5. Actor scaling on hum-large (top) and puzzle-4x4 (bottom) with the deep critic fixed. Performance saturates at modest actor sizes; LAC-S and LAC-L bracket the plateau.
Configuration
H-large
A-giant
P-4x4
Avg
LAC ( n -step + categorical)
79±6
86±3
55±6
73
−n -step (use 1 -step TD)
42±3
38±15
18±2
33
− Categorical (use MSE)
7±12
72±12
32±15
37
− Both ( 1 -step TD + MSE)
15±5
1±1
12±3
9
Table 2. Component ablation on the task1 variant of each environment. Each row uses a 32 -layer residual critic with different combinations of n -step bootstrap and categorical loss.
Figure 6. Spatial value landscape on hum-large . For each spatial location (x,y) , we average the predicted Q -value over actions sampled at that location. The shallow 4 -layer critic (left) yields a blurred landscape, whereas the 32 -layer LAC critic (right) produces a sharper gradient aligned with the distance to the goal (star).
Method
Actor (inference)
Latency (ms)
Speedup
IQL
Gaussian, 1 step
0.30
1.25×
ReBRAC
Deterministic, 1 step
0.27
1.13×
IDQL
Diffusion, 10 steps
0.95
3.96×
SRPO
Diffusion, 1 step
0.28
1.17×
CAC
Distilled Diffusion, 1 step
0.31
1.29×
FBRAC
Flow, 10 steps
0.55
2.29×
Table 3. Per-action inference latency on a single NVIDIA RTX 3090, batch size 1 . Speedup is relative to LAC-S.
Method
Baseline
+ LAC critic
Gain
TD3+BC (Deterministic)
10
46
+36
FAWAC (flow-matching)
6
30
+24
FQL (Distilled Flow)
19
57
+38
IDQL (Diffusion)
1
31
+30
CAC (Distilled Diffusion)
38
97
+59
Table 4. Replacing the critic of various offline RL methods with the LAC critic, while keeping their actor parametrization and policy-extraction objective intact. Success rates ( % ) on the task1 variant of hum-medium .
Hyperparameter
Value
Critic (ResMLP with categorical head)
Embedding dimension
256
Number of residual blocks
8
Sub-layers per block
4
Total dense layers
32
Activation
ReLU ( Nair and Hinton, 2010 )
Table 5. LAC hyperparameters used across all OGBench environments.
Gaussian / Det.
Diffusion
Flow
Ours
Env
Task
BC
TD3+BC
IQL
ReBRAC
IDQL
SRPO
CAC
FAWAC
FBRAC
IFQL
FQL
L-S
L-L
ant-l
1
0 ± 0
86 ± 2
48 ± 9
91 ± 10
0 ± 0
0 ± 0
42 ± 7
1 ± 1
70 ± 20
24 ± 17
80 ± 8
92 ± 4
96 ± 0
2
6 ± 3
77 ± 3
42 ± 6
88 ± 4
14 ± 8
4 ± 4
1 ± 1
0 ± 1
35 ± 12
8 ± 3
57 ± 10
83 ± 3
94 ± 2
3
29 ± 5
92 ± 0
72 ± 7
51 ± 18
26 ± 8
3 ± 2
49 ± 10
12 ± 4
83 ± 15
52 ± 17
93 ± 3
97 ± 2
98 ± 2
4
8 ± 3
79 ± 3
51 ± 9
84 ± 7
62 ± 25
45 ± 19
17 ± 6
10 ± 3
37 ± 18
18 ± 8
80 ± 4
91 ± 3
95 ± 0
5
10 ± 3
78 ± 3
54 ± 22
90 ± 2
2 ± 2
1 ± 1
55 ± 6
9 ± 5
76 ± 8
38 ± 18
83 ± 4
95 ± 1
97 ± 1
Table 6. Full OGBench success rates ( % ) on all 35 tasks. The best result in each row is shown in bold , and the second-best is underlined .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 7. The 7 OGBench environments used in our experiments.
The dataset distributions in offline reinforcement learning (RL) often exhibit complex and multi-modal distributions, necessitating expressive policies to capture such distributions beyond widely-used Gaussian policies. To handle such complex and multi-modal datasets, in this paper, we propose Flow Actor-Critic, a new actor-critic method for offline RL, based on recent flow policies. The proposed method not only uses the flow model for actor as in previous flow policies but also exploits the expressive flow model for conservative critic acquisition to prevent Q-value explosion in out-of-data regions. To this end, we propose a new form of critic regularizer based on the flow behavior proxy model obtained as a byproduct of flow-based actor design. Leveraging the flow model in this joint way, we achieve new state-of-the-art performance for test datasets of offline RL including the D4RL and recent OGBench benchmarks.
Scaling critic capacity is a promising direction for improving off-policy reinforcement learning (RL). However, recent work shows that larger critics are prone to overfitting and instability in replay-based bootstrapped training. In this paper, we propose using Low-Rank Adaptation (LoRA) as a structural regularizer for critic learning. Our approach freezes randomly initialized base matrices and optimizes only the corresponding low-rank adapters, thereby constraining critic updates to a low-dimensional subspace. We evaluate our method across different off-policy RL algorithms, including SAC and FastTD3 based on different network architectures. Empirically, LoRA efficiently reduces critic loss during training and improves overall policy performance, achieving the best or competitive results on most tasks. Extensive experiments demonstrate that our low-rank updates provide a simple and effective form of structural regularization for critic learning in off-policy RL.
Yuan Zhuang, Yuexin Bian, Sihong He +7
University of Connecticut · University of California San Diego · University of Texas at Arlington +1
One-step offline RL actors are attractive because they avoid backpropagating through long iterative samplers and keep inference cheap, but they still have to improve under a critic without drifting away from actions that the dataset can support. In recent one-step extraction pipelines, a strong iterative teacher provides one target action for each latent draw, and the same student output is asked to do both jobs: move toward higher Q and stay near that paired endpoint. If those two directions disagree, the loss resolves them as a compromise on that same sample, even when a nearby better action remains locally supported by the data. We propose DROL, a latent-conditioned one-step actor trained with top-1 dynamic routing. For each state, the actor samples K candidate actions from a bounded latent prior, assigns each dataset action to its nearest candidate, and updates only that winner with Behavior Cloning and critic guidance. Because the routing is recomputed from the current candidate geometry, ownership of a supported region can shift across candidates over the course of learning. This gives a one-step actor room to make local improvements that pointwise extraction struggles to capture, while retaining single-pass inference at test time. On OGBench and D4RL, DROL is competitive with the one-step FQL baseline, improving many OGBench task groups while remaining strong on both AntMaze and Adroit. Project page: https://muzhancun.github.io/preprints/DROL.
Zhancun Mu, Guangyu Zhao, Yiwu Zhong +1
School of Intelligence Science and Technology, Peking University