Generative Value Classifier

Alexander Swerdlow; Perry Dong; Chelsea Finn

cs.LGOct 15, 2026

Generative Value Classifier

Authors: Alexander Swerdlow, Perry Dong, Chelsea Finn

Organizations: Stanford University

Abstract

Q-learning is premised on learning a function to predict the value of a state-action pair, with the goal of using such a function to guide policy improvement. Most approaches learn this function from scratch using scalar or distributional return targets. This discriminative approach leaves another source of supervision for learning representations unused: the actions themselves. Rather than directly predict the return of a state-action pair, we ask the reverse question: how likely is the action under each possible return? This perspective provides a critic whose generative component is a return-conditioned policy, which can share the actor's architecture. Training this policy supervises the critic with every action rather than with returns alone, and a squared-error loss anchors the critic's Q-value to the bootstrapped target. We evaluate our approach on both offline and online RL tasks and show gains over leading discriminative methods, including distributional RL methods.

Figures & tables

Figure 2: Generative Value Classifier at inference time. The actor proposes candidate actions (left). For each action, we compute the return-conditioned action likelihood, along with the return prior. Normalizing their product over return bins (center) provides a return distribution whose mean gives

Q(s,a)

, which we use to rank the action candidates (right).

Figure 3: Online RL on RoboMimic. Success rate versus environment steps (in millions) on Lift, Can, Square, and Tool Hang, from left to right. We compare GVC with QC, RLPD, IDQL, FQL, QAM, and QSM. Shaded regions show one standard deviation across seeds.

Gaussian Policies

Flow Policies

Environment

Task

IQL

ReBRAC

FBRAC

IFQL

FQL

C51

IQN

CODAC

Value Flows

BFN

GVC

scene-sparse

task1

19\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 6

}}

0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 0

}}

9\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 8

}}

\mathbf{100}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 1

}}

\mathbf{96}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 3

}}

65\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 16

}}

19\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 6

}}

\mathbf{99}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 2

}}

\mathbf{100}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 1

}}

\mathbf{100}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 1

}}

\mathbf{100}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 1

}}

\mathbf{99}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 1

}}

task2

1\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 1

}}

0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 0

}}

66\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 19

}}

86\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 10

}}

83\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 5

}}

52\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 12

}}

2\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 2

}}

\mathbf{98}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 2

}}

\mathbf{98}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 3

}}

\mathbf{99}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 1

}}

\mathbf{100}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 1

}}

\mathbf{99}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 2

}}

task3

1\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 1

}}

0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 0

}}

0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 0

}}

\mathbf{94}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 8

}}

56\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 8

}}

60\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 2

}}

2\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 2

}}

93\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 6

}}

72\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 38

}}

89\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 5

}}

\mathbf{94}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 6

}}

\mathbf{99}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 2

}}

task4

2\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 2

}}

0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 0

}}

0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 0

}}

77\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 34

}}

72\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 12

}}

69\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 16

}}

4\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 4

}}

82\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 15

}}

76\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 30

}}

92\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 10

}}

90\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 4

}}

\mathbf{98}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 3

}}

task5

0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 0

}}

0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 0

}}

0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 0

}}

75\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 19

}}

52\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 8

}}

19\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 5

}}

1\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 2

}}

57\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 30

}}

66\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 20

}}

91\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 7

}}

45\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 14

}}

\mathbf{97}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}

\pm 3

}}

Table 1: Offline evaluation on OGBench. Success rate (%) per task. We bold values within

95\%

of the best performance for each task.

Task	IQN	IFQL	FQL	RLPD	IQL	Value Flows	BFN	GVC
antmaze-large	$55\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 5 $}}$ $\rightarrow$ $91\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 1 $}}$	$11\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 3 $}}$ $\rightarrow$ $3\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 2 $}}$	$\mathbf{87}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 7 $}}$ $\rightarrow$ $\mathbf{100}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 1 $}}$	$0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$ $\rightarrow$ $\mathbf{99}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 2 $}}$	$41\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 5 $}}$ $\rightarrow$ $45\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 5 $}}$	$\mathbf{72}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 4 $}}$ $\rightarrow$ $\mathbf{96}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 1 $}}$	$\mathbf{66}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 19 $}}$ $\rightarrow$ $\mathbf{99}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 1 $}}$	$\mathbf{77}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 9 $}}$ $\rightarrow$ $\mathbf{97}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 2 $}}$
humanoidmaze-medium	$23\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 3 $}}$ $\rightarrow$ $14\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 2 $}}$	$56\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 12 $}}$ $\rightarrow$ $82\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 7 $}}$	$12\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 7 $}}$ $\rightarrow$ $22\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 12 $}}$	$0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$ $\rightarrow$ $8\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 3 $}}$	$21\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 5 $}}$ $\rightarrow$ $16\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 3 $}}$	$29\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 6 $}}$ $\rightarrow$ $86\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 3 $}}$	$32\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 8 $}}$ $\rightarrow$ $89\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 4 $}}$	$\mathbf{94}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 2 $}}$ $\rightarrow$ $\mathbf{99}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 1 $}}$
cube-double	$29\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 4 $}}$ $\rightarrow$ $42\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 7 $}}$	$12\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 3 $}}$ $\rightarrow$ $41\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 2 $}}$	$40\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 11 $}}$ $\rightarrow$ $92\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 3 $}}$	$0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$ $\rightarrow$ $0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$	$0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$ $\rightarrow$ $0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$	$65\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 7 $}}$ $\rightarrow$ $79\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 6 $}}$	$79\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 7 $}}$ $\rightarrow$ $90\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 7 $}}$	$\mathbf{64}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 21 $}}$ $\rightarrow$ $\mathbf{98}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 1 $}}$
cube-triple	$25\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 5 $}}$ $\rightarrow$ $14\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 6 $}}$	$2\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 1 $}}$ $\rightarrow$ $7\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 5 $}}$	$4\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 1 $}}$ $\rightarrow$ $83\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 12 $}}$	$0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$ $\rightarrow$ $0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$	$3\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 1 $}}$ $\rightarrow$ $0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$	$29\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 8 $}}$ $\rightarrow$ $70\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 7 $}}$	$21\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 10 $}}$ $\rightarrow$ $90\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 17 $}}$	$\mathbf{46}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 21 $}}$ $\rightarrow$ $\mathbf{100}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$
puzzle-4x4	$22\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 2 $}}$ $\rightarrow$ $6\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 1 $}}$	$23\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 2 $}}$ $\rightarrow$ $19\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 12 $}}$	$8\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 3 $}}$ $\rightarrow$ $38\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 52 $}}$	$\mathbf{0}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$ $\rightarrow$ $\mathbf{100}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$	$5\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 1 $}}$ $\rightarrow$ $0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$	$14\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 3 $}}$ $\rightarrow$ $51\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 12 $}}$	$24\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 6 $}}$ $\rightarrow$ $68\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 39 $}}$	$\mathbf{25}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 8 $}}$ $\rightarrow$ $\mathbf{100}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 1 $}}$
scene	$0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$ $\rightarrow$ $0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$	$0\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$ $\rightarrow$ $60\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 14 $}}$	$\mathbf{82}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 11 $}}$ $\rightarrow$ $\mathbf{100}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 1 $}}$	$\mathbf{0}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$ $\rightarrow$ $\mathbf{100}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$	$15\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 4 $}}$ $\rightarrow$ $10\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 3 $}}$	$\mathbf{92}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 23 $}}$ $\rightarrow$ $\mathbf{100}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$	$\mathbf{51}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 16 $}}$ $\rightarrow$ $\mathbf{97}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 4 $}}$	$\mathbf{96}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 9 $}}$ $\rightarrow$ $\mathbf{100}\,\text{\scalebox{.80}{\color[rgb]{0.35,0.35,0.35}$ \pm 0 $}}$

Table 2: Offline-to-online evaluations on OGBench tasks. Success rate (%) on 6 OGBench tasks after offline training

\rightarrow

after online finetuning. Mean

\pm

standard deviation over 8 seeds, with values within

95\%

of the best final performance for each task in bold.

Figure 5: Ablations on Tool Hang. Success rate (%) vs. environment steps. (a) Action loss: varying flow loss coefficients,

\lambda_{F}

. (b) Flow loss: GVC where the flow loss is weighted by the target return distribution

\widehat{p}_{i}

(blue), uniform weighting across return bins (coral), and the flow loss removed (teal). (c) Scoring: different

T

values, with and without shared noise. Shaded regions show one standard deviation across seeds.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Ablations on Square. Success rate (%) versus environment steps. (a) Action loss: different flow-loss coefficients

\lambda_{F}

, compared with the default. (b) Flow loss: GVC with the default flow loss weighted by the target return distribution

\widehat{p}_{i}

(blue), uniform weighting across return bins (coral), and the flow loss removed (teal). (c) Scoring: different

T

values, with and without shared noise. Shaded regions show one standard deviation across seeds.

Figure 7: Value-anchor ablation on Square, Tool Hang, and puzzle-4x4 . Success rate versus environment steps, or offline updates on puzzle-4x4 ( task4 ), with the anchor coefficient

\lambda_{V}

scaled to

10

0.1

0.01

, and

0

times its default (

1\times

, Table 4 ),

8

seeds per setting.

Figure 8: Best-of-

N

selection versus a distilled one-step actor. Success rate of GVC with its default best-of-

N

extraction (solid) and with a one-step actor (shortcut model) trained by direct Q-gradients under GVC , for a range of distillation coefficients

\alpha

(dashed). Left and center: online RL on Square and Tool Hang. Right: offline RL on puzzle-4x4 , averaged over its five tasks.

\alpha=0

removes the behavior anchor from the one-step actor.

Figure 9: Offline RL training step time. Wall-clock time per gradient update on puzzle-4x4 on a single H100, mean and standard deviation over

3

seeds.

Hyperparameter	Value
Optimizer	Adam
Learning rate	$3\times 10^{-4}$
Batch size	$256$
Target network update rate	$0.005$
Actor $f_{\phi}$	MLP, $4$ hidden layers of width $512$ , no layer normalization
Action model $g_{\theta}$ trunk	MLP, $3$ hidden layers of width $512$ , layer normalization, GELU

Appendix

Table 3: Common GVC hyperparameters. Values shared by all experiments. Domain-specific values are given in Table 4 .

Domain	$\gamma$	$K$	$\beta_{\mathrm{ret}}$	$\lambda_{V}$	$N$	Critics
RoboMimic, online ( $h=5$ )
lift , can , square (MH)	$0.99$	$5$	$40$	$0.15$	$8$	$2$ , min
tool-hang (PH)	$0.99$	$5$	$40$	$0.05$	$8$	$2$ , min
OGBench, offline ( $h=1$ )
cube-double-play	$0.995$	$21$	$40$	$0.2$	$8$	$2$ , mean
cube-triple-play	$0.995$	$21$	$10$	$0.2$	$8$	$2$ , mean

Appendix

Table 4: GVC hyperparameters.

K

is the number of return bins;

N

is the number of candidates for best-of-

N

selection; “Critics” gives the ensemble size and aggregation.

Method	Hyperparameter	Search values
IQL	AWR inverse temperature	$\{0.3,1,3,10\}$
ReBRAC	Actor BC coefficient	$\{0.003,0.01,0.03,0.1,0.3,1\}$
	Critic BC coefficient	$\{0,0.001,0.01,0.1\}$
IFQL	Action candidates	$\{32,64,128\}$
FBRAC	BC coefficient	$\{1,3,10,30,100,300\}$
FQL	BC distillation coefficient	$\{3,10,30,100,300,1000\}$

Appendix

Table 5: Hyperparameter search spaces for tuning our baselines, following Value Flows [ 14 ] . BC denotes behavioral cloning; BCFM denotes bootstrapped conditional flow matching.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,704 papers

Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning
Apr 12, 2026Zikang Shan, Han Zhong, Liwei Wang +1RL for Language Models Reinforcement Learning
Peking University · Microsoft Research Asia · Shanghai Jiao Tong University
Per-Domain Generalizing Policies: On Learning Efficient and Robust Q-Value Functions (Extended Version with Technical Appendix)
Mar 18, 2026Nicola J. Müller, Moritz Oster, Isabel Valera +2Generalization in RL Q-Learning
German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany · Saarland University, Saarland Informatics Campus, Saarbrücken, Germany · Center for European Research in Trusted Artificial Intelligence (CERTAIN)
Decoupling Policy Extraction for Offline Reinforcement Learning
Aug 21, 2026Xuyao Lin, Yixiang Shan, Jinru Duan +7Offline RL Policy Learning
Simplexity Robotics, Rensselaer Polytechnic Institute · Simplexity Robotics · Simplexity Robotics, Northeastern University

Apr 12, 2026J/K move · Enter open · S save

Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning

Zikang Shan, Han Zhong, Liwei Wang, Li Zhao

Peking University · Microsoft Research Asia · Shanghai Jiao Tong University

RL for Language Models Reinforcement Learning Value Function Estimation Credit Assignment in RL Actor-Critic Methods

Open paper AskPDF arXiv

Credit assignment is a central challenge in reinforcement learning (RL). Classical actor-critic methods address this challenge through fine-grained advantage estimation based on a learned value function. However, learned value models are often avoided in modern large language model (LLM) RL because conventional discriminative critics are difficult to train reliably. We revisit value modeling and argue that this difficulty is partly due to limited expressiveness. In particular, representation complexity theory suggests that value functions can be hard to approximate under the one-shot prediction paradigm used by existing value models, and our scaling experiments show that such critics do not improve reliably with scale. Motivated by this observation, we propose Generative Actor-Critic (GenAC), which replaces one-shot scalar value prediction with a generative critic that performs chain-of-thought reasoning before producing a value estimate. We further introduce In-Context Conditioning, which helps the critic remain calibrated to the current actor throughout training. GenAC improves value approximation, ranking reliability, and out-of-distribution generalization, and these gains translate into stronger downstream RL performance than both value-based and value-free baselines. Overall, our results suggest that stronger value modeling is a promising direction for improving credit assignment in LLM reinforcement learning.

Abstract

Reinforcement Learning

Explore similar work

Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning

Per-Domain Generalizing Policies: On Learning Efficient and Robust Q-Value Functions (Extended Version with Technical Appendix)

Decoupling Policy Extraction for Offline Reinforcement Learning