Selective Growing Q-Networks (SGQN): An Analysis of Adaptive Action Grids in Decoupled Q-Learning

Parham Nooranbakht; Seyed Mahdi Basiri Azad; Joschka Bödecker

cs.LGOct 14, 2026

Selective Growing Q-Networks (SGQN): An Analysis of Adaptive Action Grids in Decoupled Q-Learning

Authors: Parham Nooranbakht, Seyed Mahdi Basiri Azad, Joschka Bödecker

Organizations: Faculty of Engineering, Department of Computer Science Albert-Ludwigs-Universität Freiburg, Germany

Abstract

Value-based Q-learning for continuous control can discretise the action space, but the resolution of the discretisation trades early exploration against final precision. Growing Q-Networks (GQN) grow the resolution uniformly from coarse to fine. We introduce Selective Growing Q-Networks (SGQN), which keeps decoupled Q-learning and replaces uniform growth with an adaptive action grid: bins are added next to bins with high estimated Q-value, bins with low Q-value are pruned, and actions are chosen epsilon-greedily. On eight DeepMind Control Suite (DMControl) tasks with 2 to 12 action dimensions, SGQN has the highest mean final return among the compared methods on four, is within 1.1% of the best method on three more, and is 3.4% below Bang-Bang MPO on Walker Run, using on average 7.5 of 9 available bins per dimension. We then test whether the adaptive grid explains these results, and find that it does not. Growth activates every candidate bin and behaves like uniform growth, and removing pruning changes the return by at most 1.1%. Two other choices matter: starting from a coarse grid, since starting with all 9 bins produces collapsed seeds, and epsilon-greedy selection, since an annealed softmax lowers the return by 23% to 100%. SGQN is insensitive to its growth and pruning hyperparameters. It also differs from the GQN and DecQN baselines in other respects, notably one gradient update per environment step instead of one per eight, which we did not control for, so we do not attribute its advantage to a single component. We present SGQN as a competitive value-based method and as an analysis of which ingredients of a growing action grid matter.

Figures & tables

Component	SGQN	GQN	DecQN	CQN
Observations	State	State	State	Pixels in our runs (84 $\times$ 84, 3 frames); the paper also reports state-based tasks
Two averaged critics	Yes	Yes (C); double Q-learning (P)	Yes (C, P)	Not stated
$n$ -step, $\alpha$	3, 0.6	3, 0.6	3, 0.6 (C, P)	3 (1 on Walker tasks)
Replay exponent $\beta$	0.4, constant	0.2, constant	0.2 (C, P)	Not applicable
Network	Layer-norm MLP trunk, 2 $\times$ 512	Residual layer-norm network, 512	Same as GQN (C); 500 hidden units (P)	Pixel encoder
Updates per environment step	1	1/8 (C); not stated (P)	1/8 (C); not stated (P)	1 per 2 agent steps

Table 1: Components of SGQN and of the Q-learning baselines. C: our implementation, P: the original paper. “Not stated” means the paper does not specify it. Bang-Bang PPO and MPO are policy-gradient methods and are omitted.

Panel of Figure 1 — Figure 1: Learning curves on eight DeepMind Control Suite tasks (mean $\pm$ standard deviation over the seeds that produced data, 7 for most entries, see Table 2 ; 1M environment steps).

Method	Cheetah Run	Fish Swim	Hopper Hop	Quadruped Walk	Reacher Easy	Reacher Hard	Walker Run	Walker Walk
SGQN (Ours)	788 $\pm$ 37	715 $\pm$ 17	257 $\pm$ 11	924 $\pm$ 10	962 $\pm$ 12	951 $\pm$ 16	713 $\pm$ 13	957 $\pm$ 3
SGQN without pruning (old code)	741 $\pm$ 42	662 $\pm$ 51	63 $\pm$ 76	901 $\pm$ 15 †	967 $\pm$ 6	924 $\pm$ 67	710 $\pm$ 22	941 $\pm$ 5
GQN	603 $\pm$ 14	561 $\pm$ 19	174 $\pm$ 55	739 $\pm$ 208	966 $\pm$ 4	953 $\pm$ 4	585 $\pm$ 11	948 $\pm$ 2
DecQN	172 $\pm$ 18	80 $\pm$ 4	1 $\pm$ 0	168 $\pm$ 30	182 $\pm$ 89	9 $\pm$ 2	119 $\pm$ 8	389 $\pm$ 93
CQN	179 $\pm$ 10 †	79 $\pm$ 7	10 $\pm$ 4	179 $\pm$ 23 †	168 $\pm$ 61	11 $\pm$ 7	144 $\pm$ 18	413 $\pm$ 49
Bang-Bang PPO	515 $\pm$ 97 †	193 $\pm$ 123	27 $\pm$ 31	396 $\pm$ 197	754 $\pm$ 46 †	–	258 $\pm$ 49	548 $\pm$ 186

Table 2: Mean training episode return over the final 100 episodes (mean

\pm

standard deviation over seeds). Bold : highest mean per task over all rows. † : 6 instead of 7 seeds, because the remaining run produced no data (Bang-Bang PPO: Cheetah Run and Reacher Easy; Bang-Bang MPO: Cheetah Run; CQN: Cheetah Run and Quadruped Walk; earlier SGQN version: Quadruped Walk). –: none of the seven runs produced data. CQN learns from pixels, all other methods from the state.

Figure 2: Active action bins of the first action dimension over training, for one seed per task. Rows are the nine bins (action value increasing upwards), and cell colour is the share of the selections of that dimension that went to the bin in each interval. White cells are bins that have not yet been grown into the grid, gray cells are bins that were pruned, and triangles mark bins that were added (upward) or removed (downward) at that checkpoint.

Figure 3: Return (top of each block, mean

\pm

standard deviation over seeds, window of 20 episodes) and number of active bins per dimension (bottom of each block, mean over seeds and dimensions) over training for the eight tasks. The step profile of the bin count is the same on every task.

Task	Action dim.	Active bins	Alignment	Q spread	Return
Cheetah Run	6	7.5 / 9	1.000	0.8%	788
Fish Swim	5	7.5 / 9	1.000	1.0%	715
Hopper Hop	4	7.5 / 9	1.000	2.9%	257
Quadruped Walk	12	7.5 / 9	1.001	2.6%	924
Reacher Easy	2	7.5 / 9	1.000	0.5%	962
Reacher Hard	2	7.5 / 9	1.000	0.6%	951

Table 3: Action-space resolution of SGQN at the end of training. Active bins is the number of active bins per dimension averaged over the last 300 episodes, which is one full growth interval (the count rises when growth fires and falls at each pruning event, so the last checkpoint alone depends on where in that cycle training stopped). The grid has

N=9

bins per dimension. Alignment is the mean Q of the active bins divided by the mean Q of the full grid at the last checkpoint, and Q spread is the range of Q over the nine bins of a dimension as a share of its mean (median over dimensions). Return is the mean over the final 100 training episodes. All values are means over 7 seeds.

Figure 4: Which bins are pruned. (a) Share of all pruning events that removed each bin position of the grid, with the share on each individual task as dots. Anchor bins are protected and are never removed. (b) Distribution over the fifths of the Q-ranking (lowest fifth on the left) of the bins that were removed, with the share expected under random removal, computed per event from the number of candidates, as ticks.

Variant	Fish Swim	Hopper Hop	Quadruped Walk	Walker Run
Full method	718 (13)	254 (12)	927 (10)	713 (15)
No pruning	726 (12)	255 (12)	928 (8)	710 (16)
Uniform instead of value-guided growth	718 (13)	254 (12)	926 (11)	713 (15)
Start with all 9 bins (no growth)	722 (14)	252 (32)	818 (132)	684 (49)
Annealed softmax instead of $\varepsilon$ -greedy	508 (19)	0 (0)	250 (44)	547 (21)

Table 4: Component ablation. Each row changes one design choice and leaves the rest of the full method unchanged. Values are the mean return over the final 100 training episodes, with the standard deviation across seeds in parentheses. All variants use the same 5 seeds.

Figure 5: Component ablation on four tasks with 5 seeds each. Gray dots are single seeds, dark markers are means with one standard deviation, and the shaded band is the range of the full method over the same seeds. The percentage at the right of each row is the mean relative to the full method. Uniform growth and no pruning stay inside the band, starting with all 9 bins produces individual seeds far below it, and annealed softmax is far below it on every task.

Hyperparameter	Value	Hopper Hop	Walker Run
prune threshold	0.25	257 (15)	710 (7)
	0.5 $\ast$	256 (15)	712 (8)
	0.75	257 (13)	713 (11)
growth threshold	0.7	256 (15)	712 (8)
	0.85 $\ast$	256 (15)	712 (8)
	0.95	254 (14)	712 (8)

Table 5: Hyperparameter sensitivity. Each hyperparameter is varied alone with the rest at their defaults, which are marked

\ast

. Values are the mean return over the final 100 training episodes, with the standard deviation across seeds in parentheses.

Figure 6: Hyperparameter sensitivity on Hopper Hop and Walker Run with 3 seeds. Each panel varies one hyperparameter with the others at their defaults. The blue marker is the default, open markers are other values, gray dots are single seeds and bars are one standard deviation. The values on the horizontal axis are evenly spaced in increasing order.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Full grid bins $N$	9
Initial bins $n_{0}$	3 (boundary and centre bins, which are protected anchors)
Growth interval $\Delta_{g}$	300 episodes
Pruning interval $\Delta_{p}$	50 episodes
Growth threshold $\tau_{Q}$	0.85 (fraction of the best running mean Q)
Pruning threshold $\tau_{p}$	0.5 (quantile of the mature bins)

Appendix

Table 6: SGQN-specific hyperparameters. All values are passed to the training script as explicit command line options (initial bins, pruning interval, growth interval, minimum age, maximum lives, pruning threshold, growth threshold, regrow cooldown, selection strategy and

\varepsilon

decay); the remaining options keep the defaults of our code.

Hyperparameter	Value
Optimiser	Adam
Learning rate	$1\times 10^{-4}$
Discount $\gamma$	0.99
Replay buffer size	$10^{6}$
Batch size	256
$n$ -step returns	3

Appendix

Table 7: Network and optimisation hyperparameters of SGQN. The baselines use the settings stated in their original publications and the defaults of our code where a publication does not state a value (Table 1 ).

Task	Action Dim.	State Dim.
Reacher Easy	2	7
Reacher Hard	2	7
Hopper Hop	4	15
Fish Swim	5	26
Cheetah Run	6	17
Walker Walk	6	24

Appendix

Table 8: DeepMind Control Suite task details.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,705 papers

Accelerating Q-learning through Efficient Value-Sharing across Actions
Jun 29, 2026Prabhat Nagarajan, Brett Daley, Martha White +1Deep Q-Learning Value Function Estimation
Department of Computing Science, University of Alberta, Edmonton, AB, Canada. · Alberta Machine Intelligence Institute · Meta (independent work, not on Meta’s behalf). +1
Revisiting Overestimation Bias Problem of Q-learning: Settling Large Discrete Action Space via Action Intersection
Aug 13, 2026Pu Li, Tao Tan, Hong Xie +2Reinforcement Learning Deep Q-Learning
Chongqing Institute of Green and Intelligent Technology, Chongqing, China · University of Science and Technology of China, Hefei, Anhui, China
Revisiting TD Target Aggregation under Uncertainty in Q-Learning
Aug 4, 2026Lipeng Zu, Xiaonan ZhangDeep Q-Learning Value Function Estimation
Computer Science Department, Florida State University, Tallahassee, Florida, USA

Jun 29, 2026J/K move · Enter open · S save

Accelerating Q-learning through Efficient Value-Sharing across Actions

Prabhat Nagarajan, Brett Daley, Martha White, Marlos C. Machado

Department of Computing Science, University of Alberta, Edmonton, AB, Canada. · Alberta Machine Intelligence Institute · Meta (independent work, not on Meta’s behalf).+1

Deep Q-Learning Value Function Estimation Q-Learning

Open paper AskPDF arXiv

Action values are foundational to many control algorithms such as Q-learning. Therefore, efficient action-value learning is central to reinforcement learning (RL). However, learning them can be slow, requiring many updates to move values from their initialization, typically near zero, to their true values, which may be far from zero. Moreover, action-value learning algorithms typically update each state-action pair independently, without learning a value that is common to all actions within a state. In this paper, we address these inefficiencies by introducing the mean-expansion layer, which accelerates action-value learning by sharing values across actions within a state and by changing the problem from directly learning potentially large action-values to learning a lower-norm representation of them. In deep RL, this layer can be applied as a parameter-free addition to Q-network architectures without altering the underlying algorithm. Applied to deep Q-networks and implicit quantile networks, it improves aggregate performance across 57 Atari 2600 games while increasing action gaps and dramatically reducing value overestimation.

Abstract

Reinforcement Learning

Explore similar work

Accelerating Q-learning through Efficient Value-Sharing across Actions

Revisiting Overestimation Bias Problem of Q-learning: Settling Large Discrete Action Space via Action Intersection

Revisiting TD Target Aggregation under Uncertainty in Q-Learning