Q($γ_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning

Arwa Albelaihi; Abdulrahman Altahhan

cs.LGOct 15, 2026

Q( $γ_H$ ): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning

Authors: Arwa Albelaihi, Abdulrahman Altahhan

Organizations: School of Computer Science, University of Leeds, Leeds LS2 9JT, UK

Abstract

A fixed discount factor in Q-learning imposes a single planning horizon throughout training, despite the differing demands of early exploration and long-term credit assignment. We propose Q( $γ_H$ ), a multi-horizon Q-learning framework that maintains multiple discount-specific heads and adaptively combines them at control time using an entropy-derived confidence signal. The resulting state-dependent mixing weights shift control from shorter- to longer-horizon heads as policy confidence increases, enabling adaptive horizon mixing without modifying individual Bellman updates. Across classical control and deep reinforcement-learning benchmarks, Q( $γ_H$ ) shows its strongest improvements on delayed-credit tasks relative to strong fixed-discount baselines. By decoupling horizon adaptation from the underlying learning updates, Q( $γ_H$ ) provides a lightweight framework for adaptive planning-horizon control in reinforcement learning, avoiding the need for manual discount-factor selection while remaining compatible with standard value-learning methods.

Figures & tables

Method	Multi-	State-	Gate	Bellman	Control
	head	dep.	signal	changed	mix
Q( $\gamma_{H}$ ) (ours)	✓	✓	Entropy	No	✓
Ogawa et al. [ 13 ]	$\times$	$\times$	Critic loss	Yes	$\times$
Moghimi et al. [ 11 ]	✓	$\times$	None	No	$\times$
Nafi et al. [ 12 ]	✓	$\times$	None	No	$\times$
Chen et al. [ 2 ]	✓	$\times$	None	No	$\times$

Table 1: Comparison of Q(

\gamma_{H}

) with closely related methods. State-dep. = state-dependent mixing or adaptation. Bellman changed = whether Bellman targets are modified. Control mix = whether discount-specific heads are mixed at action selection.

Environment	Type	$\|\mathcal{A}\|$	Repr.	Key challenge
Tabular / linear
MountainCar	Continuous	3	TC (IHT 2048)	Delayed credit
Acrobot	Continuous	3	TC (IHT 4096)	Delayed credit
LunarLander	Continuous	4	TC (IHT 8192)	Dense reward
Gridworld	Navigation	4	Tabular	Deterministic
WindyGridworld	Navigation	8	Tabular	Stochastic wind

Table 2: Experimental environments used in evaluations. TC denotes tile coding, IHT denotes index hash table, and FC denotes a fully connected neural network.

Shared (tabular/linear & deep)
Parameter	Symbol	Value	Notes
Initial exploration	$\varepsilon_{0}$	$0.10$	$\varepsilon$ -greedy
Exploration decay	$\delta_{\varepsilon}$	$0.985$	Per-episode
Min. exploration	$\varepsilon_{\min}$	$0.05$
Tabular/linear only
Discount set	$\Gamma$	$\{0.70,0.80,0.90,0.97,0.99\}$	$K=5$ heads

Table 3: Default hyperparameters used in the experiments.

Delayed-credit environments
Environment	Method	$\gamma^{*}$	Early	Tail	AUC
MountainCar	Q( $\gamma_{H}$ )	—	366.36	136.45	203.58
	Q-Learning	0.97	402.42	136.16	217.12
	SARSA	0.97	388.81	143.95	216.51
	Exp. SARSA	0.97	401.04	139.79	218.13
	Double Q	0.97	580.43	146.54	280.95

Table 4: Q(

\gamma_{H}

) vs. oracle-tuned fixed-

\gamma

baselines. Values are means over 10 seeds. Steps

\downarrow

except LunarLander return

\uparrow

Figure 1: Mechanism analysis on MountainCar using the same 10 canonical training seeds. Left: normalised entropy decreases while $\bar{\gamma}$ increases; shaded bands denote 95% confidence intervals across seeds. Right: normalised entropy versus episode length across all seed–episode observations ( $N=2{,}000$ , Pearson $r=0.306$ ).

Method	CartPole-v1 $\uparrow$		Acrobot-v1 $\uparrow$		MountainCar-v0 $\uparrow$
Method	AUC	Gain	AUC	Gain	AUC	Gain
Q( $\gamma$ H)-DDQN	40,476	—	$-$ 34,660	—	$-$ 55,185	—
Q( $\gamma$ H)-DQN	38,804	$+4.3\%$	$-$ 40,242	$+13.9\%$	$-$ 58,312	$+5.4\%^{*}$
DDQN	38,084	$+6.3\%$	$-$ 37,900	$+8.5\%$	$-$ 59,314	$+7.0\%^{**}$
DQN	26,851	$+50.7\%$	$-$ 35,634	$+2.7\%$	$-$ 59,112	$+6.6\%^{**}$
Hyperbolic [ 4 ]	34,616	$+16.9\%$	$-$ 38,169	$+9.2\%^{**}$	$-$ 58,025	$+4.9\%^{*}$

Table 5: Deep Q(

\gamma

H)-DDQN vs. all baselines (10 seeds, 300 episodes). AUC

\uparrow

higher is better. Gain is computed as

100\times(\mathrm{AUC}_{Q(\gamma H)}-\mathrm{AUC}_{\mathrm{base}})/|\mathrm{AUC}_{\mathrm{base}}|

; positive values indicate that Q(

\gamma

H)-DDQN performs better.

{}^{*}p{<}0.05

{}^{**}p{<}0.01

{}^{***}p{<}0.001

(paired

t

-test, 10 seeds). Bold = best per column.

Figure 2: Learning curves on Acrobot-v1 (10 seeds, 300 episodes, shaded $=95\%$ CI, reward $\uparrow$ ). Q( $\gamma$ H)-DDQN (solid blue) converges faster and achieves the best cumulative reward ( $-34{,}660$ ). It outperforms Scheduled- $\gamma$ by $48.0\%$ ( $p{<}0.001$ ), Ogawa-Adapt by $21.4\%$ ( $p{<}0.01$ ), and DDQN by $8.5\%$ . The annotation marks the gain over Scheduled- $\gamma$ relative to the episode-wise peak gap. Since Acrobot assigns $-1$ per timestep, values closer to zero indicate faster task completion.

Signal dynamics (first 50 $\to$ last 50 episodes)
	MountainCar	Acrobot
Mean head disagreement	$15.53\to 27.77$	$1.03\to 8.15$
Normalised entropy $H_{n}$	$0.960\to 0.800$	$0.969\to 0.878$
Discount summary $\bar{\gamma}$	$0.782\to 0.816$	$0.780\to 0.800$
Pearson $r$ (episode-wise, 200 episodes)
Disagreement $\to H_{n}$	$-0.883$	$-0.969$

Table 6: Signal dynamics and correlations on delayed-credit tasks (10 seeds). Pearson

r

is computed episode-wise over 200 episodes.

Method	Type	Early $\downarrow$	Tail $\downarrow$	AUC $\downarrow$	$\bar{\gamma}$	$H_{n}$
MaxMin linear	Aggregated	366.36	136.45	203.58	0.816	0.800
Mean Q	Aggregated	373.1	137.7	206.4	0.802	0.863
Max Q optimistic	Aggregated	370.7	138.9	206.8	0.789	0.926
Fixed middle	Fixed head	379.1	139.9	208.3	0.798	0.882
Fixed short	Fixed head	375.4	139.2	208.5	0.789	0.924
Min Q pessimistic	Aggregated	379.1	140.7	209.0	0.842	0.670

Table 7: Reference construction comparison on MountainCar (10 seeds, steps

\downarrow

). Final

\bar{\gamma}

and

H_{n}

are means over the last 50 episodes.

Signal	Early $\downarrow$	Tail $\downarrow$	AUC $\downarrow$	$\bar{\gamma}$ first 50	$\bar{\gamma}$ last 50
Entropy	366.36	136.45	203.58	0.782	0.816
Variance	372.79	138.70	205.25	0.795	0.784
Disagreement	375.35	138.47	206.41	0.796	0.787

Table 8: Gating signal ablation on MountainCar (10 seeds, MaxMin linear reference, steps

\downarrow

). Best value per column in bold .

Seeds	Config	Early $\downarrow$	Tail $\downarrow$	AUC $\downarrow$	$\bar{\gamma}$ first/last 50
10	K=3	367.20	139.39	204.16	0.755 / 0.790
10	K=5	366.36	136.45	203.58	0.782 / 0.816
20	K=3	367.36	139.94	204.62	0.755 / 0.790
20	K=5	370.36	140.60	205.60	0.780 / 0.811

Table 9: Head count ablation on MountainCar (steps

\downarrow

), 10 and 20 seeds. Best value per column in bold .

Param	Value	Early $\downarrow$	Tail $\downarrow$	Default	Robust range	Sensitivity
$\beta$	3.0	366.36	136.45	3.0	$[1.5,\ 3.0]$	Moderate
	1.5	388.69	140.0
	0.5	403.03	141.9
$\tau$	0.25	366.36	136.45	0.25	$[0.25,\ 0.50]$	Low
	0.50	368.12	137.8
	0.10	369.92	140.1

Table 10: Sensitivity analysis on MountainCar (10 seeds, steps

\downarrow

). Robust ranges are determined from the corresponding AUC results. Best value per block in bold .

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,705 papers

Adaptive Multi-Horizon Reinforcement Learning
Jul 22, 2026Manoosh Samiei, Doina Precup, Paul MassetContinual RL
Department of Computer Science, McGill University, Canada · 2Mila - Quebec Artificial Intelligence Institute · Department of Psychology, McGill University, Canada
Gated Q-learning: Add Off-Policy Bias to Taste
Jul 31, 2026Brett DaleyCredit Assignment in RL Step-Level Credit Assignment in RL
Meta
Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
May 7, 2026Armaan A. Abraham, Lucy Xiaoyang Shi, Chelsea FinnQ-Learning Temporal-Difference Learning
Stanford University

Jul 22, 2026J/K move · Enter open · S save

Adaptive Multi-Horizon Reinforcement Learning

Manoosh Samiei, Doina Precup, Paul Masset

Department of Computer Science, McGill University, Canada · 2Mila - Quebec Artificial Intelligence Institute · Department of Psychology, McGill University, Canada

Continual RL

Open paper AskPDF arXiv

Effective decision-making in complex and changing environments requires balancing short-term and long-term consequences. In reinforcement learning (RL), this trade-off is typically controlled through a fixed discount factor, which imposes a single exponentially discounted temporal horizon. However, biological agents exhibit flexible and adaptive temporal discounting, suggesting that effective planning requires multiple timescales. Here, we propose a multi-horizon approach that adaptively selects and combines temporal horizons, enabling robust adaptation to changes in reward structure without manual discount-factor tuning. This flexibility makes the method particularly suitable for continual learning scenarios involving task switches and varying environmental configurations. Empirically, we demonstrate that our approach identifies effective discount factors across a range of MiniGrid environments, including continual settings composed of three sequentially changing tasks. These results suggest that adaptive temporal discounting can improve parameter efficiency and enhance adaptability in both artificial and biologically inspired learning systems.

Abstract

Reinforcement Learning

Explore similar work

Adaptive Multi-Horizon Reinforcement Learning

Gated Q-learning: Add Off-Policy Bias to Taste

Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities