Short-to-Long: Detached Features for Long-Horizon Value Learning

Tong Che; Boris Ivanovic; Marco Pavone

cs.LGOct 21, 2026

Short-to-Long: Detached Features for Long-Horizon Value Learning

Authors: Tong Che, Boris Ivanovic, Marco Pavone

Organizations: NVIDIA Research

Abstract

Learning useful representations from continually changing value targets remains a challenge in deep reinforcement learning. We study the temporal horizon of this supervision: smaller discounts make these targets less sensitive to distant bootstrap changes, but using them for behavior changes the control objective. We introduce Short-to-Long (S2L), which supplies a long-horizon behavior learner with features trained by smaller-discount source networks. Detached connections prevent the behavior value loss from updating the sources, allowing each source to follow its own value objective while the behavior discount remains unchanged. Across Atari, online continuous control, and offline continuous control, the approach yields aggregate performance gains, improving the paired terminal median on 40 of 57 Atari games. Matched Atari ablations support the roles of feature transfer, smaller source discounts, and gradient isolation. Fixed-trajectory measurements further show reduced drift in source value targets with mixed discounts.

Figures & tables

Panel of Figure 1 — Figure 1: S2L-4: learn separately, transfer features from short to long. All four learners train on the same transitions. At each residual depth, a learner receives its predecessor’s activation through a detached addition. Each value loss updates only its own network. The final $\gamma=.99$ learner supplies action values or the actor objective.

Figure 2: S2L-4 improves the paired median on 40 of 57 Atari games. Bars show median paired terminal HNS differences from scaled PQN, ordered from the smallest to largest effect across the two columns. Blue, gray, and orange indicate values above

+.01

, within

\pm.01

, and below

-.01

. The dashed line marks the median across games. Effects outside

[-.30,+1.20]

are annotated with their actual values.

Figure 3: The S2L-4 return advantage grows during training. Both panels use the same 720 matched pairs. Left: mean return over the second half of training. Right: mean paired difference over the full run, overall and by algorithm. The shaded window marks the final quarter of training, over which the mean paired difference is

+34.99

Subset	$n$	Final return		$\Delta$ AUC
		Mean $\Delta$	W	0–1M	500k–1M	750k–1M
All	720	39.09	465	7.16	27.03	34.99
DDPG	360	47.42	242	17.29	40.66	46.66
SAC	360	30.76	223	$-2.97$	13.40	23.31
AdamW	360	47.49	252	16.84	36.61	42.79
Kron	360	30.68	213	$-2.52$	17.45	27.18

Table 1: S2L gains over 720 matched pairs. Entries combine all 720 pairs. Differences are S2L-4 minus SimBa; AUCs are divided by their window length. W counts positive endpoint pairs.

Metric	S2L-4	SimBa	Paired W/L/T
Local fluctuation (detrended MAD)	12.08	16.17	404/316/0
Normalized maximum drawdown	0.137	0.164	426/276/18
Cumulative negative change	38.26	45.32	412/290/18
Worst 50k-step drop	23.85	26.62	415/287/18

Table 2: Late-training return dynamics over 720 completed matched pairs. Statistics use eleven evaluations from 500k to 1M steps. Lower detrended MAD means less local fluctuation; lower values in the other rows mean smaller downward return changes. The last column counts pairs with a lower S2L-4 value / lower SimBa value / an exact tie.

Target drift	Mean $[95\%\ \mathrm{CI}]$	Median	Pairs $<0$	Games $<0$
Normalized (primary)	$-9.689\;[-17.638,-3.389]$	$-2.073$	$64/64$	$16/16$
Raw RMS (secondary)	$-.170\;[-.249,-.101]$	$-.127$	$64/64$	$16/16$

Table 3: Fixed-trajectory Q(

\lambda

)-target drift. Entries are mixed-discount minus all-

.99

differences in time-averaged consecutive Q(

\lambda

)-target drift, so negative values favor mixed discounts. The normalized measure divides each learner and probe stratum by its fixed iteration-1 target standard deviation. Intervals use 10,000 whole-game bootstrap resamples, retaining all four paired seeds.

(a) Original reference scores
Method	Endpoint HNS	Full AUC	Second-half AUC
Scaled PQN	$0.9323$	$0.3161$	$0.5809$
S2L-4	$1.0648$	$0.3999$	$0.6827$
(b) Variant minus full S2L-4
Variant	$\Delta$ endpoint	$\Delta$ full AUC	$\Delta$ second-half AUC
No gradient isolation	$-.0971$	$-.0731$	$-.1530$

Table 4: S2L-4 reference scores and ablations. Panel (a) gives the original 57-game, four-seed mean HNS scores. Panel (b) gives changes relative to the corresponding full S2L-4 model. Endpoint HNS uses iteration 610; AUCs are divided by their window length. Intervals for the original paired controls appear in Table 11 .

Figure 4: S2L-4 versus the shared-trunk MDH-4 baseline. Left: IQM HNS over 228 game–seed runs per method. Right: IQM of paired HNS differences, S2L-4 minus MDH-4. Shading gives 95% game-cluster bootstrap intervals.

Summary	Pair wins	Game wins	IQM $\Delta$	95% CI
Terminal HNS	147/228	35/57	+.0323	$[+.0135,\,+.0667]$
Full-training AUC	149/228	39/57	+.0188	$[+.0089,\,+.0343]$
Late-half AUC	153/228	39/57	+.0250	$[+.0124,\,+.0489]$
Terminal last-10	152/228	38/57	+.0329	$[+.0146,\,+.0726]$

Table 5: Shared-head control on ALE-57.

\Delta

is S2L-4 minus the matched-discount MDH-4 baseline in human-normalized score units; positive values favor S2L-4. Intervals are 95% game-cluster bootstrap intervals for the paired-cell IQM (10,000 resamples preserving four seeds per sampled game). Wins are listed as seed pairs and game medians. Terminal HNS has 5 seed-pair ties and 1 game-median tie; Terminal last-10 has 3 seed-pair ties.

Environment	Simple	Medium	Expert	Mean
HalfCheetah	$-1.63$ (0/5)	$+7.32$ (5/5)	$+17.78$ (5/5)	$+7.82$ (10/15)
Hopper	$+3.39$ (5/5)	$+2.61$ (3/5)	$-0.34$ (2/5)	$+1.88$ (10/15)
Walker2d	$-0.39$ (1/5)	$+0.82$ (4/5)	$+1.00$ (3/5)	$+0.47$ (8/15)
All	$+0.45$ (6/15)	$+3.58$ (12/15)	$+6.14$ (10/15)	$+3.39$ (28/45)

Table 6: Offline continuous control. Entries are S2L minus its matched TD3+BC control in normalized-score AUC; parentheses give positive paired seeds. All nine Minari MuJoCo conditions use five paired seeds and 200k updates.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Algorithm	PQN with Q( $\lambda$ )
Main / short discount	.99 / .90
$\lambda$	.65
CNN	Atari, medium: 32–64–64 channels
MLP	width 2560, five two-linear residual blocks
Stabilizers	full MultiSkip, LayerNorm, Kron

Appendix

Table 7: Base configuration for two-learner Atari experiments.

	Qbert (4 seeds)		Phoenix (4 seeds)		Pooled (8)
Transferred feature	GM	Wins	GM	Wins	GM	Wins
None (baseline)	1.000	–	1.000	–	1.000	–
Short input	1.173	4/4	1.173	4/4	1.173	8/8
Aligned absolute	1.239	4/4	1.237	4/4	1.238	8/8
Aligned delta	1.188	4/4	1.199	4/4	1.194	8/8
Dense all-layer mean	1.277	4/4	1.240	4/4	1.258	8/8

Appendix

Table 8: Paired training-return AUC relative to the scaled PQN base agent of Castanyer et al. (2025) . GM is the geometric mean across the listed paired seeds. Every method uses the same

\gamma=.99

target construction and action-selection rule.

Figure 5: Feature-transfer comparison. Bars show paired geometric-mean training-return AUC relative to the

\gamma=.99

base agent on Qbert and Phoenix. Every two-learner variant has the same parameter count, target construction, optimizer configuration, and action-selection rule; only the transferred feature changes.

Transferred feature	Qbert GM	Phoenix GM	Pooled GM	Positive pairs
Short input	1.000	1.000	1.000	–
Aligned absolute	1.0565	1.0546	1.0556	6/8
Aligned delta	1.0129	1.0221	1.0175	4/8
Dense all-layer mean	1.0887	1.0566	1.0725	6/8
Prefix mean	1.0420	1.0773	1.0595	7/8

Appendix

Table 9: Feature-transfer effects relative to the detached short-input variant. Parameter counts, optimizer settings, target construction, and action-selection rules match.

Game	Selected $\gamma_{s}$	Effective horizon	Median SNI
Qbert	.875	8	.256
Phoenix	.900	10	.154
Asterix	.875	8	.106
Breakout	.984375	64	.064
Seaquest	.96875	32	.341

Appendix

Table 10: Per-game source-discount-factor choices from the five-game sweep. Median SNI compares the selected source with the scaled PQN parent of Castanyer et al. (2025) over four paired seeds.

Figure 6: Source-discount selection. Left: four-seed median paired SNI against the $\gamma=.99$ base agent across the eligible grid; outlined cells are in-sample per-game maxima and $H=8$ is selected globally on the same panel. Right: seed-level paired SNI for that selected source against the matched $\gamma_{s}=.99$ duplicate source. SNI is defined in Appendix B .

Contrast	Endpoint HNS	Full AUC	Second-half AUC
Transfer $-$ no transfer	$+.101\;[-.121,+.282]$	$+.080\;[+.005,+.165]$	$+.091\;[-.024,+.207]$
Mixed discounts $-$ all $.99$	$+.060\;[-.101,+.202]$	$+.065\;[+.023,+.112]$	$+.086\;[+.028,+.153]$
Detached $-$ backpropagated	$+.114\;[+.006,+.256]$	$+.082\;[+.019,+.174]$	$+.122\;[+.025,+.275]$

Appendix

Table 11: Matched Atari ablations. Entries are mean paired human-normalized-score differences. Brackets give 95% intervals from 10,000 bootstrap resamples of games, retaining all four paired seeds. The first two rows cover all 57 games; the last covers the 52 games not used for development. Positive values favor the first method.

Summary	S2L-4	MDH-4	MDH-4 (wide)
Parameters (approx.)	321.4M	80.4M	321.4M
Endpoint HNS	$1.0648\,\pm\,2.2205$	$1.0181\,\pm\,2.7107$	$1.0286\,\pm\,2.7339$
Full-training AUC	$0.3997\,\pm\,0.9289$	$0.3627\,\pm\,0.8405$	$0.3511\,\pm\,0.9083$
Late-half AUC	$0.6827\,\pm\,1.3488$	$0.6400\,\pm\,1.4027$	$0.6174\,\pm\,1.4369$
Terminal last-10	$1.0278\,\pm\,2.1008$	$0.9896\,\pm\,2.5857$	$0.9319\,\pm\,2.6126$

Appendix

Table 12: Parameter-matched shared-trunk control on ALE-57. Entries are arithmetic means

\pm

standard deviations across 57 games and four seeds, in HNS units. Wide MDH-4 matches the total parameter count of S2L-4. Standard deviations describe score dispersion across game–seed cells.

Setting	Value
Tasks	dog-run, dog-trot, humanoid-run, humanoid-walk
Algorithms / optimizers	SAC, DDPG / AdamW, Kron
Environment steps / action repeat	1,000,000 / 2
Train environments / updates per interaction	1 / 2
Replay capacity / warm-up / batch	1M / 5,000 / 256
Evaluation	10 episodes every 50,000 environment steps

Appendix

Table 13: Shared settings for the S2L-4–SimBa DMC matrix.

Game	Seed	A	B	E	F
Qbert	1	4.194	4.888	5.754	5.532
	2	3.973	5.363	5.330	4.760
	3	4.722	4.950	5.468	5.138
	4	4.365	5.003	5.441	5.658
Phoenix	1	6.956	8.190	8.086	8.430
	2	6.972	8.464	8.619	8.576

Appendix

Table 14: Training-return AUC (

10^{9}

) for baseline (A), simple short input (B), dense mean (E), and prefix mean (F).

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,705 papers

Target-Aligned Reinforcement Learning
Mar 31, 2026Leonard S. Pleiss, James Harrison, Maximilian SchifferReinforcement Learning RL Control
Google DeepMind
Horizon Adaptive Offline Policy Learning via Value Stitching
Jun 19, 2026Kexin Zheng, Xianyuan Zhan, Xintao YanValue Function Estimation Offline RL
The University of Hong Kong · Tsinghua University
Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
May 7, 2026Armaan A. Abraham, Lucy Xiaoyang Shi, Chelsea FinnQ-Learning Temporal-Difference Learning
Stanford University

Mar 31, 2026J/K move · Enter open · S save

Target-Aligned Reinforcement Learning

Leonard S. Pleiss, James Harrison, Maximilian Schiffer

Google DeepMind

Reinforcement Learning RL Control RL Stability

Open paper AskPDF arXiv

Many value-based deep reinforcement learning algorithms rely on target networks - lagged copies of the online network - to stabilize training. While effective, this mechanism introduces a fundamental stability-recency tradeoff: slower target updates improve stability but reduce the recency of learning signals, hindering convergence speed. We propose Target-Aligned Reinforcement Learning (TARL), a simple drop-in refinement for existing algorithms that emphasizes transitions for which the target and online network estimates are highly aligned. By focusing updates on well-aligned targets, TARL mitigates the adverse effects of stale target estimates while retaining the stabilizing benefits of target networks. We empirically demonstrate consistent improvements within discrete and continuous control algorithms across various benchmark environments without any hyperparameter tuning, including a 38.18% peak score gain on Atari-10, while incurring less than a 4% increase in wall-clock time.

Abstract

Reinforcement Learning

Explore similar work

Target-Aligned Reinforcement Learning

Horizon Adaptive Offline Policy Learning via Value Stitching

Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities