Multi-Horizon Future Discovery for Physics-Informed Exploration

Shuwei Wang; Qiang Guan; Brian Qiu; Haoxiang Huang; Haoxi Wang; Chenyu Li; Bo Xu; Sen Cui

cs.LGOct 26, 2026

Multi-Horizon Future Discovery for Physics-Informed Exploration

Authors: Shuwei Wang, Qiang Guan, Brian Qiu, Haoxiang Huang, Haoxi Wang, Chenyu Li, Bo Xu, Sen Cui

Organizations: University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences (CASIA) · School of Artificial Intelligence, University of Chinese Academy of Sciences · University of Science and Technology of China ·

Abstract

Exploration in physical control environments requires agents to identify actions whose consequences reveal previously unexplored aspects of the underlying dynamics. However, many intrinsic motivation methods rely on immediate prediction errors, state novelty, or local physical changes, which may overlook informative outcomes that emerge across extended temporal horizons. We investigate whether learning structured representations of reachable futures can provide a more effective basis for exploration. To this end, we introduce Hamiltonian Temporal Contrastive Curiosity (HTC $^2$ ), a framework that combines multi-horizon temporal representation learning with Hamiltonian phase-space structure. A horizon-conditioned contrastive critic learns temporal relationships from future-state pairs sampled within the same episode. The distance between positive-pair representations is then used as an intrinsic reward, encouraging the agent to explore futures that remain insufficiently organized in the learned representation. To incorporate information about physical dynamics, we additionally modulate this reward using bounded multi-step Hamiltonian variation, rather than directly rewarding changes in the Hamiltonian itself. We integrate HTC $^2$ with both Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO), and evaluate it across six DeepMind Control Suite tasks. Comparisons with ICM, RIDE, E3B, Plan2Explore, and a single-step Hamiltonian curiosity baseline are complemented by state-coverage evaluations, component ablations, and phase-space diagnostics to characterize the contributions of temporal learning and physical structure to exploratory behavior.

Figures & tables

Figure 1: Overview of Hamiltonian Temporal Contrastive Curiosity (HTC 2 ). Observations are encoded into Hamiltonian phase states; same-episode future pairs train a horizon-conditioned temporal contrastive critic; and the resulting temporal discovery signal is modulated by bounded multi-step Hamiltonian variation.

Figure 2: Control performance of HTC 2 on six DMC tasks with SAC and PPO. The figure summarizes episode-return performance across five random seeds for ICM, RIDE, E3B, Plan2Explore, and HTC 2 . Numerical results are reported as mean

\pm

standard deviation in Appendix C.1 .

Figure 3: Physical-state coverage and episode reward during training. Top: cumulative unique discretized states; bottom: episode reward on cartpole_swingup , ball_in_cup_catch , and finger_spin . Curves show mean

\pm

one standard deviation over four random seeds.

Figure 4: Hamiltonian energy dynamics and phase-subspace geometry. A : open-loop learned Hamiltonian evolution under zero, random, and policy actions. B–C : PCA projections of the coordinate-like latent $q$ and momentum-like latent $p$ , colored by $H_{\varphi}(q,p)$ .

Figure 5: Zero-shot policy retention under parametric physical shifts. Bars and error bars show mean

\pm

standard deviation over four random seeds. Higher values indicate greater return retention.

Variant	Walker	Hopper	Cheetah
w/o $\mathcal{L}_{\mathrm{phys}}$	$693.166\pm 203.967$	$245.416\pm 14.651$	$566.209\pm 159.871$
w/o $G_{H}$	$718.941\pm 16.122$	$235.203\pm 1.896$	$609.108\pm 16.185$
one-step $\Delta H$	$761.051\pm 195.324$	$1.843\pm 1.183$	$510.093\pm 128.266$
w/o RND	$785.069\pm 93.809$	$227.774\pm 0.719$	$578.816\pm 9.910$
w/o repr. norm	$757.758\pm 114.117$	$144.466\pm 85.169$	$594.697\pm 16.876$
HTC 2 full	$\mathbf{814.342\pm 162.857}$	$\mathbf{250.865\pm 9.598}$	$\mathbf{659.810\pm 33.525}$

Table 1: HTC 2 ablations. Final return (mean

\pm

std, four seeds).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

HTC 2		Intrinsic-reward baselines
Hyperparameter	Value	Hyperparameter	ICM	E3B	RIDE
Intrinsic scale $\beta$	0.05	Intrinsic scale $\beta$	0.05	0.05	0.05
Cold-start reward	RND	Latent dim	256	256	256
Warmup steps	50K	Reward lr	$10^{-3}$	$10^{-3}$	$10^{-3}$
Annealing steps	200K	Batch size	256	256	256
World-model lr	$3\times 10^{-4}$	Update proportion	1.0	1.0	1.0

Appendix

Table 2: Core hyperparameters for HTC 2 and intrinsic-reward baselines.

Hyperparameter	SAC	PPO
Learning rate	$1\times 10^{-4}$	$1.5\times 10^{-5}$
Feature dim	50	512
Hidden dim	1024	1024
Batch size	256	256
Discount $\gamma$	0.99	0.99
Initial random steps	5000	–

Appendix

Table 3: SAC and PPO hyperparameters.

RL	Exploration	Walker Walk	Walker Run	Walker Stand	Hopper Hop	Hopper Stand	Cheetah Run
SAC	ICM	$677.8{\pm}230.2$	$356.7{\pm}52.9$	$936.9{\pm}31.7$	$237.5{\pm}9.1$	$863.3{\pm}29.5$	$631.9{\pm}45.4$
	RIDE	$717.9{\pm}67.5$	$364.9{\pm}117.7$	$967.1{\pm}3.3$	$239.0{\pm}11.3$	$871.3{\pm}22.1$	$624.7{\pm}44.0$
	E3B	$679.2{\pm}199.4$	$401.6{\pm}92.1$	$961.3{\pm}8.1$	$235.4{\pm}24.7$	$858.0{\pm}22.1$	$619.5{\pm}40.2$
	HTC 2 (ours)	$827.0{\pm}115.2$	$415.8{\pm}111.4$	$966.5{\pm}3.8$	$248.8{\pm}6.1$	$883.8{\pm}12.5$	$656.5{\pm}45.0$
PPO	ICM	$601.9{\pm}179.6$	$269.1{\pm}47.1$	$921.0{\pm}25.4$	$89.5{\pm}61.2$	$637.6{\pm}55.6$	$451.5{\pm}69.2$
	RIDE	$641.1{\pm}153.5$	$254.2{\pm}23.1$	$877.4{\pm}82.0$	$83.4{\pm}68.5$	$565.8{\pm}174.0$	$425.8{\pm}43.8$

Appendix

Table 4: Final control performance on six DMC tasks. Episode return over five random seeds, reported as mean

\pm

standard deviation. Within each optimizer block and task, red, orange, and yellow denote the first, second, and third-ranked exploration methods, respectively. Higher is better.

Figure 6: SAC learning curves on six DMC tasks. Curves show the mean episode return over five random seeds, with shaded regions indicating one standard deviation, for SAC combined with ICM, RIDE, E3B, and HTC 2 .

Figure 7: PPO learning curves on six DMC tasks. Curves show the mean episode return over five random seeds, with shaded regions indicating one standard deviation, for PPO combined with ICM, RIDE, E3B, Plan2Explore, and HTC 2 .

Environment	Metric	Statistic	HTC 2 (Ours) + SAC	ICM + SAC	RIDE + SAC	E3B + SAC
Cartpole Swingup	Visited States	Final	$4320.000\pm 0.000$	$3862.250\pm 486.344$	$3759.500\pm 249.116$	$3384.750\pm 399.951$
	Visited States	AUC	$3525.652\pm 71.624$	$3187.527\pm 163.460$	$3200.982\pm 70.494$	$2976.313\pm 208.753$
	Episode Reward	Final	$579.550\pm 273.080$	$126.855\pm 253.710$	$0.000\pm 0.000$	$0.000\pm 0.000$
	Episode Reward	AUC	$134.662\pm 87.884$	$15.452\pm 30.791$	$0.058\pm 0.042$	$0.017\pm 0.013$
Ball in Cup Catch	Visited States	Final	$2655.000\pm 55.154$	$2680.500\pm 3.536$	$2731.500\pm 14.849$	$2664.500\pm 12.021$
	Visited States	AUC	$2415.110\pm 30.751$	$2406.878\pm 69.883$	$2499.878\pm 14.418$	$2334.732\pm 19.627$

Appendix

Table 5: Physical-state coverage and downstream control performance on three DMC environments. Final performance and area under the training curve (AUC) are reported as mean

\pm

standard deviation over four random seeds. For each row, the best and second-best mean values are highlighted in green and yellow, respectively. The highlighting reflects ranking by the reported means. Higher values are better for all metrics.

Figure 8: Separate PCA projections of coordinate-like $q$ and momentum-like $p$ latents on walker_walk . Each subspace is visualized by learned Hamiltonian energy $H_{\varphi}(q,p)$ , normalized episode time, and projected trajectory traces. The learned phase representation exhibits smooth energy organization and temporally coherent evolution in both subspaces.

Figure 9: Action-conditioned phase geometry on walker_walk . The phase PCA is colored by action magnitude and learned control push. One-step phase updates are grouped by control strength, and the high-control group produces larger phase-step magnitudes. This shows that the learned phase dynamics are modulated by action-conditioned control.

Figure 10: Dimension-wise $q$ – $p$ coupling across DMC tasks. Each heatmap shows the absolute correlation between the 16 dimensions of coordinate-like $q$ and the 16 dimensions of momentum-like $p$ . The correlations are local and dimension-selective rather than uniformly high, indicating that $q$ and $p$ remain dynamically related but do not collapse into a single fully mixed latent representation.

Model	Gravity 1.5 $\times$	Friction 2.0 $\times$	Mass 1.5 $\times$	Damping 2.0 $\times$
SAC+ICM	69.58 $\pm$ 3.02	94.10 $\pm$ 6.05	64.64 $\pm$ 13.19	101.45 $\pm$ 0.08
SAC+RIDE	58.79 $\pm$ 8.01	95.85 $\pm$ 0.21	62.80 $\pm$ 13.66	101.73 $\pm$ 0.13
SAC+E3B	70.08 $\pm$ 16.26	97.36 $\pm$ 1.56	70.65 $\pm$ 14.16	100.46 $\pm$ 1.79
SAC+HTC 2	78.70 $\pm$ 11.94	100.01 $\pm$ 0.83	82.38 $\pm$ 10.40	97.68 $\pm$ 3.05

Appendix

Table 6: Parametric OOD zero-shot policy retention on walker_walk . Values report return retention relative to the base environment (%, mean

\pm

std).

Model	Gravity 1.5 $\times$	Friction 2.0 $\times$	Mass 1.5 $\times$	Damping 2.0 $\times$
SAC+ICM	5.42 $\pm$ 6.95	99.11 $\pm$ 1.21	40.76 $\pm$ 0.90	99.45 $\pm$ 1.37
SAC+RIDE	52.37 $\pm$ 14.84	96.55 $\pm$ 1.28	64.32 $\pm$ 15.83	100.44 $\pm$ 0.30
SAC+E3B	4.23 $\pm$ 3.48	94.85 $\pm$ 2.71	32.28 $\pm$ 17.39	99.60 $\pm$ 0.23
SAC+HTC 2	30.21 $\pm$ 5.63	99.15 $\pm$ 1.36	85.73 $\pm$ 10.99	97.65 $\pm$ 3.06

Appendix

Table 7: Parametric OOD zero-shot policy retention on hopper_stand . Values report return retention relative to the base environment (%, mean

\pm

std).

Model	Gravity 1.5 $\times$	Friction 2.0 $\times$	Mass 1.5 $\times$	Damping 2.0 $\times$
SAC+ICM	70.93 $\pm$ 2.46	54.84 $\pm$ 25.55	89.33 $\pm$ 17.98	43.74 $\pm$ 9.29
SAC+RIDE	77.72 $\pm$ 9.16	92.31 $\pm$ 8.88	100.90 $\pm$ 7.78	63.70 $\pm$ 4.68
SAC+E3B	74.47 $\pm$ 27.46	61.43 $\pm$ 7.62	96.43 $\pm$ 18.18	67.63 $\pm$ 6.52
SAC+HTC 2	84.18 $\pm$ 2.15	64.67 $\pm$ 38.44	107.85 $\pm$ 8.09	59.89 $\pm$ 4.71

Appendix

Table 8: Parametric OOD zero-shot policy retention on cheetah_run . Values report return retention relative to the base environment (%, mean

\pm

std).

Figure 11: Ablation learning curves. Curves report mean performance over 4 seeds, and shaded regions denote standard deviation. Full HTC 2 is compared with variants removing the physical consistency loss, Hamiltonian energy gate, RND warm start, representation normalization, and with a one-step

\Delta H

reward.

Figure 12: HTC 2 policy rollouts on six DMC tasks. Each row corresponds to one task, and columns show twelve temporally ordered sampled frames ( $t=0,\ldots,11$ ) from the final trained policy checkpoint. The visualization is used only as a qualitative inspection of policy behavior.

RL Exploration

Reinforcement learning exploration studies how agents choose informative actions and states to learn effective policies under uncertainty.

See all 221 papers

Mar 2, 2026

Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewards

Effective exploration in reinforcement learning requires not only tracking where an agent has been, but also understanding how the agent perceives and represents the world. To learn powerful representations, an agent should actively explore states that contribute to its knowledge of the environment. Temporal representations can capture the information necessary to solve a wide range of potential tasks while avoiding the computational cost associated with full state reconstruction. In this paper, we propose an exploration method that leverages temporal contrastive representations to guide exploration, prioritizing states with unpredictable future outcomes. We demonstrate that such representations can enable the learning of complex exploratory x in locomotion, manipulation, and embodied-AI tasks, revealing capabilities and behaviors that traditionally require extrinsic rewards. Unlike approaches that rely on explicit distance learning or episodic memory mechanisms (e.g., quasimetric-based methods), our method builds directly on temporal similarities, yielding a simpler yet effective strategy for exploration.

Faisal Mohamed, Catherine Ji, Benjamin Eysenbach +1

Mila-Quebec AI Institute · Université de Montréal · Princeton University

RL Exploration Robotic RL Reward-Free RL Representation Learning for RL

Ask PDF

Mar 8, 2026

DreamSAC: Learning Hamiltonian World Models via Symmetry Exploration

Learned world models excel at interpolative generalization but fail at extrapolative generalization to novel physical properties. This limitation arises because they learn statistical correlations rather than the environment's underlying generative rules, such as physical invariances and conservation laws. We argue that learning these invariances is key to robust extrapolation. To achieve this, we first introduce Symmetry Exploration, an unsupervised exploration strategy where an agent is intrinsically motivated by a Hamiltonian-based curiosity bonus to actively probe and challenge its understanding of conservation laws, thereby collecting physically informative data. Second, we design a Hamiltonian-based world model that learns from the collected data, using a novel self-supervised contrastive objective to identify the invariant physical state from raw, view-dependent pixel observations. Our framework, DreamSAC, trained on this actively curated data, significantly outperforms state-of-the-art baselines in 3D physics simulations on tasks requiring extrapolation.

Jinzhou Tang, Fan Feng, Minghao Fu +3

UC San Diego · Sun Yat-sen University

World Model Learning World Models Generalization in RL Physics-Informed RL Representation Learning for RL

Ask PDF

Sep 29, 2026

Learning to Plan from Random Exploration

Random exploration reveals how an environment can be traversed before a goal is specified. Can this experience support long-range planning without policy-improvement training? Our random-walk analysis explains what temporal relations contain: short horizons reveal geodesic geometry in the diffusion limit, while longer horizons reveal connectivity between regions before mixing removes these distinctions. We learn these relations with a conditional energy-based model that estimates temporal log-density ratios through horizon-conditioned embeddings. The model is trained on observation pairs by noise-contrastive estimation, without action or reward labels. The planner queries these learned relations at different horizons as it moves toward the goal. At test time, a separate local dynamics model predicts candidate action outcomes, and the temporal model evaluates their progress toward the goal by selecting or aggregating estimated improvements across horizons. The agent executes one action and replans with both models fixed. Experiments demonstrate long-range maze planning from random exploration using states and images. Learned score fields, embedding probes, and planned routes exhibit properties of a multiscale cognitive map. We further demonstrate egocentric navigation from random exploration and manipulation planning from suboptimal data.

Deqian Kong, Guangyan Sun, Sheng Cheng +6

UCLA · University of Minnesota · Amazon AGI +3

Long-Horizon Planning Model-Based Planning Long-Horizon Reasoning

Ask PDF

Abstract

RL Exploration

Explore similar work

Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewards

DreamSAC: Learning Hamiltonian World Models via Symmetry Exploration

Learning to Plan from Random Exploration