Bellman Meets Lyapunov: Unsupervised Reinforcement Learning via Mastering Chaos

Tristan Shah; Wooyoung Chung; Volodomyr Makarenko; Juan Wachs; Stas Tiomkin

cs.LGOct 1, 2026

Bellman Meets Lyapunov: Unsupervised Reinforcement Learning via Mastering Chaos

Authors: Tristan Shah, Wooyoung Chung, Volodomyr Makarenko, Juan Wachs, Stas Tiomkin

Organizations: Texas Tech University · Purdue University

Abstract

Reinforcement learning (RL) is a powerful paradigm for training agents, yet its success rests on domain expertise of human engineers who design informative reward signals for every new task. Unsupervised RL aims to reduce this engineering with intrinsic motivation (IM): reward signals that emerge from the agent environment interaction itself. Existing IM objectives, however, involve the selection of information variables, which re-introduces domain expertise the field has sought to eliminate. We introduce Forward CIP (F-CIP), an RL-native formulation of the Controllable Information Production (CIP) objective, which is defined by the system's dynamics alone and requires no such selection. We prove that F-CIP is compatible with RL and demonstrate its effectiveness with existing algorithms. Training agents with F-CIP results in unsupervised discovery of primitive behaviors such as balancing and maintaining controllability, which are essential for more complex robot behaviors. Paired with a simple forward-velocity reward, our method produces coordinated gaits such as hopping and running which otherwise require reward engineering to learn.

Figures & tables

Panel of Figure 1 — Figure 1: Emergent locomotion strategies on walker and hopper. Training an agent with F-CIP and a forward velocity reward yields upright running and hopping ( teal ), whereas the velocity reward alone collapses to bouncing and scooting along the ground ( gray ).

Figure 2: Schematic of our method. The environment returns the Jacobian

f_{x_{t}}

, which is used to emit reward

r_{t}

and then update the memory

S_{t}

Panel of Figure 3 — Figure 3: Learning $\Lambda^{+}$ on the Lorenz attractor . (a) convergence of the learned value function to the known $\Lambda^{+}$ for a range of $\gamma\in[0.0,0.9]$ ; (b) the attractor colored by the learned $V_{\phi}(z)$ for $\gamma=0.9$ where bright green indicates higher value.

	Intrinsic Motivation Objective
System	F-CIP (PPO)	F-CIP (APPO)	DIAYN	SMM	ICM	APT
Cart Pole	$\mathbf{0.99}$	$\mathbf{0.99}$	$0.26$	$0.68$	$0.07$	$0.64$
Double Pendulum	$\mathbf{0.99}$	$\mathbf{0.99}$	$0.44$	$0.06$	$0.05$	$0.25$
Triple Pendulum	$\mathbf{0.99}$	$\mathbf{0.99}$	$0.38$	$0.14$	$0.09$	$0.48$
Gibbon	$\mathbf{0.98}$	$\mathbf{0.99}$	$0.51$	$0.52$	$0.05$	$0.41$

Table 1: Comparison of IM on pendulum environments . We report mean extremity height over the final

10

% of the episode averaged over

10

random seeds. Values are min-max normalized so that

0.0

corresponds to the initial hanging configuration and

1.0

corresponds to the fully upright pose.

Figure 4: Unsupervised standing up behavior via F-CIP maximization . This behavior is achieved through maximization of F-CIP alone without any extrinsic reward.

	Hopper			Walker
Method	Task Return $\uparrow$	Fwd. Vel. (m/s)	Height (m)	Task Return $\uparrow$	Fwd. Vel. (m/s)	Height (m)
Ours	$\mathbf{0.39\pm 0.38}^{*}$	$1.47\pm 0.33$	$0.73\pm 0.15$	$\mathbf{0.60\pm 0.03}$	$4.53\pm 0.11$	$1.19\pm 0.04$
Fwd. Vel. only	$0.00\pm 0.00$	$0.77\pm 0.16$	$0.33\pm 0.03$	$0.10\pm 0.04$	$4.41\pm 0.22$	$0.46\pm 0.10$
ICM †	$0.00\pm 0.00$	$1.52\pm 0.55$	$0.22\pm 0.05$	$0.09\pm 0.04$	$3.85\pm 0.41$	$0.48\pm 0.12$
APT †	$0.00\pm 0.00$	$1.63\pm 0.29$	$0.23\pm 0.07$	$0.10\pm 0.05$	$3.38\pm 1.14$	$0.47\pm 0.14$
DIAYN †	$0.00\pm 0.00$	$1.22\pm 0.17$	$0.29\pm 0.12$	$0.08\pm 0.03$	$3.29\pm 0.33$	$0.42\pm 0.09$

Table 2: Reward simplification on Hopper and Walker. Mean and standard deviation calculated over

10

evaluations of the trained policies. Ours pre-trains with F-CIP alone, then fine-tunes on the forward-velocity reward while continuing to maximize F-CIP (Algorithm 1 ). Task return is the comparison metric and is measured with the full hand-designed reward; bold marks the best performance among the methods. Forward velocity and torso height are reported as diagnostics of the emergent gait. Each pretraining baseline (

\dagger

) is run in its intended configuration: the intrinsic objective is maximized during pretraining and then replaced by the velocity reward during fine-tuning. In contrast, Ours continues to maximize F-CIP throughout fine-tuning. (

*

) The high variance on Hopper arises because the oracle reward is zero whenever the torso drops below a fixed height threshold. All 10 seeds hop upright, but some adopt a deeper crouch that falls below this threshold. This strategy allows our policy to hop faster than the oracle (

1.47

vs.

1.04

m/s ).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	CP	DP	TP	G	H	W
Environment-specific
$T_{\mathrm{pre}}$	3500	3800	10000	10000	6000	1500
$T_{\mathrm{adapt}}$	0	0	0	0	7000	7000
$T_{\mathrm{warm}}$	0	0	0	0	5000	5000
$\beta_{\max}$	0	0	0	0	0.1	0.1
Rollout steps	64	64	64	256	256	256

Appendix

Table 3: Hyperparameters. Environments: CP (cart pole), DP (double pendulum), TP (triple pendulum), G (gibbon), H (hopper), W (walker).

T_{\mathrm{pre}}

T_{\mathrm{adapt}}

, and

T_{\mathrm{warm}}

are measured in PPO iterations, and

\beta_{\max}

is the maximum extrinsic reward coefficient. Batch size denotes the minibatch size used for each gradient step.

Explore similar work

Stability of Control Lyapunov Function Guided Reinforcement Learning
May 3, 2026Zachary Olkin, William D. Compton, Aaron D. AmesReinforcement Learning Control Lyapunov Function
The authors are with the Department of Control and Dynamical Systems at the California Institute of Technology.
Can We Really Learn One Representation to Optimize All Rewards?
Feb 11, 2026Chongyi Zheng, Royina Karegoudra Jayanth, Benjamin EysenbachRepresentation Learning Low-Rank Structure
Princeton University
Learning to Perceive the World Through Control: Empowerment-Based Representation Learning
May 28, 2026Mahsa Bastankhah, Sophie Broderick, Benjamin EysenbachRepresentation Learning World
Princeton University, USA

May 3, 2026, eess.SYJ/K move · Enter open · S save

Stability of Control Lyapunov Function Guided Reinforcement Learning

Zachary Olkin, William D. Compton, Aaron D. Ames

The authors are with the Department of Control and Dynamical Systems at the California Institute of Technology.

Reinforcement Learning Control Lyapunov Function

Open paper AskPDF arXiv

Reinforcement learning (RL) has become the de facto method for achieving locomotion on humanoid robots in practice, yet stability analysis of the corresponding control policies is lacking. Recent work has attempted to merge control theoretic ideas with reinforcement learning through control guided learning. A notable example of this is the use of a control Lyapunov function (CLF) to synthesize the reinforcement learning rewards, a technique known as CLF-RL, which has shown practical success. This paper investigates the stability properties of optimal controllers using CLF-RL with the goal of bridging experimentally observed stability with theoretical guarantees. The RL problem is viewed as an optimal control problem and exponential stability is proven in both continuous and discrete time using both core CLF reward terms and the additional terms used in practice. The theoretical bounds are numerically verified on systems such as the double integrator and cart-pole. Finally, the CLF guided rewards are implemented for a walking humanoid robot to generate stable periodic orbits.

Abstract

Explore similar work

Stability of Control Lyapunov Function Guided Reinforcement Learning

Can We Really Learn One Representation to Optimize All Rewards?

Learning to Perceive the World Through Control: Empowerment-Based Representation Learning