Intrinsic Goals for Autonomous Agents: Hierarchical Exploration Captures Future Volitional Action in Drosophila

Declan Kutscher; Reece Keller; Alyn Kirsch; Felix Pei; Aran Nayebi

q-bio.NCOct 25, 2026

Intrinsic Goals for Autonomous Agents: Hierarchical Exploration Captures Future Volitional Action in Drosophila

Authors: Declan Kutscher, Reece Keller, Alyn Kirsch, Felix Pei, Aran Nayebi

Organizations: Department of Computer Science, University of Maryland, College Park · Machine Learning Department, Carnegie Mellon University · Neuroscience Institute, Carnegie Mellon University · Robotics Institute, Carnegie Mellon University

Abstract

Animals have a remarkable capacity for autonomy---driven by intrinsic motivation---without external sensory cues or extrinsic objectives, yet the computational mechanisms behind this ability largely remain a mystery. We combine whole-brain recordings in the fruit fly (Drosophila melanogaster), which reveal neural representations of spontaneous exploratory decisions more than 10 seconds in advance, with a neuromechanical digital twin to test whether optimizing intrinsic goals yields realistically embodied artificial agents that quantitatively match fly behavior and internal representations of future volitional action during exploration. Our hierarchical policy selects from a discrete library of low-level motor primitives, consistent with ethological theories of behavioral syllables. We train agents without external reward using 20 intrinsic motivation algorithms spanning curiosity, entropy, and competency methods. The agents are evaluated in a virtual tethered-ball assay modeled on the fly experiment, comparing spontaneous locomotor statistics and internal representations and decoding future turn direction from kinematics and neural activity on the same timescales observed in flies. DIAYN, a hierarchical competency algorithm learning discrete policies that maximize behavioral entropy and the separability of induced sensory observations, achieves the strongest joint behavioral and neural prospective coding performance relative to flies. Its learned policies closely capture individual variability in fly exploration strategies, suggesting that this diversity serves to maximize state-space coverage at the population level. DIAYN's latent activity also closely captures the sum-difference rule for interhemispheric encoding of kinematics in biological flies. These results establish hierarchical, intrinsically motivated agents as a powerful framework for reverse-engineering animal autonomy.

Figures & tables

Figure 1: Environment Overview. (a) The fly policy is trained in an environment with obstacles and multiple odor sources. (b) The policy is then evaluated in a tethered-ball environment where the fly is head-fixed. The angular and forward velocities are read from the ball, analogous to the biological experiment of Brezovec et al. (2023) . (c) The policy receives olfaction as a weighted sum of intensities sampled at the antennae and maxillary palps. (d) The policy’s proprioceptive input comes directly from NeuroMechFly and includes joint angles, joint velocities, and body-frame gravity, height, and linear and angular velocities. (e) Vision is also provided to the policy by simulating the ommatidia mapping of pale and yellow type cells to pixels.

Figure 2: Agent Architecture Overview. (a) The FlyGym simulator provides visual, proprioceptive, and olfactory observations from a biomechanically realistic fly body. Modality-specific CNN and MLP encoders embed these observations into a shared representation $\phi_{t}$ , which is provided to the actor–critic and intrinsic drive module. When applicable, the policy is additionally conditioned on a sampled skill $z$ . The intrinsic drive module computes an intrinsic reward from the encoded state transition and selected action. The actor outputs a categorical distribution over discrete motor primitives, while the critic estimates the state value. The policy selects a new motor primitive every 100 ms. (b) Locomotor primitives are translated into joint-level commands by a low-level central pattern generator (CPG) controller, while grooming is executed using a kinematic replay. These motor commands are applied within the FlyGym physics simulation, which is integrated with a timestep of 100 $\mu$ s. Thus, each policy action is executed for 1,000 physics steps before the next action is selected.

Figure 3: Turn-direction decoding from neural and behavioral data. Prospective coding is quantified as balanced accuracy in predicting future left/right turns. (a,c) Neural (a) and behavioral (c) prospective coding in the fly and models using elastic-net logistic regression in the

-15

-8\,\mathrm{s}

window. (b) Neural and behavioral decoding before (

-15

-8\,\mathrm{s}

) and during (

-0.4

+0.4\,\mathrm{s}

) turns. (d) Behavioral versus neural prospective coding. The dashed line defines the fly’s behavioral-to-neural accuracy ratio, which DIAYN closely matches. (e) Models with no turns or only one turn direction, precluding stratified cross-validation.

Figure 4: Turning Bias. (a) Distribution of turning bias across evaluation rollouts for each method. Points indicate individual rollouts, with the mean and range shown for each method. DIAYN produces a broad distribution of turning biases spanning more of the fly-to-fly variability than the other evaluated methods. Methods with a rollout containing no turns or turns in only one direction are excluded because $\log(N_{R}/N_{L})$ is undefined for those rollouts. (b) Virtual x-y trajectories integrated from the ball state of the nine flies and the nine DIAYN evaluation skills drawn from a common origin.

Figure 5: Low-dimensional internal structures relate to $\omega$ and $v$ . (a) Reconstructed from Brezovec et al. (2023) , the IPS of the fly brain has an interhemispheric relationship of z-scored activity that is correlated with angular (interhemispheric difference) and forward velocity (interhemispheric sum). (b) As a functional analogue of the bilateral supercluster pairs in Brezovec et al. (2023) , we use PCA to find principal components in the network activation of the LSTM layer of the Actor in the policy whose difference correlates with angular velocity and whose sum correlates with forward velocity. This structural relationship appears in DIAYN’s Actor LSTM layer (b) but, does not appear in (c) the randomly initialized DIAYN model. (d) We rank each IDM by the mean of the squared Pearson correlations of its angular and forward heatmaps with the fly’s. Higher is a closer match.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Index	Primitive	$d_{L}$	$d_{R}$	Controller
0	Left	$-1.0$	$1.2$	CPG turning
1	Forward	$1.0$	$1.0$	CPG walking
2	Right	$1.2$	$-1.0$	CPG turning
3	Stop	$0$	$0$	CPG, amplitude decays to zero
4	Groom	$0$	$0$	Joint replay overrides CPG
5	Hard left	$-1.2$	$1.2$	CPG turning

Appendix

Table 1: Seven discrete behavioral primitives. The sign of

d

sets the stepping direction of that side’s legs (negative is backward) and

|d|

sets the stepping amplitude.

Component	Architecture
Visual encoder (shared across eyes)	Conv( $2\!\to\!32$ , $5\times 5$ , stride 2) – Conv( $32\!\to\!64$ , $3\times 3$ , stride 2) – Conv( $64\!\to\!64$ , $3\times 3$ , stride 2) – FC( $1024\!\to\!64$ )
Visual embedding	$2\times 64=128$
Olfactory encoder	MLP $2\!\to\!32\!\to\!16$ , concatenated with raw input ( $18$ )
Proprioceptive encoder	MLP $100\!\to\!128\!\to\!64$ , concatenated with raw input ( $164$ )
Base features $\phi_{t}$	$310$
Actor and critic LSTMs	One layer, 256 units each

Appendix

Table 2: Policy network architecture.

Method	Family	Skill prior (resampling)	Trainable IDM parameters
ICM	Curiosity	–	543,495
RND	Curiosity	–	276,992
Disagreement	Curiosity	–	1,134,350
$\gamma$ -progress	Curiosity	–	226,870
MAX	Curiosity	–	40,603,008
NGU	Curiosity	–	449,063

Appendix

Table 3: Intrinsic drive methods. Parameter counts exclude frozen targets, lagged copies, and memories.

Method	Hours	Decisions/s	Method	Hours	Decisions/s
ICM	124.1	112.3	DIAYN	145.9	95.5
RND	144.2	96.6	VIC	127.2	109.6
Disagreement	121.1	115.1	DADS	155.1	89.9
$\gamma$ -progress	162.8	85.6	rDADS	107.7	129.4
MAX	139.4	100.0	VALOR	125.1	111.4
NGU	175.3	79.5	CSF	146.1	95.4

Appendix

Table 4: Recorded training time for the IDM-trained policies. Hours sum the final rollout-end timer reading from each process, rounded to 0.1 hour, and are lower bounds on full job time. Decisions/s is

50{,}176{,}000

divided by that summed time.

Recurrent PPO
Setting	Value
Parallel environments	2,048
Decisions per environment per rollout	100
Transitions per rollout	204,800
Minibatch size	4,096 (50 minibatches per epoch)
Epochs per rollout	10

Appendix

Table 5: PPO training and environment hyperparameters.

Fly	$N_{L}$	$N_{R}$	$\log(N_{R}/N_{L})$
087	222	519	$0.85$
089	38	33	$-0.14$
094	140	2	$-4.25$
097	47	105	$0.80$
098	178	291	$0.49$
099	27	543	$3.00$

Appendix

Table 6: Turns per fly in the biological dataset.

Method	$N_{L}$	$N_{R}$	Turns/min	Turn bias $\log(N_{R}/N_{L})$	Sel. layer	Layer acc. ( $-15..-8$ s)	Behav. acc. ( $-15..-8$ s)	Margin	Internal ( $\pm 0.4$ s)	Behav. ( $\pm 0.4$ s)	Unscored reason
ICM	0	14997	55.55 $\pm$ 1.30	–	–	–	–	–	–	–	only right turns
RND	0	0	0.00 $\pm$ 0.00	–	–	–	–	–	–	–	no turns
Disagreement	0	62288	230.70 $\pm$ 12.13	–	–	–	–	–	–	–	only right turns
$\gamma$ -progress	10320	295	39.32 $\pm$ 9.03	-3.78 $\pm$ 0.96	A-LSTM+Pol	0.603 $\pm$ 0.022	0.724 $\pm$ 0.012	-0.122	0.864 $\pm$ 0.028	1.000 $\pm$ 0.000	–
RE3	179	5379	20.59 $\pm$ 1.90	3.47 $\pm$ 0.49	A-LSTM (8 tied) ‡	0.500 $\pm$ 0.000	0.531 $\pm$ 0.050	-0.031	0.756 $\pm$ 0.019	1.000 $\pm$ 0.000	–
APT	252	534	2.91 $\pm$ 0.51	0.79 $\pm$ 0.27	All	0.516 $\pm$ 0.018	0.534 $\pm$ 0.031	-0.019	0.964 $\pm$ 0.013	1.000 $\pm$ 0.000	–

Appendix

Table 7: Extended per-method results. Accuracy is balanced accuracy; uncertainty is fold s.d. where shown.

\dagger

Turning bias averaged over only the indicated number of rollouts; the remaining rollouts turned in only one direction (or not at all), so their bias is undefined.

N_{L}

and

N_{R}

are summed over all nine rollouts.

\ddagger

marks a tie at maximum accuracy.

Method	A-LSTM	C-LSTM	Pol	Val	Actor	Critic	All	Raw obs.
ICM	–	–	–	–	–	–	–	–
RND	–	–	–	–	–	–	–	–
Disagreement	–	–	–	–	–	–	–	–
$\gamma$ -progress	0.579	0.529	0.592	0.500	0.603	0.541	0.584	0.508
RE3	0.500	0.500	0.500	0.500	0.500	0.500	0.500	0.500
APT	0.500	0.503	0.500	0.500	0.498	0.502	0.516	0.505

Appendix

Table 8: Early-window (

-15

-8\,\mathrm{s}

) balanced decoding accuracy for each candidate network feature set and the raw-observation control. A-LSTM and C-LSTM denote the actor and critic LSTM hidden states; Pol and Val denote the policy- and value-MLP hidden representations; Actor concatenates A-LSTM+Pol; Critic concatenates C-LSTM+Val; and All concatenates all five recorded internal representations, including the learned observation embedding. Raw obs. denotes the proprioceptive input with vision and odor set to zero and is distinct from the learned observation embedding. Bold denotes the selected network representation (argmax over the seven network feature sets only).

\dagger

indicates that the raw-observation control achieved higher decoding accuracy than the selected internal representation. – indicates that decoding was not scored because fewer than two bouts were available in one class.

Intrinsic Motivation

Intrinsic motivation studies internal objectives that drive agents to explore, learn skills, and seek informative experiences without explicit task rewards.

See all 39 papers

Intrinsic Motivation in Reinforcement Learning: A Research Agenda for Adaptive Self-Organisation
Sep 15, 2026Anatoly BelikovIntrinsic Motivation Intrinsic Reward Methods
SingularityNET Foundation
Efficient Exploration Is Enough
Sep 7, 2026Mikel Malagón, Jon Vadillo, Josu Ceberio +2Intrinsic Reward Methods RL Exploration
University of the Basque Country (UPV/EHU) · University of Alberta · Alberta Machine Intelligence Institute +2
Modeling Fly Social Behavior with Agent-Centric Pose Forecasting
Jul 21, 2026Eyrun Eyjolfsdottir, Kristin BransonMulti-Agent Trajectory Prediction Animal Behavior Analysis
HHMI Janelia Research Campus, Ashburn, VA 20147

Sep 15, 2026J/K move · Enter open · S save

Intrinsic Motivation in Reinforcement Learning: A Research Agenda for Adaptive Self-Organisation

Anatoly Belikov

SingularityNET Foundation

Intrinsic Motivation Intrinsic Reward Methods Agent Skill Learning Emergent Behavior in Multi-Agent Systems

Open paper AskPDF arXiv

Biological cells can be viewed as individual, interacting agents whose collective dynamics give rise to adaptive behaviour at multiple levels of organisation, from individual cells through tissues to whole multicellular organisms. In this perspective and tutorial article we discuss whether intrinsic rewards in artificial neural systems can support adaptation, functional specialisation and higher-level self-organisation without a shared external objective. We review empowerment, curiosity, learning progress, information gain, unsupervised skill discovery, mutual information estimation and the use of world models for intrinsic reward computation. Particular attention is given to failure modes showing when such objectives do not produce sustained exploration or increasingly complex behaviour. We argue that more capable systems may require complementary objectives, communication, memory, learning at multiple temporal scales and environmental constraints. Based on this perspective, we outline three experimental directions. These include a resource-constrained environment in which otherwise stable behavioural attractors become unsustainable, allowing us to test whether environmental constraints can mitigate characteristic failure modes of intrinsic objectives. The network of recurrent agents with per-agent intrinsic rewards, and a hierarchical world-model agent in which exploratory motor competence develops before goal-directed behaviour. These experiments are intended to test whether intrinsic learning can lead to adaptive organisation at progressively higher levels.

Abstract

Intrinsic Motivation

Explore similar work

Intrinsic Motivation in Reinforcement Learning: A Research Agenda for Adaptive Self-Organisation

Efficient Exploration Is Enough

Modeling Fly Social Behavior with Agent-Centric Pose Forecasting