Organizations: Department of Computer Science, University of Maryland, College Park · Machine Learning Department, Carnegie Mellon University · Neuroscience Institute, Carnegie Mellon University · Robotics Institute, Carnegie Mellon University
Animals have a remarkable capacity for autonomy---driven by intrinsic motivation---without external sensory cues or extrinsic objectives, yet the computational mechanisms behind this ability largely remain a mystery. We combine whole-brain recordings in the fruit fly (Drosophila melanogaster), which reveal neural representations of spontaneous exploratory decisions more than 10 seconds in advance, with a neuromechanical digital twin to test whether optimizing intrinsic goals yields realistically embodied artificial agents that quantitatively match fly behavior and internal representations of future volitional action during exploration. Our hierarchical policy selects from a discrete library of low-level motor primitives, consistent with ethological theories of behavioral syllables. We train agents without external reward using 20 intrinsic motivation algorithms spanning curiosity, entropy, and competency methods. The agents are evaluated in a virtual tethered-ball assay modeled on the fly experiment, comparing spontaneous locomotor statistics and internal representations and decoding future turn direction from kinematics and neural activity on the same timescales observed in flies. DIAYN, a hierarchical competency algorithm learning discrete policies that maximize behavioral entropy and the separability of induced sensory observations, achieves the strongest joint behavioral and neural prospective coding performance relative to flies. Its learned policies closely capture individual variability in fly exploration strategies, suggesting that this diversity serves to maximize state-space coverage at the population level. DIAYN's latent activity also closely captures the sum-difference rule for interhemispheric encoding of kinematics in biological flies. These results establish hierarchical, intrinsically motivated agents as a powerful framework for reverse-engineering animal autonomy.
Figures & tables
Figure 1: Environment Overview. (a) The fly policy is trained in an environment with obstacles and multiple odor sources. (b) The policy is then evaluated in a tethered-ball environment where the fly is head-fixed. The angular and forward velocities are read from the ball, analogous to the biological experiment of Brezovec et al. (2023) . (c) The policy receives olfaction as a weighted sum of intensities sampled at the antennae and maxillary palps. (d) The policy’s proprioceptive input comes directly from NeuroMechFly and includes joint angles, joint velocities, and body-frame gravity, height, and linear and angular velocities. (e) Vision is also provided to the policy by simulating the ommatidia mapping of pale and yellow type cells to pixels.
Figure 2: Agent Architecture Overview. (a) The FlyGym simulator provides visual, proprioceptive, and olfactory observations from a biomechanically realistic fly body. Modality-specific CNN and MLP encoders embed these observations into a shared representation ϕt , which is provided to the actor–critic and intrinsic drive module. When applicable, the policy is additionally conditioned on a sampled skill z . The intrinsic drive module computes an intrinsic reward from the encoded state transition and selected action. The actor outputs a categorical distribution over discrete motor primitives, while the critic estimates the state value. The policy selects a new motor primitive every 100 ms. (b) Locomotor primitives are translated into joint-level commands by a low-level central pattern generator (CPG) controller, while grooming is executed using a kinematic replay. These motor commands are applied within the FlyGym physics simulation, which is integrated with a timestep of 100 μ s. Thus, each policy action is executed for 1,000 physics steps before the next action is selected.
Figure 3: Turn-direction decoding from neural and behavioral data. Prospective coding is quantified as balanced accuracy in predicting future left/right turns. (a,c) Neural (a) and behavioral (c) prospective coding in the fly and models using elastic-net logistic regression in the −15 to −8s window. (b) Neural and behavioral decoding before ( −15 to −8s ) and during ( −0.4 to +0.4s ) turns. (d) Behavioral versus neural prospective coding. The dashed line defines the fly’s behavioral-to-neural accuracy ratio, which DIAYN closely matches. (e) Models with no turns or only one turn direction, precluding stratified cross-validation.
Figure 4: Turning Bias. (a) Distribution of turning bias across evaluation rollouts for each method. Points indicate individual rollouts, with the mean and range shown for each method. DIAYN produces a broad distribution of turning biases spanning more of the fly-to-fly variability than the other evaluated methods. Methods with a rollout containing no turns or turns in only one direction are excluded because log(NR/NL) is undefined for those rollouts. (b) Virtual x-y trajectories integrated from the ball state of the nine flies and the nine DIAYN evaluation skills drawn from a common origin.
Figure 5: Low-dimensional internal structures relate to ω and v . (a) Reconstructed from Brezovec et al. (2023) , the IPS of the fly brain has an interhemispheric relationship of z-scored activity that is correlated with angular (interhemispheric difference) and forward velocity (interhemispheric sum). (b) As a functional analogue of the bilateral supercluster pairs in Brezovec et al. (2023) , we use PCA to find principal components in the network activation of the LSTM layer of the Actor in the policy whose difference correlates with angular velocity and whose sum correlates with forward velocity. This structural relationship appears in DIAYN’s Actor LSTM layer (b) but, does not appear in (c) the randomly initialized DIAYN model. (d) We rank each IDM by the mean of the squared Pearson correlations of its angular and forward heatmaps with the fly’s. Higher is a closer match.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Index
Primitive
dL
dR
Controller
0
Left
−1.0
1.2
CPG turning
1
Forward
1.0
1.0
CPG walking
2
Right
1.2
−1.0
CPG turning
3
Stop
0
0
CPG, amplitude decays to zero
4
Groom
0
0
Joint replay overrides CPG
5
Hard left
−1.2
1.2
CPG turning
Appendix
Table 1: Seven discrete behavioral primitives. The sign of d sets the stepping direction of that side’s legs (negative is backward) and ∣d∣ sets the stepping amplitude.
Table 4: Recorded training time for the IDM-trained policies. Hours sum the final rollout-end timer reading from each process, rounded to 0.1 hour, and are lower bounds on full job time. Decisions/s is 50,176,000 divided by that summed time.
Setting
Value
Recurrent PPO
Parallel environments
2,048
Decisions per environment per rollout
100
Transitions per rollout
204,800
Minibatch size
4,096 (50 minibatches per epoch)
Epochs per rollout
10
Appendix
Table 5: PPO training and environment hyperparameters.
Fly
NL
NR
log(NR/NL)
087
222
519
0.85
089
38
33
−0.14
094
140
2
−4.25
097
47
105
0.80
098
178
291
0.49
099
27
543
3.00
Appendix
Table 6: Turns per fly in the biological dataset.
Method
NL
NR
Turns/min
Turn bias log(NR/NL)
Sel. layer
Layer acc. ( −15..−8 s)
Behav. acc. ( −15..−8 s)
Margin
Internal ( ±0.4 s)
Behav. ( ±0.4 s)
Unscored reason
ICM
0
14997
55.55 ± 1.30
–
–
–
–
–
–
–
only right turns
RND
0
0
0.00 ± 0.00
–
–
–
–
–
–
–
no turns
Disagreement
0
62288
230.70 ± 12.13
–
–
–
–
–
–
–
only right turns
γ -progress
10320
295
39.32 ± 9.03
-3.78 ± 0.96
A-LSTM+Pol
0.603 ± 0.022
0.724 ± 0.012
-0.122
0.864 ± 0.028
1.000 ± 0.000
–
RE3
179
5379
20.59 ± 1.90
3.47 ± 0.49
A-LSTM (8 tied) ‡
0.500 ± 0.000
0.531 ± 0.050
-0.031
0.756 ± 0.019
1.000 ± 0.000
–
APT
252
534
2.91 ± 0.51
0.79 ± 0.27
All
0.516 ± 0.018
0.534 ± 0.031
-0.019
0.964 ± 0.013
1.000 ± 0.000
–
Appendix
Table 7: Extended per-method results. Accuracy is balanced accuracy; uncertainty is fold s.d. where shown. † Turning bias averaged over only the indicated number of rollouts; the remaining rollouts turned in only one direction (or not at all), so their bias is undefined. NL and NR are summed over all nine rollouts. ‡ marks a tie at maximum accuracy.
Method
A-LSTM
C-LSTM
Pol
Val
Actor
Critic
All
Raw obs.
ICM
–
–
–
–
–
–
–
–
RND
–
–
–
–
–
–
–
–
Disagreement
–
–
–
–
–
–
–
–
γ -progress
0.579
0.529
0.592
0.500
0.603
0.541
0.584
0.508
RE3
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
APT
0.500
0.503
0.500
0.500
0.498
0.502
0.516
0.505
Appendix
Table 8: Early-window ( −15 to −8s ) balanced decoding accuracy for each candidate network feature set and the raw-observation control. A-LSTM and C-LSTM denote the actor and critic LSTM hidden states; Pol and Val denote the policy- and value-MLP hidden representations; Actor concatenates A-LSTM+Pol; Critic concatenates C-LSTM+Val; and All concatenates all five recorded internal representations, including the learned observation embedding. Raw obs. denotes the proprioceptive input with vision and odor set to zero and is distinct from the learned observation embedding. Bold denotes the selected network representation (argmax over the seven network feature sets only). † indicates that the raw-observation control achieved higher decoding accuracy than the selected internal representation. – indicates that decoding was not scored because fewer than two bouts were available in one class.
Intrinsic motivation studies internal objectives that drive agents to explore, learn skills, and seek informative experiences without explicit task rewards.
Biological cells can be viewed as individual, interacting agents whose collective dynamics give rise to adaptive behaviour at multiple levels of organisation, from individual cells through tissues to whole multicellular organisms. In this perspective and tutorial article we discuss whether intrinsic rewards in artificial neural systems can support adaptation, functional specialisation and higher-level self-organisation without a shared external objective. We review empowerment, curiosity, learning progress, information gain, unsupervised skill discovery, mutual information estimation and the use of world models for intrinsic reward computation. Particular attention is given to failure modes showing when such objectives do not produce sustained exploration or increasingly complex behaviour. We argue that more capable systems may require complementary objectives, communication, memory, learning at multiple temporal scales and environmental constraints. Based on this perspective, we outline three experimental directions. These include a resource-constrained environment in which otherwise stable behavioural attractors become unsustainable, allowing us to test whether environmental constraints can mitigate characteristic failure modes of intrinsic objectives. The network of recurrent agents with per-agent intrinsic rewards, and a hierarchical world-model agent in which exploratory motor competence develops before goal-directed behaviour. These experiments are intended to test whether intrinsic learning can lead to adaptive organisation at progressively higher levels.
This work introduces an alternative view of efficient exploration and studies its theoretical and empirical implications in the absence of extrinsic rewards. Specifically, we define efficient explorers as agents that prioritize generating generalizable experience, i.e., data that supports learning models capable of predicting and adapting across the environment. This allows us to analyze efficient exploration through the lens of prediction and generalization. Theoretically, we demonstrate that optimally efficient explorers naturally schedule their trajectories to visit the most informative and learnable regions first. Empirically, we show that optimizing for these agents gives rise to an automatic curriculum of progressively more complex behaviors, even in relatively simple environments. These results indicate that pursuing this purely intrinsic objective alone is enough to drive the emergence of highly sophisticated behaviors. We believe that this new framework provides a principled mechanism by which agent-environment systems may sustain an open-ended process of increasingly complex behavior without external rewards, tasks, or objectives.
Mikel Malagón, Jon Vadillo, Josu Ceberio +2
University of the Basque Country (UPV/EHU) · University of Alberta · Alberta Machine Intelligence Institute +2
Understanding animal behavior at an algorithmic level -- what animals attend to, how they form internal representations, and how this maps to action -- remains a central challenge in neuroethology. We explore how well large generative models capture properties of fly social behavior. We combine transformers with agent-centric representations. Models input egocentric sensory observations and output egocentric movements, mirroring biological constraints. Social behavior emerges from agents independently sensing and responding to one another. We demonstrate that this simple model captures many properties of fly behavior, outperforming the world-centric representation and achieving SOTA on MABe 2022-fly. We systematically evaluate properties of the representation important for capturing the behavior distribution through a series of lesioning experiments. These comparisons were enabled by a new, general-purpose library for animal-centric pose forecasting.