Learning Actionable Information Landscapes for Multimodal Active Sensing in Hawkmoths
Authors: Abdelrahman Sharafeldin, Yaqing Wang, Simon Sponberg, Hannah Choi
Organizations: Machine Learning Center, Georgia Institute of Technology, Atlanta, GA 30332 · School of Physics, Georgia Institute of Technology, Atlanta, GA 30332 · School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332
Animals integrate information across sensory modalities to guide exploration, suggesting that they maintain internal estimates of where actions are expected to reduce uncertainty. However, how such action-conditioned information maps are learned from multimodal sensory experience remains unclear, as previous studies have been largely confined to unimodal sensing frameworks. We introduce a multimodal predictive-coding framework for learning uncertainty-reduction maps from visual and mechanosensory observations in hawkmoth flower interaction tasks. A variational generative model trained on simulated environments recovers modality-specific information landscapes over the action space, capturing how visual patterns, surface geometry, and mechanosensory cues shape the value of exploratory actions. These learned maps are consistent with several features of hawkmoth behavior, including nonzero tracking offsets, probing along visual patterns, and geometry-dependent exploration. Applying the model to behavioral data from flower tracking and nectary search, we find that natural trajectories tend to occupy regions of high uncertainty reduction in these learned maps. Building on these observations, we develop an uncertainty-gated lexicographic algorithm that switches between information gathering and reward exploitation based on perceptual uncertainty. In nectary search, uncertainty gating improves learning relative to exploit-only, ε-greedy, fixed-time, and uncertainty-calibrated non-lexicographic controls, with better generalization and robustness to sensory and curvature shifts. These results link information landscapes learned through generative modeling to biological sensing and embodied exploration.
Figure 3: Uncertainty-reduction maps in nectary search. (a) Mechanosensory and visual UR maps across trajectories and flower conditions. (b) Mean map UR along radial and circular trajectories. (c) Angular anisotropy of final UR maps: variance of mean UR across 16 equal angular sectors around the flower center, weighted by pixel counts and normalized by total map variance. (d) Modality weights along an example trajectory and mean mechanosensory weight with and without contact. (e) Mean visual, mechanosensory, weighted-unimodal, and aggregate-entropy UR across flower curvatures. (f) Realized information gain along biological and matched random trajectories. Shading and error bars in (b–e) show SEM. In (f), ∗∗p<0.01 (t-tests).
Figure 4: UR in flower tracking. (a) Action-conditioned UR map with flower and moth trajectories. (b,d) Mean realized UR for D. elpenor and M. stellatarum at medium and high brightness, corresponding to different levels of visual noise. (c) Biological tracking compared with exact flower tracking and shuffled moth trajectories. ∗∗p<0.01 (t-tests).
Figure 5: Aggregate-entropy control and baselines. (a) Training success rate, mean episode length, and posterior uncertainty across flower curvatures and visual patterns (icons). Curves show 25-epoch moving averages; shading denotes SEM across independent training seeds. (b) Validation success rates (mean ± SEM) under curvature changes and sensory degradation for cross-pattern flowers.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Example training trajectories from the realistic trajectory sampler
Figure S2: Example behavioral trajectories from the Deora et al. [ 4 ] dataset.
Figure S3: Additional training metrics corresponding to Figure 5 . (a) Mean aggregate-entropy reduction per exploration step for adaptive and fixed-time switching. Only these baselines are plotted here since they use the exploration head. (b) Episode return. (c) Exploration fraction. Shading denotes SEM across independent training runs.
Figure S4: Two successful aggregate-entropy control episodes on a funnel flower with a cross pattern. Left: head (circles) and proboscis-tip (triangles) trajectories, colored purple during exploration and green during exploitation. Right: smoothed aggregate entropy (orange), selected mode (blue), and switching thresholds (dashed), with matching phase shading. The examples switch at steps 11 and 8 and reach the nectary at steps 16 and 21, respectively; steps are indexed from zero.
Environment
Gate
Hysteresis
Fixed step
η
Funnel + cross
530
60
33
0.07
Funnel
535
60
40
0.05
Flat + cross
520
50
40
0.0065
Flat
550
52
10
0.004
Appendix
Table 1: Settings for the control experiments in Figure 5 .
Figure S5: Weighted-unimodal control. (a) Training success and episode length in four flower conditions. (b) Curvature and sensory-shift validation after funnel-cross training.