Olfactory pursuit: catching a moving odor source in complex flows
Authors: Maurizio Carbone, Lorenzo Piro, Robin A. Heinonen, Luca Biferale, Massimo Cencini, Antonio Celani
Organizations: Istituto dei Sistemi Complessi, CNR, Via dei Taurini 19, 00185 Rome, Italy and INFN “Tor Vergata", Via della Ricerca Scientifica 1, 00133 Rome, Italy · Department of Physics & INFN, Tor Vergata University of Rome, Via della Ricerca Scientifica 1, 00133 Rome, Italy · Machine Learning Genoa Center (MaLGa) & Department of Civil, Chemical and Environmental Engineering, University of Genoa, Genoa, Italy · Quantitative Life Sciences, The Abdus Salam International Centre for Theoretical Physics, 34151 Trieste, Italy · Department of Oncology, University of Torino, Torino, Italy
Locating and intercepting a moving target from possibly delayed, intermittent sensory signals is a paradigmatic problem in decision-making under uncertainty, and a fundamental challenge for, e.g., animals seeking prey or mates and autonomous robotic systems. Odor signals are intermittent, strongly mixed by turbulent-like transport, and typically lag behind the true target position, thereby complicating localization. Here, we formulate olfactory pursuit as a partially observable Markov decision process in which an agent maintains a joint belief over the target's position and velocity. Using a discrete run-and-tumble model, we compute quasi-optimal policies by numerically solving the Bellman equation and benchmark them against well-established information-theoretic strategies such as Infotaxis. We show that purely exploratory policies are near-optimal when the target frequently reorients, but fail dramatically when the target exhibits persistent motion. We thus introduce a computationally efficient hybrid policy that combines the information-gain drive of Infotaxis with a "greedy" value function derived from an associated fully observable control problem. Our heuristic achieves near-optimal performance across all persistence times and substantially outperforms purely exploratory approaches. Moreover, our proposal demonstrates strong robustness even in more complex search scenarios, including continuous run-and-tumble prey motion with moderate persistence time, model mismatch, and more accurate plume dynamics representation. Our results identify predictive inference of target motion as the key ingredient for effective olfactory pursuit and provide a general framework for search in information-poor, dynamically evolving environments.
Figures & tables
Figure 1: Sketch of typical olfactory pursuit. The target source (blue circle) performs a run-and-tumble motion and emits a blue chemical plume along its trajectory. The agent (red dashed trajectory) pursues the moving target by inferring its position and velocity from discrete odor detections (indicated by the red triangles). The agent maintains a probability map over the target position Xs and velocity Us , namely the joint belief bt(x,u) about its state at time t represented in the thought clouds. The search begins with an initial detection at t=0 , after which the agent’s position belief corresponds to the likelihood of a detection centered at the agent’s position; the estimated target velocity is initially random uniform over all directions. As time progresses, the target moves and emits, while the agent updates its belief to account for the target source’s motion and accumulates information about its state through successive detections. Over time, the agent’s belief becomes more peaked around the true target position, and its guess of the target velocity gets more accurate. This allows the agent to confidently pursue and intercept the target.
Figure 2: Results from the discrete run-and-tumble model, with both agent and target moving on a periodic lattice. Panel A shows the average search time ⟨T⟩ as a function of the target persistence time τp , averaged over 104 search episodes starting from a detection: the black curve indicates the quasi-optimal strategy obtained from solving the POMDP; blue curve the infotactic policy ( w=0 ); gray curve refers to random search, obtained with the agent performing a persistent random walk with no olfactory cues; circles indicate the hybrid heuristic policy 3 for the optimal weight w , yielding the lowest average search time, color-coded as in the right bar. The insets show the complementary cumulative distribution function (CCDF) of the search times for the best hybrid heuristic and Infotaxis, at τp=2 and 25 . Panel B details the effect of the policy weight w on the average search time, normalized with the best average search time, for τp=2 and 25 . Panels C , D compare the average search times of the best hybrid heuristic with the quasi-optimal policy ( C ) and Infotaxis ( D ). For this set of simulations, the side of the square lattice is L=51 with spacing Δx=1 , the agent and target speed is U=1 , the characteristic length scale of the likelihood of a detection is λ=3 , the emission rate R=1 , and the agent finds the source when ∥Xag−Xs∥≤2 .
Figure 3: Results from the continuous run-and-tumble model, with the target performing a continuous run-and-tumble on a periodic square and the agent moving on a periodic lattice. Panel A shows the average search time ⟨T⟩ as a function of the target persistence time, averaged over 104 search episodes starting from a detection: circles indicate the hybrid heuristic policy 3 for the best weight w , color-coded as in the right bar; Infotaxis in blue corresponds to w=0 ; the gray line refers to random search, obtained with the agent performing a persistent random walk without olfactory cues. The inset compares the average search times of the best hybrid heuristic with Infotaxis. Panel B shows the effect of the policy weight w on the normalized average search time for two persistence times of the target motion. Panel C shows the normalized average search time as a function of Uτd/λ , i.e. the ratio between the agent speed U and the characteristic speed of the chemical diffusion λ/τd , at emission rate R=36/τd . Results in panels A and B correspond to Uτd/λ=0.1 at R=120 . For the results reported here, the agent and target speed is U=1 , the side of the lattice domain is L=51 , the detection likelihood length scale is λ=3 , and the agent finds the source when ∥Xag−Xs∥≤2 .
Finding an odor source in a turbulent flow requires effectively leveraging the history of olfactory observations into a robust navigation strategy. In this work, we use tabular Q-learning to train an olfactory search agent with a minimal memory of past observations: only a running clock since the last whiff. This agent learns an interpretable strategy to recover the plume which combines well-known behaviors observed in insects: surging, casting, and a return downwind. While achieving good performance on data from direct numerical simulations of turbulence, the agent is limited by an inability to adapt its strategy to the local intermittency level; we show that providing more flexibility improves robustness.
Marco Rando, Robin A. Heinonen, Yujia Qi +1
Universit´e Cˆote d’Azur, Inria, CNRS, Laboratoire J.A. Dieudonn´e, 28 avenue Valrose, 06108, Nice, France · Machine Learning Genoa Center & Department of Civil, Chemical and Environmental Engineering, University of Genova Via Montallegro 1, 16145 Genoa, Italy · Machine Learning Genoa Center & Department of Civil, Chemical and Environmental Engineering, University of Genova, Via Montallegro 1, 16145 Genoa, Italy +1
Agents tasked with intercepting a moving target must learn how to reach it, ideally as efficiently as possible. As observers of animal pursuit behavior have noticed, different strategies are possible: on one extreme, the agent reactively moves toward the target's current location; on the other, the agent predicts the target's future location and moves directly there. Motivated by the desire to understand how such strategies might be learned, we introduce a continuous-time linear-quadratic model of open-loop target pursuit whose optimal control strategies interpolate between these possibilities. Usefully, a small number of interpretable parameters control which type of strategy is optimal, and it is possible to derive closed-form solutions for optimal strategies, policy learning dynamics, and value learning dynamics. Exploiting our model's linear structure, we find that the time scales of learning precisely correspond to the eigenvalues of certain matrices, and that relevant eigenvalue spectra generically have a gap. We show that this gap indicates that agents tend to learn how to reach their target before optimizing their movement. Our results provide a detailed mathematical characterization of pursuit behavior and its learning dynamics, which can both serve as a benchmark for empirical work, and as a foundation for more elaborate theoretical treatments of pursuit.
John J. Vastola, Kanaka Rajan
Department of Neurobiology, Harvard Medical School · Department of Psychology and Center for Brain Science, Harvard University · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University
In dynamic flow fields, various animals exhibit remarkable odor search capabilities despite relying on stochastic detections. Interestingly, there exists an optimal time window for integrating these detections that maximizes search efficiency. To understand the underlying mechanism, we investigate the navigation performance of Reinforcement Learning (RL) agents in unsteady flows under varying memory lengths and flow conditions. Without any predefined models, the agents develop a flow-assisted casting strategy and adaptively adjust both the geometry of their search trajectories and the concentration threshold for initiating casting to maximize the success rate. The agent's average speed toward the odor source exhibits a non-monotonic dependence on memory length, which can be explained by the "sector-search" model.
Changxu Zhao, Dongxiao Zhao, Xin Bian +1
State Key Laboratory of Ocean Engineering, School of Ocean and Civil Engineering, Shanghai Jiao Tong University, Shanghai 200240, People’s Republic of China · State Key Laboratory of Fluid Power and Mechatronic Systems, Department of Engineering Mechanics, Zhejiang University, Hangzhou 310027, People’s Republic of China