Active search for recovering objects of interest through online, adaptive decision making with autonomous agents requires trading off exploration of unknown environments with exploitation of prior observations in the search space. Prior work has proposed information gain and Thompson sampling based myopic, greedy approaches for agents to actively decide query or search locations when the number of targets is unknown. Decision making algorithms in such partially observable environments have also shown that agents capable of lookahead over a finite horizon outperform myopic policies for active search. Unfortunately, lookahead algorithms typically rely on building a computationally expensive search tree that is simulated and updated based on the agent's observations and a model of the environment dynamics. Instead, in this work, we leverage the sequence modeling abilities of diffusion models to sample lookahead action sequences that balance the exploration-exploitation trade-off for active search without building an exhaustive search tree. We identify the optimism bias in prior diffusion based reinforcement learning approaches when applied to the active search setting and propose mitigating solutions for efficient cost-aware decision making with both single and multi-agent teams. Our proposed algorithm outperforms standard baselines in offline reinforcement learning in terms of full recovery rate and is computationally more efficient than tree search in cost-aware active decision making.
Figures & tables
Fig. 1 : (a) Problem setup. Agents sense different parts of the environment looking for OOIs. True OOIs are crossed in black. Targets detected by the agent in its field of view are crossed in red. (b) Modeling for cost-aware diffusion in active search. Agents sample lookahead action sequences of length H conditioned on the current belief state and with gradient guidance from the estimated cumulative discounted return over the entire sequence.
Fig. 2 : Optimism bias in samples generated by diffuser [ 22 ] for active search. Top row in each block (for T=0…5 ) is the posterior mean estimate (darker shade implies higher value) and bottom row indicates true target location and the FOV of the sensing action. Diffusion modeling over joint state-action space with gradient guided sampling of state-action sequences learns to generate deterministic target detection after the first generated action. In other words, generated samples are biased to assume the target will be detected in the first timestep. Such sampled action sequences do not balance exploration-exploitation, explaining why diffuser is unable to generate optimal coverage action sequences for active search.
Fig. 3 : Lookahead vs myopic decision making in 1D search space of size n=16 . J=1 agent. k=1 target. Low ( σ=161 ) and high ( σ=0.2 ) observation noise. Our diffusion based approach (D-AS) recovers the optimal sequence of actions and achieves full recovery faster than the myopic and shallow search tree based baselines. Plots show mean and standard error over 20 trials.
Fig. 4 : Cost-aware decision making in 1D search space of size n=16 . J=1 agent. k=1 target. Low ( σ=161 ) and high ( σ=0.2 ) observation noise. Our diffusion based approach called CD-AS incurs smaller or competitive total cost compared to CAST when sensing cost is higher than traveling ( cs=50 s). But when sensing cost is low, CAST selects sensing actions which incur a smaller total cost. Plots show mean and standard error over 20 trials.
Fig. 5 : Lookahead vs. myopic decision making in 2D search space of size 8×8 . J=1 agent. k=1 target. Low ( σ=161 ) and high ( σ=0.2 ) observation noise. Our diffusion based approach (D-AS) samples the optimal sequence of actions and achieves full recovery with fewer measurements compared to myopic active search (EIG), offline RL (IQL), behavior cloning based diffusion (diffusion policy) and shallow online tree search (CAST) baselines. Plots show mean and standard error over 10 trials.
Fig. 6 : Cost-aware decision making in a 2D search space of size 8×8 . J=1 agent. k=1 target. σ=161 . Our diffusion based approach CD-AS incurs a higher total cost in seconds compared to CAST which combines cost-optimization with tree search based planning. Plots show mean and standard error over 10 trials.
Fig. 7 : Lookahead vs myopic decision making in 2D 8×8 search space. J=3 agents. k=4 targets. σ=161 . Our diffusion based approach recovers the optimal sequence of actions for full recovery faster compared to shallow lookahead multi-agent active search with CAST. Plots show mean and standard error over 10 trials.
Fig. 8 : Cost-aware decision making in 2D search space of size 8×8 . J=3 agents. k=4 targets. σ=161 . Our diffusion based approach CD-AS incurs a higher total cost compared to CAST which combines ground truth cost-awareness with search tree based planning. Plots show mean and standard error over 10 trials.
nℓ×nw
CAST
CD-AS
1×16
120.40
78.68
8×8
405.47
132.75
TABLE I : Comparing the average wall-clock time in seconds per decision making step with CAST and CD-AS in both 1D and 2D search spaces. CD-AS amortizes the time complexity of rolling out state-action sequences in tree search and leads to more than 30% improvement per lookahead decision step.
Compositional diffusion models offer a promising route to long-horizon planning by denoising multiple overlapping sub-trajectories while ensuring that together they constitute a global solution. However, enforcing local behavior over long chains is often insufficient for a coherent global structure to emerge. Recent works tackle this limitation through intrinsic search, which explores multiple paths during the denoising process. While intrinsic search improves global coherence, it comes at the cost of repeated evaluations of an already compute-heavy model. In this work, we argue that extrinsic search, performed outside the denoising process, offers a more effective mode of exploration for long-horizon planning while naturally enabling the use of classical algorithms to solve unseen combinatorial tasks at test time. Our eXtrinsic search-guided Diffuser (XDiffuser) first computes a plan over a state-space graph -- serving as a lightweight local connectivity oracle for the diffusion model. The plan is then used to guide denoising for a single trajectory, effectively offloading the burden of exploration. XDiffuser outperforms diffusion-based baselines on long-horizon tasks, with particularly large gains in the low-quality data regime and on unseen tasks beyond goal-reaching, including multi-agent coordination and TSP-style reasoning. Project website: https://yanivhass.github.io/XDiffuser-site/
Visual active search (VAS) has been introduced as a modeling framework that leverages visual cues to direct aerial (e.g., UAV-based) exploration and pinpoint areas of interest within extensive geospatial regions. Potential applications of VAS include detecting hotspots for rare wildlife poaching, aiding search-and-rescue missions, and uncovering illegal trafficking of weapons, among other uses. Previous VAS approaches assume that the entire search space is known upfront, which is often unrealistic due to constraints such as a restricted field of view and high acquisition costs, and they typically learn policies tailored to specific target objects, which limits their ability to search for multiple target categories simultaneously. In this work, we propose DiffVAS, a target-conditioned policy that searches for diverse objects simultaneously according to task requirements in partially observable environments, which advances the deployment of visual active search policies in real-world applications. DiffVAS leverages a diffusion model to reconstruct the entire geospatial area from sequentially observed partial glimpses, which enables a target-conditioned reinforcement learning-based planning module to effectively reason and guide subsequent search steps. Extensive experiments demonstrate that DiffVAS excels in searching diverse objects in partially observable environments, significantly surpassing state-of-the-art methods on several datasets.
Active multi-target tracking requires a mobile robot to balance exploration for undetected targets with exploitation of uncertain tracked ones. Diffusion policies have emerged as a powerful approach for capturing diverse behavioral strategies by learning action sequences from expert demonstrations. However, existing methods implicitly select among strategies through the denoising process, without uncertainty quantification over which strategy to execute. We formulate expert selection for diffusion policies as an offline contextual bandit problem and propose a Bayesian framework for pessimistic, uncertainty-aware strategy selection. A multi-head Variational Bayesian Last Layer (VBLL) model predicts the expected tracking performance of each expert strategy given the current belief state, providing both a point estimate and predictive uncertainty. Following the pessimism principle for offline decision-making, a Lower Confidence Bound (LCB) criterion then selects the expert whose worst-case predicted performance is best, avoiding overcommitment to experts with unreliable predictions. The selected expert conditions a diffusion policy to generate corresponding action sequences. Experiments on simulated indoor tracking scenarios demonstrate that our approach outperforms both the base diffusion policy and standard gating methods, including Mixture-of-Experts selection and deterministic regression baselines.
Haotian Xiang, Qin Lu, Yaakov Bar-Shalom
School of ECE, University of Georgia, Athens, GA 30602, USA · Dept. of ECE, University of Connecticut, Storrs, CT 06269, USA