Hybrid Belief Reinforcement Learning for Efficient Coordinated Spatial Exploration

Danish Rizvi; David Boyle

cs.LGMar 4, 2026

Hybrid Belief Reinforcement Learning for Efficient Coordinated Spatial Exploration

Authors: Danish Rizvi, David Boyle

Organizations: Imperial College London, UK

Abstract

Coordinating multiple autonomous agents to explore and serve spatially heterogeneous demand requires jointly learning unknown spatial patterns and planning trajectories that maximize task performance. Pure model-based approaches provide structured uncertainty estimates but lack adaptive policy learning, while deep reinforcement learning often suffers from poor sample efficiency when spatial priors are absent. This paper presents a hybrid belief-reinforcement learning (HBRL) framework to address this gap. In the first phase, agents construct spatial beliefs using a Log-Gaussian Cox Process (LGCP) and execute information-driven trajectories guided by a Pathwise Mutual Information (PathMI) planner with multi-step lookahead. In the second phase, trajectory control is transferred to a Soft Actor-Critic (SAC) agent, warm-started through dual-channel knowledge transfer: belief state initialization supplies spatial uncertainty, and replay buffer seeding provides demonstration trajectories generated during LGCP exploration. A variance-normalized overlap penalty enables coordinated coverage through shared belief state, permitting cooperative sensing in high-uncertainty regions while discouraging redundant coverage in well-explored areas. The framework is evaluated on a multi-UAV wireless service provisioning task. Results show 10.8% higher cumulative reward and 38% faster convergence over baselines, with ablation studies confirming that dual-channel transfer outperforms either channel alone.

Figures & tables

Ref.	Approach	UD	MA	NM	SU	SE
Krause et al. (2008) Krause et al. (2008)	Greedy Submodular placement	✓	✗	✗	✓	n/a
Wu et al. (2018) Wu et al. (2018)	Joint traj. & resource	✗	✓	✓	✗	n/a
Zeng et al. (2019) Zeng et al. (2019)	Convex Opt. Min. flight energy	✗	✗	✓	✗	n/a
Wang et al. (2022) Wang et al. (2022)	DRL traj. for MEC	✓	✓	✓	✗	✗
Cao et al. (2023) Cao et al. (2023)	Learning-based IPP	✓	✗	✓	✓	✓
Zhou et al. (2024) Zhou et al. (2024)	Heterogeneous MARL	✓	✓	✓	✗	✗

TABLE I : Summary of Related Work Highlighting Limitations. UD: Unknown Demand, MA: Multiple mobile Agents, NM: Non-Myopic Planning, SU: Spatial Uncertainty, SE: Sample Efficiency.

Fig. 1 : Multiple mobile agents spatial exploration over a discretized operational area with unknown demand, modeled via a spatial belief process. The illustrated scenario depicts UAVs as the instantiated agents, with further details provided in Section 4.

Symbol	Description	Symbol	Description
$N$	Number of agents	$T$	Mission horizon
$\mathcal{A}$	Service area	$\mathcal{G}$	Discretized grid
$\Delta$	Grid resolution	$G_{x},G_{y}$	Grid size
$\mathbf{p}_{i}(t)$	Position of agent $i$	$d_{\max}$	Max. displacement
$r_{c}$	sensing radius	$\mathcal{C}(t)$	Covered cells at $t$
$y_{c}(t)$	Observed event count	$N_{\text{req}}(t)$	Total requests

TABLE II : Summary of Key Notations

Cell History	$\tilde{\sigma}_{c}^{2}(t)$	Var.-Norm.	Fixed
Never visited	1.0	0.0	1.0
Recently visited	0.1	0.9	1.0
Stale (grown variance)	0.5	0.5	1.0

TABLE III : Comparison of overlap penalty strategies for a cell covered by two agents.

Fig. 2 : Overview of the proposed HBRL framework. Phase 1 employs LGCP-based belief inference and PathMI planning to guide information-driven exploration. Phase 2 performs policy optimization using Soft Actor-Critic (SAC), warm-started via dual-channel knowledge transfer: (i) belief state initialization and (ii) replay buffer seeding with LGCP-generated trajectories.

Parameter	Symbol	Value
Service area	$\mathcal{A}$	$2000\times 2000$ m 2
Grid resolution	$\Delta$	$20$ m
Grid dimensions	$G_{x}\times G_{y}$	$100\times 100$
Number of UAVs	$N$	$2$ – $4$
Coverage radius	$r_{c}$	$100$ m
Max. displacement per step	$d_{\max}$	$15$ m

TABLE IV : Simulation and Algorithm Parameters

Fig. 4 : Reward comparison between Pure LGCP, Pure RL, Behavior Cloning and HBRL frameworks.

Fig. 5 : Posterior Variance comparison between Pure LGCP, Pure RL and HBRL. Lower values indicate higher confidence in the inferred demand field.

Fig. 6 : Comparison of reward and episodes to reach Pure RL reward for all three transfer channel scenarios.

Fig. 7 : Effect of LGCP warm-start duration on SAC training. Different warm-start lengths determine the transition point from LGCP exploration (Phase-1) to SAC optimization(Phase-2).

Fig. 8 : Effect of PathMI planning horizon on final reward.

Fig. 9 : Learning performance comparison under varying numbers of UAVs. Increasing the number of agents improves overall reward but exhibits sub-linear scaling due to coordination overhead and redundant coverage

Fig. 10 : Comparison of reward under various overlap penalty scenarios.

Fig. 11 : Impact of temporal decay on HBRL performance: (a) reward convergence and (b) belief uncertainty evolution. The dashed line denotes the warm-start transition point.

Fig. 12 : Reward weight sensitivity analysis. (a) Effect of exploration weight $\omega_{2}$ on final reward with coordination weight fixed at $\omega_{3}=1.0$ . The shaded region indicates the optimal range $\omega_{2}\in[0.4,0.6]$ . (b) Effect of coordination weight $\omega_{3}$ on final reward with exploration weight fixed at $\omega_{2}=0.4$ . (c) Reward heatmap over the $(\omega_{2},\omega_{3})$ configuration space with $\omega_{1}=5$ . The star indicates the default configuration.

Fig. 13 : Training curves for different

p_{\mathrm{loss}}

values

Fig. 14 : Final Reward and Convergence vs.

p_{\mathrm{loss}}

Fig. 15 : Learned belief intensity maps at episode 100 and convergence for HBRL and Pure RL, compared against the ground truth

Fig. 16 : Strategy vs. Correlation

Fig. 17 : Strategy vs. Runtime

UAV Communication

See all 8 papers

Mar 3, 2026

Heterogeneous Agent Collaborative Reinforcement Learning

We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional homogeneous teacher-to-student transfer. Building on this problem, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO with double rollouts by an average of 3.6% while using only half the rollout cost.

Zhixia Zhang, Zixuan Huang, Gongxun Li +10

Beihang University · Bytedance China · Apple +2

Multi-Agent Collaboration Heterogeneous MARL Cooperative MARL Reinforcement Learning with Verifiable Rewards

Ask PDF

May 3, 2026

Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning

Cooperative multi-agent reinforcement learning (MARL) requires agents to discover joint strategies in a combinatorially large state-action space, yet effective coordination configurations are exceedingly rare. Intrinsic motivation, which augments task rewards with novelty bonuses, is a popular approach for driving exploration, but its effectiveness hinges on the exploration intensity

β

, where too large a value overwhelms the task signal and causes coordination collapse, while too small a value prevents discovery of rare strategies. We address two complementary challenges: adapting

β

globally over training, and allocating the exploration budget across agents whose intrinsic reward signals vary in reliability. Our framework combines a return-conditioned sigmoid schedule (RCB) for global intensity control with a per-agent Reward Signal Quality (RSQ) metric that concentrates the exploration budget on agents with reliable signals. The core insight is that agents receiving noisy intrinsic rewards should explore less aggressively, and this allocation can be determined automatically from signal-to-noise statistics. Successor Distance (SD), a quasimetric intrinsic reward, naturally produces distinguishable per-agent signal quality, completing the framework with convergence and ordering preservation guarantees. On seven cooperative benchmarks (MPE, SMAX, MABrax), our method achieves top-tier returns across all environments.

Dahyun Oh, Minhyuk Yoon, H. Jin Kim

Department of Aerospace Engineering, Seoul National University, Seoul, Republic of Korea

Multi-Agent Coordination Intrinsic Reward Methods Reinforcement Learning Intrinsic Motivation RL Exploration Cooperative MARL

Ask PDF

Mar 23, 2026

Human-Inspired Pavlovian and Instrumental Learning for Autonomous Agent Navigation

Autonomous agents operating in uncertain environments must balance fast responses with goal-directed planning. Classical MF RL often converges slowly and may induce unsafe exploration, whereas MB methods are computationally expensive and sensitive to model mismatch. This paper presents a human-inspired hybrid RL architecture integrating Pavlovian, Instrumental MF, and Instrumental MB components. Inspired by Pavlovian and Instrumental learning from neuroscience, the framework considers contextual radio cues, here intended as georeferenced environmental features acting as CS, to shape intrinsic value signals and bias decision-making. Learning is further modulated by internal motivational drives through a dedicated motivational signal. A Bayesian arbitration mechanism adaptively blends MF and MB estimates based on predicted reliability. Simulation results show that the hybrid approach accelerates learning, improves operational safety, and reduces navigation in high-uncertainty regions compared to standard RL baselines. Pavlovian conditioning promotes safer exploration and faster convergence, while arbitration enables a smooth transition from exploration to efficient, plan-driven exploitation. Overall, the results highlight the benefits of biologically inspired modularity for robust and adaptive autonomous systems under uncertainty.

Jingfeng Shan, Francesco Guidi, Mehrdad Saeidi +8

WiLab, University of Bologna, via dell’Università 50, 47521 Cesena, Italy · National Research Council of Italy (CNR), viale del Risorgimento 2, 40136 Bologna, Italy · University of Bologna, Department of Psychology “Renzo Canestrari”, 47521 Cesena, Italy

Reinforcement Learning Model-Based RL Safe RL Model-Free RL

Ask PDF

Abstract

UAV Communication

Explore similar work

Heterogeneous Agent Collaborative Reinforcement Learning

Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning

Human-Inspired Pavlovian and Instrumental Learning for Autonomous Agent Navigation