Hilbert-Augmented Reinforcement Learning for Scalable Multi-Robot Coverage and Exploration

Tamil Selvan Gurunathan; Aryya Gangopadhyay

cs.ROFeb 23, 2026

Hilbert-Augmented Reinforcement Learning for Scalable Multi-Robot Coverage and Exploration

Authors: Tamil Selvan Gurunathan, Aryya Gangopadhyay

Abstract

We present a coverage framework that integrates Hilbert space-filling priors into decentralized multi-robot learning and execution. We augment DQN and PPO with Hilbert-based spatial indices to structure exploration and reduce redundancy in sparse-reward environments, and we evaluate scalability in multi-robot grid coverage. We further describe a waypoint interface that converts Hilbert orderings into curvature-bounded, time-parameterized SE(2) trajectories (planar (x, y, θ)), enabling onboard feasibility on resource-constrained robots. Experiments show improvements in coverage efficiency, redundancy, and convergence speed over DQN/PPO baselines. In addition, we validate the approach on a Boston Dynamics Spot legged robot, executing the generated trajectories in indoor environments and observing reliable coverage with low redundancy. These results indicate that geometric priors improve autonomy and scalability for swarm and legged robotics.

Figures & tables

Panel of Figure 1 — Fig. 1 : Flowcharts illustrating the integration of Hilbert index into the H-DQN (left) and H-PPO (right) architectures.

Method	4 Agents	8 Agents	12 Agents	16 Agents
DQN	0.676	0.729	0.721	0.589
H-DQN (ours)	0.831	0.882	0.890	0.895
PPO	0.712	0.760	0.772	0.683
H-PPO (ours)	0.871	0.904	0.917	0.927

TABLE I : Average coverage ratio across agent team sizes.

Method	4 Agents	8 Agents	12 Agents	16 Agents
DQN	3.477	2.964	3.142	4.173
H-DQN (ours)	2.395	2.184	2.071	1.948
PPO	2.962	2.816	2.742	3.354
H-PPO (ours)	1.985	1.891	1.841	1.768

TABLE II : Average redundancy (visits per unique cell) across agent team sizes.

Fig. 2 : Real-robot execution of the PPO policy on a Boston Dynamics Spot in a $10\,\mathrm{m}\times 10\,\mathrm{m}$ area (5 $\times$ 5 grid). Policy-generated waypoints (markers) are converted to discrete $\mathrm{SE}(2)$ commands (linear steps and $\pm 30^{\circ}$ turns) via the Spot SDK; the overlaid trace shows onboard odometry during execution. PPO covers all cells but exhibits additional revisits and a longer traversal compared to the Hilbert-guided variant in Fig. 3.

Fig. 3 : Real-robot execution of the H - PPO policy under the same conditions as Fig. 2 . Hilbert-index augmentation yields a locality-preserving sweep with fewer revisits and shorter coverage time. Planned waypoints align closely with the executed odometry, indicating high path fidelity at medium walking speed.

Fig. 4 : Average cumulative reward across different agent counts for DQN, H-DQN, PPO, and H-PPO. Hilbert-augmented agents consistently achieve higher reward, particularly as the team size increases.

Fig. 5 : Comparison between H-DQN and standard DQN across cumulative reward, coverage ratio, and redundancy. Hilbert-DQN shows superior scalability and efficiency, especially at higher agent counts.

Fig. 6 : Average cumulative reward across agent counts for H-PPO and its ablated variants. Both state augmentation and curve-guided sampling contribute to improved reward.

Fig. 7 : Coverage ratio comparison for H-PPO and ablations. Structured exploration improves spatial efficiency, while removing exploration bias degrades performance.

Fig. 8 : Redundancy across agent counts. H-PPO maintains low redundancy due to better coverage coordination, while ablated variants show increased overlap.

Multi-Robot Exploration

Multi-robot exploration coordinates robot teams to map unknown environments, find targets, and gather information efficiently and safely.

See all 54 papers

VORL-EXPLORE: A Hybrid Learning Planning Approach to Multi-Robot Exploration in Dynamic Environments
Mar 9, 2026Ning Liu, Sen Shen, Zheng Li +4Multi-Robot Exploration Multi-Robot Motion Planning
The University of Western Australia · The Chinese University of Hong Kong · Karlsruhe Institute of Technology +1
Mobility-Budget-Constrained Submodular Maximization for Distributed Semantic Multi-Robot Exploration
Oct 13, 2026Changxuan Yao, Sandilya Sai Garimella, Lu Gan +1Multi-Robot Exploration Informative Path Planning
The authors are with the Institute of Robotics and Intelligent Machines, Georgia Institute of Technology, Atlanta, GA 30332, USA.
Hybrid Belief Reinforcement Learning for Efficient Coordinated Spatial Exploration
Mar 4, 2026Danish Rizvi, David BoyleUAV Communication Reinforcement Learning
Imperial College London, UK

Mar 9, 2026J/K move · Enter open · S save

VORL-EXPLORE: A Hybrid Learning Planning Approach to Multi-Robot Exploration in Dynamic Environments

Ning Liu, Sen Shen, Zheng Li, Sheng Liu+3

The University of Western Australia · The Chinese University of Hong Kong · Karlsruhe Institute of Technology+1

Multi-Robot Exploration Multi-Robot Motion Planning Multi-Robot Task Allocation Robot Policy Learning

Open paper AskPDF arXiv

Hierarchical multi-robot exploration decouples frontier allocation from local navigation, making systems brittle in dynamic environments. Without awareness of execution difficulty, allocators may send robots toward frontiers whose routes are congested or unsafe, causing bottlenecks, oscillatory replanning, and redundant coverage. We propose VORL-EXPLORE, a hybrid learning and planning framework that makes exploration execution-aware through execution fidelity, an estimate of local navigability and execution reliability. This fidelity score is used to penalize risky or crowded frontiers in a fidelity-coupled Voronoi allocation objective with inter-robot repulsion, and to arbitrate between global

A^*

guidance and a reactive reinforcement learning policy during navigation. As a result, robots follow efficient global plans in reliable regions and rely more on safer local reactions in crowded or changing areas. The fidelity model is recalibrated using self-supervised pseudo-labels derived from recent progress and safety outcomes. Quantitative experiments on randomized dynamic grids and an ablation under severe traffic show higher success rates, shorter exploration lengths, and lower inter-robot coverage overlap. A Gazebo factory scenario is included as a deployment illustration. The source code is available at https://github.com/21ning/VORL-EXPLORE.

Abstract

Multi-Robot Exploration

Explore similar work

VORL-EXPLORE: A Hybrid Learning Planning Approach to Multi-Robot Exploration in Dynamic Environments

Mobility-Budget-Constrained Submodular Maximization for Distributed Semantic Multi-Robot Exploration

Hybrid Belief Reinforcement Learning for Efficient Coordinated Spatial Exploration