An Integrated Open-Source Stack for Vision-Based Autonomous Navigation of Wheeled Rovers

Iaroslav Kolomiets; Iaroslav Sheipak; Mikhail Reppo; Vsevolod Egorov; Viktor Chertopolokhov; Isaac Chairez

cs.RONov 5, 2026

An Integrated Open-Source Stack for Vision-Based Autonomous Navigation of Wheeled Rovers

Authors: Iaroslav Kolomiets, Iaroslav Sheipak, Mikhail Reppo, Vsevolod Egorov, Viktor Chertopolokhov, Isaac Chairez

Organizations: MSU Rover Team · Faculty of Mechanics and Mathematics, Moscow State University, Moscow, Russia · Department of Mechatronics and Automation, Tecnologico de Monterrey, Mexico

Abstract

Vision-based rover autonomy requires integrating perception, localization, planning and platform-specific control under limited sensing and computing resources. We present an open-source ROS 2 stack for following directional arrows, reaching a terminal cone and returning autonomously to the start. It connects YOLOv8n detection and geometric direction, range and bearing estimation with RGB-D SLAM, wheel-inertial state estimation, kinematics-aware planning, behavior-tree mission management and STM32 actuation. Standard interfaces separate reusable autonomy from the hardware layer; a proportional path-following plugin supports low-speed rover motion. Public resources include software, configuration, a labeled dataset, trained weights and evaluation logs. On 478 held-out images, the detector achieves [email protected] of 0.720 under the Ultralytics evaluator and outperforms a classical OpenCV baseline under a shared evaluation protocol. The reported perception latency is 25 ms on a Jetson Orin NX. Three missions, one indoor and two outdoor, demonstrate marker-guided navigation and autonomous return without operator motion commands or reported collisions. Field results document integration failures and corrective choices; repeated instrumented trials remain necessary to quantify physical navigation errors and reliability.

Figures & tables

Panel of Figure 1 — Figure 1: Deployed perception-to-action stack. Boxes with a gray fill were developed by the authors; dashed boxes are reused open-source components; rounded boxes are sensors. The path-following controller is a proportional (P) controller; its displayed gain is from the inspected configuration (Sect. 2.2 ).

Layer	Component	Origin	Repository / file
Actuation	STM32 firmware: motor PID/STC loops, steering position control, encoders	developed	-DC-STM32
HAL	ROS 2 serial driver and steering-geometry node (Ackermann, point turn, strafe)	developed	-DC-ROS2/ eureka_movement_2
Sensing	IMU driver (WitMotion WT901)	developed	-IMU
Localization	Wheel odometry + IMU EKF; RGB-D SLAM	reused	robot_localization , RTAB-Map; -SLAM/ eureka_localization/ config/
Mapping	Occupancy grid from depth cloud	reused	RTAB-Map Grid/* params in -SLAM/.../ nav2tune.launch.py
Planning	Smac Hybrid-A* planner, costmaps, recoveries	reused	Nav2; -SLAM/ eureka_navigation/ config/nav2.yaml

Table 1: Components of the deployed stack, their origin and where they are configured.

Panel of Figure 2 — Figure 2: Validation-split evaluation. Left: $F_{1}$ –confidence curves of the deployed checkpoint (thick: all classes, maximum 0.81 at 0.258). Right: column-normalized confusion matrix at confidence 0.25 / IoU 0.45; each column sums to one over the predicted classes.

Metric	Value	Scope
[email protected] / [email protected]:0.95	0.826 / 0.612	validation split
[email protected] / [email protected]:0.95	0.720 / 0.532	held-out test split
Arrow [email protected], cone [email protected]	0.94, 0.71 / 0.74, 0.70	validation / test
Arrow P / R, cone P / R	0.92 / 0.89, 0.81 / 0.65	validation, conf. 0.258
Arrow-direction accuracy	not independently established	no verified aggregate reported
Range / bearing accuracy	not established over full operating range	near-range jitter test only

Table 2: Perception metrics of the deployed model. Navigation outcomes are reported per mission in Table 4 .

Detector	Class	[email protected]	Precision	Recall
YOLOv8n (deployed)	arrow	0.68	0.89	0.62
	cone	0.68	0.83	0.63
Classical CV	arrow	0.23	0.56	0.33
	cone	0.05	0.24	0.12

Table 3: Learned detector vs. hand-crafted OpenCV baseline on the same held-out test split (478 images), scored using one greedy IoU-0.5 matcher, with P/R at each detector’s best-

F_{1}

point. The same checkpoint and images yield different AP in Table 2 because that table uses the Ultralytics evaluator rather than this common matcher.

Panel of Figure 3 — Figure 3: (a) Field detection with class, distance and bearing overlay; (b) the rover in field configuration; (c) final outdoor campaign on unprepared terrain.

	M1 (indoor)	M2 (outdoor)	M3 (outdoor)
Date (session length)	25 Nov 2025 (2 h)	4 Dec 2025 (2 h, M2+M3)	4 Dec 2025
Conditions	indoor, dust, $>$ 25 ∘ C	grass, leaves, drizzle	grass, leaves, drizzle
Verified marker count	n/r	n/r	n/r
Max. marker distance	$\leq$ 3 m	up to 10 m	$\approx$ 10 m
Route length (goal polyline)	n/r	n/r	$\approx$ 17 m
Autonomous duration	n/r	n/r	n/r

Table 4: Per-mission summary (“n/r” = not recorded). The checklist score (Table 5 ) is listed apart from the navigation outcomes.

Criterion	Weight	M1	M2	M3
Recognition of all visual markers	20	20	20	20
Autonomous navigation, return, reusable map	20	16	19	20
Low-level motion control	20	20	20	20
Hardware reliability	20	20	20	20
Operator telemetry and video console	6	6	6	6
Real-time integration of all components	10	10	10	10

Table 5: Specification-based acceptance checklist (weights sum to 100). Deductions concern only the reusable route map: not exercised in M1, generated but not saved in M2.

Robot Navigation

Robot navigation studies how robots perceive, localize, plan, and move safely toward goals in static or changing environments.

See all 570 papers

The Unified Autonomy Stack: Toward a Blueprint for Generalizable Robot Autonomy
May 12, 2026Mihir Dharmadhikari, Nikhil Khedekar, Mihir Kulkarni +7Robotics Safe Robot Navigation
Autonomous Robots Lab, Norwegian University of Science and Technology (NTNU), Trondheim, Norway · Department of Engineering Cybernetics, Norwegian University of Science and Technology, Høgskoleringen 1, Trondheim 7034, Norway
CrossMaps: Confidence-Aware Open-Vocabulary Semantic Mapping for Rover Navigation
Jun 15, 2026Jan-Niklas Klein, Sona Ghahremani, Christian Medeiros Adriano +1Robot Navigation Semantic Mapping
Hasso Plattner Institute for Digital Engineering, Potsdam, Germany.
Multi-Task Visual Perception Network with LLM Conditioning for Autonomous Navigation
Sep 13, 2026Praveen Kumar, K. R. Guruprasad, Tushar SandhanRobot Navigation Visual Robot Navigation
Department of Electrical Engineering, Indian Institute of Technology Kanpur, India · Department of Mechanical Engineering, Indian Institute of Technology Kanpur, India

May 12, 2026J/K move · Enter open · S save

The Unified Autonomy Stack: Toward a Blueprint for Generalizable Robot Autonomy

Mihir Dharmadhikari, Nikhil Khedekar, Mihir Kulkarni, Morten Nissov+6

Autonomous Robots Lab, Norwegian University of Science and Technology (NTNU), Trondheim, Norway · Department of Engineering Cybernetics, Norwegian University of Science and Technology, Høgskoleringen 1, Trondheim 7034, Norway

Robotics Safe Robot Navigation Robot Motion Planning Robotic Perception

Open paper AskPDF arXiv

We introduce and open-source the Unified Autonomy Stack, a system-level solution that enables resilient autonomy across diverse aerial and ground robot morphologies. The architecture centers on three synergistic modules -- multi-modal perception, multi-behavior planning, and multi-layered safe navigation -- that together deliver comprehensive mission autonomy. The stack fuses data from LiDAR, radar, vision, and inertial sensing, enabling (a) robust localization and mapping through factor graph-based fusion, (b) semantic scene understanding, (c) motion and informative path planning through sampling-based techniques adaptive across spatial scales, as well as (d) multi-layered safe navigation both through planning on the online reconstructed map and deep learning-driven exteroceptive policies alongside last-resort safety filters using control barrier functions. The resulting behaviors include safe GNSS-denied navigation into unknown and perceptually-degraded regions, exploration of complex environments, object discovery, and efficient inspection planning. The stack has been field-tested and validated on both aerial (rotorcraft) and ground (legged) robots operating in a host of demanding environments, including self-similar and smoke-filled settings, with complex geometries and high obstacle clutter. These tests demonstrate resilient performance in challenging conditions. To facilitate ease of adoption, we open-source the implementation alongside supporting documentation, validation, and evaluation datasets https://github.com/ntnu-arl/unified_autonomy_stack. A video giving the overview of the paper and the field experiments is available at https://youtu.be/l8Su8OXsM-E.

Abstract

Robot Navigation

Explore similar work

The Unified Autonomy Stack: Toward a Blueprint for Generalizable Robot Autonomy

CrossMaps: Confidence-Aware Open-Vocabulary Semantic Mapping for Rover Navigation

Multi-Task Visual Perception Network with LLM Conditioning for Autonomous Navigation