PAANI : On Device Visual Evidence Fusion and Explainable Guidance for River Robot Simulation

Savio Cardoz; Santhiya Rajan

cs.AISep 17, 2026

PAANI : On Device Visual Evidence Fusion and Explainable Guidance for River Robot Simulation

Authors: Savio Cardoz, Santhiya Rajan

Organizations: ACL Digital, Pune · Multiverse Computing, Spain · Ph.D. (Part-time) Research Scholar, PSG College of Technology, Coimbatore, India

Abstract

Mobile river monitoring robots must interpret obstacles and water boundaries that geographic waypoints alone cannot describe. On resource constrained platforms, converting imperfect visual predictions into timely and inspectable guidance is a distinct challenge. An object label or steering command does not explain which evidence supports a decision or when that evidence is unreliable. We present PAANI, an on-device perception to guidance architecture that combines a project trained YOLO11n detector and a custom MobileNetV3 Small semantic segmenter with timestamp aligned evidence fusion on Arduino UNO Q. Bounded tracking supplies object persistence, while an explicit corridor policy combines surface labels, accepted detections, urgency and mask uncertainty. Each final advisory exposes its contributing evidence and policy reasons. ROS 2 interfaces connect the local AI pipeline to a separate Gazebo vessel, localization and control testbed. Training uses 10,000 WaterScenes images for four-class detection and 1,127 MaSTr1325 images for segmentation, including 198 segmentation validation images. The selected FP32 ONNX models occupy 14.817 MB. Detector checkpoint test mAP at 0.5 IoU is 0.7388, while the separately evaluated rectangular ONNX export achieves validation mAP at 0.5 IoU of 0.7367. Segmentation ONNX validation mIoU is 0.9750. A five-minute UNO Q recording produced median and 95th percentile pipeline latencies of 467.8 ms and 580.3 ms at a configured 0.5 Hz cadence. The evaluation also identifies black input misclassification and a sampling rate mismatch that prevents the diagnostic apparent motion estimator from collecting sufficient evidence. These results support an inspectable and reusable edge robotics foundation while clearly distinguishing model accuracy and on-board execution from validated on-water collision avoidance.

Figures & tables

Figure 1: UNO Q infrastructure mapping. Linux inference and ROS communication are separate from the sketch’s indication role. The App Lab workflow does not imply use of a managed AI Brick.

Figure 2: Overall compute and robotics partition. The camera’s independent footage is not rendered from the simulated pose. The board executes vision and guidance; the external host supplies simulation and sensor feedback. MCU indication is distinct from propulsion control.

Interface	Producer and content	Consumer / purpose
Camera image stream	Camera; image + ROS header	Latest-frame inference; valid calibration is not established
Frame-timing stream	Camera/replay; source time, epoch	Matched time base for diagnostic motion
Perception evidence	Perception; boxes, lane summaries, timing	Tracking and visual policy
Tracked observations	Tracker; IDs, acceptance, motion fields	Dashboard and available journal subscribers
Explained advisory	Advisor; direction, reasons, components	Navigator, dashboard and guidance bridge
Navigation command	Navigator; atomic heading/speed/enable	Heading/speed controller

Table 1: Principal data interfaces in the selected graph. JSON is carried in ROS String messages.

Property	Detector	Segmenter
Architecture	YOLO11n bounding-box detector	MobileNetV3-Small + 64-channel light FPN
Training framework	PyTorch / Ultralytics	PyTorch / torchvision
Class order	person, vessel, pier, buoy	water, obstacle, sky
Deployment	FP32 ONNX opset 17	FP32 ONNX opset 17
Input	$[1,3,256,320]$	$[1,3,256,320]$
Output	$[1,8,1680]$ raw predictions	$[1,3,256,320]$ logits

Table 2: Selected learned models. ONNX is a serialization format; ONNX Runtime is the board inference engine. Decimal MB are file sizes, not RAM.

Figure 3: High-level AI architecture. The two learned branches express dependencies, not concurrent execution. The new apparent-motion states are a diagnostic side branch. Legacy urgency remains an input to the existing risk model. Detailed contracts appear in Tables 1 and 2 .

Split	Images	Person	Vessel	Pier	Buoy
Training	10,000	5,594	9,484	28,301	11,538
Validation	10,824	1,609	11,254	24,334	3,320
Test	5,412	835	5,679	12,186	1,680

Table 3: Prepared WaterScenes sample and annotation counts. The selected subset is not the entire public dataset.

Figure 4: Complete detector training history from 60 recorded epochs. Accuracy and loss are dataset-fitting diagnostics, not closed-loop navigation metrics. The evaluation geometry of the selected rectangular export is reported separately.

Figure 5: Complete 50-epoch segmentation history, with epoch labels converted from zero-based logs to one-based display. No validation-loss series is inferred. High water/sky IoU should be considered together with obstacle and boundary quality.

Artifact	Split	[email protected]	[email protected]:0.95
Selected PyTorch checkpoint	Validation	0.743199	0.463867
Selected PyTorch checkpoint	Test	0.738781	0.457888
Rectangular FP32 ONNX	Validation	0.736668	0.456929

Table 4: Detection results. Separate artifacts and evaluation splits are not interchangeable.

Class	Checkpoint val.	Checkpoint test	ONNX val.
Person	.366075	.365398	.359960
Vessel	.609288	.609214	.599580
Pier	.421294	.412297	.418510
Buoy	.458810	.444641	.449664

Table 5: Per-class detector [email protected]:0.95. The results reveal category variation hidden by a single aggregate.

Artifact	mIoU	Water IoU	Obstacle IoU	Sky IoU	Edge F1
PyTorch	.975038	.991244	.938622	.995249	.876991
FP32 ONNX	.975037	.991245	.938620	.995246	.876919
INT8 ONNX	.730281	Not used for deployment			.231788

Table 6: Segmentation validation on 198 images. INT8 is the rejected PTQ candidate.

Figure 6: Per-stage sample outputs for WaterScenes image 01703. A–C are saved image/model outputs; D–F render matching recorded telemetry. The mask is predicted, not ground truth. The Windows-host replay used threshold 0.25, eight repeated still observations and controlled timestamps; the board uses threshold 0.40. Neither physical motion nor field generalization is established.

Input	L	C	R	Advice	Interpretation
47266	.3413	.2941	.1611	STRAIGHT	Center below blocking threshold
01703	.4976	.5114	.4573	RIGHT	Center blocked; lower-risk permitted right side
27011	.3367	.4716	.6280	STRAIGHT	Center remains below 0.48
47357	.6615	.6025	.4489	STOP	Occupancy prevents a permitted safer turn
Black	.0288	.0004	.0029	STRAIGHT	Unusable-input failure, not safe water

Table 7: Final historical offline decisions. L/C/R are corridor scores, not probabilities. All five stale probes produced STOP after the recorded 9.3-s age.

Figure 7: Saved multi-object, occupied-corridor and black-input examples. Cyan denotes predicted water, orange obstacle/environment, and dark blue sky. These selected illustrations are not a held-out qualitative test set. The last row exposes an invalid-input failure rather than a successful negative control.

Metric	Value
Polling samples / distinct inference results	300 / 151
Pipeline latency median / p95 / maximum (ms)	467.842 / 580.341 / 643.857
Detector median / segmenter execution median (ms)	254.408 / 165.593
Effective inference cadence (Hz)	0.500
Measured capture rate, approximately (FPS)	2.58
Frame age at completion median / p95 (s)	0.6838 / 0.8888

Table 8: Recorded five-minute two-thread UNO Q benchmark. Timing and thermal observations do not constitute power or long-duration qualification.

Figure 8: Replotted raw operational telemetry. Inference/age use 151 distinct results; CPU and temperature use 300 polls. Detector timing includes its runtime preprocessing/decoding, while segmenter timing is the recorded session execution interval; their medians need not sum to total pipeline median.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Epochs / batch / workers	60 / 32 / 4
Initialization / input	Pretrained YOLO11n; square 320 training input
Optimizer / initial LR / weight decay	AdamW / 0.001 / 0.0005
Cosine final LR fraction / warmup	0.01 / 3 epochs
Momentum / warmup momentum	0.937 / 0.8
Warmup bias LR / nominal batch	0.1 / 64

Appendix

Table 9: Recorded detector optimization and augmentation configuration. Numeric settings are not a claim of a tuned optimum.

Setting	Value
Segmenter epochs / batch / workers	50 / 16 / 4
Decoder / input	64 channels; $320\times 256$
Optimizer / LR / weight decay	AdamW / 0.0003 / 0.0001
Schedule / loss	Cosine $T_{\max}=50$ ; CE + Dice (coefficient 1)
Ignore label / seed / AMP	255 / 42 / enabled
Training loader	Shuffled; incomplete final batch dropped

Appendix

Table 10: Recorded segmentation configuration and selected runtime thresholds.

Setting	Value / scope
Mass / equilibrium height	12.4 kg / 0.177 m; uncalibrated model
Heave stiffness / damping	650 / 90; simulator coefficient units
Roll-pitch restoring / damping	90 / 24; simulator coefficient units
Surge linear / quadratic drag	4 / 20
Sway linear / quadratic drag	20 / 80
Yaw linear / quadratic drag	4 / 25

Appendix

Table 11: Provisional robotics parameters. Metric-prototype parameters are not active camera-simulation evidence.

Robotics

Robotics studies the design and control of machines that perceive, move, and act in physical environments.

See all 755 papers

Oct 2, 2026

Lightweight and Resource-Efficient Perception for Robotic Guide Dogs

Robotic guide dogs should understand their surroundings, objects, and potential risks. Prior research has focused on raw sensor data from cameras and 2D or 3D LiDAR, which precisely measure distance points rather than provide a semantic understanding of the scene. While these physical measurements are effective for robot-centric collision avoidance and robot safety, they are not suitable for human-centric guidance. The system should recognize the type and relevance of obstacles and explain them, clearly and actionably, in terms of their spatial relation to the user. We present complete on-device perception modules that fuse a 360 camera and a 2D LiDAR for reliable collision avoidance, with moving-object detection and tracking for human-centric guidance. Finally, in walking-impossible situations, a vision--language model delivers pathway explanations as a safety mechanism to reduce user anxiety. In experiments, verification of fused 360 camera--LiDAR depth shows reliable near-range perception but inherent mid-range bias, while the system as a whole sustained real-time performance under 55 W. On the real-world egocentric GuideDogQA benchmark, our system achieved 83.8% accuracy, compared with 67.1% for GPT-4o. These results demonstrate that practical human-centric guidance with real-time on-device inference is feasible even on quadrupeds.

Jinse Kwon, Yoojin Lim, Choonghan Lee +3

Electronics and Telecommunications Research Institute (ETRI), Korea · School of Computer Science and Engineering, Pusan National University, Korea · Division of Electronics and Information Engineering, Jeonbuk National University, Korea

Camera-LiDAR Fusion Assistive Robotics Robot Collision Avoidance

Ask PDF

Oct 1, 2026

UniTrackPLA: Unified Panorama-Language-Action Model for Instruction-Guided Navigation and Dynamic Person Tracking

General-purpose embodied robots should support both navigation toward language-specified destinations and dynamic person tracking under arbitrary initial target azimuths. However, existing methods typically rely on forward-facing observations and address these tasks with separate policies, limiting omnidirectional perception and unified closed-loop control. We present UniTrackPLA, a unified panorama-language-action model for instruction-guided navigation and dynamic person tracking. Its Panoramic-Aware Encoding (PAE) preserves the temporal and azimuthal structure of perspective views projected from each panorama, enabling perspective-pretrained visual encoders to process omnidirectional observations. A shared vision-language backbone grounds instructions in the panoramic context and predicts continuous robot-centric waypoint chunks for both tasks. World-Action Consistency (WAC) further predicts action-conditioned future visual states and verifies waypoint prefixes online, allowing reliable actions to be reused while triggering replanning upon inconsistency. We also introduce OmniTrackNav-Bench, comprising 5,000 simulated tracking trajectories, 10,000 simulated VLN routes, and 96 verified real-world routes, providing 919,978 waypoint-supervision instances. UniTrackPLA improves overall tracking SR from 23.50% to 35.00% and Omni-VLN SR/SPL from 13.00%/12.77% to 19.75%/19.29%. Incorporating 76 real-world routes further improves held-out [email protected] from 42.92% to 92.08%. Closed-loop experiments on a Go2-W robot demonstrate unified panoramic tracking and navigation across indoor and outdoor environments. The project page is at https://tw5775.github.io/UniTrackPLA.

Pengfei Qi, Haoran Lin, Sizhuang Chen +7

College of Integrated Circuits, Hunan University, Changsha, China · School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha, China · School of Advanced Technology, Xi’an Jiaotong-Liverpool University, Suzhou, China +1

Visual Object Tracking Robot Navigation Vision-Language Navigation VLMs for Robotics

Ask PDF

Jun 1, 2026

Tiny Collaborative Inference for Occlusion-Robust Object Detection

Edge AI nodes for search and rescue are increasingly expected to run computer vision locally, yet ultra-low-end hardware imposes hard constraints on memory, compute, and inter-device communication. This work addresses occlusion-robust object detection on devices with less than 1 MB SRAM by combining an MCUNet backbone, a YOLOv2 detection head, and Lite quantisation. Two collaborative inference strategies are evaluated: feature-level fusion, concatenating intermediate feature maps, and decision-level fusion via Weighted Boxes Fusion (WBF). WBF outperforms feature-level fusion under all tested occlusion conditions, yielding gains of up to +0.2736 mAP in asymmetric scenarios. Extending fusion to three views improves accuracy further (up to +0.3827 mAP) at modest communication overhead (~1.3 KB per exchange). Hardware experiments progress from a host-assisted USB-relay baseline to a Wi-Fi peer-to-peer deployment on two Coral Dev Board Micro units, where WBF executes on-device with negligible communication energy relative to inference. In a 301.9 s autonomous session of 108 frames, fused output is produced on 61 frames versus 47 for a single board - a coverage gain of +29.8%. A decentralised federated learning feasibility note is included but not treated as a primary result, as performance remains limited under non-iid data. The results support decision-level fusion as a viable option for improving occlusion robustness in small-scale edge object detection, including host-free multi-board operation on ultra-low-end hardware.

Chieh-Tung Cheng, Mustafa Aslanov, Eiman Kanjo

Imperial College London, United Kingdom · Nottingham Trent University, United Kingdom

Edge Inference Efficient Object Detection Object Detection

Ask PDF

Abstract

Robotics

Explore similar work

Lightweight and Resource-Efficient Perception for Robotic Guide Dogs

UniTrackPLA: Unified Panorama-Language-Action Model for Instruction-Guided Navigation and Dynamic Person Tracking

Tiny Collaborative Inference for Occlusion-Robust Object Detection