An Affordable AI-Integrated Smart Cane for Multimodal Mobility Assistance of Visually Impaired Users

Ali Akarma; Adeel Ahmad; Toqeer Ali Syed

cs.AISep 12, 2026

An Affordable AI-Integrated Smart Cane for Multimodal Mobility Assistance of Visually Impaired Users

Authors: Ali Akarma, Adeel Ahmad, Toqeer Ali Syed

Organizations: AI Center, Faculty of Computer and Information Systems, Islamic University of Madinah, Madinah 42351, Saudi Arabia · AI V&V Lab, King Fahd University of Petroleum and Minerals, Dhahran 31261, Saudi Arabia

Abstract

Visual impairment affects over 2.2 billion people worldwide, yet conventional white canes cannot detect elevated hazards or provide semantic environmental context. Existing AI-assisted navigation systems typically rely on expensive hardware or cloud connectivity, limiting accessibility in resource-constrained settings. This paper presents an affordable ($88 USD), fully offline AI-integrated smart cane designed for multimodal mobility assistance on an ultra-low-power Raspberry Pi Zero 2W. The system fuses RGB vision sensing with Time-of-Flight (ToF) distance estimation, pairing an INT8-quantized SSD MobileNet V1 model with distance-aware vibrotactile feedback and real-time audio alerts. To ensure operational robustness on constrained hardware, a multiprocessing architecture isolates sensor acquisition, neural inference, and haptic feedback into independent processes with fail-safe sensing support. Experimental evaluation across indoor mobility scenarios demonstrates a macro-averaged F1-score of 0.82 (precision: 0.85, recall: 0.81), a mean end-to-end latency of 330,ms, and a peak power draw of 2.8,W. A preliminary usability study with 12 participants (SUS: 78.5, NASA-TLX) demonstrated positive user perception and enhanced obstacle awareness. The proposed prototype validates the feasibility of deploying privacy-preserving, edge-native assistive intelligence for cost-sensitive mobility assistance.

Figures & tables

System / Work	Primary Modality	Processing Platform	Typical Latency	Offline Operation	Estimated Cost	Primary Limitations
NavBelt [ 8 ]	Ultrasonic sensing	Microcontroller	$<$ 50 ms	Yes	$\sim$ $50	Limited semantic awareness and difficulty distinguishing obstacle context.
Tapu et al. [ 9 ]	Wearable camera-based vision	High-end smartphone	$\sim$ 200 ms	Partial	$\sim$ $800	Increased thermal load, ergonomic limitations, and battery drain during continuous operation.
Upadhyaya et al. [ 11 ]	RGB vision (YOLOv8)	Raspberry Pi 4	$\sim$ 250 ms	Yes	$\sim$ $200	Higher power consumption and larger hardware footprint compared to lightweight assistive systems.
OrCam MyEye [ 4 ]	Wearable vision sensing	Proprietary embedded hardware	$\sim$ 100 ms	Yes	$\sim$ $3,500–4,500	High commercial cost and limited accessibility in low-resource settings.
Cloud-Assisted Frameworks [ 10 ]	RGB + LiDAR	Edge-to-cloud infrastructure	$>$ 500 ms	No	Variable	Dependence on network connectivity and variable communication latency.
Proposed System	RGB + ToF fusion	Raspberry Pi Zero 2W	$\sim$ 330 ms	Yes	$\sim$ $88	Reduced frame rate and limited field-of-view under constrained hardware conditions.

TABLE I: Comparison of Representative Assistive Mobility Systems. Cost figures are approximate and sourced from cited literature.

Fig. 1: Overview of the proposed AI-integrated smart cane architecture. Sensing, inference, and assistive feedback are separated into isolated processes. Audio feedback ( espeak-ng ) is triggered directly by the inference process for low-latency verbal notifications, while haptic control operates as an independent daemon communicating via a lightweight tmpfs -based IPC mechanism. This separation enables continued vibrotactile feedback during inference slowdowns or partial subsystem interruptions.

Fig. 2: Sensor placement and spatial sensing coverage. The wider RGB field-of-view ( $62^{\circ}$ ) supports semantic scene awareness, while the narrower ToF beam ( $25^{\circ}$ ) provides localized depth estimation for nearby obstacle detection.

Distance Range	Pulse Duration (ms)	Pulse Interval (ms)	Alert Priority
$<$ 50 cm	500	100	Critical
50–80 cm	300	200	High
80–120 cm	200	400	Moderate
120–150 cm	100	800	Low
$>$ 150 cm	Audio only	—	Informational

TABLE II: Vibrotactile Feedback Encoding Scheme Based on Obstacle Proximity Distance

Subsystem	CPU Usage (%)	RAM (MB)	Power (W)
Background OS Services	7–10	105–120	0.7–0.9
Camera Capture + Preprocessing	12–18	40–50	0.3–0.5
TFLite INT8 Inference	60–70	170–190	1.1–1.3
ToF + Haptic Feedback Daemon	1–3	6–10	0.3–0.4
Peak Concurrent System Load	85–92	330–360	2.6–2.9

TABLE III: Approximate System Resource Profiling on Raspberry Pi Zero 2W During Continuous Operation

Fig. 3: Approximate end-to-end latency breakdown for a representative semantic detection cycle on the Raspberry Pi Zero 2W platform. Stage boundaries and widths are proportional to measured duration on the horizontal time axis (ms). Neural inference remains the dominant computational bottleneck, contributing approximately 120–140 ms of the total 300–350 ms pipeline latency.

Obstacle Class	Precision	Recall	F1-Score	FP per 100 neg.
Person	0.94	0.91	0.92	1.7
Chair	0.88	0.85	0.86	3.3
Desk / Table	0.86	0.82	0.84	4.2
Doorway	0.82	0.79	0.80	5.8
Potted Plant	0.75	0.68	0.71	7.5
Macro Average	0.85	0.81	0.82	—

TABLE IV: Semantic Obstacle Detection Performance Across Indoor Categories (N

\approx

120 per class, 600 total encounters)

Panel of Figure 8 — Fig. 4: Representative indoor corridor evaluation scenario illustrating semantic and spatial obstacle monitoring along the user walking path. The shaded cone represents the approximate RGB camera field-of-view during forward navigation.

Scenario	Avoidance Rate (95% CI)	Mean Reaction Time
Cluttered Corridor	93.3% [78.7%, 98.1%] (28/30)	1.1 s
Elevated Hazard	86.7% [70.3%, 94.7%] (26/30)	1.4 s
Dynamic Pedestrian	80.0% [62.7%, 90.5%] (24/30)	1.8 s

TABLE V: Mobility Assistance Evaluation Results (N=30 trials per scenario; 95% CI computed using Wilson score method)

Metric	Mean Score (95% CI)	Interpretation
SUS Score	78.5 ( $\pm$ 4.2) [71.0, 86.0]	Above average usability (threshold: 68)
NASA-TLX Mental Demand	45 / 100	Moderate workload
NASA-TLX Frustration	22 / 100	Relatively low frustration
Perceived Safety	4.2 / 5.0	Positive user perception

TABLE VI: Preliminary Usability Assessment Results (N=12 blindfold-assisted participants)

Panel of Figure 11 — Fig. 5: Illustrative failure pathway associated with rapid cane motion. Motion blur introduced by the rolling-shutter camera reduces semantic inference confidence below the 0.50 acceptance threshold, increasing dependence on short-range ToF-based obstacle awareness as a fallback mechanism.

Failure Scenario	Observed Impact and Current Mitigation
Rapid Cane Motion	Fast sweeping motion introduces motion blur and intermittent semantic detection instability. Partial mitigation is provided through independent ToF-based proximity sensing.
Low-Light Environments	RGB semantic inference performance degrades substantially under poor illumination conditions. Under these scenarios, the system relies primarily on short-range ToF sensing.
Reflective or Glass Surfaces	Specular reflections may distort ToF measurements and produce unstable distance estimates. This limitation remains only partially addressed in the current implementation.
Narrow Field-of-View	Obstacles outside the combined RGB-ToF sensing region may not be detected consistently during rapid directional changes.
Embedded Processing Constraints	Limited inference throughput ( $\sim$ 2–3 FPS) may introduce delayed semantic feedback during fast user movement or dynamic obstacle encounters.

TABLE VII: Observed Failure Modes and Deployment Limitations

Visually Impaired Navigation Assistance

See all 22 papers

VisionAId: An Offline-First Multimodal Android Assistant for People with Visual Impairment, Featuring Personalized Object Retrieval
Jul 2, 2026Cristian-Gabriel Florea, Stelian SpînuAssistive Technology for Blind and Low-Vision People Visually Impaired Navigation Assistance
The Military Technical Academy “Ferdinand I”, Bucharest, Romania
Assisted Spatial Cognition Through Vision-Language Models
Sep 11, 2026H. Riaz, J. B. Fernandez, I. Mills +3Visual Spatial Reasoning Assistive Technology for Blind and Low-Vision People
School of Electronic Engineering, Dublin City University, Dublin, Ireland · Insight Research Ireland Centre for Data Analytics, Dublin, Ireland · South East Technological University, Waterford, Ireland
VisionAssist: An Open-Source Smartphone Assistant for AI-Based Visual Accessibility
Jul 18, 2026Ayşe Özlem Çalışkan, Jordi Sanchez-RieraAssistive Technology for Blind and Low-Vision People
Institut de Robòtica i Informàtica Industrial (CSIC-UPC)

Jul 2, 2026J/K move · Enter open · S save

VisionAId: An Offline-First Multimodal Android Assistant for People with Visual Impairment, Featuring Personalized Object Retrieval

Cristian-Gabriel Florea, Stelian Spînu

The Military Technical Academy “Ferdinand I”, Bucharest, Romania

Assistive Technology for Blind and Low-Vision People Visually Impaired Navigation Assistance

Open paper AskPDF arXiv

Over 285 million people worldwide live with a visual impairment, for whom everyday tasks such as avoiding obstacles, locating personal belongings, recognizing familiar faces, or handling cash remain persistent obstacles to personal autonomy. Existing assistive applications are typically limited to recognizing predefined categories, depend heavily on cloud connectivity, or require dedicated hardware. We present VisionAId, an Android application that turns a commodity smartphone into a real-time visual assistant. The system integrates six on-device deep learning models (metric monocular depth estimation, instance segmentation, visual and facial embeddings, face detection, and a custom banknote detector) running entirely through ONNX Runtime, with an optional cloud large language model (Google Gemini Flash) used only for narrative scene description and automatic object labeling. A distinctive contribution is a few-shot pipeline for personal objects: the user photographs an object from several angles, and the system later locates that specific instance in the environment, guiding the user toward it with augmented-reality markers, spatial audio, and distance-proportional haptics. All feedback is multimodal (Romanian speech synthesis, voice commands, vibration). On a reference device (Samsung Galaxy S21 Ultra), INT8 quantization reduces depth latency from ~1200 ms to ~491 ms, the custom banknote detector reaches an mAP@50 of 0.986, and metric depth is calibrated to below 1 cm of error within 3 m.

Abstract

Visually Impaired Navigation Assistance

Explore similar work

VisionAId: An Offline-First Multimodal Android Assistant for People with Visual Impairment, Featuring Personalized Object Retrieval

Assisted Spatial Cognition Through Vision-Language Models

VisionAssist: An Open-Source Smartphone Assistant for AI-Based Visual Accessibility