Multi-Agent Object Detection Framework Based on Raspberry Pi YOLO Detector and Slack-Ollama Natural Language Interface
Authors: Vladimir Kalušev, Branko Brkljač, Milan Brkljač
Organizations: The Institute for Artificial Intelligence Research and Development of Serbia Fruškogorska 1, 21000 Novi Sad, Republic of Serbia · Department of Power, Electronic and Telecommunication Engineering Faculty of Technical Sciences, University of Novi Sad Trg Dositeja Obradovića 6, 21000 Novi Sad, Republic of Serbia · Faculty of Finance, Banking and Auditing, Alfa BK University Bulevar maršala Tolbuhina 8, 11070 Belgrade, Republic of Serbia
The paper presents design and prototype implementation of an edge based object detection system within the new paradigm of AI agents orchestration. It goes beyond traditional design approaches by leveraging on LLM based natural language interface for system control and communication and practically demonstrates integration of all system components into a single resource constrained hardware platform. The method is based on the proposed multi-agent object detection framework which tightly integrates different AI agents within the same task of providing object detection and tracking capabilities. The proposed design principles highlight the fast prototyping approach that is characteristic for transformational potential of generative AI systems, which are applied during both development and implementation stages. Instead of specialized communication and control interface, the system is made by using Slack channel chatbot agent and accompanying Ollama LLM reporting agent, which are both run locally on the same Raspberry Pi platform, alongside the dedicated YOLO based computer vision agent performing real time object detection and tracking. Agent orchestration is implemented through a specially designed event based message exchange subsystem, which represents an alternative to completely autonomous agent orchestration and control characteristic for contemporary LLM based frameworks like the recently proposed OpenClaw. Conducted experimental investigation provides valuable insights into limitations of the low cost testbed platforms in the design of completely centralized multi-agent AI systems. The paper also discusses comparative differences between presented approach and the solution that would require additional cloud based external resources.
Figures & tables
Figure 1: Developed fast prototyping hardware testbed: (a) custom Raspberry Pi camera setup, (b) MIPI CSI camera board with Sony IMX219 color sensor.
Figure 2: Multi-agent system architecture: functional elements of the proposed object detection framework and implemented agentic components.
2.4 GHz and 5.0 GHz IEEE 802.11ac wireless, Bluetooth 5.0
Ethernet
Gigabit Ethernet
USB ports
2 USB 3.0 ports; 2 USB 2.0 ports
GPIO
Raspberry Pi standard 40 pin GPIO header
Table 1: Hardware testbed specifications.
Specification
Details, Raspberry Pi module v2
Resolution
8 megapixels
Max Video
1080p47
Still Modes
1640 × 1232p41; 640 × 480p206
Sensor
Sony IMX219
Sensor Resolution
3280 × 2464 pixels
Table 2: Camera board specifications.
Figure 3: Software components implementing proposed multi-agent architecture in Fig. 2 : (a) modules structure, (b) code snippet from "main.py" illustrating system initialization phase with reporting and communication agent subscription to agent specific router events.
Figure 4: Illustration of the implemented functionalities: (a) vision agent’s object tracking, and (b) report agent’s asynchronous LLM system call.
Figure 5: Characteristics of the messaging application: socket mode communication, message events subscription and the bot token scopes defining what functionality communication and control agents can ask for from the chatbot instance.
Figure 6: Multi-agent system operation with user interaction through created natural language interface based on the Slack channel and Ollama event report generation. Both notification and control commands are provided through familiar messaging application, while the object detection and tracking events are triggering combined image and text system responses.
Figure 7: Illustration of system configuration without LLM based reporting agent: (a) object tracking and video encoding, (b) CPU load and memory usage.
Robust human-robot interaction in complex environments requires accurate gesture perception, semantic scene understanding, and reliable task planning under limited onboard computing resources. This paper presents a cloud-edge multimodal interaction framework that integrates an enhanced YOLO-based gesture detector with coordinated large language model (LLM) and vision-language model (VLM) agents. The proposed detector, incorporates the Convolutional Block Attention Module (CBAM) into the neck and replaces the baseline bounding-box regression objective with Distance-IoU (DIoU) loss. These modifications improve feature discrimination and localization for small or partially occluded gestures in complex backgrounds. The cloud layer performs gesture detection, scene understanding, multimodal fusion, and action planning, whereas the TonyPi robot locally handles data acquisition, communication, action execution, and feedback. Experiments on a public gesture dataset and a custom dataset show that YOLO-DC achieves precision values of 98.9% and 95.0%, with [email protected] values of 90.7% and 92.7%, respectively. System-level evaluation yields success rates of 95%, 88%, and 82% for single-action, composite-action, and vision-dependent tasks. A 30 participant evaluation yields an overall mean satisfaction score of 3.69 out of 5. These results demonstrate the feasibility of combining refined gesture detection with multimodal agents for resource-constrained robotic interaction.
Recent surges in LLM-driven intelligent systems largely overlook decades of foundational multi-agent systems (MAS) research, resulting in frameworks with critical limitations such as centralization and inadequate trust and communication protocols. This paper introduces the Fetch.ai architecture, an industrial-strength platform designed to bridge this gap by facilitating the integration of classical MAS principles with modern AI capabilities. We present a novel, multi-layered solution built on a decentralized foundation of on-chain blockchain services for verifiable identity, discovery, and transactions. This is complemented by a comprehensive development framework for creating secure, interoperable agents, a cloud-based platform for deployment, and an intelligent orchestration layer where an agent-native LLM translates high-level human goals into complex, multi-agent workflows. We demonstrate the deployed nature of this system through a decentralized logistics use case where autonomous agents dynamically discover, negotiate, and transact with one another securely. Ultimately, the Fetch.ai stack provides a principled architecture for moving beyond current agent implementations towards open, collaborative, and economically sustainable multi-agent ecosystems.
Michael J. Wooldridge, Attila Bagoly, Jonathan J. Ward +2
Recent advances in artificial intelligence (AI) have enabled effective perception and language models for robots, but their deployment remains computationally expensive, increasing latency and energy use. This work presents the Open Robotics Inference and Control Framework (ORICF), a modular, declarative, and model-agnostic platform for composing multimodal robotic inference pipelines. ORICF integrates input/output (I/O) adapters, pluggable inference back ends, and post-processing logic, while lightweight YAML specifications allow models, hardware targets, and data channels to be changed without code modification. The framework also supports edge offloading, i.e., executing inference on nearby external computers instead of onboard the robot. ORICF is evaluated on a mobile robot that answers spoken queries about people detected in its camera stream by combining automatic speech recognition (ASR), a large language model (LLM), and a convolutional neural network (CNN) detector through Robot Operating System 2 (ROS2). Compared with onboard execution, ORICF-based edge deployment reduces robot-side compute utilization by up to 83.16% and estimated energy consumption by 65.8%, while preserving modularity and reproducibility.
Andrés Meseguer Valenzuela, Luís Miguel Bartolín Arnau
Instituto Tecnológico de Informática (ITI), Paterna, Spain