IoT-Brain: Grounding LLMs for Semantic-Spatial Sensor Scheduling
Organizations: University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · The Chinese University of Hong Kong · The Hong Kong University of Science and Technology
Abstract
Intelligent systems powered by large-scale sensor networks are shifting from predefined monitoring to intent-driven operation, revealing a critical Semantic-to-Physical Mapping Gap. While large language models (LLMs) excel at semantic understanding, existing perception-centric pipelines operate retrospectively, overlooking the fundamental decision of what to sense and when. We formalize this proactive decision as Semantic-Spatial Sensor Scheduling (S3) and demonstrate that direct LLM planning is unreliable due to inherent gaps in representation, reasoning, and optimization. To bridge these gaps, we introduce the Spatial Trajectory Graph (STG), a neuro-symbolic paradigm governed by a verify-before-commit discipline that transforms open-ended planning into a verifiable graph optimization problem. Based on STG, we implement IoT-Brain, a concrete system embodiment, and construct TopoSense-Bench, a campus-scale benchmark with 5,250 natural-language queries across 2,510 cameras. Evaluations show that IoT-Brain boosts task success rate by 37.6% over the strongest search-intensive methods while running nearly 2 times faster and using 6.6 times fewer prompt tokens. In real-world deployment, it approaches the reliability upper bound while reducing 4.1 times network bandwidth, providing a foundational framework for LLMs to interact with the physical world with unprecedented reliability and efficiency.
Figures & tables
| Knowledge Base Statistics | Value | ||
| Buildings / Floor Plans / Outdoor Segments | 33 / 161 / 53 | ||
| Total Topological Scenarios / Deployed Cameras | 7,832 / 2,510 | ||
| Query Dataset Statistics | Count (%) | ||
| Tier 1: Intra-Zone Perception | |||
| T1.a: Focal Scene Awareness | 1,433 (27.3%) | ||
| e.g., "verify activity near the door to room 5F 1" | |||
| Paradigm | TCR (%) | Latency (s) | Bandwidth (GB) | TFP (frames) |
| Static Scheduling | 3.61 | 403.42 | 0.138 | 179 |
| Naive Parallel | 65.64 | 927.99 | 0.540 | 704 |
| IoT-Brain (Ours) | 49.84 | 413.69 | 0.131 | 166 |
LLM Reasoning with Graphs
Large language model reasoning with graphs uses structured relationships to support multi-step inference, evidence retrieval, and planning.
See all 210 papers