cs.CVFeb 14, 2025

MITO: A Millimeter-Wave Dataset and Simulator for Non-Line-of-Sight Perception

Authors: Tara Boroushaki, Laura Dodds, Cusuh Ham, Fadel Adib

Organizations: Massachusetts Institute of Technology · Adobe Research · Cartesian Systems

Abstract

The ability to observe the world is fundamental to reasoning and making informed decisions on how to interact with the environment. However, optical perception can often be disrupted due to common occurrences like occlusions, which can pose challenges to existing vision systems. We present MITO, the first millimeter-wave (mmWave) dataset of diverse, everyday objects, collected using a UR5 robotic arm with two mmWave radars operating at different frequencies and an RGB-D camera. Unlike visible light, mmWave signals can penetrate common occlusions (e.g., cardboard boxes, fabric, plastic). MITO captures over 24 million mmWave frames and uses them to generate 550 high-resolution mmWave images in line-of-sight and non-light-of-sight (NLOS), as well as RGB-D images, segmentation masks, and raw mmWave signals, taken from 76 different objects. We develop an open-source simulation tool that can be used to generate synthetic mmWave images for any 3D triangle mesh. Finally, we establish benchmarks for NLOS segmentation and classification to demonstrate the utility of our dataset and simulator for enabling broader NLOS perception.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Jun 24, 2026

RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception

Millimeter-wave (mmWave) radar perception is limited by data scarcity: models trained on existing radar datasets fail to generalize to new objects, environments, and sensing trajectories. We present RadarTwin, a framework for generating deployment-specific radar training data before real data collection. Given a 3D reconstruction of a target space (phone LiDAR, robot-mounted sensing, or RGB-to-3D), RadarTwin uses a vision-language model to infer radar-relevant surface materials and a physics-based ray tracer to synthesize raw frequency-modulated continuous-wave (FMCW) radar measurements with multi-bounce propagation. To study what transfers from simulation to reality, we collect a paired real-simulated dataset spanning household objects, material classes, distances, rotations, translations, and mobile sensing trajectories. We show that simulated and real radar share the same object-discriminative shape and material features, and that modeling the environment's multipath is essential to matching real measurements. A representation trained on simulation alone recognizes real objects at 2.5 times chance with no real radar labels, and a few labeled examples raise this to 95.3% on a 12-way recognition task. RadarTwin enables training radar perception for a new space before any real radar data is collected there.
Aug 14, 2026

Can Language Models Understand mmWave Data? Benchmarking Large Language Models for mmWave Radar-Based Human Understanding

Large language models (LLMs) have shown remarkable reasoning and generative capabilities, motivating their use as universal reasoning engines for perception. While modern approaches such as vision-language models (VLMs) have attempted to incorporate reasoning capabilities into visual sensing, the integration of LLMs with the millimeter-wave (mmWave) modality-despite its unique advantages under low light and occlusion-remains largely unexplored. The principal bottlenecks stem from the scarcity of radar language pairs, severe cross-dataset heterogeneity, and the absence of a foundational mmWave encoder. We address this gap through a minimal textualization interface that serializes each mmWave point cloud into concise natural language, allowing off-the-shelf LLMs to operate in a question answering (QA) setting. Building on this, we present mmWave-QA, the first benchmark for language-conditioned mmWave human perception. mmWave-QA aggregates heterogeneous public mmWave datasets and harmonizes them via calibration-aware preprocessing and global taxonomy alignment, while providing natural language QA. Spanning six scenarios and five QA tasks, the benchmark enables standardized evaluation across diverse mmWave hardware and experimental conditions, establishing a foundation for scalable research on mmWave-LLM integration. We further evaluate and analyze LLMs on our mmWave-QA, highlighting their zero-shot reasoning potential for radar perception, as well as their robustness under visual degradation.
Feb 21, 2026

WiCompass: Oracle-driven Data Scaling for mmWave Human Pose Estimation

Millimeter-wave Human Pose Estimation (mmWave HPE) promises privacy but suffers from poor generalization under distribution shifts. We demonstrate that brute-force data scaling is ineffective for out-of-distribution (OOD) robustness; efficiency and coverage are the true bottlenecks. To address this, we introduce WiCompass, a coverage-aware data-collection framework. WiCompass leverages large-scale motion-capture corpora to build a universal pose space ``oracle'' that quantifies dataset redundancy and identifies underrepresented motions. Guided by this oracle, WiCompass employs a closed-loop policy to prioritize collecting informative missing samples. Experiments show that WiCompass consistently improves OOD accuracy at matched budgets and exhibits superior scaling behavior compared to conventional collection strategies. By shifting focus from brute-force scaling to coverage-aware data acquisition, this work offers a practical path toward robust mmWave sensing.