Accurate material recognition is a fundamental capability for intelligent perception systems to interact safely and effectively with the physical world. For instance, distinguishing visually similar objects like glass and plastic cups is critical for safety but challenging for vision-based methods due to specular reflections, transparency, and visual deception. While millimeter-wave (mmWave) radar offers robust material sensing regardless of lighting, existing camera-radar fusion methods are limited to closed-set categories and lack semantic interpretability. In this paper, we introduce VLMaterial, a training-free framework that fuses vision-language models (VLMs) with domain-specific radar knowledge for physics-grounded material identification. First, we propose a dual-pipeline architecture: an optical pipeline uses the segment anything model and VLM for material candidate proposals, while an electromagnetic characterization pipeline extracts the intrinsic dielectric constant from radar signals via an effective peak reflection cell area (PRCA) method and weighted vector synthesis. Second, we employ a context-augmented generation (CAG) strategy to equip the VLM with radar-specific physical knowledge, enabling it to interpret electromagnetic parameters as stable references. Third, an adaptive fusion mechanism is introduced to intelligently integrate outputs from both sensors by resolving cross-modal conflicts based on uncertainty estimation. We evaluated VLMaterial in over 120 real-world experiments involving 41 diverse everyday objects and 4 typical visually deceptive counterfeits across varying environments. Experimental results demonstrate that VLMaterial achieves a recognition accuracy of 96.08%, delivering performance on par with state-of-the-art closed-set benchmarks while eliminating the need for extensive task-specific data collection and training.
Figures & tables
Work
Modality
Contactless
Category
Material
C. & M.
Training-free
Li et al. [ 4 ]
Infrared
✓
✗
✓
✗
✗
Wani et al. [ 5 ]
✓
✓
✗
✗
✗
Akte-Liquid [ 6 ]
Acoustic
✓
✗
✓
✗
✗
Liu et al. [ 7 ]
Tactile
✗
✗
✓
✗
✗
Yang et al. [ 8 ]
Camera
✓
✗
✓
✗
✗
Beyond Appearances [ 10 ]
✓
✗
✓
✗
✗
TABLE I: Comparison of Object Recognition Modalities.
Fig. 1: All typical daily objects: 41 items across 6 materials and 7 categories (the paper kettle was excluded due to its rarity). For clarity, images are annotated with dimensions (where L, W, H, and D represent Length, Width, Height, and Diameter, respectively) to provide scale context, with index number marked in the bottom-left (size not provided to the VLM).
Fig. 2: Validating inference stability via deterministic output verification.
Fig. 3: Experimental setups for data collection in different scenarios and detailed equipment configuration.
Fig. 4: Overall object material recognition (VLM-only).
Fig. 5: Evaluation of VLMs on four radar data representations (Raw data, Point cloud, Heatmap, and ERF).
Material awareness can improve robotic navigation and interaction, particularly in conditions where cameras and LiDAR degrade. We present a lightweight mmWave radar material classification pipeline designed for ultra-low-power edge devices (TI IWRL6432), using compact range-bin intensity descriptors and a Multilayer Perceptron (MLP) for real-time inference. While the classifier reaches a macro-F1 of 94.2% under the nominal training geometry, we observe a pronounced performance drop under realistic geometry shifts, including sensor height changes and small tilt angles. These perturbations induce systematic intensity scaling and angle-dependent radar cross section (RCS) effects, pushing features out of distribution and reducing macro-F1 to around 68.5%. We analyze these failure modes and outline practical directions for improving robustness with normalization, geometry augmentation, and motion-aware features.
Jannik Hohmann, Dong Wang, Andreas Nüchter
Department of Informatics XVII (Robotics), Julius-Maximilians-University Würzburg, 97074 Würzburg, Germany. · Zentrum für Telematik e.V., Würzburg · U2IS, ENSTA, Institut Polytechnique de Paris, France.
Cameras and LiDAR degrade in rain, fog, and snow, while millimeter-wave radar remains largely unaffected. We align a radar encoder to frozen SigLIP vision embeddings and decode structured scene captions through a frozen vision-language model (VLM) with approximately 7M trainable parameters. On K-RADAR with held-out fog, light snow, and heavy snow sequences, all radar configurations outperform a camera baseline that collapses to over 90% hallucination. We identify a token-norm mismatch as the dominant failure mode when bridging radar to a frozen VLM and show that projector-output LayerNorm resolves it. Analysis of encoder complexity, caption format, and pooling strategy reveals tradeoffs that inform future radar-VLM pipeline design.
Kali Hamilton, Christoffer Heckman
Autonomous Robotics and Perception Group at the University of Colorado Boulder, Boulder, CO 80309, USA.
Accurate 3D object detection is critical for autonomous driving, necessitating reliable, cost-effective sensors capable of operating in adverse weather conditions. Camera and millimeter-wave radar fusion has emerged as a promising solution; however, these methods often rely on finely annotated radar data, which is scarce and labor-intensive to produce. To address this challenge, we present CLLAP, a Contrastive Learning-based LiDAR-Augmented Pretraining framework that enhances the performance of existing radar-camera fusion methods for 3D object detection. CLLAP leverages abundant LiDAR data to generate pseudo-radar data using the proposed L2R (LiDAR-to-Radar) Sampling method. Then, it incorporates this data into a novel dual-stage, dual-modality contrastive learning strategy, enabling effective self-supervised learning from paired pseudo-radar and image data. This approach facilitates effective pretraining of existing radar-camera fusion models in a plug-and-play manner, enhancing their feature extraction capabilities and improving detection accuracy and robustness. Experimental results using NuScenes and Lyft Level 5 datasets demonstrate significant performance improvements across three baseline models, highlighting CLLAP's effectiveness in advancing radar-camera fusion for autonomous driving applications.
Bingyi Liu, Chuanhui Zhu, Hongfei Xue +5
Wuhan University of Technology · University of North Carolina at Charlotte · The Hong Kong Polytechnic University +2