Taming Vision Priors for Data Efficient mmWave Channel Modeling

Zhenlin An; Longfei Shangguan; John Kaewell; Philip Pietraski; Jelena Senic; Camillo Gentile; Nada Golmie; Kyle Jamieson

cs.CVMar 11, 2026

Taming Vision Priors for Data Efficient mmWave Channel Modeling

Authors: Zhenlin An, Longfei Shangguan, John Kaewell, Philip Pietraski, Jelena Senic, Camillo Gentile, Nada Golmie, Kyle Jamieson

Organizations: University of Georgia · University of Pittsburgh · Interdigital · NIST ·

Abstract

Accurately modeling millimeter-wave (mmWave) propagation is essential for real-time AR and autonomous systems. Differentiable ray tracing offers a physics-grounded solution but still facing deployment challenges due to its over-reliance on exhaustive channel measurements or brittle, hand-tuned scene models for material properties. We present VisRFTwin, a scalable and data-efficient digital-twin framework that integrates vision-derived material priors with differentiable ray tracing. Multi-view images from commodity cameras are processed by a frozen Vision-Language Model to extract dense semantic embeddings, which are translated into initial estimates of permittivity and conductivity for scene surfaces. These priors initialize a Sionna-based differentiable ray tracer, which rapidly calibrates material parameters via gradient descent with only a few dozen sparse channel soundings. Once calibrated, the association between vision features and material parameters is retained, enabling fast transfer to new scenarios without repeated calibration. Evaluations across three real-world scenarios, including office interiors, urban canyons, and dynamic public spaces show that VisRFTwin reduces channel measurement needs by up to 10 $\times$ while achieving a 59% lower median delay spread error than pure data-driven deep learning methods.

Figures & tables

Fig. 1 . System Workflow. VisRFTwin leverages 3D scene reconstruction (NeRF) and pretrained vision–language models to extract semantic material features from images. These vision priors are translated into electromagnetic parameters to bootstrap differentiable ray tracing, reducing the amount of channel measurements needed for channel modeling.

Approach	Model	Physics Fidelity	RF Data Needed	Vision Modality	Scene Dynamics	Multipath Delay	Scattering Physics
PointNet ( Mi et al., 2024 )	e2e DL	Low	Dense	✗	✗	✓	✗
Sionna ( Hoydis et al., 2023 )	DiffRT (RF Prop.)	High	Medium	✗	✗	✓	✓
RadioTwin ( An et al., 2025 )	DiffRT (RF Prop.)	High	Dense	✗	✗	✓	✓
NeRF2 ( Zhao et al., 2023 )	RM (Optical Prop.)	Medium	Dencs	✗	✗	✗	✗
RF-3DGS ( Zhang et al., 2025 )	RM (Optical Prop.)	Medium	Medium	✓	✗	✗	✗
RFCanvas ( Chen et al., 2024 )	RM (Optical Prop.)	Medium	Sparse	✓	✓	✗	✗

Table 1. Comparison of channel modeling methods.

Panel of Figure 3 — Fig. 2 . Differentiable ray tracing framework. (a) Real-world ray tracing visualization with rays, transmitters, receivers, and interaction points. (b) End-to-end differentiable training loop using channel mismatch as gradient feedback.

Fig. 3 . Comparison of material understanding methods in indoor environments. (a) Input image of a complex indoor scene. (b) Classic material segmentation models struggle with generalization, producing coarse and inaccurate segmentation that fails to adapt to the diversity of real-world materials. (c) The Segment Anything Model (SAM) ( Cen et al., 2024 ) offers high-quality instance boundaries but lacks the ability to differentiate materials, limiting its applicability for material-aware tasks. (d, e) Our vision-guided model extracts rich semantic features using CLIP and simultaneously predicts accurate depth maps, enabling projection onto 3D surfaces.

Fig. 5 . 3D Semantic Field. The colors encode semantic features, with different hues indicating distinct material categories across the scene.

Fig. 6 . Pipeline for radio material identification.

Fig. 7 . Lambertian scattering model. An incident ray striking a perfectly diffuse surface results in scattered rays distributed uniformly over the hemisphere

Material	Ra (µm)	Scattering Coeff. $S$
Wall Plasterboard	2.309	0.07–0.1
Chipboard	2.695	0.1–0.2
Cardboard	3.252	0.1–0.2
Ceiling Plasterboard	11.85	0.2–0.4
Brick	14.68	0.3–0.5

Table 2. Scattering Coefficient of Typical Materials. We list a few examples for illustration.

Fig. 8 . Scattering coefficient prediction using vision-language similarity. Vision features extracted from material images are compared with text embeddings of roughness-related prompts to infer scattering levels.

Fig. 9 . Few - Shot Vision - Guided Radio Digital - Twin Training. After running the vision based radio twin, a few channel sounding probes anchor this twin, and differentiable path tracing compares predicted channel with measurements. The loss is back - propagated to calibrate the object material.

Panel of Figure 12 — Fig. 10 . Evaluation Scenarios. (a) Indoor conference room with tables and chairs; (b) Indoor lobby space with sofas, tables, and glass partitions used for millimeter-wave measurements; (c) Outdoor scenario besides a parking lot; (d) Measured multipath profile depicting received power across individual paths.

Panel of Figure 13 — Fig. 11 . Snapshot-level performance of the proposed model on the four canonical channel-metrics.

Fig. 12 . Compared with SoTA Solution. Delay Spread.

Metric	Definition
Path Loss (PL)	$\mathrm{PL}=-10\mathop{\mathrm{log}}_{10}\Bigl(\sum_{i=1}^{N}P_{i}\Bigr)$
RMS Delay Spread ( $\tau_{\mathrm{RMS}}$ )	$\tau_{\mathrm{RMS}}=\sqrt{\tfrac{\sum_{i}P_{i}\tau_{i}^{2}}{\sum_{i}P_{i}}-\Bigl(\tfrac{\sum_{i}P_{i}\tau_{i}}{\sum_{i}P_{i}}\Bigr)^{2}}$
Angular Spread (AS)	$\mathrm{AS}=\sqrt{-2\mathop{\mathrm{ln}}\Bigl\|\tfrac{\sum_{i}P_{i}e^{j\phi_{i}}}{\sum_{i}P_{i}}\Bigr\|}\,[^{\circ}]$
Rician $K$ -factor	$K=10\mathop{\mathrm{log}}_{10}\Bigl(\tfrac{P_{\mathrm{LoS}}}{\sum_{i\neq\mathrm{LoS}}P_{i}}\Bigr)$

Table 3. Snapshot-level metrics (M1–M4).

Fig. 13 . Spatial Distribution of RMS delay estimation

Fig. 14 . Multipath Profile Estimation. Power profile and delay profile comparison under zero shot estimation.

Fig. 19 . Dynamic Evlauation. (left) Lab measurement scene with a transmitter (TX) and receiver (RX) placed along a 4 m LoS while a human subject walks through the link. (right) Ray-tracing results.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 20 . Progressive refinement of NeRF-derived point clouds. (1) raw point cloud generated by querying NeRF with virtual camera depth rendering, (2) filtered and down-sampled point cloud, and (3) final mesh obtained through Possion surface reconstruction.

Panel of Figure 21 — Fig. 21 . Illustration of two reconstructed indoor scenes shown within a single column. (a) a second view emphasizing spatial layout and camera placement (b) A first-person reconstruction highlighting interior structures.

Fig. 22 . Vertex-wise geometry error of the mesh with respect to the LiDAR ground truth. left—spatial error map; right: mesh to mesh distance histogram.

Fig. 23 . Outdoor dataset Reconstruction

Fig. 24 . Material proposal prompt for GPT-4o.

Fig. 25 . Appearance of materials with different roughness. As roughness increases, visual cues become more pronounced, enabling vision-based inference of surface roughness and corresponding scattering coefficients.

Fig. 26 . Measured Multipath Profile

Differentiable Ray Tracing

See all 12 papers

RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception
Jun 24, 2026Emily Bejerano, Federico Tondolo, Devang Gupta +5Sim-to-Real Transfer Millimeter-Wave Radar
Columbia University New York, NY, USA · University of California, Merced Merced, CA, USA
Differentiable Ray Tracing with Gaussians for Unified Radio Propagation Simulation and View Synthesis
May 8, 2026Niklas Vaara, Lam Huynh, Pekka Sangi +23D Gaussian Splatting Differentiable Rendering
Center for Machine Vision and Signal Analysis, University of Oulu, Finland · CubiCasa Oy, Finland
mmRadarTwin: A Measurement-Calibrated Signal-Level Digital Twin Platform for Indoor mmWave Radar
Jul 30, 2026Jianyi Zhou, Chenghao Zhang, Yanli Li +1Millimeter-Wave Radar mmWave Radar Sensing
The University of Sydney · Sydney, NSW, Australia

Jun 24, 2026J/K move · Enter open · S save

RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception

Emily Bejerano, Federico Tondolo, Devang Gupta, Aaron Mano Cherian+4

Columbia University New York, NY, USA · University of California, Merced Merced, CA, USA

Sim-to-Real Transfer Millimeter-Wave Radar mmWave Radar Sensing Synthetic Data Generation for Robotics

Open paper AskPDF arXiv

Millimeter-wave (mmWave) radar perception is limited by data scarcity: models trained on existing radar datasets fail to generalize to new objects, environments, and sensing trajectories. We present RadarTwin, a framework for generating deployment-specific radar training data before real data collection. Given a 3D reconstruction of a target space (phone LiDAR, robot-mounted sensing, or RGB-to-3D), RadarTwin uses a vision-language model to infer radar-relevant surface materials and a physics-based ray tracer to synthesize raw frequency-modulated continuous-wave (FMCW) radar measurements with multi-bounce propagation. To study what transfers from simulation to reality, we collect a paired real-simulated dataset spanning household objects, material classes, distances, rotations, translations, and mobile sensing trajectories. We show that simulated and real radar share the same object-discriminative shape and material features, and that modeling the environment's multipath is essential to matching real measurements. A representation trained on simulation alone recognizes real objects at 2.5 times chance with no real radar labels, and a few labeled examples raise this to 95.3% on a 12-way recognition task. RadarTwin enables training radar perception for a new space before any real radar data is collected there.

Abstract

Differentiable Ray Tracing

Explore similar work

RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception

Differentiable Ray Tracing with Gaussians for Unified Radio Propagation Simulation and View Synthesis

mmRadarTwin: A Measurement-Calibrated Signal-Level Digital Twin Platform for Indoor mmWave Radar