cs.CVMar 11, 2026

Taming Vision Priors for Data Efficient mmWave Channel Modeling

Authors: Zhenlin An, Longfei Shangguan, John Kaewell, Philip Pietraski, Jelena Senic, Camillo Gentile, Nada Golmie, Kyle Jamieson

Organizations: University of Georgia · University of Pittsburgh · Interdigital · NIST · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Accurately modeling millimeter-wave (mmWave) propagation is essential for real-time AR and autonomous systems. Differentiable ray tracing offers a physics-grounded solution but still facing deployment challenges due to its over-reliance on exhaustive channel measurements or brittle, hand-tuned scene models for material properties. We present VisRFTwin, a scalable and data-efficient digital-twin framework that integrates vision-derived material priors with differentiable ray tracing. Multi-view images from commodity cameras are processed by a frozen Vision-Language Model to extract dense semantic embeddings, which are translated into initial estimates of permittivity and conductivity for scene surfaces. These priors initialize a Sionna-based differentiable ray tracer, which rapidly calibrates material parameters via gradient descent with only a few dozen sparse channel soundings. Once calibrated, the association between vision features and material parameters is retained, enabling fast transfer to new scenarios without repeated calibration. Evaluations across three real-world scenarios, including office interiors, urban canyons, and dynamic public spaces show that VisRFTwin reduces channel measurement needs by up to 10×\times while achieving a 59% lower median delay spread error than pure data-driven deep learning methods.

Figures & tables

Fig. 1 . System Workflow. VisRFTwin leverages 3D scene reconstruction (NeRF) and pretrained vision–language models to extract semantic material features from images. These vision priors are translated into electromagnetic parameters to bootstrap differentiable ray tracing, reducing the amount of channel measurements needed for channel modeling.
ApproachModelPhysics FidelityRF Data NeededVision ModalityScene DynamicsMultipath DelayScattering Physics
PointNet ( Mi et al., 2024 )e2e DLLowDense✗✗✓✗
Sionna ( Hoydis et al., 2023 )DiffRT (RF Prop.)HighMedium✗✗✓✓
RadioTwin ( An et al., 2025 )DiffRT (RF Prop.)HighDense✗✗✓✓
NeRF2 ( Zhao et al., 2023 )RM (Optical Prop.)MediumDencs✗✗✗✗
RF-3DGS ( Zhang et al., 2025 )RM (Optical Prop.)MediumMedium✓✗✗✗
RFCanvas ( Chen et al., 2024 )RM (Optical Prop.)MediumSparse✓✓✗✗
Table 1. Comparison of channel modeling methods.
Fig. 2 . Differentiable ray tracing framework. (a) Real-world ray tracing visualization with rays, transmitters, receivers, and interaction points. (b) End-to-end differentiable training loop using channel mismatch as gradient feedback.
Fig. 3 . Comparison of material understanding methods in indoor environments. (a) Input image of a complex indoor scene. (b) Classic material segmentation models struggle with generalization, producing coarse and inaccurate segmentation that fails to adapt to the diversity of real-world materials. (c) The Segment Anything Model (SAM) ( Cen et al., 2024 ) offers high-quality instance boundaries but lacks the ability to differentiate materials, limiting its applicability for material-aware tasks. (d, e) Our vision-guided model extracts rich semantic features using CLIP and simultaneously predicts accurate depth maps, enabling projection onto 3D surfaces.
Fig. 4 . Feature Re-projection.
Fig. 5 . 3D Semantic Field. The colors encode semantic features, with different hues indicating distinct material categories across the scene.
Fig. 6 . Pipeline for radio material identification.
Fig. 7 . Lambertian scattering model. An incident ray striking a perfectly diffuse surface results in scattered rays distributed uniformly over the hemisphere
MaterialRa (µm)Scattering Coeff. SS
Wall Plasterboard2.3090.07–0.1
Chipboard2.6950.1–0.2
Cardboard3.2520.1–0.2
Ceiling Plasterboard11.850.2–0.4
Brick14.680.3–0.5
Table 2. Scattering Coefficient of Typical Materials. We list a few examples for illustration.
Fig. 8 . Scattering coefficient prediction using vision-language similarity. Vision features extracted from material images are compared with text embeddings of roughness-related prompts to infer scattering levels.
Fig. 9 . Few - Shot Vision - Guided Radio Digital - Twin Training. After running the vision based radio twin, a few channel sounding probes anchor this twin, and differentiable path tracing compares predicted channel with measurements. The loss is back - propagated to calibrate the object material.
Fig. 10 . Evaluation Scenarios. (a) Indoor conference room with tables and chairs; (b) Indoor lobby space with sofas, tables, and glass partitions used for millimeter-wave measurements; (c) Outdoor scenario besides a parking lot; (d) Measured multipath profile depicting received power across individual paths.
Fig. 11 . Snapshot-level performance of the proposed model on the four canonical channel-metrics.
Fig. 12 . Compared with SoTA Solution. Delay Spread.
MetricDefinition
Path Loss (PL)PL=−10log10(∑i=1NPi)\mathrm{PL}=-10\mathop{\mathrm{log}}_{10}\Bigl(\sum_{i=1}^{N}P_{i}\Bigr)
RMS Delay Spread ( τRMS\tau_{\mathrm{RMS}} )τRMS=∑iPiτi2∑iPi−(∑iPiτi∑iPi)2\tau_{\mathrm{RMS}}=\sqrt{\tfrac{\sum_{i}P_{i}\tau_{i}^{2}}{\sum_{i}P_{i}}-\Bigl(\tfrac{\sum_{i}P_{i}\tau_{i}}{\sum_{i}P_{i}}\Bigr)^{2}}
Angular Spread (AS)AS=−2ln∣∑iPiejϕi∑iPi∣ [∘]\mathrm{AS}=\sqrt{-2\mathop{\mathrm{ln}}\Bigl|\tfrac{\sum_{i}P_{i}e^{j\phi_{i}}}{\sum_{i}P_{i}}\Bigr|}\,[^{\circ}]
Rician KK -factorK=10log10(PLoS∑i≠LoSPi)K=10\mathop{\mathrm{log}}_{10}\Bigl(\tfrac{P_{\mathrm{LoS}}}{\sum_{i\neq\mathrm{LoS}}P_{i}}\Bigr)
Table 3. Snapshot-level metrics (M1–M4).
Fig. 13 . Spatial Distribution of RMS delay estimation
Fig. 14 . Multipath Profile Estimation. Power profile and delay profile comparison under zero shot estimation.
Figure 18
Fig. 19 . Dynamic Evlauation. (left) Lab measurement scene with a transmitter (TX) and receiver (RX) placed along a 4 m LoS while a human subject walks through the link. (right) Ray-tracing results.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 20 . Progressive refinement of NeRF-derived point clouds. (1) raw point cloud generated by querying NeRF with virtual camera depth rendering, (2) filtered and down-sampled point cloud, and (3) final mesh obtained through Possion surface reconstruction.
Fig. 21 . Illustration of two reconstructed indoor scenes shown within a single column. (a) a second view emphasizing spatial layout and camera placement (b) A first-person reconstruction highlighting interior structures.
Fig. 22 . Vertex-wise geometry error of the mesh with respect to the LiDAR ground truth. left—spatial error map; right: mesh to mesh distance histogram.
Fig. 23 . Outdoor dataset Reconstruction
Fig. 24 . Material proposal prompt for GPT-4o.
Fig. 25 . Appearance of materials with different roughness. As roughness increases, visual cues become more pronounced, enabling vision-based inference of surface roughness and corresponding scattering coefficients.
Fig. 26 . Measured Multipath Profile
More in this topic

Differentiable Ray Tracing

See all 12 papers

Explore similar work

CardsList
Jun 24, 2026

RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception

Millimeter-wave (mmWave) radar perception is limited by data scarcity: models trained on existing radar datasets fail to generalize to new objects, environments, and sensing trajectories. We present RadarTwin, a framework for generating deployment-specific radar training data before real data collection. Given a 3D reconstruction of a target space (phone LiDAR, robot-mounted sensing, or RGB-to-3D), RadarTwin uses a vision-language model to infer radar-relevant surface materials and a physics-based ray tracer to synthesize raw frequency-modulated continuous-wave (FMCW) radar measurements with multi-bounce propagation. To study what transfers from simulation to reality, we collect a paired real-simulated dataset spanning household objects, material classes, distances, rotations, translations, and mobile sensing trajectories. We show that simulated and real radar share the same object-discriminative shape and material features, and that modeling the environment's multipath is essential to matching real measurements. A representation trained on simulation alone recognizes real objects at 2.5 times chance with no real radar labels, and a few labeled examples raise this to 95.3% on a 12-way recognition task. RadarTwin enables training radar perception for a new space before any real radar data is collected there.

Emily Bejerano, Federico Tondolo, Devang Gupta +5

Columbia University New York, NY, USA · University of California, Merced Merced, CA, USA

Sim-to-Real TransferMillimeter-Wave RadarmmWave Radar SensingSynthetic Data Generation for Robotics
AskPDF
May 8, 2026

Differentiable Ray Tracing with Gaussians for Unified Radio Propagation Simulation and View Synthesis

Explicit neural representations such as 3D Gaussian Splatting (3DGS) enable high-fidelity and real-time novel view synthesis, yet optimize for alpha-composited optical appearance rather than ray-intersectable geometry. In contrast, radio-frequency (RF) digital twins require deterministic multi-bounce paths, where the geometry dictates trajectories and their associated attenuation and delay. We introduce a framework enabling differentiable RF propagation simulation directly within visually reconstructed neural scenes, allowing point-to-point path computation between arbitrary 3D locations while preserving high-quality visual rendering. Unlike conventional RF simulation pipelines that rely on manually constructed meshes, we embed Gaussian primitives into a hardware-accelerated ray tracing structure as the underlying spatial representation. By extracting physically meaningful channel impulse responses from visual-only reconstructions, we provide cross-modal evidence that neural reconstructions can serve as unified spatial representations for both electromagnetic propagation simulation and photorealistic view synthesis.

Niklas Vaara, Lam Huynh, Pekka Sangi +2

Center for Machine Vision and Signal Analysis, University of Oulu, Finland · CubiCasa Oy, Finland

3D Gaussian SplattingDifferentiable RenderingDifferentiable Ray TracingWireless CommunicationsRadio Propagation ModelingDigital Twins
AskPDF
Jul 30, 2026

mmRadarTwin: A Measurement-Calibrated Signal-Level Digital Twin Platform for Indoor mmWave Radar

Indoor mmWave radar perception is difficult to reproduce because measured range-angle responses depend on scene geometry, material response, multipath, hardware conventions, and signal processing. Existing ray-tracing and digital-twin tools often expose rendering, channel, or path-level quantities, while radar sensing requires complex signal products that can be processed and compared in the same domain as real FMCW measurements. We present mmRadarTwin, a signal-level and path-attributed digital-twin platform for indoor mmWave radar. mmRadarTwin links a real radar measurement branch with an Unreal Engine scene-simulation branch through a shared receive-channel and range-angle processing interface. The simulator writes complex multi-channel receive grids and exports per-path contribution records that identify the actor, material tag, propagation event, and output-bin support of each simulated return. We evaluate mmRadarTwin in an office deployment using a commodity monostatic mmWave radar and mobile scene-capture hardware. Across 154 measured poses spanning 22 radar locations, the current physics-only path-basis simulator recalls 70.8% of measurement-active geometry-supported response regions in the central usable field of view while exposing residuals caused by weak or missing path support, shifted responses, unsupported anchors, and missing physical mechanisms. Rather than claiming complete radar-map reconstruction or cross-room generalization, mmRadarTwin establishes a practical systems workflow for constructing, comparing, and diagnosing indoor radar digital twins.

Jianyi Zhou, Chenghao Zhang, Yanli Li +1

The University of Sydney · Sydney, NSW, Australia

Millimeter-Wave RadarmmWave Radar SensingActive SensingRadio Propagation ModelingDigital Twins
AskPDF