mmIR: Frequency-Space Inverse Rendering for 3D Millimeter-Wave Radar ADC Synthesis

Adnan Armouti; Yixuan Gao; Rajalakshmi Nandakumar

cs.CVAug 28, 2026

mmIR: Frequency-Space Inverse Rendering for 3D Millimeter-Wave Radar ADC Synthesis

Authors: Adnan Armouti, Yixuan Gao, Rajalakshmi Nandakumar

Organizations: Cornell Tech, New York, NY, USA

Abstract

High-resolution 3D radar data is scarce. Commodity mmWave sensors use small antenna arrays that limit angular resolution to several degrees, and existing datasets provide only 2D range-azimuth maps or sparse point clouds rather than raw analog-to-digital converter (ADC) signals. Hardware scaling is expensive, synthetic-aperture scanning is impractical at fleet scale, and learned synthesis methods are bottlenecked by the very data shortage they aim to address. We present mmIR, an open-source differentiable frequency-modulated continuous-wave (FMCW) radar inverse renderer that fits a physics-based forward model to real captures and re-renders from dense virtual apertures to synthesize high-resolution 3D radar data. Because radar resolution is too coarse to recover geometry directly, mmIR performs LiDAR-assisted inverse rendering: using LiDAR-derived meshes as a geometric scaffold, mmIR optimizes per-vertex International Telecommunication Union (ITU) physics materials, vertex normals, and antenna beam patterns through end-to-end automatic differentiation of a phase-coherent multiple-input multiple-output (MIMO) forward model with multi-bounce propagation, polarization, and free-space diffraction. On seven outdoor and six indoor ColoRadar scenes, mmIR achieves 0.914 mean Pearson correlation on range-azimuth maps versus 0.307 for Sionna-RT. Scenes trained on a cascaded imaging radar transfer to a co-located single-chip radar without re-training (0.554 correlation), and dense virtual arrays (100x100 elements) produce single-frame 3D occupancy validated against LiDAR. Project page: https://mmwave-inverse-rendering.github.io/

Figures & tables

Figure 1: Given a LiDAR mesh and GT real cascaded-radar RA map (left), mmIR fits per-vertex materials, normals, and beam patterns (center), then re-renders a dense 100 $\times$ 100 virtual array for single-frame 3D occupancy (right).

Method

Open

Repr.

Signal

Multi-b.

Diffr.

Polar.

Geom.

Norm.

Mat.

Pose

Beam

Per-vtx

Bound.

\nabla

Real

Sionna-RT [ 18 ]

\checkmark

Mesh

CIR

\checkmark

\checkmark

\checkmark

–

No BRDF

\checkmark

–

SFCW Inv. [ 17 ]

\checkmark

Mesh

–

\checkmark

–

Shading

\checkmark

Pos.

–

\checkmark

–

\checkmark

InverTwin [ 10 ]

–

Mesh

CIR

\checkmark

\checkmark

–

\checkmark

–

No BRDF

\checkmark

–

\checkmark

\checkmark

mmIR (Ours)

\checkmark

Mesh

ADC

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

Table 1: Feature comparison of physics-based differentiable radar renderers. Signal: ADC = raw FMCW, CIR = channel impulse response, IF = intermediate-frequency phasors. Multi-b. = multi-bounce; Diffr. = edge diffraction; Polar. = polarization; Geom. = differentiable geometry; Norm. = differentiable normals; Mat. = physics material model; Per-vtx = per-vertex materials; Bound.

\nabla

= boundary gradients; Real = validated on real data. Checkmarks denote supported pipeline capabilities; see Sec. in the supplement for which parameters are active in our experiments.

Figure 2: mmIR pipeline. Given a LiDAR mesh and sensor parameters, the differentiable model traces phase-coherent MIMO paths (reflection + diffraction) into raw FMCW ADC. End-to-end AD (ray tracing, BSDF, phasor accumulation, ADC) optimizes per-vertex materials, normals, and antenna beam patterns from a range–azimuth loss.

Figure 3: Antenna configurations: cascaded MMWCAS (12Tx $\times$ 16Rx with 136 unique virtual elements) used for training, single-chip AWR1843 (3Tx $\times$ 4Rx with 12 unique virtual elements) used for transfer evaluation, dense array (100Tx $\times$ 100Rx, 10,000 virtual elements) used for 3D inference. Top: physical Tx/Rx; bottom: virtual aperture. The cascade’s near-1D virtual aperture yields $256{\times}127{\times}1$ RAE resolution (Sec. , ); only the dense array resolves true elevation at $256{\times}127{\times}127$ RAE resolution.

Outdoor (7 scenes)
	RA Correlation $\uparrow$		RA PSNR $\uparrow$		RA SSIM $\uparrow$		RA RMSE $\downarrow$		ADC Log-Mag MSE $\downarrow$		ADC MM-Mag MSE $\downarrow$
Scene	Ours	Sionna-RT	Ours	Sionna-RT	Ours	Sionna-RT	Ours	Sionna-RT	Ours	Sionna-RT	Ours	Sionna-RT
S0-F135	0.921	0.418	37.4	21.4	0.938	0.509	0.0135	0.0850	48.3271	244.8361	0.0421	0.0469
S0-F390	0.936	0.242	41.0	23.4	0.937	0.550	0.0089	0.0674	5.0250	294.3032	0.0289	0.0417
S1-F185	0.954	0.312	39.8	18.9	0.926	0.326	0.0102	0.1138	32.3385	282.6453	0.0398	0.0461
S1-F438	0.942	0.346	37.7	19.9	0.926	0.401	0.0130	0.1008	32.5274	288.9782	0.0369	0.0435

Table 2: Range–azimuth (RA) and ADC training evaluation on seven outdoor and six indoor scenes. mmIR is compared against Sionna-RT [ 18 ] . Metrics: Pearson correlation (Corr), PSNR, SSIM, RMSE, and ADC-level magnitude error. Best in bold . Results use the min–max-normalized RA loss (Eq. 8 ).

Figure 4: Qualitative RA on the seven outdoor scenes, sorted by mmIR correlation. Rows: scene mesh (red sphere = radar, green arrow = boresight), GT, mmIR, Sionna-RT. mmIR captures dominant scatterers and multipath; Sionna-RT is weaker and misaligned.

Figure 5: As Fig. 4 , for six indoor scenes (lab, hallway, classroom). mmIR tracks the GT returns; Sionna-RT over-responds in dense indoor multipath.

	RA Correlation $\uparrow$		RA PSNR $\uparrow$		RA SSIM $\uparrow$		RA RMSE $\downarrow$		ADC Log-Mag MSE $\downarrow$		ADC MM-Mag MSE $\downarrow$
Scene	Ours	Bench.	Ours	Bench.	Ours	Bench.	Ours	Bench.	Ours	Bench.	Ours	Bench.
S0-F135	0.533	0.002	25.6	17.8	0.505	0.464	0.0524	0.1283	10.9639	301.4341	0.2194	0.0668
S0-F390	0.464	0.328	21.2	24.2	0.529	0.566	0.0873	0.0619	12.9644	331.2549	0.0630	0.0840
S1-F185	0.567	0.136	22.4	20.1	0.422	0.480	0.0759	0.0990	0.5754	324.8441	0.1993	0.0771
S1-F438	0.684	0.079	23.2	13.5	0.674	0.202	0.0694	0.2119	21.8672	348.0237	0.1507	0.0662
S2-F105	0.608	0.084	24.7	22.8	0.609	0.690	0.0583	0.0721	3.4458	289.7929	0.1959	0.1068

Table 3: Cross-sensor transfer evaluation. Parameters trained on the cascade (12Tx

\times

16Rx) are frozen and used to render the single-chip AWR1843 (3Tx

\times

4Rx) without re-training. Best in bold .

	Precision $\uparrow$		Recall $\uparrow$		F1 $\uparrow$		Accuracy $\uparrow$		RMSE $\downarrow$		R-CD $\downarrow$
Scene	Ours	ColoR	Ours	ColoR	Ours	ColoR	Ours	ColoR	Ours	ColoR	Ours	ColoR
S0-F135	0.077	0.223	0.018	0.054	0.029	0.087	0.044	0.057	2.324	3.302	0.248	0.228
S0-F390	0.481	0.045	0.110	0.014	0.179	0.021	0.212	0.017	2.692	2.745	0.216	0.207
S1-F185	0.997	0.483	0.566	0.082	0.722	0.141	0.703	0.103	0.909	2.380	0.067	0.236
S1-F438	0.633	0.433	0.265	0.117	0.373	0.185	0.362	0.131	1.511	3.339	0.116	0.268
S2-F105	0.771	0.154	0.162	0.025	0.268	0.044	0.340	0.029	1.099	3.402	0.113	0.282

Table 4: 3D occupancy vs. GT LiDAR. Baseline: ColoRadar’s cascaded-radar point cloud with elevation extrusion added in post-processing. Ours: mmIR’s RAE rendered via the dense virtual aperture (100

\times

100 elements), resolving true elevation. KD-tree match at

\tau{=}0.5

m. Best in bold .

Figure 6: Single-frame 3D occupancy on the LiDAR mesh. Top: mmIR’s rendered RAE via the dense virtual aperture (100 $\times$ 100 elements), thresholded at the 99.9th intensity percentile. Bottom: ColoRadar-provided RAE, obtained from the near-1D cascaded radar with elevation extrusion added in post-processing. Red sphere = radar, green arrow = boresight.

Explore similar work

Jun 24, 2026cs.CV

RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception

Millimeter-wave (mmWave) radar perception is limited by data scarcity: models trained on existing radar datasets fail to generalize to new objects, environments, and sensing trajectories. We present RadarTwin, a framework for generating deployment-specific radar training data before real data collection. Given a 3D reconstruction of a target space (phone LiDAR, robot-mounted sensing, or RGB-to-3D), RadarTwin uses a vision-language model to infer radar-relevant surface materials and a physics-based ray tracer to synthesize raw frequency-modulated continuous-wave (FMCW) radar measurements with multi-bounce propagation. To study what transfers from simulation to reality, we collect a paired real-simulated dataset spanning household objects, material classes, distances, rotations, translations, and mobile sensing trajectories. We show that simulated and real radar share the same object-discriminative shape and material features, and that modeling the environment's multipath is essential to matching real measurements. A representation trained on simulation alone recognizes real objects at 2.5 times chance with no real radar labels, and a few labeled examples raise this to 95.3% on a 12-way recognition task. RadarTwin enables training radar perception for a new space before any real radar data is collected there.

Emily Bejerano, Federico Tondolo, Devang Gupta +5

Columbia University New York, NY, USA · University of California, Merced Merced, CA, USA

Sim-to-Real Transfer Millimeter-Wave Radar mmWave Radar Sensing Synthetic Data Generation for Robotics

Ask PDF

Sep 10, 2026cs.CV

3D Point Splatting for mmWave Radar Novel View Synthesis

Solving novel view synthesis (NVS) for millimeter-wave (mmWave) radar requires a renderer that is physically faithful, complex-valued, and multi-viewpoint-tractable. No prior method achieves these three properties simultaneously. Differentiable Monte Carlo (MC) ray tracers implement the radar forward model directly with explicit material modeling and complex outputs, but do not scale to the multi-view optimization NVS demands. Optical-NVS ports of NeRF, hash grids, and 3D Gaussians train fast but discard phase and replace explicit material modeling with opaque learned features, restricting them to power-only range-azimuth (RA) magnitudes. We propose 3D Point Splatting (3DPS), the first differentiable point renderer for radar, derived directly from the standard solid-angle form of the radar equation. Each oriented 3D point carries an ITU-R P.2040 material model, evaluated in closed form, with the resulting complex phasor splatted into range bins through a precomputed point spread function (PSF). The complex-valued output makes the renderer product-agnostic. The same optimized scene yields analog-to-digital converter (ADC), complex range profile (CRP), and RA outputs through standard fast Fourier transform (FFT) pipelines without retraining for each format. On six outdoor ColoRadar scenes, 3DPS reaches 0.587 mean Pearson correlation on held-out RA images. This is between 1.7x and 5.2x the three optical-NVS baselines (RadarSplat, Radar Fields, DART). Training takes approximately 3 minutes per scene on a single RTX 4090.

Adnan Armouti, Yixuan Gao, Rajalakshmi Nandakumar

Cornell Tech New York, NY, USA

Differentiable Rendering Millimeter-Wave Radar mmWave Radar Sensing Novel View Synthesis

Ask PDF

Jul 30, 2026cs.CV

mmRadarTwin: A Measurement-Calibrated Signal-Level Digital Twin Platform for Indoor mmWave Radar

Indoor mmWave radar perception is difficult to reproduce because measured range-angle responses depend on scene geometry, material response, multipath, hardware conventions, and signal processing. Existing ray-tracing and digital-twin tools often expose rendering, channel, or path-level quantities, while radar sensing requires complex signal products that can be processed and compared in the same domain as real FMCW measurements. We present mmRadarTwin, a signal-level and path-attributed digital-twin platform for indoor mmWave radar. mmRadarTwin links a real radar measurement branch with an Unreal Engine scene-simulation branch through a shared receive-channel and range-angle processing interface. The simulator writes complex multi-channel receive grids and exports per-path contribution records that identify the actor, material tag, propagation event, and output-bin support of each simulated return. We evaluate mmRadarTwin in an office deployment using a commodity monostatic mmWave radar and mobile scene-capture hardware. Across 154 measured poses spanning 22 radar locations, the current physics-only path-basis simulator recalls 70.8% of measurement-active geometry-supported response regions in the central usable field of view while exposing residuals caused by weak or missing path support, shifted responses, unsupported anchors, and missing physical mechanisms. Rather than claiming complete radar-map reconstruction or cross-room generalization, mmRadarTwin establishes a practical systems workflow for constructing, comparing, and diagnosing indoor radar digital twins.

Jianyi Zhou, Chenghao Zhang, Yanli Li +1

The University of Sydney · Sydney, NSW, Australia

Millimeter-Wave Radar mmWave Radar Sensing Active Sensing Radio Propagation Modeling Digital Twins

Ask PDF