mmIR: Frequency-Space Inverse Rendering for 3D Millimeter-Wave Radar ADC Synthesis

Adnan Armouti; Yixuan Gao; Rajalakshmi Nandakumar

cs.CVAug 28, 2026

mmIR: Frequency-Space Inverse Rendering for 3D Millimeter-Wave Radar ADC Synthesis

Authors: Adnan Armouti, Yixuan Gao, Rajalakshmi Nandakumar

Organizations: Cornell Tech, New York, NY, USA

Abstract

High-resolution 3D radar data is scarce. Commodity mmWave sensors use small antenna arrays that limit angular resolution to several degrees, and existing datasets provide only 2D range-azimuth maps or sparse point clouds rather than raw analog-to-digital converter (ADC) signals. Hardware scaling is expensive, synthetic-aperture scanning is impractical at fleet scale, and learned synthesis methods are bottlenecked by the very data shortage they aim to address. We present mmIR, an open-source differentiable frequency-modulated continuous-wave (FMCW) radar inverse renderer that fits a physics-based forward model to real captures and re-renders from dense virtual apertures to synthesize high-resolution 3D radar data. Because radar resolution is too coarse to recover geometry directly, mmIR performs LiDAR-assisted inverse rendering: using LiDAR-derived meshes as a geometric scaffold, mmIR optimizes per-vertex International Telecommunication Union (ITU) physics materials, vertex normals, and antenna beam patterns through end-to-end automatic differentiation of a phase-coherent multiple-input multiple-output (MIMO) forward model with multi-bounce propagation, polarization, and free-space diffraction. On seven outdoor and six indoor ColoRadar scenes, mmIR achieves 0.914 mean Pearson correlation on range-azimuth maps versus 0.307 for Sionna-RT. Scenes trained on a cascaded imaging radar transfer to a co-located single-chip radar without re-training (0.554 correlation), and dense virtual arrays (100x100 elements) produce single-frame 3D occupancy validated against LiDAR. Project page: https://mmwave-inverse-rendering.github.io/

Figures & tables

Figure 1: Given a LiDAR mesh and GT real cascaded-radar RA map (left), mmIR fits per-vertex materials, normals, and beam patterns (center), then re-renders a dense 100 $\times$ 100 virtual array for single-frame 3D occupancy (right).

Method

Open

Repr.

Signal

Multi-b.

Diffr.

Polar.

Geom.

Norm.

Mat.

Pose

Beam

Per-vtx

Bound.

\nabla

Real

Sionna-RT [ 18 ]

\checkmark

Mesh

CIR

\checkmark

\checkmark

\checkmark

–

No BRDF

\checkmark

–

SFCW Inv. [ 17 ]

\checkmark

Mesh

–

\checkmark

–

Shading

\checkmark

Pos.

–

\checkmark

–

\checkmark

InverTwin [ 10 ]

–

Mesh

CIR

\checkmark

\checkmark

–

\checkmark

–

No BRDF

\checkmark

–

\checkmark

\checkmark

mmIR (Ours)

\checkmark

Mesh

ADC

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

\checkmark

Table 1: Feature comparison of physics-based differentiable radar renderers. Signal: ADC = raw FMCW, CIR = channel impulse response, IF = intermediate-frequency phasors. Multi-b. = multi-bounce; Diffr. = edge diffraction; Polar. = polarization; Geom. = differentiable geometry; Norm. = differentiable normals; Mat. = physics material model; Per-vtx = per-vertex materials; Bound.

\nabla

= boundary gradients; Real = validated on real data. Checkmarks denote supported pipeline capabilities; see Sec. in the supplement for which parameters are active in our experiments.

Figure 2: mmIR pipeline. Given a LiDAR mesh and sensor parameters, the differentiable model traces phase-coherent MIMO paths (reflection + diffraction) into raw FMCW ADC. End-to-end AD (ray tracing, BSDF, phasor accumulation, ADC) optimizes per-vertex materials, normals, and antenna beam patterns from a range–azimuth loss.

Figure 3: Antenna configurations: cascaded MMWCAS (12Tx $\times$ 16Rx with 136 unique virtual elements) used for training, single-chip AWR1843 (3Tx $\times$ 4Rx with 12 unique virtual elements) used for transfer evaluation, dense array (100Tx $\times$ 100Rx, 10,000 virtual elements) used for 3D inference. Top: physical Tx/Rx; bottom: virtual aperture. The cascade’s near-1D virtual aperture yields $256{\times}127{\times}1$ RAE resolution (Sec. , ); only the dense array resolves true elevation at $256{\times}127{\times}127$ RAE resolution.

Outdoor (7 scenes)
	RA Correlation $\uparrow$		RA PSNR $\uparrow$		RA SSIM $\uparrow$		RA RMSE $\downarrow$		ADC Log-Mag MSE $\downarrow$		ADC MM-Mag MSE $\downarrow$
Scene	Ours	Sionna-RT	Ours	Sionna-RT	Ours	Sionna-RT	Ours	Sionna-RT	Ours	Sionna-RT	Ours	Sionna-RT
S0-F135	0.921	0.418	37.4	21.4	0.938	0.509	0.0135	0.0850	48.3271	244.8361	0.0421	0.0469
S0-F390	0.936	0.242	41.0	23.4	0.937	0.550	0.0089	0.0674	5.0250	294.3032	0.0289	0.0417
S1-F185	0.954	0.312	39.8	18.9	0.926	0.326	0.0102	0.1138	32.3385	282.6453	0.0398	0.0461
S1-F438	0.942	0.346	37.7	19.9	0.926	0.401	0.0130	0.1008	32.5274	288.9782	0.0369	0.0435

Table 2: Range–azimuth (RA) and ADC training evaluation on seven outdoor and six indoor scenes. mmIR is compared against Sionna-RT [ 18 ] . Metrics: Pearson correlation (Corr), PSNR, SSIM, RMSE, and ADC-level magnitude error. Best in bold . Results use the min–max-normalized RA loss (Eq. 8 ).

Figure 4: Qualitative RA on the seven outdoor scenes, sorted by mmIR correlation. Rows: scene mesh (red sphere = radar, green arrow = boresight), GT, mmIR, Sionna-RT. mmIR captures dominant scatterers and multipath; Sionna-RT is weaker and misaligned.

Figure 5: As Fig. 4 , for six indoor scenes (lab, hallway, classroom). mmIR tracks the GT returns; Sionna-RT over-responds in dense indoor multipath.

	RA Correlation $\uparrow$		RA PSNR $\uparrow$		RA SSIM $\uparrow$		RA RMSE $\downarrow$		ADC Log-Mag MSE $\downarrow$		ADC MM-Mag MSE $\downarrow$
Scene	Ours	Bench.	Ours	Bench.	Ours	Bench.	Ours	Bench.	Ours	Bench.	Ours	Bench.
S0-F135	0.533	0.002	25.6	17.8	0.505	0.464	0.0524	0.1283	10.9639	301.4341	0.2194	0.0668
S0-F390	0.464	0.328	21.2	24.2	0.529	0.566	0.0873	0.0619	12.9644	331.2549	0.0630	0.0840
S1-F185	0.567	0.136	22.4	20.1	0.422	0.480	0.0759	0.0990	0.5754	324.8441	0.1993	0.0771
S1-F438	0.684	0.079	23.2	13.5	0.674	0.202	0.0694	0.2119	21.8672	348.0237	0.1507	0.0662
S2-F105	0.608	0.084	24.7	22.8	0.609	0.690	0.0583	0.0721	3.4458	289.7929	0.1959	0.1068

Table 3: Cross-sensor transfer evaluation. Parameters trained on the cascade (12Tx

\times

16Rx) are frozen and used to render the single-chip AWR1843 (3Tx

\times

4Rx) without re-training. Best in bold .

	Precision $\uparrow$		Recall $\uparrow$		F1 $\uparrow$		Accuracy $\uparrow$		RMSE $\downarrow$		R-CD $\downarrow$
Scene	Ours	ColoR	Ours	ColoR	Ours	ColoR	Ours	ColoR	Ours	ColoR	Ours	ColoR
S0-F135	0.077	0.223	0.018	0.054	0.029	0.087	0.044	0.057	2.324	3.302	0.248	0.228
S0-F390	0.481	0.045	0.110	0.014	0.179	0.021	0.212	0.017	2.692	2.745	0.216	0.207
S1-F185	0.997	0.483	0.566	0.082	0.722	0.141	0.703	0.103	0.909	2.380	0.067	0.236
S1-F438	0.633	0.433	0.265	0.117	0.373	0.185	0.362	0.131	1.511	3.339	0.116	0.268
S2-F105	0.771	0.154	0.162	0.025	0.268	0.044	0.340	0.029	1.099	3.402	0.113	0.282

Table 4: 3D occupancy vs. GT LiDAR. Baseline: ColoRadar’s cascaded-radar point cloud with elevation extrusion added in post-processing. Ours: mmIR’s RAE rendered via the dense virtual aperture (100

\times

100 elements), resolving true elevation. KD-tree match at

\tau{=}0.5

m. Best in bold .

Figure 6: Single-frame 3D occupancy on the LiDAR mesh. Top: mmIR’s rendered RAE via the dense virtual aperture (100 $\times$ 100 elements), thresholded at the 99.9th intensity percentile. Bottom: ColoRadar-provided RAE, obtained from the near-1D cascaded radar with elevation extrusion added in post-processing. Red sphere = radar, green arrow = boresight.

Explore similar work

RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception
Jun 24, 2026Emily Bejerano, Federico Tondolo, Devang Gupta +5Sim-to-Real Transfer Millimeter-Wave Radar
Columbia University New York, NY, USA · University of California, Merced Merced, CA, USA
3D Point Splatting for mmWave Radar Novel View Synthesis
Sep 10, 2026Adnan Armouti, Yixuan Gao, Rajalakshmi NandakumarDifferentiable Rendering Millimeter-Wave Radar
Cornell Tech New York, NY, USA
mmRadarTwin: A Measurement-Calibrated Signal-Level Digital Twin Platform for Indoor mmWave Radar
Jul 30, 2026Jianyi Zhou, Chenghao Zhang, Yanli Li +1Millimeter-Wave Radar mmWave Radar Sensing
The University of Sydney · Sydney, NSW, Australia

Jun 24, 2026, cs.CVJ/K move · Enter open · S save

RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception

Emily Bejerano, Federico Tondolo, Devang Gupta, Aaron Mano Cherian+4

Columbia University New York, NY, USA · University of California, Merced Merced, CA, USA

Sim-to-Real Transfer Millimeter-Wave Radar mmWave Radar Sensing Synthetic Data Generation for Robotics

Open paper AskPDF arXiv

Millimeter-wave (mmWave) radar perception is limited by data scarcity: models trained on existing radar datasets fail to generalize to new objects, environments, and sensing trajectories. We present RadarTwin, a framework for generating deployment-specific radar training data before real data collection. Given a 3D reconstruction of a target space (phone LiDAR, robot-mounted sensing, or RGB-to-3D), RadarTwin uses a vision-language model to infer radar-relevant surface materials and a physics-based ray tracer to synthesize raw frequency-modulated continuous-wave (FMCW) radar measurements with multi-bounce propagation. To study what transfers from simulation to reality, we collect a paired real-simulated dataset spanning household objects, material classes, distances, rotations, translations, and mobile sensing trajectories. We show that simulated and real radar share the same object-discriminative shape and material features, and that modeling the environment's multipath is essential to matching real measurements. A representation trained on simulation alone recognizes real objects at 2.5 times chance with no real radar labels, and a few labeled examples raise this to 95.3% on a 12-way recognition task. RadarTwin enables training radar perception for a new space before any real radar data is collected there.

Abstract

Explore similar work

RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception

3D Point Splatting for mmWave Radar Novel View Synthesis

mmRadarTwin: A Measurement-Calibrated Signal-Level Digital Twin Platform for Indoor mmWave Radar