Millimeter-wave radar resolves the micrometer displacements of a moving body as micro-Doppler, enabling gait, gesture, activity, pose and vital-sign sensing. Each task relies on a downstream model trained on real data, yet radar datasets lag orders of magnitude behind vision and language. Brute-force collection will not close that gap, and existing synthesis methods (forward-only simulators, learned generators and static-scene inverse renderers) cannot fit a real capture of a moving body, leaving a sim2real gap. We present 4D Point Splatting (4DPS), a physics-based differentiable renderer that fits an explicit moving scene to a real capture. Using the synchronized camera to initialize the room and body, we represent the scene as oriented points carrying material, normal and velocity, and splat each point into range and Doppler bins. Our output is the raw complex return a real radar records, so we apply the same signal processing on the rendered data to obtain any representation for training any SOTA downstream model. We fit 235 sequences on HuPR and 100 on RT-Pose, two datasets with different radars. Our reconstructions set the state of the art, with median correlations of 0.940 and 0.830 on range-azimuth, 0.829 and 0.695 on range-Doppler, and 0.988 and 0.966 on micro-Doppler. Our renders also set the state of the art in downstream sim2real gap. On RT-Pose, a pose model trained on them reaches 21.9 cm MPJPE on real data, against 16.7 cm for real training and 127.6 cm for the RF-Genesis baseline, where both the rendered- and real-trained models learn nearly the same features. Our custom CUDA kernels fit a sequence in 5.7 minutes on HuPR and 12.9 on RT-Pose on one H100, making dataset-scale inverse rendering of mmWave radar possible for the first time. This work can enable synthetic data generation for any human-centered downstream task and model, and scene editing for data augmentation.
Figures & tables
Figure 1: 4DPS fits a moving scene to a real radar capture. Top: from a camera-initialized scene of oriented points, 4DPS fits materials, normals and velocities (color: radial velocity) until the rendered range–azimuth–elevation–Doppler tensor matches the recording. Bottom: the fitted scene can be edited (App. G ) and rendered as training data for pose, vital-sign, activity and fall models.
Method
Repr.
Signal
Complex
Mat.
Vel.
Doppler
Real
Edit
Code
Differentiable radar inverse rendering
mmIR ( Armouti et al., 2026b )
Mesh
ADC
✓
✓
–
–
✓
✓
✓
3DPS ( Armouti et al., 2026a )
Points
ADC
✓
✓
–
–
✓
✓
✓
Human micro-Doppler simulation and synthesis
RF-Genesis ( Chen & Zhang, 2024 )
Mesh
ADC
✓
–
–
✓
✓
–
✓
HybridSim ( Xiong et al., 2026 )
Mesh
ADC
✓
✓
–
✓
✓
✓
–
Table 1: Comparison of radar rendering and synthesis methods. Signal: CIR or raw ADC. Mat.: explicit materials; Vel.: velocity estimated; Doppler: from subject motion; Real: validated on real captures; Edit: re-renderable scene; Code: released. Radar ports of neural fields and Gaussian splatting are omitted, because they supervise post-FFT power of a static scene.
Figure 2: 4DPS overview. Camera-initialized oriented point sets for a static scene and a moving person carry normals, materials and, for the person, velocities. For every antenna pair, 4DPS splats each point into its Doppler and range bins with an amplitude and a carrier phase, forming the RAED tensor that L compares with the recording. Green: forward pass; red: gradients; gray: fixed inputs.
Range–angle
Range–Doppler
Micro-Doppler
Method
ρ↑
ρc↑
ρM↑
LU→1
ρμ↑
ρσ↑
ρS↑
HuPR , two single-chip radars
RF-Genesis ( Chen & Zhang, 2024 )
0.134
0.339
0.518
0.05
0.820
0.063
0.671
4DPS (unfit)
0.878
0.270
0.537
0.57
0.930
0.283
0.754
4DPS (Ours)
0.940
0.666
0.829
0.92
0.993
0.966
0.988
RT-Pose , cascade, one-person sequences
Table 2: Rendering fidelity. Medians over sequences (HuPR pools both radars) of the Pearson correlation with the recording: range–angle ( ρ ; ρc clutter-removed), occupied range–Doppler cells ( ρM ), and micro-Doppler centroid, spread and spectrogram ( ρμ , ρσ , ρS ); LU : rendered-to-real noise-floor level. Bold: best per section. App. D gives the other DSP representations.
Figure 3: Real, 4DPS and RF-Genesis on three RT-Pose test sequences. Rows: micro-Doppler for the full sequence; typical range–Doppler frame; and typical range–azimuth frame; the number is the render’s sequence-level correlation with the recording. Activities: 159 stand and lift a leg, 147 stand and wave, 166 walk and wave. See App. J for more details.
Figure 4: The sim-to-real gap. Left: real-data MPJPE on RT-Pose ( ↓ ) and AP on HuPR ( ↑ ) after training on real data, 4DPS or RF-Genesis (four seeds, ± 1 std; bars on their true scale). Right: retention of real-trained accuracy per keypoint group (S → R over R → R, the inverse for MPJPE); on HuPR the 4DPS deficit sits in the arms (shaded). Table 3 gives every metric.
Feature similarity
Linear readout
CKA first
CKA middle
CKA backbone
CKA logits
Procrustes every layer
Rendered- trained
Untrained backbone
RT-Pose
1.35 , 1.17
0.98–1.10
0.98 †
0.99 †
0.98–1.05
1.16 [1pt][1.09, 1.28]
1.25
HuPR
1.15 1.37
undefined
1.08 † 1.10 †
1.37 [1pt][1.08, 1.70]
1.05–1.17
1.30 [1pt][1.20, 1.38]
2.50
Table 4: Do rendered- and real-trained models learn the same features? Each cell is the real–rendered distance over the real–real distance between seeds (1.0: as close as two real-trained models), with its 95% interval; the readout columns are error ratios. Bold: the interval lies above the 1.2× margin fixed before the analysis. † : uninformative, as an untrained network is about as close. HuPR’s middle layers are undefined (dead temporal merges); App. F gives every layer and the error-consistency analysis.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Quantity
Shape
Note
Learned
joint velocity scales sj , residual cjk
(14), (bands, 14, 3)
each residual component bounded by 0.75∥vjkin∥ ; 3.5 Hz cap
person normals ni
(visible points, 3) per frame
one fused parameter over the training frames
person material θ
(6)
one six-vector for the body, initialized by the person stage
scene materials θi , normals
(N,6), (N,4)
normals as quaternions; N=20,000
receiver gain g , joint gains γj
(1) per radar, (14)
log-parameterized; ∏jγj=1
Appendix
Table 5: Learned and fixed quantities of the forward model. Materials and normals are parameterized as in Armouti et al. (2026a) . Learning rates: velocity coefficients 10−2 , person normals 10−3 , scene materials 10−2 , scene normals 5⋅10−3 , person material 10−3 , gains 10−2 , noise floor 3⋅10−2 (Adam).
Dataset
ST
ADC
Cam
Pose
Pub
Code
Selected
RT-Pose ( Ho et al., 2025 )
✓
✓
✓
✓
✓
✓
HuPR ( Lee et al., 2023 )
✓
✓
✓
✓
✓
✓
No slow time (human sensing)
MMVR ( Rahman et al., 2025 )
×
×
×
✓
✓
∘
HIBER ( Wu et al., 2023 )
×
×
✓
✓
∘
∘
Appendix
Table 6: Dataset survey. ST : slow time retained, at least two chirps per frame. ADC : raw ADC or I/Q samples, not heatmaps or point clouds. Cam : colocated camera imagery released, not merely used to generate labels. Pose : human pose labels (keypoints or a body mesh). Pub : openly downloadable. Code : reference downstream model code released. ✓ satisfied, × not satisfied, ∘ partially satisfied (access by request or agreement, a withheld or blurred modality, an expired mirror, chirps per frame undocumented, a released cube already transformed along range, angle and Doppler, or model code without training scripts or only from later work). Only RT-Pose and HuPR satisfy all six. Pretrained weights are not a criterion: RT-Pose publishes none, so we train its reference model ourselves (App. E ).
Method
ρenv↑
PSNR ↑
SSIM ↑
RMSE ↓
Δϕt↑
Δϕt ( ∘ ) ↓
HuPR , horizontal single-chip radar (H)
RF-Genesis ( Chen & Zhang, 2024 )
0.649
14.0
0.37
0.205
-0.21
109
4DPS (Ours)
0.857
21.2
0.58
0.088
0.55
63
HuPR , vertical single-chip radar (V)
RF-Genesis
0.629
14.0
0.34
0.203
0.11
90
4DPS (Ours)
0.772
19.0
0.48
0.114
0.44
72
Appendix
Table 7: Rendering fidelity on the ADC samples. Table 2 of the main text carries the seven columns its prose reads; this table and Tables 8 – 10 carry all of them, grouped by representation, over the same sequences and the same renders, with H, V and C as there. ρenv is the envelope correlation after the rank-1 phase correction; Δϕt is the relative-phase coherence along fast time ( 1 is the recording’s phase step, 0 random), with its weighted RMSE in degrees. The unfit render is not scored. Every column here is scale-free, so RF-Genesis is scored as rendered. The ADC phase depends on the scene: on RT-Pose its median Δϕt is 0.51 on the clean recordings and 0.23 on the cluttered ones. Bold: the better of the two methods in each column of a row section.
Method
ρ↑
PSNR ↑
SSIM ↑
RMSE ↓
Δϕr↑
Δϕr ( ∘ ) ↓
Δϕv↑
Δϕv ( ∘ ) ↓
HuPR , horizontal single-chip radar (H)
RF-Genesis
0.246
20.0
0.28
0.102
-0.63
147
0.52
67
4DPS (Ours)
0.856
26.0
0.65
0.052
0.66
56
0.75
48
HuPR , vertical single-chip radar (V)
RF-Genesis
0.274
20.0
0.19
0.102
-0.52
139
0.31
82
4DPS (Ours)
0.785
24.8
0.56
0.061
0.55
65
0.47
71
Appendix
Table 8: Rendering fidelity on the complex range profile (CRP). Sequences, renders and row sections as in Table 7 . Δϕr and Δϕv are the relative-phase coherences along range and virtual channel ( 1 is the recording’s phase step, 0 random), each with its weighted RMSE in degrees. The unfit render is not scored. Every column here is scale-free, so RF-Genesis is scored as rendered. Its negative Δϕr is a sampling convention: its range profiles step by π per bin against the recording’s, and multiplying them by (−1)k makes the coherence positive and changes no other column; we report it as run. Bold: the better of the two methods in each column of a row section.
R → R model
Dataset
Retention
Information gap
Realism gap
Reverse gap
50/50 test
Real test
HuPR (AP, pts)
0.59
6.0
21.2
16.4
55.8 (1.1)
65.6 (1.7)
RT-Pose (MPJPE, cm)
0.76
1.0
6.1
3.4
18.93 (1.44)
16.67 (0.42)
Appendix
Table 12: Downstream transfer summary. Retention and the three gaps are defined in the text and read in the dataset’s own unit (AP ×100 on HuPR, MPJPE in cm on RT-Pose); the last two columns score the real-trained model on a 50/50 real/rendered test set and on the real test set, mean (std) over four seeds.
HuPR keypoint (AP)
R → R
S → R
R → S
S → S
R+S → R
R_Hip
90.1 (0.6)
72.1 (2.9)
82.7 (1.7)
91.5 (0.8)
89.4 (0.2)
R_Knee
84.5 (0.7)
65.3 (2.8)
76.0 (1.3)
87.7 (0.6)
83.8 (0.5)
R_Ankle
73.3 (1.7)
50.2 (2.6)
73.9 (2.2)
73.6 (1.3)
71.8 (1.6)
L_Hip
90.6 (1.3)
71.7 (3.3)
83.0 (1.4)
90.8 (1.3)
88.9 (0.6)
L_Knee
83.7 (2.2)
61.1 (3.3)
75.7 (1.0)
85.2 (2.6)
82.1 (0.9)
L_Ankle
77.5 (1.3)
63.0 (2.0)
66.2 (1.3)
80.8 (1.0)
75.0 (1.0)
Appendix
Table 13: Per-keypoint and per-joint downstream accuracy , per arm (train → test; R real, S rendered by 4DPS), same protocol as Table 12 : HuPR AP ×100 per keypoint and [email protected] (top), RT-Pose MPJPE in cm per joint and PCK at 10 and 15 cm (bottom). MPJPE is taken relative to the pelvis, so the pelvis reads 0; PCK on RT-Pose is scored on real test data only (–). Mean (std) over four seeds.
RT-Pose MPJPE (cm) by group
R → R
S → R
R+S → R
activity
Stand and Lift Leg
9.81 (0.20)
14.45 (1.60)
9.77 (0.27)
activity
Stand and Wave Hand
17.08 (1.08)
22.03 (0.81)
18.66 (1.70)
activity
Walk
21.67 (2.14)
25.69 (2.54)
19.23 (0.43)
activity
Walk and Sit
10.43 (0.51)
25.28 (1.00)
10.77 (1.20)
activity
Walk and Wave Hand
14.58 (0.28)
17.05 (1.31)
14.18 (0.41)
location
Indoor
19.40 (0.88)
23.89 (1.63)
17.92 (1.09)
Appendix
Table 14: RT-Pose MPJPE (cm) by group : the test sequences’ activity, location, occlusion and scenario, for the three arms scored on real test data, same protocol as Table 12 ; mean (std) over four seeds.
Row
Cell
Pairs
Differs in
Reference
same init
real k vs rendered k
4
data
(none trained)
diff. init
real i vs real j
10
init, order
itself
real i vs rendered j
16
init, order, data
real–real, diff. init
beside the design
mixed vs real
4, 16
data (and init)
real–real, diff. init
rendered vs rendered
6
init, order
real–real, diff. init
untrained vs real
5
training itself
real–real, diff. init
Appendix
Table 15: The cells compared , with what differs inside each. Ratios are taken against the real–real cell of the matched row. No real arm was trained twice from one initialization, so the first row has no real–real reference and its cross pairs are read against the different-initialization real–real mean, which bounds their ratio from above.
Frame error corr.
Joint error agr.
RT-Pose
1.13 [1pt][0.99, 1.31]
1.23 [1pt][1.21, 1.26]
HuPR
2.20 [1pt][2.07, 2.34]
2.94 [1pt][2.78, 3.11]
Appendix
Table 17: Do rendered- and real-trained models fail on the same frames? Error consistency as the real–rendered value over the real–real value between seeds, with its 95% interval (1.0: as consistent as two real-trained models). Bold: the interval lies above the 1.2× margin. The cells behind each ratio follow.
debiased CKA (similarity ↑ )
Procrustes
Tap
real–real
real–rend.
floor
ratio
ratio
H1h chirp output
0.976
0.973
0.858
1.15 [0.97, 1.35]
1.08
H2h, H3h, H2v, H3v
undefined: the tap is dead in some checkpoints
–
H4h encoder level 3
0.287
0.229
0.227
1.08 [0.93, 1.27]
1.05
H4v encoder level 3
0.164
0.084
0.044
1.10 [0.96, 1.30]
1.15
H5 decoder logits
0.674
0.554
0.040
1.37 [1.08, 1.70]
1.17
Appendix
Table 18: Representational similarity by tap (HuPR, real test frames), as Table 16 for RT-Pose. H1 is the chirp network’s output, H4 the tensor the decoder reads and H5 its pre-sigmoid heatmap; h and v are the horizontal and vertical boards. The level-1 and level-2 encoder taps are undefined, because they carry no signal in some checkpoints (App. F , dead temporal merges). The rendered probe reproduces every computable cell to within 0.03.
range–azimuth
range–Doppler
micro-Doppler
ρ
ρc
ρM
LU
ρμ
ρσ
ρS
n
RT-Pose, 24 pairs, both directions
Separated ( A ’s scene, B ’s fit)
0.756
0.484
0.696
0.97
0.962
0.679
0.962
48
B ’s own fit
0.843
0.540
0.697
0.97
0.962
0.679
0.962
48
Retention (%)
90
90
100
100
100
100
100
B ’s unfit render
0.660
0.120
0.282
0.58
0.581
0.168
0.497
44
Appendix
Table 19: Separability of the scene and the person. B ’s complete fit rendered in A ’s fitted static scene (“Separated”), scored against B ’s real recording with the evaluation of Table 2 , beside B ’s own fit, B ’s unfit render and RF-Genesis driven by B ’s registered motion, on the same renders and frames. Medians over renders; HuPR pools its two boards, as Table 2 does, so n counts (render, board) products. Retention is the median of the separated render over the median of B ’s own fit, on the same renders. RF-Genesis’s LU comes from its level-matched scoring, as in Table 2 . A column undefined on a render (no frame of the evaluation’s motion subset) has a smaller n than the row’s, which the column shows.
Figure 5: Separability, RT-Pose (top) and HuPR’s horizontal board (bottom). Each block renders the person of one recording in the fitted scene of another recording of the same environment. Columns: the recording (Real), the person’s complete fit rendered in the other scene (Separated), the person’s own fit, and RF-Genesis driven by the same registered motion. Rows, frame, normalization and scores are those of Fig. 3 . We chose the pairs by a rule: the renders at the 25th, 50th and 75th percentile of the mean ratio of the separated render’s three panel scores to the own fit’s.
Figure 6: Novel motions in the fitted scene of RT-Pose sequence 142 (indoor, cluttered) (not scored). Per motion: the posed mesh at four evenly spaced frames, seen from 45 ∘ between the radar’s view and the side, above the rendered micro-Doppler over the whole clip (time across, radial velocity up) and the rendered range–Doppler and range–azimuth at the clip’s frame of largest pelvis speed (range across, over the person’s range ± 2 m). Panels are linear magnitude normalized to their own range, as in Fig. 3 . Falls: forward and backward generated by MDM ( Tevet et al., 2023 ) , the fall from a pulled rug from CMU ( Mahmood et al., 2019 ) ; exercises from CMU.
Figure 7: Novel motions in the fitted scene of RT-Pose sequence 159 (outdoor) (not scored). Per motion: the posed mesh at four evenly spaced frames, seen from 45 ∘ between the radar’s view and the side, above the rendered micro-Doppler over the whole clip (time across, radial velocity up) and the rendered range–Doppler and range–azimuth at the clip’s frame of largest pelvis speed (range across, over the person’s range ± 2 m). Panels are linear magnitude normalized to their own range, as in Fig. 3 . Falls: forward and backward generated by MDM ( Tevet et al., 2023 ) , the fall from a pulled rug from CMU ( Mahmood et al., 2019 ) ; exercises from CMU.
Figure 8: Novel motions in the fitted scene of RT-Pose sequence 186 (indoor, clean) (not scored). Per motion: the posed mesh at four evenly spaced frames, seen from 45 ∘ between the radar’s view and the side, above the rendered micro-Doppler over the whole clip (time across, radial velocity up) and the rendered range–Doppler and range–azimuth at the clip’s frame of largest pelvis speed (range across, over the person’s range ± 2 m). Panels are linear magnitude normalized to their own range, as in Fig. 3 . Falls: forward and backward generated by MDM ( Tevet et al., 2023 ) , the fall from a pulled rug from CMU ( Mahmood et al., 2019 ) ; exercises from CMU.
Figure 9: Novel motions in the fitted scene of HuPR sequence 183 (horizontal board) (not scored). Per motion: the posed mesh at four evenly spaced frames, seen from 45 ∘ between the radar’s view and the side, above the rendered micro-Doppler over the whole clip (time across, radial velocity up) and the rendered range–Doppler and range–azimuth at the clip’s frame of largest pelvis speed (range across, over the person’s range ± 2 m). Panels are linear magnitude normalized to their own range, as in Fig. 3 . Falls: forward and backward generated by MDM ( Tevet et al., 2023 ) , the fall from a pulled rug from CMU ( Mahmood et al., 2019 ) ; exercises from CMU.
Figure 10: Novel motions in the fitted scene of HuPR sequence 41 (horizontal board) (not scored). Per motion: the posed mesh at four evenly spaced frames, seen from 45 ∘ between the radar’s view and the side, above the rendered micro-Doppler over the whole clip (time across, radial velocity up) and the rendered range–Doppler and range–azimuth at the clip’s frame of largest pelvis speed (range across, over the person’s range ± 2 m). Panels are linear magnitude normalized to their own range, as in Fig. 3 . Falls: forward and backward generated by MDM ( Tevet et al., 2023 ) , the fall from a pulled rug from CMU ( Mahmood et al., 2019 ) ; exercises from CMU.
Term removed
ρenv↑
PSNR ↑
SSIM ↑
RMSE ↓
Δϕt↑
Δϕt ( ∘ ) ↓
HuPR , horizontal single-chip radar (H)
None (full objective)
0.857
21.2
0.58
0.088
0.55
63
−LRAE,0
0.742
18.4
0.45
0.121
0.63
56
−LRAE,pl
0.856
21.2
0.58
0.088
0.54
64
−(LRAE,0win+LRAE,plwin)
0.856
21.2
0.58
0.088
0.55
64
HuPR , vertical single-chip radar (V)
Appendix
Table 21: Objective ablation on the ADC samples. Columns as Table 7 ( ρenv the envelope correlation after the rank-1 phase correction, Δϕt the fast-time relative-phase coherence with its weighted RMSE in degrees); rows as Table 20 , over the same sequences (HuPR 235 per board, RT-Pose 63 one-person). Bold: the best value in each column of a row section, by the column’s arrow.
Term removed
ρ↑
PSNR ↑
SSIM ↑
RMSE ↓
Δϕr↑
Δϕr ( ∘ ) ↓
Δϕv↑
Δϕv ( ∘ ) ↓
HuPR , horizontal single-chip radar (H)
None (full objective)
0.856
26.0
0.65
0.052
0.66
56
0.75
48
−LRAE,0
0.667
22.4
0.44
0.076
0.64
58
0.66
57
−LRAE,pl
0.850
25.9
0.62
0.052
0.65
56
0.75
48
−(LRAE,0win+LRAE,plwin)
0.854
26.0
0.65
0.052
0.64
57
0.75
48
HuPR , vertical single-chip radar (V)
Appendix
Table 22: Objective ablation on the complex range profile (CRP). Columns as Table 8 ( Δϕr and Δϕv the relative-phase coherences along range and virtual channel, each with its weighted RMSE in degrees); rows, sequences and bold as Table 21 .
Data
Method
GPU
Sequences
Per sequence (min)
Total (GPU-h)
Step (ms)
HuPR
4DPS
H100
235
5.7
22.3
19.4
4DPS (unfit)
H100
235
1.5
5.9
–
RF-Genesis
RTX 4090
235
5.3
21.0
–
RT-Pose
4DPS
H100
100
12.9
21.6
19.3
4DPS (unfit)
H100
57
1.9
1.9
–
RF-Genesis
RTX 4090
100
6.0
10.6
–
Appendix
Table 25: Reconstruction cost (Sec. 4.1 ): the median wall time per sequence and the total over the corpus. RT-Pose’s unfit render covers the 57 split sequences it is scored on. The 4DPS step is the median over sequences of the training step’s median. RF-Genesis ran on RTX 4090s and 4DPS on H100s, so the two methods’ times are not a like-for-like speed comparison.
HuPR
RT-Pose
Arm
Renderer
GPU
h / training
Epochs
GPU-h
h / training
Epochs
GPU-h
Real
none (real)
H100
6.8 (0.1)
200
27.1
6.5 (0.1)
50
26.2
Rendered
4DPS
H100
6.7 (0.0)
200
26.8
8.4 (0.0)
50
33.7
Mixed
4DPS
H100
6.7 (0.0)
200
26.9
8.2 (0.2)
50
32.9
Rendered
RF-Genesis
H100
6.7 (0.0)
200
26.7
7.7 (0.1)
50
31.0
Mixed
RF-Genesis
H100
6.7 (0.0)
200
26.7
7.9 (0.3)
50
31.5
Appendix
Table 26: Downstream training cost (Sec. 4.2 ): hours per training, mean (std) over the four seeds, and the total over the four seeds. Every training ran on one H100. Evaluating Table 3 ’s five cells over the four seeds adds 0.5 (4DPS) and 0.5 (RF-Genesis) GPU-h on HuPR, and 0.7 and 0.7 on RT-Pose.
HuPR
RT-Pose
Stage
Method / arm
GPU
median
max
median
max
Reconstruction
4DPS
H100
5.7
6.7
18.7
19.6
(Sec. 4.1 )
RF-Genesis
RTX 4090
–
3.3 †
–
10.8 †
Training
real
H100
–
14.9 †
–
11.2
(Sec. 4.2 )
4DPS rendered
H100
–
14.9 †
–
11.2
4DPS mixed
H100
–
14.9 †
–
11.2
Appendix
Table 27: Peak GPU memory (GiB): the median and the maximum over sequences for reconstruction, and the maximum over the four seeds for training. Every 4DPS cell and every training cell is PyTorch’s allocator peak ( max_memory_allocated ), as the 4DPS runs record it per sequence and RT-Pose’s trainer logs it; the reserved pool is larger. The RF-Genesis reconstruction cells are instead the card-level peak from nvidia-smi, because its ray tracer allocates memory outside PyTorch (PyTorch alone records 1.1 and 1.6 GiB), so they are not the same quantity as the 4DPS cells. The cells marked † were measured separately from the runs that produced the paper’s numbers, with the same code, protocol and data: RF-Genesis as the maximum over the shortest, median and longest sequence of each dataset on an RTX 4090, and HuPR training over 300 steps and one validation pass per arm on an H100; the 5 HuPR arms are identical (21.9 GiB at card level).
Figure 11: Qualitative gallery, RT-Pose (1 of 3): sequences 159, 147 and 166 (top) and 180, 148 and 160 (bottom) (4DPS-fidelity ranks 8, 1, 16, 4, 9 and 13 of 63). The top three are the sequences of Fig. 3 . Selection rule, rows, columns, frames and scores are given at the start of App. J .
Figure 12: Qualitative gallery, RT-Pose (2 of 3): sequences 144, 183 and 179 (top) and 152, 110 and 116 (bottom) (4DPS-fidelity ranks 18, 22, 26, 30, 34 and 38 of 63). As Fig. 11 .
Figure 13: Qualitative gallery, RT-Pose (3 of 3): sequences 151, 113 and 178 (top) and 1, 10 and 108 (bottom) (4DPS-fidelity ranks 42, 46, 50, 54, 58 and 62 of 63). As Fig. 11 .
Figure 14: Qualitative gallery, HuPR, horizontal board (1 of 3): sequences 247, 41 and 248 (top) and 192, 172 and 164 (bottom) (4DPS-fidelity ranks 11, 57, 13, 8, 26 and 41 of 235). The top three are the HuPR test sequences ranked like those of Fig. 3 (4DPS fidelity plus rendered-to-real OKS). A single board resolves elevation poorly, so the range–azimuth–elevation rows are coarse along that axis.
Figure 15: Qualitative gallery, HuPR, horizontal board (2 of 3): sequences 107, 30 and 96 (top) and 165, 93 and 27 (bottom) (4DPS-fidelity ranks 58, 73, 89, 104, 120 and 135 of 235). As Fig. 14 .
Figure 16: Qualitative gallery, HuPR, horizontal board (3 of 3): sequences 194, 252 and 56 (top) and 158, 67 and 60 (bottom) (4DPS-fidelity ranks 150, 166, 181, 197, 212 and 228 of 235). As Fig. 14 .
Solving novel view synthesis (NVS) for millimeter-wave (mmWave) radar requires a renderer that is physically faithful, complex-valued, and multi-viewpoint-tractable. No prior method achieves these three properties simultaneously. Differentiable Monte Carlo (MC) ray tracers implement the radar forward model directly with explicit material modeling and complex outputs, but do not scale to the multi-view optimization NVS demands. Optical-NVS ports of NeRF, hash grids, and 3D Gaussians train fast but discard phase and replace explicit material modeling with opaque learned features, restricting them to power-only range-azimuth (RA) magnitudes. We propose 3D Point Splatting (3DPS), the first differentiable point renderer for radar, derived directly from the standard solid-angle form of the radar equation. Each oriented 3D point carries an ITU-R P.2040 material model, evaluated in closed form, with the resulting complex phasor splatted into range bins through a precomputed point spread function (PSF). The complex-valued output makes the renderer product-agnostic. The same optimized scene yields analog-to-digital converter (ADC), complex range profile (CRP), and RA outputs through standard fast Fourier transform (FFT) pipelines without retraining for each format. On six outdoor ColoRadar scenes, 3DPS reaches 0.587 mean Pearson correlation on held-out RA images. This is between 1.7x and 5.2x the three optical-NVS baselines (RadarSplat, Radar Fields, DART). Training takes approximately 3 minutes per scene on a single RTX 4090.
High-resolution 3D radar data is scarce. Commodity mmWave sensors use small antenna arrays that limit angular resolution to several degrees, and existing datasets provide only 2D range-azimuth maps or sparse point clouds rather than raw analog-to-digital converter (ADC) signals. Hardware scaling is expensive, synthetic-aperture scanning is impractical at fleet scale, and learned synthesis methods are bottlenecked by the very data shortage they aim to address. We present mmIR, an open-source differentiable frequency-modulated continuous-wave (FMCW) radar inverse renderer that fits a physics-based forward model to real captures and re-renders from dense virtual apertures to synthesize high-resolution 3D radar data. Because radar resolution is too coarse to recover geometry directly, mmIR performs LiDAR-assisted inverse rendering: using LiDAR-derived meshes as a geometric scaffold, mmIR optimizes per-vertex International Telecommunication Union (ITU) physics materials, vertex normals, and antenna beam patterns through end-to-end automatic differentiation of a phase-coherent multiple-input multiple-output (MIMO) forward model with multi-bounce propagation, polarization, and free-space diffraction. On seven outdoor and six indoor ColoRadar scenes, mmIR achieves 0.914 mean Pearson correlation on range-azimuth maps versus 0.307 for Sionna-RT. Scenes trained on a cascaded imaging radar transfer to a co-located single-chip radar without re-training (0.554 correlation), and dense virtual arrays (100x100 elements) produce single-frame 3D occupancy validated against LiDAR. Project page: https://mmwave-inverse-rendering.github.io/
Radars are an ideal complement to cameras: both are inexpensive, solid-state sensors, with cameras offering fine angular resolution, while radars provide metric depth and robustness under adverse weather. However, radar data is more difficult to interpret than camera images and varies significantly between sensors, necessitating increased reliance on simulation for prototyping sensors and processing pipelines. Recent work treating radar reconstruction as a novel view synthesis problem has shown great promise in reconstructing radar-relevant geometry and simulating low-level radar data. However, such methods are constrained by the low spatial resolution of the underlying radar. To address this, we propose a unified differentiable renderer, RadarSim, which leverages the high angular resolution of RGB cameras to generate Doppler radar range images from a camera-initialized neural field. Using a novel data set of calibrated radar camera recordings from a custom hand-held rig, we demonstrate that RadarSim produces sharper geometry and Doppler range frames than radar-only reconstructions.
Chuhan Chen, Tianshu Huang, Akarsh Prabhakara +5
Carnegie Mellon University · Bosch Research · University of Wisconsin–Madison