Learning the Missing Derivative: Neural Multiple-Scattering Operators for Inverse Volume Rendering
Organizations: Department of Computer Science, Florida State University
Abstract
Reconstructing optically thick media from photographs requires modeling multiple scattering and its response to changes in extinction. Frozen-source tomography discards this derivative; differentiable path tracing estimates it through expensive, noisy sampling. We instead predict the multiple-scattering field with a neural operator and differentiate through it during inverse rendering, while computing transmittance and single scattering directly. A further challenge arises during optimization: an operator that predicts the field accurately on held-out media can still produce poor derivatives at the density states visited by the optimizer. We address this by fine-tuning on states generated during inversions driven by the operator, using Monte Carlo scattering-field supervision. On held-out optically thick clouds, our method improves relighting over frozen-source tomography at equal time and outperforms differentiable path tracing given substantially more time. Training on inversion trajectories also improves derivative alignment at visited states and reduces optimization drift.
Figures & tables
| Method | Jacobian | 30 s | 2 min | 8 min | 16 min |
|---|---|---|---|---|---|
| Median PSNR [dB] | |||||
| Frozen Source [ 27 ] | 31.3 | 35.1 | 40.1 | 40.0 | |
| Detached Operator | 32.2 | 32.8 | 38.6 | 39.9 | |
| PRB [ 48 ] | MC est. of | 24.4 | 28.4 | 31.2 | 32.3 |
| Ours | 37.3 | 42.5 | 42.9 | 42.9 | |
| Median paired gain of Ours [dB] | |||||
| operator quality (development) | inversion performance | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| field error | cosine | drift | final iterate, main | selected, 16 min | |||||
| Operator | generic | visited | next step | controlled | main | 8 min | 16 min | controlled | main |
| generic | 0.0228 | 0.0725 | 0.15 | 6.42 | – | – | – | 42.8 | – |
| generic+ (equal steps) | 0.0229 | 0.0693 | – | 6.74 | – | – | – | 43.2 | – |
| scope | 0.0222 | 0.0630 | 0.13 | 4.09 | 7.66 | 40.0 | 30.5 | 43.7 | 42.4 |
| scope+ (equal steps) | 0.0214 | 0.0597 | 0.15 | 3.74 | 5.54 | 40.3 | 30.8 | 43.9 | 42.6 |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Operator (30k steps) | field rel. | image PSNR | MS PSNR |
|---|---|---|---|
| reference (U-Net) | 0.0411 | 52.52 | 48.00 |
| world-frame outputs | 0.0484 | 50.28 | 45.76 |
| source predicted directly (no analytic ) | 0.0419 | 52.46 | 47.94 |
| no forward-cascade features | 0.0432 | 52.08 | 47.56 |
| fixed offline set (4000 media) | 0.0369 | 53.50 | 48.98 |
| FNO backbone | 0.0537 | 49.95 | 45.43 |
| Operator | starts from | training data | steps |
|---|---|---|---|
| generic | – | generic (on line) | |
| generic+ | generic | generic | |
| scope | generic | generic scope | |
| scope+ | scope | generic scope | |
| trajectory | scope | generic scope visited |
| value error | derivative (traj.) | far field | image change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| # | family | traj. | scope | generic | cosine | share | cosine | Frozen Source | ours, cos. | ||
| 0 | procedural | 89.1 | 0.025 | 0.027 | 0.043 | 0.95 (0.98) | 0.25 | 0.49 | 0.97 | 0.1% | 0.91 |
| 1 | procedural | 45.2 | 0.025 | 0.027 | 0.036 | 0.99 (1.00) | 0.13 | 0.32 | 0.99 | 1.2% | 0.99 |
| 2 | procedural | 51.1 | 0.031 | 0.033 | 0.042 | 0.98 (0.99) | 0.16 | 0.37 | 0.99 | 1.1% | 0.99 |
| 3 | procedural | 64.1 | 0.041 | 0.047 | 0.064 | 0.98 (0.99) | 0.12 | 0.25 | 0.99 | 1.5% | 0.99 |
| 4 | procedural | 50.3 | 0.032 | 0.037 | 0.047 | 0.97 (0.99) | 0.21 | 0.58 | 0.99 | 1.2% | 0.98 |
| gen. | scope | visited | visited, by density error | |||
|---|---|---|---|---|---|---|
| operator | val | val | states | |||
| generic | 0.0228 | 0.0441 | 0.0725 | 0.045–0.057 | 0.079 | 0.119 |
| generic+ | 0.0229 | 0.0420 | 0.0693 | 0.043–0.054 | 0.076 | 0.115 |
| scope | 0.0222 | 0.0337 | 0.0630 | 0.035–0.046 | 0.072 | 0.114 |
| scope+ | 0.0214 | 0.0313 | 0.0597 | 0.032–0.044 | 0.068 | 0.109 |
| trajectory | 0.0216 | 0.0315 | 0.0446 | 0.030–0.038 | 0.048 | 0.064 |
| Direction | density error | generic | scope | scope+ | trajectory | |
|---|---|---|---|---|---|---|
| next step | 6 | / | / | / | / | |
| 10 | / | / | / | / | ||
| 8 | / | / | / | / | ||
| trajectory below scope on 24/24 (median ratio 0.82), below scope+ on 24/24 (0.84) | ||||||
| drift | 5 | / | / | / | / | |
| 7 | / | / | / | / | ||
| selected | final iterate | traj., final | ||||
|---|---|---|---|---|---|---|
| Budget | scope | traj. | scope | Frozen Source | selected | scope |
| 30 s | 41.04 | 41.32 | 41.29 | 33.10 | (6) | (2) |
| 2 min | 44.12 | 44.52 | 44.07 | 34.08 | (5) | (6) |
| 8 min | 44.16 | 43.91 | 40.08 | 37.93 | (5) | (8) |
| 16 min | 44.16 | 42.30 | 35.79 | 38.30 | (2) | (8) |
| Config. | spp | lr | c2f | Adam | 30 s | 120 s | 480 s | 960 s | 1800 s |
|---|---|---|---|---|---|---|---|---|---|
| P16b | 16 | 0.04 | fast | 28.37 | 29.62 | 27.74 | – | – | |
| P16 | 16 | 0.01 | slow | 26.26 | 29.29 | 30.64 | 30.24 | 29.40 | |
| P30d | 64 | 0.16 | fast | 29.65 | 26.83 | – | – | – | |
| P30c | 64 | 0.08 | fast | 29.46 | 29.35 | 25.71 | – | – | |
| P30b | 64 | 0.04 | fast | 28.72 | 30.84 | 29.18 | 27.28 | 25.32 | |
| P30 | 64 | 0.02 | fast | 27.13 | 29.87 | 31.05 | 30.19 | 29.17 |
| Claim | criterion on the 12 media of the main evaluation | development basis | main evaluation |
|---|---|---|---|
| P1 | Ours Frozen Source (better prior) after 2 min: median 4 dB and ahead; after 16 min: median 2.5 dB | , 8; , 8 | , 12; , 12; pass |
| P2 | Ours Detached Operator after 2 min: median 3.5 dB | , 8 | , 12; pass |
| P3 | Ours after 2 min PRB (better configuration) after 30 min on | – | 12 ( ); pass |
| P4 | Ours Ours with the scope operator, mean over 2–16 min (estimate) | (4 Cloudy) | [0.26, 1.14], 11 |
| P5 | trajectory scope, final iterates after 8 min: median 0.5 dB and ahead | , 8 | , 11; pass |
| P5b | trajectory scope+, the same | , 8 | , 11; pass |
| 8 min | 16 min | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| # | family | Ours | traj. | scope | scope+ | Ours | traj. | scope | scope+ | |
| 0 | procedural | 89.1 | 41.40 | 41.65 † | 41.55 † | 41.03 † | 41.40 | 37.34 † | 30.79 † | 30.76 † |
| 1 | procedural | 45.2 | 46.77 | 46.93 | 47.22 | 47.33 | 46.77 | 47.03 | 45.13 | 46.82 |
| 2 | procedural | 51.1 | 45.07 | 45.10 | 44.32 † | 44.40 † | 45.30 | 45.22 | 29.77 † | 29.91 † |
| 3 | procedural | 64.1 | 49.14 | 47.95 | 39.62 † | 41.18 † | 49.14 | 47.16 | 33.93 † | 34.58 † |
| 4 | procedural | 50.3 | 42.55 | 43.22 † | 42.24 † | 42.66 † | 42.55 | 41.81 † | 32.50 † | 36.54 † |
| Comparison | 30 s | 2 min | 8 min | 16 min |
|---|---|---|---|---|
| Ours Frozen Source (TV ) | +6.4 [5.0, 8.9] 12/12 | +10.7 [7.2, 13.9] 12/12 | +4.4 [3.0, 5.7] 12/12 | +4.2 [2.9, 5.2] 12/12 |
| Ours Frozen Source (TV ) | +5.3 [4.9, 7.1] 12/12 | +6.9 [6.1, 10.7] 12/12 | +6.2 [3.6, 7.7] 12/12 | +6.3 [3.6, 7.8] 12/12 |
| Ours Frozen Source (better prior) | +5.1 [4.7, 7.1] 12/12 | +6.8 [5.5, 10.7] 12/12 | +4.4 [2.4, 5.7] 12/12 | +4.0 [2.3, 5.1] 12/12 |
| Ours Detached Operator | +5.2 [3.7, 7.1] 12/12 | +10.2 [5.3, 12.3] 12/12 | +5.9 [4.0, 8.8] 12/12 | +5.1 [3.4, 6.1] 12/12 |
| Ours Ours w/ scope op. | +0.0 [ 0.1, 0.3] 7/12 | +0.5 [ 0.1, 0.7] 9/12 | +0.9 [0.4, 1.5] 11/12 | +1.0 [0.4, 1.5] 11/12 |
| Ours Ours w/ scope+ op. | +0.1 [0.0, 0.3] 9/12 | +0.3 [0.0, 0.5] 9/12 | +0.7 [0.2, 1.0] 9/12 | +0.7 [0.1, 1.0] 9/12 |
| Arm | iterations | evaluated density error | current, 16 min | div. | anchor error | ||
|---|---|---|---|---|---|---|---|
| 2 min | 16 min | 2 min | 16 min | median ( ) | |||
| Ours (trajectory op.) | 1960 | 16180 | 0.55 | 0.70 | 1.49 (4) | 0 | 20% |
| Ours w/ scope op. | 1992 | 15754 | 0.58 | 0.58 | 7.66 (11) | 0 | 67% |
| Ours w/ scope+ op. | 1989 | 16198 | 0.53 | 0.60 | 5.54 (10) | 0 | 54% |
| trajectory, final iterate | 2030 | 16897 | 0.56 | 1.32 | 1.32 (4) | 4 | 21% |
| scope, final iterate | 1972 | 15924 | 0.60 | 7.35 | 7.35 (11) | 11 | 61% |
| Arm | 2 min | 8 min | 16 min | 30 min |
|---|---|---|---|---|
| Ours (trajectory op.) | 0.55 / 0.20 | 0.70 / 0.24 | ||
| Ours w/ scope op. | 0.58 / 0.21 | 0.58 / 0.21 | ||
| trajectory, final iterate | 0.81 / 0.31 | 1.32 / 0.49 | ||
| scope, final iterate | 2.05 / 0.65 | 7.35 / 3.61 | ||
| scope+, final iterate | 1.80 / 0.56 | 6.92 / 3.37 | ||
| Frozen Source (TV ) | 1.29 / 0.68 | 0.57 / 0.31 |
| Ours, 2 min | Frozen Source, 2 min | Detached Operator | PRB, 2 min | PRB | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # | family | traj. | scope | scope+ | better | 2 min | lr .005 | lr .01 | better | 30 min | |||
| 0 | procedural | 89.1 | 40.42 | 39.56 | 40.22 | 27.22 | 27.40 | 27.40 | 27.14 | 24.40 | 25.98 | 25.98 | 28.41 |
| 1 | procedural | 45.2 | 45.94 | 45.84 | 45.99 | 40.87 | 39.26 | 40.87 | 40.20 | 29.94 | 32.33 | 32.33 | 37.40 |
| 2 | procedural | 51.1 | 43.53 | 43.41 | 43.51 | 41.34 | 38.72 | 41.34 | 40.85 | 29.47 | 31.24 | 31.24 | 36.73 |
| 3 | procedural | 64.1 | 47.44 | 47.81 | 48.13 | 38.13 | 41.19 | 41.19 | 43.16 | 33.52 | 35.23 | 35.23 | 39.82 |
| 4 | procedural | 50.3 | 43.38 | 43.71 | 43.18 | 39.03 | 39.40 | 39.40 | 38.61 | 25.69 | 27.74 | 27.74 | 32.93 |
| Comparison | 30 s | 2 min | 8 min | 16 min |
|---|---|---|---|---|
| Ours Frozen Source (TV ) | +6.5 [2.3, 10.7] 11/12 | +8.4 [4.1, 15.2] 12/12 | +5.8 [1.8, 8.3] 12/12 | +5.7 [1.7, 7.0] 12/12 |
| Ours Frozen Source (TV ) | +6.1 [3.5, 7.5] 12/12 | +7.6 [4.0, 9.2] 12/12 | +7.0 [3.3, 8.4] 12/12 | +6.8 [3.3, 8.3] 12/12 |
| Ours Frozen Source (better prior) | +5.2 [2.3, 7.5] 11/12 | +6.6 [3.1, 9.2] 12/12 | +5.8 [1.5, 7.8] 12/12 | +5.7 [1.6, 7.0] 12/12 |
| Ours Detached Operator | +5.5 [1.4, 9.7] 12/12 | +7.1 [2.3, 14.3] 12/12 | +6.1 [1.7, 8.4] 12/12 | +6.1 [1.5, 7.6] 12/12 |
| Ours Ours without | 0.1 [ 0.3, 0.1] 4/12 | 0.1 [ 0.2, 0.1] 5/12 | 0.3 [ 0.4, 0.1] 5/12 | 0.3 [ 0.4, 0.1] 5/12 |
| Ours Ours without anchor | 0.5 [ 0.6, 0.3] 2/12 | +0.8 [0.4, 1.2] 10/12 | +0.9 [0.5, 1.3] 11/12 | +0.9 [0.5, 1.3] 11/12 |
| Controlled set | 30 s | 2 min | 8 min | 16 min |
| Ours (scope operator) | 39.3 | 43.5 | 43.7 | 43.7 |
| without | 39.4 | 43.5 | 44.1 | 44.1 |
| without anchor ( ) | 39.7 | 43.3 | 42.9 | 42.9 |
| Detached Operator (no ) | 34.1 | 35.5 | 38.4 | 38.9 |
| Frozen Source (no operator) | 34.7 | 36.9 | 38.0 | 39.1 |
| Monte Carlo, photons / evaluation | time | error | cosine |
|---|---|---|---|
| (3 seeds) | 1.9 s | 3.60–3.61 | 0.264–0.271 |
| (3 seeds) | 5.6 s | 2.08–2.10 | 0.421–0.423 |
| (3 seeds) | 18 s | 1.15 | 0.650–0.659 |
| (3 seeds) | 55 s | 0.662–0.667 | 0.828–0.832 |
| (3 seeds) | 3.0 min | 0.371–0.381 | 0.934–0.937 |
| (2 seeds) | 9.1 min | 0.228–0.230 | 0.974–0.975 |
Inverse Rendering
Inverse rendering estimates scene geometry, materials, lighting, and other causes of images from observed visual or sensor data.
See all 76 papers