Towards 3D Scene Understanding of Gas Plumes in LWIR Hyperspectral Images Using Neural Radiance Fields
Authors: Scout Jarman, Zigfried Hampel-Arias, Adra Carr, Kevin R. Moon
Organizations: Utah State University, Department of Mathematics & Statistics, Logan, UT USA · Los Alamos National Laboratory, Intelligence and Space Research Division, Los Alamos, NM USA
Hyperspectral images (HSI) have many applications, ranging from environmental monitoring to national security, and can be used for material detection and identification. Longwave infrared (LWIR) HSI can be used for gas plume detection and analysis. Oftentimes, only a few images of a scene of interest are available and are analyzed individually. The ability to combine information from multiple images into a single, cohesive representation could enhance analysis by providing more context on the scene's geometry and spectral properties. Neural radiance fields (NeRFs) create a latent neural representation of volumetric scene properties that enable novel-view rendering and geometry reconstruction, offering a promising avenue for hyperspectral 3D scene reconstruction. We explore the possibility of using NeRFs to create 3D scene reconstructions from LWIR HSI and demonstrate that the model can be used for the basic downstream analysis task of gas plume detection. The physics-based DIRSIG software suite was used to generate a synthetic multi-view LWIR HSI dataset of a simple facility with a strong sulfur hexafluoride gas plume. Our method, built on the standard Mip-NeRF architecture, combines state-of-the-art methods for hyperspectral NeRFs and sparse-view NeRFs, along with a novel adaptive weighted MSE loss. Our final NeRF method requires around 50% fewer training images than the standard Mip-NeRF and achieves an average PSNR of 39.8 dB with as few as 30 training images. Gas plume detection applied to NeRF-rendered test images using the adaptive coherence estimator achieves an average AUC of 0.821 when compared with detection masks generated from ground-truth test images.
Figures & tables
Figure 1: Example of the gas plume detection process. (Left): False-coloringing HSI using wavelengths 10.4, 8.1, and 8.5 μ m for the red, green, and blue channels. (Middle): ACE detection score map with background μ and Σ estimated using all pixels, and using the SF 6 absorption spectrum for t . (Right): Plume mask created by thresholding the ACE scores at 0.6 .
Figure 2: Example of the adaptive weighted L2 loss weight calculation. 1. provides example false-colored training images with their associated NeRF renderings. 2. shows the squared channel residuals for each pixel. 3. shows the weighting for each channel after averaging the residuals and scaling to sum to one.
Figure 3: (a) The visible light (RGB) coloring of our simulated scene, showing the stack, facility, road, and building. (b) The absorption spectrum of SF 6 used in the plume simulation. (c) The hemisphere of images captures the scene. Each image points to the base of the stack.
Metric
# Images
Mip-NeRF
Ours
PSNR
20
33.7 ± 0.43
36.7 ± 0.77
30
34.5 ± 0.46
39.6 ± 0.72
40
35.4 ± 1.00
41.3 ± 0.41
50
36.4 ± 0.59
42.8 ± 0.33
75
42.0 ± 0.68
45.3 ± 0.32
100
45.1 ± 0.20
47.1 ± 0.34
Table 1: Average and standard deviation of PSNR and SSIM for Mip-NeRF and our method across different training set sizes. Our method consistently outperforms Mip-NeRF across all sizes. Larger numbers are better for PSNR and SSIM.
Figure 4: False-color renderings compared to ground truth images. The first row shows the ground truth, the second row shows the renderings from our method, and the third row shows the renderings from Mip-NeRF. The first column is for 20 training images, followed by 40, 50, and 100 training images. The seed that produced the highest geometric average of PSNR/55, SSIM, AUC, TPR, and 1-FPR for each image was used for rendering; these represent the best-case scenario for performance. The PSNR and SSIM are printed above each rendered image. See Video 1 (MP4, 4.7 MB) for “drone path,” false-color rendered videos of the eight models used to generate these still images.
Metric
# Images
Mip-NeRF
Ours
AUC
20
0.588 ± 0.048
0.615 ± 0.076
30
0.638 ± 0.060
0.821 ± 0.081
40
0.750 ± 0.047
0.892 ± 0.041
50
0.832 ± 0.025
0.913 ± 0.009
75
0.940 ± 0.023
0.968 ± 0.013
100
0.949 ± 0.026
0.987 ± 0.010
Table 2: Average and standard deviation of AUC, TPR, and FPR rates for Mip-NeRF and our method across different training set sizes. Our method often outperforms Mip-NeRF, particularly for 30 training images. Larger numbers are better for AUC and TPR, and smaller numbers are better for FPR.
Figure 5: Plot of Mip-NeRF and our method’s image reconstruction and detection performance (slight jitter applied to x coordinates). The lines and error bars show the average model performance, with standard deviation, from Tables 1 and 2 . Each of the five experiments is plotted to show the spread in model performance across different training samples. With the detection metrics AUC and TPR, a single experiment can produce high detection performance. This can be seen with our method on 20 training images, and with Mip-NeRF on 40 training images.
Metric
# Img
Mip-NeRF
Ours
PSNR
20
33.38, 34.57
35.24, 37.37
30
33.89, 35.34
38.30, 40.40
40
33.97, 36.74
40.53, 41.72
50
35.47, 37.06
42.26, 43.10
75
41.00, 42.95
44.92, 45.71
100
44.81, 45.38
46.78, 47.69
Table 3: Minimum and maximum model performance for all metrics. Values are organized as “min, max”. These values can be found in Figure 5 . Our model’s worst performance is frequently greater than Mip-NeRF’s best performance. Larger numbers are better for PSNR, SSIM, AUC, and TPR. Smaller numbers are better for FPR.
Figure 6: ACE detection scores for SF 6 applied to NeRF rendered images compared to ACE detection applied to the ground truth images. The rows show the ground truth, our method, and Mip-NeRF detection results. The columns show results for training set sizes of 20, 40, 50, and 100 images. These detections are applied to the same images as in Figure 4 . The AUC, TPR, and FPR rates are printed above each reconstructed image. See Video 2 (MP4, 10.7 MB) for “drone path,” ACE rendered videos of the eight models used to generate these still images.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Image reconstruction and detection performance comparison when using just an L2 loss, and when adding SAM and AWL2. The green line shows default Mip-NeRF performance, while the orange line shows the change in performance when including SAM and AWL2. Additionally, the pink line shows performance with MD and GR, while the purple line shows performance with MD and GR when adding SAM and AWL2. The x’s indicate the seed with the best model performance, and the o indicates the worst.
Figure 8: Performance comparison of various modeling changes compared to the default Mip-NeRF. All of these models use our proposed SAM and AWL2 losses. The red line shows the Mip-NeRF performance. The blue line shows performance when using multi-channel density. The green line shows the Mip-NeRF when adding geometry regularization and sample space annealing. The purple line shows the performance when combining MD and GR, and is the model we present as “ours” in Section 4 . The x’s show the best-performing models, and the o’s show the worst-performing models.
Run Time
GPU Memory
Method
Time (s)
% Inc.
Mem. (Gb)
% Inc.
Mip-NeRF
3,190
13.0
Mip + Losses
3,439
7.8%
13.0
0.0%
MD
4,339
36.0%
20.8
60.0%
MD + GR
5,525
73.2%
22.4
72.3%
Appendix
Table 4: Timing and GPU memory usage when training each method for 30,000 iterations, with 50 training images and 31 evaluation images.