Towards 3D Scene Understanding of Gas Plumes in LWIR Hyperspectral Images Using Neural Radiance Fields
Authors: Scout Jarman, Zigfried Hampel-Arias, Adra Carr, Kevin R. Moon
Organizations: Utah State University, Department of Mathematics & Statistics, Logan, UT USA · Los Alamos National Laboratory, Intelligence and Space Research Division, Los Alamos, NM USA
Hyperspectral images (HSI) have many applications, ranging from environmental monitoring to national security, and can be used for material detection and identification. Longwave infrared (LWIR) HSI can be used for gas plume detection and analysis. Oftentimes, only a few images of a scene of interest are available and are analyzed individually. The ability to combine information from multiple images into a single, cohesive representation could enhance analysis by providing more context on the scene's geometry and spectral properties. Neural radiance fields (NeRFs) create a latent neural representation of volumetric scene properties that enable novel-view rendering and geometry reconstruction, offering a promising avenue for hyperspectral 3D scene reconstruction. We explore the possibility of using NeRFs to create 3D scene reconstructions from LWIR HSI and demonstrate that the model can be used for the basic downstream analysis task of gas plume detection. The physics-based DIRSIG software suite was used to generate a synthetic multi-view LWIR HSI dataset of a simple facility with a strong sulfur hexafluoride gas plume. Our method, built on the standard Mip-NeRF architecture, combines state-of-the-art methods for hyperspectral NeRFs and sparse-view NeRFs, along with a novel adaptive weighted MSE loss. Our final NeRF method requires around 50% fewer training images than the standard Mip-NeRF and achieves an average PSNR of 39.8 dB with as few as 30 training images. Gas plume detection applied to NeRF-rendered test images using the adaptive coherence estimator achieves an average AUC of 0.821 when compared with detection masks generated from ground-truth test images.
Figures & tables
Figure 1: Example of the gas plume detection process. (Left): False-coloringing HSI using wavelengths 10.4, 8.1, and 8.5 μ m for the red, green, and blue channels. (Middle): ACE detection score map with background μ and Σ estimated using all pixels, and using the SF 6 absorption spectrum for t . (Right): Plume mask created by thresholding the ACE scores at 0.6 .
Figure 2: Example of the adaptive weighted L2 loss weight calculation. 1. provides example false-colored training images with their associated NeRF renderings. 2. shows the squared channel residuals for each pixel. 3. shows the weighting for each channel after averaging the residuals and scaling to sum to one.
Figure 3: (a) The visible light (RGB) coloring of our simulated scene, showing the stack, facility, road, and building. (b) The absorption spectrum of SF 6 used in the plume simulation. (c) The hemisphere of images captures the scene. Each image points to the base of the stack.
Metric
# Images
Mip-NeRF
Ours
PSNR
20
33.7 ± 0.43
36.7 ± 0.77
30
34.5 ± 0.46
39.6 ± 0.72
40
35.4 ± 1.00
41.3 ± 0.41
50
36.4 ± 0.59
42.8 ± 0.33
75
42.0 ± 0.68
45.3 ± 0.32
100
45.1 ± 0.20
47.1 ± 0.34
Table 1: Average and standard deviation of PSNR and SSIM for Mip-NeRF and our method across different training set sizes. Our method consistently outperforms Mip-NeRF across all sizes. Larger numbers are better for PSNR and SSIM.
Figure 4: False-color renderings compared to ground truth images. The first row shows the ground truth, the second row shows the renderings from our method, and the third row shows the renderings from Mip-NeRF. The first column is for 20 training images, followed by 40, 50, and 100 training images. The seed that produced the highest geometric average of PSNR/55, SSIM, AUC, TPR, and 1-FPR for each image was used for rendering; these represent the best-case scenario for performance. The PSNR and SSIM are printed above each rendered image. See Video 1 (MP4, 4.7 MB) for “drone path,” false-color rendered videos of the eight models used to generate these still images.
Metric
# Images
Mip-NeRF
Ours
AUC
20
0.588 ± 0.048
0.615 ± 0.076
30
0.638 ± 0.060
0.821 ± 0.081
40
0.750 ± 0.047
0.892 ± 0.041
50
0.832 ± 0.025
0.913 ± 0.009
75
0.940 ± 0.023
0.968 ± 0.013
100
0.949 ± 0.026
0.987 ± 0.010
Table 2: Average and standard deviation of AUC, TPR, and FPR rates for Mip-NeRF and our method across different training set sizes. Our method often outperforms Mip-NeRF, particularly for 30 training images. Larger numbers are better for AUC and TPR, and smaller numbers are better for FPR.
Figure 5: Plot of Mip-NeRF and our method’s image reconstruction and detection performance (slight jitter applied to x coordinates). The lines and error bars show the average model performance, with standard deviation, from Tables 1 and 2 . Each of the five experiments is plotted to show the spread in model performance across different training samples. With the detection metrics AUC and TPR, a single experiment can produce high detection performance. This can be seen with our method on 20 training images, and with Mip-NeRF on 40 training images.
Metric
# Img
Mip-NeRF
Ours
PSNR
20
33.38, 34.57
35.24, 37.37
30
33.89, 35.34
38.30, 40.40
40
33.97, 36.74
40.53, 41.72
50
35.47, 37.06
42.26, 43.10
75
41.00, 42.95
44.92, 45.71
100
44.81, 45.38
46.78, 47.69
Table 3: Minimum and maximum model performance for all metrics. Values are organized as “min, max”. These values can be found in Figure 5 . Our model’s worst performance is frequently greater than Mip-NeRF’s best performance. Larger numbers are better for PSNR, SSIM, AUC, and TPR. Smaller numbers are better for FPR.
Figure 6: ACE detection scores for SF 6 applied to NeRF rendered images compared to ACE detection applied to the ground truth images. The rows show the ground truth, our method, and Mip-NeRF detection results. The columns show results for training set sizes of 20, 40, 50, and 100 images. These detections are applied to the same images as in Figure 4 . The AUC, TPR, and FPR rates are printed above each reconstructed image. See Video 2 (MP4, 10.7 MB) for “drone path,” ACE rendered videos of the eight models used to generate these still images.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Image reconstruction and detection performance comparison when using just an L2 loss, and when adding SAM and AWL2. The green line shows default Mip-NeRF performance, while the orange line shows the change in performance when including SAM and AWL2. Additionally, the pink line shows performance with MD and GR, while the purple line shows performance with MD and GR when adding SAM and AWL2. The x’s indicate the seed with the best model performance, and the o indicates the worst.
Figure 8: Performance comparison of various modeling changes compared to the default Mip-NeRF. All of these models use our proposed SAM and AWL2 losses. The red line shows the Mip-NeRF performance. The blue line shows performance when using multi-channel density. The green line shows the Mip-NeRF when adding geometry regularization and sample space annealing. The purple line shows the performance when combining MD and GR, and is the model we present as “ours” in Section 4 . The x’s show the best-performing models, and the o’s show the worst-performing models.
Run Time
GPU Memory
Method
Time (s)
% Inc.
Mem. (Gb)
% Inc.
Mip-NeRF
3,190
13.0
Mip + Losses
3,439
7.8%
13.0
0.0%
MD
4,339
36.0%
20.8
60.0%
MD + GR
5,525
73.2%
22.4
72.3%
Appendix
Table 4: Timing and GPU memory usage when training each method for 30,000 iterations, with 50 training images and 31 evaluation images.
Advances in hyperspectral imaging (HSI) and 3D reconstruction have enabled accurate, high-throughput characterization of agricultural produce quality and plant phenotypes, both essential for advancing agricultural sustainability and breeding programs. HSI captures detailed biochemical features of produce, while 3D geometric data substantially improves morphological analysis. However, integrating these two modalities at scale remains challenging, as conventional approaches involve complex hardware setups incompatible with automated phenotyping systems. Recent advances in neural radiance fields (NeRF) offer computationally efficient 3D reconstruction but typically require moving-camera setups, limiting throughput and reproducibility in standard indoor agricultural environments. To address these challenges, we introduce HSI-SC-NeRF, a stationary-camera multi-channel NeRF framework for high-throughput hyperspectral 3D reconstruction targeting postharvest inspection of agricultural produce. Multi-view hyperspectral data is captured using a stationary camera while the object rotates within a custom-built Teflon imaging chamber providing diffuse, uniform illumination. Object poses are estimated via ArUco calibration markers and transformed to the camera frame of reference through simulated pose transformations, enabling standard NeRF training on stationary-camera data. A multi-channel NeRF formulation optimizes reconstruction across all hyperspectral bands jointly using a composite spectral loss, supported by a two-stage training protocol that decouples geometric initialization from radiometric refinement. Experiments on three agricultural produce samples demonstrate high spatial reconstruction accuracy and strong spectral fidelity across the visible and near-infrared spectrum, confirming the suitability of HSI-SC-NeRF for integration into automated agricultural workflows.
Kibon Ku, Talukder Z. Jubery, Adarsh Krishnamurthy +1
Temperature-emissivity-texture (TeX) decomposition seeks to recover object heat state, material spectral response, and visible-like geometric texture from long-wave infrared hyperspectral imaging (LWIR HSI). Existing TeX pipelines are mainly scene-specific inverse solvers, and the lack of paired LWIR HSI-TeX supervision has limited learning-based decomposition. To address this gap, we introduce TeX-1500, a large-scale paired LWIR HSI-TeX dataset and benchmark for supervised HSI-to-TeX decomposition. TeX-1500 contains 1,522 calibrated real-scene pairs from DARPA Invisible Headlights (DARPA IH) pushbroom imagery and our FTIR acquisitions, covering five locations, four seasons, diverse acquisition times, heterogeneous wavelength layouts, and two sensor families. Each sample stores a calibrated valid-band radiance cube, calibrated wavelength positions, and aligned temperature, emissivity, and texture supervision constructed through a consistent restoration and TeX-construction protocol. We further provide TeX-UNet, a simple wavelength-aware baseline that maps calibrated HSI bands and wavelength positions to TeX fields. Experiments on the held-out DARPA IH pushbroom scenes and zero-/few-shot transfer to FTIR scenes show that TeX-1500 provides usable paired supervision and a measurable benchmark for data-driven physical-property-centered thermal perception.
Cheng Dai, Jiale Lin, Hongyi Xu +3
School of Science, Westlake University, Hangzhou 310030, China · School of Engineering, Westlake University, Hangzhou 310030, China
Neural radiance field (NeRF) and 3D Gaussian splatting (3DGS) are two mainstream approaches for novel view synthesis. They often show complementary performance, i.e., 3DGS demonstrating faster rendering speed and NeRF demonstrating higher rendering quality. Motivated by this, we propose leveraging NeRF-rendered images for 3DGS. Specifically, we target street scenes and utilize a pre-trained street-specific NeRF method to produce training images for a target 3DGS method. In our 3DGS training, NeRF-rendered images are used to remove transient objects in street-level input views and to generate bird's-eye views as additional views, inheriting the higher-quality rendering of NeRF into 3DGS. We further incorporate a diffusion-based image enhancement to improve the image quality of the additional views. Experimental results on one synthetic and two real datasets demonstrate that our proposed method improves street-scene rendering while preserving the speed of 3DGS and the quality of NeRF.