HS-3D-NeRF: 3D Surface and Hyperspectral Reconstruction From Stationary Hyperspectral Images Using Multi-Channel NeRFs

Kibon Ku; Talukder Z. Jubery; Adarsh Krishnamurthy; Baskar Ganapathysubramanian

cs.CVFeb 18, 2026

HS-3D-NeRF: 3D Surface and Hyperspectral Reconstruction From Stationary Hyperspectral Images Using Multi-Channel NeRFs

Authors: Kibon Ku, Talukder Z. Jubery, Adarsh Krishnamurthy, Baskar Ganapathysubramanian

Organizations: Iowa State University, Ames, Iowa, USA

Abstract

Advances in hyperspectral imaging (HSI) and 3D reconstruction have enabled accurate, high-throughput characterization of agricultural produce quality and plant phenotypes, both essential for advancing agricultural sustainability and breeding programs. HSI captures detailed biochemical features of produce, while 3D geometric data substantially improves morphological analysis. However, integrating these two modalities at scale remains challenging, as conventional approaches involve complex hardware setups incompatible with automated phenotyping systems. Recent advances in neural radiance fields (NeRF) offer computationally efficient 3D reconstruction but typically require moving-camera setups, limiting throughput and reproducibility in standard indoor agricultural environments. To address these challenges, we introduce HSI-SC-NeRF, a stationary-camera multi-channel NeRF framework for high-throughput hyperspectral 3D reconstruction targeting postharvest inspection of agricultural produce. Multi-view hyperspectral data is captured using a stationary camera while the object rotates within a custom-built Teflon imaging chamber providing diffuse, uniform illumination. Object poses are estimated via ArUco calibration markers and transformed to the camera frame of reference through simulated pose transformations, enabling standard NeRF training on stationary-camera data. A multi-channel NeRF formulation optimizes reconstruction across all hyperspectral bands jointly using a composite spectral loss, supported by a two-stage training protocol that decouples geometric initialization from radiometric refinement. Experiments on three agricultural produce samples demonstrate high spatial reconstruction accuracy and strong spectral fidelity across the visible and near-infrared spectrum, confirming the suitability of HSI-SC-NeRF for integration into automated agricultural workflows.

Figures & tables

Figure 1. Schematic of the stationary camera imaging system for hyperspectral NeRF-based point cloud reconstruction in high-throughput plant phenotyping. In this setup, each plant is conveyed to a rotating turntable marked against a Teflon (PTFE) background. Over a full rotation, a tripod-mounted stationary camera captures high-resolution images that serve as input for HS-3D-NeRF techniques to generate 3D reconstructions. This streamlined approach eliminates the need for complex moving-camera rigs, aligning with the objectives of efficient, scalable agricultural imaging. The right shows the different PCD reconstructions using HS-3D-NeRF and spectral signatures on the circled regions. Teaser Overview of HS-3D-NeRF

Figure 2. Motivation for hyperspectral imaging (HSI) in 3D. Two views of the same apple show viewpoint-dependent bruise visibility: one ROI is visible in view (A) but occluded in view (B), motivating HSI with 3D reconstruction for more complete surface assessment.

Panel of Figure 3 — Figure 3. Workflow of the HSI-SC-NeRF pipeline. The process consists of three main steps: (A) Dataset Acquisition, where the experimental environment is set up and multi-view hyperspectral image data is collected using a stationary camera; (B) Data Preprocessing, involving white-reference spectral calibration and COLMAP-based pose estimation to ensure radiometric and geometric consistency; and (C) NeRF-Based HS PCD, where a multi-channel NeRF model is trained for scene representation, followed by hyperspectral point cloud reconstruction and refinement to generate high-quality 3D hyperspectral point clouds. The graphical abstract (bottom) illustrates the end-to-end pipeline: the imaging chamber with rotating turntable (A), an RGB timelapse frame from the acquisition session (B), the resulting hyperspectral point cloud at 480 $\mathrm{n}\mathrm{m}$ (C), and per-region reflectance spectra extracted from the reconstructed point cloud (D).

Figure 4. Teflon studio chamber for uniform hyperspectral illumination. (A) The chamber is designed from PTFE (Teflon) sheets to provide diffuse illumination. (B) Inside the chamber and (C) the outside.

Figure 5. White reference (WR)–based spectral calibration. (A) Coarse WR region of interest (ROI) extraction. (B) Automatically generated central WR mask after 70th-percentile-based spatial deviation filtering. (C) Pseudo-RGB rendering of the raw hyperspectral cube before calibration. (D) Final spectrally calibrated result after WR-based normalization and reflectance clipping.

D(x,y)=\frac{1}{L}\sum_{\lambda}\left|\frac{I_{\mathrm{WR}}(x,y,\lambda)-\mu_{\mathrm{WR}}(\lambda)}{\mu_{\mathrm{WR}}(\lambda)}\right|,

Algorithm 1 Spectral calibration using a white reference (WR) tarp with automatic spatial mask generation

Figure 6. Camera poses and viewing directions estimated by COLMAP for 64 hyperspectral images. Black squares show camera positions, and red arrows indicate viewing directions toward the object center. (A) Pear, (B) Maize, and (C) Apple.

Figure 7. HSI-SC-NeRF two-stage training protocol. The model is first pre-trained on full-frame HSI, then fine-tuned with foreground masks and the camera optimizer disabled using the selected loss-weight setting

(\lambda_{\mathrm{ang}},\lambda_{\mathrm{hsi}})

. Iteration counts are shown in the diagram.

Figure 8. Spatial validation of maize ear reconstruction. (A) Precision–recall curves as functions of the distance threshold $\epsilon$ , with the optimal threshold ( $\epsilon=0.002$ m) indicated by the black dashed line, yielding an F-score of 97.31. Point-to-surface distances were computed after ICP registration in CloudCompare, resulting in a final RMS error of $0.001$ m over approximately 50,000 points. (B) Reference PCD obtained using a rotating RGB camera (iPhone 13 Pro, $2532\times 1170$ ). (C) Predicted PCD generated from hyperspectral imagery captured inside the Teflon imaging chamber using a stationary SPECIM camera ( $512\times 512$ ).

Figure 9. Evaluation images. Six representative frames (10% of the 64 total frames) are shown. In each maize ear, the left image shows the ground-truth reference and the right image shows the prediction.

HSI Eval Metrics	Maize	Pear	Apple
SAM [rad] $\downarrow$	$0.02921\pm 0.00457$	$0.04577\pm 0.01281$	$0.06420\pm 0.03807$
RMSE $\downarrow$	$0.03051\pm 0.01141$	$0.03242\pm 0.00754$	$0.04383\pm 0.02415$
SSIM $\uparrow$	$0.88823\pm 0.05618$	$0.83259\pm 0.03064$	$0.74229\pm 0.11584$
PSNR [dB] $\uparrow$	$29.13709\pm 2.55173$	$27.47112\pm 3.20361$	$24.49213\pm 5.85267$
Number of evaluation rays	$262{,}144$

Table 1. HSI evaluation metrics with the loss-weight setting

(\lambda_{\mathrm{ang}},\lambda_{\mathrm{hsi}})=(0.25,0.75)

on the 204-band hyperspectral datasets. SAM, RMSE, PSNR, and SSIM are computed directly over the full 204-band hyperspectral renderings. Values are reported as mean

\pm

SD over held-out views.

(Angular, HSI)	SAM [0–1] $\downarrow$		RMSE [0–1] $\downarrow$		SSIM [0–1] $\uparrow$		PSNR [dB] $\uparrow$
	Pre-train	Fine-tune	Pre-train	Fine-tune	Pre-train	Fine-tune	Pre-train	Fine-tune
(0,1) HSI-only	$0.0622\pm 0.0113$	$0.0422\pm 0.0192$	$0.0731\pm 0.0134$	$0.0360\pm 0.0141$	$0.3905\pm 0.1377$	$0.8564\pm 0.0718$	$22.11\pm 1.63$	$27.27\pm 4.43$
(0.25,0.75)	$0.0588\pm 0.0091$	$\mathbf{0.0292\pm 0.0046}$	$0.0715\pm 0.0098$	$\mathbf{0.0305\pm 0.0114}$	$0.4356\pm 0.1161$	$\mathbf{0.8882\pm 0.0562}$	$22.17\pm 1.34$	$\mathbf{29.14\pm 2.55}$
(0.5,0.5)	$0.0808\pm 0.0130$	$0.0523\pm 0.0204$	$0.0922\pm 0.0085$	$0.0468\pm 0.0162$	$0.1758\pm 0.0972$	$0.7896\pm 0.0978$	$19.81\pm 1.11$	$24.45\pm 3.59$
(0.75,0.25)	$0.0498\pm 0.0087$	$0.0428\pm 0.0169$	$0.0616\pm 0.0141$	$0.0421\pm 0.0138$	$0.5212\pm 0.1676$	$0.8534\pm 0.0802$	$23.78\pm 1.91$	$25.89\pm 3.94$
(1,0) Angular-only	$0.0707\pm 0.0249$	$0.0381\pm 0.0162$	$0.0893\pm 0.0120$	$0.0732\pm 0.0096$	$0.2811\pm 0.1276$	$0.5984\pm 0.0342$	$19.97\pm 1.51$	$21.65\pm 1.69$

Table 2. Pre-trained vs. fine-tuned results on the maize dataset across loss weights

(\lambda_{\mathrm{ang}},\lambda_{\mathrm{hsi}})

, reported as mean

\pm

SD over held-out views. Metrics are grouped into spectral fidelity (SAM, RMSE; lower is better) and spatial/image-quality proxies (SSIM, PSNR; higher is better). Best fine-tuned values are highlighted in bold.

Figure 10. Apple visualization and spectra. (A) Pseudo-RGB (598, 548, 449 nm). (B) Bruise-contrast composite (801, 708, 551 nm). (C) Mean reflectance spectra from the marked healthy and bruised regions.

Figure 11. Maize reconstructions under different band triplets. (A) Visible composite [650, 540, 470] $\mathrm{n}\mathrm{m}$ . (B) NIR-inclusive maize triplet [850, 580, 490] $\mathrm{n}\mathrm{m}$ . (C) Spectral reflectance curves extracted from the highlighted surface regions in (A) and (B).

Figure 12. Pear visualization and spectra. (A) Pseudo-RGB composite. (B) Disease-oriented composite (980, 710, 550 nm). (C) Mean reflectance spectra from the marked healthy and diseased regions.

Crop	Band composite [nm]	Area (m 2 )	Volume (L)
Apple	Pseudo-RGB: [598, 548, 449]	0.176	0.225
	Bruise-contrast: [801, 708, 551]	0.118	0.245
Maize	Visible: [650, 540, 470]	0.147920	0.395930
	NIR-inclusive: [850, 580, 490]	0.147920	0.395930
Pear	Pseudo-RGB	0.0881612	0.125526
	Disease-oriented: [550, 710, 980]	0.0881611	0.125525

Table 3. Surface area and volume estimates for Poisson meshes (octree depth 8) reconstructed from different 3-channel composites (Figs. 10 – 12 A–B). Area is reported in m 2 and volume in liters (L).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 13. Automatic central WR mask generated using the 70th-percentile deviation threshold. (A) First-pass spatial relative deviation map computed over the WR region of interest. (B) Refined deviation map and resulting automatic central WR mask after percentile thresholding and morphological filtering. (C) Example acquisition showing the WR tarp and the object positioned at the center of the scene.

Figure 14. Justification of the 70th-percentile threshold for automatic WR mask generation. In each pair, the top plot shows the spatial relative deviation map computed within the WR region of interest, and the bottom image shows the corresponding binary WR mask projected onto the original image. (A), (B), and (C) correspond to the 65th-, 70th-, and 75th-percentile thresholds, respectively.

Explore similar work

Mar 5, 2026

Towards 3D Scene Understanding of Gas Plumes in LWIR Hyperspectral Images Using Neural Radiance Fields

Hyperspectral images (HSI) have many applications, ranging from environmental monitoring to national security, and can be used for material detection and identification. Longwave infrared (LWIR) HSI can be used for gas plume detection and analysis. Oftentimes, only a few images of a scene of interest are available and are analyzed individually. The ability to combine information from multiple images into a single, cohesive representation could enhance analysis by providing more context on the scene's geometry and spectral properties. Neural radiance fields (NeRFs) create a latent neural representation of volumetric scene properties that enable novel-view rendering and geometry reconstruction, offering a promising avenue for hyperspectral 3D scene reconstruction. We explore the possibility of using NeRFs to create 3D scene reconstructions from LWIR HSI and demonstrate that the model can be used for the basic downstream analysis task of gas plume detection. The physics-based DIRSIG software suite was used to generate a synthetic multi-view LWIR HSI dataset of a simple facility with a strong sulfur hexafluoride gas plume. Our method, built on the standard Mip-NeRF architecture, combines state-of-the-art methods for hyperspectral NeRFs and sparse-view NeRFs, along with a novel adaptive weighted MSE loss. Our final NeRF method requires around 50% fewer training images than the standard Mip-NeRF and achieves an average PSNR of 39.8 dB with as few as 30 training images. Gas plume detection applied to NeRF-rendered test images using the adaptive coherence estimator achieves an average AUC of 0.821 when compared with detection masks generated from ground-truth test images.

Scout Jarman, Zigfried Hampel-Arias, Adra Carr +1

Utah State University, Department of Mathematics & Statistics, Logan, UT USA · Los Alamos National Laboratory, Intelligence and Space Research Division, Los Alamos, NM USA

3D Scene Understanding 3D Reconstruction Neural Radiance Fields Hyperspectral Imaging Sparse-View 3D Reconstruction

Ask PDF

May 25, 2026

Broadband Hyperspectral 3D Imaging using Dispersed Structured Light

Hyperspectral 3D imaging enables the capture of dense spectral information and scene geometry but has traditionally been confined to narrow spectral windows, typically the visible range. In this work, we introduce a broadband hyperspectral 3D imaging (BH3D) method to extend this capability across the full visible-near-infrared and short-wavelength infrared (SWIR) spectrum (450-1500 nm). This broad coverage is critical as it captures complementary physical cues: visible wavelengths reveal surface appearance, while SWIR bands provide insight into subsurface properties and material composition. However, realizing BH3D is challenging due to fundamental sensor constraints between visible-spectrum silicon and SWIR-spectrum InGaAs sensors, which necessitate complex multi-spectrograph designs. Here we propose a single-spectrograph BH3D system, using a stereo setup comprising visible and SWIR cameras, that reconstructs dense broadband hyperspectral reflectance together with accurate 3D geometry. Our key idea is to extend dispersed structured light to the broadband regime using a single spectrograph. We model the image formation of broadband dispersed structured light, and estimate hyperspectral reflectance and depth. We validate our approach on diverse real-world scenes, demonstrating accurate reconstruction with a mean spectral angle mapper of 0.13 rad, root mean square error of 0.03, and mean depth error of 4.5 mm. We further demonstrate identifying metameric materials, performing imaging through opaque layers, uncovering hidden features on banknotes, and revealing blood vessels.

Suhyun Shin, Yunseong Moon, Ryota Maeda +3

POSTECH, South Korea · University of Hyogo, Japan · University of Toronto, Canada

3D Reconstruction Hyperspectral Imaging

Ask PDF

Sep 4, 2026

Learning Spatial-Spectral Refinement and Calibrating Complementary Observations for Hyperspectral Image Super-Resolution

Hyperspectral and multispectral image fusion (HMIF) aims to reconstruct a high-resolution hyperspectral image (HR-HSI) by combining the fine spatial details of a high-resolution multispectral image (HR-MSI) with the rich spectral information of a low-resolution hyperspectral image (LR-HSI). Recent advances in implicit neural representations (INRs) have enabled flexible coordinate-based modeling for HMIF; however, existing INR-based approaches may not fully capture fine-grained spatial structures and rich spectral dependencies. Moreover, the LR-HSI and HR-MSI are primarily incorporated through degradation-consistency constraints, leaving their complementary information underexploited. To address these limitations, we propose Two-Stage Reconstruction with Implicit Tensor Neural Representation (TSR-ITNR), a unified self-supervised framework integrating representation refinement and observation-guided calibration. In Stage 1, TSR-ITNR learns an implicit Tucker representation and refines its low-rank spatial coefficient tensor and spectral basis to better capture fine spatial structures and interband correlations. A fixed pretrained denoiser further provides a deep prior for the preliminary reconstruction. In Stage 2, parameter-free calibration derives complementary and noninterfering corrections from both observations to recover information insufficiently captured in Stage 1. Theoretical analysis establishes the geometry-preserving property of spectral refinement and the orthogonal complementarity of calibration. Extensive experiments on multiple benchmark datasets demonstrate strong quantitative, visual, and spectral reconstruction performance without ground-truth HR-HSI supervision. Beyond conventional reconstruction metrics, we further assess the effectiveness of TSR-ITNR using downstream semantic segmentation accuracy.

Liqian Yang, Xingchi Chen, Xinfeng Gui +2

School of Management, Zhengzhou University, Zhengzhou, 450001, China · Center for Intelligent Decision-making and Machine Learning, Xi’an Jiaotong University, Xi’an, 710049, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, China

Hyperspectral Image Super-Resolution Image Reconstruction Multi-Source Image Fusion Implicit Neural Representations

Ask PDF