Accelerating 4D Hyperspectral Imaging through Physics-Informed Neural Representation and Adaptive Sampling
Authors: Chi-Jui Ho, Harsh Bhakta, Wei Xiong, Nicholas Antipa
Organizations: Dept. of Electrical and Computer Engineering, UC San Diego, La Jolla, CA 92093, USA · Dept. of Chemistry, UC San Diego, La Jolla, CA 92093, USA · Materials Science and Engineering Program, UC San Diego, La Jolla, CA, 92093, USA
High-dimensional hyperspectral imaging (HSI) enables the visualization of ultrafast molecular dynamics and complex, heterogeneous spectra. However, applying this capability to resolve spatially varying vibrational couplings in two-dimensional infrared (2DIR) spectroscopy, a type of coherent multidimensional spectroscopy (CMDS), necessitates prohibitively long data acquisition, driven by dense Nyquist sampling requirements and the need for extensive signal accumulation. To address this challenge, we introduce a physics-informed neural representation approach that efficiently reconstructs dense spatially-resolved 2DIR hyperspectral images from sparse experimental measurements. In particular, we used a multilayer perceptron (MLP) to model the relationship between the sub-sampled 4D coordinates and their corresponding spectral intensities, and recover densely sampled 4D spectra from limited observations. The reconstruction results demonstrate that our method, using a fraction of the samples, faithfully recovers both oscillatory and non-oscillatory spectral dynamics in experimental measurement. Moreover, we develop a loss-aware adaptive sampling method to progressively introduce potentially informative samples for iterative data collection while conducting experiments. Experimental results show that the proposed approach achieves high-fidelity spectral recovery using only 1/32 of the sampling budget, as opposed to exhaustive sampling, effectively reducing total experiment time by up to 32-fold. This framework offers a scalable solution for accelerating any experiments with hypercube data, including multidimensional spectroscopy and hyperspectral imaging, paving the way for rapid chemical imaging of transient biological and material systems.
Figures & tables
Figure 1: Hyper-spectrum reconstruction through neural rendering. We sparsely sample the hyper-spectrum collected from the interaction between ultrafast pulses and samples, and then use the proposed neural rendering framework to reconstruct the dense hyper-spectrum data in both oscillatory and non-oscillatory regimes.
Figure 2: Dimensionality of Spatially-Resolved 2DIR. a) The system diagram. b) Illustration of the four coordinate axes defining the hyperspectral volume. Each 2DIR frame parameterized by (ω1,ω3) , while the temporal and spatial variations are recorded in different waiting time t2 and spatial position x .
Figure 3: Our MLP-based neural representation model. We construct neural representations for signals subsampled along either the population (slow varying) or coherence (fast varying) time dimensions. Population: A coordinate-based MLP is trained to learn the temporal dynamics along population time from sparsely sampled data and to generalize this knowledge to unsampled points during inference. Coherence: The MLP is optimized in the frequency domain, after which an IRFT is applied to recover estimated raw signals for loss computation, excluding intensities at unsampled coordinates. During inference, the trained network reconstructs rapidly varying 4D signals across the full coordinate domain.
Figure 4: 2D representation of 4DIR data. A bounding box is defined around the dominant peak in each (ω1,ω3) slice. The intensities enclosed within this region are averaged to reduce the four-dimensional data to a two-dimensional representation along the (x,t2) axes, which is subsequently used for evaluation.
Figure 5: Our loss-driven adaptive sampling method. Beginning with the initially sampled coordinates (blue dots), the MLP is trained to model the spectral dynamics and the loss is evaluated along the population time t2 . The interval exhibiting the highest average prediction error is identified, and its midpoint is selected as the next sample.
Figure 6: Summary of results under different numbers of repeated samplings. (a) Measurements averaged over varying data averaging counts ( r ) and reconstructed spectra under the same count. (c) Using the least noisy case (highest r ) as reference, we compute the MSE of other measurements and predictions against it. (d) Mean and (e) standard deviation profiles of reconstructions at different repeat counts, with the reference shown as a solid black line.
r
Intensity MSE
Mean profile MSE
Std profile MSE
2
1.320e-02
1.280e-06
3.820e-07
5
8.330e-03
6.600e-07
1.240e-07
10
8.250e-03
6.050e-07
9.990e-08
20
4.990e-03
4.360e-07
4.860e-08
Table 1: Quantitative evaluation under different data averaging counts ( r ).
Figure 7: Summary of results under different sampling intervals (SI) along the population time. (a) Reference and reconstructed spectra obtained at various sampling rates along the population time axis. (b–d) Peak intensity, mean and standard deviation profiles of reconstructions at different sampling intervals, with the reference shown as a solid black line.
SI
Intensity MSE
Mean profile MSE
Std profile MSE
1
4.990e-03
4.360e-07
4.860e-08
2
5.590e-03
5.120e-07
6.390e-08
4
1.390e-02
1.290e-06
1.480e-07
8
1.680e-02
2.860e-06
1.570e-07
16
1.570e-01
3.490e-06
3.000e-07
Table 2: Quantitative evaluation under different sampling intervals (SI) along the population time.
Figure 8: Reconstruction results from subsampled t1 domain data. (a) Reconstructed 2DIR spectrum compared against the ground truth. (b) Temporal raw measurements showing the fully sampled ground truth and the subsampled input. (c) A 1D intensity profile extracted from (b) demonstrating signal recovery. To visualize oscillations beyond the native samples, the ground truth is densified via sinc interpolation, while the dense prediction is obtained by evaluating the IRFT with a finer t1 sampling.
Figure 9: Reconstruction results from subsampled t1 domain data using different algorithms. (a) The 2DIR spectrum and (b) temporal measurements generated from different reconstruction algorithms compared against the ground truth.
Algorithm
Spectrum MSE
Unsampled Temporal MSE
RFFT
1.333e-02
1.583e-04
GIRAF-2D [ 10 ]
8.607e-02
3.443e-03
Fourier-features [ 45 ]
1.615e-01
6.190e-03
Table 3: Reconstruction performance on undersampled t1 data across different algorithms.
Figure 10: Reconstruction performance at different t1 sampling rates. We display the reconstructed 2DIR spectrum under different sampling rates and associated temporal data. SR: sampling rate.
Figure 11: Reconstruction 2DIR Spectrum under low accumulation and sampling rates. With r=5 and undersampled t1,t2 measurement, we display an unsampled t2 slice of 2DIR spectrum and a slice of (t2,ω3) projection.
t1 Sampling Rate
Spectrum MSE
Unsampled Temporal MSE
80%
1.333e-02
1.583e-04
60%
1.663e-02
1.824e-04
40%
3.369e-02
3.825e-04
20%
2.503e-01
2.429e-03
Table 4: RFFT reconstruction error under different t1 sampling rates.
Figure 12: Comparison of 2DIR Spectrum Reconstruction Under Differing Sampling Strategies. (a)-(c) Fully sampled temporal data and representative sampling masks for the two evaluated strategies. (d) Temporal reconstruction statistics across 10 independent trials. (e)-(g) Corresponding reconstructed 2DIR spectra for the examples shown in (a)-(c). (h) Summary of spectral reconstruction performance. (i)-(j) 1D illustrations of the reconstruction performance under the (i) Keep and (j) Drop strategies. Omitting the early t1 samples leaves the reconstruction underconstrained, leading the model to hallucinate spurious initial oscillations.
Figure 13: Reconstruction results from different sampling strategies. We overlay (a) peak-intensity, (b) mean, and (c) standard-deviation profiles derived from spectra reconstructed using different sampling strategies.
Method
Intensity MSE
Mean profile MSE
Std profile MSE
Ours
6.190e-02
2.710e-06
2.320e-07
Random
7.010e-02
4.950e-06
4.870e-07
Uniform
3.430e-01
4.870e-06
3.590e-07
Table 5: Quantitative comparison of different adaptive sampling strategies.
Figure 14: Effect of statistical moment-matching. Spectral recovery results obtained with (orange) and without (blue) moment-matching constraints for (a) the mean and (b) the standard deviation profiles.
Single-pixel imaging (SPI) offers a cost-effective route to hyperspectral acquisition but struggles to recover high-fidelity spatial and spectral details under extremely low sampling rates, a severely ill-posed inverse problem. While deep learning has shown potential, existing data-driven methods demand large-scale pretraining datasets that are often impractical in hyperspectral imaging. To overcome this limitation, we propose an end-to-end physics-informed framework that leverages untrained neural networks and RGB guidance for joint hyperspectral reconstruction and super-resolution without any external training data. The framework comprises three physically grounded stages: (1) a Regularized Least-Squares method with RGB-derived Grayscale Priors (LS-RGP) that initializes the solution by exploiting cross-modal structural correlations; (2) an Untrained Hyperspectral Recovery Network (UHRNet) that refines the reconstruction through measurement consistency and hybrid regularization; and (3) a Transformer-based Untrained Super-Resolution Network (USRNet) that upsamples the spatial resolution via cross-modal attention, transferring high-frequency details from the RGB guide. Extensive experiments on benchmark datasets demonstrate that our approach significantly surpasses state-of-the-art algorithms in both reconstruction accuracy and spectral fidelity. Moreover, a proof-of-concept experiment using a physical single-pixel imaging system validates the framework's practical applicability, successfully reconstructing a 144-band hyperspectral data cube at a mere 6.25% sampling rate. The proposed method thus provides a robust, data-efficient solution for computational hyperspectral imaging.
Hao Zhang, Bilige Xu, Lichen Wei +2
College of Science, Northwest Agriculture & Forestry University, Yangling 712100, China · Key Laboratory of Photoelectronic Imaging Technology and System, School of Optics and Photonics, Beijing Institute of Technology, Beijing, China · State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan 430074, China
Recent advances have demonstrated that coded aperture snapshot spectral imaging (CASSI) systems show great potential for capturing 3D hyperspectral images (HSIs) from a single 2D measurement. Despite the inherent spectral continuity of scenes captured by CASSI, most existing reconstruction methods are restricted to fixed, discrete spectral outputs, thereby precluding continuous spectral reconstruction or spectral super-resolution. To address this challenge, we propose Phy-CoSF, which synergizes deep unfolding networks with implicit neural representations, establishing a new paradigm for continuous spectral reconstruction and super-resolution in CASSI. Specifically, we propose a two-phase architecture that bridges discrete-wavelength training with continuous spectral rendering, enabling the synthesis of high-fidelity HSIs at arbitrary target wavelengths. At the core of our framework lies the continuous spectral fields (CoSF) module, embedded within each unfolding stage as a dynamic prior, which comprises a triple-branch cross-domain feature mixer for comprehensive spatial-frequency-channel feature fusion, alongside a spectral synthesis head that generates spectral intensities by querying continuous wavelength coordinates. Extensive experimental results demonstrate that Phy-CoSF not only achieves continuous modeling at arbitrary spectral resolutions but also outperforms many state-of-the-art methods in both reconstruction fidelity and spectral detail preservation. Our code and more results are available at: https://github.com/PaiDii/Phy-CoSF.git.
Wudi Chen, Zhiyuan Zha, Xin Yuan +6
College of Communication Engineering, Jilin University, Changchun 130012, China. · School of Engineering, Westlake University, Hangzhou, Zhejiang 310024, China. · School of Electrical & Electronic Engineering, Nanyang Technological University, Singapore 639798. +4
Hyperspectral image super-resolution is essential for enhancing the spatial fidelity of HSI data, yet existing deep learning methods often struggle with substantial spectral redundancy and the limited non-linear modeling capacity of standard feed-forward networks (FFNs). To address these challenges, we propose Spectral Dynamic Attention Network (SDANet), a framework designed to adaptively suppress redundant spectral interactions. SDANet integrates two key components: 1) Dynamic Channel Sparse Attention (DCSA) module that computes channel-wise correlations and selectively preserves the most informative attention responses through dynamic and data-dependent sparsification. 2) Frequency-Enhanced Feed-Forward Network (FE-FFN) that jointly models spatial and frequency-domain representations to enhance non-linear expressiveness. Extensive experiments on two benchmark datasets demonstrate that SDANet achieves state-of-the-art HISR performance while maintaining competitive efficiency. The code will be made publicly available at https://github.com/oucailab/SDANet.
Tengya Zhang, Feng Gao, Lin Qi +2
Sanya Oceanographic Institution, Ocean University of China, Sanya, China · State Key Laboratory of Physical Oceanography, Ocean University of China, Qingdao 266100, China · Department of Electrical and Computer Engineering, Mississippi State University, Starkville, MS 39762 USA