Universal computational thermal imaging overcoming the ghosting effect
Authors: Hongyi Xu, Du Wang, Chenjun Zhao, Jiashuo Chen, Jiale Lin, Liqin Cao, Yanfei Zhong, Yiyuan She, +1 more
Organizations: Department of Electronic and Information Engineering, School of Engineering and Research Center for Industries of the Future, Westlake University, Hangzhou 310030, China · State Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan University, Wuhan 430079, China · Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China · Department of Physics, School of Science, Westlake University, Hangzhou 310030, China · School of Resource and Environmental Sciences, Wuhan University, Wuhan 430079, China · Institute of Theoretical Science, Westlake University, Hangzhou 310030, China · Institute of Natural Sciences, Westlake Institute for Advanced Study, Hangzhou 310024, China
Thermal imaging is crucial for night vision but fundamentally hampered by the ghosting effect, a loss of detailed texture in cluttered photon streams. While conventional ghosting mitigation has relied on data post-processing, the recent breakthrough in heat-assisted detection and ranging (HADAR) opens a promising frontier for hyperspectral computational thermal imaging that produces night vision with day-like visibility. However, universal anti-ghosting imaging remains elusive, as state-of-the-art HADAR applies only to limited scenes with uniform materials, whereas material non-uniformity is ubiquitous in the real world. Here, we propose a universal computational thermal imaging framework, TAG (thermal anti-ghosting), to address material non-uniformity and overcome ghosting for high-fidelity night vision. TAG takes hyperspectral photon streams for nonparametric texture recovery, enabling our experimental demonstration of unprecedented expression recovery in thus-far-elusive ghostly human faces -- the archetypal, long-recognized ghosting phenomenon. Strikingly, TAG not only universally outperforms HADAR across various scenes, but also reveals the influence of material non-uniformity, shedding light on HADAR's effectiveness boundary. We extensively test facial texture and expression recovery across day and night, and demonstrate, for the first time, thermal 3D topological alignment and mood detection. This work establishes a universal foundation for high-fidelity computational night vision, with potential applications in autonomous navigation, reconnaissance, healthcare, and wildlife monitoring.
Figures & tables
Figure 1: Universal thermal anti-ghosting (TAG) for high-fidelity night vision with day-like visibility. a, The TAG framework. TAG takes hyperspectral thermal imagery for nonparametric TeX decomposition and produces high-fidelity textures, along with non-uniform temperature and emissivity. b, The SLOT principle. To break TeX degeneracy, SLOT uses a B-spline basis expansion for continuous emissivity and imposes a smoothness constraint, universally applicable to non-uniform materials. In contrast, HADAR relies on rigid material categorization and suffers from classification errors when material non-uniformity is present. c, Traditional thermal imaging cannot separate ambient scattering from direct emission, yielding textureless silhouettes that resemble ghosts. d, TAG vividly recovers otherwise hidden geometric textures and facial expressions that are crucial for various perception tasks. From left to right: neutral (sad), frown with eyes open (angry), grin with eyes and mouth open (happy), and pout.
Figure 2: The challenge of TeX degeneracy for texture recovery. a, Illustration of TeX degeneracy, where multiple TeX triplets produce an identical thermal radiation spectrum (taken from the red cross in b). b, Texture comparison. Left: poor texture recovery by panchromatic thermal imaging and CLAHE failing to break the TeX degeneracy. Right: vivid texture recovery by TAG, which successfully breaks the TeX degeneracy.
Figure 3: Shedding light on the effectiveness boundary of HADAR. a, Comparable texture recovery between HADAR and TAG when intra-class material non-uniformity is smaller (negligible) than the inter-class emissivity contrast. b, Failure of HADAR TeX vision when intra-class material non-uniformity is larger (significant) than the inter-class emissivity contrast. c, HADAR is robust to emissivity/library inaccuracy when material non-uniformity is negligible. The leftmost panel uses the baseline candidate emissivity estimated by the HADAR approach. The middle and right panels use candidate curves from TAG controlled by the regularization parameter ( λ=10−3,106 ).
Figure 4: Zero-shot cross-modal machine perception enabled by TAG. a, Input data comparison showing ghosting traditional thermal imaging and TAG with vivid textures. b, Task I: AI colorization. With textures, TAG perfectly accommodates color mapping and preserves spatial patterns. Without textures, traditional thermal images produce blurry color artifacts. c, Task II: 3D topological alignment. TAG recovers textures to flawlessly anchor a 468-point facial mesh, whereas alignment completely fails or drifts on traditional thermal images. d, Task III: Semantic perception and recognition. Standard RGB-trained vision engines successfully localize faces and predict nuanced human emotions ( i.e. , Sad, Happy) on TAG images, but suffer catastrophic breakdowns on thermal images.
Figure 5: Passive recovery of facial texture and expression with high-fidelity night vision in pitch darkness. a, Ghosting thermal imaging at night. b, Nighttime facial expression recovery by TAG. Left: neutral expression with mouth and eyes closed. Right: Smiling with mouth and eyes open.
Figure 6: Robustness of TAG in complex open-world field tests. a, Traditional thermal imaging from the DARPA Invisible Headlights dataset, where the ghosting effect blurs critical details. b, TAG consistently recovers great textures of unknown materials across vast scenes without requiring prior material libraries (surpassing HADAR), both during the day and at night.
Metrics
EN ↑
AG ↑
SF ↑
SD ↑
Male
IR
7.349
0.476
1.331
51.491
TAG (Ours)
7.491
1.916
5.933
53.079
Female
IR
6.761
1.329
3.408
50.523
TAG (Ours)
7.412
4.157
8.363
52.004
Table 1: Texture quantification comparing two universal modalities, the traditional thermal imaging (IR) and our TAG, in the human region in Fig. 4 a.
Thermal imaging remains effective under adverse illumination, yet passive long-wave infrared (LWIR) measurements often lack fine texture. Existing thermal texture imaging approaches commonly rely on spectral sensing or registered auxiliary modalities, incurring substantial data throughput or vulnerability to cross-modal degradation. We introduce T2exture, a sparsely perturbed thermal texture imaging framework that aims to reconstruct temporally dense thermal texture sequences from densely sampled passive frames and a few actively perturbed keyframes. We define thermal texture as the residual between a source-on observation and its corresponding source-off passive state. Under sparse LWIR illumination and rapid quasi-steady paired acquisition, this residual attenuates the passive-emission background and approximates a source-induced reflected response, exposing localized material- and geometry-dependent texture. T2exture reconstructs a dense sequence of this source-conditioned response through two stages. Stage 1 estimates the unobserved source-off passive state at each active instant from neighboring passive frames to obtain reliable differential texture anchors. Stage 2 combines sparse anchors with passive structural context near each target time to reconstruct the dense sequence. On the simulated benchmark, T2exture adds only 0.20M parameters to AMT-L while improving PSNR by 6.66 dB. Extensive evaluations on simulated and real acquisitions further show clearer texture recovery and stronger structural preservation than representative VFI baselines. These results establish T2exture as a practical framework for thermal texture imaging under sparse active acquisition.
Jiashuo Chen, Cheng Dai, Yanan Hu +1
School of Engineering, Westlake University · School of Science, Westlake University
Thermal-infrared (TIR) hyperspectral imagery (HSI) provides critical scene information for various applications. However, its practical utility is severely limited by unique sensor degradations beyond the capabilities of existing restoration methods, which are ignorant of underlying thermal physics. Here, we propose HAIR (HADAR-based Image Restoration) as a physics-driven framework for ground-based TIR-HSI restoration. HAIR utilizes the HADAR rendering equation (HRE) and combines it with the atmospheric downwelling radiative transfer equation (RTE) to model TIR-HSI using temperature, emissivity, and texture (TeX) physical triplets. This physical model leads to a TeX decompose-synthesize strategy that guarantees physical consistency and spatio-spectral noise resilience, in stark contrast to existing approaches. Moreover, our framework uses a forward-modeled atmospheric downwelling reference, along with spectral smoothness of emissivity and blackbody radiation, to enable spectral calibration and generation that would otherwise be elusive. Our extensive experiments on the outdoor DARPA Invisible Headlights dataset and in-lab FTIR measurements show that HAIR consistently outperforms state-of-the-art methods across denoising, inpainting, spectral calibration, and spectral super-resolution, establishing a benchmark in objective accuracy and visual quality.
Cheng Dai, Jiale Lin, Bingxuan Song +4
School of Science, Westlake University, Hangzhou 310030, China
Thermal cameras provide reliable visibility in darkness and adverse conditions, but thermal imagery remains significantly harder to use for novel view synthesis (NVS) than visible-light images. This difficulty stems primarily from two characteristics of affordable thermal sensors. First, thermal images have extremely low dynamic range, which weakens appearance cues and limits the gradients available for optimization. Second, thermal data exhibit rapid frame-to-frame photometric fluctuations together with slow radiometric drift, both of which destabilize correspondence estimation and create high-frequency floater artifacts during view synthesis, particularly when no RGB guidance (beyond camera pose) is available. Guided by these observations, we introduce a lightweight preprocessing and splatting pipeline that expands usable dynamic range and stabilizes per-frame photometry. Our approach achieves state-of-the-art performance across thermal-only NVS benchmarks, without requiring any dataset-specific tuning.
M. Kerem Aydin, Vishwanath Saragadam, Emma Alexander
Northwestern University · University of California, Riverside