Physically Guided TeX Generation for Low-Light Visual Tasks
Authors: Daoheng Li, Jianjun Qian, Mengkai Yan, Jian Yang
Organizations: PCA Lab, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China · College of Computer and Software, Hohai University, Nanjing, China · PCA Lab, School of Intelligence Science and Technology, Nanjing University, Nanjing, China
Low-light conditions reduce visible contrast and obscure details, while infrared images lack color and texture, motivating the fusion of infrared and visible information for visual tasks. Existing fusion methods primarily optimize image appearance or task-specific features without explicitly disentangling the coupled factors underlying thermal observations, limiting their physical interpretation under thermal degeneracy. Hyperspectral thermal measurements enable physical decomposition, but they require specialized imaging hardware, while pixel-level physical annotations for supervision or evaluation incur additional costs. We propose TeX-Gen, a physically guided framework for TeX generation from paired visible and infrared images in low-light visual tasks. TeX Generation constructs complementary multi-view features and employs independent branch-specific gates to route cues into the Temperature, Emissivity, and Texture maps. TeXMetric integrates radiometric reconstruction, channel structure, and material consistency to assess the generated maps without requiring pixel-level physical labels. TeX Tune improves performance through task-guided adjustments, jointly optimizing the adjustment modules and task network while keeping the generator frozen. Experiments show that TeX-Gen achieves competitive task performance in low-light environments on the FLIR, LLVIP, M3FD, and MSRS datasets. Experiments on HADAR also examine the association between TeXMetric scores and downstream detection performance.
Figures & tables
Fig. 2: Overview of TeX-Gen. TeX Generation constructs multi-view features online and selects complementary evidence through channel-specific gates to form the baseline TeX Zb under TeXMetric constraints. With the generator frozen, TeX Tune applies task-specific corrections to Zb through Multi-scale Guide, Local Activity Gate, and Region-Aware processing before passing the tuned TeX Za to the complete downstream task network. Task supervision jointly optimizes the channel-tuning modules and task network, whereas the task penalty updates only the channel-tuning modules.
Fig. 3: Radiometric reconstruction in TeXMetric. Emitted and reflected environmental contributions form a normalized infrared reconstruction for comparison with the observed image.
Fig. 4: Channel-structure constraints involving all three channels. Shared source structure guides T , e , and X . Candidate material edges are attenuated in zX , while eligible pairs with similar ze guide temperature ordering in zT .
Fig. 5: Material consistency through fixed-interval emissivity decoding, edge-aware local smoothness, and local ordering. The ordering schematic uses visible contrast to guide relative ze responses for infrared-similar pixel pairs.
Method
Year
Modality
mAP50
mAP75
mAP50:95
DDQ-DETR [ 55 ]
2023
RGB
73.9
32.2
37.1
YOLOv5 [ 56 ]
2020
RGB
67.8
25.9
31.8
DDQ-DETR [ 55 ]
2023
IR
64.9
24.5
30.9
YOLOv5 [ 56 ]
2020
IR
73.9
35.7
39.5
GAFF [ 5 ]
2021
IR+RGB
74.6
31.3
37.4
ProbEn [ 9 ]
2022
IR+RGB
75.5
31.8
37.9
TABLE I: Detection results on FLIR (%). N/A indicates a value not reported in the corresponding publication.
Method
Year
Modality
mAP50
mAP75
mAP50:95
DDQ-DETR [ 55 ]
2023
RGB
86.1
45.8
46.7
YOLOv5 [ 56 ]
2020
RGB
90.8
51.9
50.0
DDQ-DETR [ 55 ]
2023
IR
93.9
64.6
58.6
YOLOv5 [ 56 ]
2020
IR
94.6
72.2
61.9
GAFF [ 5 ]
2021
IR+RGB
94.0
60.2
55.8
ProbEn [ 9 ]
2022
IR+RGB
93.4
50.2
51.5
TABLE II: Detection results on LLVIP (%). N/A indicates a value not reported in the corresponding publication.
Method
Year
Modality
mAP50
mAP50:95
CFT [ 6 ]
2022
IR+RGB
89.8
60.7
TarDAL [ 7 ]
2022
IR+RGB
86.0
56.0
MetaFusion [ 18 ]
2023
IR+RGB
86.7
56.8
CDDFuse [ 58 ]
2023
IR+RGB
87.0
56.5
ICAFusion [ 27 ]
2024
IR+RGB
87.4
57.2
Text-IF [ 59 ]
2024
IR+RGB
N/A
40.4
TABLE III: Aggregate detection results on M3FD (%). N/A indicates a value not reported in the corresponding publication.
Method
Year
Modality
Unlabelled
Car
Person
Bike
Curve
Car Stop
Guardrail
Color Cone
Bump
mIoU
EMMA [ 64 ]
2024
IR+RGB
98.37
89.32
71.15
70.58
60.71
70.37
69.59
64.75
75.50
74.48
DCINN [ 65 ]
2024
IR+RGB
98.34
89.08
70.94
69.19
59.31
73.27
72.21
64.02
72.76
74.35
MRFS [ 66 ]
2024
IR+RGB
98.32
88.80
70.55
70.41
58.76
73.85
74.66
62.80
72.37
74.50
TIMFusion [ 67 ]
2024
IR+RGB
98.30
89.19
68.68
69.64
59.55
71.94
69.48
63.69
71.75
73.58
TDFusion [ 68 ]
2025
IR+RGB
98.38
89.75
71.76
70.36
59.77
72.40
75.06
64.61
73.73
75.09
KDFuse [ 69 ]
2025
IR+RGB
98.20
87.50
69.70
60.50
64.70
60.60
85.50
46.90
82.90
73.00
TABLE IV: Nine-class MSRS semantic-segmentation results (%). Unlabelled denotes Background, and mIoU includes all nine classes.
Fig. 6: Detection comparison summarized from Tables I through III . Solid blue and hatched orange bars denote Ours and MVR-MoE, respectively, and Δ is Ours minus MVR-MoE in AP percentage points.
Variant
mAP50
mAP75
mAP50:95
mAPs
mAPm
mAPl
Full TeX Tune
87.5
54.2
52.9
39.1
59.0
73.3
w/o penalty term
86.0
49.0
50.0
36.4
56.0
71.6
w/o multi-scale guide
86.2
49.0
49.8
36.5
55.5
70.9
TABLE V: Ablation of TeX Tune components on FLIR.
Generation criterion
mAP50
mAP75
mAP50:95
mAPs
mAPm
mAPl
Radiometric reconstruction
80.5
45.4
46.2
32.1
52.5
70.9
Reconstruction + X structure
81.8
46.0
47.3
33.5
53.9
68.3
Reconstruction + material consistency
81.9
45.1
47.0
33.3
53.2
69.4
TABLE VI: Ablation of TeXMetric constraint groups on FLIR.
Fig. 7: Downstream responses to TeXMetric constraints on FLIR, summarized from Table VI .
Direct-input control and task-adjusted TeX
Condition
mAP50
mAP75
mAP50:95
Generated TeX (without adjustment)
85.5
47.8
49.3
Task-adjusted TeX
87.5
54.2
52.9
Joint tuning ( mAP50:95 )
Trainable components
Best epoch
Best before end Best epoch before final epoch
Channel adjustment only (detector frozen)
100
No
TABLE VII: FLIR input-configuration comparison and joint task tuning.
Fig. 8: Joint task tuning on FLIR dataset during TeX Tune. Updating channel corrections alone with the inherited detector frozen makes limited progress, whereas jointly updating the corrections and task branch enables the task network to learn from the tuned TeX.
Fig. 9: Local TeX changes after task tuning on LLVIP. Brighter pixels in the cropped absolute-difference maps indicate larger changes.
Fig. 10: TeXMetric improvement and downstream detection on HADAR. (a) Separate plots of TeXMetric score (lower is better) and AP under YOLOv8-S detector (higher is better), evaluated at the same generator checkpoints. (b) Ground-truth labels and generated maps at epochs 50, 100, 150, and 200 for street, crowded-street, and highway views.
Generation objective
mAP50
mAP75
mAP50:95
Radiometric reconstruction
80.5
45.4
46.2
Radiometric reconstruction + Entropy
81.4
43.8
45.7
Radiometric reconstruction + Gradient energy
80.9
43.3
45.4
Radiometric reconstruction + MI
81.9
44.6
46.2
Radiometric reconstruction + SSIM
83.9
44.8
47.1
Radiometric reconstruction + PSNR
82.5
43.5
45.9
TABLE VIII: FLIR detection results for radiometric reconstruction with conventional image-quality constraints.
TeX generation objective
MSE ↓
RMSE ↓
mAP50
mAP75
mAP50:95
AR100
Radiometric reconstruction
0.1357
0.3684
50.4
26.5
28.7
40.2
Radiometric reconstruction + pixel MSE
0.0555
0.2355
54.0
25.0
27.4
41.8
Radiometric reconstruction + pixel RMSE
0.0535
0.2313
42.4
16.9
21.5
36.4
TeXMetric
0.1319
0.3632
54.1
38.1
34.4
43.4
TABLE IX: Pixel-level supervision versus TeXMetric on HADAR under matched training budgets.
Fig. 11: Downstream predictions and map results on HADAR for TeX generated under TeXMetric constraints versus pixel-level supervision (MSE and RMSE).
Infrared and visible images provide complementary cues for low-light image generation, and physics-based methods can better exploit this complementarity. However, existing physics-based methods still rely heavily on paired data. To address these limitations, we propose Multi-View Routing Mixture-of-Experts (MVR-MoE), a framework for physically consistent TeX generation that learns multi-view structural representations. MVR-MoE mines intrinsic structural cues from infrared and visible images to organize them into multi-view representations, while the proposed multi-view hyper connections expand the feature space and preserve both shared evidence and modality-specific details. A hierarchical routing mechanism further regulates feature utilization at both the structural-view level and the expert level, enabling adaptive exploitation of heterogeneous cues. The predicted temperature, emissivity, and texture maps are constrained by the Stefan-Boltzmann law to ensure physically consistent. Extensive experiments on challenging datasets demonstrate that MVR-MoE achieves superior robustness and discriminability while remaining lightweight.
Daoheng Li, Jianjun Qian*, Mengkai Yan +1
Nanjing University of Science and Technology, Nanjing, Jiangsu, China · Hohai University, Nanjing, Jiangsu, China
Temperature-emissivity-texture (TeX) decomposition seeks to recover object heat state, material spectral response, and visible-like geometric texture from long-wave infrared hyperspectral imaging (LWIR HSI). Existing TeX pipelines are mainly scene-specific inverse solvers, and the lack of paired LWIR HSI-TeX supervision has limited learning-based decomposition. To address this gap, we introduce TeX-1500, a large-scale paired LWIR HSI-TeX dataset and benchmark for supervised HSI-to-TeX decomposition. TeX-1500 contains 1,522 calibrated real-scene pairs from DARPA Invisible Headlights (DARPA IH) pushbroom imagery and our FTIR acquisitions, covering five locations, four seasons, diverse acquisition times, heterogeneous wavelength layouts, and two sensor families. Each sample stores a calibrated valid-band radiance cube, calibrated wavelength positions, and aligned temperature, emissivity, and texture supervision constructed through a consistent restoration and TeX-construction protocol. We further provide TeX-UNet, a simple wavelength-aware baseline that maps calibrated HSI bands and wavelength positions to TeX fields. Experiments on the held-out DARPA IH pushbroom scenes and zero-/few-shot transfer to FTIR scenes show that TeX-1500 provides usable paired supervision and a measurable benchmark for data-driven physical-property-centered thermal perception.
Cheng Dai, Jiale Lin, Hongyi Xu +3
School of Science, Westlake University, Hangzhou 310030, China · School of Engineering, Westlake University, Hangzhou 310030, China
Vision-language models (VLMs) often fail under low illumination because their visual grounding is learned predominantly from RGB imagery, whereas thermal infrared preserves complementary scene structure when visible cues degrade. We present Thermo-VL, a wavelength-aware VLM that augments a frozen Molmo-7B backbone with a trainable thermal encoder and a text-guided dual-attention fusion module. Given aligned RGB tokens, thermal tokens, and prompt embeddings, the fusion module conditions thermal features on both language and RGB context, then injects a gated residual into the frozen RGB stream so thermal evidence can be incorporated without disrupting Molmo's pretrained RGB-language interface. We train the model with the standard language-modeling objective together with auxiliary alignment and regularization losses that improve cross-modal grounding and reduce over-reliance on RGB. We also introduce a pixel-aligned RGB-thermal instruction-tuning dataset and Thermo-VL-Bench, a manually screened RGB-thermal VQA benchmark for low-light and cross-spectrum reasoning. Experiments show strong gains on challenging thermal-only and RGB+thermal reasoning tasks, highlighting the value of prompt-conditioned multispectral fusion. Our dataset and code are publicly available at: https://thusharakart.github.io/Thermo-VL
Rusiru Thushara, Yasiru Ranasinghe, Jay Paranjape +1
Department of Electrical & Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA