Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models

Baoheng Zhang; Jiahui Liu; Gui Zhao; Weizhou Zhang; Yixuan Ma; Jun Jiang; Yingxian Chen; Wilton W. T. Fok

cs.CVMar 29, 2026

Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models

Authors: Baoheng Zhang, Jiahui Liu, Gui Zhao, Weizhou Zhang, Yixuan Ma, Jun Jiang, Yingxian Chen, Wilton W. T. Fok,

Organizations: The University of Hong Kong

Abstract

Multimodal Large Language Models (MLLMs) perform strong vision-language reasoning under standard conditions but fail in extreme illumination, where RGB inputs lose irrevocable structure and semantics. We propose Event-MLLM, an event-enhanced model that performs all-light visual reasoning by dynamically fusing event streams with RGB frames. Two key components drive our approach: an Illumination Indicator - a learnable signal derived from a DINOv2 branch that represents exposure degradation and adaptively modulates event-RGB fusion - and an Illumination Correction Loss that aligns fused features with non-degraded (normal-light) semantics in the latent space, compensating for information lost in extreme lighting. We curate the first multi-illumination event-instruction corpus for MLLMs, with 2,241 event-RGB samples (around 6 QA pairs each) across diverse scenes and 17 brightness rates (0.05x - 20x), plus an instruct-following benchmark for reasoning, counting, and fine-grained recognition under extreme lighting. Experiments show that Event-MLLM markedly outperforms general-purpose, illumination-adaptive, and event-only baselines, setting a new state of the art in robust multimodal perception and reasoning under challenging illumination.

Figures & tables

Figure 1: Extreme lighting scenarios are demonstrated. When the normal (a) RGB frame can not be obtained, only (c) overexposed or (d) underexposed frame can be used as input. Existing baselines often produce illusions due to information degradation, while our method cleverly incorporates information from the (b) event frame, enabling the model to provide more accurate responses.

Figure 2: An overview of our proposed model is illustrated: (a) Features are extracted from extreme-light RGB frames by the original vision encoder and the DINOv2 vision encoder, respectively. These features are then fused with the features extracted from the event frame using our designed MLPs with our proposed Illumination Indicator ( $F_{\text{illu}}$ ) to obtain fused features $F_{\text{fusion}}$ (Section 3.2 ). (b) Features are extracted from normal-light RGB frames and used for representation learning with the fused features using our proposed Illumination Correction Loss $\mathcal{L}_{\mathrm{IC}}$ (Section 3.3 ). (c) During training, the fused features participate in the fine-tuning of the entire MLLM, while (a) and (c) participate in inference (Section 3.4 ).

MLLM Type	Method	Multi-Choice		Object-Counting
MLLM Type	Method	Accuracy (%) $\uparrow$	F1 Score (%) $\uparrow$	Accuracy (%) $\uparrow$	MAE $\downarrow$
General Model	LLaVA-7B [ 32 ]	18.43	65.39	67.84	0.9957
	Deepseek-VL [ 56 ]	7.00	55.70	54.87	1.0544
	InternVL [ 72 ]	31.50	75.46	72.56	0.8641
Event-specialized Model	EventGPT [ 37 ]	3.16	73.60	67.41	1.0015
Illumination-aware Model	Q-Instruct [ 53 ]	15.86	65.09	66.87	0.9803

Table 1: Compared to existing MLLMs of different types, our method significantly outperforms all other methods to achieve state-of-the-art performance on both multi-choice and object-counting tasks on all metrics. Furthermore, we take experiments with different sizes of Qwen [ 51 ] as baselines and apply our design, where consistently achieves performance improvements (Best results are shown in bold ).

Figure 3: Qualitative comparison under various illumination levels for the multi-choice task (top) and the object-counting task (bottom). Under severe underexposure and overexposure, the baseline model frequently produces incorrect or hallucinated predictions, whereas our illumination-guided event-enhanced model remains stable and accurate across all brightness ratios. These examples illustrate the robustness of our method in extreme-light conditions.

Task	Metric	Backbone: Qwen-3B			Backbone: Qwen-7B
Task	Metric	Baseline	Ill-Correction	Ours	Baseline	Ill-Correction	Ours
Multi-Choice	Accuracy (%) $\uparrow$	29.86	32.95	34.48	44.71	51.90	53.13
Multi-Choice	F1 Score (%) $\uparrow$	77.04	79.13	80.35	81.95	84.83	85.43
Object-Counting	Accuracy (%) $\uparrow$	69.66	70.48	71.08	72.73	74.23	74.66
Object-Counting	MAE $\downarrow$	0.6655	0.6194	0.6127	0.5303	0.4645	0.4557

Table 2: Ablation studies on illumination-aware adaptive feature fusion via Illumination-Correction (Section 3.3 ) and LoRA fine-tuning (Section 3.4 ). Starting from the baseline (w/o both components), we evaluate the contribution of Illumination-Correction alone (w/o LoRA fine-tuning), and the full two-stage pipeline (w. Both Illumination-Correction and LoRA fine-tuning) on multi-choice and object-counting tasks. (Best results are shown in bold ).

Backbone: Qwen-3B
Method	Multi-Choice QA		Object-Counting
Method	Acc. (%) $\uparrow$	F1 (%) $\uparrow$	Acc. (%) $\uparrow$	MAE $\downarrow$
Pre-fusion	19.58	73.96	68.21	0.7698
Post-fusion	26.17	77.51	70.52	0.6427
Ours	34.48	80.35	71.08	0.6127
Backbone: Qwen-7B

Table 3: Ablation studies on direct training strategies: Pre-fusion (pixel-level summation), Post-fusion (feature-level addition), and our 2 staged training method on multi-choice and object-counting tasks. We evaluate the performance comparison across Qwen-3B and Qwen-7B backbones (Best results are shown in bold ).

Backbone	Brightness Rate (Underexposure)	0.05	0.08	0.1	0.125	0.2	0.4	0.5	0.75
Qwen-3B	w/o Illumination Correction	0.6597	0.7429	0.7782	0.8069	0.8593	0.9342	0.9554	0.9886
Qwen-3B	with Illumination Correction	0.9273	0.9368	0.9405	0.9439	0.9508	0.9619	0.9652	0.9710
Qwen-7B	w/o Illumination Correction	0.7420	0.8011	0.8272	0.8520	0.8981	0.9551	0.9706	0.9927
Qwen-7B	with Illumination Correction	0.9185	0.9311	0.9359	0.9416	0.9508	0.9615	0.9647	0.9697
w/o	Brightness Rate (Overexposure)	2	3	5	7.5	8	10	15	20
Qwen-3B	w/o Illumination Correction	0.9741	0.9556	0.9245	0.8722	0.8618	0.8187	0.7240	0.6510

Table 4: Cosine similarity comparison of vision encoder features between original and varying brightness ratios with/without illumination correction (Section 3.2 ) across Qwen-3B/7B backbones.

Figure 4: Visualization results of features before and after fusion with t-SNE. (a) Before fusion: event features and RGB features with different brightness ratios. (b) After fusion, fusion features under different brightness ratios cluster around the normal-light feature.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning
Apr 20, 2026Abrar Majeedi, Zhiyuan Ruan, Ziyi Zhao +3Cross-Modal Learning Visual Question Answering
University of Wisconsin-Madison · Amazon · Northeastern University
RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding
May 19, 2026Hanqing Liu, Mingjie Liu, Luoping Cui +3Visual Question Answering Event-Based Vision
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China · School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China 2State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China
Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification
Mar 27, 2026Shuai Lv, Chang Liu, Feng Tang +5Multimodal Large Language Models Multimodal Reasoning
University of Science and Technology of China, Hefei, China · Huawei Foundation Model Department, Shanghai, China · The Chinese University of Hong Kong, Hong Kong, China +1

Apr 20, 2026J/K move · Enter open · S save

DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning

Abrar Majeedi, Zhiyuan Ruan, Ziyi Zhao, Hongcheng Wang+2

University of Wisconsin-Madison · Amazon · Northeastern University

Cross-Modal Learning Visual Question Answering VLM Robustness Large Vision-Language Models Multimodal Large Language Models Multimodal Reasoning+2

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have achieved impressive performance on visual perception and reasoning tasks with RGB imagery, yet they remain fragile under common degradations, such as fog, blur, or low-light conditions. Infrared (IR) imaging, a well-established complement to RGB, offers inherent robustness in these conditions, but its integration into MLLMs remains underexplored. To bridge this gap, we propose DUALVISION, a lightweight fusion module that efficiently incorporates IR-RGB information into MLLMs via patch-level localized cross-attention. To support training and evaluation and to facilitate future research, we also introduce DV-204K, a dataset of ~25K publicly available aligned IR-RGB image pairs with 204K modality-specific QA annotations, and DV-500, a benchmark of 500 IR-RGB image pairs with 500 QA pairs designed for evaluating cross-modal reasoning. Leveraging these datasets, we benchmark both open- and closed-source MLLMs and demonstrate that DUALVISION delivers strong empirical performance under a wide range of visual degradations. Our code and dataset are available at https://abrarmajeedi.github.io/dualvision.

Abstract

Vision-Language Models

Explore similar work

DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning

RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding

Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification