cs.CVNov 14, 2026

Beyond Paired Inputs: Learning to Leverage RGB-T Cross-Modal Relations in Multimodal Large Language Models

Authors: Tianyi Zhao, Qihao Xu, Jiawen Xi, Duanni Meng, Bo Zhao, Maoxun Yuan, Xingxing Wei, Bo Li

Organizations: School of Artificial Intelligence, Beihang University · School of Robotics and Intelligent Manufacturing, Fudan University

Abstract

RGB and thermal infrared (TIR) sensors provide complementary observations, yet their distinct sensing characteristics make effective cross-modal coordination challenging. Although existing multimodal large language models (MLLMs) can process paired RGB-T inputs, they often fail to translate cross-modal relations into effective task-specific decisions, leaving the potential of these relations to improve task performance and robustness largely untapped. Motivated by this observation, we introduce RelMLLM, an RGB-T MLLM that explicitly learns to exploit cross-modal relations in task-specific decision making. It represents modality reliability, the cross-modal relation, and the dominant modality as a verifiable decision state. We derive task-grounded supervision for this state by comparing RGB-only, TIR-only, and paired RGB-T task outcomes. A hierarchical relation-guided reward supervises modality-specific evidence, relation-state prediction, and the final task output, while encouraging consistency across evidence, relation, and decision. We evaluate RelMLLM on natural RGB-T grounding, RGB-T visual question answering, and held-out sensor degradation. RelMLLM improves grounding accuracy and degradation robustness over its Qwen3-VL baselines while preserving performance on general RGB-T reasoning benchmarks. These results demonstrate that effectively exploiting cross-modal relations leads to more accurate and robust RGB-T decisions. Code and model weights are publicly available.

Figures & tables

Explore similar work

CardsList
  1. DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning

    Apr 20, 2026Abrar Majeedi, Zhiyuan Ruan, Ziyi Zhao +3Cross-Modal LearningVisual Question Answering

  2. Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks

    Jun 9, 2026Miaoxin Cai, Guanqun Wang, Wei Zhang +6Multimodal UnderstandingMultimodal Grounding

  3. RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation

    Mar 4, 2026Hao Li, Yuhao Wang, Wenning Hao +3Visual Object TrackingMultimodal RAG