Beyond Paired Inputs: Learning to Leverage RGB-T Cross-Modal Relations in Multimodal Large Language Models
Organizations: School of Artificial Intelligence, Beihang University · School of Robotics and Intelligent Manufacturing, Fudan University
Abstract
RGB and thermal infrared (TIR) sensors provide complementary observations, yet their distinct sensing characteristics make effective cross-modal coordination challenging. Although existing multimodal large language models (MLLMs) can process paired RGB-T inputs, they often fail to translate cross-modal relations into effective task-specific decisions, leaving the potential of these relations to improve task performance and robustness largely untapped. Motivated by this observation, we introduce RelMLLM, an RGB-T MLLM that explicitly learns to exploit cross-modal relations in task-specific decision making. It represents modality reliability, the cross-modal relation, and the dominant modality as a verifiable decision state. We derive task-grounded supervision for this state by comparing RGB-only, TIR-only, and paired RGB-T task outcomes. A hierarchical relation-guided reward supervises modality-specific evidence, relation-state prediction, and the final task output, while encouraging consistency across evidence, relation, and decision. We evaluate RelMLLM on natural RGB-T grounding, RGB-T visual question answering, and held-out sensor degradation. RelMLLM improves grounding accuracy and degradation robustness over its Qwen3-VL baselines while preserving performance on general RGB-T reasoning benchmarks. These results demonstrate that effectively exploiting cross-modal relations leads to more accurate and robust RGB-T decisions. Code and model weights are publicly available.
Figures & tables
| Benchmark | Test Samples | Task | Metric |
|---|---|---|---|
| RGBT-GroundBench | 10,124 | Ground. | [email protected] |
| RGB-TH-Bench | 1,856 | VQA | QAcc |
| DV-1K. | 1,000 | VQA | Acc. |
| Degraded-1K | 1,000 | Ground. | [email protected] |
| Method | RGB-TH-Bench Overall QAcc (%) | DV-1K Overall Acc. (%) | Degraded-1K [email protected] (%) | |||
|---|---|---|---|---|---|---|
| RGB deg. | TIR deg. | Both deg. | Overall | |||
| Vanilla GRPO-only | 74.41 | 78.10 | 31.23 | 48.65 | 12.57 | 30.80 |
| Relation-GRPO (Ours) | 75.50 (+1.09) | 78.10 (+0.00) | 57.06 (+25.83) | 74.17 (+25.52) | 50.30 (+37.73) | 60.50 (+29.70) |
| Vanilla DAPO-only | 74.52 | 78.10 | 30.03 | 48.95 | 13.17 | 30.70 |
| Relation-DAPO (Ours) | 75.38 (+0.86) | 78.60 (+0.50) | 57.66 (+27.63) | 74.17 (+25.22) | 50.60 (+37.43) | 60.80 (+30.10) |
| Vanilla GSPO-only | 75.49 | 77.80 | 32.73 | 48.05 | 14.37 | 31.70 |
| Variant | RGB-TH-Bench Overall QAcc (%) | DV-1K Overall Acc. (%) | Degraded-1K [email protected] (%) | |||
|---|---|---|---|---|---|---|
| RGB deg. | TIR deg. | Both deg. | Overall | |||
| Zero-shot | 73.40 | 77.4 | 33.33 | 35.14 | 12.87 | 27.10 |
| SFT only | 73.04 | 77.7 | 43.84 | 52.85 | 38.62 | 45.10 |
| w/o Relation-State | 75.44 | 78.2 | 45.05 | 55.86 | 41.02 | 47.30 |
| w/o Evidence Cons. | 75.50 | 78.6 | 55.56 | 72.07 | 47.90 | 58.50 |
| RelMLLM-8B | 75.61 | 78.3 | 57.36 | 74.47 | 51.50 | 61.10 |
| Model | RGB-TH-Bench QAcc. (%) | ||
|---|---|---|---|
| RGB-Txt | RGB-Th-Txt | Overall | |
| Random Baseline | 50.00 | 50.00 | 50.00 |
| InternVL3-2B | 77.59 | 59.74 | 68.67 |
| Qwen2.5-VL-3B | 78.93 | 57.62 | 68.28 |
| Qwen3-VL-4B | 82.95 | 63.13 | 73.04 |
| InternVL2.5-4B | 78.26 | 59.15 | 68.70 |
| Model | DV-1K Acc. (%) | ||
|---|---|---|---|
| DV-RGB (DV-500) | DV-TIR | Overall | |
| InternVL2.5-4B | 87.2 | 58.4 | 72.8 |
| InternVL3-2B | 88.4 | 56.0 | 72.2 |
| InternVL3-8B | 88.0 | 67.8 | 77.9 |
| LLaVA-OV-1.5-4B | 88.2 | 66.6 | 77.4 |
| LLaVA-OV-1.5-8B | 86.0 | 68.0 | 77.0 |
| Model | Degraded-1K [email protected] (%) | |||
|---|---|---|---|---|
| RGB deg. | TIR deg. | Both deg. | Overall | |
| Qwen3-VL-4B | 8.71 | 18.02 | 22.46 | 16.40 |
| 53.15 | 64.86 | 28.14 | 48.70 | |
| RelMLLM-4B | (+44.44) | (+46.84) | (+5.68) | (+32.30) |
| Qwen3-VL-8B | 33.33 | 35.14 | 12.87 | 27.10 |
| 57.36 | 74.47 | 51.50 | 61.10 | |
| Model | Rank | Test Overall | RefFLIR [email protected] (%) | RefM3FD [email protected] (%) | RefMFAD [email protected] (%) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| val | test | testA | testB | testC | val | test | testA | testB | testC | val | test | testA | testB | testC | |||
| (A) VLM-Based RGB-T Visual Grounding Models | |||||||||||||||||
| MV-CLIP-VG † | 16 | 44.00 | 45.57 | 46.01 | 72.62 | 41.56 | 16.06 | 34.52 | 38.92 | 63.42 | 50.05 | 10.01 | 47.02 | 47.52 | 82.76 | 45.07 | 23.96 |
| MV-HiVG-B † | 8 | 69.47 | 75.33 | 69.20 | 85.80 | 66.72 | 50.20 | 69.64 | 72.35 | 93.99 | 79.43 | 49.40 | 67.07 | 67.04 | 91.01 | 64.52 | 51.53 |
| MV-OneRef-B † | 9 | 61.90 | 63.82 | 62.05 | 86.19 | 57.34 | 35.43 | 62.50 | 63.98 | 89.20 | 69.34 | 37.93 | 61.62 | 59.97 | 89.25 | 56.85 | 41.58 |
| UniRGB-IR | 7 | 69.89 | 70.23 | 69.87 | 90.32 | 59.06 | 47.46 | 71.43 | 73.07 | 94.40 | 77.06 | 51.89 | 68.07 | 67.06 | 90.49 | 64.11 | 50.71 |
Multimodal Large Language Models
Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.
See all 1,147 papers