RAVEN-Bench: A Paired EO-IR Video QA Benchmark for Aerial Multimodal Understanding
Organizations: The University of British Columbia, Okanagan, Kelowna, BC, Canada · The Ohio State University, Columbus, OH, USA · TerraSense Analytics, Kelowna, BC, Canada
Abstract
Multimodal large language models (MLLMs) have advanced rapidly in image and video understanding, yet their ability to reason over paired aerial electro-optical and infrared (EO-IR) videos remains poorly understood. Such video poses a harder test than standard RGB video QA: targets are often small, evidence may appear only briefly, viewpoints change with the platform, and useful cues may appear in only one stream. We introduce RAVEN-Bench (Reasoning over Aerial Videos with EO-IR aligNment), a paired EO-IR video QA benchmark for low-altitude aerial multimodal understanding. RAVEN-Bench contains 100 temporally aligned EO-IR video pairs and 1,200 human-authored four-way multiple-choice questions, each independently audited for answer keys, modality requirements, evidence intervals, and group validity. It supports EO-only, IR-only, and paired EO+IR evaluation, and uses hierarchical capability levels and structured question groups to diagnose paired-input gain beyond the stronger unimodal condition, unimodal reliance, target consistency, and evidence-chain coherence. We evaluate frontier, open-weight, and remote-sensing MLLMs, showing that aggregate accuracy can overstate evidence-supported EO-IR reasoning. RAVEN-Bench provides controlled modality conditions and group-level diagnostics for auditing whether progress on RGB video benchmarks transfers to aerial cross-spectral understanding. Code and data are available at https://github.com/yuhu990424/RAVEN-Bench and https://huggingface.co/datasets/TerraSenseAnalytics/RAVEN-Bench.
Figures & tables
| Benchmark | Input | Sensor pairing | Labels | Corr. | Task | Scale |
| General video QA [ 14 , 5 , 15 ] | RGB video | – | QA, groups a | – | MCQ | – |
| Long-video QA [ 22 , 6 , 7 , 23 ] | RGB video | – | QA | – | VQA | – |
| RS VLMs [ 8 , 9 , 16 , 17 , 24 , 25 ] | RS image b | Multisensor | Cap., box, QA | – | Cap., VQA | – |
| EO–IR perception [ 13 , 12 ] | EO–IR/MS | Paired | Box, mask, track | – | Det., seg., track | – |
| Traffic-VQA [ 18 ] | UAV image | Optical–thermal | QA | – | VQA | 8.2K / 1.3M |
| WildFireVQA [ 19 ] | UAV image | RGB–thermal | QA | – | MCQ | 6.1K / 207K |
| Modality setting | Capability level under EO+IR | Consistency and coherence | |||||||||
| Model | |||||||||||
| Frontier API models | |||||||||||
| GPT-5.4 [ 27 ] | 45.2 | 41.6 | 46.1 | 0.9 | 51.8 | 46.7 | 39.8 | 12.0 | 30.6 | 22.8 | 23.3 |
| Gemini 3.1 Pro [ 28 ] | 46.2 | 44.9 | 49.6 | 3.4 | 52.2 | 52.2 | 44.4 | 7.8 | 36.8 | 30.2 | 19.4 |
| Gemini 3 Flash [ 29 ] | 40.7 | 43.3 | 42.6 | -0.7 | 46.1 | 43.3 | 38.4 | 7.7 | 25.4 | 20.3 | 22.3 |
| Open-weight models, model size 4B | |||||||||||
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Statistic | Value |
| EO–IR video pairs | 100 |
| Modality-specific video streams | 200 |
| QA items | 1,200 |
| Questions per video pair | 12 |
| Answer options per question | 4 |
| Total modality-stream duration | 17,939.0 s |
| Audit item | Annotation-derived count |
| Retained annotation files | 100 files |
| QA items with four answer options and one answer key | 1,200 QA items |
| Capability-level coverage | L1: 386; L2: 560; L3: 254 |
| Modality-requirement coverage | EO: 656; IR: 229; EO+IR: 315 |
| Grouped-question metadata | 1,200 QA items include group identifiers, group family, and group step |
| Consistency groups | 100 groups / 800 QA items / 8 QA items per group |
| Question-type tag | Count | QA items (%) |
| spatial_reasoning | 350 | 29.17 |
| temporal_reasoning | 348 | 29.00 |
| entity_grounding | 322 | 26.83 |
| cross_modal_phenomenon_explanation | 187 | 15.58 |
| camera_motion_vs_target_motion | 119 | 9.92 |
| thermal_evidence_interpretation | 115 | 9.58 |
| Field | Type | Description |
| sample_id | string | Unique identifier for the EO–IR video pair. |
| type | categorical | Released fine-grained retained scenario label used for coverage analysis. |
| main_event | text | Internal event descriptor used during annotation and auditing. |
| event_description | text | Internal summary of the target-centric event. |
| scenario_type | categorical | Legacy internal annotation descriptor used during QA construction; not used as an evidence-condition label or scoring slice. |
| notes | object | Annotation warnings, design notes, and acceptance gates for prompt-leakage and ambiguity checks. |
| Field | Type | Description |
| uid | string | Unique identifier for the QA item within a sample. |
| group_id | string | Identifier for the consistency or coherence group. |
| group_family | categorical | consistency or coherence . |
| group_focus | text | Short internal descriptor of the group target or evidence pattern. |
| group_step | integer | Ordered step index used for grouped scoring. |
| capability_level | categorical | L1 , L2 , or L3 . |
| Screening category | Rejection or revision rationale |
| Unsupported visual evidence | The proposed answer cannot be verified from the supplied EO or IR video evidence. |
| Multiple plausible answers | More than one option remains visually defensible after reviewing the relevant time interval. |
| Modality requirement mismatch | The annotated required modality does not match where the decisive evidence appears. |
| Language-only shortcut | The answer can be inferred from wording, option priors, or commonsense without visual evidence. |
| Pixel-registration requirement | The item requires exact EO–IR pixel alignment rather than event-level correspondence. |
| Unavailable sensor or platform metadata | The item depends on geolocation, sensor parameters, flight path, altitude, or platform details not released for evaluation. |
| Audit result | Value |
| Coverage | 100/100 pairs; 1,200/1,200 items |
| Answer-key agreement | 96.5% |
| Modality agreement | 94.4% |
| Group-validity agreement | 93.6% |
| Wording or distractor revisions | 114 items (9.5%) |
| Answer-key corrections | 42 items (3.5%) |
| Type | Level | Question or group item | Correct option / evidence |
| EO-only | L1 | In the first close EO pass, which ground relation places the aircraft? | A; EO 00:17–00:29. |
| IR-only | L1 | In the close IR pass, where is the bright pavement relative to the aircraft body? | A; IR 00:28–00:52. |
| EO+IR-required | L3 | Which cross-modal chain avoids the paint-color shortcut? | A; EO 00:17–02:12 and IR 00:28–01:25. |
| Consistency group | L1/L2 | TerraSense_3 C1 : eight grouped questions about orbit-induced screen-side changes and EO–IR placement cues. | Annotation answer keys for steps 1–8; EO and IR intervals vary by item. |
| Coherence group | L2/L3 | TerraSense_3 H1 : four ordered questions from unsafe EO motion interpretation to cross-modal full-clip reading. | Annotation answer keys for steps 1–4; EO 00:17–02:12 and IR 00:28–01:25. |
| Step | Req. | Level | Question | Consistency role |
| 1 | EO | L1 | In the first close EO pass, which ground relation places the aircraft? | Grounds the target in EO using nearby layout. |
| 2 | IR | L1 | In the close IR pass, where is the bright pavement relative to the aircraft body? | Checks matching target placement in IR. |
| 3 | EO | L1 | In the later EO pass after the orbit, which screen-side layout is visible? | Rechecks placement after viewpoint change. |
| 4 | EO | L2 | Which EO sequence describes the pavement shifting around a grass-side aircraft? | Separates camera orbit from target travel. |
| 5 | IR | L2 | Which IR sequence tracks the aircraft through the scale change? | Tests IR persistence of the same target. |
| 6 | EO+IR | L2 | Which EO detail becomes a weak direct cue in IR? | Tests whether an EO cue transfers poorly to IR. |
| Step | Req. | Level | Question | Coherence role |
| 1 | EO | L2 | When the view changes, which cue is checked against the nearby alternative? | Verifies EO tug-to-barge tow geometry against the cooler barge-pair alternative. |
| 2 | IR | L2 | Which IR cue pairing appears in this interval? | Verifies warm hull or disturbed-water contrast paired with the tow geometry. |
| 3 | EO+IR | L3 | Which combined EO/IR sequence account is visible? | Combines EO tow geometry with IR thermal/wake evidence. |
| 4 | EO+IR | L3 | For the full clip, which cue chain is visible? | Tests the final maritime event-level conclusion over the full paired observation. |
| Model | |||
| Gemini 3.1 Pro | 57.2 | 36.8 | 21.2 |
| Qwen3-VL-235B | 54.5 | 33.4 | 19.1 |
| UniRS | 57.9 | 37.2 | 23.4 |
| LLaVA-Video-7B-Qwen2 | 25.3 | 8.4 | 2.2 |
| Step | Requirement | Level | Question | Evidence role |
| 1 | EO | L2 | Which early-to-late EO chain would be unsafe to treat as aircraft travel? | Distinguishes platform orbit from target travel in EO. |
| 2 | IR | L2 | In the later IR view, which paved-surface layout surrounds the aircraft? | Checks the IR spatial layout around the same target. |
| 3 | EO+IR | L3 | Which cross-modal chain avoids the paint-color shortcut? | Requires combining EO temporal evidence with IR evidence. |
| 4 | EO+IR | L3 | Which full-clip reading uses the orbit and IR evidence together? | Tests the final event-level interpretation over the full paired observation. |
| Setting | Protocol |
| Task | Zero-shot four-way multiple-choice QA. |
| Input settings | EO-only, IR-only, and paired EO+IR. |
| Question set | The same retained 1,200 QA items and answer-option order are used across input settings. |
| Temporal budget | 16 synchronized event-relative timestamps per modality: 16 frames for EO-only and IR-only; the matched EO and IR frames at each timestamp, up to 32 modality-labeled frames, for EO+IR. |
| Visual annotations | No target boxes, crops, masks, capability labels, modality requirements, group identifiers, or answers are shown to models. |
| Prompt template | The input-specific templates in Appendix C.2 . |
| Model | Checkpoint or API snapshot | Interface | Decoding | Hardware/runtime notes |
| GPT-5.4 | OpenAI API snapshot gpt-5.4-2026-03-05 | Responses API /v1/responses ; sampled image frames | Temperature 0; max output 16 | Provider-hosted; hardware not disclosed. |
| Gemini 3.1 Pro | gemini-3.1-pro-preview | Gemini API; multimodal input | Temperature 0; max output 16 | Provider-hosted; hardware not disclosed. |
| Gemini 3 Flash | gemini-3-flash-preview | Gemini API; multimodal input | Temperature 0; max output 16 | Provider-hosted; hardware not disclosed. |
| SmolVLM2-2.2B | HuggingFaceTB/SmolVLM2-2.2B-Instruct | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| Qwen3-VL-2B | Qwen/Qwen3-VL-2B-Instruct | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| VideoLLaMA3-2B | DAMO-NLP-SG/VideoLLaMA3-2B | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| Model | Checkpoint or API snapshot | Interface | Decoding | Hardware/runtime notes |
| Keye-VL-1.5-8B | Kwai-Keye/Keye-VL-1.5-8B | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| MiMo-VL-7B-RL-2508 | XiaomiMiMo/MiMo-VL-7B-RL-2508 | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| VideoLLaMA3-7B | DAMO-NLP-SG/VideoLLaMA3-7B | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| InternVideo2.5-Chat-8B | OpenGVLab/InternVideo2_5_Chat_8B | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| mPLUG-Owl3-7B | mPLUG/mPLUG-Owl3-7B-241101 | Transformers/ModelScope; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| LLaVA-Video-7B-Qwen2 | lmms-lab/LLaVA-Video-7B-Qwen2 | LLaVA-NeXT checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| Input setting | Timestamps | Frames per sensor | Total frames |
| EO-only | 16 | 16 EO | 16 |
| IR-only | 16 | 16 IR | 16 |
| EO+IR | 16 | 16 EO + 16 IR | up to 32 |
| Model output pattern | Parsed prediction |
| A | A |
| The answer is B. | B |
| Option C is correct. | C |
| No valid option found | Incorrect |
| Multiple valid options found | First valid option |
| Setting | Visual input | |||
| Random | None | 24.8 | 10.4 | 8.5 |
| Answer-prior | None | 26.1 | 10.8 | 8.8 |
| Text-only (Qwen3.5-9B [ 34 ] ) | None | 22.9 | 8.3 | 6.6 |
| Text-only (best over evaluated models) | None | 28.4 | 12.1 | 9.8 |
| Human subset (120 QA / 10 pairs) | EO+IR | 82.3 | 77.1 | 69.6 |
| Best MLLM on same subset | EO+IR | 50.4 | 38.6 | 31.8 |
| Condition | Control purpose | ||||
| Aligned EO+IR | 43.5 | +3.1 | 45.7 | 23.4 | Main paired-input setting. |
| EO+EO | 41.1 | +0.7 | 42.8 | 20.6 | Controls for two-stream input budget using duplicated EO evidence. |
| IR+IR | 40.7 | +0.3 | 43.1 | 21.1 | Controls for two-stream input budget using duplicated IR evidence. |
| Modality-shuffled EO+IR | 39.6 | -0.8 | 41.2 | 18.3 | Tests whether true EO–IR correspondence is required. |
| Temporally misaligned EO+IR | 40.1 | -0.3 | 39.9 | 17.4 | Tests whether synchronized EO–IR timestamps are required. |
| Temporal-shuffle | 39.2 | -1.2 | 37.4 | 15.8 | Tests whether ordered temporal evidence is required. |
| 4 | 38.4 | +0.9 | 39.6 | 16.8 | 21.6 |
| 8 | 41.3 | +2.1 | 43.1 | 20.1 | 21.2 |
| 16 | 43.5 | +3.1 | 45.7 | 23.4 | 20.1 |
| 32 | 44.1 | +3.2 | 46.3 | 23.9 | 20.2 |
| Evaluation set | MC Main | FF Main | Answer+Evidence | FF Cons. | FF Coh. |
| RAVEN-Bench (100 pairs) | 43.5 | 34.8 | 27.1 | 24.6 | 19.3 |
| Error mechanism | Physical condition | Model behavior | Primary readouts |
| Occlusion-driven identity swap | Foreground trees, moving occluders, nearby vehicles, viewpoint changes, or scale changes interrupt the target track. | The model binds post-occlusion evidence to a visually similar nearby object instead of preserving the same target identity. | , |
| Tiny-target evidence loss | The target occupies only a few pixels after resizing, frame sampling, or video-token compression; glare, reflection, or surface texture further reduces usable evidence. | The model detects a local cue or salient artifact but fails to preserve the target across frames or modalities. | , |
| Sensor-salience distractor | EO glare, shadows, wakes, hot backgrounds, low thermal contrast, or thermal crossover dominates the visible or thermal salience field. | The model follows the visually or thermally dominant distractor rather than the smaller target cue. | , |
| Cross-modal binding failure | EO and IR provide asymmetric evidence: EO may preserve shape, color, texture, or wake geometry, while IR may emphasize thermal contrast or suppress visible-spectrum cues. | The model relies on one stream, switches between inconsistent candidates, or pairs an EO cue with the wrong IR evidence. | , |
Video Question Answering
Video question answering studies how AI systems use visual and audio evidence over time to answer questions about video.
See all 197 papers