Firebolt-VL: Efficient Vision-Language Understanding with Cross-Modality Modulation

Quoc-Huy Trinh; Mustapha Abdullahi; Bo Zhao; Debesh Jha

cs.CVApr 6, 2026

Firebolt-VL: Efficient Vision-Language Understanding with Cross-Modality Modulation

Authors: Quoc-Huy Trinh, Mustapha Abdullahi, Bo Zhao, Debesh Jha

Organizations: Aalto University · University of South Dakota

Abstract

Recent advances in multimodal large language models (MLLMs) have enabled impressive progress in vision-language understanding, yet their high computational cost limits deployment in resource-constrained scenarios such as personal assistants, document understanding, and smart cameras. Most existing methods rely on Transformer-based cross-attention, whose quadratic complexity hinders efficiency. Moreover, small vision-language models often struggle to precisely capture fine-grained, task-relevant visual regions, leading to degraded performance on fine-grained reasoning tasks that limit their effectiveness in the real world. To address these issues, we introduce Firebolt-VL, an efficient vision-language model that replaces the Transformer-based decoder with a Liquid Foundation Model (LFM) decoder. To further enhance visual grounding, we propose a Token-Grid Correlation Module, which computes lightweight correlations between text tokens and image patches and modulates via the state-space model with FiLM conditioning. This enables the model to selectively emphasize visual regions relevant to the textual prompt while maintaining linear-time inference. Experimental results across multiple benchmarks demonstrate that Firebolt-VL achieves accurate, fine-grained understanding with significantly improved efficiency. Our model and code are available at: https://fireboltvl.github.io

Figures & tables

Panel of Figure 1 — Figure 1: Performance and latency efficiency of Firebolt-VL on multiple tasks compared with state-of-the-art baselines. Firebolt-VL is both competitive and efficient, demonstrating strong generalization across diverse tasks.

Figure 2: Overview of the Firebolt-VL architecture. The Cross-Modal Modulator (CMM) fuses textual instructions with the visual representations of the query image to produce conditioned tokens, which are then processed by the Liquid Foundation Model (LFM). The model is trained in two stages: (1) CMM pre-training to initialize modulation parameters, and (2) end-to-end training of the full framework.

Method	LLM	Parameters	VQAv2	POPE	AI2D	MMMU ${}_{\text{val}}$	MME p	SQA I	MMB ${}_{\text{dev}}$
IDEFICS [ 21 ]	LLaMA	9.0B	60.0	81.9	42.2	18.4	1177.3	53.5	45.3
OpenFlamingo v2 [ 3 ]	MPT	9.0B	60.4	52.6	31.7	28.8	607.2	44.8	–
MiniGPT-4-v1 [ 47 ]	Vicuna	8.0B	–	34.6	28.4	23.6	1047.4	39.6	–
MiniGPT-4-v2 [ 5 ]	LLaMA 2	8.0B	–	60.0	30.5	25.0	968.4	54.7	–
Chameleon [ 37 ]	LLaMA 2	7.0B	–	19.4	46.0	22.4	202.7	46.8	–
FastV [ 6 ]	Vicuna	7.0B	55.0	48.0	42.7	22.0	873.2	51.1	–

Table 1: Quantitative comparison of the proposed Firebolt-VL model with existing MLLMs across seven benchmarks. The superscript

p

denotes the perception score on the MME benchmark, while SQA I refers the ScienceQA-IMG [ 30 ] . The best results are shown in bold , and the second-best results are underlined . “–” indicates results not reported in the original papers.

Figure 3: Qualitative comparison of responses from Firebolt-VL with recent efficient vision–language models, including MobileVLM V2 [ 10 ] and SmolVLM2 [ 31 ] , on detail-dependent question-answering tasks. Firebolt-VL demonstrates stronger fine-grained grounding and more accurate, instruction-aligned responses.

Figure 4: Accuracy–latency comparison of compact MLLMs. Bubble area denotes parameter count (B), and annotations indicate the exact model size. Firebolt-VL provides a favorable accuracy–latency trade-off, achieving a higher MME p perception score at lower inference latency.

Approach	POPE	AI2D	MMMU ${}_{\text{val}}$	Average
Mamba [ 15 ]	57.4	39.9	23.6	40.3
S4D [ 16 ]	69.9	45.6	23.7	46.4
S4 [ 17 ]	69.4	46.2	26.4	47.3

Table 2: Performance comparison of Mamba, S4D, and S4 state-space models on POPE [ 24 ] , AI2D [ 20 ] , and MMMU

{}_{\text{val}}

[ 45 ] benchmarks. Structured models (S4/S4D) outperform Mamba.

Approach	MME	AI2D	MMMU ${}_{\text{val}}$
Prepend	981.5	24.1	22.1
Cross-attended	1036.5	45.1	24.4
CMM (Ours)	1376.2	46.2	26.4

Table 3: Performance of different connector methods on MME [ 13 ] , AI2D [ 20 ] , and MMMU

{}_{\text{val}}

[ 45 ] . Our proposed CMM outperforms both Prepend and Cross-attention fusion across all benchmarks.

Dataset	$k=1$	$k=2$	$k=3$	$k=4$	$k=5$
MME	1039.8	1192.0	1258.6	1376.2	1289.4
AI2D	43.9	44.1	45.4	46.2	45.6
MMMU ${}_{\text{val}}$	22.2	24.3	25.8	26.4	26.0

Table 4: Effect of top-

k

grid selections on MME [ 13 ] , AI2D [ 20 ] , and MMMU

{}_{\text{val}}

[ 45 ] . Performance improves up to

k=4

and slightly drops at

k=5

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

Distilling Visual Reasoning into Text Space
Sep 28, 2026Wenhan Yang, Nilay Naharas, Ali Payani +1CoT Distillation VLM Distillation
University of California, Los Angeles · Cisco Research
VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions
Mar 24, 2026Adrian Bulat, Alberto Baldrati, Ioannis Maniadis Metaxas +2Efficient VLM Inference Vision-Language Models
Samsung AI Cambridge · Technical University of Iasi · Queen Mary University of London
LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models
Feb 26, 2026Yuxiang Shen, Hailong Huang, Zhenkun Gao +6VLM Reasoning Large Vision-Language Models
the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China +3

Sep 28, 2026J/K move · Enter open · S save

Distilling Visual Reasoning into Text Space

Wenhan Yang, Nilay Naharas, Ali Payani, Baharan Mirzasoleiman

University of California, Los Angeles · Cisco Research

CoT Distillation VLM Distillation Large Vision-Language Models VLM Reasoning

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have shown strong promise for multimodal reasoning, yet often struggle with tasks requiring concepts beyond what is directly observable in the input image. Existing methods generate intermediate images or latent visual tokens to guide reasoning, but these representations can introduce errors and increasingly interfere with textual reasoning as reasoning progresses. We propose Visual-to-Text Chain-of-Thought Distillation (V2T), a framework that enables LVLMs to internalize visual reasoning without generating intermediate visual representations at inference time. V2T first trains a teacher LVLM using interleaved visual and textual chains of thought, and then uses knowledge distillation to train a student LVLM using the teacher's logits and cross-entropy supervision from ground-truth textual reasoning. When reasoning images can be mapped to the original image, V2T can additionally distill the teacher's attention to corresponding regions, while ground-truth bounding boxes can further guide a subsequent reinforcement learning stage. Experiments across multiple multimodal reasoning benchmarks show that V2T consistently outperforms the teacher and existing baselines, improving average accuracy by 14.3% on a held-out set and 2.7% on the broader visual evaluation suite. Moreover, lightweight SFT and substantially reduced RL make V2T up to 42x faster to train than state-of-the-art baselines.

Abstract

Efficient VLM Inference

Explore similar work

Distilling Visual Reasoning into Text Space

VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models