SpikeMLLM: Spike-based Multimodal Large Language Models via Modality-Specific Temporal Scales and Temporal Compression

Han Xu; Zhiyong Qin; Di Shang; Jiahong Zhang; Xuerui Qiu; Bo Lei; Tiejun Huang; Bo Xu

cs.NEApr 13, 2026

SpikeMLLM: Spike-based Multimodal Large Language Models via Modality-Specific Temporal Scales and Temporal Compression

Authors: Han Xu, Zhiyong Qin, Di Shang, Jiahong Zhang, Xuerui Qiu, Bo Lei, Tiejun Huang, Bo Xu,

Organizations: Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence · Zhongguancun Academy ·

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable progress but incur substantial computational overhead and energy consumption during inference, limiting deployment in resource-constrained environments. Spiking Neural Networks (SNNs), with their sparse event-driven computation, offer inherent energy efficiency advantages on neuromorphic hardware, yet extending them to MLLMs faces two key challenges: heterogeneous modalities make uniform spike encoding insufficient, and high-resolution image inputs amplify timestep unfolding overhead. We propose SpikeMLLM, the first spike-based framework for MLLMs, which unifies existing ANN quantization methods in the spiking representation space and incorporates Modality-Specific Temporal Scales (MSTS) guided by Modality Evolution Discrepancy (MED) and Temporally Compressed LIF (TC-LIF) for timestep compression from T=L-1 to T=log2(L)-1. Experiments on four representative MLLMs across diverse multimodal benchmarks show that SpikeMLLM maintains near-lossless performance under aggressive timestep compression (Tv/Tt=3/4), with average gaps of only 0.72% and 1.19% relative to the FP16 baseline on InternVL2-8B and Qwen2VL-72B. We further develop a dedicated RTL accelerator tailored to the spike-driven datapath, observing 9.06x higher throughput and 25.8x better power efficiency relative to an FP16 GPU baseline under a deployment-oriented co-design setting, suggesting the promise of algorithm-hardware co-design for efficient multimodal intelligence.

Figures & tables

Figure 1. Cross-layer MED discrepancy between visual and text tokens in MLLMs, visualized on Qwen-VL-Chat-9.6B ( Bai et al., 2023 ) . Text tokens exhibit higher MED than visual tokens on average, motivating differentiated timestep allocation across modalities in MSTS.

Figure 2. Comparison between Integer-to-Spike LIF and TC-LIF. Integer-to-Spike LIF requires $T=L-1$ timesteps with uniform spike sequences, whereas TC-LIF reduces timesteps to $T=\log_{2}(L)-1$ via temporal weighted integration.

Figure 3. Microarchitecture of Spike-driven Dot-product PE

Method	Spiking	Time Step	OCRBench	MME	TextVQA	DocVQA	ScienceQA	Avg gap
Qwen2VL-7B-FP16	$\times$	N/A	835	2285	84.11	93.95	85.67	–
MSTS	$\checkmark$	2/3	717	1984	78.28	84.20	80.37	10.15
MSTS	$\checkmark$	2.3/3.3	765	2109	80.97	89.12	81.95	5.86
MSTS-reverse	$\checkmark$	2.3/3.3	752	2083	80.07	87.56	81.71	7.00
MSTS	$\checkmark$	3/4	823	2236	83.46	93.31	85.57	1.03
InternVL2-8B-FP16	$\times$	N/A	795	2211	77.71	91.05	97.07	–

Table 3. Analysis of MED-guided layer-wise timestep allocation on Qwen2VL-7B and InternVL2-8B. MSTS-reverse assigns higher timesteps to lower-MED layers under the same budget. Additional results are provided in Supplementary Table 13 .

Figure 4. Ablation study of MSTS: Normalized performance of SpikeMLLM(QuaRot+MSTS+TC-LIF) under different modality-specific timestep allocations

T_{v}/T_{t}

(

T\in\{3,4\}

) on Qwen2VL-7B and InternVL2-8B.

Figure 5. Ablation study of MSTS: Normalized performance of SpikeMLLM(QuaRot+MSTS+TC-LIF) under different modality-specific timestep allocations

T_{v}/T_{t}

(

T\in\{2,3\}

) on Qwen2VL-7B and InternVL2-8B.

Figure 6. Ablation study of TC-LIF: Normalized performance of SpikeMLLM(QuaRot +MSTS+TC-LIF) as timestep

T_{v}/T_{t}

increases on Qwen2VL-7B and InternVL2-8B.

Panel of Figure 7 — Figure 7. Firing rate distribution across layers for SpikeMLLM(QuaRot) ( $T=15$ ) and SpikeMLLM(QuaRot+MSTS+TC-LIF) ( $T_{v}/T_{t}=3/4$ ) on Qwen2VL-7B.

Figure 8. Overall Architecture of Co-design Accelerator

Panel of Figure 9 — Figure 9. Per-layer firing rate distribution of different methods on Qwen2VL-7B.

Figure 11. Example responses generated by SpikeMLLM(QuaRot+MSTS+TC-LIF, $T_{v}/T_{t}=3/4$ ) and the FP16 baseline on representative multimodal inputs. The responses remain highly consistent under aggressive timestep compression.

Efficient Multimodal Inference

Efficient multimodal inference reduces the compute, memory, latency, and energy needed to process image, video, audio, text, and sensor inputs.

See all 227 papers

QuantaSpike: Short-Window Spike-Driven Quantization for Large Language Models
Sep 28, 2026Bang Hu, Guowei Zhu, Changze Lv +4LLM Quantization Spiking Neural Networks
School of Computer Science, Fudan University, Shanghai, China
Spike-driven Large Language Model
Apr 11, 2026Han Xu, Xuerui Qiu, Baiyu Chen +7Spiking Language Models Spiking Neural Networks
Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Brain-Inspired Models Group, Beijing Academy of Artificial Intelligence, Beijing 100862, China +7
Large Language Models with At Most One Spike per Neuron
Sep 4, 2026Zhuoya Zhao, Parsa Omidi, Aref Jafari +1Spiking Language Models Spiking Neural Networks

Sep 28, 2026J/K move · Enter open · S save

QuantaSpike: Short-Window Spike-Driven Quantization for Large Language Models

Bang Hu, Guowei Zhu, Changze Lv, Xiaoqing Zheng+3

School of Computer Science, Fudan University, Shanghai, China

LLM Quantization Spiking Neural Networks Efficient Language Model Inference Neural Network Quantization Spiking Language Models Spike-Based Signal Encoding

Open paper AskPDF arXiv

Large language models (LLMs) achieve strong performance across many tasks but rely on dense multiply-accumulate (MAC) operations during inference, resulting in high energy cost. Spiking neural networks (SNNs) offer an event-driven alternative in which synaptic integration uses lightweight accumulation. However, spike-driven LLM inference remains difficult because outlier-heavy activations typically require long firing windows or auxiliary non-spiking paths. We propose QuantaSpike, a short-window spike-driven quantization framework for LLMs built around Logarithmic Ternary Integrate-and-Fire (LTIF) neurons. LTIF uses ternary events with power-of-two membrane-response quanta, improving the information represented by each firing step while retaining shift-ACC-compatible computation. QuantaSpike combines this neuron with group-adaptive gain and selective outlier admission: normal values use residual LTIF steps, whereas admitted outliers receive one additional onset spike before entering the same residual dynamics. Across OPT and Llama-2, QuantaSpike achieves state-of-the-art or competitive perplexity and zero-shot accuracy among spike-driven LLM quantization methods. It also transfers to newer dense LLMs, remaining close to the FP16 reference on Llama-3-8B and Qwen3-8B under the same four-step firing window. Analytical linear-energy projections show that QuantaSpike reduces the energy of one linear transformation by about

80.0\%

on OPT models and

67.1\%

on Llama-2 models relative to SpikeQuant, providing an accurate and energy-efficient spike-driven path for LLM inference.

Abstract

Efficient Multimodal Inference

Explore similar work

QuantaSpike: Short-Window Spike-Driven Quantization for Large Language Models

Spike-driven Large Language Model

Large Language Models with At Most One Spike per Neuron