Brain-Inspired Multimodal Spiking Neural Network for Image-Text Retrieval

Xintao Zong; Xian Zhong; Wenxuan Liu; Jianhao Ding; Zhaofei Yu; Tiejun Huang

cs.CVMar 25, 2026

Brain-Inspired Multimodal Spiking Neural Network for Image-Text Retrieval

Authors: Xintao Zong, Xian Zhong, Wenxuan Liu, Jianhao Ding, Zhaofei Yu, Tiejun Huang

Organizations: Hubei Key Laboratory of Transportation Internet of Things, Wuhan University of Technology · State Key Laboratory for Multimedia Information Processing, Peking University

Abstract

Spiking neural networks (SNNs) have recently shown strong potential in unimodal visual and textual tasks, yet building a directly trained, low-energy, and high-performance SNN for multimodal applications such as image-text retrieval (ITR) remains highly challenging. Existing artificial neural network (ANN)-based methods often pursue richer unimodal semantics using deeper and more complex architectures, while overlooking cross-modal interaction, retrieval latency, and energy efficiency. To address these limitations, we present a brain-inspired Cross-Modal Spike Fusion network (CMSF) and apply it to ITR for the first time. The proposed spike fusion mechanism integrates unimodal features at the spike level, generating enhanced multimodal representations that act as soft supervisory signals to refine unimodal spike embeddings, effectively mitigating semantic loss within CMSF. Despite requiring only two time steps, CMSF achieves top-tier retrieval accuracy, surpassing state-of-the-art ANN counterparts while maintaining exceptionally low energy consumption and high retrieval speed. This work marks a significant step toward multimodal SNNs, offering a brain-inspired framework that unifies temporal dynamics with cross-modal alignment and provides new insights for future spiking-based multimodal research. The code is available at https://github.com/zxt6174/CMSF.

Figures & tables

Figure 1: Brain-Inspired Multimodal SNN. Sensory-specific cortical regions process unimodal information, while higher-order areas such as the audiovisual (AV) cortex integrate multimodal semantics. This hierarchical mechanism inspires our spike-level fusion strategy for constructing a multimodal SNN framework.

Figure 2: Brain-Inspired Multimodal Spiking Neural Network for Image-Text Retrieval. (a) Overview of CMSF. (b) Working principle of the Spike Generator. (c,d) Details of Intra-modal Attention blocks. (e) One implementation of the Spike Fusion module. Pre-extracted region and word features are converted into spikes for unimodal semantic modeling within a sparse embedding space. The Spike Fusion module performs spike-level cross-modal interaction, generating enhanced embeddings as soft labels to guide unimodal encoders. This stage is excluded during inference. Bidirectional Hard-Alignment computes the fine-grained similarity matrix, while Early-Alignment and Soft-Label Alignment optimize unimodal encoders from both input and output perspectives. Spike Comb Cross Attention (SCCA) is detailed in § 3.5 , with additional fusion variants in the Supplementary Material A.

Figure 3: Illustration of Our Bidirectional Hard Alignment. It identifies fine-grained region-word hard matches and integrates them to compute the overall image-text similarity.

Figure 4: Details of Our Spike Comb Cross Attention Structure. With the mask operation, it achieves lower time complexity and reduced energy consumption.

Figure 5: Illustration of Spike Fusion Soft-Label Alignment Strategy. The left part represents the similarity computation involved in both training and inference; the right part is training-only.

Methods	MSCOCO 1K Test Set							Flickr30K 1K Test Set
	Image-to-Text			Text-to-Image			R@Sum	Image-to-Text			Text-to-Image			R@Sum
	R@1	R@5	R@10	R@1	R@5	R@10	R@Sum	R@1	R@5	R@10	R@1	R@5	R@10	R@Sum
Faster R-CNN + BiGRU
SCAN $\ast$ [ 18 ]	72.7	94.8	98.4	58.8	88.4	94.8	507.9	67.4	90.3	95.8	48.6	77.7	85.2	465.0
VSRN $\ast$ [ 21 ]	76.2	94.8	98.2	62.8	89.7	95.1	516.8	71.3	90.6	96.0	54.7	81.8	88.2	482.6
VSE $\infty$ [ 4 ]	78.5	96.0	98.7	61.7	90.3	95.6	520.8	76.5	94.2	97.7	56.4	83.4	89.9	498.1

Table 1: Results on MSCOCO 1K and Flickr30K 1K Test Sets. Best results are in bold , and the second-best scores are underlined .

\ast

indicates ensemble results. Additional results on pretrained VLMs ( e.g. , CLIP [ 35 ] ) are reported in Supplementary Material B.

Methods	MSCOCO 5K Test Set
	Image-to-Text			Text-to-Image			R@Sum
	R@1	R@5	R@10	R@1	R@5	R@10	R@Sum
Fast R-CNN + BiGRU
SCAN $\ast$ [ 18 ]	50.4	82.2	90.0	38.6	69.3	80.4	410.9
VSRN $\ast$ [ 21 ]	53.0	81.1	89.4	40.5	70.6	81.1	415.7
VSE $\infty$ [ 4 ]	56.6	83.6	91.4	39.3	69.9	81.1	421.9

Table 2: Results on MSCOCO 5K Test Set. Best results are in bold , and the second-best scores are underlined .

Figure 6: Computation profile of CMSF’s encoding stage (left) and inference speed comparison with ANN models (right). AC: accumulate, MAC: multiply-and-accumulate.

\triangle

\square

, and

\lozenge

represent results on MSCOCO 1K, Flickr30K , and MSCOCO 5K, respectively. The vertical axis shows R@Sum and the horizontal axis shows retrieval time (closer to the upper-left corner is better).

Methods	Architecture	Param (M)	Ops (M)	Energy (mJ)	R@Sum
VSE $\infty$ [ 4 ]	RNN	113.97	162.87	0.749	513.5
USER [ 48 ]	RNN	122.37	318.05	1.463	519.9
HREM [ 9 ]	Vanilla-Attn	126.55	621.50	2.859	520.4
CMSF (Ours)	Spike-Attn	179.30	265.04	0.626	523.9

Table 3: Theoretical inference energy comparison. Ops denote synaptic operations for SNNs and floating-point operations for ANNs. Best results are in bold.

Alignment Method	Image-to-Text			Text-to-Image			R@Sum
Alignment Method	R@1	R@5	R@10	R@1	R@5	R@10	R@Sum
LSE	75.9	93.6	96.7	59.5	84.5	90.4	500.6
VHA	77.2	95.4	98.1	61.4	86.9	92.3	511.3
THA	76.3	94.8	96.7	61.0	86.2	91.8	506.9
BiHA (Ours)	82.1	96.3	98.0	65.9	88.4	93.2	523.9

Table 4: Effect of alignment strategies on Flickr30K 1K test set. Best results are in bold.

Early-Align	Spike Fusion	R@1 (I2T)	R@1 (T2I)	R@Sum
○	○	76.5	63.0	511.6
○	●	78.5	63.3	514.2
●	○	79.7	64.1	517.0
●	●	82.1	65.9	523.9

Table 5: Ablation of Early Alignment and Spike Fusion Soft Alignment on Flickr30K 1K Test Set. Best results are in bold.

Figure 7: Effect of Different Time Steps. CMSF performance across various time steps on Flickr30K and MSCOCO during training.

Figure 8: Detailed structure of our Spike Cross Attention. Embeddings from both modalities are fused at the spike level via cross-attention matrix multiplication.

Method	Time Complexity	Space Complexity
SCA	$O(ND^{2})$	$O(D^{2}+ND)$
SCSA	$O((N+L)D^{2})$	$O(D^{2}+(N+L)D)$
SCCA	$O(N)$	$O(D)$

Table 6: Time and space complexities of SCA, SCSA, and SCCA.

N

(regions) and

L

(words) are of similar magnitude.

Methods	Image-to-Text			Text-to-Image			R@Sum
Methods	R@1	R@5	R@10	R@1	R@5	R@10	R@Sum
SCA	81.3	95.8	98.5	64.9	88.3	93.4	522.1
SCSA	81.0	96.4	98.1	64.9	88.4	93.2	522.0
SCCA	82.1	96.3	98.0	65.9	88.4	93.2	523.9

Table 7: Results of different fusion methods on Flickr30K 1K test set. R@K denotes Recall@K, and R@Sum is the sum of R@1, R@5, and R@10 for both retrieval directions.

Methods	Structure	Image-to-Text			Text-to-Image			R@Sum
Methods	Structure	R@1	R@5	R@10	R@1	R@5	R@10	R@Sum
Flickr30K 1K Test Set
CLIP	ViT-B/32	78.7	95.4	98.0	66.3	88.6	93.1	520.0
CLIP	ViT-L/14	87.3	99.0	99.5	76.4	94.8	97.4	554.5
CMSF	frcnn+bigru	80.7	95.0	97.6	61.3	85.9	91.3	511.8
CMSF	frcnn+bert	82.1	96.3	98.0	65.9	88.4	93.2	523.9

Table 8: Results Compared to Pre-trained Methods. frcnn means FasterRCNN [ 36 ] .

Neural Layer	Theoretical Consumption
Region Linear	$E_{\mathrm{AC}}\cdot T\cdot R_{r}\cdot FL_{r}$
Word Linear	$E_{\mathrm{AC}}\cdot T\cdot R_{w}\cdot FL_{w}$
Q, K, V	$E_{\mathrm{AC}}\cdot T\cdot R_{0}\cdot 3ND^{2}$
SCA	$E_{\mathrm{AC}}\cdot T_{1}\cdot R_{1}\cdot ND^{2}$
SCSA	$E_{\mathrm{AC}}\cdot T_{2}\cdot R_{2}\cdot(N+L)D^{2}$
SCCA	$E_{\mathrm{AC}}\cdot T_{3}\cdot R_{3}\cdot N$

Table 9: Detailed calculation formulas. The theoretical energy consumption of each CMSF layer.

Spike Generator	Image-to-Text			Text-to-Image			R@Sum
Spike Generator	R@1	R@5	R@10	R@1	R@5	R@10	R@Sum
Conv-BN	77.9	95.8	98.6	60.8	85.8	91.5	510.4
Delta-BN	76.1	95.4	97.8	58.5	84.0	90.2	502.0
Linear-BN	80.3	96.5	98.2	64.8	87.5	92.7	520.0
Linear-LN	81.3	96.1	97.7	64.4	88.6	93.2	521.3
Repeat-BN	80.2	95.9	97.5	64.4	88.2	93.4	519.6

Table 10: Ablation of Spike Generator designs on Flickr30K 1K. R@K denotes Recall@K for image-to-text (first three columns) and text-to-image (next three columns); R@Sum is the total.

Head Num	Image-to-Text			Text-to-Image			R@Sum
Head Num	R@1	R@5	R@10	R@1	R@5	R@10	R@Sum
$h=2$	79.4	96.5	97.7	63.9	88.0	93.0	518.6
$h=3$	80.5	96.1	98.0	64.4	88.5	93.3	520.8
$h=4$	80.2	96.3	98.2	64.9	88.4	93.2	521.2
$h=6$	82.1	96.3	98.0	65.9	88.4	93.2	523.9
$h=9$	80.2	95.9	98.0	64.3	88.3	92.9	519.5

Table 11: Ablation of comb head numbers on Flickr30K 1K. R@K denotes Recall@K for image-to-text (columns 2-4) and text-to-image (columns 5-7); R@Sum is the total across all six metrics.

Figure 9: Visualization of retrieval results. Top: image-to-text retrieval examples. Bottom: text-to-image retrieval examples by CMSF on Flickr30K .

Image-Text Retrieval

Image-text retrieval matches images with natural-language queries, enabling search across visual collections using either modality.

See all 38 papers

MSGAT: Multi-Head Spiking Graph Attention with Similarity-Space Fusion for Image-Text Retrieval
Oct 8, 2026Xintao Zong, Wenxuan Liu, Jianhao Ding +2Image-Text Retrieval Cross-Modal Retrieval
Peking University
SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks
Aug 3, 2026Xiubo Liang, Jinxing Han, Yuke Li +3Spiking Transformers Spiking Neural Networks
School of Software Technology, Zhejiang University, Ningbo, China · NetEase Yidun AI Lab, Hangzhou, China
Current Injection Spiking Neural Network for Infrared and Visible Image Fusion
Jul 22, 2026Rui Zhao, Zhuoyuan Li, Wenrui Li +4Efficient Neural Network Inference Multi-Source Image Fusion
College of Computing and Data Science, Nanyang Technological University, Singapore · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong · Department of Computer Science and Technology, Harbin Institute of Technology, Harbin, China +2

Oct 8, 2026J/K move · Enter open · S save

MSGAT: Multi-Head Spiking Graph Attention with Similarity-Space Fusion for Image-Text Retrieval

Xintao Zong, Wenxuan Liu, Jianhao Ding, Zhaofei Yu+1

Peking University

Image-Text Retrieval Cross-Modal Retrieval Spiking Neural Networks Multimodal Fusion Graph Attention Networks

Open paper AskPDF arXiv

Spiking neural networks (SNNs) offer an energy-efficient computing paradigm through sparse event-driven computation, showing great potential for efficient multimodal learning. However, applying SNNs to high-level multimodal tasks, such as image-text retrieval (ITR), remains challenging, since sparse spike representations make it difficult to capture semantic structures required for cross-modal alignment. Existing spiking ITR methods rely on local alignment and additional soft-label supervision during training, while lacking awareness of structural and multi-granularity relationships. To address these issues, we propose a Multi-head Spiking Graph Attention Network (MSGAT) for structural modeling and equip it with dynamic attention heads to capture complementary relational patterns and enable spike-driven graph reasoning and aggregation. However, within a two-branch multi-granularity fusion framework, the fine-grained spike representations generated by MSGAT are sparse and discrete, whereas the global representations are continuous, making conventional feature-level fusion susceptible to interference across heterogeneous representations. Therefore, we introduce Sim-Fuse, a similarity-space fusion alignment strategy integrating coarse- and fine-grained matching relations while avoiding direct fusion of heterogeneous representations. Experiments on Flickr30K and MSCOCO show our method outperforms ANN methods under matched settings and existing SNN retrieval baselines. Moreover, with only two time steps, our SNN achieves comparable or superior performance to its ANN counterpart while reducing theoretical module-level energy by 55%. The code is provided in the Supplementary Materials.

Abstract

Image-Text Retrieval

Explore similar work

MSGAT: Multi-Head Spiking Graph Attention with Similarity-Space Fusion for Image-Text Retrieval

SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks

Current Injection Spiking Neural Network for Infrared and Visible Image Fusion