Search Multilayer Perceptron-Based Fusion for Efficient and Accurate Siamese Tracking

Tianqi Shen; Huakao Lin; Ning An

cs.CVMar 2, 2026

Search Multilayer Perceptron-Based Fusion for Efficient and Accurate Siamese Tracking

Authors: Tianqi Shen, Huakao Lin, Ning An

Organizations: Institute of Mining Artificial Intelligence, Chinese Institute of Coal Science Department of Computer Science, City University of Hong Kong Beijing 100013, China / Hong Kong SAR · Image Processing Center, School of Astronautics, Beihang University Beijing 100191, China · Institute of Mining Artificial Intelligence, Chinese Institute of Coal Science State Key Laboratory of Intelligent Coal Mining and Strata Control Beijing 100013, China

Abstract

Siamese visual trackers have recently advanced through increasingly sophisticated fusion mechanisms built on convolutional or Transformer architectures. However, both struggle to deliver pixel-level interactions efficiently on resource-constrained hardware, leading to a persistent accuracy-efficiency imbalance. Motivated by this limitation, we redesign the Siamese neck with a simple yet effective Multilayer Perception (MLP)-based fusion module that enables pixel-level interaction with minimal structural overhead. Nevertheless, naively stacking MLP blocks introduces a new challenge: computational cost can scale quadratically with channel width. To overcome this, we construct a hierarchical search space of carefully designed MLP modules and introduce a customized relaxation strategy that enables differentiable neural architecture search (DNAS) to decouple channel-width optimization from other architectural choices. This targeted decoupling automatically balances channel width and depth, yielding a low-complexity architecture. The resulting tracker achieves state-of-the-art accuracy-efficiency trade-offs. It ranks among the top performers on four general-purpose and three aerial tracking benchmarks, while maintaining real-time performance on both resource-constrained Graphics Processing Units (GPUs) and Neural Processing Units (NPUs).

Figures & tables

Figure 1: Comparison between the conventional pipeline and the proposed pipeline. In subfigure (

\boldsymbol{a}

), the template feature

\boldsymbol{F}^{t}

and search feature

\boldsymbol{F}^{s}

are processed using manually designed module(s). Alternatively, (

\boldsymbol{b}

) integrates the fusion architecture into a searchable space, enabling the discovery of a lightweight yet effective neck design.

Figure 2: ( $\boldsymbol{a}$ ) CNN-based fusion uses template feature maps $\boldsymbol{F}^{t}$ as kernels sliding over search feature maps $\boldsymbol{F}^{s}$ , performing patch-level fusion efficiently on hardware. ( $\boldsymbol{b}$ ) Transformer-based fusion applies cross-attention between template tokens $\boldsymbol{F}^{t}$ and search tokens $\boldsymbol{F}^{s}$ , achieving pixel-level fusion but with slower hardware execution. ( $\boldsymbol{c}$ ) Our MLP-based fusion first performs coarse fusion on $\boldsymbol{F}^{t}$ and $\boldsymbol{F}^{s}$ , followed by refinement using Wave-MLP blocks, resulting in hardware-efficient pixel-level fusion. ( $\boldsymbol{d}$ ) The MLP-based approach yields a better accuracy–speed trade-off than its CNN and Transformer counterparts.

Figure 3: Overview of our proposed tracker and MCAS. i. Versatile architecture with two backbones: [ 16 ] for resource-constrained GPU platforms and [ 38 ] for resource-limited NPU deployment. ii. The primary MLP-based components in the MCAS include the Coarse Fusion MLP (CFM) Module and the Refine Fusion MLP (RFM) Module. CFM performs coarse fusion to generate the heatmap, while RFM refines it to produce the response map. iii. Tracking-head network transforms the response map into classification and regression response maps, representing the predictive classification and bounding box of the target.

Figure 4: Structure of the harmonization block

H_{i}

and its basic layer

C_{i1}

. Top: Basic layers are categorized into CH layers (

C_{i1},\cdots,C_{ik},\cdots,C_{ii}

with

1\leq k\leq i

) and PH layers (

P_{i1},P_{i2},P_{i3}

). Bottom:

C_{i1}

consists of 12 Wave-MLP blocks with different configurations. "wavemlp_k5_t4_dw" refers to a Wave-MLP block where the PATM module uses a

5{\times}5

convolution kernel ("k5"), the MLP expansion ratio is 4 ("t4"), and depthwise separable convolution is applied ("dw"). Each CH or PH layer is a basic layer.

$H_{1}$	CH	$C_{11}(256,256)$	-	-
$H_{1}$	PH	$P_{11}(256,256)$	$P_{12}(256,256)$	$P_{13}(256,256)$
$H_{2}$	CH	$C_{22}(256,320)$	$C_{21}(256,320)$	-
$H_{2}$	PH	$P_{21}(320,320)$	$P_{22}(320,320)$	$P_{23}(320,320)$
$H_{3}$	CH	$C_{33}(256,384)$	$C_{32}(256,384)$	$C_{31}(320,384)$
$H_{3}$	PH	$P_{31}(384,384)$	$P_{32}(384,384)$	$P_{33}(384,384)$

Table 1: CH and PH Configurations of Each Harmonization Block

Type	Expansion Ratio	Kernel Size	Depthwise
k3_t4	4	3	✗
k3_t4_dw	4	3	✓
k3_t8	8	3	✗
k3_t8_dw	8	3	✓
k5_t4	4	5	✗
k5_t4_dw	4	5	✓

Table 2: Configurations of Different Wave-MLP Variants

Routing Path	Number of Architectures
$0\to 1$	$12\times 13\times 13\times 13=26,364$
$0\to 1\to 2$	$(12\times 13\times 13\times 13)^{2}=695,060,496$
$0\to 2$	$12\times 13\times 13\times 13=26,364$
$0\to 1\to 2\to 3$	$(12\times 13\times 13\times 13)^{3}=18,324,574,916,544$
$0\to 1\to 3$	$(12\times 13\times 13\times 13)^{2}=695,060,496$
$0\to 2\to 3$	$(12\times 13\times 13\times 13)^{2}=695,060,496$

Table 3: Number of All Possible Architectures

				VOT2019					FPS ↑	Latency ↓ (ms)
Orientation	Trackers	Year	Type	EAO↑	Robustness↓	OTB 2015	GOT10K TEST	NFS 30	FPS ↑	Latency ↓ (ms)
	ECO	2017	-	-	-	64.3	-	46.6	-	-
	LightTrack(M)	2021	cnn	3 3.3	3 2.1	65.8	61.1	56.3	50.20	19.9
	FEAR(XS)	2022	cnn	25.2	56.2	66.7	6 1.9	58.3	1 23.33	8 .1
	E.T.Track	2023	trans	24.1	57.2	6 7.8	58.3	5 9.0	48.15	20.8
GPU-constrained platforms	SEAT_LT(ours)	2023	mlp	3 4.0	3 1.6	6 9.7	6 3.6	6 1.1	8 0.54	1 2.4

Table 4: Comparison of Trackers for Lightweight GPU Use on OTB2015, GOT10KTEST and VOT2019

Figure 5: Comparative analysis of tracker attributes on VOT2019. The radar chart annotates the minimum and maximum EAO scores per attribute. SEAT_LT attains consistently high EAO scores across attributes—except for illumination change—and surpasses other lightweight GPU-oriented trackers, with particularly strong gains under occlusion.

Figure 6: The comprehensive assessment of trackers tailored for on-board deployment encompasses three renowned aerial tracking benchmarks.

	MACs	Params	FPS	Average Latency	Subgraphs	DDR Latency	UAV20L Accuracy
SEAT_AL( ours )	8.13G	11.47M	20.41	195.47ms	13	892.43ms	0.593
SiamAPN++ [ 5 ]	10.25G $\uparrow$	15.15M $\uparrow$	23.57 $\uparrow$	168.64ms $\downarrow$	8 $\downarrow$	624.77ms $\downarrow$	0.533 $\downarrow$
HiFT [ 36 ]	4.83G $\downarrow$	9.98M $\downarrow$	4.30 $\downarrow$	921.41ms $\uparrow$	28 $\uparrow$	681.40ms $\downarrow$	0.566 $\downarrow$
TCTrack [ 37 ]	3.37G $\downarrow$	7.09M $\downarrow$	0.71 $\downarrow$	5608.77ms $\uparrow$	52 $\uparrow$	3461.16ms $\uparrow$	0.516 $\downarrow$

Table 5: COMPARISON of MODEL DEPLOYMENT on HORIZON X3 CHIP among DIFFERENT ON-BOARD TRACKERS

Figure 7: In real-world scenarios involving bus (left) and pedestrian (right), SEAT_AL successfully addresses challenges related to similar objects and camera motion, maintaining robust tracking without any target loss. In contrast, SiamAPN++ [ 5 ] struggles, losing the target in both cases.

					SEAT_LT
Ablation Study	Pre- Fusion	Fusion -Type	Post- Fusion	Fusion -Type	GOT10K (AO)%	OTB2015 (AUC)%	VOT2019 (EAO)%	FPS	Latency (ms)
Ab1	cnn	patch	mlp	pixel	52.2	62.8	22.3	69.63	14.4
Ab2	transformer	pixel	mlp	pixel	62.6	65.7	30.8	61.23	16.3
Ab0	mlp	pixel	-	-	63.5	68.1	33.7	89.39	11.2
Ab3	mlp	pixel	cnn	patch	63.1	69.6	31.5	83.82	11.9
Ab4	mlp	pixel	transformer	pixel	6 3.8	69.6	33.6	77.99	12.8

Table 6: ABLATION ANALYSIS of THE PROPOSED MODULES on GOT10K, OTB2015 and VOT2019

Figure 8: Visualization of ablation study. We adopt EigenCAM [ 60 ] to visualize of feature maps outputted by collector layer in ablation experiments of SEAT_LT. Original images are chosen from GOT10KTEST, OTB2015 and VOT2019. Our tracker yields feature maps whose attention gathers at target region better.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Module	Block Type	FLOPs Expression
CFM	Convolution-based	$H_{s}W_{s}\cdot H_{t}W_{t}\cdot C\cdot C^{\prime}$
	Cross-attention-based	$2H_{s}W_{s}\cdot H_{t}W_{t}\cdot C+H_{s}W_{s}\cdot C\cdot C^{\prime}$
	CFM (Ours)	$H_{s}W_{s}\cdot H_{t}W_{t}\cdot(C+C^{\prime})+H_{t}W_{t}\cdot C$
RFM	Standard CNN (3 $\times$ 3)	$9\cdot H_{z}W_{z}\cdot C\cdot C^{\prime}$
	Depthwise CNN	$H_{z}W_{z}\cdot(9C+C\cdot C^{\prime})$
	Transformer (4 heads, w/o FFN)	$4\cdot(H_{z}W_{z})^{2}\cdot C+4\cdot H_{z}W_{z}\cdot C\cdot C^{\prime}$

Appendix

Table 7: FLOPs comparison of different fusion blocks used in CFM and RFM

Figure 9: Performance comparison of trackers in terms of frames per second (FPS) and Success Rate. The visualizations represent computational complexity (FLOPs) by color intensity and model size by circle size. The green line marks the real-time FPS threshold.

Figure 10: Qualitative Attribute-based comparisons of trackers for NPU deployment on UAV20L. The top five trackers are displayed in a bar chart, illustrating their success scores across 12 different attributes, with each bar representing a specific attribute.

Figure 11: Visualization of the MCAS-relaxed supernet and derived architectures for resource-constrained targets (SEAT_LT for GPU, SEAT_AL for NPU).

Figure 12: Comparison of the search processes between differentiable neural architecture search (DNAS) and evolutionary neural architecture search (ENAS). The x-axis denotes the number of search iterations, while the y-axis represents the Intersection over Union (IOU) metric on the validation set (left) and the total chained latency cost (right).

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 175 papers

SpikeTrack: A Spike-driven Framework for Efficient Visual Tracking
Feb 27, 2026Qiuyang Zhang, Jiujun Cheng, Qichao Mao +5Visual Object Tracking Spiking Neural Networks
Tongji University · Nova University of Lisbon · Stockholm University +2
SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker
Apr 14, 2026Junbin Su, Ziteng Xue, Shihui Zhang +3Multimodal Visual Object Tracking Efficient Multimodal Inference
School of Artificial Intelligence (School of Software), Yanshan University · School of Software, Beihang University · Hebei Key Laboratory of Computer Virtual Technology and System Integration +3
UTPTrack: Towards Simple and Unified Token Pruning for Visual Tracking
Feb 27, 2026Hao Wu, Xudong Wang, Jialiang Zhang +5Visual Object Tracking Visual Token Pruning
Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · Shanghai Jiao Tong University +2

Feb 27, 2026J/K move · Enter open · S save

SpikeTrack: A Spike-driven Framework for Efficient Visual Tracking

Qiuyang Zhang, Jiujun Cheng, Qichao Mao, Cong Liu+4

Tongji University · Nova University of Lisbon · Stockholm University+2

Visual Object Tracking Spiking Neural Networks

Open paper AskPDF arXiv

Spiking Neural Networks (SNNs) promise energy-efficient vision, but applying them to RGB visual tracking remains difficult: Existing SNN tracking frameworks either do not fully align with spike-driven computation or do not fully leverage neurons' spatiotemporal dynamics, leading to a trade-off between efficiency and accuracy. To address this, we introduce SpikeTrack, a spike-driven framework for energy-efficient RGB object tracking. SpikeTrack employs a novel asymmetric design that uses asymmetric timestep expansion and unidirectional information flow, harnessing spatiotemporal dynamics while cutting computation. To ensure effective unidirectional information transfer between branches, we design a memory-retrieval module inspired by neural inference mechanisms. This module recurrently queries a compact memory initialized by the template to retrieve target cues and sharpen target perception over time. Extensive experiments demonstrate that SpikeTrack achieves the state-of-the-art among SNN-based trackers and remains competitive with advanced ANN trackers. Notably, it surpasses TransT on LaSOT dataset while consuming only 1/26 of its energy. To our knowledge, SpikeTrack is the first spike-driven framework to make RGB tracking both accurate and energy efficient. The code and models are available at https://github.com/faicaiwawa/SpikeTrack.

Abstract

Visual Object Tracking

Explore similar work

SpikeTrack: A Spike-driven Framework for Efficient Visual Tracking

SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker

UTPTrack: Towards Simple and Unified Token Pruning for Visual Tracking