S$^3$E-Det: A Strict Spatial-Spectral Equivariant Detector for Multispectral Object Detection

Peng Zhang; Tingfa Xu; Jinrui Sui; Yuyang Tian; Jianan Li

cs.CVOct 20, 2026

S $^3$ E-Det: A Strict Spatial-Spectral Equivariant Detector for Multispectral Object Detection

Authors: Peng Zhang, Tingfa Xu, Jinrui Sui, Yuyang Tian, Jianan Li

Organizations: Beijing Institute of Technology, Beijing 100081, China. · Big Data and Artificial Intelligence Laboratory, Beijing Institute of Technology Chongqing Innovation Center, Chongqing 401135, China.

Abstract

Multispectral object detection requires detectors to exploit material-dependent spectral responses while maintaining consistent localization under geometric variations. Existing MOD detectors learn discriminative spectral representations, yet lack a unified mechanism that preserves spectral order, adapts spectral-context fusion across pyramid levels, and regulates geometric responses under rotation. To address these limitations, we propose S $^3$ E-Det, a strict spatial-spectral equivariant detector that couples ordered spectral modeling with transformation-consistent prediction. Spectral Order-aware Sampling Calibration (SOSC) imposes local order constraints on input-derived spectral features through first-order variation, while Common-Differential Equivariant Fusion (CDEF) adapts spectral-context aggregation across pyramid levels. A compact Geometric Alignment Head (GAH) reconciles invariant and equivariant output properties within a unified prediction stage. Across MODA, HOD3K, and DroneVehicle, S $^3$ E-Det combines high detection accuracy with a compact parameter footprint and reduced rotation-induced variation. Code is available at https://github.com/Riiluo/S3E-Det.

Figures & tables

Fig. 1: (a) Conventional spatial–spectral modeling does not preserve the ordered spectral structure and uses fixed feature fusion. (b) Ordered adaptive modeling preserves spectral order and adapts fusion across feature levels. (c) Strict rotational response consistency makes feature responses transform predictably under exact quarter-turn rotations.

Fig. 2: mAP 50 vs. Params on MODA; bubble area: FLOPs.

Fig. 3: Overall network framework of S 3 E-Det. SOSC blocks perform staged equivariant feature extraction, CDEF conducts multi-scale fusion, and GAH decodes invariant categories and aligned oriented boxes at three scales. (a) Stem maps multispectral inputs into group-indexed feature fields; (b) CDEF separates common context from differential spectral detail before an agreement-gated residual update.

Fig. 4: Structure of the SOSC block. It calibrates ordered spectral responses through neighboring-support slicing, derivative-aware interpolation, residual update, and RE-FFN.

Fig. 5: Illustration of GAH. The head assigns invariant readouts to category scores and canonical distances, while equivariant group-phase decoding produces the aligned object angle.

	Methods	Car	Bus	Van	Awn. bike	Truck	Tricycle	Bicycle	Pedestrian	mAP ${}_{50}\uparrow$	mAP ${}_{75}\uparrow$	mAP $\uparrow$	#F $\downarrow$	#P $\downarrow$
two-stage	LSKNet [ 19 ]	90.4	88.5	74.5	72.5	66.0	52.5	41.0	28.2	64.2	42.8	39.4	221.9G	32.6M
	O-RCNN [ 18 ]	90.5	89.7	74.6	72.5	66.6	54.7	45.1	30.4	65.5	44.0	40.9	244.7G	55.3M
	StripRCNN [ 20 ]	90.5	89.0	76.1	73.6	67.1	56.5	45.2	30.7	66.1	44.0	41.0	231.8G	45.2M
	ReDet [ 13 ]	90.6	89.6	76.8	72.9	69.6	52.4	41.5	41.0	66.8	44.1	41.3	245.2G	31.6M
one-stage	R3Det-KLD [ 22 ]	90.4	88.7	74.8	66.6	60.9	42.6	29.8	35.5	61.2	39.4	36.8	306.7G	39.6M
	S 2 ANet [ 21 ]	90.4	88.7	74.4	69.1	62.7	48.9	30.1	40.7	63.1	38.7	36.7	216.5G	38.8M

TABLE I: Comparison with representative methods on MODA.

\dagger

indicates methods originally designed for multispectral object detection.

# Input: pyramid features

\{F_{l}\}_{l=3}^{5}

, group basis

\{\vartheta_{g}\}

# Output: class logits

\{p_{l}\}

and aligned boxes

\{B_{l}\}

for

l\in\{3,4,5\}

P_{l}

= EquivariantStem (

F_{l}

Q_{l}

= GroupPool (

P_{l}

)

\triangleright

Invariant readout

p_{l}

= ClassReadout (

Q_{l}

d_{l}

= DistanceReadout (

Q_{l}

)

\triangleright

Equivariant phase decoding

Table 7

Fig. 6: Visualization comparison of detection results and feature maps obtained by different methods on MODA.

Model	Params (M)	FLOPs (G)	Size (MB)	Peak GPU Mem (GB)	Time (min/epoch)
YOLOv8n	3.2	12.6	5.75	14	10
S 3 E-Det ( $C_{4}$ )	2.4	35.6	5.43	20	12

TABLE II: Model size and computational resource profile.

Fig. 7: Feature responses of S 3 E-Det: (a) input image; (b)–(d) with/without SOSC, CDEF, and GAH. Yellow / orange dashed ellipses mark enhanced target/suppressed background responses, respectively.

Equiv.	SOSC	CDEF	GAH	mAP ${}_{50}\uparrow$	mAP ${}_{75}\uparrow$	mAP $\uparrow$
$\times$	$\times$	$\times$	$\times$	62.0	48.0	43.3
$\checkmark$	$\times$	$\times$	$\times$	66.8	54.1	47.7
$\checkmark$	$\checkmark$	$\times$	$\times$	69.6	55.5	50.0
$\checkmark$	$\times$	$\checkmark$	$\times$	68.7	54.9	49.4
$\checkmark$	$\times$	$\times$	$\checkmark$	68.3	55.2	49.2
$\checkmark$	$\checkmark$	$\checkmark$	$\times$	71.3	58.3	52.3

TABLE III: Ablation studies on the key components of S 3 E-Det.

Spectral kernel	Residual gate $\gamma$	Derivative gate $\eta$	mAP ${}_{50}\uparrow$	mAP $\uparrow$
1	$\checkmark$	$\checkmark$	71.3	52.6
3	$\checkmark$	$\checkmark$	72.6	54.0
5	$\checkmark$	$\checkmark$	72.4	53.8
7	$\checkmark$	$\checkmark$	72.2	53.5
3	$\times$	$\checkmark$	71.4	52.8
3	$\checkmark$	$\times$	71.0	52.3

TABLE IV: Hyperparameter ablation of SOSC on MODA.

Reduction ratio	Spatial kernel	Agreement gate	mAP ${}_{50}\uparrow$	mAP $\uparrow$
2	3	$\checkmark$	72.3	53.7
4	3	$\checkmark$	72.6	54.0
8	3	$\checkmark$	71.9	53.3
4	5	$\checkmark$	72.4	53.8
4	3	$\times$	71.6	52.9

TABLE V: Hyperparameter ablation of CDEF on MODA.

Group order	mAP ${}_{50}\uparrow$	mAP ${}_{75}\uparrow$	mAP $\uparrow$	Params $\downarrow$	FLOPs $\downarrow$
$C_{1}$	62.0	48.0	43.3	0.6M	8.5G
$C_{2}$	70.8	57.9	51.5	1.2M	17.3G
$C_{4}$	72.6	59.7	54.0	2.4M	35.6G
$C_{8}$	72.7	59.8	54.0	5.0M	73.0G

TABLE VI: Ablation of rotation group order on MODA.

Operator	Params $\downarrow$ (2,445,776)	FLOPs $\downarrow$ (35.60G)	Cumulative $\bar{e}_{C_{4}}\downarrow$	Local $\bar{c}_{C_{4}}\downarrow$
L0 Stem (group lift)	2,080	0.54G	0.000000	0.000000
L1 SOSC-1	10,307	2.59G	0.000023	0.000004
L2 Patch merging-1	2,112	0.34G	0.000034	0.000004
L3 SOSC-2	38,883	2.51G	0.000227	0.000004
L4 Patch merging-2	8,320	0.34G	0.000059	0.000001
L5 SOSC-3	151,331	2.46G	0.000509	0.000005

TABLE VII: Layer-wise cost and exact-

C_{4}

residuals (mean over three non-identity elements). Means are computed from unrounded residuals; all displayed values lie in

[0,1]

Angle	$e_{\rm int}\downarrow$	L8 backbone $\downarrow$	P3–P5 mean $\downarrow$	$e_{\rm raw}\downarrow$	Decoded OBBs $\downarrow$	Post-NMS predictions $\downarrow$
$30^{\circ}$	0.013557	0.456471	0.376044	0.348256	0.432972	0.658733
$60^{\circ}$	0.013669	0.547496	0.396743	0.359207	0.456248	0.663884
$90^{\circ}$	0.000000	0.001776	0.001008	0.000890	0.022366	0.024853
$120^{\circ}$	0.013473	0.467767	0.381041	0.336827	0.503154	0.768568
$150^{\circ}$	0.013331	0.533103	0.383906	0.342756	0.451895	0.575583
$180^{\circ}$	0.000000	0.001838	0.001013	0.000934	0.034079	0.033145

TABLE VIII: Angle-resolved implementation conformance on a

30^{\circ}

grid. Quarter turns are exact

C_{4}

elements; the remaining angles evaluate off-grid rotations.

e_{\rm raw}

averages classification, canonical DFL, and phase-angle residuals. Mean rows are computed from unrounded residuals; all displayed values lie in

[0,1]

Method	mAP ${}_{50}\uparrow$	mAP ${}_{75}\uparrow$	mAP $\uparrow$	FLOPs $\downarrow$	Params $\downarrow$
CO-DETR	77.9	57.3	51.7	200.7G	64.5M
DINO	89.1	61.6	56.3	114.7G	47.6M
Faster R-CNN	89.4	62.7	56.9	206.7G	41.1M
ATSS	91.2	63.0	57.4	110.6G	32.2M
TOOD	90.3	65.6	57.8	108.5G	32.1M
RepPoints	92.0	65.5	58.4	104.5G	36.9M

TABLE IX: Comparison with state-of-the-art detectors on HOD3K.

Method	Car	Bus	Truck	Fre.	Van	mAP ${}_{50}\uparrow$	mAP $\uparrow$
C 2 Former [ 28 ]	90.2	89.8	68.3	64.4	58.5	74.2	47.3
YOLOv8l-RGB [ 29 ]	92.5	91.7	68.8	47.8	50.2	70.2	48.6
YOLOv8l-IR [ 29 ]	93.4	91.9	69.3	53.7	51.1	71.9	49.1
SLBAF-Net [ 32 ]	90.2	89.9	76.0	68.2	59.9	76.8	49.5
CSOM-ODAF [ 33 ]	90.1	89.8	75.6	68.2	61.8	77.1	50.1
CMA [ 34 ]	95.8	93.1	75.9	59.8	59.4	76.8	50.4

TABLE X: Performance comparison on DroneVehicle.

Fig. 10: Detection result comparison on DroneVehicle. Red boxes indicate the ground truth, while green boxes denote the predictions.

Configuration	Mean (ms/image) $\downarrow$	FPS $\uparrow$	Peak Memory (MiB) $\downarrow$	Avg. Board Power (W) $\downarrow$	Energy (J/image) $\downarrow$
Rotation-group variants
$C_{1}$	28.2	35.5	82	5.7	0.16
$C_{2}$	56.0	17.9	168	8.0	0.45
$C_{4}$	89.3	11.2	312.4	9.2	0.82
$C_{8}$	181.0	5.5	595	11.2	2.03
Component variants at $C_{4}$

TABLE XI: Measured FP16 RKNN deployment costs of rotation-group and component variants.

Multispectral Object Detection

Multispectral object detection combines information from visible, infrared, and other spectral bands to locate objects in challenging scenes.

See all 43 papers

Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detection
Jul 6, 2026Peng Zhang, Tingfa Xu, Shuaihao Han +1Multispectral Object Detection Equivariant Neural Networks
Beijing Institute of Technology, Beijing, China
SGLDet: Spectral-aware Global-Local Dynamic Modeling for Multispectral Aerial Object Detection
Oct 20, 2026Peng Zhang, Shuaihao Han, Tingfa Xu +1Multispectral Object Detection Object Detection
Beijing Institute of Technology, Beijing 100081, China · Beijing Institute of Technology, Beijing 100081, China, and also with Chongqing Innovation Center, Beijing Institute of Technology, Chongqing 401135, China
Dual Sparse Aggregation Transformer for Multispectral Object Detection
Jun 30, 2026Wencong Wu, Xiuwei Zhang, Hanlin Yin +2Multispectral Object Detection Detection Transformer
School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, China · School of Computer Science, the Shaanxi Provincial Key Laboratory of Speech and Image Information Processing and the National Engineering Laboratory for Integrated Aerospace-Ground-Ocean Big Data Application Technology, Northwestern Polytechnical University, Xi’an 710072, China · School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China

Jul 6, 2026J/K move · Enter open · S save

Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detection

Peng Zhang, Tingfa Xu, Shuaihao Han, Jianan Li

Beijing Institute of Technology, Beijing, China

Multispectral Object Detection Equivariant Neural Networks Multi-Scale Feature Fusion Object Detection

Open paper AskPDF arXiv

Existing multispectral detectors are limited by discrete spectral processing, a scale-dependent shift in the relative reliability of spectral and spatial cues across pyramid levels, and the lack of explicit rotation-equivariant geometric priors for arbitrarily oriented objects. To tackle these limitations, we propose FressDet, a fully rotation-equivariant spectral-spatial learning framework for multispectral object detection, capable of capturing the continuous, ordered nature of spectral structure and enabling reliable spectral-spatial fusion across pyramid levels under arbitrary in-plane rotations. FressDet integrates three complementary components. Spectral Implicit Warp (SpeIW) enables query-based spectral resampling via a coordinate-conditioned implicit field, yielding a monotone, order-preserving warp. Rotation-Equivariant Consistency Weighting (ReCoW) adaptively fuses spectral and spatial branches based on branch reliability, reinforcing informative cues while suppressing noise across pyramid levels. The oriented-aware head exploits group-indexed features to stably predict oriented objects without parameter replication. Taken together, FressDet learns more discriminative and robust spectral-spatial representations even under rotational perturbations. By achieving state-of-the-art performance with 93% fewer parameters on three public benchmarks, FressDet demonstrates its effectiveness and generalizability.

Abstract

Multispectral Object Detection

Explore similar work

Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detection

SGLDet: Spectral-aware Global-Local Dynamic Modeling for Multispectral Aerial Object Detection

Dual Sparse Aggregation Transformer for Multispectral Object Detection