Aligned Consensus Teaching for Label-Efficient Oriented Object Detection in Weakly-Aligned Visible-Infrared Imagery

Qi Ming; Xiaxin Yuan; Jiahuan Zhou; Jiangmeng Li; Xudong Zhao; Zhanchao Huang; Juan Fang; Shaoguang Huang

cs.CVSep 16, 2026

Aligned Consensus Teaching for Label-Efficient Oriented Object Detection in Weakly-Aligned Visible-Infrared Imagery

Authors: Qi Ming, Xiaxin Yuan, Jiahuan Zhou, Jiangmeng Li, Xudong Zhao, Zhanchao Huang, Juan Fang, Shaoguang Huang,

Organizations: Beijing University of Technology · University of Science and Technology of China · Peking University · Institute of Software, Chinese Academy of Sciences ·

Abstract

Visible-infrared object detection (VIOD) detects objects with oriented bounding boxes from paired visible and infrared images. Existing methods depend on costly dual-modality annotations. Semi-supervised learning can reduce this burden, but extending it from single-modal detection to VIOD is challenging. In the practical image-pair-level setting considered here, only a few pairs are labeled in both modalities, while the rest are completely unlabeled. This limited supervision creates three challenges: (i) too few labeled boxes for robust cross-modal alignment; (ii) pseudo-label errors caused by branch-wise misses accumulate during self-training; and (iii) tail-class annotations become critically scarce as the labeling budget decreases. We propose Aligned Consensus Teacher (ACT) for label-efficient VIOD in this setting. Its Cycle-Consistent Region Alignment (CRA) combines cycle consistency and sparse anchors with reliability-weighted regional matching. Cross-Modal Consensus Mean-Teacher (CMC-MT) forms consensus pseudo labels under pair-preserving views to recover branch-wise misses and supervise unlabeled pairs. Text-Guided Cross-Modal Instance Augmentation (TG-CMIA) uses a vision-language scene prior to compose tail-class instance pairs while preserving RGB--IR offsets. To the best of our knowledge, ACT is the first framework to study semi-supervised VIOD under this image-pair-level setting. Experiments on DroneVehicle and VEDAI show consistent gains across annotation ratios. With 10% labeled pairs on DroneVehicle, ACT reaches 94.3% of the mAP obtained by the same detector under full supervision. Code and models will be available on GitHub to facilitate future work.

Figures & tables

Figure 1: Motivation on DroneVehicle. (a) A $1\%$ labeling budget provides about $90\times$ fewer matched boxes for RGB–IR alignment. (b) A branch-wise miss yields incomplete pseudo labels whose errors propagate through EMA self-training. (c) ACT recovers class-wise AP toward the full-supervision reference at the $1\%$ budget.

Figure 2: Training and inference overview of ACT. CRA learns regional correspondence from labeled pairs, CMC-MT converts unlabeled pairs into consensus supervision, and TG-CMIA composes tail-class RGB–IR instance pairs on the labeled branch. Only the fused EMA teacher is retained for inference.

Table 3

Figure 3: Quantitative analysis on DroneVehicle. (a) Label efficiency across annotation ratios. (b) Teacher pseudo-label quality across RGB, IR, and fused branches. (c) Class-wise effects of TG-CMIA under the two lowest labeling budgets.

Method	car	truck	bus	van	freight	mAP
C 2 Former [ 41 ]	0.896	0.442	0.872	0.340	0.326	0.575
DMM [ 47 ]	0.900	0.555	0.888	0.449	0.521	0.663
SM3Det [ 18 ]	0.897	0.474	0.881	0.360	0.434	0.609
M2D-LIF [ 45 ]	0.889	0.508	0.778	0.412	0.423	0.602
MT [ 31 ]	0.890	0.492	0.880	0.364	0.435	0.612
SOOD [ 11 ]	0.897	0.462	0.862	0.367	0.383	0.594

Table 3: Per-category AP on DroneVehicle at the

10\%

annotation ratio.

Components			Annotation ratio
CMC-MT	TG-CMIA	CRA	1%	5%	10%	100%
			0.299	0.485	0.575	0.724
✓			0.489	0.619	0.665	–
✓	✓		0.510	0.639	0.679	–
✓	✓	✓	0.511	0.644	0.683	–

Table 4: Component ablation on DroneVehicle.

Figure 4: Qualitative results on DroneVehicle. Rows show day and night scenes; columns show RGB input, baseline predictions (orange), and ACT predictions (green). Red dashed boxes mark targets recovered by ACT.

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 629 papers

Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration
Aug 11, 2026Qi Ming, Yuyang Wang, Mingjing Zhao +7Multispectral Object Detection Cross-Modal Alignment
Beijing University of Technology, China · Central South University, China · Beijing Electronics Science & Technology Institute +4
Residual Optimal Transport-Based Experts Collaboration Towards Modality-Aware Infrared-Visible Object Detection
Sep 3, 2026Yue Zhao, Hua Yu, Yukun Zhao +6Missing-Modality Learning Multimodal Robustness
School of Electronic Engineering, Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University, Xi’an 710071, China · College of Computing and Data Science, Nanyang Technological University, Singapore · School of Electronics and Communication Engineering, Sun Yat-sen University, Guangzhou 510275, China +3
Not All Patches are Equal: Sampling Matters for Visible-Infrared Pre-Training
Jul 22, 2026Qiwei Ma, Bin Deng, Junjie Zhu +5Visual Representation Learning Self-Supervised Pre-Training
School of Artificial Intelligence and Robotics, Hunan University, Changsha, 410082, Hunan, China · Intelligent Game and Decision Lab, Beijing, 100166, Beijing, China · Yuelushan Center for Industrial Innovation, Changsha, 410082, Hunan, China

Aug 11, 2026J/K move · Enter open · S save

Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration

Qi Ming, Yuyang Wang, Mingjing Zhao, Yifan Xiao+6

Beijing University of Technology, China · Central South University, China · Beijing Electronics Science & Technology Institute+4

Multispectral Object Detection Cross-Modal Alignment Infrared Object Detection Cross-Modal Image Registration Multimodal Sensor Fusion

Open paper AskPDF arXiv

Visible-infrared object detection relies on complementary RGB and thermal cues, but its performance is often degraded by cross-modal spatial misalignment. Most existing methods rely on implicit feature adaptation to handle weakly misaligned scenarios, while large-offset geometric discrepancies remain insufficiently addressed. In this paper, we propose a Joint Feature-domain Registration and Detection network (JFRDet), an end-to-end visible-infrared oriented object detector tailored for severely cross-modal geometric discrepancies. JFRDet introduces a Cross-Modal Affine Alignment (CMAA) module to estimate an image-level affine transformation for explicit multi-level feature alignment. Note that illumination changes directly affect the reliability of RGB cues, an Illumination-Guided Complementary Fusion (IGCF) module adaptively exploits modality reliability under varying illumination conditions for cross-modal fusion. Then, an Alignment Quality-Consistency Gating (AQCG) strategy stabilizes joint optimization by modulating detection supervision according to alignment reliability and gradient consistency. We further construct DroneVehicle Misaligned (DVMA), a benchmark for evaluating visible-infrared oriented object detection under severe cross-modal geometric misalignment. The proposed JFRDet achieves 69.7%

\mathrm{mAP}_{50}

on DVMA, which represents state-of-the-art (SOTA) performance. The code and dataset will be available on GitHub.

Abstract

Object Detection

Explore similar work

Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration

Residual Optimal Transport-Based Experts Collaboration Towards Modality-Aware Infrared-Visible Object Detection

Not All Patches are Equal: Sampling Matters for Visible-Infrared Pre-Training