cs.CVSep 16, 2026

Aligned Consensus Teaching for Label-Efficient Oriented Object Detection in Weakly-Aligned Visible-Infrared Imagery

Authors: Qi Ming, Xiaxin Yuan, Jiahuan Zhou, Jiangmeng Li, Xudong Zhao, Zhanchao Huang, Juan Fang, Shaoguang Huang, 

Organizations: Beijing University of Technology · University of Science and Technology of China · Peking University · Institute of Software, Chinese Academy of Sciences · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Visible-infrared object detection (VIOD) detects objects with oriented bounding boxes from paired visible and infrared images. Existing methods depend on costly dual-modality annotations. Semi-supervised learning can reduce this burden, but extending it from single-modal detection to VIOD is challenging. In the practical image-pair-level setting considered here, only a few pairs are labeled in both modalities, while the rest are completely unlabeled. This limited supervision creates three challenges: (i) too few labeled boxes for robust cross-modal alignment; (ii) pseudo-label errors caused by branch-wise misses accumulate during self-training; and (iii) tail-class annotations become critically scarce as the labeling budget decreases. We propose Aligned Consensus Teacher (ACT) for label-efficient VIOD in this setting. Its Cycle-Consistent Region Alignment (CRA) combines cycle consistency and sparse anchors with reliability-weighted regional matching. Cross-Modal Consensus Mean-Teacher (CMC-MT) forms consensus pseudo labels under pair-preserving views to recover branch-wise misses and supervise unlabeled pairs. Text-Guided Cross-Modal Instance Augmentation (TG-CMIA) uses a vision-language scene prior to compose tail-class instance pairs while preserving RGB--IR offsets. To the best of our knowledge, ACT is the first framework to study semi-supervised VIOD under this image-pair-level setting. Experiments on DroneVehicle and VEDAI show consistent gains across annotation ratios. With 10% labeled pairs on DroneVehicle, ACT reaches 94.3% of the mAP obtained by the same detector under full supervision. Code and models will be available on GitHub to facilitate future work.

Figures & tables

Figure 1: Motivation on DroneVehicle. (a) A 1%1\% labeling budget provides about 90×90\times fewer matched boxes for RGB–IR alignment. (b) A branch-wise miss yields incomplete pseudo labels whose errors propagate through EMA self-training. (c) ACT recovers class-wise AP toward the full-supervision reference at the 1%1\% budget.
Figure 2: Training and inference overview of ACT. CRA learns regional correspondence from labeled pairs, CMC-MT converts unlabeled pairs into consensus supervision, and TG-CMIA composes tail-class RGB–IR instance pairs on the labeled branch. Only the fused EMA teacher is retained for inference.
Table 3
Figure 3: Quantitative analysis on DroneVehicle. (a) Label efficiency across annotation ratios. (b) Teacher pseudo-label quality across RGB, IR, and fused branches. (c) Class-wise effects of TG-CMIA under the two lowest labeling budgets.
MethodcartruckbusvanfreightmAP
C 2 Former [ 41 ]0.8960.4420.8720.3400.3260.575
DMM [ 47 ]0.9000.5550.8880.4490.5210.663
SM3Det [ 18 ]0.8970.4740.8810.3600.4340.609
M2D-LIF [ 45 ]0.8890.5080.7780.4120.4230.602
MT [ 31 ]0.8900.4920.8800.3640.4350.612
SOOD [ 11 ]0.8970.4620.8620.3670.3830.594
Table 3: Per-category AP on DroneVehicle at the 10%10\% annotation ratio.
ComponentsAnnotation ratio
CMC-MTTG-CMIACRA1%5%10%100%
   0.2990.4850.5750.724
✓  0.4890.6190.665–
✓✓ 0.5100.6390.679–
✓✓✓0.5110.6440.683–
Table 4: Component ablation on DroneVehicle.
Figure 4: Qualitative results on DroneVehicle. Rows show day and night scenes; columns show RGB input, baseline predictions (orange), and ACT predictions (green). Red dashed boxes mark targets recovered by ACT.
More in this topic

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 629 papers

Explore similar work

CardsList
Aug 11, 2026

Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration

Visible-infrared object detection relies on complementary RGB and thermal cues, but its performance is often degraded by cross-modal spatial misalignment. Most existing methods rely on implicit feature adaptation to handle weakly misaligned scenarios, while large-offset geometric discrepancies remain insufficiently addressed. In this paper, we propose a Joint Feature-domain Registration and Detection network (JFRDet), an end-to-end visible-infrared oriented object detector tailored for severely cross-modal geometric discrepancies. JFRDet introduces a Cross-Modal Affine Alignment (CMAA) module to estimate an image-level affine transformation for explicit multi-level feature alignment. Note that illumination changes directly affect the reliability of RGB cues, an Illumination-Guided Complementary Fusion (IGCF) module adaptively exploits modality reliability under varying illumination conditions for cross-modal fusion. Then, an Alignment Quality-Consistency Gating (AQCG) strategy stabilizes joint optimization by modulating detection supervision according to alignment reliability and gradient consistency. We further construct DroneVehicle Misaligned (DVMA), a benchmark for evaluating visible-infrared oriented object detection under severe cross-modal geometric misalignment. The proposed JFRDet achieves 69.7% mAP50\mathrm{mAP}_{50} on DVMA, which represents state-of-the-art (SOTA) performance. The code and dataset will be available on GitHub.

Qi Ming, Yuyang Wang, Mingjing Zhao +7

Beijing University of Technology, China · Central South University, China · Beijing Electronics Science & Technology Institute +4

Multispectral Object DetectionCross-Modal AlignmentInfrared Object DetectionCross-Modal Image RegistrationMultimodal Sensor Fusion
AskPDF
Sep 3, 2026

Residual Optimal Transport-Based Experts Collaboration Towards Modality-Aware Infrared-Visible Object Detection

Infrared-visible object detection (IVOD) integrates complementary evidence from visible and infrared sensors for reliable perception in challenging scenes. In practice, sensors may fail or drop frames, leaving one modality unavailable or intermittent. Existing methods for IVOD assume both modalities are always present, and fixed fusion collapses when one stream is missing. Furthermore, it remains a critical challenge to reliably estimate semantic correlation across heterogeneous modalities, especially under spectral distribution discrepancy. We present FlexibleFusion, a unified and adaptive method that flexibly allocates integration pathways and fusion strength, operating seamlessly across complete and missing-modality regimes. At its core, the Modality-Aware Experts Collaboration (MAEC) mechanism selectively activates and aggregates cross-modal or intra-modal expert pathways. It allows cross-modal fusion when full modalities are available and falls back to self-fusion under missing conditions. Additionally, we design Residual Self-Paced Entropic Optimal Transport (RSPEOT) to align heterogeneous feature distributions from a transport perspective. Instead of relying on the fixed sparsity coefficient in standard entropic optimal transport (EOT), RSPEOT introduces a residual-driven self-paced update that prioritizes reliable matches and progressively refines harder ones. This design alleviates the additional optimization burden of standard EOT while preserving reliable semantic alignment. Comprehensive experiments under complete and missing-modality protocols show consistent performance across arbitrary modality configurations. Code will be released upon publication.

Yue Zhao, Hua Yu, Yukun Zhao +6

School of Electronic Engineering, Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University, Xi’an 710071, China · College of Computing and Data Science, Nanyang Technological University, Singapore · School of Electronics and Communication Engineering, Sun Yat-sen University, Guangzhou 510275, China +3

Missing-Modality LearningMultimodal RobustnessMultispectral Object DetectionCross-Modal AlignmentObject DetectionOptimal Transport
AskPDF
Jul 22, 2026

Not All Patches are Equal: Sampling Matters for Visible-Infrared Pre-Training

Visible-infrared (VIS-IR) alignment is a key pre-training task for robust multi-sensor perception. Most existing methods use uniform patch-wise contrastive learning, but this can be unreliable in VIS-IR data because imaging-physics differences make some spatially paired regions inherently less comparable, and aligning them with equal strength hinders representation learning and downstream transfer. In this paper, we revisit VIS-IR pre-training from a sampling perspective and propose Importance-Aware Sampling (IAS), which adjusts training emphasis based on patch reliability. Specifically, IAS (i) derives patch weights from infrared structural cues and uses them to reweight the contrastive objective; (ii) learns a soft importance mask with a lightweight sampler, optionally warm-started from the hand-crafted prior; and (iii) employs a patch curriculum learning strategy that gradually expands from high-reliability regions to harder patches. It is worth noting that IAS is plug-and-play and works with both patch-/correlation-level alignment (e.g., UNIV-style) and image-level contrastive baselines (e.g., ImageBind-style). Extensive experiments on multiple VIS-IR benchmarks demonstrate consistent improvements over strong baselines, including for IR semantic segmentation, IR object detection and VIS semantic segmentation and cross-modal retrieval task. Code will be released on https://github.com/KlayMa527/IAS.

Qiwei Ma, Bin Deng, Junjie Zhu +5

School of Artificial Intelligence and Robotics, Hunan University, Changsha, 410082, Hunan, China · Intelligent Game and Decision Lab, Beijing, 100166, Beijing, China · Yuelushan Center for Industrial Innovation, Changsha, 410082, Hunan, China

Visual Representation LearningSelf-Supervised Pre-TrainingCross-Modal AlignmentMultimodal Contrastive Learning
AskPDF