JMT-YOLO: Multi-task Domain Adaptation via Segmentation-Guided Consistency Learning

Zhenhua Yu; Jiahao Zhang; Wenchao Zhang; Ou Ye; Hengxiang Zhao

cs.CVOct 9, 2026

JMT-YOLO: Multi-task Domain Adaptation via Segmentation-Guided Consistency Learning

Authors: Zhenhua Yu, Jiahao Zhang, Wenchao Zhang, Ou Ye, Hengxiang Zhao

Organizations: Institute of Systems Security and Control, College of Artificial Intelligence and Computer Science, Xi’an University of Science and Technology, Xi’an 710054, China

Abstract

Domain adaptive object detection (DAOD) aims to reduce performance degradation due to distribution shifts between training (source) and testing (target) datasets. However, owing to insufficient modeling of fine-grained features and limited instance-level alignment, existing approaches become less effective under complex environmental conditions, such as extreme weather conditions and object occlusion. We overcome these challenges by proposing JMT-YOLO, a novel joint multi-task learning framework for DAOD. JMT-YOLO is based on the Mean Teacher architecture. It introduces an auxiliary semantic segmentation task and a masking mechanism to enhance the modeling of foreground-background semantic boundaries in the target domain. Specifically, fine-grained feature degradation is reduced by introducing a multiscale context-aware segmentation head equipped with a dense connection structure to effectively capture local and global contextual features. A mask-consistency loss is formulated by applying random masking operations to target images, thereby guiding the network to learn structural context and improving instance-level feature alignment. Extensive experiments on three benchmark domain adaptation tasks (Pascal VOC to Clipart1k, Cityscapes to Foggy Cityscapes, and KITTI to Cityscapes) demonstrate that JMT-YOLO achieves detection performances of 46.9%, 57.3%, and 61.0%, respectively. These results demonstrate superior performance over existing state-of-the-art and baseline methods, thereby validating the effectiveness and generalization of JMT-YOLO.

Figures & tables

Figure 1: Overall architecture of the proposed JMT-YOLO framework. Left : pretrained contrastive unpaired translation (CUT) [ 32 ] model is utilized to generate target-like images $x^{SF}$ from source images $x^{S}$ and source-like images $x^{TF}$ from target images $x^{T}$ . Concurrently, a masked target image $x^{TM}$ is generated by applying a masking operation to $x^{T}$ . Middle : a multi-task network, based on the MT framework, is constructed using the YOLOv5 [ 33 ] detector. Right : corresponding loss functions are formulated based on predictions from multiple inputs (differentiated by color pathways) to support semi-supervised learning.

Figure 2: Pseudo-images generated from KITTI and Cityscapes datasets. (a) Original images $x^{S}$ from the KITTI domain. (b) KITTI images translated into the target (Cityscapes) style images $x^{SF}$ . (c) Cityscapes images translated into the source (KITTI) style images $x^{TF}$ . (d) Original images $x^{T}$ from the Cityscapes domain.

Figure 3: Overview of the teacher-student network built on YOLOv5. A shared backbone and FPN–PAN neck feed two parallel heads: an unchanged detect head and an auxiliary Segment Head integrates the proposed DenseRFB for multiscale, context-aware features. Both the branches share the backbone/neck and are jointly optimized. “AC” denotes dilated convolution.

Method	Venue	aero	bcycle	bird	boat	bottle	bus	car	cat	chair	cow	table	dog	hrs	bike	prsn	plnt	sheep	sofa	train	tv	all
SWDA [ 6 ]	CVPR19	26.2	48.5	32.6	33.7	38.5	54.3	37.1	18.6	34.8	58.3	17.0	12.5	33.8	65.5	61.6	52.0	9.3	24.9	54.1	49.1	38.1
SCL [ 49 ]	arXiv19	44.7	50.0	33.6	27.4	42.2	55.6	38.3	19.2	37.9	69.0	30.1	26.3	34.4	67.3	61.0	47.9	21.4	26.3	50.1	47.3	41.5
DM [ 15 ]	CVPR19	25.8	63.2	24.5	42.4	47.9	43.1	37.5	9.1	47.0	46.7	26.8	24.9	48.1	78.7	63.0	45.0	21.3	36.1	52.3	53.4	41.8
CRDA [ 50 ]	CVPR20	28.7	55.3	31.8	26.0	40.1	63.6	36.6	9.4	38.7	49.3	17.6	14.1	33.3	74.3	61.3	46.3	22.3	24.3	49.1	44.3	38.3
HTCN [ 51 ]	CVPR20	33.6	58.9	34.0	23.4	45.6	57.0	39.8	12.0	19.7	51.3	21.1	20.1	39.1	72.8	63.0	43.1	19.3	30.1	50.2	51.8	40.3
MEAA [ 52 ]	ACMMM20	31.3	53.5	38.0	17.8	38.5	69.9	38.2	23.8	38.3	58.1	14.6	18.1	33.8	88.1	60.3	42.1	7.8	30.8	61.1	58.7	41.1

Table 1: The transfer results of PascalVOC → Clipart1k. The bold indicates the best performance, and underline indicates the second-best.

Figure 4: Detection results of different methods on the Clipart1k dataset when PascalVOC is used as the source domain. JMT-YOLO localizes even better and accurately with fewer false positives.

Method	Venue	Detector	Bus	Bicycle	Car	Mcycle	Person	Rider	Train	Truck	All
DA-Faster [ 5 ]	CVPR18	FR-CNN	35.3	27.1	40.5	20.0	25.0	31.0	20.2	22.1	27.6
SWDA [ 6 ]	CVPR19	FR-CNN	36.2	35.3	43.5	30.0	29.9	42.3	32.6	24.5	34.3
ATF [ 53 ]	ECCV20	FR-CNN	43.3	38.8	50.0	33.4	34.6	47.0	38.7	23.7	38.7
UMT [ 36 ]	CVPR21	FR-CNN	56.5	37.3	48.6	30.4	33.0	46.7	46.8	34.1	41.7
AT [ 37 ]	CVPR22	FR-CNN	58.4	53.0	63.8	34.4	43.2	52.7	34.5	32.4	46.6
DSCA [ 19 ]	PR 2024	FR-CNN	52.6	35.2	68.8	52.7	53.9	61.3	24.3	32.5	47.6

Table 2: The transfer results of Cityscapes → Foggy Cityscapes. The bold indicates the best performance, and underline indicates the second-best. FR-CNN refers to the use of the Faster R-CNN detector.

Figure 5: Detection results of different methods on Foggy Cityscapes dataset when Cityscapes is used as the source domain. JMT-YOLO shows improved robustness in detecting distant and occluded objects.

Method	Venue	Detector	Car AP
EPM [ 22 ]	ECCV20	YOLOv5-L	58.1
DAYOLO [ 12 ]	PMLR21	YOLOv5-L	57.4
MSDAYOLO [ 58 ]	ICIP21	YOLOv5-L	56.5
AT [ 37 ]	CVPR22	YOLOv5-L	59.0
CMT [ 59 ]	CVPR23	YOLOv5-L	53.2
VT [ 39 ]	PR25	YOLOv5-L	61.9

Table 3: The transfer results of KITTI → Cityscapes. The bold indicates the best performance, and underline indicates the second-best.

Figure 6: Detection results of different methods on Cityscapes dataset when KITTI is used as the source domain. Our method JMT-YOLO achieves more precise localization and fewer false detection.

Figure 7: Effect of different mask ratios on model training. Moderate masking improves model performance, while excessive masking considerably declines the performance.

Figure 8: Effect of $\lambda_{\mathrm{det}}$ and $\lambda_{\mathrm{seg}}$ combinations on the mAP of the model in the target domain. When the weights are set to $\lambda_{\mathrm{det}}$ = 0.6 and $\lambda_{\mathrm{seg}}$ = 0.4, the model achieves the best performance (57.3%).

Source Only	Baseline	Masking	DenseRFB	[email protected](%)	Gain(%)
✓				35.9	-
✓	✓			55.9	+20.0
✓	✓	✓		56.8	+20.9
✓	✓		✓	57.2	+21.3
✓	✓	✓	✓	57.3	+21.4

Table 4: Impact of different modules on detection performance. Starting from Source Only, adding Baseline (SSDA-YOLO), Masking (masking mechanism), and DenseRFB modules (auxiliary segmentation head) progressively improves results.

Figure 9: Grad-CAM visualizations on Foggy Cityscapes. With masking, the activations become more compact and aligned with object regions, while the unmasked model shows weaker responses.

Status	[email protected](%)	mAP@.5:.95
Baseline	59.6	29.8
Ours (200 GT masks)	61.0	30.6
Ours (GT+SAM)	62.0	32.4
Ours (auto box+SAM)	61.7	32.3
Ours (text+SAM)	60.9	31.1

Table 5: Comparison of SAM prompting strategies for pseudo-mask supervision on KITTI → Cityscapes (car), including GT box prompts, auto-predicted box prompts, and a box-free text prompt.

Figure 10: Qualitative comparison of SAM pseudo masks generated with different prompting strategies on KITTI (car). GT box prompts generally produce more complete and boundary-aligned masks. Auto box prompts remain effective but can introduce slight boundary noise. Text prompts are less spatially constrained, leading to occasional missing or fragmented masks in complex scenes.

Status	[email protected](%)	mAP@.5:.95
Baseline	59.6	29.8
Ours (GT+SAM)	62.0	32.4
Ours (noisy $\pm$ 10%)	61.0	31.8
Ours (noisy $\pm$ 30%)	60.7	30.9
Ours (noisy $\pm$ 50%)	60.4	29.7
Ours (noisy $\pm$ 70%)	58.6	28.6

Table 6: Effect of box-prompt noise on SAM-based pseudo-mask supervision on KITTI → Cityscapes (car), where noisy

\pm r\%

denotes

\pm r\%

center-shift perturbation of GT box prompts.

Figure 11: Effect of box-prompt center-shift noise on SAM pseudo masks (noisy $\pm r\%$ ) on KITTI (car). Mild shifts preserve sufficient overlap and yield usable masks, whereas large shifts drift toward background or adjacent regions, resulting in fragmented or incorrect masks.

Figure 12: t-SNE visualization comparison of intermediate feature distributions. (a) Source Only model, source domain (blue), and target domain (red) features are clearly separated, (b) SSDA-YOLO model, source and target domain features begin to merge but differences still exist, (c) our proposed method JMT-YOLO, source and target domain feature distributions highly overlapped, indicating stronger domain alignment capability.

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 619 papers

TestMate: Test-Time Domain Adaptation Aided by Lightweight Vision Foundation Model
Jul 4, 2026Dimitrios Fotiou, Vasileios Mygdalis, Ioannis PitasSource-Free Domain Adaptation Test-Time Adaptation
Aristotle University of Thessaloniki
DA-Mamba: Learning Domain-Aware State Space Model for Global-Local Alignment in Domain Adaptive Object Detection
Mar 19, 2026Haochen Li, Rui Zhang, Hantao Yao +5Object Detection Unsupervised Domain Adaptation
Intelligent Software Research Center, Institute of Software, CAS · University of Chinese Academy of Sciences · State Key Lab of Processors, Institute of Computing Technology, CAS +1
Contrastive Hypergraph Source-free Domain Adaptive Object Detection in Adverse Weather Conditions--Extended Version
Oct 17, 2026Jicheng Yuan, Tung Kieu, Manh Nguyen-Duc +2Object Detection Source-Free Domain Adaptation
Open Distributed Systems, Technical University of Berlin · Department of Computer Science, Aalborg University, Denmark · Fraunhofer Institute for Open Communication Systems, Berlin, Germany

Jul 4, 2026J/K move · Enter open · S save

TestMate: Test-Time Domain Adaptation Aided by Lightweight Vision Foundation Model

Dimitrios Fotiou, Vasileios Mygdalis, Ioannis Pitas

Aristotle University of Thessaloniki

Source-Free Domain Adaptation Test-Time Adaptation Semantic Segmentation

Open paper AskPDF arXiv

Test-Time Domain Adaptation (TTDA) aims to adapt Deep Neural Networks to distribution shifts using only streaming, unlabeled test data in real time. Current methods for semantic segmentation tasks suffer from critical limitations. Entropy minimization techniques require costly backpropagation, risking catastrophic forgetting and producing noisy segmentation boundaries. Memory-bank methods, while backpropagation-free, exhibit slow adaptation, requiring numerous samples to converge and struggle to handle continuous domain shifts. We introduce TestMate, a novel, real-time, and backpropagation-free TTDA framework that overcomes these issues. TestMate leverages generalization capability of a lightweight Visual Foundation Model to guide the adaptation. We use a zero-shot instance segmentation YOLOv8-seg based model to generate unlabeled mask proposals for objects and their parts at multiple scales in real time. These proposals are fused with the primary model via a heuristic, size-ordered competitive scheme, where small, high-confidence regions dominate and refine predictions in surrounding larger, less certain areas. This paremeter-free mechanism enables immediate adaptation from the first frame, inherently avoids catastrophic forgetting and effectively preserves fine object details and boundaries, even for small objects. TestMate can be used as a standalone, efficient refinement module or seamlessly integrated into existing TTDA methods to significantly boost their performance. We demonstrate state-of-the-art results across two benchmark datasets, proving TestMate's effectiveness in three distinct adaptation tasks: TTDA, Source-Free Domain Adaptation (SFDA), and online-TTDA. Code is available.

Abstract

Object Detection

Explore similar work

TestMate: Test-Time Domain Adaptation Aided by Lightweight Vision Foundation Model

DA-Mamba: Learning Domain-Aware State Space Model for Global-Local Alignment in Domain Adaptive Object Detection

Contrastive Hypergraph Source-free Domain Adaptive Object Detection in Adverse Weather Conditions--Extended Version