AHMAD: Adaptive Hybrid Multi-task Vision Learning with Assisted Distillation for Keypoint Detection

Mohammad Mahdi; Nedyalko Prisadnikov; Yuqian Fu; Carmelo Scribano; Danda Pani Paudel; Luc Van Gool

cs.CVSep 28, 2026

AHMAD: Adaptive Hybrid Multi-task Vision Learning with Assisted Distillation for Keypoint Detection

Authors: Mohammad Mahdi, Nedyalko Prisadnikov, Yuqian Fu, Carmelo Scribano, Danda Pani Paudel, Luc Van Gool

Organizations: INSAIT, Sofia University “St. Kliment Ohridski”

Abstract

Generalist multitasking vision models aim to unify multiple vision tasks within a single framework, enabling more efficient and versatile learning. However, handling diverse vision tasks -- spanning dense and sparse predictions -- remains challenging due to their inherently varying output structures. In this paper, we propose AHMAD, a simple yet effective framework for generalist multitask learning that integrates different key vision tasks: semantic segmentation, instance segmentation, depth estimation, keypoint detection, and object detection. Our approach incorporates these five tasks into a unified structure: a shared encoder-decoder with several lightweight task-specific projectors. Under the multitask learning paradigm, we observed a complementary performance gain, achieving a state-of-the-art PQ of 53.1 and an mIoU of 66.5 for COCO-val panoptic and semantic segmentation, respectively. Additionally, for top-down keypoint detection, which typically incurs high computational overhead due to multiple forward passes, we introduce a knowledge distillation-based method that enables a single forward pass over the entire image, greatly improving efficiency. Ultimately, our model delivers a lightweight yet effective generalist multitask learning framework, demonstrating strong performance across five vision tasks.

Figures & tables

Figure 1 : Our Proposed Training Framework . We leverage a powerful vision transformer encoder combined with a CNN-based decoder to generate a rich feature map, which contains representations useful for various tasks. Task-specific projectors are then applied to perform different tasks.

Figure 2 : Method Overview . In each iteration, a random vision task—illustrated here with object detection—is selected. After patch tokenization, the backbone processes the tokens, and outputs from four intermediate layers are concatenated to form $z$ . This is then decoded to a shared feature map $F_{x}=D_{\phi}(z)$ , which is further processed by the corresponding task-specific projector. For the first three tasks, EDS is applied during loss minimization (*). The gradient-colored backbone highlights layer decay in $B_{\theta}$ , where later layers receive stronger updates while earlier layers remain more stable, promoting better feature adaptation. Gray/dim arrows indicate inactive tasks, and for task projectors, input and outer arrows represent the forward and backward propagation, respectively.

Figure 3 : KD * Overview. Blue arrows indicate paths that are inactive during inference, while red arrows represent the active inference path. By enabling feature cropping, we reduce the number of forward passes from $N$ to 1. In this process, ${}_{4}z$ represents the latent image embedding obtained from the last of the four intermediate layers, $w^{\prime}=4w$ , and $h^{\prime}=4h$ . The green lines indicate contributions to the loss function, while the dashed red line represents the point at which we obtain the final output during the inference stage.

Model	Resolution	#Params	Backbone	Segmentation		Depth	Detection		Keypoint
				PQ $\uparrow$	mIoU $\uparrow$	RMSE $\downarrow$	mAP 50 $\uparrow$	mAP all $\uparrow$	OKS-mAP 50 $\uparrow$	OKS-mAP all $\uparrow$
Specialized Models
MaskFormer	$640\times 640$	212M	SWIN-L	52.7	-	$\times$	$\times$	$\times$	$\times$	$\times$
Mask2Former	$640\times 640$	216M	SWIN-L	57.8	-	$\times$	$\times$	$\times$	$\times$	$\times$
OneFormer	$640\times 640$	223M	DINAT-L	58.0	67.4	$\times$	$\times$	$\times$	$\times$	$\times$
Faster-RCNN [ 14 ]	$600\times 1000$	60M	ResNet-101	$\times$	$\times$	$\times$	-	44.0	$\times$	$\times$

Table 1 : Comparison of multi-task models across various evaluation metrics on the COCO-val split. GiT performs object detection and instance segmentation at a resolution of

1120\times 1120

, while semantic segmentation is conducted at

672\times 672

. Additionally, Unified-IO is evaluated on the GRIT benchmark, which includes the COCO dataset. ”

\times

” indicates that the model does not support the task, while ”-” means the task is supported, but no performance numbers for the COCO dataset are provided in the paper.

Figure 4 : Qualitative Results. We show our results on five different tasks with diverse examples.

Method	Input	#Projector Params	OKS-mAP $\uparrow$	Avg Run/Img	Decoder
Top-down KD	$C_{280\times 280}$	440K	62.6	2.7	✓
Bypassed KD	$C_{280\times 280}$	2.5K	60.1	2.7	$\times$
KnowDist KD	$G_{280\times 280}$	2.5K	41.32	1.0	✓

Table 2 : A comparative analysis of the standard top-down Keypoint detection, the decoder-bypassed variant, and our proposed Knowledge Distillation approach.

G

denotes the global image input, while

C

represents the cropped image input. The last column shows a

\checkmark

for methods that run the decoder, and

\times

otherwise.

Panoptic Segmentation
EDS	(u,v) Encoding	PQ $\uparrow$
$\times$	2D Regression	38.01
$\times$	PE L=4	44.86
$\checkmark$	2D Regression	41.61
$\checkmark$	Cross-Entropy	39.87
$\checkmark$	PE L=3	47.38

Table 3 : Panoptic Segmentation Ablation. Effects of EDS and Positional Embedding (PE). In 2D Regression, we directly regressed

(u,v)

. The ablation is performed on a ViT-Base model with a resolution of

420\times 420

Conf ID	Config	Loss Function	OKS-mAP $\uparrow$
CID-1	Stand-alone feature cropping	$\mathcal{L}^{*}_{2}$	34.2
CID-2	CID-1, RGB concatenation	$\mathcal{L}^{*}_{2}$	34.6
CID-3	Distillation, Shared Projector	$\mathcal{L}^{*}_{1}+\mathcal{L}^{d}_{1}$	35.2
CID-4	CID-3	$\mathcal{L}^{}_{1}+\mathcal{L}^{d}_{1}+\mathcal{L}^{}_{2}$	35.6
CID-5	CID-4, Different Projectors	$\mathcal{L}^{}_{1}+\mathcal{L}^{d}_{1}+\mathcal{L}^{}_{2}$	36.2
CID-6	CID-5, TFU added	$\mathcal{L}^{}_{1}+\mathcal{L}^{d}_{1}+\mathcal{L}^{}_{2}$	39.8

Table 4 : KD * Ablation. Comparison of different architectures and loss functions on OKS-mAP performance. For CID-1 and CID-2, no distillation is performed. In CID-2, the cropped image is directly concatenated with the feature map from the input. In CID-3, both the student and teacher utilize the same projector,

{H}^{5}_{\psi_{5}}

. Conversely, in CID-5, the student and teacher employ different but architecturally identical projectors,

{H}^{5}_{\psi_{5}}

and

{H^{\prime}}^{5}_{\psi_{5}}

. Observe how our TFU function effectively enhances performance.

Explore similar work

Jul 1, 2026cs.CV

GKDT: General Keypoint Detection Transformer

With the emergence of various pre-trained vision and language models, computer vision is shifting from narrow-domain to open-domain recognition. The construction of a more powerful yet general keypoint detection (GKD) model to support diverse tasks has become increasingly important in the field. To this end, we firstly present a large-scale unified keypoint dataset called MegaKPT. The dataset is composed of over 1.3 million diverse object instances from twenty-nine existing datasets, and enjoys high-quality unified annotations with keypoint text descriptions. Based on MegaKPT, we develop GKDT, a simple, flexible and powerful DINOv3 based Transformer model for General Keypoint Detection. Our GKDT supports visual prompts, text prompts, or both. To enhance model training, we also propose a suite of useful strategies such as mix-modal prompted training and dynamic importance sampling. By testing over 22 test sets with seen or unseen objects, our single GKDT model shows strong performance and generality in detecting keypoints on broad categories, with most categories over 90% PCK@0.1 accuracy, offering high practical applicability to real-world problems. The dataset, models, and codes will be released at https://github.com/AlanLuSun/General-Keypoint-Detection.

Changsheng Lu, Yuxin Chen, Haokun Gui +5

Hong Kong University of Science and Technology, Hong Kong, China · Australian National University, Canberra, Australia · Tencent Inc., China +1

Keypoint Detection Open-Vocabulary Object Detection Transformer Architectures Human Annotations

Ask PDF

Jul 17, 2026cs.CV

DPNeXt: A Lightweight Multi-Scale Feature Fusion Framework for Efficient ViT-Based Multi-Task Dense Prediction

Multi-Task Learning (MTL) in robotics perception systems supports comprehensive 3D spatial scene understanding by integrating semantic segmentation and depth estimation. While Vision Foundation Models (VFMs) are increasingly adopted as robust feature encoders, existing decoding strategies present a critical bottleneck. To address this, we propose DPNeXt, a streamlined multi-scale feature fusion decoder and efficient alternative to the standard Dense Prediction Transformer (DPT). DPNeXt uses dual depthwise separable inverted bottlenecks to improve frozen VFM utilization through fusion-centric decoding and independent task modularization. To further mitigate negative inductive transfer between tasks, we introduce the Multi-Task Boundary Guidance (MTBG) strategy. Unlike prior boundary-aware methods that add fusion modules or gating, MTBG applies symmetric boundary-focused supervision to encourage geometric consistency without extra annotation or inference cost. Experiments on Cityscapes show that DPNeXt-S outperforms prior state-of-the-art (SOTA) MTL models, while DPNeXt-B further improves the overall performance and achieves the best results among the compared methods. On NYUv2, DPNeXt-B also achieves the best semantic segmentation and depth estimation results among the compared methods while requiring substantially fewer trainable parameters than prior large-scale MTL models. Compared with the standard DPT, DPNeXt-S reduces trainable parameters by 78.6% and achieves the fastest inference speed among the compared models on resource-constrained laptop hardware. The source code, model checkpoints, and a demo video will be made available at https://github.com/kangjehun/DPNeXt.

Jehun Kang, Jungha Wang, Youngjun Hwang +1

School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea

Dense Prediction Depth Estimation Multi--Task Learning Efficient Semantic Segmentation Next

Ask PDF

Jun 29, 2026cs.CV

MUSE: Unlocking Timestep as Native Task Steering for One-Step Dense Prediction

Monocular dense prediction has recently seen remarkable success by repurposing pre-trained diffusion models. This opens a promising yet challenging avenue for more efficient multi-task learning paradigm. However, existing multi-task diffusion methods often introduce parameter-heavy adapters, experts, or learnable task tokens, leading to computational redundancy. In this paper, we reveal an inherent mechanism within one-step diffusion models: the native, fixed sinusoidal timestep embedding can be repurposed as an endogenous task steering signal. Based on this discovery, we propose Multi-task Unified eStimation via timestep Embedding (MUSE), a parameter-free, single-model multi-tasking approach for dense prediction. We interpret this mechanism via Manifold Decoupling, where discrete, fixed timestep values deterministically steer the generation process towards decoupled, task-specific manifolds in the latent space. Extensive experiments across 10 datasets demonstrate that MUSE achieves highly competitive performance on both monocular depth and normal estimation, and its efficacy generalizes across U-Net and DiT architectures. Our work offers a concise and efficient path toward generalist vision models by simply unlocking the latent potential of existing generation infrastructure.

Shuo Zhou, Zhaoxin Li, Xiujuan Chai

Agricultural Information Institute, Chinese Academy of Agricultural Sciences, Beijing, China. · Key Laboratory of Agricultural Big Data, Ministry of Agriculture and Rural Affairs, Beijing, China.

Dense Prediction Multi--Task Learning Timestep Diffusion Models Timestep Embeddings Latent Space+1

Ask PDF