AHMAD: Adaptive Hybrid Multi-task Vision Learning with Assisted Distillation for Keypoint Detection

Mohammad Mahdi; Nedyalko Prisadnikov; Yuqian Fu; Carmelo Scribano; Danda Pani Paudel; Luc Van Gool

cs.CVSep 28, 2026

AHMAD: Adaptive Hybrid Multi-task Vision Learning with Assisted Distillation for Keypoint Detection

Authors: Mohammad Mahdi, Nedyalko Prisadnikov, Yuqian Fu, Carmelo Scribano, Danda Pani Paudel, Luc Van Gool

Organizations: INSAIT, Sofia University “St. Kliment Ohridski”

Abstract

Generalist multitasking vision models aim to unify multiple vision tasks within a single framework, enabling more efficient and versatile learning. However, handling diverse vision tasks -- spanning dense and sparse predictions -- remains challenging due to their inherently varying output structures. In this paper, we propose AHMAD, a simple yet effective framework for generalist multitask learning that integrates different key vision tasks: semantic segmentation, instance segmentation, depth estimation, keypoint detection, and object detection. Our approach incorporates these five tasks into a unified structure: a shared encoder-decoder with several lightweight task-specific projectors. Under the multitask learning paradigm, we observed a complementary performance gain, achieving a state-of-the-art PQ of 53.1 and an mIoU of 66.5 for COCO-val panoptic and semantic segmentation, respectively. Additionally, for top-down keypoint detection, which typically incurs high computational overhead due to multiple forward passes, we introduce a knowledge distillation-based method that enables a single forward pass over the entire image, greatly improving efficiency. Ultimately, our model delivers a lightweight yet effective generalist multitask learning framework, demonstrating strong performance across five vision tasks.

Figures & tables

Figure 1 : Our Proposed Training Framework . We leverage a powerful vision transformer encoder combined with a CNN-based decoder to generate a rich feature map, which contains representations useful for various tasks. Task-specific projectors are then applied to perform different tasks.

Figure 2 : Method Overview . In each iteration, a random vision task—illustrated here with object detection—is selected. After patch tokenization, the backbone processes the tokens, and outputs from four intermediate layers are concatenated to form $z$ . This is then decoded to a shared feature map $F_{x}=D_{\phi}(z)$ , which is further processed by the corresponding task-specific projector. For the first three tasks, EDS is applied during loss minimization (*). The gradient-colored backbone highlights layer decay in $B_{\theta}$ , where later layers receive stronger updates while earlier layers remain more stable, promoting better feature adaptation. Gray/dim arrows indicate inactive tasks, and for task projectors, input and outer arrows represent the forward and backward propagation, respectively.

Figure 3 : KD * Overview. Blue arrows indicate paths that are inactive during inference, while red arrows represent the active inference path. By enabling feature cropping, we reduce the number of forward passes from $N$ to 1. In this process, ${}_{4}z$ represents the latent image embedding obtained from the last of the four intermediate layers, $w^{\prime}=4w$ , and $h^{\prime}=4h$ . The green lines indicate contributions to the loss function, while the dashed red line represents the point at which we obtain the final output during the inference stage.

Model	Resolution	#Params	Backbone	Segmentation		Depth	Detection		Keypoint
				PQ $\uparrow$	mIoU $\uparrow$	RMSE $\downarrow$	mAP 50 $\uparrow$	mAP all $\uparrow$	OKS-mAP 50 $\uparrow$	OKS-mAP all $\uparrow$
Specialized Models
MaskFormer	$640\times 640$	212M	SWIN-L	52.7	-	$\times$	$\times$	$\times$	$\times$	$\times$
Mask2Former	$640\times 640$	216M	SWIN-L	57.8	-	$\times$	$\times$	$\times$	$\times$	$\times$
OneFormer	$640\times 640$	223M	DINAT-L	58.0	67.4	$\times$	$\times$	$\times$	$\times$	$\times$
Faster-RCNN [ 14 ]	$600\times 1000$	60M	ResNet-101	$\times$	$\times$	$\times$	-	44.0	$\times$	$\times$

Table 1 : Comparison of multi-task models across various evaluation metrics on the COCO-val split. GiT performs object detection and instance segmentation at a resolution of

1120\times 1120

, while semantic segmentation is conducted at

672\times 672

. Additionally, Unified-IO is evaluated on the GRIT benchmark, which includes the COCO dataset. ”

\times

” indicates that the model does not support the task, while ”-” means the task is supported, but no performance numbers for the COCO dataset are provided in the paper.

Figure 4 : Qualitative Results. We show our results on five different tasks with diverse examples.

Method	Input	#Projector Params	OKS-mAP $\uparrow$	Avg Run/Img	Decoder
Top-down KD	$C_{280\times 280}$	440K	62.6	2.7	✓
Bypassed KD	$C_{280\times 280}$	2.5K	60.1	2.7	$\times$
KnowDist KD	$G_{280\times 280}$	2.5K	41.32	1.0	✓

Table 2 : A comparative analysis of the standard top-down Keypoint detection, the decoder-bypassed variant, and our proposed Knowledge Distillation approach.

G

denotes the global image input, while

C

represents the cropped image input. The last column shows a

\checkmark

for methods that run the decoder, and

\times

otherwise.

Panoptic Segmentation
EDS	(u,v) Encoding	PQ $\uparrow$
$\times$	2D Regression	38.01
$\times$	PE L=4	44.86
$\checkmark$	2D Regression	41.61
$\checkmark$	Cross-Entropy	39.87
$\checkmark$	PE L=3	47.38

Table 3 : Panoptic Segmentation Ablation. Effects of EDS and Positional Embedding (PE). In 2D Regression, we directly regressed

(u,v)

. The ablation is performed on a ViT-Base model with a resolution of

420\times 420

Conf ID	Config	Loss Function	OKS-mAP $\uparrow$
CID-1	Stand-alone feature cropping	$\mathcal{L}^{*}_{2}$	34.2
CID-2	CID-1, RGB concatenation	$\mathcal{L}^{*}_{2}$	34.6
CID-3	Distillation, Shared Projector	$\mathcal{L}^{*}_{1}+\mathcal{L}^{d}_{1}$	35.2
CID-4	CID-3	$\mathcal{L}^{}_{1}+\mathcal{L}^{d}_{1}+\mathcal{L}^{}_{2}$	35.6
CID-5	CID-4, Different Projectors	$\mathcal{L}^{}_{1}+\mathcal{L}^{d}_{1}+\mathcal{L}^{}_{2}$	36.2
CID-6	CID-5, TFU added	$\mathcal{L}^{}_{1}+\mathcal{L}^{d}_{1}+\mathcal{L}^{}_{2}$	39.8

Table 4 : KD * Ablation. Comparison of different architectures and loss functions on OKS-mAP performance. For CID-1 and CID-2, no distillation is performed. In CID-2, the cropped image is directly concatenated with the feature map from the input. In CID-3, both the student and teacher utilize the same projector,

{H}^{5}_{\psi_{5}}

. Conversely, in CID-5, the student and teacher employ different but architecturally identical projectors,

{H}^{5}_{\psi_{5}}

and

{H^{\prime}}^{5}_{\psi_{5}}

. Observe how our TFU function effectively enhances performance.

Explore similar work

GKDT: General Keypoint Detection Transformer
Jul 1, 2026Changsheng Lu, Yuxin Chen, Haokun Gui +5Keypoint Detection Open-Vocabulary Object Detection
Hong Kong University of Science and Technology, Hong Kong, China · Australian National University, Canberra, Australia · Tencent Inc., China +1
DPNeXt: A Lightweight Multi-Scale Feature Fusion Framework for Efficient ViT-Based Multi-Task Dense Prediction
Jul 17, 2026Jehun Kang, Jungha Wang, Youngjun Hwang +1Dense Prediction Depth Estimation
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea
MUSE: Unlocking Timestep as Native Task Steering for One-Step Dense Prediction
Jun 29, 2026Shuo Zhou, Zhaoxin Li, Xiujuan ChaiDense Prediction Multi--Task Learning
Agricultural Information Institute, Chinese Academy of Agricultural Sciences, Beijing, China. · Key Laboratory of Agricultural Big Data, Ministry of Agriculture and Rural Affairs, Beijing, China.

Jul 1, 2026, cs.CVJ/K move · Enter open · S save

GKDT: General Keypoint Detection Transformer

Changsheng Lu, Yuxin Chen, Haokun Gui, Rong Wang+4

Hong Kong University of Science and Technology, Hong Kong, China · Australian National University, Canberra, Australia · Tencent Inc., China+1

Keypoint Detection Open-Vocabulary Object Detection Transformer Architectures Human Annotations

Open paper AskPDF arXiv

With the emergence of various pre-trained vision and language models, computer vision is shifting from narrow-domain to open-domain recognition. The construction of a more powerful yet general keypoint detection (GKD) model to support diverse tasks has become increasingly important in the field. To this end, we firstly present a large-scale unified keypoint dataset called MegaKPT. The dataset is composed of over 1.3 million diverse object instances from twenty-nine existing datasets, and enjoys high-quality unified annotations with keypoint text descriptions. Based on MegaKPT, we develop GKDT, a simple, flexible and powerful DINOv3 based Transformer model for General Keypoint Detection. Our GKDT supports visual prompts, text prompts, or both. To enhance model training, we also propose a suite of useful strategies such as mix-modal prompted training and dynamic importance sampling. By testing over 22 test sets with seen or unseen objects, our single GKDT model shows strong performance and generality in detecting keypoints on broad categories, with most categories over 90% PCK@0.1 accuracy, offering high practical applicability to real-world problems. The dataset, models, and codes will be released at https://github.com/AlanLuSun/General-Keypoint-Detection.

Abstract

Explore similar work

GKDT: General Keypoint Detection Transformer

DPNeXt: A Lightweight Multi-Scale Feature Fusion Framework for Efficient ViT-Based Multi-Task Dense Prediction

MUSE: Unlocking Timestep as Native Task Steering for One-Step Dense Prediction