Multi-task Just Recognizable Difference for Video Coding for Machines: Database, Model, and Coding Application

Junqi Liu; Yun Zhang; Xiaoxia Huang; Long Xu; Weisi Lin

eess.IVApr 10, 2026

Multi-task Just Recognizable Difference for Video Coding for Machines: Database, Model, and Coding Application

Authors: Junqi Liu, Yun Zhang, Xiaoxia Huang, Long Xu, Weisi Lin

Organizations: the School of Electronics and Communication Engineering, Shenzhen Campus, Sun Yat-Sen University, Shenzhen 518107, China. · the National Astronomical Observatories, Chinese Academy of Sciences, Beijing, 100012, China. · the School of Computer Science and Engineering, Nanyang Technological University, Jurong West, Singapore

Abstract

Just Recognizable Difference (JRD) boosts coding efficiency for machine vision through visibility threshold modeling, but is currently limited to a single-task scenario. To address this issue, we propose a Multi-Task JRD (MT-JRD) dataset and an Attribute-assisted MT-JRD (AMT-JRD) model for Video Coding for Machines (VCM), enhancing both prediction accuracy and coding efficiency. First, we construct a dataset comprising 27,264 JRD annotations from machines, supporting three representative tasks including object detection, instance segmentation, and keypoint detection. Secondly, we propose the AMT-JRD prediction model, which integrates Generalized Feature Extraction Module (GFEM) and Specialized Feature Extraction Module (SFEM) to facilitate joint learning across multiple tasks. Thirdly, we innovatively incorporate object attribute information into object-wise JRD prediction through the Attribute Feature Fusion Module (AFFM), which introduces prior knowledge about object size and location. This design effectively compensates for the limitations of relying solely on image features and enhances the model's capacity to represent the perceptual mechanisms of machine vision. Finally, we apply the AMT-JRD model to VCM, where the accurately predicted JRDs are applied to reduce the coding bit rate while preserving accuracy across multiple machine vision tasks. Extensive experimental results demonstrate that AMT-JRD achieves precise and robust multi-task prediction with a mean absolute error of 3.781 and error variance of 5.332 across three tasks, outperforming the state-of-the-art single-task prediction model by 6.7% and 6.3%, respectively. Coding experiments further reveal that compared to the baseline VVC and JPEG, the AMT-JRD-based VCM improves an average of 3.861% and 7.886% Bjontegaard Delta-mean Average Precision (BD-mAP), respectively.

Figures & tables

Panel of Figure 1 — Fig. 1: Visualized examples of MT-JRD. (a) Performance degradation in multi-task scenarios. In subfigures (b)-(m), the first column displays the original image or the distorted image compressed with MT-JRDs; the second and third columns present the corresponding OD/IS and KPD results.

Fig. 2: Paradigm of MT-JRD including database, model, and coding application.

Fig. 3: The pipeline for constructing the MT-JRD dataset.

Panel of Figure 4 — Fig. 4: MT-JRD quantity distribution under different $T$ s. (a) MT-JRD distribution when $T=0.75$ , (b) OD JRD, (c) IS JRD, (d) KPD JRD.

Panel of Figure 5 — Fig. 5: Distortion distribution of the MT-JRD dataset.

Fig. 6: Relationship between object size and JRD.

Panel of Figure 7 — Fig. 7: Relationship between object location and JRD.(a) OD, (b) IS, (c) KPD

Panel of Figure 8 — Fig. 8: Architectures of MT-JRD prediction models. (a) Independent single-task architecture, (b) Unified GFEM-based architecture, (c) Joint GFEM-SFEM architecture.

Fig. 9: The proposed AMT-JRD model, which consists of GFEM, SFEM, AFFM, and multi-task classification heads.

Panel of Figure 10 — Fig. 10: The processing workflow for JRD-based VCM optimization. (a) JRD-based VCM [ 37 ] , (b) JRD-based VVC [ 37 ] , (c) JRD-based JPEG.

Model	OD			IS			KPD			Avg.
Model	$E_{A}$	$E_{[27,51]}$	$\sigma_{e}$	$E_{A}$	$E_{[27,51]}$	$\sigma_{e}$	$E_{A}$	$E_{[27,51]}$	$\sigma_{e}$	$E_{A}$	$E_{[27,51]}$	$\sigma_{e}$
EL-JRD (IJCV 21’) [ 35 ]	5.933	5.721	7.949	5.613	5.122	7.495	5.778	5.215	7.651	5.775	5.353	7.698
BC-JRD (TMM 23’) [ 36 ]	4.555	4.283	6.572	4.326	3.875	5.954	4.276	3.866	6.297	4.386	4.008	6.274
DT-JRD (TMM 25’) [ 37 ]	3.929	3.442	5.564	4.001	3.428	5.648	4.225	3.632	5.867	4.052	3.501	5.693
MT-JND (ICIP 23’) [ 14 ]	4.586	4.022	6.102	4.757	4.113	6.302	4.637	3.868	6.387	4.660	4.001	6.264
MJ-REC (TCSVT 24’) [ 16 ]	4.709	4.182	6.276	4.863	4.235	6.440	5.080	4.282	6.823	4.884	4.233	6.513

TABLE I: Average accuracy and standard deviation comparison between the proposed AMT-JRD and the state-of-the-art JRD models.

Model	EL-JRD [ 35 ]	BC-JRD [ 36 ]	DT-JRD [ 37 ]	MT-JND [ 14 ]	MJ-REC [ 16 ]	AMT-JRD
$\lvert\Delta$ PSNR $\rvert$ $\downarrow$	0.617 / 0.686 / 0.745	0.502 / 0.535 / 0.482	0.485 / 0.502 / 0.545	0.559 / 0.611 / 0.612	0.579 / 0.606 / 0.665	0.441 / 0.468 / 0.450
$\lvert\Delta$ SSIM $\rvert(\times 10^{-3})$ $\downarrow$	9.693 / 11.052 / 11.584	7.921 / 8.618 / 8.237	7.655 / 8.143 / 8.876	8.983 / 10.237 / 9.629	9.207 / 9.966 / 10.761	6.916 / 7.770 / 7.476
$\lvert\Delta$ LPIPS $\rvert(\times 10^{-3})$ $\downarrow$	5.297 / 5.851 / 5.761	4.413 / 4.532 / 4.402	4.382 / 4.339 / 4.440	5.184 / 5.484 / 4.673	5.278 / 5.203 / 5.310	4.176 / 4.151 / 4.099
$R^{2}$ of PSNR $\uparrow$	0.921 / 0.914 / 0.870	0.948 / 0.942 / 0.955	0.952 / 0.951 / 0.932	0.939 / 0.930 / 0.917	0.934 / 0.936 / 0.901	0.961 / 0.954 / 0.961
$R^{2}$ of SSIM $\uparrow$	0.953 / 0.935 / 0.918	0.971 / 0.957 / 0.960	0.971 / 0.963 / 0.952	0.959 / 0.945 / 0.951	0.959 / 0.963 / 0.935	0.978 / 0.967 / 0.967
$R^{2}$ of LPIPS $\uparrow$	0.898 / 0.868 / 0.876	0.932 / 0.919 / 0.925	0.931 / 0.927 / 0.923	0.904 / 0.892 / 0.932	0.900 / 0.900 / 0.901	0.945 / 0.936 / 0.936

TABLE II: Multi-task accuracy comparison in terms of differences and

R^{2}

of PSNR [dB], SSIM and LPIPS.

Architecture		OD			IS			KPD			Avg.
Architecture		$E_{A}$	$E_{[27,51]}$	$\sigma_{e}$	$E_{A}$	$E_{[27,51]}$	$\sigma_{e}$	$E_{A}$	$E_{[27,51]}$	$\sigma_{e}$	$E_{A}$	$E_{[27,51]}$	$\sigma_{e}$
Single-task	–	3.947	3.560	5.503	4.016	3.586	5.515	3.851	3.438	5.353	3.938	3.528	5.457
Multi-task	GFEM	3.957	3.564	5.530	3.994	3.525	5.462	4.017	3.561	5.512	3.989	3.550	5.501
	GFEM+SFEM	3.855	3.459	5.409	3.900	3.445	5.431	3.816	3.406	5.338	3.857	3.437	5.393
	GFEM+SFEM+AFFM	3.796	3.395	5.415	3.818	3.337	5.365	3.728	3.286	5.217	3.781	3.339	5.332

TABLE III: Accuracy comparison of MT-JRD components in three tasks.

Panel of Figure 14 — Fig. 11: Coding gain comparison among JRD models in VVC. (a) OD, (b) IS, (c) KPD.

Panel of Figure 15 — Fig. 12: Coding gain comparison among JRD models in JPEG. (a) OD, (b) IS, (c) KPD.

Base codec	Method	OD	IS	KPD	Avg.
VVC	EL-JRD	2.058	2.346	2.823	2.409
	BC-JRD	2.589	2.638	4.169	3.132
	DT-JRD	2.694	2.233	3.783	2.903
	MT-JND	0.883	1.310	3.117	1.770
	MJ-REC	1.190	2.303	3.195	2.229
	AMT-JRD	3.801	3.579	4.202	3.861

TABLE IV: BD-mAP gains of JRD-based VCM on different JRD models, tasks, and base codecs[Unit:%].

Task\Method	AMT-JRD			GT-JRD
Task\Method	OD	IS	KPD	OD	IS	KPD
OD	3.801	2.844	3.219	5.515	3.977	2.810
IS	3.088	3.579	3.574	4.815	5.951	3.665
KPD	3.715	3.580	4.202	4.047	4.096	4.917

TABLE V: BD-mAP of JRD-based VCM on cross-task validation[Unit:%].

Model	HVS	Multi-Task	Params (M)	FLOPs (G)	$E_{A}$
EL-JRD	✗	✗	484.200( $\times$ 3)	936.408( $\times$ 3)	5.775
BC-JRD	✗	✗	31.587 ( $\times$ 3)	989.184( $\times$ 3)	4.386
DT-JRD	✗	✓	305.655	15.264	4.052
MT-JND	✓	✓	34.082	30.752	4.660
MJ-REC	✓	✓	11.085	58.076	4.884
AMT-JRD (Ours)	✗	✓	77.374	9.934	3.781

TABLE VI: Comparison of JRD model on accuracy, size, and computational cost.

Fig. 13: Complexity and accuracy of JRD models on the MT-JRD dataset.

Panel of Figure 20 — Fig. 14: Visualized machine analytical results of compressed images from different JRD-based coding methods. The first, second, and third lines correspond to OD, IS, and KPD, respectively. (a)(i)(q) original images, (b)(j)(r) VVC all-intra, (c)(k)(s) EL-JRD-based VCM, (d)(l)(t) BC-JRD-based VCM, (e)(m)(u) DT-JRD-based VCM, (f)(n)(v) MT-JND-based VCM, (g)(o)(w) MJ-REC-based VCM, (h)(p)(x) AMT-JRD-based VCM. The bit rate, confidence, and similarity metric of the detected objects are listed as triples. (b) {0.2757, 0.9500, 0.8881}, (c) {0.2665, 0.9719, 0.9335}, (d) {0.2710, 0.9705, 0.9380}, (e) {0.2660, 0.9744, 0.9623}, (f) {0.2710, 0.9705, 0.9380}, (g) {0.2751, 0.9749, 0.9521}, (h) {0.2660, 0.9744, 0.9623}, (j) {0.2854, 0.9990, 0.9644}, (k) {0.2803, 0.9991, 0.9679}, (l) {0.2768, 0.9990, 0.9649}, (m) {0.2713, 0.9990, 0.9701}, (n) {0.2731, 0.9993, 0.9723}, (o) {0.2768, 0.9990, 0.9649}, (p) {0.2676, 0.9990, 0.9746}, (r) {0.4627, 0.9982, 0.9745}, (s) {0.4089, 0.9995, 0.7651}, (t) {0.4089, 0.9995, 0.7651}, (u) {0.3934, 0.9993, 0.9075}, (v) {0.4301, 0.9995, 0.9876}, (w) {0.4301, 0.9995, 0.9876}, (x) {0.4301, 0.9995, 0.9876}

Multi-Task Learning

Multi-task learning trains models to solve related tasks together by sharing representations, parameters, or supervision.

See all 391 papers

Apr 14, 2026

PAT-VCM: Plug-and-Play Auxiliary Tokens for Video Coding for Machines

Existing video coding for machines is often trained for a specific downstream task and model. As a result, the compressed representation becomes tightly coupled to the end task, making it difficult to scale across multiple tasks or adapt to model updates. We propose PAT-VCM, a plug-and-play auxiliary-token framework for video coding for machines. PAT-VCM keeps a shared baseline compressed stream and augments it with lightweight task-aware auxiliary tokens, allowing different downstream tasks to recover the information they need without retraining a separate codec for each task. The framework supports three forms of auxiliary information: visual residual tokens, prompt/control tokens, and semantic tokens. We evaluate PAT-VCM on segmentation, depth estimation, and semantic recognition. A shared detection-oriented auxiliary branch provides a reusable first refinement, task-specific visual branches improve segmentation and depth, prompt tokens provide further segmentation gains at negligible bitrate, and semantic tokens achieve strong recognition performance with extremely low overhead. These results suggest that a shared compressed representation, combined with lightweight task-aware auxiliary tokens, is a practical and scalable alternative to tightly task-coupled VCM design.

Wei Jiang, Wei Wang

Futurewei Technologies Inc. San Jose, CA 95131

Video Compression Neural Video Compression

Ask PDF

Apr 7, 2026

CI-ICM: Channel Importance-driven Learned Image Coding for Machines

Traditional human vision-centric image compression methods are suboptimal for machine vision centric compression due to different visual properties and feature characteristics. To address this problem, we propose a Channel Importance-driven learned Image Coding for Machines (CI-ICM), aiming to maximize the performance of machine vision tasks at a given bitrate constraint. First, we propose a Channel Importance Generation (CIG) module to quantify channel importance in machine vision and develop a channel order loss to rank channels in descending order. Second, to properly allocate bitrate among feature channels, we propose a Feature Channel Grouping and Scaling (FCGS) module that non-uniformly groups the feature channels based on their importance and adjusts the dynamic range of each group. Based on FCGS, we further propose a Channel Importance-based Context (CI-CTX) module to allocate bits among feature groups and to preserve higher fidelity in critical channels. Third, to adapt to multiple machine tasks, we propose a Task-Specific Channel Adaptation (TSCA) module to adaptively enhance features for multiple downstream machine tasks. Experimental results on the COCO2017 dataset show that the proposed CI-ICM achieves BD-mAP@50:95 gains of 16.25

\%

in object detection and 13.72

\%

in instance segmentation over the established baseline codec. Ablation studies validate the effectiveness of each contribution, and computation complexity analysis reveals the practicability of the CI-ICM. This work establishes feature channel optimization for machine vision-centric compression, bridging the gap between image coding and machine perception.

Yun Zhang, Junle Liu, Huan Zhang +3

School of Electronics and Communication Engineering, Shenzhen Campus, Sun Yat-Sen University, Shenzhen 518107, China. · School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China. · School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China. +2

Learned Image Compression

Ask PDF

Sep 28, 2026

AHMAD: Adaptive Hybrid Multi-task Vision Learning with Assisted Distillation for Keypoint Detection

Generalist multitasking vision models aim to unify multiple vision tasks within a single framework, enabling more efficient and versatile learning. However, handling diverse vision tasks -- spanning dense and sparse predictions -- remains challenging due to their inherently varying output structures. In this paper, we propose AHMAD, a simple yet effective framework for generalist multitask learning that integrates different key vision tasks: semantic segmentation, instance segmentation, depth estimation, keypoint detection, and object detection. Our approach incorporates these five tasks into a unified structure: a shared encoder-decoder with several lightweight task-specific projectors. Under the multitask learning paradigm, we observed a complementary performance gain, achieving a state-of-the-art PQ of 53.1 and an mIoU of 66.5 for COCO-val panoptic and semantic segmentation, respectively. Additionally, for top-down keypoint detection, which typically incurs high computational overhead due to multiple forward passes, we introduce a knowledge distillation-based method that enables a single forward pass over the entire image, greatly improving efficiency. Ultimately, our model delivers a lightweight yet effective generalist multitask learning framework, demonstrating strong performance across five vision tasks.

Mohammad Mahdi, Nedyalko Prisadnikov, Yuqian Fu +3

INSAIT, Sofia University “St. Kliment Ohridski”

Multi-Task Learning Keypoint Detection

Ask PDF

Abstract

Multi-Task Learning

Explore similar work

PAT-VCM: Plug-and-Play Auxiliary Tokens for Video Coding for Machines

CI-ICM: Channel Importance-driven Learned Image Coding for Machines

AHMAD: Adaptive Hybrid Multi-task Vision Learning with Assisted Distillation for Keypoint Detection