MTLSI-Net: A Linear Semantic Interaction Network for Parameter-Efficient Multi-Task Dense Prediction

Chen Liu; Hengyu Man; Xiaopeng Fan; Debin Zhao

cs.CVApr 2, 2026

MTLSI-Net: A Linear Semantic Interaction Network for Parameter-Efficient Multi-Task Dense Prediction

Authors: Chen Liu, Hengyu Man, Xiaopeng Fan, Debin Zhao

Organizations: Harbin Institute of Technology · Peng Cheng Laboratory · Harbin Institute of Technology SuZhou Research Institute

Abstract

Multi-task dense prediction aims to perform multiple pixel-level tasks simultaneously. However, capturing global cross-task interactions remains non-trivial due to the quadratic complexity of standard self-attention on high-resolution features. To address this limitation, we propose a Multi-Task Linear Semantic Interaction Network (MTLSI-Net), which facilitates cross-task interaction through linear attention. Specifically, MTLSI-Net incorporates three key components: a Multi-Task Multi-scale Query Linear Fusion Block, which captures cross-task dependencies across multiple scales with linear complexity using a shared global context matrix; a Semantic Token Distiller that compresses redundant features into compact semantic tokens, distilling essential cross-task knowledge; and a Cross-Window Integrated attention Block that injects global semantics into local features via a dual-branch architecture, preserving both global consistency and spatial precision. These components collectively enable the network to capture comprehensive cross-task interactions at linear complexity with reduced parameters. Extensive experiments on NYUDv2 and PASCAL-Context demonstrate that MTLSI-Net achieves state-of-the-art performance, validating its effectiveness and efficiency in multi-task learning.

Figures & tables

Fig. 1: Overview of the proposed coarse-to-fine multi-task learning framework. Given an input image, a shared backbone first extracts multi-scale features. Subsequently, preliminary decoders generate initial task-specific representations and coarse predictions. Next, to enable cross-task interaction, the MT-MQLFB processes the concatenated features and predictions. Finally, semantic decoders, incorporating the Semantic Token Distiller and Cross-Window Integrated Attention Block (CWIB), refine these features to produce fine-grained predictions for all tasks.

Fig. 2: Architecture of the Multi-task Multi-scale Query Linear Fusion Block.

Method	Backbone	Seg.(mIoU $\uparrow$ )	Depth(RMSE $\downarrow$ )	Normal(mErr $\downarrow$ )	Boundary(odsF $\uparrow$ )	Params(M)
PAD-Net [ 24 ]	HRNet-18	36.70	0.6264	20.85	76.50	5.02
PSD [ 35 ]	HRNet-18	36.69	0.6246	20.87	76.42	4.71
ATRC [ 2 ]	HRNet-18	38.90	0.6010	20.48	76.34	5.06
Ours	HRNet-18	42.89	0.5892	20.12	76.40	13.94
MQTransformer [ 26 ]	Swin-T	43.61	0.5979	20.05	76.20	32.91
DeMT [ 27 ]	Swin-T	46.36	0.5871	20.65	76.90	32.07

TABLE I: Performance comparison with SOTA methods using different backbones on NYUDv2.

Method	Seg.	Parsing	Sal.	Normal	Boundary	Params	FLOPs
Method	mIoU $\uparrow$	mIoU $\uparrow$	maxF $\uparrow$	mErr $\downarrow$	odsF $\uparrow$	M	G
PAD-Net [ 24 ]	78.01	67.12	79.21	14.37	72.60	330	773
MTI-Net [ 18 ]	78.31	67.40	84.75	14.67	73.00	851	774
ATRC [ 2 ]	77.11	66.84	81.20	14.23	72.10	340	871
MQTransformer [ 26 ]	78.93	67.41	83.58	14.21	73.90	-	-
TaskExpert [ 29 ]	80.64	69.42	84.87	13.56	73.30	420	622

TABLE II: Performance comparison with SOTA methods on PASCAL-Context.

Fig. 3: Ablation study on the number of semantic tokens $K$ on NYUDv2.

Dataset	scale	Seg	Depth	Normal	Boundary
Dataset	scale	mIoU $\uparrow$	RMSE $\downarrow$	mErr $\downarrow$	odsF $\uparrow$
NYUDv2 ( $K=1024$ )	1	44.17	0.6254	20.51	76.00
	3	44.10	0.5985	20.35	76.02
	5	43.85	0.6014	20.34	75.96
	(1,3,5)	46.03	0.5968	20.10	76.34

TABLE III: Ablation study on the effect of multi-scale query configurations in MT-MQLFB.

Fig. 4: Visual comparison on the NYUDv2 dataset.

Multi-Task Learning

Multi-task learning trains models to solve related tasks together by sharing representations, parameters, or supervision.

See all 391 papers

$\mathcal{B}^{3}$ -Net: Controlled Posterior Bridge Learning for Multi-Task Dense Prediction
May 7, 2026Meihua Zhou, Li YangMulti-Task Learning Dense Prediction
Wannan Medical University, Wuhu, China · University of Chinese Academy of Sciences, Beijing, China
Task-State Adaptation with Prototype Memory for Multi-Task Dense Prediction
Aug 28, 2026Yangyang Xu, Haobo Yuan, Yuzhu Wang +4Multi-Task Learning Vision Foundation Model Adaptation
Tsinghua University · University of California, Merced · Shanghai Jiao Tong University
DPNeXt: A Lightweight Multi-Scale Feature Fusion Framework for Efficient ViT-Based Multi-Task Dense Prediction
Jul 17, 2026Jehun Kang, Jungha Wang, Youngjun Hwang +1Multi-Task Learning Multi-Scale Feature Fusion
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea

May 7, 2026J/K move · Enter open · S save

$\mathcal{B}^{3}$ -Net: Controlled Posterior Bridge Learning for Multi-Task Dense Prediction

Meihua Zhou, Li Yang

Wannan Medical University, Wuhu, China · University of Chinese Academy of Sciences, Beijing, China

Multi-Task Learning Dense Prediction Negative Transfer

Open paper AskPDF arXiv

Multi-task dense prediction solves complementary pixel-level tasks in a unified model, such as semantic segmentation, depth estimation, surface normal estimation, and edge detection. Existing decoder-side interactions use attention, prompts, routing, diffusion, Mamba, or bridge features to exchange task evidence, but most of them organize this evidence implicitly. They usually fuse task features by similarity or affinity, without explicitly modeling that evidence reliability varies across tasks and spatial locations. As a result, unreliable evidence may contaminate the shared representation and intensify negative transfer. We propose

\mathcal{B}^{3}

-Net, a controlled posterior bridge learning framework for multi-task dense prediction. Our method decomposes decoder-side interaction into reliability estimation, posterior bridge construction, and bounded redistribution. The Precision Field Estimator estimates patch-wise evidence precision from task-reference alignment and local variation. The Posterior Bridge Operator builds a precision-weighted posterior bridge through heteroscedastic evidence fusion, yielding a shared state more reliable than uniform or heuristic mixtures. The Contractive Dispatch Operator redistributes the bridge to each task branch through a bounded update, reducing uncontrolled feature injection. Experiments on NYUD-v2, PASCAL-Context, and Cityscapes show that

\mathcal{B}^{3}

-Net achieves competitive or superior trade-offs over representative CNN-, Transformer-, diffusion-, Mamba-, and bridge-feature-based methods. Backbone-matched comparisons and extensive analyses further verify that the gains arise from controlled posterior bridge learning rather than backbone capacity or decoder scale.

Abstract

Multi-Task Learning

Explore similar work

B3\mathcal{B}^{3}B3-Net: Controlled Posterior Bridge Learning for Multi-Task Dense Prediction

Task-State Adaptation with Prototype Memory for Multi-Task Dense Prediction

DPNeXt: A Lightweight Multi-Scale Feature Fusion Framework for Efficient ViT-Based Multi-Task Dense Prediction

$\mathcal{B}^{3}$ -Net: Controlled Posterior Bridge Learning for Multi-Task Dense Prediction