Frequency Switching Mechanism for Parameter-E!cient Multi-Task Learning

Shih-Wen Liu; Yen-Chang Chen; Wei-Ta Chu; Fu-En Yang; Yu-Chiang Frank Wang

cs.CVMar 22, 2026

Frequency Switching Mechanism for Parameter-E!cient Multi-Task Learning

Authors: Shih-Wen Liu, Yen-Chang Chen, Wei-Ta Chu, Fu-En Yang, Yu-Chiang Frank Wang

Organizations: National Cheng Kung University, Tainan, Taiwan · NVIDIA Research, Taiwan

Abstract

Multi-task learning (MTL) aims to enable a single model to solve multiple tasks efficiently; however, current parameter-efficient fine-tuning (PEFT) methods remain largely limited to single-task adaptation. We introduce Free Sinewich, a parameter-efficient multi-task learning framework that enables near-zero-cost weight modulation via frequency switching (Free). Specifically, a Sine-AWB (Sinewich) layer combines low-rank factors and convolutional priors into a single kernel, which is then modulated elementwise by a sinusoidal transformation to produce task-specialized weights. A lightweight Clock Net is introduced to produce bounded frequencies that stabilize this modulation during training. Theoretically, sine modulation enhances the rank of low-rank adapters, while frequency separation decorrelates the weights of different tasks. On dense prediction benchmarks, Free Sinewich achieves state-of-the-art performance-efficiency trade-offs (e.g., up to +5.39% improvement over single-task fine-tuning with only 6.53M trainable parameters), offering a compact and scalable paradigm based on frequency-based parameter sharing. Project page: https://casperliuliuliu.github.io/projects/Free-Sinewich.

Figures & tables

Figure 1: Overall comparison of different PEFT-MTL methods. $r$ denotes the rank for the low-rank decomposition modules.

Panel of Figure 2 — Figure 2: Illustration of frequency switching. (a) The base matrix space $X$ represents a shared parameter substrate reused across tasks. Each task-specific matrix space $\mathcal{M}_{t}$ denotes the domain where the transformed matrix resides. Different sine transformations with task-dependent frequencies $\omega_{t}$ map the same base matrix into $M_{t}$ , producing task-specific matrices with distinct properties. (b) Each $\omega_{t}$ defines a unique sine wave, inducing a nonlinear mapping $\mathcal{F}_{\omega_{t}}$ that transforms the shared base matrix into task-specific representations elementwise. This illustrates how varying $\omega_{t}$ enables task specialization without parameter duplication.

Panel of Figure 3 — Figure 3: Overview of the Free Sinewich framework: (a) A Swin Transformer Tiny [ 26 ] serves as the shared encoder. The encoder receives image patch tokens with task tokens prepended to them. Following the VPT-shallow strategy [ 17 ] , the task tokens are introduced only at the first Transformer stage. Task-agnostic features are extracted through the task-agnostic module (TA-Module) in all Transformer blocks except for the last block, while the last block employs a task-specific module (TS-Module) to capture task-dependent representations. (b) Details of the TA-Module, which are basically the same as LoRA. (c) The TS-Module includes a lightweight Clock Net (LCN) that takes task token $\boldsymbol{p}_{t}$ as input and determines a task frequency $\omega_{t}$ . This frequency is used in the sine transformation to enhance the representational power of the low-rank adapter. Conceptually, we switch the shared base matrix $M_{\mathsf{AWB}}$ into different transient, task-specialized matrices $M_{t}$ ’s (ghost icon). The transient $M_{t}$ ’s are instantiated on-the-fly to extract task-specific features, enabling efficient and scalable PEFT-MTL.

Method	SemSeg	Human Parts	Saliency	Normals	$\Delta m$ (%)	Trainable
Method	(mIoU ↑)	(mIoU ↑)	(mIoU ↑)	(rmse ↓)	$\Delta m$ (%)	Parameters (M)
Single Task	67.21	61.93	62.35	17.97	0	112.62
MTL - Tuning Decoders Only	65.09	53.48	57.46	20.69	-9.95	1.94
MTL - Full Fine Tuning	67.56	60.24	65.21	16.64	+2.23	30.06
Adapter [ 13 ]	69.21	57.38	61.28	18.83	-2.71	11.24
Bitfit [ 4 ]	68.57	55.99	60.64	19.42	-4.60	2.85

Table 1: Comparison of parameter-efficient multi-task learning methods on the PASCAL-Context benchmark. Using Swin Transformer Tiny pretrained on the ImageNet-1K dataset as the backbone,

\Delta m

denotes the average relative improvement over the single-task baseline (positive indicates better), and

\uparrow

\downarrow

indicate higher / lower is better. Across all ranks, Free Sinewich delivers the best trade-off between performance and efficiency. * means the method uses the weights of Swin Transformer Tiny pretrained on the ImageNet-22k [ 7 ] dataset.

Method	SemSeg	Depth	Edge	Normals	$\Delta m$ (%)	Trainable
Method	(mIoU ↑)	(rmse ↓)	(odsF ↑)	(rmse ↓)	$\Delta m$ (%)	Parameters (M)
Single Task	42.59	66.08	59.80	22.58	0	112.64
MTL - Tuning Decoders Only	34.34	84.47	57.40	32.93	-24.26	1.96
MTL - Full Fine Tuning	42.07	65.36	58.80	23.74	-1.73	30.08
MTLoRA [ 1 ] ( $r=64$ )	41.11	65.49	58.30	24.55	-3.45	8.36
DiTASK [ 27 ] - MTL	37.36	75.00	57.10	28.63	-14.27	3.57

Table 2: Performance comparison on the NYUDv2 [ 28 ] dataset. All methods utilize the Swin Transformer Tiny [ 26 ] backbone pre-trained on the ImageNet-22K [ 7 ] dataset, with HRNet as the decoder for fair comparison.

Method	SemSeg	Human Parts	Saliency	Normals	$\Delta m$ (%)	Trainable
Method	(mIoU ↑)	(mIoU ↑)	(mIoU ↑)	(rmse ↓)	$\Delta m$ (%)	Parameters (M)
Free Sinewich (Ours)	71.25	61.38	66.24	16.14	+5.39	6.53
Ours w/o LCN	70.83	61.37	66.09	16.17	+5.12	6.51
Ours w/o Low-pass filter	70.95	61.33	65.44	16.22	+4.82	6.53
Ours w/o Sine	69.68	60.69	64.91	16.37	+3.67	6.53

Table 3: Performance variations when different components are used. Removing key components such as the LCN, the low-pass filter, or the sine transformation leads to performance degradation, confirming that each element contributes to overall performance and efficiency.

Method	Decoder	SemSeg	Human Parts	Saliency	Normals	$\Delta m$ (%)	Trainable Param. (M)
Method	Decoder	(mIoU ↑)	(mIoU ↑)	(mIoU ↑)	(rmse ↓)	$\Delta m$ (%)	Decoder / All
TADFormer [ 3 ]	HRNet [ 33 ]	72.05	61.60	65.45	16.70	+4.67	1.94 / 4.78
	SegFormer [ 38 ]	72.33	61.16	65.80	16.87	+4.51	2.08 / 4.91
	ASPP [ 5 ]	73.66	60.37	65.27	16.43	+5.09	12.44 / 15.27
Free Sinewich	HRNet [ 33 ]	72.65	62.57	65.55	16.36	+5.80	1.07 / 4.04
	SegFormer [ 38 ]	72.68	62.67	66.16	16.53	+5.86	1.08 / 4.05

Table 4: Performance variations when different configurations are used for the decoder. All methods use the Swin Transformer Tiny backbone pre-trained on the ImageNet-22K dataset, with rank

r{=}32

for a fair comparison on PASCAL-Context dataset.

Method	SemSeg	Human Parts	Saliency	Normals	$\Delta m$ (%)	Trainable
Method	(mIoU ↑)	(mIoU ↑)	(mIoU ↑)	(rmse ↓)	$\Delta m$ (%)	Parameters (M)
Shared Base	71.25	61.38	66.24	16.14	+5.39	6.53
Independent Base	70.81	61.56	65.42	16.09	+5.03	10.22
Independent Decoder	70.91	61.57	66.03	16.10	+5.31	7.41

Table 5: Performance variations when different sharing schemes are employed. Performance comparison among the shared base (Free Sinewich), independent base, and independent decoder variants on the PASCAL-Context dataset. Sharing a single base matrix across tasks consistently improves the overall performance with fewer parameters.

Figure 4: Visual comparison of semantic segmentation on the Pascal-Context dataset. The Free Sinewich method yields fuller and more coherent segmentation masks, capturing finer-grained boundary details than TADFormer.

Panel of Figure 5 — Figure 5: Effect of backbone pretraining and model capacity on Free Sinewich. (a) Comparison between Swin-T pretrained on ImageNet-1K and ImageNet-22K. (b) Performance comparison between Swin-T and Swin-B backbones (both pretrained on ImageNet-22K).

Sweep over kernel size $K$ (fixed $\sigma=1.0$ )
Kernel	Standard	SemSeg	Human Parts	Saliency	Normals	$\Delta m$ (%)
Size $k$	Deviation $\sigma$	(mIoU ↑)	(mIoU ↑)	(mIoU ↑)	(rmse ↓)	$\Delta m$ (%)
$K=5$	$\sigma=1.0$	71.20	61.37	66.16	16.15	+5.32
$K=7$	$\sigma=1.0$	71.25	61.38	66.24	16.14	+5.39
$K=9$	$\sigma=1.0$	71.06	61.46	66.07	16.11	+5.32
Sweep over $\sigma$ (fixed $K=7$ )

Table 6: Ablation over low-pass filter hyperparameters. Effect of kernel size

K

and standard deviation

\sigma

on performance.

Figure 6: Pairwise gradient cosine similarity measured on the shared LoRA base matrix across training epochs (x-axis). Our method maintains near-orthogonal inter-task gradients with reduced variance and lower conflict rate compared to TADFormer.

Method	SemSeg	Depth	$\Delta m$ (%)	Trainable
Method	(mIoU ↑)	(rmse ↓)	$\Delta m$ (%)	Param. (M)
Single Task	63.98	5.91	0	56.00
MTL - Dec	53.30	8.54	-30.59	0.97
MTL - Full	61.03	5.76	-1.03	28.49
DiTASK [ 27 ]	56.08	6.35	-9.89	2.58
MTLoRA [ 1 ]	60.88	5.84	-1.83	7.23

Table 7: Results on the Cityscapes dataset. All methods use a Swin-Tiny backbone [ 26 ] pretrained on ImageNet-22K [ 7 ] with an HRNet decoder. MTL-Dec denotes training the decoder only, while MTL-Full denotes full fine-tuning.

\Delta m

denotes the average relative improvement over the single-task baseline.

Multi-Task Learning

Multi-task learning trains models to solve related tasks together by sharing representations, parameters, or supervision.

See all 377 papers

FAAR: Efficient Frequency-Aware Multi-Task Fine-Tuning via Automatic Rank Selection
Mar 20, 2026Maxime Fontana, Michael Spratling, Miaojing ShiNonlinear LoRA Parameter-Efficient Fine-Tuning
King’s College London London, UK · University of Luxembourg Esch-sur-Alzette, Luxembourg · Tongji University Shanghai, China
PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Prompts
May 13, 2026Anjir Ahmed Chowdhury, Syed Zawad, Xiaolong Ma +2LLM Fine-Tuning Multi-Task Learning
Department of Computer Science University of Houston · IBM Research · Argonne National Laboratory
From Isolation to Alignment: Unified LoRA for Efficient Multi-Task Learning
Aug 7, 2025Jinda Liu, Yi Chang, Yuan WuMulti-Task Learning Low-Rank Adaptation
School of Artificial Intelligence, Jilin University · 2Southern University of Science and Technology · 3International Center of Future Science, Jilin University +1

Mar 20, 2026J/K move · Enter open · S save

FAAR: Efficient Frequency-Aware Multi-Task Fine-Tuning via Automatic Rank Selection

Maxime Fontana, Michael Spratling, Miaojing Shi

King’s College London London, UK · University of Luxembourg Esch-sur-Alzette, Luxembourg · Tongji University Shanghai, China

Nonlinear LoRA Parameter-Efficient Fine-Tuning Multi-Task Learning Low-Rank Adaptation

Open paper AskPDF arXiv

Adapting models pre-trained on large-scale datasets is a proven way to reach strong performance quickly for down-stream tasks. However, the growth of state-of-the-art mod-els makes traditional full fine-tuning unsuitable and difficult, especially for multi-task learning (MTL) where cost scales with the number of tasks. As a result, recent studies investigate parameter-efficient fine-tuning (PEFT) using low-rank adaptation to significantly reduce the number of trainable parameters. However, these existing methods use a single, fixed rank, which may not be optimal for differ-ent tasks or positions in the MTL architecture. Moreover, these methods fail to learn spatial information that cap-tures inter-task relationships and helps to improve diverse task predictions. This paper introduces Frequency-Aware and Automatic Rank (FAAR) for efficient MTL fine-tuning. Our method introduces Performance-Driven Rank Shrink-ing (PDRS) to allocate the optimal rank per adapter location and per task. Moreover, by analyzing the image frequency spectrum, FAAR proposes a Task-Spectral Pyramidal Decoder (TS-PD) that injects input-specific context into spatial bias learning to better reflect cross-task relationships. Experiments performed on dense visual task benchmarks show the superiority of our method in terms of both accuracy and efficiency compared to other PEFT methods in MTL. FAAR reduces the number of parameters by up to 9 times compared to traditional MTL fine-tuning whilst improving overall performance. Our code is available.

Abstract

Multi-Task Learning

Explore similar work

FAAR: Efficient Frequency-Aware Multi-Task Fine-Tuning via Automatic Rank Selection

PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Prompts

From Isolation to Alignment: Unified LoRA for Efficient Multi-Task Learning