Deconstructing Multi-Task Active Learning: The Paradox of Gradient Conflict and Orthogonal Decomposition

Juho Jung; Seongjae Kang; Jinyoung Han

cs.LGOct 16, 2026

Deconstructing Multi-Task Active Learning: The Paradox of Gradient Conflict and Orthogonal Decomposition

Authors: Juho Jung, Seongjae Kang, Jinyoung Han

Organizations: Sungkyunkwan University · KAIST

Abstract

Existing multi-task active learning (MTL-AL) methods predominantly rely on linear scalarization of heterogeneous objectives such as uncertainty, diversity, and gradient conflict. We argue that this entangled formulation is fundamentally limited, as linear combination fails to capture Pareto-optimal trade-offs under conflicting objectives. We propose a shift from combination to deconstruction and introduce a principled framework that rethinks MTL-AL along three orthogonal dimensions. First, we identify the Conflict Paradox: although the optimization literature treats gradient conflict as interference to be suppressed, penalizing conflict when ranking candidates is measurably worse than not scoring it at all. We therefore decouple the two concerns: the acquisition function does not score conflict, and the resulting interference is resolved by gradient surgery during training. Second, we introduce Orthogonal Decomposition, separating acquisition into Informativeness and Exploration. This two-stage process, informativeness filtering followed by coverage maximization, prevents "diversity dilution" by excluding uninformative outliers before enforcing diversity. Third, we propose Cross-Task Mutual Information (CTMI), a covariance-based statistic of cross-task dependency computed from MC-dropout task entropies. Extensive experiments on COCO, Cityscapes, and NYUv2, including comparisons against recent MTL-AL and data-selection baselines, demonstrate consistent improvements over state-of-the-art methods, highlighting that structural rethinking of acquisition yields larger gains than incremental heuristic design. Code is available at https://github.com/Juho-Jung/multitask_active_learning_official.

Figures & tables

Figure 1: Visualizing the Conflict Paradox. Retrospective oracle analysis on 1,000 COCO benchmark samples using a multi-task model pretrained on a 1% seed set . Gradient alignment (cosine similarity, x-axis) is negatively correlated with actual loss reduction after training (y-axis) under AdamW with PCGrad off, ruling out the resolution mechanism as a confound: high-conflict samples yield larger information gain once basic features are learned.

Figure 2: Overview of the Orthogonal Decomposition Selection (ODS) Framework. (Left) The Conflict Paradox: Conflict-aware selection treats gradient conflict as an informative signal. (Middle) Orthogonal Decomposition: Acquisition is decomposed into Informativeness Filtering (Hardness and CTMI) followed by Coverage Maximization within the retained candidates. (Right) Decoupled Optimization: Whatever gradient interference the selected batch contains is resolved at training time by PCGrad .

Round	1	5	10	20	30	40	49
Pearson $r$	$-0.18$	$-0.11$	$-0.18$	$-0.09$	$-0.12$	$-0.02$	$-0.09$
Spearman $\rho$	$-0.15$	$-0.07$	$-0.17$	$-0.07$	$-0.09$	$+0.00$	$-0.07$

Table 1: Correlation between Informativeness

\mathcal{I}

and Exploration

\mathcal{E}

on COCO (1,500 unlabeled samples). Both Pearson

r

and Spearman

\rho

stay near zero across the full 50-round trajectory and decay toward zero as the model converges, supporting Assumption 1 .

Figure 3: The Orthogonal Decomposition Framework. Instead of a single weighted sum, we process objectives sequentially. Stage 1 projects data onto the High-Informativeness Subspace ( $\mathcal{M}_{\mathcal{I}}$ ) (filtering out easy/redundant samples). Stage 2 maximizes coverage within this subspace, preventing “ Diversity Dilution .”

Uncertainty-based
Method	Cls Acc (%)	Det mAP	Seg mIoU	Best MTL Score ( $\Delta m\%$ ) $\uparrow$	Last MTL Score ( $\Delta m\%$ ) $\uparrow$	Avg Rank $\downarrow$
Random Sampling	38.42 $\pm$ 0.15	25.21 $\pm$ 0.20	15.10 $\pm$ 0.35	0.000 (-)	0.000 (-)	6.33
Entropy Sampling [ 37 ]	38.30 $\pm$ 0.12	25.13 $\pm$ 0.18	14.97 $\pm$ 0.22	0.000 (-)	-0.005 (-0.5%)	9.67
Margin	38.25 $\pm$ 0.15	25.16 $\pm$ 0.21	15.06 $\pm$ 0.25	0.000 (-)	-0.003 (-0.3%)	8.67
Learning Loss [ 50 ]	38.36 $\pm$ 0.15	25.18 $\pm$ 0.21	15.05 $\pm$ 0.25	0.045 (+4.5%)	0.038 (+3.8%)	7.67
Diversity-based

Table 2: Main Results on COCO Multi-Task Benchmark (Round 30,

3

seeds:

\{42,123,456\}

). All methods use the same shared MTL backbone (~2.40M params) and identical training protocol; only the acquisition function differs. We report mean

\pm

std on the constituent tasks, the Best and Last MTL Composite Score, and the Average Rank across tasks. Note that Hekimoglu et al. [17] additionally trains an acquisition-only autoencoder (~1.04M params per round); all other methods incur no extra trainable module.

Figure 4: Evolution of MTL Composite Score ( $\Delta m\%$ ) on COCO. The plot visualizes the relative improvement over the Random Sampling baseline ( $y=0$ ). Our method (ODS) consistently leads performance throughout training. In contrast, standard baselines like Entropy and CoreSet fall into the negative gain zone ( $y<0$ ), failing to outperform random sampling.

Method	Best $\Delta m$ (%) per seed			Best $\Delta m$ (%)	Trajectory $\Delta m$ (%)
	42	123	456	mean $\pm$ std	mean $\pm$ std
FASS [ 45 ] (adapted)	$+6.00$	$+3.17$	$+2.75$	$3.97\pm 1.44$	$-0.15\pm 0.12$
Cluster-Margin [ 8 ] (adapted)	$+5.70$	$+2.27$	$+2.50$	$3.49\pm 1.57$	$+0.32\pm 0.10$
PPAL [ 49 ] (adapted)	$+5.98$	$+1.68$	$+1.11$	$2.92\pm 2.17$	$+0.08\pm 0.52$
$\mathbb{D}^{2}$ -Pruning [ 25 ] (adapted)	$+7.12$	$+1.55$	$+1.64$	$3.44\pm 2.61$	$+0.09\pm 0.02$
SES [ 48 ] (adapted)	$+7.42$	$+0.91$	$+4.12$	$4.15\pm 2.66$	$+0.41\pm 0.62$

Table 3: Adapted multi-stage baselines under a unified protocol (COCO, seeds

\{42,123,456\}

, 31 rounds).

\Delta m

is measured per seed against the same-seed Random run; Best is the peak over rounds and Trajectory the mean over all rounds (area under the label-efficiency curve). This protocol differs from Table 2 , which reports the original single-protocol comparison against classical baselines; the two sets of numbers are not interchangeable.

ID	Strategy	Selection	Optimization	MTL Comp.
(A)	Penalize Conflict	Unified ( $1-\|C_{g}\|$ )	AdamW	0.0589
(B)	Ignore Conflict	Unified (No $C_{g}$ )	AdamW	0.0756
(C)	Embrace (No Stage 2)	Embrace	PCGrad	0.0932
(D)	Ours (Full)	Embrace + 2-Stage	PCGrad	0.105

Table 4: Verifying the Conflict Paradox. Comparing conflict handling strategies. (A) Penalizing conflict (conventional wisdom) yields the worst performance. (B) Simply ignoring it is better. (C) Embracing high-conflict samples without Stage 2 already surpasses both. (D) The full ODS pipeline (embrace + Stage 2 K-Center) achieves SOTA, isolating the contribution of structural diversity on top of conflict-aware selection.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Protocol	$m$	$K$	$\alpha$	$\gamma$	Results
Original	10	20	1.8	0.7	Table 2 , Table 4
Unified (multi-seed)	15	15	2.5	0.5	Table 3

Appendix

Table 5: Stage-1 settings per protocol.

m

is the pool factor,

K

the number of MC-dropout passes, and

(\alpha,\gamma)

the fixed informativeness weights.

Single-Task Active Learning (Baselines)
Method	Uncertainty	Diversity	Gradient Conflict	Inter-Task MI	MTL-Specific
Random Sampling	✗	✗	✗	✗	✗
Entropy Sampling [ 37 ]	✓	✗	✗	✗	✗
CoreSet [ 34 ]	✗	✓	✗	✗	✗
BADGE [ 2 ]	✓	✓	✗	✗	✗
Reliable AL (Influence) [ 46 ]	✓	✗	✗	✗	✗

Appendix

Table 6: Comprehensive Comparison of Related Methodologies. We categorize all relevant methods by their consideration of key factors. Note that while many methods address Multi-Task Learning (MTL) or Data Efficiency, most do not satisfy the specific requirements of Multi-Task Active Learning (MTL-AL) for vision tasks. “(training)” denotes methods that handle conflict only during gradient updates, not selection. “(pruning)” denotes data pruning on labeled datasets, distinct from active learning.

Round	1	5	10	15	20	25	30	35	40	45	49
Pearson $r$	$-0.179$	$-0.106$	$-0.180$	$-0.121$	$-0.087$	$-0.079$	$-0.124$	$-0.088$	$-0.020$	$-0.034$	$-0.089$
Spearman $\rho$	$-0.148$	$-0.069$	$-0.174$	$-0.111$	$-0.072$	$-0.060$	$-0.089$	$-0.070$	$+0.001$	$-0.018$	$-0.065$

Appendix

Table 7: Per-round correlation of

\mathcal{I}

and

\mathcal{E}

on COCO (1,500 unlabeled images). The persistent negative sign in early/mid rounds is exactly why linear scalarization fails: the two scores trade off, so a weighted sum slides into the diversity-dilution regime. The two-stage decomposition avoids this by structurally restricting Stage 2 to high-

\mathcal{I}

candidates.

Round	Task	$W$	Skew	Kurtosis
1	Cls	0.979	$-0.02$	$-0.88$
1	Det	0.898	$-1.72$	$+6.56$
1	Seg	0.963	$-0.68$	$\phantom{+}0.00$
10	Cls	0.979	$+0.86$	$-0.10$
10	Det	0.908	$-1.65$	$+6.74$
10	Seg	0.990	$-0.20$	$-0.53$

Appendix

Table 8: Shapiro-Wilk

W

statistic, skewness, and excess kurtosis of

z

-score normalized per-task entropies on COCO.

W

closer to 1.0 indicates a closer fit to a Gaussian distribution. Segmentation entropies converge to nearly perfect Gaussianity; classification develops a slight right tail; detection exhibits heavier tails but maintains

W>0.90

throughout.

Rd	Pair	Pearson	Spearman	dCorr	$\big\|\|r\|{-}\text{dCorr}\big\|$
1	Cls-Det	$+0.515$	$+0.517$	0.488	0.027
1	Cls-Seg	$+0.264$	$+0.264$	0.245	0.019
1	Det-Seg	$+0.448$	$+0.447$	0.422	0.026
10	Cls-Det	$+0.338$	$+0.367$	0.330	0.008
10	Det-Seg	$+0.205$	$+0.200$	0.188	0.017
29	Cls-Det	$+0.412$	$+0.483$	0.406	0.006

Appendix

Table 9: Pearson vs. Spearman vs. distance correlation of per-task entropies (COCO).

|\,\text{Pearson}|

and dCorr agree within

\leq\!0.03

at every round, demonstrating that task-uncertainty dependencies in MTL are predominantly linear. Consequently, CTMI loses negligible signal by relying on covariance.

Dataset	Method	Task 1	Task 2	Task 3	Best MTL Score
COCO (Tasks: Cls / Det / Seg)
		Cls (Acc) $\uparrow$	Det (mAP) $\uparrow$	Seg (mIoU) $\uparrow$	Comp. ( $\Delta\%$ )
COCO	Random	38.4	25.2	15.1	0.0%
	BatchBALD	39.4	25.4	16.4	+4.0%
	MT-BADGE	41.2	25.8	16.8	+7.0%
	ODS	42.1	26.5	17.4	+10.0%

Appendix

Table 10: Detailed Cross-Domain Performance (Round 30). (

\uparrow

: higher is better,

\downarrow

: lower is better.)

Method	Cls Acc $\uparrow$	Det Acc $\uparrow$	Seg Dice $\uparrow$	Seg mIoU $\uparrow$
Random	0.505 $\pm$ .011	0.372 $\pm$ .011	0.316 $\pm$ .021	0.274 $\pm$ .029
CoreSet	0.508 $\pm$ .006	0.376 $\pm$ .012	0.315 $\pm$ .019	0.272 $\pm$ .027
Uncertainty	0.511 $\pm$ .006	0.373 $\pm$ .013	0.316 $\pm$ .028	0.276 $\pm$ .037
Diversity	0.504 $\pm$ .010	0.372 $\pm$ .003	0.313 $\pm$ .014	0.273 $\pm$ .023
ODS (Ours)	0.521 $\pm$ .010	0.377 $\pm$ .004	0.326 $\pm$ .022	0.289 $\pm$ .029

Appendix

Table 11: COCO with ImageNet-pretrained ResNet-50 (3 seeds). ODS retains rank 1 across individual tasks; gaps narrow as expected when using a strong pretrained backbone.

Method	Mean	Median
Random	225	204
Uncertainty	226	197
CoreSet	205	165
BADGE	655	636
ODS	995	1114

Appendix

Table 12: Per-round runtime on COCO (seconds). “Acquisition” isolates the cost of the AL selection step (excluding training).

Scenario A: Standard Optimization
Strategy	Optimizer	Best Score ( $\Delta m\%$ )
Random	AdamW	—
BatchBALD	AdamW	0.0311 (+3.1%)
MT-BADGE	AdamW	0.0516 (+5.2%)
ODS (Ours)	AdamW	0.0816 ( +8.2% )
Scenario B: Gradient Surgery

Appendix

Table 13: PCGrad ablation on COCO. ODS significantly outperforms all baselines regardless of the underlying optimizer, while PCGrad provides consistent supplementary gains.

Configuration	Hard.	CTMI	K-C	MTL Comp.
H only (no KC)	✓	✗	✗	0.087
CTMI only (no KC)	✗	✓	✗	0.064
H+CTMI (no KC)	✓	✓	✗	0.093
H only + KC	✓	✗	✓	0.094
CTMI only + KC	✗	✓	✓	0.082
Full ODS	✓	✓	✓	0.105

Appendix

Table 14: Component-wise ablation. The full ODS dominates each leave-one-out variant. Removing K-Center Stage 2 (“no KC”) causes the largest single-component drop, confirming that the structural decoupling is the dominant source of gain.

Rule	Form	Final MTL Comp.
Weighted sum (default)	$\alpha H+\gamma\,C$	$0.723\pm 0.016$
Rank-sum	$\operatorname{rank}(H)+\operatorname{rank}(C)$	$0.731\pm 0.001$
$z$ -score sum	$z(H)+z(C)$	$0.731\pm 0.001$
Product	$H\cdot C$	$0.726\pm 0.008$

Appendix

Table 15: Combination-rule ablation (COCO, 3 seeds:

\{42,123,456\}

). Final MTL Composite (mean

\pm

std). All four rules perform within

0.008

of each other—well within seed-to-seed variation—indicating no statistically distinguishable difference.

Combination strategy	Cls Acc	Det Acc	Seg Dice	MTL Comp.
Linear: $(1\!-\!\delta)(\alpha H{+}\gamma\mathrm{CTMI})+\delta\,\mathrm{Div}$	38.8	24.00	11.24	0.04
ODS (two-stage)	42.14	26.52	17.44	0.105
$\Delta$	$+3.34$	$+2.52$	$\mathbf{+6.2}$	$+0.065$

Appendix

Table 16: Linear scalarization vs. ODS, identical scores.

Table 21

$(\alpha,\gamma)$	$(0,1)$	$(0.5,1)$	$(1,0)$	$(1,0.5)$	$(1,1)$	$(1,2)$	$(2,1)$
MTL Comp.	0.082	0.099	0.094	0.102	0.105	0.103	0.104

Appendix

Table 19: Sensitivity of ODS to

(\alpha,\gamma)

on COCO. Equal weighting (

\alpha=\gamma=1

) is near-optimal and robust to a

4\times

swing in either weight.

Figure 5: t-SNE Distributional Analysis across Active Learning Rounds (1–8). We visualize the embedding space of selected samples from different acquisition strategies, overlaid on the unlabeled pool (gray). Red dashed circles highlight regions of redundant selection by baselines. (Baseline Inefficiency): Diversity-driven methods such as CoreSet and BADGE repeatedly select samples concentrated within narrow, low-density regions, leading to redundant over-sampling of visual outliers (e.g., isolated bottom clusters in R2, R3, R5, and R7). (Controlled Exploration by ODS): Although ODS ( $\color[rgb]{1,0,0}\bullet$ ) also explores these peripheral regions to ensure coverage, it does so sparsely , avoiding the clustering behavior observed in baselines. (Balanced Allocation): As a result, ODS allocates most of its acquisition budget to the high-density informative manifold , achieving a more effective balance between exploration of outliers and exploitation of the main data distribution.

Figure 6: Qualitative Comparison of Selected Samples (COCO, Rounds 1–2). Each row shows images selected by a different acquisition strategy. Red boxes mark redundant selections by baselines, where visually or semantically similar samples are repeatedly acquired within the same round. In contrast, ODS avoids such early-stage redundancy , allocating its budget to more diverse and complementary visual content.

Figure 7: Qualitative Comparison of Selected Samples (COCO, Rounds 3–4). Red boxes highlight structural and textural redundancies among samples selected by baseline methods, such as repeated acquisition of near-identical scenes or objects. Despite operating at later rounds with a more informative pool, baselines continue to exhibit inefficient budget usage, whereas ODS maintains diverse coverage without collapsing onto narrow visual patterns .

Figure 8: Annotation Density Analysis (COCO, Rounds 1 & 2). We visualize the ground-truth bounding boxes and segmentation masks for selected samples. ODS (Top Row): consistently selects images with high instance density (e.g., the cluttered desk in R1, the crowd in R2), providing rich supervision for multi-task learning. Baselines (Bottom Rows): frequent failure modes include selecting “empty” images with sparse or no annotations (e.g., Entropy selecting less informative rooms in R2, Random selecting simple single-object images). This confirms that ODS maximizes information gain per pixel.

Figure 9: Annotation Density Analysis (COCO, Rounds 3 & 4). Overlays reveal the semantic complexity of selected samples. ODS (Top Rows): selects structurally complex scenes with overlapping objects, forcing the model to resolve difficult occlusions. Baselines: CoreSet and BatchBALD often select images with large, simple textures or single large objects, which offer limited geometric challenges compared to the dense scenes selected by ODS.

Active Learning

Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.

See all 280 papers

Information-Theoretic Requirements for Gradient-Based Task Affinity Estimation in Multi-Task Learning
Apr 9, 2026Jasper Zhang, Bryan ChengMulti-Task Learning Task Transferability Estimation
Great Neck South High School
CORE-MTL: Rethinking Gradient Balancing via Causal Orthogonal Representations
Jun 1, 2026Chengfeng Wu, Tao Zou, Yanru Wu +1Disentangled Representation Learning Causal Representation Learning
Shenzhen Key Laboratory of Ubiquitous Data Enabling, Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · State Key Laboratory of Complex & Critical Software Environment, Beihang University, Beijing, China
Gradient-Discrepancy Acquisition for Pool-Based Active Learning
May 4, 2026Mohamadsadegh Khosravani, Sandra ZillesActive Learning
Department of Computer Science University of Regina · University of Regina

Apr 9, 2026J/K move · Enter open · S save

Information-Theoretic Requirements for Gradient-Based Task Affinity Estimation in Multi-Task Learning

Jasper Zhang, Bryan Cheng

Great Neck South High School

Multi-Task Learning Task Transferability Estimation

Open paper AskPDF arXiv

Multi-task learning shows strikingly inconsistent results -- sometimes joint training helps substantially, sometimes it actively harms performance -- yet the field lacks a principled framework for predicting these outcomes. We identify a fundamental but unstated assumption underlying gradient-based task analysis: tasks must share training instances for gradient conflicts to reveal genuine relationships. When tasks are measured on the same inputs, gradient alignment reflects shared mechanistic structure; when measured on disjoint inputs, any apparent signal conflates task relationships with distributional shift. We discover this sample overlap requirement exhibits a sharp phase transition: below 30% overlap, gradient-task correlations are statistically indistinguishable from noise; above 40%, they reliably recover known biological structure. Comprehensive validation across multiple datasets achieves strong correlations and recovers biological pathway organization. Standard benchmarks systematically violate this requirement -- MoleculeNet operates at <5% overlap, TDC at 8-14% -- far below the threshold where gradient analysis becomes meaningful. This provides the first principled explanation for seven years of inconsistent MTL results.

Abstract

Active Learning

Explore similar work

Information-Theoretic Requirements for Gradient-Based Task Affinity Estimation in Multi-Task Learning

CORE-MTL: Rethinking Gradient Balancing via Causal Orthogonal Representations

Gradient-Discrepancy Acquisition for Pool-Based Active Learning