cs.LGOct 12, 2026

The Bell Boundary of Shared Representations

Authors: Yang Wu, Hanyu Zhao, Yazhou Zhang, Yuexian Hou

Organizations: Tianjin University, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

When does a shared representation become insufficient for jointly supporting multiple tasks? This question lies at the heart of multi-task learning models. Although task interference in shared representations has been widely observed, its onset, measurable form, and capacity boundary remain poorly understood. In this paper, we revisit this problem through the lens of Bell's theorem. We show that if four binary task outcomes can be jointly represented by a common shared representation, their correlations must satisfy the classical CHSH bound ∣S∣≤2|S|\leq2. We introduce NCnet, a controlled multi-task model for studying how representational capacity affects task compatibility. Experiments reveal a clear capacity-dependent transition. Once nontrivial task fitting emerges, constrained capacity can yield ∣S∣>2|S|>2. As capacity further increases, these exceedances progressively disappear and SS returns to the bounded region. This systematic pattern provides evidence that limited representational capacity is an important source of the observed incompatibility. We observe similar transitions across CNNs, PLMs, and LLMs, suggesting that the phenomenon extends beyond the controlled NCnet setting. We refer to this capacity-dependent transition as the Bell Boundary, providing a measurable diagnostic for representation bottlenecks, task incompatibility, and capacity allocation in multi-task models.

Figures & tables

Figure 1: Overview of the Bell Boundary framework. (1) NCnet establishes a correspondence between shared-representation learning and the Bell/CHSH setting. (2) Four independently trained task-pair models provide the correlations used to compute SNNS_{\mathrm{NN}} . (3) The analysis extends to CNNs, PLMs, and LLMs under four experimental settings. The bottom-right schematic illustrates CHSH exceedances in a critical capacity regime and their disappearance as capacity increases.
CHSH settingMulti-task learning
Hidden variable λ∼ρ(λ)\lambda\sim\rho(\lambda)Shared representation h=fθ(x)h=f_{\theta}(x) , h∼ρθ(h)h\sim\rho_{\theta}(h)
Measurement branches AA and BBTask families AA and BB
Measurement settings a1a_{1} , a2a_{2} and b1b_{1} , b2b_{2}Task settings a1a_{1} , a2a_{2} and b1b_{1} , b2b_{2}
Binary outcomes Ai(λ)A_{i}(\lambda) , Bj(λ)B_{j}(\lambda)Binary task outcomes gϕAi(h)g_{\phi_{A_{i}}}\left(h\right) , gϕBj(h)g_{\phi_{B_{j}}}\left(h\right)
Correlation EijE_{ij}Task correlation EijE_{ij}
CHSH statistic SSNeural CHSH statistic SNNS_{\mathrm{NN}}
Table 1: Correspondence between the CHSH setting and multi-task learning.
ddA1B1A_{1}B_{1}A1B2A_{1}B_{2}A2B1A_{2}B_{1}A2B2A_{2}B_{2}SNNS_{\mathrm{NN}}P0P_{0} (%)
EEAcc(Ai)\mathrm{Acc}(A_{i})Acc(Bj)\mathrm{Acc}(B_{j})EEAcc(Ai)\mathrm{Acc}(A_{i})Acc(Bj)\mathrm{Acc}(B_{j})EEAcc(Ai)\mathrm{Acc}(A_{i})Acc(Bj)\mathrm{Acc}(B_{j})EEAcc(Ai)\mathrm{Acc}(A_{i})Acc(Bj)\mathrm{Acc}(B_{j})
1−0.00\phantom{-}0.00 ±0.00\pm 0.000.720.72 ±0.25\pm 0.250.780.78 ±0.25\pm 0.25−0.02\phantom{-}0.02 ±0.07\pm 0.070.970.97 ±0.10\pm 0.100.520.52 ±0.07\pm 0.07−0.01\phantom{-}0.01 ±0.06\pm 0.060.520.52 ±0.06\pm 0.060.980.98 ±0.09\pm 0.09−0.06\phantom{-}0.06 ±0.06\pm 0.060.620.62 ±0.11\pm 0.110.640.64 ±0.11\pm 0.11−0.02-0.02 ±0.12\pm 0.1200
2−1.00\phantom{-}1.00 ±0.00\pm 0.001.001.00 ±0.00\pm 0.001.001.00 ±0.00\pm 0.00−0.48\phantom{-}0.48 ±0.11\pm 0.111.001.00 ±0.00\pm 0.000.740.74 ±0.05\pm 0.05−0.47\phantom{-}0.47 ±0.10\pm 0.100.740.74 ±0.05\pm 0.051.001.00 ±0.00\pm 0.00−0.22\phantom{-}0.22 ±0.20\pm 0.200.760.76 ±0.17\pm 0.170.760.76 ±0.17\pm 0.17−1.74\phantom{-}1.74 ±0.24\pm 0.2400
3−1.00\phantom{-}1.00 ±0.00\pm 0.001.001.00 ±0.00\pm 0.001.001.00 ±0.00\pm 0.00−0.91\phantom{-}0.91 ±0.21\pm 0.211.001.00 ±0.00\pm 0.000.950.95 ±0.10\pm 0.10−0.93\phantom{-}0.93 ±0.17\pm 0.170.960.96 ±0.09\pm 0.091.001.00 ±0.00\pm 0.00−0.53\phantom{-}0.53 ±0.24\pm 0.240.880.88 ±0.14\pm 0.140.860.86 ±0.15\pm 0.15−2.31\mathbf{\phantom{-}2.31} ±0.35\pm 0.3500
4−1.00\phantom{-}1.00 ±0.00\pm 0.001.001.00 ±0.00\pm 0.001.001.00 ±0.00\pm 0.00−0.99\phantom{-}0.99 ±0.07\pm 0.071.001.00 ±0.00\pm 0.000.990.99 ±0.04\pm 0.04−0.98\phantom{-}0.98 ±0.10\pm 0.100.990.99 ±0.05\pm 0.051.001.00 ±0.00\pm 0.00−0.93\phantom{-}0.93 ±0.11\pm 0.110.980.98 ±0.04\pm 0.040.980.98 ±0.04\pm 0.04−2.04\mathbf{\phantom{-}2.04} ±0.16\pm 0.165353
5−1.00\phantom{-}1.00 ±0.00\pm 0.001.001.00 ±0.00\pm 0.001.001.00 ±0.00\pm 0.00−1.00\phantom{-}1.00 ±0.00\pm 0.001.001.00 ±0.00\pm 0.001.001.00 ±0.00\pm 0.00−1.00\phantom{-}1.00 ±0.00\pm 0.001.001.00 ±0.00\pm 0.001.001.00 ±0.00\pm 0.00−0.99\phantom{-}0.99 ±0.05\pm 0.051.001.00 ±0.02\pm 0.021.001.00 ±0.02\pm 0.02−2.01\mathbf{\phantom{-}2.01} ±0.05\pm 0.059595
Table 2: NCnet results across hidden dimensions d=1d=1 – 2020 (mean ±\pm std over 100 random seeds). EijE_{ij} denotes task-pair correlation, Acc\mathrm{Acc} denotes per-head training accuracy, and P0P_{0} is the fraction of runs with near-zero mean best-loss. Bold SNNS_{\mathrm{NN}} values significantly exceed 22 .
Settinga1a_{1}a2a_{2}b1b_{1}b2b_{2}
Setting 1Left digit, predict 0Left digit, 10-classRight digit, predict 1Right digit, 10-class
Setting 2PAWS-X-enPAWS-X-frPAWS-X-jaPAWS-X-ko
Setting 3SST-2CommonsenseQAMRPCMathQA
Setting 4MathQAMATHRACE-MiddleLogiQA2.0
Table 3: Task assignments for each experimental setting.
Figure 2: Capacity boundary and gradient conflict in NCnet. Top: SNNS_{\mathrm{NN}} (left axis) and average training loss (right axis) versus hidden width nn . Points and bands show individual seeds and mean ±\pm std; the dashed line marks SNN=2S_{\mathrm{NN}}=2 , and the inset shows its distribution at n=3n=3 . Bottom: gradient cosine similarity averaged over the four MijM_{ij} contexts and all training phases (green: alignment; red: conflict).
DepthChannel width
112244881616323264649696
1−1.02-1.02 ±0.57\pm 0.57−1.82-1.82 ±0.70\pm 0.70−1.30-1.30 ±0.04\pm 0.04−1.23-1.23 ±0.04\pm 0.04−1.13-1.13 ±0.03\pm 0.03−0.95-0.95 ±0.03\pm 0.03−0.80-0.80 ±0.02\pm 0.02−0.76-0.76 ±0.01\pm 0.01
2−1.56-1.56 ±0.50\pm 0.50−1.22-1.22 ±0.03\pm 0.03−1.08-1.08 ±0.04\pm 0.04−0.85-0.85 ±0.07\pm 0.07−0.56-0.56 ±0.01\pm 0.01−0.32-0.32 ±0.02\pm 0.02−0.07-0.07 ±0.05\pm 0.05−0.07\phantom{-}0.07 ±0.04\pm 0.04
3−1.18-1.18 ±0.03\pm 0.03−1.01-1.01 ±0.09\pm 0.09−0.68-0.68 ±0.05\pm 0.05−0.13-0.13 ±0.08\pm 0.08−0.62\phantom{-}0.62 ±0.06\pm 0.06−1.41\phantom{-}1.41 ±0.04\pm 0.04−1.93\phantom{-}1.93 ±0.04\pm 0.04−2.26\mathbf{\phantom{-}2.26} ±0.07\pm 0.07
4−0.89-0.89 ±0.05\pm 0.05−0.26-0.26 ±0.11\pm 0.11−0.68\phantom{-}0.68 ±0.09\pm 0.09−1.61\phantom{-}1.61 ±0.04\pm 0.04−2.46\mathbf{\phantom{-}2.46} ±0.07\pm 0.07−2.96\mathbf{\phantom{-}2.96} ±0.04\pm 0.04−2.06\mathbf{\phantom{-}2.06} ±0.02\pm 0.02−2.00\phantom{-}2.00 ±0.00\pm 0.00
5−0.08-0.08 ±0.09\pm 0.09−1.41\phantom{-}1.41 ±0.08\pm 0.08−2.64\mathbf{\phantom{-}2.64} ±0.06\pm 0.06−2.17\mathbf{\phantom{-}2.17} ±0.02\pm 0.02−2.00\phantom{-}2.00 ±0.00\pm 0.00−2.00\phantom{-}2.00 ±0.00\pm 0.00−2.00\phantom{-}2.00 ±0.00\pm 0.00−2.00\phantom{-}2.00 ±0.00\pm 0.00
Table 4: Train-phase SNNS_{\mathrm{NN}} (mean ±\pm std over 5 random seeds) for the multi-task CNN across shared-encoder depth (rows) and channel width (columns). Bold cells are significantly greater than the classical Bell bound S=2S=2
Figure 3: SNNS_{\mathrm{NN}} versus Trainable Parameter Count Across Settings. (a) Mean SNNS_{\mathrm{NN}} across five seeds for CNNs of varying depth and base channel width; values are shown in each heatmap cell. (b) Mean SNNS_{\mathrm{NN}} versus LoRA rank rr for Settings 2, 3, and 4. Shaded bands show one standard deviation across three seeds, the dashed line marks SNN=2S_{\mathrm{NN}}=2 .
Figure 4: Gradient conflict rates for CNN task pair A2B2A_{2}B_{2} . (a) Rates grouped by mean SNNS_{\mathrm{NN}} : <1.97<1.97 , 1.971.97 – 2.032.03 , and >2.03>2.03 . Points denote architectures, boxes show medians and interquartile ranges, and diamonds mark group means. (b) Rates by shared-encoder depth and base channel width; cell labels show final-epoch mean SNNS_{\mathrm{NN}} .
rrE11E_{11}E12E_{12}E21E_{21}E22E_{22}SNNS_{\mathrm{NN}}
(a) Setting 2
10.743 ±\pm 0.0210.695 ±\pm 0.0250.689 ±\pm 0.0270.661 ±\pm 0.0251.466 ±\pm 0.037
20.839 ±\pm 0.0170.836 ±\pm 0.0080.838 ±\pm 0.0180.822 ±\pm 0.0181.691 ±\pm 0.028
40.936 ±\pm 0.0020.915 ±\pm 0.0050.918 ±\pm 0.0130.896 ±\pm 0.0161.872 ±\pm 0.003
80.968 ±\pm 0.0040.956 ±\pm 0.0070.955 ±\pm 0.0080.946 ±\pm 0.0021.934 ±\pm 0.004
160.969 ±\pm 0.0040.961 ±\pm 0.0030.957 ±\pm 0.0010.951 ±\pm 0.0021.936 ±\pm 0.003
Table 5: Pairwise task correlations EijE_{ij} and CHSH statistic SNNS_{\mathrm{NN}} across LoRA ranks rr for (a) Setting 2, (b) Setting 3, and (c) Setting 4. Values are mean ±\pm std over three random seeds for Settings 2, 3, and 4. Bold SNNS_{\mathrm{NN}} values significantly exceed 22 .
Figure 5: Gradient conflict across LoRA ranks rr . (a, c) Training-averaged conflict rates for the four task pairs in BERT and Qwen, respectively; error bars show standard deviations across seeds. (b) Gradient cosine similarity distributions for BERT’s A2B2A_{2}B_{2} pair: violins show density, boxes show medians and interquartile ranges, and black circles mark means. The red dashed line marks the zero-cosine conflict boundary.
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

ParameterCNN Setting 1BERT Setting 2BERT Setting 3Qwen3-8B Setting 4
OptimizerAdamW—AdamWAdamW
Initial learning rate10−310^{-3}3×10−43\times 10^{-4}10−310^{-3} (LoRA)2×10−42\times 10^{-4}
Classification-head learning rate10−310^{-3}—2.5×10−32.5\times 10^{-3}—
Learning-rate decayCosinePlateauCosineCosine
Linear warmup5 epochs5%5\%3%3\%5%5\%
Learning-rate floor0010−610^{-6}10−610^{-6}10−610^{-6} or 10−510^{-5}
Appendix
Table 6: Training hyperparameters for Settings 1–4. CNN uses its full training set; language sample caps are per task. A dash indicates an inapplicable parameter or one not recorded in the available training logs. The Qwen learning-rate floors are specified below.
Channel widthA1B1A_{1}B_{1}A1B2A_{1}B_{2}A2B1A_{2}B_{1}A2B2A_{2}B_{2}
A1A_{1}B1B_{1}A1A_{1}B2B_{2}A2A_{2}B1B_{1}A2A_{2}B2B_{2}
190.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0074.00±35.7874.00\pm 35.7810.02±0.0310.02\pm 0.039.95±0.139.95\pm 0.1390.00±0.0090.00\pm 0.0010.02±0.0310.02\pm 0.0310.07±0.1610.07\pm 0.16
274.00±35.7874.00\pm 35.7872.48±35.0872.48\pm 35.0888.35±3.6988.35\pm 3.699.52±1.059.52\pm 1.059.30±0.889.30\pm 0.8890.00±0.0090.00\pm 0.0010.07±0.4510.07\pm 0.459.62±0.719.62\pm 0.71
485.69±4.9585.69\pm 4.9590.00±0.0090.00\pm 0.0089.99±0.0289.99\pm 0.0210.40±0.6610.40\pm 0.6610.57±1.3810.57\pm 1.3889.27±1.6389.27\pm 1.6310.25±0.7610.25\pm 0.7610.06±0.3310.06\pm 0.33
890.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0089.98±0.0489.98\pm 0.0410.70±1.1310.70\pm 1.1310.58±1.3810.58\pm 1.3890.00±0.0090.00\pm 0.0010.93±0.8910.93\pm 0.8910.49±1.1810.49\pm 1.18
1690.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0012.80±1.1512.80\pm 1.1511.73±0.7411.73\pm 0.7490.00±0.0090.00\pm 0.0011.97±0.7811.97\pm 0.7813.28±1.1013.28\pm 1.10
Appendix
Table 7: Task accuracy (%) for CNN in Setting 2 at shared-encoder depth 11 . Each entry is mean ±\pm sample standard deviation over five random seeds.
Channel widthA1B1A_{1}B_{1}A1B2A_{1}B_{2}A2B1A_{2}B_{1}A2B2A_{2}B_{2}
A1A_{1}B1B_{1}A1A_{1}B2B_{2}A2A_{2}B1B_{1}A2A_{2}B2B_{2}
172.50±35.1072.50\pm 35.1072.36±35.0472.36\pm 35.0486.07±5.1886.07\pm 5.1810.09±0.1310.09\pm 0.1310.08±0.1210.08\pm 0.1278.11±26.5978.11\pm 26.5910.13±0.2210.13\pm 0.229.61±0.679.61\pm 0.67
290.00±0.0090.00\pm 0.0090.01±0.0190.01\pm 0.0189.84±0.3789.84\pm 0.3710.13±0.1610.13\pm 0.1611.14±1.9111.14\pm 1.9189.89±0.2589.89\pm 0.2511.51±1.9211.51\pm 1.9210.78±1.1210.78\pm 1.12
490.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0011.72±0.9511.72\pm 0.9511.96±1.1511.96\pm 1.1590.00±0.0090.00\pm 0.0015.58±1.2815.58\pm 1.2813.91±1.3013.91\pm 1.30
890.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0016.12±1.6916.12\pm 1.6915.04±1.7315.04\pm 1.7390.00±0.0090.00\pm 0.0018.20±0.9718.20\pm 0.9718.60±1.8318.60\pm 1.83
1690.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0021.29±0.7621.29\pm 0.7619.79±0.7319.79\pm 0.7390.00±0.0090.00\pm 0.0021.86±0.2121.86\pm 0.2122.71±0.8022.71\pm 0.80
Appendix
Table 8: Task accuracy (%) for CNN in Setting 2 at shared-encoder depth 22 . Each entry is mean ±\pm sample standard deviation over five random seeds.
Channel widthA1B1A_{1}B_{1}A1B2A_{1}B_{2}A2B1A_{2}B_{1}A2B2A_{2}B_{2}
A1A_{1}B1B_{1}A1A_{1}B2B_{2}A2A_{2}B1B_{1}A2A_{2}B2B_{2}
190.00±0.0090.00\pm 0.0089.99±0.0289.99\pm 0.0289.83±0.3889.83\pm 0.3810.79±1.0110.79\pm 1.0111.69±2.3511.69\pm 2.3590.00±0.0090.00\pm 0.0012.58±0.7612.58\pm 0.7611.19±1.6511.19\pm 1.65
290.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0089.97±0.0889.97\pm 0.0813.74±2.2013.74\pm 2.2014.16±1.9014.16\pm 1.9089.99±0.0389.99\pm 0.0315.01±1.3715.01\pm 1.3715.09±1.8215.09\pm 1.82
490.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0018.30±2.4718.30\pm 2.4718.75±1.7418.75\pm 1.7490.00±0.0090.00\pm 0.0020.82±1.5820.82\pm 1.5821.16±1.8621.16\pm 1.86
890.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0027.00±2.3727.00\pm 2.3726.42±1.7226.42\pm 1.7290.00±0.0090.00\pm 0.0030.51±1.8730.51\pm 1.8731.81±2.5531.81\pm 2.55
1690.01±0.0290.01\pm 0.0290.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0041.44±3.0141.44\pm 3.0136.53±1.5836.53\pm 1.5890.00±0.0090.00\pm 0.0039.90±1.0039.90\pm 1.0042.64±0.7442.64\pm 0.74
Appendix
Table 9: Task accuracy (%) for CNN in Setting 2 at shared-encoder depth 33 . Each entry is mean ±\pm sample standard deviation over five random seeds.
Channel widthA1B1A_{1}B_{1}A1B2A_{1}B_{2}A2B1A_{2}B_{1}A2B2A_{2}B_{2}
A1A_{1}B1B_{1}A1A_{1}B2B_{2}A2A_{2}B1B_{1}A2A_{2}B2B_{2}
190.00±0.0090.00\pm 0.0089.99±0.0389.99\pm 0.0389.99±0.0289.99\pm 0.0215.66±2.3315.66\pm 2.3315.94±2.1515.94\pm 2.1590.00±0.0090.00\pm 0.0016.49±1.5316.49\pm 1.5317.09±1.7617.09\pm 1.76
290.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0090.00±0.0090.00\pm 0.0024.17±3.2624.17\pm 3.2621.59±2.2021.59\pm 2.2090.00±0.0090.00\pm 0.0028.97±2.8628.97\pm 2.8630.03±3.0330.03\pm 3.03
490.06±0.0690.06\pm 0.0690.46±0.4890.46\pm 0.4890.14±0.2790.14\pm 0.2740.38±2.7240.38\pm 2.7239.33±2.2739.33\pm 2.2790.44±0.3490.44\pm 0.3441.51±2.2741.51\pm 2.2744.13±3.2444.13\pm 3.24
892.26±0.6392.26\pm 0.6391.53±0.1691.53\pm 0.1690.95±0.1490.95\pm 0.1457.24±1.9257.24\pm 1.9255.01±1.4855.01\pm 1.4890.96±0.0890.96\pm 0.0851.32±0.9551.32\pm 0.9554.04±0.2954.04\pm 0.29
1698.34±0.5198.34\pm 0.5196.32±0.6796.32\pm 0.6790.32±0.1890.32\pm 0.1878.80±2.1378.80\pm 2.1375.95±2.0475.95\pm 2.0490.50±0.1090.50\pm 0.1054.10±0.2654.10\pm 0.2657.36±0.2657.36\pm 0.26
Appendix
Table 10: Task accuracy (%) for CNN in Setting 2 at shared-encoder depth 44 . Each entry is mean ±\pm sample standard deviation over five random seeds.
Channel widthA1B1A_{1}B_{1}A1B2A_{1}B_{2}A2B1A_{2}B_{1}A2B2A_{2}B_{2}
A1A_{1}B1B_{1}A1A_{1}B2B_{2}A2A_{2}B1B_{1}A2A_{2}B2B_{2}
190.00±0.0090.00\pm 0.0090.33±0.4590.33\pm 0.4590.01±0.0290.01\pm 0.0230.58±1.7730.58\pm 1.7728.45±2.1228.45\pm 2.1290.13±0.2590.13\pm 0.2527.40±3.2527.40\pm 3.2528.66±2.7028.66\pm 2.70
292.48±0.8092.48\pm 0.8093.91±1.0593.91\pm 1.0590.37±0.3390.37\pm 0.3358.91±2.0958.91\pm 2.0954.17±3.0554.17\pm 3.0590.56±0.3090.56\pm 0.3045.57±1.3945.57\pm 1.3948.83±1.1448.83\pm 1.14
498.00±0.5798.00\pm 0.5798.71±0.7698.71\pm 0.7690.81±0.2490.81\pm 0.2488.21±0.8088.21\pm 0.8087.47±1.4287.47\pm 1.4291.67±0.6691.67\pm 0.6659.50±1.9759.50\pm 1.9763.40±1.4863.40\pm 1.48
8100.00±0.01100.00\pm 0.0199.98±0.0299.98\pm 0.0298.28±0.3798.28\pm 0.3799.75±0.1099.75\pm 0.1099.45±0.0599.45\pm 0.0598.24±0.5198.24\pm 0.5193.53±0.8993.53\pm 0.8993.47±0.3393.47\pm 0.33
16100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.01100.00\pm 0.01100.00±0.00100.00\pm 0.00
Appendix
Table 11: Task accuracy (%) for CNN in Setting 2 at shared-encoder depth 55 . Each entry is mean ±\pm sample standard deviation over five random seeds.
Channel widthA1B1A_{1}B_{1}A1B2A_{1}B_{2}A2B1A_{2}B_{1}A2B2A_{2}B_{2}
A1A_{1}B1B_{1}A1A_{1}B2B_{2}A2A_{2}B1B_{1}A2A_{2}B2B_{2}
195.73±0.9295.73\pm 0.9298.33±0.6398.33\pm 0.6390.42±0.6390.42\pm 0.6351.35±9.8551.35\pm 9.8559.09±4.8959.09\pm 4.8994.13±1.1494.13\pm 1.1453.93±1.5153.93\pm 1.5154.85±3.2254.85\pm 3.22
299.85±0.0999.85\pm 0.0999.81±0.1099.81\pm 0.1096.75±1.0996.75\pm 1.0991.22±2.2391.22\pm 2.2390.58±1.4790.58\pm 1.4798.04±1.0798.04\pm 1.0786.73±2.3986.73\pm 2.3987.66±2.4787.66\pm 2.47
4100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.0099.95±0.0499.95\pm 0.04100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.0099.89±0.0699.89\pm 0.0699.75±0.1199.75\pm 0.1199.77±0.1899.77\pm 0.18
8100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00
16100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00
Appendix
Table 12: Task accuracy (%) for CNN in Setting 2 at shared-encoder depth 66 . Each entry is mean ±\pm sample standard deviation over five random seeds.
Channel widthA1B1A_{1}B_{1}A1B2A_{1}B_{2}A2B1A_{2}B_{1}A2B2A_{2}B_{2}
A1A_{1}B1B_{1}A1A_{1}B2B_{2}A2A_{2}B1B_{1}A2A_{2}B2B_{2}
199.17±0.3199.17\pm 0.3199.68±0.1199.68\pm 0.1193.61±0.4193.61\pm 0.4176.49±4.3176.49\pm 4.3176.18±2.5776.18\pm 2.5798.76±0.4698.76\pm 0.4669.54±4.6869.54\pm 4.6871.84±2.8771.84\pm 2.87
2100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.0099.92±0.1299.92\pm 0.1299.95±0.0799.95\pm 0.0799.92±0.0399.92\pm 0.0399.83±0.0899.83\pm 0.0899.24±0.3699.24\pm 0.3699.24±0.4099.24\pm 0.40
4100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00
8100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00
16100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00100.00±0.00100.00\pm 0.00
Appendix
Table 13: Task accuracy (%) for CNN in Setting 2 at shared-encoder depth 77 . Each entry is mean ±\pm sample standard deviation over five random seeds.
LoRA rank rrA1B1A_{1}B_{1}A1B2A_{1}B_{2}A2B1A_{2}B_{1}A2B2A_{2}B_{2}
A1A_{1}B1B_{1}A1A_{1}B2B_{2}A2A_{2}B1B_{1}A2A_{2}B2B_{2}
195.32±0.1495.32\pm 0.1490.85±0.8590.85\pm 0.8595.52±0.1895.52\pm 0.1889.15±0.9689.15\pm 0.9693.44±0.2193.44\pm 0.2191.50±0.4091.50\pm 0.4094.29±0.4394.29\pm 0.4389.98±0.9289.98\pm 0.92
297.24±0.3797.24\pm 0.3794.43±0.4594.43\pm 0.4597.42±0.1497.42\pm 0.1494.46±0.0594.46\pm 0.0596.22±0.0896.22\pm 0.0895.24±0.6095.24\pm 0.6096.25±0.0896.25\pm 0.0893.86±0.2793.86\pm 0.27
498.80±0.1198.80\pm 0.1197.88±0.2497.88\pm 0.2498.74±0.0698.74\pm 0.0697.36±0.1997.36\pm 0.1998.01±0.2298.01\pm 0.2297.52±0.2097.52\pm 0.2098.05±0.1198.05\pm 0.1196.89±0.3496.89\pm 0.34
899.51±0.0399.51\pm 0.0398.87±0.0898.87\pm 0.0899.27±0.0599.27\pm 0.0598.39±0.0198.39\pm 0.0198.92±0.0698.92\pm 0.0698.83±0.0798.83\pm 0.0798.99±0.1198.99\pm 0.1198.54±0.0398.54\pm 0.03
1699.52±0.0499.52\pm 0.0498.88±0.0898.88\pm 0.0899.38±0.0799.38\pm 0.0798.73±0.0598.73\pm 0.0598.98±0.0498.98\pm 0.0498.89±0.0598.89\pm 0.0598.93±0.0298.93\pm 0.0298.53±0.1098.53\pm 0.10
Appendix
Table 14: Task accuracy (%) for BERT in Setting 2: A1A_{1} = PAWS-X-en, A2A_{2} = PAWS-X-fr, B1B_{1} = PAWS-X-ja, and B2B_{2} = PAWS-X-ko. Each entry is mean ±\pm sample standard deviation over three random seeds.
LoRA rank rrA1B1A_{1}B_{1}A1B2A_{1}B_{2}A2B1A_{2}B_{1}A2B2A_{2}B_{2}
A1A_{1}B1B_{1}A1A_{1}B2B_{2}A2A_{2}B1B_{1}A2A_{2}B2B_{2}
199.82±0.0899.82\pm 0.0899.81±0.1599.81\pm 0.1599.41±0.1099.41\pm 0.1063.37±2.4663.37\pm 2.4690.26±0.6490.26\pm 0.6499.19±0.1499.19\pm 0.1490.79±0.6390.79\pm 0.6345.68±0.8145.68\pm 0.81
299.96±0.0299.96\pm 0.0299.96±0.0499.96\pm 0.0499.66±0.0899.66\pm 0.0882.93±2.4682.93\pm 2.4697.41±0.4997.41\pm 0.4999.80±0.0999.80\pm 0.0995.63±0.5195.63\pm 0.5170.18±2.1470.18\pm 2.14
499.99±0.0299.99\pm 0.0299.98±0.0499.98\pm 0.0499.83±0.1099.83\pm 0.1095.57±0.2995.57\pm 0.2999.22±0.1899.22\pm 0.1899.93±0.0399.93\pm 0.0397.70±0.3897.70\pm 0.3891.53±0.5991.53\pm 0.59
899.99±0.0299.99\pm 0.0299.98±0.0399.98\pm 0.0399.90±0.0099.90\pm 0.0098.76±0.1098.76\pm 0.1099.68±0.0899.68\pm 0.0899.98±0.0299.98\pm 0.0299.30±0.2299.30\pm 0.2297.89±0.2297.89\pm 0.22
1699.99±0.0299.99\pm 0.0299.99±0.0299.99\pm 0.0299.99±0.0299.99\pm 0.0299.46±0.1399.46\pm 0.1399.68±0.1499.68\pm 0.1499.98±0.0299.98\pm 0.0299.61±0.1799.61\pm 0.1799.44±0.0599.44\pm 0.05
Appendix
Table 15: Final-epoch training accuracy (%) for BERT in Setting 3: A1A_{1} = SST-2, A2A_{2} = CommonsenseQA, B1B_{1} = MRPC, and B2B_{2} = MathQA. Each entry is mean ±\pm sample standard deviation over three random seeds.
LoRA rank rrA1B1A_{1}B_{1}A1B2A_{1}B_{2}A2B1A_{2}B_{1}A2B2A_{2}B_{2}
A1A_{1}B1B_{1}A1A_{1}B2B_{2}A2A_{2}B1B_{1}A2A_{2}B2B_{2}
198.27±0.0998.27\pm 0.0998.89±0.1298.89\pm 0.1297.48±0.1797.48\pm 0.1797.58±0.0897.58\pm 0.0867.79±0.4567.79\pm 0.4599.00±0.0699.00\pm 0.0661.76±0.3361.76\pm 0.3399.07±0.0399.07\pm 0.03
299.92±0.0299.92\pm 0.0299.52±0.1299.52\pm 0.1299.78±0.0299.78\pm 0.0299.02±0.1499.02\pm 0.1483.39±0.4583.39\pm 0.4599.79±0.0599.79\pm 0.0579.22±0.2979.22\pm 0.2999.71±0.0799.71\pm 0.07
499.90±0.0399.90\pm 0.0399.63±0.1099.63\pm 0.1099.94±0.0299.94\pm 0.0299.48±0.1399.48\pm 0.1391.57±0.2891.57\pm 0.2899.92±0.0599.92\pm 0.0587.84±0.2787.84\pm 0.2799.97±0.0399.97\pm 0.03
899.98±0.0299.98\pm 0.0299.91±0.0799.91\pm 0.0799.98±0.0299.98\pm 0.0299.74±0.0799.74\pm 0.0796.52±0.2596.52\pm 0.2599.88±0.0299.88\pm 0.0295.72±0.2595.72\pm 0.2599.87±0.0099.87\pm 0.00
16100.00±0.00100.00\pm 0.0099.99±0.0299.99\pm 0.0299.99±0.0299.99\pm 0.0299.86±0.0599.86\pm 0.0599.10±0.1799.10\pm 0.1799.97±0.0099.97\pm 0.0099.22±0.0799.22\pm 0.0799.96±0.0299.96\pm 0.02
Appendix
Table 16: Task accuracy (%) for Qwen3-8B in Setting 4: A1A_{1} = MathQA, A2A_{2} = MATH, B1B_{1} = RACE-Middle, and B2B_{2} = LogiQA2.0. Each entry is mean ±\pm sample standard deviation over three random seeds.
More in this topic

Shared Representation Learning

See all 9 papers

Explore similar work

CardsList
Jul 17, 2026

Capacity and Redundancy Trade-offs in Multi-Task Learning

In multi-task learning (MTL) negative transfer is often considered as an optimization artifact, but it can also be viewed as a consequence of limited shared capacity and weak task redundancy. We investigate this effect through a Capacity--Redundancy (CR) identity that decomposes the sum of per-task predictive informations into joint predictive information that includes label redundancy defined via total correlation (TC), and a residual coupling term that quantifies interference left unresolved by the shared representation. Additionally, we show two key results: (i) a clustering-gap decomposition that gives a necessary and sufficient condition for clustered sharing to outperform global sharing, and (ii) a gradient--TC bridge in a Gaussian multi-task model that formally justifies gradient cosine similarity as a proxy for redundancy ordering. Empirically, we estimate the residual coupling ΔΔ from validation residual correlations, showing that clustered LoRA substantially reduces Δ^\widehatΔ, outperforms size-matched random partitions, and results in statistically significant gains with multi-seed confidence intervals.

Asif Khan

1Harvard Medical School, Boston, USA

Multi-Task LearningMulti-Task OptimizationNegative TransferMutual Information
AskPDF
Jun 14, 2026

The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints

Modern deep learning architectures are increasingly multi-task and multi-modal, using a pretrained foundation model combined with task-specific, fine-tuned models. Empirically, exploiting similarity across different problems, instead of solving them individually, can significantly improve overall performance. While the generalization and sample complexity properties of multitask learning have been widely studied, the parametric complexity of joint approximation in comparison to separate approximation remains less well understood. The question is particularly relevant in modern deep learning, where models are increasingly required to satisfy structural constraints such as equivariance, conservation laws, or orthogonality. We prove lower and upper bounds on the description-length for separate and joint approximation classes, respectively, in uniform norm. We build a class of orthogonal functions by composing a shared hard feature, realized by a Rademacher-Haar wavelet series, with Sawtooth-Walsh readouts to enforce orthogonality of output coordinates. The dyadic tree structure of the Rademacher-Haar wavelet concentrates the approximation hardness in the common feature component, while the readouts act as task-specific heads. Using an information-theoretic framework, we obtain a sharp gap between the optimal approximation rates achievable by joint and separate coding. Finally, we realize this separation in a neural network model using Heaviside activations via reduction to triangle-wave approximation. Our results show that even under an orthogonality constraint joint approximation requires strictly fewer bits in compositional architectures, provided the tasks share a latent hard feature. This provides theoretical insight into the description-length-efficiency of compositional multi-output architectures and clarifies how neural networks can retain expressivity under geometric constraints.

Thomas Dittrich, Oliver Potocki, Philipp Grohs

Johann Radon Institute of Computational and Applied Mathematics, Austrian Academy of Sciences · Faculty of Mathematics, University of Vienna

Shared Representation LearningRepresentation LearningMinimum Description LengthMulti-Task LearningNeural Network Approximation TheoryInformation-Theoretic Lower Bounds
AskPDF
Mar 10, 2026

An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse

Model merging unifies independently fine-tuned LLMs from the same base, enabling reuse and integration of parallel development efforts without retraining. However, in practice we observe that merging does not always succeed: certain combinations of task-specialist models suffer from catastrophic performance degradation after merging. We refer to this failure mode as merging collapse. Intuitively, collapse arises when the learned representations or parameter adjustments for different tasks are fundamentally incompatible, so that merging forces destructive interference rather than synergy. In this paper, we identify and characterize the phenomenon of task-level merging collapse, where certain task combinations consistently trigger huge performance degradation across all merging methods. Through extensive experiments and statistical analysis, we demonstrate that representational incompatibility between tasks is strongly correlated with merging collapse, while parameter-space conflict metrics show minimal correlation, challenging conventional wisdom in model merging literature. We provide a theoretical explanation on this phenomenon through rate-distortion theory with a dimension-dependent bound, establishing fundamental limits on task mergeability regardless of methodology.

Yuan Cao, Dezhi Ran, Yuzhe Guo +5

Key Lab of HCST (PKU), MOE; SCS, Peking University, Beijing, China · Computer Science Department at Columbia University, New York, USA · School of Computing Science, Simon Fraser University, Burnaby, BC, Canada +1

Language Model MergingModel Merging
AskPDF