Task-Relevant Null-Space Residuals for Non-Injective Neural Mappings

Bizu Feng; Zhimu Yang; Shuming Wang; Yuan Cheng; Shaode Yu; Xiaojun Qian; Zixin Hu

cs.AISep 29, 2026

Task-Relevant Null-Space Residuals for Non-Injective Neural Mappings

Authors: Bizu Feng, Zhimu Yang, Shuming Wang, Yuan Cheng, Shaode Yu, Xiaojun Qian, Zixin Hu

Organizations: Institute of Artificial Intelligence Innovation and Industry, Fudan University, Shanghai, China · Shanghai Academy of AI for Science, Shanghai, China · Human Phenome Institute, Fudan University, Shanghai, China · School of Information and Communication Engineering, Communication University of China, Beijing, China

Abstract

Non-injective mappings in neural networks map distinct inputs to the same representation, thereby implicitly inducing equivalence relations in the input space. However, the input differences eliminated by these mappings may still be required by downstream tasks, creating a mismatch between operator-induced indistinguishability and task-required distinctions. For non-injective linear operators realized in the current forward pass, their null spaces exactly characterize these invisible input variations. We propose Task-Relevant Null-Space Residuals (NSR), a general residual framework for non-injective linear mappings. NSR combines null-space component extraction from pre-mapping representations, member-level encoding and gating, and application-specific integration to exploit potentially task-relevant information under downstream supervision while preserving the original aggregation or merging rules. We evaluate NSR in two structurally different settings: token merging and graph aggregation. In token merging, NSR achieves higher semantic segmentation performance than the corresponding compressed baselines in 34 out of 36 evaluated configurations, with a maximum observed gain of 31.51 mIoU points under strong compression. In graph aggregation, NSR achieves 100% training accuracy on Tree-NeighborsMatch at depths d=2--6 across three backbones, alongside gains on heterophilic node classification and molecular graph regression. Together, these results support null-space residuals as a practical complement to non-injective linear mappings, enabling downstream models to learn from input distinctions invisible in the original operator's output.

Figures & tables

Figure 1: Operator-induced indistinguishability need not align with downstream task requirements: distinct inputs may produce the same mapped representation while requiring different task outputs.

Figure 2: Overview of NSR. Null-space residuals are extracted from pre-mapping representations based on the current linear operator and transformed into usable complementary information through a learnable branch under task supervision. The original mapping branch retains its computation rule.

			Original		+NSR
Dataset	Method	WP	GFLOPs $\downarrow$	mIoU $\uparrow$	GFLOPs $\downarrow$	mIoU $\uparrow$	$\Delta_{\mathrm{NSR}}\uparrow$
VOC	Full-ViT	$1.0\times$	1.254	64.85	–	–	–
	ToMe	$1.8\times$	0.698	62.81	0.750	64.23	+1.43
		$2.5\times$	0.492	44.58	0.533	59.54	+14.95
		$3.4\times$	0.373	18.31	0.407	49.82	+31.51
		$3.8\times$	0.333	16.98	0.365	46.23	+29.25

Table 1: Semantic-segmentation results across merging methods and compression configurations. Bold marks the higher mIoU in each Original–NSR pair.

\Delta_{\mathrm{NSR}}

is computed from unrounded mIoU.

Figure 3: Training accuracy on Tree-NeighborsMatch across tree depths. Dashed and solid curves denote the original GCN, GIN, and GraphSAGE models and their NSR variants, respectively. Each point is the highest training accuracy attained in the corresponding run.

Backbone	Variant	Roman-empire Acc. (%) $\uparrow$	Amazon-ratings Acc. (%) $\uparrow$	Minesweeper ROC-AUC (%) $\uparrow$	Tolokers ROC-AUC (%) $\uparrow$	Questions ROC-AUC (%) $\uparrow$
GCN	Original	$72.05\pm 0.64$	$49.41\pm 0.57$	$90.20\pm 0.66$	$\mathbf{84.90\pm 0.90}$	$75.32\pm 1.35$
GCN	+NSR	$\mathbf{82.90\pm 0.71}$	$\mathbf{49.67\pm 0.57}$	$\mathbf{92.09\pm 0.63}$	$84.36\pm 0.73$	$\mathbf{78.25\pm 1.21}$
GIN	Original	$74.00\pm 0.79$	$49.56\pm 0.55$	$86.90\pm 0.57$	$83.06\pm 0.88$	$74.81\pm 1.57$
GIN	+NSR	$\mathbf{82.25\pm 0.65}$	$\mathbf{50.47\pm 0.57}$	$\mathbf{89.41\pm 1.09}$	$\mathbf{83.18\pm 0.87}$	$\mathbf{75.06\pm 1.40}$
GraphSAGE	Original	$81.59\pm 0.55$	$52.33\pm 0.37$	$\mathbf{93.37\pm 0.40}$	$83.28\pm 0.56$	$74.82\pm 0.91$
GraphSAGE	+NSR	$\mathbf{87.71\pm 0.61}$	$\mathbf{52.72\pm 0.75}$	$92.75\pm 0.48$	$\mathbf{84.50\pm 0.63}$	$\mathbf{75.07\pm 2.16}$

Table 2: Node-classification results on the five heterophilic graph benchmarks. Values are the mean and sample standard deviation (%) over the ten official splits. The higher score within each backbone pair is shown in bold.

Figure 4: Within each official split, test nodes are divided into ten approximately equal-frequency bins, with the horizontal axis ordering the bins from low to high scores. Lines show the mean accuracy of the corresponding bin across the ten official splits, and shaded bands of the same color indicate the mean

\pm

one sample standard deviation.

Backbone	Original	+NSR
GCN	$0.4732\pm 0.0067$	$\mathbf{0.2947\pm 0.0050}$
GIN	$0.3451\pm 0.0093$	$\mathbf{0.3091\pm 0.0093}$
GraphSAGE	$0.4373\pm 0.0086$	$\mathbf{0.3831\pm 0.0125}$

Table 3: Test MAE on the ZINC subset. Values are the mean and standard deviation over ten random seeds; lower values are better.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Tree depth	Generated samples	Message-passing layers	Nominal effective batch size
2	96	3	64
3	8,000	4	64
4	16,000	5	1,024
5	32,000	6	1,024
6	32,000	7	1,024
7	32,000	8	2,048

Appendix

Table 4: Tree-NeighborsMatch configurations. Effective batches larger than

256

are implemented using gradient accumulation.

Method	Merge location	Within-group weights
ToMe	After the attention residual update and before the MLP	Normalized token-size weights
PiToMe	After the attention residual update and before the MLP	Normalized token-size weights
MPM	Before a designated Transformer block	Equal weights of $1/2$ within each pair

Appendix

Table 5: Integration of NSR with the three token-merging methods. Each method retains its merge location and weighting rule. Token size denotes the number of original patches represented by a token.

Backbone	Members	Aggregation coefficients
GCN	Incoming messages in the normalized backbone graph, including self-loops	Degree-normalized GCN coefficients
GraphSAGE	Neighbor messages specified by the input edge list	Neighborhood-mean weights
GIN	Neighbor messages and a separate root message	$1$ for neighbors and $1+\epsilon$ for the root

Appendix

Table 6: Member sets and aggregation coefficients used by the NSR graph branches.

Figure 5: Relative null-space residual magnitude

\rho_{Z}=\|Z\|_{F}/\|X\|_{F}

for the trained

+\textsc{NSR}

token-merging models on (a) ADE20K, (b) Pascal VOC, and (c) Cityscapes. For each compression setting, the solid line reports the mean across merging blocks in which NSR is constructed, and the shaded region shows the corresponding mean

\pm

one standard deviation.

Dataset	ToMe	PiToMe	MPM
ADE20K	$9.02\times 10^{-7}$	$9.54\times 10^{-7}$	$9.54\times 10^{-7}$
Pascal VOC	$7.38\times 10^{-7}$	$6.28\times 10^{-7}$	$4.77\times 10^{-7}$
Cityscapes	$1.76\times 10^{-6}$	$9.54\times 10^{-7}$	$4.77\times 10^{-7}$

Appendix

Table 7: Maximum null-space violation

\epsilon_{\mathrm{null}}=\|AZ\|_{\infty}

for the trained

+\textsc{NSR}

models. Each entry is the maximum observed value over all evaluated compression settings and merging blocks.

			Pixel Acc. $\uparrow$		Params
Dataset	Method	WP	Original	+NSR	Original	+NSR	Gap Closure $\uparrow$
VOC	Full-ViT	$1.0\times$	0.9062	–	5,528,853	–	–
	ToMe	$1.8\times$	0.8988	0.9030	5,528,853	6,094,485	$69.7\%$
		$2.5\times$	0.8248	0.8873	5,528,853	6,094,485	$73.8\%$
		$3.4\times$	0.7131	0.8607	5,528,853	6,048,373	$67.7\%$
		$3.8\times$	0.7111	0.8487	5,528,853	6,002,261	$61.1\%$

Appendix

Table 8: Additional semantic-segmentation results for token merging with and without NSR. WP denotes the computation-reduction operating point of the original compressed baseline relative to Full-ViT. Pixel Acc. and Params are reported separately for Original and +NSR. Within each Original–NSR pair, the higher Pixel Acc. is shown in bold. Gap Closure is computed from unrounded mIoU values according to Eq. ( 14 ) and reported to one decimal place.

Figure 6: Mean accuracy gain from NSR across diagnostic-score bins on Roman-empire. Bins are the same as those in Figure 4 . Within each official split, the gain is computed as the accuracy of the NSR-augmented model minus that of the Original model. The figure reports the mean over the ten official splits.

Dataset	Target	Selected $r$	Achieved	GFLOPs
VOC2012	$1.8\times$	14	$1.797\times$	0.6980
	$2.5\times$	21	$2.550\times$	0.4918
	$3.4\times$	29	$3.367\times$	0.3725
	$3.8\times$	33	$3.768\times$	0.3329
ADE20K	$1.8\times$	66	$1.799\times$	5.8169
	$2.5\times$	98	$2.490\times$	4.2015

Appendix

Table 9: ToMe compression configurations on the three semantic segmentation datasets. Target and Achieved denote the target and realized computation-reduction factors, respectively. Selected

r

is the planned number of tokens merged per layer. GFLOPs denotes the computation of the corresponding baseline.

Dataset	Target	Retention ratio $q$	Achieved	GFLOPs
VOC2012	$1.8\times$	0.895	$1.796\times$	0.6983
	$2.5\times$	0.829	$2.503\times$	0.5010
	$3.4\times$	0.754	$3.407\times$	0.3681
	$3.8\times$	0.725	$3.783\times$	0.3316
ADE20K	$1.8\times$	0.908	$1.804\times$	5.8011
	$2.5\times$	0.849	$2.503\times$	4.1799

Appendix

Table 10: PiToMe compression configurations on the three semantic segmentation datasets.

q

denotes the retention ratio. All operating points use the same margin and matching mode schedules. GFLOPs denotes the computation of the corresponding baseline.

Dataset	Target	Reported point	Insertion blocks	Search	Achieved	GFLOPs
VOC2012	$1.8\times$	$1.8\times$	[0,4,5,9]	$1.801\times$	$1.793\times$	0.6994
	$2.5\times$	$2.4\times$	[0–1,3,5–6,10]	$2.494\times$	$2.449\times$	0.5122
	$3.0\times$	$3.0\times$	[0–2,4–6,8–11]	$3.004\times$	$2.926\times$	0.4286
	$3.8\times$	$3.3\times$	[0–11]	$3.438\times$	$3.332\times$	0.3764
ADE20K	$1.8\times$	$1.8\times$	[2,4,7,9,11]	$1.803\times$	$1.803\times$	5.8041
	$2.5\times$	$2.4\times$	[0–1,4,7,9,11]	$2.507\times$	$2.355\times$	4.4431

Appendix

Table 11: MPM compression configurations on the three semantic segmentation datasets. Target denotes the search target, while Reported point is the operating-point label used in the final experiments. Search and Achieved denote the computation-reduction factors obtained during calibration and final validation, respectively. GFLOPs denotes the computation of the final baseline. A range

i

–

j

in Insertion blocks includes all consecutive block indices from

i

j

, inclusive.

Figure 7: Layer-wise token-merging distributions of ToMe and PiToMe on VOC2012, ADE20K, and Cityscapes. The top and bottom rows correspond to ToMe and PiToMe, respectively, while the three columns correspond to VOC2012, ADE20K, and Cityscapes from left to right. The horizontal axis indicates the Transformer block index, and the vertical axis indicates the target computation-reduction factor. Color represents the normalized number of merged tokens, $M_{l}=r_{l}/T_{0}$ . All subplots share the same color scale, with darker colors indicating a larger fraction of the initial patch tokens merged at the corresponding block.

Compressor	Compression	Base	Post-merge-only	Full NSR
ToMe	$1.8\times$	0.6281	0.6274	0.6423
	$2.5\times$	0.4458	0.4395	0.5954
	$3.4\times$	0.1831	0.1778	0.4982
	$3.8\times$	0.1698	0.1683	0.4623
PiToMe	$1.8\times$	0.6148	0.6181	0.6244
	$2.5\times$	0.5577	0.5593	0.5919

Appendix

Table 12: Comparison with a local residual correction based only on merged representations. Best and second-best results at each operating point are shown in bold and underlined, respectively.

Compressor	Compression	Base	NSR-Feature	NSR-Spatial	Full NSR
ToMe	$1.8\times$	0.6281	0.6351	0.6308	0.6423
	$2.5\times$	0.4458	0.4347	0.5985	0.5954
	$3.4\times$	0.1831	0.1693	0.4823	0.4982
	$3.8\times$	0.1698	0.1632	0.4576	0.4623
PiToMe	$1.8\times$	0.6148	0.6138	0.6258	0.6244
	$2.5\times$	0.5577	0.5687	0.5873	0.5919

Appendix

Table 13: Ablation of Feature Integration and Spatial Routing. Best and second-best results at each operating point are shown in bold and underlined, respectively.

Compressor	WP	Base	RawSource	Full NSR
ToMe	$1.8\times$	0.6281	0.6349	0.6423
	$2.5\times$	0.4458	0.5930	0.5954
	$3.4\times$	0.1831	0.4912	0.4982
	$3.8\times$	0.1698	0.4490	0.4623
PiToMe	$1.8\times$	0.6148	0.6201	0.6244
	$2.5\times$	0.5577	0.5899	0.5919

Appendix

Table 14: Input-source ablation within the token-merging instantiation of NSR on Pascal VOC. RawSource directly encodes pre-merge member features while retaining the complete NSR architecture. Base denotes the corresponding original merging baseline. WP follows the operating-point labels used in the main experiments. Values are mIoU on a 0–1 scale; higher is better. The best result at each operating point is shown in bold.

Configuration	GCN	GraphSAGE	GIN
Base	$72.05\pm 0.64$	$81.59\pm 0.55$	$74.00\pm 0.79$
Post-aggregation-only	$75.30\pm 0.69$	$84.00\pm 0.46$	$76.01\pm 0.62$
RawSource	$79.21\pm 0.41$	$85.57\pm 0.64$	$80.69\pm 0.51$
Full NSR	$\boldsymbol{82.90\pm 0.71}$	$\boldsymbol{87.71\pm 0.61}$	$\boldsymbol{82.25\pm 0.65}$

Appendix

Table 15: Graph-aggregation ablations on Roman-empire. Values are test accuracy (%), reported as the mean and sample standard deviation over the ten official splits. The best result in each column is shown in bold.

Statistic	Roman-empire	Amazon-ratings	Minesweeper	Tolokers	Questions
Nodes	22662	24492	10000	11758	48921
Edges	32927	93050	39402	519000	153540
Avg. degree	2.91	7.60	7.88	88.28	6.28
Global clustering	0.29	0.32	0.43	0.23	0.02
Avg. local clustering	0.39	0.58	0.44	0.53	0.03
Diameter	6824	46	99	11	16

Appendix

Table 16: Statistics of the five heterophilic graph benchmarks, as reported by Platonov et al. [2023b] .

Explore similar work

Aug 11, 2026cs.LG

HyperFix: Combinatorial Nonlinear Correction for Task Vector Merging

Task vectors enable model merging without joint retraining. In practice, the subset of task vectors to be merged may vary, but many existing methods use scalar tuning for a particular subset, requiring repeated tuning across subsets and restricting task vector merging to linear rescaling. We therefore formulate merging across varying task subsets as a combinatorial correction problem and introduce HyperFix, a lightweight hypernetwork that predicts subset-conditioned nonlinear corrections in weight space. Trained once on singleton, pair, and triple subsets from a task bank, HyperFix generalizes to larger subsets without per-subset optimization. Our local perturbation analysis bounds the residual correction beyond linear merging and motivates learning it from small task updates. Experiments across diverse benchmarks show that HyperFix outperforms existing task vector merging methods while reducing tuning cost.

Hyo Seo Kim, Ren Wang

Illinois Institute of Technology

Continual Model Merging Multi--Task Learning Hypernetwork Efficient Task Adaptation Subspace

Ask PDF

May 18, 2026cs.LG

Attention Sinks and Outliers in Attention Residuals

We propose OASIS, an outlier- and sink-aware technique built on inter-layer null signaling. As AttnResidual architectures introduce an additional depth-wise normalization channel, they improve inter-layer routing flexibility but also exacerbate attention sinks, activation outliers, and the resulting degradation in inference stability and quantization robustness. OASIS addresses this issue by introducing a Softmax1-based null space and coupling token-level null evidence to depth routing through an inter-layer null signal, thereby reducing sink-dominated routing and improving structural robustness. Theoretically, we show that the dual-normalization design of AttnResidual intensifies sink formation and quantization brittleness. Experimentally, we compare OASIS against five baselines on three real-world datasets and observe consistent improvements in both attention sink and post-quantization performance. Notably, OASIS achieves an average reduction of 9.26% in maximum infinity norm and 2.60% in average kurtosis across the evaluated settings, while lowering perplexity by 75.85% under W8A8 and improving GSM8K Pass@1 by 12.42% under W4A4.

Haozheng Luo, Haoran Dai, Shaoyang Zhang +10

Department of Computer Science, Northwestern University · Department of Computer Science, Illinois Institute of Technolog · Department of Computer Science and Engineering, University of Michigan +5

Outliers

Ask PDF

May 13, 2026cs.LG

Delta Attention Residuals

Attention Residuals replace standard additive residual connections with learned softmax attention over previous layer outputs, enabling selective cross-layer routing. However, standard Attention Residuals still attend over cumulative hidden states in previous layers, which are highly redundant. We show that this redundancy leads to routing collapse in deeper layers: attention weights become low-contrast and closer to uniform (max weight

{\approx}

0.2), limiting the model's ability to select informative states in previous layers. This raises a key but underexplored design question: what layer-wise representations should be routed in Attention Residuals? To answer this question, we propose Delta Attention Residuals, which attend over deltas -- the change introduced by each sublayer (

\mathbf{v}_i = \mathbf{h}_{i+1} - \mathbf{h}_i

) -- instead of cumulative states. Delta representations are structurally diverse and yield higher-contrast attention distributions (max weight

{\approx}

0.6), enabling more selective and effective routing across layers. This principle applies at both per-sublayer and block granularity. Across all tested scales (220M--7.6B), Delta Attention Residuals consistently outperform both standard residuals and Attention Residuals, with 1.7--8.2% validation perplexity gains. Delta Attention Residuals also enables converting pretrained checkpoints into Delta Attention Residuals via standard fine-tuning. Code is available at https://github.com/wdlctc/delta-attention-residuals-code.

Cheng Luo, Zefan Cai, Junjie Hu

Independent Researcher · University of Wisconsin–Madison

Layer-Wise Cross-Layer Interactions

Ask PDF