Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs

Mingyu Jin; Yutong Yin; Jingcheng Niu; Qingcheng Zeng; Wujiang Xu; Mengnan Du; Wei Cheng; Zhaoran Wang

cs.CLMar 3, 2026

Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs

Authors: Mingyu Jin, Yutong Yin, Jingcheng Niu, Qingcheng Zeng, Wujiang Xu, Mengnan Du, Wei Cheng, Zhaoran Wang,

Organizations: Rutgers University · Northwestern University · UKP Lab, TU Darmstadt · New Jersey Institute of Technology ·

Abstract

In this work, we investigate how Large Language Models (LLMs) adapt their internal representations when encountering inputs of increasing difficulty, quantified as the degree of out-of-distribution (OOD) shift. We reveal a consistent and quantifiable phenomenon: as task difficulty increases, whether through harder reasoning questions, longer contexts, or adding answer choices, the last hidden states of LLMs become substantially sparser. In short, the farther the shift, the sparser the representations. This sparsity--difficulty relation is observable across diverse models and domains, suggesting that language models respond to unfamiliar or complex inputs by concentrating computation into specialized subspaces in the last hidden state. Through a series of controlled analyses with a learning dynamic explanation, we demonstrate that this sparsity is not incidental but an adaptive mechanism for stabilizing reasoning under OOD. Leveraging this insight, we design Sparsity-Guided Curriculum In-Context Learning (SG-ICL), a strategy that explicitly uses representation sparsity to schedule few-shot demonstrations, leading to considerable performance enhancements. Our study provides new mechanistic insights into how LLMs internalize OOD challenges. The source code is available at the URL: https://github.com/MingyuJ666/sparsityLLM.

Figures & tables

Figure 1: Harder Inputs Induce Sparser Representations. Across all four controlled difficulty axes, the last hidden states become progressively sparser as tasks get harder. Results are shown for Qwen2.5-3B using Top-10% Energy; nevertheless, the same trend holds across difficulty datasets and settings, sparsity metrics, LLM family, and LLM sizes.

Panel of Figure 2 — Figure 2: Overview of Sparsity Analysis. Both subfigures show that sparsity significantly correlates with difficulty for Qwen2.5-7B across two datasets: MATH-500 (left) and DeepMATH-103K (right).

Figure 3: Sparsity Metrics under Answer Choice Expansion. Bar plots show mean sparsity across 14 disciplines for four metrics under Normal (+0), Moderate Expansion (+5), and Large Expansion (+10) on Qwen2.5-3B . Error bars indicate the minimum and maximum across disciplines. Increasing task difficulty leads to higher sparsity.

Figure 4: Sparsity Differences under Knowledge Conflict. Across four sparsity metrics, the harder conflict ( ) condition yields consistently sparser last-layer representations than the non-conflict ( ) condition on Qwen2.5-3B . Arrows indicate the sparsity direction of each metric.

Figure 5: Layer-wise Sparsity across Context Lengths. While intermediate layers show minimal variation across contexts, the final layers exhibit sharp divergence: longer contexts consistently produce sparser representations. The experiment is conducted on LongReasonQA ( Ling et al., 2025 ) , where the background context length can be controlled. Line color indicates the input context length: 8K, 16K, 32K, 64K.

Model	Params	Training	Accuracy (%, $\uparrow$ )			Top-5 Energy (%, $\uparrow$ )			Top-10 Energy (%, $\uparrow$ )			Eff. Rank ( $\downarrow$ )
Model	Params	Steps	$\mathcal{D}_{\text{mem}}$	$\mathcal{D}_{\text{ID}}$	$\mathcal{D}_{\text{OOD}}$	$\mathcal{D}_{\text{mem}}$	$\mathcal{D}_{\text{ID}}$	$\mathcal{D}_{\text{OOD}}$	$\mathcal{D}_{\text{mem}}$	$\mathcal{D}_{\text{ID}}$	$\mathcal{D}_{\text{OOD}}$	$\mathcal{D}_{\text{mem}}$	$\mathcal{D}_{\text{ID}}$	$\mathcal{D}_{\text{OOD}}$
Trans-8-8	0.02B	2500	64.00	42.00	31.00	26.11	26.03	26.26	42.12	42.00	42.34	0.502	0.504	0.500
Trans-16-8	0.04B	2000	71.00	50.00	24.00	25.96	25.94	26.06	41.98	41.98	42.15	0.503	0.503	0.501
Trans-16-16	0.17B	2000	85.00	56.00	44.00	27.70	27.70	27.78	43.70	43.66	43.76	0.483	0.484	0.482
Trans-16-16	0.17B	2500	81.00	58.00	37.00	27.64	27.58	27.76	43.56	43.52	43.69	0.484	0.485	0.483
Trans-32-32	1.34B	1500	78.00	57.00	37.00	30.28	30.54	30.84	46.44	46.72	47.04	0.454	0.451	0.448

Table 1: Emergence of the Sparsity Phenomenon. We report metrics across 3 levels. The gradient blue shading, observed uniquely in Trans-32-32, illustrates that the “harder-is-sparser” mechanism is a learned behavior.

Figure 6: Learning dynamics via Top-

k

Energy. Top-

k

Energy first rises during early training, indicating feature selection and increasing activation concentration, and then gradually decreases while accuracy remains high, suggesting feature consolidation into a more distributed and robust representation.

Figure 7: Performance Comparison of Reasoning Strategies. Our SG-ICL achieves an accuracy of 76.60% on MATH-500 with Qwen2.5-7B , substantially outperforming standard CoT baselines (zero-shot and few-shot) and the Auto-CoT baseline (75.20%).

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Sparsity analysis across different models. (a) Visualizes the sparsity pattern of Llama3.1-70B. (b) and (c) compare the Qwen2.5 series models (32B and 70B).

Context Setting	Context Prompt	Query Prompt
Related	On the inside, bananas are white.	What color are mangoes on the inside? They are	white		orange
Irrelevant	The capital of Canada is Ottawa.	What color are mangoes on the inside? They are	Ottawa		orange
Random	Promotion	What color are mangoes on the inside? They are	Promotion		orange
Distraction over Counterfactual Context
Counterfactual	On the inside, bananas are green.	What color are mangoes on the inside? They are	green	white		orange

Appendix

Table 2: Prompt Setup ( Niu et al., 2025 ) . We follow the exact same prompt setup as Niu et al. (2025) . The emojis represent the target tokens: : counterfactual; : distracting; : correct.

Context Setting	$L_{1}$ Norm	Top-5% Energy	Top-10% Energy	Eff. Rank
Related	5066.96	0.5723	0.6715	0.6715
Irrelevant	5072.15	0.5816	0.6772	0.6764
Random	3689.13	0.7600	0.8203	0.5987
Counterfactual	4938.74	0.5978	0.6937	0.6680

Appendix

Table 3: Sparsity metrics averaged over samples across different context settings in Qwen2.5-1.5B.

Figure 10: Accuracy–Sparsity Correlation. Accuracy strongly correlates with Last Hidden State sparsity on MATH-500 for Qwen2.5-7B (

L_{1}

r

=0.897; Top-10% Energy:

r

=-0.915).

Figure 12: Sparsity Metrics vs. Problem Difficulty on DeepMath-103K for Qwen2.5-1.5B in Layer8. We plot several 8-layer sparsity metrics against problem difficulty levels on DeepMath-103K.

Setting	$\boldsymbol{\ell_{1}}$	Top-5% $\uparrow$	Top-10% $\uparrow$	Eff. Rank $\downarrow$
Last layer, 3B	$-0.076/-0.104$	$0.129/0.103$	$0.130/0.103$	$-0.126/-0.083$
Last layer, 1.5B	$-0.384/-0.378$	$0.271/0.234$	$0.271/0.235$	$-0.135/-0.086$
Layer 8, 1.5B	$0.027/0.105$	$-0.072/-0.116$	$-0.089/-0.129$	$0.052/0.102$

Appendix

Table 4: DeepMath-103K correlations between problem difficulty and sparsity metrics. Each entry reports Pearson

r

/ Spearman

\rho

. The expected sparsification pattern is lower effective rank and higher Top-

k

energy as difficulty increases.

Figure 13: Per-domain sparsity statistics for Qwen2.5-3B-Instruct on MMLU-Robust.

Panel of Figure 14 — Figure 14: Average sparsity statistics on MMLU-Robust for more LMs.

Figure 15: Task complexity induces sparsity. Representation activation showing increased sparsity in harder tasks (32k context, blue) compared to original settings (Short context, red) . Three metrics consistently demonstrate higher sparsity for challenging inputs.

Figure 16: Layer-wise Sparsity across Context Lengths. While intermediate layers show minimal variation across contexts, the final layers exhibit sharp divergence: longer contexts consistently produce sparser representations. This experiment is conducted on LongReasonQA ( Li et al., 2025a ) , which can control the background context length.

Figure 17: Layer-wise Sparsity across Context Lengths. While intermediate layers show minimal variation across contexts, the final layers exhibit sharp divergence: longer contexts consistently produce sparser representations. This experiment is conducted on LongReasonQA ( Li et al., 2025a ) , which can control the background context length.

Panel of Figure 18 — Figure 18: A LongReason 8K instance. We show one example used in our long-context evaluation. For readability, the background is truncated (head + tail); the full context is available in the dataset.

Panel of Figure 19 — Figure 19: A LongReason 8K instance (paraphrased). We follow the same presentation format as our long-context example: truncated background + multiple-choice inquiry + gold label.


Stage	Synthetic KG construction (as implemented)

Universe	Entities are symbolic nodes $Q_{0},Q_{1},\ldots,Q_{n-1}$ ; relations are $P_{0},P_{1},\ldots,P_{n_{r}-1}$ .
Rule sampling	Sample a pool of candidate rules as relation chains: $(r_{0}\Leftarrow r_{1},r_{2},\ldots,r_{\ell})$ with $\ell\in[L_{\min},L_{\max}]$ . Optionally apply length-weighted sampling with temperature $\tau$ (shorter rules sampled more often). Acyclicity is enforced via a dependency graph over relations.
Backbone graph initialization	Construct an initial directed multigraph by instantiating each sampled rule on fresh entity nodes: create a path $Q\xrightarrow{r_{1}}\cdots\xrightarrow{r_{\ell}}Q^{\prime}$ and add the implied edge $Q\xrightarrow{r_{0}}Q^{\prime}$ . Maintain per-relation entity pools (“repeated entities”) for later attachment.
Graph growth (structural prior)	Iteratively add new nodes until reaching $n$ entities. Each new node attaches $m$ edges by sampling relations constrained by an adjacency map derived from rule co-occurrence (also supporting a power-law attachment mode).

Appendix

Table 5: Synthetic knowledge graph generation via a latent-rule process: rule sampling, graph growth, deductible edge injection, and difficulty-controlled evaluation splits.

Figure 20: Learning dynamics across sparsity metrics and accuracy. We track Top-5% Energy, Top-10% Energy, accuracy, and normalized Effective Rank across training checkpoints for the controlled graph experiment. Early training shows increasing energy concentration and decreasing effective rank, while later training shows partial recovery of effective rank and stabilization of accuracy, supporting the two-phase feature-selection and feature-consolidation interpretation.

Figure 21: Learning dynamics for Trans-16-16. We track Top-5% Energy, Top-10% Energy, accuracy, and normalized Effective Rank across training checkpoints in the controlled graph environment. Compared with the larger Trans-32-32 model, Trans-16-16 exhibits a similar early concentration phase, where Top-

k

Energy increases and Effective Rank decreases, followed by a relaxation phase in which energy concentration weakens and Effective Rank recovers. This provides additional evidence that the sparsity dynamics are tied to feature selection and consolidation during training, rather than being an artifact of a single model scale.

OOD Generalization

Out-of-distribution generalization studies how machine-learning models perform when test data, domains, tasks, or environments differ from training.

See all 395 papers

Tool Use Reduces Depth-Induced Collapse in OOD Reasoning
Feb 24, 2026David Koplow, Tomer Galanti, Tomaso PoggioOOD Generalization LLM Tool Use
Massachusetts Institute of Technology · Texas A&M University
How Language Models Process Out-of-Distribution Inputs: A Two-Pathway Framework
Apr 30, 2026Hamidreza SaghirLLM Interpretability Attention Mechanisms
Independent Researcher
Curriculum Learning Teaches LLMs Long-Horizon Implicit Reasoning, Up To A Limit
Oct 19, 2026Jimson Huang, Maxwell Nickell, Ruowang Zhang +1Long-Horizon LLM Reasoning Implicit Reasoning in Language Models
Purdue University West Lafayette, IN 47907, USA

Feb 24, 2026J/K move · Enter open · S save

Tool Use Reduces Depth-Induced Collapse in OOD Reasoning

David Koplow, Tomer Galanti, Tomaso Poggio

Massachusetts Institute of Technology · Texas A&M University

OOD Generalization LLM Tool Use Algorithmic Reasoning

Open paper AskPDF arXiv

Humans can apply ideas learned in one context to substantially different situations. We call this process of searching for and constructing novel recombinations of learned relationships to solve new problems out-of-distribution (OOD) reasoning. The capacity for large language models (LLMs) to support OOD reasoning underpins proposals for generally intelligent systems. However, this property is challenging to measure because most problems admit many decompositions, some involving shallow subproblems and others involving subproblems that may have been memorized from the training data. This makes it difficult to determine how much compositional reasoning a model must actually perform. Uncertainty about training distributions, how to measure a datapoint's distance from a training distribution, and the exponential number of ways to decompose most tasks make it intractable to robustly measure any modern LLM's OOD-reasoning capacity on standard natural-language tasks. In this work, we introduce a benchmark in which a model progressively solves a Boolean circuit over

GF(2)

from data. This benchmark is both minimal, isolating OOD reasoning from these confounding factors, and general, as any computable function can be represented as a sufficiently large

GF(2)

polynomial. We find that standalone models' next-step accuracy collapses as depth grows. In contrast, tool use through the synthesis and execution of code prevents this collapse in both small and frontier LLMs. These results indicate that synthesizing tools play a crucial role in supporting OOD reasoning.

Abstract

OOD Generalization

Explore similar work

Tool Use Reduces Depth-Induced Collapse in OOD Reasoning

How Language Models Process Out-of-Distribution Inputs: A Two-Pathway Framework

Curriculum Learning Teaches LLMs Long-Horizon Implicit Reasoning, Up To A Limit