Organizations: Rutgers University · Northwestern University · UKP Lab, TU Darmstadt · New Jersey Institute of Technology · NEC Laboratories America · University of North Carolina at Chapel Hill
In this work, we investigate how Large Language Models (LLMs) adapt their internal representations when encountering inputs of increasing difficulty, quantified as the degree of out-of-distribution (OOD) shift. We reveal a consistent and quantifiable phenomenon: as task difficulty increases, whether through harder reasoning questions, longer contexts, or adding answer choices, the last hidden states of LLMs become substantially sparser. In short, the farther the shift, the sparser the representations. This sparsity--difficulty relation is observable across diverse models and domains, suggesting that language models respond to unfamiliar or complex inputs by concentrating computation into specialized subspaces in the last hidden state. Through a series of controlled analyses with a learning dynamic explanation, we demonstrate that this sparsity is not incidental but an adaptive mechanism for stabilizing reasoning under OOD. Leveraging this insight, we design Sparsity-Guided Curriculum In-Context Learning (SG-ICL), a strategy that explicitly uses representation sparsity to schedule few-shot demonstrations, leading to considerable performance enhancements. Our study provides new mechanistic insights into how LLMs internalize OOD challenges. The source code is available at the URL: https://github.com/MingyuJ666/sparsityLLM.
Figures & tables
Figure 1: Harder Inputs Induce Sparser Representations. Across all four controlled difficulty axes, the last hidden states become progressively sparser as tasks get harder. Results are shown for Qwen2.5-3B using Top-10% Energy; nevertheless, the same trend holds across difficulty datasets and settings, sparsity metrics, LLM family, and LLM sizes.
Figure 2: Overview of Sparsity Analysis. Both subfigures show that sparsity significantly correlates with difficulty for Qwen2.5-7B across two datasets: MATH-500 (left) and DeepMATH-103K (right).
Figure 3: Sparsity Metrics under Answer Choice Expansion. Bar plots show mean sparsity across 14 disciplines for four metrics under Normal (+0), Moderate Expansion (+5), and Large Expansion (+10) on Qwen2.5-3B . Error bars indicate the minimum and maximum across disciplines. Increasing task difficulty leads to higher sparsity.
Figure 4: Sparsity Differences under Knowledge Conflict. Across four sparsity metrics, the harder conflict ( ) condition yields consistently sparser last-layer representations than the non-conflict ( ) condition on Qwen2.5-3B . Arrows indicate the sparsity direction of each metric.
Figure 5: Layer-wise Sparsity across Context Lengths. While intermediate layers show minimal variation across contexts, the final layers exhibit sharp divergence: longer contexts consistently produce sparser representations. The experiment is conducted on LongReasonQA ( Ling et al., 2025 ) , where the background context length can be controlled. Line color indicates the input context length: 8K, 16K, 32K, 64K.
Model
Params
Training
Accuracy (%, ↑ )
Top-5 Energy (%, ↑ )
Top-10 Energy (%, ↑ )
Eff. Rank ( ↓ )
Steps
Dmem
DID
DOOD
Dmem
DID
DOOD
Dmem
DID
DOOD
Dmem
DID
DOOD
Trans-8-8
0.02B
2500
64.00
42.00
31.00
26.11
26.03
26.26
42.12
42.00
42.34
0.502
0.504
0.500
Trans-16-8
0.04B
2000
71.00
50.00
24.00
25.96
25.94
26.06
41.98
41.98
42.15
0.503
0.503
0.501
Trans-16-16
0.17B
2000
85.00
56.00
44.00
27.70
27.70
27.78
43.70
43.66
43.76
0.483
0.484
0.482
2500
81.00
58.00
37.00
27.64
27.58
27.76
43.56
43.52
43.69
0.484
0.485
0.483
Trans-32-32
1.34B
1500
78.00
57.00
37.00
30.28
30.54
30.84
46.44
46.72
47.04
0.454
0.451
0.448
Table 1: Emergence of the Sparsity Phenomenon. We report metrics across 3 levels. The gradient blue shading, observed uniquely in Trans-32-32, illustrates that the “harder-is-sparser” mechanism is a learned behavior.
Figure 6: Learning dynamics via Top- k Energy. Top- k Energy first rises during early training, indicating feature selection and increasing activation concentration, and then gradually decreases while accuracy remains high, suggesting feature consolidation into a more distributed and robust representation.
Figure 8
Figure 7: Performance Comparison of Reasoning Strategies. Our SG-ICL achieves an accuracy of 76.60% on MATH-500 with Qwen2.5-7B , substantially outperforming standard CoT baselines (zero-shot and few-shot) and the Auto-CoT baseline (75.20%).
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Scaling of computational depth (Layers) and parallel attention capacities (Heads) across Llama models. Both metrics show a step-wise increase consistent with the model’s total parameter growth.
Figure 11Figure 12
Figure 9: Sparsity analysis across different models. (a) Visualizes the sparsity pattern of Llama3.1-70B. (b) and (c) compare the Qwen2.5 series models (32B and 70B).
Context Setting
Context Prompt
Query Prompt
Related
On the inside, bananas are white.
What color are mangoes on the inside? They are
white
orange
Irrelevant
The capital of Canada is Ottawa.
What color are mangoes on the inside? They are
Ottawa
orange
Random
Promotion
What color are mangoes on the inside? They are
Promotion
orange
Distraction over Counterfactual Context
Counterfactual
On the inside, bananas are green.
What color are mangoes on the inside? They are
green
white
orange
Appendix
Table 2: Prompt Setup ( Niu et al., 2025 ) . We follow the exact same prompt setup as Niu et al. (2025) . The emojis represent the target tokens: : counterfactual; : distracting; : correct.
Context Setting
L1 Norm
Top-5% Energy
Top-10% Energy
Eff. Rank
Related
5066.96
0.5723
0.6715
0.6715
Irrelevant
5072.15
0.5816
0.6772
0.6764
Random
3689.13
0.7600
0.8203
0.5987
Counterfactual
4938.74
0.5978
0.6937
0.6680
Appendix
Table 3: Sparsity metrics averaged over samples across different context settings in Qwen2.5-1.5B.
Figure 10: Accuracy–Sparsity Correlation. Accuracy strongly correlates with Last Hidden State sparsity on MATH-500 for Qwen2.5-7B ( L1 : r =0.897; Top-10% Energy: r =-0.915).
Figure 11: Sparsity Metrics vs. Problem Difficulty on DeepMath-103K. We plot several last-layer sparsity metrics against problem difficulty levels for Qwen2.5-3B and Qwen2.5-1.5B. Across both models, harder problems tend to induce progressively sparser final-layer representations, showing a consistent dataset-level relationship between reasoning difficulty and representational sparsity.
Figure 12: Sparsity Metrics vs. Problem Difficulty on DeepMath-103K for Qwen2.5-1.5B in Layer8. We plot several 8-layer sparsity metrics against problem difficulty levels on DeepMath-103K.
Setting
ℓ1
Top-5% ↑
Top-10% ↑
Eff. Rank ↓
Last layer, 3B
−0.076/−0.104
0.129/0.103
0.130/0.103
−0.126/−0.083
Last layer, 1.5B
−0.384/−0.378
0.271/0.234
0.271/0.235
−0.135/−0.086
Layer 8, 1.5B
0.027/0.105
−0.072/−0.116
−0.089/−0.129
0.052/0.102
Appendix
Table 4: DeepMath-103K correlations between problem difficulty and sparsity metrics. Each entry reports Pearson r / Spearman ρ . The expected sparsification pattern is lower effective rank and higher Top- k energy as difficulty increases.
Figure 20Figure 21
Figure 13: Per-domain sparsity statistics for Qwen2.5-3B-Instruct on MMLU-Robust.
Figure 14: Average sparsity statistics on MMLU-Robust for more LMs.
Figure 15: Task complexity induces sparsity. Representation activation showing increased sparsity in harder tasks (32k context, blue) compared to original settings (Short context, red) . Three metrics consistently demonstrate higher sparsity for challenging inputs.
Figure 16: Layer-wise Sparsity across Context Lengths. While intermediate layers show minimal variation across contexts, the final layers exhibit sharp divergence: longer contexts consistently produce sparser representations. This experiment is conducted on LongReasonQA ( Li et al., 2025a ) , which can control the background context length.
Figure 17: Layer-wise Sparsity across Context Lengths. While intermediate layers show minimal variation across contexts, the final layers exhibit sharp divergence: longer contexts consistently produce sparser representations. This experiment is conducted on LongReasonQA ( Li et al., 2025a ) , which can control the background context length.
Figure 18: A LongReason 8K instance. We show one example used in our long-context evaluation. For readability, the background is truncated (head + tail); the full context is available in the dataset.
Figure 19: A LongReason 8K instance (paraphrased). We follow the same presentation format as our long-context example: truncated background + multiple-choice inquiry + gold label.
Stage
Synthetic KG construction (as implemented)
Universe
Entities are symbolic nodes Q0,Q1,…,Qn−1 ; relations are P0,P1,…,Pnr−1 .
Rule sampling
Sample a pool of candidate rules as relation chains: (r0⇐r1,r2,…,rℓ) with ℓ∈[Lmin,Lmax] . Optionally apply length-weighted sampling with temperature τ (shorter rules sampled more often). Acyclicity is enforced via a dependency graph over relations.
Backbone graph initialization
Construct an initial directed multigraph by instantiating each sampled rule on fresh entity nodes: create a path Qr1⋯rℓQ′ and add the implied edge Qr0Q′ . Maintain per-relation entity pools (“repeated entities”) for later attachment.
Graph growth (structural prior)
Iteratively add new nodes until reaching n entities. Each new node attaches m edges by sampling relations constrained by an adjacency map derived from rule co-occurrence (also supporting a power-law attachment mode).
Appendix
Table 5: Synthetic knowledge graph generation via a latent-rule process: rule sampling, graph growth, deductible edge injection, and difficulty-controlled evaluation splits.
Figure 20: Learning dynamics across sparsity metrics and accuracy. We track Top-5% Energy, Top-10% Energy, accuracy, and normalized Effective Rank across training checkpoints for the controlled graph experiment. Early training shows increasing energy concentration and decreasing effective rank, while later training shows partial recovery of effective rank and stabilization of accuracy, supporting the two-phase feature-selection and feature-consolidation interpretation.
Figure 21: Learning dynamics for Trans-16-16. We track Top-5% Energy, Top-10% Energy, accuracy, and normalized Effective Rank across training checkpoints in the controlled graph environment. Compared with the larger Trans-32-32 model, Trans-16-16 exhibits a similar early concentration phase, where Top- k Energy increases and Effective Rank decreases, followed by a relaxation phase in which energy concentration weakens and Effective Rank recovers. This provides additional evidence that the sparsity dynamics are tied to feature selection and consolidation during training, rather than being an artifact of a single model scale.
Humans can apply ideas learned in one context to substantially different situations. We call this process of searching for and constructing novel recombinations of learned relationships to solve new problems out-of-distribution (OOD) reasoning. The capacity for large language models (LLMs) to support OOD reasoning underpins proposals for generally intelligent systems. However, this property is challenging to measure because most problems admit many decompositions, some involving shallow subproblems and others involving subproblems that may have been memorized from the training data. This makes it difficult to determine how much compositional reasoning a model must actually perform. Uncertainty about training distributions, how to measure a datapoint's distance from a training distribution, and the exponential number of ways to decompose most tasks make it intractable to robustly measure any modern LLM's OOD-reasoning capacity on standard natural-language tasks. In this work, we introduce a benchmark in which a model progressively solves a Boolean circuit over GF(2) from data. This benchmark is both minimal, isolating OOD reasoning from these confounding factors, and general, as any computable function can be represented as a sufficiently large GF(2) polynomial. We find that standalone models' next-step accuracy collapses as depth grows. In contrast, tool use through the synthesis and execution of code prevents this collapse in both small and frontier LLMs. These results indicate that synthesizing tools play a crucial role in supporting OOD reasoning.
David Koplow, Tomer Galanti, Tomaso Poggio
Massachusetts Institute of Technology · Texas A&M University
Recent white-box OOD detection methods for LLMs -- including CED, RAUQ, and WildGuard confidence scores -- appear effective, but we show they are structurally confounded by sequence length (|r| >= 0.61) and collapse to near-chance under length-matched evaluation. Even raw attention entropy (mean H(alpha) across heads and layers), a natural baseline we include for completeness, shows the same confound. The confound stems from attention's Theta(log T) dependence on input length. To identify genuine OOD signals after deconfounding, we propose a two-pathway framework: embeddings capture what text is about (effective for topic shifts), while the processing trajectory -- hidden-state evolution across layers -- captures how the model processes input. The relative power of each pathway varies along a vocabulary-transparency spectrum: embedding methods excel on vocabulary-distinctive OOD, while trajectory features detect covert-intent inputs that share vocabulary with normal text (0.721 avg AUROC; Jailbreak: 0.850). Three evidence lines support this framework: (1) a crossover between k-NN and trajectory scoring across 6 tasks, where each pathway wins on different OOD types; (2) a per-layer analysis showing that layer-0 k-NN signal is almost entirely a length artifact (Jailbreak: 0.759 raw -> 0.389 matched) -- processing constructs genuine OOD signal from near-chance embeddings; and (3) circuit attribution showing adversarial tasks engage attention circuits more than semantic tasks (p = 0.022; Jailbreak patching p < 0.001), with partial cross-model replication. Code release upon publication.
LLMs "reason" by externalizing intermediate steps as tokens through chain-of-thought (CoT), yet even frontier reasoning models hit abrupt performance cliffs on long-horizon tasks. Implicit reasoning (reasoning within a single forward pass) is useful during CoT generation, as the model can "search ahead" to decide the correct next move without explicit search. Models with weak implicit reasoning are more likely to commit to wrong paths and must attempt to recover via backtracking, while models with strong implicit reasoning identify the correct proof strategy directly, shortening the CoT. However, prior work shows LLMs struggle to reason implicitly beyond shallow depths, with no relief from scale. We investigate whether curriculum learning can extend this depth by fine-tuning on examples with progressively increasing vocabulary size and lookahead (the number of steps a model must search ahead to identify the correct next move). At matched compute, curriculum-trained models achieve lookaheads where no-curriculum training fails. Qwen3-0.6B reaches 98% accuracy on problems with lookahead up to 256 with curriculum, compared to 45% without. However, progress slows as fine-tuning continues, following a saturating exponential that suggests a maximum intrinsic lookahead to which models can learn to reason implicitly. Larger models reach higher fitted ceilings, fine-tuning pretrained models far outpaces training from scratch, and curriculum step sizes that are too large stall progression. Finally, the capability acquired on synthetic data transfers to natural language reasoning, with curriculum-trained models outperforming models trained without curriculum at matched compute on 11 natural language reasoning benchmarks.