Recent work has shown that large language models (LLMs) exhibit strong numerical sequence modeling capabilities and show promise in time-series prediction. While LLMs display in-context learning capabilities, the mechanisms with which they accomplish time-series prediction remain unclear. Specifically, whether they truly understand the underlying structure, which at a minimum requires reasoning over first differences in the sequence of numbers. To study this, we investigate Llama 3.1-8B from a mechanistic interpretability point of view. Mechanistic interpretability is an emerging field concerned with the reverse engineering of the algorithms learned by neural networks such as LLMs. To assess Llamas' numerical sequence modeling capabilities and to facilitate our mechanistic interpretability analysis, we create a sequence modeling task that cannot be solved without picking up structural cues. Specifically, we sample n random numbers and repeat them with an offset. We find that Llama displays strong performance on our tasks suggesting that it can pick up on the underlying structure. To understand the mechanisms that allow it to do so, we perform probing experiments and activation patching based counterfactual analysis. Probing reveals that the model computes and stores first differences in its internal representations without explicit supervision, indicating that it tracks structural information about the sequence. Activation patching reveals that Llama retrieves the relevant first-difference with a mechanism similar to an induction circuit and subsequently adds it to the current value. Notably, our work represents one of the first studies to identify this form of concept induction in LLMs.
Figures & tables
Figure 1. Illustrates how the model identifies a repeated first-difference (delta) pattern after differencing, then extrapolates by locating and copying the delta that immediately follows the final observed one, and adding it to last number.
Figure 2. (a) Shows the input signal, which consists of all unique values which means no number appears twice in the sequence. (b) Shows the deltas of the input signal, revealing that a cycle or pattern emerges only after applying the differencing operation.
Index
0
1
2
3
4
5
23
24
25
26
27
31
32
33
34
35
36
37
55
56
57
58
59
Token
S
403
,
405
,
400
411
,
417
,
409
416
,
436
,
438
,
433
444
,
450
,
442
Delta
2
-5
7
6
-8
-2
20
2
-5
7
6
-8
Table 1. Table shows selected index positions, tokens, and their corresponding first differences from the tokenized input. A recurrence in the first difference after token 33 marks the onset of a repeating cycle.
Figure 3. Layerwise linear decodability of first differences and final numbers. Mean absolute error (MAE; top) and coefficient of determination ( R2 ; bottom) for linear probes trained to decode delta (left) at 27 th position and final number (right) at 57 th position from hidden states across layers. Solid green lines denote performance on true targets; dashed red lines denote control probes trained on shuffled labels. Delta show consistently strong linear decodability, while final number representations gradually lose decodability.
Figure 4. Illustrates the query-hidden state patching technique that we use to test whether the model performs arithmetic composition by retrieving a first difference from an earlier position. The clean sequence (bottom) contains a structured delta pattern, while the corrupt sequence (top) is flat-valued. We transplant activations from the clean run into the corrupt run from a given layer l onward. We patch hidden states (shown with blue arrows) at layers l to L at one token position at a time except the beginning-of-sequence and final two tokens. We patch the query vector of the final token (shown with orange arrows) from the clean run at layers l+1 to L , ensuring the model attends to the correct location encoding the target first difference. This setup isolates and tests the causal contribution of induction and addition behaviors in the model’s prediction pipeline.
Figure 5. Heatmap of probability difference scores ( ΔP ) across layers and token positions, following patching of the final query vector from layers l+1 to L and hidden states from layers l to L . A sharp causal effect emerges at token 27, highlighted in red, indicating that the final query attends to position 27 to retrieve the locally stored first difference that follows the last observed one.
Figure 6. Token importance scores ( TIt ) across sequence positions. Peaks highlight the model’s use of structural alignment for pattern detection and delta retrieval.
Figure 7. Shows key-swap intervention shifts confidence from ground truth to counterfactual, revealing the role of position-aware attention alignment in delta retrieval.
Figure 8. Shows ground truth remains more probable post key-swap, indicating the model prioritizes delta-based composition over redirected attention.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9. Illustrates the head-level patching intervention that we use to isolate the functional role of individual attention heads in pattern recognition and delta retrieval. The clean sequence (bottom) contains structured first-difference patterns, while the corrupt sequence (top) is flat-valued. At each layer l , we patch only the output of a single attention head (at the final token position) from the clean forward pass into the corrupt one (shown with orange arrow), while all other heads and components remain unchanged. This targeted intervention tests whether the selected head causally contribute to the recognition of repeating structure, retrieval of a critical first difference, and contributes to accurate extrapolation in the final prediction.
Figure 10. Illustrates the key-swapping intervention that we use to disentangle the model’s reliance on phase (positional) cues versus delta-based representations during extrapolation. We swap keys from token positions 25 and 27—both identified as causally important—are swapped from layer l onward (indicated by orange and gray arrows), while preserving the respective value vectors from layer l onward. This setup tests whether the model selects the correct first difference for addition based on the original delta identity stored in the value vector or shifts its prediction in response to the redirected positional cue, thereby revealing its bias toward phase.
Does structure in representations imply structure in computation? We study how Llama-3.1-8B reasons over cyclic concepts (e.g., "what month is six months after August?"). Even though Llama-3.1-8B's representations for these concepts are circularly structured, we find that instead of directly computing modular addition in the period of the cyclic concept (e.g., 12 for months), the model re-uses a generic addition mechanism across tasks that operates independently of concept-specific geometry. First, it computes the sum of its two inputs using base-10 addition (six + August=14). Then, it maps this sum back to cyclic concept space (14->February). We show that Llama-3.1-8B uses task-agnostic Fourier features to compute these sums--in fact, these features have periods that respect standard base-10 addition, e.g., 2, 5, and 10, rather than the cyclic concept period (e.g., 12 for months). Furthermore, we identify a sparse set of 28 MLP neurons re-used across all tasks (approximately 0.2% of the MLP at layer 18) that can be partitioned into disjoint clusters, each computing the sum for a Fourier feature with a different period. Our work highlights how an interplay between causal abstraction and feature geometry can deepen our mechanistic understanding of LMs.
Sheridan Feucht, Tal Haklay, Usha Bhalla +9
aNortheastern University · bTechnion IIT · cHarvard University +1
Pretrained large language models (LLMs) have demonstrated in-context learning (ICL) capabilities for numerical inference over sequences serialized as text. Prior work has identified and characterized this form of numerical inference primarily through output-level evaluations such as prediction error. However, how numerical information is organized within LLM representations remains much less understood. To study this internal organization, we adopt a graph signal processing perspective in which attention induces a weighted graph over tokens, while token hidden states define signals on its nodes. Quantitative graph-spectral diagnostics and qualitative token-graph visualizations reveal that representations become more clearly differentiated by input dynamical complexity as context length increases. Simpler inputs produce attention-induced token graphs with stronger global connectivity and smoother, spectrally concentrated hidden-state signals, whereas more complex inputs produce more localized graphs and hidden-state signals with broader spectral support and greater high-frequency energy. Together, these findings point to systematic, context-dependent internal signatures associated with numerical ICL that are conserved across model families.
Jiajun Bao, Zihao Qi, Toni J. B. Liu +4
1Cornell University · 2Goodfire AI · 3Imperial College London
Time Series Foundation Models (TSFMs) excel at numerical forecasting but operate as black boxes lacking qualitative reasoning. Conversely, applying LLMs directly to temporal data introduces a modality gap: text tokenizers fragment continuous numerical values, degrading mathematical relationships and exploding sequence lengths, leading to computational overhead. To resolve this, we introduce STRIDE (Strategic Time-series Reasoning Injected via Distilled Embeddings), a novel framework natively integrating LLM reasoning into the continuous embedding space of TSFMs. Instead of discrete tokens, STRIDE distills reasoning traces into a lightweight LLM and dynamically projects its pooled hidden states as a continuous cross-modal prior into the numerical encoder. Jointly optimized via cross-entropy and quantile losses, STRIDE readily attaches to diverse architectures (e.g., Chronos-2, Timer-S1). STRIDE ranks #1 on the GIFT-Eval leaderboard (0.625 MASE, 0.423 CRPS) when ensembled with Synapse. On TFRBench, STRIDE consistently outperforms baselines in both forecasting accuracy and reasoning quality, with ablations confirming gains stem from strategic reasoning. Thus, injecting semantic reasoning as a continuous prior equips TSFMs with human-interpretable reasoning while improving predictive accuracy.