Large language models (LLMs) make next-token predictions based on clues present in their context, such as semantic descriptions and in-context examples. Yet, elucidating which prior tokens most strongly influence a given prediction remains challenging due to the proliferation of layers and attention heads in modern architectures. We propose Jacobian Scopes, a suite of gradient-based, token level causal attribution methods for interpreting LLM predictions. Grounded in perturbation theory and information geometry, Jacobian Scopes quantify how input tokens influence various aspects of a model's prediction, such as specific logits, the full predictive distribution, and model uncertainty (effective temperature). Through case studies spanning instruction understanding, translation, and in-context learning (ICL), we demonstrate how Jacobian Scopes reveal implicit political biases, uncover word- and phrase-level translation strategies, and shed light on recently debated mechanisms underlying in-context time-series forecasting. On token-ablation faithfulness benchmarks (LAMBADA and IWSLT2017) across six LLMs, the distribution-level Fisher and Temperature Scopes significantly outperform established gradient-based attribution methods in 10 of 12 model-dataset settings. To facilitate exploration of Jacobian Scopes on custom text, we open-source our implementations and provide a cloud-hosted interactive demo at https://huggingface.co/spaces/Typony/JacobianScopes.
Figures & tables
Figure 1: Overview of Jacobian Scopes. (a) An autoregressive LLM maps an input token sequence to a hidden state y , which is decoded into a predictive distribution over the vocabulary. (b) A small perturbation to any input token is propagated to the output via the Jacobian Jt , then projected onto a direction v in output space; the magnitude of this projection defines the token’s influence score, computable in a single backward pass. The three Scopes correspond to v=wtarget (target unembedding row), v=u1 (principal Fisher direction), and v=y^ (normalized hidden state). (c) The three Jacobian Scope variants each target a different feature of the model’s output: Semantic Scope asks which input tokens drive the predicted probability of a specific target word; Fisher Scope asks which tokens most change the overall shape of the predicted distribution; and Temperature Scope asks which tokens control the model’s confidence, i.e., how sharply peaked the distribution is.
Figure 2: Semantic Scope reveals the potential for political bias that may be present in models such as LLaMA-3.2. The prediction for the subject being a “liberal” is attributed to the input token “Columbia”, while the prediction for “conservative” is attributed to the token pair “the South”.
Figure 3: Fisher Scope visualizes LLaMA-3.2 3B’s strategy for translating Dante’s Inferno [ 2 ] . Left: Magnitude of influence from each source word (blue) on the predicted distribution at each target position (red). Right: Probability distribution of the top 7 words predicted at each target position. (a) LLaMA assigns high probability to “world”, the unambiguous literal translation of “mondo”. (b) When translating the phrase “priegoti ch’a la mente altrui mi rechi” (literally: “I pray thee to bring me to another’s mind”), LLaMA’s predicted distribution is highly uncertain, with each top token hinting at a different phrasing strategy.
Figure 4: Left: Temperature Scope reveals different strategies employed by LLaMA-3.2 during time-series forecasting: (a) When continuing the dynamics of a partially observed Lorenz system, LLaMA attends to regions in the input history that exhibit patterns similar to those near the cutoff. (b) When extrapolating unstructured Brownian motion, LLaMA shows diminished influence of early context. Right: Predicted probability distribution over the top 30 most probable tokens.
1B/1.5B-scale
LLaMA-3.2 1B
Qwen2.5-1.5B
Gemma-3 1B
Method
LMB
IWS
LMB
IWS
LMB
IWS
Random
−0.26±0.01
−0.30±0.01
−0.35±0.01
−0.25±0.01
−0.42±0.01
−0.41±0.01
Integrated Grads
−1.19±0.01
−1.04±0.01
−1.71±0.01
−1.36±0.01
−0.69±0.01
−0.87±0.01
Input × Grad
−1.28±0.01
−1.04±0.01
−1.72±0.01
−1.35±0.01
−1.56±0.01
−1.36±0.01
Semantic Scope
−1.30±0.01
−1.06±0.01
−1.68±0.01
−1.24±0.01
−1.64±0.01
−1.30±0.01
Table 1: AOPC (higher magnitude is better) on LAMBADA (LMB) and IWSLT2017 DE → EN (IWS) for all six models, averaged over ablation fractions k∈{5%,10%,20%} via the trapezoidal rule over 1000 test passages.
Figure 5: Performance of Fisher Scope improves slightly with increasing rank r in the low-rank approximation ( eq. 23 ), on LLaMA-3.2 1B and LAMBADA (1000 passages). Dashed lines show rank-free methods; shaded bands are ±1 standard error. Rank 1, the default Fisher Scope, captures nearly all of the attainable faithfulness with a single backward pass.
Semantic Scope
Fisher Scope
Temperature Scope
Quantity explained
target logit z
full distribution p
confidence ∥y∥2
Influence score
∥wtarget⊺Jt∥2
∥u1⊺Jt∥2
∥y^⊺Jt∥2
Recommended use case
semantic reasoning & bias analysis
NLP tasks with non-unique predictions
in-context learning on numerical data
Table 2: Comparison of the Jacobian Scopes methods introduced. wtarget denotes the unembedding vector of the target token, u1 the principal Fisher direction, and y^ the normalized hidden direction.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Fisher Scope attribution examples on (a) LAMBADA and (b) IWSLT2017 DE → EN, using LLaMA-3.2 3B. Darker blue indicates higher influence; the target token is highlighted in red. Right: Probability distribution of the top 7 words predicted at the target position.
1B/1.5B-scale
LLaMA-3.2 1B
Qwen2.5-1.5B
Gemma-3 1B
Method
LMB
IWS
LMB
IWS
LMB
IWS
Random
−0.41±0.01
−0.36±0.01
−0.51±0.02
−0.43±0.02
−0.63±0.01
−0.85±0.01
Integrated Grads
−1.36±0.02
−1.27±0.02
−2.06±0.02
−2.08±0.03
−0.81±0.01
−1.02±0.01
Input × Grad
−1.61±0.02
−1.46±0.02
−2.06±0.02
−2.09±0.03
−0.80±0.01
−1.03±0.01
Semantic Scope
−1.63±0.02
−1.49±0.02
−2.00±0.02
−1.92±0.03
−0.80±0.01
−1.03±0.02
Appendix
Table 3: AOPC under PAD-token ablation (higher magnitude is better) on LAMBADA (LMB) and IWSLT2017 DE → EN (IWS), averaged over k∈{5%,10%,20%} via the trapezoidal rule. Best or tied-best entries in each column are bolded.
Model
Dataset
Scope
Baseline
Mean Δ [95% CI]
p
LLaMA-3.2 1B
LAMBADA
Fisher
Input × Grad
−0.036[−0.048,−0.024]
1.2×10−6
LLaMA-3.2 1B
IWSLT
Temperature
IG
−0.049[−0.073,−0.025]
4.0×10−4
LLaMA-3.2 3B
LAMBADA
Fisher
Input × Grad
−0.055[−0.071,−0.039]
1.1×10−11
LLaMA-3.2 3B
IWSLT
Fisher
Input × Grad
−0.029[−0.042,−0.015]
3.9×10−4
Qwen2.5 1.5B
LAMBADA
Temperature
Input × Grad
−0.141[−0.168,−0.114]
3.1×10−21
Qwen2.5 1.5B
IWSLT
Temperature
IG
−0.025[−0.052,−0.002]
0.12
Appendix
Table 4: Paired AOPC tests (best Scope vs. best baseline from table 1 ). Negative Δ favors the Scope; bold entries are significant at p<0.05 (two-sided Wilcoxon).
Figure 7: Semantic Scope visualizes how LLaMA-3.2 understands system-prompt-style instructions. The logit assigned to the token “truthful” is most prominently attributed to the input token “deceive”, and “nice” to “argue”.
Figure 8: Fisher Scope reveals that translated words, highlighted in red, are primarily influenced by their lexical counterparts in the source language.
Figure 9: Semantic Scope attribution for detective micro-fiction. The attribution graph illustrates how LLaMA-3.2 3B connects suspects to their associated clues when reasoning about potential perpetrators and weapons in a novel-like setting. Attributions clarify the model’s process for linking evidence to conclusions across a long narrative context.
Figure 10: Temperature Scope shows LLaMA-3.2 attending to similar motifs in history when extrapolating a logistic map trajectory.
Figure 11: Temperature Scope attribution graph of LLaMA-3.2 extrapolating a partially observed Lorenz system, with a different initial condition from that in Figure 4 (a).
Figure 12: Temperature Scope attribution graph of LLaMA-3.2 extrapolating (a) a partially observed Lorenz system, and (b) the same Lorenz system with uniform drift.
Figure 13: Additional Temperature Scope attributions on LLaMA-3.2 extrapolating Brownian motion in context.
Figure 14: Left: (a) Temperature Scope, (b) Semantic Scope, and (c) Fisher Scope attributions for the same Lorenz system. Right: The predicted probability mass over the top 20 candidate tokens at the target position, ordered by their numerical values. The ground-truth token (shown in red) is used as the target for Semantic Scope.
Figure 15: Causal attribution using (a) Semantic Scope and (b) its path-integrated variant based on Integrated Gradients.
Figure 16: Token-wise ℓ2 norm of the Integrated Gradient integrand ∇z(X~(α)) along the interpolation path α∈[0,1] .
Figure 17: Justification for the rank-1 approximation used in Fisher Scope, evaluated on LLaMA-3.2 1B and LAMBADA. Top: Eigenvalue spectrum of F across 10 prompts on a log-log scale, exhibiting a consistent power-law decay spanning 6 orders of magnitude. Middle: Mean cumulative explained variance ∑i=1rλi/tr(F) averaged over 1000 passages (shaded band: ±1σ ); u1 alone accounts for ≈ 37% of total sensitivity (1 σ : 25–50%). Bottom: AOPC as a function of rank r∈{1,4,16,64,256} , with baselines shown as dashed lines (reproduced in Figure 5 ). Rank-1 Fisher Scope already outperforms Input × Gradient and Semantic Scope; increasing r yields only marginal improvement at r -fold computational cost.
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
The generative nature of Large Language Models (LLMs) is reflected in the conditional probabilities they compute to sample each response token given the previous tokens. These probabilities encode the distributional structure that the model learns in training and exploits in inference. In this work, we use these probabilities to situate LLMs within the mathematical theory of stochastic processes. We use this framework to design a model-agnostic probabilistic token attribution measure, using Bayes rule to invert the next-token log-probabilities so as to capture the models internal representation of the distribution over token sequences. The representation is independent of the models computational structure. This representation yields the conditional probability of the response given the prompt, and of the response given the prompt with a token marginalized away. Our attribution score is the log of the ratio of these probabilities. We further compute the entropies of a single prompts token distributions, conditioned on the remaining context. The interplay between entropy and attribution score sheds light on LLM behavior. We evaluate 8 models across 7 prompts and investigate anomalies, token sensitivity, response stability, model stability, and training convergence, thereby improving interpretability and guiding users to focus on uncertain or unstable parts of the generation.
Shilpika Shilpika, Carlo Graziani, Bethany Lusch +2
Argonne Leadership Computing Facility Argonne National Laboratory Lemont, IL, USA · Mathematics and Computer Science Division Argonne National Laboratory Lemont, IL, USA · Department of Computer Science University of Illinois Chicago Chicago, IL, USA
Large Language Models (LLMs) are increasingly deployed across diverse applications, raising critical questions for governance, accountability, and data provenance. Understanding which training data most influenced a model's output remains a fundamental open problem. We address this challenge through training data attribution (TDA) for auto-regressive LLMs by expanding upon the inverse formulation: How would training data be affected if the model had seen the generated output during training? Our method perturbs the base model using bidirectional gradient optimization (gradient ascent and descent) on a generated text sample and measures the resulting change in loss across training samples. Our framework supports attribution at arbitrary data granularity, enabling both factual and stylistic attribution. We evaluate our method against baselines on pretrained models with known datasets, and show that it outperforms previous work on influence metrics, thereby enhancing model interpretability, an essential requirement for accountable AI systems.
Frédéric Berdoz, Luca A. Lanzendörfer, Kaan Bayraktar +1
A critical step for reliable large language models (LLMs) use in healthcare is to attribute predictions to their training data, akin to a medical case study. This requires token-level precision: pinpointing not just which training examples influence a decision, but which tokens within them are responsible. While influence functions offer a principled framework for this, prior work is restricted to autoregressive settings and relies on an implicit assumption of token independence, rendering their identified influences unreliable. We introduce a flexible framework that infers token-level influence through a latent mediation approach for general prediction tasks. Our method attaches sparse autoencoders to any layer of a pretrained LLM to learn a basis of approximately independent latent features. Unlike prior methods where influence decomposes additively across tokens, influence computed over latent features is inherently non-decomposable. To address this, we introduce a novel method using Jacobian-vector products. Token-level influence is obtained by propagating latent attributions back to the input space via token activation patterns. We scale our approach using efficient inverse-Hessian approximations. Experiments on medical benchmarks show our approach identifies sparse, interpretable sets of tokens that jointly influence predictions. Our framework enhances trust and enables model auditing, generalizing to high-stakes domain requiring transparent and accountable decisions.
Shixing Yu, Promit Ghosal, Kyra Gan
Electrical and Computer Engineering · Cornell Tech · Department of Statistics +2