RIFT: Token Influence Follows the Path of Least Resistance

Tingan Jin; Shuhang Dong; Haosong Li

cs.CLOct 12, 2026

RIFT: Token Influence Follows the Path of Least Resistance

Authors: Tingan Jin, Shuhang Dong, Haosong Li

Organizations: University of California, Los Angeles Los Angeles, CA, USA · Independent Researcher

Abstract

Which context tokens drive a language model's answer? Graph-based tools such as attention rollout and attention flow answer this from a single forward pass, yet their faithfulness in decoder-only models is rarely tested against interventions. We view the layers of a decoder as a causal network whose conductances are attention weights, and read a token's influence on the answer from the resistance between them. We propose RIFT (Resistance-based Information Flow in Transformers), which scores a token by the series resistance of its path of least resistance to the answer, found exactly by a classic shortest-path dynamic program. On synthetic in-context retrieval tasks and extractive question answering over Wikipedia paragraphs, RIFT predicts the effect of replacing a token with a Spearman correlation of 0.52-0.68; predicting a single replacement, it reaches 77-96% of the agreement between two different random replacements. It needs no gradients and no extra forward passes, scoring every token of a prompt in 0.9-3.4 ms on one CPU thread, whereas the replacement experiment needs a forward pass per token. On 576 held-out prompts from four models of two families (Qwen3 and Llama 3.2), it outperforms token distance, attention rollout and attention flow in every pre-specified comparison, by 0.23 to 0.88 in mean Spearman correlation, and it is the best or tied-best of all attention-based scores we test. Aggregates that add up all routes (attention flow, effective resistance and rollout) instead mostly encode token position: as a simple proposition shows, early tokens have many more parallel routes to the answer.

Figures & tables

Panel of Figure 1 — Fig. 1: The resistance view. A decoder unrolled over layers is a causal network; node $(\ell,i)$ is token $i$ after layer $\ell$ , and edges carry attention mixing weights (plus the residual) as conductances. An early token reaches the answer through many parallel routes (gray), which all-path aggregates add up. RIFT scores only the strongest single route, the path of least resistance (blue).

								Advantage of RIFT over
	Model	Prompts per task	Layout	Acc. %	Rel.	RIFT 1	RIFT	distance	rollout	attention flow
Synthetic tasks: confirmation (held out, pre-specified)
A	1.7B base	single 32, first 32	mixed	95	0.61	0.49	0.53	+0.41 [+0.32, +0.49]	+0.47 [+0.39, +0.55]	+0.64 [+0.55, +0.74]
B	1.7B instruct	single 43, two-hop 21	mixed	92	0.71	0.60	0.63	+0.63 [+0.53, +0.72]	+0.34 [+0.26, +0.41]	+0.63 [+0.55, +0.72]
C	4B instruct	single 26, two-hop 13, first 25	mixed	97	0.67	0.51	0.53	+0.42 [+0.33, +0.50]	+0.50 [+0.43, +0.57]	+0.65 [+0.57, +0.72]
D	1.7B base	single 32, first 32	interleaved	86	0.67	0.53	0.55	+0.64 [+0.52, +0.75]	+0.40 [+0.31, +0.48]	+0.47 [+0.37, +0.57]

TABLE I: Predicting causal token influence: mean within-prompt Spearman correlation with the replacement label (the mean of two replacements), and the paired advantage of RIFT over each baseline with 95% bootstrap intervals. Acc.: share of the scored prompts answered correctly. Rel.: agreement between two single replacements; RIFT 1 : RIFT against a single replacement, the like-for-like counterpart of Rel. A–E and H use Qwen3, F, G and I Llama-3.2-3B-Instruct; H and I are correctly answered SQuAD questions, with the log-probability label. In the Qwen3 datasets, attention flow ranks tokens from earliest to latest (exactly in all but one prompt), so its correlation with the label is the negative of that of distance.

Task	$N$	Rel.	RIFT	Distance	Rollout	Attn. flow
Single-hop	283	0.62	0.55	$-$ 0.11	0.23	0.08
Two-hop	34	0.84	0.68	0.50	$-$ 0.03	$-$ 0.50
First-value	131	0.65	0.56	0.04	0.21	$-$ 0.04
SQuAD	128	0.65	0.68	$-$ 0.15	0.32	0.14

TABLE II: Confirmation results by task family (prompt-weighted means over the nine datasets;

N

is the number of prompts). Rel.: agreement between two single replacements; the score columns use their mean.

Panel of Figure 4 — Fig. 2: Confirmatory results for the nine held-out datasets (A–E and H: Qwen3; F, G and I: Llama-3.2-3B-Instruct; H and I: SQuAD). RIFT (blue) is far above token distance and rollout in every dataset; the widest path (violet), its single-route sibling and a diagnostic outside the decision rule, comes close. Attention flow is omitted (for Qwen3 it is the mirror image of distance); Table I gives all values with the label reliability.

Panel of Figure 5 — Fig. 3: A representative SQuAD question for Qwen3-1.7B (dataset H; 207 tokens): “What type of animal did onlookers claim was piloting the boat?” (answer: monkey ). Columns are the 12 scored words in position order (subscript: position; the answer word and its question words in bold); darker cells mark tokens ranked as more influential within the prompt.

Panel of Figure 6 — Fig. 4: Accuracy versus cost on the SQuAD prompts (datasets H and I). Accuracy: Spearman correlation with the effect of one replacement (for the replacement experiment, the agreement between two replacements). Cost: median time to score every token of a prompt. RIFT, rollout and distance are timed on one CPU thread, attention flow and effective resistance from their per-source solver times, and the replacement experiment from its per-pass GPU time (one forward pass per token).

	$-$ max. attention	$-$ widest path	$-$ multiplicative path
A	+0.05 [+0.01, +0.10]	+0.05 [+0.02, +0.08]	+0.03 [ $-$ 0.01, +0.06]
B	+0.20 [+0.15, +0.24]	+0.00 [ $-$ 0.03, +0.03]	+0.10 [+0.08, +0.13]
C	+0.15 [+0.11, +0.20]	$-$ 0.01 [ $-$ 0.03, +0.02]	+0.09 [+0.06, +0.12]
D	+0.03 [ $-$ 0.01, +0.06]	+0.05 [+0.01, +0.08]	+0.01 [ $-$ 0.02, +0.04]
E	+0.18 [+0.14, +0.23]	$-$ 0.01 [ $-$ 0.04, +0.03]	+0.09 [+0.05, +0.12]
F	+0.05 [+0.01, +0.09]	+0.04 [ $-$ 0.01, +0.08]	+0.04 [+0.01, +0.08]

TABLE III: Advantage of RIFT over other attention-based scores (paired difference in mean Spearman with 95% intervals; computed after confirmation, diagnostic).

Score	Position $\rho$	Label $\rho$
Token distance	1.00 / 1.00	0.13 / 0.00
Attention rollout	$-$ 0.73 / $-$ 0.60	0.02 / 0.28
Attention flow	$-$ 1.00 / $-$ 1.00	$-$ 0.13 / 0.00
Eff. resistance, attention	$-$ 0.96 / $-$ 0.91	$-$ 0.14 / $-$ 0.01
Eff. resistance, Jacobian	$-$ 0.99 / $-$ 0.98	$-$ 0.14 / $-$ 0.02
Structural null	$-$ 1.00 / $-$ 1.00	$-$ 0.13 / 0.00

TABLE IV: Position dependence versus validity (exploration, 192 prompts per 1.7B model, base / instruct). Position: Spearman with inverse distance to the answer; label: Spearman with the replacement label. Structural null: effective resistance with every conductance set to the same constant.

Panel of Figure 9 — Fig. 5: Position dependence as the attention graph deepens (mean of the two 1.7B models). All-path aggregates become functions of position within a few layers; attention flow (hollow diamonds) and effective resistance nearly coincide. RIFT does not. Negative values mean that earlier tokens score higher.

Transformer Interpretability

Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.

See all 338 papers

Oct 19, 2026

DeltaTrace: Efficient Signed Attribution for Reasoning Language Models

We study how words in the context support or oppose a language model's response and present DeltaTrace, a novel attribution framework that explains the effect of input tokens on a fixed model response. DeltaTrace measures how the response's log-probability changes when the context tokens being explained are masked. To distribute this difference among source tokens, it constructs operator-specific local rules for finite changes and composes them backward through the computational graph using effective secant slopes. Each token is assigned a signed attribution as the inner product of its embedding displacement and the corresponding backward-propagated slope. Local conservation makes these contributions sum to the measured score difference and preserves their meaning when aggregated over sentences or passages. DeltaTrace supports both attention and linear-attention models, with auxiliary memory growing linearly with sequence length at fixed model dimensions and chunk size. On the evaluated attention model, DeltaTrace achieves the lowest deletion-faithfulness scores (RISE and MAS) on most retrieval and reasoning tasks and the highest needle-in-a-haystack Recall, exceeding FlashTrace by 5.67-24.33 percentage points. On the evaluated hybrid attention and linear-attention model, DeltaTrace lowers RISE on 9 tasks and MAS on 12 relative to FlashTrace.

Chen Liang, Daoping Wu, Donghua Yang +4

Harbin Institute of Technology · Zhejiang Lab

Feature Attribution LLM Interpretability Faithfulness of Language Model Explanations

Ask PDF

Jun 24, 2025

RCStat: A Statistical Framework for using Relative Contextualization in Transformers

Identifying which tokens and activations truly influence model predictions is critical for both the efficiency and interpretability of large auto-regressive language models. Yet in practice, token importance is typically inferred from attention distributions that entangle contextual relevance with softmax-normalization effects, obscuring relative influence across tokens and heads. We propose

\textbf{RCStat}

, a statistical framework for quantifying contextual influence in attention mechanisms. At its core

\textbf{Relative Contextualization (RC)}

is a random variable that measures how strongly one subset of tokens contributes to another under the model's internal scoring scheme. RCStat admits computationally efficient bounds on expected influence that can be estimated at inference time. No retraining is required. We apply RCStat to two tasks. For

\textbf{attribution}

, attention heads with high expected RC accurately identify the prompt spans that drive generation, enabling reliable span-level explanations. For

\textbf{key-value cache eviction}

, RC-based adaptive thresholding selectively evicts low-impact KV entries, substantially reducing cache size while preserving generation quality. Across question answering, summarization, and attribution benchmarks, RCStat achieves consistent gains, improving generation quality by 15-40% with upto 36% error reduction for KV eviction, and attribution accuracy by 2-16%. These results demonstrate that explicitly modeling contextual influence provides a principled and practical alternative to attention-based heuristics.

Debabrata Mahapatra, Shubham Agarwal, Apoorv Saxena +1

Adobe Research, India

Attention Mechanisms Transformer Interpretability LLM Inference Acceleration KV-Cache Eviction

Ask PDF

Aug 20, 2026

Ask Self, Ask Others: Relation Is All You Need

Attention dominates token mixing, but it collapses relation formation and flow allocation into a single score-to-flow step. We introduce Relation, which separates them by first organizing pairwise evidence into explicit Self and Exchange relations and deriving information flow afterward. Relation first decides whether a token should rely on itself or draw from its history, and if it draws from history, where to look. This relational organization gives rise to Full Relation, FlashRelation, Linear Relation, and Hybrid Relation. Across matched decoder-only models, Full Relation achieves lower mean final-validation NLL than MHA and reaches the paired MHA final training loss with 4.5-7.3% fewer tokens. Structural diagnostics further show that Relation learns a distinct depth organization: the first layer acts as a current-token anchor and a high-rank router, while later layers shift strongly toward history. In a fixed-context reference benchmark, FlashRelation is 4.17-5.28x faster than the materialized Full Relation implementation. Across scale-matched production workloads, it reaches 89.7-92.9% of PyTorch FlashAttention throughput while executing the exact Full Relation operator. Hybrid Relation demonstrates that Full and Linear Relation layers can be composed within a single decoder. These results support a relation-first view of token mixing: ask Self, ask Others, then let Flow follow Relation.

Yuting Ge, Pengju Yang, Mingkai Nie

Department of Electrical Engineering City University of Hong Kong · Jilin University · National University of Singapore

Self-Attention Decoder-Only Language Models Attention Mechanisms Efficient Language Model Inference Hybrid Attention

Ask PDF

Abstract

Transformer Interpretability

Explore similar work

DeltaTrace: Efficient Signed Attribution for Reasoning Language Models

RCStat: A Statistical Framework for using Relative Contextualization in Transformers

Ask Self, Ask Others: Relation Is All You Need