Shortcut Before Circuit: Document Statistics Time In-Context Conflict Resolution

Yijun Liao; Fanwei Liang

cs.CLAug 25, 2026

Shortcut Before Circuit: Document Statistics Time In-Context Conflict Resolution

Authors: Yijun Liao, Fanwei Liang

Abstract

When a context asserts two values for one fact, a model commits to a cue -- recency, repetition, position -- but natural data rarely makes these disagree, so behavior cannot reveal which. We train 26M-parameter transformers on a synthetic language where recency and rarity are exactly coextensive, and separate them with a minimal causal edit that inverts one cue while holding the truth, token count and answer position fixed. All 75 runs reach accuracy >= 0.999, including where the trivial heuristic fails, so no held-in evaluation distinguishes them. Under intervention the per-cell readout does not replicate: 13 of 25 cells differ by more than 0.3 in sign fraction across three seeds, the largest by 0.879 against a standard error of 0.025. The construction predicts this -- coextensive rules leave the objective indifferent between them -- and the variance is ordered by how much of the optimization each comparison releases. What replicates is timing: escape from a positional shortcut with a closed-form ceiling, monotone in redundancy. Probed before that escape, attribution reverses sign in 32 of 75 runs at unchanged accuracy, and gating on circuit formation is necessary but not sufficient. The corpus fixes when a mechanism appears, not which one -- a criterion for when mechanistic attribution to data is available at all, and our construction makes the unavailable case exact.

Figures & tables

Figure 1: (a) Sign fraction at 16000 steps for the 25 cells under three seeds, joined per cell; filled circles seed 0, open squares seed 1, open triangles seed 2. Line length is the range across seeds, reaching 0.879 against a binomial standard error of 0.025. (b) The

R_{\mathrm{old}}=3

\Delta D=3

run of seed 1 (§ 6.3 ):

d\,\mathrm{loss}/d\log\mathrm{step}

(left axis) and sign fraction (right axis). The interior peak marks circuit formation near step 4600; the readout crosses near step 12000 at unchanged accuracy (§ 6.3 ).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

$R_{\mathrm{old}}$	$\Delta D$	bindMax	$r_{\mathrm{len}}$	updDens	adj q	adj f	ant q	ant f
3	2	3	$-0.024$	1.029	0.056	0.078	13.32	10.92
3	3	3	$-0.023$	1.020	0.058	0.077	13.21	10.95
3	5	3	$+0.028$	1.022	0.056	0.074	13.09	11.00
3	8	3	$-0.035$	1.021	0.060	0.077	12.55	10.92
3	16	3	$-0.002$	0.984	0.074	0.076	11.49	11.03
5	2	3	$-0.003$	1.074	0.087	0.104	9.57	8.29

Appendix

Table 1: Generator invariants, 1500 documents per cell. Columns are defined in the text below: bindMax cross-document binding reuse,

r_{\mathrm{len}}

correlation between realized

\Delta D

and token count, ansLast trivial-copy documents, updDens pre-query update density in the final decile, adj / ant same-slot adjacency rate and antecedent distance for the queried (

q

) and filler (

f

) slots. The fifth invariant holds only partially: ant q exceeds ant f by 2–22%.

$R_{\mathrm{old}}$	$\Delta D$	support	posCeil	slots	$R_{\mathrm{real}}$	domain	tail-0
3	2	$[1,3]$	0.333	27.6	2.52	0.92	0.66
3	3	$[2,4]$	0.333	27.6	2.47	0.91	0.52
3	5	$[2,8]$	0.143	27.6	2.39	0.89	0.37
3	8	$[4,12]$	0.111	27.6	2.23	0.81	0.19
3	16	$[8,24]$	0.059	27.9	1.91	0.66	0.04
5	2	$[1,3]$	0.333	20.2	3.99	0.99	0.71

Appendix

Table 2: Covariates that shift with the axes, 1500 documents per cell. Columns are defined in the text above.

$R_{\mathrm{old}}$	$\Delta D$	rec	rar	frq	glu	pos
3	2	1.000	1.000	0.085	0.651	0.341
3	3	1.000	1.000	0.099	0.528	0.345
3	5	1.000	1.000	0.120	0.359	0.163
3	8	1.000	1.000	0.163	0.183	0.127
3	16	1.000	1.000	0.309	0.042	0.070
5	2	1.000	1.000	0.024	0.722	0.339

Appendix

Table 3: Fraction of documents on which each candidate rule selects the ground truth, 1500 documents per cell. rec recency, rar rarity, frq frequency, pri primacy, glu global last update, ltk last value token before the query, pos best fixed offset from the end.

run	$\gamma$	escape	mean	max	std	bound
$R_{\mathrm{old}}=3,\Delta D=5$ , s0	1.0	none	0.848	1.703	0.345	22.9
$R_{\mathrm{old}}=3,\Delta D=5$ , s1	1.0	none	0.958	2.156	0.425	36.9
$R_{\mathrm{old}}=16,\Delta D=2$ , s0	1.0	1000	1.067	1.602	0.248	20.5
$R_{\mathrm{old}}=16,\Delta D=2$ , s1	1.0	1000	1.023	1.727	0.274	23.6
main grid, 50 runs	2.0	1000–10000	1.47–1.74	2.11–2.55	0.15–0.35	33–52

Appendix

Table 4: Terminal QK-norm gains. mean , max and std are over all

2\,d_{h}\,n_{\mathrm{layer}}=1024

components; bound is

\max_{i}\lvert w^{(q)}_{i}w^{(k)}_{i}\rvert\sqrt{d_{h}}

, the right-hand side of Equation 4 , against the 7.6 that Equation 5 requires at

n=224

p=0.9

. Initial bounds are 8.0 at

\gamma=1

and 32.0 at

\gamma=2

. escape is the 1000-step gate. Main-grid rows are the range over the 50 runs of seeds 0 and 1.

cell	$\gamma$	escape	acc	copy	mass	median	frac+
$R_{\mathrm{old}}=3,\Delta D=5$ , s0	2.0	5000	1.000	1.000	1.00	$-0.087$	0.126
$R_{\mathrm{old}}=3,\Delta D=5$ , s0	1.0	none	0.092	0.132	0.15	—	—
$R_{\mathrm{old}}=3,\Delta D=5$ , s1	2.0	5000	1.000	1.000	0.66	$+1.964$	0.972
$R_{\mathrm{old}}=3,\Delta D=5$ , s1	1.0	none	0.194	0.244	0.15	—	—
$R_{\mathrm{old}}=16,\Delta D=2$ , s0	2.0	1000	1.000	1.000	0.98	$+1.827$	0.998
$R_{\mathrm{old}}=16,\Delta D=2$ , s0	1.0	1000	1.000	1.000	0.97	$+2.095$	0.993

Appendix

Table 5: The

\gamma=1

arm. escape is the first evaluation at which the copy diagnostic exceeds 0.95; none means it is never crossed within 16000 steps. mass is mean contrast-pair mass and median is ungated, both as in Table 15 . Readouts are omitted at

R_{\mathrm{old}}=3

\gamma=1

: mass is 0.15 there, with 0.11 and 0.02 of documents clearing the per-document gate, so those two runs report on circuit formation and not on rule attribution.

layers	$R_{\mathrm{old}}$	seed 0	seed 1
4	3	3400	5100
4	8	400	500 †
4	16	300 †	400
8	3	4200	4600
8	8	800	900
8	16	400	400

Appendix

Table 6: Loss-derivative peak position in the depth arm, both seeds, on the 100-step recording grid. † marks runs where the cosine schedule’s final decay exceeds the escape peak, so the search returns a position inside the decay rather than at formation (§ 6.2 ); those values are not usable. The 8-layer rows are the main grid’s (Table 9 ).

$R_{\mathrm{old}}$	$\Delta D$	support	$\lvert\mathrm{supp}\rvert$	posCeil	acc	copy	ratio
2	2	$[1,3]$	3	0.333	0.3255	0.002	98%
2	3	$[2,4]$	3	0.333	0.3253	0.001	98%
3	5	$[2,8]$	7	0.143	0.1045	0.002	73%
2	8	$[4,12]$	9	0.111	0.0773	0.046	70%
2	5	$[2,8]$	7	0.143	0.0018	0.003	1%

Appendix

Table 7: Positional ceiling against observed plateau accuracy. Accuracy is taken at the last evaluation before the copy diagnostic leaves chance, except in the last row (

R_{\mathrm{old}}=2

\Delta D=5

), where it never leaves chance and the value is the terminal one at 12000 steps.

$\Delta D$	support	posCeil	acc	copy	$\Delta$ / sign frac	state
2	$[1,3]$	0.333	0.3255	0.002	$-0.47$ / 0.05	positional, 98% of ceiling
3	$[2,4]$	0.333	0.3253	0.001	$-0.44$ / 0.01	positional, 98%
5	$[2,8]$	0.143	0.0018	0.003	$-0.00$ / 0.22	neither, 1%
8	$[4,12]$	0.111	0.0773	0.046	$-1.30$ / 0.02	positional, 70%
16	$[8,24]$	0.059	1.0000	0.999	$+0.33$ / 0.72	retrieval, escaped at 5000

Appendix

Table 8: The excluded

R_{\mathrm{old}}=2

column at 12000 steps, seed 0. Four of the five never acquire the retrieval circuit within the budget: three terminate positional and one neither . The probe reading on the three positional runs is the artifact analyzed in § 6.3 .

$R_{\mathrm{old}}$	$\Delta D$	gate	peak	$h$	gate	peak	$h$
		seed 0			seed 1
3	2	8000	7500	9.86	7000	6100	8.12
3	3	6000	5600	6.66	5000	4600	6.98
3	5	5000	4200	7.15	5000	4600	5.77
3	8	7000	6800	8.80	6000	5600	7.55
3	16	4000	3400	5.11	10000	9600	14.48

Appendix

Table 9: Escape step located two ways, both seeds. gate is the first evaluation at which the copy diagnostic exceeds 0.95 and is resolved only to the 1000-step evaluation grid. peak is the position of the single interior maximum of

d\,\mathrm{loss}/d\log\mathrm{step}

, recorded every 100 steps and involving no threshold, with its height

h

. The peak resolves escape ten times more finely and separates four levels of

R_{\mathrm{old}}

(

3

5

8

and

\{12,16\}

) where the gate separates three. Peak height increases with escape step (Spearman

\rho=0.933

over these 50 runs); the cosine schedule’s final decay exceeds the escape peak in the three cells marked † , so the unthresholded argument is weak there and we rely on the gate. Nineteen of the 50 runs share a peak position of 400 steps, the floor of the 100-step recording grid; the Spearman coefficient of § 6.2 uses average ranks over that tie group. The table covers seeds 0 and 1; seed 2’s gate and peak columns are in the text below. No seed-1 run is affected by the cosine tail.

Figure 2: Escape located from the peak of

d\,\mathrm{loss}/d\log\mathrm{step}

, recorded every 100 steps with no threshold; filled markers seed 0, open markers seed 1. Left: peak position against

R_{\mathrm{old}}

, with the 1000-step evaluation grid marked. Right: peak height against peak position over all 50 runs.

cell	budget	escape	median	sign fraction
$R_{\mathrm{old}}=16$ , $\Delta D=2$	4000	1000	$+2.363$	0.995
	16000	1000	$+1.811$	0.998
	32000	2000	$+2.433$	1.000
$R_{\mathrm{old}}=3$ , $\Delta D=2$	12000	2508	$-0.000$	0.43
	16000	8000	$-0.011$	0.354
	32000	6000	$+0.262$	0.907

Appendix

Table 10: Readout against step budget. The first two cells vary the budget at fixed seed; the last three hold the escape step fixed at 1000 under both budgets, so any difference between their rows is in the schedule rather than in when the circuit formed. Medians are gated at mass

\geq 0.5

. The 12000-step row is from the earlier round of this appendix and is resolved on a finer evaluation grid, hence its non-multiple-of-1000 escape step.

cell	slots	$q_{\mathrm{kept}}$	tokens	median	frac $+$	$p$	medHi	medLo
$R_{\mathrm{old}}=8$ main, s0	14.59	5.75	204	$+0.037$	0.66	9e-11	$+0.086$	$+0.003$
$R_{\mathrm{old}}=8$ main, s1	14.59	5.75	204	$-0.005$	0.45	0.06	—	—
$R_{\mathrm{old}}=8$ shortened, s0	8.95	4.02	116	$+0.337$	0.95	2e-81	$+0.413$	$+0.172$
$R_{\mathrm{old}}=8$ shortened, s1	8.95	4.02	116	$+0.821$	0.99	4e-104	$+1.070$	$+0.432$
$R_{\mathrm{old}}=8$ shortened, s2	8.95	4.02	116	$+1.351$	0.98	3e-96	$+1.683$	$+1.109$
$R_{\mathrm{old}}=16$ main, s0	8.73	8.94	204	$+1.811$	1.00	3e-118	$+2.274$	$+1.351$

Appendix

Table 11: Slot-matched arm. The shortened cell has the slot count of

R_{\mathrm{old}}=16

and most of its copy dispersion (Table 12 ) while retaining

R_{\mathrm{old}}=8

. The median column is gated at contrast-pair mass

\geq 0.5

; medHi and medLo split all documents with edit domain at each cell’s own

q_{\mathrm{kept}}

median and are ungated, so they are not directly comparable to the median column where mass is low. The two reference rows carry no split here (—); their within-cell splits are in Table 14 , where the seed-1 row at

R_{\mathrm{old}}=8

is one of the six cells whose pooled median falls below the 0.15-nat floor of § 5.4 . The seed-1 reference row is itself one of the 13 non-replicating cells (

+0.113

at 0.74, against

+1.811

at 1.00 in seed 0), so percentages in the text are computed against the seed-0 reference and baseline and the seed-1 and seed-2 rows are read as directional replication.

cell	$p_{\mathrm{update}}$	slots	q_gap	q_kept	tokens	tail0	$\rho$
$R_{\mathrm{old}}=8$	0.28	18.87	7.60	5.69	205.3	0.840	0.086
main	0.50	14.60	7.37	5.84	204.4	0.822	0.100
arm	0.90	11.51	7.48	5.94	205.2	0.791	0.110
shortened	0.50	8.94	5.82	3.98	117.3	0.821	0.100
$R_{\mathrm{old}}=16$ arm	0.28	11.55	5.51	8.90	205.2	0.900	0.051
main	0.50	8.69	5.23	9.04	205.0	0.898	0.056

Appendix

Table 12: Calibration of the

p_{\mathrm{update}}

arm, 2000 documents per configuration. q_gap is the generator’s per-document antecedent distance for the queried slot, the same quantity as ant q in Table 1 ; the two measurement paths read 0.4–0.5 apart on the cells they share, so dispersion is compared within a path throughout. q_kept is realized redundancy, tail0 the fraction of documents with no update after the queried rebinding, and

\rho=p_{\mathrm{update}}/(1+p_{\mathrm{update}}R_{\mathrm{old}})

the filler-side update density. The shortened row is the arm of App. G.1 , remeasured here so that all four dispersion figures come from one path; it reproduces that table’s slot count, realized redundancy and token count to within 1.1% and its dispersion figure does not.

cell	$p_{\mathrm{update}}$	seed	median	frac $+$	mass	medHi	medLo
$R_{\mathrm{old}}=8$ main	0.50	0	—	0.662	—	—	—
	0.50	1	—	0.450	—	—	—
	0.50	2	—	0.763	—	—	—
$R_{\mathrm{old}}=8$ arm	0.90	0	$-0.055$	[0.154]	1.00	$-0.063$	$-0.047$
	0.90	1	$+2.852$	1.000	0.66	$+4.371$	$+3.540$
	0.90	2	$+1.159$	0.997	0.97	$+1.288$	$+0.914$

Appendix

Table 13: Readouts in the

p_{\mathrm{update}}

arm. The median column is gated at contrast-pair mass

\geq 0.5

; medHi and medLo split all documents with edit domain at each cell’s own realized-redundancy median and are ungated, so they are not comparable to the median column where mass is low. Reference rows give the main grid’s sign fraction only (Table 16 ); their medians and controls are in Table 15 and Table 14 . The seed-0 row of the

R_{\mathrm{old}}=8

leg carries a control of

-0.020

against its median of

-0.055

and is not interpretable; its sign fraction is bracketed.

		seed 0		seed 1		seed 2
$R_{\text{old}}$	$\Delta D$	medHi	medLo	medHi	medLo	medHi	medLo
3	2	$-0.016$	$-0.005$	$-0.051$	$-0.041$	$+0.029$	$+0.002$
3	3	$-0.046$ a	$-0.047$	$+2.879$	$+2.592$	$+1.472$	$+0.467$
3	5	$-0.120$	$-0.069$	$+7.458$	$+0.430$	$-0.017$	$-0.011$
3	8	$-0.041$	$-0.015$	$-0.001$ a	$-0.001$	$+1.486$	$+0.870$
3	16	$-0.022$ a	$-0.024$	$-0.032$	$-0.017$	$-0.032$	$-0.007$

Appendix

Table 14: Within-cell dose-response, three seeds. Each cell is split at its own median realized redundancy; medHi and medLo are the medians of the two halves, over all documents with edit domain and ungated, so they are not comparable to the gated medians of Table 15 where mass is low. The gradient runs in the direction of the cell’s own effect in 65 of the 75 runs. a marks the six exceptions whose pooled median is below 0.15 nats in absolute value, where there is no effect to stratify. b marks the four that invert a gradient while carrying an effect: the inversion is 8% and 3% of the pooled median in seeds 0 and 1, and 17% and 10% in the two contributed by seed 2. The two-seed version of this table carried no inversion above 8%, so the claim the third seed removes is that inversions are confined to gradients negligible relative to their cell; what survives is that the split tracks the size of the effect, at Spearman

\rho=0.865

over the 50 runs of seeds 0 and 1.

Figure 3: Per-document readout in three cells along the

\Delta D=5

column. Red line at zero, blue dashed at the median. Note the asymmetry in shape: the negative-side distribution is concentrated near zero with a thin tail, while the positive-side distributions are broad. Axes are not shared.

$R$	$\Delta D$	$n$	yield	median	IQR low	IQR high	frac $+$	$p$	mean	trim	ctrl	mass	mOK
3	2	364	0.91	$-0.011$	$-0.043$	$+0.007$	0.35	3e-08	$-0.109$	$-0.034$	$+0.000$	1.00	1.00
3	3	362	0.91	$-0.046$	$-0.213$	$+0.055$	0.36	3e-07	$-0.170$	$-0.115$	$+0.001$	1.00	1.00
3	5	364	0.91	$-0.087$	$-0.219$	$-0.030$	0.13	4e-51	$-0.218$	$-0.150$	$+0.004$	1.00	1.00
3	8	328	0.82	$-0.024$	$-0.075$	$-0.009$	0.10	1e-54	$-0.073$	$-0.055$	$+0.000$	1.00	1.00
3	16	274	0.69	$-0.023$	$-0.104$	$+0.038$	0.41	2e-03	$-0.094$	$-0.058$	$+0.001$	1.00	1.00
5	2	397	0.99	$+0.066$	$+0.008$	$+0.137$	0.78	2e-30	$+0.087$	$+0.076$	$-0.006$	1.00	1.00

Appendix

Table 15: Per-cell readouts, main grid at 16000 steps, seed 0, 400 held-out documents per cell.

n

is the number with edit domain.

$R_{\mathrm{old}}$	$\Delta D$	s0	s1	s2	range	$R_{\mathrm{old}}$	$\Delta D$	s0	s1	s2	range
3	8	0.098	0.477	0.977	0.879	16	2	0.998	0.736	0.972	0.261
3	5	0.126	0.972	0.270	0.845	3	16	0.405	0.167	0.349	0.238
5	2	0.781	0.175	0.967	0.792	5	8	0.800	0.798	0.931	0.133
3	3	0.365	0.934	0.930	0.569	8	5	0.932	1.000	0.947	0.068
16	16	0.977	0.469	0.969	0.508	8	8	0.970	0.997	0.934	0.063
3	2	0.354	0.138	0.625	0.487	5	16	0.796	0.844	0.784	0.060

Appendix

Table 16: Sign fraction per cell under three seeds at 16000 steps, 400 held-out documents each, ordered by the range across seeds. Thirteen of 25 cells span more than 0.3; the binomial standard error at

n=400

is at most 0.025, so the largest range is 35 standard errors. All 75 runs reach in-distribution accuracy

\geq 0.999

and terminate in the retrieval state, so no held-in evaluation separates any pair of entries. § 5.2 treats a cell-to-cell difference as meaningful only when it exceeds the range in this table. Adding the third seed widened the maximum from 0.845 to 0.879 and moved the cell carrying it from

(3,5)

(3,8)

$R_{\mathrm{old}}$	step	half	mass	alias	t-diff	margin	t-diff	margin
				$\lvert\cos(g_{L},g_{\Delta})\rvert$			ratio to alias
3	5000	0.599	1.00	0.00877	0.03965	0.03347	$4.5\times$	$3.8\times$
3	6000	0.535	1.00	0.00317	0.02532	0.02199	$8.0\times$	$6.9\times$
3	8000	0.350	1.00	0.01437	0.00317	0.00576	$0.2\times$	$0.4\times$
3	10000	0.405	1.00	0.00060	0.00138	0.00058	$2.3\times$	$1.0\times$
16	1000	0.514	0.99	0.01838	0.04785	0.04113	$2.6\times$	$2.2\times$

Appendix

Table 17: Eight readouts at the ten checkpoints where every readout with a contrast pair is valid (mass

\geq 0.5

) and

g_{L}

is estimable (

\mathrm{half}=\cos(g_{L}^{A},g_{L}^{B})\geq 0.3

). Columns alias , t-diff and margin are

\lvert\cos(g_{L},g_{\Delta})\rvert

for the multiplicity inversion, for the paired difference whose two ground truths differ, and for the single-document margin; chance is

2.0\times 10^{-4}

throughout and mass is the aliased readout’s. The two ratio columns are t-diff and margin over alias. The five remaining edits are omitted here and summarized in the text: their

\lvert\cos\rvert

is not ordered by collision rate at any checkpoint.

Shortcut Learning

Shortcut learning occurs when models exploit easy but unreliable cues instead of the task-relevant evidence needed to generalize.

See all 159 papers

Aug 11, 2026

Reasoning Shortcuts and Value Symmetries: What Symmetry Permits, Architecture Realizes, and Optimization Selects

Reasoning shortcuts are rule solutions that reach correct predictions through unintended concepts. A recent framework of Takemura, Inoue, and Nishino analyzes them through an automorphism group of value relabelings, asking when rules pin concepts down. Its key definition, one value permutation shared across all positions, does not apply as stated to any of its four heterogeneous benchmarks, and the most direct embedding, padding, produces confident false pathology: 90.91% of solution pairs unexplained on CLE4EVR, versus 0% under every well-defined rung of the componentwise hierarchy we introduce; the padded verdict rotates under configuration-file ordering. Across eleven rule families under fifteen pre-specified predictions (thirteen confirmed), unexplained-pair rates span 0% to 99.9999% and track provable structure: six theorems give sufficient conditions for transitivity and its failure. For circuit-given rules, symmetry-inertness of a coordinate is coNP-complete; automorphism existence is coNP-hard under randomized reductions, lies in

Σ_2^p

, is not

Σ_2^p

-complete in the Boolean case unless PH collapses, and is coNP-complete on monotone circuits. Boolean transitivity is classified exactly: automorphisms explain everything iff the solution set is an affine coset. Weakly supervised models place all 94 observed shortcuts at the one level the theory flags, none at the 48 it certifies transitive, and none at twelve typed-ambiguous levels. Relocating the absorbing element moves every shortcut with it; a confusion null attributes the location to geometry while the observed rate exceeds it by half again. Trained end to end on CLE4EVR's rule and heterogeneous domains through a synthetic prototype front end, models produce 20,223 label-preserving errors with zero different-orbit exceptions, as transitivity predicts, where the padded instrument would misreport 78-88% of them.

Xin Xu

Carnegie Mellon University · University of Pennsylvania

Symbolic Reasoning Neuro-Symbolic Reasoning

Ask PDF

Aug 27, 2026

Circuit Condensation: Post-Training that Concentrates a Behavior's Causal Circuit

One approach in mechanistic interpretability explains model behavior through circuits: the components and connections that carry it. Frozen discovery often returns hundreds of edges, making them hard to inspect, compare, or verify exhaustively. We introduce Circuit Condensation, which post-trains models to concentrate behaviors into smaller causal graphs. Each round prunes low-attribution edges and trains a low-rank adapter to match the original through what remains. It keeps the cut only if task accuracy and held-out perplexity remain within tolerance. Across four behaviors and eight models, condensed circuits are smaller than the strongest published frozen-weight baseline in 30 of 32 settings, by 8.1x on average and up to 316x. Repeating the search without weight updates produces larger circuits in 29 of 32 settings, showing that weight updates drive the reduction rather than search alone. For 19 small circuits, exhaustive subset tests show that 11 are exactly minimal and identify removable edges in the rest. Pair ablations reveal dependencies that single-edge tests miss. On indirect object identification, condensation isolates 24 heads, including 17 with documented roles, whereas a matched frozen circuit needs 61 heads, 36 without documented roles. The result is a sufficient subcircuit of the published mechanism rather than a reconstruction of it. The resulting circuit tracks the original model's next-token distribution and, at matched edge count, predicts its errors better than a frozen circuit in 10 of 11 evaluable settings.

Sai Adith Senthil Kumar

George Mason University

Mechanistic Interpretability Circuits in Language Models Causal Interpretability Neural Network Pruning

Ask PDF

Oct 19, 2026

DeltaTrace: Efficient Signed Attribution for Reasoning Language Models

We study how words in the context support or oppose a language model's response and present DeltaTrace, a novel attribution framework that explains the effect of input tokens on a fixed model response. DeltaTrace measures how the response's log-probability changes when the context tokens being explained are masked. To distribute this difference among source tokens, it constructs operator-specific local rules for finite changes and composes them backward through the computational graph using effective secant slopes. Each token is assigned a signed attribution as the inner product of its embedding displacement and the corresponding backward-propagated slope. Local conservation makes these contributions sum to the measured score difference and preserves their meaning when aggregated over sentences or passages. DeltaTrace supports both attention and linear-attention models, with auxiliary memory growing linearly with sequence length at fixed model dimensions and chunk size. On the evaluated attention model, DeltaTrace achieves the lowest deletion-faithfulness scores (RISE and MAS) on most retrieval and reasoning tasks and the highest needle-in-a-haystack Recall, exceeding FlashTrace by 5.67-24.33 percentage points. On the evaluated hybrid attention and linear-attention model, DeltaTrace lowers RISE on 9 tasks and MAS on 12 relative to FlashTrace.

Chen Liang, Daoping Wu, Donghua Yang +4

Harbin Institute of Technology · Zhejiang Lab

Feature Attribution LLM Interpretability Faithfulness of Language Model Explanations

Ask PDF

Abstract

Shortcut Learning

Explore similar work

Reasoning Shortcuts and Value Symmetries: What Symmetry Permits, Architecture Realizes, and Optimization Selects

Circuit Condensation: Post-Training that Concentrates a Behavior's Causal Circuit

DeltaTrace: Efficient Signed Attribution for Reasoning Language Models