SCIT: Testing Causal Cache Carriers in Latent Chain-of-Thought Models

Yi Ding; Lijun Huang; Menglin Yang

cs.CLAug 27, 2026

SCIT: Testing Causal Cache Carriers in Latent Chain-of-Thought Models

Authors: Yi Ding, Lijun Huang, Menglin Yang

Organizations: The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China

Abstract

Latent chain-of-thought models move intermediate reasoning from emitted text into continuous states, improving compactness but hiding the causal object. We introduce SCIT, the Suffix Cache Interchange Test, a causal protocol that constructs exact source-recipient counterfactuals, patches declared cache segments, and identifies which transformer object carries the counterfactual computation. SCIT combines sufficiency tests with K/V component splits, hidden-state controls, semantic source controls, decoded validation, and matched corruption. On CODI-GPT2 and a Sim-CoT-style GPT-2 reproduction, counterfactual arithmetic transfers primarily through value-cache suffix trajectories rather than hidden states, keys, reusable answer slots, or single-token triggers. Complete sufficiency-and-necessity evidence for the late-value-suffix mechanism holds for the main CODI-GPT2 checkpoint; the Sim-CoT-style checkpoint shows the same sufficiency and decoded-control pattern but insufficient matched-corruption evidence for a necessity call. Beyond these local arithmetic cells, SCIT reveals carrier-regime shifts: arithmetic-like GPT-2/1B cells preserve latent-tail value/KV transfer, whereas competent 8B and repaired non-arithmetic cells route through prompt-prefix or full-cache K/V; boundary cells receive no mechanism call. SCIT therefore contributes a cache-level diagnostic, a checkpoint-specific GPT-2 arithmetic mechanism, and a competence-gated carrier map rather than a universal latent-tail claim.

Figures & tables

Figure 1: SCIT carrier-test protocol. A source cache slice and recipient-side fields are combined into a patched recipient rollout, then evaluated under a fixed intervention grid. The patched rollout is validated with target-win, margin, free-generation, and matched-corruption readouts before assigning an active-carrier, carrier-shift, or no-call status. Segment calls use the smallest gated disjoint segment; whole-cache replacement is an upper-bound control. TW: target win; dLP: log-probability drop.

Claim	Verdict	Key evidence and risk closed	Where
Local arithmetic carrier	LATENT-TAIL	GPT-2 arithmetic passes competence; value suffixes transfer (all/late V $0.996$ – $1.000$ , layer 8–9 V $0.875$ – $0.992$ ), while key/ $h$ and semantic controls are weak. CODI corruption closes necessity (TW $0.211$ ); Sim-CoT is sufficiency-only/directional.	Tables 19 , 20 , 21 , 54
Arithmetic scale check	LATENT-TAIL	1B arithmetic-like cells preserve the active latent carrier (active $1.00$ vs inactive $0.00$ ; corruption active TW $0.00$ vs inactive $1.00$ ). The same arithmetic-style diagnostic shifts at 8B.	Fig. 3 ; App. Tables 14 , 44
8B field-routed cells	PREFIX SHIFT	Competent 8B arithmetic/entity/relation/expression cells route through prompt-prefix/full K/V: direct-pre sufficiency is already $.969$ – $1.000$ , latent $0.00$ , active-corruption dLP $13.64$ – $21.07$ . In arithmetic/expression/list/relation follow-ups, post-prefix zero and field ablations collapse readout from visible fields while latent-suffix ablation does not.	Fig. 3 ; Table 2 ; App. Tables 9 , 10 , 11 , 15 , 27
Boundary discipline	NO CALL	Qwen3-4B stays near chance ( $0.25$ – $0.33$ ); graph/finite-state/OOD/support-limited cells fail competence, support, or decoded-readout gates. No carrier is interpreted below gate.	Tables 14 , 27 , 31
Training-path sensitivity	PATH-SENSITIVE	T16 and new CODI seed audits pass clean gates but enter different basins; new CODI seeds show only $0.049$ – $0.138$ late-8–9 value sufficiency and $0.026$ – $0.089$ corruption drops.	Tables 42 , 43
Metric robustness	AUDITED	Competence sweep $.70/.80/.90$ is stable; 16 carrier-threshold settings and 12/12 large- $N$ calls are unchanged; exact/near ties are rare ( $0.063\%/0.072\%$ ).	Table 16

Table 1: Compressed claim dashboard. Each row gives the mechanism verdict, the shortest evidence bundle needed to interpret it, and the numerical source; Appendix Table 14 gives the full gate/sufficiency/corruption/decoded ledger.

Figure 2: Core arithmetic mechanism evidence. The panels read as a four-step evidence chain for the GPT-2 arithmetic row in Table 1 : the sufficient restricted object is the middle-to-late value block, semantic source controls fall below exact counterfactual sources, suffix-window curves show that the effect is not merely the final cache token, and matched corruption closes necessity only for CODI-GPT2. Unless otherwise marked, plotted values are final-step target-win rates. Numeric provenance is direct: layer values come from Table 19 ; component controls from Tables 51 , 49 , and 50 ; semantic controls from Table 20 ; suffix-window curves from Table 33 ; and corruption from Tables 21 and 58 .

Figure 3: Competence-gated carrier-regime map. The segment heatmap shows final-step target-win rates for latent-tail, prompt-prefix, and full-cache K/V; full-cache replacement is an upper-bound control, while the verdict follows the smallest gated disjoint segment. The 8B time scan shows prompt-prefix sufficiency from the first scanned latent step, the boundary panel lists no-call cells blocked by competence or support gates, and the field-routing inset summarizes the visible prompt spans behind the 8B prefix-shift rows.

8B cell	Lat.	Pref.	Best visible field(s)
2-hop arithmetic	.00	1.00	facts / initial clause: 1.00
Entity lookup	.00	1.00	lookup entry / facts: 1.00
Relation chain	.00	1.00	query: 1.00; tables: .00
Expression tree	.00	.99	facts: .98; bindings: .63
List filter/sum	.00	1.00	rules/facts: 1.00; threshold: .44

Table 2: Main-text slice of 8B field routing. Lat. and Pref. are oracle target-win rates for latent-tail and prompt-prefix K/V replacement. Visible fields are the strongest prompt-span patches; source-control rows, OOD variants, and post-prefix readout ablations are in Appendix Tables 15 and 11 .

Appendix figures & tables58 assets

Supplementary material from the paper’s appendix.

Appendix

Appendix block	Tables / figures	Purpose
Method/setup	Main-text Figure 1 ; Appendix Figure 4 ; Tables 4 – 7	Defines the intervention grid, prompt templates, scoring/battery, and sample sizes moved out of the main text.
Carrier-regime audit	Tables 8 – 18	Gives the carrier-status index, 8B time scan, no-latent truncation and post-prefix readout probes, no-call rows, concern-to-evidence map, ledger, field routing, threshold/tie audit, and method-positioning map.
Core arithmetic	Tables 19 – 21 ; Figure 5	Supplies the GPT-2 localization, semantic-source controls, matched corruption, and candidate-set stress check behind the local mechanism claim.
Scope/boundaries	Tables 22 – 31	Separates task-breadth calibration, repaired cells, support boundaries, and base-model sanity checks.
Trajectory/auxiliary checks	Tables 32 – 43	Collects trajectory-length, head, geometry, noise, gradient, ablation, horizon, and seed/path diagnostics.
Carrier-scale/component details	Tables 44 – 58	Expands segment splits, GPT-2 block scans, OOD/1B checks, vector/cache controls, cross-template controls, decoded validation, and Sim-CoT corruption.

Appendix

Table 3: Appendix table map. Each row names the appendix block and the tables or figures that support it.

Name	Cache segment	Component operation	Current state	Interpretation
Hidden-only	Recipient cache unchanged	No K/V replacement	Replace $h_{t}(r)$ with $h_{t}(s)$	Tests whether the current hidden vector alone carries the counterfactual.
Cache-only	Selected segment from source	Replace K and V jointly unless restricted below	Keep $h_{t}(r)$	Tests whether the cache trajectory is sufficient without copying the source hidden state.
Cache+h	Selected segment from source	Replace K and V jointly unless restricted below	Replace $h_{t}(r)$ with $h_{t}(s)$	Strong closed-loop interchange; interpreted only with hidden-only and cache-only controls.
Key-only	Selected segment from source	Replace K; keep recipient V	As specified by cache-only/cache+h	Surgical routing test; an artificial state used to test key-only sufficiency.
Value-only	Selected segment from source	Replace V; keep recipient K	As specified by cache-only/cache+h	Surgical content test; success means source value content is usable under recipient keys.
Full-cache	Prompt prefix plus latent tail	Replace all available K/V cache entries	As specified by cache-only/cache+h	Upper-bound interchange check and comparison point for segment-local claims.

Appendix

Table 4: Intervention definitions. The rows are not all mutually exclusive experimental families: key-only, value-only, and K/V are component choices that can be crossed with cache-only or cache+h state handling. Segment choices such as latent-tail, prompt-prefix, and whole-cache determine where the component operation is applied. Cache-only rows copy K/V entries at the patch point but keep the recipient’s current hidden state

h_{t}

; hidden-only rows copy

h_{t}

but leave all K/V entries unchanged.

Family	Prompt template
balls	Question: Alice has {A} red balls and {B} blue balls. She gives away {C} red balls. How many balls does she have now? Answer:
books	Question: Alice has {A} science books and {B} history books. She lends out {C} science books. How many books does she have now? Answer:
coins	Question: Alice has {A} silver coins and {B} gold coins. She gives away {C} silver coins. How many coins does she have now? Answer:

Appendix

Table 5: Arithmetic prompt templates. The implementation places Answer: on a new line; the table uses one line only for compactness. If

P

is the post-transfer count, the variables satisfy

A=P+C

and the answer is

P+B

. Books and coins preserve the same computation while changing the surface noun family and transfer verb.

Family	Cells	Intervention/readout	Evidence role
Core localization	CODI-GPT2 and Sim-CoT-GPT2 on balls/books/coins	Latent-tail cache+h over layer groups; cache-only, hidden-only, key-only, value-only splits	Identifies the arithmetic carrier as a value-cache trajectory rather than the current hidden vector or key routing.
Semantic controls	Exact, same-answer, partial-variable, cross-template, random, greedy, sampled, enlarged-candidate checks	Same patched state scored by target win, margin, and parsed free generation	Tests whether transfer is full-context counterfactual behavior rather than answer-slot reuse or a closed-set artifact.
Necessity	Matched random corruption plus zero/mean ablations	Recipient-answer preservation and log-probability drop after damaging the same region	Separates sufficiency from requirement and checks whether the localized region is globally fragile.
Carrier-regime calibration	1B/8B arithmetic, three-hop, entity, relation, expression/list, state, graph, rule cells	Latent tail, prompt prefix, whole cache, component, decoding, support, and OOD gates	Maps whether the carrier persists, shifts, or becomes uninterpretable; these cells are not treated as broad benchmark wins.
8B hardening	Field-level 8B arithmetic, entity, relation, expression/list probes with source controls	Field K/V patches over lookup, query, fact, rule, operand, threshold spans	Decomposes prefix or full-cache positives into visible-field carrier evidence rather than leaving 8B as an opaque stress test.
Reporting audits	Threshold sweeps, tie audit, head controls, seed/path probes	Fixed decision grid, tie counts, ranked versus random head patches, checkpoint scans	Checks that carrier calls do not depend on arbitrary thresholds, tie order, or a single training path.

Appendix

Table 6: Experimental battery by evidential role. The main mechanism claim uses the core localization, semantic controls, decoded validation, and matched corruption rows. Broader 1B/8B and task-breadth rows are competence-gated calibration.

Diagnostic	Main cells	$N$ / seeds	Randomization and readout
GPT-2 localization	CODI-GPT2 and Sim-CoT-GPT2, balls/books/coins	CODI balls: 240 examples per cell; other rows: two 60-example seeds	Held-out source–recipient pairs; target win over recipient, donor, oracle, and random labels; steps 5/6 reported.
Semantic controls	Same models/templates	Same as localization	Same-answer, same-intermediate, partial-variable, cross-template, and random sources; fifth label added when needed.
Matched corruption	CODI-GPT2 main; Sim-CoT-GPT2 replication	360 examples aggregated over balls/books/coins per table	Same-template and same-range random cache sources; recipient-answer preservation and log-probability drop.
Decoded validation	Greedy and sampled free generation	Greedy: 60 examples per template; sampling: five samples per example	Parsed free-generation answer compared with counterfactual, recipient, and out-of-set labels.
Carrier-regime calibration	1B/8B arithmetic, entity, relation, expression/list, state cells	Usually $N=128$ held-out pairs per carrier-map task; field rows caption exact $N$	Competence-gated latent-tail, prompt-prefix, and whole-cache segment splits with source controls where available.

Appendix

Table 7: Experimental design details. Appendix captions give row-specific SEMs and exceptions; this table records the core

N

, seed, and randomization choices.

Call type	Cells	Interpretation
Latent-tail positive	GPT-2 arithmetic; clean three-hop; 1B arithmetic-like	Arithmetic carrier persists, but layer locus need not be scale-invariant.
Carrier shift	8B arithmetic, entity, relation; repaired expression/list; repaired relation-chain; state controls	Competent cells route through prompt-prefix or full-cache K/V; no-latent and post-prefix ablations show the prompt-prefix state is sufficient before rollout and visible fields remain readout-active afterward.
No-call boundary	Qwen3-4B, graph path, finite-state, ultra-OOD expression, list controls, fixed-rule holdouts	Competence, support, or source-control gates block mechanism interpretation.
Reporting audit	Threshold sweep, tie audit, head Pareto, seed/path probes	Checks that calls are not artifacts of cutoffs, tie order, or one training path.

Appendix

Table 8: Calibration-suite status. These cells calibrate carrier regimes; they are not treated as broad benchmark-generalization wins.

8B follow-up cell	$N$	Latent-tail TW	Prompt-prefix TW	Prompt $\Delta_{\mathrm{cf-rec}}$	Top-8 attn. Lat./Pref.	Interpretation
Two-hop arithmetic	48	0.000–0.000	1.000–1.000	26.2–26.2	.013 / .960	Prefix from step 1.
Entity lookup	48	0.000–0.000	1.000–1.000	31.8–31.8	.001 / .876	Prefix lookup binding.
Expression tree	48	0.000–0.000	1.000–1.000	35.6–35.6	.004 / .987	Prefix operand/fact binding.
Relation chain	48	0.000–0.000	1.000–1.000	41.9–41.9	.031 / .962	Prefix query binding.

Appendix

Table 9: Time-resolved 8B prompt-prefix follow-up. Each row scans latent steps 1–6 with oracle cache-only K/V patches over the latent suffix and prompt prefix. Target-win ranges are the minimum and maximum over the six scanned steps: prompt-prefix K/V is sufficient from the first scanned latent step, while latent-suffix K/V is inactive throughout. The attention column reports final-step answer-readout mass for top-8 heads selected by an independent prompt-prefix K/V head scan, averaged across recipient and oracle-counterfactual states.

8B truncation cell	$N$	Direct pre-recipient TW	Direct pre-oracle TW	Full-cache TW	Interpretation
Two-hop arithmetic	32	1.000	1.000	1.000	Prefix state is sufficient before rollout.
Expression tree	32	1.000	0.969	0.969	One oracle miss; same qualitative prefix regime.
List filter/sum	32	1.000	1.000	1.000	Prefix state dominates latent tail.
Relation chain	32	1.000	1.000	1.000	Query/source binding is pre-rollout sufficient.

Appendix

Table 10: 8B no-latent truncation probe. Direct-pre rows score candidate answers from the state returned by forward_until_step before completing the latent rollout; latent-tail and full-cache columns are step-1 oracle K/V intervention controls from the same held-out run. The prompt-prefix state already selects the recipient answer under the recipient prompt and the counterfactual answer under the oracle prompt, while latent-tail patches remain inactive. This rules out a necessary latent-tail accumulation account; by itself, the truncation probe does not separate early completion from later visible-field readout.

8B readout cell	$N$	Latent-suffix zero TW	Prompt-prefix zero TW	Best field zero TW	Unaffected field	Interpretation
Two-hop arithmetic	24/12	1.000	.417 / .417	initial/facts: .417–.500	query: 1.000	Numeric facts remain required at readout.
Expression tree	24/12	1.000	.458 / .458	rules: .250; inputs: .417	query: 1.000	Rules and bindings, not latent suffix, support answer readout.
List filter/sum	24/12	1.000	.125 / .125	values: .583; facts: .750	query/rules: .917–1.000	Value bindings dominate the visible-field dependency.
Relation chain	24/12	1.000	.250 / .250	room/query: .167–.250	person table: 1.000	Readout depends on the queried relation fields.

Appendix

Table 11: 8B post-prefix readout ablations. The first two target-win columns use source-free zero ablation over the recipient latent suffix or prompt prefix at latent step 6, shown as no-continue / continue values for K/V ablation. The field column uses a smaller localization probe (

N=12

) that zeroes visible prompt spans at direct-pre, no-continue, and post-latent-final stages; the displayed ranges summarize the damaging K/V field spans across those stages. These are harsher ablations than matched corruption, so we use them as readout-sensitivity evidence rather than counterfactual-transfer evidence. They show that latent suffix entries are not required, while prompt-prefix and task-relevant visible fields remain required after latent encoding.

Boundary cell	Gate failure	Interpretation
Qwen3-4B probes	Closed-set competence near chance	No interpolation claim between 1B and 8B.
Graph path	Weak decoded competence	Path-following training support is insufficient.
Broader finite-state	Short/long sequence gates fail	State tracking does not generalize from narrow supports.
Expression ultra-OOD	Free-generation and source-control gates fail	Repaired carrier is support-limited.
List hard cells	Source controls remain high	Prompt-prefix copying cannot be separated cleanly.
Fixed-rule holdouts	Leave-one/class holdouts fail	Matched-support fitting, not robust rule execution.

Appendix

Table 12: No-call boundary cells. Each row fails a prerequisite gate, so SCIT withholds a latent-tail mechanism call.

Weakness	Where to look	What the evidence supports
The main task is too narrow	Tables 22 , 23 , 25 , 27	The paper is a causal diagnostic, not a broad reasoning benchmark; added cells test depth, branching, list aggregation, relation binding, and state tracking under competence gates.
Scale generalization is unresolved	Tables 47 , 44 , 15 , 9 , 10 , 11 , 27	1B arithmetic-like cells preserve value/KV transfer, but 8B and repaired task cells shift to prompt-prefix/full-cache carriers that appear already before latent rollout and remain sensitive to visible fields at readout.
Carrier calls depend on competence gates, thresholds, or tie order	Table 16	The large- $N$ 1B/8B carrier split is unchanged under $.70/.80/.90$ competence gates and stricter sufficiency/competitor thresholds; exact target-score ties are rare in the cache-interchange state-score rows.
The late-layer split could be arbitrary	Tables 45 , 25	GPT-2 has a stable middle-to-late restricted block, but architecture-relative localization shifts at 1B.
Closed candidate sets inflate random baselines	Figure 5 ; Tables 47 , 56	Enlarged candidate sets and sampled decoding preserve the main exact-context separation; wide-OOD random wins are treated as high-baseline cases and interpreted through continuous margins.
The effect may be only hidden-state replacement	Tables 48 , 49 , 50 , 51	Hidden-only and key-only patches are weak; value/cache replacement carries the arithmetic transfer, with checkpoint-dependent readout support.

Appendix

Table 13: Concern-to-evidence map for the appendix. Each row points to compact diagnostic evidence for a likely weakness.

Cell	Breadth	Gate	Active carrier	Oracle suff. TW	Matched corruption	Decoded CF	Call
1B arithmetic-like	two-hop, three-hop, conditional	FG 1.00 ( $N=192$ )	latent-tail KV	1.00 vs 0.00 ( $N=192$ )	corrupt 0.00 vs 1.00; dLP 12.79 ( $N=384$ )	1.00 vs 0.00 ( $N=192$ )	latent-tail repl.
1B entity lookup	single visible binding	FG pass 1.00 ( $N=64$ )	prompt-prefix KV	1.00 vs 0.00 ( $N=64$ )	corrupt 0.00 vs 1.00; dLP 21.87 ( $N=128$ )	0.88 vs 0.00 ( $N=64$ )	carrier shift
1B relation chain	12-room relation repair	FG 1.00 ( $N=48$ )	prompt-prefix KV	1.00 vs 0.00 ( $N=48$ )	corrupt 0.00 vs 1.00; dLP 20.69 ( $N=128$ )	1.00 vs 0.00 ( $N=48$ )	carrier shift
8B arithmetic-like	two-hop, three-hop, conditional	FG 0.99 ( $N=120$ )	prompt-prefix KV	1.00 vs 0.00 ( $N=120$ )	corrupt 0.00 vs 0.99; dLP 13.64 ( $N=192$ )	1.00 vs 0.00 ( $N=120$ )	scale shift
8B entity lookup	single visible binding	FG 1.00 ( $N=40$ )	prompt-prefix KV	1.00 vs 0.00 ( $N=40$ )	corrupt 0.00 vs 1.00; dLP 16.25 ( $N=64$ )	1.00 vs 0.00 ( $N=40$ )	carrier shift
8B repair/binding	expression, list, relation	FG 1.00 ( $N=120$ )	prompt-prefix KV	1.00 vs 0.00 ( $N=120$ )	corrupt 0.00 vs 1.00; dLP 21.07 ( $N=192$ )	1.00 vs 0.00 ( $N=120$ )	shift

Appendix

Table 14: Comprehensive carrier-evidence ledger generated from the retained run196–run198 HPC3 summaries. Gate reports unpatched recipient free-generation accuracy where decoded validation is available. Oracle sufficiency and decoded counterfactual (CF) columns report active versus inactive segment rates. Matched corruption reports recipient target-win after corrupting the active versus inactive segment and the active-segment mean log-probability drop (dLP). The final two rows make failed-gate and reporting-rule audits visible in the same table rather than relegating them to prose.

8B cell	Split	$N$	Pref.	Lat.	Src. ctrl	Best visible field TW	Call
2-hop arithmetic	in-domain	40	1.000	0.000	1.000	initial_clause:1.00; facts_only:1.00; initial_red:1.00	visible source-binding
3-hop arithmetic	in-domain	40	1.000	0.000	1.000	initial_clause:1.00; facts_only:1.00; initial_red:1.00	visible source-binding
entity lookup	in-domain	40	1.000	0.000	1.000	lookup_table:1.00; matching_lookup_entry:1.00; facts_only:1.00	visible source-binding
relation chain	in-domain	152	1.000	0.000	1.000	query:1.00; person_table:0.00; room_table:0.00	visible source-binding
expression tree	far-OOD	81	0.988	0.000	0.988	facts_only:0.98; input_bindings:0.63; left_inputs:0.59	visible source-binding
expression tree	ultra-OOD	81	0.185	0.136	0.200	facts_only:0.22; input_bindings:0.20; left_inputs:0.20	boundary/no call

Appendix

Table 15: 8B field-routing hardening for prompt-prefix carrier shifts. Prefix (Pref.) and latent (Lat.) columns report oracle cache-only K/V target-win rates; Src. ctrl reports the larger source-label win rate among random and non-oracle prompt-prefix patches. The field column reports the strongest visible prompt spans from the same held-out run. Successful 8B cells are not opaque full-cache wins: lookup entries, facts, rules, or query fields recover transfer while latent-suffix K/V remains inactive. High Src. ctrl is interpreted as visible source-field copying rather than oracle-specific latent reasoning; weak-prefix rows remain no-call boundary cells. Table 11 gives the inverse readout-sensitivity check on recipient states.

Audit	Rows / sweep	Outcome
Carrier threshold grid	Sufficiency floor $\{.75,.80,.85,.90\}$ crossed with competitor ceiling $\{.10,.20,.25,.30\}$ on $12$ large- $N$ 1B/8B carrier cells	Every grid point gives the same calls: $5$ latent-tail cells and $7$ prompt-prefix cells; no row becomes mixed or inconclusive.
Competence gate grid	Competence floor $\{.70,.80,.90\}$ on the same $12$ large- $N$ 1B/8B carrier cells, using unpatched clean closed-set competence	Every grid point keeps all $12$ cells and the same $5$ latent-tail / $7$ prompt-prefix calls; raw competence is $.992$ – $1.000$ .
Low-competence boundary check	Qwen3-4B sanity probes under the same closed-set competence rule	Four sanity probes remain below even a $.70$ gate ( $.25$ – $.29$ ), with best oracle target win only $.25$ – $.33$ ; no mechanism call is unlocked by relaxing the default gate.
Default carrier threshold	Sufficiency $\geq.80$ , competitor $\leq.25$	The default is a reporting rule tied to a chance-level competitor ceiling; it is not needed to separate the endpoint large- $N$ split rows.
Target-win tie audit	$855{,}200$ state-score cache-interchange rows in the retained JSONL outputs	Exact top-score ties occur in $540$ rows ( $.063\%$ ); top-two score gaps within $10^{-3}$ occur in $617$ rows ( $.072\%$ ).

Appendix

Table 16: Metric audit for competence-gate, carrier-threshold, and tie-order reporting risks. The audit is generated from retained summaries and JSONL score rows by summarize_scit_metric_audit.py . It does not replace margins, free generation, or matched corruption for high-baseline cells; it shows that the headline large-

N

carrier split and cache-interchange target-win rows are not artifacts of the stated competence gate, carrier threshold, or frequent exact ties.

Diagnostic	Typical question	SCIT use	Why it matters here
Activation patching / causal tracing	Which hidden site changes an output?	Interchange source cache suffixes under exact counterfactual semantics	Keeps the high-level variable fixed by the task rather than inferred after observing a patching effect.
Residual hidden-state replacement	Is a single current vector sufficient?	Hidden-only and cache-only baselines	Separates vector-state accounts from accumulated cache-trajectory accounts.
Key/value component split	Is transfer routing-side or content-side?	Patch keys, values, K/V together, and matched corruptions	Shows whether the source effect follows attention routing or value content.
Linear probes / readout directions	Is a variable readable from a state?	Value probes and probe-direction interventions	Tests content without treating decodability as causal modularity.
Free-generation validation	Does a closed-set win survive decoding?	Greedy and sampled patched decoding on headline controls	Guards against teacher-forced candidate-set artifacts.
Matched random corruption	Is the localized region needed?	Replace recipient regions with in-distribution random-source regions	Adds a necessity check that is closer to the patch distribution than zero ablation.

Appendix

Table 17: Positioning SCIT relative to standard diagnostics. The contribution is the combined causal contract for latent-CoT rollouts: exact counterfactual source construction, component and segment splits, semantic controls, decoded validation, and matched corruption.

Issue	Diagnostic	Takeaway
Carrier	Layer-group value-cache interchange on matched counterfactual pairs	In GPT-2-scale checkpoints, value patches in a middle-to-late window transfer the counterfactual answer; nonlate values and keys are weak.
Hidden state	Hidden-state-only, cache-only, and key/value separation	The current hidden vector alone is weak; the effect follows the value cache, with checkpoint-dependent readout support from the source hidden state.
Semantic slot	Same-answer, partial-variable, and exact-context source controls	Exact counterfactual trajectories dominate semantic controls, arguing against an answer-only or modular intermediate-slot account.
Necessity	Matched random corruption of recipient cache entries	Corrupting the same GPT-2 value region selectively damages recipient-answer preservation; the Sim-CoT replication is directional but weaker.
Boundary behavior	Greedy/sampled decoding, 1B/8B calibration cells, and targeted task repairs	Arithmetic cells preserve the decoded exact-context pattern, while expression/list, relation-chain, and 8B cells show that the carrier segment can shift or fail OOD.

Appendix

Table 18: Qualitative evidence map. The main text foregrounds the compact numerical evidence in Table 1 ; this table lists the role of the supporting experiments.

Model / Template	All	Late	Late 8–9	Late 10–11	Nonlate
CODI-GPT2 / balls	1.000/0.996	1.000/0.996	0.992/0.883	0.546/0.029	0.492/0.008
CODI-GPT2 / books	1.000/1.000	1.000/1.000	0.992/0.875	0.567/0.017	0.475/0.008
CODI-GPT2 / coins	1.000/1.000	1.000/1.000	1.000/0.908	0.492/0.017	0.400/0.000
Sim-CoT-GPT2 / balls	0.983/1.000	0.942/1.000	0.750/0.992	0.033/0.133	0.033/0.175
Sim-CoT-GPT2 / books	0.992/1.000	0.950/1.000	0.833/0.958	0.075/0.192	0.108/0.208
Sim-CoT-GPT2 / coins	0.975/1.000	0.942/1.000	0.792/0.992	0.017/0.133	0.067/0.183

Appendix

Table 19: Exact late-value localization results. Entries are target-win rates at latent steps 5/6 for oracle counterfactual value-suffix patching with the joint cache–state intervention. CODI-GPT2 balls has 240 examples per cell; the other rows have two 60-example seeds. Across rows, step-6 SEM is at most 0.004 for late and 0.037 for nonlate.

Model / Template	Oracle	Same CF Ans.	Same Partial	Same Int. / Wrong Var.
CODI-GPT2 / balls	1.000/0.988	0.404/0.400	0.346/0.325	0.192/0.175
CODI-GPT2 / books	0.992/0.992	0.437/0.429	0.367/0.367	0.117/0.100
CODI-GPT2 / coins	1.000/1.000	0.450/0.450	0.308/0.308	0.125/0.142
Sim-CoT-GPT2 / balls	0.925/1.000	0.235/0.353	0.242/0.375	0.175/0.225
Sim-CoT-GPT2 / books	0.892/1.000	0.325/0.483	0.292/0.433	0.150/0.217
Sim-CoT-GPT2 / coins	0.908/1.000	0.237/0.356	0.217/0.367	0.133/0.192

Appendix

Table 20: Exact semantic source controls. Entries are target-win rates at latent steps 5/6 for late-value joint cache–state patching. CODI-GPT2 balls aggregates 240 examples per cell; the other rows use two 60-example seeds. Across reported cells, step-6 SEMs are at most 0.046. Because candidate-set scoring can give nonzero wins to random or mismatched sources, the main statistic is the oracle-control margin, not raw distance from nominal chance.

Panel of Figure 5 — Figure 5: Enlarged-candidate semantic-control check. Bars show oracle target-win minus the best non-oracle semantic-control target-win after expanding the scored answer set to about 12–13 candidates. The margins remain large, reducing the concern that semantic-control separation is an artifact of a small four- or five-answer candidate set.

Region	Comp.	Target Win	Rec. Drop
Late	Value	0.767/0.211	2.431/13.655
Nonlate	Value	1.000/1.000	0.006/0.013
Late 8–9	Value	0.906/0.319	0.874/9.837
Late 10–11	Value	1.000/0.986	0.008/0.186
Late	Key	1.000/1.000	0.002/0.004
Nonlate	Key	1.000/1.000	0.000/0.000

Appendix

Table 21: Exact necessity results via matched random suffix corruption on CODI-GPT2, aggregated over balls/books/coins. Entries are target-win rates and recipient-answer log-probability drops at latent steps 5/6; each row aggregates 360 examples.

Axis	Covered in this diagnostic	Not established
Task family	Main diagnostic: one two-hop arithmetic program expressed as balls, books, and coins templates; SCIT-Bench v2 adds clean three-hop arithmetic, conditional arithmetic, entity lookup, graph path, relation chain, finite-state tracking, fixed-rule state updates, expression trees, and list filter/sum as calibration diagnostics; competence-gated repairs expose expression/list, relation-chain, and state-update carrier shifts, plus fixed-rule sequence-holdout failure	Open-domain task families and robust long-horizon state tracking
Counterfactual semantics	Exact recipient, oracle, partial-variable, same-answer, and random sources	Naturally occurring benchmark items without constructed source controls
Numerical robustness	Wider OOD numeric values with post-transfer and held-fixed counts in 100–500 and transfer counts in 10–99	Long arithmetic, negatives, carry/borrow-heavy calculations, or symbolic algebra
Model scale	Two GPT-2-scale checkpoints plus public/matched LLaMA3-1B calibration diagnostics and matched 1B/8B carrier-split probes, including relation-chain and expression/list repair	A stable latent-tail carrier across large latent-CoT model families
Behavioral readout	Teacher-forced target win, greedy and limited temperature-sampling diagnostics, and prompt-only base-model sanity checks	Open-ended downstream deployment behavior or long-horizon reasoning

Appendix

Table 22: Benchmark scope. The SCIT task suite is designed as a controlled causal diagnostic with exact source semantics, not as a general reasoning benchmark.

Requirement	Needed for a strong generalization claim	Current status in this paper
Task structure	A balanced set of exact-counterfactual families: deeper linear arithmetic, branching arithmetic, list aggregation, symbolic binding, path/state tracking, and mixed composition	Positive latent-tail evidence is strongest for arithmetic-like cells; repaired expression/list and relation-chain cells are competent at 8B or 1B/8B but shift to prompt-prefix K/V; minimal and fixed-rule state-update cells also shift to prompt-prefix or remain finite-support bound
Model scale	The same diagnostic battery across small, 1B, mid-scale, and 8B-or-larger checkpoints, preferably with at least two architecture families	GPT-2 and 1B arithmetic-like cells support latent-tail value/KV transfer; matched 8B, expression/list repair, and relation-chain probes support full-cache/prompt-prefix transfer but not latent-tail transfer
Competence gate	Free-generation and teacher-forced competence before interpreting any causal intervention	Applied throughout the calibration cells; failed or weak cells are treated as competence-limited rather than mechanism failures
Carrier contract	For each competent cell, report hidden-only, cache-only, cache-plus-hidden, latent-tail, prompt-prefix, full-cache, key/value/KV, semantic controls, corruption, and free-generation transfer	Implemented for the main arithmetic cells and the current 1B/8B carrier-split probes; repaired expression/list and relation-chain cells have source controls, matched span corruption, prompt-segment or prompt-field split, and patched free generation; minimal and fixed-rule state-update cells have objective, horizon, segment, and sequence-holdout checks
Success criterion	Either stable latent-tail value/KV transfer across the grid, or a reproducible map of when the carrier shifts across task and scale	Current evidence supports a limited carrier-regime map, not a universal late-value mechanism

Appendix

Table 23: Generalization roadmap. A full solution to the task/scale objection requires a balanced competence-gated matrix, not additional surface variants of one program.

Checkpoint / renderings	Recip./oracle FG	Recip./oracle TW	Oracle latent/prompt/full K/V TW	Decision
Single-template: original, unseen compact	$48/48,48/48$	$48/48,48/48$	$0/1/1$ ( $N=48$ each)	prompt-prefix call retained
Single-template: unseen brief	$31/48,31/48$	$37/48,36/48$	–	fails competence; scan withheld
Multi-template: original, compact, unseen brief	$48/48,48/48$ per form	$48/48,48/48$ per form	$0/1/1$ ( $N=48$ each)	prompt-prefix call retained

Appendix

Table 24: Relation-chain surface-form pilot on the same 48 executable worlds under original, compact, and brief renderings. The five competence-passing checkpoint–form cells preserve prompt-prefix rather than latent-suffix transfer, and random prompt-prefix patches select the donor label 48/48 in every passing cell. This is competence-conditional stability under three controlled renderings, not evidence of unrestricted paraphrase, discourse, or strategy robustness.

Model	Task cell	Oracle TW	Best group	Margin	Free G/S	Corr. TW
CODI-GPT2	balls/books/coins	1.000	q3 09–11	0.380–0.463	1.000 / 0.978–1.000	0.183
Sim-CoT-GPT2	balls/books/coins	0.975–1.000	q3 09–11	0.250–0.363	1.000 / 1.000	0.283–0.367
Sim-CoT-LLaMA3-1B	balls/books/coins	1.000	q2 08–11	0.412–0.450	1.000 / 1.000	0.117–0.183
CODI-GPT2	three-hop	0.863	q2 06–08	0.350	0.467 / 0.511	0.167
Sim-CoT-GPT2	three-hop	0.713	q2 06–08	0.375	0.600 / 0.600	0.367
Sim-CoT-LLaMA3-1B	three-hop	1.000	q3 12–15	0.000	1.000 / 1.000	0.000

Appendix

Table 25: SCIT-Bench v2 calibration summary at the final latent step. Oracle TW is teacher-forced target win for all-layer value-cache oracle patching with the cache+h intervention. Best group is a coarse calibration-run partition, not the final GPT-2 block-localization result. Margin is oracle minus the best non-oracle semantic source. Free G/S gives greedy/sample parsed-answer success for oracle-patched free generation. Corr. TW is recipient target-win after all-value matched random corruption, where lower values indicate stronger necessity. Core arithmetic rows report ranges over balls, books, and coins. The public 1B three-hop row supports deeper arithmetic transfer but not semantic-source separation, because its semantic controls are not below the oracle. Entity and graph rows for the original current checkpoints fail the competence screen and are not used for mechanism claims.

Task cell	Recip. G/S	Oracle G/S	TF Oracle	Margin	Interpretation
balls/books/coins	1.000 / 1.000	1.000 / 1.000	1.000	0.000	Arithmetic transfer remains saturated, but semantic controls also saturate after augmentation.
three-hop	1.000 / 0.989	1.000 / 1.000	0.988	-0.012	Deeper arithmetic remains transferable; semantic separation is not clean.
conditional	1.000 / 1.000	0.333 / 0.333	0.863	-0.025	Competence is repaired, but free-generation transfer is weak and controls have high baselines.
entity lookup	1.000 / 1.000	0.000 / 0.000	0.000	-1.000	Competence without cache transfer; a clear boundary for the current mechanism claim.
graph path	0.167 / 0.133	0.200 / 0.089	0.138	-0.138	Mostly competence-limited, so not interpretable as a mechanism failure.

Appendix

Table 26: Competence-gated SCIT-Bench v2 augmentation on a matched CODI-LLaMA3-1B checkpoint. Recip. G/S is greedy/sample recipient free-generation accuracy. Oracle G/S is greedy/sample success after oracle all-layer value-cache patching with cache+h. TF Oracle is teacher-forced target win for the same oracle patch. Margin is oracle target win minus the best non-oracle semantic source.

Task	Scale / repair	Gate	Carrier readout
Expression/list	1B targeted repair	Weak in-domain and fails OOD gate	Targeted repair improves teacher-forced scores but does not yield an interpretable mechanism cell.
Expression/list	8B targeted repair	Passes in-domain; OOD fails or remains control-limited	Prompt-prefix/full-cache K/V; latent-tail transfer absent.
Relation chain	1B repaired	Passes held-out free generation; $N=128$ 12-room carrier refresh	Prompt-prefix/query/full-cache K/V; latent-suffix transfer absent.
Relation chain	8B repaired	Passes held-out free generation; $N=128$ 12-room carrier refresh	Same prompt-prefix/query/full-cache field-binding regime at larger scale.
Relation chain OOD	1B repaired	Fails OOD vocab/template gate	Limits repaired-task generalization.
Graph path	1B repaired	Fails short/long hop gates	Path-following competence boundary.

Appendix

Table 27: Targeted repair boundary tests. Expression/list repair supplies the cleanest in-domain exact-counterfactual task-breadth extension at 8B scale, but it exposes a prompt-prefix/full-cache carrier rather than the arithmetic latent-tail carrier and remains OOD-limited. Repaired relation-chain supplies the main non-arithmetic task-breadth supplement with the same carrier shift; full-vocabulary, sampled-decoding, large-

N

carrier refreshes, and source-field controls interpret this as visible prompt/source binding. The minimal finite-state and fixed-rule rows are competent only on narrow matched supports and point to prompt binding, not latent state tracking. Rows that fail the free-generation gate or held-out support checks are treated as boundaries rather than broad task-generalization evidence.

Cell	Gate / stability outcome	Source-control outcome	Interpretation
SCIT-Aug 1B/8B baseline	Fails greedy recipient gate	Oracle does not cleanly separate from controls	No zero-shot task-breadth evidence for expression trees or list aggregation.
1B targeted repair	Weak in-domain; fails OOD gate	Oracle is weak or tied with controls	Not a positive mechanism cell; competence repair is insufficient at this scale.
8B expression-tree repair	In-support and far-OOD sampled gates pass after wide-support repair; ultra-OOD fails	Oracle improves over controls in support, but random prompt-prefix sources remain competitive	Support-distance success with a prompt-prefix carrier, not robust OOD generalization.
8B list-filter/sum repair	In-support and far-OOD gates pass; ultra-OOD is partial	Oracle separates in support, while far/ultra-OOD are limited by strong random-source prompt-prefix patches	Same boundary: repair gives wider support competence but not a broad task-generalization claim.

Appendix

Table 28: Expression-tree and list-filter/sum repair summary. The two tasks have exact counterfactual semantics but differ from the original two-hop arithmetic templates: expression-tree uses two intermediate branches before a composed final answer, and list-filter/sum uses threshold selection plus aggregation. The 8B in-support and far-OOD cells pass recipient and oracle-patched free generation after wide-support repair, but segment tests and matched corruption localize the effective carrier to prompt-prefix/full-cache K/V rather than latent-tail K/V. A second hard-cell replicate and enlarged-candidate check preserve this prompt-prefix/source-answer pattern while showing that ultra-OOD remains a competence or source-control boundary.

Prompt-only baseline	Expression tree	List filter/sum	Paper use
1B base, answer-only	Fails in-domain and OOD	Fails in-domain and OOD	Numeric answer-only readout is a strict deployment setting, not a strong natural baseline.
8B base, answer-only	Fails in-domain and OOD	Partial but not clean	Same readout caveat; this does not by itself prove the task family is difficult.
1B base, CoT prompt	Strong in-domain and OOD with a longer output cap	Weak to partial	Expression-tree is feasible with a scratchpad, so the repair result should not be framed as natural task difficulty.
8B base, CoT prompt	Strong in-domain, OOD, and far-OOD with a longer output cap	Partial to strong, including far-OOD	Ordinary scratchpad readout can solve these task families; expression/list is best used to test carrier shifts under latent-CODI repair.

Appendix

Table 29: Prompt-only base-model sanity check for expression-tree and list-filter/sum. The table summarizes held-out base LLaMA 1B/8B runs with an answer-only numeric prompt and a scratchpad CoT prompt; the CoT rows include longer-output sanity checks, an additional seed for the 8B cells, and a farther 8B numeric-range calibration. It is a task-feasibility calibration, not a cache-intervention result.

Cell	Active prompt fields	Inactive fields	Interpretation
Expression tree, train/far support	Fact spans and changed input bindings, especially the left-branch inputs	Rule-only, query, answer-prefix, and latent-suffix spans	Transfer is tied to visible source operands rather than to a latent-tail trajectory.
Expression tree, ultra-OOD	The same fact/input ordering weakens with the failed deployment gate	Latent suffix remains ineffective	The field split mirrors the ultra-OOD competence boundary rather than rescuing it.
List filter/sum, train/far/ultra	Fact/rule spans, with threshold/filter fields as partial carriers	Raw value bindings, answer-prefix, query, and latent-suffix spans	Transfer follows the visible filtering rule that determines which items enter the aggregation.

Appendix

Table 30: Prompt-field ablation for the repaired 8B expression/list cells. The table reports gate-level qualitative outcomes to avoid another dense numeric grid. The active 8B carrier decomposes into visible prompt fields: expression-tree transfer follows source operand/fact bindings inside support, while list-filter/sum follows rule/fact and threshold/filter fields. Latent-suffix and answer-prefix spans do not support transfer.

Probe	Held-out structure	Deployment result	Interpretation
Matched 8B two-step repair	No same-horizon sequence holdout	Passes the matched two-step gate; fails shorter/longer horizon checks	A useful competent calibration cell, but its carrier is visible start/query and prompt-prefix cache rather than latent-tail rule execution.
Replicated sequence holdout	Three separate splits, each removing multiple bigrams	Seen supports pass; held-out supports fail	The matched success is not stable once small same-horizon regions are removed.
Leave-one bigram grid	Train on eight of nine two-symbol sequences	All seen splits pass; every held-out split fails the free-generation gate	Rules out the objection that the earlier held-out splits removed too much support. Partial cells remain below the deployment gate.
Net-displacement class holdout	Remove same-net or reverse bigram classes together	Seen supports pass; held-out classes collapse	The partial leave-one signals are best explained by local interpolation from nearby seen bigrams, not by algorithmic state updating.

Appendix

Table 31: Fixed-rule support-generalization boundary for the high-update 8B two-step repair. The table intentionally reports gate-level outcomes rather than another dense numeric grid: the conclusion is qualitative and stable across the leave-one and class-holdout variants.

Probe	What it tests	Observed pattern	Interpretation
Block-wise value probes	Whether value blocks contain readable arithmetic variables	Input/context variables are strongly decodable; intermediate and outcome variables are only partial	The trajectory contains task information, but not a modular symbolic trace.
Probe-direction intervention	Whether a decoded variable direction is a causal control direction	Probe margins can be moved strongly, but the intended counterfactual answer does not overtake the recipient answer	Decodability is not enough for causal modularity.
Semantic source controls	Whether matching an answer or partial variable is sufficient for transfer	Exact counterfactual context dominates same-answer and partial-variable controls	The causal carrier is context-bound rather than a reusable answer or intermediate slot.
Cross-template sources	Whether exact context transfer is tied to surface wording	Exact arithmetic contexts transfer across balls/books/coins wording	The trajectory is not merely lexical-template binding.
Suffix-window, head, and geometry scans	Whether transfer is a last-token, single-head, or purely lexical trigger	Multi-token suffixes are needed; ranked multi-head patches beat random; exact cross-template pairs align more than off-diagonal pairs	The causal effect is accumulated, sparse-distributed, and partly representation-aligned over the value trajectory.

Appendix

Table 32: Qualitative trajectory-content checks. The table summarizes probe and source-control evidence without presenting the appendix as a separate probing benchmark. The main conclusion is negative but useful: the value-cache trajectory carries readable arithmetic information, yet the readable axes do not form independently transplantable symbolic variables, and the transfer is not explained by a single final token or a single value head.

Model	$k=0$	$k=1$	$k=2$	$k\geq 3$
CODI	0.000	0.203	0.203	0.984–1.000
Sim-CoT	0.026	0.516	0.995	1.000

Appendix

Table 33: Final-step suffix-window target-win rates for oracle cache+h patches, aggregated over balls/books/coins with 192 examples per row.

k

is the number of latent cache positions replaced;

k=0

applies no cache-tail replacement and therefore isolates current-hidden replacement under the cache+h variant. CODI denotes CODI-GPT2; Sim-CoT denotes Sim-CoT-GPT2.

Patch	All /48	Equal /21	Unequal /27
Full latent K/V+ $h$	47	21	26
$k=2$ latent K/V+ $h$	47	21	26
$k=1$ latent K/V+ $h$	44	20	24
Full latent V+ $h$	47	21	26
Full latent K+ $h$	18	8	10
Full latent K/V only	27	10	17

Appendix

Table 34: Adaptive-stop exact-oracle target-win successes at the frozen clean recipient stop. Among 96 trajectories, realized stops are

T=3/4/5

for

35/27/34

trajectories; 27/48 source–recipient pairs have unequal stops. Reverse-aligned full and

k=2

latent K/V+

h

each succeed on 47/48 pairs, but K/V-only and

h

-only fail the sufficiency gate. Value+

h

success does not establish a value-specific carrier because key+

h

exceeds the inactivity ceiling. The pilot supports joint-object sufficiency, not constituent necessity or intervention-induced re-halting.

Patch	1h	3h	5h	8h
Ranked, cache+h	0.009	0.255	0.833	0.982
Random, cache+h	0.000	0.003	0.005	0.023
Ranked, cache-only	0.023	0.287	0.824	0.968
Random, cache-only	0.002	0.005	0.011	0.020

Appendix

Table 35: Value-head Pareto diagnostic for CODI-GPT2 arithmetic at the final latent step. Entries are target-win rates after patching the top-ranked or random value heads in the late value window, aggregated over balls/books/coins. Single heads are not sufficient, but a small ranked set recovers the transfer far faster than random same-size sets. The effect is therefore sparse-distributed across value heads rather than a named single-head circuit or a uniform all-head effect.

Cell	Active carrier	Top-1	Top-4	Top-8	Top-16	Random-64
1B arithmetic	Latent-tail value	0.000	0.727	1.000	1.000	0.445
1B entity	Prompt-prefix K/V	0.023	0.781	1.000	1.000	0.531
8B arithmetic	Prompt-prefix K/V	0.996	1.000	1.000	1.000	0.254
8B entity	Prompt-prefix K/V	0.000	0.504	0.996	1.000	0.117
8B expression	Prompt-prefix K/V	0.453	0.793	0.883	0.965	0.160
8B list	Prompt-prefix K/V	0.027	0.527	0.734	0.996	0.141

Appendix

Table 36: Cross-scale ranked-head Pareto diagnostic on the active carrier for each cell. Entries are final-step target-win rates under cache-only patches on two held-out evaluation replicates (

N=256

pooled per row), using head rankings selected on a separate small scan set. The active carrier is not uniformly spread across all heads: ranked heads recover transfer much faster than random head sets. Some 8B prompt-prefix cells are highly concentrated, while list-filter/sum remains more distributed and reaches full recovery only at larger ranked-head budgets.

Cell	SCIT carrier	$\Delta$ KV stable rank, Lat./Pref.	$\Delta$ KV norm, Lat./Pref.	Top-8 attn., Lat./Pref.	Time scan
1B arithmetic	Latent-tail value/KV	12.1 / 11.8	128 / 131	.111 / .781	Latent c-only 4/6; latent +h 6/6. Prefix c-only 0/6.
8B arithmetic	Prompt-prefix K/V	36.9 / 12.4	5.9 / 320	.011 / .973	Prefix c-only/+h 6/6; latent 0/6.
8B expression	Prompt-prefix K/V	33.9 / 15.3	12.9 / 418	.006 / .978	Prefix c-only/+h 6/6; latent 0/6.
8B list	Prompt-prefix K/V	40.7 / 30.7	39.2 / 374	.013 / .949	Prefix c-only/+h 6/6; latent 0/6.
8B relation	Prompt-prefix K/V	20.9 / 35.3	141 / 304	.033 / .959	Prefix c-only/+h 6/6; latent 0/6.

Appendix

Table 37: Auxiliary carrier-shift diagnostics.

\Delta

KV is the source-minus-recipient K/V matrix at the final latent step; stable rank is

\|M\|_{F}^{2}/\|M\|_{2}^{2}

, and norm is Frobenius norm. The full diagnostic file also reports centered effective rank and attention-entropy bootstrap intervals; these are summarized here only through the compact stable-rank, norm, and readout-mass columns. Top-8 attention reports answer-readout mass to the latent suffix and prompt prefix for heads ranked by an independent carrier-head scan. The time scan counts the number of latent steps, out of six, where oracle K/V patching reaches saturated target-win in a required-coverage run that compares cache-only and cache+h patches; Table 9 gives the later

N=48

four-cell follow-up. These diagnostics support the prompt-prefix carrier in the 8B cells, but they are not causal proof by themselves. The 1B row is intentionally mixed: SCIT patches identify a latent-tail carrier, while answer-readout attention remains prompt-heavy and prompt-prefix cache+h has isolated hidden-state-assisted positives.

Block	CKA	Pair	Offdiag	Gap
Nonlate	.999	.996	.720	.277
Late 8–9	.999	.995	.563	.432
Late 10–11	.999	.996	.432	.564

Appendix

Table 38: Cross-template value-geometry check at the final latent step. Rows average balls/books/coins pairs matched by the same arithmetic state. CKA is saturated and should not be over-interpreted; the useful signal is the paired-minus-offdiagonal cosine gap, which is larger in the value blocks implicated by the SCIT transfer than in the nonlate block. This is representation corroboration, not causal proof.

Model	All	Late	8–9	10–11	Nonlate
CODI	1.000	0.996	0.917	0.017	0.000
Sim-CoT	1.000	1.000	0.984	0.142	0.134

Appendix

Table 39: Benign prefix-noise robustness. Entries are final-step target-win rates averaged over balls/books/coins after prepending a neutral natural-language sentence to the prompt. The same late and middle-to-late value-cache localization remains active, while final-only and nonlate patches stay weak. This argues against an absolute-position or rigid-template explanation of the arithmetic localization.

Task	L9H1	L8H1	L8H8
Balls	1.225	1.060	0.865
Books	1.428	1.167	0.914
Coins	1.305	1.102	0.915

Appendix

Table 40: Value-gradient readout diagnostic for CODI-GPT2 under late-value patching. Entries are average gradient-times-value magnitudes for the answer-margin objective at recurring high-sensitivity heads. Sensitivity concentrates in the same layers 8–9 value region implicated by suffix interchange. This is a readout-sensitivity check, not proof of a single-head circuit.

Span	Ablation	Rec. win	Rec. drop
Latent value	zero	0.425	11.43
Latent K/V	zero	0.388	12.49
Latent key	zero	0.900	1.02
Prompt K/V	zero	0.254	13.81
Whole K/V	mean	0.279	30.17

Appendix

Table 41: Source-free cache-ablation specificity check on CODI-GPT2 arithmetic. Entries average balls/books/coins at the final latent step. Direct zero or mean replacement is harsher and more out-of-distribution than matched random corruption, so it is not the primary necessity evidence. It nevertheless shows that damaging value/K/V cache spans disrupts recipient-answer readout much more than latent-key-only zeroing, while broad prompt or whole-cache ablation unsurprisingly causes larger global damage.

Probe	Positive signal	Latent-tail result	Interpretation
Quick GPT-2 re-training dynamics	Oracle/full-cache target win rises from 0.58 at checkpoint 100 to 0.81 at checkpoint 440; prompt-prefix tracks the same curve	Latent-tail-only stays weak, roughly 0.14–0.23 target win with near-zero margin	This quick path does not show a clean ontogeny in which prompt binding compresses into a latent-tail carrier.
Stronger true-final $T=6$ recipe	Whole-cache and oracle transfer are high; a new-seed replicate preserves the effect	Latent-tail-plus-hidden recovers most oracle transfer, and suffix windows rise smoothly from near zero to high transfer	The arithmetic latent-tail regime is recoverable under stronger training, but it is not automatic.
Adjacent $T=6$ recipes	Higher learning rate and wider numeric support keep whole-cache transfer high	Latent-tail suffix patches remain weak while prompt-prefix/full-cache transfer dominates	Carrier choice is training-recipe sensitive even within GPT-2 arithmetic.
Stronger true-final $T=12$	Whole-cache replacement remains strong	Sliding suffix patches from $k=0$ to $k=11$ stay near chance, while prompt-prefix transfer is high	Longer latent rollout alone does not imply latent-tail accumulation.
Stronger true-final $T=16$	Oracle and whole-cache transfer are near saturated; a larger replicate preserves the effect	Latent-tail-plus-hidden is near saturated, and suffix windows rise gradually from one-token failure to full-suffix recovery	Longer horizons can recover a trajectory regime, but only for some recipes; horizon scaling is not monotonic.
Range-augmented $T=6$ support-wide	Recipient and oracle free-generation pass on the expanded numeric support across a larger evaluation replicate	Latent-tail-plus-hidden and full-cache transfer are saturated, while prompt-prefix remains weak; prompt-field splits localize the transfer to latent-suffix value/KV, not visible facts	Numeric support expansion can repair the old wide-range boundary inside support and restore the arithmetic latent-tail carrier.

Appendix

Table 42: Exploratory training-path, latent-horizon, and numeric-support probes. These newly trained GPT-2 CODI checkpoints are not used to replace the original CODI-GPT2 result. They calibrate its scope: stronger training and range augmentation can recover a latent-tail trajectory in selected arithmetic recipes, but adjacent recipes, one competent

T=16

seed replicate, and farther OOD cells expose prompt-prefix/facts, weak-transfer, or deployment-readout boundary regimes. The T16 seed-replicate dynamics favor early carrier-basin bifurcation over either method instability or a universal late-training compression into latent-tail values.

Seed	Clean recipient TW	Late 8–9 suff.	Late 10–11 suff.	All-value suff.	Late 8–9 corrupt TW	Drop
s204a	.951 [.927,.971]	.138 [.104,.172]	.328 [.281,.375]	.464 [.414,.510]	.904 [.875,.932]	.047
s204b	.938 [.911,.961]	.049 [.029,.070]	.143 [.109,.177]	.214 [.172,.253]	.849 [.812,.883]	.089
s204c	.987 [.974,.997]	.135 [.102,.169]	.516 [.464,.562]	.479 [.427,.526]	.961 [.940,.979]	.026

Appendix

Table 43: CODI-GPT2 multi-seed replication audit. Rows pool balls/books/coins with

N=384

examples per seed and report bootstrap 95% intervals in brackets. Sufficiency columns are oracle value-cache target-win rates under the layer-suffix intervention; corruption target win is recipient-answer preservation after matched random late-8–9 value corruption. A follow-up latent/prompt/whole-cache K/V scan gives

.370/.203/.870

(s204a),

.208/.427/.865

(s204b), and

.427/.141/.906

(s204c). Whole-cache transfer is high, but neither disjoint segment reaches the

.80

local-carrier gate, so the protocol returns no localized carrier and does not run finalist semantic or decoded tests. These competent seeds are therefore seed-sensitivity evidence, not positive replications of the headline CODI-GPT2 mechanism.

Checkpoint	Task cell	Latent-tail KV	Prompt-prefix KV	Full KV	Call
1B + SCIT aug.	balls/books/coins	1.000	0.000	1.000	latent tail
1B + SCIT aug.	three-hop	1.000	0.000	1.000	latent tail
1B + SCIT aug.	conditional	1.000	0.000	1.000	latent tail
1B + SCIT aug.	entity lookup	0.000	1.000	1.000	prompt prefix
1B relation repair	relation chain	0.000	1.000	1.000	prompt prefix
1B finite-state tiny	one-symbol/one-step	0.000	1.000	1.000	prompt prefix

Appendix

Table 44: Carrier-split diagnostic at latent step 6. Entries are teacher-forced target-win rates for K/V cache replacement by segment. The augmented 1B/8B arithmetic, conditional, and entity rows use large-

N

held-out refreshes (

N=128

per task). The repaired relation-chain rows use

N=128

per scale with the full 12-room candidate vocabulary; query-only K/V patches also reach

1.000

target win, confirming a visible query/source-field carrier. For the 1B augmented checkpoint, value-only latent-tail patches are also sufficient for balls/books/coins/three-hop and near-sufficient for conditional; entity lookup instead requires prompt-prefix K/V. Expression/list repair passes the in-domain gate at 8B and also requires prompt-prefix/full-cache K/V. Relation-chain repairs pass the competence gate at both 1B and 8B scale, but again require prompt-prefix/full-cache K/V. The tiny finite-state control is competent only in a one-symbol/one-step setting and also follows prompt-prefix K/V. For the 8B augmented checkpoint, latent-tail key, value, and K/V patches are ineffective across the tested cells, while prompt-prefix K/V succeeds.

Model	All	Late	Nonlate	6–8	9–11	6–7	8–9	10–11	6–9
CODI-GPT2	1.000	0.994	0.000	0.211	0.639	0.000	0.900	0.011	0.967
Sim-CoT-GPT2	1.000	1.000	0.161	0.717	0.956	0.133	0.983	0.117	0.989

Appendix

Table 45: GPT-2 contiguous layer-block scan. Entries are final-step target-win rates for oracle counterfactual value-suffix patching with cache+h, aggregated over balls/books/coins with 180 examples per row. Late is layers 8–11 and nonlate is 0–7; the remaining columns are contiguous three-layer groups, two-layer blocks, and the 6–9 sliding window.

Model / Range	All	Late	Late 8–9	Late 10–11	Nonlate
CODI-GPT2 / wide OOD	0.794/0.797	0.797/0.761	0.719/0.594	0.453/0.133	0.386/0.081
Sim-CoT-GPT2 / wide OOD	0.572/0.692	0.506/0.686	0.453/0.583	0.183/0.217	0.203/0.217

Appendix

Table 46: OOD numeric-range localization, aggregated over balls/books/coins. Entries are target-win rates at latent steps 5/6 for oracle counterfactual value-suffix patching with cache+h. OOD examples use post-transfer and held-fixed counts in 100–500 and transfer counts in 10–99.

A. 1B vector-versus-cache diagnostic, step 6 target win
Model	Task scope	Hidden only	Latent cache	Cache+h	Last cache+h
Sim-CoT-LLaMA3-1B	balls/books/coins	0.000	1.000	1.000	0.000
Sim-CoT-LLaMA3-1B	+ three-hop	0.000	1.000	1.000	0.000
CODI-LLaMA3-1B	balls/books/coins	0.000	0.933	0.933	0.008
CODI-LLaMA3-1B	+ three-hop	0.000	0.950	0.950	0.006
B. 1B wide numeric OOD, all-layer value cache+h, step 6 target win

Appendix

Table 47: 1B calibration diagnostics. Panel A reports oracle counterfactual vector-versus-cache target-win rates. Panel B reports wide numeric OOD examples with post-transfer and held-fixed counts in 100–500 and transfer counts in 10–99. The random-source average in Panel B is high, so the wide-OOD result is used as robustness evidence for all-layer transfer rather than as a clean semantic-control result.

Intervention	Target Win	Target Delta
Late value cache-only	0.469/0.998	12.755/15.618
Late value cache+h	1.000/0.998	15.642/15.618
Late 8–9 cache-only	0.152/0.887	9.554/14.946
Late 8–9 cache+h	0.994/0.887	15.616/14.951
Nonlate cache-only	0.002/0.006	0.774/1.391
Nonlate cache+h	0.465/0.006	12.482/1.402

Appendix

Table 48: Cache patch diagnostic on CODI-GPT2, aggregated over balls/books/coins. Entries are latent steps 5/6.

Intervention	Target Win	CF Gap Delta
Hidden state only	0.246/0.000	12.116/0.003
Latent cache only	0.704/1.000	19.470/31.898
Latent cache + h	1.000/1.000	31.893/31.893
Last latent cache + h	0.267/0.196	12.254/11.521
Full cache only	0.758/1.000	20.564/32.523
Full cache + h	1.000/1.000	32.522/32.522

Appendix

Table 49: Vector-versus-cache baseline on CODI-GPT2, aggregated over balls/books/coins. Entries are latent steps 5/6.

Intervention	Target Win	CF Gap Delta
Hidden state only	0.017/0.033	4.869/6.270
Latent cache only	0.361/0.928	12.806/21.897
Latent cache + h	0.944/1.000	21.961/28.198
Last latent cache + h	0.672/0.589	16.751/15.700
Full cache only	0.983/0.961	24.197/23.052
Full cache + h	1.000/1.000	29.022/29.022

Appendix

Table 50: Vector-versus-cache baseline on Sim-CoT-GPT2, aggregated over balls/books/coins. Entries are latent steps 5/6.

Intervention	TW s5	TW s6	$\Delta$ s5	$\Delta$ s6
Key suffix cache-only	0.000	0.000	0.380	0.786
Value suffix cache-only	0.750	1.000	14.704	16.408
KV suffix cache-only	0.583	1.000	14.164	16.414
Key suffix cache+h	0.175	0.000	9.966	0.777
Value suffix cache+h	1.000	1.000	16.411	16.409
KV suffix cache+h	1.000	1.000	16.414	16.414

Appendix

Table 51: Key-versus-value sufficiency on CODI-GPT2 balls. TW is target win;

\Delta

is target delta. Splitting latent steps 5 and 6 into separate columns avoids ambiguous step-pair rendering in narrow appendix columns.

Recipient	Source	Target Win	Target Delta
balls	In-template oracle	1.000/1.000	16.227/16.206
balls	Same-template partial-variable	0.308/0.258	2.751/3.295
balls	Cross-template books	1.000/0.992	16.226/16.202
balls	Cross-template coins	1.000/0.992	16.226/16.203
balls	Cross-template partial books	0.333/0.300	2.720/3.219
balls	Cross-template partial coins	0.300/0.267	3.103/3.481

Appendix

Table 52: Cross-template controls on CODI-GPT2. Entries are latent steps 5/6 for late-value joint cache–state patching. Cross-template exact sources express the same counterfactual arithmetic context in another lexical domain; cross-template partial sources preserve only the partial-variable match while changing another arithmetic input.

Recip.	Oracle	Exact	Same Part.	Part.
balls	0.983/1.000	0.992/1.000	0.233/0.317	0.208/0.308
books	0.967/1.000	0.958/1.000	0.217/0.317	0.200/0.317
coins	0.933/0.983	0.933/0.983	0.183/0.283	0.208/0.308

Appendix

Table 53: Sim-CoT-GPT2 cross-template controls. Entries are target-win rates at latent steps 5/6 for late-value cache+h patching. Exact Avg. averages the two exact cross-template source domains; Part. Avg. averages the two partial-variable cross-template controls.

Recipient	Source	N	Free CF	Free Recip.	Teacher Target
balls	Oracle	60	0.967	0.017	0.983
balls	Exact x-template avg	120	0.975	0.008	0.992
balls	Same partial vars	60	0.000	0.100	0.367
balls	Partial x-template avg	120	0.000	0.058	0.292
balls	Random source	60	0.050	0.017	0.583
books	Oracle	60	0.983	0.000	1.000

Appendix

Table 54: Greedy free-generation validation on CODI-GPT2 at latent step 6 using late-value cache+h patching. Free CF is the fraction of patched greedy decodes whose parsed answer equals the counterfactual answer; Free Recip. is the corresponding recipient-answer rate. Teacher Target reports the teacher-forced target-win rate for the same generated state.

Recip.	Oracle	Exact	Part.	Rand.
balls	1.000	1.000	0.000	0.033
books	0.983	0.983	0.008	0.050
coins	1.000	1.000	0.000	0.067

Appendix

Table 55: Sim-CoT-GPT2 greedy free-generation validation at latent step 6 using late-value cache+h patching. Entries are Free CF rates: the fraction of patched greedy decodes whose parsed answer equals the counterfactual answer.

Model	Source group	Samples	Free CF	Free Recip.	Out-of-set	Entropy
CODI-GPT2	Oracle	300	0.967	0.020	0.013	3.844
CODI-GPT2	Exact x-template	600	0.970	0.022	0.008	3.849
CODI-GPT2	Same partial vars	300	0.000	0.050	0.867	3.671
CODI-GPT2	Partial x-template	600	0.000	0.057	0.848	3.782
CODI-GPT2	Random source	300	0.017	0.023	0.843	3.677
Sim-CoT-GPT2	Oracle	300	0.993	0.000	0.007	3.793

Appendix

Table 56: Sampled free-generation diagnostic at latent step 6 with late-value cache+h patching. Each patched state is decoded with five samples using temperature 0.7, top- p 0.95, and top- k 40. Free CF and Free Recip. are parsed-answer rates; Out-of-set counts parsed answers outside the candidate answer set. Entropy is answer entropy in bits, averaged over task-level summaries.

Model	Patch	Oracle	Exact Avg.	Same Part.	Part. Avg.	Random
CODI-GPT2	cache only	0.978	0.978	0.006	0.006	0.033
CODI-GPT2	cache+h	0.978	0.978	0.006	0.006	0.033
Sim-CoT-GPT2	cache only	0.636	0.632	0.031	0.025	0.031
Sim-CoT-GPT2	cache+h	0.992	0.992	0.000	0.003	0.028

Appendix

Table 57: Cache-only free-generation diagnostic. Entries are model-level averages over balls, books, and coins at latent step 6. CODI-GPT2 preserves the decoded counterfactual effect under cache-only late-value patching. Sim-CoT-GPT2 retains a substantial cache-only effect, but source hidden-state replacement improves decoded readout.

Region	Comp.	Target Win	Rec. Drop
Late	Value	0.975/0.606	0.351/3.912
Nonlate	Value	1.000/1.000	0.001/0.004
Late 8–9	Value	0.986/0.753	0.226/2.353
Late 10–11	Value	1.000/1.000	0.000/0.004
Late	Key	1.000/1.000	-0.000/0.001
Nonlate	Key	1.000/1.000	0.000/0.000

Appendix

Table 58: Sim-CoT-GPT2 necessity via matched random suffix corruption, aggregated over balls, books, and coins. Entries are latent steps 5/6.

Transformer Interpretability

Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.

See all 339 papers

Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models
Feb 3, 2026Anish Sathyanarayanan, Aditya Nagarsekar, Aarush RathoreCoT Faithfulness Faithfulness of Language Model Explanations
BITS Pilani, K. K. Birla Goa Campus
Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning
May 22, 2026Jinghan Jia, Joe Benton, Eric EasleyCoT Faithfulness LLM Interpretability
†Dept. CSE, Michigan State University · ⋆Anthropic Fellows Program · ‡Anthropic
From Concept Alignment to Causal Grounding: An Intervention Test of Chain-of-Thought Faithfulness
Sep 19, 2026Qianli Wang, Yilong Wang, Dennis Wei +5CoT Faithfulness LLM Interpretability
Technische Universität Berlin · German Research Center for Artificial Intelligence (DFKI) · IBM Research +4

Feb 3, 2026J/K move · Enter open · S save

Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models

Anish Sathyanarayanan, Aditya Nagarsekar, Aarush Rathore

BITS Pilani, K. K. Birla Goa Campus

CoT Faithfulness Faithfulness of Language Model Explanations LLM Auditing Implicit Reasoning in Language Models Causal Interpretability Causal Interventions in Language Models+2

Open paper AskPDF arXiv

Chain-of-thought (CoT) prompting is widely used as a reasoning aid and is often treated as a transparency mechanism. Yet behavioral gains under CoT do not imply that the model's internal computation causally depends on the emitted reasoning text, i.e. models may produce fluent rationales while routing decision-critical computation through latent pathways. We introduce a causal, layerwise audit of CoT faithfulness based on activation patching. Our key metric, the CoT Mediation Index (CMI), isolates CoT-specific causal influence by comparing performance degradation from patching CoT-token hidden states against matched control patches. Across multiple model families (Phi, Qwen, DialoGPT) and scales, we find that CoT-specific influence is typically depth-localized into narrow ''reasoning windows,'' and we identify bypass regimes where CMI is near-zero despite plausible CoT text. We further observe that models tuned explicitly for reasoning tend to exhibit stronger and more structured mediation than larger untuned counterparts, while Mixture-of-Experts models show more distributed mediation consistent with routing-based computation. Overall, our results show that CoT faithfulness varies substantially across models and tasks and cannot be inferred from behavior alone, motivating causal, layerwise audits when using CoT as a transparency signal.

Abstract

Transformer Interpretability

Explore similar work

Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models

Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning

From Concept Alignment to Causal Grounding: An Intervention Test of Chain-of-Thought Faithfulness