When Does Correction Become Repair? Mechanistic Auditing of Internal Interventions in Tool-Using LLMs

Jiayi Li; Ruizhe Li

cs.CLSep 28, 2026

When Does Correction Become Repair? Mechanistic Auditing of Internal Interventions in Tool-Using LLMs

Authors: Jiayi Li, Ruizhe Li

Organizations: University of the Chinese Academy of Sciences, China · School of Computer Science, University of Birmingham, UK

Abstract

Before invoking external tools, an agentic LLM must select among a K-way action space: executing a call, seeking clarification, answering directly, or declining. While internal activation steering can alter these pre-execution decisions, conventional aggregate metrics obscure where altered states land and what collateral damage they inflict. We present SAKIKO, an auditing framework that formalizes representation repair via directional error discovery, router-conditioned intervention, destination-resolved verification, and prospectively frozen statistical licensing. Across seven LLMs on When2Call and MetaTool, channel-keyed interventions induce direction-specific net gains in five models; across three sealed evaluations, none of 59 budget-matched random directions matches calibrated target gain. Crucially, destination auditing shows that behavioral movement does not equal repair: an intervention achieving +55 net gain corrupts over half of the baseline-correct decisions it touches, and promising point estimates on Qwen3-4B and Gemma-2-9B are formally declined due to finite-sample uncertainty. SAKIKO establishes the necessity of outcome-resolved adjudication before claiming internal repair. Code: https://github.com/ruizheliUOA/mechanistic-tool-use-llm.

Figures & tables

Figure 1: Overview of SAKIKO. The framework discovers directional pre-execution tool-decision error channels, detects channel-associated states per input, applies the corresponding channel-keyed intervention, verifies the post-intervention destination and collateral on initially correct decisions, and licenses only the strongest repair claim supported by the evidence.

Figure 2: Model-side view of SAKIKO. (a) Over a frozen model, a first pass reads hidden state at observation layer $\ell_{\mathrm{obs}}$ and a channel Router decides whether to fire; on a miss the baseline decision $\hat{y}_{0}$ stands, and on a hit the input is rerun so the correction can be applied at the injection layer $\ell_{\mathrm{inj}}$ , which precedes $\ell_{\mathrm{obs}}$ in the stack. (b) Each baseline error is keyed by its direction $c=(y^{\star}\rightarrow\hat{y}_{0})$ and corrected by adding $q_{c}s_{c}d_{c}$ at that site, with $d_{c}$ unit-norm, $s_{c}$ a local activation scale and $q_{c}$ a dose frozen before evaluation. (c) The decision precedes any tool call: a frozen readout scores the action space $\mathcal{Y}$ and its argmax gives $\hat{y}_{0}$ , and $\hat{y}_{1}$ after intervention, with destination, collateral and licensing.

Figure 3: Correction is direction-specific. (a) On Phi-3.5 locked test, calibrated direction yields a net gain of

+55

, whereas random and sign-reversed vectors yield

+14

, layer shifts yield

+3

, and mismatched channel vectors incur

-17

(arms colored by corrupted component; four arms show aggregate counts). (b) Under sealed protocol, all 59 budget-matched random vectors passing the Router gate trail calibrated target gain (add-one

p=0.017

across all settings).

Figure 4: Correction is not repair. (a) Outcome-resolved steering on Phi-3.5-mini. Of 200 routed errors, 107 reach the target while 46 shift into alternative errors; concurrently, 52 of 93 exposed correct decisions break, despite a net

+55

gain. (b) Target-hit rates (

95\%

bootstrap CIs) against the frozen

0.50

criterion. Qwen3-8B clears the gate; Qwen3-4B and Gemma-2-9B yield favorable point estimates whose confidence intervals cross the threshold, warranting formal DECLINE verdicts due to evidentiary shortfall rather than uncorrectability.

Figure 5: Destination correctness and preservation are separate properties. (a) On Qwen3-8B, activation steering and logit shifting yield similar target arrivals but differ descriptively in lateral error spillage (14 vs. 21 exits;

\mathrm{TH}_{c}=0.731

vs.

0.638

), without surviving multiplicity correction (McNemar

p=1.0

and

p=0.189

). (b) Collateral damage across denominators vs. a

5\%

tolerance: Phi-3.5-mini incurs

52/93

exposed (

\mathrm{E2}

) and

52/264

overall (

\mathrm{E1}

) breaks; Qwen3-8B incurs

0/211

overall (6 exposed); Qwen3-4B incurs

1/50

exposed; and Gemma-2-9B incurs

1/11

exposed (

\mathrm{E2}=9.1\%>5\%

). By construction,

\mathrm{E2}\geq\mathrm{E1}

. (See Figure 4 b for licensing intervals).

Figure 6: One question under three input variants (panels A–C). Open circles and solid dots indicate pre- and post-intervention decisions, showing an identical shift across panels ( tool_call

\rightarrow

request_for_info ). Filled labels denote the ground-truth target; in panel B, the post-intervention dot lands above the target, capturing an exit without target arrival. The bottom bands contrast readout resolutions: transition is unsegmented; accuracy captures A and C but misses B (consistently incorrect); destination resolves all three distinct outcomes. Visualizations are illustrative; counts represent frozen outcome-class totals rather than specific transition frequencies.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Population	Outcome Class	Condition
Baseline Channel Errors ( $N_{c}$ , where $\hat{y}_{0}=s$ )	SOURCE RETAINED ( $S_{c}$ )	$\hat{y}_{1}=s$
	GOLD ARRIVAL ( $A_{c}$ )	$\hat{y}_{1}=g$
	OTHER WRONG ( $O_{c}$ )	$\hat{y}_{1}\notin\{g,s\}$
Exposed Baseline Correct ( $C_{\mathrm{exp}}$ , where $\hat{y}_{0}=y^{\star}$ )	CORRECT RETAINED	$\hat{y}_{1}=y^{\star}$
	BROKEN ( $B$ )	$\hat{y}_{1}\neq y^{\star}$

Appendix

Table 1: The five mutually exclusive post-intervention outcome classes. Aggregate net gain (

G=\sum_{c}A_{c}-B

) leaves exits to other incorrect actions (

O_{c}

) unobserved and allows target arrivals (

A_{c}

) and collateral breaks (

B

) to artificially cancel.

Symbol	Quantity	Population (denominator)	Defined in
$c=(g\rightarrow s)$	directional error channel	—	equation 1
$N_{c}$	channel errors	—	equation 2
$E_{c}^{\mathrm{rt}}$	routed channel errors	—	equation 3
$C$	baseline-correct decisions	—	equation 4
$C_{\mathrm{exp}}$	exposed correct decisions	—	equation 5
$S_{c}$	source retained	$N_{c}$	equation 6

Appendix

Table 2: Notation and endpoint definitions. The population column is the denominator; quoting a rate without it is ambiguous. Populations are defined in equation 2 –equation 5 .

Methodology	Intervention Paradigm	Action Space	Dest.	Lic.
ToA Position ( Wang et al., 2026a )	Conceptual framework	Epistemic boundary	×	×
Tool Hallucination Probes ( Healy et al., 2026 )	Passive linear probing	Binary (error / valid)	×	×
Beyond the Black Box ( Tatsat and Shater, 2026 )	Passive SAE $+$ probing	Binary / ternary risk	×	×
Action-Class Diagnostic ( Zhao et al., 2026 )	Diagnostic decomposition (no intervention)	Four-class	×	×
CAST ( Lee et al., 2025 )	Conditional steering	Binary (refuse / comply)	×	×
Tool Steering Probes ( Wu et al., 2026b )	Additive steering	Pairwise (tool A $\to$ B)	×	×

Appendix

Table 3: Comparison of SAKIKO with the agent interpretability and steering frameworks we survey. Unlike the passive diagnostic tools and binary or pairwise steering methods listed here, SAKIKO addresses multiclass action spaces, tracks post-intervention destination distributions, verifies baseline preservation, and formally adjudicates repair claims under sample uncertainty. Dest. is destination verification and Lic. is evidential licensing. The four-class row is diagnostic only: it decomposes failures without intervening, so no destination distribution is produced to verify.

Stage	Phase	Requires labels	Data it may access
Discover	offline	yes	train split
Support	offline	yes	train, validation
Detect	offline (fit) / online (apply)	fit only	train, validation
Estimate	offline	yes	train only
Intervene	offline (calibrate) / online (apply)	calibrate only	development
Verify	offline	yes	evaluation

Appendix

Table 4: Which stages require labelled data and which run at inference time. Only Detect and Intervene have an online counterpart; everything that consumes reference labels happens before deployment.

Model	Protocol	Effect	Controls	Specific	Verdict
Phi-3.5-mini	historical	net +55	random +14, reversed +14, wrong layer +3, mismatched channel $-17$ ; five seeds +53 to +67	yes	not sealed
Qwen2.5-7B	historical	net +79	reversed +16; ten random directions, mean +25.2, largest +50	yes	not sealed
Llama-3.1-8B	historical	— †	as many as 5 of 20 random directions reach the real effect	no	not sealed
Mistral-7B	historical	net +12	random mean +29, 17 of 20 reach the real effect; reversed +19	no	not sealed
Qwen3-8B	sealed	0.276	0 of 59 random directions reach it (largest 0.023)	yes	ADMIT
Qwen3-4B	sealed	0.081	0 of 59 (largest 0.000)	yes	DECLINE

Appendix

Table 5: Correction across seven evaluated settings. Effect is net gain on the locked test (historical) or the target gain on the evaluation population (sealed). Controls summarises the comparison that decides direction specificity. Historical and sealed settings use different protocols and are not compared with each other. Historical settings carry no formal verdict: the frozen gate was written afterwards and was never run on them, and a DECLINE denotes insufficient evidentiary support under the protocol, not uncorrectability. Figure 3 , Appendix J and Appendix K give each setting’s full control battery. † Secondary audit records report only the comparison with random directions.

Study Property
Property Level	Operational Criterion
ADJUDICABLE	Action space $\|\mathcal{Y}\|\geq 3$ , with defined gold labels, baseline errors, and an exposed reference partition.
Representation Properties
READABLE	Error channel $c$ is linearly separable from reference states at observation layer $\ell_{\mathrm{obs}}$ .
STEERABLE	Directional intervention outperforms zero-magnitude, matched-random, reversed and layer-transposed controls, with cross-channel substitution added where the protocol records it.
Intervention Properties (Necessary for Repair)

Appendix

Table 6: The prospectively frozen SAKIKO evidence hierarchy. A formal repair claim is not a standalone property, but the strict conjunction

\textsc{REPAIR}(\pi)\equiv\textsc{CORRECTABLE}(\pi)\wedge\textsc{PRESERVING}_{\text{pop}}(\pi)\wedge\textsc{LICENSABLE}(\pi)

, whose preservation term is the population-level form.

#	Condition	Quantity	Population	Threshold	Layer
1	channel support	$N_{c}$	evaluation	$\geq 30$	integrity
9	zero arm exactness	all endpoints	routed	exact match	integrity
10	no structural failure	run integrity	—	none	integrity
4	specificity	add-one $p$ over $K=59$	$N_{c}$	$\leq 0.05$	specificity
2	target gain, point	$\mathrm{TG}_{c}$	$N_{c}$	$>0$	destination, point
5	target-hit, point	$\mathrm{TH}_{c}$	$X_{c}$	$>0.50$	destination, point

Appendix

Table 7: The canonical definition of a SAKIKO licence. Conditions are numbered as in the frozen artifacts and grouped here by what they check. An ADMIT requires all ten; the rule recorded in the freeze is 10/10; 9/10 = DECLINE . Destination intervals are bootstrap intervals over channel errors with

10{,}000

draws; the collateral interval is the one-sided upper bound from the same draws. Apart from the zero boundary in Conditions 2 and 3,

0.50

is the only destination threshold the gate uses.

Figure 7: What the licence adds over simpler reporting rules. (a) Every evaluated unit against eight alternative reporting rules, applied retrospectively; the sequence is not nested, since R4 drops specificity rather than adding a requirement, and passing a rule here is a diagnostic reading rather than a formal verdict. Units span both protocol generations and include one development-stage setting ( Gemma ca

\rightarrow

direct DEV ), which is retained here because it is the clearest case of a vacuous preservation bound: zero exposed correct decisions. (b) How many units each rule accepts. A rule based on aggregate gain alone accepts all nine; the full frozen conjunction accepts one. R4 rises because it drops the specificity requirement rather than because the evidence improves.

Benchmark	Actions	Gold	Dest.	Interv.	Status
When2Call	4 (3 gold)	yes	yes	yes	primary; all formal outcomes
MetaTool	2	yes	no	yes	historical; breadth of effect
ACEBench	4	yes	yes	no	framework transfer; support NO-GO

Appendix

Table 8: Benchmark roles. Only When2Call carries formal licence outcomes. Gold is whether gold labels are recoverable, Dest. whether the destination question is posable, Interv. whether an intervention was run. A dataset entering the table is not thereby evidence of breadth; the status column records what each one actually supports.

Model	Protocol	$\ell_{\mathrm{obs}}$	$\ell_{\mathrm{inj}}$	Estimator	Dose $q$	Channels
Phi-3.5-mini	historical	18	14 / 16 ‡	DiffMean / PCA-1	$\alpha=10.0$	3
Qwen2.5-7B	historical	20	16	DiffMean / PCA-1	N/V	3
Llama-3.1-8B	historical	N/V	N/V	N/V	N/V	N/V
Mistral-7B-v0.3	historical	N/V	N/V	N/V	N/V	N/V
Qwen3-8B	sealed	26	21	$d_{\mathrm{grad}}$	1.0	1
Qwen3-4B	sealed	26	21	$d_{\mathrm{grad}}$	0.25	1

Appendix

Table 9: Evaluated settings and their frozen configuration. The horizontal rule separates the two protocol generations, which are never pooled. Entries marked N/V have no retained artifact to verify them against: the historical settings predate the configuration-freeze policy that governs the sealed rows, and we leave them unstated rather than quote an unverified value.

Baseline state	Final destination	Count	Population
channel error	source retained (routed)	20	routed, 72
channel error	GOLD ARRIVAL	38	routed, 72
channel error	OTHER WRONG	14	routed, 72
channel error	unrouted, returned unchanged	15	$N_{c}=87$
baseline correct	retained correct	6 exposed, 0 broken	$C_{\mathrm{exp}}=6$
baseline correct	not exposed	205	$\|C\|=211$

Appendix

Table 10: Qwen3-8B formal run, every decision in the two adjudicated cohorts resolved; the remaining evaluation rows are neither channel errors nor baseline-correct. Of

87

channel errors,

72

were routed; the

15

unrouted errors are returned unchanged by construction. Note the two Net conventions: whole counts every fixed decision in the evaluation population, channel counts arrivals in the adjudicated channel net of breaks, which is why the score-space comparator’s

37

arrivals and

2

breaks appear as

+35

in Table 13 . The paper quotes both and never substitutes one for the other.

#	Condition	Layer	Threshold	Qwen3-8B	Qwen3-4B
1	channel support	integrity	$\geq 30$	pass	pass
9	zero arm exactness	integrity	exact	pass	pass
10	no structural failure	integrity	—	pass	pass
4	specificity, $K=59$	specificity	$p\leq 0.05$	pass	pass
2	target gain, point	destination	$>0$	pass	pass
5	target-hit, point	destination	$>0.50$	pass	pass

Appendix

Table 11: The two sealed Qwen3 settings against the frozen conjunction. Both clear every point estimate and both satisfy the frozen preservation conditions, which are written on

\mathrm{E1}

; on

\mathrm{E2}

the point estimates are below the bound but the interval evidence is not (

0.3930

and

0.0914

). The verdicts diverge on the two destination-interval conditions and on nothing else. Aggregate gain does not separate them:

+40

against

+39

Figure 8: The historical record behind Sections 4 and 5 . (a) E1 collateral for each of the five artifact-backed Phi-3.5 seeds against the

5\%

bound, with each run’s Net gain: breakage ranges from

16

52

264

baseline-correct decisions while the Net gain stays between

+53

and

+67

, and every realisation exceeds the bound. (b) The Qwen2.5-7B locked battery: the real direction (

+79

) against its reversal (

+16

) and the mean of ten budget-matched random directions (

+25.2

, the largest reaching

+50

). Both panels are historical protocol and both survive as aggregate counts except where noted; they carry no formal verdict.

Arm	Budget / site	Population	Alternative removed
baseline	none	all 548 rows	— defines both populations
zero	none	routed	that endpoints are computed on a mis-specified pipeline; must reproduce the baseline exactly before any endpoint is read
real $d_{\mathrm{grad}}$	$q_{c}s_{c}$ at $\ell_{\mathrm{inj}}$	routed	— primary arm
$K=59$ fresh randoms	$q_{c}s_{c}$ at $\ell_{\mathrm{inj}}$	routed	that any perturbation of this magnitude would move the decision; this is the primary null
reverse $-d_{\mathrm{grad}}$	$q_{c}s_{c}$ at $\ell_{\mathrm{inj}}$	routed	that the axis matters but its sign does not
wrong layer	same $q_{c}s_{c}$ at $\ell_{\mathrm{obs}}$	routed	that the direction works anywhere in the stack; the budget is not renormalised to the other layer’s activation norm, which would change site and budget together

Appendix

Table 12: Control arms in the sealed protocol, transcribed from the frozen control specification. The alternative removed column states what each arm exists to rule out. Budget and gate are identical to the real arm unless noted. Qwen3-8B values are quoted; Gemma-2-9B uses the same inventory at

\ell_{\mathrm{inj}}=24

with wrong-layer at

\ell=30

Quantity	Activation ( $d_{\mathrm{grad}}$ )	Score-space
gold arrivals $A_{c}$	38	37
other wrong $O_{c}$	14	21
source exits $X_{c}$	52	58
target-hit $\mathrm{TH}_{c}$	0.7308	0.6379
target gain $\mathrm{TG}_{c}$	0.2759	0.1839
Fixed / Broke (whole)	40 / 0	41 / 2

Appendix

Table 13: Activation intervention against the score-space comparator on the same channel and population. The aggregates are close; the destinations are not. The paired tests are not significant after multiplicity correction and the comparison was not prespecified, so this contrast is descriptive : it establishes neither that activation intervention is better nor that the two differ in general.

true target-hit	0.73	0.68	0.63	0.60	0.58	0.55
source exits needed	30	49	93	158	245	620

Appendix

Table 14: Source exits required to certify a true target-hit above

0.50

. This approximates the gate’s bootstrap criterion rather than replacing it, and we apply it only to the settings in this paper.

Quantity	Superseded	Authoritative	Reason
Qwen3-4B target-hit CI lower	0.4561	0.4559	index carried an independent recomputation; the frozen verdict artifact is authoritative
Qwen3-4B target-hit CI upper	0.6984	0.6970	same
Gemma target-hit CI lower	0.4400	0.4444	moved by $0.0044$ , beyond the Monte-Carlo noise the captions had allowed
Phi collateral denominator	$52/264$ as the rate	$52/93$ as the estimand	$264$ arises from using the channel-assignment field as the firing decision; the firing rule is route $\neq$ none , giving $293=200+93$
Qwen3-4B, Gemma rung label	“admitted at Level-1”	FORMAL DECLINE	falsified by the primary verdict artifacts
“Level-1 / Level-2”	used as licence tiers	post-hoc descriptive only	no prospectively frozen definition exists in any artifact

Appendix

Table 15: Historical values that later audits corrected. Each correction makes a derived index agree with the frozen artifact it cites; no frozen artifact was modified and no verdict changed.

Result	Artifact	Row-level
Qwen3-8B formal	final/results/qwen3_stage2_formal/	yes
Qwen3-4B formal	final/results/qwen3_4b_w2c_formal_v1/	yes
Gemma-2-9B formal	.../gemma/formal_freeze/	yes
Phi-3.5 multi-seed	final/results/clean/p1_multiseed_table.csv	seed 42 only
Mistral-7B	final/results/mistral7b_w2c_sakiko_ca/	no
Llama-3.1-8B	final/results/phase8_prospective_llama/	no

Appendix

Table 16: Where each result in the paper is recorded. Paths are relative to the repository root. Row-level records exist only where indicated; the remaining historical settings survive as aggregate counts and cannot be re-resolved by destination.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 639 papers

Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence
Oct 15, 2026Arijit Dey, Bhartendu Kumar, Ramya Hebbalaguppe +1Confidence Estimation in Language Models Language Model Calibration
MBZUAI · TCS Research Labs, India · IIT Delhi
Structured Feedback Improves Repair in an LLM Agent Loop
Jul 15, 2026Jaideep Ray, Ankit GoyalLLM Agent Workflow Optimization LLM Agent Evaluation
Independent Researcher
Diagnosis Is Not Prescription: Linguistic Co-Adaptation Explains Patching Hazards in LLM Pipelines
May 21, 2026Yoon Jeonghun, Kim DongchanLLM Agents Causal Interventions in Language Models
KAIST (Korea Advanced Institute of Science and Technology), Republic of Korea · NAVER Corp., Seongnam, Republic of Korea · NAVER Corp., Bellevue, WA, USA

Oct 15, 2026J/K move · Enter open · S save

Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence

Arijit Dey, Bhartendu Kumar, Ramya Hebbalaguppe, Khuushi Maheshwari

MBZUAI · TCS Research Labs, India · IIT Delhi

Confidence Estimation in Language Models Language Model Calibration Confidence Estimation Probability Calibration LLM Auditing Isotonic Regression

Open paper AskPDF arXiv

Confidence scores determine whether an LLM answer is accepted, reviewed, or regenerated, yet scalar recalibration can only correct errors visible through the score itself. We introduce RADAR (Repair, Adopt, and Diagnose After Recalibration), a reliability framework that separates confidence repair from deciding whether to deploy the repair and auditing the errors that remain. Under squared loss, we decompose prediction risk into outcome noise, information hidden by the score, and mapping error; restricting repair to order-preserving maps additionally introduces a pooling gap. For repair, we propose Probe-Anchored Isotonic Calibration (PAIC), which learns a monotone map from correctness-probe predictions on unlabeled target prompts. PAIC's excess risk depends on the probe's score-conditional bias and fitting error rather than its overall accuracy. For adoption, we derive a paired procedure with finite-sample control of false improvements. For auditing, we develop a prompt-clustered kernel test that detects feature-aligned residual errors while accounting for dependent answers from the same prompt. Across multiple LLM checkpoints, benchmarks, and distribution shifts, richer predictors can improve confidence estimation, but projecting their information back to a scalar score can lose these gains, and lower prediction risk need not yield better review decisions. Across 47,400 replayed raw-score allocations, paired betting adopts candidates with lower held-out test risk in 42.95% of allocations, versus 15.09% for one-sided empirical Bernstein. RADAR thus provides a principled workflow to repair what a scalar score can express, adopt a repair only when supported by evidence, and diagnose systematic errors that recalibration leaves behind. Code and data will be released upon acceptance.

Abstract

LLM Auditing

Explore similar work

Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence

Structured Feedback Improves Repair in an LLM Agent Loop

Diagnosis Is Not Prescription: Linguistic Co-Adaptation Explains Patching Hazards in LLM Pipelines