Beyond Action Entropy: Quotient-Space Exploration for Genome-Scale Metabolic Model Repair

Xuan Gong; Hanbo Huang; Wenbin Dai; Jing Wang; Lei Bai; Xiang Xiao; Weishu Zhao; Shiyu Liang

cs.LGOct 8, 2026

Beyond Action Entropy: Quotient-Space Exploration for Genome-Scale Metabolic Model Repair

Authors: Xuan Gong, Hanbo Huang, Wenbin Dai, Jing Wang, Lei Bai, Xiang Xiao, Weishu Zhao, Shiyu Liang

Organizations: Shanghai Jiao Tong University · Shanghai AI Laboratory

Abstract

Repairing scientific models from functional observations differs fundamentally from supervised prediction: feedback may certify a solution without revealing which structural correction is responsible. We study this setting for genome-scale metabolic model (GEM) repair, where multiple reaction edits can explain the same phenotypes and many apparently distinct edits correspond to the same biological mechanism. This many-to-one structure creates a hidden failure mode for conventional exploration: diversity in the output space need not translate into diversity of scientific hypotheses. We introduce QuotientPO, which collapses equivalent repairs into canonical mechanisms and optimizes exploration directly over the resulting quotient space. To make quotient exploration informative under finite rollouts, we derive a kernelized Rényi estimator that resolves graded crowding among distinct repair cores beyond coarse exact-match counts. On 2,212 held-out GEMs, QuotientPO improves Success@32 from 17.93% to 20.10% (+12.1% relative) while consistently increasing distinct successful-core discovery under the same sampling budget. These results establish quotient-space exploration as a principled approach to mechanism-level discovery under verifier-induced equivalence.

Figures & tables

Figure 1: Overview. QuotientPO canonicalizes equivalent surface edits to explore distinct mechanistic hypotheses rather than redundant syntactic variants for GEM repair.

Figure 2: Theoretical intuition. (a) Each core

m

contains

n_{m}

equivalent raw edits contracted by

\kappa

into a single point, preventing large

n_{m}

from distorting exploration. (b) Bar heights show exploration credit. Exact matching gives identical credit to equal-count cores, whereas kernel similarity favors isolated discoveries (

m_{1}

) over clustered candidates (

m_{2}

–

m_{4}

Method	Greedy	Success@1	Success@8	Success@32	Improved (mean@32)	Success Cores@32
MILP (Oracle)	15.01	-	-	-	25.63	0.15
SFT-Hard	1.72 ${}_{\pm\text{0.16}}$	0.51 ${}_{\pm\text{0.03}}$	1.96 ${}_{\pm\text{0.11}}$	4.49 ${}_{\pm\text{0.05}}$	1.64 ${}_{\pm\text{0.03}}$	0.09 ${}_{\pm\text{0.01}}$
SFT-Soft	2.65 ${}_{\pm\text{0.35}}$	2.40 ${}_{\pm\text{0.05}}$	5.27 ${}_{\pm\text{0.41}}$	7.52 ${}_{\pm\text{0.49}}$	9.92 ${}_{\pm\text{0.18}}$	0.16 ${}_{\pm\text{0.01}}$
Vanilla GRPO	7.97 ${}_{\pm\text{0.29}}$	7.43 ${}_{\pm\text{0.03}}$	13.79 ${}_{\pm\text{1.09}}$	17.93 ${}_{\pm\text{1.54}}$	30.21 ${}_{\pm\text{0.59}}$	0.50 ${}_{\pm\text{0.01}}$
MaxEnt	7.29 ${}_{\pm\text{1.16}}$	5.97 ${}_{\pm\text{0.86}}$	12.69 ${}_{\pm\text{1.27}}$	17.12 ${}_{\pm\text{1.41}}$	29.35 ${}_{\pm\text{2.03}}$	0.48 ${}_{\pm\text{0.05}}$
GFlowNet	3.20 ${}_{\pm\text{0.23}}$	3.23 ${}_{\pm\text{0.15}}$	8.35 ${}_{\pm\text{0.38}}$	13.10 ${}_{\pm\text{0.17}}$	14.91 ${}_{\pm\text{0.49}}$	0.34 ${}_{\pm\text{0.01}}$

Table 1: Main results. MILP is an oracle with test-time phenotype access. SFT methods use checkpoints selected on the validation split, while RL methods are evaluated at the final checkpoint after an identical 864-update training budget. All results are evaluated on the test set, reporting mean

\pm

standard deviation across three seed runs.

Panel of Figure 4 — Figure 4: Test-time scaling and training dynamics. (a) Repair success (Success@ $k$ ) and distinct successful-core discovery (Success cores@ $k$ ). (b) Validation curves across different RL methods.

Figure 5: Landscape of sampled repairs ( Streptomyces axinellae ).

Panel of Figure 6 — Figure 6: Exploration across raw and quotient spaces. (a) MDS projections of sampled repairs before (top) and after (bottom) canonicalization on ( S. griseoflavus ). (b) Funnel transitions from distinct raw repairs ( $k=32$ ) to verifier-validated cores across the test set.

	Training (200→864)		Successful Cores
Method	Exact ties (%)	Resolved (%)	Collision $\downarrow$	Mean $d_{J}$ $\uparrow$
Vanilla	8.19 $\rightarrow$ 8.59	60.55 $\rightarrow$ 84.86	0.5769	0.2408
Shannon	8.60 $\rightarrow$ 9.72	59.23 $\rightarrow$ 80.20	0.5687	0.2431
Exact R2	7.34 $\rightarrow$ 10.23	59.71 $\rightarrow$ 78.20	0.5285	0.2861
Kernel R2	8.57 $\rightarrow$ 7.99	63.81 $\rightarrow$ 72.78	0.5176	0.3008

Table 2: Finite-sample credit resolution and successful-core diversity. Training diagnostics use all rollouts.

Figure 7: Building an exploration-ready policy. (a) Pretraining and ESM-2 enhance the local repair prior. (b) Set-level SFT unlocks rollout reward variance for RL. Error bars denote standard deviation across 8 rollout batches.

Param.	Setting	S@32
–	Default	20.10
$h(0.25)$	0.125	19.41
$h(0.25)$	0.5	18.57
$\beta(0.02)$	0.01	19.05
$\beta(0.02)$	0.04	20.00

Table 3: Robustness and ablation studies. S@32 denotes Success@32 (%).

Panel of Figure 8 — Figure 8: Case studies of mechanistic repair.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Assay category	BacDive families
Carbon source	glucose, maltose, sucrose, glycerol, acetate, lactate, pyruvate, succinate, citrate, glutamate
Nitrogen source	ammonium, urea, nitrate, nitrite
Sulfur source	sulfate
Phosphorus source	phosphate

Appendix

Table 4: Nutrient-growth assay coverage.

Figure 9: Dataset characterization and split diagnostics.

Figure 10: Benchmark validation.

Anchor $\gamma$	Best	Best step	Last	Last ADD
$0$ (no anchor)	7.91	15	5.80	7367.57
$0.4$	8.46	745	7.64	23.05
$\mathbf{0.6}$	$\mathbf{9.38}$	864	$\mathbf{9.38}$	10.98
$1.0$	8.00	715	7.27	6.05

Appendix

Table 5: SFT-anchor selection for the GFlowNet baseline. Anchored variants use the same final training recipe and complete all 864 updates. The unanchored run is incomplete and is included only to show object-space cardinality degeneration. Success values are validation Success@8 (%).

Figure 12: Scaling behavior of GEM pretraining.

Configuration	Hit@1 $\uparrow$	dev-select loss $\downarrow$
Pretrained + ESM2	0.207	3.248
Random + ESM2	0.114	4.524
Pretrained, no ESM2	0.143	3.922

Appendix

Table 6: Direct SFT-A development results. Hit rates are operation-macro, model-equal retrieval scores on certified partial labels (seed 42).

Parameter	Setting
Actor initialization	Soft-SFT
Total / trainable parameters	92.85M / 59.46M
Training / validation GEMs	9,196 / 1,087
RL seeds	42, 43, 44
Training devices	8 GPU ranks
GEMs per update	32

Appendix

Table 7: Shared RL configuration.

Method	Update (s)	FBA drain (s)	Overhead
Vanilla GRPO	21.323	2.229	–
QuotientPO	23.218	4.186	+8.9%

Appendix

Table 8: Per-update runtime under identical hardware and rollout settings.

Template	Method	Success@1	Success@8	Success@32	Greedy	Improved	Success Cores@32
Default ( $N=697$ )
	Vanilla GRPO	7.37 $\pm$ 0.58	12.72 $\pm$ 0.79	16.79 $\pm$ 1.08	7.84 $\pm$ 0.36	28.83 $\pm$ 1.08	0.516 $\pm$ 0.003
	MaxEnt	5.69 $\pm$ 0.96	11.81 $\pm$ 1.58	15.93 $\pm$ 1.28	7.32 $\pm$ 1.52	28.04 $\pm$ 2.33	0.462 $\pm$ 0.088
	GFlowNet	1.77 $\pm$ 0.22	6.07 $\pm$ 0.54	10.38 $\pm$ 0.33	2.54 $\pm$ 0.30	11.48 $\pm$ 0.70	0.265 $\pm$ 0.017
	QuotientPO	7.89 $\pm$ 0.50	14.25 $\pm$ 0.22	18.75 $\pm$ 0.82	7.80 $\pm$ 0.33	30.01 $\pm$ 1.19	0.582 $\pm$ 0.006
Gram-positive ( $N=491$ )

Appendix

Table 9: Performance across GEM reconstruction templates. Results are mean

\pm

standard deviation across three seeds.

$r$	Paired GEMs (42/43/44)	Vanilla GRPO	QuotientPO
2	295/269/323	$1.425\pm 0.019$	$\mathbf{1.461\pm 0.020}$
4	247/228/258	$1.960\pm 0.037$	$\mathbf{2.050\pm 0.056}$
8	209/186/206	$2.659\pm 0.076$	$\mathbf{2.782\pm 0.114}$
16	147/123/130	$3.256\pm 0.033$	$\mathbf{3.502\pm 0.203}$

Appendix

Table 10: Success-controlled mechanism coverage. Expected distinct canonical cores (

D_{r}

) evaluated across

r

successful draws on paired cohorts.

	Exact-credit ties (%)				Kernel-resolved ties (%)
Method	0	200	400	864	0	200	400	864
Vanilla GRPO	5.67	8.19	8.31	8.59	30.61	60.55	57.50	84.86
Discrete Shannon	5.67	8.60	9.99	9.72	30.61	59.23	59.77	80.20
Exact Rényi-2	5.67	7.34	9.06	10.23	30.61	59.71	63.71	78.20
Kernel Rényi-2	5.67	8.57	8.60	7.99	30.61	63.81	57.23	72.78

Appendix

Table 11: Full training trajectory of finite-sample credit resolution. Exact-count tie rates measure distinct-core pairs indistinguishable under multiplicity credit within rollout groups; kernel-resolved rates report the fraction of these ties separated by graded kernel credit.

Method	Success@32 $\uparrow$	Success Cores@32 $\uparrow$
Vanilla GRPO	17.93 $\pm$ 1.54	0.50 $\pm$ 0.01
Discrete Shannon	18.61 $\pm$ 1.09	0.52 $\pm$ 0.01
Exact Rényi-2	18.04 $\pm$ 1.08	0.54 $\pm$ 0.03
Kernel Rényi-2	20.10 $\pm$ 0.52	0.57 $\pm$ 0.02

Appendix

Table 12: Downstream performance of quotient exploration objectives.

Method	Success@32
Vanilla	15.75 ${}_{\pm\text{1.20}}$
MaxEnt	14.29 ${}_{\pm\text{0.95}}$
GFlowNet	11.57 ${}_{\pm\text{0.14}}$
QuotientPO	17.21 ${}_{\pm\text{0.81}}$

Appendix

Table 13: Multi-seed robustness and ablation results. Success@32 (%) are reported as mean

\pm

standard deviation over three seeds.

Figure 13: Cases of SFT versus RL Optimization.

Figure 14: Cases of solution-space visualization across RL baselines.

Repair alternatives & encoded chemistry	Functional verification & policy diagnosis
Case 1 (C1): Mycobacterium terrae Target: Urea utilization Focal pipeline ( $R\to S\to C$ ): $V$ : $13\to 2\to 0$ vs $Q$ : $13\to 2\to 1$ Yield ( $N_{\mathrm{succ}}\mid K_{\mathrm{core}}$ ): $V$ : $4\mid 1$ $Q$ : $4\mid 2$
Shared deletions (3): -HEX1 , -MALTabc , -NARK Pathway 1: Uptake blockade -UREAt $\mathrm{urea}_{e}\longrightarrow\mathrm{urea}_{c}$ Pathway 2: Carboxylation blockade (focal) -UREASE $\displaystyle\mathrm{urea}+\mathrm{ATP}+\mathrm{HCO}_{3}^{-}$ $\displaystyle\longrightarrow\mathrm{allophanate}$ $\displaystyle+\mathrm{ADP}+\mathrm{P}_{\mathrm{i}}+\mathrm{H}^{+}$ Note: UREASE encodes urea carboxylase in the GEM. Both canonical cores contain exactly four deletions.	Phenotypic verification Both cores reduce urea-supported growth from $0.202$ to zero while preserving nitrate-supported growth of $0.545$ and all remaining targets. Reintroducing the focal deletion restores the failed urea-growth phenotype, confirming its functional necessity. Search behavior diagnosis Vanilla produces successful raw proposals containing this intracellular branch ( $S_{V}=2$ ), but completely eliminates it during canonical reduction ( $C_{V}=0$ ). In contrast, QuotientPO successfully retains both functional intervention sites ( $C_{Q}=1$ ).
Case 2 (C2): Streptomyces inusitatus Target: Alternative proline supply Focal pipeline ( $R\to S\to C$ ): $V$ : $32\to 32\to 0$ vs $Q$ : $32\to 30\to 2$ Yield ( $N_{\mathrm{succ}}\mid K_{\mathrm{core}}$ ): $V$ : $32\mid 2$ $Q$ : $30\mid 2$
Shared deletion (1): -UREAt Pathway 1: Direct conversion +ORNCD $\mathrm{ornithine}\longrightarrow\mathrm{proline}+\mathrm{NH}_{4}^{+}$ Pathway 2: Semialdehyde/P5C route (focal) +G5SADs $\displaystyle\mathrm{GSA}$ $\displaystyle\longrightarrow\mathrm{P5C}+\mathrm{H}_{2}\mathrm{O}+\mathrm{H}^{+}$ $\displaystyle\mathrm{P5C}$ $\displaystyle\xrightarrow{\text{existing P5CR/P5CRx}}\mathrm{proline}$ Note: GSA denotes glutamate-5-semialdehyde; P5C denotes $\Delta^{1}$ -pyrroline-5-carboxylate. Both canonical cores contain two edits.	Phenotypic verification Both two-edit cores restore proline supply and recover citrate-supported growth from zero to $0.831$ while preserving the inactive urea target. Flux witnesses independently confirm active throughput along both pathways. Search behavior diagnosis Vanilla already contains the focal G5SADs route in successful proposals but loses it after canonicalization ( $C_{V}=0$ ); QuotientPO retains it ( $C_{Q}=2$ ). Thus, this is a retention rather than discovery gap. ORNCD and ORNCD $\_$ 2 are treated as one functional pathway family.

Appendix

Table 14: Tier 1: retained-solution coverage. The focal alternative appears in successful Vanilla raw proposals (

S_{V}>0

) but is retained only by QuotientPO after canonical reduction (

C_{V}=0

C_{Q}>0

Repair alternatives & encoded chemistry	Functional verification & policy diagnosis
Case 3 (C3): Lactobacillus farraginis Target: Alternative phosphate donors for UTP synthesis Focal pipeline ( $R\to S\to C$ ): $V$ : $4\to 0\to 0$ vs $Q$ : $10\to 1\to 1$ Yield ( $N_{\mathrm{succ}}\mid K_{\mathrm{core}}$ ): $V$ : $0\mid 0$ $Q$ : $2\mid 2$
Shared edits (3): +MOHMT , +NTRIR2y , -NTRIR3pp Pathway 1: ATP donor +NDPK2 $\mathrm{ATP}+\mathrm{UDP}\longrightarrow\mathrm{ADP}+\mathrm{UTP}$ Pathway 2: PEP donor (focal) +PYK2 $\displaystyle\mathrm{PEP}+\mathrm{UDP}+\mathrm{H}^{+}$ $\displaystyle\longrightarrow\mathrm{pyruvate}+\mathrm{UTP}$ Both complete cores contain four edits.	Phenotypic verification The alternatives provide UDP-to-UTP phosphorylation using different phosphate donors. Both QuotientPO cores satisfy all six targets, with glucose growth $0.540171$ and $0.539515$ , respectively, and nitrate growth $0.888279$ . Flux witnesses carry approximately $0.16655$ through NDPK2 or $0.16635$ through PYK2 . Search behavior diagnosis Vanilla proposes PYK2 four times, but these proposals lack the coordinated repair required for success. Adding NTRIR2y restores the nitrate-positive target.
Case 4 (C4): Streptomyces ferralitis Target: Alternative nitrogen donors for asparagine Focal pipeline ( $R\to S\to C$ ): $V$ : $24\to 0\to 0$ vs $Q$ : $29\to 3\to 1$ Yield ( $N_{\mathrm{succ}}\mid K_{\mathrm{core}}$ ): $V$ : $0\mid 0$ $Q$ : $3\mid 3$
Shared edits (3): +ARGDC , -FFSD , -SUCRtex Pathway 1: Glutamine donor +ASNS1 $\displaystyle\mathrm{Asp}+\mathrm{Gln}+\mathrm{ATP}+\mathrm{H}_{2}\mathrm{O}$ $\displaystyle\longrightarrow\mathrm{Asn}+\mathrm{Glu}$ $\displaystyle+\mathrm{AMP}+\mathrm{PP}_{\mathrm{i}}+\mathrm{H}^{+}$ Pathway 2: Ammonium donor (focal) +ASNS2 $\displaystyle\mathrm{Asp}+\mathrm{NH}_{4}^{+}+\mathrm{ATP}$ $\displaystyle\longrightarrow\mathrm{Asn}+\mathrm{AMP}$ $\displaystyle+\mathrm{PP}_{\mathrm{i}}+\mathrm{H}^{+}$	Phenotypic verification The two reactions complete asparagine synthesis using different nitrogen donors. Both QuotientPO cores satisfy the glucose-positive and sucrose-negative targets, with glucose growth $1.342551$ and $1.345126$ . Search behavior diagnosis Vanilla frequently proposes the focal ASNS2 branch, but the corresponding repair sets remain unsuccessful. One such proposal also introduces extracellular sucrose hydrolysis through SUCRe ; removing this additional bypass restores the sucrose-negative target. Thus, successful use of the focal biosynthetic branch depends on its surrounding repair combination.

Appendix

Table 15: Tier 2: successful-combination coverage. Vanilla proposes the focal module (

R_{V}>0

) but does not place it in a successful complete repair (

S_{V}=0

), whereas QuotientPO does.

Repair alternatives & encoded chemistry	Functional verification & policy diagnosis
Case 5 (C5): Streptomyces olivochromogenes Target: Alternative control of citrate-supported growth Focal pipeline ( $R\to S\to C$ ): $V$ : $0\to 0\to 0$ vs $Q$ : $2\to 2\to 2$ Yield ( $N_{\mathrm{succ}}\mid K_{\mathrm{core}}$ ): $V$ : $2\mid 1$ $Q$ : $8\mid 3$
Shared edit (1): -UREAt Alternative 1: Entry restriction -CITt14 , -FE3t CITt14 directly supports citrate uptake, while FE3t provides a reversible iron route coupled to the remaining FEDCabc citrate-entry pathway. Alternative 2: Intracellular node (focal) -SUCOAS $\mathrm{succinyl\text{-}CoA}+\mathrm{ADP}+\mathrm{P_{i}}\longrightarrow\mathrm{succinate}+\mathrm{CoA}+\mathrm{ATP}.$	Phenotypic verification Both cores eliminate the initial urea and citrate false positives while preserving glucose- and sucrose-supported growth. The entry-restriction core retains glucose/sucrose growth at $0.630548/1.261097$ , whereas the SUCOAS core retains $0.436023/0.872046$ . Restoring either CITt14 or FE3t breaks only the citrate-negative target in the first core. Restoring SUCOAS does the same in the focal core. Restoring UREAt affects only the urea-negative target. Search behavior diagnosis Vanilla never proposes -SUCOAS in this rollout panel. QuotientPO discovers it twice. Both samples are successful and canonicalize to the same focal core. Thus, this comparison exposes an intracellular repair alternative absent from Vanilla raw proposals, rather than two distinct mechanisms.
Case 6 (C6): Streptomyces durbertensis Target: Alternative quinone-precursor supply Focal pipeline ( $R\to S\to C$ ): $V$ : $0\to 0\to 0$ vs $Q$ : $5\to 1\to 1$ Yield ( $N_{\mathrm{succ}}\mid K_{\mathrm{core}}$ ): $V$ : $1\mid 1$ $Q$ : $7\mid 5$
Shared edits (5): +HSERTA , +HSST , +THZPSN2_SC , -NO3abc , -NO3tex Pathway 1: Aromatic branch +PHPYROX $\displaystyle\mathrm{phenylpyruvate}\longrightarrow\mathrm{benzaldehyde}$ $\displaystyle\longrightarrow\mathrm{benzoate}\longrightarrow\mathrm{benzoyl\text{-}CoA}$ $\displaystyle\xrightarrow{\text{reverse BSCT}}\mathrm{OSB\text{-}CoA}$ Pathway 2: Three-ADD branch (focal) +SHCHCS , +SUCBZS , +SUCBZL_1 $\displaystyle\mathrm{isochorismate}+\mathrm{SSA}$ $\displaystyle\longrightarrow\mathrm{SHCHC}\longrightarrow\mathrm{OSB}$ $\displaystyle\longrightarrow\mathrm{OSB\text{-}CoA}$ Note: SSA denotes succinic semialdehyde and OSB denotes $o$ -succinylbenzoate.	Phenotypic verification Both branches converge on OSB-CoA and downstream menaquinone-precursor supply. The resulting cores satisfy all four targets and yield glucose growth of approximately $0.454946$ . Minimum-total-flux witnesses use the corresponding precursor branch at approximately $4.55\times 10^{-5}$ , matching the quinone demand induced by biomass production. Search behavior diagnosis The complete three-ADD module does not occur in Vanilla raw proposals, whereas QuotientPO proposes it five times and retains one successful canonical core. Its five total distinct cores therefore correspond to multiple edit sets, rather than five distinct quinone-supply mechanisms.

Appendix

Table 16: Tier 3: raw-module coverage. The complete focal module is absent from Vanilla raw proposals (

R_{V}=0

) but appears in successful QuotientPO repairs.

Explore similar work

Aug 13, 2026cs.AI

Capability Sheaves for Compositional Agent-Harness Repair: Controlled Quotients and a Real-Repository Stress Test

Agent harnesses combine retrieval, routing, state, provenance, and verification, but locally successful components may disagree on shared state. We model this failure with a finite \emph{capability sheaf}: stalks encode typed behavior signatures, restriction maps retain shared fields, and accepted runs are useful global sections. An exact finite constraint-satisfaction problem (CSP) defines acceptance, while a linearized relative cohomology class provides a diagnostic and search feature. A controlled experiment over 20 task clusters introduces hidden interior mediators whose raw states are nuisance variables. Quotienting their coboundaries reduces the candidate budget from 2,000 to 1,000 per cluster; aligning the hidden state removes the gap. Exact CSP matches the quotient, so the result demonstrates invariance to stale representatives, not superiority over exact reasoning. We then test the method on a discovery split from the SWE-bench Multilingual pool of PatchFuseBench: 160 issues from 20 repositories, 875 real candidate patches, 2,579 source-aware edit atoms, and 153 newly executed patches. A first pool-level construction is constant because

[b-Dx]=[b]

\operatorname{coker}D

and therefore cannot rank configurations. A candidate-indexed repair is nontrivial on 848/875 candidates and varies within 120/160 issues. It resolves 118 issues versus 116 for a matched noncohomological selector, but the difference is not supported across repositories (exact sign-flip

p=0.75

). A leave-one-repository-out abstention gate reaches 127/160, tying the strong anchor and exceeding its matched gate by one issue (

p=1.0

). The discovery gate therefore fails and the confirmatory split remains sealed. The study supports the controlled invariance mechanism and an identifiability correction, but not a real-world cohomological advantage.

Saveliy Batruin

Independent researcher

Automated Program Repair Sheaf Theory LLM Agent Harnesses

Ask PDF

Oct 1, 2026cs.LG

Are We Recovering Mechanisms? Objective-Level Recovery Gaps in Mechanistic Interpretability

Mechanistic interpretability aims to recover the internal computations responsible for model behavior. Progress in automated circuit discovery is often framed as a search problem: better attribution or optimization should identify better mechanisms. This assumes that the evaluation objective can recognize a better circuit once it is found. We show that intervention-defined faithfulness can instead prefer an equally sized circuit that reproduces the model's behavior less well, creating an objective-level recovery gap. Across four human-reference tasks and InterpBench, we compare validation faithfulness with behavior on held-out prompts under fixed ordinary resampling. The behavioral criterion is agreement with the intact model, including its mistakes, except on Greater-Than, where we use semantic accuracy. Controlled reference edits reveal misranking without any discovery algorithm, and outputs of EAP, EAP-IG, ACDC, and Edge-SP exhibit the same failure. Under resampling, KL misranks 9.4%-41.2% of candidate pairs across these methods on the human-reference tasks. We investigate context distortion as an explanation: replacing excluded signals changes the inputs on which retained components operate. Restoring selected signals from the recipient's intact-model execution repairs 96 of 100 persistent KL misrankings from the discovery pool on both validation and held-out prompts. The circuits and their original behavioral scores remain unchanged. These findings show why better discovery alone is insufficient when its objective rewards the wrong candidate.

Chuqin Geng, Li Zhang, Haolin Ye +3

University of Toronto · McGill University

Circuit Discovery Mechanistic Interpretability Explainability Evaluation

Ask PDF

Jun 30, 2026cs.SE

Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Models

In deployment settings where retraining is infeasible, small frozen code models are routinely asked to repair a failed program after seeing their own failing output, usually treated as a retry mechanism. From a Popperian view, a generated program is a conjecture and a test-execution violation is an oracle-relative, executable counterexample, so feedback's value should be attributed not to re-exposure to failing code but to whether the conjecture is opened to external, executable criticism. As the third stage of a falsification-centered measurement program, this study builds a placebo-controlled instrument that decomposes the feedback packet against a blind-resampling baseline at matched output-generation budget and against content-free, shape-matched placebos. The contribution is not a new repair algorithm but a reflexive methodology (packet decomposition, placebo mirroring, matched-budget discordant-pair tests, fresh-generation confirmation, executable audits) that makes both the model's program conjecture and the researcher's "feedback content works" claim falsifiable. Across six HumanEval+/MBPP+ cells with three 0.5B-1.5B frozen models, 290 dead task-cell units (no best-of-8 candidate passing the public tier) were evaluated; the main run produced 7,000 fresh generations and a preregistered follow-up 1,400 more. Blind resampling exceeded bare-code retry by +18 net unlocks (25/7, Holm p=0.0021). Code-plus-facts recovered +18 over bare code (21/3, p=0.00042) and +15 over a generic-bullet placebo (p=0.0041). An instruction-only effect was not distinguishable (+3, p=0.36). Code-plus-facts and blind resampling tied at 26 unlocks each (not equivalence). Six external-controller follow-ups tied a content-free shape placebo. In this regime, falsification helped not as vocabulary or self-critique, but as comparison with external, executable counterexamples.

Mehmet Iscan

PythaLab, Yıldız Technical University, Istanbul, Turkey

LLM Self-Correction Automated Program Repair Execution-Guided Code Generation Code Generation Evaluation

Ask PDF