From Outcomes to Components: A Counterfactual Evaluation Framework for LLM-Proposed Repairs

Hengyu Ye; Songhua Hu

cs.AIOct 19, 2026

From Outcomes to Components: A Counterfactual Evaluation Framework for LLM-Proposed Repairs

Authors: Hengyu Ye, Songhua Hu

Organizations: ShanghaiTech University · City University of Hong Kong

Abstract

An LLM-generated repair action specifies which records to change and by how much. A failed action may still contain a useful target set or numerical update, depending on how the two are paired. We evaluate this problem in traffic networks and databases, where aggregate observations obscure the records that need correction. We examine the pairings by executing each combination from the same initial state, using a reference action that restores the affected records. In 76 of 114 traffic actions, the gain from correcting both parts exceeds the sum of the gains from correcting each part alone. Six of these actions become worse after either correction alone, but improve when both parts are corrected. In database repair, replacing only the proposed update with the reference update worsens 17 of 108 model actions, all of which select unaffected records. We also test whether parts of two proposals can be reused. On held-out observations, selecting among the proposals and their recombined target-update pairs recovers 61.6% of the reference action's loss reduction, compared with 58.0% from selecting between the proposals alone.

Figures & tables

Figure 1: From a repair outcome to a diagnosis of its decisions. A submitted repair supplies selected records and a numerical update. Four executions combine proposed and reference components, changing the update horizontally and the targets vertically. The illustrated selection errors lead to different diagnoses: missed records can hide a useful update, while excess records can make a stronger correction harmful. Record diagrams are schematic. Reference components enter the evaluation after the model submits its action.

Analysis	Included observations
Traffic survey	77 episodes from 11 models and seven conditions. Five numerical-repair conditions yield 55 actions. Sensor bias and the healthy control contribute recognition outcomes.
Traffic components	140 demand episodes yield 117 demand-scaling actions. Three empty target sets leave 114 for component analysis. A sparse-target subset contains 54 actions and 28 distinct state–target combinations for search.
Database components	108 actions from three models, six faults, two evidence conditions, and three generations.
Traffic preview	72 directional evaluations from three models, four faults, three generations, and two panel directions. Averaging directions yields 36 paired results.

Table 1: Analysis sets and eligibility. Counts refer to actions unless stated otherwise.

Figure 2: Fault recognition coexists with widely varying recovery rates across 11 models. Each row shows one model across seven conditions. Filled and open circles denote correct and incorrect recognition on 77 interactive episodes. Adjacent colored tiles show recovery rates for all 55 numerical-repair episodes across recognition outcomes. Colors saturate outside $[-1.5,1]$ . Sensor bias and the healthy control show recognition only.

Figure 3: Component replacement and fixed-target search on sparse targets. Left: recovery rates across 54 actions. Horizontal bars and numbers above the groups show means. Right: recovery rates before and after search for 28 distinct state–target combinations. The before-search rate uses the first recorded action for each combination. Each search evaluates 28 multipliers while preserving the selected targets.

Model	Excess targets	Harm among all actions	Harm with excess targets	Harm counts by repetition
DeepSeek-Flash	21/36	15/36	15/21	7, 3, 5
K3	4/36	2/36	2/4	1, 0, 1
GLM-5.3	1/36	0/36	0/1	0, 0, 0

Table 2: Database target errors and update-replacement harm. Each model contributes 36 actions: six faults, two evidence conditions, and three repetitions. Harm means that replacing only the update decreases the recovery rate by more than 0.01. Repetitions share the same faults.

Action	DeepSeek-Flash	GLM-5.3
Agent action	0.427	$-1.973$
Replace targets	0.517	$-1.973$
Replace update	$-1.032$	1.000
Reference action	1.000	1.000

Table 3: Replacing the update can improve or worsen a repair. Recovery rates for two database repairs. DeepSeek-Flash selects a broad group (target F1 0.396), and GLM-5.3 selects the exact affected set. Each column compares four executions of one proposal. Appendix A.3.1 identifies the actions.

Candidate set	Candidates	Mean recovery rate $R$
Original record-level proposal	1	0.030
Original grouped proposal	1	0.506
Two original proposals	2	0.580
Originals + midpoint updates	4	0.605
Originals + crossed components	4	0.616

Table 4: Execution preview. Mean evaluation-panel recovery rate across 36 model–fault–repetition results, each averaged over both panel directions.

Figure 4: Component-reuse gains by model and fault. Three points per model–fault cell show gain over selection between the originals. Each point averages both panel directions. Crossing and midpoint use two proposals and four candidate executions.

Candidate	Targets	Multiplier	Selection loss	Held-out $R$
Record targets, record update	5	1.250	40.224	0.046
Grouped targets, grouped update	103	1.450	71.762	0.218
Record targets, grouped update	5	1.450	35.856	0.079
Grouped targets, record update	103	1.250	13.749	0.680

Table 5: Four combinations of two GLM-5.3 proposals. Three-origin demand deficit, repetition 2, selection panel A. Target counts are positive-demand OD pairs. The lowest selection loss determines the preview choice.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Target F1	Actions	$\Delta_{S}$	$\Delta_{u}$
Below 0.5	60	0.583	0.247
0.5 to below 0.9	18	0.449	0.224
0.9 and above	36	0.000	0.063

Appendix

Table A1: Component effects by target accuracy across 114 demand actions.

Model	Eligible actions	Agent $R$	$\Delta_{S}$	$\Delta_{u}$
K3	23	0.664	0.274	0.062
DeepSeek-Flash	23	0.345	0.487	0.168
GLM-5.3	23	0.575	0.229	0.196
GLM-Flash	23	0.465	0.378	0.157
MiMo-Pro	22	0.122	0.527	0.351

Appendix

Table A2: Component effects by model in the difficulty study. Effects are mean marginal changes in recovery rate under the full two-factor comparison.

Execution	Mean $R$	Median $R$	Max $R$	$R\geq 0.9$
Agent action	0.113	0.117	0.291	0/54
Replace update	0.206	0.176	0.432	0/54
Fixed-target search	0.437	0.450	0.696	0/54
Replace targets	0.794	0.867	1.000	22/54
Reference action	1.000	1.000	1.000	54/54

Appendix

Table A3: Component replacements and fixed-target search. Each row specifies which components of the agent action are replaced. The search row holds agent-selected targets fixed and searches the legal multiplier range.

Figure A1: Recognition, target accuracy, and recovery rate form different difficulty profiles. Each block contains seven conditions varying fault magnitude, the number and arrangement of affected OD pairs, or the share of affected sensors. Panels show (a) recognition, with blue cells denoting correct recognition and crosses marking incorrect recognition, (b) target F1, and (c) recovery rate with colors saturated outside $[-1.5,1]$ and minus signs for harmful actions. The gray block in (c) denotes sensor repair, evaluated by target accuracy. Condition settings are listed in Appendix A.4 . The figure shows 210 fault episodes. Five additional healthy controls bring the study total to 215.

Fault	Affected intersection	Injected change
Amount $+27\%$	Shipping mode and return status	Multiply amount by 1.270
Rate $+0.023$	Shipping mode and return status	Add 0.023 to adjustment rate
Amount $-21\%$	Handling instruction and return status	Multiply amount by 0.790
Rate $-0.018$	Handling instruction and return status	Subtract 0.018 from adjustment rate
Amount $+34\%$	Shipping mode and handling instruction	Multiply amount by 1.340
Rate $+0.017$	Shipping mode and handling instruction	Add 0.017 to adjustment rate

Appendix

Table A4: Database fault definitions. Each intersection selects one category of each named attribute. Amount changes are multiplicative, and adjustment-rate changes are additive.

Model	Round	Excess	Harm/all	Harm/excess	Mean $\Delta R$	Median $\Delta R$
DeepSeek-Flash	1	8/12	7/12	7/8	-0.570	-0.042
	2	6/12	3/12	3/6	0.137	0.001
	3	7/12	5/12	5/7	7.192	0.000
K3	1	1/12	1/12	1/1	-0.112	0.000
	2	2/12	0/12	0/2	0.449	0.000
	3	1/12	1/12	1/1	0.034	0.000

Appendix

Table A5: Database replacement outcomes across three generation rounds. Each model contributes 12 actions per round (six faults, two evidence conditions). Excess means that selected rows include unaffected rows. Harm means

R_{01}-R_{00}<-0.01

. The final columns describe the recovery-rate effect of update replacement over all 12 actions.

Repeat	Fault	Evidence	Target F1	Excess	$R_{00}$	$R_{10}$	$R_{01}$	$R_{11}$
1	Amount $+27\%$	Direct	1.000	0	1.000	1.000	1.000	1.000
1	Amount $+27\%$	Time-grouped	0.395	64572	0.341	0.760	-2.036	1.000
1	Rate $+0.023$	Direct	0.673	42266	0.055	0.051	-0.008	1.000
1	Rate $+0.023$	Time-grouped	0.941	5402	0.973	1.000	0.951	1.000
1	Amount $-21\%$	Direct	1.000	0	1.000	1.000	1.000	1.000
1	Amount $-21\%$	Time-grouped	0.393	113858	-3.507	-0.946	-6.873	1.000

Appendix

Table A6: DeepSeek-Flash: all database component outcomes. Faults are named by the change to the amount or adjustment rate, as defined in Table A4 . Excess counts selected unaffected rows. Displayed values are rounded to three decimals. Classification uses unrounded values.

Repeat	Fault	Evidence	Target F1	Excess	$R_{00}$	$R_{10}$	$R_{01}$	$R_{11}$
1	Amount $+27\%$	Direct	1.000	0	-0.828	-0.828	1.000	1.000
1	Amount $+27\%$	Time-grouped	0.225	145540	-0.369	0.807	-3.844	1.000
1	Rate $+0.023$	Direct	1.000	0	1.000	1.000	1.000	1.000
1	Rate $+0.023$	Time-grouped	1.000	0	1.000	1.000	1.000	1.000
1	Amount $-21\%$	Direct	1.000	0	1.000	1.000	1.000	1.000
1	Amount $-21\%$	Time-grouped	0.931	0	0.646	0.691	0.953	1.000

Appendix

Table A7: K3: all database component outcomes. Faults are named by the change to the amount or adjustment rate, as defined in Table A4 . Excess counts selected unaffected rows. Displayed values are rounded to three decimals. Classification uses unrounded values.

Repeat	Fault	Evidence	Target F1	Excess	$R_{00}$	$R_{10}$	$R_{01}$	$R_{11}$
1	Amount $+27\%$	Direct	0.932	0	0.949	0.996	0.955	1.000
1	Amount $+27\%$	Time-grouped	1.000	0	0.589	0.589	1.000	1.000
1	Rate $+0.023$	Direct	1.000	0	0.907	0.907	1.000	1.000
1	Rate $+0.023$	Time-grouped	0.941	5402	0.766	0.904	0.951	1.000
1	Amount $-21\%$	Direct	0.931	0	0.953	1.000	0.953	1.000
1	Amount $-21\%$	Time-grouped	1.000	0	-1.973	-1.973	1.000	1.000

Appendix

Table A8: GLM-5.3: all database component outcomes. Faults are named by the change to the amount or adjustment rate, as defined in Table A4 . Excess counts selected unaffected rows. Displayed values are rounded to three decimals. Classification uses unrounded values.

Case	Condition	Injected multiplier	Planted targets
Broad deficit	Demand deficit	0.616	168 OD pairs
Broad surplus	Demand surplus	1.268	168 OD pairs
Capacity reduction	Capacity reduction	0.781	3 links
Localized deficit	Demand deficit	0.532	48 OD pairs
Localized surplus	Demand surplus	1.878	48 OD pairs
Sensor bias	Sensor bias	1.878	6 sensors

Appendix

Table A9: Conditions in the fixed-evidence evaluation. Each state is checked for fault-class visibility and recovery rate under the reference action before model execution.

Model	Recognized	Recognized numerical cases	Mean $R$	$R\geq 0.9$	$R<0$
K3	4/7	2/5	0.023	0/5	0/5
GLM-Flash	4/7	2/5	0.036	0/5	0/5
DeepSeek-Flash	2/7	0/5	0.000	0/5	0/5
GLM-5.3	6/7	4/5	0.083	0/5	0/5
MiMo-Pro	5/7	3/5	0.015	0/5	0/5

Appendix

Table A10: Fixed-evidence results by model. Numerical recognition and recovery rate use the five numerical conditions. The mean recovery rate includes no-ops and incorrect diagnoses.

Short name	Endpoint identifier	Evaluation date (2026)
Astra	openai/gpt-6-astra	Sep 12
Sol	openai/gpt-5.6-sol	Sep 12
Terra	openai/gpt-5.6-terra	Sep 12
Opus	anthropic/claude-opus-4.8	Sep 12
Sonnet	anthropic/claude-sonnet-4.6	Sep 12
Gemini	google/gemini-3.8-flash	Sep 12

Appendix

Table A11: Endpoint identifiers for the 11-model traffic evaluation.

Model	Endpoint identifier	Date (2026)	Max tokens	Reasoning
K3	kimi-k3	Sep 14	16,384	unspecified
DeepSeek-Flash	deepseek-flash	Oct 17	8,192	disabled
GLM-Flash	z-ai/glm-5.3-flash	Sep 14	16,384	high
GLM-5.3	z-ai/glm-5.3	Sep 15	16,384 †	high †
MiMo-Pro	xiaomi/mimo-v2.5-pro	Sep 15	16,384	unspecified

Appendix

Table A12: Endpoint and decoding record for the fixed-evidence evaluation. † GLM-5.3’s broad surplus condition uses 8,192 tokens and low reasoning. All other GLM-5.3 cells use 16,384 tokens and high reasoning.

Model	Conditions	Returned provider
K3	All	native endpoint
DeepSeek-Flash	All	native endpoint
GLM-Flash	All	StreamLake
GLM-5.3	Broad deficit, broad surplus	DeepInfra
GLM-5.3	Capacity reduction, localized deficit, sensor bias	Reka
GLM-5.3	Localized surplus, healthy	Morph

Appendix

Table A13: Provider routes for the fixed-evidence evaluation.

Model	Endpoint identifier	Gateway	Max tokens	Reasoning
K3	moonshotai/kimi-k3	OpenRouter	49,152	high
DeepSeek-Flash	deepseek-flash	native endpoint	65,536	high
GLM-5.3	z-ai/glm-5.3	OpenRouter	49,152	high
GLM-Flash	z-ai/glm-5.3-flash	OpenRouter	49,152	high
MiMo-Pro	xiaomi/mimo-v2.5-pro	OpenRouter	49,152	provider default

Appendix

Table A14: Endpoint and decoding record for the difficulty study. Each row uses one gateway and configuration over the complete 43-condition matrix.

Counterfactual Evaluation

Counterfactual evaluation tests how models, explanations, and policies behave under hypothetical changes to inputs, evidence, actions, or environments.

See all 93 papers

Causal Episodic Memory for Feedback-Driven Agent Repair
Aug 6, 2026Khang Nhat Hoang Vo, Tam Minh Chu, Anh Trac Duc Dinh +2LLM Self-Correction LLM Agent Memory
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), VNU-HCM, Ho Chi Minh City, Vietnam
Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair
Oct 11, 2026Elad Sharony, Ido Greenberg, Jialin Song +3Automated Program Repair
Technion · NVIDIA
LLM-Only PDDL Domain Repair with Open-Weight Models
Aug 18, 2026Nader Karimi Bavandpour, Pascal BercherAutomated Program Repair Symbolic Planning
School of Computing, The Australian National University, Canberra, Australia

Aug 6, 2026J/K move · Enter open · S save

Causal Episodic Memory for Feedback-Driven Agent Repair

Khang Nhat Hoang Vo, Tam Minh Chu, Anh Trac Duc Dinh, Thuyen Vinh Ha Bui+1

Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), VNU-HCM, Ho Chi Minh City, Vietnam

LLM Self-Correction LLM Agent Memory Text-to-SQL

Open paper AskPDF arXiv

LLM agents that repair failures often discard successful corrections, forcing later episodes to rediscover similar solutions. We study whether finalized repair outcomes can improve subsequent Text-to-SQL episodes without parameter updates. We introduce MERIT, a training-free agent that maintains an online dual-polarity memory of oracle-verified corrections and observed unsuccessful directions. Under oracle-assisted benchmark feedback, only memories from earlier finalized episodes are eligible for retrieval. A deterministic classifier assigns a coarse failure type, which conditions a hybrid lexical-dense retriever before the frozen model generates each revision. Using Qwen2.5-7B-Instruct with identical initial predictions and repair budgets, \method{} improves execution accuracy over stateless iterative repair from

66.34\%

69.79\%

on Spider and from

47.35\%

48.44\%

on BIRD. Paired analyses provide clear evidence for the Spider gain but weaker evidence on BIRD. MERIT is not reliably separated from untyped dynamic retrieval on either benchmark, while Reflexion-style memory reaches

51.24\%

on BIRD at substantially higher inference cost. Ablations show that negative memory contributes modestly, the value of type conditioning and lexical-dense ranking is dataset dependent, and schema-local experience provides the most consistent benefit. These results clarify when causal cross-query memory improves repair and when broader memory representations remain preferable. Our implementation is available here:

Abstract

Counterfactual Evaluation

Explore similar work

Causal Episodic Memory for Feedback-Driven Agent Repair

Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair

LLM-Only PDDL Domain Repair with Open-Weight Models