Counterfactual Self-Evolving Agents for Evidence-Grounded Reasoning

Xing Han; Yuxin Wang; Chen Chen; Wei Dai; Gautham Krishna Gudur; Shijun Li; Hsing-Huan Chung; Gregory D. Hager

cs.AISep 26, 2026

Counterfactual Self-Evolving Agents for Evidence-Grounded Reasoning

Authors: Xing Han, Yuxin Wang, Chen Chen, Wei Dai, Gautham Krishna Gudur, Shijun Li, Hsing-Huan Chung, Gregory D. Hager, +3 more

Organizations: Johns Hopkins University · MIT · University of Texas at Austin · Bayesian Health

Abstract

Self-play proposer--solver methods improve reasoning by generating tasks and learning from verified solutions. However, for evidence-identifiable tasks, where case-specific evidence and domain knowledge determine a checkable answer, self-play requires generating plausible cases whose answers can be independently verified. We introduce counterfactual self-evolution, which generates counterfactual context for reconsidering the original case. A trainable Proposer constructs targeted evidence edits and describes potential outcome changes with causal explanations. We handcraft an expert-verified counterfactual instruction-tuning dataset to teach the Proposer to generate high-quality counterfactuals across a broad range of action--outcome scenarios. Each counterfactual instruction-tuning example specifies an edit within a defined category and explains its hypothesized causal effect on the decision, teaching the Proposer to reason systematically about what changes and why. We instruction-tune the Proposer on these examples, then formulate a fine-tuning reward that integrates feedback from the Solver and Verifier. Across diverse counterfactual scenarios, this reward favors high-quality counterfactuals and warranted revisions, while penalizing changes that overturn correct decisions. The counterfactual context aims to correct errors and strengthen confidence in correct decisions. Accepted counterfactuals accumulate in memory that supplies in-context evidence to the frozen Solver; the Solver adapts through evolving context rather than weight updates. We apply the framework to clinical reasoning, fact verification, and business reasoning. Our evaluation tracks performance over successive rounds as counterfactual memory grows, including transfer to harder cases. Our method achieves superior results across diverse frontier models.

Figures & tables

Figure 1: A multimodal clinical example with diverse counterfactual context. Using a case from our differential diagnosis evaluation ( Strickland et al., 2023 ) , we illustrate four types of counterfactual context for the agent. (a) Removing imaging hides a heart-scan result while retaining oxygen readings and prior knowledge of how an opening in the heart (PFO) affects blood flow. (b) Shifting the scan result’s arrival time tests when its evidence is available to the agent. (c) Changing the test reason replaces concern about low oxygen with a routine checkup in the ordering note, keeping the test time and result fixed. (d) Changing the doctor varies the care and test-ordering policy for the same patient. These contexts prompt the agent to reason about which evidence is available, when and why it was collected, and how clinicians’ choices shape it. Verified counterfactuals support self-evolution by providing feedback to train the Proposer and reusable context to help the Solver correct errors and preserve sound decisions on later cases.

Figure 2: Counterfactual self-evolution framework. The Proposer generates counterfactuals from the Solver’s decision and selected reasoning traces. The Verifier checks plausibility, and accepted counterfactuals enter memory as reusable Solver context. The Proposer learns through CF instruction tuning and feedback combining Solver’s correctness/confidence with Verifier’s plausibility.

Figure 3: CF context supports reconsideration. (a,b) Crafted CF context corrects an IHM prediction. (c) Test accuracy with zero-shot inference and crafted CF context.

Figure 4: Decision scenarios emphasized in CF instruction tuning. (a) Binary decision transitions favor correction (Type 3) and preservation (Type 2, strengthen the confidence). (b) Multiple-choice questions (MCQ) also allow switches between wrong options; A denotes the correct option. Strong CFs remove nonessential evidence while retaining diagnostic support; weak CFs remove a diagnostic pillar and weaken support without specifying a replacement answer.

Figure 5: CF instruction tuning produces high-quality CF context. (a) We instruction-tune a Qwen2.5-7B Proposer on increasing numbers of training CFs to guide a frozen Solver. (b) We categorize the adopted CF context by the causal test it performs mentioned in Section 3.1 : Observe changes the available evidence or its timing; Explain changes the causal hypothesis while holding evidence fixed; and Intervene changes an action. Scores are shown on a 0–100 scale.

Figure 6: Comprehensive evaluations of CF-Evolve. We instruction-tune diverse base models to generate CF context that guides a frozen Solver, followed by memory evolution to optimize the involved context. Results show that adding high-quality CF context consistently improves performance across tasks. Task-specific baselines are shown only where applicable.

Figure 7: Evaluate CF-Evolve on harder problems. (a, b) CF-Evolve accumulates memory across patient-difficulty subgroups for clinical mortality prediction at increasing prediction horizons. (c, d) CF-Evolve accumulates memory across CF banks for MMMU/MMMU-Pro business questions of increasing difficulty. Across both domains, richer context yields limited gains on easier problems but increasingly benefits harder ones. Easy decisions already have sufficient evidence, whereas difficult cases benefit from counterfactual experience that supplies missing or ambiguous reasoning paths.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Training	Validation	Test
DiagnosisArena	640	137	138
MMMU Business	677	145	145

Appendix

Table 1: Benchmark splits for MCQ evaluation.

Proposer	Solver	Accuracy	AUROC	AUPRC	F1
None (zero-shot)	Qwen	.5839	.7000	.2252	.3052
SFT	Qwen	.7372	.7543	.2985	.3723
None	Qwen SFT	.8786	.7584	.3128	.1216
Solver-specific SFT	Qwen SFT	.8630	.7595	.3212	.3521
SFT + GRPO	Qwen	.8417	.7553	.3237	.3720

Appendix

Table 2: Component comparison on full-test IHM (RQ1). SFT Proposer and SFT+GRPO are evaluated with the frozen Qwen2.5-7B Solver; the Solver-specific row uses a Proposer conditioned on the SFT Solver’s draft. Higher is better.

Configuration	Test correct	Accuracy (%)	Corrected	Harmed
Zero-shot Solver	48	34.8	—	—
CF-IT, seed 20260728	84	60.9	36	0
CF-IT, seed 20260730	87	63.0	41	2
CF-IT, seed 20260731	81	58.7	34	1
Three-seed mean	84.0	$60.9\pm 2.2$	37.0	1.0
Validation-selected consensus	87	63.0	39	0

Appendix

Table 3: DiagnosisArena CF instruction-tuning results.

Training records	Test correct	Accuracy (%)	Corrected	Harmed	Different wrong option
133	48	34.8	1	1	0
266	48	34.8	0	0	0
400	77	55.8	36	7	6
533	84	60.9	36	0	1

Appendix

Table 4: DiagnosisArena training-scale results.

Figure 9: Structured clinical CF instruction target. Current evidence and the original Solver draft condition an evidence audit, a hypothetical edit with unaffected paths held fixed, its expected effect, and an applicability boundary. Optional memory returns to the original case; abstention leaves it empty. The schema is condensed and is not a complete training record.

Setting	Value
Base model	Qwen2.5-7B-Instruct
Training scale	$3{,}000$ records; $2$ epochs
Adapter	LoRA rank $16$ , scaling $32$ , dropout $0.05$
Target projections	Attention: q, k, v, o ; MLP: gate, up, down
Trainable parameters	$40{,}370{,}176$
Learning rate and schedule	$10^{-4}$ ; $3\%$ warm-up; cosine decay

Appendix

Table 5: Full-scale clinical Proposer instruction-tuning settings.

CF role	Initial answer	Intended effect
Safe-preserving	Correct	Preserve support for the reference answer after a non-decisive evidence change.
Sufficient corrective	Incorrect	Address the key reasoning error and make the reference answer clearly best.
Insufficient corrective	Incorrect	Address the same diagnostic boundary while leaving the reference answer non-unique.

Appendix

Table 6: CF roles in MCQ supervision.

Operation	Required fields	Effect
delete	Exact target_span	Remove the specified source text.
add	Exact anchor_span , position , new_text	Insert explicitly hypothetical evidence at the specified location.
replace	Exact target_span , new_text	Replace a finding with a hypothetical alternative.

Appendix

Table 7: Executable CF edits in DiagnosisArena.

Base model	Type 2 preserve	Type 3 correct	Paired Type 3	Pool	Train / val.
Qwen2.5-7B	200	426	426	626	533 / 93
Llama3-8B	204	428	324	632	569 / 63
MedGemma-4B	255	375	296	630	567 / 63

Appendix

Table 8: DiagnosisArena supervision pools.

Base model	Type 2 preserve	Type 3 correct	Pool	Train / val.
Qwen2.5-7B	352	181	533	480 / 53
Llama3-8B	216	228	444	400 / 44
MedGemma-4B	213	248	461	415 / 46

Appendix

Table 9: MMMU Business supervision pools.

Target field	Learning target
boundary_verdict	Predict CROSSED , NOT_CROSSED , or UNRESOLVED .
proposed_answer	Predict the proposed A–D answer.
closest_competitor	Identify the competing option relevant to the evidence boundary.
think	Explain the reasoning error or retained support and cite relevant evidence IDs.
x_cf.primary , x_cf.contrast	Generate the primary executable CF and an optional insufficient contrast.
y_cf	Explain the expected effects on decision support.

Appendix

Table 10: MCQ Proposer target fields.

DiagnosisArena
Base model	Training records	After oversampling	Stopping epoch
Qwen2.5-7B	533	703	4
Llama3-8B	569	749	3
MedGemma-4B	567	567	3
MMMU Business
Qwen2.5-7B	480	648	3

Appendix

Table 11: Model-specific Proposer training settings.

Base model	Embedding checkpoint	Dim.
Qwen2.5-7B	Qwen/Qwen3-Embedding-0.6B	1,024
Llama3-8B	nvidia/llama-nemotron-embed-1b-v2	2,048
MedGemma-4B	google/embeddinggemma-300m	768

Appendix

Table 12: Embedding models for MCQ retrieval.

Knowledge corpus	Collections	Chunks
DiagnosisArena medical	39	84,572
MMMU Business	5	16,302

Appendix

Table 13: Knowledge-base size per MCQ index.

Business collection	Chunks
business_accounting	4,189
business_economics	4,767
business_finance	2,789
business_management	2,182
business_marketing	2,375
Total	16,302

Appendix

Table 14: Business corpus after cleaning.

Dataset / task	Metric	Base model	Zero-shot	RAG	GSEM	MILM	Ours
MIMIC-IV / IHM	AUROC	Qwen2.5-7B	0.7000	–	–	0.7235	0.7553
		Llama3-8B	0.7025	–	–	0.7089	0.7466
		MedGemma-4B	0.7117	–	–	0.7157	0.7374
MIMIC-IV / LOS	MAE $\downarrow$	Qwen2.5-7B	69.76	–	–	59.77	58.60
		Llama3-8B	59.42	–	–	65.26	55.20
		MedGemma-4B	56.29	–	–	60.14	55.42

Appendix

Table 15: Extended robustness and baseline comparisons (RQ2). Metrics follow Figure 6 , with additional clinical baselines. LOS is regression. A dash marks a matched result still in progress. Section E.6 specifies the MCQ acceptance policies, retrieval baselines, and Business output processing.

Solver	Proposer context	Accuracy	AUROC	AUPRC
Qwen	None (zero-shot)	0.5839	0.7000	0.2252
Qwen	Heuristic CF	0.5839	0.6968	0.2225
Qwen	SFT Proposer	0.7372	0.7543	0.2985
Qwen	SFT + GRPO	0.8417	0.7553	0.3237
Qwen SFT	None	0.8786	0.7584	0.3128
Qwen SFT	Solver-specific SFT	0.8630	0.7595	0.3212

Appendix

Table 16: Full-test IHM decomposition of Proposer instruction tuning, outcome-guided RL, and direct Solver fine-tuning. Comparisons are controlled within each Solver pair.

Figure 11: Analysis of CF supervision. (a) IHM performance with zero-shot inference and CF Proposers trained on 1k, 2k, and 3k examples. (b) Adopted CFs by unified edit-level type (Appendix E.4 ) differ across tasks. IHM excludes 1,460 no_cf outputs (

n=9{,}011

actionable CFs); LOS reports its full evaluation (

n=14{,}825

Edit type	What changes	Clinical (MIMIC-IV)	Span edit (MCQ)
Remove	An evidence item or channel is withheld	modality_ablation	delete
Add	Hypothetical evidence or an alternative explanation is introduced	evidence_addition , cause_swap	add
Replace	A finding, observation time, treatment, or support state is substituted	action_change , temporal_shift	replace
Mixed	Several atomic edits of different types	– (single edit)	combination

Appendix

Table 17: Unified edit-level CF types and their dataset-specific operations.

Base model	Context	Accuracy	AUROC	AUPRC
Qwen2.5-7B	None / learned CF	0.5839 / 0.7372	0.7000 / 0.7543	0.2252 / 0.2985
Llama3-8B	None / learned CF	0.6065 / 0.7424	0.7025 / 0.7466	0.2268 / 0.2830
MedGemma-4B	None / learned CF	0.7620 / 0.8027	0.7117 / 0.7374	0.2344 / 0.2906

Appendix

Table 18: Full IHM metrics for Table 15 .

	Accuracy (%)				Answer changes
Base model	Zero-shot	RAG	GSEM	Ours	Corrected	Harmed
DiagnosisArena ( $n=138$ )
Qwen2.5-7B	34.8	26.8	26.8	66.7	47	3
Llama3-8B	30.4	29.7	31.2	61.6	43	0
MedGemma-4B	37.7	33.3	33.3	63.0	35	3
MMMU Business ( $n=145$ )

Appendix

Table 19: MCQ results across tasks and base models.

Base model	Initial correct	Final correct	Corrected	Harmed	Gain (pp)
DiagnosisArena ( $n=138$ )
Qwen2.5-7B	48	92	47	3	+31.9
Llama3-8B	42	85	43	0	+31.2
MedGemma-4B	55	87	35	3	+23.2
MMMU Business ( $n=145$ )
Qwen2.5-7B	89	92	4	1	+2.1

Appendix

Table 20: Paired correction and preservation results. Initial correct is measured after each run’s output processing; Gain measures the subsequent CF revision step. Separate zero-shot accuracies appear in Table 19 .

Category	Count	Share (%)
Adopted outcomes ( $n=41$ )
Wrong $\rightarrow$ correct	39	95.1
Wrong $\rightarrow$ different wrong	2	4.9
Correct $\rightarrow$ wrong	0	0.0
Primary CF operation ( $n=41$ )
Replace	19	46.3

Appendix

Table 21: Adopted DiagnosisArena CF interventions.

Figure 12: Error types in contributing CF memories. The 104 DiagnosisArena seed-memory records include 44.2% missed evidence and 34.6% misweighted evidence.

Policy	Val. acc.	Test acc.	Adopted	Corrected	Harmed	Preserved (%)
Ignore RAG signal	67.9	66.7	55	47	3	93.8
Veto explicit REJECT	56.9	52.9	32	27	2	95.8
Positive-only (PASS/FLAG)	52.6	48.6	21	19	0	100.0
Single-vote + REJECT veto	67.2	66.7	51	46	2	95.8
Single-vote + positive support	66.4	64.5	44	41	0	100.0

Appendix

Table 22: RAG-Verifier acceptance policies on DiagnosisArena.

DiagnosisArena ( $n=138$ )
Base model	Zero-shot	Solver-SFT	Ours
Qwen2.5-7B	34.8	74.6	66.7
Llama3-8B	30.4	79.0	61.6
MedGemma-4B	37.7	70.3	63.0
MMMU Business ( $n=145$ )
Qwen2.5-7B	61.4	62.1	63.4

Appendix

Table 23: Comparison with direct Solver fine-tuning (accuracy, %).

Figure 13: Difficulty-ordered accumulation of CF context. Patients are grouped by frozen zero-shot confidence, and CF context is admitted cumulatively from Easy to Medium to Hard. Adding Easy context alone has little effect, whereas the larger gains emerge after Medium and Hard cases are included, especially for the longer-horizon tasks. Darker lines are longer (harder) horizons; the shaded region marks the Easy-only round, and labels in (a) and (b) give the final gain over no CF.

Figure 14: Held-out accuracy with CF memory. Each cell shows the change in test accuracy over the empty bank (bold) and the resulting test accuracy (below).

Figure 15: Memory fixes and breaks answers, and the net gain is uncertain. (a, c) Corrections (wrong

\rightarrow

correct) and damage (correct

\rightarrow

wrong) relative to the empty bank. (b, d) Net change in test accuracy with 95% paired confidence intervals and exact McNemar

p

-values.

Figure 16: Content audit of the full memory bank. Share of the 517 memories with each property; a memory can fall into several categories.

Explore similar work

CoBe: A Benchmark For Conversational Counterfactual Text Editing
Oct 9, 2026Arvind Raghavan, Inwoo Hwang, Shreyas Havaldar +6Causal Counterfactual Generation Counterfactual Reasoning in Language Models
Causal Artificial Intelligence Lab Columbia University
The Illusion of $\textit{What If}$ : Evaluating the Breakdown of Counterfactual Reasoning in LLMs
Aug 28, 2026Yucheng Wang, Yuetian Du, Zhengyi Liu +8Counterfactual Reasoning in Language Models Counterfactual Reasoning
Zhejiang University · Alibaba Group · City University of Hong Kong
COUNTERMEM: World-Model Verified Counter-Factual Memory for Language Agents
Sep 25, 2026Hongji Pu, Ruixiang Tang, Yongfeng ZhangMemory-Augmented Language Model Agents Persistent Memory for Language Models
University of Illinois Urbana–Champaign · Department of Computer Science, Rutgers University

Oct 9, 2026J/K move · Enter open · S save

CoBe: A Benchmark For Conversational Counterfactual Text Editing

Arvind Raghavan, Inwoo Hwang, Shreyas Havaldar, Kai-Zhan Lee+5

Causal Artificial Intelligence Lab Columbia University

Causal Counterfactual Generation Counterfactual Reasoning in Language Models Counterfactual Evaluation

Open paper AskPDF arXiv

Counterfactual text editing is the ability to revise a given text in light of a hypothetical change while preserving facts not causally affected by that change. When asked what would have happened if a doctor had not recommended surgery, the model output should retain the patient's medical history and modify only the consequences of the decision. This requires causal understanding: a model must determine, without being explicitly instructed to, what events are affected by the hypothetical change and what should remain fixed. This skill is central to human reasoning, especially in safety-critical domains like healthcare or law. By contrast, the task of associational text generation merely generates text statistically associated with the altered condition. In the medical example, an associational response might make the patient's symptoms less severe because milder cases correlate with no surgery. Such responses produce a plausible scenario, but not the counterfactual version of the given one. Prior counterfactual-text datasets do not clearly separate these modes of reasoning. We introduce the Counterfactual Benchmark (CoBe), a testbed for models' zero-shot ability to counterfactually edit a given textual scenario when prompted to do so in a conversationally natural way. CoBe contains 2000 questions spanning healthcare, science and engineering, business, and everyday scenarios. We test a suite of frontier and smaller models, with frontier systems achieving an average score of

58.05\%

. Our findings indicate that models frequently default to associational generation rather than preserving the causal structure of the original scenario. Moreover, performance does not scale consistently with either model size or output token length, suggesting that the failure may reflect model training objectives rather than insufficient model capacity.

Abstract

Explore similar work

CoBe: A Benchmark For Conversational Counterfactual Text Editing

The Illusion of What If\textit{What If}What If: Evaluating the Breakdown of Counterfactual Reasoning in LLMs

COUNTERMEM: World-Model Verified Counter-Factual Memory for Language Agents

The Illusion of $\textit{What If}$ : Evaluating the Breakdown of Counterfactual Reasoning in LLMs