Impact Is Not Invalidation: Ask About the Claim, Not the Diff

Atul Anand

cs.CLSep 20, 2026

Impact Is Not Invalidation: Ask About the Claim, Not the Diff

Authors: Atul Anand

Organizations: Thomson Reuters Bangalore, India

Abstract

Memory systems for coding agents must decide, when a repository changes, which of their stored claims have become false. Content anchoring invalidates a claim whenever the artifact it came from changes, which fires constantly. Semantic-equivalence classification asks whether a diff preserves behavior, a question about the diff rather than about any stored claim. We show the second signal fails for a reason unrelated to model capability: asked whether a commit preserves behavior, five models spanning a 40x price range fire on 59-72% of real commits and reach precisions of only 0.291 to 0.329 against a 0.25 base rate. Asked instead whether one specific claim still holds, the same models on the same diffs reach 0.705 to 0.974. A control that hands the behavior-preservation judge the claim text, changing only the question, moves precision by 0.010 and 0.016; changing the question moves it by 0.49 and 0.65. We also compare against pytest-testmon, a deployed regression-test selector with coverage-derived dependency data: it reaches 0.868 recall at 0.415 precision, so near-complete knowledge of what a change can reach does not identify what it falsifies. Ground truth is execution, not annotation: a claim is a test function passing at commit t, and it has flipped if that same assertion text fails at t+1. Building this required an observation we did not find in prior work. On a CI-gated mainline a commit that leaves a pre-existing test failing cannot merge, so the naive construction has an empty positive class by design. We report 10,369 claims with 184 execution-verified flips mined from 23 Python libraries, splits held out by repository, a post-knowledge-cutoff split, a shuffled-diff null, a paraphrase control, and a leave-one-repository-out analysis over 17 repositories.

Figures & tables

Fig. 1: A real case from our benchmark. The commit removes an argument check from nth_combination , so it does change behaviour and a semantic-equivalence judge is right to say so. The stored claim is about all_equal , which the commit never touches, and execution confirms it still holds. Answering “did behaviour change” correctly is not the same as answering “is this claim still true”.

Repository	Commits	Claims	Flips	Flip rate	Discarded
arrow	26	1547	19	1.2%	73
boltons	6	360	14	3.9%	0
cachetools	18	1080	43	4.0%	180
click	18	1068	8	0.7%	0
freezegun	7	309	1	0.3%	0
inflection	2	120	0	0.0%	0

TABLE I: The mined corpus. Ten of the twenty-three repositories produced any flips at all; the empty ones are listed because the rate at which mining comes up dry is part of what the construction costs.

Arm	Signal	Precision	Recall	TP/FP/FN
A3 file anchor	structural anchor	0.21 [0.16,0.26]	0.30 [0.23,0.38]	46/173/105
A4 symbol anchor	structural anchor	0.39 [0.32,0.45]	0.54 [0.46,0.62]	81/128/70
A4b testmon	structural anchor	0.41 [0.36,0.47]	0.87 [0.81,0.92]	131/185/20
A5 sem-equiv strict	diff-level LLM	0.29 [0.24,0.34]	0.69 [0.61,0.76]	104/254/47
A6 risk-framed	claim-level LLM	0.79 [0.72,0.86]	0.72 [0.64,0.79]	108/28/43
A4 $\cup$ A6	union	0.45 [0.39,0.51]	0.81 [0.74,0.87]	122/149/29

TABLE II: Held-out split: 604 claims, 151 flips, 17 repositories, stratified to a 25% flip rate. Intervals are 95% bootstrap.

Fig. 2: Held-out arms with bootstrap 95% confidence intervals. The dashed line is the base rate, which is what always-invalidate achieves. Both the file-anchoring and semantic-equivalence arms sit on or below it.

Arm	Precision	Recall	Claims re-verified	Flips served stale
A4b testmon	0.415	0.868	52%	13%
A6 risk-framed	0.794	0.715	22%	28%

TABLE III: Held-out, 604 claims. Neither arm dominates: testmon buys low staleness with a large verification budget, A6 buys a small budget at higher staleness. testmon supplied a verdict for 80% of claims; the rest are charged to it as holds.

		held-out		post-cutoff
Information	Question	flash-lite	opus-5	flash-lite	opus-5
diff only	behavior-preserving?	0.291	0.309	0.338	0.278
claim + diff	behavior-preserving?	0.301	0.325	0.349	0.289
claim + diff	does this claim hold?	0.794	0.974	0.750	0.913
adding the claim is worth		+0.01	+0.02	+0.01	+0.01
changing the question is worth		+0.49	+0.65	+0.40	+0.62

TABLE IV: Precision; the base rate is 0.25 in every column. The design replicates on the post-cutoff split, whose commits postdate the training data of every model evaluated, so the effect is not recall of the repository.

Fig. 3: The same data as Table IV for the budget model. Reading across the top row, information buys almost nothing. Reading down the right column, the question buys 0.59.

… 13 further repositories, gap between +37.7 and +43.4
Repository excluded	A6 precision	A4 precision	Gap
none (all held-out)	0.794	0.388	+40.7
cachetools (43 flips)	0.775	0.431	+34.4
more-itertools (41 flips)	0.705	0.238	+46.7
markdown (28 flips)	0.793	0.382	+41.1
structlog (14 flips)	0.788	0.345	+44.3

TABLE V: Leave-one-repository-out over the held-out split. Removing any single repository, including either of the two largest contributors of positives, leaves the gap far above the pre-registered threshold of +15.

Arm	Claims re-verified	Flips served stale	Precision at natural rate
A0 always-invalidate	100%	0%	0.018
A3 file anchor	36%	70%	0.014
A4 symbol anchor	35%	46%	0.033
A4b testmon	52%	13%	0.037
A5 sem-equiv strict	59%	31%	0.022
A6 claim-relative	22%	28%	0.174

TABLE VI: Held-out, 604 claims. The last column reweights held claims to the observed 1.78% natural flip rate, which is what a deployment faces.

Claim given to the arm	Precision	Recall	TP/FP/FN
test function (as elsewhere)	0.794	0.715	108/28/43
prose note	0.711	0.669	101/41/50
for reference: A4b testmon	0.415	0.868	131/185/20
for reference: A5 sem-equiv	0.291	0.689	104/254/47

TABLE VII: Held-out, 601 claims that rendered, 100% coverage. Rewriting the claim as prose costs precision but does not remove the effect.

		A5 sem-equiv		A6 claim-relative
Model	Tier	precision	fires on	precision	fires on
gemini-3.1-flash-lite	floor	0.291	59%	0.794	23%
gemini-2.5-flash	floor	0.329	64%	0.810	23%
claude-haiku-4.5	mid	0.321	65%	0.705	29%
claude-sonnet-5	mid	0.322	63%	0.952	17%
claude-opus-5	ceiling	0.309	72%	0.974	19%

TABLE VIII: Held-out, 604 claims, base rate 0.25. A5 spans 0.291 to 0.329 across a roughly 40

\times

price range and never separates from the base rate; A6 spans 0.705 to 0.974 on the same claims and diffs.

Split	$n$	flips	A4	A4b	A5	A6
post-cutoff, click	62	8	0.00/0.00	0.21/1.00	0.20/1.00	0.53/1.00
held-out, click	68	8	0.00/0.00	0.19/1.00	0.19/1.00	0.50/1.00
post-cutoff, boltons	26	14	0.75/0.86	0.29/0.29	0.58/1.00	1.00/0.93
post-cutoff, both repos	88	22	0.60/0.55	0.23/0.55	0.34/1.00	0.75/0.95
held-out, all 17 repos	604	151	0.39/0.54	0.41/0.87	0.29/0.69	0.79/0.72

TABLE IX: Holding the repository fixed across the training cutoff. Cells are precision/recall for the budget model. click is the one repository present on both sides, and every arm reproduces there to within 0.03. The reordering seen on the full post-cutoff split is carried by boltons , which appears only after the cutoff and has a 54% flip rate.

Claim Verification

Claim verification assesses whether statements are supported, contradicted, or unresolved by relevant evidence.

See all 59 papers

Aug 4, 2026

EA-Graph: Artifact-Anchored Verification Memory for Coding Agents under Upstream Drift

Coding agents increasingly work across sessions, but prose notes can preserve a conclusion without the program state that supported it. After an upstream change, a repository may still build even though earlier verification claims are no longer valid. EA-Graph is an artifact-anchored memory for verification claims. It represents artifacts at sub-path granularity, resolves aliases to leaf definitions, anchors each claim to the content used to establish it, and keeps evidence strength separate from freshness. When replacement content is unavailable, the claim becomes unprovable rather than guessed. EA-Graph is evaluated on generated repositories whose behavior-to-artifact ground truth is known by construction. The task is to classify prior claims as unaffected, affected, or unprovable after value drift, logic drift, and deliberately withheld upstream content. The analysis covers 42 sessions across seven clean worlds, 14 model-world instances, three memory conditions, and two model tiers. In the Haiku round, artifact-anchored memory outscored prose notes and no persistent memory in all seven worlds; each exact paired Wilcoxon comparison yielded p = 0.0156. In the Sonnet round, the anchored condition was perfect, but frequent control ceilings left the preregistered contrasts non-significant. No session fabricated withheld content. These results support a bounded claim: artifact-anchored memory improved the smaller model's provability judgments in this testbed. An exploratory comparison further suggests that structured claim memory may narrow a capability gap by externalizing in-session re-derivation, but it does not establish cross- model equivalence. The study makes no claim about efficiency or repair quality.

Hwai-Jung Hsu, Cheng-Jan Chi, Hanna Everett

Sustainable and Peer Education Center, National Yang Ming Chiao Tung University · Office of AI Affairs, National Yang Ming Chiao Tung University

Agent Memory Software Engineering Agents Memory Agent Benchmarks Agent Memory Management

Ask PDF

Jul 14, 2026

SemaDiff: Identifying Semantic-Changing Commits with Generated Code and Tests

Distinguishing semantic-preserving commits from changing ones remains an open challenge in software repository mining. While existing approaches detect refactoring commits accurately, they cannot ensure that a commit is purely semantic-preserving, without any interleaving behaviour-changing modification. This limitation can impact several tasks, such as debugging, fault localisation, bug dataset construction, rollback analysis, and bug fixes backporting. To fill this gap, we propose SemaDiff, a novel approach for identifying semantic-preserving commits through behaviour-based analysis; comparison of similar test execution on pre- and post-commit versions. As code impacted by the refactoring is often hard to test and different accross both versions, we propose generating additional calling methods to that code, which serve as testing target. Given a commit, SemaDiff analyses the diff to identify modified code and extracts unchanged dependent code that calls it. It then generates an additional dependent class using a large language model to exercise the changed code in both versions, and automatically generates tests for the dependent code. This way, we obtain the same tests for the different code versions, enabling the behavioural-difference detection. The commit is classified as semantic-preserving only if all generated tests produce identical outcomes across the two versions. To evaluate SemaDiff, we construct and annotate manually a dataset of 183 commits, gathered from well-known open-source Java projects. The obtained results show that SemaDiff distinguishes accurately semantic-preserving from -- changing commits in about 76% of the cases, with a 100% precision in semantic-changing commit detection.

Maha Ayub, Michael Konstantinou, Ahmed Khanfir +2

SnT, University of Luxembourg, Luxembourg · RIADI, ENSI, University of Manouba, Tunisia · Concordia University, Canada

Differential Testing Semantic Change Detection

Ask PDF

Jul 17, 2026

Presentation, Not Mechanism: A Render Confound in Deprecation-Aware Memory Evaluation

AI systems increasingly retrieve from records that revise themselves: issue threads, encyclopedic histories, policy logs, and long conversations. The challenge is not only finding relevant evidence, but deciding which claims remain in force, which were superseded, and when to abstain. Structured memories promise to solve this with typed edges, temporal updates, and conflict status, yet evaluations often change mechanism and prompt presentation together. We study this as Evidence-State Revision, comparing flat retrieval, coarse edge invalidation, and fine-grained RevisionLedger on 2,907 high-agreement questions from GitHub, multi-repo issue histories, Wikipedia, and DyKnow-style temporal streams. A render-matched control (same layout, deprecation disabled) reveals the central confound: when a value is changed and later restored, RevisionLedger appears to beat a flat baseline by +0.182, but almost all the gain comes from easier presentation; the fine-grained mechanism residual is indistinguishable from zero (+0.021 to +0.025 across two judge families). After presentation is controlled, coarse invalidation is the only mechanism that pays for current-state queries, beating the fine ledger by 0.084; the same query-sufficiency principle says provenance mainly needs retained invalidated evidence, not richer typing. Memory evaluations should hold render fixed, and deprecation-aware systems should deploy the coarsest retained state that covers their queries.

Zhaoyang Jiang, Zhizhong Fu, Zicheng Li +5

School of Health & Wellbeing, University of Glasgow, Glasgow, UK · School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China · Institute of Health Informatics, University College London, London, UK +1

LLM Memory Question Answering State Tracking IR Evaluation Evidence Retrieval Temporal QA

Ask PDF

Abstract

Claim Verification

Explore similar work

EA-Graph: Artifact-Anchored Verification Memory for Coding Agents under Upstream Drift

SemaDiff: Identifying Semantic-Changing Commits with Generated Code and Tests

Presentation, Not Mechanism: A Render Confound in Deprecation-Aware Memory Evaluation