The Right Fact, the Wrong Decision: SciValidityBench for Scientific Agent Memory

Yuexi Pan; Yongchao Chen

cs.AIOct 26, 2026

The Right Fact, the Wrong Decision: SciValidityBench for Scientific Agent Memory

Authors: Yuexi Pan, Yongchao Chen

Organizations: Tsinghua University · University of Science and Technology of China

Abstract

Most agent-memory benchmarks frame long-term memory as retrieval: can an agent recover the relevant record from a long and evolving history? Scientific agents face a stricter problem: a correctly remembered fact may still be invalid to use. The same reported test result can support confirmation when the outcome and stopping rule were fixed in advance, yet only exploration when the outcome was selected from twenty inspected results. We call this scientific use validity: whether remembered evidence is eligible to influence a decision under the state governing its interpretation. We introduce SciValidityBench, an executable benchmark factory with 64 TaskSpecs spanning five validity modules and six scientific domains. TaskSpecs couple generated experimental histories with scientific-state definitions, programmatic decision oracles, controlled interventions, and automatic scoring. The SciStateForge compiler-reader framework turns distributed histories into explicit decision-governing state. Counterfactual pairs and targeted corruption-repair controls then test whether models use those dependencies and localize failures. Across four model families, compiled scientific state improves valid decision-making by 9.2-18.8 percentage points over a generic Event Graph. Retrieval alone does not reproduce this cross-family gain, while content-identical flat and typed views show that the improvement comes from exposing governing state rather than a particular graph serialization. SciValidityBench thus shifts memory evaluation from finding the right fact to deciding whether that fact should control the decision.

Figures & tables

Figure 1: The right fact can license different decisions. Retrieval alone cannot distinguish the two worlds because the reported value is identical. SciValidityBench evaluates whether an agent reconstructs the cross-record state that determines how the evidence may be used. This is the distinction between remembering a result and using it validly.

Figure 2: The SciStateForge pipeline. A domain-specific history is normalized, compiled into decision-governing state, rendered for a model, and scored by an executable oracle. The worked pair changes one scientific-state condition while keeping the task semantics fixed.

Module	Governing information	Invalid use under test
Inferential Ledger	Search family, stopping rule, predeclared analysis scope, and evidence consumption	Promoting an adaptively selected or repeatedly consumed result to confirmation
Blind Memory	Plan lock, disclosure time, and authorized access roles	Treating an outcome seen too early as an uncontaminated confirmatory analysis
Construct Drift	Metric definition, population, replicate meaning, and version relation	Pooling measurements that share a name but are only convertible or incommensurate
Absence Algebra	Attempt status, instrument outcome, censoring, and detection mechanism	Collapsing not-run, failure, censoring, and confirmed zero into one kind of blank
Sunk Cost	Prospective value and the relevance of historical investment	Allowing irrecoverable past effort to alter a decision with identical future payoffs

Table 1: Five modules, one use-validity construct. Each module isolates a distinct reason why a true record may or may not govern an action.

Domain	Scientific surface	Domain	Scientific surface
Machine learning	Datasets, checkpoints, seeds, metric implementations	A/B testing	Variants, traffic slices, endpoints, scheduled looks
Clinical research	Protocols, cohorts, outcomes, censoring records	Bioinformatics	Assays, samples, replicates, pipeline versions
Systems	Hardware, workloads, runtime logs, benchmark configurations	Causal inference	Exposures, estimands, adjustment sets, identification choices

Table 2: Six domain surfaces. The adapter changes scientific objects and language while the module retains the governing validity primitive.

Model	Raw	BM25	Event	Flat	SVL	SVL $-$ Event (95% CI)
Qwen3-14B	0.798	0.802	0.794	0.991	0.981	+0.188 [+0.120, +0.255]
Gemma-3-12B	0.733	0.719	0.719	0.903	0.902	+0.183 [+0.116, +0.250]
Mistral-7B	0.556	0.586	0.566	0.692	0.678	+0.113 [+0.059, +0.166]
Phi-3.5-mini	0.480	0.475	0.478	0.609	0.570	+0.092 [+0.055, +0.134]

Table 3: Compiled state improves valid decisions across four model families. Each absolute-rate cell contains all 64 TaskSpecs at ten unseen seeds. The final column first averages paired seed differences within each TaskSpec and then weights TaskSpecs equally.

Figure 3: Main effect and independent replications. Panel A reports the TaskSpec-balanced primary contrast. Panel B shows the same contrast in two hosted seed blocks (H1/H2) and a local Qwen3-14B replication (L).

Model	Flat $-$ Event Graph	Flat $-$ Matched Source	SVL $-$ Flat
Qwen3-14B	+0.174 [+0.056, +0.300]	+0.194 [+0.066, +0.335]	+0.003 [+0.000, +0.010]
Gemma-3-12B	+0.133 [+0.000, +0.273]	+0.011 [ $-$ 0.136, +0.162]	+0.240 [+0.125, +0.356]
Mistral-7B	+0.207 [+0.078, +0.347]	+0.193 [+0.064, +0.333]	$-$ 0.005 [ $-$ 0.023, +0.009]
Phi-3.5-mini	+0.189 [+0.045, +0.375]	$-$ 0.295 [ $-$ 0.460, $-$ 0.099]	+0.488 [+0.276, +0.696]

Table 4: Source- and serialization-level controls reveal reader-specific mechanisms. Effects are balanced over 12 design families, with 95% family-resampling intervals. Matched Source and Flat SSG have exactly equalized full-chat token counts.

Figure 4: State-use and failure diagnosis. Panel A reports module-level validity gains (SVL $-$ Event Graph, percentage points) and shows where compilation helps. Value pairs test whether behavior follows a changed state. Target and matched repairs distinguish restoration of the governing information from a generic edit effect. All panels use a shared reader order and Panel A labels every module–reader contrast.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Module	ML	A/B	Clinical	Bioinfo.	Systems	Causal	Total
Inferential	4	4	4	4	4	4	24
Blind	2	2	2	0	0	2	8
Construct	2	2	2	2	2	2	12
Absence	2	2	2	2	2	2	12
Sunk Cost	2	2	0	2	2	0	8
Total	12	12	10	10	10	10	64

Appendix

Table 5: Module–domain coverage of the 64 TaskSpecs. Inferential Ledger, Construct Drift, and Absence Algebra cover all six domains. Blind Memory and Sunk Cost use domains where their interventions have a direct operational interpretation.

Component	Output	Function
Domain adapter $D_{\tau}$	Entities, units, and event language	Changes the scientific surface while preserving the validity primitive
World generator $W_{\tau}$	Hidden state and public history	Uses a declared seed and version to produce reproducible event sequences
State compiler $C_{\tau}$	Public nodes, attributes, and relations	Derives decision-governing state only from model-visible records
Action space $\mathcal{A}_{\tau}$	Finite legal decisions	Fixes the response contract before evaluation
Oracle $O_{\tau}$	Valid action	Computes the answer from the governing state without a free-form model judge
Intervention $I_{\tau}$	Counterfactual and corrupted worlds	Changes a decisive value or state component under a declared relation

Appendix

Table 6: Executable TaskSpec contract. The components map a generated history to compiled state, controlled interventions, valid actions, and diagnostic scores.

Module	Decisive scientific state	Decision change
Inferential Ledger	Whether the result followed the declared search family and stopping rule	Confirmation becomes exploratory after adaptive selection
Blind Memory	Whether outcome access occurred before the analysis plan was locked	A confirmatory analysis becomes compromised after premature access
Construct Drift	Whether two identically named measurements share a comparable definition	Results may be pooled only when their constructs are equivalent or validly converted
Absence Algebra	Whether a missing record means not run, failure, censoring, or a true zero	The next action changes among initiating, repeating, censor-aware analysis, and recording zero
Sunk Cost	Whether historical investment is relevant to prospective value	The preferred project remains unchanged when only irrecoverable past effort changes

Appendix

Table 7: One decision pattern per validity module. Each example uses the same generator–compiler–oracle interface while changing the state relation that licenses the action.

Field	Role
Stable identifiers	Bind the world, episode, event, actor, and referenced objects
Sequence and visibility	Determine event order and when a record becomes public
Versioned payload	Store domain values under a declared schema
Cost and provenance	Retain resource use and the source that produced the event
Event type	Normalize Plan , Run , Observe , Access , Write , Retrieve , and Decide

Appendix

Table 8: Canonical event fields. Normalization makes domain histories replayable before scientific-state relations are derived.

View	Model-visible content	Controlled question
Raw Events	Complete ordered public event sequence	Can the reader derive the valid action directly from the full history?
BM25 Events	Top five request-relevant event chunks	Does conventional retrieval recover the governing evidence?
Event Graph	All public events and objects with generic sequence and mention edges	Does organizing events as a graph recover use validity?
Flat SSG	Every public compiled node, attribute, and relation as flat records	What changes when decision-governing state is explicit without graph syntax?
Typed SVL	The same compiled content as typed nodes and relations	How does a structured scientific ledger affect different readers?

Appendix

Table 9: Five primary memory views. Flat SSG and typed SVL expose the same compiled nodes, attributes, and relations. Event Graph preserves all source events but omits scientific-state derivation.

Model	Raw	BM25	Event Graph	Flat SSG	SVL
Qwen3-14B	631	718	1,044	460	439
Gemma-3-12B	678	775	1,112	498	468
Mistral-7B	790	902	1,326	598	568
Phi-3.5-mini	798	909	1,344	613	580

Appendix

Table 10: Median input tokens in the primary representation matrix. Each model–view cell contains 640 prompts. Token counts use the reader’s own tokenizer. Event Graph is longest for every model.

Purpose	Tasks	Seeds	Readers	Conditions	Decisions
Primary representations and value pairs	64	10	4	Five original views plus companion worlds for the two compiled views	17,920
Corruption and repair	24	6	4	Intact, corrupted, exact repair, and matched irrelevant repair	4,896
Additional memory policies	24	10	4	Seven memory conditions	6,720
Independent hosted replication	64	Two blocks	2	Primary representation contrasts	17,920
Independent Qwen replication	64	Matched	1	Original views and value pairs	4,480
Source and serialization controls	64	4	4	Eight views over two source worlds	16,384

Appendix

Table 11: Evaluation matrices. Each matrix targets a distinct question: representation quality, state use, failure localization, generalization, or source-level control.

Model	Raw	BM25	Event Graph	Flat SSG	SVL
Qwen3-14B	0.798	0.802	0.794	0.991	0.981
Gemma-3-12B	0.733	0.719	0.719	0.903	0.902
Mistral-7B	0.556	0.586	0.566	0.692	0.678
Phi-3.5-mini	0.480	0.475	0.478	0.609	0.570

Appendix

Table 12: Original-world validity over all 64 TaskSpecs and ten seeds. Each model–view cell contains 640 decisions.

Model	BM25 $-$ Raw	Event $-$ Raw
Qwen3-14B	+0.003 [-0.003, +0.011]	-0.005 [-0.013, +0.000]
Gemma-3-12B	-0.014 [-0.031, -0.002]	-0.014 [-0.033, +0.002]
Mistral-7B	+0.030 [+0.005, +0.059]	+0.009 [-0.023, +0.044]
Phi-3.5-mini	-0.005 [-0.016, +0.003]	-0.002 [-0.011, +0.006]

Appendix

Table 13: TaskSpec-balanced representation contrasts. Intervals come from 20,000 TaskSpec resamples. Retrieval does not reproduce the cross-family compiled-state advantage.

Model	Module	Raw	BM25	Event	Flat	SVL
Qwen3-14B	Inferential	0.750	0.750	0.750	1.000	1.000
Qwen3-14B	Blind	0.512	0.512	0.475	1.000	1.000
Qwen3-14B	Construct	0.750	0.767	0.750	1.000	1.000
Qwen3-14B	Absence	1.000	1.000	1.000	1.000	1.000
Qwen3-14B	Sunk Cost	1.000	1.000	1.000	0.925	0.850
Gemma-3-12B	Inferential	0.633	0.625	0.617	0.875	0.875

Appendix

Table 14: Validity by reader, module, and representation. Each cell uses ten seeds and every primary TaskSpec in the corresponding module.

Model	ML	A/B	Clinical	Bioinfo.	Systems	Causal
Qwen3-14B	+0.183	+0.183	+0.250	+0.120	+0.120	+0.270
Gemma-3-12B	+0.192	+0.167	+0.260	+0.110	+0.120	+0.250
Mistral-7B	+0.100	+0.100	+0.090	+0.190	+0.070	+0.130
Phi-3.5-mini	+0.092	+0.108	+0.110	+0.060	+0.070	+0.110

Appendix

Table 15: Typed SVL minus Event Graph by scientific domain. TaskSpecs are equally weighted within each reader–domain cell.

Model	Families	Positive	Effect	95% interval	Family LOGO
Qwen3-14B	12	5	+0.188	[+0.033, +0.342]	[+0.155, +0.232]
Gemma-3-12B	12	6	+0.181	[+0.025, +0.336]	[+0.151, +0.227]
Mistral-7B	12	6	+0.112	[+0.017, +0.217]	[+0.083, +0.134]
Phi-3.5-mini	12	4	+0.099	[+0.021, +0.194]	[+0.064, +0.108]

Appendix

Table 16: Sensitivity to design-family dependence. Effects weight 12 module–design families equally. The final column is the range obtained by leaving out one family at a time. LOGO denotes leave-one-family-out.

Model	Tasks	Positive	Effect	95% interval	LOTO range
Qwen3-14B	42	12	+0.143	[+0.083, +0.214]	[+0.134, +0.146]
Gemma-3-12B	42	13	+0.148	[+0.081, +0.219]	[+0.137, +0.151]
Mistral-7B	42	22	+0.131	[+0.057, +0.202]	[+0.122, +0.146]
Phi-3.5-mini	42	12	+0.083	[+0.043, +0.126]	[+0.076, +0.085]

Appendix

Table 17: Exact-choice-overlap sensitivity. The primary contrast is recomputed after removing all tasks with literal equality between a compiled state value and a legal action. LOTO denotes leave-one-task-out.

Model	Flat relation acc.	SVL relation acc.	Flat both-valid	SVL both-valid
Qwen3-14B	1.000	0.998	0.991	0.981
Gemma-3-12B	0.906	0.905	0.856	0.855
Mistral-7B	0.561	0.550	0.442	0.412
Phi-3.5-mini	0.436	0.361	0.327	0.270

Appendix

Table 18: Aggregate state-use metrics over 640 pairs per reader–compiled-view cell. Both-valid requires correct actions on both sides. Relation accuracy also recognizes valid invariance.

Relation accuracy (Flat/SVL)
Model	Infer.	Blind	Construct	Absence	Sunk
Qwen3-14B	1.000/1.000	1.000/1.000	1.000/1.000	1.000/1.000	1.000/0.988
Gemma-3-12B	0.750/0.750	1.000/1.000	1.000/1.000	1.000/1.000	1.000/0.988
Mistral-7B	0.408/0.263	0.013/0.388	0.500/0.500	1.000/1.000	1.000/0.975
Phi-3.5-mini	0.300/0.275	0.988/0.613	0.300/0.300	0.100/0.000	1.000/1.000
Both-world validity (Flat/SVL)

Appendix

Table 19: Counterfactual relation accuracy and both-world validity by module. Each cell reports Flat/SVL. Module sizes are 240/80/120/120/80 pairs in the displayed order.

Operator	State change
Omit	Remove one target node and its incident relations
Unlink	Remove a governing relation while retaining both endpoints
Relabel	Replace a target type or relation label with an incompatible one
Merge	Collapse two distinct scientific objects into one identity
Duplicate	Add a second copy that creates ambiguous identity or evidence use
Inject	Add a plausible but irrelevant node or relation

Appendix

Table 20: Six reversible scientific-state corruptions. An operator is used only when the TaskSpec assigns it a defined scientific interpretation.

Model	Overall	Infer.	Blind	Construct	Absence	Sunk
Qwen3-14B	+0.269	+0.000	+0.500	+0.681	+0.833	-0.333
Gemma-3-12B	+0.328	+0.000	+0.500	+0.667	+0.875	+0.083
Mistral-7B	+0.150	+0.007	+0.438	+0.000	+0.667	+0.000
Phi-3.5-mini	-0.003	-0.021	+0.042	+0.000	+0.000	+0.000

Appendix

Table 21: Exact target repair minus matched irrelevant repair. Module columns reveal where the reader-specific recovery originates.

Model	Inject	Merge	Omit	Relabel	Unlink	Duplicate
Qwen3-14B	1.000 ∗	1.000 ∗	1.000 ∗	0.789 ∗	0.866 ∗	0.000
Gemma-3-12B	0.942 ∗	1.000 ∗	1.000 ∗	0.789 ∗	1.000 ∗	0.000
Mistral-7B	0.000	0.000	0.809 ∗	0.830 ∗	0.000	0.000
Phi-3.5-mini	0.000	0.000	0.628 ∗	0.000	0.397	0.333

Appendix

Table 22: Cross-domain corruption-fingerprint stability. A star marks cosine similarity of at least 0.50.

Reader	Flat $-$ Event	Flat $-$ Matched Source	Opaque Flat $-$ Event
Qwen3-14B	+0.174 [+0.056, +0.300]	+0.194 [+0.066, +0.335]	+0.099 [ $-$ 0.066, +0.257]
Gemma-3-12B	+0.133 [+0.000, +0.273]	+0.011 [ $-$ 0.136, +0.162]	+0.280 [+0.153, +0.405]
Mistral-7B	+0.207 [+0.078, +0.347]	+0.193 [+0.064, +0.333]	+0.115 [+0.031, +0.215]
Phi-3.5-mini	+0.189 [+0.045, +0.375]	$-$ 0.295 [ $-$ 0.460, $-$ 0.099]	+0.550 [+0.439, +0.668]

Appendix

Table 23: Source- and serialization-level contrasts. Effects are balanced across 12 module–design families. Brackets give 95% family-resampling intervals. Matched Source and Flat SSG have equal full-chat token counts.

Data block	Reader	Typed SVL $-$ Event Graph	Flat $-$ Typed SVL	BM25 $-$ Raw
Block 1	gpt-5.6-sol	+0.486 [+0.434, +0.534]	$-$ 0.011 [ $-$ 0.022, $-$ 0.002]	$-$ 0.053 [ $-$ 0.083, $-$ 0.027]
Block 1	gpt-5.5	+0.483 [+0.430, +0.533]	$-$ 0.009 [ $-$ 0.025, +0.002]	$-$ 0.028 [ $-$ 0.052, $-$ 0.008]
Block 2	gpt-5.6-sol	+0.477 [+0.425, +0.523]	$-$ 0.023 [ $-$ 0.052, $-$ 0.002]	$-$ 0.039 [ $-$ 0.070, $-$ 0.009]
Block 2	gpt-5.5	+0.470 [+0.419, +0.519]	$-$ 0.013 [ $-$ 0.028, $-$ 0.002]	$-$ 0.022 [ $-$ 0.039, $-$ 0.006]

Appendix

Table 24: Independent hosted-reader replication. Brackets give 95% TaskSpec-resampling intervals. Flat SSG and typed SVL differ by at most 2.3 points.

Reader	Typed SVL $-$ Event Graph	Flat $-$ Typed SVL	BM25 $-$ Raw
Qwen3-14B	+0.186 [+0.120, +0.253]	+0.009 [+0.000, +0.023]	+0.008 [+0.000, +0.019]

Appendix

Table 25: Independent Qwen replication. Brackets give 95% TaskSpec-resampling intervals.

Reader	Effect	95% interval	Positive tasks	Leave-one-task-out range
Qwen3-14B	+0.285	[+0.228, +0.335]	20/24	[+0.279, +0.298]
Gemma-3-12B	+0.299	[+0.238, +0.354]	20/24	[+0.291, +0.312]
Mistral-7B	+0.144	[+0.051, +0.238]	12/24	[+0.129, +0.160]
Phi-3.5-mini	+0.111	[+0.060, +0.167]	10/24	[+0.102, +0.116]

Appendix

Table 26: TaskSpec-balanced effect against the four conventional memory policies. The effect compares SVL with the mean of BM25, summary, selective, and provenance conditions.

Reader	Inferential	Blind	Construct	Absence	Sunk Cost
Qwen3-14B	+0.292	+0.306	+0.375	+0.406	+0.000
Gemma-3-12B	+0.304	+0.306	+0.392	+0.444	+0.000
Mistral-7B	+0.100	$-$ 0.150	+0.429	+0.219	+0.000
Phi-3.5-mini	+0.000	+0.300	+0.246	+0.000	+0.000

Appendix

Table 27: Module-level effects on the additional TaskSpec set. Entries use the same SVL-versus-conventional-memory contrast as Table 26 .

Policy	Nodes	Edges	Node recall	Edge recall	Required-edge recall	Endpoints
None	0.00	0.00	0.000	0.000	0.000	0.000
BM25	2.65	0.68	0.601	0.246	0.186	0.000
Summary	2.25	0.67	0.510	0.271	0.111	0.000
Selective	1.58	0.42	0.379	0.167	0.111	0.000
Provenance	4.46	1.04	1.000	0.368	0.000	1.000
ECM	3.96	1.04	0.886	0.368	0.000	0.556

Appendix

Table 28: Structural content preserved by each memory policy. The audit covers 480 TaskSpec–seed worlds. SVL is the only view that retains all nodes, governing relations, and complete relation endpoints.

Scientific Agent Benchmarks

See all 36 papers

MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
Jul 1, 2026Zhishang Xiang, Zerui Chen, Yunbo Tang +5Agent Memory LLM Sycophancy
Xiamen University · Jilin University
Beyond Downstream QA: Rethinking How We Evaluate Agent Memory
Oct 15, 2026Sebastian-Antonio Toma, Jean de Kernier, Mike Ply +1Agent Memory LLM Agent Memory
Amazon Web Services · Technical University of Cluj-Napoca, Romania
ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence
Oct 9, 2026Yiwei Chen, Amila Weerasinghe, Siddharth Biswal +4AI Agent Benchmarks Causal Reasoning in Language Models
UC San Diego · Amazon

Jul 1, 2026J/K move · Enter open · S save

MemSyco-Bench: Benchmarking Sycophancy in Agent Memory

Zhishang Xiang, Zerui Chen, Yunbo Tang, Zhimin Wei+4

Xiamen University · Jilin University

Agent Memory LLM Sycophancy LLM Agent Evaluation AI Agent Benchmarks Memory Agent Benchmarks

Open paper AskPDF arXiv

Memory has emerged as a cornerstone of modern LLM-based agents, supporting their evolution from single-turn assistants to long-term collaborators. However, memory is not always beneficial: retrieved memories often induce a critical issue of sycophancy, causing agents to over-align with the user at the cost of factual accuracy or objective reasoning. Despite this emerging risk, existing memory benchmarks primarily evaluate whether memories are correctly stored, retrieved, or updated, while overlooking how retrieved memories influence downstream reasoning and decision-making. To bridge this gap, we propose MemSyco-Bench, a comprehensive benchmark for evaluating memory-induced sycophancy in agent systems. MemSyco-Bench measures when memory should influence a decision and how valid memory should be used. Specifically, it covers five tasks that assess whether agents can reject memory as factual evidence, respect its applicable scope, resolve conflicts between memory and objective evidence, track memory updates, and use valid memory for personalization. All related resources are collected for the community at https://github.com/XMUDeepLIT/MemSyco-Bench.

Abstract

Scientific Agent Benchmarks

Explore similar work

MemSyco-Bench: Benchmarking Sycophancy in Agent Memory

Beyond Downstream QA: Rethinking How We Evaluate Agent Memory

ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence