DreamBench-SWE: A Multi-Session Memory-Hygiene Benchmark for Software Agents

Sarthak Singh

cs.AIAug 21, 2026

DreamBench-SWE: A Multi-Session Memory-Hygiene Benchmark for Software Agents

Authors: Sarthak Singh

Organizations: Independent Researcher

Abstract

DreamBench-SWE is a multi-session benchmark for software-agent memory hygiene in which later software tasks depend on non-inferable evidence from earlier sessions and are scored by executable hidden oracles. We report the original scaled v2 fold and a separately preregistered v2.1 successor audit designed after that study but frozen before successor outcome inspection. The successor run completed 360/360 work units and 720/720 S3 cells across four conditions. In the original fold, the primary DF-hybrid--B5 contrast was null (95/180 versus 89/180; clustered p=.518, Holm p=1), not evidence of equivalence, and C9/C10 retained B0-headroom limitations. In the successor, no external memory achieved 21/180 passes (rate 0.1167), deterministic verbatim event memory 82/180 (rate 0.4556), the typed-plus-raw reference probe 83/180 (rate 0.4611), and one pinned hosted Mem0 literal-storage configuration 97/180 (rate 0.5389). The registered six-slot Family A retained unavailable slots at p=1; all three available comparisons against no memory rejected after Holm correction. Both preregistered mechanism contrasts were unavailable after pre-evaluation conformance rejection. The secondary literal-storage-versus-verbatim comparison was nonconfirmatory and sensitivity-dependent, while the comparison with the reference probe did not reject. The audit therefore supports DreamBench-SWE as a discriminating executable profile benchmark and characterizes one exact hosted-memory configuration, but it does not establish an external-system mechanism, superiority among memory-bearing conditions, equivalence, or broad product generality. The original v2.0.5 findings and artifacts remain unchanged.

Figures & tables

Claim family	Tested?	Evidence	Result	Section
Benchmark validity	V2 fold complete	60 traps, 3 seeds, 1890 condition-level result files; admission funnel 30 authored, 30 dry-valid, 30 live-valid, 30 admitted	Completed measurement fold, narrowed by C9 and C10 B0-headroom failures	6
H1 / v2 P1: hybrid typed-plus-raw memory vs. strong verbatim event memory	V1 tested; v2 clustered P1 tested	V1 reference-probe hybrid $52/66=0.788$ vs. B5 $48/66=0.727$ , Holm $p{=}1.0$ ; v2 reference-probe hybrid $95/180=0.528$ vs. B5 $89/180=0.494$ , signed $+6$ , permutation $p{=}0.518$ , Holm $p{=}1.0$	Separation not established; failure to reject, not equivalence	6.2
Structured memory anchor	V1 and v2 tested	V1 typed-only vs. B3 positive with clustered caveats; v2 DF vs. B3 signed $+13$ , permutation $p{=}0.279$ , Holm $p{=}1.0$	V2 does not reject; v1 anchor remains historical, not headline	6.2
Hygiene diagnostics	Diagnostic	V2 oracle metrics: IrrelevantImportRate $1434/3024=0.474$ , OverscopeRate $378/2268=0.167$ , RepeatedErrorRate $1512/2646=0.571$ , StaleUseRate $756/3780=0.200$	Executable diagnostics expose failure modes; no judge-based hygiene headline	6.4
Cost and resource diagnostics	Diagnostic only	V2 cost per successful task: reference-probe raw-only $0.0356$ , reference-probe hybrid $0.0383$ , B5 $0.0395$	Resource evidence constrains interpretation; it is not a headline claim	8
V2.1 successor discrimination	Separate preregistered audit	Family A: B5, DF-hybrid, and B5-MEM0-LIT each reject versus B0 after fixed-six-slot Holm correction; Family B has no available contrast	Benchmark discrimination and one bounded external profile; no mechanism, superiority, equivalence, or product ranking	6.6

Table 1: Re-anchored claim ledger. Confirmatory claims are limited to canonical analyzer outputs; nulls are failures to reject, not equivalence claims.

Work	What prior work owns	What DreamBench-SWE/reference probe adds
LoCoMo, LongMemEval, BEAM, AMA-Bench, MemoryAgentBench [ 16 , 36 , 30 , 43 , 8 ]	Long-history conversational memory, temporal reasoning, updates, incremental interaction, selective forgetting, and agentic QA.	Repository-state continuation in which earlier-session evidence controls executable code-oracle success rather than only answer quality.
MemoryArena [ 7 ]	Interdependent multi-session Memory-Agent-Environment loops in web navigation, planning, search, and formal reasoning.	SWE-specific hidden-evidence traps, production code edits, repository oracles, and contamination controls.
EvoMemBench [ 34 ]	In-/cross-episode and knowledge-/execution-oriented evaluation of 15 representative memory methods.	A narrower controlled test of repository continuity and hygiene pathologies, with trap-clustered inference over executable S3 outcomes.
WorldMemArena [ 15 ]	Multimodal action-world interaction with stage-level diagnosis of memory writing, maintenance, retrieval, and use.	Text-and-code repository tasks whose decisive evidence is hidden from S3 and whose success is checked by software oracles.
STATE-Bench [ 17 ]	Stateful enterprise workflows with database actions, simulated users, and an Agent Learning track for reusable memories, skills, or prompt optimizations.	Multi-session software-repository memory hygiene with frozen earlier evidence, code diffs, and hidden repository assertions.
Mem0 / Mem0 g [ 2 ]	Online extraction, consolidation, retrieval, and graph memory for long-term agent memory; B5-MEM0 and B5-MEM0-LIT are two pinned configurations of one hosted external system family.	Multi-session SWE hygiene traps with hidden CSPRNG tokens and executable oracles; offline typed provenance, contradiction repair, grounded replay, stale suppression, and a retrieval gate.

Table 2: Closest prior work and claim boundary. The table does not assert generic benchmark priority. Rows other than Mem0 are positioning references, not additional external memory-system baselines in the reported v2 fold.

Panel of Figure 1 — Figure 1: Reference-probe system architecture for the wake/sleep/read loop. Raw episodes are append-only; sleep operators write or update only derived-memory records.

Operation	Write or transition	Ablation	Fold status
Typed consolidation	Candidate semantic-project, procedural, human-feedback, constraint, and episodic-derived memories with scope, provenance, confidence, utility, risk, staleness, tags, and write reason	A1 no typed consolidation; A11 forced consolidation	A1 designed/not run; A11 run/descriptive; ladder isolates typed pipeline
Causal failure extraction	Failure memories grounded in failed trajectories, command output, tests, diffs, logs, or feedback	A3 no causal failure extraction	Designed/not run
Counterfactual replay	Dream artifacts and replay-derived failure or suppression decisions, accepted only when grounded in executable evidence, raw trace evidence, or human-audited causal support	A4 no counterfactual replay	Run/descriptive
Contradiction repair	Contradiction records, supersession links, status transitions, and validity-interval updates over derived memories	A2 no contradiction repair	Run/descriptive
Stale suppression and forgetting	stale , superseded , or logical deleted status for derived memories; retrieval exclusion without deleting raw evidence	A5 no stale suppression	Run/descriptive; stale activation evaluated through proxies
Provenance gate	active versus requires_review transition based on whether the derived claim is supported by raw episodes	A8 no provenance gate	Designed/not run

Table 3: Implemented operators and ablation coverage. A0 episodic-only disables the derived store entirely and is run as a descriptive floor. The confirmatory fold isolates ladder-level channels; it does not statistically isolate every named operator.

Quantity	Definition in the confirmatory fold
Trap universe	V2 primary fold: 60 S3 traps, each the third session of a three-session sequence; historical v1 fold: 22 S3 traps.
Seeds	3 seeds per condition.
Complete S3 cells	180 seed-trap S3 task cells for complete v2 conditions; 66 seed-trap S3 task cells for complete historical v1 conditions.
Warmup denominator	Diagnostic pre-S3 trajectory counts reported by the analyzer; warmup rates do not replace the S3 task-cell denominator.
Hygiene denominator	Unique analyzer-contributing result records for the relevant fold; hygiene means and count sums are deduped by the canonical analyzer before paper tables are generated.
Validity exclusions	Record-level exclusions when error_type contains task_exception , codex_exec_failed , or isolation_unavailable ; exclusions are not automatically lost S3 cells.

Table 4: Unit of analysis and aggregation rules. Source: CONFIRMATORY-FOLD.md .

Panel of Figure 2 — Figure 2: DreamBench-SWE sequence-template skeleton used for the reported confirmatory fold. Tokens shown in the schematic are synthetic or redacted placeholders; the per-trap audit is in Appendix A .

Code	Canonical label	Contract
B0	No external memory	No external memory reads or writes.
B1	Raw episodic retrieval	Retrieves raw prior trajectories.
B2	Vector trace retrieval	Deterministic token-vector retrieval over prior trace chunks.
B3	Reflection-only vector retrieval	Free-form verbal reflection lessons after sessions, retrieved by vector similarity.
B4	Untyped summary memory	Deterministic untyped session summaries.
B5	Verbatim event-memory substitute	Offline instance-memory substitute; one deterministic verbatim injected-event memory per trajectory; not stock Mem0.

Table 5: Condition and baseline contract for the main paper. Source definitions: CONFIRMATORY-FOLD.md and PREREGISTRATION.md .

Item	Reported fold setting
Wake agent and model	Codex CLI agent using gpt-5.5 .
Provider route	Hosted OpenAI/Codex route through the configured Codex CLI credentials.
Sleep/judge model	--judge-model codex-gpt-5.5 , mapped by the harness to gpt-5.5 .
Agent container	@openai/codex 0.142.0 in the scripts/Dockerfile.codex-agent image on node:22-slim .
Primary fold run date	2026-07-06, recorded by run stamp 20260706T074759Z .
Seeds	Seeds 1, 2, and 3 control the harness seed, condition namespace, task ordering within the recorded grid, and any seeded sampling in the harness; hosted model nondeterminism can still remain.

Table 6: D&B setup essentials for the reported confirmatory fold. Source pointers are the launch script, Codex-agent Dockerfile, and reproduction manifest.

Condition	Wake backbone(s)	Traps	Cells	Pass@1	Clustered 95% CI
B0 No memory	Codex / GPT-5.5	60	180	21/180 (0.117)	[0.050, 0.200]
B1 Raw episodic	Codex / GPT-5.5	60	180	21/180 (0.117)	[0.050, 0.200]
B2 Vector traces	Codex / GPT-5.5	60	180	20/180 (0.111)	[0.033, 0.194]
B3 Reflection-vector	Codex / GPT-5.5	60	180	67/180 (0.372)	[0.250, 0.489]
B4 Untyped summary	Codex / GPT-5.5	60	180	21/180 (0.117)	[0.050, 0.200]
B5 Instance memory	Codex / GPT-5.5	60	180	89/180 (0.494)	[0.378, 0.617]

Table 7: V2 main ablation ladder S3 Pass@1 for the primary wake backbone. Hosted live-memory baselines are excluded from this headline table and reserved for failure analysis. Intervals are trap-cluster bootstrap 95% CIs; seeds are repeated observations within traps.

Construct	Traps	Cells	B5	Probe hybrid	Best observed	Small-n
C1 Verbatim retention	18	1026	48/54 (0.889)	46/54 (0.852)	B5 Instance memory 48/54 (0.889)	no
C2 Retrieval precision	6	342	15/18 (0.833)	12/18 (0.667)	B5 Instance memory 15/18 (0.833)	no
C3 Staleness/supersession	7	399	13/21 (0.619)	8/21 (0.381)	B5 Instance memory 13/21 (0.619)	no
C4 Update propagation	1	57	0/3 (0.000)	0/3 (0.000)	A4 1/3 (0.333)	yes
C5 Scope discipline	6	342	4/18 (0.222)	3/18 (0.167)	B5 Instance memory 4/18 (0.222)	no
C6 Contradiction/provenance	5	285	3/15 (0.200)	3/15 (0.200)	A2 4/15 (0.267)	yes

Table 8: V2 per-construct S3 strata C1–C10, descriptive only. Hosted live-memory baselines are excluded from headline construct summaries. Small-n is flagged when a construct has fewer than 6 trap clusters with observed cells.

ID	Comparison	Signed statistic	Permutation $p$	Holm $p$	Reject	First/second wins
P1	reference-probe hybrid vs. B5	$+6$	0.518419	1.000	no	21 / 15
P2	reference-probe typed-only vs. B5	$-9$	0.482597	1.000	no	22 / 31
P3	reference-probe raw-only vs. B5	$-5$	0.489746	1.000	no	7 / 12
P4	reference-probe hybrid vs. reference-probe raw-only	$+11$	0.127867	0.767	no	22 / 11
P5	reference-probe hybrid vs. reference-probe typed-only	$+15$	0.134251	0.767	no	33 / 18
P6	reference-probe typed-only vs. B3	$+13$	0.279096	1.000	no	32 / 19

Table 9: V2 trap-clustered P-family with Holm correction. Signed statistic is positive when the first condition has more trap-level wins. Null outcomes are failures to reject, not equivalence tests. Source: canonical clustered analyzer JSON.

Stratum	Max condition rate	Spread	B0 rate	Holds?	Note
C2	0.833	0.833	0/18 = 0.000	yes	retrieval precision
C3	0.667	0.667	0/15 = 0.000	yes	staleness/supersession
C5	0.222	0.222	0/18 = 0.000	yes	scope discipline
C6	0.250	0.250	0/12 = 0.000	yes	contradiction/provenance
C7	0.333	0.333	0/9 = 0.000	yes	cross-session synthesis
C9	1.000	1.000	12/12 = 1.000	no	invalid: B0-headroom fail

Table 10: V2 construct-validity headroom summary. The overall strengthened F7 result is false because C9 fails; C10 B0 headroom also fails. C9 and C10 are invalid as anti-hoarding or abstention evidence in this fold by the B0-headroom criterion. Source: canonical post-fold analysis packet.

Metric	Numerator	Denominator	Rate
IrrelevantImportRate	1434	3024	0.474
OverscopeRate	378	2268	0.167
RepeatedErrorRate	1512	2646	0.571
StaleUseRate	756	3780	0.200

Table 11: Executable v2 hygiene-oracle aggregate metrics. Source: canonical v2 hygiene-oracle JSON.

Condition	Passed	Valid S3	Rate	Trap-clustered 95% CI
B0	21	180	0.1167	[0.0348, 0.1986]
B5	82	180	0.4556	[0.3304, 0.5807]
DF-hybrid	83	180	0.4611	[0.3545, 0.5677]
B5-MEM0-LIT	97	180	0.5389	[0.4165, 0.6613]

Table 12: Canonical v2.1 successor condition profile. Seeds are repeated observations within 60 trap clusters. Intervals are equal-trap-weighted normal intervals over trap means, not pairwise equivalence intervals. Marginal rates are descriptive; inferential claims are limited to registered comparisons.

Family	Registered comparison	Availability	Raw clustered $p$	Holm $p$	Interpretation
A	B5 vs. B0	available	0.000324063	0.00129625	rejects
A	DF-hybrid vs. B0	available	1.56049e-06	9.36294e-06	rejects
A	B5-MEM0 vs. B0	unavailable	1	1	pre-evaluation rejection
A	B5-MEM0-LIT vs. B0	available	7.85204e-06	3.92602e-05	rejects
A	B5-SM-LOCAL vs. B0	unavailable	1	1	pre-evaluation rejection
A	B5-SM-LOCAL-DOC vs. B0	unavailable	1	1	pre-evaluation rejection

Table 13: Preregistered successor comparison families. Unavailable slots retain raw and adjusted

p=1

; neither family was shrunk. Family A supports benchmark discrimination against B0. Family B contains no evaluable mechanism contrast.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Sequence	Class	Required S3 dependency	B5 S3	reference-probe typed-only S3
config-convention-sections	Recall-verbatim	Section-list record and a redacted schema marker appear in event text	3/3	3/3
config-freeze-provenance	Recall-verbatim	freeze_tag() must return the redacted freeze tag	3/3	2/3
config-reviewer-coerce-tag	Recall-verbatim	Bad-bool rejection format with a redacted reviewer code is stated in event text	0/3	1/3
config-reviewer-dupkey	Recall-verbatim	Duplicate-key line-numbered message format is stated in event text	3/3	3/3
config-reviewer-strict-csv	Recall-verbatim	Strict CSV width message format is stated in event text	3/3	3/3
config-stale-merge	Recall-verbatim	Redacted deep-merge source token is stated in event text	3/3	2/3

Appendix

Table 14: Per-trap recall-verbatim audit for the 22 v1.0 S3 traps. Outcome columns count S3 successes across seeds 1, 2, and 3 from per_trap_matrix_and_leakage.json . The table has 18 recall-verbatim rows and four synthesis/apply rows, matching the paper’s 18/22 evidence-location disclosure; the label does not imply that a solution patch is available for copying.

Artifact	SHA-256
analysis/fold/v2_fold.json	bfd63c8090c8b44650d56b582dac207faa99210bf25f66a911fe24c1d7f6f491
analysis/fold/v2_confirmatory_clustered.json	2294352db444fdcb0ee367f7c8b2a3682848cd66311f35ba04fb346ca4a71425
analysis/fold/v2_hygiene_oracle.json	77cd239c9d4097535d511aa6bc89adb0ec06f29c844cb6e4bf3b390c7740358f
analysis/fold/v2_cost_frontier.json	60e82b31126930642141c042f20052108147e4ab700d01bfdd77f4c4743b7662
analysis/fold/admission_funnel.json	229375cea802c0dcbc2576397ba73023b6e42c2fc8f59a616ef958904552fb52
analysis/fold/v2_tables.tex	2c1118dac674fbbb6d09875eb209cf621278cb8851b929901517cf52485f0b9a

Appendix

Table 15: Checksums for the v2 analyzer bundle artifacts in this checkout.

Condition	Definition
B0	No external memory reads or writes.
B1	Raw prior trajectory retrieval only.
B2	Deterministic token-vector retrieval over prior trace chunks.
B3	Reflection-only vector retrieval: free-form verbal reflection lessons after sessions, retrieved by vector similarity.
B4	Deterministic untyped session summaries.
B5	Offline instance-memory substitute; one deterministic verbatim injected-event memory per trajectory.

Appendix

Table 16: Baseline condition definitions.

Condition	Definition
DF	Typed consolidation, contradiction repair, counterfactual replay, local maintenance, and retrieval gating.
DF-raw-only	Typed consolidation disabled; raw-evidence capsules enabled; contradiction records excluded from implementation reads.
DF-hybrid	DF plus raw-evidence capsules; contradiction records excluded from implementation reads.
DF-strict	DF with $\theta_{\mathrm{admit}}=1.5$ , read limit 3, and read-token budget 600.
DF-strict-hybrid	DF-hybrid with the same strict retrieval settings as DF-strict.
A0	Episodic-only raw episode writes; no typed consolidation, repair, or replay.

Appendix

Table 17: Reference-probe ladder and ablation definitions.

Construct	Construct name	Target $n$	Role in V2
C1	Verbatim retention	18	V1 continuity and B5 calibration anchor
C2	Retrieval precision under interference	6	Anti-hoarding and executable precision under distractors
C3	Staleness detection and supersession	7	Stale suppression and multi-hop supersession
C4	Update propagation / regression avoidance	1	Derived update propagation probe
C5	Scope discipline	6	Overgeneralization resistance
C6	Contradiction handling / provenance conflict	5	Authority and precedence under conflict

Appendix

Table 18: Frozen V2 construct quota. The arithmetic is

18+6+7+1+6+5+7+4+4+2=60

ID	Comparison	Purpose
P1	DF-hybrid vs. B5	Headline comparison against strongest V1 baseline
P2	DF-typed-only vs. B5	Typed consolidation alone against verbatim memory
P3	DF-raw-only vs. B5	Raw evidence retention against B5
P4	DF-hybrid vs. DF-raw-only	Incremental value of typed maintenance above raw evidence
P5	DF-hybrid vs. DF-typed-only	Incremental value of raw evidence above typed-only
P6	DF-typed-only vs. B3	Structured typed memory against reflection/vector anchor

Appendix

Table 19: Frozen V2 primary S3 comparison family.

Falsifier	Frozen threshold or claim rule	V2 value
B0 headroom	B0 S3 pass rate $\leq 0.40$ preferred; $\leq 0.50$ maximum acceptable.	Overall B0 $21/180=0.117$ passes; C9 $12/12$ and C10 $6/6$ B0 headroom fail.
Useful memory precision	DF-hybrid $\geq 0.45$ and no more than 0.15 below B5.	FAIL: DF-hybrid =0.264; B5=0.556.
Seed spread	DF-hybrid max(seed) minus min(seed) $\leq 0.12$ preferred; $\leq 0.16$ maximum for stability claim.	PASS: rates $31/60,32/60,32/60$ ; spread $0.017$ .
Regression and repeated error	DF-hybrid repeated-error rate $\leq$ B5 and $\leq 0.15$ ; regression-after-update $\leq 0.08$ .	MIXED/FAIL: repeated error 0.142 passes relative to B5 0.225 and threshold 0.15; regression-after-update 0.106 fails.
Strict-hybrid recovery	If included, DF-strict-hybrid S3 pass rate $\geq 0.60$ and no more than 0.12 below DF-hybrid .	FAIL: $78/180=0.433$ .
Construct dependence	General memory-hygiene claim requires nonnegative P1 direction on anti-hoarding aggregate and on C7.	No general memory-hygiene claim is made; primary P1 is null and F7 is false.

Appendix

Table 20: Frozen V2 falsifiers and claim constraints with resolved v2 values from the canonical analyzer bundle.

Construct	Construct name	One-line definition	Pure-trap criterion	Anti-hoarding?	v2 trap count
C1	Verbatim retention	Preserve a non-inferable exact token, format, or rule from one prior event and reproduce it when relevant.	A single prior event contains the decisive arbitrary token or format, and storing that event is necessary and sufficient.	No	18
C2	Retrieval precision under interference	Select the one relevant stored fact among many similar stored facts under a fixed read-token budget.	The sequence injects many same-vocabulary distractor events, only one event is load-bearing, and the budget admits only a small subset.	Yes	6
C3	Staleness detection and supersession	Use the superseding fact and suppress the stale fact, even when the stale fact is more salient.	An older token or rule is repeated and lexically attractive, a later event supersedes it, and the oracle fails any use of the old value.	Yes	7
C4	Update propagation / regression avoidance	Propagate a changed fact to a derived later location that the superseding event did not directly patch.	The superseding event gives the new rule but not the S3 answer location; the oracle requires applying the rule consistently downstream.	No	1
C5	Scope discipline	Apply feedback inside its stated scope and refuse to apply it outside that scope.	The trap includes an in-scope check where memory must be used and an out-of-scope check where the same memory becomes a failure.	Yes	6
C6	Contradiction handling / provenance conflict	Resolve conflicting sources by an established authority or precedence rule rather than recency or arbitrary choice.	Two sources conflict without temporal order deciding the answer; a separate precedence rule determines which source the oracle accepts.	Yes	5

Appendix

Table 21: Construct taxonomy and frozen v2 design quota. “Anti-hoarding” records the preregistered allocation: C2, C3, C5, C6, C9, and C10 contribute

6+7+6+5+4+2=30

designed traps, exceeding the 20-of-60 floor. Individual C7 traps can also be tagged anti-hoarding when they meet the preregistered paraphrase-retention rule. This is allocation arithmetic, not an outcome guarantee: the observed C9 and C10 strata fail B0-headroom validity and do not support anti-hoarding or abstention claims.

Nonzero trap clusters $k$	Minimum split	Exact $p$	One discordant seed/trap	Net margin	All three seeds discordant/trap
20	15–5	0.041389	$20/180=0.111$	$10/180=0.056$	$60/180=0.333$ , net $30/180=0.167$
30	21–9	0.042774	$30/180=0.167$	$12/180=0.067$	$90/180=0.500$ , net $36/180=0.200$
40	27–13	0.038477	$40/180=0.222$	$14/180=0.078$	$120/180=0.667$ , net $42/180=0.233$
50	33–17	0.032839	$50/180=0.278$	$16/180=0.089$	$150/180=0.833$ , net $48/180=0.267$
60	39–21	0.027340	$60/180=0.333$	$18/180=0.100$	$180/180=1.000$ , net $54/180=0.300$

Appendix

Table 22: Equal-weight sign-test thresholds for the 60-trap, 180-cell v2 design. The cell rates are descriptive translations of trap-level splits, not the primary independent sample size.

Nonzero clusters $k$	Rejection threshold	Power at 0.65	Power at 0.70	Power at 0.75
20	$\geq 15/20$	0.245	0.416	0.617
30	$\geq 21/30$	0.358	0.589	0.803
40	$\geq 27/40$	0.441	0.703	0.897
50	$\geq 33/50$	0.506	0.782	0.945
60	$\geq 39/60$	0.559	0.838	0.970

Appendix

Table 23: Frozen approximate power under the equal-weight trap-level sign model, parameterized by the true probability that a nonzero trap favors the reference probe.

Symbol or term	Meaning
$A$	Software-engineering agent under evaluation.
$R$	Repository for a task or sequence.
$T$	Task prompt or task instance.
$S_{i}$	The $i$ th bounded session in a multi-session sequence. In the memory-hygiene objective, $S_{i}\subseteq R_{i}$ is also used for the subset of retrieved memories that should have been suppressed; context disambiguates the two uses.
Sequence	Ordered list of sessions over one repository or a controlled family of related repositories.
$\tau$	Agent trajectory, $\tau=(\mathrm{meta},e_{1},e_{2},\ldots,e_{n},\mathrm{outcome})$ .

Appendix

Table 24: Base notation from the problem formulation and method sections.

Read-score symbol	Meaning
$J(T_{q},T_{m})$	Tag-overlap similarity between task-context tags and memory tags.
$F(q,m)$	File-scope compatibility score.
$Y(q,m)$	Symbol-scope compatibility score.
$K(q,m)$	Task-scope compatibility score.
$\pi_{\mathrm{type}}(m)$	Condition-specific memory-type prior.
$c_{m}$ , $u_{m}$ , $v_{m}$ , $p_{m}$	Confidence, utility, human-verification, and provenance-strength metadata.

Appendix

Table 25: Retrieval-gate notation from the method section.

Term	Definition
S1, S2	Setup or reinforcement sessions that create prior sequence state and memory evidence before the scored trap.
S3	Trap session. In the analyzer, S3 means oracle_id ends with -s3 or ordinal == 3 . The reported headline outcome is S3 pass_at_1 .
Warmup	Valid non-S3 records. Grouped runs can contribute warmup ordinals beyond 1 and 2; warmup is not the S3 headline outcome.
pass_at_1	Result-record field used by scripts/analyze_confirmatory.py for S3 outcomes.
Pass@1	Fraction of executable sessions whose first completed attempt passes the configured verification suite. The paper states that Pass@1 excludes timeout-then-pass records.
final_passed	Final executable oracle result for a record.

Appendix

Table 26: Session, outcome, and aggregation terms used in the fold.

Code	Canonical label	Definition
B0	No external memory	No external memory reads or writes.
B1	Raw episodic retrieval	Retrieves raw prior trajectories.
B2	Vector trace retrieval	Deterministic token-vector retrieval over prior trace chunks.
B3	Reflection-only vector retrieval	Free-form verbal reflection lessons after sessions, retrieved by vector similarity.
B4	Untyped summary memory	Deterministic untyped session summaries.
B5	Verbatim event-memory substitute	Offline instance-memory substitute; one deterministic verbatim injected-event memory per trajectory; not stock Mem0.

Appendix

Table 27: Condition-code glossary from the condition contract and artifact appendix.

Designed-not-run ablation	Meaning in this fold
A1	No typed consolidation.
A3	No causal failure extraction.
A7	Similarity-only retrieval.
A8	No provenance gate.
A10	No raw episodic fallback at read time.
A12	No scope fields.

Appendix

Table 28: Ablation labels mentioned as designed-but-not-run in the reported fold.

Term	Definition
R1	Adds a verbatim raw-evidence capsule: one EPISODIC memory per episode carrying injected_memory_event.content through the normal retrieval gate.
R1b	Exempts EPISODIC records from contradiction repair so raw evidence is not rewritten or deleted by repair.
R2	Excludes CONTRADICTION -type records from implementation reads through the existing allowed_types gate.
Typed consolidation	Writes candidate semantic-project, procedural, human-feedback, constraint, and episodic-derived memories with scope, provenance, confidence, utility, risk, staleness, tags, and write reason.
Causal failure extraction	Writes failure memories grounded in failed trajectories, command output, tests, diffs, logs, or feedback.
Counterfactual replay	Produces dream artifacts and replay-derived failure or suppression decisions, accepted only when grounded in executable evidence, raw trace evidence, or human-audited causal support.

Appendix

Table 29: Repair labels and sleep/read operators used by reference-probe variants.

ID	Name	Definition
C1	Verbatim retention	Preserve a non-inferable exact token, format, or rule from one prior event and reproduce it when relevant.
C2	Retrieval precision under interference	Select the one relevant stored fact among many similar stored facts under a fixed read-token budget.
C3	Staleness detection and supersession	Use the superseding fact and suppress the stale fact, even when the stale fact is more salient.
C4	Update propagation / regression avoidance	Propagate a changed fact to a derived later location that the superseding event did not directly patch.
C5	Scope discipline	Apply feedback inside its stated scope and refuse to apply it outside that scope.
C6	Contradiction handling / provenance conflict	Resolve conflicting sources by an established authority or precedence rule rather than recency or arbitrary choice.

Appendix

Table 30: Construct labels from the construct-taxonomy appendix. V2 construct-profile outcomes are reported separately by the canonical analyzer artifacts; this table defines the labels.

Metric or count	Definition	Fold handling	Direction
TaskSuccess	$\sum_{i}\mathrm{TaskSuccess}_{i}/N$ ; in prose, the fraction of sessions whose final patch satisfies the task oracle.	Task outcome	Higher
Pass@1 / pass_at_1	$\sum_{i}\mathrm{Pass@1}_{i}/N_{\mathrm{exec}}$ ; the analyzer uses S3 pass_at_1 as the headline outcome.	Primary S3 outcome	Higher
final_passed	Final executable oracle result.	Reported task field	Higher
UsefulMemory Precision	$\sum_{i}\|U_{i}\|/\sum_{i}\|R_{i}\|$ : useful admitted memories divided by admitted memories. The label-grounded scorer also describes the v2 form as useful retrieved memories divided by all retrieved memories for human-feedback opportunities.	Hygiene panel	Higher
HarmfulMemory Rate	$\sum_{i}\|H_{i}\|/\sum_{i}\|R_{i}\|$ : harmful admitted memories divided by admitted memories. The v2 scorer definition is harmful memory retrieved into active context and acted on.	Diagnostic in this fold; 0.000 for every ladder condition	Lower
RepeatedError Rate	Count of repeated error events divided by opportunities to avoid repetition. The v2 scorer definition is that the same labeled error appears before the active session and recurs in the active patch.	Hygiene panel	Lower

Appendix

Table 31: Task, hygiene, and cost metric glossary. Hygiene values in the confirmatory fold are deterministic offline re-scores from result records; the paper treats hygiene claims as diagnostic because several metrics are task-coupled, collinear, or judge-dependent.

Term	Definition
Pooled exact McNemar family	Secondary paired exact two-sided McNemar comparisons over S3 seed-trap cells, with Holm correction across P1–P6.
Supplemental McNemar comparison	A comparison rendered by the analyzer but not part of the pre-registered Holm-corrected McNemar family; it does not change the family’s adjusted $p$ -values.
Paired $n$	Number of S3 seed-trap cells present for both conditions in a comparison after the validity gate.
$b$	Discordant-pair count where the first condition passes and the second condition fails.
$c$	Discordant-pair count where the second condition passes and the first condition fails.
Discordant pairs	$b+c$ , the paired cells where the two conditions differ.

Appendix

Table 32: Paired and clustered sensitivity terminology from the confirmatory and clustered-sensitivity analyzers and Section 6.2 .

Memory Agent Benchmarks

Memory agent benchmarks evaluate how AI agents store, retrieve, update, and use information across interactions and changing tasks.

See all 164 papers

MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
Jul 1, 2026Zhishang Xiang, Zerui Chen, Yunbo Tang +5Agent Memory LLM Sycophancy
Xiamen University · Jilin University
DolphinBench: Mapping the Pareto Frontier of Agent Memory
Sep 21, 2026Soumil Rathi, Deshraj Yadav, Taranjeet SinghAgent Memory Persistent Memory for Language Models
Mem0
MemSecBench: Tracking Agent Memory Poisoning from Persistence to Consequence and Repair
Jul 29, 2026Xuanze Chen, Xukang Xie, Wentao Fu +3Agent Memory Poisoning Memory Agent Benchmarks
Institute of Cyberspace Security, Zhejiang University of Technology, Hangzhou 310023, China · Binjiang Institute of Artificial Intelligence, ZJUT, Hangzhou 310056, China · College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China

Jul 1, 2026J/K move · Enter open · S save

MemSyco-Bench: Benchmarking Sycophancy in Agent Memory

Zhishang Xiang, Zerui Chen, Yunbo Tang, Zhimin Wei+4

Xiamen University · Jilin University

Agent Memory LLM Sycophancy LLM Agent Evaluation AI Agent Benchmarks Memory Agent Benchmarks

Open paper AskPDF arXiv

Memory has emerged as a cornerstone of modern LLM-based agents, supporting their evolution from single-turn assistants to long-term collaborators. However, memory is not always beneficial: retrieved memories often induce a critical issue of sycophancy, causing agents to over-align with the user at the cost of factual accuracy or objective reasoning. Despite this emerging risk, existing memory benchmarks primarily evaluate whether memories are correctly stored, retrieved, or updated, while overlooking how retrieved memories influence downstream reasoning and decision-making. To bridge this gap, we propose MemSyco-Bench, a comprehensive benchmark for evaluating memory-induced sycophancy in agent systems. MemSyco-Bench measures when memory should influence a decision and how valid memory should be used. Specifically, it covers five tasks that assess whether agents can reject memory as factual evidence, respect its applicable scope, resolve conflicts between memory and objective evidence, track memory updates, and use valid memory for personalization. All related resources are collected for the community at https://github.com/XMUDeepLIT/MemSyco-Bench.

Abstract

Memory Agent Benchmarks

Explore similar work

MemSyco-Bench: Benchmarking Sycophancy in Agent Memory

DolphinBench: Mapping the Pareto Frontier of Agent Memory

MemSecBench: Tracking Agent Memory Poisoning from Persistence to Consequence and Repair