cs.AISep 25, 2026

A Benchmark and Diagnostic Study of Epistemic Admission in Shared Agent Memory

Authors: Xiaoyang Li, Yiqi Wang, Chencheng Zhu, KE XU, Wencheng Yang, Zequn Sun, Pingan Song, Yiqun Duan, 

Organizations: University of Southern Queensland · University of New South Wales · Aikaier · Nanjing University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Evaluating claim admission in shared agent memory is challenging because repeated claims may be mistaken for independent evidence. An agent may copy or paraphrase a retrieved belief, while admitting a false claim exposes subsequent agents to it. To study this problem, we introduce the Correlated Promotion Benchmark (CPB), which evaluates whether candidate claims should be admitted to shared memory.CPB-Static constructs a frozen test split from publicly annotated sources with fixed gold actions. CPB-Live runs multi-agent teams over a shared store, records all writes and retrievals, and tracks source lineage defined by each scenario. A separate consumer answers from the store alone. We evaluate eight admission policies across four agent families. Our results show that policies which deduplicate sources reject many true claims alongside false ones, whereas policies preserving answer coverage admit nearly as many false claims as unrestricted sharing. Gating on declared source type reduces false adoption to 0.06--0.09, compared with 0.22--0.47 for other answering policies. Once an uncontested false belief enters memory, the consumer asserts it in 0.97--0.99 of probes across all families. No non-oracle policy consistently rejects false claims across verbatim copies, paraphrases, and paraphrases declared authoritative. These findings reveal the limitations of admission policies without access to source lineage.

Figures & tables

Figure 1: The instrument. Left: a policy chooses one of four actions for a candidate and its evidence, and only promote writes the store. Middle: CPB-Static collapses lineage, so three supports from one source count as one, and scores against a fixed gold rule with simulated propagation. Right: CPB-Live reads reach and exposure from the store’s log and adoption from the consumer’s answer.
 Qwen3.5-9BGemma-3-12BLlama-3.1-8BClaude Opus 5
PolicyDmg ↓\downarrowFA ↓\downarrowTA ↑\uparrowDmg ↓\downarrowFA ↓\downarrowTA ↑\uparrowDmg ↓\downarrowFA ↓\downarrowTA ↑\uparrowDmg ↓\downarrowFA ↓\downarrowTA ↑\uparrow
Private only0.0000.0000.0000.0000.0000.0000.0000.0000.0000.0000.0000.000
Share all1.0001.0001.0001.0001.0001.0001.0001.0001.0001.0001.0001.000
Majority vote1.0001.0000.9640.9850.6490.4590.9810.1080.1000.8640.6720.529
LLM judge1.0001.0001.0000.9850.9880.9750.4830.3250.3230.4720.4730.435
Governance rule0.1520.1610.4810.1370.1450.4630.1510.1370.1330.1120.1200.348
Table 1: CPB-Live, the 100 configurations of each family pooled. Dmg : damage as a share of share-all’s. FA and TA : false and true admissions as shares of the false and true candidates the policy saw. The last block is the two published systems of Section 5.2 .
Figure 2: False adoption and correctness on CPB-Live, the eight policies on the 100 configurations of each family. Values in Appendix E , where the diagnostic arms and the frontier judge over Qwen appear on their own pools.
Figure 3: Consumer answers with exactly one visible false belief, the nine policies’ probes on the pool of Table 5 , by family. Left bars show stores without a competing true belief, and right bars include one. Answers naming both values count as voicing the false value.
Figure 4: Dual-source copy regimes. Left: false admissions summed over the nine policies, by regime and family. Right: downstream adoption by policy and regime, pooled over the families’ configurations.
Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

PolicyStatic, 400 episodesLive
Private only, share all, majority vote, governance ruleyesfour families, 100 configurations
Independence vote, DEPEN, review all correlatedyesfour families, 100 configurations
Stage-6 cells of the cross, seven policiesnoQwen and Claude Opus 5, 30 configurations
Lineage oracle, semantic collapsenoQwen and Claude Opus 5, 70 dual-source configurations
Share all without the agents’ retrievalnoQwen and Claude Opus 5, 60 base configurations
Confidence threshold, unanimity voteyesno
Appendix
Table 2: Which policy ran on which half. Static is the frozen test split of 400 episodes, and Live is 100 committed configurations per agent family, composed as Section 4.2 states, plus the 30 stage-6 configurations on two families.
 Claude Opus 5
PolicyDmg ↓\downarrowFA ↓\downarrowTA ↑\uparrow
Independent-source vote0.2430.2500.108
DEPEN0.2240.2600.111
Review all correlated0.2800.3330.108
Appendix
Table 3: The three rules of the independence class where they separate. On the three open families they decide identically on every episode and share the independence row of Table 1 . Columns as in that table.
PolicyEp.RefuseHoldAdoptCorrAbsorb
Qwen3.5-9B
Share all1000.000.620.450.65–
Mem0 default write path1000.000.620.510.630.05
Governance rule1000.840.100.090.63–
Mem0 behind the governance rule1000.840.100.090.630.00
A-MemGuard consistency check1000.000.620.450.63–
Appendix
Table 4: Mem0 and A-MemGuard compared with reference policies by agent family. Ep. : configurations. Refuse : proportion of false candidates rejected. Hold : proportion of configurations with false memory at probe time. Adopt : false-answer rate. Corr : true-answer rate among scenarios with a true value. Absorb : proportion of Mem0 admissions merged into existing memories. Mem0 replaces the store under share-all or governance, and A-MemGuard supplies the admission check. Both use the family’s backbone.
 Qwen3.5-9BGemma-3-12BLlama-3.1-8BClaude Opus 5
PolicyAdopt ↓\downarrowCorr ↑\uparrowAdopt ↓\downarrowCorr ↑\uparrowAdopt ↓\downarrowCorr ↑\uparrowAdopt ↓\downarrowCorr ↑\uparrow
Private only0.000.000.000.000.000.000.000.00
Share all0.450.650.320.770.470.680.440.73
Majority vote0.450.650.340.620.450.770.410.62
LLM judge0.450.650.330.680.220.780.190.40
Frontier judge over Qwen0.200.56––––––
Appendix
Table 5: Downstream adoption on CPB-Live. The nine policies pool the 100 configurations of each family, the two diagnostic arms the 40 dual-source configurations of stages 4 and 5, and the frontier judge its 60 base configurations on Qwen. Adopt : the share of configurations in which the consumer asserted a false value when answering from shared memory. Corr : the share of configurations whose scenario carries a true value in which the consumer asserted it. The consumer runs on the family’s own backbone and receives no source of its own, so shared memory is its only information. Where a policy promotes nothing, both columns are zero by construction.
 Qwen3.5-9BGemma-3-12BLlama-3.1-8BClaude Opus 5
PolicyAdoptnormCorrnormAdoptnormCorrnormAdoptnormCorrnormAdoptnormCorrnorm
Private only0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00
Share all0.450.470.650.670.320.320.770.780.470.470.680.680.440.430.730.77
Majority vote0.450.470.650.670.340.350.620.630.450.450.770.770.410.410.620.63
LLM judge0.450.470.650.670.330.330.680.700.220.220.780.780.190.210.400.45
Frontier judge over Qwen0.200.200.560.58––––––––––––
Appendix
Table 6: Downstream adoption under the rule of record and under the robustness matcher, on the pools of Table 5 , whose Adopt and Corr columns repeat here. norm marks the same quantity after answer and gold value are mapped to a canonical form by a fixed table, so that a fraction, its percentage and its hyphenated spelling count as one value and a leading article is ignored. Every difference between a plain column and its norm column is an answer the rule of record graded as other because the value was stated in another form.
Visible false beliefsTrue belief also visibleProbesAsserted the false value
0no17290.01
0yes7070.00
1no5530.98
1yes1600.15
2no220.95
2yes80.00
Appendix
Table 7: Adoption as a function of what shared memory holds when the consumer is asked, pooled over policies and agent families. A competing true belief is strongly protective, and the protection weakens as false beliefs accumulate.
PolicyQwen3.5-9BGemma-3-12BLlama-3.1-8BClaude Opus 5
Private only1.00 [62]1.00 [62]1.00 [67]1.00 [48]
Share all0.00 [62]0.00 [81]0.00 [735]0.00 [56]
Majority vote0.00 [62]0.35 [94]0.89 [604]0.33 [64]
LLM judge0.00 [62]0.01 [81]0.68 [345]0.53 [55]
Frontier judge over Qwen0.51 [35]–––
Governance rule0.84 [62]0.85 [62]0.86 [73]0.88 [50]
Appendix
Table 8: Refusal conditioned on a false candidate arriving: of the false candidates a policy was asked to decide, false by the assertion grade, the share it did not promote. The denominator is in brackets and is each policy’s own stream over the pool of Table 5 . Conditioning removes the families’ difference in verbosity and not the difference between the streams the policies faced. Private only and Share all bound the range by construction.
Figure 5: Refusal among false candidates on each policy’s own stream, the eight policies on the 100 configurations of each family. Bars distinguish agent families, and Table 8 gives the denominators and the rows of the diagnostic arms and the frontier judge over Qwen on their own pools.
PolicyI-CORRD-CONCmixedCOPYINDPARAPARA-AUTHTCOPYTPARAFIND
Private only0 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 0–––
Share all50 ⋅\cdot 10 ⋅\cdot 1072 ⋅\cdot 18 ⋅\cdot 2025 ⋅\cdot 5 ⋅\cdot 1540 ⋅\cdot 10 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 1036 ⋅\cdot 9 ⋅\cdot 040 ⋅\cdot 10 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 100 ⋅\cdot 0 ⋅\cdot 1040 ⋅\cdot 10 ⋅\cdot 0
Majority vote50 ⋅\cdot 10 ⋅\cdot 1072 ⋅\cdot 18 ⋅\cdot 1825 ⋅\cdot 5 ⋅\cdot 1540 ⋅\cdot 10 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 1036 ⋅\cdot 9 ⋅\cdot 040 ⋅\cdot 10 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 100 ⋅\cdot 0 ⋅\cdot 1040 ⋅\cdot 10 ⋅\cdot 0
LLM judge50 ⋅\cdot 10 ⋅\cdot 1072 ⋅\cdot 18 ⋅\cdot 2025 ⋅\cdot 5 ⋅\cdot 1540 ⋅\cdot 10 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 1036 ⋅\cdot 9 ⋅\cdot 040 ⋅\cdot 10 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 100 ⋅\cdot 0 ⋅\cdot 1040 ⋅\cdot 10 ⋅\cdot 0
Governance rule0 ⋅\cdot 0 ⋅\cdot 100 ⋅\cdot 0 ⋅\cdot 180 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 100 ⋅\cdot 0 ⋅\cdot 040 ⋅\cdot 10 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 040 ⋅\cdot 10 ⋅\cdot 0
Independent-source vote0 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 1036 ⋅\cdot 9 ⋅\cdot 040 ⋅\cdot 10 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 1040 ⋅\cdot 10 ⋅\cdot 0
Appendix
Table 9: Qwen3.5-9B: damage, false writes and true writes per scenario class, the totals of Table 1 attributed to the classes that produced them. I-CORR, D-CONC and mixed are the 60 base configurations. COPY is the dual-source set with a verbatim copy of a false claim and IND the one with an independent restatement of a true claim. PARA is a reworded copy of a false claim as web text and PARA-AUTH the same copy typed authoritative. TCOPY and TPARA are a verbatim and a reworded copy of a true claim and FIND an independent restatement of a false claim. These three classes are outside the pool of Table 1 .
PolicyI-CORRD-CONCmixedCOPYINDPARAPARA-AUTH
Private only0 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 0
Share all50 ⋅\cdot 10 ⋅\cdot 5372 ⋅\cdot 35 ⋅\cdot 10920 ⋅\cdot 6 ⋅\cdot 5340 ⋅\cdot 10 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 1040 ⋅\cdot 10 ⋅\cdot 040 ⋅\cdot 10 ⋅\cdot 0
Majority vote50 ⋅\cdot 10 ⋅\cdot 1572 ⋅\cdot 18 ⋅\cdot 2220 ⋅\cdot 4 ⋅\cdot 1540 ⋅\cdot 10 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 1036 ⋅\cdot 9 ⋅\cdot 040 ⋅\cdot 10 ⋅\cdot 0
LLM judge50 ⋅\cdot 10 ⋅\cdot 5372 ⋅\cdot 35 ⋅\cdot 8020 ⋅\cdot 6 ⋅\cdot 5340 ⋅\cdot 10 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 1036 ⋅\cdot 9 ⋅\cdot 040 ⋅\cdot 10 ⋅\cdot 0
Governance rule0 ⋅\cdot 0 ⋅\cdot 100 ⋅\cdot 0 ⋅\cdot 180 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 100 ⋅\cdot 0 ⋅\cdot 036 ⋅\cdot 9 ⋅\cdot 0
Independent-source vote0 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 1028 ⋅\cdot 7 ⋅\cdot 036 ⋅\cdot 9 ⋅\cdot 0
Appendix
Table 10: Gemma-3-12B: damage, false writes and true writes per scenario class, the totals of Table 1 attributed to the classes that produced them. I-CORR, D-CONC and mixed are the 60 base configurations. COPY is the dual-source set with a verbatim copy of a false claim and IND the one with an independent restatement of a true claim. PARA is a reworded copy of a false claim as web text and PARA-AUTH the same copy typed authoritative.
PolicyI-CORRD-CONCmixedCOPYINDPARAPARA-AUTH
Private only0 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 0
Share all40 ⋅\cdot 61 ⋅\cdot 16180 ⋅\cdot 170 ⋅\cdot 23125 ⋅\cdot 76 ⋅\cdot 20440 ⋅\cdot 132 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 14340 ⋅\cdot 182 ⋅\cdot 040 ⋅\cdot 114 ⋅\cdot 0
Majority vote35 ⋅\cdot 7 ⋅\cdot 1980 ⋅\cdot 20 ⋅\cdot 2325 ⋅\cdot 5 ⋅\cdot 1440 ⋅\cdot 10 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 1040 ⋅\cdot 12 ⋅\cdot 040 ⋅\cdot 11 ⋅\cdot 0
LLM judge20 ⋅\cdot 5 ⋅\cdot 8232 ⋅\cdot 26 ⋅\cdot 3620 ⋅\cdot 17 ⋅\cdot 428 ⋅\cdot 15 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 2216 ⋅\cdot 25 ⋅\cdot 032 ⋅\cdot 24 ⋅\cdot 0
Governance rule0 ⋅\cdot 0 ⋅\cdot 100 ⋅\cdot 0 ⋅\cdot 200 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 100 ⋅\cdot 0 ⋅\cdot 040 ⋅\cdot 10 ⋅\cdot 0
Independent-source vote0 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 1028 ⋅\cdot 7 ⋅\cdot 040 ⋅\cdot 10 ⋅\cdot 0
Appendix
Table 11: Llama-3.1-8B: damage, false writes and true writes per scenario class, the totals of Table 1 attributed to the classes that produced them. I-CORR, D-CONC and mixed are the 60 base configurations. COPY is the dual-source set with a verbatim copy of a false claim and IND the one with an independent restatement of a true claim. PARA is a reworded copy of a false claim as web text and PARA-AUTH the same copy typed authoritative.
PolicyI-CORRD-CONCmixedCOPYINDPARAPARA-AUTHTCOPYTPARAFIND
Private only0 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 0–––
Share all50 ⋅\cdot 10 ⋅\cdot 1940 ⋅\cdot 12 ⋅\cdot 7925 ⋅\cdot 6 ⋅\cdot 2124 ⋅\cdot 6 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 2536 ⋅\cdot 9 ⋅\cdot 039 ⋅\cdot 13 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 120 ⋅\cdot 0 ⋅\cdot 1040 ⋅\cdot 14 ⋅\cdot 0
Majority vote45 ⋅\cdot 9 ⋅\cdot 1040 ⋅\cdot 10 ⋅\cdot 1420 ⋅\cdot 4 ⋅\cdot 1224 ⋅\cdot 6 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 932 ⋅\cdot 8 ⋅\cdot 024 ⋅\cdot 6 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 80 ⋅\cdot 0 ⋅\cdot 936 ⋅\cdot 9 ⋅\cdot 0
LLM judge35 ⋅\cdot 9 ⋅\cdot 1244 ⋅\cdot 12 ⋅\cdot 2815 ⋅\cdot 3 ⋅\cdot 140 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 104 ⋅\cdot 1 ⋅\cdot 03 ⋅\cdot 1 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 123 ⋅\cdot 8 ⋅\cdot 0
Governance rule0 ⋅\cdot 0 ⋅\cdot 40 ⋅\cdot 0 ⋅\cdot 160 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 110 ⋅\cdot 0 ⋅\cdot 024 ⋅\cdot 6 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 036 ⋅\cdot 9 ⋅\cdot 0
Independent-source vote0 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 1024 ⋅\cdot 6 ⋅\cdot 028 ⋅\cdot 7 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 00 ⋅\cdot 0 ⋅\cdot 1032 ⋅\cdot 8 ⋅\cdot 0
Appendix
Table 12: Claude Opus 5: damage, false writes and true writes per scenario class, the totals of Table 1 attributed to the classes that produced them. I-CORR, D-CONC and mixed are the 60 base configurations. COPY is the dual-source set with a verbatim copy of a false claim and IND the one with an independent restatement of a true claim. PARA is a reworded copy of a false claim as web text and PARA-AUTH the same copy typed authoritative. TCOPY and TPARA are a verbatim and a reworded copy of a true claim and FIND an independent restatement of a false claim. These three classes are outside the pool of Table 1 .
 VerbatimRewordedIndependent
Policyfalse ↓\downarrowtrue ↑\uparrowfalse ↓\downarrowtrue ↑\uparrowfalse ↓\downarrowtrue ↑\uparrow
Qwen3.5-9B
Private only0.00–0.00––0.00
Share all1.001.000.901.001.001.00
Majority vote1.001.000.901.001.001.00
LLM judge1.001.000.901.001.001.00
Appendix
Table 13: Writes of the target proposition on CPB-Live by the form of the second source and the truth of the claim. Each cell is the share of configurations of that class in which the policy wrote the proposition. A verbatim pair is a claim and a copy of itself under another source id, a reworded pair restates the claim in other words from the same lineage, and an independent pair adds a genuinely independent restatement. The false cells come from the COPY, PARA and FIND classes and the true cells from the TCOPY, TPARA and IND classes. A lower false share and a higher true share are better. The judge row is each family’s own model. A missing cell is a class the family has not run.
ArmEp.Cand.False cand.WritesHoldDamageDecl.Adopt
Qwen3.5-9B
Share all601.30.6330.5514700.30
Share all, no shared retrieval601.70.6330.5514700.25
Claude Opus 5
Share all608.30.5280.421151890.27
Share all, no shared retrieval602.20.5270.4512200.23
Appendix
Table 14: The shared memory writeback ablation on the stage 2 configurations. Share all is run once with every emitting agent retrieving shared memory and once with that retrieval switched off while the consumers still read the store. Ep. counts configurations. Cand. is the number of candidates per configuration and False cand. the number of false candidates per configuration. Writes is the number of false beliefs written. Hold is the proportion of configurations holding a false belief. Damage is the number of consumer reads that touched a false belief. Decl. is the number of candidates that declared a shared belief as their basis. Adopt is the false answer rate at the probe. Families that did not run the ablated arm are not listed.
A \\backslash Graderasserts falseasserts trueneithertotal
asserts false230427
asserts true042244
neither114749
total244353120
Appendix
Table 15: The assertion grade against two annotators on 120 pairs, 30 per family. Left: annotator A’s reviewed label by row and the grader’s by column. Middle: pairs on which the two agree, per family. Right: agreement and Cohen’s kappa for every pairing of labels, blind and reviewed; the blind rows are the independent validation. Of the pairs the grader called false, 1 of 24 are not assertions to the annotator, a rate whose Wilson interval runs from 0.01 to 0.20.
A \\backslash rulefalse valuetrue valuebothabstainsothertotal
false value240001034
true value02410631
both00220022
abstains00124126
other000077
total2424242424120
Appendix
Table 16: The consumer probe’s grading rule against two annotators on 120 answers, six per family and per category of the rule. Left: annotator A’s reviewed label by row and the rule’s by column. Middle: answers on which the two agree, per family. Right: agreement and Cohen’s kappa for every pairing of labels, blind and reviewed; the blind rows are the independent validation, and A’s blind shortfall is the 24 answers reading unknown that A had labelled by the stored value.
FamilyGradeTraceContentMissedUncitedAdmittedDmg gradeDmg traceDmg both
Qwen3.5-9B626258000263263263
Gemma-3-12B818751000262275275
Llama-3.1-8B735176818676666676265270270
Claude Opus 556324198676767214275275
Appendix
Table 17: Assertion grading versus historical scoring rules. Top: share-all false-candidate counts under Grade, Trace and Content, with damage under Grade, Trace and the union of Trace and Content, written Both. Missed denotes content-detected cases absent from the trace, Uncited those with neither citations nor declared dependencies, and Admitted those promoted. Bottom: historical refusal rates with denominators in brackets. Current refusal appears in Figure 5 .
JudgeAgentsDmg ↓\downarrowCeilingFW ↓\downarrowTW ↑\uparrowWrites/cand.Order sens.Parse fail.
Qwen3.5-9BQwen3.5-9B147147334578/780.000
Gemma-3-12BGemma-3-12B14214251186253/2580.700
Llama-3.1-8BLlama-3.1-8B7214548160251/20020.200
Claude Opus 5Qwen3.5-9B78147173451/880.200
Claude Opus 5Claude Opus 5941152454229/3590.802
Appendix
Table 18: Five LLM judges on CPB-Live over the 60 base configurations. Ceiling is share-all’s damage in the same agent family.
Judge backboneFPR ↓\downarrowFPR on MW ↓\downarrowFP/FN [CI]P/R/K/AABSTAIN rec.Acc.
Qwen3.5-9B0.2280.228 [0.17, 0.29]1.60 [1.02, 2.52]0.44/0.56/0.00/0.0030.000.427
Gemma-3-12B0.2850.285 [0.23, 0.35]2.92 [1.89, 4.89]0.50/0.40/0.10/0.0000.000.448
Llama-3.1-8B0.2280.228 [0.18, 0.28]1.33 [0.87, 2.03]0.42/0.27/0.31/0.0000.000.477
Claude Opus 50.0770.077 [0.05, 0.11]0.23 [0.13, 0.36]0.22/0.24/0.54/0.0000.000.412
Appendix
Table 19: The same judge prompt on four backbones on the frozen Static test split of 400 episodes. FPR is the false-promotion rate, MW the majority-wrong subset, the episodes on which a support-minus-contradiction proxy of the majority vote disagrees with the gold action, which on the frozen test split is every episode whose gold action is not promote, so FPR on MW equals FPR, FP/FN false promotions over false withholdings, P/R/K/A the shares of the four actions, and the last two columns the abstain recall on the high-stakes family, where the governance rule scores 0.10, and the accuracy.
ModelReferencennκ\kappa [CI]Bias [CI]FP/FN
Qwen3.5-9Bblind-100 (annotator A)1000.36 [0.17, 0.54]2.86 [1.25, 10.00]20/7
Qwen3.5-9BA/B-agreed subset910.55 [0.32, 0.74]1.80 [0.57, 9.00]9/5
Claude Opus 5blind-100 (annotator A)1000.40 [0.21, 0.56]5.25 [2.17, 24.00]21/4
Claude Opus 5A/B-agreed subset910.65 [0.44, 0.83]1.75 [0.50, 9.00]7/4
Qwen3.5-9Bgold, CPB-Static test400–1.60 [1.02, 2.52]–
Gemma-3-12Bgold, CPB-Static test400–2.92 [1.89, 4.89]–
Appendix
Table 20: Over-assertion of the judgment layer. Top: the same-proposition judgment against the human annotators, where the bias is false same-judgments over false different-judgments, so a value above one says same more often than the human. Bottom: the promotion direction against gold on CPB-Static, false promotions over false withholdings, so a value above one over-promotes.
#ComparisonQwen (confirmatory)Holm ppGemmaLlamaOpus 5
2Judge vs Majority vote, FPR on the majority-wrong subset0.228 vs 0.000; diff 0.228 [0.169, 0.288]0.0000.285 vs 00.228 vs 00.077 vs 0
3Judge direction, FP/FN of PROMOTE ( >> 1 over-promotes)1.60 [1.02, 2.52]; supportedby CI2.92 [1.89, 4.89]1.33 [0.87, 2.03]0.23 [0.13, 0.36]
4Governance rule vs judge, ABSTAIN recall on R-STAKE0.10 vs 0.00; diff 0.10 [0.03, 0.18]0.0120.10 vs 0.000.10 vs 0.000.10 vs 0.00
Appendix
Table 21: The three pre-registered comparisons that involve the judge, the second, third and fourth of the six, Qwen3.5-9B confirmatory with Holm-adjusted pp over the six, and the descriptive replication on three further judge backbones.
ArtifactFrozen, UTCBehaviour hash, first 16 hex
cpb-static-test-split2026-09-09 14:27:023cc2607d164831ad
gold-action-mapping-rule2026-08-12 03:49:162d6e8d476bfb9f87
pre-registered-comparisons-p142026-08-31 12:38:588ae0f8bc5504b040
robustness-anchor-p122026-08-31 14:27:2381cb4354f5c1283e
semantic-gate-p4b2026-08-17 13:53:22d5b5b9d01237bc89
Appendix
Table 22: The pre-registration freezes. Each artifact was hashed and its freeze file committed at the UTC time shown, before the runs it governs; the behaviour hash binds the artifact’s content and the source hash the files that produce it. The freeze files and the tool that verifies them ship with the code release.
More in this topic

Memory Agent Benchmarks

Memory agent benchmarks evaluate how AI agents store, retrieve, update, and use information across interactions and changing tasks.

See all 164 papers

Explore similar work

CardsList
Jun 17, 2026

GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents

Memory benchmarks for LLM agents largely assume single-user settings, leaving shared assistants for hospitals, workplaces, campuses, and households understudied. In these deployments, multiple principals write to a common memory pool and query it under different roles, scopes, and relationships, so memory quality requires governance as well as recall. We introduce GateMem, a benchmark for multi-principal shared-memory agents. GateMem jointly evaluates utility for legitimate long-horizon requests with state updates, access control across contextual authorization boundaries, and agent-facing active forgetting after explicit deletion requests. It spans medical, office, education, and household domains, with long-form multi-party episodes, incremental memory injection, hidden checkpoints, structured judging, and leak-target annotations. Across diverse baselines and backbone models, no method simultaneously achieves strong utility, robust access control, and reliable forgetting. Long-context prompting often yields the best governance score at high token cost, while retrieval-based and external-memory methods reduce cost yet still leak unauthorized or deleted information. These results show current memory agents remain far from reliable shared institutional deployment.

Zhe Ren, Yibo Yang, Yimeng Chen +7

School of Artificial Intelligence, Jilin University · 2Shanghai Jiao Tong University · 3King Abdullah University of Science and Technology (KAUST) +2

LLM Agent MemoryAgent Memory GovernanceAI Agent SecurityAI Agent Security BenchmarksMemory Agent BenchmarksAI Agent Governance+3
AskPDF
Aug 20, 2026

Beyond Memory Majority: Latent-Source Reasoning for Multi-Agent Memory Arbitration

Long-term multi-agent systems continuously accumulate the memories produced by different agents. Existing memory methods typically treat retrieved memories as independent evidence and combine them through voting or weighting. However, this independence assumption often fails in multi-agent settings: memories written by different agents may inherit the same upstream source or shared bias, causing correlated evidence to be repeatedly counted and creating a false majority. We term this failure mode Memory Correlation Bias. To address the issue, we propose the Correlation-Aware Memory Arbitration (CAMA) framework that jointly decouples retrieved memories and recovers missing independent evidence. We model the retrieved memories as query-conditioned evidence groups and combine neural dependency inference with provenance-based symbolic priors to estimate the effective number of independent evidence sources, thereby preventing correlated memories from forming a false majority. Since critical independent evidence may be absent from the initial retrieval set, CAMA further learns a sequential recovery policy that actively retrieves alternative evidence or traces upstream sources before making the final decision, aiming to recover sufficient independent evidence for reliable arbitration while minimizing retrieval cost. Experiments on multiple benchmarks demonstrate the superiority of our method over the state-of-the-art baseline methods, suppressing false majorities induced by correlated memories.

Chenchen Lin, Wenhao Yuan, Xuehe Wang +1

Department of Electrical and Computer Engineering, The University of Hong Kong · School of Artificial Intelligence, Sun Yat-sen University · Southern Marine Science and Engineering Guangdong Laboratory (Zhuhai) +1

Agent Memory ManagementInformation Retrieval
AskPDF
Oct 5, 2026

The Right Memory in the Wrong Context: Verifying Retrieval Admissibility in Long-Term Agent Memory

Long-term-memory agents can retrieve relevant information that is inadmissible for the current request because it belongs to another principal, violates policy, or reflects an incompatible lifecycle state. Recall and final-answer accuracy do not reveal this: a route can appear safe by missing required evidence, while a correct answer may follow inadmissible prompt exposure. We introduce a retrieval-admissibility verification framework that assigns each memory-query pair one of three statuses (admissible, inadmissible, or unresolved), compares routes at matched required-evidence recall with bounds for unresolved cases, and tracks memory IDs through prompt exposure while linking exposure to target-level disclosure. We evaluate its stages on separate, non-pooled populations. A post-hoc top-20 reanalysis of frozen rankings from two public long-term-memory benchmarks, RHELM and MemOps, covers 3,767 queries. All released anchors lie within trusted query namespaces; with within-namespace scores unchanged, off-namespace filtering cannot lower their ranks. Top-20 anchor recall increases from 0.432 to 0.533, 80% recall feasibility from 0.237 to 0.311, and exact similarity evaluations decrease by 98.3%. In a frozen 72-case development diagnostic, a released-metadata reference preserves required evidence, whereas neither text-only verifier detects violations under the 1% required-anchor false-denial limit. Across 1,523 paired benchmark-native cases, namespace routing is associated with judged-accuracy gains of 0.053-0.068 across three readers; recall also changes, so this comparison is observational. In 16 controlled exposure scenarios, only one of four reader-specific 95% confidence intervals excludes zero for relevant-inadmissible literal disclosure (+0.156, 95% CI [0.031, 0.312]). Results motivate separate verification of candidate support, admissibility, prompt exposure, and answer disclosure.

Zi Wang, Xingqiao Wang, Emmanuel Addai +2

University of Arkansas at Little Rock

Memory Agent BenchmarksLong-Term Conversational MemoryAgent Memory Management
AskPDF