Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

Anouar Imel; Imad Charradi; Abderrahman Sghir; Khalil El Khammar

cs.AIOct 27, 2026

Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

Authors: Anouar Imel, Imad Charradi, Abderrahman Sghir, Khalil El Khammar

Organizations: School of Information and Communication Engineering, University of Electronic Science and Technology of China (UESTC) · Xidian University · Nanjing University of Aeronautics and Astronautics · School of Artificial Intelligence, Nanjing University of Information Science and Technology

Abstract

When an LLM group's discussion is compressed into a short memory for a final decision maker, what should the memory keep? Two natural heuristics keep the rarest claims or the claims relevant to the decision criteria. On hidden-profile tasks both look attractive, because how widely a fact is held is confounded with whether it helps. We introduce CrossProfile, a diagnostic benchmark of 60 templates with automatically verified answer keys, in which the ownership of supporting, decoy, and irrelevant facts is crossed while their text is held fixed (480 tasks). Across Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models, both heuristics fail in predictable places. Selecting by relevance readmits decoys, because decoys are on topic: even a perfect relevance filter loses 23 to 31 points to rarest-first selection in the classic hidden-profile configuration. Rarest-first selection is no better than random selection on average and wins only where the rare facts happen to be the useful ones. In both cases accuracy follows the balance of evidence the memory keeps, not the policy that kept it. The decision model adds a third failure, a form of social proof: it gives extra weight to facts attributed to many agents. Removing the attribution tags, or reassigning them at random with line order fixed, raises accuracy in the classic configuration by 13 to 24 points. In natural discussions the evidence is lost before any memory acts: the complete transcript yields lower accuracy than no notes at all.

Figures & tables

Figure 1: Rarity and decision usefulness. Fact text stays fixed while the ownership of support and decoy facts changes; here irrelevant facts stay private. Filled dots mark the agents holding each fact, and the boxes on the right show what rarest-first selection prioritizes. (c) The three ownership factors are crossed. Decoys are true, criterion-relevant facts that favor a competing option. Role labels are never shown to the models.

Figure 2: Pipeline. Facts from template M01 (Table 4 ) are distributed among the agents; HiddenBench tasks have three or four agents and CrossProfile tasks have four. After the discussion, a memory policy compresses it under a token budget $B$ , either by summarizing the transcript (B3) or by selecting whole claims from a claim inventory (R1, R1-rel, B4, Random). A separate call with the same model weights answers from the task and the memory alone.

Figure 3: Who wins depends on who holds what. CrossProfile accuracy at

r=.6

by support and decoy ownership (each point averages the two irrelevant-fact assignments, 60 templates, three seeds). R1 and B4 are mirror images; R1-rel varies little across cells but falls well below R1 in the classic hidden-profile column (shaded: private support, shared decoys), where even the all-facts memory (dashed) drops. Per-cell intervals for Qwen and Llama are in Appendix A.2 .

Policy	Acc.	Private	Shared	Gap	Tokens
Qwen2.5-7B-Instruct
B3	51.3	–	–	–	108
B4	22.1	26.3	85.8	59.5	136
R1	88.2	98.3	12.0	$-86.3$	139
Random	42.1	52.2	58.4	6.2	136
Llama-3.1-8B-Instruct

Table 1: Controlled HiddenBench at

r=.6

(65 tasks, three seeds): decision accuracy (%), exact-copy retention of private and shared facts (%), retention gap (shared minus private, points), and median memory length (tokens).

Memory	Qwen	Llama
Evidence summary (B3)	72.0	58.7
First mention (B4)	61.0	61.5
Random	64.5	63.4
Rarest first (R1)	64.0	63.0
Relevance first (R1-rel)	72.6	74.6
All facts (unbudgeted)	73.1	72.6

Table 2: CrossProfile accuracy (%) at

r=.6

, averaged over 60 templates, eight ownership variants, and three seeds.

Figure 4: Criterion relevance is insufficient in the classic hidden-profile cell (private support, shared decoys, private irrelevant facts,

r=.6

). (a) Accuracy differences from R1 for R1-rel and for oracle relevance on Qwen, Llama, and DeepSeek-V4.1-Flash (Flash), with 95% template-bootstrap intervals. (b) Mean number of selected facts by role on Qwen and Llama: relevance-first priority admits shared decoys.

Figure 5: Holder tags mislead the decision model. Accuracy in the classic cell (private support, shared decoys, private irrelevant facts) for the all-facts memory with and without holder tags, and for R1 at

r=.6

(60 templates, three seeds; Flash: DeepSeek-V4.1-Flash).

Model	Vote	REF	Empty	All	Discl.
Qwen	23.6	22.1	33.8	70.3	0.0 †
Llama	31.8	33.8	36.9	61.5	2.4 †
Flash	24.6	25.6	35.9	89.7	1.7

Table 3: Natural HiddenBench discussions (65 tasks, 195 discussions per model). Vote: accuracy of the group’s majority vote after two rounds. REF, Empty, All: decision accuracy from the full transcript, an empty memory, and all facts (%). Discl.: private facts stated in full in some message (%, string-matching heuristic). † Disclosure for Qwen and Llama from one seed (65 discussions). Flash: DeepSeek-V4.1-Flash.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Role	Fact text
f01	decoy	Hallum Diagnostics has passed every electrical safety inspection on its scanners.
f02	support	Hallum Diagnostics took nine days to answer urgent repair callouts last year.
f03	support	Ostrava Clinical passed every leakage test on its scanners during the last audit.
f04	support	Ostrava Clinical engineers reach any ward within four hours for repairs.
f05	irrelevant	Hallum Diagnostics has run an apprenticeship scheme for local school leavers since 2015.
f06	irrelevant	Ostrava Clinical’s headquarters was flooded in a burst pipe incident in 2019.

Appendix

Table 4: All facts of template M01 (hospital scanner procurement; correct option: Ostrava Clinical; decoy: Hallum Diagnostics). Roles are evaluator annotations and are never given to the models.

Model / ratio	H1	H2	H3	H3, shared irrelevant
Qwen / .4	$+65.3$ [60.1, 70.1]	$+13.1$ [9.7, 16.5]	$+20.8$ [16.0, 25.8]	$+8.1$ [3.9, 12.2]
Qwen / .6	$+64.6$ [60.3, 69.0]	$+5.0$ [1.2, 8.9]	$+8.5$ [3.5, 13.6]	$+8.7$ [4.6, 12.8]
Qwen / .8	$+35.3$ [30.3, 40.4]	$+2.4$ [ $-1.1$ , 5.7]	$+2.6$ [ $-1.7$ , 6.9]	$+2.9$ [ $-0.6$ , 6.4]
Llama / .4	$+67.4$ [62.6, 72.1]	$+6.4$ [2.4, 10.4]	$+15.3$ [11.2, 19.4]	$+12.5$ [9.0, 16.0]
Llama / .6	$+62.6$ [56.8, 68.2]	$+4.9$ [ $-0.3$ , 10.0]	$+15.6$ [11.1, 19.9]	$+7.6$ [4.2, 11.2]
Llama / .8	$+29.4$ [24.7, 34.2]	$-1.5$ [ $-5.3$ , 2.4]	$+3.8$ [0.3, 7.4]	$+3.2$ [0.1, 6.4]

Appendix

Table 5: CrossProfile contrasts (points) with 95% template-bootstrap intervals. Only H1–H3 at

r=.6

belong to the primary Holm family.

Figure 6: Primary contrasts at

r=.6

. H1: dependence of the R1 advantage over B4 on decoy ownership. H2: R1 accuracy with shared minus private irrelevant facts. H3: R1-rel minus R1 with private irrelevant facts. Points and whiskers show estimates and 95% template-bootstrap intervals. Filled markers are significant after Holm correction over H1–H3 within each model; Llama H2 is not.

	Qwen			Llama
Policy	$r=.4$	$r=.6$	$r=.8$	$r=.4$	$r=.6$	$r=.8$
B3	68.9	72.0	72.6	52.8	58.7	60.6
B4	51.5	61.0	68.0	50.5	61.5	70.9
Random	57.8	64.5	71.6	55.1	63.4	70.6
R1	54.6	64.0	71.9	53.6	63.0	72.6
R1-rel	69.0	72.6	74.7	67.5	74.6	76.0

Appendix

Table 6: CrossProfile accuracy (%) at all budget ratios.

Figure 7: CrossProfile accuracy across budget ratios (Table 6 ). Points are means over templates, variants, and seeds; horizontal lines mark the unbudgeted all-facts and empty controls.

S / D / I	R1	R1-rel	B4	Random
sh / sh / sh	67.2 [60.0, 74.4]	75.6 [68.9, 81.7]	67.2 [60.0, 74.4]	60.0 [52.2, 67.8]
sh / sh / pr	58.3 [50.6, 66.1]	68.9 [60.6, 76.7]	66.1 [57.8, 74.4]	62.2 [54.4, 69.4]
sh / pr / sh	35.0 [28.3, 41.7]	73.3 [66.1, 80.0]	92.2 [88.3, 96.1]	72.2 [64.4, 79.4]
sh / pr / pr	22.2 [14.4, 30.6]	67.2 [60.6, 73.9]	96.1 [92.8, 98.9]	76.1 [68.9, 83.3]
pr / sh / sh	92.2 [87.8, 96.1]	75.0 [68.3, 81.7]	13.9 [8.3, 20.0]	58.9 [51.7, 66.1]
pr / sh / pr	96.7 [93.3, 99.4]	74.4 [67.8, 80.6]	26.1 [19.4, 33.3]	53.9 [46.1, 61.7]

Appendix

Table 7: Qwen, copy policies: accuracy (%) by ownership cell at

r=.6

, with 95% template-bootstrap intervals. S/D/I: ownership of support, decoy, and irrelevant facts (sh: shared, pr: private).

S / D / I	B3	All facts
sh / sh / sh	72.2 [65.0, 79.4]	78.9 [71.7, 85.6]
sh / sh / pr	81.7 [76.1, 87.2]	75.0 [68.3, 81.7]
sh / pr / sh	86.1 [79.4, 91.7]	89.4 [83.9, 94.4]
sh / pr / pr	91.1 [86.7, 95.0]	91.1 [86.1, 95.6]
pr / sh / sh	46.7 [40.0, 53.9]	51.1 [42.2, 60.0]
pr / sh / pr	56.1 [47.2, 65.0]	54.4 [46.1, 62.8]

Appendix

Table 8: Qwen, summary and all-facts memories: accuracy (%) by ownership cell at

r=.6

, with 95% template-bootstrap intervals. Notation as in Table 7 .

S / D / I	R1	R1-rel	B4	Random
sh / sh / sh	63.9 [56.1, 71.7]	72.8 [66.7, 78.9]	62.8 [55.0, 70.6]	58.3 [50.0, 66.1]
sh / sh / pr	53.3 [45.0, 61.1]	75.6 [69.4, 81.7]	68.9 [61.7, 76.1]	59.4 [52.8, 66.1]
sh / pr / sh	40.0 [31.7, 48.9]	73.9 [67.2, 80.6]	91.7 [87.2, 95.6]	69.4 [61.7, 77.2]
sh / pr / pr	18.3 [12.2, 24.4]	78.3 [71.7, 84.4]	95.6 [92.2, 98.3]	72.2 [65.0, 79.4]
pr / sh / sh	90.0 [85.0, 94.4]	72.8 [65.0, 80.0]	10.6 [6.1, 15.0]	57.8 [49.4, 65.6]
pr / sh / pr	97.2 [93.9, 99.4]	77.2 [70.0, 83.9]	31.1 [23.9, 38.3]	62.2 [54.4, 70.0]

Appendix

Table 9: Llama, copy policies: accuracy (%) by ownership cell at

r=.6

, with 95% template-bootstrap intervals. Notation as in Table 7 .

S / D / I	B3	All facts
sh / sh / sh	63.3 [55.6, 70.6]	72.2 [64.4, 80.0]
sh / sh / pr	65.0 [56.7, 73.3]	75.6 [67.8, 83.3]
sh / pr / sh	71.1 [64.4, 77.8]	85.0 [77.8, 91.7]
sh / pr / pr	71.1 [62.2, 79.4]	85.0 [78.9, 90.6]
pr / sh / sh	38.3 [31.1, 45.6]	58.9 [50.0, 67.8]
pr / sh / pr	50.6 [43.3, 58.3]	55.6 [47.2, 63.3]

Appendix

Table 10: Llama, summary and all-facts memories: accuracy (%) by ownership cell at

r=.6

, with 95% template-bootstrap intervals. Notation as in Table 7 .

Figure 8: R1 accuracy at $r=.6$ by support and decoy ownership, separately for each model and irrelevant-fact assignment. Every cell averages the same 60 templates and three seeds; intervals are in Tables 7 and 9 .

Contrast	Qwen	Llama
H1 with Random as reference	$+40.7$ [34.6, 47.1]	$+34.2$ [27.8, 41.0]
H1 with seven retained items	$+66.8$ [61.9, 71.7]	$+72.6$ [67.2, 78.1]
Oracle relevance $-$ R1, classic cell	$-23.3$ [ $-30.6$ , $-16.1$ ]	$-31.1$ [ $-38.9$ , $-23.9$ ]
Tagged $-$ untagged support gap, all facts	$+13.8$ [8.1, 19.3]	$+6.1$ [1.0, 11.2]
Tagged $-$ shuffled-tag support gap, all facts	$+6.4$ [1.9, 11.1]	$+7.4$ [2.2, 12.6]
Untagged $-$ tagged, classic cell, all facts	$+14.4$ [7.8, 21.1]	$+13.3$ [6.7, 20.0]

Appendix

Table 11: Attribution controls (points) with 95% template-bootstrap intervals. Classic cell: private support, shared decoys, private irrelevant facts.

Policy	N1	N2	N3	N4	N5
B3	$-1.9$ [ $-7.7$ , 4.1]	$-1.0$ [ $-8.7$ , 6.7]	$+2.1$ [ $-5.3$ , 9.6]	–	–
B4	$-1.2$ [ $-7.5$ , 5.1]	$+1.4$ [ $-5.3$ , 8.2]	$-3.1$ [ $-9.4$ , 3.1]	$-0.2$ [ $-3.2$ , 2.9]	0.0 [ $-1.9$ , 1.9]
R1	$-2.9$ [ $-8.5$ , 2.7]	$-0.7$ [ $-7.5$ , 6.3]	$+1.4$ [ $-5.3$ , 7.9]	$+1.0$ [ $-2.4$ , 4.4]	$+0.3$ [ $-1.9$ , 2.4]
REF	$+0.5$ [ $-5.6$ , 6.7]	$+0.5$ [ $-7.2$ , 8.2]	$+6.2$ [ $-1.5$ , 13.8]	–	–

Appendix

Table 12: Qwen: change in natural-discussion accuracy relative to the original runs (points), averaged over budget ratios, with 95% task-bootstrap intervals. Dashes mark memories that N4 and N5 do not rebuild.

Policy	N1	N2	N3	N4	N5
B3	$+1.4$ [ $-4.6$ , 7.3]	$+5.1$ [ $-1.2$ , 11.5]	$+20.9$ [14.2, 27.7]	–	–
B4	$+3.8$ [ $-3.2$ , 10.9]	$+6.5$ [ $-0.2$ , 13.2]	$-6.5$ [ $-12.7$ , $-0.2$ ]	$-2.7$ [ $-8.2$ , 2.4]	$-0.3$ [ $-2.9$ , 2.2]
R1	$+5.0$ [ $-1.0$ , 11.1]	$+3.4$ [ $-3.6$ , 10.8]	$+27.9$ [20.2, 35.7]	$-2.6$ [ $-7.9$ , 2.7]	0.0 [ $-2.1$ , 1.9]
REF	$-1.0$ [ $-9.7$ , 7.2]	$-1.0$ [ $-8.7$ , 6.7]	$+30.8$ [21.5, 39.5]	–	–

Appendix

Table 13: Llama: change in natural-discussion accuracy relative to the original runs (points), averaged over budget ratios, with 95% task-bootstrap intervals. Dashes mark memories that N4 and N5 do not rebuild.

Arm	Accuracy	$\Delta$ vs. A0	Accuracy	$\Delta$ vs. A0
	Qwen		Llama
A0: original	30.3 [22.6, 37.9]	–	30.8 [23.1, 39.0]	–
A1: evidence first, no recommendation	28.2 [21.5, 35.4]	$-2.1$ [ $-10.3$ , 5.6]	59.0 [50.8, 67.7]	$+28.2$ [17.9, 39.0]
A2: evidence first	35.9 [28.2, 43.6]	$+5.6$ [ $-1.0$ , 12.3]	55.9 [48.2, 63.6]	$+25.1$ [15.9, 34.4]
A3: no recommendation	22.6 [15.9, 29.7]	$-7.7$ [ $-15.9$ , 0.0]	40.5 [33.3, 47.7]	$+9.7$ [0.5, 18.5]
A4: neutral length request	31.3 [23.6, 39.5]	$+1.0$ [ $-4.1$ , 6.7]	38.5 [31.8, 45.6]	$+7.7$ [ $-1.5$ , 16.9]

Appendix

Table 14: Discussion-prompt ablation: REF accuracy (%) and paired differences from A0 (points), over 65 tasks and seeds 7, 23, and 58, with 95% task-bootstrap intervals.

Figure 9: Discussion-prompt ablation: change in REF accuracy relative to A0, with 95% task-bootstrap intervals. Qwen does not reproduce Llama’s large gain from evidence-first wording.

Model	Condition / contrast	Effect [95% CI]	Holm $p$
Qwen	Headroom: all facts $-$ empty	$+36.4$ [25.1, 47.2]	–
Qwen	Controlled, known: R1 $-$ B3	$+29.2$ [22.9, 35.4]	$<.001$
Qwen	Controlled, known: R1 $-$ B4	$+55.4$ [49.4, 61.4]	$<.001$
Qwen	Controlled, known: M1 $-$ B4	$-4.6$ [ $-8.2$ , $-1.2$ ]	.008
Qwen	Controlled, extracted: R1 $-$ B3	$+25.8$ [18.8, 32.6]	$<.001$
Qwen	Controlled, extracted: R1 $-$ B4	$+53.2$ [47.5, 58.6]	$<.001$

Appendix

Table 15: All HiddenBench headroom and memory contrasts (points).

Comparison	Qwen	Llama
Controlled: R1 $-$ B3	$+29.2$ [22.9, 35.4]	$+26.7$ [20.2, 33.2]
Controlled: R1 at B3 length $-$ B3	$+25.6$ [17.6, 33.7]	$+25.8$ [19.1, 32.5]
Natural: R1 $-$ B3	$-1.4$ [ $-5.0$ , 2.4]	$-3.1$ [ $-9.2$ , 2.7]

Appendix

Table 16: HiddenBench accuracy differences (points) with 95% task-bootstrap intervals over 65 tasks; seeds and the three budget ratios are averaged within task. The controlled R1

-

B3 contrast belongs to the pre-specified Holm family.

Model / policy	Tokens / budget (%)	95% CI
Qwen / R1	95.0	[94.8, 95.3]
Qwen / R1-rel	95.7	[95.5, 95.8]
Qwen / B4	95.8	[95.7, 96.0]
Qwen / Random	95.6	[95.5, 95.8]
Qwen / B3	79.3	[77.8, 80.8]
Llama / R1	95.1	[94.9, 95.4]

Appendix

Table 17: Realized memory length relative to the budget in CrossProfile at

r=.6

, averaged within template, with 95% template-bootstrap intervals.

Run	Contrast	Effect [95% CI]	Templates / tasks
DeepSeek-V4.1-Flash, CrossProfile	H1	$+62.6$ [56.7, 68.6]	60
DeepSeek-V4.1-Flash, CrossProfile	H2	$+11.5$ [7.5, 15.6]	60
DeepSeek-V4.1-Flash, CrossProfile	H3	$+16.7$ [12.8, 20.7]	60
DeepSeek-V4.1-Flash, CrossProfile	H3, shared irrelevant	$+6.8$ [3.2, 10.4]	60
DeepSeek-V4.1-Flash, CrossProfile	H1 with Random as reference	$+28.3$ [22.1, 34.7]	60
DeepSeek-V4.1-Flash, CrossProfile	R1-rel $-$ R1, classic cell	$-32.2$ [ $-40.6$ , $-24.4$ ]	60

Appendix

Table 18: Hosted-model contrasts (points) with 95% bootstrap intervals.

Run	Valid	Invalid
DeepSeek-V4.1-Flash, CrossProfile	8,466	174
DeepSeek-V4.1-Flash, additional controls	4,184	136
DeepSeek-V4.1-Flash, natural	564	21
DeepSeek-V4-Pro, CrossProfile subset	2,880	0

Appendix

Table 19: Hosted-model decision counts. As in the local runs, a decision is invalid only after one format retry, and invalid decisions are scored as wrong.

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 246 papers

When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model
Sep 28, 2026Rishabh Sharma, Rishika LallConversational Memory LLM Reranking
Independent Researcher
Overwhelmed by Choice: Studying LLM Decision Making at Scale
Sep 26, 2026Yu-Chi Lin, Aryan Seth, Anshul Aravind +4LLM Evaluation LLM Decision-Making
University of California, Los Angeles · University of Cincinnati
Superficial Beliefs in LLM Decision-Making
Jun 9, 2026Gabriel Freedman, Francesca ToniLLM Decision-Making Metacognition in Language Models
Department of Computing Imperial College London

Sep 28, 2026J/K move · Enter open · S save

When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model

Rishabh Sharma, Rishika Lall

Independent Researcher

Conversational Memory LLM Reranking Memory Agent Benchmarks Long-Term Conversational Memory

Open paper AskPDF arXiv

Does conversational memory need LLM-extracted facts, or is selecting the right raw turns enough? Published results disagree. Extraction-based systems report gains from distilled facts. Recent studies find raw history with good ranking does as well, but disagree about whether ranking matters. We ran a pre-registered study on held-out LoCoMo conversations and LongMemEval. At a tight budget on LoCoMo, raw turns selected by a single call to Jev, a typed decision model, are non-inferior to an LLM-extraction memory (one-sided 95% bound -3.0 points against a -5-point margin). Blind human grading narrows the margin but does not change the result. Raw turns cost 3,061 times less to write, and the result holds with a second answer model. Within this study, reranking's gain shrinks as the budget grows. It adds 17.4 points on LoCoMo and 9.1 on LongMemEval when three of 30 candidates are kept. At generous budgets it adds 1.5 and 1.1, and extraction systems are more accurate. This suggests why published results disagree. At matched context, Jev selects as accurately as an LLM reranker (non-inferiority bound -2.0) at a third of the latency, and more accurately than a multi-call graph traversal. Reranking lowers correct abstention. Plans, code and graded answers are released.

Abstract

LLM Decision-Making

Explore similar work

When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model

Overwhelmed by Choice: Studying LLM Decision Making at Scale

Superficial Beliefs in LLM Decision-Making