Correct Answers from Insufficient Histories: A Controlled Study of Agent Memory Evaluation
Organizations: Huawei Technologies Ltd.
Abstract
An answer can match the reference for a complete agent history even when the reader's visible history does not determine it. We study this distinction using executable device logs. Our protocol separates reference agreement, certified evidence sufficiency, and context-appropriate behavior. Insufficiency requires two legal histories yielding byte-identical visible contexts but different answers; public chronology and query cutoffs certify sufficient prefixes. On 80 confirmation histories, Qwen3-32B matches 87 hidden references among 203 insufficient raw-prefix instances, with only seven appropriate abstentions. A fixed-input DeepSeek-V4-Flash replication matches 85 and abstains on 21 of the same 203 instances. Each reader makes 1,600 requests. Compact CSV's estimated appropriateness effects remain inconclusive: +3.16 percentage points, clustered 95% interval [0.00,6.57], for Qwen and +2.46, [-1.84,6.94], for DeepSeek. A separate held-out judge study achieves 99.38% correctness but retains support errors and truncations; a deterministic baseline solves all constructed cases. These results support an evaluation diagnosis across two hosted reader configurations, not gains for a general memory system or natural-language evidence assessment.
Figures & tables
| Study | Histories / status | Model requests | Interpretation |
|---|---|---|---|
| Outcome masking | 40 development | 480 | Equal-length evidence intervention; four queries per history |
| Unordered packing | Same 40 development | 2,080 | Budget and format exploration, including supplement; shared requests |
| Judge confirmation | 120 reserved | 2,400 | Constructed candidates; no new reader answers |
| Chronological Qwen | 16 development + 80 confirmation | 1,920 | Five contexts and four queries; 1,600 confirmation requests |
| DeepSeek replication | Same 80, Qwen-exposed | 1,600 | Identical visible inputs; second reader configuration |
| Qwen3-32B | DeepSeek-V4-Flash | Shared evidence counts | |||||
|---|---|---|---|---|---|---|---|
| Context | Reference | Appropriate | Reference | Appropriate | Partial S | I | U |
| Raw prefix | 181/320 | 90/307 | 172/320 | 98/307 | 104 | 203 | 13 |
| Whole-record raw | 189/320 | 89/307 | 172/320 | 94/307 | 104 | 203 | 13 |
| CSV prefix | 195/320 | 108/298 | 167/320 | 111/298 | 119 | 179 | 22 |
| Full raw | 166/320 | 166/320 | 198/320 | 198/320 | — | 0 | 0 |
| Full CSV | 188/320 | 188/320 | 196/320 | 196/320 | — | 0 | 0 |
| Certified raw-prefix subset | Qwen | DeepSeek |
|---|---|---|
| Sufficient: correct | 83/104 | 77/104 |
| Sufficient: abstains | 4/104 | 23/104 |
| Insufficient: concrete | 196/203 | 182/203 |
| Insufficient: abstains | 7/203 | 21/203 |
| Insufficient: ref. hit | 87/203 | 85/203 |
| Evaluator | Coverage | Joint correct | False accepts |
|---|---|---|---|
| Finite program | 1,680 | 1,680 | 0/480 |
| Extended program | 1,920 | 1,920 | 0/480 |
| Reference only | 1,680 | 1,320 | 120/480 |
| LLM low | 1,918 | 1,908 | 0/480 |
| Program-first + low | 1,920 | 1,920 | 0/480 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Context | Operator | Reference | Appropriate |
|---|---|---|---|
| Raw prefix | count | 40/80 | 13/76 |
| first | 63/80 | 40/79 | |
| last | 43/80 | 19/76 | |
| state | 35/80 | 18/76 | |
| Whole-record | count | 40/80 | 13/76 |
| first | 63/80 | 40/79 |
| Reader | Certified subset | Raw | Whole | CSV |
|---|---|---|---|---|
| Qwen | S: correct | 83/104 | 87/104 | 99/119 |
| S: abstain | 4/104 | 0/104 | 1/119 | |
| I: concrete | 196/203 | 201/203 | 170/179 | |
| I: abstain | 7/203 | 2/203 | 9/179 | |
| I: ref. hit | 87/203 | 91/203 | 75/179 | |
| DeepSeek | S: correct | 77/104 | 74/104 | 86/119 |
| DeepSeek contrast | pp | 95% CI | |
|---|---|---|---|
| CSV–raw, appropriate | 285 | ||
| CSV–raw, reference | 320 | ||
| Whole–raw, appropriate | 307 | ||
| Full CSV–raw, reference | 320 |
| Reference agreement | Context appropriateness | |||||
|---|---|---|---|---|---|---|
| Context | pp | 95% CI | pp | 95% CI | ||
| Raw prefix | 320 | 307 | ||||
| Whole-record raw | 320 | 307 | ||||
| CSV prefix | 320 | 298 | ||||
| Full raw | 320 | 320 | ||||
| Full CSV | 320 | 320 | ||||
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,297 papers