cs.AIAug 18, 2026
SaveExplicit State Elicitation Is Not Enough: A Controlled Audit of Memory-Policy Classification
Abstract
Personalized agents must decide whether retrieved user memory should be used, ignored, updated, or queried before it affects a current task. We use this setting to develop an empirical audit protocol for structured intermediate outputs: first audit dataset shortcuts, then isolate bundled prompt changes, check whether intermediate labels are answer-associated, test decomposed semantic evidence, and audit provider-level execution failures. A 480-example synthetic development set initially suggested large gains from a state-structured prompt bundle, but TF-IDF diagnostics showed lexical separability and no positive standalone Ignore cases. We therefore construct a frozen 160-example controlled counterfactual set with 40 matched four-way families and rule-derived reference policies. On this set, exposing the four state definitions improves accuracy, but an isolated explicit state-output field does not significantly improve policy accuracy for Llama-3.3-70B and gives only a marginal, non-significant gain for GPT-OSS-120B. Supplying benchmark-associated state labels shifts policy predictions, but because those labels deterministically map to policies, this is a label-conditioning diagnostic rather than evidence of a faithful internal mechanism. Family-level and seed-stability analyses further show that example-level accuracy overstates counterfactual consistency: complete four-way family success is rare. An exploratory follow-up that elicits decomposed semantic evidence also fails to improve routing for the cleanly evaluated endpoint; the corresponding GPT-OSS condition was unavailable because of provider-side request validation. We evaluate policy classification only, not downstream responses, tool actions, or memory-store mutation.
Figures & tables
Figure 1: Five-stage audit protocol instantiated in this paper. The label-conditioning diagnostic is separated from semantic-evidence interventions because supplied benchmark-state labels map deterministically to policy labels and therefore do not test mechanism faithfulness.
| Model | Clean acc. | Taxonomy acc. | Taxonomy vs clean | State acc. | State vs clean |
|---|---|---|---|---|---|
| Llama-3.3-70B | 44.0% | 53.1% | +9.17 pp [5.00, 13.33] raw p=0.0003; Holm p=0.0012 | 44.6% | 0.6 pp [-2.9, 4.0] raw p=0.8084; Holm p=0.8084 |
| GPT-OSS-120B | 54.8% | 59.8% | +5.00 pp [1.67, 8.33] raw p=0.0045; Holm p=0.0135 | 58.1% | 3.3 pp [0.0, 6.7] raw p=0.0710; Holm p=0.1420 |
Table 1: Primary matched ablation on the frozen controlled set. Accuracies are exact-match policy accuracies. is each arm minus clean policy-only in percentage points; parse failures count as incorrect. Parse failures are 0 for clean policy-only and state-output on both endpoints; taxonomy-only has 1 parse failure for Llama and 0 for GPT-OSS. Intervals are 95% scenario-family-cluster bootstrap intervals; p-values are paired family-level permutation tests. Holm adjustment is over the four arm-versus-clean contrasts shown for taxonomy-only and state-output across the two endpoints; the state-output Holm values are unchanged under this expanded family. All reported percentage-point differences in the paper are computed from unrounded accuracies; displayed rounded endpoints may not reproduce them exactly.
| Model | Policy-only | Supplied ref. | Supplied conflict | State-first |
|---|---|---|---|---|
| Llama-3.3-70B | 45.4% | 52.7% | 37.5% | 42.7% |
| GPT-OSS-120B | 53.1% | 65.0% | 50.0% | 59.2% |
| Model | Ref. vs policy | Conflict vs ref. | Conflict vs policy | State-first ( ) |
| Llama-3.3-70B | 7.3 pp [2.5, 12.1] p=0.0065 | -15.2 pp [-20.4, -10.0] | -7.9 pp [-11.9, -4.2] | -2.7 pp p=0.0825 |
| GPT-OSS-120B | 11.9 pp [8.3, 15.4] | -15.0 pp [-20.0, -10.2] | -3.1 pp [-7.1, 0.8] p=0.1582 | 6.0 pp p=0.0025 |
Table 2: Label-conditioning sensitivity diagnostic on the frozen controlled set. The supplied-reference and supplied-conflict columns are absolute accuracies when the prompt receives a benchmark-associated state label. Delta columns are percentage-point differences on matched controlled-set decisions; conflict-vs-reference is relative to supplied-reference routing, not policy-only. Because controlled-set state labels map deterministically to policies, this diagnostic tests sensitivity to supplied benchmark-associated labels rather than an internal causal state mechanism.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Paper-facing name | Artifact ID |
|---|---|
| Policy-only matched arm | Clean_Policy_Only |
| Explicit state-output matched arm | State_Output_No_Rationale |
| State-only routing with policy definitions | State_Only_Deterministic_Routing (matched-ablation prompt) |
| State-only routing without policy definitions | State_Only_Deterministic_Routing (label-conditioning prompt) |
| Supplied reference label | Forced_Correct_State |
| Supplied conflicting label | Forced_Wrong_State |
Table 3: Reader-facing names for internal artifact identifiers. Exact machine IDs are stored in the raw-result metadata. The two state-only deterministic-routing artifact conditions share a historical ID but use different prompts; they are therefore named separately in prose.
| Model | Arm | Calls | Accuracy | Macro F1 | Parse fail |
|---|---|---|---|---|---|
| Llama-3.3-70B | CLEAN_POLICY_ONLY | 480 | 44.0% | 36.3% | 0 |
| Llama-3.3-70B | TAXONOMY_NO_STATE_OUTPUT | 480 | 53.1% | 46.5% | 1 |
| Llama-3.3-70B | STATE_OUTPUT_NO_RATIONALE | 480 | 44.6% | 35.4% | 0 |
| Llama-3.3-70B | STATE_OUTPUT_WITH_RATIONALE | 480 | 47.9% | 40.4% | 0 |
| Llama-3.3-70B | STATE_ONLY_DETERMINISTIC_ROUTING | 480 | 45.0% | 35.9% | 0 |
| GPT-OSS-120B | CLEAN_POLICY_ONLY | 480 | 54.8% | 47.1% | 0 |
Table 4: Full frozen-V3 matched prompt-ablation results.
| Model | Condition | Calls | Accuracy | Macro F1 | State agreement | Parse fail |
|---|---|---|---|---|---|---|
| Llama-3.3-70B | POLICY_ONLY | 480 | 45.4% | 36.3% | n/a | 0 |
| Llama-3.3-70B | STATE_FIRST | 480 | 42.7% | 32.6% | 41.5% | 0 |
| Llama-3.3-70B | POLICY_FIRST | 480 | 48.8% | 40.1% | 44.6% | 0 |
| Llama-3.3-70B | FORCED_CORRECT_STATE | 480 | 52.7% | 49.2% | n/a | 0 |
| Llama-3.3-70B | FORCED_WRONG_STATE | 480 | 37.5% | 30.4% | n/a | 0 |
| Llama-3.3-70B | STATE_ONLY_DETERMINISTIC_ROUTING | 480 | 46.9% | 36.1% | 46.9% | 0 |
Table 5: Full frozen-V3 label-conditioning diagnostic results. State agreement is shown only for conditions that ask the model to emit a state. Forced-state conditions supply benchmark-associated labels and are not interpreted as internal state mechanisms.
| Model | Policy | 95% CI | Holm | |
|---|---|---|---|---|
| Llama-3.3-70B | Use | +17.5 pp ∗ | [6.67, 28.33] | 0.010 |
| Llama-3.3-70B | Ignore | n.s. | n.s. | n.s. |
| Llama-3.3-70B | Update | n.s. | n.s. | n.s. |
| Llama-3.3-70B | Ask | -14.2 pp ∗ | [-22.50, -6.67] | 0.010 |
| GPT-OSS-120B | Use | n.s. | n.s. | n.s. |
| GPT-OSS-120B | Ignore | +15.8 pp ∗ | [7.50, 25.00] | 0.003 |
Table 6: Per-policy decomposition of the primary matched ablation. Deltas are State_Output_No_Rationale minus Clean_Policy_Only ; CIs are scenario-family-cluster bootstrap intervals; is Holm-adjusted across the four policies within each model; parse failures count as incorrect. Only Holm-significant cells are numerically reported here; full per-policy metrics and confusion matrices appear below. Gains and losses are reported together and the overall effect is null. ∗ marks Holm-significant cells.
| Model | Arm | Accuracy | Macro F1 | Parse fail |
|---|---|---|---|---|
| llama-3.1-8b-instant | Clean policy-only | 25.6% | 11.3% | 0/480 |
| llama-3.1-8b-instant | State-output no rationale | 37.3% | 27.4% | 18/480 |
| llama-3.1-8b-instant | Delta (state-output minus policy-only) | +11.7 pp | 95% CI [8.54, 14.79] | p=0.0001 |
Table 7: Additive third-endpoint replication. The candidate qwen/qwen3.6-27b was excluded after an HTTP 400 json_validate_failed smoke failure, mirroring the GPT-OSS explicit-evidence provider failure documented in the main text. llama-3.1-8b-instant is a small model included only to test cross-model direction, not as a headline result. The state-output arm had 18/480 parse failures, counted as incorrect, so this delta must not be read as a clean gain.
| Model | Condition | Family exact |
|---|---|---|
| Llama | Matched policy-only | 0.0% [0.0, 2.5] |
| Llama | State output | 0.0% [0.0, 2.5] |
| GPT-OSS | Matched policy-only | 1.7% [0.0, 5.0] |
| GPT-OSS | State output | 5.0% [0.8, 10.0] |
Table 8: Family exact-match rates on the frozen controlled set. A family is correct only when all four matched variants are predicted correctly. Intervals resample scenario families, with seeds averaged inside each family. For 0.0% entries, the 3/n upper bound uses n=120 family-seed units, an optimistic independence assumption.
| Model | Condition | Family exact |
|---|---|---|
| Llama | Diagnostic policy-only | 0.0% [0.0, 2.5] |
| Llama | Supplied reference label | 1.7% [0.0, 4.2] |
| Llama | Supplied conflicting label | 0.0% [0.0, 2.5] |
| GPT-OSS | Diagnostic policy-only | 1.7% [0.0, 4.2] |
| GPT-OSS | Supplied reference label | 11.7% [5.0, 19.2] |
| GPT-OSS | Supplied conflicting label | 0.8% [0.0, 2.5] |
Table 9: Family exact-match rates for the separately executed label-conditioning diagnostic. The policy-only row comes from a different frozen protocol source and is labeled by protocol to avoid conflation. Intervals resample scenario families, with seeds averaged inside each family. For 0.0% entries, the 3/n upper bound uses n=120 family-seed units, an optimistic independence assumption.
| Model | Accuracy | Macro F1 | Parse failures | Delta vs clean | 95% CI | |
|---|---|---|---|---|---|---|
| llama-3.3-70b-versatile | 35.42% | 30.36% | 0 | -8.54 pp | [-13.96, -3.54] | 0.003400 |
| openai/gpt-oss-120b | 46.88% | 44.08% | 0 | -7.92 pp | [-12.08, -3.54] | 0.001200 |
Table 10: Normalized-input probe results. Deltas are normalized-input minus clean policy-only.
| Error mode | Count | Eligible | Detail |
|---|---|---|---|
| conflict over detection | 2428 | 5121 | |
| task neglect | 1583 | 5121 | |
| recency overreach | 1543 | 5121 | |
| ask defaulting | 873 | 5121 | |
| irrelevance to ask confusion | 760 | 5121 | |
| state policy inconsistency | 276 | 3840 | 138 final policy correct |
Table 11: Rule-based error-mode counts across final scored controlled-set model-arm-example-seed records. Eligible denotes the count of final scored records for which the category is applicable (5121 for arms scored on all records; 3840 for state-emitting arms only). Counts are non-exclusive and must not be summed.
| Model | Arm | Policy | Precision | Recall | F1 |
|---|---|---|---|---|---|
| Llama-3.3-70B | Clean policy-only | Use | 85.1 | 61.7 | 71.5 |
| Llama-3.3-70B | Clean policy-only | Ignore | 100.0 | 0.8 | 1.7 |
| Llama-3.3-70B | Clean policy-only | Update | 34.6 | 98.3 | 51.2 |
| Llama-3.3-70B | Clean policy-only | Ask | 35.3 | 15.0 | 21.1 |
| Llama-3.3-70B | State-output | Use | 84.8 | 79.2 | 81.9 |
| Llama-3.3-70B | State-output | Ignore | 83.3 | 4.2 | 7.9 |
Table 12: Per-policy precision, recall, and F1 for the two primary matched-ablation arms. Values are percentages. Parse failures count as incorrect.
| Model | Generation category | Active | Stale | Conflicting | Underspecified | Parse fail | |
|---|---|---|---|---|---|---|---|
| Llama-3.3-70B | Stable | 348 | 0 | 3 | 9 | 0 | 360 |
| Llama-3.3-70B | Contextual | 343 | 0 | 5 | 12 | 0 | 360 |
| Llama-3.3-70B | Updated | 33 | 4 | 323 | 0 | 0 | 360 |
| Llama-3.3-70B | Ambiguous | 108 | 20 | 102 | 130 | 0 | 360 |
| Qwen3-32B | Stable | 354 | 0 | 1 | 3 | 2 | 360 |
| Qwen3-32B | Contextual | 358 | 0 | 0 | 2 | 0 | 360 |
Table 13: State-structured bundle state confusion matrix pooled over three seeds. Rows are gold benchmark categories; columns are predicted memory states. Overall benchmark-mapped state agreement under the mapping Stable/Contextual Active, Updated Conflicting, and Ambiguous Underspecified is 79.4% for Llama-3.3-70B and 73.1% for Qwen3-32B.
| Model | Arm | Overall | Updated | Parse fail | vs Historical | vs Cleaned |
|---|---|---|---|---|---|---|
| Llama-3.3-70B | Historical no-state-output | 70.3±0.5% | 46.7±3.8% | 0.0±0.0% | – | – |
| Llama-3.3-70B | Cleaned no-state-output | 71.7±0.6% | 48.6±3.2% | 0.0±0.0% | – | – |
| Llama-3.3-70B | State-structured bundle | 85.1±0.2% | 89.7±2.7% | 0.0±0.0% | 3.6e-37 | 2.4e-32 |
| Qwen3-32B | Historical no-state-output | 65.3±1.8% | 30.3±5.4% | 0.2±0.2% | – | – |
| Qwen3-32B | Cleaned no-state-output | 64.3±0.8% | 33.6±2.7% | 0.0±0.0% | – | – |
| Qwen3-32B | State-structured bundle | 75.5±1.2% | 75.3±3.8% | 0.3±0.3% | 1.5e-18 | 2.9e-26 |
Table 14: Three-arm ablation comparing the historical implicit-state/no-state-output prompt, the cleaned implicit-state/no-state-output prompt, and the state-structured prompt bundle. -values are pooled exact McNemar diagnostics over 1440 matched example-seed decisions, with parse failures counted as wrong; cluster-aware comparisons are reported separately.
| Model | Arm | Overall | Stable | Contextual | Updated | Ambiguous | Parse fail |
|---|---|---|---|---|---|---|---|
| Llama-3.3-70B | State-structured bundle | 85.1±0.2% | 96.4±0.5% | 95.8±0.8% | 89.7±2.7% | 58.3±2.2% | 0.0±0.0% |
| Llama-3.3-70B | Historical oracle-worded routing | 86.9±2.2% | 99.2±0.0% | 97.5±0.0% | 72.5±3.0% | 78.3±6.0% | 0.0% |
| Qwen3-32B | State-structured bundle | 75.5±1.2% | 98.6±1.3% | 99.2±0.8% | 75.3±3.8% | 28.9±1.0% | 0.3±0.3% |
| Qwen3-32B | Historical oracle-worded routing | 73.5±2.1% | 99.7±0.5% | 100.0±0.0% | 50.0±3.0% | 44.2±6.6% | 0.0% |
Table 15: State-structured bundle versus Historical oracle-worded routing . This historical language-model routing diagnostic uses oracle-worded prompt text and should not be interpreted as a neutral mapped-state intervention or a deterministic oracle.
| Model | Arm | Tokens | Tokens/call | Accuracy |
|---|---|---|---|---|
| Llama-3.3-70B | Direct policy (no classification) | 219,903 | 152.7 | 69.8±1.6% |
| Llama-3.3-70B | State-structured bundle | 475,302 | 330.1 | 85.1±0.2% |
| Qwen3-32B | Direct policy (no classification) | 563,470 | 391.3 | 66.7±1.7% |
| Qwen3-32B | State-structured bundle | 811,965 | 563.9 | 75.5±1.2% |
Table 16: Token cost and strict accuracy for direct policy prediction versus explicit state classification. Token counts are aggregated over 1440 calls.
| Agent | Generation category | Use | Ask | Ignore | Update | Parse fail | |
|---|---|---|---|---|---|---|---|
| Historical no-state-output | Stable | 355 | 2 | 1 | 2 | 0 | 360 |
| Historical no-state-output | Contextual | 356 | 2 | 2 | 0 | 0 | 360 |
| Historical no-state-output | Updated | 157 | 7 | 28 | 168 | 0 | 360 |
| Historical no-state-output | Ambiguous | 165 | 133 | 62 | 0 | 0 | 360 |
| Cleaned no-state-output | Stable | 354 | 3 | 3 | 0 | 0 | 360 |
| Cleaned no-state-output | Contextual | 357 | 0 | 3 | 0 | 0 | 360 |
Table 17: Policy confusion matrix for Llama-3.3-70B. Rows are gold benchmark categories; columns are predicted policies plus parse failures.
| Agent | Generation category | Use | Ask | Ignore | Update | Parse fail | |
|---|---|---|---|---|---|---|---|
| Historical no-state-output | Stable | 353 | 5 | 1 | 0 | 1 | 360 |
| Historical no-state-output | Contextual | 359 | 1 | 0 | 0 | 0 | 360 |
| Historical no-state-output | Updated | 186 | 16 | 49 | 109 | 0 | 360 |
| Historical no-state-output | Ambiguous | 211 | 119 | 28 | 0 | 2 | 360 |
| Cleaned no-state-output | Stable | 357 | 3 | 0 | 0 | 0 | 360 |
| Cleaned no-state-output | Contextual | 357 | 3 | 0 | 0 | 0 | 360 |
Table 18: Policy confusion matrix for Qwen3-32B. Rows are gold benchmark categories; columns are predicted policies plus parse failures.
| Protocol | Calls | Hosted cost |
|---|---|---|
| Matched prompt ablation | 4,800 | $0.8887 |
| Label-conditioning diagnostic | 5,760 | $0.9118 |
| Normalized-input probe | 960 | 0.066606 |
| Semantic-evidence follow-up | 2,880 | $0.3977 |
| Smoke tests | 2 | $0.0002 |
| Frozen-core total including smoke | 10,562 | $1.8007 |
Table 19: Hosted call and cost ledger by protocol. The normalized-input row lists stored per-model costs rather than a stored protocol-total cost.
| Policy | History records | Rule-derived relation |
|---|---|---|
| Use | Jordan saved an earlier entry: flight seat choice uses an aisle seat for its flight seat choice option. The later note records an aisle seat for the flight seat choice option of flight seat choice. Case A: Jordan noted that the train seat option for train seat is a window seat. Case B: the profile lists I sit near an exit as the train seat option for train seat. | One current task-relevant preference is unique; temporal and contextual distractors do not change it. |
| Ignore | The earlier note records a window seat for the train seat option of train seat. Later, Jordan noted that the train seat option for train seat is I sit near an exit. Case A: the profile lists I sit near an exit as the train seat option for train seat. Jordan saved the case b entry: train seat uses a window seat for its train seat option. | The four records concern distractor attributes; no task-relevant memory is needed. |
| Update | Earlier, Jordan noted that the flight seat choice option for flight seat choice is a window seat. Later, the profile lists a bulkhead aisle seat as the flight seat choice option for flight seat choice. Jordan saved the case a entry: train seat uses a window seat for its train seat option. The case b note records I sit near an exit for the train seat option of train seat. | The later temporal task-relevant record supersedes the earlier task-relevant record. |
| Ask | Earlier, the profile lists a window seat as the train seat option for train seat. Jordan saved a later entry: train seat uses I sit near an exit for its train seat option. The case a note records an aisle seat for the flight seat choice option of flight seat choice. Case B: Jordan noted that the flight seat choice option for flight seat choice is a window seat. | Two context-specific task-relevant preferences are present, but the task omits the deciding context. |
Table 20: Representative four-way family from the frozen controlled set. The task is byte-identical for all four variants: “Choose the preference option for Jordan’s flight seat choice.” The example is shown to expose the controlled synthetic structure, not as a naturalistic conversation sample.
More in this topic
AI Agent Auditing
AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.
See all 262 papers