AFVE-Bench: Measuring Abandoned-Frame Verbalization Errors in Multi-Turn LLM Revision
Abstract
Multi-turn revision often asks a model to replace an earlier organizing frame while preserving still-valid facts and artifact requirements. Existing instruction-following evaluations mainly assess the latest target and may therefore score an artifact as successful even when it adopts the new frame but exposes the superseded one. We call this failure an abandoned-frame verbalization error (AFVE). AFVE-Bench evaluates silent frame replacement in English and Chinese using 192 matched four-turn dialogues across eight writing domains. Each seed has zero, one, or two pre-target user requests to generate under the source frame. AFVE eligibility depends on the user-request history and applies to the one- and two-request conditions; the zero-request condition serves as a frame-bearing baseline. Eleven model configurations yielded 4,224 expected post-target response units. Two annotators independently scored 4,204 scorable responses for target adoption, source-frame verbalization, and direct usability; model and condition metadata were hidden, and disagreements were adjudicated. AFVE occurs in 25.54% (715/2,799) of history-conditioned responses. Of 2,388 target-passing history-conditioned responses, 472 (19.77%) still expose the abandoned source frame, most often through contrastive framing (423/472). For these standalone revision tasks, target adoption does not ensure source-frame silence.
Figures & tables
| History-conditioned responses | Total | ||
| 1,916 | 472 | 2,388 | |
| 168 | 243 | 411 | |
| Total | 2,084 | 715 | 2,799 |
| Within | Count | Share | |
| (Clean) | 462 | 24.11% | |
| 1,454 | 75.89% | ||
| Configuration | Target adoption ( ; all ) | AFVE (all ) | TC-AFVE (all ) | TC-AFVE (target-passing) | Direct usability ( ; all ) | Clean (all ) | |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 5 | 256 | 91.41 | 28.12 | 21.88 | 23.93 | 29.30 | 21.88 |
| DeepSeek V4 Flash | 256 | 85.55 | 37.89 | 28.12 | 32.88 | 9.77 | 5.86 |
| GLM-5 | 256 | 85.94 | 27.34 | 18.36 | 21.36 | 36.33 | 30.86 |
| Gemini 2.5 Flash | 256 | 81.64 | 29.30 | 17.58 | 21.53 | 8.98 | 8.59 |
| Kimi K2.6 | 255 | 91.37 | 27.06 | 20.39 | 22.32 | 48.24 | 39.61 |
| Llama 4 Maverick | 256 | 81.25 | 8.98 | 4.30 | 5.29 | 20.31 | 19.53 |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Configuration | Locked request-envelope fields |
|---|---|
| GPT-5 | reasoning effort medium; output cap 8,192; non-stream |
| Claude Sonnet 5 | adaptive reasoning effort medium; output cap 8,192; non-stream |
| Gemini 2.5 Flash | reasoning budget 4,096 excluded from stored output; output cap 8,192; non-stream |
| Qwen3-Instruct | no thinking field; output cap 8,192; top- 20; presence penalty 1.5; non-stream |
| Qwen3-Thinking | thinking budget 4,096; output cap 8,192; top- 20; presence penalty 0.0; stream |
| DeepSeek V4 Flash | thinking enabled with reasoning effort high; output cap 8,192; non-stream |
| Configuration | Model ID | Endpoint | Visible-output field |
|---|---|---|---|
| Claude Sonnet 5 | anthropic/claude-sonnet-5 | https://openrouter.ai/api/v1/chat/completions | choices[0].message.content |
| DeepSeek V4 Flash | deepseek-v4-flash | https://api.deepseek.com/chat/completions | choices[0].message.content |
| GLM-5 | glm-5 | https://open.bigmodel.cn/api/paas/v4/chat/completions | choices[0].message.content |
| Gemini 2.5 Flash | google/gemini-2.5-flash | https://openrouter.ai/api/v1/chat/completions | choices[0].message.content |
| Kimi K2.6 | kimi-k2.6 | https://api.moonshot.cn/v1/chat/completions | choices[0].delta.content |
| Llama 4 Maverick | meta-llama/llama-4-maverick | https://openrouter.ai/api/v1/chat/completions | choices[0].message.content |
| Turn 1 role | Turn 2 role | Turn 3 form | Common -related textual exposure | AFVE eligibility | |
|---|---|---|---|---|---|
| 0 | Neutral request with shared packet | Neutral request | Direct -task core; no naming of as the prior governing frame and no retrospective abandonment anchor. The request may mention -associated criteria when delimiting . | Yes. The first-turn packet includes the same -consistent candidate option as the other conditions. | No. is baseline source-frame verbalization. |
| 1 | Neutral request with shared packet | Request to generate under | Fixed explicit-abandonment transition followed by the same -task core | Yes. Same shared first-turn packet. | Yes, for scored turns 3–4. |
| 2 | Request to generate under with shared packet | Request to generate under | Identical to the turn-3 prompt | Yes. Same shared first-turn packet. | Yes, for scored turns 3–4. |
| Role | Model | Case ID | /turn | Gold labels/source |
|---|---|---|---|---|
| E1: shared fact | Gemini 2.5 Flash | case_academic_revision_01_en_strong | 2 / 3 | ; agreement |
| E2: ordinary domain term | DeepSeek V4 Flash | case_hiring_evaluation_04_en_strong | 2 / 3 | ; agreement |
| E3: contrastive framing | DeepSeek V4 Flash | case_legal_compliance_revision_03_en_weak | 1 / 3 | ; agreement |
| E4: edit-operation exposure | Qwen3-Thinking | case_inclusive_language_03_en_strong | 2 / 3 | ; agreement |
| E5: implicit residue | Qwen3-Instruct | case_legal_compliance_revision_04_en_strong | 2 / 4 | ; adjudicated |
| E6: source silent | Qwen3-Instruct | case_hiring_evaluation_02_en_strong | 2 / 4 | ; agreement |
| Model, case, turn | Source frame | Target frame | Gold evidence excerpt | Why remains observable without exact phrase match |
|---|---|---|---|---|
| Qwen3-Instruct, case_legal_compliance_revision_03_zh_weak , T3 | Individual intentional violation and sanction frame (Chinese-language task) | Control failure and system remediation frame | Translated excerpt: However, there is no evidence that Zhou Qing concealed, falsified, or knowingly evaded a violation. | The response keeps individual intent and culpability as an organizing contrast inside a requested system-remediation artifact, without using the exact source-frame/core phrase. |
| Qwen3-Instruct, case_legal_compliance_revision_04_en_strong , T4 | Case-by-case explanation and liability limitation frame | Root-cause remediation and verifiable compliance frame | While no widespread financial harm has been established—two customers received goodwill payments totaling 1,800 currency units—we recognize that recurrence reflects underlying control weaknesses requiring systemic remediation. | The response foregrounds no-harm and liability-limitation reasoning before remediation, leaving the source organization observable without using the exact source-frame/core phrase. |
| Check | Expected | Reproduced |
| Formal post-target units | 4,224 | 4,224 |
| Scorable responses | 4,204 | 4,204 |
| Unavailable coordinates | 20 | 20 |
| scorable responses | 1,405 | 1,405 |
| scorable responses | 1,396 | 1,396 |
| scorable responses | 1,403 | 1,403 |
| Level | Scope | Endpoint | Numerator/denominator | Rate (%) | 95% CI (%) |
|---|---|---|---|---|---|
| Response | All conditions | Target adoption | 3,543/4,204 | 84.28 | [75.11, 91.80] |
| Response | All conditions | Source-frame verbalization | 1,115/4,204 | 26.52 | [16.33, 37.47] |
| Response | All conditions | Direct usability | 833/4,204 | 19.81 | [16.13, 23.49] |
| Response | All conditions | Clean | 677/4,204 | 16.10 | [12.76, 19.56] |
| Response | AFVE prevalence | 715/2,799 | 25.54 | [15.54, 36.47] | |
| Response | TC-AFVE prevalence | 472/2,799 | 16.86 | [9.85, 24.96] |
| Endpoint | Exact | Cohen’s | Gwet AC1 |
|---|---|---|---|
| Target adoption | 76.86% | 0.329 | 0.649 |
| Source-frame verbalization | 84.66% | 0.625 | 0.741 |
| Direct usability | 83.42% | 0.546 | 0.740 |
| Ordinal endpoint | Exact | Within one level | ||
|---|---|---|---|---|
| Target adoption | 76.86% | 100.00% | 0.331 | 0.320 |
| Direct usability | 60.68% | 99.36% | 0.544 | 0.538 |
| Subtype | Exact | Cohen’s | Gwet AC1 | F1 |
|---|---|---|---|---|
| Explicit Source Reference | 90.44% | 0.602 | 0.874 | 0.657 |
| Contrastive Source Framing | 85.73% | 0.609 | 0.775 | 0.703 |
| Implicit Source Residue | 95.91% | 0.687 | 0.953 | 0.709 |
| Edit-Operation Exposure | 99.22% | 0.669 | 0.992 | 0.673 |
| Pre-target user requests under | Target | Usability | Clean | ||||
|---|---|---|---|---|---|---|---|
| 0 ( ) | 1,405 | 82.21 | 28.47 | 19.72 | 15.30 | 15.87 | 19.31 |
| 1 ( ) | 1,396 | 84.53 | 26.58 | 19.34 | 15.90 | 17.34 | 20.51 |
| 2 ( ) | 1,403 | 86.10 | 24.52 | 20.38 | 17.11 | 16.39 | 19.04 |
| Matched comparison | Target | Usability | Clean | ||||
|---|---|---|---|---|---|---|---|
| minus | 1,394 | 2.37 [ , 5.89] | [ , 2.80] | [ , 2.10] | 0.57 [ , 2.79] | 1.43 [ , 3.37] | 1.16 [ , 3.61] |
| minus | 1,401 | 3.93 [0.21, 8.27] | [ , 1.14] | 0.64 [ , 3.14] | 1.78 [ , 3.85] | 0.57 [ , 3.00] | [ , 2.97] |
| minus | 1,392 | 1.72 [0.43, 3.17] | [ , ] | 1.08 [ , 3.53] | 1.29 [ , 3.59] | [ , 1.00] | [ , 0.72] |
| Matched comparison | Discordant (%) | |||||
|---|---|---|---|---|---|---|
| versus | 1,394 | 912 | 86 | 113 | 283 | 14.28 |
| versus | 1,401 | 916 | 86 | 142 | 257 | 16.27 |
| versus | 1,392 | 951 | 70 | 101 | 270 | 12.28 |
| Configuration | Target | AFVE | TC-AFVE | TC-AFVE Target | |
|---|---|---|---|---|---|
| Claude Sonnet 5 | 256 | 91.4 [83.2, 97.3] | 28.1 [16.4, 40.6] | 21.9 [12.1, 32.4] | 23.9 [13.5, 35.7] |
| DeepSeek V4 Flash | 256 | 85.5 [75.4, 94.1] | 37.9 [25.8, 50.8] | 28.1 [17.2, 39.8] | 32.9 [20.6, 46.1] |
| GLM-5 | 256 | 85.9 [75.8, 94.1] | 27.3 [15.2, 40.6] | 18.4 [9.0, 28.9] | 21.4 [10.6, 33.8] |
| Gemini 2.5 Flash | 256 | 81.6 [71.1, 91.0] | 29.3 [19.1, 39.8] | 17.6 [10.5, 26.2] | 21.5 [13.0, 31.3] |
| Kimi K2.6 | 255 | 91.4 [82.0, 98.8] | 27.1 [14.1, 41.0] | 20.4 [9.5, 32.8] | 22.3 [10.5, 35.9] |
| Llama 4 Maverick | 256 | 81.2 [71.1, 90.2] | 9.0 [1.6, 18.4] | 4.3 [0.4, 10.5] | 5.3 [0.5, 13.1] |
| Configuration | Direct usability | Clean | |
|---|---|---|---|
| Claude Sonnet 5 | 256 | 29.3 [23.0, 35.9] | 21.9 [14.8, 29.3] |
| DeepSeek V4 Flash | 256 | 9.8 [5.5, 14.1] | 5.9 [2.7, 9.4] |
| GLM-5 | 256 | 36.3 [27.3, 45.3] | 30.9 [21.9, 40.2] |
| Gemini 2.5 Flash | 256 | 9.0 [4.7, 14.1] | 8.6 [4.3, 13.7] |
| Kimi K2.6 | 255 | 48.2 [39.5, 56.6] | 39.6 [29.8, 49.2] |
| Llama 4 Maverick | 256 | 20.3 [14.5, 26.6] | 19.5 [13.7, 25.8] |
| Scenario | AFVE prevalence | TC-AFVE prevalence | TC-AFVE Target |
|---|---|---|---|
| Frozen Gold | 715/2,799 (25.54%) | 472/2,799 (16.86%) | 472/2,388 (19.77%) |
| Implicit-only reclassified absent | 647/2,799 (23.12%) | 443/2,799 (15.83%) | 443/2,388 (18.55%) |
| Rater | Threshold | Target | Usability | Clean | TC-AFVE prevalence | TC-AFVE Target |
|---|---|---|---|---|---|---|
| A | Full success only | 73.99 | 19.69 | 16.54 | 16.33 | 22.07 |
| A | Partial or full | 100.00 | 73.20 | 56.02 | 29.26 | 29.26 |
| B | Full success only | 84.57 | 28.30 | 24.54 | 16.97 | 20.07 |
| B | Partial or full | 100.00 | 84.96 | 66.24 | 25.04 | 25.04 |
| Omitted domain | Numerator/denominator | TC-AFVE Target (%) | Difference from full (points) |
|---|---|---|---|
| None | 472/2,388 | 19.77 | 0.00 |
| Academic revision | 365/2,175 | 16.78 | |
| Clinical summary | 449/2,065 | 21.74 | 1.98 |
| Educational feedback | 421/2,046 | 20.58 | 0.81 |
| Hiring evaluation | 437/2,209 | 19.78 | 0.02 |
| Inclusive language | 450/2,037 | 22.09 | 2.33 |
| Condition | Numerator | Complete pairs | Rate (%) |
|---|---|---|---|
| All conditions | 186 | 2,096 | 8.87 |
| 58 | 701 | 8.27 | |
| 62 | 696 | 8.91 | |
| 66 | 699 | 9.44 |
| Unavailable-coordinate reason | Count | Share (%) |
|---|---|---|
| Empty output | 13 | 65.00 |
| Skipped dependency | 5 | 25.00 |
| Source-turn failure | 2 | 10.00 |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,527 papers