Organizations: School of Artificial Intelligence, Jilin University · College of Software, Jilin University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · BiliBili Inc. · University of Ottawa · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University · International Center of Future Science, Jilin University
Reliable use of large language models requires responses to satisfy formatting requirements across changing instructions. Standard compliance scores do not distinguish preserving a model's observed format from correcting it, or reveal whether that correction holds across variants. We introduce ReFormBench, a benchmark that uses each model's unconstrained response to the same question as its formatting baseline. It links targets by feature and require/forbid mode, evaluating their instantiated requirements across variants. ReFormBench comprises 5,000 difficulty-screened three-constraint test instances derived from 910 source questions and covering 66 formatting features. TCPR measures complete instruction compliance, while EMCR measures individual and joint correction of targets opposed to the annotated baseline state. Across ten models, EMCR ranges from 34.04% to 64.72%, and the highest three-target correction estimate is 50.02%. Compliance and correction scores yield different model rankings and task-level profiles. A prompting experiment on Qwen3.5-9B improves both metrics, and the catalog covers over 95% of extracted instances on four additional models. ReFormBench makes response-conditioned correction across instruction variants a measurable dimension of format following. The code and data are available at https://anonymous.4open.science/r/ReFormBench/.
Figures & tables
Figure 1: Illustration of correcting a model’s observed format with an explicit constraint.
Figure 2: Benchmark construction and evaluation. Source questions support a shared formatting constraint space and model-specific response baselines. The finalized benchmark contains 5,000 three-constraint instruction variants. Quality control covers instruction validity, difficulty screening, and response judging; evaluation measures complete compliance (TCPR) and individual and joint correction across variants (EMCR).
Benchmark
FF
McC
StC
PrB
IFEval ( Zhou et al., 2023 )
△
△
×
×
FollowBench ( Jiang et al., 2024 )
△
△
△
×
ComplexBench ( Wen et al., 2024 )
△
△
△
×
EIFBench ( Zou et al., 2025 )
△
✓
×
×
FOFO ( Xia et al., 2024 )
✓
✓
×
×
SchemaBench ( Lu et al., 2025 )
✓
✓
×
×
Table 1: Comparison of benchmark designs.
EMCR components
Model
TCPR
EMCR
K=1
K=2
K=3
Kimi K3
82.82
64.72
80.22
63.92
50.02
GPT-5.6 Sol
81.99
62.31
78.55
61.95
46.42
Gemini 3.5 Flash
78.08
60.63
78.28
60.20
43.42
Qwen3.6-35B-A3B
77.82
60.34
76.83
60.46
43.71
GLM-5.2
75.76
55.50
74.33
55.09
37.07
Table 2: Format following and sustained correction across variants for ten models (%). EMCR is the equal-weight mean of the three K -level correction rates.
Figure 3: Task-level format following across ten models and seven task types. TCPR (top) and EMCR (bottom), shown as calibrated point estimates (%). Both panels use the same model order and a shared 0–100 color scale. Values are rounded to one decimal place; task summaries and scores to two decimal places appear in Tables A5 and A6 .
Measure
Separate
Joint
Δ (pp)
All-pass (%)
70.48
64.43
−6.05
Target pass (%)
Baseline-opposed
86.80
84.05
−2.75
Baseline-aligned
91.33
87.30
−4.03
Group gap (pp)
−4.53
−3.25
+1.28
Table 3: Separate versus joint constraint satisfaction on Qwen3.6-27B. Δ is Joint minus Separate (pp); group gap is baseline-opposed minus baseline-aligned target pass. Separate all-pass averages products of calibrated pass probabilities. Differences use unrounded estimates.
Figure 4: Prompting configuration improvement for Qwen3.5-9B. Paired dots show TCPR and EMCR, followed by EMCR’s three K -level components. Bars show gains in percentage points, calculated from unrounded estimates.
Model
Cov. (%)
Catalog features
ρ
Gemini 3.1 Pro
95.05
64/66
0.737
DeepSeek-V3.2
96.03
64/66
0.777
gpt-oss-120b
96.54
65/66
0.805
Doubao-Seed-2.1-Pro
95.21
62/66
0.776
Four models pooled
95.75
65/66
0.804
Table 4: Catalog coverage on four additional models, each with 910 responses. “Cov.” gives the percentage of valid extracted feature instances covered by the catalog. Catalog features are distinct entries (union for the pooled row). Spearman ρ compares feature frequencies with the ten-model reference; the pooled row uses four-model mean frequencies and aggregated instance counts.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
a
b
c
Other passes
v1
1
1
1
—
v2
1
0
—
1
v3
1
—
—
2
Appendix
Table A1: Illustrative target judgments for one model–question pair. Each variant contains three constraints. Other passes count satisfied baseline-aligned constraints.
Model
K=1
K=2
K=3
Kimi K3
1.41
1.61
1.90
GPT-5.6 Sol
1.32
1.52
1.74
Gemini 3.5 Flash
1.42
1.68
1.98
Qwen3.6-35B-A3B
1.35
1.55
1.91
GLM-5.2
1.58
1.91
2.31
DeepSeek-V4-Pro
1.48
1.85
2.41
Appendix
Table A2: Mean number of instruction variants with a target violation per unsuccessful evaluation unit at each K level.
Evaluated model
Dual-judge agreement
Disagreements
Adjudicated samples
Agrees with GPT count (%)
Agrees with Gemini count (%)
Kimi K3
91.74%
1,239
576
568 (98.61%)
8 (1.39%)
GPT-5.6 Sol
91.28%
1,308
588
491 (83.50%)
97 (16.50%)
Gemini 3.5 Flash
90.56%
1,416
609
563 (92.45%)
46 (7.55%)
Qwen3.6-35B-A3B
90.83%
1,375
602
578 (96.01%)
24 (3.99%)
GLM-5.2
89.71%
1,544
632
598 (94.62%)
34 (5.38%)
DeepSeek-V4-Pro
86.60%
2,010
698
653 (93.55%)
45 (6.45%)
Appendix
Table A3: Dual-judge agreement and human adjudication of sampled response–constraint disagreements for the ten models in the main evaluation. Agreement rates use 15,000 constraint judgments per model as the denominator. Adjudication percentages use the corresponding model’s reviewed sample count; the total row pools the sample counts. The fixed pool instead uses 582, 1,000, and 789 records for Kimi K3, GPT-5.6 Sol, and Claude Opus 4.6, respectively; the other seven models have the same adjudication counts.
Additional model
Extracted
Excluded
Valid
Covered
Uncovered
Coverage (%)
Gemini 3.1 Pro
9,085
42
9,043
8,595
448
95.05
DeepSeek-V3.2
8,967
48
8,919
8,565
354
96.03
gpt-oss-120b
11,601
24
11,577
11,177
400
96.54
Doubao-Seed-2.1-Pro
10,259
28
10,231
9,741
490
95.21
Pooled
39,912
142
39,770
38,078
1,692
95.75
Appendix
Table A4: Raw counts for catalog instance coverage on four additional models, each evaluated on 910 source questions. Excluded instances are classified as non-format; all mapping-error counts are zero. Covered instances include exact, subtype, and composite mappings. Coverage is covered divided by valid instances.
Task type
Source questions
Instruction variants
Mean TCPR
Mean EMCR
Highest EMCR (model: score)
Knowledge QA
163
754
73.54
56.59
Kimi K3: 73.63
Concept Explanation
105
559
73.22
56.00
GPT-5.6 Sol: 71.74
Information Extraction
93
362
73.00
59.59
GPT-5.6 Sol: 71.72
Decision Analysis
133
835
70.96
51.87
Qwen3.6-35B-A3B: 69.74
Solution Design
167
1008
68.10
50.23
Kimi K3: 65.72
Creative Writing
133
848
66.50
48.73
Kimi K3: 65.60
Appendix
Table A5: Format following across seven task types (%). Mean TCPR and mean EMCR assign equal weight to the ten models. The final column gives the highest EMCR point estimate and the corresponding model for each task. The seven tasks contain 910 source questions and 5,000 instruction variants; each variant has three constraints.
Model
Metric
Knowledge QA
Concept explanation
Information extraction
Decision analysis
Solution design
Creative writing
Code generation
Kimi K3
TCPR
87.69
83.43
84.76
84.05
79.95
83.01
78.06
EMCR
73.63
65.36
64.42
65.97
65.72
65.60
54.32
GPT-5.6 Sol
TCPR
85.92
85.93
87.37
84.91
80.12
80.86
71.40
EMCR
69.20
71.74
71.72
59.22
63.87
60.20
50.66
Gemini 3.5 Flash
TCPR
80.55
78.56
82.48
78.91
75.31
81.24
71.32
EMCR
62.23
66.66
67.88
59.38
56.25
63.56
55.24
Appendix
Table A6: Three-Constraint Pass Rate (TCPR, %) and Equal-Weight Multi-Constraint Correction Rate (EMCR, %) for ten models across seven task types. Bold indicates the highest value for each metric within each task type.
Model
Knowledge QA
Concept explanation
Information extraction
Decision analysis
Solution design
Creative writing
Code generation
Kimi K3
256 385 103
165 238 57
116 164 39
259 372 86
311 482 108
290 468 125
254 424 141
GPT-5.6 Sol
249 369 99
172 265 73
132 193 53
261 381 85
291 439 101
295 459 115
269 461 154
Gemini 3.5 Flash
211 284 67
144 201 46
104 152 36
238 318 62
282 410 83
299 488 134
253 405 135
Qwen3.6-35B-A3B
221 294 66
143 190 40
113 166 49
239 330 67
288 454 111
290 495 144
264 452 153
GLM-5.2
205 266 56
146 204 43
111 144 34
228 332 73
285 413 97
291 464 133
237 365 113
DeepSeek-V4-Pro
233 314 71
154 221 50
126 187 54
244 349 79
277 407 86
302 528 161
265 440 141
Appendix
Table A7: Eligible correction evaluation units by model and task. Read each cell from top to bottom: it lists the numbers of eligible target sets at K=1 / K=2 / K=3 , which form the denominators of the corresponding correction rates.
Model
Positive annotations
K=1
K=2
K=3
Kimi K3
8327
1,651 (3.99)
2,533 (6.66)
659 (8.92)
GPT-5.6 Sol
7559
1,669 (4.00)
2,567 (6.67)
680 (9.08)
Gemini 3.5 Flash
10970
1,531 (3.91)
2,258 (6.49)
563 (8.51)
Qwen3.6-35B-A3B
11642
1,558 (3.90)
2,381 (6.46)
630 (8.60)
GLM-5.2
10155
1,503 (3.88)
2,188 (6.38)
549 (8.40)
DeepSeek-V4-Pro
9052
1,601 (3.98)
2,446 (6.67)
642 (9.18)
Appendix
Table A8: Baseline annotations and correction evaluation units. Each model has 910 unconstrained responses. K-level cells give the eligible unit count followed by mean variant coverage ∣Iq,S∣ in parentheses. Each unit count is the denominator of its overall correction rate. Coverage counts a variant once even if several selected targets occur in it.
Table A9: Source-question counts by domain and task. Each of the 910 questions is counted once; zeros indicate empty cells. Domains are ordered by total count. Task order and abbreviations follow Figure 3 .
Dataset
Questions
License / use
WildBench
217
CC BY 4.0
ThoughtTrace
186
CC BY 4.0
Inverse IFEval
71
Evaluation use
ELI5
33
Unstated
ExpertQA
46
MIT
Appendix
Table A10: Sources of the 553 retained public-source questions. Licenses are those stated on the linked dataset pages; “Evaluation use” is a documented intended use rather than a license, and “Unstated” denotes no specified data license.
Variants
Questions
Target groups
1
256
3,008
2
147
1,139
3
98
683
4–5
119
610
6–10
152
466
>10
138
119
Appendix
Table A11: Variant multiplicity in the shared instruction set before applying model-specific baseline eligibility. The second column counts questions with that many retained variants; the third counts targets appearing in that many variants. Of 6,025 targets, 3,017 recur. The maximum is 30 variants per question and 24 per target.
Feature
Generic presence definition
Req.
Forb.
Bilingual parallel layout
Aligns corresponding source and translated text.
52
9
Column alignment
Organizes content in parallel columns or aligned rows.
1
2
Dialogue turns
Separates speakers or dialogue turns onto distinct lines.
219
83
Line breaks and spacing
Uses line breaks, blank lines, or compact spacing.
1,110
63
Paragraph or sentence count
Explicitly controls the number of paragraphs or sentences.
22
0
Poetry lines and stanzas
Groups verse into lines or stanzas.
3
3
Appendix
Table A12: Complete formatting catalog of 66 features in eight categories. Names and definitions are English translations of the catalog. Req. and Forb. count require and forbid occurrences in the retained 5,000 instructions; these counts describe instruction composition rather than model-specific correction eligibility.
Model
Temperature
Top- p
Top- k
Reasoning control
Output limit (tokens)
Thinking budget (tokens)
Qwen3.6-27B
0
1
0
Enabled
65,536
32,768
Qwen3-235B-A22B-Thinking
0
high
16,384
Qwen3.5-9B
0
1
0
Enabled
65,536
32,768
Appendix
Table A13: Generation settings for Qwen3.6-27B, Qwen3-235B-A22B-Thinking, and Qwen3.5-9B. Each row reports the predominant client-side configuration. high denotes the requested reasoning-effort level; “Enabled” denotes thinking mode with the specified token budget. Blank cells indicate parameters not explicitly set in the request. Top- k=0 disables top- k filtering. Each input is submitted as one user message without an additional system prompt, and one response is retained per benchmark item.
Item
Setting
Model scale
Public model identifiers/cards report Qwen3.6-27B (27B), Qwen3.6-35B-A3B (35B total, 3B active), Qwen3-235B-A22B-Thinking (235B/22B), Qwen3.5-9B (9B), and gpt-oss-120b (116.8B/5.1B).
Client requests
One user message per item, no additional system prompt; one final response retained. Explicit Qwen sampling and token settings are in Table A13 ; unspecified parameters follow the serving implementation.
Final-response workload
50,000 main, 9,100 unconstrained-baseline, 3,640 coverage, 15,000 single-constraint, and 5,000 prompting generations (82,740 in total).
Hardware
Locally deployed models used RTX 4080 SUPER and RTX PRO 6000 GPUs.
Software
The released inference and feature-extraction runners require Python 3.10+ and use only the Python standard library.
Appendix
Table A14: Reproduction settings and final answer-generation workload.
Figure 5: Example 1. Agreement preserves a passing response. Complete instruction, model response, independent judgments, and calibrated result.
Figure 6: Example 2. An unreviewed conflict receives a stratum estimate. Complete instruction, model response, independent judgments, and calibrated result.
Figure 7: Prompt 1: Core-catalog and open feature extraction (part 1 of 3). System prompt for annotating format features in unconstrained responses. English translation.
Figure 8: Prompt 1: Core-catalog and open feature extraction (part 2 of 3). System prompt for annotating format features in unconstrained responses. English translation.
Figure 9: Prompt 1: Core-catalog and open feature extraction (part 3 of 3). System prompt for annotating format features in unconstrained responses. English translation.
Figure 10: Prompt 2: Open feature extraction with eight categories. System prompt for open feature discovery on additional models. English translation.
Figure 11: Prompt 3: Three-constraint format grading (part 1 of 4). Shared grading prompt for GPT-5.4 Mini and Gemini 3.1 Flash-Lite. English translation.
Figure 12: Prompt 3: Three-constraint format grading (part 2 of 4). Shared grading prompt for GPT-5.4 Mini and Gemini 3.1 Flash-Lite. English translation.
Figure 13: Prompt 3: Three-constraint format grading (part 3 of 4). Shared grading prompt for GPT-5.4 Mini and Gemini 3.1 Flash-Lite. English translation.
Figure 14: Prompt 3: Three-constraint format grading (part 4 of 4). Shared grading prompt for GPT-5.4 Mini and Gemini 3.1 Flash-Lite. English translation.
Figure 15: Prompt 4: Single-constraint question reconstruction (part 1 of 3). Reconstruction prompt for preparing the single-constraint ablation questions. English translation.
Figure 16: Prompt 4: Single-constraint question reconstruction (part 2 of 3). Reconstruction prompt for preparing the single-constraint ablation questions. English translation.
Figure 17: Prompt 4: Single-constraint question reconstruction (part 3 of 3). Reconstruction prompt for preparing the single-constraint ablation questions. English translation.
Figure 18: Prompt 5: Qwen3.5-9B prompt enhancement. Meta-prompt appended to the question in the same user message for Qwen3.5-9B. English translation.
Practitioners make three prompt-design decisions with almost no controlled evidence behind them: how to format instructions and context (markdown, plain text, prose, or tabular), how many simultaneous instructions a system prompt can carry before compliance degrades, and how much context a model can hold before recall and honesty degrade. We report two controlled experiments crossing all three factors on one held, contamination-free synthetic corpus (the "Book of Veyra," 8,780 uniquely-named entities, deterministically regenerable from a fixed seed), evaluated across five models. Experiment 1 (960 calls/model) measures instruction-following decay as rule count N grows from 10 to 160, crossed with four formats and system-prompt vs. user-turn placement. Perfect-response rate collapses to zero by N=80 for every model, format, and placement. Placement produces effects at least as large as format at N=160 in most models, but the direction is model-specific. No model shows a reliable markdown advantage; one 35B model favors plain text instead. Experiment 2 (5,520 calls/model) measures recall accuracy, false-premise sycophancy, and absent-fact fabrication across a 2k-to-512k-token context ladder in the same four formats. Recall stays near ceiling through 64-128k tokens, then degrades sharply and format-dependently: one model's accuracy spread reaches 48 points at 128k tokens. Fabrication never occurs (0/5,760 probes), and sycophancy stays negligible (<=8.3%). What rises sharply near each model's context ceiling is outright refusal to answer (0% to 79-90%), distinct from sycophancy or fabrication. Neither pre-registered format ordering holds, and token overhead (+22% to +37% over plain text) further changes which format is preferable where accuracy spread is genuine. We release the full harness, corpus generator, and raw results (VeyraBench): https://github.com/iNetanel/veyrabench
Large language models often fail to satisfy formatting instructions when they must simultaneously perform demanding tasks. We study this behaviour through a prospective memory inspired lens from cognitive psychology, using a controlled paradigm that combines verifiable formatting constraints with benchmark tasks of increasing complexity. Across three model families and over 8,000 prompts, compliance drops by 2-21% under concurrent task load. Vulnerability is highly type-dependent: terminal constraints (requiring action at the response boundary) degrade most, with drops up to 50%, while avoidance constraints remain comparatively robust. A salience-enhanced format (explicit instruction framing plus a trailing reminder) recovers much of the lost compliance, restoring performance to 90-100% in many settings. Interference is bidirectional: formatting constraints can also reduce task accuracy, with one model's GSM8K accuracy dropping from 93% to 27%. In additional stacking experiments, joint compliance declines sharply as constraints accumulate. All results use deterministic programmatic checkers without an LLM-as-judge component on publicly available datasets.
Large language models often remain sensitive to answer format: a question solved correctly in one form may fail in another semantically equivalent form. To study this gap, we define cross-format robustness as the extent to which a model answers the same underlying question consistently across formats. We then compare full-format training with FormatMix, which expands only a subset of training items into multiple equivalent formats using either random or targeted selection. Across GLM4 and Llama-3.1, multi-format supervision consistently improves both task performance and cross-format robustness, whereas Multiple-choice question (MCQ)-only supervision alone brings little benefit and can even reduce robustness. We further find that expanding only about 30% of the training set into multiple formats often recovers most of the gain from full-format training, and this effect appears across the model families and sizes we study. These results suggest that format diversity, rather than additional supervision alone, is the key driver of robustness. That lightweight multi-format augmentation is a practical way to make LLMs less sensitive to answer format without changing the base model.
June M. Liu, Shaomian Zheng, He Cao +3
Ant Group · International Digital Economy Academy (IDEA)