We introduce CAPITU, a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese. Unlike existing benchmarks that focus on English or use generic prompts, CAPITU contextualizes all tasks within eight canonical works of Brazilian literature, combining verifiable instruction constraints with culturally-grounded content. The benchmark comprises 59 instruction types organized into seven categories, all designed to be automatically verifiable without requiring LLM judges or human evaluation. Instruction types include Portuguese-specific linguistic constraints (word termination patterns like -ando/-endo/-indo, -inho/-inha, -mente) and structural requirements. We evaluate 18 state-of-the-art models across single-turn and multi-turn settings. Our results show that frontier reasoning models achieve strong performance (GPT-5.2 with reasoning: 98.5% strict accuracy), while Portuguese-specialized models offer competitive cost-efficiency (Sabiazinho-4: 87.0% at $0.13 vs Claude-Haiku-4.5: 73.5% at $1.12). Multi-turn evaluation reveals significant variation in constraint persistence, with conversation-level accuracy ranging from 60% to 96% across models. We identify specific challenges in morphological constraints, exact counting, and constraint persistence degradation across turns. We release the complete benchmark, evaluation code, and baseline results to facilitate research on instruction-following in Portuguese.
Figures & tables
Figure 1: Overview of CAPITU’s main contributions.
Category
Benchmark
Year
PT-BR
Instruction Types
Multi-turn
Cultural
English IF
IFEval
2023
25
FollowBench
2024
5 levels
IFBench
2024
58
InFoBench
2024
Decomposed
Multilingual IF
Multi-IF
2024
†
25
Native Cultural
CIF-Bench (ZH)
2024
Cultural
Table 1: Instruction-following benchmarks comparison: scope and evaluation characteristics.
Figure 2: Overview of the CAPITU methodology
Category
Count
Example Instructions
Count
17
Word count range, exact word count, sentence count, paragraph count, character count
Words
11
Include word, word frequency, conjunctions, connectives, temporal markers
Pattern
8
Words ending in -ando/-endo/-indo, -inho/-inha, -ão/-ões, -mente
Forbidden
7
Forbidden word, no numbers, no first person, no questions
Structure
6
Start with word, end with word, acrostic, line prefix
Punctuation
5
Only declarative, include question, use semicolon
Table 2: Instruction categories in CAPITU with representative examples.
Table 4: Difficulty levels and constraint composition.
Figure 3: Example of an assembled prompt showing task template (top) and instruction constraints (bottom).
Figure 8
Benchmark Design
Single-turn
Multi-turn
Instruction types
59
Easy
70
Conversations
100
Instruction categories
7
Medium
70
Turns each
3
Literary works
8
Hard
60
Total turns
300
Literary movements
5
Total
200
Difficulty levels
3
Table 5: CAPITU benchmark statistics.
Model
Strict
Loose
Inst.
Coh.
Final
Proprietary Models
GPT-5 †
98.0
99.0
99.0
99.0
99.1
GPT-5-mini †
98.0
98.0
99.0
98.1
98.9
GPT-5.2 †
98.5
98.5
99.3
97.6
98.8
GPT-5.2
91.5
91.5
95.1
98.4
97.1
Gemini-3-Pro-Preview †
92.5
95.5
95.3
98.7
97.1
Table 6: Single-turn results on CAPITU. Strict / Loose = prompt-level accuracy (all instructions correct; Loose tests response variants with first/last line removal and markdown stripping); Instr. = instruction-level accuracy; Coh. = coherence score (GPT-4o-mini judge); Final = average of per-prompt scores computed via Equation ( 1 ). Cell backgrounds indicate performance.
Model
Easy
Medium
Hard
Δ (E → H)
Proprietary Models
GPT-5 †
100.0
97.1
96.7
–3.3
GPT-5-mini †
100.0
98.6
95.0
–5.0
GPT-5.2 †
100.0
98.6
96.7
–3.3
GPT-5.2
100.0
91.4
81.7
–18.3
Gemini-3-Pro-Preview †
95.7
94.3
86.7
–9.0
Table 7: Strict accuracy (%) by difficulty level. Easy = 1 instruction; Medium = 2 instructions; Hard = 3–4 instructions. Cell backgrounds indicate performance (see color scale below table).
Model
Conver.
Turn 1
Turn 2
Turn 3
Proprietary Models
GPT-5-mini †
96.0
100.0
98.0
98.0
Gemini-3-Pro-Preview †
95.0
100.0
99.0
96.0
Gemini-3-Flash-Preview
95.0
100.0
99.0
95.0
GPT-5 †
90.0
100.0
91.0
97.0
Claude-Sonnet-4.5
87.0
96.0
92.0
88.0
Table 8: Multi-turn evaluation. Conver. = conversation accuracy (all turns correct); Turn columns show per-turn strict accuracy with accumulating constraints. Cell backgrounds indicate performance (see color scale below table).
Instruction
Avg. Accuracy (%)
count:exact_word_count
21.1
words:max_word_repeat
31.6
count:character_count_range
47.4
structure:acrostic
66.3
pattern:terminacao_inho_inha_min
67.8
punctuation:include_quote
73.7
Table 9: Top 10 hardest instructions (average accuracy across all models).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Instruction ID
Description
Count (17)
count:word_count_range
Response must have between X and Y words
count:exact_word_count
Response must have exactly N words
count:min_word_count
Response must have at least N words
count:max_word_count
Response must have at most N words
count:unique_word_count
Use at least N unique words
Appendix
Table 10: Complete instruction types in CAPITU (59 total).
Figure 4: Complete coherence evaluation prompt structure. The system message defines the judge’s role; the user message provides the original prompt, the model’s response, and literary metadata as context; the judge returns a JSON score with justification.
Instruction-following benchmarks remain predominantly English-centric, leaving a critical evaluation gap for the hundreds of millions of Indic language speakers. We introduce IndicIFEval, a benchmark evaluating constrained generation of LLMs across 14 Indic languages using automatically verifiable, rule-based instructions. It comprises around 800 human-verified examples per language spread across two complementary subsets: IndicIFEval-Ground, translated prompts from IFEval (Zhou et al., 2023) carefully localized for Indic contexts, and IndicIFEval-Ground, synthetically generated instructions grounded in native Indic content. We conduct a comprehensive evaluation of major open-weight and proprietary models spanning both reasoning and non-reasoning models. While models maintain strong adherence to formatting constraints, they struggle significantly with lexical and cross-lingual tasks -- and despite progress in high-resource languages, instruction-following across the broader Indic family lags significantly behind English. We release IndicIFEval and its evaluation scripts to support progress on multilingual constrained generation (http://github.com/ai4bharat/IndicIFEval).
Thanmay Jayakumar, Mohammed Safi Ur Rahman Khan, Raj Dabre +2
Nilekani Centre at AI4Bharat · Indian Institute of Technology Madras, India · IT University of Copenhagen +1
Instruction-following is a foundational capability of large language models (LLMs), with its improvement hinging on scalable and accurate feedback from judge models. However, the reliability of current judge models in instruction-following remains underexplored due to several deficiencies of existing meta-evaluation benchmarks, such as their insufficient data coverage and oversimplified pairwise evaluation paradigms that misalign with model optimization scenarios. To this end, we propose IF-RewardBench, a comprehensive meta-evaluation benchmark for instruction-following that covers diverse instruction and constraint types. For each instruction, we construct a preference graph containing all pairwise preferences among multiple responses based on instruction-following quality. This design enables a listwise evaluation paradigm that assesses the capabilities of judge models to rank multiple responses, which is essential in guiding model alignment. Extensive experiments on IF-RewardBench reveal significant deficiencies in current judge models and demonstrate that our benchmark achieves a stronger positive correlation with downstream task performance compared to existing benchmarks. Our codes and data are available at https://github.com/thu-coai/IF-RewardBench.
Bosi Wen, Yilin Niu, Cunxiang Wang +5
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University · Zhipu AI · University of Electronic Science and Technology of China
Common instruction following (IF) benchmarks, such as IFEval, evaluate the ability of language models (LMs) to follow simple constraints that can easily be verified programmatically. These same constraints, in turn, are used as reward signals for reinforcement learning. This design may lead to reward hacking of constraints, pushing models away from the main task. Recent attempts to address this resulted in benchmarks with richer constraints, but required expert labor, remained closed-access, and cannot be easily extended. We present Lift-Bench, an IF benchmark that asks a model to draft a document given a topic and a structured outline. Constraints are scoped to certain document sections, providing finer control over constraint density. Lift-Bench includes both verifiable and judged constraints, as well as a training counterpart, Lift-Train. Importantly, Lift-Bench provides both main task and instruction-following constraints, mitigating shallow reward hacking. On experiments with multiple models, we find that frontier models struggle with the harder subsets of our benchmark and analyze model performance along multiple axes. We also find that training on Lift-Train is much less susceptible to reward hacking than training on the common IFTrain. Lift-Bench and Lift-Train are publicly available, as well as the data generating pipeline.