We introduce CAPITU, a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese. Unlike existing benchmarks that focus on English or use generic prompts, CAPITU contextualizes all tasks within eight canonical works of Brazilian literature, combining verifiable instruction constraints with culturally-grounded content. The benchmark comprises 59 instruction types organized into seven categories, all designed to be automatically verifiable without requiring LLM judges or human evaluation. Instruction types include Portuguese-specific linguistic constraints (word termination patterns like -ando/-endo/-indo, -inho/-inha, -mente) and structural requirements. We evaluate 18 state-of-the-art models across single-turn and multi-turn settings. Our results show that frontier reasoning models achieve strong performance (GPT-5.2 with reasoning: 98.5% strict accuracy), while Portuguese-specialized models offer competitive cost-efficiency (Sabiazinho-4: 87.0% at $0.13 vs Claude-Haiku-4.5: 73.5% at $1.12). Multi-turn evaluation reveals significant variation in constraint persistence, with conversation-level accuracy ranging from 60% to 96% across models. We identify specific challenges in morphological constraints, exact counting, and constraint persistence degradation across turns. We release the complete benchmark, evaluation code, and baseline results to facilitate research on instruction-following in Portuguese.
Figures & tables
Figure 1: Overview of CAPITU’s main contributions.
Category
Benchmark
Year
PT-BR
Instruction Types
Multi-turn
Cultural
English IF
IFEval
2023
25
FollowBench
2024
5 levels
IFBench
2024
58
InFoBench
2024
Decomposed
Multilingual IF
Multi-IF
2024
†
25
Native Cultural
CIF-Bench (ZH)
2024
Cultural
Table 1: Instruction-following benchmarks comparison: scope and evaluation characteristics.
Figure 2: Overview of the CAPITU methodology
Category
Count
Example Instructions
Count
17
Word count range, exact word count, sentence count, paragraph count, character count
Words
11
Include word, word frequency, conjunctions, connectives, temporal markers
Pattern
8
Words ending in -ando/-endo/-indo, -inho/-inha, -ão/-ões, -mente
Forbidden
7
Forbidden word, no numbers, no first person, no questions
Structure
6
Start with word, end with word, acrostic, line prefix
Punctuation
5
Only declarative, include question, use semicolon
Table 2: Instruction categories in CAPITU with representative examples.
Table 4: Difficulty levels and constraint composition.
Figure 3: Example of an assembled prompt showing task template (top) and instruction constraints (bottom).
Figure 8
Benchmark Design
Single-turn
Multi-turn
Instruction types
59
Easy
70
Conversations
100
Instruction categories
7
Medium
70
Turns each
3
Literary works
8
Hard
60
Total turns
300
Literary movements
5
Total
200
Difficulty levels
3
Table 5: CAPITU benchmark statistics.
Model
Strict
Loose
Inst.
Coh.
Final
Proprietary Models
GPT-5 †
98.0
99.0
99.0
99.0
99.1
GPT-5-mini †
98.0
98.0
99.0
98.1
98.9
GPT-5.2 †
98.5
98.5
99.3
97.6
98.8
GPT-5.2
91.5
91.5
95.1
98.4
97.1
Gemini-3-Pro-Preview †
92.5
95.5
95.3
98.7
97.1
Table 6: Single-turn results on CAPITU. Strict / Loose = prompt-level accuracy (all instructions correct; Loose tests response variants with first/last line removal and markdown stripping); Instr. = instruction-level accuracy; Coh. = coherence score (GPT-4o-mini judge); Final = average of per-prompt scores computed via Equation ( 1 ). Cell backgrounds indicate performance.
Model
Easy
Medium
Hard
Δ (E → H)
Proprietary Models
GPT-5 †
100.0
97.1
96.7
–3.3
GPT-5-mini †
100.0
98.6
95.0
–5.0
GPT-5.2 †
100.0
98.6
96.7
–3.3
GPT-5.2
100.0
91.4
81.7
–18.3
Gemini-3-Pro-Preview †
95.7
94.3
86.7
–9.0
Table 7: Strict accuracy (%) by difficulty level. Easy = 1 instruction; Medium = 2 instructions; Hard = 3–4 instructions. Cell backgrounds indicate performance (see color scale below table).
Model
Conver.
Turn 1
Turn 2
Turn 3
Proprietary Models
GPT-5-mini †
96.0
100.0
98.0
98.0
Gemini-3-Pro-Preview †
95.0
100.0
99.0
96.0
Gemini-3-Flash-Preview
95.0
100.0
99.0
95.0
GPT-5 †
90.0
100.0
91.0
97.0
Claude-Sonnet-4.5
87.0
96.0
92.0
88.0
Table 8: Multi-turn evaluation. Conver. = conversation accuracy (all turns correct); Turn columns show per-turn strict accuracy with accumulating constraints. Cell backgrounds indicate performance (see color scale below table).
Instruction
Avg. Accuracy (%)
count:exact_word_count
21.1
words:max_word_repeat
31.6
count:character_count_range
47.4
structure:acrostic
66.3
pattern:terminacao_inho_inha_min
67.8
punctuation:include_quote
73.7
Table 9: Top 10 hardest instructions (average accuracy across all models).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Instruction ID
Description
Count (17)
count:word_count_range
Response must have between X and Y words
count:exact_word_count
Response must have exactly N words
count:min_word_count
Response must have at least N words
count:max_word_count
Response must have at most N words
count:unique_word_count
Use at least N unique words
Appendix
Table 10: Complete instruction types in CAPITU (59 total).
Figure 4: Complete coherence evaluation prompt structure. The system message defines the judge’s role; the user message provides the original prompt, the model’s response, and literary metadata as context; the judge returns a JSON score with justification.