Instruction-following benchmarks remain predominantly English-centric, leaving a critical evaluation gap for the hundreds of millions of Indic language speakers. We introduce IndicIFEval, a benchmark evaluating constrained generation of LLMs across 14 Indic languages using automatically verifiable, rule-based instructions. It comprises around 800 human-verified examples per language spread across two complementary subsets: IndicIFEval-Ground, translated prompts from IFEval (Zhou et al., 2023) carefully localized for Indic contexts, and IndicIFEval-Ground, synthetically generated instructions grounded in native Indic content. We conduct a comprehensive evaluation of major open-weight and proprietary models spanning both reasoning and non-reasoning models. While models maintain strong adherence to formatting constraints, they struggle significantly with lexical and cross-lingual tasks -- and despite progress in high-resource languages, instruction-following across the broader Indic family lags significantly behind English. We release IndicIFEval and its evaluation scripts to support progress on multilingual constrained generation (http://github.com/ai4bharat/IndicIFEval).
Figures & tables
Figure 1: Overview of the dataset construction pipeline
Model
as
bn
gu
hi
kn
ml
mr
ne
or
pa
sa
ta
te
ur
en
Avg
gemini/gemini-3-flash
90.0
90.0
89.7
89.7
86.3
84.7
88.8
89.7
86.9
89.4
79.4
85.7
89.4
85.4
96.3
87.5
gemini/gemini-3-pro
89.4
92.5
91.3
91.0
91.3
86.9
91.3
92.2
86.0
93.2
86.3
89.7
90.3
89.1
95.3
90.0
openai/gpt-5-2025-08-07
88.8
90.7
89.4
92.8
89.1
90.0
89.7
91.0
89.1
90.3
80.4
86.9
88.5
87.5
95.0
88.9
openai/gpt-5-mini-2025-08-07
77.9
86.0
84.4
88.8
72.9
76.3
81.6
82.9
75.4
84.1
70.1
75.1
80.1
81.3
91.9
79.8
CohereLabs/aya-expanse-8b
19.0
24.0
17.4
53.6
20.2
21.8
24.3
20.2
19.0
18.7
21.2
24.9
18.4
38.9
70.4
24.4
CohereLabs/aya-expanse-32b
33.0
42.7
33.0
70.4
30.8
39.9
32.7
37.7
35.5
32.1
29.9
41.4
26.5
57.6
77.3
38.8
Table 1: Overall Results for IndicIFEval-Trans (avg calculated only for Indic languages)
Figure 2: Effect of Increasing Model Parameters. The higher the better – The Gemma-3-27B-IT model performs the best, followed by Llama-4-Scout-17B-16E.
Figure 4
Figure 3: Indic - English disparity comparing Language vs Model Family. The lower the better – the Aya family exhibits the highest gap in general, but performs moderately well for Hindi, Bengali, Urdu, and Tamil.
Figure 4: Indic - English gap comparing Model Family vs Instruction Category. Lower the better – the Llama family and Gemma family perform the best and worst respectively on Language constraint within the family.
Model
as
bn
gu
hi
kn
ml
mr
ne
or
pa
sa
ta
te
ur
Avg
gemini/gemini-3-flash
64.6
67.0
60.7
61.9
58.1
56.2
62.9
66.9
62.8
64.5
55.6
64.9
64.3
59.6
62.1
gemini/gemini-3-pro
62.3
66.0
61.0
58.9
61.0
59.4
62.0
64.2
67.9
65.5
50.9
66.0
70.4
60.2
62.6
openai/gpt-5-2025-08-07
73.2
72.6
77.5
77.9
73.6
70.6
78.0
72.4
80.4
78.5
78.3
78.1
78.4
72.7
75.9
openai/gpt-5-mini-2025-08-07
75.5
71.2
74.5
75.5
71.5
68.2
81.7
75.7
78.8
77.0
79.5
79.1
78.7
73.6
75.8
CohereLabs/aya-expanse-8b
38.1
46.0
38.2
48.4
41.1
43.2
42.0
40.5
39.1
43.8
25.6
52.1
39.5
40.4
41.3
CohereLabs/aya-expanse-32b
59.9
56.2
46.1
54.9
45.8
55.5
53.1
50.7
47.8
42.5
36.0
58.4
48.0
42.4
49.8
Table 2: Overall Results for IndicIFEval-Ground
Figure 5: Indic - English disparity comparing Instruction Category vs Language. Lower the better – Hindi (‘hi’) is visually the lightest (lowest Δ ) across all categories horizontally and darkest (highest Δ ) across all languages vertically.
Figure 6: Indic vs English Disparity comparing Open-Weight and Proprietary. Lower the better – Hindi (‘hi’) exhibits the lowest gap with English in both proprietary (red) and open-weight settings (blue).
Figure 7: Indic vs English Disparity comparing Thinking and Non-Thinking Modes. A steeper negative slope indicates that thinking more effectively narrows the performance gap between English and Indic languages.
Table 3: Taxonomy of IFEval instruction constraints grouped into five high-level categories for analysis.
Table 4: Illustrative examples of translation errors observed during our automatic translation from the English prompts.
Figure 8: Human Verification for IndicIFEval-Trans conducted on Google Sheets.
Figure 9: Human Verification for IndicIFEval-Ground conducted on Google Sheets.
IndicIFEval-Ground Instruction Tasks
News Article
Step-by-Step Instructions
Articles or Blog Post
Social Media Post
Essay
Technical Report
Poem
Question Generation
Creative Story
Email
Dialogue Generation
Summarization
Appendix
Table 5: List of generation tasks used for prompt diversification of the IndicIFEval-Ground dataset. Red color indicates instructions that require the context be included in the generated prompt; blue represents tasks that require no context for answering.
Figure 10: Proportion of the instruction tasks in IndicIFEval-Ground , averaged across languages.
Language
# Correct Prompts
Assamese
257/439
Bengali
409/502
Gujarati
377/446
Hindi
457/527
Kannada
341/392
Malayalam
350/446
Appendix
Table 6: Per-language prompt verification statistics for IndicIFEval-Ground .
Language
# Correct Translations (Total 490)
# Keyword Corrections (Total 200)
Assamese
440
38
Bengali
454
21
Gujarati
482
8
Hindi
487
9
Kannada
457
27
Malayalam
490
5
Appendix
Table 7: Per-language prompt verification statistics for IndicIFEval-Trans . Parallely across languages, there are 321 correct translations.
Figure 11: Prompt template used to generate instructions for Keyword Inclusion & Frequency.
Figure 12: Prompt template used to generate instructions for First Word constraints.
Figure 13: Prompt template used to generate instructions for Sentence & Paragraph Count constraints.
Figure 14: Prompt template used to generate instructions Forbidden Keyword constraints.
Table 8: Examples of prompt-constraint pairs from the IndicIFEval-Trans dataset. The final column contains the original, unmodified prompt from the IFEval dataset Zhou et al. (2023) .
Table 9: Examples of prompt-constraint pairs from the IndicIFEval-Ground dataset.