IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models

Saurabh Bharti; Gaurav Azad; Abhinaw Jagtap; Nachiket Tapas

cs.CLFeb 18, 2026

IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models

Authors: Saurabh Bharti, Gaurav Azad, Abhinaw Jagtap, Nachiket Tapas

Organizations: Department of Computer Sciencce and Engineering, Chhattisgarh Swami Vivekanand Technical University Bhilai, India

Abstract

The rapid advancement of large language models (LLMs) necessitates evaluation frameworks that reflect real-world academic rigor and multilingual complexity. This paper introduces IndicEval, a scalable benchmarking platform designed to assess LLM performance using authentic high-stakes examination questions from UPSC, JEE, and NEET across STEM and humanities domains in both English and Hindi. Unlike synthetic benchmarks, IndicEval grounds evaluation in real examination standards, enabling realistic measurement of reasoning, domain knowledge, and bilingual adaptability. The framework automates assessment using Zero-Shot, Few-Shot, and Chain-of-Thought (CoT) prompting strategies and supports modular integration of new models and languages. Experiments conducted on Gemini 2.0 Flash, GPT-4, Claude, and LLaMA 3-70B reveal three major findings. First, CoT prompting consistently improves reasoning accuracy, with substantial gains across subjects and languages. Second, significant cross-model performance disparities persist, particularly in high-complexity examinations. Third, multilingual degradation remains a critical challenge, with marked accuracy drops in Hindi compared to English, especially under Zero-Shot conditions. These results highlight persistent gaps in bilingual reasoning and domain transfer. Overall, IndicEval provides a practice-oriented, extensible foundation for rigorous, equitable evaluation of LLMs in multilingual educational settings and offers actionable insights for improving reasoning robustness and language adaptability.

Figures & tables

Framework	Real Exam Data	Multilingual	Bilingual Cognition	Domain Depth	Exam Realism
GLUE	$\times$	Limited	$\times$	Low	$\times$
SuperGLUE	$\times$	Limited	$\times$	Low	$\times$
AGIEval	$\checkmark$	$\times$	$\times$	Moderate	Partial
HELM	$\times$	Partial	$\times$	Low	$\times$
M3Exam	$\checkmark$	$\checkmark$	$\times$	Moderate	Partial
SciExam	$\checkmark$	Limited	$\times$	High (STEM)	Partial

Table 1: Comparison of Major LLM Benchmarking Frameworks

Examination	Subject	English	Hindi
UPSC	Polity	125	120
UPSC	History	110	115
JEE	Physics	95	90
JEE	Chemistry	100	100
JEE	Mathematics	85	80
NEET	Biology	105	110

Table 2: Dataset Composition by Examination, Subject, and Language

Model	Subject	Language	Zero-Shot	CoT
Gemini 2.0 Flash	Chemistry	Hindi	44.00	71.05
Gemini 2.0 Flash	UPSC	English	87.00	89.00
LLaMA 3-70B	UPSC	English	67.00	80.00
LLaMA 3-70B	UPSC	Hindi	39.53	68.60
Gemini 2.0 Flash	JEE Physics	English	67.59	75.93
Gemini 2.0 Flash	NEET Biology	Hindi	86.02	91.40

Table 3: Effect of Chain-of-Thought Prompting on Accuracy (%)

Subject	Language	Gemini 2.0 Flash			LLaMA 3-70B
Subject	Language	CoT	Few	Zero	CoT	Few	Zero
JEE Physics	English	75.93	51.85	67.59	68.52	48.15	61.11
JEE Physics	Hindi	77.06	59.63	52.29	64.22	43.12	57.80
JEE Chemistry	English	76.32	69.30	67.54	67.54	55.26	70.18
JEE Chemistry	Hindi	71.05	57.89	64.91	60.53	42.98	54.39
NEET Physics	English	80.95	78.57	85.71	64.29	52.38	71.43

Table 4: Accuracy (%) Across Subjects, Languages, and Prompting Strategies

Education

See all 46 papers

Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context
Jul 28, 2026Abhishek Kumar Singh, Shrey Nag, Sachita +2LLM Safety Benchmarks Multilingual Language Model Evaluation
GTBIT¹ · TIET² · IGDTUW³ +2
L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages
Aug 16, 2026Rinit Jain, Tirthraj Mahajan, Advait Joshi +1Multilingual QA Factual Knowledge in Language Models
Pune Institute of Computer Technology, Pune · L3Cube Labs, Pune · Indian Institute of Technology Madras
IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages
Feb 25, 2026Thanmay Jayakumar, Mohammed Safi Ur Rahman Khan, Raj Dabre +2Instruction Following Multilingual Language Model Evaluation
Nilekani Centre at AI4Bharat · Indian Institute of Technology Madras, India · IT University of Copenhagen +1

Jul 28, 2026J/K move · Enter open · S save

Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context

Abhishek Kumar Singh, Shrey Nag, Sachita, Lipi Goel+1

GTBIT¹ · TIET² · IGDTUW³+2

LLM Safety Benchmarks Multilingual Language Model Evaluation LLM Evaluation Cross-Cultural Language Model Evaluation Cultural Bias in Language Models LLM Safety Evaluation

Open paper AskPDF arXiv

India is a vast nation of over 1.4 billion people, varied by hundreds of diverse and locally specific traditions and cultures and 22 officially recognized languages. Large language models (LLMs) are now being deployed on a massive scale throughout the mainland as well as in remote villages. However, the common benchmarks - MMLU, BIG-Bench, and TruthfulQA are almost exclusively English- and Western-centric. They do not identify those safety, fairness, and accuracy failures unique to the Indian context. That is the gap Inspect India Evals seeks to fill. It is an open-source framework built on top of UK AISI's Inspect AI platform. It has six benchmarks: Multilingual MMLU across sixteen Indian languages, BharatBBQ (our adaptation of BBQ for Indian social bias), a safety evaluation for Digital Public Infrastructure, a multilingual safety test using harmful prompts in Indian languages, a multi-turn jailbreak resistance test, and an Indian cultural knowledge benchmark scored using LLM-as-judge rubrics. In this study, we tested five open-weight models ranging from 8B to 32B parameters. Sarvam-M 24B and Gemma 2 27B came out on top, both scoring 80% on the composite India Fairness Index, with Sarvam-M even beating larger 32B models on Indian cultural knowledge and DPI safety compliance. All models scored 100% refusal on Multilingual Safety, whereas DPI safety varied from 20% to 100%. The framework is public. It's built to work with the UK AISI registry. Anyone can reproduce or extend this work.

Abstract

Education

Explore similar work

Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context

L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages

IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages