IndicIPR-QA: A Benchmark for Extractive Question Answering on Indian IPR Laws

M. Mohaiminul Islam; Muhammad Abulaish

cs.CLOct 29, 2026

IndicIPR-QA: A Benchmark for Extractive Question Answering on Indian IPR Laws

Authors: M. Mohaiminul Islam, Muhammad Abulaish

Organizations: Department of Computer Science and Engineering, South Asian University, New Delhi, India

Abstract

Legal Question Answering (QA) over statutory texts is a challenging task in Natural Language Processing (NLP), especially in low-resource settings with complex legal systems. It requires answers that are precise and grounded in the law. However, to the best of current knowledge, no existing extractive QA benchmark provides exact answer spans from statutory texts across all seven Indian Intellectual Property Rights (IPR) Acts. This paper presents IndicIPR-QA1, an extractive QA benchmark for Indian IPR laws. The proposed dataset contains 4,462 QA pairs from 1,116 statutory contexts across seven Acts. Questions and answers are generated using GPT-4o and validated through nine automatic quality checks, single and multi-model LLM evaluations, and a 200-pair human audit. Section-level partitioning prevents source-context overlap across the training, validation, and test sets. Further analysis examines answer span length, lexical overlap, and multi-clause structure across the corpus and individual Acts. BM25 achieves a token-level F1 score of 20.21, while fine-tuned InLegalBERT reaches 85.02 under a unified sliding-window inference protocol. The large performance gap between BM25 (20.21 F1) and InLegalBERT (85.02 F1) shows that simple word matching is not enough to answer these legal questions. Performance also varies across the seven Acts, indicating substantial differences in answer-extraction performance.

Figures & tables

Fig. 1: End-to-end IndicIPR-QA construction. Human-audit percentages denote joint acceptability on the 200-pair sample; LLM scores are post-hoc diagnostics.

ID	Filter criterion	Sequential filtering		Per-filter ablation
ID	Filter criterion	Removed	Retained	Recovered pairs	Accepted
F1	Required fields	0	5,721	0 (0.0%)	4,689
F2	Answer in context	0	5,721	0 (0.0%)	4,689
F3	Answer length 5–350 †	595	5,126	368 (7.8%)	5,057
F4	Coverage $<$ 85% ‡	417	4,709	417 (8.9%)	5,106
F5	Question quality	13	4,696	13 (0.3%)	4,702

TABLE I: Quality-filter analysis of the 5,721 raw GPT-4o generated candidate QA pairs. Sequential filtering applies filters F1 to F9 in order. Per-filter ablation omits each filter individually while keeping the remaining filters active.

Training variant	F1	Char-F1	$\Delta$ F1
clean selected (ablation control)	84.11	86.30	—
without F3+F4 combined	84.58	87.25	$+$ 0.47
without F3 (answer length)	84.56	87.13	$+$ 0.45
without F4 (coverage)	83.90	86.66	$-$ 0.21
raw candidates (unfiltered)	83.44	85.64	$-$ 0.67
heuristic all (no selection)	82.77	86.59	$-$ 1.34

TABLE II: Model level filter ablation using InLegalBERT on different training data variants. All archived checkpoints are evaluated with the unified decoder;

\Delta

F1 is relative to the clean selected run.

	Single Judge		Ensemble
Criterion	Avg. score	Flagged	Avg. score	Flagged
Question specificity	2.92	1.1%	2.93	0.0%
Answer sufficiency	2.92	2.8%	2.99	0.4%
Answer minimality	2.52	4.2%	2.85	0.4%
Legal relevance	2.97	0.7%	3.00	0.0%
Flagged QA pairs	222 pairs (5.0%)		26 pairs (0.6%)

TABLE III: LLM as Judge evaluation with a single judge and a three model ensemble. Scores range from 1 (poor/flagged) to 3 (good).

Criterion	$\kappa_{\text{UW}}$	$\kappa_{\text{QW}}$	Accept. %
Question Clarity	$-0.008$	0.051	99.0% †
Answer Correctness	0.117	0.328	95.0%
Answer Minimality	0.312	0.613	75.5% ‡

TABLE IV: Human inter-annotator agreement on 200 QA pairs.

\kappa_{\text{UW}}

= unweighted Cohen’s

\kappa

;

\kappa_{\text{QW}}

= quadratic-weighted Cohen’s

\kappa

Fig. 2: Criterion-wise inter-annotator agreement for the human validation study. Answer minimality has the strongest agreement, while question clarity shows a ceiling effect due to skewed marginal distributions.

Property	SQ1	SQ2	CUAD	JEC	IPB	IndicIPR-QA (Ours)
QA pairs	87.6k	150k	13.1k	26.4k	10.4k	4.5k
Contexts	18.9k	35k	510	—	—	1,116
QA/context	4.6	4.3	25.7	—	—	4.0
Ans len (wds)	3.2	3.2	30.0	MCQ	Mixed	35.6
Ctx len (wds)	117	117	200	—	200	171.3
Q len (wds)	10.1	10.1	9.5	—	—	11.9

TABLE V: Comparison of IndicIPR-QA with related QA benchmarks (SQ1 = SQuAD 1.1 [ 1 ] ; SQ2 = SQuAD 2.0 [ 2 ] ; CUAD [ 3 ] ; JEC = JEC-QA [ 4 ] ; IPB = IPBench [ 10 ] ). “—” indicates unavailable or not directly comparable statistics. Values follow the reporting conventions of each benchmark’s source and may use different dataset split definitions.

Act	$\mathbf{n_{all}}$	$\mathbf{n_{test}}$	Avg. ans. (words)	Q-Ctx	F1
Designs	173	24	43.4	0.320	100.00
Patents	1,143	69	35.8	0.429	98.40
Semicond.	585	47	22.1	0.418	91.53
Trade Marks	1,106	88	31.9	0.416	83.83
GI	551	51	51.8	0.458	83.66
Copyright	349	25	62.3	0.447	79.41

TABLE VI: Per-Act IndicIPR-QA statistics, sorted by test F1.

n_{\mathrm{all}}

and

n_{\mathrm{test}}

denote full-dataset and test-set counts; Q-Ctx denotes stopword removed question-context unigram overlap.

Lexical retrieval
System	EM	F1	Char-F1
BM25 retrieval	0.00	20.21	77.10
BM25 oracle window	14.24	51.30	85.23
Fine-tuned encoders
BERT-base	63.66	83.33	86.38
LegalBERT-US	65.12	84.66	86.67

TABLE VII: Baseline performance on the IndicIPR-QA test split (344 QA pairs). ZS = zero-shot; FS = 3-shot few-shot.

Fig. 3: Token-level F1 across encoder and lexical baselines and the best prompted setting for each open LLM on the IndicIPR-QA test split. The large gap between BM25 and InLegalBERT shows that lexical matching alone is insufficient for the benchmark.

Fig. 4: Novel test-context rate by Act. The 100% rate confirms that no source context is shared between the training and test splits.

Question Answering

Question answering studies how AI systems interpret questions and produce accurate, evidence-grounded answers across text, dialogue, and multimodal data.

See all 263 papers

AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal System
Jul 21, 2026Shubham Kumar Nigam, Shubham Kumar Mishra, Noel Shallum +2Retrieval-Augmented Generation Legal NLP
Computer Science and Engineering, Indian Institute of Technology, Kanpur, 208016, Uttar Pradesh, India. · School of Computer Science, University of Birmingham, Dubai, UAE. · Law, Symbiosis Law School, Pune, 411014, Maharashtra, India. +1
L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages
Aug 16, 2026Rinit Jain, Tirthraj Mahajan, Advait Joshi +1Multilingual QA Factual Knowledge in Language Models
Pune Institute of Computer Technology, Pune · L3Cube Labs, Pune · Indian Institute of Technology Madras
ViLegalExpert: A Large-Scale Benchmark for Vietnamese Legal Retrieval and Question Answering from Real-World Consultations
Sep 30, 2026Dat Tien Nguyen, Nghia Hieu Nguyen, Anh Thi-Hoang Nguyen +3Legal QA Legal NLP
University of Information Technology · Vietnam National University, Ho Chi Minh city, Viet Nam

Jul 21, 2026J/K move · Enter open · S save

AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal System

Shubham Kumar Nigam, Shubham Kumar Mishra, Noel Shallum, Kripabandhu Ghosh+1

Computer Science and Engineering, Indian Institute of Technology, Kanpur, 208016, Uttar Pradesh, India. · School of Computer Science, University of Birmingham, Dubai, UAE. · Law, Symbiosis Law School, Pune, 411014, Maharashtra, India.+1

Retrieval-Augmented Generation Legal NLP Legal QA RAG Evaluation

Open paper AskPDF arXiv

This comprehensive study introduces an advanced Artificial Intelligence for Indian Legal Question Answering (AILQA) system tailored to the Indian legal context. AILQA leverages a variety of embedding and generative models, including recent Large Language Models (LLMs), to address the unique challenges posed by the intricate and diverse nature of Indian legal texts and to enhance the accuracy and reliability of responses to legal questions. We conducted rigorous evaluations using both lexical and semantic metrics, enriched by expert legal feedback, to ensure relevance and accuracy. Our findings underscore the effectiveness of the Retrieval-Augmented Generation (RAG) paradigm in improving answer quality, particularly in complex legal domains. Additionally, we assessed performance on standardized tests such as the All India Bar Examination (AIBE), thereby providing a robust benchmark for practical applications. Under the study's evaluation protocol, some AI-generated responses received higher ratings than the available reference answers, particularly when they contained accurate and relevant supporting details. This finding is specific to the evaluated dataset and rating criteria and should not be interpreted as evidence that the models generally outperform qualified legal professionals. We also discuss the challenges encountered, such as the need for precise context and the risks of model hallucination, and propose directions for future research to further refine AI capabilities in the legal field. This study aims to pave the way for enhanced legal decision-support systems, making them more accessible and effective for legal professionals and the public alike.

Abstract

Question Answering

Explore similar work

AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal System

L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages

ViLegalExpert: A Large-Scale Benchmark for Vietnamese Legal Retrieval and Question Answering from Real-World Consultations