IndoNutriQA: An Indonesian Clinical Nutrition Question Answering Benchmark

Tirana Noor Fatyanosa; Ummu Ditya Erliana; Mahendra Data; Muhammad Nurwegiono; Eleazar Tadeo Eman; Alham Fikri Aji

cs.CLOct 22, 2026

IndoNutriQA: An Indonesian Clinical Nutrition Question Answering Benchmark

Authors: Tirana Noor Fatyanosa, Ummu Ditya Erliana, Mahendra Data, Muhammad Nurwegiono, Eleazar Tadeo Eman, Alham Fikri Aji

Organizations: Universitas Brawijaya · Universitas Ma Chung · MBZUAI

Abstract

Clinical Question Answering (QA) underpins Clinical Intelligent Decision Support Systems, but progress is severely limited for low-resource languages such as Indonesian by the absence of public domain-specific data. We introduce IndoNutriQA, an expert-curated Indonesian clinical QA benchmark of 911 question-answer pairs built from real clinical-nutrition reports. Rather than a large-scale training corpus, it is positioned as a high-quality evaluation resource: every pair is grounded in a clinically relevant context, enriched with structured metadata (anthropometric assessment, medical diagnosis, document provenance, chunk segmentation). Eight standardised question types following the Nutrition Care Process (NCP), including the Problem, Etiology, Sign and Symptom (PES) diagnosis statement, are answered for every report, which was written by a graduate-level dietetic intern, by one of three nutritionists. Answers are free-form clinical prose rather than extractive spans, and an expert validation study finds 6-10% of reference answers contestable. The dataset supports generative, extractive and retrieval-augmented QA as well as zero- and few-shot LLM evaluation. We report fine-tuned baselines (IndoBERT variants, mT5, RAG) with 5-fold cross-validation, learning curves and bootstrap 95% confidence intervals, and benchmark eight contemporary LLMs zero- and five-shot. We further calibrate our LLM-as-judge rubric against three expert raters (one RD, two nutritionists), finding it human-comparable on factuality and completeness but unreliable on clinical safety, and a human error analysis shows it misses unsupported content that the raters identify. IndoNutriQA is released to foster research in Indonesian clinical NLP and nutritional decision support.

Figures & tables

1. Screening method	5. Dietary goals
2. Nutritional status	6. Dietary principles
3. Abnormal biochemistry	7. Dietary requirements
4. PES nutrition diagnosis	8. Dietary guidelines

Table 1: The eight standardised questions, applied uniformly to every source report; each maps onto one NCP / PES section, which bounds the acceptable answer space. Verbatim Indonesian wording is in Appendix I .

Model	Family	Shots	EM	F1	BLEU	R-1	R-2	R-L
Fine-tuned baselines (this work)
Rifky_IndoBERT_QA	IndoBERT (extr.)	–	12.41	36.34	12.93	37.28	34.31	36.72
IndoLEM_IndoBERT	IndoBERT (extr.)	–	5.84	23.78	10.60	24.44	22.87	24.17
Cahya_IndoBERT	IndoBERT (extr.)	–	4.38	16.56	6.14	17.02	16.01	16.92
XLM-R-large	XLM-R (extr.)	–	1.46	12.60	8.62	12.62	11.45	12.45
mDeBERTa-v3-base	mDeBERTa (extr.)	–	5.84	19.35	14.03	19.51	18.28	19.19

Table 2: All evaluated systems on the IndoNutriQA test split (

n=137

). The top block reports fine-tuned baselines (extractive, generative, retrieval-augmented); the rest report LLMs in zero- and five-shot settings under an identical Indonesian system prompt, temperature

0

, fixed few-shot sampling seed, and no parameter updates. Bold marks the best score per column.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Value
Total QA Pairs	911
Total Source Reports	120
Average Question Length	8.32 tokens
Average Answer Length	70.66 tokens
Average Context Length	131.28 tokens
Unique Medical Diagnoses	128

Appendix

Table 3: Dataset Statistics Overview

Model	F1 % [95% CI]
Fine-tuned baselines
Rifky_IndoBERT_QA	36.34 [29.03, 43.59]
IndoLEM_IndoBERT	23.78 [17.48, 29.75]
Cahya_IndoBERT	16.56 [11.20, 21.96]
XLM-R-large	12.60 [ 8.05, 17.26]
mDeBERTa-v3-base	19.35 [13.34, 25.42]

Appendix

Table 4: F1 with bootstrap

95\%

confidence intervals over

1{,}000

resamples of the test set (

n=137

). LLM rows distinguish 0-shot and 5-shot. Bold marks the best point estimate.

Model	EM (%)	F1 (%)	ROUGE-L (%)
Rifky_IndoBERT_QA	10.20 $\pm$ 1.37	32.62 $\pm$ 1.29	33.23 $\pm$ 1.62
IndoLEM_IndoBERT	5.59 $\pm$ 1.62	21.69 $\pm$ 2.37	22.25 $\pm$ 2.53
Cahya_IndoBERT	5.70 $\pm$ 0.94	23.75 $\pm$ 2.65	24.40 $\pm$ 3.04
mT5 (Generative)	0.00 $\pm$ 0.00	6.60 $\pm$ 2.66	5.66 $\pm$ 2.30

Appendix

Table 5:

5

-fold cross-validation, folds stratified by source report. Mean

\pm

standard deviation across folds.

Figure 1: Learning curve: F1 on the held-out test set as a function of training-set fraction.

Benchmark	Indo.	Nutr.	Clin. docs
SQuAD	–	–	–
TyDi QA	$\surd$	–	–
PubMedQA	–	–	–
MedQA	–	–	–
MedMCQA	–	–	–
MedQuAD	–	–	–

Appendix

Table 6: Positioning of IndoNutriQA against representative QA benchmarks. “Clinical docs” marks resources grounded in authentic clinical documentation rather than exam items or literature abstracts. The full nine-column comparison, including sample counts and question/answer lengths, is in Appendix C .

Dataset	Language	Domain	# Samples	Q Type	A Type	Source	Avg Q Length	Avg A Length
MedQA (USMLE) Jin et al. (2021)	EN	General Med	12,723	MCQ	Single	Exam	116.6 tokens	3.5 tokens
MedQA (MCMLE) Jin et al. (2021)	ZH	General Med	34,251	MCQ	Single	Exam	45.7 chars	7.3 chars
MedQA (TWMLE) Jin et al. (2021)	ZH	General Med	14,123	MCQ	Single	Exam	61.0 chars	20.6 chars
MedMCQA Pal et al. (2022)	EN	General Med	194,000+	MCQ	Single	Exam	12.71 tokens	2.70 tokens
PubMedQA (PQA-L) Jin et al. (2019)	EN	Biomedical	1,000+	Y/N/Maybe	Short	Literature	14.4 tokens	43.2 tokens
PubMedQA (PQA-U) Jin et al. (2019)	EN	Biomedical	61,200+	Y/N/Maybe	Short	Literature	15.0 tokens	45.9 tokens

Appendix

Table 7: Combined overview and statistical comparison of medical QA datasets.

Metric	Score % [95% CI]
Generator-side
Faithfulness ( $n=105$ )	12.86 [–, –]
Answer relevancy	0.24 [ 0.00, 0.60]
Retriever-side
Context precision	73.43 [66.85, 80.23]
Context recall	35.98 [29.61, 42.86]

Appendix

Table 8: RAGAS metrics for the RAG baseline on the test split (

n=137

). Scores in percent (

\times 100

) with bootstrap

95\%

CIs. Faithfulness reported on the

105

examples for which RAGAS could parse atomic claims;

32

examples produced near-empty outputs on which faithfulness is undefined.

Model	Lexical F1	BERTScore F1	Embed. cos	Judge fact.	Judge compl.	Judge safety	Judge overall
Fine-tuned baselines
Rifky_IndoBERT_QA	36.34	45.94 [38.41, 54.07]	45.27 [39.31, 51.74]	3.14 [2.84, 3.45]	2.84 [2.53, 3.13]	4.89 [4.80, 4.96]	3.62 [3.43, 3.83]
IndoLEM_IndoBERT	23.78	35.42 [27.87, 42.57]	36.23 [30.51, 41.84]	2.61 [2.34, 2.91]	2.33 [2.07, 2.60]	4.85 [4.77, 4.92]	3.26 [3.09, 3.45]
Cahya_IndoBERT	16.56	25.78 [19.24, 32.26]	28.61 [23.64, 33.79]	2.46 [2.18, 2.75]	2.19 [1.93, 2.45]	4.85 [4.77, 4.92]	3.17 [2.99, 3.36]
XLM-R-large	12.60	23.16 [16.74, 29.64]	27.61 [23.02, 32.62]	2.34 [2.05, 2.64]	2.12 [1.88, 2.38]	4.84 [4.75, 4.92]	3.10 [2.92, 3.29]
mDeBERTa-v3-base	19.35	28.47 [21.53, 35.90]	32.18 [26.71, 38.05]	2.54 [2.28, 2.83]	2.32 [2.07, 2.59]	4.89 [4.82, 4.94]	3.25 [3.08, 3.44]

Appendix

Table 9: Semantic and LLM-judge metrics on the IndoNutriQA test split (

n=137

), with bootstrap

95\%

CIs.

Dimension	Quadratic $\kappa$	Within-1 pt.	Mean abs. diff.
Factuality	0.584	83.3%	0.77
Completeness	0.711	84.6%	0.71
Safety	0.251	80.7%	0.66
Pooled (all dims)	0.652	82.8%	0.71

Appendix

Table 10: Inter-judge agreement between GPT-4o-mini and Claude Haiku 4.5 across the three rubric dimensions. Quadratic-weighted Cohen’s

\kappa

over the

1

–

5

Likert ordinal scale; “within-1” = fraction of paired judgements that differ by

\leq 1

Likert point.

Model	Judge fact.	Judge compl.	Judge safety	Judge overall
Fine-tuned baselines
Rifky_IndoBERT_QA	3.98 [3.68, 4.29]	2.93 [2.60, 3.26]	4.28 [4.02, 4.53]	3.73 [3.46, 3.98]
IndoLEM_IndoBERT	3.80 [3.49, 4.10]	2.35 [2.05, 2.66]	4.17 [3.93, 4.41]	3.44 [3.20, 3.68]
Cahya_IndoBERT	3.85 [3.52, 4.18]	2.23 [1.90, 2.57]	4.21 [3.93, 4.45]	3.43 [3.14, 3.69]
XLM-R-large	3.81 [3.51, 4.12]	1.96 [1.69, 2.29]	4.12 [3.86, 4.38]	3.30 [3.07, 3.55]
mDeBERTa-v3-base	3.87 [3.58, 4.17]	2.20 [1.88, 2.54]	4.18 [3.93, 4.42]	3.42 [3.18, 3.65]

Appendix

Table 11: Claude Haiku 4.5 LLM-as-judge scores (the second-judge cross-validation of Table 9 ). Scale:

1

–

5

Likert with bootstrap

95\%

CIs over

1{,}000

resamples. Bold marks the strongest model per dimension.

	Fact.	Compl.	Safety
Inter-rater ( $n=50$ )
Quadratic $\kappa$	$0.780$	$0.752$	$0.160$
Krippendorff $\alpha$	$0.592$	$0.712$	$0.250$
Judge vs rater consensus (quadratic $\kappa$ )
GPT-4o-mini	$0.705$	$0.623$	$0.133$
Claude Haiku 4.5	$0.674$	$0.585$	$0.480$

Appendix

Table 12: Expert-rater calibration of the LLM-as-judge rubric on

50

stratified units. Inter-rater

\kappa

is the reference pair; Krippendorff’s ordinal

\alpha

pools all three raters. Judge agreement on factuality and completeness approaches the inter-rater ceiling, whereas safety agreement is low and both judges score safety substantially above the rater consensus.

	Mean judge gap
Dimension	Same prov.	Other	DiD	$p$
	( $n{=}4$ )	( $n{=}18$ )
Factuality	$+0.018$	$-0.438$	$+0.456$	$0.103$
Completeness	$+0.491$	$+0.313$	$+0.178$	$0.225$
Safety	$+0.425$	$+0.684$	$-0.259$	$0.092$
Overall	$+0.311$	$+0.186$	$+0.125$	$0.340$

Appendix

Table 13: Judge self-preference test. “Judge gap” is the per-system mean score under GPT-4o-mini minus the mean under Claude Haiku

4.5

on commonly scored examples; positive values mean GPT-4o-mini is more lenient. “Same prov.” groups the four systems sharing the judge’s provider (GPT-4o-mini and GPT-4o, zero- and five-shot). DiD is the difference-in-differences estimate of self-preference;

p

is a two-sided permutation test over

20{,}000

relabellings. No dimension shows a significant effect, and the factuality and safety estimates carry opposite signs.

	Units		Mean rater score
Human error category	$n$	%	Fact.	Compl.	Safe.
Omission	$18$	$36$	$3.32$	$2.58$	$2.53$
Non-answer / truncated	$8$	$16$	$1.75$	$1.50$	$1.50$
NCP/PES violation	$6$	$12$	$3.20$	$2.60$	$2.40$
Reference itself wrong	$5$	$10$	$2.83$	$3.67$	$2.50$
Unsupported addition	$5$	$10$	$4.17$	$3.83$	$4.00$

Appendix

Table 14: Human error analysis:

53

free-text notes from three expert raters over

37

50

audited units, coded into nine categories. % is of the

50

units; counts are lower bounds. Details are in Appendix F .

Figure 2: Headline findings on the IndoNutriQA test split (

n=137

). (a) Lexical vs. BERTScore F1, markers shaped by model family: LLMs cluster above the “metrics agree” diagonal and mT5 falls far off it (high BERTScore from generic-token alignment, near-zero lexical), so no single metric suffices. (b) LLM-judge error taxonomy over

50

sampled outputs per model (three representative rows): the extractive baseline is dominated by noise ([CLS]-token collapse), the LLMs by partial answer, and the judge’s hallucination label is unreliable, as the human error analysis of Section 6.8 shows. (c) Five-shot prompting lifts Llama-3.1-8B by

+17.7

F1 versus

+7.0

for GPT-4o-mini, consistent with demonstrations mainly calibrating answer style.

Model	Factual ( $n{=}38$ )	Inferential ( $n{=}39$ )	Recommendation ( $n{=}60$ )
Fine-tuned baselines
Rifky_IndoBERT_QA	12.78	29.80	55.51
IndoLEM_IndoBERT	5.07	17.18	39.92
Cahya_IndoBERT	2.55	3.55	33.89
XLM-R-large	2.09	4.43	24.56
mDeBERTa-v3-base	3.07	6.05	38.30

Appendix

Table 15: F1 by question category on the IndoNutriQA test split. Categories from Appendix I : factual (

n=38

), inferential (

n=39

), recommendation (

n=60

). The fourth category proposed in Appendix A.1 (comparative) is not represented because the standardized questions all target a single time point per patient. Bold marks the strongest model per category.

Figure 3: F1 by disease category for each evaluated model. Rows banded by family (fine-tuned baselines above the solid line, LLMs below); cells colored on a common scale. The respiratory column (the largest category, $28.33\%$ of source reports) is not systematically darker than the columns for categories with far fewer test-split reports (ENT, infectious, nutrition/growth), supporting the Section 5.1 claim that the dataset imbalance does not translate to biased generalization.

Model	Resp. ( $n{=}42$ )	Gastro. ( $n{=}27$ )	Neuro. ( $n{=}25$ )	Hem./Onc. ( $n{=}22$ )	Nutr./Gr. ( $n{=}7$ )	Infect. ( $n{=}7$ )	ENT ( $n{=}7$ )
Fine-tuned baselines
Rifky_IndoBERT_QA	35.71	41.96	31.83	34.23	23.77	41.15	48.91
IndoLEM_IndoBERT	24.51	26.76	27.48	18.78	5.71	13.65	38.57
Cahya_IndoBERT	18.37	18.72	9.35	13.89	8.37	27.27	29.00
XLM-R-large	13.77	19.83	4.87	10.67	2.09	16.54	17.85
mDeBERTa-v3-base	20.22	19.40	23.29	13.32	0.68	16.53	40.24

Appendix

Table 16: F1 by disease category on the IndoNutriQA test split. Categories follow the Section 5.1 treemap; the test split contains the seven categories shown. Cardiac and renal/urologic are not represented in this test split. Bold marks the strongest model per category.

Model	Correct	Partial	Wrong span	Term. mismatch	Hallucination	Noise
Fine-tuned baselines
Rifky_IndoBERT_QA	32.0	14.0	0.0	2.0	0.0	52.0
XLM-R-large	10.0	14.0	0.0	0.0	0.0	76.0
mDeBERTa-v3-base	12.0	16.0	0.0	0.0	0.0	72.0
Closed-source LLM (5-shot)
GPT-4o-mini	30.0	70.0	0.0	0.0	0.0	0.0

Appendix

Table 17: Error taxonomy over a sampled subset of

n=50

test examples (seed

42

), classified by GPT-4o-mini. Cells show % of the

50

sampled outputs per row; each row sums to

100\%

. Bold marks the strongest model on correct. The Hallucination column is a judge label that human audit shows to under-detect unsupported content (Section 6.8 ); it is not evidence of absence.

Disease category	Source reports		QA pairs
Disease category	$n$	%	$n$	%
Respiratory	$34$	$28.33$	$273$	$29.97$
Infectious	$21$	$17.50$	$157$	$17.23$
Gastrointestinal	$15$	$12.50$	$104$	$11.42$
Nutrition/growth	$13$	$10.83$	$92$	$10.10$
Neurologic	$12$	$10.00$	$86$	$9.44$

Appendix

Table 18: Disease-category distribution of IndoNutriQA. Each source report is assigned a single dominant category, so the columns partition the corpus; percentages are rounded and may not sum exactly to

100

. The same categorisation backs the stratified results in Section 6.7 .

Figure 4: Distribution of disease categories in IndoNutriQA.

Panel of Figure 5 — Figure 5: Demographic and anthropometric characteristics of IndoNutriQA cases, and distribution of token-level syntactic divergence between model predictions and reference answers.

Clinical QA

Clinical question answering uses AI to answer medical and patient-specific questions from clinical knowledge, records, guidelines, and evidence.

See all 63 papers

L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages
Aug 16, 2026Rinit Jain, Tirthraj Mahajan, Advait Joshi +1Multilingual QA Factual Knowledge in Language Models
Pune Institute of Computer Technology, Pune · L3Cube Labs, Pune · Indian Institute of Technology Madras
Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines
May 1, 2026Hugo Abonizio, Filipe Rocha Lopes, Roberto Lotufo +1Knowledge Augmentation for Language Models Healthcare
FEEC, UNICAMP, Campinas, Brazil · NoHarm.ai, Brazil · Maritaca AI, Brazil
FAM-Bench: Benchmarking Condition-Aware Food-as-Medicine Reasoning in Vision-Language Models
May 29, 2026Mingyang Mao, Bhargav Rishi Medisetti, Utkarsh Grover +4Healthcare Clinical Decision Support
Department of Electrical Engineering, University of South Florida · Muma College of Business, University of South Florida · Computer Science, University of Copenhagen

Aug 16, 2026J/K move · Enter open · S save

L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages

Rinit Jain, Tirthraj Mahajan, Advait Joshi, Raviraj Joshi

Pune Institute of Computer Technology, Pune · L3Cube Labs, Pune · Indian Institute of Technology Madras

Multilingual QA Factual Knowledge in Language Models Multilingual Language Model Evaluation Cultural Knowledge in Language Models

Open paper AskPDF arXiv

We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs). The benchmark comprises 3,471 curriculum-grounded English question--answer pairs spanning nine domains, curated from educational curricula, competitive examination materials, and domain-specific reference books. We introduce a practical hybrid construction strategy that combines context-grounded LLM-based question generation and validation with semantic deduplication and human verification, enabling scalable creation of benchmark data while preserving annotation quality. The benchmark is translated into 19 Indic languages, yielding a publicly released multilingual dataset of 69,420 question--answer pairs across 20 languages. We evaluate six LLMs under three protocols: LLM-as-a-judge and two deterministic lexical criteria, exact-substring and word-overlap matching. All three produce almost the same model ranking, showing that the results do not depend on the choice of judge. The frontier commercial model leads by a wide margin, and among open-weight models Gemma4 31B outperforms the Indic-specialised Sarvam 30B in every evaluated Indic language.

Abstract

Clinical QA

Explore similar work

L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages

Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines

FAM-Bench: Benchmarking Condition-Aware Food-as-Medicine Reasoning in Vision-Language Models