IndoNutriQA: An Indonesian Clinical Nutrition Question Answering Benchmark

Tirana Noor Fatyanosa; Ummu Ditya Erliana; Mahendra Data; Muhammad Nurwegiono; Eleazar Tadeo Eman; Alham Fikri Aji

cs.CLOct 22, 2026

IndoNutriQA: An Indonesian Clinical Nutrition Question Answering Benchmark

Authors: Tirana Noor Fatyanosa, Ummu Ditya Erliana, Mahendra Data, Muhammad Nurwegiono, Eleazar Tadeo Eman, Alham Fikri Aji

Organizations: Universitas Brawijaya · Universitas Ma Chung · MBZUAI

Abstract

Clinical Question Answering (QA) underpins Clinical Intelligent Decision Support Systems, but progress is severely limited for low-resource languages such as Indonesian by the absence of public domain-specific data. We introduce IndoNutriQA, an expert-curated Indonesian clinical QA benchmark of 911 question-answer pairs built from real clinical-nutrition reports. Rather than a large-scale training corpus, it is positioned as a high-quality evaluation resource: every pair is grounded in a clinically relevant context, enriched with structured metadata (anthropometric assessment, medical diagnosis, document provenance, chunk segmentation). Eight standardised question types following the Nutrition Care Process (NCP), including the Problem, Etiology, Sign and Symptom (PES) diagnosis statement, are answered for every report, which was written by a graduate-level dietetic intern, by one of three nutritionists. Answers are free-form clinical prose rather than extractive spans, and an expert validation study finds 6-10% of reference answers contestable. The dataset supports generative, extractive and retrieval-augmented QA as well as zero- and few-shot LLM evaluation. We report fine-tuned baselines (IndoBERT variants, mT5, RAG) with 5-fold cross-validation, learning curves and bootstrap 95% confidence intervals, and benchmark eight contemporary LLMs zero- and five-shot. We further calibrate our LLM-as-judge rubric against three expert raters (one RD, two nutritionists), finding it human-comparable on factuality and completeness but unreliable on clinical safety, and a human error analysis shows it misses unsupported content that the raters identify. IndoNutriQA is released to foster research in Indonesian clinical NLP and nutritional decision support.

Figures & tables

1. Screening method	5. Dietary goals
2. Nutritional status	6. Dietary principles
3. Abnormal biochemistry	7. Dietary requirements
4. PES nutrition diagnosis	8. Dietary guidelines

Table 1: The eight standardised questions, applied uniformly to every source report; each maps onto one NCP / PES section, which bounds the acceptable answer space. Verbatim Indonesian wording is in Appendix I .

Model	Family	Shots	EM	F1	BLEU	R-1	R-2	R-L
Fine-tuned baselines (this work)
Rifky_IndoBERT_QA	IndoBERT (extr.)	–	12.41	36.34	12.93	37.28	34.31	36.72
IndoLEM_IndoBERT	IndoBERT (extr.)	–	5.84	23.78	10.60	24.44	22.87	24.17
Cahya_IndoBERT	IndoBERT (extr.)	–	4.38	16.56	6.14	17.02	16.01	16.92
XLM-R-large	XLM-R (extr.)	–	1.46	12.60	8.62	12.62	11.45	12.45
mDeBERTa-v3-base	mDeBERTa (extr.)	–	5.84	19.35	14.03	19.51	18.28	19.19

Table 2: All evaluated systems on the IndoNutriQA test split (

n=137

). The top block reports fine-tuned baselines (extractive, generative, retrieval-augmented); the rest report LLMs in zero- and five-shot settings under an identical Indonesian system prompt, temperature

0

, fixed few-shot sampling seed, and no parameter updates. Bold marks the best score per column.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Value
Total QA Pairs	911
Total Source Reports	120
Average Question Length	8.32 tokens
Average Answer Length	70.66 tokens
Average Context Length	131.28 tokens
Unique Medical Diagnoses	128

Appendix

Table 3: Dataset Statistics Overview

Model	F1 % [95% CI]
Fine-tuned baselines
Rifky_IndoBERT_QA	36.34 [29.03, 43.59]
IndoLEM_IndoBERT	23.78 [17.48, 29.75]
Cahya_IndoBERT	16.56 [11.20, 21.96]
XLM-R-large	12.60 [ 8.05, 17.26]
mDeBERTa-v3-base	19.35 [13.34, 25.42]

Appendix

Table 4: F1 with bootstrap

95\%

confidence intervals over

1{,}000

resamples of the test set (

n=137

). LLM rows distinguish 0-shot and 5-shot. Bold marks the best point estimate.

Model	EM (%)	F1 (%)	ROUGE-L (%)
Rifky_IndoBERT_QA	10.20 $\pm$ 1.37	32.62 $\pm$ 1.29	33.23 $\pm$ 1.62
IndoLEM_IndoBERT	5.59 $\pm$ 1.62	21.69 $\pm$ 2.37	22.25 $\pm$ 2.53
Cahya_IndoBERT	5.70 $\pm$ 0.94	23.75 $\pm$ 2.65	24.40 $\pm$ 3.04
mT5 (Generative)	0.00 $\pm$ 0.00	6.60 $\pm$ 2.66	5.66 $\pm$ 2.30

Appendix

Table 5:

5

-fold cross-validation, folds stratified by source report. Mean

\pm

standard deviation across folds.

Figure 1: Learning curve: F1 on the held-out test set as a function of training-set fraction.

Benchmark	Indo.	Nutr.	Clin. docs
SQuAD	–	–	–
TyDi QA	$\surd$	–	–
PubMedQA	–	–	–
MedQA	–	–	–
MedMCQA	–	–	–
MedQuAD	–	–	–

Appendix

Table 6: Positioning of IndoNutriQA against representative QA benchmarks. “Clinical docs” marks resources grounded in authentic clinical documentation rather than exam items or literature abstracts. The full nine-column comparison, including sample counts and question/answer lengths, is in Appendix C .

Dataset	Language	Domain	# Samples	Q Type	A Type	Source	Avg Q Length	Avg A Length
MedQA (USMLE) Jin et al. (2021)	EN	General Med	12,723	MCQ	Single	Exam	116.6 tokens	3.5 tokens
MedQA (MCMLE) Jin et al. (2021)	ZH	General Med	34,251	MCQ	Single	Exam	45.7 chars	7.3 chars
MedQA (TWMLE) Jin et al. (2021)	ZH	General Med	14,123	MCQ	Single	Exam	61.0 chars	20.6 chars
MedMCQA Pal et al. (2022)	EN	General Med	194,000+	MCQ	Single	Exam	12.71 tokens	2.70 tokens
PubMedQA (PQA-L) Jin et al. (2019)	EN	Biomedical	1,000+	Y/N/Maybe	Short	Literature	14.4 tokens	43.2 tokens
PubMedQA (PQA-U) Jin et al. (2019)	EN	Biomedical	61,200+	Y/N/Maybe	Short	Literature	15.0 tokens	45.9 tokens

Appendix

Table 7: Combined overview and statistical comparison of medical QA datasets.

Metric	Score % [95% CI]
Generator-side
Faithfulness ( $n=105$ )	12.86 [–, –]
Answer relevancy	0.24 [ 0.00, 0.60]
Retriever-side
Context precision	73.43 [66.85, 80.23]
Context recall	35.98 [29.61, 42.86]

Appendix

Table 8: RAGAS metrics for the RAG baseline on the test split (

n=137

). Scores in percent (

\times 100

) with bootstrap

95\%

CIs. Faithfulness reported on the

105

examples for which RAGAS could parse atomic claims;

32

examples produced near-empty outputs on which faithfulness is undefined.

Model	Lexical F1	BERTScore F1	Embed. cos	Judge fact.	Judge compl.	Judge safety	Judge overall
Fine-tuned baselines
Rifky_IndoBERT_QA	36.34	45.94 [38.41, 54.07]	45.27 [39.31, 51.74]	3.14 [2.84, 3.45]	2.84 [2.53, 3.13]	4.89 [4.80, 4.96]	3.62 [3.43, 3.83]
IndoLEM_IndoBERT	23.78	35.42 [27.87, 42.57]	36.23 [30.51, 41.84]	2.61 [2.34, 2.91]	2.33 [2.07, 2.60]	4.85 [4.77, 4.92]	3.26 [3.09, 3.45]
Cahya_IndoBERT	16.56	25.78 [19.24, 32.26]	28.61 [23.64, 33.79]	2.46 [2.18, 2.75]	2.19 [1.93, 2.45]	4.85 [4.77, 4.92]	3.17 [2.99, 3.36]
XLM-R-large	12.60	23.16 [16.74, 29.64]	27.61 [23.02, 32.62]	2.34 [2.05, 2.64]	2.12 [1.88, 2.38]	4.84 [4.75, 4.92]	3.10 [2.92, 3.29]
mDeBERTa-v3-base	19.35	28.47 [21.53, 35.90]	32.18 [26.71, 38.05]	2.54 [2.28, 2.83]	2.32 [2.07, 2.59]	4.89 [4.82, 4.94]	3.25 [3.08, 3.44]

Appendix

Table 9: Semantic and LLM-judge metrics on the IndoNutriQA test split (

n=137

), with bootstrap

95\%

CIs.

Dimension	Quadratic $\kappa$	Within-1 pt.	Mean abs. diff.
Factuality	0.584	83.3%	0.77
Completeness	0.711	84.6%	0.71
Safety	0.251	80.7%	0.66
Pooled (all dims)	0.652	82.8%	0.71

Appendix

Table 10: Inter-judge agreement between GPT-4o-mini and Claude Haiku 4.5 across the three rubric dimensions. Quadratic-weighted Cohen’s

\kappa

over the

1

–

5

Likert ordinal scale; “within-1” = fraction of paired judgements that differ by

\leq 1

Likert point.

Model	Judge fact.	Judge compl.	Judge safety	Judge overall
Fine-tuned baselines
Rifky_IndoBERT_QA	3.98 [3.68, 4.29]	2.93 [2.60, 3.26]	4.28 [4.02, 4.53]	3.73 [3.46, 3.98]
IndoLEM_IndoBERT	3.80 [3.49, 4.10]	2.35 [2.05, 2.66]	4.17 [3.93, 4.41]	3.44 [3.20, 3.68]
Cahya_IndoBERT	3.85 [3.52, 4.18]	2.23 [1.90, 2.57]	4.21 [3.93, 4.45]	3.43 [3.14, 3.69]
XLM-R-large	3.81 [3.51, 4.12]	1.96 [1.69, 2.29]	4.12 [3.86, 4.38]	3.30 [3.07, 3.55]
mDeBERTa-v3-base	3.87 [3.58, 4.17]	2.20 [1.88, 2.54]	4.18 [3.93, 4.42]	3.42 [3.18, 3.65]

Appendix

Table 11: Claude Haiku 4.5 LLM-as-judge scores (the second-judge cross-validation of Table 9 ). Scale:

1

–

5

Likert with bootstrap

95\%

CIs over

1{,}000

resamples. Bold marks the strongest model per dimension.

	Fact.	Compl.	Safety
Inter-rater ( $n=50$ )
Quadratic $\kappa$	$0.780$	$0.752$	$0.160$
Krippendorff $\alpha$	$0.592$	$0.712$	$0.250$
Judge vs rater consensus (quadratic $\kappa$ )
GPT-4o-mini	$0.705$	$0.623$	$0.133$
Claude Haiku 4.5	$0.674$	$0.585$	$0.480$

Appendix

Table 12: Expert-rater calibration of the LLM-as-judge rubric on

50

stratified units. Inter-rater

\kappa

is the reference pair; Krippendorff’s ordinal

\alpha

pools all three raters. Judge agreement on factuality and completeness approaches the inter-rater ceiling, whereas safety agreement is low and both judges score safety substantially above the rater consensus.

	Mean judge gap
Dimension	Same prov.	Other	DiD	$p$
	( $n{=}4$ )	( $n{=}18$ )
Factuality	$+0.018$	$-0.438$	$+0.456$	$0.103$
Completeness	$+0.491$	$+0.313$	$+0.178$	$0.225$
Safety	$+0.425$	$+0.684$	$-0.259$	$0.092$
Overall	$+0.311$	$+0.186$	$+0.125$	$0.340$

Appendix

Table 13: Judge self-preference test. “Judge gap” is the per-system mean score under GPT-4o-mini minus the mean under Claude Haiku

4.5

on commonly scored examples; positive values mean GPT-4o-mini is more lenient. “Same prov.” groups the four systems sharing the judge’s provider (GPT-4o-mini and GPT-4o, zero- and five-shot). DiD is the difference-in-differences estimate of self-preference;

p

is a two-sided permutation test over

20{,}000

relabellings. No dimension shows a significant effect, and the factuality and safety estimates carry opposite signs.

	Units		Mean rater score
Human error category	$n$	%	Fact.	Compl.	Safe.
Omission	$18$	$36$	$3.32$	$2.58$	$2.53$
Non-answer / truncated	$8$	$16$	$1.75$	$1.50$	$1.50$
NCP/PES violation	$6$	$12$	$3.20$	$2.60$	$2.40$
Reference itself wrong	$5$	$10$	$2.83$	$3.67$	$2.50$
Unsupported addition	$5$	$10$	$4.17$	$3.83$	$4.00$

Appendix

Table 14: Human error analysis:

53

free-text notes from three expert raters over

37

50

audited units, coded into nine categories. % is of the

50

units; counts are lower bounds. Details are in Appendix F .

Figure 2: Headline findings on the IndoNutriQA test split (

n=137

). (a) Lexical vs. BERTScore F1, markers shaped by model family: LLMs cluster above the “metrics agree” diagonal and mT5 falls far off it (high BERTScore from generic-token alignment, near-zero lexical), so no single metric suffices. (b) LLM-judge error taxonomy over

50

sampled outputs per model (three representative rows): the extractive baseline is dominated by noise ([CLS]-token collapse), the LLMs by partial answer, and the judge’s hallucination label is unreliable, as the human error analysis of Section 6.8 shows. (c) Five-shot prompting lifts Llama-3.1-8B by

+17.7

F1 versus

+7.0

for GPT-4o-mini, consistent with demonstrations mainly calibrating answer style.

Model	Factual ( $n{=}38$ )	Inferential ( $n{=}39$ )	Recommendation ( $n{=}60$ )
Fine-tuned baselines
Rifky_IndoBERT_QA	12.78	29.80	55.51
IndoLEM_IndoBERT	5.07	17.18	39.92
Cahya_IndoBERT	2.55	3.55	33.89
XLM-R-large	2.09	4.43	24.56
mDeBERTa-v3-base	3.07	6.05	38.30

Appendix

Table 15: F1 by question category on the IndoNutriQA test split. Categories from Appendix I : factual (

n=38

), inferential (

n=39

), recommendation (

n=60

). The fourth category proposed in Appendix A.1 (comparative) is not represented because the standardized questions all target a single time point per patient. Bold marks the strongest model per category.

Figure 3: F1 by disease category for each evaluated model. Rows banded by family (fine-tuned baselines above the solid line, LLMs below); cells colored on a common scale. The respiratory column (the largest category, $28.33\%$ of source reports) is not systematically darker than the columns for categories with far fewer test-split reports (ENT, infectious, nutrition/growth), supporting the Section 5.1 claim that the dataset imbalance does not translate to biased generalization.

Model	Resp. ( $n{=}42$ )	Gastro. ( $n{=}27$ )	Neuro. ( $n{=}25$ )	Hem./Onc. ( $n{=}22$ )	Nutr./Gr. ( $n{=}7$ )	Infect. ( $n{=}7$ )	ENT ( $n{=}7$ )
Fine-tuned baselines
Rifky_IndoBERT_QA	35.71	41.96	31.83	34.23	23.77	41.15	48.91
IndoLEM_IndoBERT	24.51	26.76	27.48	18.78	5.71	13.65	38.57
Cahya_IndoBERT	18.37	18.72	9.35	13.89	8.37	27.27	29.00
XLM-R-large	13.77	19.83	4.87	10.67	2.09	16.54	17.85
mDeBERTa-v3-base	20.22	19.40	23.29	13.32	0.68	16.53	40.24

Appendix

Table 16: F1 by disease category on the IndoNutriQA test split. Categories follow the Section 5.1 treemap; the test split contains the seven categories shown. Cardiac and renal/urologic are not represented in this test split. Bold marks the strongest model per category.

Model	Correct	Partial	Wrong span	Term. mismatch	Hallucination	Noise
Fine-tuned baselines
Rifky_IndoBERT_QA	32.0	14.0	0.0	2.0	0.0	52.0
XLM-R-large	10.0	14.0	0.0	0.0	0.0	76.0
mDeBERTa-v3-base	12.0	16.0	0.0	0.0	0.0	72.0
Closed-source LLM (5-shot)
GPT-4o-mini	30.0	70.0	0.0	0.0	0.0	0.0

Appendix

Table 17: Error taxonomy over a sampled subset of

n=50

test examples (seed

42

), classified by GPT-4o-mini. Cells show % of the

50

sampled outputs per row; each row sums to

100\%

. Bold marks the strongest model on correct. The Hallucination column is a judge label that human audit shows to under-detect unsupported content (Section 6.8 ); it is not evidence of absence.

Disease category	Source reports		QA pairs
Disease category	$n$	%	$n$	%
Respiratory	$34$	$28.33$	$273$	$29.97$
Infectious	$21$	$17.50$	$157$	$17.23$
Gastrointestinal	$15$	$12.50$	$104$	$11.42$
Nutrition/growth	$13$	$10.83$	$92$	$10.10$
Neurologic	$12$	$10.00$	$86$	$9.44$

Appendix

Table 18: Disease-category distribution of IndoNutriQA. Each source report is assigned a single dominant category, so the columns partition the corpus; percentages are rounded and may not sum exactly to

100

. The same categorisation backs the stratified results in Section 6.7 .

Figure 4: Distribution of disease categories in IndoNutriQA.

Panel of Figure 5 — Figure 5: Demographic and anthropometric characteristics of IndoNutriQA cases, and distribution of token-level syntactic divergence between model predictions and reference answers.

Clinical QA

Clinical question answering uses AI to answer medical and patient-specific questions from clinical knowledge, records, guidelines, and evidence.

See all 63 papers

Aug 16, 2026

L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages

We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs). The benchmark comprises 3,471 curriculum-grounded English question--answer pairs spanning nine domains, curated from educational curricula, competitive examination materials, and domain-specific reference books. We introduce a practical hybrid construction strategy that combines context-grounded LLM-based question generation and validation with semantic deduplication and human verification, enabling scalable creation of benchmark data while preserving annotation quality. The benchmark is translated into 19 Indic languages, yielding a publicly released multilingual dataset of 69,420 question--answer pairs across 20 languages. We evaluate six LLMs under three protocols: LLM-as-a-judge and two deterministic lexical criteria, exact-substring and word-overlap matching. All three produce almost the same model ranking, showing that the results do not depend on the choice of judge. The frontier commercial model leads by a wide margin, and among open-weight models Gemma4 31B outperforms the Indic-specialised Sarvam 30B in every evaluated Indic language.

Rinit Jain, Tirthraj Mahajan, Advait Joshi +1

Pune Institute of Computer Technology, Pune · L3Cube Labs, Pune · Indian Institute of Technology Madras

Multilingual QA Factual Knowledge in Language Models Multilingual Language Model Evaluation Cultural Knowledge in Language Models

Ask PDF

May 1, 2026

Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines

Brazil's Unified Health System (SUS) relies on official clinical guidelines that define diagnostic criteria, treatments, dosages, and monitoring procedures for over 200 million citizens. Yet current LLMs perform poorly on this guideline-specific knowledge, and no benchmark evaluates clinical recall grounded in Brazilian Portuguese protocols. We address this gap by adapting Qwen2.5-14B-Instruct to the Brazilian clinical domain. From 178 official guidelines (~5.4M tokens), we generate ~70M tokens of synthetic data in three formats -- rephrases, wiki-style articles, and question-answer pairs -- using four generator LLMs. We then apply continual pre-training followed by Group Relative Policy Optimization (GRPO). We introduce HealthBench-BR, with 1,780 balanced true/false clinical assertions, and PCDT-QA, with 890 open-ended clinical questions scored by an LLM judge. Our best model achieves 83.9% on HealthBench-BR and 85.4% on PCDT-QA, outperforming GPT-5.2, Claude Sonnet 4.6, Gemini 3.1 Pro, and Google AI Overview's web-grounded RAG despite having only 14B parameters. Ablations show that generator diversity and reinforcement learning are critical to these gains. We release all datasets, benchmarks, and model weights to support reproducible clinical NLP research for Brazilian Portuguese. Code, data, and model weights are available at https://github.com/hugoabonizio/clinical-protocols-br

Hugo Abonizio, Filipe Rocha Lopes, Roberto Lotufo +1

FEEC, UNICAMP, Campinas, Brazil · NoHarm.ai, Brazil · Maritaca AI, Brazil

Knowledge Augmentation for Language Models Healthcare Clinical QA LLM Post-Training Clinical Language Model Evaluation Clinical NLP+1

Ask PDF

May 29, 2026

FAM-Bench: Benchmarking Condition-Aware Food-as-Medicine Reasoning in Vision-Language Models

Food AI can recognize dishes, ingredients, and nutrients. However, these abilities do not show whether a dish is suitable for a person with a specific health condition. They also do not test whether such decisions are grounded in the relevant food evidence. To address this gap, we introduce FAM-Bench, a Food-as-Medicine benchmark comprising 2,500 question instances constructed from 1,811 recipes across 13 diet-related health conditions. Gold annotations are derived from rule-based dietary guidance informed by published sources and nutrition professionals. The benchmark contains two complementary tasks: dish-level suitability assessment and comparative dish analysis. We evaluate open-weight and proprietary vision-language models under four prompting modes. Results reveal a substantial gap between suitability decisions and their supporting rationales, as well as limited performance on comparative ranking. These findings highlight persistent challenges in grounded, condition-aware food reasoning.

Mingyang Mao, Bhargav Rishi Medisetti, Utkarsh Grover +4

Department of Electrical Engineering, University of South Florida · Muma College of Business, University of South Florida · Computer Science, University of Copenhagen

Healthcare Clinical Decision Support Multimodal Reasoning Multimodal Model Evaluation

Ask PDF

Abstract

Clinical QA

Explore similar work

L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages

Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines

FAM-Bench: Benchmarking Condition-Aware Food-as-Medicine Reasoning in Vision-Language Models