PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoning

Taegyun Kim; Youngwook Ham; Jungwook Rhim; Ju-Hyun An; Sungkyu Park; Kunwoo Park

cs.CLSep 4, 2026

PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoning

Authors: Taegyun Kim, Youngwook Ham, Jungwook Rhim, Ju-Hyun An, Sungkyu Park, Kunwoo Park

Organizations: Department of Intelligent Semiconductors, Soongsil University · Kangwon National University · KDI School of Public Policy and Management · School of AI Convergence, Soongsil University

Abstract

We introduce PetQA, a Korean long-form question-answering (QA) benchmark for evaluating veterinary knowledge and clinical reasoning in large language models (LLMs) and large vision-language models (LVLMs). PetQA contains 10,076 text-only and 8,751 multimodal QA pairs derived from real-world questions about dogs and cats, paired with answers from expert veterinarians. Its test split, PetQA-Bench, further includes annotations for question types and clinical conditions. We evaluate eighteen models using ROUGE, BERTScore, and LLM-as-a-judge metrics for factuality and helpfulness under three settings: zero-shot inference, retrieval-augmented generation (RAG), and supervised fine-tuning (SFT). The benchmarking results provide an overview of the strengths and limitations of current models in addressing veterinary clinical queries and highlight the need for more effective adaptation methods to develop clinically reliable AI systems for veterinary care. To facilitate broader use, we additionally provide translated versions of PetQA-Bench in five languages.

Figures & tables

Dataset	Source	Answer Format	Language	Modality	Size	Target Subject
BioASQ Tsatsaronis et al. (2015)	Medical Article	Span-based, Binary	En	Text	4,721	Human
PubMedQA Jin et al. (2019)	Medical Article	Span-based, Binary	En	Text	1,000
MMLU (Med.) Hendrycks et al. (2021)	Medical Licensing Examination	Multiple-choice	En	Text	1,089
MedQA Jin et al. (2021)			En, Zh	Text	61,097
MedMCQA Pal et al. (2022)			En	Text	193,155
CMExam Liu et al. (2023)			Zh	Text	68,119

Table 1: Comparison of PetQA with existing medical benchmarks.

Figure 1: A raw data sample (left) and its corresponding example from PetQA (right).

Metric	Text	Multimodal
Target species
Dog	1,672	1,650
Cat	328	350
Question type
Diagnosis	1,231	1,699
Treatment	377	129

Table 2: Descriptive statistics of PetQA-Bench .

Figure 2: Distribution of the top-five clinical conditions in PetQA-Bench .

Model	Text				Multimodal
Model	ROUGE	BERTScore	Factuality	Helpfulness	ROUGE	BERTScore	Factuality	Helpfulness
Closed large vision-language models
GPT-4.1 mini	0.303 ( $\pm$ 0.001)	0.746 ( $\pm$ 0.000)	0.551 ( $\pm$ 0.002)	4.728 ( $\pm$ 0.002)	0.237 ( $\pm$ 0.001)	0.726 ( $\pm$ 0.000)	0.531 ( $\pm$ 0.002)	4.597 ( $\pm$ 0.005)
GPT-4.1	0.290 ( $\pm$ 0.000)	0.743 ( $\pm$ 0.000)	0.569 ( $\pm$ 0.001)	4.880 ( $\pm$ 0.003)	0.234 ( $\pm$ 0.001)	0.726 ( $\pm$ 0.001)	0.531 ( $\pm$ 0.002)	4.715 ( $\pm$ 0.042)
Gemini 2.5 Flash	0.279 ( $\pm$ 0.000)	0.731 ( $\pm$ 0.000)	0.541 ( $\pm$ 0.001)	4.914 ( $\pm$ 0.007)	0.224 ( $\pm$ 0.001)	0.709 ( $\pm$ 0.000)	0.495 ( $\pm$ 0.004)	4.679 ( $\pm$ 0.011)
Gemini 2.5 Pro	0.280 ( $\pm$ 0.001)	0.738 ( $\pm$ 0.000)	0.550 ( $\pm$ 0.002)	4.923 ( $\pm$ 0.000)	0.246 ( $\pm$ 0.000)	0.726 ( $\pm$ 0.001)	0.524 ( $\pm$ 0.008)	4.814 ( $\pm$ 0.012)

Table 3: Zero-shot performance measured on PetQA-Bench . Bold and underlined values indicate the best and second-best results, respectively. Results for closed LVLMs are averaged over three runs.

Figure 3: Performance by question type on PetQA-Bench . For each metric, the x-axis denotes question type: D (Diagnosis), T (Treatment), B (Basic veterinary knowledge), and M (Miscellaneous).

Figure 4: Clinical condition inclusion performance on the Text set.

Figure 5:

\Delta

factuality between RAG and zero-shot inference, where positive values indicate the superiority of RAG.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Train		Validation
	Text	Multimodal	Text	Multimodal
Target species
Dog	5,098	3,871	1,675	1,595
Cat	978	880	325	405
Text tokens
Questions

Appendix

Table A1: Descriptive statistics of PetQA for the train and validation sets.

Judge Model	Pearson	Spearman	Kendall
GPT-4o	0.2255	0.2239	0.2144
GPT-5	0.1544	0.2218	0.2097
Gemini 2.5 Pro	0.0967	0.1231	0.1149
EXAONE-3.5-32B	-0.1097	-0.0781	-0.0771

Appendix

Table A2: Correlation between human evaluation scores and LLM-as-a-judge scores for helpfulness.

Judge Model	Pearson	Spearman
Gemini 2.5 Pro	0.4087	0.3922
Gemini 2.5 Flash	0.4065	0.3784
GPT-4o-mini	0.1514	0.0995

Appendix

Table A3: Correlations between human evaluation scores and LLM-as-a-judge factuality scores for the factuality.

Model	Model ID	Params
Closed large vision-language models
GPT-4.1 mini	gpt-4.1-mini-2025-04-14	–
GPT-4.1	gpt-4.1-2025-04-14	–
Gemini 2.5 Flash	gemini-2.5-flash	–
Gemini 2.5 Pro	gemini-2.5-pro	–
Open-weight large vision-language models

Appendix

Table A4: Models used in experiments, along with their corresponding model identifiers and parameter sizes.

Figure A1: Clinical condition inclusion performance on the Multimodal set.

Model	Text			Multimodal
	P	R	F1	P	R	F1
Closed large vision-language models
GPT-4.1 mini	0.528	0.647	0.551	0.562	0.572	0.531
GPT-4.1	0.535	0.678	0.569	0.538	0.590	0.531
Gemini 2.5 Flash	0.493	0.682	0.541	0.479	0.586	0.495
Gemini 2.5 Pro	0.502	0.679	0.550	0.512	0.625	0.524

Appendix

Table A5: Breakdown of factuality into precision and recall.

Metric	Text	Multimodal
Diagnosis questions only
ROUGE	0.2807	0.2298
BERTScore	0.7360	0.7196
Factuality	0.5560	0.5005
Helpfulness	4.5557	4.3285
Shared question type distribution

Appendix

Table A6: Comparison of mean performance on PetQA-Bench after controlling for question type. The first analysis includes only diagnosis questions. The second applies the Multimodal set’s question type distribution to both sets.

Model	ROUGE	BERTScore	Factuality	Helpfulness
Text
Qwen3-8B	0.294	0.746	0.514	3.754
w/ Reasoning	0.256	0.727	0.502	4.527
Qwen3-32B	0.282	0.742	0.530	4.433
w/ Reasoning	0.250	0.727	0.504	4.779
Multimodal

Appendix

Table A7: Effects of enhanced reasoning on veterinary QA performance.

Model	Text				Multimodal
Model	ROUGE	BERTScore	Factuality	Helpfulness	ROUGE	BERTScore	Factuality	Helpfulness
Closed large vision-language models
GPT-4.1 mini	0.226	0.725	0.509	4.886	0.211	0.708	0.473	4.766
GPT-4.1	0.223	0.722	0.526	4.928	0.206	0.706	0.482	4.795
Gemini 2.5 Flash	0.211	0.722	0.507	4.636	0.191	0.700	0.448	4.226
Gemini 2.5 Pro	0.213	0.718	0.518	4.912	0.207	0.708	0.496	4.678

Appendix

Table A8: Zero-shot performance on the English version of PetQA-Bench . Bold and underlined indicate the best and second-best results, respectively.

Table A9: Zero-shot performance on the Chinese, German, Indonesian, and Arabic versions of PetQA-Bench . Bold and underlined indicate the best and second-best results, respectively.

Model	Text				Multimodal
Model	ROUGE	BERTScore	Factuality	Helpfulness	ROUGE	BERTScore	Factuality	Helpfulness
Closed large vision-language models
GPT-4.1 mini	0.299 (-0.004)	0.745 (-0.001)	0.569 (+0.018)	4.827 (+0.099)	0.235 (-0.002)	0.724 (-0.002)	0.534 (+0.003)	4.780 (+0.183)
GPT-4.1	0.287 (-0.003)	0.741 (-0.002)	0.571 (+0.002)	4.931 (+0.051)	0.228 (-0.006)	0.722 (-0.004)	0.531 (0.000)	4.809 (+0.094)
Gemini 2.5 Flash	0.276 (-0.003)	0.729 (-0.002)	0.536 (-0.005)	4.777 (-0.137)	0.221 (-0.003)	0.702 (-0.007)	0.497 (+0.002)	4.727 (+0.048)
Gemini 2.5 Pro	0.292 (+0.012)	0.741 (+0.003)	0.556 (+0.006)	4.921 (-0.002)	0.248 (+0.002)	0.727 (+0.001)	0.539 (+0.015)	4.866 (+0.052)

Appendix

Table A10: Performance of RAG on PetQA-Bench . Parenthetical values report

\Delta=\text{RAG}-\text{zero-shot}

, with blue text indicating positive values and red text indicating a negative value. Bold and underlined indicate the best and second-best results, respectively.

Model	Text				Multimodal
Model	ROUGE	BERTScore	Factuality	Helpfulness	ROUGE	BERTScore	Factuality	Helpfulness
Open-weight large vision-language models
Qwen3-VL-8B	0.341 (+0.070)	0.771 (+0.034)	0.477 (-0.022)	2.649 (-1.312)	0.239 (+0.035)	0.698 (-0.023)	0.317 (-0.144)	1.978 (-1.098)
Qwen3-VL-32B	0.354 (+0.096)	0.776 (+0.046)	0.513 (0.000)	2.880 (-1.987)	0.266 (+0.051)	0.724 (+0.012)	0.379 (-0.110)	2.343 (-2.451)
Gemma-3-12B	0.360 (+0.073)	0.775 (+0.035)	0.498 (-0.012)	2.788 (-1.773)	0.299 (+0.056)	0.745 (+0.022)	0.409 (-0.081)	2.463 (-1.792)
Gemma-3-27B	0.366 (+0.086)	0.777 (+0.040)	0.507 (-0.012)	2.897 (-1.973)	0.321 (+0.077)	0.757 (+0.033)	0.433 (-0.063)	2.499 (-2.127)

Appendix

Table A11: SFT performance measured on PetQA-Bench . Parenthetical values report

\Delta=\text{SFT}-\text{zero-shot}

, with blue text indicating positive values and red text indicating negative values. Bold and underlined indicate the best and second-best results, respectively.

Panel of Figure 21 — Figure A2: Qualitative example in which retrieval-augmented generation (RAG) produces an accurate response by jointly reasoning over the clinical image and retrieved documents.

Panel of Figure 22 — Figure A3: Qualitative example showing that retrieval-augmented generation (RAG) prioritizes retrieved text over visual context, leading to an incorrect diagnosis.

Panel of Figure 23 — Figure A4: Qualitative example showing that supervised fine-tuning (SFT) produces a more generic and conservative response with reduced image-grounded reasoning.

Panel of Figure 24 — Figure A5: LLM-based preprocessing prompt.

Panel of Figure 25 — Figure A6: LLM-as-a-judge prompt used for factuality evaluation.

Panel of Figure 26 — Figure A7: LLM-as-a-judge prompt used for helpfulness evaluation.

Panel of Figure 27 — Figure A8: Zero-shot prompt for question answering. The same prompt is used under supervised fine-tuning.

Panel of Figure 28 — Figure A9: Prompt used for retrieval-augmented generation (RAG).

Panel of Figure 29 — Figure A10: Interfaces used for dataset annotation and expert validation.

Explore similar work

VetScore: Risk-Weighted Fact Verification for Veterinary Long-Form QA with Citations
Aug 4, 2026Ivan Kartáč, Jan Tovarys, Mateusz Lango +1LLM Answer Verification Factual Consistency Evaluation
PrimVeterinary · Institute of Formal and Applied Linguistics, Charles University
BAGEL: Benchmarking Animal Knowledge Expertise in Language Models
Apr 17, 2026Jiacheng Shen, Masato Hagiwara, Milad Alizadeh +9LLM Evaluation Question Answering
2NYU Center for Data Science; NYU Shanghai · 1Earth Species Project · 3NYU Shanghai Center for Data Science; NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai
K9-Bench: Evaluating Multimodal LLMs on Canine-Centric Videos
Jul 2, 2026Khush Attarde, Yusuf Ali, Megha Thukral +3Video Understanding Multimodal Large Language Models
Ogmen Robotics Inc., Massachusetts, USA · Georgia Institute of Technology, Atlanta, USA

Aug 4, 2026, cs.CLJ/K move · Enter open · S save

VetScore: Risk-Weighted Fact Verification for Veterinary Long-Form QA with Citations

Ivan Kartáč, Jan Tovarys, Mateusz Lango, Ondřej Dušek

PrimVeterinary · Institute of Formal and Applied Linguistics, Charles University

LLM Answer Verification Factual Consistency Evaluation Question Answering

Open paper AskPDF arXiv

Citation excerpts can be used to increase the reliability of generated outputs and their faithfulness to cited sources, which is especially important in high-stakes domains such as human and veterinary medicine. However, this does not guarantee that generated claims are faithful to the provided excerpts. We present VetScore, a multi-step evaluation method for veterinary long-form question answering, designed to assess how well are generated claims supported by the provided excerpts, weighing this information by each claim's harm potential. VetScore first segments the output and decomposes it into individual claims, then scores each claim with respect to its harm potential and evaluates its faithfulness to source excerpts, and finally calculates the overall risk-adjusted score. We collect an expert-annotated meta-evaluation dataset, evaluate our approach with a range of judge models, and show that it achieves high correlations with veterinary experts even with small judge models, while offering explainability across multiple dimensions.

Abstract

Explore similar work

VetScore: Risk-Weighted Fact Verification for Veterinary Long-Form QA with Citations

BAGEL: Benchmarking Animal Knowledge Expertise in Language Models

K9-Bench: Evaluating Multimodal LLMs on Canine-Centric Videos