SommBench: Assessing Sommelier Expertise of Language Models

William Brach; Tomas Bedej; Jacob Nielsen; Jacob Pichna; Juraj Bedej; Eemeli Saarensilta; Julie Dupouy; Gianluca Barmina

cs.CLMar 12, 2026

SommBench: Assessing Sommelier Expertise of Language Models

Authors: William Brach, Tomas Bedej, Jacob Nielsen, Jacob Pichna, Juraj Bedej, Eemeli Saarensilta, Julie Dupouy, Gianluca Barmina,

Organizations: Slovak University of Technology, Bratislava, Slovakia · sommify, Helsinki, Finland · University of Southern Denmark, Odense, Denmark

Abstract

With the rapid advances of large language models, it becomes increasingly important to systematically evaluate their multilingual and multicultural capabilities. Previous cultural evaluation benchmarks focus mainly on basic cultural knowledge that can be encoded in linguistic form. Here, we propose SommBench, a multilingual benchmark to assess sommelier expertise, a domain deeply grounded in the senses of smell and taste. While language models learn about sensory properties exclusively through textual descriptions, SommBench tests whether this textual grounding is sufficient to emulate expert-level sensory judgment. SommBench comprises three main tasks: Wine Theory Question Answering (WTQA), Wine Feature Completion (WFC), and Food-Wine Pairing (FWP). SommBench is available in multiple languages: English, Slovak, Swedish, Finnish, German, Danish, Italian, and Spanish. This helps separate a language model's wine expertise from its language skills. The benchmark datasets were developed in close collaboration with a professional sommelier and native speakers of the respective languages, resulting in 1,024 wine theory question-answering questions, 1,000 wine feature-completion examples, and 1,000 food-wine pairing examples. We provide results for the most popular language models, including closed-weights models such as Gemini 2.5, and open-weights models, such as GPT-OSS and Qwen 3. Our results show that the most capable models perform well on wine theory question answering (up to 97% correct with a closed-weights model), yet feature completion (peaking at 65%) and food-wine pairing show (MCC ranging between 0 and 0.39) turn out to be more challenging. These results position SommBench as an interesting and challenging benchmark for evaluating the sommelier expertise of language models. The benchmark is publicly available at https://github.com/sommify/sommbench.

Figures & tables

Figure 1: Performance of leading open and closed-source language models on the SommBench benchmark. The radar chart shows model accuracy (the higher, the better) in SommBench tasks revealing key differences in competencies between tasks and models.

Figure 2: Taxonomy of Sommelier expertise, categorized into knowledge, profiling, and food wine pairing.

Model	WTQA	WFC	FWP	Score
Closed-Weights Models
gemini-2.5-flash	0.9	0.63	0.39	0.65
gpt-4.1	0.9	0.62	0.25	0.59
gpt-4o	0.9	0.63	0.19	0.57
gemini-2.5-pro	0.96	0.62	0.12	0.57
gpt-5	0.97	0.57	0.17	0.57

Table 1: Performance of language models on SommBench. Models are evaluated using the SommBench score calculated from WFC, WTQA and FWP tasks. The Qwen models were employed without reasoning.

Model	EN	SK	SV	FI	DE	DA	IT	ES
Closed-Weights Models
grok-4	0.98	0.96	0.97	0.95	0.97	0.97	0.98	0.94
gpt-5	0.98	0.98	0.99	0.95	0.97	0.98	0.98	0.93
gemini-2.5-pro	0.98	0.95	0.98	0.96	0.97	0.96	0.96	0.95
grok-4-fast	0.94	0.94	0.93	0.91	0.94	0.92	0.94	0.91
gemini-2.5-flash	0.91	0.92	0.93	0.92	0.93	0.94	0.93	0.89

Table 2: Accuracy scores for language models on the Wine Theory Question-Answering (WTQA) task across eight languages (EN: English, SK: Slovak, SV: Swedish, FI: Finnish, DE: German, DA: Danish, IT: Italian, ES: Spanish). The Qwen models were employed without reasoning.

Model	EN	SK	SV	FI	DE	DA	IT	ES
Closed-Weights Models
gpt-5	0.57	0.56	0.59	0.55	0.56	0.58	0.59	0.57
gemini-2.5-pro	0.62	0.61	0.62	0.62	0.62	0.63	0.62	0.61
grok-4	0.62	0.6	0.59	0.62	0.59	0.61	0.6	0.62
gemini-2.5-flash	0.64	0.65	0.61	0.63	0.63	0.63	0.65	0.62
gpt-4.1	0.61	0.64	0.62	0.63	0.62	0.63	0.62	0.63

Table 3: Wine feature completion results of language models across eight languages. Thinking is disabled for qwen models.

Model	TPR	TNR	MCC
Closed-Weights Models
gemini-2.5-flash	0.59	0.79	0.39
gpt-4.1	0.72	0.53	0.25
gpt-4.1-mini	0.78	0.40	0.20
gpt-4o	0.72	0.46	0.19
gpt-5	0.58	0.59	0.17

Table 4: Performance of language models on the FWP task. Models are evaluated using True Positive Rate (TPR) to measure the identification of good pairings, True Negative Rate (TNR) for spotting bad pairings, and the Matthews Correlation Coefficient (MCC) for a balanced assessment. Thinking is disabled for qwen models.

Figure 3: Scaling behaviour. SommBench score (y-axis) against number of parameters in billions (x-axis). gemini-2.5-flash and gemini-2.5-pro are plotted without a specific x-coordinate, as their parameter counts are undisclosed; they are included as performance reference points only.

Multilingual Language Model Evaluation

Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.

See all 485 papers

Aug 20, 2026

OenoBench: A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models

We introduce OenoBench, a wine-domain knowledge benchmark of 3,266 multiple-choice questions across six pillars (regions, grape varieties, viticulture, winemaking, producers, business) and four difficulty tiers. The corpus is built from 38,104 atomic, source-anchored facts extracted by 35 provenance-verified scrapers from government registries (INAO, TTB, OIV), peer-reviewed journals, and Wikipedia/Wikidata. Our methodological contribution is an LLM-driven pipeline in which language models reformat verified facts and audit the result, but never serve as the source of truth: every claim traces to a URL, every question is generated by one of five strategies across five generator families, and every question is scored by a nine-agent audit calibrated against a human gold sheet via Cohen's

κ

. Evaluating sixteen frontier configurations, we find: (i) overall accuracy spans 53%-84%, led by o3 at 83.6%; (ii) reasoning-mode lift concentrates in DeepSeek R1 (+6.8pp) and is absent in Claude Opus and Gemini Pro; (iii) Anthropic shows +9pp self preference on its own questions while Google shows -8pp inverse preference; (iv) frontier open-weight models share the cost-vs-accuracy Pareto frontier with proprietary reasoning models; and (v) every config gains around 33pp on closed-book solvable items, revealing a parametric-recall ceiling that only the contextual slice avoids. We release corpus, audit findings, human-review app, and construction code under CC-BY-SA-4.0.

Nikita Khudov

StrategAI

LLM Evaluation Factual Knowledge in Language Models Knowledge-Intensive QA

Ask PDF

Apr 21, 2026

Lost in Translation: Do LVLM Judges Generalize Across Languages?

Automatic evaluators such as reward models play a central role in the alignment and evaluation of large vision-language models (LVLMs). Despite their growing importance, these evaluators are almost exclusively assessed on English-centric benchmarks, leaving open the question of how well these evaluators generalize across languages. To answer this question, we introduce MM-JudgeBench, the first large-scale benchmark for multilingual and multimodal judge model evaluation, which includes over 60K pairwise preference instances spanning 25 typologically diverse languages. MM-JudgeBench integrates two complementary subsets: a general vision-language preference evaluation subset extending VL-RewardBench, and a chart-centric visual-text reasoning subset derived from OpenCQA, enabling systematic analysis of reward models (i.e., LVLM judges) across diverse settings. We additionally release a multilingual training set derived from MM-RewardBench, disjoint from our evaluation data, to support domain adaptation. By evaluating 22 LVLMs (15 open-source, 7 proprietary), we uncover substantial cross-lingual performance variance in our proposed benchmark. Our analysis further shows that model size and architecture are poor predictors of multilingual robustness, and that even state-of-the-art LVLM judges exhibit inconsistent behavior across languages. Together, these findings expose fundamental limitations of current reward modeling and underscore the necessity of multilingual, multimodal benchmarks for developing reliable automated evaluators.

Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5

York University · University of Alberta · Nanyang Technological University +1

Multimodal Model Evaluation Multilingual VLM Evaluation Multimodal Reward Modeling VLM Evaluation Vision-Language Model-as-a-Judge Cross-Lingual Consistency in Language Models+1

Ask PDF

Jun 2, 2026

CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks

Selecting a pretrained language model, or evaluating a fine-tuned one, for a specific application is a high-value decision, yet the public benchmarks used to make it are poorly suited: a generic benchmark need not reflect a particular sub-domain or sub-task, and its scores are suspect when its items have leaked into pretraining and are recalled rather than solved. We present CoEval, an open framework that supplies a trustworthy, task-specific signal through ensemble self-evaluation: from a task or domain description, a pool of models rotates through all three roles, teacher, student, and judge, to generate a fresh, contamination-free benchmark, answer it, and score one another, with no human labels or raters. Because every model also answers as a student, the responses are the data that weight each question by its discriminative power and each judge by its consensus with the panel. Where ground truth exists, CoEval recovers the true ranking and tracks objective correctness at \r{ho}=0.86, and the weighting recovers the gold ranking of thirteen models at Spearman 0.95. Reliability comes from panel composition, not size: this label-free weighting zeroes out broken judges and down-weights saturated questions, so neither distorts the ranking. Generated items show zero verbatim overlap with five public benchmarks, the panel cancels verbosity bias and precludes same-family self-preference, and rankings are domain-specific: three different models top four de-novo domains, so a generic leaderboard misdirects most practitioners. The same pipeline reruns on each model release, giving any team a contamination-free leaderboard for its application.

Alexander Apartsin, Yehudit Aperstein

Holon Institute of Technology, Israel · Afeka Tel Aviv Academic College of Engineering, Israel

Model Selection LLM Evaluation Language Model Self-Assessment Synthetic Benchmark Generation Language Model Ensembles

Ask PDF

Abstract

Multilingual Language Model Evaluation

Explore similar work

OenoBench: A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models

Lost in Translation: Do LVLM Judges Generalize Across Languages?

CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks