Recent progress in Natural Language Processing (NLP) has been driven by the emergence of Large Language Models (LLMs), which exhibit remarkable generative and reasoning capabilities. However, despite their success, evaluating the true semantic understanding of these models remains a persistent challenge. Traditional benchmarks such as Word-in-Context (WiC) effectively probe this capability, but their creation is resource-intensive and often limited to high-resource languages. In this paper, we introduce SemBench, a framework for automatically generating synthetic benchmarks that assess the semantic competence of LLMs using only dictionary sense definitions and a sentence encoder. This approach eliminates the need for curated example sentences, making it both scalable and language-independent. We evaluate SemBench in three languages (English, Spanish, and Basque) spanning different levels of linguistic resources, and across a wide range of LLMs. Our results show that rankings derived from SemBench strongly correlate with those obtained from standard WiC datasets. Furthermore, our analysis demonstrates that only a small number of examples is required to achieve stable and meaningful rankings. Overall, SemBench provides a lightweight, adaptable, and data-efficient framework for cross-lingual evaluation of semantic understanding in LLMs.
Figures & tables
Figure 1: A general overview of the SemBench framework.
Language
Reference WiC
Dictionary
Sentence Encoder
(EN) English
Pilehvar and Camacho-Collados (2019)
Oxford Dictionary of English (ODE) 1 1 1 www.oed.com ; through Valera and Rigau (2021) .
EmbeddingGemma 300M
Vera et al. (2025)
(ES) Spanish
Vázquez Abuín and Garcia (2025)
Diccionario de la RAE (DRAE) 2 2 2 www.rae.es/ ; accessed via the public website.
(EU) Basque
Urbizu et al. (2022)
Egungo Euskararen Hiztegia (EEH) 3 3 3 www.ehu.eus/eeh ; SQL dump 2024-01-26 .
Table 1: Resources employed in the experimentation for each language.
Statistic
English
Spanish
Basque
Sense density
6.12
± 3.5
6.92
± 5.9
2.07
± 0.3
Definition length
64.77
± 27.7
59.77
± 31.7
39.99
± 28.1
Example length
106.10
± 42.5
n/a
n/a
Table 2: Statistics of SemBench rand . Lengths are reported in terms of number of characters.
Figure 2: Accuracy (Acc) correlation between SemBench and English WiC using 5-shot configuration. On the left, we compare SemBench rand starting from definitions, whereas on the right, we compare the variant starting from examples.
Figure 3: Accuracy (Acc) correlation between SemBench Def with Spanish (left) and Basque (right) WiC datasets on the 5-shot settings.
Figure 4: Spearman’s ρ correlation curves between different SemBench variants and WiC with varying data-points. Confidence intervals are obtained by applying bootstrapping through 100 iterations.
Figure 5: Spearman’s ρ correlation comparison between 0-Shot and 5-Shot in SemBench Def (left) and SemBench Ex (right) against WiC .
SemBench Ex
SemBench Def
Model Name
easy
med
hard
easy
med
hard
Gemma 3 4B
78.10
67.20
61.40
79.30
68.80
63.70
Qwen3 4B
91.20
86.20
82.00
92.00
86.40
83.50
Llama 2 7B
78.70
68.50
66.00
75.50
64.60
60.80
Llama 3.1 8B
90.80
84.00
81.60
91.20
84.90
82.20
Latxa 8B
87.60
81.10
77.00
86.00
77.90
74.60
Table 3: SemBench English results in 5-Shot scenario in the different difficulty levels. Bold-case indicates the best results by group. Underline indicates the best results overall.
English
Spanish
Basque
Model Name
SB
WiC
SB
WiC
SB
WiC
Gemma 3 4B
70.70
60.16
62.60
53.90
55.08
50.20
Qwen3 4B
88.80
65.16
78.50
57.60
59.82
52.00
Llama 2 7B
67.00
57.36
62.40
54.10
55.53
49.95
Llama 3.1 8B
87.90
65.80
72.70
56.50
59.82
52.60
Latxa 8B
81.80
62.51
69.70
55.40
63.43
51.60
Table 4: Multilingual results in WiC and SemBench Ex (SB) with rand strategy at 5-shot scenario. Bold-case indicates the best results by group. Underline indicates the best results overall.
In frame semantics, sentence comprehension is assumed to proceed by relating lexical meaning to background knowledge called semantic frames, thereby enabling readers to implicitly enrich the text with unstated information. Recent large language models (LLMs) have achieved strong performance across a wide range of downstream tasks. However, it remains unclear whether they can reproduce the kinds of implicit enrichment that humans naturally make during comprehension. To address this question, we introduce FrameBench, a benchmark grounded in frame semantics. FrameBench consists of multiple-choice questions that test whether models distinguish the frames evoked by the same verb across contexts. We construct the benchmark for English and Japanese using FrameNet-style resources and a generation-and-verification pipeline with native-speaker judgments. Our experiments on a diverse set of models reveal challenges for small models, while several large models surpass the human reference scores. We release the constructed FrameBench dataset and the code for dataset construction and evaluation at https://github.com/SasanoLab/FrameBench.
We present SemanticQA, an evaluation suite designed to assess language models (LMs) in semantic phrase processing tasks. The benchmark consolidates existing multiword expression (MwE) resources and reorganizes them into a unified testbed. It covers both general lexical phenomena, such as lexical collocations, and three fine-grained categories: idiomatic expressions, noun compounds, and verbal constructions. Through SemanticQA, we assess LMs of diverse architectures and scales in extraction, classification, and interpretation tasks, as well as sequential task compositions. We reveal substantial performance variation, particularly on tasks requiring semantic reasoning, highlighting differences in reasoning efficacy and semantic understanding of LMs, providing insights for pushing LMs with stronger comprehension on non-trivial semantic phrases. The evaluation harness and data of SemanticQA are available at https://github.com/jacklanda/SemanticQA.
Yang Liu, Hongming Li, Melissa Xiaohui Qin +2
University of Science and Technology Beijing · 2State Key Laboratory of General Artificial Intelligence, BIGAI
Evaluating the multilingual and multicultural capabilities of Large Language Models (LLMs) is essential for their global utility. However, current benchmarks face three critical limitations: (1) fragmented evaluation dimensions that often neglect deep cultural nuances; (2) insufficient language coverage in subjective tasks relying on low-quality machine translation; and (3) shallow analysis that lacks diagnostic depth beyond simple rankings. To address these, we introduce GaoYao, a comprehensive benchmark with 182.3k samples, 26 languages and 51 nations/areas. First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers (General Multilingual, Cross-cultural, Monocultural) and nine cognitive sub-layers. Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective benchmarks into 19 languages and synthesizing cross-cultural test sets for 34 cultures, surpassing prior coverage by up to 111%. Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs. Our findings reveal significant geographical performance disparities and distinct gaps between tasks, offering a reliable map for future work. We release the benchmark (https://github.com/lunyiliu/GaoYao).