Recent progress in Natural Language Processing (NLP) has been driven by the emergence of Large Language Models (LLMs), which exhibit remarkable generative and reasoning capabilities. However, despite their success, evaluating the true semantic understanding of these models remains a persistent challenge. Traditional benchmarks such as Word-in-Context (WiC) effectively probe this capability, but their creation is resource-intensive and often limited to high-resource languages. In this paper, we introduce SemBench, a framework for automatically generating synthetic benchmarks that assess the semantic competence of LLMs using only dictionary sense definitions and a sentence encoder. This approach eliminates the need for curated example sentences, making it both scalable and language-independent. We evaluate SemBench in three languages (English, Spanish, and Basque) spanning different levels of linguistic resources, and across a wide range of LLMs. Our results show that rankings derived from SemBench strongly correlate with those obtained from standard WiC datasets. Furthermore, our analysis demonstrates that only a small number of examples is required to achieve stable and meaningful rankings. Overall, SemBench provides a lightweight, adaptable, and data-efficient framework for cross-lingual evaluation of semantic understanding in LLMs.
Figures & tables
Figure 1: A general overview of the SemBench framework.
Language
Reference WiC
Dictionary
Sentence Encoder
(EN) English
Pilehvar and Camacho-Collados (2019)
Oxford Dictionary of English (ODE) 1 1 1 www.oed.com ; through Valera and Rigau (2021) .
EmbeddingGemma 300M
Vera et al. (2025)
(ES) Spanish
Vázquez Abuín and Garcia (2025)
Diccionario de la RAE (DRAE) 2 2 2 www.rae.es/ ; accessed via the public website.
(EU) Basque
Urbizu et al. (2022)
Egungo Euskararen Hiztegia (EEH) 3 3 3 www.ehu.eus/eeh ; SQL dump 2024-01-26 .
Table 1: Resources employed in the experimentation for each language.
Statistic
English
Spanish
Basque
Sense density
6.12
± 3.5
6.92
± 5.9
2.07
± 0.3
Definition length
64.77
± 27.7
59.77
± 31.7
39.99
± 28.1
Example length
106.10
± 42.5
n/a
n/a
Table 2: Statistics of SemBench rand . Lengths are reported in terms of number of characters.
Figure 2: Accuracy (Acc) correlation between SemBench and English WiC using 5-shot configuration. On the left, we compare SemBench rand starting from definitions, whereas on the right, we compare the variant starting from examples.
Figure 3: Accuracy (Acc) correlation between SemBench Def with Spanish (left) and Basque (right) WiC datasets on the 5-shot settings.
Figure 4: Spearman’s ρ correlation curves between different SemBench variants and WiC with varying data-points. Confidence intervals are obtained by applying bootstrapping through 100 iterations.
Figure 5: Spearman’s ρ correlation comparison between 0-Shot and 5-Shot in SemBench Def (left) and SemBench Ex (right) against WiC .
SemBench Ex
SemBench Def
Model Name
easy
med
hard
easy
med
hard
Gemma 3 4B
78.10
67.20
61.40
79.30
68.80
63.70
Qwen3 4B
91.20
86.20
82.00
92.00
86.40
83.50
Llama 2 7B
78.70
68.50
66.00
75.50
64.60
60.80
Llama 3.1 8B
90.80
84.00
81.60
91.20
84.90
82.20
Latxa 8B
87.60
81.10
77.00
86.00
77.90
74.60
Table 3: SemBench English results in 5-Shot scenario in the different difficulty levels. Bold-case indicates the best results by group. Underline indicates the best results overall.
English
Spanish
Basque
Model Name
SB
WiC
SB
WiC
SB
WiC
Gemma 3 4B
70.70
60.16
62.60
53.90
55.08
50.20
Qwen3 4B
88.80
65.16
78.50
57.60
59.82
52.00
Llama 2 7B
67.00
57.36
62.40
54.10
55.53
49.95
Llama 3.1 8B
87.90
65.80
72.70
56.50
59.82
52.60
Latxa 8B
81.80
62.51
69.70
55.40
63.43
51.60
Table 4: Multilingual results in WiC and SemBench Ex (SB) with rand strategy at 5-shot scenario. Bold-case indicates the best results by group. Underline indicates the best results overall.