In Bangla, a main verb is often followed by one of a small set of vector verbs. These verbs lose their own meaning and add a shade of meaning instead, such as completion. For example, likhe phello means that someone finished writing, not that they wrote and threw something. The same two verbs can also describe two separate events, so only the context shows which meaning is meant. This makes compound verbs a hard test of how language models handle multiword expressions. We present BanglaCV-Probe, a benchmark of 17 vector verbs with 1,044 minimal pairs, 204 entailment items and 197 pairs of correct and incorrect combinations, all checked by native speakers. We test 26 models (encoders, LLMs and translation systems) on four skills: telling the two meanings apart internally, understanding what a vector adds, preferring correct combinations and keeping the vector's meaning in translation. Probes on five encoders tell the two meanings apart and still reach up to 93% accuracy when trained on pairs without surface cues. Prompted models rarely use this knowledge. Entailment accuracy stays between 59% and 81%. Several LLMs answer Yes or No to almost every item, and nine of ten models miss that someone who ate up the rice left none. Most instruction-tuned LLMs translate vectors literally in 15-20% of vector sentences and in up to 55% for mora (die). On correct combinations, a small Bangla-only model (88%) matches BLOOM-7B1 (87%) and both beat Qwen2.5-7B (74%). Data and code: https://github.com/ashuvo25/Pole_OR_Vector
Figures & tables
Figure 1: The same V2 \bengalifont ফেলা (throw) as a vector ( v ) and as a full verb ( f ). Probes on the V2’s hidden state get both sentences right in all five encoders. The LLM Gemma-2-2B-IT still translates both with threw . Translations are real model outputs.
Part
Items
Used for
Minimal pairs (vector vs. full verb)
1,044
RQ1 RQ4
adjacent / hard (cue-free)
74
from real text
103
Entailment items
204
RQ2
Selectional pairs (valid vs. invalid)
197
RQ3
Total
1,445
Table 1: BanglaCV-Probe . All parts cover the 17 vector verbs (64–95 items per verb). Colours match the research questions in Table 2 .
RQ1
RQ2
RQ3
RQ4
probe
NLI
surpr.
MT
Encoders
BanglaBERT, mBERT, MuRIL
∙
∙ †
XLM-R base, large
∙
∙
NLI classifiers
mDeBERTa ( × 2), XLM-R-l
∙
Table 2: The 26 models and where they are used. ∗ Instruction-tuned version (BLOOMZ for BLOOM); RQ4 uses only Qwen2.5-7B. † In RQ3, BanglaBERT is replaced by its generator, which has a masked-language-model head.
Figure 2: Probe accuracy on the test sets. Lines join results of the same probe, trained on ordinary pairs. The surface cue (dashed) is near-perfect on corpus pairs but at chance on hard pairs, and the transferred probe falls to the level of random-weight copies (grey). Probes re-trained within the hard pairs (5-fold, right) reach 0.80–0.93.
Hard pairs
Model
ℓ
Test
Corpus
transfer
5-fold
rand.
BanglaBERT
4
.97
.87
.64
.93
.51
mBERT
9
.93
.67
.59
.80
.51
MuRIL
2
.96
.82
.54
.91
.49
XLM-R base
6
.95
.79
.61
.85
.56
XLM-R large
9
.95
.78
.55
.86
.60
Table 3: Probe accuracy. ℓ : layer chosen on dev. Transfer: probe trained on ordinary pairs, at layer ℓ ; 5-fold: trained and tested within the hard pairs; rand.: the same 5-fold probe on a random-weight copy. The surface cue is correct on 205 of 206 corpus sentences (.995). All pretrained–random differences in the 5-fold setting are significant ( p<0.001 ).
Figure 3: Share of Yes answers per item category, with Wilson 95% intervals. Hollow diamonds mark the gold answer. Shaded: literal traps, where the hypothesis states the V2’s lexical meaning and the gold answer is No . Several LLMs give nearly the same answer in every category.
Model
Acc
AUC
Yes
Trap
mDeBERTa-NLI (2mil7)
.77
.85
.45
.72
mDeBERTa-NLI (XNLI)
.77
.84
.53
.59
XLM-R-large-XNLI
.75
.86
.61
.50
Qwen2.5-0.5B-Inst
.62
.66
.48
.73
Qwen2.5-1.5B-Inst
.75
.81
.53
.73
Qwen2.5-3B-Inst
.59
.79
.05
1.00
Table 4: Entailment results. Yes = share of Yes answers (gold: 0.47); Trap = accuracy on the 64 literal traps. LLM scores average an English and a Bangla prompt.
Figure 4: Selectional accuracy (filled) against each model’s fewer-tokens baseline (hollow), with 95% CIs. All models beat their baseline.
Model
Acc
Per tok.
Base
Benef.
BanglaBERT-gen
.88
.82
.55
.70
mBERT
.64
.59
.55
.30
MuRIL
.80
.77
.55
.83
XLM-R base
.68
.65
.55
.65
XLM-R large
.73
.70
.55
.57
Qwen2.5-0.5B
.60
.55
.51
.39
Table 5: Selectional accuracy (197 pairs). Per tok.: per-token log-probability; Base: the model’s fewer-tokens baseline; Benef.: the 23 benefactive pairs.
Figure 5: Share of English translations that contain the V2’s literal meaning, by sentence type, with Wilson 95% intervals. Hollow diamonds: human reference translations. For vector sentences, lower is better; for full-verb sentences (right), higher is better.
Literal
Meaning
chrF++
System
error ↓
lost ↓
vec.
full
NLLB-200 600M
9.6
27.1
45.9
50.9
NLLB-200 1.3B
6.5
27.9
48.9
54.6
NLLB-200 3.3B
6.5
26.4
50.3
55.7
BanglaT5-NMT
2.8
25.6
49.9
59.7
Gemma-2-2B-IT
18.0
34.7
41.6
41.1
Table 6: Translation results (%). Literal error: vector sentences translated with the V2’s literal meaning. Meaning lost: full-verb sentences whose literal meaning is missing. chrF++ against our reference translations.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Layer-wise probe accuracy. Grey: test accuracy, which saturates by layer 2. Coloured: probe trained on ordinary pairs and applied to hard pairs; shading marks the gain over the random-weight copy (dashed). Triangles: layer chosen on dev; circles: best layer on hard pairs.
Figure 7: RQ1 probe accuracy per vector verb. n = number of pairs. The hard pairs are few per verb, so per-verb differences there are indicative only.
Figure 8: Accuracy on vector-meaning items (gold Yes ) vs. literal traps (gold No ). Points near the dashed line do not discriminate.
Figure 9: RQ3 accuracy of causal models by size. The Qwen2.5 series is flat up to 3B; BLOOM-7B1 clearly exceeds Qwen2.5-7B.
Figure 10: RQ4 literal error rate per vector verb (values ≥ .15 printed). \bengalifont মরা ‘die’ is read literally by almost every system.
Figure 11: RQ4 trade-off between literal errors on vector sentences and lost meaning on full-verb sentences. BLOOMZ avoids literal errors by dropping meaning.
Authored
Corpus
System
separated
same V2
adjacent
(real text)
n=735
n=58
n=66
n=89
MT systems
NLLB-0.6B
7
38
9
10
NLLB-1.3B
4
33
8
10
NLLB-3.3B
4
24
9
11
Appendix
Table 17: RQ4 literal error rate (%) by subset: vector sentences whose reference contains no literal word, but whose translation does.
Frequent verbs such as 'have' and 'make' can function either as collocates in light-verb constructions or as full lexical predicates, as in 'make a decision' vs. 'make a cake'. Whether language models represent this distinction, and whether such representations vary across languages, remains unclear. We introduce a large-scale controlled dataset in English, Spanish, and French, comprising minimally varying sentence series in which the same context contains the same verb in light-verb and full-verb uses. Two probing experiments show that language models differentiate between these uses even in minimal contexts and exhibit separable patterns across object types. We release the dataset, generation code, and materials as a reusable resource. The framework supports extensions to broader contexts, additional verbs, and other languages.
Francesca Franzon, Nicolas Rosàs Gómez, Leo Wanner
Universitat Pompeu Fabra (UPF) · Catalan Institute for Research and Advanced Studies (ICREA)
Many Bangla words are at once personal names and culturally loaded common nouns, "Maya" is both a girl's name and a word for affectionate compassion. Choosing the right reading demands cultural knowledge that is scarce in the pretraining data of modern language models. We introduce Culturally Entangled Homograph (CEH) disambiguation and build a Bangla benchmark of 1,516 expert-verified sentences (3,032 labelled occurrences) in which one word appears twice with two distinct readings, each labelled with a culturally grounded category and an explanation of the reasoning behind it. Across open- and closed-source models, we find a systematic dominant-meaning bias: models default to the common-noun sense and overlook the name. A Bangla-specific model fails under every prompting regime we test, showing that language-specific pretraining alone does not confer cultural grounding. We further show that contrastive chain-of-thought prompting can sharply reduce this bias without training, and that distilling cultural explanations teaches small (1-3B) models to reason toward the correct reading rather than memorise labels, cutting dominant-meaning bias from as high as 100% to under 5% and turning the failed Bangla-specific model into our strongest system. Dataset and code are available at https://github.com/ashuvo25/BanglaCEH.
Bangla culture is richly expressed through region, dialect, history, food, politics, media, and everyday visual life, yet it remains underrepresented in multimodal evaluation. To address this gap, we introduce BanglaVerse, a culturally grounded benchmark for evaluating multilingual vision-language models (VLMs) on Bengali culture across historically linked languages and regional dialects. Built from 1,152 manually curated images across nine domains, the benchmark supports visual question answering and captioning, and is expanded into four languages and five Bangla dialects, yielding ~32.2K artifacts. Our experiments show that evaluating only standard Bangla overestimates true model capability: performance drops under dialectal variation, especially for caption generation, while historically linked languages such as Hindi and Urdu retain some cultural meaning but remain weaker for structured reasoning. Across domains, the main bottleneck is missing cultural knowledge rather than visual grounding alone, with knowledge-intensive categories. These findings position BanglaVerse as a more realistic test bed for measuring culturally grounded multimodal understanding under linguistic variation.
Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Shubhashis Roy Dipta +6