In Bangla, a main verb is often followed by one of a small set of vector verbs. These verbs lose their own meaning and add a shade of meaning instead, such as completion. For example, likhe phello means that someone finished writing, not that they wrote and threw something. The same two verbs can also describe two separate events, so only the context shows which meaning is meant. This makes compound verbs a hard test of how language models handle multiword expressions. We present BanglaCV-Probe, a benchmark of 17 vector verbs with 1,044 minimal pairs, 204 entailment items and 197 pairs of correct and incorrect combinations, all checked by native speakers. We test 26 models (encoders, LLMs and translation systems) on four skills: telling the two meanings apart internally, understanding what a vector adds, preferring correct combinations and keeping the vector's meaning in translation. Probes on five encoders tell the two meanings apart and still reach up to 93% accuracy when trained on pairs without surface cues. Prompted models rarely use this knowledge. Entailment accuracy stays between 59% and 81%. Several LLMs answer Yes or No to almost every item, and nine of ten models miss that someone who ate up the rice left none. Most instruction-tuned LLMs translate vectors literally in 15-20% of vector sentences and in up to 55% for mora (die). On correct combinations, a small Bangla-only model (88%) matches BLOOM-7B1 (87%) and both beat Qwen2.5-7B (74%). Data and code: https://github.com/ashuvo25/Pole_OR_Vector
Figures & tables
Figure 1: The same V2 \bengalifont ফেলা (throw) as a vector ( v ) and as a full verb ( f ). Probes on the V2’s hidden state get both sentences right in all five encoders. The LLM Gemma-2-2B-IT still translates both with threw . Translations are real model outputs.
Part
Items
Used for
Minimal pairs (vector vs. full verb)
1,044
RQ1 RQ4
adjacent / hard (cue-free)
74
from real text
103
Entailment items
204
RQ2
Selectional pairs (valid vs. invalid)
197
RQ3
Total
1,445
Table 1: BanglaCV-Probe . All parts cover the 17 vector verbs (64–95 items per verb). Colours match the research questions in Table 2 .
RQ1
RQ2
RQ3
RQ4
probe
NLI
surpr.
MT
Encoders
BanglaBERT, mBERT, MuRIL
∙
∙ †
XLM-R base, large
∙
∙
NLI classifiers
mDeBERTa ( × 2), XLM-R-l
∙
Table 2: The 26 models and where they are used. ∗ Instruction-tuned version (BLOOMZ for BLOOM); RQ4 uses only Qwen2.5-7B. † In RQ3, BanglaBERT is replaced by its generator, which has a masked-language-model head.
Figure 2: Probe accuracy on the test sets. Lines join results of the same probe, trained on ordinary pairs. The surface cue (dashed) is near-perfect on corpus pairs but at chance on hard pairs, and the transferred probe falls to the level of random-weight copies (grey). Probes re-trained within the hard pairs (5-fold, right) reach 0.80–0.93.
Hard pairs
Model
ℓ
Test
Corpus
transfer
5-fold
rand.
BanglaBERT
4
.97
.87
.64
.93
.51
mBERT
9
.93
.67
.59
.80
.51
MuRIL
2
.96
.82
.54
.91
.49
XLM-R base
6
.95
.79
.61
.85
.56
XLM-R large
9
.95
.78
.55
.86
.60
Table 3: Probe accuracy. ℓ : layer chosen on dev. Transfer: probe trained on ordinary pairs, at layer ℓ ; 5-fold: trained and tested within the hard pairs; rand.: the same 5-fold probe on a random-weight copy. The surface cue is correct on 205 of 206 corpus sentences (.995). All pretrained–random differences in the 5-fold setting are significant ( p<0.001 ).
Figure 3: Share of Yes answers per item category, with Wilson 95% intervals. Hollow diamonds mark the gold answer. Shaded: literal traps, where the hypothesis states the V2’s lexical meaning and the gold answer is No . Several LLMs give nearly the same answer in every category.
Model
Acc
AUC
Yes
Trap
mDeBERTa-NLI (2mil7)
.77
.85
.45
.72
mDeBERTa-NLI (XNLI)
.77
.84
.53
.59
XLM-R-large-XNLI
.75
.86
.61
.50
Qwen2.5-0.5B-Inst
.62
.66
.48
.73
Qwen2.5-1.5B-Inst
.75
.81
.53
.73
Qwen2.5-3B-Inst
.59
.79
.05
1.00
Table 4: Entailment results. Yes = share of Yes answers (gold: 0.47); Trap = accuracy on the 64 literal traps. LLM scores average an English and a Bangla prompt.
Figure 4: Selectional accuracy (filled) against each model’s fewer-tokens baseline (hollow), with 95% CIs. All models beat their baseline.
Model
Acc
Per tok.
Base
Benef.
BanglaBERT-gen
.88
.82
.55
.70
mBERT
.64
.59
.55
.30
MuRIL
.80
.77
.55
.83
XLM-R base
.68
.65
.55
.65
XLM-R large
.73
.70
.55
.57
Qwen2.5-0.5B
.60
.55
.51
.39
Table 5: Selectional accuracy (197 pairs). Per tok.: per-token log-probability; Base: the model’s fewer-tokens baseline; Benef.: the 23 benefactive pairs.
Figure 5: Share of English translations that contain the V2’s literal meaning, by sentence type, with Wilson 95% intervals. Hollow diamonds: human reference translations. For vector sentences, lower is better; for full-verb sentences (right), higher is better.
Literal
Meaning
chrF++
System
error ↓
lost ↓
vec.
full
NLLB-200 600M
9.6
27.1
45.9
50.9
NLLB-200 1.3B
6.5
27.9
48.9
54.6
NLLB-200 3.3B
6.5
26.4
50.3
55.7
BanglaT5-NMT
2.8
25.6
49.9
59.7
Gemma-2-2B-IT
18.0
34.7
41.6
41.1
Table 6: Translation results (%). Literal error: vector sentences translated with the V2’s literal meaning. Meaning lost: full-verb sentences whose literal meaning is missing. chrF++ against our reference translations.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Layer-wise probe accuracy. Grey: test accuracy, which saturates by layer 2. Coloured: probe trained on ordinary pairs and applied to hard pairs; shading marks the gain over the random-weight copy (dashed). Triangles: layer chosen on dev; circles: best layer on hard pairs.
Figure 7: RQ1 probe accuracy per vector verb. n = number of pairs. The hard pairs are few per verb, so per-verb differences there are indicative only.
Figure 8: Accuracy on vector-meaning items (gold Yes ) vs. literal traps (gold No ). Points near the dashed line do not discriminate.
Figure 9: RQ3 accuracy of causal models by size. The Qwen2.5 series is flat up to 3B; BLOOM-7B1 clearly exceeds Qwen2.5-7B.
Figure 10: RQ4 literal error rate per vector verb (values ≥ .15 printed). \bengalifont মরা ‘die’ is read literally by almost every system.
Figure 11: RQ4 trade-off between literal errors on vector sentences and lost meaning on full-verb sentences. BLOOMZ avoids literal errors by dropping meaning.
Authored
Corpus
System
separated
same V2
adjacent
(real text)
n=735
n=58
n=66
n=89
MT systems
NLLB-0.6B
7
38
9
10
NLLB-1.3B
4
33
8
10
NLLB-3.3B
4
24
9
11
Appendix
Table 17: RQ4 literal error rate (%) by subset: vector sentences whose reference contains no literal word, but whose translation does.