Literal Lock-In: How Test Language Shifts Literal Attraction in Typed Decision Models on Bengali Idioms

Adib Sakhawat; Syed Rifat Raiyan

cs.CLOct 15, 2026

Literal Lock-In: How Test Language Shifts Literal Attraction in Typed Decision Models on Bengali Idioms

Authors: Adib Sakhawat, Syed Rifat Raiyan

Organizations: Systems and Software Lab (SSL) Department of Computer Science and Engineering Islamic University of Technology, Dhaka, Bangladesh

Abstract

Decision models, non-generative scorers that report a probability over caller-supplied options and emit no text, are increasingly deployed as routers, classifiers and guardrails. Because they generate nothing, rationales and chain-of-thought do not apply. We audit three: an encoder scorer, a decoder scorer and a closed API, entirely through interventions on their input, using expert-annotated Bengali idioms carrying a gold figurative and a gold literal gloss. Dropping an idiom's own literal gloss into the option list draws every arm toward it, taking 65% to 96% of each arm's remaining Bengali errors. A model-by-language matrix over one shared item set shows how far this depends on the test language: every arm is 1.9 to 6.4 times more captured in Bengali than in English on the same paired entries, the commercial model's apparent resistance is a property of its test language, and capability does not order the failure. Calibration degrades in all three arms, and on the open arms confidence inverts, so gating on it is worse than gating at random. On a semantics-free option list, where accuracy is at chance, the best-calibrated arm still reports mean confidence 0.82. Code and data: https://github.com/sakhadib/literal-lock-in.

Figures & tables

Figure 1: The model takes the words. Laya-ml, Lod-lille and Jev-1.13 keep these three markers throughout the paper. The Bengali idiom \bn অগ্নিবল ( agnibal ) is built from \bn অগ্নি ( fire ) and \bn বল ( strength ) and means intense hunger. With both readings in the option list, all three models select the literal gloss in Bengali, at $p=1.00$ , $0.97$ and $0.36$ . Asked as the paired English entry, with the English glosses of the same four options, Lod-lille and Jev-1.13 recover and Laya-ml does not. Across the $3{,}265$ paired items literal attraction falls from $0.731$ , $0.918$ and $0.304$ in Bengali to $0.379$ , $0.241$ and $0.047$ in English (§ 5.1 ). Every item in that pool carries an English parenthetical inside its Bengali literal option by construction, this one included; § 5.6 measures what that cue is worth and shows capture without it.

Figure 2: The instrument in one view. The audit proceeds from the Bangla Bagdhara corpus through six stages: (1) corpus , which supplies 10,361 expert-annotated idioms with gold figurative and literal glosses; (2) distractor family , the variable E1 manipulates, with d-lit and d-hard injecting the expert literal gloss; (3) item prompt , which fixes the typed question,

k=4

options, instructions, and per-item order seed; (4) models , comprising Laya-ml, Lod-lille, and Jev-1.13; (5) scoring , which records the four-option probability vector,

\arg\max

, and confidence without requiring generated text; and (6) metrics , which evaluate accuracy, LAR, ECE, AURC, agreement, and order stability. The figure’s visual encoding distinguishes the varied distractor family, literal-bearing conditions, models under test, and measured outcomes.

Figure 3: The distractor ladder (left) and literal attraction rate (right) for the two open arms. Laya-ml (

n=8{,}876

), Lod-lille (

n=888

); the black dashed line marks the four-option chance level (

0.250

) in both panels; error bars are 95% bootstrap CIs. Surface- and tag-matched distractors cost a few points; the idiom’s own literal gloss ( d-lit ) collapses both models far below chance, and the stronger model harder.

Distractors	Laya	Lod	$\boldsymbol{\Delta}$
random	0.351	0.484	$+0.133^{***}$
surface	0.339	0.412	$+0.073^{***}$
tag	0.325	0.446	$+0.120^{***}$
$+$ literal	0.136	0.066	$-0.070^{***}$
hard	0.149	0.063	$-0.086^{***}$
lar (lit.)	0.658	0.885

Table 1: The distractor ladder and literal attraction. Every cell is the 888 items the two arms share, so the printed values and

\Delta

have one base (

\Delta

is computed before rounding); Laya-ml’s full-set figures are in Appendix E and differ by at most

0.016

(Appendix O ). McNemar is paired on those items ( ∗∗∗

p<0.001

). Laya-ml, Lod-lille; the darker fill marks the higher value in a row. The sign of

\Delta

reverses at the literal boundary: the stronger arm leads on the first three families and trails on the two containing a literal gloss.

Outcome	Laya	Lod	Jev
correct	0.432	0.453	0.935
captured	0.596	0.787	0.737
other error	0.421	0.399	0.746
ECE ( d-rand )	0.071	0.073	0.005
ECE ( d-lit )	0.385	0.679	0.014
AURC (conf.)	0.924	0.989	0.013

Table 2: Confidence by outcome, pooled over the two literal-bearing families d-lit and d-hard , with calibration below. Bold marks, per model, whether it is more confident when correct or when captured. Each column is on that arm’s own items, so compare pattern rather than level. Both open arms are more confident when captured, and their confidence-gated AURC is worse than the random-gate baseline on the row below it, which inverts selective prediction. Both AURC rows are the d-lit condition, and the random-gate value is the expected AURC of a confidence signal drawn at random on the same items.

	Laya	Lod
Bengali lar under d-lit , by cue
idiom not in gloss	0.616	0.866
idiom in gloss	0.767	0.986
difference	$+0.151$	$+0.120$
no Latin-script cue	0.581	0.871
Latin-script cue	0.746	0.909

Table 3: Two lexical cues are real; neither carries the result. Laya-ml, Lod-lille. Each cue would let a lexical matcher pick the literal option without reading it. Verbatim overlap moves lar on both arms and the Latin-script cue on Laya-ml only, its Lod-lille interval covering zero. The last block keeps items that are multiword, whose literal gloss does not contain the idiom and carries no English parenthetical, so neither lexical cue is available: lar stays far above the

0.250

chance level on both arms. Length is not removed by that subset and is reported separately in § 5.6 , where it does not predict capture. The commercial arm cannot be split this way, since its only Bengali literal run is the matrix pool, where every item carries an English parenthetical by construction.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

	Laya-ml	Lod-lille	Jev-1.13
backbone	mmBERT	Qwen3-0.6B	closed
scoring	marker	independent	unknown
parameters	322M	0.6B	n/a
items	8,876	888	8,761
latency	27.2 ms	86.1 ms	API
weights	open	open	closed

Appendix

Table 4: The three model arms. Item counts are each arm’s main run: the full ladder for Laya-ml, a fixed 10% random subset for Lod-lille, and the English pool for Jev-1.13. Every arm also ran the

3{,}265

-item capture matrix in full. Parameter counts are the model cards’. Appendix O bounds the subset’s drift at

0.016

Arm	Instruction string
Bengali	\bn নিচের বাংলা বাগধারাটির প্রকৃত ভাবার্থ কোনটি?
English	“What is the actual figurative meaning of this idiom?”

Appendix

Table 5: The two instruction strings. Each is one sentence and is byte-identical across families, experiments and arms; the Bengali string is the direct equivalent of the English one. Which string a call receives follows the language of the cell, on every arm: Bengali cells receive the Bengali string and English cells the English one, including the open arms’ English cells in E4 and in the capture matrix. The open arms’ task configuration records only the Bengali string because their English cells set the English one in the run code. The matched condition of Appendix L therefore shares its instruction across the three arms, and differs only in the sampled distractors, which the commercial arm drew under its own seed namespace.

	option text, as supplied	role	Laya-ml $p$	Jev-1.13 $p$
A	\bn একগর্ভা; এক সন্তানের জননী; যে নারী একবার মাত্র গর্ভ ধারণ করে	distractor	0.00	0.02
B	\bn ওজন ( physical weight )	literal	0.99	0.15
C	\bn ক্ষমতা, গুরুত্ব, মর্যাদা	gold	0.00	0.82
D	\bn হালকাভাবে পড়া	distractor	0.01	0.01

Appendix

Table 6: One item exactly as received by the models. The state is the Bengali idiom \bn ওজন ( ojon ), literally “physical weight,” with the English equivalent carry weight . The instruction is identical across items and distractor families. The four option texts are the actual strings reconstructed with the builder and seed used for the run and checked against the recorded gold and literal keys. The probabilities are the returned values. Laya-ml places

0.99

on the literal option, which denotes the physical weight of an object, and

0.00

on the gold figurative gloss; Jev-1.13 places

0.82

on the gold option.

	Varies	Fixed	Seed
E1	distractors	idiom, instr., $k$	1000
E2	idiom surface	meaning, options	2000
E3	idiom form	options, instr.	3000
E4	task language	corpus row, family	4000
E5	nothing (derived)	n/a	n/a
E6	option text	idiom, gold	6000

Appendix

Table 7: The seven interventions. E5 is the calibration and selective-prediction analysis of § 5.2 : it issues no calls of its own and is computed from the E1 outputs. E4’s two cells are paired corpus rows rather than the same strings: the English cell presents the annotator’s English equivalent idiom and the English glosses, so what is held fixed is the row and the family, not the state (Appendix J ). Seeds are the Bengali arms’; the commercial arm uses a disjoint set (Appendix R ) because its blocks were run in a different order.

Family	Laya-ml	Lod-lille	Jev-1.13
	bn	bn	en
d-rand	0.367	0.484	0.968
d-surf	0.338	0.412	0.915
d-sem	0.333	0.446	0.829
d-lit	0.151	0.066	0.931
d-hard	0.144	0.063	0.854

Appendix

Table 8: Accuracy by distractor family, and literal attraction on the two families that contain a literal gloss. Chance is

0.250

. The commercial column is English and is the reason § 5.1 runs the matrix: compare it with the Bengali column of Table 14 , where the same model is captured at

0.304

Figure 4: Reliability, 15 equal-width bins. Panels left to right: Laya-ml, Lod-lille, Jev-1.13. circles are the random-distractor condition, squares in each panel’s own color the

+

literal one, and the gray dashed diagonal perfect calibration. Under d-lit the two open models’ curves bend downward : the model is most wrong where it reports being most sure. Jev-1.13, in English, stays on the diagonal.

	Laya-ml		Lod-lille
Coverage	d-rand	d-lit	d-rand	d-lit
0.05	0.669	0.007	0.977	0.000
0.10	0.597	0.007	0.989	0.000
0.20	0.528	0.023	0.899	0.000
0.35	0.470	0.047	0.785	0.000
full	0.367	0.151	0.484	0.066

Appendix

Table 9: Accuracy under a confidence gate. The two model arms are named in the column-group headers and also carry their usual tints, cyan for Laya-ml and magenta for Lod-lille, so the table reads without color. On random distractors, both arms behave as a caller would hope: restricting to the most confident 5% roughly doubles accuracy. On the literal-bearing condition, the curve inverts. Laya-ml falls from

0.151

at full coverage to

0.007

in its most confident 5%, while Lod-lille answers every one of its most confident 35% incorrectly. Gating on confidence is therefore worse than not gating, and worse than gating at random.

Laya-ml ( $n=8{,}876$ )
Family	acc	ECE	ACE	Brier	AUGRC	rand
d-rand	0.367	0.071	0.073	0.728	0.294	.316
d-lit	0.151	0.385	0.385	1.030	0.448	.425
d-hard	0.144	0.389	0.389	1.030	0.450	.428
Lod-lille ( $n=888$ )
d-rand	0.484	0.073	0.084	0.608	0.193	.258

Appendix

Table 10: Three calibration metrics on the same recorded distributions. Laya-ml, Lod-lille, Jev-1.13. ECE uses 15 equal-width bins, ACE the adaptive equal-mass variant of Nixon et al. (2019) ; the largest gap between them across these nine cells is

0.011

, so the ECE figures elsewhere in the paper are not artifacts of the binning. Brier is the multiclass score and needs no bins; bold marks cells above

0.750

, its value for a uniform four-way distribution, where the model’s reported distribution is less useful to a caller than no preference at all. AUGRC is the generalized risk-coverage area of Traub et al. (2024) , which does not condition risk on the accepted set and so does not over-weight high-confidence failures; the rand column is its analytic random baseline, half the error rate, and bold marks the AUGRC cells that are worse than it. No new model calls were made; every metric here is a re-reading of the released per-item probabilities.

manipulation	$p(\text{gold})$	string sent to the model
E3: the state changes idiom: \bn অগাধ জলের মাছ ( Still waters run deep )
intact	0.21	\bn অগাধ জলের মাছ
word shuffle	0.19	\bn মাছ জলের অগাধ
reversed	0.19	\bn মাছ জলের অগাধ
delete one word	0.24	\bn অগাধ জলের
substitute one word	0.24	\bn অগাধ জলের ফ্যা

Appendix

Table 11: What the interventions do to real Bengali text. The table shows the exact strings used for the idiom \bn অগাধ জলের মাছ ( still waters run deep , literally “a fish of deep water”); the tinted string in each block is the unmodified one the rest of that block is derived from. E3 changes the state supplied to the model; the reported probabilities are Laya-ml’s, on the gold figurative gloss. E6 rewrites the options rather than the state, so a single

p(\text{gold})

is not defined for its rows and the column is empty there. Shuffling and reversing the expression leave the probability essentially unchanged, whereas the carrier sentence changes it. E6 changes the option text while holding its role fixed; after the full gloss, the rows truncate it, scramble its words, translate it, and finally remove its content, ending with the semantics-free control. The other three options undergo the same transformation as the gold option. The E6 tokenizer splits on punctuation and rejoins on spaces, so the semicolons of the full gloss do not survive into the truncated and scrambled rows; the scrambled row is a strict permutation of the full gloss’s ten tokens and can be checked as one.

		Mean	Med.	$\mathbf{>}\textbf{.95}$	Ent.
Laya	d-rand	0.438	0.408	0.001	0.892
Laya	d-lit	0.536	0.479	0.039	0.778
Lod	d-rand	0.447	0.391	0.002	0.881
Lod	d-lit	0.745	0.819	0.171	0.497
Jev	d-rand	0.967	1.000	0.852	0.068
Jev	d-lit	0.931	0.990	0.708	0.131

Appendix

Table 12: Distribution of the returned maximum probability, and normalized entropy of the full vector (

1=

uniform). The literal-bearing family makes both open arms sharper , not flatter: Lod-lille’s mean peak rises from

0.447

0.745

and the share of near-deterministic answers from

0.002

0.171

, while its accuracy falls from

0.484

0.066

. The model is not confused by the literal gloss; it is convinced by it.

idiom	expert literal gloss	gold figurative gloss	English equivalent	$p(\text{literal})$
\bn জলের দাম	\bn জলের মূল্য (the price of water)	\bn অতি সস্তাদর (extremely cheap)	dirt cheap	0.99
\bn অক্ষর পরিচয়	\bn অক্ষর বা অক্ষরমালার জ্ঞান বা পরিচয় (knowledge of letters or the alphabet)	\bn সামান্যতম জ্ঞান (the slightest knowledge)	not know one’s ABCs	0.99
\bn দু কথা	\bn দুটি কথা (two words)	\bn সামান্য কথা; অল্প ও সাধারণ কথা (a trifling remark; a brief and ordinary exchange)	a word or two	0.99
\bn রাম-লক্ষণ	\bn রাম এবং লক্ষ্মণ (Rama and Lakshmana, two brothers)	\bn ভ্রাতৃভক্ত দুই ভাই (two devoted brothers)	blood brothers	1.00
\bn পেঁয়াজ পয়জার দুইই হওয়া	\bn পেঁয়াজ এবং জুতা উভয়ই হওয়া (to be both the onion and the shoe)	\bn মার ও গালাগালি খাওয়া (to take both a beating and abuse)	catch hell	0.98

Appendix

Table 13: Items where Laya-ml places almost all probability on the literal gloss. All five are drawn from the

2{,}480

items on which none of the lexical construction cues of § 5.6 is available: each idiom is multiword, its literal gloss does not contain the idiom string, and the gloss carries no English parenthetical, so no option in the list carries Latin script. Each idiom does still share a content word with its literal gloss, the hook that Table 16 tests and finds does not carry capture. Glosses and English equivalents are the corpus’s own; the English in parentheses is ours, for readability. Both the literal and the figurative gloss were in the option list, and the last column is the mass placed on the literal one under d-lit on the full Bengali run. On every one of these the model gave the gold gloss at most

0.011

Figure 5: Literal attraction by model and task language, all six cells on the same 3,265 items (error bars 95% bootstrap CIs; dashed line the four-option chance level). Laya-ml, Lod-lille, Jev-1.13; solid bars Bengali, hatched bars English. The Bengali ordering does not follow capability, and Jev-1.13’s English value matches the

0.047

of the English ladder. The cells are in Table 14 and their intervals in Appendix I .

	Bengali		English
	acc	lar	acc	lar
Laya-ml	0.127	0.731	0.417	0.379
Lod-lille	0.044	0.918	0.684	0.241
Jev-1.13	0.536	0.304	0.926	0.047

Appendix

Table 14: The literal condition as a model

\times

language matrix, all six cells measured on the same 3,265 items with one construction and one seed (

n=3{,}265

per cell; 95% CIs in Appendix I ). Bold marks the highest lar per language. Every arm is far more captured in Bengali than in English, and the Bengali ordering does not follow capability: Jev-1.13, the strongest arm, is the least captured in both languages, while Lod-lille, stronger than Laya-ml on random distractors (Table 1 ), is the more captured of the two in Bengali.

	lar 95% CI
	Bengali	English	$\boldsymbol{\Delta}$
Laya-ml	$[.716,.746]$	$[.363,.397]$	$+0.352$
Lod-lille	$[.909,.928]$	$[.227,.256]$	$+0.677$
Jev-1.13	$[.288,.319]$	$[.040,.055]$	$+0.256$

Appendix

Table 15: Bootstrap intervals on the lar cells of Table 14 and the paired within-model language effect. Intervals are 2,000-resample percentile intervals on the same

3{,}265

items;

\Delta

is the Bengali-minus-English difference on those paired items, McNemar

p<10^{-100}

in all three rows.

	Laya	Lod	Jev
Bengali lar by surface hook
hook absent ( $n=1429$ )	0.716	0.845	0.376
hook present ( $n=1836$ )	0.742	0.975	0.247
difference	$+0.026$	$+0.130$	$-0.129$
Capture gap, hook-absent items only
Bengali lar	0.716	0.845	0.376

Appendix

Table 16: The content-word hook does not carry the language gap. Columns are Laya-ml, Lod-lille and Jev-1.13. “Hook” means the Bengali literal gloss shares at least one content word with the idiom, one of the two asymmetries most likely to inflate Bengali capture. Upper block: capture is not reliably higher when the hook is present, and on the commercial arm it is lower. Lower block: restricting both language cells to hook-absent items leaves the capture gap essentially unchanged, and larger for two of three arms. The script is released with the per-item outputs.

	Laya	Lod
Bengali lar under d-lit
cue absent	0.581	0.871
$n$	5,485	560
cue present	0.746	0.909
$n$	3,391	328
difference	$+0.165$	$+0.037$

Appendix

Table 17: The Latin-script cue is real but does not carry the result. Laya-ml, Lod-lille. Items are split by whether the idiom’s literal gloss carries an English parenthetical, which is the only thing in a Bengali option list written in Latin script. Upper block: the cue is worth

+0.165

of capture on the marker scorer and little on the independent-option scorer, yet capture without it is still

0.581

and

0.871

against a chance level of

0.250

. Lower block: the same split moves accuracy on families containing no literal option by at most

0.042

, and on Laya-ml by a consistent

1.4

3.0

points, five to twelve times less than what it buys on the literal family. The commercial arm cannot be split this way, since its only Bengali literal run is the pool, where every item carries a parenthetical.

Figure 6: Accuracy by task language, grouped by model: Laya-ml (four bars), Lod-lille, Jev-1.13. Solid bars are Bengali input, hatched bars English; Laya-ml’s third and fourth bars use the English-only checkpoint, the fourth being the control. Error bars 95% bootstrap CIs, dashed line chance. Every model gains between 23 and 38 points when the paired English entry is presented instead of the Bengali idiom.

Figure 7: The semantics-free control. Per model, the bar is accuracy (95% bootstrap CI) and the colored bar is mean reported confidence ( Laya-ml, Lod-lille, Jev-1.13); the dashed line is the four-option chance level. No option carries information, so the gap between the two bars is overconfidence:

0.250

0.049

and

0.561

. Jev-1.13’s condition is English.

Model	$n$	Accuracy	95% CI
Laya-ml	8,876	0.367	[0.357, 0.377]
Lod-lille	888	0.484	[0.453, 0.518]
Jev-1.13	8,761	0.736	[0.727, 0.745]

Appendix

Table 18: The common baseline across all three arms. The condition is matched across models: a Bengali idiom followed by its Bengali figurative gloss with random distractors, the same instruction string and the same

k

. It is not byte-identical, because the commercial arm drew its sampled distractors under a disjoint seed namespace (Appendix R ), so the three option lists share the gold and differ in the three distractors. Each row is on that arm’s own items, so

n

differs. The pairwise differences are computed on the items each pair shares, which is why they do not equal the differences between rows: Jev–Laya

+0.369

, Jev–Lod

+0.259

and Lod–Laya

+0.133

(McNemar, all

p<10^{-9}

Figure 8: The matched three-way comparison. Bars left to right: Laya-ml, Lod-lille, Jev-1.13. Black dashed line is chance, error bars 95% bootstrap CIs. This capability ordering is the baseline against which the calibration failures in § 5.2 –§ 5.3 should be read.

Figure 9: Unit integrity: the idiom is perturbed, the options held fixed. Laya-ml, Lod-lille; error bars 95% bootstrap CIs, dashed line chance (left panel). Left: accuracy. Right: agreement with the answer on the intact idiom. Scrambling and reversing the expression does not meaningfully hurt either model, which is what a model reading the expression as a set of words would show.

Family	Full	Subset	Drift
d-rand	0.367	0.351	$-0.016$
d-surf	0.338	0.339	$+0.001$
d-sem	0.333	0.325	$-0.007$
d-lit	0.151	0.136	$-0.015$
d-hard	0.144	0.149	$+0.004$

Appendix

Table 19: Laya-ml on the 10% subset against its own full set, drift computed before rounding. Maximum drift is

0.016

, and the larger drifts are downward, a slight lean of under two points. Two of five subset estimates fall marginally outside the full-set bootstrap CI, which is expected at

n=888

and changes no sign or ordering. The full-set figures are therefore reported as the precise estimates, and Lod-lille’s intervals are reported wider throughout.

Option text	Laya	Lod	Jev
full gloss	0.365	0.492	0.970
truncated	0.361	0.468	0.958
scrambled	0.357	0.479	0.974
other lang.	0.347	0.398	0.963
no semantics	0.256	0.233	0.254

Appendix

Table 20: Option-side ablation. The semantics-free row removes meaningful option content and yields near-chance accuracy in all three arms. Jev-1.13’s column is English.

Figure 10: Option-side ablation. Panels left to right: Laya-ml, Lod-lille, Jev-1.13. In each panel the fifth, bar is the semantics-free control of § 5.3 ; the whole Jev-1.13 panel is hatched because its task is in English . Error bars are 95% bootstrap CIs and the black dashed line is chance. Truncating and word-scrambling the option text cost almost nothing in any of the three models.

	Laya	Lod	Jev
answer changes	0.837	0.107	0.133
correctness varies	0.686	0.070	0.120
distinct glosses	2.83	1.14	1.15
perfectly stable	0.163	0.893	0.867
$p$ (gold) range	0.346	0.029	0.128
selection spread	0.139	0.030	0.011

Appendix

Table 21: Option-order sensitivity. Each item is evaluated under all 24 orderings of the same four options (

n=992/299/300

). Laya, Lod, Jev; bold marks the largest value in each row.

p

(gold) range is the within-item range of the gold option’s probability across the 24 orderings, averaged over items; selection spread is the gap between the most and least chosen slot, pooled over orderings, and is

0

under no positional preference. The marker-based scorer changes its chosen gloss on

83.7\%

of items against

10.7\%

for the independent-option scorer. The three item samples are drawn separately per arm, so cross-arm comparisons are unpaired. Lod is documented as order-independent by construction; the permutation audit below shows it is not, which is why its entries here are small rather than zero.

	Laya	Lod	Jev
items	992	299	300
median spread	0.495	0.040	0.070
mean spread	0.507	0.046	0.139
95th pct. spread	0.772	0.105	0.480
max spread	0.928	0.245	0.820
share $<10^{-4}$	0.000	0.000	0.230

Appendix

Table 22: Permutation audit. Per item, the largest distance a single option’s own score travels across the 24 orderings of its list, after mapping each slot back to the option it displayed. Laya-ml, Lod-lille, Jev-1.13. Bold marks the largest value in each row. An order-independent scorer should sit at fp32 noise, around

10^{-7}

; the share row gives the fraction of items at or below

10^{-4}

, a threshold three orders of magnitude more permissive than that. Only Jev-1.13 has any items at that threshold, and then fewer than a quarter; its probabilities are reported to two decimal places, so for that arm the row means identical as reported rather than identical in the underlying computation. The script that produces this table is released with the per-item outputs.

Figure 11: Share of selections falling on each option slot, over the full ladder condition rather than the permutation sweep. Panels left to right: Laya-ml, Lod-lille, Jev-1.13. The gold option is placed uniformly at random, so the black dashed line at

0.25

is the expectation under no positional bias and any departure from it is bias. Laya-ml peaks on slot B, taking it

37.2\%

of the time against

14.6\%

for slot A; the other two arms sit close to the line throughout, with a mild and monotone preference for earlier slots.

Figure 12: Order sensitivity. Left: how many distinct glosses each model selects across the 24 orderings of one option set; within each group bars run Laya-ml, Lod-lille, Jev-1.13, and a value of 1 means perfect stability. Right: per model, the colored bar is the share of items whose chosen gloss is unstable and the bar is the mean within-item range of

p

(gold). Laya-ml is unstable on 83.7% of items with a mean

p

(gold) swing of

0.346

; the other two are near-stable.

Cognitive Bias in Language Models

See all 16 papers

Jun 17, 2026

G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment

Idioms are difficult to transfer across languages due to their non-compositionality and weak surface-form grounding, making literal mappings unreliable. We present G-IdiomAlign, a gloss-pivoted benchmark where each idiom is anchored by an English gloss from Wiktionary. We further construct a high-confidence reference alignment set for reproducible evaluation. G-IdiomAlign supports two protocols: (1) a controlled Multiple-Choice Idiom Equivalence with typed distractors for error attribution; and (2) a Gloss-Contrastive Generation contrasting No-gloss and With-gloss inputs to isolate the effect of an explicit semantic pivot. Across diverse LLMs, a bias to literal translation is a dominant failure mode, especially when the target is a low-resource language. Glosses consistently improve Gloss-Contrastive Generation under an embedding-based semantic proxy, but performance remains modest, indicating substantial headroom in the open output space. Subsequent analysis on Qwen3-8B further suggests that cross-condition differences are concentrated more in attention heads than in layers, while better With-gloss generations coincide with stronger gloss anchoring.

Fengying Ye, Yanming Sun, Runzhe Zhan +3

NLP2CT Lab, Department of Computer and Information Science, University of Macau · Faculty of Arts and Humanities, University of Macau

Figurative Language Understanding Multilingual Language Model Evaluation LLM Evaluation Machine Translation Cross-Lingual Transfer

Ask PDF

Apr 12, 2026

When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalities

Idiomatic reasoning, deeply intertwined with metaphor and culture, remains a blind spot for contemporary language models, whose progress skews toward surface-level lexical and semantic cues. For instance, the Bengali idiom \foreignlanguage{bengali}{\char"0986\char"0999\char"09CD\char"0997\char"09C1 \char"09B0 \char"09AB\char"09B2 \char"099F\char"0995} (angur fol tok,

grapes are sour''): it encodes denial-driven rationalization, yet naive models latch onto the literal fox-and-grape imagery. Addressing this oversight, we present

Mediom,'' a multilingual, multimodal idiom corpus of 3,533 Hindi, Bengali, and Thai idioms, each paired with gold-standard explanations, cross-lingual translations, and carefully aligned text--image representations. We benchmark both large language models (textual reasoning) and vision-language models (figurative disambiguation) on Mediom, exposing systematic failures in metaphor comprehension. To mitigate these gaps, we propose ``HIDE,'' a Hinting-based Idiom Explanation framework that leverages error-feedback retrieval and targeted diagnostic cues for iterative reasoning refinement. Collectively, Mediom and HIDE establish a rigorous test bed and methodology for culturally grounded, multimodal idiom understanding embedded with reasoning hints in next-generation AI systems.

Sarmistha Das, Shreyas Guha, Suvrayan Bandyopadhyay +3

Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, India · School of Information Technology, King Mongkut’s Institute of Technology Ladkrabang, Bangkok, Thailand

Retrieval-Augmented Reasoning Figurative Language Understanding Cross-Lingual Reasoning in Language Models Multimodal Reasoning Cross-Cultural Language Model Evaluation

Ask PDF

Oct 13, 2026

Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges

Typed decision models (TDMs) return a probability for each allowed answer instead of generating text, which makes them candidates for cheaply replacing LLM judges that compare two answers. Whether they share the biases of LLM judges is untested. We run the CALM benchmark's bias tests on eight TDMs and seven LLM judges of 2024 and 2026. Every TDM is biased, in three to eight of twelve tests. Emotional tone affects them most, and rewriting the better answer in such a tone lowers the accuracy of a typical TDM from 99% to 42%. TDMs are more biased than 2026 LLM judges. All eight change verdicts when the answers swap places, which the 2026 LLM judges hardly do, and several side with an answer that most people allegedly prefer or that a famous model allegedly wrote. Yet both kinds of judge are tripped up by the same prompts. The TDMs' probabilities show that most tests mainly flip verdicts the TDM was unsure of, and that good calibration does not protect against bias. As TDMs are more consistent and 37 times cheaper than 2026 LLM judges, our results suggest they can be attractive judges if their biases are tested for the task.

Fabian Lukassen, Christoph Weisser, Alexander Silbersdorff

University of Göttingen · Bielefeld University of Applied Sciences and Arts (HSBI)

Bias in LLM-as-a-Judge LLM-as-a-Judge LLM Evaluation Language Model Bias Evaluation

Ask PDF

Abstract

Cognitive Bias in Language Models

Explore similar work

G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment

When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalities

Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges