cs.CLOct 15, 2026
SaveLiteral Lock-In: How Test Language Shifts Literal Attraction in Typed Decision Models on Bengali Idioms
Organizations: Systems and Software Lab (SSL) Department of Computer Science and Engineering Islamic University of Technology, Dhaka, Bangladesh
Abstract
Decision models, non-generative scorers that report a probability over caller-supplied options and emit no text, are increasingly deployed as routers, classifiers and guardrails. Because they generate nothing, rationales and chain-of-thought do not apply. We audit three: an encoder scorer, a decoder scorer and a closed API, entirely through interventions on their input, using expert-annotated Bengali idioms carrying a gold figurative and a gold literal gloss. Dropping an idiom's own literal gloss into the option list draws every arm toward it, taking 65% to 96% of each arm's remaining Bengali errors. A model-by-language matrix over one shared item set shows how far this depends on the test language: every arm is 1.9 to 6.4 times more captured in Bengali than in English on the same paired entries, the commercial model's apparent resistance is a property of its test language, and capability does not order the failure. Calibration degrades in all three arms, and on the open arms confidence inverts, so gating on it is worse than gating at random. On a semantics-free option list, where accuracy is at chance, the best-calibrated arm still reports mean confidence 0.82. Code and data: https://github.com/sakhadib/literal-lock-in.
Figures & tables
Figure 1: The model takes the words. Laya-ml, Lod-lille and Jev-1.13 keep these three markers throughout the paper. The Bengali idiom \bn অগ্নিবল ( agnibal ) is built from \bn অগ্নি ( fire ) and \bn বল ( strength ) and means intense hunger. With both readings in the option list, all three models select the literal gloss in Bengali, at , and . Asked as the paired English entry, with the English glosses of the same four options, Lod-lille and Jev-1.13 recover and Laya-ml does not. Across the paired items literal attraction falls from , and in Bengali to , and in English (§ 5.1 ). Every item in that pool carries an English parenthetical inside its Bengali literal option by construction, this one included; § 5.6 measures what that cue is worth and shows capture without it.
Figure 2: The instrument in one view. The audit proceeds from the Bangla Bagdhara corpus through six stages: (1) corpus , which supplies 10,361 expert-annotated idioms with gold figurative and literal glosses; (2) distractor family , the variable E1 manipulates, with d-lit and d-hard injecting the expert literal gloss; (3) item prompt , which fixes the typed question, options, instructions, and per-item order seed; (4) models , comprising Laya-ml, Lod-lille, and Jev-1.13; (5) scoring , which records the four-option probability vector, , and confidence without requiring generated text; and (6) metrics , which evaluate accuracy, LAR, ECE, AURC, agreement, and order stability. The figure’s visual encoding distinguishes the varied distractor family, literal-bearing conditions, models under test, and measured outcomes.
Figure 3: The distractor ladder (left) and literal attraction rate (right) for the two open arms. Laya-ml ( ), Lod-lille ( ); the black dashed line marks the four-option chance level ( ) in both panels; error bars are 95% bootstrap CIs. Surface- and tag-matched distractors cost a few points; the idiom’s own literal gloss ( d-lit ) collapses both models far below chance, and the stronger model harder.
| Distractors | Laya | Lod | |
| random | 0.351 | 0.484 | |
| surface | 0.339 | 0.412 | |
| tag | 0.325 | 0.446 | |
| literal | 0.136 | 0.066 | |
| hard | 0.149 | 0.063 | |
| lar (lit.) | 0.658 | 0.885 |
Table 1: The distractor ladder and literal attraction. Every cell is the 888 items the two arms share, so the printed values and have one base ( is computed before rounding); Laya-ml’s full-set figures are in Appendix E and differ by at most (Appendix O ). McNemar is paired on those items ( ∗∗∗ ). Laya-ml, Lod-lille; the darker fill marks the higher value in a row. The sign of reverses at the literal boundary: the stronger arm leads on the first three families and trails on the two containing a literal gloss.
| Outcome | Laya | Lod | Jev |
|---|---|---|---|
| correct | 0.432 | 0.453 | 0.935 |
| captured | 0.596 | 0.787 | 0.737 |
| other error | 0.421 | 0.399 | 0.746 |
| ECE ( d-rand ) | 0.071 | 0.073 | 0.005 |
| ECE ( d-lit ) | 0.385 | 0.679 | 0.014 |
| AURC (conf.) | 0.924 | 0.989 | 0.013 |
Table 2: Confidence by outcome, pooled over the two literal-bearing families d-lit and d-hard , with calibration below. Bold marks, per model, whether it is more confident when correct or when captured. Each column is on that arm’s own items, so compare pattern rather than level. Both open arms are more confident when captured, and their confidence-gated AURC is worse than the random-gate baseline on the row below it, which inverts selective prediction. Both AURC rows are the d-lit condition, and the random-gate value is the expected AURC of a confidence signal drawn at random on the same items.
| Laya | Lod | |
|---|---|---|
| Bengali lar under d-lit , by cue | ||
| idiom not in gloss | 0.616 | 0.866 |
| idiom in gloss | 0.767 | 0.986 |
| difference | ||
| no Latin-script cue | 0.581 | 0.871 |
| Latin-script cue | 0.746 | 0.909 |
Table 3: Two lexical cues are real; neither carries the result. Laya-ml, Lod-lille. Each cue would let a lexical matcher pick the literal option without reading it. Verbatim overlap moves lar on both arms and the Latin-script cue on Laya-ml only, its Lod-lille interval covering zero. The last block keeps items that are multiword, whose literal gloss does not contain the idiom and carries no English parenthetical, so neither lexical cue is available: lar stays far above the chance level on both arms. Length is not removed by that subset and is reported separately in § 5.6 , where it does not predict capture. The commercial arm cannot be split this way, since its only Bengali literal run is the matrix pool, where every item carries an English parenthetical by construction.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| Laya-ml | Lod-lille | Jev-1.13 | |
|---|---|---|---|
| backbone | mmBERT | Qwen3-0.6B | closed |
| scoring | marker | independent | unknown |
| parameters | 322M | 0.6B | n/a |
| items | 8,876 | 888 | 8,761 |
| latency | 27.2 ms | 86.1 ms | API |
| weights | open | open | closed |
Table 4: The three model arms. Item counts are each arm’s main run: the full ladder for Laya-ml, a fixed 10% random subset for Lod-lille, and the English pool for Jev-1.13. Every arm also ran the -item capture matrix in full. Parameter counts are the model cards’. Appendix O bounds the subset’s drift at .
| Arm | Instruction string |
|---|---|
| Bengali | \bn নিচের বাংলা বাগধারাটির প্রকৃত ভাবার্থ কোনটি? |
| English | “What is the actual figurative meaning of this idiom?” |
Table 5: The two instruction strings. Each is one sentence and is byte-identical across families, experiments and arms; the Bengali string is the direct equivalent of the English one. Which string a call receives follows the language of the cell, on every arm: Bengali cells receive the Bengali string and English cells the English one, including the open arms’ English cells in E4 and in the capture matrix. The open arms’ task configuration records only the Bengali string because their English cells set the English one in the run code. The matched condition of Appendix L therefore shares its instruction across the three arms, and differs only in the sampled distractors, which the commercial arm drew under its own seed namespace.
| option text, as supplied | role | Laya-ml | Jev-1.13 | |
|---|---|---|---|---|
| A | \bn একগর্ভা; এক সন্তানের জননী; যে নারী একবার মাত্র গর্ভ ধারণ করে | distractor | 0.00 | 0.02 |
| B | \bn ওজন ( physical weight ) | literal | 0.99 | 0.15 |
| C | \bn ক্ষমতা, গুরুত্ব, মর্যাদা | gold | 0.00 | 0.82 |
| D | \bn হালকাভাবে পড়া | distractor | 0.01 | 0.01 |
Table 6: One item exactly as received by the models. The state is the Bengali idiom \bn ওজন ( ojon ), literally “physical weight,” with the English equivalent carry weight . The instruction is identical across items and distractor families. The four option texts are the actual strings reconstructed with the builder and seed used for the run and checked against the recorded gold and literal keys. The probabilities are the returned values. Laya-ml places on the literal option, which denotes the physical weight of an object, and on the gold figurative gloss; Jev-1.13 places on the gold option.
| Varies | Fixed | Seed | |
|---|---|---|---|
| E1 | distractors | idiom, instr., | 1000 |
| E2 | idiom surface | meaning, options | 2000 |
| E3 | idiom form | options, instr. | 3000 |
| E4 | task language | corpus row, family | 4000 |
| E5 | nothing (derived) | n/a | n/a |
| E6 | option text | idiom, gold | 6000 |
Table 7: The seven interventions. E5 is the calibration and selective-prediction analysis of § 5.2 : it issues no calls of its own and is computed from the E1 outputs. E4’s two cells are paired corpus rows rather than the same strings: the English cell presents the annotator’s English equivalent idiom and the English glosses, so what is held fixed is the row and the family, not the state (Appendix J ). Seeds are the Bengali arms’; the commercial arm uses a disjoint set (Appendix R ) because its blocks were run in a different order.
| Family | Laya-ml | Lod-lille | Jev-1.13 |
| bn | bn | en | |
| d-rand | 0.367 | 0.484 | 0.968 |
| d-surf | 0.338 | 0.412 | 0.915 |
| d-sem | 0.333 | 0.446 | 0.829 |
| d-lit | 0.151 | 0.066 | 0.931 |
| d-hard | 0.144 | 0.063 | 0.854 |
Table 8: Accuracy by distractor family, and literal attraction on the two families that contain a literal gloss. Chance is . The commercial column is English and is the reason § 5.1 runs the matrix: compare it with the Bengali column of Table 14 , where the same model is captured at .
Figure 4: Reliability, 15 equal-width bins. Panels left to right: Laya-ml, Lod-lille, Jev-1.13. circles are the random-distractor condition, squares in each panel’s own color the literal one, and the gray dashed diagonal perfect calibration. Under d-lit the two open models’ curves bend downward : the model is most wrong where it reports being most sure. Jev-1.13, in English, stays on the diagonal.
| Laya-ml | Lod-lille | |||
|---|---|---|---|---|
| Coverage | d-rand | d-lit | d-rand | d-lit |
| 0.05 | 0.669 | 0.007 | 0.977 | 0.000 |
| 0.10 | 0.597 | 0.007 | 0.989 | 0.000 |
| 0.20 | 0.528 | 0.023 | 0.899 | 0.000 |
| 0.35 | 0.470 | 0.047 | 0.785 | 0.000 |
| full | 0.367 | 0.151 | 0.484 | 0.066 |
Table 9: Accuracy under a confidence gate. The two model arms are named in the column-group headers and also carry their usual tints, cyan for Laya-ml and magenta for Lod-lille, so the table reads without color. On random distractors, both arms behave as a caller would hope: restricting to the most confident 5% roughly doubles accuracy. On the literal-bearing condition, the curve inverts. Laya-ml falls from at full coverage to in its most confident 5%, while Lod-lille answers every one of its most confident 35% incorrectly. Gating on confidence is therefore worse than not gating, and worse than gating at random.
| Family | acc | ECE | ACE | Brier | AUGRC | rand |
|---|---|---|---|---|---|---|
| Laya-ml ( ) | ||||||
| d-rand | 0.367 | 0.071 | 0.073 | 0.728 | 0.294 | .316 |
| d-lit | 0.151 | 0.385 | 0.385 | 1.030 | 0.448 | .425 |
| d-hard | 0.144 | 0.389 | 0.389 | 1.030 | 0.450 | .428 |
| Lod-lille ( ) | ||||||
| d-rand | 0.484 | 0.073 | 0.084 | 0.608 | 0.193 | .258 |
Table 10: Three calibration metrics on the same recorded distributions. Laya-ml, Lod-lille, Jev-1.13. ECE uses 15 equal-width bins, ACE the adaptive equal-mass variant of Nixon et al. (2019) ; the largest gap between them across these nine cells is , so the ECE figures elsewhere in the paper are not artifacts of the binning. Brier is the multiclass score and needs no bins; bold marks cells above , its value for a uniform four-way distribution, where the model’s reported distribution is less useful to a caller than no preference at all. AUGRC is the generalized risk-coverage area of Traub et al. (2024) , which does not condition risk on the accepted set and so does not over-weight high-confidence failures; the rand column is its analytic random baseline, half the error rate, and bold marks the AUGRC cells that are worse than it. No new model calls were made; every metric here is a re-reading of the released per-item probabilities.
| manipulation | string sent to the model | |
| E3: the state changes idiom: \bn অগাধ জলের মাছ ( Still waters run deep ) | ||
| intact | 0.21 | \bn অগাধ জলের মাছ |
| word shuffle | 0.19 | \bn মাছ জলের অগাধ |
| reversed | 0.19 | \bn মাছ জলের অগাধ |
| delete one word | 0.24 | \bn অগাধ জলের |
| substitute one word | 0.24 | \bn অগাধ জলের ফ্যা |
Table 11: What the interventions do to real Bengali text. The table shows the exact strings used for the idiom \bn অগাধ জলের মাছ ( still waters run deep , literally “a fish of deep water”); the tinted string in each block is the unmodified one the rest of that block is derived from. E3 changes the state supplied to the model; the reported probabilities are Laya-ml’s, on the gold figurative gloss. E6 rewrites the options rather than the state, so a single is not defined for its rows and the column is empty there. Shuffling and reversing the expression leave the probability essentially unchanged, whereas the carrier sentence changes it. E6 changes the option text while holding its role fixed; after the full gloss, the rows truncate it, scramble its words, translate it, and finally remove its content, ending with the semantics-free control. The other three options undergo the same transformation as the gold option. The E6 tokenizer splits on punctuation and rejoins on spaces, so the semicolons of the full gloss do not survive into the truncated and scrambled rows; the scrambled row is a strict permutation of the full gloss’s ten tokens and can be checked as one.
| Mean | Med. | Ent. | |||
|---|---|---|---|---|---|
| Laya | d-rand | 0.438 | 0.408 | 0.001 | 0.892 |
| d-lit | 0.536 | 0.479 | 0.039 | 0.778 | |
| Lod | d-rand | 0.447 | 0.391 | 0.002 | 0.881 |
| d-lit | 0.745 | 0.819 | 0.171 | 0.497 | |
| Jev | d-rand | 0.967 | 1.000 | 0.852 | 0.068 |
| d-lit | 0.931 | 0.990 | 0.708 | 0.131 |
Table 12: Distribution of the returned maximum probability, and normalized entropy of the full vector ( uniform). The literal-bearing family makes both open arms sharper , not flatter: Lod-lille’s mean peak rises from to and the share of near-deterministic answers from to , while its accuracy falls from to . The model is not confused by the literal gloss; it is convinced by it.
| idiom | expert literal gloss | gold figurative gloss | English equivalent | |
|---|---|---|---|---|
| \bn জলের দাম | \bn জলের মূল্য (the price of water) | \bn অতি সস্তাদর (extremely cheap) | dirt cheap | 0.99 |
| \bn অক্ষর পরিচয় | \bn অক্ষর বা অক্ষরমালার জ্ঞান বা পরিচয় (knowledge of letters or the alphabet) | \bn সামান্যতম জ্ঞান (the slightest knowledge) | not know one’s ABCs | 0.99 |
| \bn দু কথা | \bn দুটি কথা (two words) | \bn সামান্য কথা; অল্প ও সাধারণ কথা (a trifling remark; a brief and ordinary exchange) | a word or two | 0.99 |
| \bn রাম-লক্ষণ | \bn রাম এবং লক্ষ্মণ (Rama and Lakshmana, two brothers) | \bn ভ্রাতৃভক্ত দুই ভাই (two devoted brothers) | blood brothers | 1.00 |
| \bn পেঁয়াজ পয়জার দুইই হওয়া | \bn পেঁয়াজ এবং জুতা উভয়ই হওয়া (to be both the onion and the shoe) | \bn মার ও গালাগালি খাওয়া (to take both a beating and abuse) | catch hell | 0.98 |
Table 13: Items where Laya-ml places almost all probability on the literal gloss. All five are drawn from the items on which none of the lexical construction cues of § 5.6 is available: each idiom is multiword, its literal gloss does not contain the idiom string, and the gloss carries no English parenthetical, so no option in the list carries Latin script. Each idiom does still share a content word with its literal gloss, the hook that Table 16 tests and finds does not carry capture. Glosses and English equivalents are the corpus’s own; the English in parentheses is ours, for readability. Both the literal and the figurative gloss were in the option list, and the last column is the mass placed on the literal one under d-lit on the full Bengali run. On every one of these the model gave the gold gloss at most .
Figure 5: Literal attraction by model and task language, all six cells on the same 3,265 items (error bars 95% bootstrap CIs; dashed line the four-option chance level). Laya-ml, Lod-lille, Jev-1.13; solid bars Bengali, hatched bars English. The Bengali ordering does not follow capability, and Jev-1.13’s English value matches the of the English ladder. The cells are in Table 14 and their intervals in Appendix I .
| Bengali | English | |||
|---|---|---|---|---|
| acc | lar | acc | lar | |
| Laya-ml | 0.127 | 0.731 | 0.417 | 0.379 |
| Lod-lille | 0.044 | 0.918 | 0.684 | 0.241 |
| Jev-1.13 | 0.536 | 0.304 | 0.926 | 0.047 |
Table 14: The literal condition as a model language matrix, all six cells measured on the same 3,265 items with one construction and one seed ( per cell; 95% CIs in Appendix I ). Bold marks the highest lar per language. Every arm is far more captured in Bengali than in English, and the Bengali ordering does not follow capability: Jev-1.13, the strongest arm, is the least captured in both languages, while Lod-lille, stronger than Laya-ml on random distractors (Table 1 ), is the more captured of the two in Bengali.
| lar 95% CI | |||
|---|---|---|---|
| Bengali | English | ||
| Laya-ml | |||
| Lod-lille | |||
| Jev-1.13 | |||
Table 15: Bootstrap intervals on the lar cells of Table 14 and the paired within-model language effect. Intervals are 2,000-resample percentile intervals on the same items; is the Bengali-minus-English difference on those paired items, McNemar in all three rows.
| Laya | Lod | Jev | |
|---|---|---|---|
| Bengali lar by surface hook | |||
| hook absent ( ) | 0.716 | 0.845 | 0.376 |
| hook present ( ) | 0.742 | 0.975 | 0.247 |
| difference | |||
| Capture gap, hook-absent items only | |||
| Bengali lar | 0.716 | 0.845 | 0.376 |
Table 16: The content-word hook does not carry the language gap. Columns are Laya-ml, Lod-lille and Jev-1.13. “Hook” means the Bengali literal gloss shares at least one content word with the idiom, one of the two asymmetries most likely to inflate Bengali capture. Upper block: capture is not reliably higher when the hook is present, and on the commercial arm it is lower. Lower block: restricting both language cells to hook-absent items leaves the capture gap essentially unchanged, and larger for two of three arms. The script is released with the per-item outputs.
| Laya | Lod | |
|---|---|---|
| Bengali lar under d-lit | ||
| cue absent | 0.581 | 0.871 |
| 5,485 | 560 | |
| cue present | 0.746 | 0.909 |
| 3,391 | 328 | |
| difference | ||
Table 17: The Latin-script cue is real but does not carry the result. Laya-ml, Lod-lille. Items are split by whether the idiom’s literal gloss carries an English parenthetical, which is the only thing in a Bengali option list written in Latin script. Upper block: the cue is worth of capture on the marker scorer and little on the independent-option scorer, yet capture without it is still and against a chance level of . Lower block: the same split moves accuracy on families containing no literal option by at most , and on Laya-ml by a consistent to points, five to twelve times less than what it buys on the literal family. The commercial arm cannot be split this way, since its only Bengali literal run is the pool, where every item carries a parenthetical.
Figure 6: Accuracy by task language, grouped by model: Laya-ml (four bars), Lod-lille, Jev-1.13. Solid bars are Bengali input, hatched bars English; Laya-ml’s third and fourth bars use the English-only checkpoint, the fourth being the control. Error bars 95% bootstrap CIs, dashed line chance. Every model gains between 23 and 38 points when the paired English entry is presented instead of the Bengali idiom.
Figure 7: The semantics-free control. Per model, the bar is accuracy (95% bootstrap CI) and the colored bar is mean reported confidence ( Laya-ml, Lod-lille, Jev-1.13); the dashed line is the four-option chance level. No option carries information, so the gap between the two bars is overconfidence: , and . Jev-1.13’s condition is English.
| Model | Accuracy | 95% CI | |
|---|---|---|---|
| Laya-ml | 8,876 | 0.367 | [0.357, 0.377] |
| Lod-lille | 888 | 0.484 | [0.453, 0.518] |
| Jev-1.13 | 8,761 | 0.736 | [0.727, 0.745] |
Table 18: The common baseline across all three arms. The condition is matched across models: a Bengali idiom followed by its Bengali figurative gloss with random distractors, the same instruction string and the same . It is not byte-identical, because the commercial arm drew its sampled distractors under a disjoint seed namespace (Appendix R ), so the three option lists share the gold and differ in the three distractors. Each row is on that arm’s own items, so differs. The pairwise differences are computed on the items each pair shares, which is why they do not equal the differences between rows: Jev–Laya , Jev–Lod and Lod–Laya (McNemar, all ).
Figure 8: The matched three-way comparison. Bars left to right: Laya-ml, Lod-lille, Jev-1.13. Black dashed line is chance, error bars 95% bootstrap CIs. This capability ordering is the baseline against which the calibration failures in § 5.2 –§ 5.3 should be read.
Figure 9: Unit integrity: the idiom is perturbed, the options held fixed. Laya-ml, Lod-lille; error bars 95% bootstrap CIs, dashed line chance (left panel). Left: accuracy. Right: agreement with the answer on the intact idiom. Scrambling and reversing the expression does not meaningfully hurt either model, which is what a model reading the expression as a set of words would show.
| Family | Full | Subset | Drift |
|---|---|---|---|
| d-rand | 0.367 | 0.351 | |
| d-surf | 0.338 | 0.339 | |
| d-sem | 0.333 | 0.325 | |
| d-lit | 0.151 | 0.136 | |
| d-hard | 0.144 | 0.149 |
Table 19: Laya-ml on the 10% subset against its own full set, drift computed before rounding. Maximum drift is , and the larger drifts are downward, a slight lean of under two points. Two of five subset estimates fall marginally outside the full-set bootstrap CI, which is expected at and changes no sign or ordering. The full-set figures are therefore reported as the precise estimates, and Lod-lille’s intervals are reported wider throughout.
| Option text | Laya | Lod | Jev |
|---|---|---|---|
| full gloss | 0.365 | 0.492 | 0.970 |
| truncated | 0.361 | 0.468 | 0.958 |
| scrambled | 0.357 | 0.479 | 0.974 |
| other lang. | 0.347 | 0.398 | 0.963 |
| no semantics | 0.256 | 0.233 | 0.254 |
Table 20: Option-side ablation. The semantics-free row removes meaningful option content and yields near-chance accuracy in all three arms. Jev-1.13’s column is English.
Figure 10: Option-side ablation. Panels left to right: Laya-ml, Lod-lille, Jev-1.13. In each panel the fifth, bar is the semantics-free control of § 5.3 ; the whole Jev-1.13 panel is hatched because its task is in English . Error bars are 95% bootstrap CIs and the black dashed line is chance. Truncating and word-scrambling the option text cost almost nothing in any of the three models.
| Laya | Lod | Jev | |
|---|---|---|---|
| answer changes | 0.837 | 0.107 | 0.133 |
| correctness varies | 0.686 | 0.070 | 0.120 |
| distinct glosses | 2.83 | 1.14 | 1.15 |
| perfectly stable | 0.163 | 0.893 | 0.867 |
| (gold) range | 0.346 | 0.029 | 0.128 |
| selection spread | 0.139 | 0.030 | 0.011 |
Table 21: Option-order sensitivity. Each item is evaluated under all 24 orderings of the same four options ( ). Laya, Lod, Jev; bold marks the largest value in each row. (gold) range is the within-item range of the gold option’s probability across the 24 orderings, averaged over items; selection spread is the gap between the most and least chosen slot, pooled over orderings, and is under no positional preference. The marker-based scorer changes its chosen gloss on of items against for the independent-option scorer. The three item samples are drawn separately per arm, so cross-arm comparisons are unpaired. Lod is documented as order-independent by construction; the permutation audit below shows it is not, which is why its entries here are small rather than zero.
| Laya | Lod | Jev | |
|---|---|---|---|
| items | 992 | 299 | 300 |
| median spread | 0.495 | 0.040 | 0.070 |
| mean spread | 0.507 | 0.046 | 0.139 |
| 95th pct. spread | 0.772 | 0.105 | 0.480 |
| max spread | 0.928 | 0.245 | 0.820 |
| share | 0.000 | 0.000 | 0.230 |
Table 22: Permutation audit. Per item, the largest distance a single option’s own score travels across the 24 orderings of its list, after mapping each slot back to the option it displayed. Laya-ml, Lod-lille, Jev-1.13. Bold marks the largest value in each row. An order-independent scorer should sit at fp32 noise, around ; the share row gives the fraction of items at or below , a threshold three orders of magnitude more permissive than that. Only Jev-1.13 has any items at that threshold, and then fewer than a quarter; its probabilities are reported to two decimal places, so for that arm the row means identical as reported rather than identical in the underlying computation. The script that produces this table is released with the per-item outputs.
Figure 11: Share of selections falling on each option slot, over the full ladder condition rather than the permutation sweep. Panels left to right: Laya-ml, Lod-lille, Jev-1.13. The gold option is placed uniformly at random, so the black dashed line at is the expectation under no positional bias and any departure from it is bias. Laya-ml peaks on slot B, taking it of the time against for slot A; the other two arms sit close to the line throughout, with a mild and monotone preference for earlier slots.
Figure 12: Order sensitivity. Left: how many distinct glosses each model selects across the 24 orderings of one option set; within each group bars run Laya-ml, Lod-lille, Jev-1.13, and a value of 1 means perfect stability. Right: per model, the colored bar is the share of items whose chosen gloss is unstable and the bar is the mean within-item range of (gold). Laya-ml is unstable on 83.7% of items with a mean (gold) swing of ; the other two are near-stable.