One Framework to Diagnose Them All: A Stratified Benchmark for Bengali LLM Hallucination
Organizations: IEEE · Islamic University of Technology · North Dakota State University · Bangladesh University of Engineering and Technology ·
Abstract
Hallucination research in low-resource languages fragments easily: each newly named failure mode prompts its own dataset, spending scarce annotators on incomparable resources. We argue for consolidation and test it at community scale. Alikbachan is a Bengali hallucination-detection benchmark stratified along three crossed axes: a human-validated cultural-distance band, six task types, and a five-family, 32-mode hallucination taxonomy. It comprises a harvested 2,516-item test set (Phase 1) and a quota-constructed, gold-labeled 5,000-item set (Phase 2). The stratification is diagnostic. Four zero-shot LLM judges vary by over 30 macro-F1 points across strata, and the item-level predictions of 99 Phase-1 and 32 Phase-2 competition systems reproduce the profile: discourse-level omission defeats judges and most teams alike, whereas only a minority over-flag negative controls. The judges' verdict probabilities rank items well (ROC-AUC 0.74-0.95) but are overconfident (ECE 0.04-0.29). An open competition on Alikbachan drew 222 teams and 3,566 submissions, and a single-prompt 32B judge reaches the 57th percentile. Of the 25 teams in both phases, 24 regress on Phase 2 (mean points) while every zero-shot judge improves, and Phase-1 rank barely predicts Phase-2 rank (Spearman ), so a Phase-1 leaderboard alone would have crowned a different winner.
Figures & tables
| Layer | Common pitfall | Required curator check |
| Packaging | An image, option, table or answer-bearing part is missing from the released row | Test answerability on the released row itself, not on its source |
| Representation | OCR, Unicode or math notation admits competing readings | Keep raw and canonical forms; adjudicate only once one reading is fixed |
| Linguistic validity | The idiom, register, morphology or lexical relation is ill-formed | Judge grammaticality, lexical correctness and contextual support separately |
| Question contract | The answer has the wrong role, entity, place, time, unit or polarity | Record an answer-slot signature that binds every requested argument |
| Epistemic status | An estimate is treated as exact, or an unresolved future as fact | Preserve approximation, modality, authority and temporal scope |
| Reasoning | Topical overlap stands in for a comparison or composition | Require a licensed operation and reject the nearest plausible alternative |
| Verdicts | Verdict probabilities | |||||||
| System | Acc. | Macro-F1 | F1 | F1 | PR-AUC | PR-AUC | ROC-AUC | ECE |
| Phase-1 test set (2,516 items) | ||||||||
| Majority class | 0.531 | 0.347 | 0.000 | 0.693 | 0.531 | 0.469 | 0.500 | – |
| Qwen2.5-32B | 0.770 | 0.770 | 0.781 | 0.758 | 0.894 | 0.851 | 0.870 | 0.212 |
| Gemma-2-9B | 0.733 | 0.731 | 0.711 | 0.751 | 0.851 | 0.810 | 0.827 | 0.227 |
| Llama-3.1-8B | 0.631 | 0.557 | 0.376 | 0.738 | 0.810 | 0.761 | 0.776 | 0.285 |
| Detector family | What Alikbachan provides |
| Evidence-grounded | context (or [NULL] ) and response_bn for every item |
| Self-consistency | prompt_bn ; resampling needs a proxy model, as generators are not released |
| Supervised | label on 4,315 public items; strata for splits; rationale as extra supervision |
| Verdict confidence | Top-20 verdict log-probabilities from our judge scripts ( § E ) |
| White-box likelihood | Context, prompt and response for teacher forcing ( § G ) |
| Error analysis | Band, task, family, mode, provenance, source (Phase 2); task type (Phase 1) |
| over Maj. | ||||
| Failure mode | Fam. | Maj. | Judges | Teams |
| negative-control | A | 1.000 | 7.3% † | 26.2% † |
| discourse-level-omission | B | 0.487 | 0.274 | 0.164 |
| sociopragmatic-register | C | 0.394 | 0.400 | 0.343 |
| script-attention-traps | 0.375 | 0.428 | 0.380 | |
| reasoning-language- inconsistency | 0.393 | 0.431 | 0.414 | |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Term | Meaning in this paper |
| Faithful, hallucinated | Gold label 1: the response is correct and, given a context, supported by it. Gold label 0: it contains fabricated, unsupported or factually wrong content. |
| Context, [NULL] | The passage a response must be judged against; [NULL] marks a closed-book item judged against world knowledge. |
| Cultural band | C0 globally stable facts (the control band); C1 culturally situated facts, specific to Bangladesh or contrary to a global default; C2 contested, recent or time-sensitive facts. |
| Hallucination family | A content or parametric; B context or intrinsic; C linguistic or cultural; D adversarial or robustness; E meta or pipeline. The families divide into the 32 failure modes of Table 11 . |
| Theory-authored, theory-confirmed | A Phase-2 item written around a target failure mode and labeled by construction; or built from real source content and labeled by review against the source. |
| Evidential state | The audit’s four-way judgment before binary projection: Supported , Refuted , Not Entailed or Malformed ( § 5 ). |
| Field | Value |
| id | P2-00040 |
| source | wikipedia_banglapedia |
| task_type | short_cb_qa |
| cultural_band | C1 |
| family | A |
| failure_mode | cultural_default_substitution |
| Source corpus | Items | % | Public |
| Wikipedia and Banglapedia | 1,100 | 22 | 323 |
| Government publications | 900 | 18 | 273 |
| Bangladeshi statutes | 700 | 14 | 198 |
| National Curriculum and Textbook Board texts | 700 | 14 | 218 |
| Newspapers | 700 | 14 | 211 |
| e-Bangla Library texts | 550 | 11 | 165 |
| Family | ||||||
| Task type, band | A | B | C | D | E | Total |
| short-cb-qa | 939 | – | 289 | 212 | 160 | 1,600 |
| C0 | 105 | – | 50 | 31 | 21 | 207 |
| C1 | 560 | – | 197 | 147 | 120 | 1,024 |
| C2 | 274 | – | 42 | 34 | 19 | 369 |
| short-ob-qa | 366 | 441 | 207 | 152 | 134 | 1,300 |
| Task type | Band | |||||||||||
| Fam. | Failure mode | Quota | cb | ob | lob | sum | tr | adv | C0 | C1 | C2 | %y=1 |
| A | cultural-default-substitution | 400 | 179 | 28 | – | – | 184 | 9 | 68 | 295 | 37 | 31.5 |
| temporal-confusion | 350 | 258 | 29 | – | 53 | 2 | 8 | – | 102 | 248 | 46.6 | |
| entity-substitution | 300 | 224 | 23 | – | 45 | – | 8 | 39 | 221 | 40 | 26.0 | |
| numerical-precision-drift | 250 | 103 | 102 | 45 | – | – | – | 45 | 180 | 25 | 31.2 | |
| negative-control | 200 | – | – | – | – | – | 200 | 35 | 145 | 20 | 100.0 | |
| Test set | Sample set | Macro-F1 on the test set | |||||||
| Stratum | % | % | Qwen2.5 | Gemma-2 | Llama-3.1 | TigerLLM | Maj. | ||
| Task type (exact, from the context field) | |||||||||
| Closed-book QA | 1,155 | 45.9 | 169 | 56.5 | 0.624 | 0.586 | 0.389 | 0.592 | 0.335 |
| Context-grounded QA | 1,361 | 54.1 | 130 | 43.5 | 0.880 | 0.854 | 0.687 | 0.858 | 0.359 |
| Cultural-distance band (reconstructed proxy) | |||||||||
| C0 – globally stable | 2,014 | 80.0 | 243 | 81.3 | 0.739 | 0.720 | 0.504 | 0.775 | 0.345 |
| Fam. | Failure mode | Items | % faith. | Definition |
| A | cultural-default-substitution | 400 | 31.5 | A globally dominant default replaces the Bangladesh-specific correct answer. |
| temporal-confusion | 350 | 46.6 | Wrong dates or years, or events put in the wrong order. | |
| entity-substitution | 300 | 26.0 | A plausible answer that names the wrong person, place or organization. | |
| numerical-precision-drift | 250 | 31.2 | Numbers, statistics or measurements altered or wrongly rounded. | |
| negative-control | 200 | 100.0 | Unambiguously faithful items (all label 1). Health check for over-flagging. | |
| wrong-attribution | 200 | 26.0 | A work, quote or achievement credited to the wrong author or actor. |
| Pitfall | What makes the answer look plausible | Why the judgment fails or needs review |
| Topical but non-answer-bearing context | The passage discusses the same word, person or event | The requested relation is never stated or derivable; the contextual state is Not Entailed |
| Role substitution | Both candidate people and both roles occur nearby | Father and mother, creator and adopter, author and pseudonym, ruler and saint, or employer and employee are different relations |
| Jurisdiction transfer | The same event type and date occur in the passage | Evidence about one country, institution or polity cannot answer a differently scoped question |
| Near-number or near-date selection | The response lies inside a stated range or resembles a visible value | Range membership does not prove a first year; a nearby digit is not an acceptable temporal or numerical substitute |
| Answer-slot mismatch | The response is factually associated with the subject | Temporal, locative, agentive and quantitative questions require different semantic answer types before factual accuracy can be assessed |
| False multi-span composition | Every token of the answer appears somewhere in the context | Combining a birthplace with a later office, or a person with another entity’s date, creates a relation the passage never asserts |
| Dataset condition | Diagnostic question | Required treatment |
| Missing visual or table | Can the question be solved without the referenced balance, portrait, diagram, chart or marked position? | Mark Malformed until the original modality is packaged; a remembered answer or one-letter key is not evidence |
| Incomplete multiple choice | Does the stem presuppose options that are absent? | Restore all alternatives, or rewrite and independently adjudicate the item as open-ended |
| Weak or fragmentary stem | Can several completions satisfy the surviving phrase? | Recover the intended proposition or record the item as ambiguous; do not infer the missing question from the response |
| Mathematical OCR | Do superscripts, radicals, fractions, signs or token boundaries admit more than one expression tree? | Store the raw OCR, rendered equation, canonical expression tree and accepted derivation before labeling |
| Invalid quantitative premise | Does a visible value make the ordinary operation undefined or impossible? | Do not manufacture a solvable problem by averaging or silently repairing the premise |
| False or underspecified premise | Must the respondent accept a claim that is disputed, temporally unstable or undefined? | Separate premise diagnosis from answer diagnosis, and specify the authority or hypothetical world |
| Pitfall | Dataset symptom | Curation decision |
| Copied-key consensus | Several search results repeat the same short answer | Trace independence and authority; repetition is not corroboration |
| Snapshot conflict | An older passage and a newer response give different values | Declare whether the task evaluates passage faithfulness or current-world correctness |
| Stale category or count | An undated question has different historical and current answers | Specify authority, taxonomy, jurisdiction and as_of ; do not merely replace one number |
| Legal or policy drift | Publication, commencement, amendment and survey reference dates differ | Preserve the document version, effective interval, reference period and access date |
| Approximation loss | The source says ‘about’ but the response supplies unsupported exact digits | Retain the level of uncertainty; extra precision is an additional factual claim |
| Future claim | The passage says a decision or projection is unsettled | Faithful behavior is calibrated uncertainty, not the fabrication of a definite outcome |
| Phenomenon | Common curation error | Required distinction |
| Requested lexical relation | A related word or usage sentence is accepted as the requested antonym, meaning, prefix class or derivation | Make the lexical operation binding; topical relevance is insufficient |
| Idiom semantic type | The idiom appears, but a phrase denoting an object, relation or event is assigned as a personal property | Judge sentence well-formedness, contextual entailment and dictionary meaning independently |
| ( samasa , compounding) analysis | A memorized class is accepted despite an invalid or contradictory ( byasbakya , the paraphrase that expands a compound) | Require both the semantic relation and a valid expansion |
| ( guruchandali , mixed registers) | Mixed ( sadhu , formal) and ( cholito , colloquial) forms are accepted merely because the sentence is understandable | Enforce register consistency when standard Bengali is requested; tag intentional dialogue separately |
| Morphosyntax | Classifier, agreement, honorific, case or the scope of negation is ignored | Preserve the grammatical features that change participant, quantity, politeness or truth conditions |
| Unicode normalization | Differences in canonical storage are treated as spelling differences, or a meaning-changing near-homograph is silently merged | Preserve raw and canonical strings, and record whether normalization changes meaning |
| System | Acc. | Macro-F1 | F1 | F1 |
| Phase-1 test set (2,516 items) | ||||
| Majority class | 0.531 | 0.347 | 0.000 | 0.693 |
| TituLLM-1B | 0.524 | 0.415 | 0.162 | 0.668 |
| Phase 2 (5,000 items) | ||||
| Majority class | 0.543 | 0.352 | 0.704 | 0.000 |
| TituLLM-1B | 0.467 | 0.400 | 0.201 | 0.600 |
| With context | Without context | |||
| Model | Cohen’s | AUROC | Cohen’s | AUROC |
| Phase-1 test set 1,361 items with context, 1,155 without | ||||
| Qwen2.5-32B | 0.489 [0.37, 0.62] | 0.692 [0.66, 0.72] | 0.051 [ 0.05, 0.17] | 0.520 [0.48, 0.55] |
| Gemma-2-9B | 0.755 [0.65, 0.87] | 0.775 [0.75, 0.80] | 0.063 [ 0.06, 0.18] | 0.521 [0.49, 0.55] |
| Llama-3.1-8B | 0.645 [0.53, 0.78] | 0.773 [0.75, 0.79] | 0.052 [ 0.06, 0.15] | 0.500 [0.47, 0.53] |
| Gemma-4-12B † | 0.031 [ 0.08, 0.13] | 0.505 [0.47, 0.53] | 0.066 [ 0.04, 0.18] | 0.517 [0.49, 0.55] |
| Fam. | Failure mode | Maj. | Qwen2.5 | Gemma-2 | Llama-3.1 | TigerLLM | |
| A | cultural-default-substitution | 400 | 0.407 | 0.858 | 0.764 | 0.529 | 0.820 |
| temporal-confusion | 350 | 0.348 | 0.612 | 0.595 | 0.591 | 0.681 | |
| entity-substitution | 300 | 0.425 | 0.859 | 0.676 | 0.500 | 0.822 | |
| numerical-precision-drift | 250 | 0.408 | 0.900 | 0.830 | 0.687 | 0.824 | |
| negative-control | 200 | 1.000 | 0.465 | 0.481 | 0.497 | 0.481 | |
| wrong-attribution | 200 | 0.425 | 0.918 | 0.821 | 0.681 | 0.880 |
| Qwen2.5 | Gemma-2 | Llama-3.1 | TigerLLM | ||||||
| Fam. | Failure mode | FF | Miss | FF | Miss | FF | Miss | FF | Miss |
| A | cultural-default-substitution | 12.7 | 12.8 | 4.0 | 31.8 | 1.6 | 67.5 | 1.6 | 24.5 |
| temporal-confusion | 52.8 | 24.1 | 43.6 | 37.4 | 4.9 | 66.8 | 13.5 | 47.1 | |
| entity-substitution | 19.2 | 8.1 | 20.5 | 32.9 | 1.3 | 67.1 | 9.0 | 17.6 | |
| numerical-precision-drift | 9.0 | 8.7 | 3.8 | 21.5 | 0.0 | 45.3 | 1.3 | 23.8 | |
| negative-control | 13.0 | – | 7.5 | – | 1.0 | – | 7.5 | – | |
| Best macro-F1 | Phase-1 teams | Phase-2 teams |
| 18 | 6 | |
| – | 53 | 2 |
| – | 77 | 5 |
| – | 42 | 12 |
| 32 | 7 | |
| Total teams | 222 | 32 |
| Team | Phase 1 | Phase 2 | Change |
| ExtremeAI | 0.983 | 0.812 | 0.172 |
| Huntrix | 0.983 | 0.901 | 0.081 |
| BUET_NeuralX | 0.962 | 0.954 | 0.008 |
| IUFree | 0.959 | 0.891 | 0.068 |
| Ain’t No Way | 0.959 | 0.381 | 0.578 |
| Bitwiz | 0.936 | 0.921 | 0.015 |
Hallucination Detection
Hallucination detection identifies unsupported, fabricated, or evidence-inconsistent content in language and multimodal model outputs.
See all 84 papers