cs.CLNov 19, 2026

One Framework to Diagnose Them All: A Stratified Benchmark for Bengali LLM Hallucination

Authors: Asif Shahriyar Sushmit, Shefayat E Shams Adib, Abdullah Al Jubaer Gem, Ahmed Shafin Ruhan, MD. Nazmuddoha Ansary, Biplab Kumar Sarkar, Syed Rifat Raiyan, Ajwad Abrar, 

Organizations: IEEE · Islamic University of Technology · North Dakota State University · Bangladesh University of Engineering and Technology · 

Abstract

Hallucination research in low-resource languages fragments easily: each newly named failure mode prompts its own dataset, spending scarce annotators on incomparable resources. We argue for consolidation and test it at community scale. Alikbachan is a Bengali hallucination-detection benchmark stratified along three crossed axes: a human-validated cultural-distance band, six task types, and a five-family, 32-mode hallucination taxonomy. It comprises a harvested 2,516-item test set (Phase 1) and a quota-constructed, gold-labeled 5,000-item set (Phase 2). The stratification is diagnostic. Four zero-shot LLM judges vary by over 30 macro-F1 points across strata, and the item-level predictions of 99 Phase-1 and 32 Phase-2 competition systems reproduce the profile: discourse-level omission defeats judges and most teams alike, whereas only a minority over-flag negative controls. The judges' verdict probabilities rank items well (ROC-AUC 0.74-0.95) but are overconfident (ECE 0.04-0.29). An open competition on Alikbachan drew 222 teams and 3,566 submissions, and a single-prompt 32B judge reaches the 57th percentile. Of the 25 teams in both phases, 24 regress on Phase 2 (mean −17.6-17.6 points) while every zero-shot judge improves, and Phase-1 rank barely predicts Phase-2 rank (Spearman ρ=0.35ρ=0.35), so a Phase-1 leaderboard alone would have crowned a different winner.

Figures & tables

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali

    May 29, 2026Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +3LLM ReliabilityMultilingual Language Model Evaluation

  2. HalluScore: Large Language Model Hallucination Question Answering Benchmark

    May 16, 2026Aisha Alansari, Hamzah LuqmanMultilingual Language Model EvaluationLLM Evaluation