Multilingual Benchmark

Ask about this topic

Momentum

19 papers in the last four weeks, up 73% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 151

All topics
CardsList

  1. BabelFake: A Multilingual Audio-Visual DeepFake Benchmark

    Oct 5, 2026Carlotta Segna, Joel Tschesche, Anna RohrbachMultilingual BenchmarkMultilingual Automatic Speech Recognition

    TU Darmstadt & Hessian.AI, Germany

  2. SHAMS: An Audio-Grounded Pronunciation Benchmark for Levantine Arabic

    Oct 1, 2026Ben Sapirstein, Roy Mattar, Guy Mor-Lan +3Arabic Natural Language ProcessingArabic

    Reichman University · Independent Researcher · The Hebrew University of Jerusalem +2

  3. Precision over Scale: A Polish-Silesian Benchmark and a Translation System Outperforming Open-Source and Commercial Models

    Oct 1, 2026Grzegorz Kulik, Mikołaj Pokrywka, Adam Jatowt +1Multilingual BenchmarkPrecision

    NASK National Research Institute, Poland · University of Innsbruck, Austria

  4. Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning

    Sep 30, 2026Tyler Skow, Shravan Chaudhari, Rama Chellappa +1Large Language Model UnlearningMultilingual Benchmark

    Johns Hopkins University

  5. Billiger.de Products: A Bilingual Entity Matching Benchmark

    Sep 29, 2026Aaron Steiner, Ksenia Elagin, Ralph Peeters +2Multilingual Benchmark

    University of Mannheim, Data and Web Science Group, B6, 26, 68159 Mannheim, Germany · solute GmbH, Zeppelinstraße 15, 76185 Karlsruhe, Germany

  6. Benchmarking Automatic Speech Recognition Tools for Iberian Languages

    Sep 29, 2026Fernando López, Pablo Gómez, David Solans +2Automatic Speech Recognition EvaluationMultilingual Automatic Speech Recognition

    Telefónica Innovación Digital, Spain · Universidad Autónoma de Madrid, Spain

  7. Almieyar: A Culturally Grounded Benchmark for Multi-Dialect Arabic Speech Recognition

    Sep 28, 2026Omid Ghahroodi, Anas Madkoor, Dima Faris Al Saudi +37Arabic Natural Language ProcessingArabic

    QCRI, HBKU · Qatar University · UDST +15

  8. BaatCheet: A Multilingual Corpus for Dialogue Translation in Indian Languages

    Sep 27, 2026Priyanka Dasari, Yuvrajsinh D. Bodana, Vandan Mujadia +3Translation QualityMultilingual Benchmark

    Language Technologies Research Centre, IIIT Hyderabad, India

  9. Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure

    Sep 24, 2026Fardeen Sadab, Adib SakhawatMultilingual BenchmarkHuman Annotators

    Department of Computer Science and Engineering Islamic University of Technology, Dhaka, Bangladesh

  10. MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents

    Sep 21, 2026Chenxu Xiong, Dongming Shen, Yuzhi Tang +3Full-Duplex Voice AgentsDialogue Benchmarks

    Boson AI Santa Clara, CA 95054

  11. Vimarsha: Faithful ASR Evaluation for Indian Languages with Demographic Diversity, In-the-Wild Audio and Spelling Variations

    Sep 21, 2026Kaushal Santosh Bhogale, Srija Anand, Sadakopa Ramakrishnan Thothathiri +3Automatic Speech Recognition EvaluationIndian Languages

    Indian Institute of Technology, Madras, India · Sarvam AI, India

  12. BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents

    Sep 20, 2026Peng Kuang, Yuchun Fan, Jiangnan Li +7Multilingual AgentsMultilingual Benchmark

    Alibaba Token Hub, Alibaba Group · University of Illinois Urbana-Champaign · Northeastern University +2

  13. MuLA-Bench: A Multilingual Long-Form Audio Understanding Benchmark via Multi-Tier Auditing

    Sep 20, 2026Zeyu Yang, Xinyu Zhang, Zibo Bi +5Multilingual BenchmarkAcoustic Latent Space

    The Chinese University of Hong Kong, Shenzhen · Alibaba Token Hub, Alibaba Group · Xi’an Jiaotong University

  14. TeochewBench: A Human-Reviewed Benchmark for Teochew Hanzi Translation

    Sep 16, 2026Jianan WuMachine Translation QualityMultilingual Benchmark

    Peking University

  15. Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech

    Sep 12, 2026Chibuzor Okocha, Christan Earl GrantCode-SwitchingAutomatic Speech Recognition Evaluation

    Department of Computer & Information Science & Engineering University of Florida, Gainesville, FL, USA

  16. Prompt Revision as a Source of Cultural Bias in Text-to-Image Systems

    Sep 12, 2026Aleksandra Urman, Elsa Lichtenegger, Salima Jaoua +7Stereotype MitigationText Generation

    University of Zurich · University of Helsinki · ETH Zurich +1

  17. 5-Dialects-BN: Unmasking the Impact of Transliteration on Bangla Dialectal LLMs

    Sep 9, 2026Md Mahir Jawad, Galib Mahmud Jim, Rafid Ahmed +3DialectsBangla

    BRAC University · Penta Global Limited · University of Central Florida +1

  18. Improving Cross-Lingual Token Representations by Adding a Pinch of SALT

    Sep 9, 2026Guillem RamírezCross-EncoderMotion-Language Alignment

    ILCC, University of Edinburgh

  19. SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia

    Sep 9, 2026Jingyi Liao, Wenyu Zhang, Zhuohan Liu +6Multilingual BenchmarkThai

    Institute of Advanced Intelligence and Computing, A*STAR · Nanyang Technological University · Center for AI Safety +1

  20. BanglaMemeX: Advancing Cultural Metaphoric Image Interpretation in Bangla with a Multimodal Explainable Dataset

    Sep 7, 2026Md. Sadman Sakib, Zisan Mahmud, Md. Fahim Arefin +1BanglaMetaphors

    University of Dhaka · University of Texas at Dallas

  21. Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks

    Sep 3, 2026Oline Ranum, Edward Fish, Simon Hadfield +1Gloss-Free Sign Language TranslationSign Language Translation

    Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey

  22. MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models

    Sep 1, 2026Tawsif Tashwar Dipto, Mehedi Ahamed, Radib Bin Kabir +5MemesCultural Awareness

    1Islamic University of Technology · 2South East University · 3Vector Institute +3

  23. EDRAC: Benchmarking Arabic Dialect Reading Comprehension

    Sep 1, 2026Noor Abo Mokh, Kirill Chirkunov, Teresa Lynn +15Arabic Natural Language ProcessingArabic

    Mohamed bin Zayed University of Artificial Intelligence · New York University Abu Dhabi · IBM Research AI +1

  24. WorldBench: Culturally Grounded Benchmark for Multilingual Agents

    Sep 1, 2026Leonardo Ranaldi, Sherrie Shen, Jushi Kai +1Multilingual AgentsAgentic Benchmarks

    (•) ILCC, School of Informatics, University of Edinburgh · (•,◦) ILCC, School of Informatics, University of Edinburgh · (◦) School of Artificial Intelligence, Shanghai Jiao Tong University

  25. Latent Mechanisms of Language Control in Multilingual Language Models

    Aug 31, 2026Ryo Mitsuhashi, Sabri Boughorbel, Majd HawaslyMultilingual Language ModelsCode-Switching

    Princeton University New Jersey, USA · Prince Sattam bin Abdulaziz University Riyadh, Saudi Arabia · QCRI, Hamad Bin Khalifa University, Doha, Qatar

  26. Evaluating and Mitigating Anti-LGBTQ Biases in German and Multilingual Language Models

    Aug 31, 2026Melina Morch, Daniel BraunLarge Language Model BiasMultilingual Benchmark

    Marburg University · Computer Science · Cultural Data Studies +2

  27. BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian

    Aug 13, 2026Jophin John, Michael Hoffmann, Jan Fillies +2Multilingual BenchmarkDialects

    Leibniz Supercomputing Centre (LRZ), Garching, Germany · Stanford University, Stanford, California, USA · Freie Universität Berlin, Germany +2

  28. VICBench: A Multi-Language Benchmark for Code Vulnerability Detection

    Aug 12, 2026Jin Lu, Xuening Han, Yang Zhong +4Common Vulnerabilities And ExposureBenchmark Datasets

    Amazon Web Services · University of Pittsburgh · Purdue University

  29. From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models

    Aug 11, 2026Si'an Xie, Jiaxun Liu, Biao Yang +4Large Language Model BenchmarksReasoning Skills

    Beijing University of Posts and Telecommunications · Peking University · Kuaishou Technology

  30. VoxSumm: A Multilingual Corpus of Long-Form Spoken News for Joint Summarization and Translation

    Aug 11, 2026Yejin Jeon, Marie Maltais, Virginia Ceccatelli +2SummarizationMultilingual Benchmark

    Mila - Quebec AI Institute · McGill University, Canada · Google DeepMind +1

  31. Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness

    Aug 10, 2026Pinzhen Chen, Koel Dutta Chowdhury, Xiaoya Xu +20Multilingual BenchmarkContrastive Learning

    Queen’s University Belfast · University of Technology Nuremberg · Saarland University +12

  32. SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs

    Aug 8, 2026Debopriyo Banerjee, Kapil Rajesh Kavitha, Angana Borah +11Multilingual SafetyLarge Language Model Safety

    Inception · Mohamed bin Zayed University of Artificial Intelligence · University of Michigan +4

  33. Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluation

    Aug 7, 2026Anna Kołos, Grzegorz Statkiewicz, Karolina Seweryn +3Multilingual BenchmarkMultimodal Benchmarks

    NASK National Research Institute, Warsaw, Poland

  34. MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning

    Aug 4, 2026Martin Böckling, Elizaveta Nosova, Heiko Paulheim +1Geospatial ReasoningMultilingual Benchmark

    Data and Web Science Group, University of Mannheim, Germany

  35. M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

    Aug 4, 2026Tomáš Burkert, Angelika Peljak-Łapińska, David ZelenýMultilingual BenchmarkMultilingual Language Models

    RWS TrainAI

  36. ConlangBench: Exploring Language Knowledge and Learning in LLMs through Diverse Constructed Languages

    Aug 4, 2026Jinhong Jeong, Seungyeop Yi, Sangah Lee +1Multilingual BenchmarkLarge Language Model Benchmarks

    1Yonsei University · 2Seoul National University

  37. M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Models

    Jul 31, 2026Ryo Fukuda, Atsushi Ando, Hiroki Kanagawa +4Full-Duplex Speech ModelsTurn-Taking

    NTT, Inc., Kyoto, Japan

  38. Indic DiarBench: A Multilingual Joint Diarization and ASR Benchmark for Indian Languages

    Jul 26, 2026Deovrat Mehendale, Aditya Mehndiratta, Dhruv Rathi +2Speaker DiarizationMultilingual Automatic Speech Recognition

    Sarvam AI, India · AI4Bharat, IIT Madras, India

  39. PatiGonit22K: A Comprehensive Dataset for Solving Complex Bengali MWPs

    Jul 24, 2026Swastika Kundu, Azizul Hakim Fayaz, Tashreef MuhammadBanglaMath Problems

    Department of Computer Science and Engineering Ahsanullah University of Science and Technology, Dhaka, Bangladesh · Department of Computer Science and Engineering Southeast University, Dhaka, Bangladesh

  40. When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs

    Jul 23, 2026Anna Mosolova, Djamé SeddahLarge Language Model BenchmarksMultilingual Benchmark

    INRIA Paris

  41. Overview of FinMMEval 2026 Task 1: Multilingual Financial Multiple-Choice Question Answering

    Jul 22, 2026Zhuohan Xie, Yuyang Dai, Rania Elbadry +18Multilingual BenchmarkFinancial Question Answering

    Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria · University of Arizona, Tucson, United States +6

  42. ESCUCHA: A Spanish Speech Benchmark for Heterogeneous Acoustic Conditions

    Jul 20, 2026Fernando López, Ana Ayala, Guillermo Segovia +4Automatic Speech Recognition EvaluationLarge Audio Language Models

    Telefónica Innovación Digital · Universidad Autónoma de Madrid · Universidad Complutense de Madrid

  43. KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding

    Jul 19, 2026Timur Turatali, Aida Turdubaeva, Rustem Izmailov +2Multilingual Benchmark

    1The Cramer Project · Institute of IT, Kyrgyz State Technical University named after I. Razzakov · School of Computer Science, University of Windsor +2

  44. Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

    Jul 16, 2026Saima Afrin, Alessandro Midolo, Camilo Escobar-Velásquez +5Code GenerationMultilingual Benchmark

    William & Mary, VA, USA · University of Catania, Italy · Universidad de los Andes, Colombia +2

  45. Meta-Learning Preferences for Multilingual LLM Alignment

    Jul 14, 2026Jiaying Lin, Seongho Son, Nam Phuong Tran +3Large Language Model AlignmentMotion-Language Alignment

    Department of Computer Science, University of Warwick, Coventry, United Kingdom · Department of Computer Science, University College London, London, United Kingdom · Department of Mathematics and Computer Science, University of Basel, Basel, Switzerland

  46. Beyond Benchmarks: Exposing the Hidden Crisis in Bangla Hate Speech Detection

    Jul 13, 2026Faria Afrin Tisha, Fariya Tabassum, Hafsa Binte Kibria +2Hate Speech DetectionBangla

    Department of Electrical and Computer Engineering, Rajshahi University of Engineering and Technology, Rajshahi, 6204, Bangladesh. · Department of Computer Science, University of York, Deramore Lane, Heslington, York YO10 5GH, UK.

  47. Do LLMs Fabricate Legal Citations? A Bilingual Benchmark on Saudi Data Protection Law and the GDPR

    Jul 13, 2026Noura Suliman AlrajehLegal DomainCitations

    IT Department, King Abdulaziz University, Jeddah, Saudi Arabia

  48. Test-Time Scaling for Small VLMs on Multilingual Visual MCQ

    Jul 10, 2026Spiros Baxevanakis, Peng-Jian YangTest-Time ScalingMultilingual Benchmark

    University of Amsterdam, Science Park 904, 1098 XH Amsterdam, The Netherlands

  49. Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration

    Jul 8, 2026Weicheng Ma, John Guerrerio, Soroush VosoughiStereotype MitigationMultilingual Benchmark

    College of Computing, Georgia Institute of Technology · Computer Science Department, Dartmouth College

  50. Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability

    Jul 7, 2026Alicia Parrish, Rajat Shinde, Sanket Badhe +57Multilingual SafetyMultilingual Benchmark

  51. PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages

    Jul 7, 2026Daryna Dementieva, Nikolay Babakov, Kathy Hämmerl +14Multilingual BenchmarkTable Reasoning Datasets

    Technical University of Munich (TUM) · Munich Center for Machine Learning (MCML) · Independent Researcher +7

  52. MORE: A Multilingual Document Parsing Benchmark and Evaluation

    Jul 3, 2026Long Xu, Binghong Wu, Tinghao Yu +6Multilingual BenchmarkMultilingual

    Tencent, Shenzhen, China

  53. MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark

    Jul 1, 2026Xianru Chen, Yukai Huang, Mingxiang Chen +6Multilingual BenchmarkMultilingual

    ByteDance Seed · Nanjing University · M-A-P

  54. Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions

    Jun 29, 2026Avisha Das, Mihir Parmar, Mohana Ramnath +1Multilingual BenchmarkMultilingual

    Shiv Nadar University Chennai · Google Cloud AI Research · IIT Madras

  55. EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbots

    Jun 29, 2026Camilo Chacón SartoriEmotional Support Conversation TaskEmpathy

    MindSurf

  56. BackTranslation2.0 -- A Linguistically Motivated Metric to Assess Sign Language Production

    Jun 27, 2026Oliver Cory, Maksym Ivashechkin, Karahan Sahin +6Sign Language TranslationMultilingual Benchmark

    Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK

  57. Multilingual Reasoning Cascades Need More Context

    Jun 25, 2026Arnav Mazumder, Dengjia Zhang, Shuyue Stella Li +2Multilingual Benchmark

    University of Washington · Johns Hopkins University

  58. RedVox: Safety and Fairness Gaps in Speech Models Across Languages

    Jun 25, 2026Beatrice Savoldi, Sara Papi, Wafa Aissa +2Multilingual SafetyLarge Language Model Safety

    Fondazione Bruno Kessler, Italy

  59. Multilingual Hematology Visual Question Answering Dataset

    Jun 24, 2026Hajra Malik, Hafiza Tooba Aftab, Abdul Rehman +2Multilingual Healthcare Q&AUrdu

    Information Technology University, Lahore · King Edward Medical University, Lahore

  60. Cross-Lingual Exploration for Parametric Knowledge

    Jun 23, 2026Elisha Diskind, Itamar Trainin, Uri Shaham +3Cross-Lingual ConsistencyCross-Lingual Transfer

    The Hebrew University of Jerusalem · Google Research · MIT, IBM-MIT Watson AI Lab