Multimodal Model Evaluation

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 48% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 496

All topics

KhatianDoc: A Human-Verified Benchmark Diagnosing Multimodal LLM Failure on Bengali Legal Land Records
Sep 3, 2026Tasmiad Hasan, Arafat Zaman Ratul, Sarker Sadman Saalim +3Legal Document Analysis Legal QA
North South University, Dhaka, Bangladesh
CulturalMenuBench: Probing the Knowledge-Application Gap in Multimodal Culinary Reasoning
Sep 3, 2026Bo Zeng, Linfeng Gao, Peiqin Lin +9Multimodal Reasoning Multilingual VLM Evaluation
Alibaba Group · Xiamen University · University of Hamburg
MedQA-MM: Shortcuts Behind Medical Visual Reasoning
Sep 3, 2026Benlu Wang, Yifan Zhang, Jiaqing Yu +7Medical VQA Shortcut Learning
University of Massachusetts Amherst · 4Yale University · University of Massachusetts Lowell +6
Deeply Interleaved Text-Image Contexts for Multimodal LLMs Assessment
Sep 2, 2026Zihao Wang, Xi Xiang, Yuwen Sun +5Vision-Language Models Multimodal Model Evaluation
1Harbin Institute of Technology · 2Huawei Noah’s Ark Lab
DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents
Sep 2, 2026Zhuoran Yu, Le Thien Phuc Nguyen, Jaden Park +5Multimodal Large Language Models Chart QA
University of Wisconsin-Madison · Massachusetts Institute of Technology · Stanford University +2
Does Playing it Safe Count as Faithfulness? Reassessing LVLM Hallucination Mitigation Methods
Sep 1, 2026Mehrdad Fazli, Sina Mansouri, Mohit Marvania +1VLM Evaluation Faithfulness in VLMs
Department of Computer Science, George Mason University Fairfax, VA 22030, USA
ExBind: A Controlled Diagnostic Benchmark for Visual-to-Executable Correspondence
Sep 1, 2026Ziqian Wang, Yuxiao Cheng, Tingxiong Xiao +1Multimodal Grounding Multimodal Model Evaluation
Department of Automation, Tsinghua University Beijing 100084, China
A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation
Sep 1, 2026Hodong Lee, Sanghee Park, Dohoon Ryu +4ML Reproducibility Multimodal Foundation Models
NAVER Cloud AI · Korea University · KAIST AI +1
SciTrue: Reliable Scientific Claim Validation with Frontier and Open Language Models at the NTCIR SciClaimEval Task
Sep 1, 2026Qiming Bao, Neşet Özkan Tan, Siyuan Wang +1Document Understanding LLM Evaluation
University of Auckland · New Zealand · The Chinese University of Hong Kong +1
Distributed Implicit Harm: A Compositional Safety Blind Spot in MLLM-Based Video Moderation
Aug 31, 2026Ruotong Wang, Zihao Zhu, Siwei Lyu +2Multimodal Large Language Models Video Content Moderation
The Chinese University of Hong Kong, Shenzhen · Kling Team, Kuaishou Technology · State University of New York at Buffalo
MMDS-Bench: Benchmarking Multimodal Large Language Models on Dynamic Stance in Social Media Interactions
Aug 31, 2026Yuzhe Ding, Kang He, Li Zheng +5Multimodal Large Language Models Multimodal Reasoning
School of Cyber Science and Engineering, Wuhan University · Shanghai Innovation Institute
SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment
Aug 31, 2026Chaohui Dang, Zheheng Jiang, James Glasbey +3Surgical Video Understanding Multimodal Model Evaluation
Department of Electronic, Electrical and Systems Engineering, School of Engineering, University of Birmingham, Birmingham, UK · School of Computing and Mathematical Science, University of Leicester, Leicester, UK · Department of Applied Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK +1
OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding
Aug 31, 2026Gengxu Li, Yuan Wu, Yi ChangVisual Reasoning Multimodal Reasoning
School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · 3International Center of Future Science, Jilin University
Fine-Grained Multi Image Object Hallucination Benchmark
Aug 31, 2026Joonki Min, Chaeyun Kim, Hyungwook Choi +4VLM Evaluation Object Hallucination in VLMs
Seoul National University · AIM Intelligence
ImageEval 2026: Culturally Grounded Arabic Multimodal Evaluation
Aug 31, 2026Samir Abdaljalil, Hunzalah Hassan Bhatti, Ahlam Bashiti +11Hallucination Detection T2I Generation
Texas A&M University · Qatar Computing Research Institute, Qatar · Birzeit University, Palestine +2
Foundation and Multimodal Large Language Models for Face Presentation and Morph Attack Detection
Aug 30, 2026Hatef Otroshi Shahreza, Asif Hussain Khan, Peter Lorenz +2Image Forgery Detection Vision Foundation Model Adaptation
Idiap Research Institute, Switzerland · Université de Lausanne, Switzerland
DVBench: Benchmarking MLLMs for Understanding Dynamic Charts and Narratives in Data Videos
Aug 30, 2026Bomiao Wang, Zekai Shao, Jiexiang Lan +3Video Understanding Video-Language Model Evaluation
Fudan University · The Hong Kong University of Science and Technology (Guangzhou)
Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning
Aug 29, 2026Zhaolu Kang, Meixin Wu, Yu Xue +6Multimodal Robustness Multimodal Large Language Models
Tencent · Peking University · Zhejiang University
WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents
Aug 28, 2026Zongkai Liu, Hui Zhang, Liqiang Niu +7Multimodal IR Multimodal Search Agents
Weixin AI, Tencent · Sun Yat-sen University
Edit2TikZ: A Comprehensive and Challenging Benchmark for Scientific Figure Editing with TikZ
Aug 13, 2026Zongyun Zhang, Jiacheng Ruan, Xian Gao +5Text-Guided Image Editing Code Generation
Shanghai Jiao Tong University
Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs
Aug 13, 2026Xinming Wang, Weinong Wang, Hongming Yang +13RL for Language Models LLM Alignment
Institute of Automation, Chinese Academy of Sciences · Tencent · Zhongguancun Academy +3
Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
Aug 12, 2026Weihao Bo, Shan Zhang, Yanpeng Sun +7Code Generation Multimodal QA
Nanjing University of Science and Technology · Baidu Inc · AIML, Adelaide University +4
Do You See What You Draw? A Semantic Closed-Loop Framework for Holistic Evaluation of Unified Multimodal Models
Aug 12, 2026Hao Zhang, Jiaxin Qi, Zhijiang Tang +1Unified Multimodal Models Multimodal Reasoning
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Computer Network Information Center, Chinese Academy of Sciences
Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System
Aug 12, 2026Haoyu Zhang, Shuoxun Zhang, Peng Ye +5Visual Question Answering Multi-Agent LLM Systems
Embedded Deep Learning and Visual Analysis Laboratory, College of Future Information Technology, Fudan University, Shanghai 200433, China · Embedded Deep Learning and Visual Analysis Laboratory, College of Future Information Technology, Fudan University, Shanghai 200433, China, and also with Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China, and also with The Chinese University of Hong Kong, Hong Kong · College of Future Information Technology, Fudan University, Shanghai 200433, China +1
MBA: Multimodal Benchmark and Agents for Real-World Business Ideation
Aug 12, 2026Hojun Choi, Jaeyo Shin, Suin Lee +1AI Agent Benchmarks Computational Creativity
KAIST AI, Republic of Korea
PRMU: A Corpus-Free Benchmark for Person-Centric Knowledge Unlearning in Multimodal Large Language Models
Aug 11, 2026Huafeng Chen, Yueming Lyu, Ziyuan Chen +4MLLM Unlearning Machine Unlearning
School of Intelligence Science and Technology, Nanjing University · Department of Computing, Imperial College London
Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenes
Aug 11, 2026Zhaoyang Wei, Bowen Jiang, Xumeng Han +6Multimodal Grounding Multimodal Robustness
University of Chinese Academy of Sciences (UCAS), Beijing, 101480, China. · Tencent CDG, Shenzhen, 518054, China. · Institute of Information Engineering, CAS, Beijing, 100085, China.
E $^3$ mo-Bench: A Scalable Benchmark for Multimodal Evoked and Expressed Emotion Understanding via Bayesian Pairwise Alignment
Aug 11, 2026Lancheng Gao, Ziheng Jia, Shengyan Li +4Multimodal Large Language Models Multimodal Emotion Recognition
Shanghai Jiao Tong University
CircuitReason-1k: Benchmarking Long-Horizon Visual-to-Symbolic Reasoning inElectrical Circuits
Aug 10, 2026Xinqi Yang, Kang An, Tengyue Wang +8Structured Reasoning Visually Grounded Reasoning
Shanghai Jiao Tong University · South China University of Technology · ModelBest +6
MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence
Aug 10, 2026Chenxu Du, Kang An, Tengyue Wang +8Multimodal Reasoning Visually Grounded Reasoning
Southwest Jiaotong University · Shanghai Jiao Tong University · South China University of Technology +7

Sep 3, 2026, cs.CLJ/K move · Enter open · S save

KhatianDoc: A Human-Verified Benchmark Diagnosing Multimodal LLM Failure on Bengali Legal Land Records

Tasmiad Hasan, Arafat Zaman Ratul, Sarker Sadman Saalim, S. M. Shah Nawaz Hossain+2

North South University, Dhaka, Bangladesh

Legal Document Analysis Legal QA Optical Character Recognition Multimodal Model Evaluation Document Image Understanding

Open paper AskPDF arXiv

Land ownership in Bangladesh is recorded in Ana-Ganda-Kora-Kranti-Til, a base-16 positional fraction system with dedicated Unicode glyphs, no mainstream font, and no coverage in any OCR pipeline or tokenizer. The handwritten records that carry these fractions, RS Khatians, are the authoritative title record for millions of parcels and a frequent subject of civil litigation, yet no benchmark has asked whether a machine can read one. We introduce KhatianDoc, a four-task benchmark built from 107 real RS Khatian records from the Vumi (land) Office of Munshiganj, Bangladesh: symbol recognition, base-16-to-decimal conversion, structured field extraction, and legal document question answering over 1,634 QA pairs. Ground truth was transcribed by hand, verified by a land-law practitioner to full agreement, and anonymized through positional tokens that keep the referential distinctions multi-hop questions depend on. We evaluate six multimodal LLMs (8B to 72B+, open and closed) under a fixed zero-shot protocol. Five QA categories, 39.3% of our stratified set, return zero correct answers from every model; on the arithmetic task, every model that emits a number does worse than a constant-mean baseline, with exact- and near-match scores coinciding: decorrelation, not approximation. Auditing our own metrics surfaced two artifacts in opposite directions: we correct a refusal-scoring bug and report the fixed scores beside the originals, and flag an inflated metadata metric as an upper bound. KhatianDoc documents not a performance gap but the absence of a capability, with verified ground truth for future systems. Code and data, with a redacted image release, are publicly available.

Momentum

Related topics

Latest papers 496