VLM Evaluation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

71 papers in the last four weeks, up 69% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 649

All topics

Benchmarking Large Vision-Language Models on CFMME: A Comprehensive Chinese Financial Multimodal Evaluation Dataset
May 28, 2026Qian Chen, Xianyin Zhang, Yanzhi Liu +3VLM Evaluation Vision-Language Models
Qwen DianJin Team, Alibaba Cloud Computing
CrystalXRD-Bench: Benchmarking Vision-Language Models for XRD Peak Indexing Across Diverse Crystalline Materials
May 28, 2026Chengliang Xu, Xiaogang Li, Peiyao Xiao +3VLM Evaluation Materials Science
Alibaba Group, Beijing, China
Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark
May 28, 2026Rahul Bissa, Abhishek Vyas, Yash JainVLM Evaluation Supervised Fine-Tuning
AprioriLabs
Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models
May 27, 2026Jiyao Zhang, Mingxu Zhang, Yitong Peng +8VLM Evaluation Spatial Reasoning Benchmarks
VLMs May Not Globally Enhance Human Alignment over LLMs During Natural Reading
May 27, 2026Jinzhou Wu, Zhengwu Ma, Jixing Li +2VLM Evaluation Vision-Language Models
Department of Mechanical and Vehicle Engineering, Chongqing University · Department of Linguistics and Translation, City University of Hong Kong · McGovern Institute for Brain Research, Massachusetts Institute of Technology
The Abstraction Gap in Vision-Language Causal Reasoning
May 27, 2026Chinh Hoang, Mohammad Rashedul HasanVLM Evaluation Causal Reasoning in Language Models
Department of Electrical and Computer Engineering, University of Nebraska–Lincoln, Lincoln, Nebraska, USA.
When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs
May 27, 2026Marcell Fekete, Johannes Bjerva, Tamás KáldiVLM Evaluation Visual Question Answering
Department of Computer Science, Aalborg University, Copenhagen, Denmark · Department of Psycholinguistics and Neurolinguistics, ELTE Research Centre for Linguistics, Budapest, Hungary · ELTE Bárczi Gusztáv Faculty of Special Needs Education, Budapest, Hungary
PointQ-Bench: Benchmarking Diagnostic and Interpretable Point Cloud Quality Assessment
May 27, 2026Duanchu Wang, Cheng Li, Junjie Yang +5VLM Evaluation Point Cloud Understanding
Xidian University · Xi’an Jiaotong University · University of Chinese Academy of Sciences +1
Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers
May 27, 2026Carmen Quiles-Ramírez, Leticia L. Rodríguez, Nicolás Martorell +1VLM Evaluation Multimodal Large Language Models
Universidad de Granada, Spain · Universidad de Buenos Aires, Argentina
Evolving to the Aesthetics of a Vision-Language Model
May 27, 2026Stephen James Krol, Jon McCormackVLM Evaluation Preference Learning
SensiLab, Monash University Melbourne, Australia
Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions
May 26, 2026Antonia Karamolegkou, Nicolas Angleraud, Benoît Sagot +1VLM Evaluation Document Understanding
Inria, Paris, France
Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery
May 26, 2026Yifan Jiang, Ruoxi Ning, Sheng Yao +1VLM Evaluation Vision-Language Models
University of Waterloo · Vector Institute
Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models
May 26, 2026Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1VLM Evaluation Data Visualization
University of Waterloo · Vector Institute · Layer 6 AI
EpiCurveBench: Evaluating VLMs on Epidemic Curve Digitization
May 26, 2026Thomas Berkane, Maimuna S. MajumderVLM Evaluation Chart Data Extraction
Computational Health Informatics Program Boston Children’s Hospital & Harvard Medical School
Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation
May 26, 2026Heng Qu, Yike Liu, Renren Jin +4VLM Evaluation Vision-Language Navigation
1Wuhan University · 2Xiaomi
On the Robustness of Machine Unlearning for Vision-Language Models
May 26, 2026Yujie Lin, Kaidi Jia, Jiayao Ma +2VLM Evaluation VLM Unlearning
1Xiamen University
METATR: A Multilingual, Evolving Benchmark for Automatic Text Recognition
May 26, 2026Mélodie Boillet, Solène Tarride, Christopher KermorvantVLM Evaluation Online Handwriting Recognition
TEKLIA, Paris, France
FTibSuite: A Comprehensive Resource Suite for Tibetan Vision-Language Modeling
May 26, 2026Guixian Xu, Yide Liang, Zeli Su +5VLM Evaluation Vision-Language Models
Hainan International College, Minzu University of China · School of Information Engineering, Minzu University of China · Institute of Automation, Chinese Academy of Sciences +1
VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenes
May 25, 2026Jingru Chen, Yiming Liu, Mingtao Chen +5VLM Evaluation Tool Use in VLMs
Hunyuan, Tencent · Peking University · Zhejiang University
LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence
May 25, 2026Xiang An, Yin Xie, Feilong Tang +27VLM Evaluation Temporal Video Grounding
Glint Lab · AIM for Health Lab · MVP Lab
Beyond Appearance: Can Multimodal Large Language Models Exploit Vertical Structure for Remote Sensing Natural Scene Understanding?
May 25, 2026Jing Huang, Duanchu Wang, Junjie Yang +5VLM Evaluation Remote Sensing Image Understanding
School of Software Engineering, Xi’an Jiaotong University, Xi’an 710049, China · Shenyang Institute of Automation, University of Chinese Academy of Sciences, Shenyang, China · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University
Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker
May 25, 2026Zongjian Wu, Lei ZhangVLM Evaluation Vision-Language Grounding
School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China
MetaphorVU: Towards Metaphorical Video Understanding
May 25, 2026Zhuoqun Li, Boxi Cao, Guiping Jiang +13VLM Evaluation Figurative Language Understanding
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Kuaishou Technology +1
Do Image-Text Metrics Respect Semantic Invariances?
May 23, 2026Amit Agarwal, Hitesh Laxmichand Patel, Meizhu Liu +9VLM Evaluation Multimodal Robustness
Oracle AI
PEDESTRIANQA: A Benchmark for Vision-Language Models on Pedestrian Intention and Trajectory Prediction
May 23, 2026Naman Mishra, Shankar Gangisetty, C. V. JawaharVLM Evaluation Pedestrian Intention Prediction
CVIT, IIIT-Hyderabad, India.
FoodMonitor: Benchmarking MLLMs for Explainable Compliance Analysis
May 23, 2026Ruihao Xu, Xingming Shui, Jingxuan Niu +4VLM Evaluation Video Anomaly Detection
Tsinghua Shenzhen International Graduate School, Tsinghua University
SPACENUM: Revisiting Spatial Numerical Understanding in VLMs
May 22, 2026Jianshu Zhang, Yijiang Li, Huifeixin Chen +4VLM Evaluation Visual Spatial Reasoning
Northwestern · UCSD · USC +1
General Hazard Detection
May 22, 2026Stephanie Ng, CP Lim, SueJen Looi +5VLM Evaluation AI Safety Evaluation
Swinburne University of Technology, VIC, Australia · National Transport Research Organisation (NTRO), Australia · Google Cloud, California, USA +1
EvalVerse: Pipeline-Aware and Expert-Calibrated Benchmarking for Professional Cinematic Video Generation
May 22, 2026Songlin Yang, Haobin Zhong, Ruilin Zhang +23VLM Evaluation Video Quality Assessment
The Hong Kong University of Science and Technology · Tencent · Tsinghua University +5
CaST-Bench: Benchmarking Causal Chain-Grounded Spatio-Temporal Reasoning for Video Question Answering
May 22, 2026Mingfang Zhang, Jingjing Pan, Ashutosh Kumar +7VLM Evaluation Temporal Video Grounding
Woven by Toyota · The University of Tokyo

May 28, 2026, cs.CVJ/K move · Enter open · S save

Benchmarking Large Vision-Language Models on CFMME: A Comprehensive Chinese Financial Multimodal Evaluation Dataset

Qian Chen, Xianyin Zhang, Yanzhi Liu, Lifan Guo+2

Qwen DianJin Team, Alibaba Cloud Computing

VLM Evaluation Vision-Language Models Multimodal QA Multimodal Model Evaluation

Open paper AskPDF arXiv

The emergence of Large Vision-Language Models (LVLMs) has substantially expanded model capabilities beyond text-only understanding, enabling unified inference across both visual and textual modalities and supporting a broader range of real-world applications. To comprehensively evaluate the perception, understanding, reasoning, and cognition capabilities of LVLMs throughout the entire financial business workflow in Chinese contexts, we introduce CFMME, a novel Chinese financial multimodal evaluation benchmark. CFMME comprises 6,052 instances spanning from fundamental academic knowledge to complex real-world applications, covering eight primary financial image modalities and four core multimodal tasks. On CFMME, we conduct a thorough evaluation of representative LVLMs. The results show that the state-of-the-art model attains an overall accuracy of 66.11% on the question answering task and an average score of 77.18 on the detection, recognition, and information extraction tasks, indicating substantial room for improvement in current LVLMs. In addition, we conduct detailed analyses of error causes, cross-modal capabilities, and multi-orientation settings, yielding valuable insights for future research. We hope that CFMME will spur further progress in LVLMs, especially by improving their performance on multiple multimodal tasks in the financial domain.

Momentum

Related topics

Latest papers 649