VLM Evaluation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

70 papers in the last four weeks, up 67% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 649

All topics

Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences
Aug 11, 2026Martina Ianaro, Guilherme Fernandes, Maurizio Gabbrielli +1VLM Evaluation Vision-Language Models
University of Bologna Bologna, Italy · NOVA School of Science and Technology NOVA Laboratory for Computer Science and Informatics Lisbon, Portugal
VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback?
Aug 11, 2026Mizanur Rahman, Arshia Azimlu, Shadikur Rahman +4VLM Evaluation Data Visualization
‡York University · $Nanyang Technological University · ¶Salesforce AI Research
More Accurate, Less Human: Gestalt Grouping in Vision Models
Aug 10, 2026Sudhanva Manjunath Athreya, Sai Phani Kumar MalladiVLM Evaluation Data Visualization
Siemens AG · University of Utah Siemens AG
Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains
Aug 10, 2026Diandian Zhang, Tingyu Song, Lin Fu +2VLM Evaluation Video Generation
Zhejiang University · UCAS · Tongji University +1
PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models
Aug 10, 2026Zhanna Mukhametsharip, Vera Demberg, Varsha SureshVLM Evaluation Pragmatic Reasoning in Language Models
Saarland University, Germany · Max Planck Institute for Informatics, Germany
VectraYX-Vision-1B: A Sub-2B Spanish/LATAM Cybersecurity Vision-Language Model with Structured Visual Reasoning and Native Tool Use
Aug 9, 2026Juan S. SantillanaVLM Evaluation Tool Use in VLMs
Globant
Explore, Map, Remember, Decide: Are Embodied VLMs Ready for Safety-Critical Scenarios?
Aug 8, 2026Gabriele La Malfa, Nitay Alon, Emanuele La Malfa +2VLM Evaluation Memory-Augmented VLMs
King’s College London, UK · Massachusetts Institute of Technology, USA · University of Oxford, UK +2
How Much Does It Cost to Answer My Question? Benchmarking Cloud VLM-based VQA Systems
Aug 8, 2026Henri Vanhuynegem, Weitao Xu, Yiran Shen +1VLM Evaluation Visual Question Answering
Department of Software Technology, Delft University of Technology, Delft, The Netherlands · Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · School of Software, Shandong University, Jinan, China +1
Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluation
Aug 7, 2026Anna Kołos, Grzegorz Statkiewicz, Karolina Seweryn +3VLM Evaluation Visual Question Answering
NASK National Research Institute, Warsaw, Poland
BRUCE: Benchmarking Robustness Under Corruption Escalation for Scientific Vision-Language Reasoning
Aug 7, 2026Saim Rehman, Muhammad ShafiqueVLM Evaluation VLM Robustness
eBRAIN Lab New York University Abu Dhabi (NYUAD) Abu Dhabi, United Arab Emirates
SABRE: Scalable and Automated Benchmarking of VLMs under Stress
Aug 7, 2026Zixuan Lan, Luzhe Sun, Matthew R. Walter +1VLM Evaluation VLM Robustness
University of Chicago · Stony Brook University · Toyota Technological Institute at Chicago
Human-AI Perceptual Alignment by Playing Hues and Cues
Aug 7, 2026Nuria Alabau-Bosque, Jorge Vila-Tomás, Paula Daudén-Oliver +3VLM Evaluation Vision-Language Grounding
Image Processing Lab, Universitat de València, Carrer del Catedrátic José Beltrán Martinez, Paterna, 46980, Spain
Beyond Fluency: A Clinical Benchmark and Anomaly-Enhanced Baseline for Spine MRI Report Generation
Aug 7, 2026Bruno Palau, Franziska Vogt, Daria Laslo +4VLM Evaluation Radiology Report Generation
ETH Zurich, Switzerland · 2Swiss Institute of Bioinformatics (SIB), Switzerland
Generative Embedding Benchmark: How Much Information Survives in a Dense Embedding?
Aug 7, 2026Yun Li, Biao Yang, Peixi Wu +5VLM Evaluation Multimodal Embedding
2Fudan University · 1Kuaishou Technology · University of Science and Technology of China +1
GraphVerse: A Comprehensive Visual Graph Reasoning Benchmark for Multimodal Large Language Models
Aug 7, 2026Yuanfu Sun, Yuanhang Ren, Kang Li +5VLM Evaluation Visual Reasoning
New York University · Sensetime Research · Tsinghua University +3
Visual Grounding in Zero-Shot Vision-Language Control
Aug 6, 2026J. de Curtò, Dayani Plasencia, Diego Sánchez +1VLM Evaluation VLMs for Autonomous Driving
Department of Computer Applications in Science & Engineering, BARCELONA Supercomputing Center, Barcelona, Spain · Escuela T´ecnica Superior de Ingenier´ıa (ICAI), Universidad Pontificia Comillas, Madrid, Spain · Estudis d’Inform`atica, Multim`edia i Telecomunicaci´o, Universitat Oberta de Catalunya, Barcelona, Spain +3
VLMs for Videogame Data Annotation
Aug 6, 2026Katrin Schmid, Iuri FrosioVLM Evaluation
NVIDIA, Australia · NVIDIA, Italy
GST-Bench: Can VLMs Develop Global Spatial Awareness from Video?
Aug 6, 2026Qifeng Zhang, Kaixiang Huang, Heng Dong +6VLM Evaluation Spatial Reasoning Benchmarks
ByteDance Seed · Zhejiang University · National University of Singapore +1
TAU-Bench: From Anomaly Instance Tracking to Fine-Grained Video Anomaly Understanding
Aug 6, 2026Kepeng Yang, Dongxuan Liu, Rongxin Gao +8VLM Evaluation Anomaly Localization
XMU · THU · CUHK +1
Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading
Aug 6, 2026Rasul Khanbayov, Hasan KurbanVLM Evaluation Vision-Language Models
College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering
Aug 6, 2026Rasul Khanbayov, Hasan KurbanVLM Evaluation
College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs
Aug 6, 2026Yanqi Wu, Runhe Lai, Xinhua Lu +5VLM Evaluation Hallucination Detection in VLMs
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Peng Cheng Laboratory, Shenzhen, China · Key Laboratory of Machine Intelligence and Advanced Computing, MOE, Guangzhou, China +2
DynaPix: Can Vision-Language Models Identify the Exact Future?
Aug 6, 2026Thong Nguyen, Vinh-Hien Do, Quynh Vo +2VLM Evaluation Vision-Language Models
Centre for AI Research, VinUniversity · National University of Singapore
Evaluating the Diagnostic Robustness of Vision-Language Models Under Visual and Textual Perturbations
Aug 5, 2026Ali Khoramfar, Mohammad Javad Dousti, Alireza Mohamadian +1VLM Evaluation VLM Robustness
Department of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran · Advanced Diagnostic and Interventional Radiology Research Center (ADIR), Tehran University of Medical Sciences, Tehran, Iran
ComplexityWorld: Benchmarking Vision-Language Models on Verifiable Visual Decision Making
Aug 5, 2026Ningxin Pan, Hanyu Li, Yehui TangVLM Evaluation Visually Grounded Reasoning
Peking University, Beijing, China.
BanglaWild: An In-the-Wild Bengali Scene Text Recognition Benchmark for OCR and Vision-Language Models
Aug 4, 2026Sadab Shiper, Tawsif Tashwar Dipto, Mir Md Inzamam +1VLM Evaluation Scene Text Recognition
Islamic University of Technology, Bangladesh · BRAC University, Bangladesh
Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation
Aug 4, 2026Chunlin Liu, Junnian Chen, Haitong Jiang +7VLM Evaluation VLM Unlearning
1Shenzhen University · 2Hebei University of Technology · 3Ludwig Maximilian University of Munich +1
COMEX: A Composition-Grounded Benchmark and Learning Framework for Explainable Aesthetic Image Cropping
Aug 4, 2026Rui Yang, Wei Zhou, Dingyong Gou +5VLM Evaluation Image Editing
State Key Laboratory of Mobile Network and Mobile Multimedia Technology, ZTE Corporation · School of Data Science and Institute of Artificial Intelligence, Chang’an University
Confident but Unreliable: A Behavioral Safety Audit of Vision-Language Models on Brain MRI
Aug 3, 2026Amir Sabbaghziarani, Mohammadsajad Abavisani, Sergey PlisVLM Evaluation VLM Hallucination
TReNDS Center Atlanta, GA, USA
Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search
Aug 3, 2026Han Wang, Alex Whitworth, Pak Ming Cheung +5VLM Evaluation Automated Evaluation
Pinterest · San Francisco, CA, USA

Aug 11, 2026, cs.CVJ/K move · Enter open · S save

Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences

Martina Ianaro, Guilherme Fernandes, Maurizio Gabbrielli, Joao Magalhaes

University of Bologna Bologna, Italy · NOVA School of Science and Technology NOVA Laboratory for Computer Science and Informatics Lisbon, Portugal

VLM Evaluation Vision-Language Models Large Vision-Language Models Video Reasoning

Open paper AskPDF arXiv

As generative multimedia evolves from static image synthesis to complex, interleaved visual narratives, a foundational bottleneck has emerged: the judgment crisis. While human perception naturally synthesizes the temporal and logical flow of a story, automated evaluation systems remain largely "blind" to sequential continuity, often failing to distinguish between a coherent narrative and a semantically shuffled or contradictory sequence. This work identifies a critical structural gap in current multimodal evaluation paradigms, arguing that the reliance on Large Vision-Language Models (LVLMs) as judges is fundamentally limited by architectural biases. Our analysis reveals a profound performance dichotomy: while models may appear competent in isolated pointwise scoring, they suffer a catastrophic collapse when required to perform pairwise discrimination of temporal order. We demonstrate that this is not merely a data-scarcity issue but a structural one. Through a series of diagnostic probes, we uncover systematic positional asymmetries, specifically primacy and recency effects, where a model's judgment of a story is significantly influenced by the placement of a frame, often more than by its semantic consistency. These biases, potentially rooted in causal masking and rotary embeddings, suggest that current transformer-based judges are inherently ill-equipped for long-form visual reasoning. By exposing these blind spots, we challenge the multimedia community to move beyond snapshot-centric metrics and instead pioneer Temporally-Aware Evaluation paradigms that treat visual sequences as unified logical structures rather than unordered collections of frames.

Momentum

Related topics

Latest papers 649