Multimodal Model Evaluation

Overview

Multimodal model evaluation studies whether systems can perceive, reason over, and generate content using combinations of text, images, audio, video, documents, and structured signals. Researchers build task-specific benchmarks and assess outputs with human judgments, programmatic checks, structured rubrics, and measures of evidence grounding, calibration, and safety. Controlled tests such as modality ablations, counterfactual inputs, and robustness checks help reveal shortcuts and failures that aggregate accuracy can hide. A central challenge is showing that a correct answer reflects reliable use of the relevant evidence, while keeping evaluations reproducible, representative, and comparable across settings.

Latest papers

CardsList

799 papers39 papers in the last four weeks, down 53% on the four weeks before. 0.3% of all new papers.

  1. NeMo: Needle in a Montage for Video-Language Understanding

    Sep 29, 2025Zi-Yuan Hu, Shuo Liang, Duo Zheng +10Temporal Video GroundingVideo-Language Model Evaluation

  2. Automotive-ENV: Benchmarking Multimodal Models in Automotive Cockpit Environments

    Sep 25, 2025Junfeng Yan, Biao Wu, Meng Fang +1Multimodal Model EvaluationHuman-Computer Interaction

  3. The Platonic Universe: Do Foundation Models See the Same Sky?

    Sep 23, 2025UniverseTBD, :, Trinidad Borrell +11Cross-Modal AlignmentMultimodal Model Evaluation

  4. Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection

    Sep 18, 2025Zhu Li, Xiyuan Gao, Yuqing Zhang +2MLLM EvaluationSarcasm Detection

  5. Measuring Epistemic Humility in Multimodal Large Language Models

    Sep 11, 2025Bingkui Tong, Jiaer Xia, Sifeng Shang +1VLM HallucinationMultimodal Large Language Models

  6. Explain Before You Answer: A Survey on Compositional Visual Reasoning

    Aug 24, 2025Fucai Ke, Joy Hsu, Zhixi Cai +10Compositional ReasoningVisual Reasoning

  7. VGGSounder: Audio-Visual Evaluations for Foundation Models

    Aug 11, 2025Daniil Zverev, Thaddäus Wiedemer, Ameya Prabhu +3Audio-Visual UnderstandingMultimodal Classification

  8. Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenes

    Jun 11, 2025Zhaoyang Wei, Bowen Jiang, Xumeng Han +6Multimodal GroundingMultimodal Robustness

  9. A Survey of Generative Categories and Techniques in Multimodal Generative Models

    May 29, 2025Longzhen Han, Awes Mubarak, Almas Baimagambetov +2Multimodal GenerationMultimodal Model Evaluation

  10. MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation

    May 23, 2025Jihan Yao, Yushi Hu, Wenyuan Wang +10Multimodal GenerationMultimodal Model Evaluation

  11. FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow

    May 23, 2025Haoyu Sun, Huichen Will Wang, Jiawei Gu +2Software EngineeringWeb Application Generation

  12. Redemption Score: A Multi-Modal Evaluation Framework for Image Captioning via Distributional, Perceptual, and Linguistic Signal Triangulation

    May 22, 2025Ashim Dahal, Ankit Ghimire, Saydul Akbar Murad +1Image CaptioningImage Captioning Evaluation

  13. HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation

    May 16, 2025Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie +6Algorithmic FairnessMultimodal Large Language Models

  14. ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models

    Feb 13, 2025Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma +31Visually Grounded ReasoningVLM Reasoning