Multimodal Model Evaluation

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 48% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 496

All topics

GaMMA: Towards Joint Global-Temporal Music Understanding in Large Multimodal Models
May 1, 2026Zuyao You, Zhesong Yu, Mingyu Liu +3Multimodal Pretraining Multimodal Large Language Models
1Fudan University · 2ByteDance
Are Multimodal LLMs Ready for Clinical Dermatology? A Real-World Evaluation in Dermatology
May 1, 2026Roy Jiang, Hyunjae Kim, Zhenyue Qin +8Healthcare Clinical Triage
Department of Dermatology, Yale School of Medicine, Yale University, New Haven, CT, USA · Department of Bioinformatics and Data Science, Yale School of Medicine, Yale University, New Haven, CT, USA · Yale School of Medicine, Yale University, New Haven, CT, USA
SpecVQA: A Benchmark for Spectral Understanding and Visual Question Answering in Scientific Images
Apr 30, 2026Jialu Shen, Han Lyu, Suyang Zhong +5Visual Question Answering Multimodal Model Evaluation
DP Technology
Decoding Scientific Experimental Images: The SPUR Benchmark for Perception, Understanding, and Reasoning
Apr 30, 2026Junpeng Ding, Zichen Tang, Haihong E +17AI for Science VLM Reasoning
Beijing University of Posts and Telecommunications
COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts
Apr 30, 2026Bingli Wang, Huanze Tang, Haijun Lv +5Vision-Language Models Vision-Language Grounding
1Southeast University · 2Shanghai AI Laboratory
TRIP-Evaluate: An Open Multimodal Benchmark for Evaluating Large Models in Transportation
Apr 29, 2026Han Gong, Zhen Zhou, Yunyang Shi +4LLM Evaluation Intelligent Transportation Systems
School of Transportation Southeast University Nanjing, Jiangsu 211189 · School of Artificial Intelligence and Computer Science Jiangnan University Wuxi, Jiangsu 214122 · Department of Civil and Environmental Engineering Hong Kong Polytechnic University Kowloon, Hong Kong
HotComment: A Benchmark for Evaluating Popularity of Online Comments
Apr 28, 2026Yafeng Wu, Yunyao Zhang, Liliang Ye +4Social Media Analysis Multimodal Model Evaluation
Huazhong University of Science and Technology Wuhan, China · Beijing Institute of Computer Technology and Applications Beijing, China
VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation
Apr 28, 2026Divake Kumar, Sina Tayebati, Devashri Naik +2VLM Evaluation Vision-Language Models
University of Illinois at Chicago · 2AI Labs at Capital One
M $^3$ -VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering
Apr 28, 2026Jiatong Ma, Longteng Guo, Yuchen Liu +4Multimodal RAG Visual Question Answering
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models
Apr 27, 2026Weixing Wang, Liudvikas Zekas, Anton Hackl +5Unified Multimodal Models Cross-Modal Alignment
Hasso Plattner Institute / University of Potsdam, Potsdam, Germany
Assessing Y-Axis Influence: Bias in Multimodal Language Models on Chart-to-Table Translation
Apr 27, 2026Seok Hwan Song, Azher Ahmed Efat, Wallapak TavanapongData Visualization Multimodal Large Language Models
Department of Computer Science, Iowa State University, Ames, Iowa, USA
K-MetBench: A Multi-Dimensional Benchmark for Fine-Grained Evaluation of Expert Reasoning, Locality, and Multimodality in Meteorology
Apr 27, 2026Soyeon Kim, Cheongwoong Kang, Myeongjin Lee +3LLM Evaluation Cultural Bias in Language Models
KAIST, INEEJI · Seongnam, Korea · KAIST +2
All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation
Apr 27, 2026Leonardo Haw-Yang Foo, Chih-Kai Yang, Chen-An Li +2Audio-Language Model Evaluation Audio QA
National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
Exploring Audio Hallucination in Egocentric Video Understanding
Apr 26, 2026Ashish Seth, Xinhao Mei, Changsheng Zhao +9Audio-Visual Understanding Hallucination in Language Models
Meta · University Of Maryland, College Park
ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction
Apr 26, 2026Zichun Guo, Yuling Shi, Wenhao Zeng +6Image Reconstruction Multimodal Large Language Models
Xidian University · Shanghai Jiao Tong University · Alibaba Qwen +1
FAIR_XAI: Improving Multimodal Foundation Model Fairness via Explainability for Wellbeing Assessment
Apr 26, 2026Sophie Chiang, Tom Brennan, Fethiye Irmak Dogan +2Depression Detection Medical Diagnosis
Department of Computer Science & Technology, University of Cambridge. · Harvard University.
EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs
Apr 25, 2026He Hu, Tengjin Weng, Zebang Cheng +5Multimodal Large Language Models Multimodal Model Evaluation
Shenzhen University Shenzhen, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) Shenzhen, China · Queen Mary University of London +3
MTT-Bench: Predicting Social Dominance in Mice via Multimodal Large Language Models
Apr 24, 2026Yunquan Chen, Haoyu ChenMultimodal Large Language Models Social Reasoning
Department of Communication Systems, KTH Royal Institute of Technology, Stockholm, Sweden · CMVS, University of Oulu, Finland
CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language
Apr 24, 2026Rui Zhao, Xuewen Zhong, Xiaoyun Zheng +2Multimodal Large Language Models Sign Language Recognition
School of Informatics, Xiamen University, China · 2Key Lab of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian-Taiwan (XMU), Ministry of Culture and Tourism, China · 3National Language Resources Monitoring and Research Center for Education and Teaching Media, Xiamen University, China
Can Multimodal Large Language Models Truly Understand Small Objects?
Apr 24, 2026Fujun Han, Junan Chen, Xintong Zhu +4Multimodal QA Multimodal Model Evaluation
Shanghai AI Laboratory · The Chinese University of Hong Kong, Shenzhen · University of Science and Technology of China +2
Beyond Single Plots: A Benchmark for Question Answering on Multi-Charts
Apr 23, 2026Azher Ahmed Efat, Seok Hwan Song, Wallapak TavanapongData Visualization Multimodal Large Language Models
Department of Computer Science, Iowa State University, Ames, Iowa, USA
ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Response
Apr 23, 2026Stephan Xie, Ben Cohen, Mononito Goswami +6Time Series Reasoning Time Series QA
Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, USA · Datadog AI Research, New York, NY, USA · Amazon Web Services, Seattle, WA, USA
Thinking Like a Botanist: Challenging Multimodal Language Models with Intent-Driven Chain-of-Inquiry
Apr 22, 2026Syed Nazmus Sakib, Nafiul Haque, Shahrear Bin Amin +4Visual Question Answering Plant Disease Classification
Department of Robotics and Mechatronics Engineering, University of Dhaka, Dhaka, Bangladesh · Department of Computer Science and Engineering, University of Dhaka, Dhaka, Bangladesh · Department of Agronomy, Gazipur Agricultural University, Gazipur, Bangladesh +1
ONOTE: Benchmarking Omnimodal Notation Processing for Expert-level Music Intelligence
Apr 22, 2026Menghe Ma, Siqing Wei, Yuecheng Xing +5Optical Music Recognition Multimodal Reasoning
Beijing University of Posts and Telecommunications Beijing, China · China Conservatory of Music Beijing, China · Nanyang Technological University Singapore, Singapore
The Expense of Seeing: Attaining Trustworthy Multimodal Reasoning Within the Monolithic Paradigm
Apr 22, 2026Karan GoyalVLM Evaluation Multimodal Reasoning
IIIT Delhi, India
Exploring Spatial Intelligence from a Generative Perspective
Apr 22, 2026Muzhi Zhu, Shunyao Jiang, Huanyi Zheng +9Spatial Reasoning Benchmarks Unified Multimodal Models
Zhejiang University, State Key Laboratory of CAD & CG · Ant Group · Zhejiang University of Technology +1
CHASM: Unveiling Covert Advertisements on Chinese Social Media
Apr 22, 2026Jingyi Zheng, Tianyi Hu, Yule Liu +5Social Media Analysis Online Advertising
Hong Kong University of Science and Technology (Guangzhou) · Aarhus Univeristy
CCTVBench: Contrastive Consistency Traffic VideoQA Benchmark for Multimodal LLMs
Apr 22, 2026Xingcheng Zhou, Hao Guo, Rui Song +5Visual Question Answering Video QA
Technical University of Munich · University of California, Los Angeles
X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosis
Apr 22, 2026Gui Wang, Zehao Zhong, YongSong Zhou +6Healthcare Medical Diagnosis
School of Computer Science and Software Engineering, Shenzhen University · 2Tsinghua University · 5Wenzhou Medical University +3
SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark
Apr 22, 2026Gui Wang, YongSong Zhou, Kaijun Deng +4Spatiotemporal Reasoning Healthcare
School of Computer Science and Software Engineering, Shenzhen University · University of Nottingham · School of AI, Shenzhen University +1

May 1, 2026, cs.SDJ/K move · Enter open · S save

GaMMA: Towards Joint Global-Temporal Music Understanding in Large Multimodal Models

Zuyao You, Zhesong Yu, Mingyu Liu, Bilei Zhu+2

1Fudan University · 2ByteDance

Multimodal Pretraining Multimodal Large Language Models Audio Understanding Audio-Language Models Music Information Retrieval Multimodal Model Evaluation+1

Open paper AskPDF arXiv

In this paper, we propose GaMMA, a state-of-the-art (SoTA) large multimodal model (LMM) designed to achieve comprehensive musical content understanding. GaMMA inherits the streamlined encoder-decoder design of LLaVA, enabling effective cross-modal learning between music and language. By incorporating audio encoders in a mixture-of-experts manner, GaMMA effectively unifies both time-series and non-time-series music understanding tasks within one set of parameters. Our approach combines carefully curated datasets at scale with a progressive training pipeline, effectively pushing the boundaries of music understanding via pretraining, supervised fine-tuning (SFT), and reinforcement learning (RL). To comprehensively assess both temporal and non-temporal capability of music LMMs, we introduce MusicBench, the largest music-oriented benchmark, comprising 3,739 human-curated multiple-choice questions covering diverse aspects of musical understanding. Extensive experiments demonstrate that GaMMA establishes new SoTA in the music domain, achieving 79.1% accuracy on MuchoMusic, 79.3% on MusicBench-Temporal, and 81.3% on MusicBench-Global, consistently outperforming previous methods.

Momentum

Related topics

Latest papers 496