cs.CVOct 30, 2026

Open-Weight MLLMs Cannot Compare What They Cannot Name

Authors: Wenxin Su, Anna Kreshuk

Organizations: European Molecular Biology Laboratory, Heidelberg, Germany

Abstract

Multimodal Large Language Models (MLLMs) are increasingly capable of understanding images and performing complex visual reasoning, suggesting the possibility of using them to automate biological image analysis. This need is becoming increasingly urgent as advances in technology continue to generate ever larger and more diverse microscopy datasets, already exceeding the scale at which manual visual analysis is feasible. With the final aim to use MLLMs for visual comparison of complex phenotypes, we investigate a very simple baseline: comparison of the size of body parts in synthetic images of ants. We find that state-of-the-art open-weight MLLMs fail for some body parts, but not for others, even though the comparison itself is visually trivial for all of them. We evaluate different in-context learning strategies and analyze performance on the related object recognition task. Our results suggest a recognition-driven grounding failure: structures that the models cannot recognize are instead associated with a recognizable semantic concept, impairing the grounding and comparison of both. Further analysis of the vision encoder is consistent with the hypothesis that fine-grained visual structure may be less well preserved without semantic support. Together, our results reveal a surprising limitation of current MLLMs: semantic recognition can gate simple visual comparisons that do not intrinsically require semantic understanding. Our code and data are available at https://github.com/kreshuklab/Open-Weight-MLLMs-Cannot-Compare-What-They-Cannot-Name

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification

    Sep 5, 2026Yixin Wan, Tianle Zheng, Kai-Wei ChangVLM EvaluationVision-Language Models

  2. What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges

    Oct 28, 2026Saumya Mundra, Ajoy Mondal, Vineeth N Balasubramanian +1Visual CountingMLLM Evaluation

  3. Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models

    Jun 29, 2026Sajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie +2Language Model Generation EvaluationMultimodal Reasoning