Open-Weight MLLMs Cannot Compare What They Cannot Name
Organizations: European Molecular Biology Laboratory, Heidelberg, Germany
Abstract
Multimodal Large Language Models (MLLMs) are increasingly capable of understanding images and performing complex visual reasoning, suggesting the possibility of using them to automate biological image analysis. This need is becoming increasingly urgent as advances in technology continue to generate ever larger and more diverse microscopy datasets, already exceeding the scale at which manual visual analysis is feasible. With the final aim to use MLLMs for visual comparison of complex phenotypes, we investigate a very simple baseline: comparison of the size of body parts in synthetic images of ants. We find that state-of-the-art open-weight MLLMs fail for some body parts, but not for others, even though the comparison itself is visually trivial for all of them. We evaluate different in-context learning strategies and analyze performance on the related object recognition task. Our results suggest a recognition-driven grounding failure: structures that the models cannot recognize are instead associated with a recognizable semantic concept, impairing the grounding and comparison of both. Further analysis of the vision encoder is consistent with the hypothesis that fine-grained visual structure may be less well preserved without semantic support. Together, our results reveal a surprising limitation of current MLLMs: semantic recognition can gate simple visual comparisons that do not intrinsically require semantic understanding. Our code and data are available at https://github.com/kreshuklab/Open-Weight-MLLMs-Cannot-Compare-What-They-Cannot-Name
Figures & tables
| Model | Param. | Date | Antennae | Mandibles | Bristles | Legs |
|---|---|---|---|---|---|---|
| Open-weight Instruction MLLMs | ||||||
| Qwen3.8 | 27B | 2026-08 | 57.0 | 38.6 | 89.9 | 90.5 |
| Gemma4 | 31B | 2026-04 | 54.6 | 50.4 | 80.1 | 93.0 |
| Qwen3.5 | 27B | 2026-02 | 51.9 | 33.2 | 83.7 | 83.8 |
| InternVL3.5 | 38B | 2025-08 | 39.4 | 34.2 | 89.6 | 61.1 |
| GLM4.1 | 9B | 2025-07 | 35.3 | 34.1 | 85.7 | 52.6 |
| Model | Baseline | Reference | Example | Object | Segmentation | Segmentation+Object |
|---|---|---|---|---|---|---|
| Open-weight Instruction MLLMs | ||||||
| Qwen3.8 | 57.0 | 51.8 (-5.2) | 60.0 (+3.0) | 55.0 (-2.0) | 75.7 (+18.7) | 78.0 (+21.0) |
| Gemma4 | 54.6 | 48.0 (-6.6) | 51.3 (-3.3) | 49.9 (-4.7) | 73.2 (+18.6) | 75.9 (+21.3) |
| Qwen3.5 | 51.9 | 48.9 (-3.0) | 53.8 (+1.9) | 46.4 (-5.5) | 72.0 (+20.1) | 71.6 (+19.7) |
| InternVL3.5 | 39.4 | 42.5 (+3.1) | 41.3 (+1.9) | 34.5 (-4.9) | 44.5 (+5.1) | 43.7 (+4.3) |
| GLM4.1 | 35.3 | 35.5 (+0.2) | 34.8 (-0.5) | 33.4 (-1.9) | 39.0 (+3.7) | 34.9 (-0.4) |
| Model | Baseline | Reference | Example | Object | Segmentation | Segmentation+Object |
|---|---|---|---|---|---|---|
| Open-weight Instruction MLLMs | ||||||
| Qwen3.8 | 38.6 | 38.6 (0.0) | 40.7 (+2.1) | 56.9 (+18.3) | 77.9 (+39.3) | 81.4 (+42.8) |
| Gemma4 | 50.4 | 44.1 (-6.3) | 47.1 (-3.3) | 48.5 (-1.9) | 69.0 (+18.6) | 73.0 (+22.6) |
| Qwen3.5 | 33.2 | 32.8 (-0.4) | 34.3 (+1.1) | 43.6 (+10.4) | 68.8 (+35.6) | 76.8 (+43.6) |
| InternVL3.5 | 34.2 | 35.4 (+1.2) | 36.6 (+2.4) | 36.6 (+2.4) | 53.7 (+19.5) | 51.6 (+17.4) |
| GLM4.1 | 34.1 | 34.2 (+0.1) | 33.7 (-0.4) | 34.1 (0.0) | 34.3 (+0.2) | 35.0 (+0.9) |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges
Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models
r/photocritique posts under eight prompt conditions. Reference-based similarity gives a misleading picture. In absolute terms the stricter lexical and learned metrics align only weakly with human critiques while a coarse embedding cosine reports broad topical overlap, yet requesting shorter critiques raises those scores and withholding the image barely changes them: the similarity reflects length, the post text, and a stable critiquing style more than image-specific observation. An LLM judge sharpens the question rather than settling it: in the primary condition all four judges prefer the frontier models' critiques to the human ones, but on the --B models they diverge wildly, from to preference on identical pairs. Asked instead how similar each pair is in substance, those judges and two human annotators agree, rating every model between and on a -- scale, close to ``mostly different''. Behaviorally, the models diverge in ways the scores do not surface: they cover nearly every aesthetic aspect where humans are selective and repeat themselves across critiques of one photo, even when prompted to write at human length. We argue that reference-based similarity rewards a fluent, comprehensive critique style rather than the selectivity and specificity of human critique.