Multimodal large language models (MLLMs) are often evaluated by converting model outputs into discrete answers. The conversion protocol determines which scores become predictions and can strongly influence the reported results. We introduce RAMP (Readout Aware Multimodal Protocol), a systematic framework that compares direct yes/no token logits, surface-form scores, and typed or reversed A/B interfaces under fixed model checkpoints and paired visual questions. On 500 POPE adversarial questions, typed versus bare readout changes accuracy by −20.2 percentage points for SmolVLM-256M, −16.4 for TinyLLaVA-1.1B, and +1.8 for Qwen2.5-VL-3B. Paired transitions reveal extensive prediction changes even when the net accuracy difference is small: 79 of 104 binary ChartQA items change correctness for SmolVLM-500M, despite a +1.0-point net difference. We further analyze prompt wording, candidate forms, option position, cross-task effects, and calibration. An independent offline replay reproduces 16,140 first-token decisions from saved raw scores with exact agreement. Our findings establish answer readout as a measurable component of multimodal evaluation and provide a reproducible protocol for analyzing model comparisons.
Figures & tables
Figure 1: Readout-sensitive examples. Each row fixes the image and base question while changing only the answer interface. Bare-token, surface-form, and typed A/B readouts can produce different predictions and score margins on the same item; here m denotes the signed yes–no score margin and P(yes)=σ(m) .
Figure 2: Overview of RAMP. A fixed image–question pair is evaluated with frozen MLLM checkpoint. Bare and surface form scoring inspect answer-token logits under the original question. Typed and reversed interfaces score option letters after independently constructed A/B prompts. Semantic mapping converts each candidate decision to a shared yes/no label. Paired analysis compares accuracy, transitions, and confidence.
Dataset / configuration
Model
n
Bare
Surface
Typed
Reverse
Typed − bare (pp)
POPE, new MPS
SmolVLM-256M
500
71.8
75.0
51.6
44.6
−20.2[−27.6,−13.2]
POPE, new MPS
TinyLLaVA-1.1B
500
82.4
78.4
66.0
49.8
−16.4[−20.8,−11.8]
POPE, new MPS
Qwen2.5-VL-3B
500
85.2
86.6
87.0
86.0
+1.8[−0.4,+4.0]
POPE, CPU/single tile
SmolVLM-500M
500
82.2
81.2
60.0
60.4
−22.2[−27.4,−17.4]
ChartQA binary
SmolVLM-256M
104
48.1
50.0
54.8
48.1
+6.7[−6.9,+19.2]
ChartQA binary
SmolVLM-500M
104
52.9
58.7
54.8
49.0
+1.9[−15.2,+17.5]
Table 1: Readout accuracy under recorded model configurations. Values are accuracy percentages on matched questions. Brackets give image-cluster 95% confidence intervals for typed-minus-bare differences in percentage points. POPE SmolVLM-500M uses CPU/fp32 single-tile processing; the first three POPE rows use their separately recorded MPS/fp16 configurations. ChartQA uses 104 native yes/no questions.
Figure 3: Archived POPE accuracy across answer readouts. The saved historical runs evaluate bare yes/no, surface-form, typed A/B, and reversed A/B extraction on POPE adversarial questions. All displayed configurations use 500 items except Qwen’s archived reversed arm (487 observed items); TinyLLaVA’s reversed arm was not recorded. Archived results are separate from the checkpoint-pinned runs in Table 1 .
Figure 4: Paired accuracy changes in archived POPE runs. Typed-minus-bare differences with image-cluster 95% bootstrap intervals on 500 matched questions.
Figure 5: Paired readout effects reconstructed from raw scores. Net typed-minus-bare accuracy changes (left) and the total number of correctness transitions (right) on matched items. The figure identifies the respective POPE and ChartQA run configurations. Large individual-question disagreement can accompany a small aggregate score difference.
Figure 6: Correctness-conditioned absolute score-margin distributions. Empirical cumulative distribution functions for correct and incorrect predictions across three archived POPE models and three readout interfaces. Curves show the margin distribution within each model–protocol panel; horizontal scales differ across protocols. Each panel contains 500 POPE adversarial questions.
Figure 7: Reliability diagrams from archived POPE readouts. Empirical correctness is plotted against predicted correctness confidence for bare, surface-form, and typed interfaces across three backbones. Each curve uses 500 evaluated questions and the saved 15-quantile-bin convention; the dashed line indicates ideal calibration. These historical runs are distinct from the checkpoint-pinned ECE comparison in Fig. 8 .
Figure 8: Calibration sensitivity in checkpoint-pinned POPE runs. Correctness ECE (15 bins; lower is better) for bare and typed interfaces, computed on 500 matched questions per configuration. The SmolVLM-500M experiment uses its separately recorded CPU/fp32 single-tile preprocessing. Values come directly from the processed run-specific metric table.
Figure 9: Readout sensitivity across visual tasks. POPE adversarial comparisons and ChartQA native yes/no subset results are shown under their separately recorded run identities. The ChartQA study contains 104 binary questions and uses the same semantic readout definitions.
Every multimodal-LLM multiple-choice benchmark ends with a scoring rule that converts a model's response, often free-form text rather than a clean option letter, into a right/wrong verdict, buried in its code. Unlike the dataset, this rule is chosen without justification and left unexamined, yet it sets the published number. We reproduce the official scorers of eight peer-reviewed benchmarks on five open vision-language models (VLMs) that fit a single 16 GB GPU, and introduce a blinded human-plus-LLM-judge adjudication (Cohen's kappa=0.956) that recovers what the model actually answered. The scoring rule alone moves accuracy by up to 22 points on identical outputs, yet exactly 0 wherever a model emits clean option letters, so it hides on strong-model leaderboards. It is the fallback and not the rambling that does the damage: the benchmark with the highest non-compliance we measure has an artifact of exactly zero, because its scorer abstains instead of guessing; a reasoning model with non-compliance 0.000 still swings 9.5 points with which regex match of its trace the parser keeps. No standard rule is reliable: snapping a non-answer to its closest option credits answers never given yet buries correct ones, abstention discards answers stated in words, and likelihood scoring follows answer-token priors, so a single grade conflates capability with luck rather than measuring either. We release all code, labels, and the protocol, and argue benchmarks should report a non-compliance rate and at least two rules, not one unexamined grade.
Multimodal Large Language Models (MLLMs) have recently emerged as general architectures capable of reasoning over diverse modalities. Benchmarks for MLLMs should measure their ability for cross-modal integration. However, current benchmarks are filled with shortcut questions, which can be solved using only a single modality, thereby yielding unreliable rankings. For example, in vision-language cases, we can find the correct answer without either the image or the text. These low-quality questions unnecessarily increase the size and computational requirements of benchmarks. We introduce a multi-modal and multidimensional item response theory framework (M3IRT) that extends classical IRT by decomposing both model ability and item difficulty into image-only, text-only, and cross-modal components. M3IRT estimates cross-modal ability of MLLMs and each question's cross-modal difficulty, enabling compact, high-quality subsets that better reflect multimodal reasoning. Across 24 VLMs on three benchmarks, M3IRT prioritizes genuinely cross-modal questions over shortcuts and preserves ranking fidelity even when 50% of items are artificially generated low-quality questions, thereby reducing evaluation cost while improving reliability. M3IRT thus offers a practical tool for assessing cross-modal reasoning and refining multimodal benchmarks.
Shunki Uebayashi, Kento Masui, Kyohei Atarashi +5
Kyoto University · CyberAgent. · The Institute of Statistical Mathematics +1
Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines. We introduce Facet-Probe, a five-facet audit (option, evidence-chunk, document-rank, image-set, and mixed-modality ordering) of 18 frontier and open-weight MLLMs. A Bayesian item-response model separates ordering noise from per-facet bias, and a same-ordering control estimates the decoder-stochastic floor for observed flips. We find that none of the 18 MLLMs we audit are order-invariant: screened per-facet panel-mean flip rates span 24-50%. A Gemini same-ordering control at temperature 0 estimates a substantial ordering excess over a same-input decoder-noise floor in verified cells. Capability predicts but does not eliminate flips; the best model still flips on 13.4% of trials. In our Gemini mitigation tests, training-free prompt changes are modality-conditional and do not transfer from text to visual reasoning. These results suggest that prompt-level mitigation alone is unlikely to provide general order robustness, motivating future work on training-time and architectural approaches. We propose cross-ordering flip rate as a standard reporting axis for MLLMs.