cs.CVNov 16, 2026

RAMP: Rethinking Multimodal Evaluation through Answer Readout Protocols

Authors: Jianye Zeng, Qiao Yikang, Duan Huang, Ziying Song

Organizations: University of Science and Technology of China · Central South University · Yanshan University

Abstract

Multimodal large language models (MLLMs) are often evaluated by converting model outputs into discrete answers. The conversion protocol determines which scores become predictions and can strongly influence the reported results. We introduce RAMP (Readout Aware Multimodal Protocol), a systematic framework that compares direct yes/no token logits, surface-form scores, and typed or reversed A/B interfaces under fixed model checkpoints and paired visual questions. On 500 POPE adversarial questions, typed versus bare readout changes accuracy by −20.2-20.2 percentage points for SmolVLM-256M, −16.4-16.4 for TinyLLaVA-1.1B, and +1.8+1.8 for Qwen2.5-VL-3B. Paired transitions reveal extensive prediction changes even when the net accuracy difference is small: 79 of 104 binary ChartQA items change correctness for SmolVLM-500M, despite a +1.0+1.0-point net difference. We further analyze prompt wording, candidate forms, option position, cross-task effects, and calibration. An independent offline replay reproduces 16,140 first-token decisions from saved raw scores with exact agreement. Our findings establish answer readout as a measurable component of multimodal evaluation and provide a reproducible protocol for analyzing model comparisons.

Figures & tables

Explore similar work

CardsList
  1. Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks

    Nov 2, 2026Ahmad Jarjis Hasan, Jonathan LwowskiMLLM EvaluationVLM Evaluation

  2. Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory

    Mar 3, 2026Shunki Uebayashi, Kento Masui, Kyohei Atarashi +5MLLM EvaluationMultimodal Model Evaluation