cs.CVNov 6, 2026

BEiTScore++: Reference-free Image Captioning Evaluation with BEiT-3 and Conformalized Quantile Regression

Authors: Gonçalo Gomes, Bruno Martins, Chrysoula Zerva

Organizations: Instituto Superior Técnico, University of Lisbon · INESC-ID · Instituto de Telecomunicações · AILS Lab; ECE, National Technical University of Athens

Abstract

Reliable uncertainty estimation is important for deploying evaluation metrics in real-world settings, where both input ambiguity and domain shifts are common. In this work, we propose an uncertainty-aware extension of the BEiTScore image captioning metric, producing conformally calibrated confidence intervals alongside alignment scores. Our approach models predictive distributions over human judgments, enabling both accurate scoring and principled uncertainty quantification in a single forward pass. We evaluate our BEiTScore++ metric across multiple benchmarks with human-annotated ratings, showing that it preserves the ranking correlation of BEiTScore, while producing intervals whose width tracks the prediction error more closely than Monte Carlo dropout on most benchmarks. The estimated uncertainty also grows under image degradation and on out-of-domain artwork images, making BEiTScore++ a practical tool for uncertainty-aware evaluation of image-caption alignment.

Figures & tables

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model

    May 20, 2026Gonçalo Gomes, Bruno Martins, Chrysoula ZervaVLM EvaluationImage Captioning

  2. DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

    Oct 2, 2026Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes +2Text-Image AlignmentT2I Generation Evaluation

  3. MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation

    May 7, 2026Shichao Kan, Xuyang Zhang, Haojie Zhang +7Cross-Modal AlignmentVision-Language Grounding