cs.CVNov 17, 2026

M3^3-Test: Coverage Gaps in Joint-Perturbation Testing of Visual Question Answering

Authors: Enhui Wan, Xiaowan Wang

Organizations: Beijing University of Posts and Telecommunications, Shahe Campus, No. 1 Nanfeng Road, Shahe Higher Education Park, Changping District, Beijing, 102206, China

Abstract

A visual question answering (VQA) model can pass every image-only and question-only test yet fail when the same changes occur together. Standard robustness reports average over conditions and therefore cannot say which accepted requests remain vulnerable. We introduce M3-Test, a coverage audit that aligns each clean input with its single-modality variants and a complete 5x5 family, 4x4 strength joint library, so that any reduced test suite can be replayed on the same requests. Across 2.6 million responses from four vision-language models, joint failures affect 26-29% of the Qwen questions that pass all 40 single-modality tests, although fewer than 0.6% of their joint records fail. Passing the 25 maximum-strength corners does not close this gap: 11-14% of the accepted questions still fail elsewhere in the library. Re-executing the archived failures reproduces about 90% of them, against a 0.3-0.6% flip rate among matched passing controls, and re-checking the most fragile failure records supports 80-89% of them. On the evidence examined, neither execution noise nor grading error can fully account for the gap. The same aligned grids make previous evaluations reusable. On held-out images, a risk order learned from other models discovers about 80% of affected Qwen questions within 25 additional queries, compared with 65-68% for maximum-first and 53-57% for pairwise covering arrays. M3-Test thus turns a passing robustness suite into a request-level account of what remains untested, supporting acceptance, regression review and test allocation.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems

    May 19, 2026Quanxing Xu, Yuhao Tian, Ling Zhou +4Visual Question AnsweringAutomated Software Testing

  2. VLTest: Finding Joint Image-Text Failures in Vision-Language Models

    Oct 22, 2026Bin Duan, Jintao Lin, Guowei YangVLM EvaluationMutation Testing

  3. VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence

    May 20, 2026Mozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu +5VLM EvaluationVisual Question Answering