M-Test: Coverage Gaps in Joint-Perturbation Testing of Visual Question Answering
Organizations: Beijing University of Posts and Telecommunications, Shahe Campus, No. 1 Nanfeng Road, Shahe Higher Education Park, Changping District, Beijing, 102206, China
Abstract
A visual question answering (VQA) model can pass every image-only and question-only test yet fail when the same changes occur together. Standard robustness reports average over conditions and therefore cannot say which accepted requests remain vulnerable. We introduce M3-Test, a coverage audit that aligns each clean input with its single-modality variants and a complete 5x5 family, 4x4 strength joint library, so that any reduced test suite can be replayed on the same requests. Across 2.6 million responses from four vision-language models, joint failures affect 26-29% of the Qwen questions that pass all 40 single-modality tests, although fewer than 0.6% of their joint records fail. Passing the 25 maximum-strength corners does not close this gap: 11-14% of the accepted questions still fail elsewhere in the library. Re-executing the archived failures reproduces about 90% of them, against a 0.3-0.6% flip rate among matched passing controls, and re-checking the most fragile failure records supports 80-89% of them. On the evidence examined, neither execution noise nor grading error can fully account for the gap. The same aligned grids make previous evaluations reusable. On held-out images, a risk order learned from other models discovers about 80% of affected Qwen questions within 25 additional queries, compared with 65-68% for maximum-first and 53-57% for pairwise covering arrays. M3-Test thus turns a passing robustness suite into a request-level account of what remains untested, supporting acceptance, regression review and test allocation.
Figures & tables
| Model | Clean(%) | Single(%) | Joint(%) | Passed( ) | Event density(%) | Affected(%) [95% interval] |
| BLIP-2 | 41.33 | 40.00 | 38.58 | 145 | 1.048 | 61.38 [53.24, 69.94] |
| LLaVA | 69.13 | 67.90 | 66.63 | 550 | 0.300 | 21.27 [17.84, 24.71] |
| Qwen2.5-VL | 71.47 | 69.38 | 67.28 | 540 | 0.514 | 28.70 [25.14, 32.61] |
| Qwen3-VL | 75.60 | 74.28 | 72.96 | 658 | 0.417 | 25.84 [22.51, 29.48] |
| Model | Evidence / reference | No failure | One failure | At least two failures |
| Qwen2.5-VL | Observed coordinates | 71.30 | 6.85 | 21.85 |
| Qwen2.5-VL | Shared-rate reference | 71.30 | 24.13 | 4.57 |
| Qwen2.5-VL | Distinct changed inputs | 71.30 | 6.85 | 21.85 |
| Qwen3-VL | Observed coordinates | 74.16 | 5.47 | 20.36 |
| Qwen3-VL | Shared-rate reference | 74.16 | 22.18 | 3.66 |
| Qwen3-VL | Distinct changed inputs | 74.16 | 5.78 | 20.06 |
| Target | Uniform(%) [95% interval] | Maximum- first(%) | Covering array(%) | Same-model history(%) | Other-model history(%) | History–maximum difference [95% interval], pp |
| BLIP-2 | 45.65 [39.77, 51.60] | 60.67 | 51.26 | 70.04 | 65.92 | +5.24 [-4.10, +15.09] |
| LLaVA | 40.32 [36.08, 44.66] | 67.52 | 55.36 | 77.78 | 78.63 | +11.11 [+2.12, +20.13] |
| Qwen2.5-VL | 43.18 [38.67, 47.82] | 65.16 | 52.65 | 81.29 | 80.00 | +14.84 [+8.12, +22.08] |
| Qwen3-VL | 43.71 [39.65, 47.66] | 67.65 | 57.05 | 80.78 | 80.20 | +12.55 [+6.71, +18.20] |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | Operator | control | Deterministic realization |
| Gaussian blur | Gaussian blur | radius | Pillow GaussianBlur; clean images are returned unchanged at . |
| Gaussian noise | Additive normal noise | intensity units | NumPy seeded generator; add in the domain, clip, and cast to uint8. |
| JPEG | Lossy re-encoding | quality | Pillow JPEG encoding; the perturbed asset remains JPEG at the selected quality. |
| Brightness | Global luminance factor | Pillow brightness factor ; sign is selected deterministically from the asset seed. | |
| Contrast | Global contrast factor | Pillow contrast factor ; sign uses a decorrelated asset seed. | |
| Typo | Token-level edit | One internal character is swapped, duplicated, dropped, or case-toggled; protected tokens are skipped. |
| Model | Scoring and screening | Passed( ) | Affected(%) | Residual after corners(%) |
| Qwen2.5-VL | Combined | 540 | 28.70 | 14.06 |
| Qwen2.5-VL | Semantic requalified | 571 | 29.77 | 15.40 |
| Qwen2.5-VL | Semantic fixed-cohort | 540 | 28.15 | 13.97 |
| Qwen3-VL | Combined | 658 | 25.84 | 11.43 |
| Qwen3-VL | Semantic requalified | 659 | 25.80 | 11.41 |
| Qwen3-VL | Semantic fixed-cohort | 658 | 25.84 | 11.43 |
| Model | Passed( ) | Eligible records | Failed records | Affected questions | Affected images |
| BLIP-2 | 145 | 29636 | 605 | 89 | 81 |
| LLaVA | 550 | 121180 | 660 | 117 | 114 |
| Qwen2.5-VL | 540 | 119610 | 1066 | 155 | 153 |
| Qwen3-VL | 658 | 147300 | 1089 | 170 | 162 |
| Model | Failures reproduced(%) | Matched controls flipped(%) | Affected questions still failing | Failures with anchors correct(%) | Corner-missed rediscovered |
| Qwen2.5-VL | 89.83 (998/1,111) | 0.27 (3/1,111) | 97.42 (151/155) | 84.43 (938/1,111) | 59/63 (59) |
| Qwen3-VL | 90.44 (993/1,098) | 0.64 (7/1,098) | 94.71 (161/170) | 85.61 (940/1,098) | 55/63 (52) |
| Maximum corners | Diagonals | |||||
| Model | Residual( ) | Accepted( ) | Risk(%) | Residual( ) | Accepted( ) | Risk(%) |
| BLIP-2 | 41 | 97 | 42.27 | 24 | 80 | 30.00 |
| LLaVA | 42 | 475 | 8.84 | 19 | 452 | 4.20 |
| Qwen2.5-VL | 63 | 448 | 14.06 | 34 | 419 | 8.11 |
| Qwen3-VL | 63 | 551 | 11.43 | 37 | 525 | 7.05 |
| Model | View | Image retention | Text retention | Joint retention |
| BLIP-2 | Combined | 90.31 [88.71, 91.82] | 88.61 [87.32, 89.79] | 97.75 [97.47, 98.03] |
| LLaVA | Combined | 98.61 [98.19, 99.01] | 97.48 [96.88, 98.00] | 99.78 [99.70, 99.86] |
| Qwen2.5-VL | Combined | 96.10 [95.09, 96.96] | 97.97 [97.43, 98.46] | 99.38 [99.18, 99.54] |
| Qwen3-VL | Combined | 98.28 [97.72, 98.76] | 98.34 [97.88, 98.75] | 99.75 [99.66, 99.84] |
| BLIP-2 | Semantic | 90.79 [89.34, 92.20] | 90.02 [88.87, 91.06] | 97.84 [97.57, 98.10] |
| LLaVA | Semantic | 98.33 [97.74, 98.80] | 97.32 [96.73, 97.85] | 99.72 [99.61, 99.81] |
| Target | Same-model calibration | Other-three calibration | Deployment screen + joints |
| BLIP-2 | 61,751 | 419,542 | 10,383 |
| LLaVA | 133,511 | 347,782 | 12,408 |
| Qwen2.5-VL | 132,183 | 349,110 | 12,358 |
| Qwen3-VL | 153,847 | 327,446 | 12,948 |
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 1,055 papers