Multi-camera driving question answering provides a discrete test of vision-language models, but post-training gains can reflect changed selected answers or improved answer extraction. We investigate this distinction within the Gemma model family using archived, task-linked WaymoQA evaluations. A zero-shot diagnostic on 5,519 test questions spans effective-small, mixture-of-experts, unified, and dense checkpoints, with serving differences disclosed rather than interpreted as architecture effects. The strongest controlled evidence is a matched-harness Gemma-4-12B base/GRPO pair: accuracy rises from 73.78% to 75.43%, a gain of 1.65 percentage points with a scene-cluster bootstrap interval of [+0.84, +2.41]. There are 215 improvements and 124 regressions; 213 improvements start from an extractable but incorrect base answer, and only two recover previously unextractable responses. Crossing correctness with extraction state shows why correct noncompliant outputs cannot be counted as prediction errors. Category counts reveal a critical safety asymmetry: routine perception gains (+5.93 pp) mask a regression in safety-critical hazard prediction (-1.82 pp), demonstrating that unweighted outcome rewards can trade off rare edge cases for aggregate accuracy. Reward saturation describes sampled training dynamics rather than an architectural ceiling. A separate archived Gemma-31B thinking comparison yields no detected average improvement, with uncertainty that does not establish equivalence. These results support a modest answer-selection gain for the recorded tuning configuration and a reproducible diagnostic accounting, without attributing corrected labels to specific grounding mechanisms or claiming superiority over matched supervised tuning.
Figures & tables
Checkpoint
Recorded service
Correct
Answered
Acc. all (%)
Gemma-4-E4B
Ollama
2,998
5,519
54.32 [E15]
Gemma-4-26B-A4B
Ollama
3,743
5,519
67.82 [E14]
Gemma-4-12B
HF / NF4
4,072
5,515
73.78 [E1]
Gemma-4-31B
API-compatible endpoint
4,261
5,518
77.21 [E13]
Table 1: Recorded zero-shot Gemma-family results on 5,519 requested items. “Answered” denotes an extracted prediction in these artifacts. Precision is recorded as NF4 for the matched 12B evaluator; it is not authenticated for the other services. The ladder is descriptive and does not isolate model size or architecture.
Outcome
Base
GRPO
Strict, correct
3,767
3,922
Strict, wrong
1,364
1,297
Fallback, correct
305
241
Fallback, wrong
79
59
Unextractable
4
0
Total
5,519
5,519
Table 2: Gemma-12B correctness crossed with archived extraction state. [E1–E2] Correct fallback answers remain correct, despite violating the requested format. The five outcomes sum to the full denominator.
Figure 1: Observed response outcomes for the matched Gemma pair. [E1–E2] Accuracy and strict compliance measure overlapping properties. Exact small-segment counts are in Table 2 ; no segment diagnoses a visual failure mechanism.
Category
n
Base correct
GRPO correct
Δ (pp)
Counterfactual
179
106
111
+2.79
Normal Perception & Prediction
691
459
500
+5.93
Normal Planning
258
190
193
+1.16
Normal scene Understanding
768
532
529
-0.39
Normal scene caption
135
106
111
+3.70
Safety-critical Perception
253
165
167
+0.79
Table 3: Gemma-12B performance across WaymoQA categories on 5,519 test items. [E1–E2] Net gains cluster in routine perception, while safety-critical prediction exhibits a regression.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Step-indexed Gemma-12B sampled-training traces. [E3] Within-group homogeneous rewards may occur at poor or high reward and do not imply an architectural ceiling.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Reasoning-enabled model derivatives must turn intermediate deliberation into a usable answer within a deployment budget. We study that transition in an Ornith checkpoint adapted to multi-camera driving question answering with verifiable rewards. Paired archived outputs show that, on a selected 400-question development screen with a 256-token cap, GRPO increases answer extraction from 78.25% to 90.25% and accuracy from 55.50% to 63.75%. Of 39 wrong-to-correct transitions, 33 begin with an unextractable base response. The base's 87 unextractable responses all reach the token cap. Checkpoint-verified reconstruction on those same questions gives a much smaller first-pass accuracy gain at 512 tokens: +0.75 percentage points, versus +8.25 at 256. A later answer-cued continuation is analyzed separately; it helps the base more and narrows the adapter difference. On the full 2,511-question development set, this recovery-enabled pipeline reduces mean generated tokens from 108.6 to 47.1, while the paired accuracy interval includes zero gain. We frame these findings as adaptation of answer delivery under finite budgets, with explicit selection and provenance limits. Ornith's self-scaffolding lineage and newer reasoning-enabled derivatives such as Bonsai motivate the problem, but our measured results concern Ornith alone and do not establish a shared architectural mechanism or cross-model transfer.
T N Rickesh, Pranav Vinodh, Sowmya Kamath S
natkal · National Institute of Technology Karnataka
Multimodal reasoning requires models to combine multiple pieces of evidence into decisions that satisfy all relevant constraints. Evaluations centered on final-answer accuracy leave unclear whether failures arise from reading individual facts or using them together. We study this question through the multi-evidence integration gap, linking local fact recovery to global decision validity on controlled subset-selection and path-finding tasks. Our evaluation combines conditional performance given correct local answers with same-grid local queries, text-oracle inputs, retrieval controls, and constraint ablations. We further examine hidden-state probes and activation replacement across network depth, token positions, and model families. Across the evaluated models, accurate local recovery coexists with substantial global failure. For example, Gemma-3-27B recovers all evaluated isolated item facts correctly but achieves only 19.75% global validity on eight-item visual instances. Probes decode information about solution membership from global-prompt states, while corresponding clean-state replacement recovers failed decisions over model-dependent depth ranges. These findings show that recovering facts and representing solution-related information are insufficient for reliable joint decisions. They motivate evaluations and model improvements that explicitly target how available evidence is combined, alongside advances in perception and model scale.
Yuyang Dai, Yuxia Wang, Zhuohan Xie
Florida State University · INSAIT, Sofia University “St. Kliment Ohridski” · MBZUAI
Multimodal large language models (MLLMs) achieve strong results on visual reasoning benchmarks, but answer accuracy alone does not indicate whether a model relied on the correct visual evidence. This gap is particularly important in multi-view driving scenes used for autonomous driving, where a model can produce a plausible answer while grounding it in the wrong camera view. We introduce a multi-view visual question answering benchmark for evaluating evidence-source identification: given six synchronized NuScenes views and a question, the model must identify the supporting camera view and answer the question. The benchmark contains 122 conflict-centric question-answer pairs from 73 scenes, spanning causality, counterfactual reasoning, and intent prediction. View labels are proposed by an automatic conflict-mining pipeline and manually verified by annotators. We evaluate three settings: camera-view selection, oracle QA given the golden view, and joint prediction in which the model selects a view and answers in one pass. Answers are evaluated in both multiple-choice and free-form formats, using exact match for structured predictions and an LLM judge for free-form responses. By explicitly separating visual-source identification from answer correctness, the benchmark exposes grounding failures that answer-only evaluation misses.