We introduce CRYSTAL (Clear Reasoning via Yielded Steps, Traceability, and Logic), a diagnostic benchmark with 6,372 instances that evaluates multimodal reasoning through verifiable intermediate steps. We propose two complementary metrics: Match F1, which scores step-level precision and recall via semantic similarity matching, and Ordered Match F1, which further penalizes disordered reasoning chains. References are constructed through a Delphi-inspired pipeline in which four independent MLLMs generate trajectories, which are then aggregated via semantic clustering and validated through human quality gates. Evaluation of 20 MLLMs, including commercial frontier systems not used during benchmark construction, reveals systematic failures that are invisible to answer accuracy: universal cherry-picking (precision far exceeds recall), non-monotonic scaling trade-offs, and disordered reasoning in which no competitive model preserves more than 60% of matched steps in the correct order. Beyond evaluation, we propose the Causal Process Reward (CPR), a multiplicative reward that couples answer correctness with step-level alignment, and CPR-Curriculum, which progressively increases reasoning difficulty during training. CPR-Curriculum achieves a 32% improvement in Match F1 via GRPO where additive reward strategies fail, improving reasoning without manual step annotation.
Figures & tables
Figure 1: The lucky guess problem. LLaVA-v1.6-7B answers correctly (C) but contradicts itself by claiming the middle console is larger while selecting it as smallest. Previous benchmarks score 100%; CRYSTAL compares the model’s predicted steps against reference reasoning steps via Match F1 (0.15), exposing flawed reasoning.
Statistic
Value
Total examples
6,372
Avg. steps
11.6
Step range
3–42
Easy / Med / Hard
48.5 / 44.1 / 7.4%
Annotation
Multi-agent + human
Table 1: Key statistics of CRYSTAL.
Figure 2: CRYSTAL spans diverse multimodal reasoning scenarios. Three examples: (Left) RealWorldQA (spatial understanding), (Middle) MMVP (fine-grained perception), (Right) ScienceQA (multi-hop logic). Parentheses give the reference step count per example.
Model
Params
Accuracy
Match F1
P
R
Steps
LIS
Ord. F1
Commercial MLLMs
GPT-5 [ 35 ]
n/a
57.99%
0.612
0.925
0.479
5.29
0.636
0.539
GPT-5-mini [ 35 ]
n/a
55.59%
0.773
0.978
0.669
7.57
0.560
0.670
GPT-5.2 Instant [ 35 ]
n/a
47.35%
0.564
0.974
0.416
4.64
0.648
0.501
Gemini 2.5 Flash [ 14 ]
n/a
53.95%
0.673
0.701
0.765
17.10
0.584
0.579
Qwen Family
Table 2: Evaluation of 20 MLLMs on CRYSTAL. Match F1 (all-distilroberta-v1, τ=0.35 ): step-level reasoning quality; P/R: step precision/recall; LIS: fraction of matched steps in correct order; Ord. F1 ( α=0.3 ): order-penalized Match F1. Top three per column in green , blue, yellow.
Figure 3: Ablation: encoder and threshold. 4 encoders across 5 thresholds, averaged over all models. (a) all-distilroberta-v1 achieves the highest Match F1 (+4.9pp at τ=0.35 ). (b–c) higher thresholds reduce precision and recall for most encoders, while DistilRoBERTa stays stable. We select τ=0.35 .
Similarity Band
Pairs
Agreement
Note
<0.20
33
100.0%
No false matches
[0.20,0.35)
14
100.0%
No false matches
[0.35,0.50)
19
68.4%
Borderline zone
[0.50,0.70)
25
64.0%
Borderline zone
≥0.70
9
88.9%
Clear matches
Overall
100
84.0%
κ=0.534
Table 3: Human agreement with embedding-based step matching. Agreement between the encoder’s decisions and a blinded human annotator on 100 step pairs stratified by cosine similarity. Agreement is perfect below the operating threshold ( τ=0.35 ); disagreements stay in the ambiguous borderline zone.
Figure 4: Order metric comparison on a representative subset of 6 models. Both metrics rise for weak models generating few steps, but LIS ratio provides wider inter-group discrimination (0.56–0.85 vs. 0.58–0.81), clearly exposing trivially ordered few-step outputs.
Configuration
Acc (%)
Match F1
Prec
Rec
Steps
LIS
Ord. F1
Qwen2.5-VL-3B
Baseline
39.85
0.480
0.898
0.347
3.73
0.723
0.434
CPR-Curriculum
47.52
0.633
0.963
0.493
5.10
0.626
0.560
Δ
+7.67
+0.153
+0.065
+0.146
+1.37
− 0.097
+0.126
InternVL3.5-4B
Baseline
37.61
0.432
0.895
0.325
3.75
0.775
0.387
Table 4: GRPO with step-level rewards. CPR-Curriculum improves both accuracy and Match F1 over the base model on two architecturally distinct backbones, using identical reward weights and protocol. All metrics use all-distilroberta-v1, τ=0.35 .
Figure 5: Training dynamics. Left: Match F1; Right: Accuracy. Composite and Answer-Only diverge to NaN gradients by step 1,500 (Composite collapses at 600), while CPR variants train stably through 2,800 steps.
Strategy
Acc (%)
Match F1
Prec
Rec
LIS
Ord. F1
Baseline
39.85
0.480
0.898
0.347
0.723
0.434
Composite
44.92
0.426
0.983
0.284
0.743
0.392
Answer-Only
44.30
0.429
0.803
0.308
0.700
0.380
CPR
41.40
0.633
0.975
0.489
0.631
0.560
CPR-Curriculum
47.52
0.633
0.963
0.493
0.626
0.560
Table 5: Reward strategy comparison. Best checkpoint per strategy on CRYSTAL test set. CPR strategies use weights aw=0.65,sw=0.35 ; a full weight ablation is presented in Table 6 . Composite and Answer-Only reach comparable accuracy but fail to improve reasoning (F1 ≤ 0.43). CPR-based strategies improve both.
Val Performance
Reward Signal
Stability
aw
sw
Acc (%)
M-F1
Rec
Discrim.
r (F1)
KL max
∇
0.80
0.20
81.79
0.676
0.553
0.865
0.338
1.274
↑
0.70
0.30
82.61
0.731
0.627
0.797
0.410
0.456
↓
0.65
0.35
82.70
0.728
0.634
0.763
0.448
0.372
↓
0.50
0.50
79.39
0.730
0.629
0.662
0.563
1.218
↑
0.35
0.65
78.53
0.736
0.642
0.560
0.670
1.684
↑
Table 6: Reward weight sensitivity. Six configurations on a 15% held-out split (4,546 samples). Both extremes are unstable ( ∇↑ ); KL max is U-shaped with its minimum at aw=0.65 , which also gives the highest accuracy and r (F1) in the stable range.
Figure S1 : Complete multi-agent reasoning generation workflow. The pipeline processes input triplets (Q,I,A) through four phases with two quality gates. Phase 1 generates candidate steps from four independent MLLMs. Phase 2 clusters steps via connected components and selects representative medoids. Phase 3 validates with a fifth MLLM; failures trigger Iteration Loop 1. Phase 4 applies human review; failures trigger Iteration Loop 2 (full restart). This iterative refinement implements Delphi-style consensus building [ 33 ] .
Figure S2 : Semantic clustering visualization (Phase 2). Scatter plot showing 48 candidate reasoning steps from four VLM generators clustered via connected components. Each colored point represents a candidate step embedded in 768-dimensional space and projected to 2D using t-SNE. Points are colored by cluster membership, with 8 clusters formed where edges connect steps with cosine similarity ≥τ=0.70 . Red stars mark cluster medoids, selected to minimize average dissimilarity within each cluster. Each cluster contains multiple paraphrastic variations (average 6 steps per cluster), demonstrating how semantically equivalent steps naturally group together. This example illustrates redundancy elimination ( 48→8 steps, 83.3% reduction) while preserving distinct reasoning operations.
Figure S3 : Human validation interface (Phase 4). Screenshot of the Gradio-based annotation tool used for quality gate verification. (a) The left panel displays the input triplet: image, question text, multiple-choice options (if present), ground-truth answer, and dataset source. Navigation controls (Prev/Next buttons and index slider) allow annotators to traverse the dataset sequentially or jump to specific examples. (b) The right panel presents the candidate reasoning steps produced by Phases 1–3, with radio buttons to mark correctness and a text area for editing steps. Annotators can save edits in memory and batch-export corrected examples to a new dataset. This interface enables efficient human review of 6,372 validation examples.
Figure S4 : Dataset generation prompt (RealWorldQA variant). The four Phase 1 generators receive the question and ground-truth answer and output a Python list of atomic evidence checks. Similar templates are adapted per source dataset.
Figure S5 : Model evaluation prompt. Unlike generation, this prompt provides only the question (no answer) and enforces JSON output. Applied to all 20 models with greedy decoding ( T=0.0 ).
Figure S6 : Perfect reasoning example. The model achieves F1 = 1.0 by systematically identifying visual elements, verifying illumination state, and concluding with the correct answer. All 8 predicted steps match the 8 references, demonstrating comprehensive reasoning without shortcuts.
Figure S7 : Sound reasoning structure with numerical errors (F1 = 0.857). Gemma3-12B covers all 6 reference step types (Recall = 1.000) but two steps contain numerical errors (Precision = 0.750): miscalculating the average as 9.5% (true: 6.294%) and misreading 2007 as 13.5% (true: 5.633%). These perceptual errors propagate through valid comparison logic, yielding answer “2” instead of “3”. Recall confirms correct algorithmic structure; Precision detects execution failures in visual data extraction.
Figure S8 : Correct answer with cherry-picked reasoning (Precision = 1.0, Recall = 0.194). The model achieves correct answer (A) with perfect precision by generating 6 high-confidence calculation steps. However, it skips 25 reference steps detailing midpoint identification, diagonal construction, and triangle congruence proofs. This exemplifies universal cherry-picking: high precision (1.0) through selective reasoning, but low recall (0.194) by omitting intermediate logical transitions. The model answers correctly through efficient shortcuts rather than comprehensive verification.
Figure S9 : Complete failure with fabricated reasoning (F1 = 0.214). MiniCPM-v2.6-8B commits a fundamental geometric error: assuming triangle ABD is equilateral when only AD=BD (two sides equal does NOT make a triangle equilateral; requires three equal sides). This false premise propagates through all subsequent reasoning. Step 3 fabricates connections to “regular hexagon” concepts that are irrelevant to the problem. The correct solution requires recognizing that AD=BD=CD implies D is the circumcenter, leading to ∠BAC=90∘ . This represents the 14.4% of predictions classified as “Failure” where both perception and logical reasoning break down catastrophically.
Figure S10 : CPR-Curriculum improves both answer and reasoning on Qwen2.5-VL-3B. The baseline outputs only 2 steps and answers “A” (wrong format). After CPR training, the model produces 8 steps covering volume comparison, composition verification, and correct conclusion (F1: 0.333 → 0.889, Recall: 0.200 → 0.800).
Figure S11 : CPR-Curriculum regression on a geometry problem. The baseline produces 10 detailed steps with explicit calculations (diameter → width → length → area) and answers correctly (F1: 0.870). After CPR training, the model generates only 6 generic steps without showing intermediate computations, misapplying the 2:1 ratio and arriving at 100 instead of 200 (F1: 0.632). This illustrates that CPR can encourage step generation at the expense of computational precision on multi-step math problems.
Figure S12 : CPR-Curriculum enables systematic elimination on InternVL3.5-4B. The baseline jumps to “E” without verifying all constraints (F1: 0.667). After CPR training, the model eliminates each option against all three constraints, covering all 10 reference types (R: 1.000) and arriving at the correct answer B (F1: 0.741).
Figure S13 : CPR-Curriculum regression on a combinatorial problem. The baseline correctly identifies the optimal 3-swap solution with high F1 (0.889). After CPR training, the model covers the same reasoning types (R: 1.000) but overcounts the required swaps, answering 4 instead of 3. Both models follow the same algorithmic structure (identify → compare → count → minimize), yet the CPR model fails at the final counting step, suggesting that step-level supervision improves reasoning coverage but does not guarantee computational accuracy on combinatorial tasks.
Qwen2.5-VL-3B
InternVL3.5-4B
Hyperparameter
Phase 1
Phase 2
Phase 1
Phase 2
Steps
1,400
2,800
200
200
Learning rate η
3×10−6
5×10−6
5×10−6
5×10−6
Per-device batch size
4
5
5
5
Candidates K
2
5
5
5
Gradient accumulation
2
2
2
2
Table S1 : Per-model GRPO hyperparameters. Phase 1 trains answer-only; Phase 2 introduces CPR. All unlisted hyperparameters are shared (see text).
Figure S14 : Mathvision example from CRYSTAL benchmark (Sample 1265). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S15 : Mathvision example from CRYSTAL benchmark (Sample 899). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S16 : Mathvision example from CRYSTAL benchmark (Sample 1947). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S17 : Plotqa example from CRYSTAL benchmark (Sample 6193). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S18 : Scienceqa example from CRYSTAL benchmark (Sample 4306). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S19 : Scienceqa example from CRYSTAL benchmark (Sample 4116). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S20 : Scienceqa example from CRYSTAL benchmark (Sample 5429). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S21 : Scienceqa example from CRYSTAL benchmark (Sample 4031). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S22 : Realwordqa example from CRYSTAL benchmark (Sample 714). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S23 : Realwordqa example from CRYSTAL benchmark (Sample 580). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S24 : MMVP example from CRYSTAL benchmark (Sample 5866). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S25 : MMVP example from CRYSTAL benchmark (Sample 6038). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S26 : Scienceqa training example (ID 29750). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S27 : Scienceqa training example (ID 25092). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S28 : Scienceqa training example (ID 24320). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S29 : Scienceqa training example (ID 26538). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S30 : Scienceqa training example (ID 26273). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S31 : Textvqa training example (ID 9690). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S32 : Textvqa training example (ID 6036). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S33 : Textvqa training example (ID 4450). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S34 : Textvqa training example (ID 23353). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S35 : Textvqa training example (ID 3762). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Reasoning evaluation measures whether AI systems solve multi-step problems reliably, use evidence and valid intermediate steps, and generalize beyond benchmarks.
Multimodal large language models often generate reasoning chains containing subtle errors that lead to incorrect answers. Current verification approaches have notable limitations. Existing approaches either require expensive labelled supervision with inconsistent cross-task performance or aggregate scores from multiple sources by simple aggregations, missing a key insight: when these scores disagree, that disagreement itself carries important information about whether a reasoning step is truly valid or not. We formalise this as a coupled scoring problem among disparate, frozen verifiers, interpretable as a coordination game with a unique closed-form equilibrium where agreement signals valid steps while disagreement reveals instability. Towards this end, we propose a training-free domain-agnostic step-wise verification approach we call VERDICT: VERification via Disagreement-Informed Coupled Thresholding. To our knowledge, VERDICT is the first training-free verifier that makes the structure of cross-modal disagreement explicit and actionable. It computes consensus scores through a closed-form solution, enabling both disagreement-aware filtering and stability-conscious ranking of reasoning steps. Evaluated across six benchmarks, \method consistently improves over the base model by up to +5.95%, and performs competitively with domain-specific critics that demand extensive supervision, demonstrating that cross-modal agreement provides robust verification signals without task-specific adaptation and Training-Free Verification
Rohit Sinha, Kunal Tilaganji, Tanuja Ganu +3
Indian Institute of Technology, Hyderabad · Microsoft Research
Visual question answering increasingly requires multi-step reasoning. Recent post-training with reinforcement learning under verifiable rewards (RLVR) and Group Relative Policy Optimization (GRPO) can improve multimodal reasoning, but most approaches rely on sparse outcome-only rewards. As a result, they struggle to tell whether an incorrect answer comes from a small mistake late in the reasoning or from an unhelpful trajectory from the start. A common solution is to train a process reward model (PRM) for step-level supervision, but this typically requires large-scale high-quality chain-of-thought annotations and additional training cost. We propose ProcessThinker, a practical post-training pipeline that provides step-level process rewards without training an explicit PRM. ProcessThinker first rewrites reasoning traces into a step-tagged format for cold-start supervised fine-tuning, then applies GRPO with a standard format reward and our rollout-based process reward. Concretely, for each intermediate step, we sample multiple continuations from that step and use the empirical success rate (final-answer verification) as the step reward. This gives dense credit assignment and encourages reasoning steps that more reliably support a correct conclusion, helping reduce inconsistent or self-contradictory progress across steps -- a key issue in logical reasoning. Across four challenging video benchmarks (Video-MMMU, MMVU, VideoMathQA, and LongVideoBench), ProcessThinker consistently improves over the baseline model Qwen3-VL-8B-Instruct
Jingpei Wu, Xiao Han, Weixiang Shen +3
LMU Munich · Harvard University · University of Cambridge +2
Multimodal large language models (MLLMs) have shown promising reasoning abilities, yet evaluating their performance in specialized domains remains challenging. STEM reasoning is a particularly valuable testbed because it provides highly verifiable feedback, but existing benchmarks often permit unimodal shortcuts due to modality redundancy and focus mainly on final-answer accuracy, overlooking the reasoning process itself. To address this challenge, we introduce StepSTEM: a graduate-level benchmark of 283 problems across mathematics, physics, chemistry, biology, and engineering for fine-grained evaluation of cross-modal reasoning in MLLMs. StepSTEM is constructed through a rigorous curation pipeline that enforces strict complementarity between textual and visual inputs. We further propose a general step-level evaluation framework for both text-only chain-of-thought and interleaved image-text reasoning, using dynamic programming to align predicted reasoning steps with multiple reference solutions. Experiments across a wide range of models show that current MLLMs still rely heavily on textual reasoning, with even Gemini 3.1 Pro and Claude Opus 4.6 achieving only 38.29% accuracy. These results highlight substantial headroom for genuine cross-modal STEM reasoning and position StepSTEM as a benchmark for fine-grained evaluation of multimodal reasoning. Source code is available at https://github.com/lll-hhh/STEPSTEM.
Jing Jin, Hao Liu, Yan Bai +9
1Tsinghua University · 3Central South University · 2Meituan Inc +3