cs.AIMar 13, 2026
SaveBeyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation
Organizations: Dartmouth College, Hanover, NH, USA
Abstract
We introduce CRYSTAL (Clear Reasoning via Yielded Steps, Traceability, and Logic), a diagnostic benchmark with 6,372 instances that evaluates multimodal reasoning through verifiable intermediate steps. We propose two complementary metrics: Match F1, which scores step-level precision and recall via semantic similarity matching, and Ordered Match F1, which further penalizes disordered reasoning chains. References are constructed through a Delphi-inspired pipeline in which four independent MLLMs generate trajectories, which are then aggregated via semantic clustering and validated through human quality gates. Evaluation of 20 MLLMs, including commercial frontier systems not used during benchmark construction, reveals systematic failures that are invisible to answer accuracy: universal cherry-picking (precision far exceeds recall), non-monotonic scaling trade-offs, and disordered reasoning in which no competitive model preserves more than 60% of matched steps in the correct order. Beyond evaluation, we propose the Causal Process Reward (CPR), a multiplicative reward that couples answer correctness with step-level alignment, and CPR-Curriculum, which progressively increases reasoning difficulty during training. CPR-Curriculum achieves a 32% improvement in Match F1 via GRPO where additive reward strategies fail, improving reasoning without manual step annotation.
Figures & tables
Figure 1: The lucky guess problem. LLaVA-v1.6-7B answers correctly (C) but contradicts itself by claiming the middle console is larger while selecting it as smallest. Previous benchmarks score 100%; CRYSTAL compares the model’s predicted steps against reference reasoning steps via Match F1 (0.15), exposing flawed reasoning.
| Statistic | Value |
|---|---|
| Total examples | 6,372 |
| Avg. steps | 11.6 |
| Step range | 3–42 |
| Easy / Med / Hard | 48.5 / 44.1 / 7.4% |
| Annotation | Multi-agent + human |
Table 1: Key statistics of CRYSTAL.
Figure 2: CRYSTAL spans diverse multimodal reasoning scenarios. Three examples: (Left) RealWorldQA (spatial understanding), (Middle) MMVP (fine-grained perception), (Right) ScienceQA (multi-hop logic). Parentheses give the reference step count per example.
| Model | Params | Accuracy | Match F1 | P | R | Steps | LIS | Ord. F1 |
|---|---|---|---|---|---|---|---|---|
| Commercial MLLMs | ||||||||
| GPT-5 [ 35 ] | n/a | 57.99% | 0.612 | 0.925 | 0.479 | 5.29 | 0.636 | 0.539 |
| GPT-5-mini [ 35 ] | n/a | 55.59% | 0.773 | 0.978 | 0.669 | 7.57 | 0.560 | 0.670 |
| GPT-5.2 Instant [ 35 ] | n/a | 47.35% | 0.564 | 0.974 | 0.416 | 4.64 | 0.648 | 0.501 |
| Gemini 2.5 Flash [ 14 ] | n/a | 53.95% | 0.673 | 0.701 | 0.765 | 17.10 | 0.584 | 0.579 |
| Qwen Family | ||||||||
Table 2: Evaluation of 20 MLLMs on CRYSTAL. Match F1 (all-distilroberta-v1, ): step-level reasoning quality; P/R: step precision/recall; LIS: fraction of matched steps in correct order; Ord. F1 ( ): order-penalized Match F1. Top three per column in green , blue, yellow.
Figure 3: Ablation: encoder and threshold. 4 encoders across 5 thresholds, averaged over all models. (a) all-distilroberta-v1 achieves the highest Match F1 (+4.9pp at ). (b–c) higher thresholds reduce precision and recall for most encoders, while DistilRoBERTa stays stable. We select .
| Similarity Band | Pairs | Agreement | Note |
|---|---|---|---|
| 33 | 100.0% | No false matches | |
| 14 | 100.0% | No false matches | |
| 19 | 68.4% | Borderline zone | |
| 25 | 64.0% | Borderline zone | |
| 9 | 88.9% | Clear matches | |
| Overall | 100 | 84.0% |
Table 3: Human agreement with embedding-based step matching. Agreement between the encoder’s decisions and a blinded human annotator on 100 step pairs stratified by cosine similarity. Agreement is perfect below the operating threshold ( ); disagreements stay in the ambiguous borderline zone.
Figure 4: Order metric comparison on a representative subset of 6 models. Both metrics rise for weak models generating few steps, but LIS ratio provides wider inter-group discrimination (0.56–0.85 vs. 0.58–0.81), clearly exposing trivially ordered few-step outputs.
| Configuration | Acc (%) | Match F1 | Prec | Rec | Steps | LIS | Ord. F1 |
|---|---|---|---|---|---|---|---|
| Qwen2.5-VL-3B | |||||||
| Baseline | 39.85 | 0.480 | 0.898 | 0.347 | 3.73 | 0.723 | 0.434 |
| CPR-Curriculum | 47.52 | 0.633 | 0.963 | 0.493 | 5.10 | 0.626 | 0.560 |
| +7.67 | +0.153 | +0.065 | +0.146 | +1.37 | 0.097 | +0.126 | |
| InternVL3.5-4B | |||||||
| Baseline | 37.61 | 0.432 | 0.895 | 0.325 | 3.75 | 0.775 | 0.387 |
Table 4: GRPO with step-level rewards. CPR-Curriculum improves both accuracy and Match F1 over the base model on two architecturally distinct backbones, using identical reward weights and protocol. All metrics use all-distilroberta-v1, .
Figure 5: Training dynamics. Left: Match F1; Right: Accuracy. Composite and Answer-Only diverge to NaN gradients by step 1,500 (Composite collapses at 600), while CPR variants train stably through 2,800 steps.
| Strategy | Acc (%) | Match F1 | Prec | Rec | LIS | Ord. F1 |
|---|---|---|---|---|---|---|
| Baseline | 39.85 | 0.480 | 0.898 | 0.347 | 0.723 | 0.434 |
| Composite | 44.92 | 0.426 | 0.983 | 0.284 | 0.743 | 0.392 |
| Answer-Only | 44.30 | 0.429 | 0.803 | 0.308 | 0.700 | 0.380 |
| CPR | 41.40 | 0.633 | 0.975 | 0.489 | 0.631 | 0.560 |
| CPR-Curriculum | 47.52 | 0.633 | 0.963 | 0.493 | 0.626 | 0.560 |
Table 5: Reward strategy comparison. Best checkpoint per strategy on CRYSTAL test set. CPR strategies use weights ; a full weight ablation is presented in Table 6 . Composite and Answer-Only reach comparable accuracy but fail to improve reasoning (F1 0.43). CPR-based strategies improve both.
| Val Performance | Reward Signal | Stability | ||||||
|---|---|---|---|---|---|---|---|---|
| Acc (%) | M-F1 | Rec | Discrim. | (F1) | KL max | |||
| 0.80 | 0.20 | 81.79 | 0.676 | 0.553 | 0.865 | 0.338 | 1.274 | |
| 0.70 | 0.30 | 82.61 | 0.731 | 0.627 | 0.797 | 0.410 | 0.456 | |
| 0.65 | 0.35 | 82.70 | 0.728 | 0.634 | 0.763 | 0.448 | 0.372 | |
| 0.50 | 0.50 | 79.39 | 0.730 | 0.629 | 0.662 | 0.563 | 1.218 | |
| 0.35 | 0.65 | 78.53 | 0.736 | 0.642 | 0.560 | 0.670 | 1.684 | |
Table 6: Reward weight sensitivity. Six configurations on a 15% held-out split (4,546 samples). Both extremes are unstable ( ); KL max is U-shaped with its minimum at , which also gives the highest accuracy and (F1) in the stable range.
Figure S1 : Complete multi-agent reasoning generation workflow. The pipeline processes input triplets through four phases with two quality gates. Phase 1 generates candidate steps from four independent MLLMs. Phase 2 clusters steps via connected components and selects representative medoids. Phase 3 validates with a fifth MLLM; failures trigger Iteration Loop 1. Phase 4 applies human review; failures trigger Iteration Loop 2 (full restart). This iterative refinement implements Delphi-style consensus building [ 33 ] .
Figure S2 : Semantic clustering visualization (Phase 2). Scatter plot showing 48 candidate reasoning steps from four VLM generators clustered via connected components. Each colored point represents a candidate step embedded in 768-dimensional space and projected to 2D using t-SNE. Points are colored by cluster membership, with 8 clusters formed where edges connect steps with cosine similarity . Red stars mark cluster medoids, selected to minimize average dissimilarity within each cluster. Each cluster contains multiple paraphrastic variations (average 6 steps per cluster), demonstrating how semantically equivalent steps naturally group together. This example illustrates redundancy elimination ( steps, 83.3% reduction) while preserving distinct reasoning operations.
Figure S3 : Human validation interface (Phase 4). Screenshot of the Gradio-based annotation tool used for quality gate verification. (a) The left panel displays the input triplet: image, question text, multiple-choice options (if present), ground-truth answer, and dataset source. Navigation controls (Prev/Next buttons and index slider) allow annotators to traverse the dataset sequentially or jump to specific examples. (b) The right panel presents the candidate reasoning steps produced by Phases 1–3, with radio buttons to mark correctness and a text area for editing steps. Annotators can save edits in memory and batch-export corrected examples to a new dataset. This interface enables efficient human review of 6,372 validation examples.
Figure S4 : Dataset generation prompt (RealWorldQA variant). The four Phase 1 generators receive the question and ground-truth answer and output a Python list of atomic evidence checks. Similar templates are adapted per source dataset.
Figure S5 : Model evaluation prompt. Unlike generation, this prompt provides only the question (no answer) and enforces JSON output. Applied to all 20 models with greedy decoding ( ).
Figure S6 : Perfect reasoning example. The model achieves F1 = 1.0 by systematically identifying visual elements, verifying illumination state, and concluding with the correct answer. All 8 predicted steps match the 8 references, demonstrating comprehensive reasoning without shortcuts.
Figure S7 : Sound reasoning structure with numerical errors (F1 = 0.857). Gemma3-12B covers all 6 reference step types (Recall = 1.000) but two steps contain numerical errors (Precision = 0.750): miscalculating the average as 9.5% (true: 6.294%) and misreading 2007 as 13.5% (true: 5.633%). These perceptual errors propagate through valid comparison logic, yielding answer “2” instead of “3”. Recall confirms correct algorithmic structure; Precision detects execution failures in visual data extraction.
Figure S8 : Correct answer with cherry-picked reasoning (Precision = 1.0, Recall = 0.194). The model achieves correct answer (A) with perfect precision by generating 6 high-confidence calculation steps. However, it skips 25 reference steps detailing midpoint identification, diagonal construction, and triangle congruence proofs. This exemplifies universal cherry-picking: high precision (1.0) through selective reasoning, but low recall (0.194) by omitting intermediate logical transitions. The model answers correctly through efficient shortcuts rather than comprehensive verification.
Figure S9 : Complete failure with fabricated reasoning (F1 = 0.214). MiniCPM-v2.6-8B commits a fundamental geometric error: assuming triangle ABD is equilateral when only (two sides equal does NOT make a triangle equilateral; requires three equal sides). This false premise propagates through all subsequent reasoning. Step 3 fabricates connections to “regular hexagon” concepts that are irrelevant to the problem. The correct solution requires recognizing that implies D is the circumcenter, leading to . This represents the 14.4% of predictions classified as “Failure” where both perception and logical reasoning break down catastrophically.
Figure S10 : CPR-Curriculum improves both answer and reasoning on Qwen2.5-VL-3B. The baseline outputs only 2 steps and answers “A” (wrong format). After CPR training, the model produces 8 steps covering volume comparison, composition verification, and correct conclusion (F1: 0.333 0.889, Recall: 0.200 0.800).
Figure S11 : CPR-Curriculum regression on a geometry problem. The baseline produces 10 detailed steps with explicit calculations (diameter width length area) and answers correctly (F1: 0.870). After CPR training, the model generates only 6 generic steps without showing intermediate computations, misapplying the 2:1 ratio and arriving at 100 instead of 200 (F1: 0.632). This illustrates that CPR can encourage step generation at the expense of computational precision on multi-step math problems.
Figure S12 : CPR-Curriculum enables systematic elimination on InternVL3.5-4B. The baseline jumps to “E” without verifying all constraints (F1: 0.667). After CPR training, the model eliminates each option against all three constraints, covering all 10 reference types (R: 1.000) and arriving at the correct answer B (F1: 0.741).
Figure S13 : CPR-Curriculum regression on a combinatorial problem. The baseline correctly identifies the optimal 3-swap solution with high F1 (0.889). After CPR training, the model covers the same reasoning types (R: 1.000) but overcounts the required swaps, answering 4 instead of 3. Both models follow the same algorithmic structure (identify compare count minimize), yet the CPR model fails at the final counting step, suggesting that step-level supervision improves reasoning coverage but does not guarantee computational accuracy on combinatorial tasks.
| Qwen2.5-VL-3B | InternVL3.5-4B | |||
| Hyperparameter | Phase 1 | Phase 2 | Phase 1 | Phase 2 |
| Steps | 1,400 | 2,800 | 200 | 200 |
| Learning rate | ||||
| Per-device batch size | 4 | 5 | 5 | 5 |
| Candidates | 2 | 5 | 5 | 5 |
| Gradient accumulation | 2 | 2 | 2 | 2 |
Table S1 : Per-model GRPO hyperparameters. Phase 1 trains answer-only; Phase 2 introduces CPR. All unlisted hyperparameters are shared (see text).
Figure S14 : Mathvision example from CRYSTAL benchmark (Sample 1265). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S15 : Mathvision example from CRYSTAL benchmark (Sample 899). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S16 : Mathvision example from CRYSTAL benchmark (Sample 1947). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S17 : Plotqa example from CRYSTAL benchmark (Sample 6193). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S18 : Scienceqa example from CRYSTAL benchmark (Sample 4306). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S19 : Scienceqa example from CRYSTAL benchmark (Sample 4116). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S20 : Scienceqa example from CRYSTAL benchmark (Sample 5429). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S21 : Scienceqa example from CRYSTAL benchmark (Sample 4031). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S22 : Realwordqa example from CRYSTAL benchmark (Sample 714). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S23 : Realwordqa example from CRYSTAL benchmark (Sample 580). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S24 : MMVP example from CRYSTAL benchmark (Sample 5866). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S25 : MMVP example from CRYSTAL benchmark (Sample 6038). Shows input image, question with multiple-choice options (if applicable), ground truth answer, and reference reasoning steps demonstrating the expected step-by-step solution path.
Figure S26 : Scienceqa training example (ID 29750). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S27 : Scienceqa training example (ID 25092). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S28 : Scienceqa training example (ID 24320). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S29 : Scienceqa training example (ID 26538). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S30 : Scienceqa training example (ID 26273). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S31 : Textvqa training example (ID 9690). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S32 : Textvqa training example (ID 6036). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S33 : Textvqa training example (ID 4450). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S34 : Textvqa training example (ID 23353). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
Figure S35 : Textvqa training example (ID 3762). Representative sample from GRPO training dataset showing input image, question with ground truth answer, and expert-generated reference reasoning steps. These step-level annotations enable reinforcement learning with Match F1 rewards without requiring manual step annotation during deployment.
More in this topic
Reasoning Evaluation
Reasoning evaluation measures whether AI systems solve multi-step problems reliably, use evidence and valid intermediate steps, and generalize beyond benchmarks.
See all 51 papers