Organizations: National Key Laboratory for Novel Software Technology, Nanjing University · Data Science & Artificial Intelligence Research Institute, China Unicom · Unicom Data Intelligence, China Unicom
Reinforcement learning has recently improved the reasoning ability of Large Language Models and Multimodal LLMs, yet prevailing reward designs emphasise final-answer correctness and consequently tolerate process hallucinations--cases where models reach the right answer while misperceiving visual evidence. We address this process-level misalignment with PaLMR, a framework that aligns not only outcomes but also the reasoning process itself. PaLMR comprises two complementary components: a perception-aligned data layer that constructs process-aware reasoning data with structured pseudo-ground-truths and verifiable visual facts, and a process-aligned optimisation layer that constructs a hierarchical reward fusion scheme with a process-aware scoring function to encourage visually faithful chains-of-thought and improve training stability. Experiments on Qwen2.5-VL-7B show that our approach substantially reduces reasoning hallucinations and improves visual reasoning fidelity, achieving state-of-the-art results on HallusionBench while maintaining strong performance on MMMU, MathVista, and MathVerse. These findings indicate that PaLMR offers a principled and practical route to process-aligned multimodal reasoning, advancing the reliability and interpretability of MLLMs.
Figures & tables
Figure 2: Overview of the proposed PaLMR framework. The model adopts a two-layer architecture: (a) the Perception-Aligned Data Layer (PaDLayer) builds process-aware multimodal data with structured pseudo ground truths and verifiable visual facts; and (b) the Process-Aligned Optimization Layer (PaOLayer) integrates perception-aware, answer, and format rewards into GRPO to enforce visually faithful and logically coherent reasoning.
Figure 3: Model-Human alignment ratio in identifying visual perception errors. The evaluation assesses visual perception errors in 100 randomly sampled responses generated from the Geo3K dataset with Qwen2.5-VL-7B, and using Qwen2.5-32B, Qwen3-30B as judgement model.
Model
#Data
MMMU val
HallusionBench
MathVerse*
MMStar
MathVista
GPT-4o [ 18 ]
-
60.0
68.0
-
-
63.8
Gemini2-Flash [ 42 ]
-
70.6
69.4
-
-
70.4
Qwen2.5-VL-72B [ 1 ]
-
68.2
71.4
-
70.8
74.8
Qwen2.5-VL-32B [ 1 ]
-
63.7
72.1
54.3
67.3
74.7
InternVL2.5-8B [ 9 ]
-
56.2
67.4
-
62.9
64.4
MM-Eureka-7B [ 32 ]
15K
55.4
69.5
46.6
64.6
73.0
Table 1: Performance comparison of MLLMs on a suite of out-of-domain benchmarks. Accuracy are reported for all benchmarks. MathVerse use vision only subset. Open-source models are evaluated using VLMEvalkit, and R1-style reasoning models are used the same template as it training if provided.
Figure 4: Data distribution after our PaDLayer data filtering . 19 distinct sub-domains are selected, and 4728 samples are finally used to generate the training dataset.
Figure 5: Qualitative comparison of reasoning chains between baseline models and PaLMR across different domains.
Model
MMMU
Hallusion
MathVerse
MMStar
MathVista
Qwen2.5-VL-3B
49.3
-
32.4
55.0
63.3
+ GRPO
53.3
-
34.0
56.7
62.8
PaLMR-3B
53.9
-
38.8
57.2
64.0
Qwen2.5-VL-7B
56.4
63.8
42.6
64.3
68.2
+ GRPO
57.8
66.7
45.9
66.0
74.1
PaLMR-7B
59.3
70.9
47.5
67.1
73.8
Table 2: Comparison of model performance across different scales on out-of-domain benchmarks. † indicates inference via vLLM.
Figure 6: (a) Comparison of average response length on the training set across different reward settings. (b) Comparison of accuracy on the MathVerse vision-only benchmark.
Model
MMMU val
MathVerse*
MMStar
MathVista
Qwen2.5-VL-7B
56.4
42.6
64.3
68.2
+ GRPO
57.8
45.9
66.0
74.1
Visual Bonus-7B
59.3
48.2
65.0
73.9
Visual Mix-7B
58.6
48.1
66.5
72.5
PaLMR -7B
59.3
47.5
67.1
73.8
Table 3: An ablation study on the formulation strategies of visual-aware rewards. Utilizing the identical configuration for training as the primary experiment, we compare the proposed PaLMR against the strategies of Visual Bonus and Visual Mix across standard benchmarks.
Figure 7: Distribution of persistent errors across data sources on the training set.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Dataset
Number of Samples
Percentage (%)
Chart VQA
ChartQA [ 29 ]
209
4.42
FigureQA [ 19 ]
348
7.36
PlotQA [ 33 ]
2
0.04
TabMWP [ 27 ]
483
10.22
Subtotal
1042
22.04
GeoVQA
GeoQA+ [ 3 ]
16
0.34
Appendix
Table 4: Detailed Dataset Composition. The distribution represents the training data after applying solvability and evaluability filtering heuristics. The ‘Subtotal’ rows indicate the aggregate number of each domain.
Figure 8: Prompt template for Visual Ground Truth Generation. The prompt template used to instruct Gemini-2.5-Flash to generate structured, question-agnostic visual descriptions.
Figure 9: System prompt template for Visual-Aware Scoring. The LLM-as-a-Judge prompt used by Qwen3-30ba3b to perform pairwise ranking between the model rollout and a reference response, conditioned on the pseudo-visual ground truth.
Figure 10: User prompt template for Visual-Aware Scoring.
Figure 11: Prompt template for Policy Model.
Figure 12: Visualization of training set.
Figure 13: Visualization of training set.
Method
Latency (s)
Overhead
MMMU
Baseline (GRPO)
397
+0%
57.8
PaLMR (w/o CoT)
400
+0.7%
57.3
PaLMR (Ours)
546
+37.5%
59.3
Appendix
Table 5: Comparison of per-step training latency, computational overhead, and downstream performance (MMMU). Evaluated on Qwen2.5-VL-7B using 8 GPUs.