Recent advances in multimodal reward modeling have been largely driven by a paradigm shift from discriminative to generative approaches. Building on this progress, recent studies have further employed reinforcement learning from verifiable rewards (RLVR) to enhance multimodal reward models (MRMs). Despite their success, RLVR-based training typically relies on labeled multimodal preference data, which are costly and labor-intensive to obtain, making it difficult to scale MRM training. To overcome this limitation, we propose a Multi-Stage Reinforcement Learning (MSRL) approach, which can achieve scalable RL for MRMs with limited multimodal data. MSRL replaces the conventional RLVR-based training paradigm by first learning a generalizable reward reasoning capability from large-scale textual preference data, and then progressively transferring this capability to multimodal tasks through caption-based and fully multimodal reinforcement-learning stages. Furthermore, we introduce a cross-modal knowledge distillation approach to improve preference generalization within MSRL. Extensive experiments demonstrate that MSRL effectively scales the RLVR-based training of generative MRMs and substantially improves their performance across both visual understanding and visual generation tasks (e.g., from 66.6% to 75.9% on VL-RewardBench and from 70.2% to 75.7% on GenAI-Bench), without requiring additional multimodal preference annotations. Our code is available at: https://github.com/wangclnlp/MSRL.
Figures & tables
Figure 1: Illustration of our multi-stage RL approach. Subfigure (a) shows that abundant textual preference data can facilitate scalable RL. Subfigure (b) demonstrates that we can effectively scale multimodal generative reward models through multi-stage RL.
Figure 2: An overview of the MSRL approach. We begin by applying RL to large-scale textual preference data (400k examples) to capture rich textual preferences. We then train an RL agent on caption-based data to generalize these preferences to multimodal tasks. During this stage, we also fine-tune the MRMs with CMKD to enhance the generalization. Subsequently, we perform RL with a limited amount of multimodal data to enable adaptation. Note that although the illustration uses image understanding as an example, MSRL is a general approach and can be applied to develop MRMs for arbitrary multimodal tasks.
Task
Input Content
Description
Image Understanding
( <Query> , <Image> , <Res A> , <Res B> )
Given an image and a textual query, the model evaluates which of the two responses better answers the query.
Image Generation
( <Caption> , <Image A> , <Image B> )
Given an image caption, the model decides which of the two generated images better reflects the described content.
Video Understanding
( <Query> , <Video> , <Res A> , <Res B> )
Given a video and a textual query, the model decides which of the two responses better answers the query.
Video Generation
( <Caption> , <Video A> , <Video B> )
Given a video caption, the model evaluates which of the two generated videos better reflects the described content.
Table 1: Description of different multimodal tasks for generative MRMs. Templates for each task are provided in Appendix A.
Method
Params.
VL-RewardBench
Multimodal RewardBench
Gen.
Hall.
Reason.
Avg.
Know.
Safe.
VQA
Other
Avg.
MLLM-as-a-Judge
GPT-4o ‡
-
49.1
67.6
70.5
62.4
72.0
74.8
87.2
65.4
74.9
GPT-5-mini
-
60.2
76.5
76.0
70.9
76.8
50.4
51.2
70.6
62.3
Gemini-1.5-Pro ‡
-
50.8
72.5
64.2
62.5
66.3
94.5
87.2
63.9
63.0
Gemini-3-Pro
-
54.6
78.9
86.7
73.4
94.2
94.7
93.0
82.4
91.1
Table 2: Accuracies (%) on VL-RewardBench and Multimodal RewardBench. The best result in each group is shown in bold. Results marked with † are taken from Wang et al. [41] on VL-RewardBench, while those marked with ‡ for both VL-RewardBench and Multimodal RewardBench are from Zhang et al. [54] . All remaining baseline results are obtained by their publicly available models. For Multimodal RewardBench, the “Other” column reports the average accuracy across the general and reasoning subsets.
Method
Params.
Image Gen.
Video Under.
Video Gen.
Average
GenAIBench
ShareGPT
GenAIBench
VideoGen
MLLM-as-a-Judge
GPT-4o
-
59.6
54.7
63.5
62.1
60.0
InternVL3.5-8B
8B
56.7
58.8
60.1
54.5
57.5
InternVL3.5-14B
14B
60.4
64.3
66.5
60.1
62.8
Open-Source Multimodal Reward Models
Table 3: Accuracies (%) on image generation, video understanding, and video generation tasks. Results marked with † are taken from Wang et al. [41] . All remaining baseline results are obtained by their publicly available models.
Method
Gen.
Hall.
Reason.
Avg.
Generative Baseline
57.2
74.5
68.2
66.6
MSRL
75.4
78.2
74.2
75.9
w/o Stage 1
62.7
73.4
70.2
68.8
w/o Stage 2 (Caption Data)
73.2
76.3
73.5
74.3
w/o Stage 2 (CMKD)
72.4
75.8
72.0
73.4
w/o Stage 3
70.8
74.6
72.4
72.6
Table 4: Ablation results on VL-RewardBench. It is worth noting that when Stage 1 is removed, the SFT-based cold-start process included in Stage 1 is also removed.
Figure 3: Performance scaling with different amounts of textual preference data on the VL-RewardBench.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Training Stage
Total
IU
IG
VU
VG
S1
S2
S3
S4
S5
S6
Stage 2 (Caption-Based Data)
19,442
5,000
2,500
2,580
4,323
2,500
2,539
Stage 3 (Multimodal Data)
20,038
8,000
2,500
2,639
2,343
2,500
2,056
Appendix
Table 5: Dataset statistics used in Stage 2 and Stage 3.
Mixing Ratio (Caption : Text)
1:0
1:1
2:1
4:1
5:1
Accuracy
74.6
73.8
74.2
75.5
75.2
Appendix
Table 6: Performance of preference generalization under different caption-to-text mixing ratios. The ratio “1:0” indicates that Stage 2 training uses only caption-based data, with no text-only preference data mixed in.
Figure 4: Template used for the image understanding task.
Figure 5: Template used for the image generation task.
Figure 6: Template used for the video understanding task.
Figure 7: Template used for the video generation task.
Figure 8: A case study illustrating the rationale generated by MSRL (8B backbone) for the image understanding task. Response A is identified as the superior answer because it provides an accurate and concise description grounded in visual evidence, whereas Response B relies on unsupported speculations regarding crowd size and the building’s historic significance. This demonstrates MSRL’s capability to discern these nuances and predict preferences with high precision effectively.
Figure 9: A case study illustrating the rationale generated by MSRL (8B backbone) for the image generation preference task. Image B is identified as the superior output because it effectively captures the prompt’s request for a “ruined stone thoroughfare” and “epic fantasy style” with dramatic lighting. In contrast, Image A is critiqued for missing the “vibrant hues” and “otherworldly elements”, presenting a setting that is too somber and ceremonial. This highlights MSRL’s ability to evaluate semantic alignment and stylistic fidelity in text-to-image generation precisely.
Figure 10: A case study illustrating the rationale generated by MSRL (8B backbone) for the video understanding task. Response A is identified as the superior answer because it accurately captures the synchronized movements and rhythmic coordination of the performers. In contrast, Response B is rejected due to visual inaccuracies, specifically the hallucinatory claim that performers are “balancing on the drums”. This demonstrates MSRL’s capability to verify visual details across temporal frames and identify factual inconsistencies.
Figure 11: A case study illustrating the rationale generated by MSRL (8B backbone) for the video generation preference task. Video A is identified as the superior output because it accurately depicts the primary subject ( i.e. , a “jellyfish”) within a detailed coral reef setting, aligning closely with the prompt. In contrast, Video B is critiqued for deviating from the prompt’s requirements, focusing instead on an ambiguous “starburst-like marine organism” rather than the requested underwater life. This demonstrates MSRL’s capability to evaluate subject fidelity and semantic alignment in generated video content.
Multimodal reward models (MRMs) play a crucial role in aligning Multimodal Large Language Models (MLLMs) with human preferences. Training a good MRM requires high-quality multimodal preference data. However, existing preference datasets face three key challenges: lack of granularity in preference strength, textual style bias, and unreliable preference signals. Besides, existing open-source multimodal preference datasets suffer from substantial noise, yet there is a lack of effective and scalable curation methods to enhance their quality. To address these limitations, we propose DT2IT-MRM, which integrates a Debiased preference construction pipeline, a novel reformulation of text-to-image (T2I) preference data, and an Iterative Training framework that curates existing multimodal preference datasets for Multimodal Reward Modeling. Our experimental results show that DT2IT-MRM achieves new state-of-the-art overall performance on three major benchmarks: VL-RewardBench, Multimodal RewardBench, and MM-RLHF-RewardBench.
Reinforcement learning (RL) with verifiable rewards (RLVR) has demonstrated the great potential of enhancing the reasoning abilities in multimodal large language models (MLLMs). However, the reliance on language-centric priors and expensive manual annotations prevents MLLMs' intrinsic visual understanding and scalable reward designs. In this work, we introduce SSL-R1, a generic self-supervised RL framework that derives verifiable rewards directly from images. To this end, we revisit self-supervised learning (SSL) in visual domains and reformulate widely-used SSL tasks into a set of verifiable visual puzzles for RL post-training, requiring neither human nor external model supervision. Training MLLMs on these tasks substantially improves their performance on multimodal understanding and reasoning benchmarks, highlighting the potential of leveraging vision-centric self-supervised tasks for MLLM post-training. We think this work will provide useful experience in devising effective self-supervised verifiable rewards to enable RL at scale. Project page: https://github.com/Jiahao000/SSL-R1.
Reinforcement Learning with Verifiable Rewards (RLVR) improves multimodal reasoning by rewarding verifiable final answers. Yet answer-correct trajectories may still rely on incomplete derivations, weak evidence, or statements that contradict their conclusions. This gap between answer correctness and reasoning validity, which we call reasoning-answer inconsistency, motivates trajectory supervision in multimodal RL. We compare two main approaches: reward models (RMs), and Generative Rewards (GRs). RMs are efficient and help early in training, but their gains weaken as the policy distribution shifts; GRs improve performance, but may give unstable rewards and computationally expensive. We therefore propose Groupwise Ranking Reward, which ranks verifier-passed trajectories for the same prompt in one pass and redistributes reward accordingly. Groupwise comparison better separates stronger and weaker correct trajectories with lower judge overhead than GRs. Experiments show that RLVR aggravates reasoning-answer inconsistency, while trajectory supervision alleviates it. Groupwise Ranking Reward performs best overall, improving reliability-conditioned accuracy from 47.4% to 54.7% over RLVR.