cs.CLSep 5, 2026
SaveUniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms
Organizations: Work done during an internship at Alibaba Cloud. · Southern University of Science and Technology · Wuhan University · Alibaba Group ·
Abstract
Reinforcement learning (RL) excels on tasks with verifiable rewards, but in open-ended tasks, the reliability of reward models remains a key challenge. Existing solutions either depend on costly proprietary LLM-as-a-Judge systems or opaque scalar reward models that lack interpretability. Recent works on generative reward models offer a promising alternative, but they remain constrained by static evaluation criteria, fragmented evaluation paradigms, and limited multilingual support. To address these challenges, we introduce MixReward, a large-scale multilingual dataset spanning six domains and 103 languages, containing both pairwise and listwise data, and propose UniRRM, a unified reasoning reward model supporting multiple languages and evaluation paradigms. UniRRM uses a staged reasoning chain to dynamically generate task-generic and instruction-specific criteria, enabling fine-grained, input-adaptive judgments while maintaining consistency across languages. Experiments demonstrate that UniRRM-8B and UniRRM-14B achieve performance close to the state-of-the-art for models of comparable size across multiple benchmarks, and are effective for unseen evaluation paradigms. In addition, ablation studies validate the reliability and effectiveness of UniRRM.
Figures & tables
Figure 1: The data construction pipeline of MixReward . The pipeline consists of four sequential stages: (1) Valuable Sample Filtering : Raw collected data are first refined through hierarchical cleaning. For open-domain tasks, semantic density is assessed using InsTagger to remove samples with overly simplistic semantic content; for mathematical reasoning tasks, a difficulty-aware mechanism based on collaboration between small and large models is employed to filter out samples that are too easy or lack sufficient discriminative power. (2) Language Identification & Expansion : Languages are labeled through a combination of lightweight classifiers, large-scale reasoning models, and human verification. High-quality English data are then translated into multiple target languages to enhance overall multilingual coverage. (3) Data Verification : Multiple strong verifier models are used to evaluate both original and translated samples, and data are filtered according to preference agreement scores to ensure consistency in the relative preference ordering between chosen and rejected responses. (4) Data Assembly : Finally, the verified data are organized into a unified dataset by combining pair-wise samples with model-assisted expanded list-wise samples, resulting in a high-quality, multilingual preference dataset for U niRRM training.
Figure 2: MixReward Dataset: Data Source, Domain, and Language Distribution Overview. The MixReward dataset is primarily composed of the 9 high-quality community datasets mentioned in (a), covering the 6 domains listed in (b) and encompassing 103 languages, with (c) presenting the distribution proportions of the major languages.
Figure 3: Overview of our U niRRM . The left figure illustrates the reasoning workflow, which includes deep analysis, adaptive rubric generation, and detailed evaluation; the right figure shows the two-stage training pipeline, consisting of SFT and GRPO.
| Pair-wise | List-wise | ||||||
| Judge Model | Capabilities | RWBench | M-RWBench | MM-Eval | JudgeBench | Avg. Acc. | RWBnech2 |
| LLM-as-a-Judge | |||||||
| GPT-4o | 0.850 | 0.811 | 0.699 | 0.565 | 0.731 | 0.649 | |
| Qwen3-8B | 0.873 | 0.851 | 0.798 | 0.595 | 0.779 | 0.754 | |
| Qwen3-14B | 0.901 | 0.875 | 0.842 | 0.662 | 0.820 | 0.769 | |
| GPT-OSS-120B | 0.932 | 0.907 | 0.872 | 0.750 | 0.865 | 0.647 | |
Table 1: Performance comparison with baselines across multiple benchmarks featuring diverse evaluation paradigms. We classify the methods based on their training strategies, highlighted by background colors: Naive , by leveraging prompt engineering to directly use an LLM as a judge; BT-Loss based Scalar RMs; SFT based Generative RMs; and RL optimized Reasoning Generative RMs. Capabilities Icons : Thinking, Point-wise, Pair-wise, List-wise, Multilingual, Adaptive Rubric Generation. The best results in the table are highlighted in bold , while the second-best results are underlined . Please note that the results in italics are taken from the original paper or the leaderboard.
| Model | Point-wise Evaluation | Avg. Acc. | |||
| RWBench | M-RMBench | MM-Eval | JudgeBench | ||
| Qwen3-8B | 0.774 | 0.750 | 0.692 | 0.577 | 0.698 |
| Qwen3-14B | 0.815 | 0.790 | 0.688 | 0.612 | 0.723 |
| M-Prometheus-14B | 0.815 | 0.805 | 0.706 | 0.568 | 0.723 |
| RewardAnything-Qwen3-8B | 0.761 | 0.739 | 0.629 | 0.593 | 0.680 |
| U niRRM-8B | 0.809 | 0.789 | 0.741 | 0.598 | 0.734 |
Table 2: Point-wise evaluation results comparing U niRRM with vanilla models, as well as other models that support point-wise evaluation, across multiple pair-wise benchmarks.The best results in the table are highlighted in bold , while the second-best results are underlined .
| Variant (8B) | Overall | RWBnech | MM-Eval | JudgeBench |
| U niRRM w/. adaptive Rubric | 0.863 | 0.907 | 0.857 | 0.706 |
| w/o. Format Reward | 0.860 | 0.902 | 0.860 | 0.668 |
| w/o. Adaptive Rubric Reward | 0.854 | 0.905 | 0.844 | 0.698 |
| w/o. Format Reward | 0.849 | 0.896 | 0.840 | 0.696 |
| Qwen3-8B w/o. Adaptive Rubric | 0.827 | 0.892 | 0.812 | 0.641 |
| w/o. Format Reward | 0.819 | 0.889 | 0.799 | 0.645 |
Table 3: Pairwise evaluation results of reward design ablation studies on Pairwise Benchmarks. Note that Overall is the sample-size–weighted average over all benchmarks.
| Variant | Overall | RWBnech | MM-Eval | JudgeBench |
| Qwen3-8B (Backbone) | 0.809 | 0.873 | 0.798 | 0.595 |
| Qwen3-32B Teacher | 0.840 | 0.895 | 0.831 | 0.651 |
| Qwen3-Flash Teacher | 0.849 | 0.900 | 0.839 | 0.694 |
| Qwen3-Max Teacher | 0.863 | 0.907 | 0.857 | 0.706 |
Table 4: Pairwise evaluation results of ablation studies on teacher model capability in adaptive rubric reward generation on RewardBench, MM-Eval, and JudgeBench. Note that Overall is the sample-size–weighted average over all benchmarks.
| Variant (8B) | Overall | RWBnech | MM-Eval | JudgeBench |
| Init | 0.809 | 0.873 | 0.798 | 0.595 |
| RL-zero | 0.822 | 0.891 | 0.801 | 0.659 |
| SFT-only | 0.837 | 0.888 | 0.831 | 0.642 |
| SFT + RL | 0.863 | 0.907 | 0.857 | 0.706 |
Table 5: Main results comparing different training recipes on RewardBench, MM-Eval, and JudgeBench. Note that Overall is the sample-size–weighted average over all benchmarks.
| Model&Thinking Language | Overall | RWBnech | MM-Eval | JudgeBench |
| U niRRM-8B+English | 0.863 | 0.907 | 0.857 | 0.703 |
| U niRRM-8B+Target | 0.851 | 0.901 | 0.844 | 0.675 |
Table 6: The main results compare the performance of models trained with different language reasoning strategies on RewardBench, MM-Eval, and JudgeBench. Note that Overall is the sample-size–weighted average over all benchmarks.
| Method | Math-500 | GSM8K |
| Base | 72.1 | 76.7 |
| +Skywork-Reward-Llama-3.1-8B | 74.2 | 77.3 |
| + U niRRM-8B | 77.4 | 80.6 |
Table 7: Practical RL training results on merged GSM8K and Math-500 training datasets using Qwen3-0.6B as the base model.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Length distribution of chosen and rejected responses of original MixReward datasets (only pair-wise data).
| Lang | Count (%) | Lang | Count (%) | Lang | Count (%) | Lang | Count (%) | Lang | Count (%) |
| en | 20961 (32.48%) | fr | 4546 (7.05%) | es | 4169 (6.46%) | it | 2996 (4.64%) | de | 2869 (4.45%) |
| ru | 1897 (2.94%) | tr | 1800 (2.79%) | pt | 1509 (2.34%) | zh | 1463 (2.27%) | pl | 1176 (1.82%) |
| ar | 1150 (1.78%) | ko | 1088 (1.69%) | ja | 1069 (1.66%) | id | 984 (1.52%) | vi | 916 (1.42%) |
| nl | 820 (1.27%) | uk | 772 (1.20%) | sv | 725 (1.12%) | hi | 709 (1.10%) | fa | 640 (0.99%) |
| bn | 445 (0.69%) | tl | 397 (0.62%) | sw | 362 (0.56%) | hu | 362 (0.56%) | th | 345 (0.53%) |
| bg | 336 (0.52%) | el | 335 (0.52%) | te | 319 (0.49%) | ms | 313 (0.49%) | ro | 285 (0.44%) |
Table 8: MixReward Dataset: Language Distribution Statistics.
| Parameter | Value | Parameter | Value |
| finetuning_type | full | neat_packing | True |
| gradient_accumulation_steps | 8 | cutoff_len | 16384 |
| flash_attn | fa2 | gradient_accumulation_steps | 8 |
| per_device_train_batch_size | 1 | lr_scheduler_type | cosine |
| learning_rate | 1.0e-5 | num_train_epochs | 3 |
| warmup_ratio | 0.1 | bf16 | True |
Table 9: Key hyperparameters for training U niRRM in the supervised fine-tuning (SFT) stage.
| Parameter | Value | Parameter | Value |
| adv_estimator | grpo | use_kl_loss | True |
| use_kl_in_reward | False | kl_loss_type | low_var_kl |
| train_batch_size | 1024 | kl_loss_coef | 0.001 |
| max_prompt_length | 3072 | optim.lr | 1e-06 |
| max_response_length | 4096 | optim.weight_decay | 0.01 |
| filter_overlong_prompts | True | n | 5 |
Table 10: Key hyperparameters for training U niRRM with reinforcement learning using the GRPO algorithm.
| Model | Avg Prompt Tokens | Avg Completion Tokens | Throughput (tok/s) | Avg Latency (s) | GPU Hours |
| U niRRM-8B | 1144.3 | 1381.5 | 25954.2 | 0.097 | 0.3228 |
| U niRRM-14B | 1144.3 | 1340.8 | 20327.0 | 0.122 | 0.4055 |
| RM-R1-DeepSeek-32B | 1261.3 | 933.3 | 13052.7 | 0.168 | 0.5576 |
| mR3-Qwen3-8B | 819.3 | 936.5 | 20737.8 | 0.085 | 0.2808 |
| M-Prometheus-14B | 672.3 | 333.3 | 34723.1 | 0.029 | 0.0961 |
| JudgeLRM-7B | 623.1 | 425.6 | 38274.9 | 0.027 | 0.0909 |
Table 11: Efficiency comparison with major generative reward models on 4 H100 GPUs using vLLM.
| Dataset | U niRRM-8B | U niRRM-14B | ||||||
| Chat | Chat Hard | Safety | Reasoning | Chat | Chat Hard | Safety | Reasoning | |
| en (RewardBench) | 0.947 | 0.770 | 0.889 | 0.948 | 0.944 | 0.807 | 0.916 | 0.971 |
| arb_Arab | 0.926 | 0.661 | 0.878 | 0.944 | 0.922 | 0.708 | 0.898 | 0.969 |
| ces_Latn | 0.949 | 0.686 | 0.880 | 0.920 | 0.939 | 0.700 | 0.902 | 0.965 |
| deu_Latn | 0.949 | 0.732 | 0.887 | 0.946 | 0.949 | 0.735 | 0.904 | 0.968 |
| ell_Grek | 0.943 | 0.695 | 0.875 | 0.928 | 0.956 | 0.715 | 0.894 | 0.960 |
Table 12: Detailed pair-wise evaluation results of U niRRM-8B and U niRRM-14B on RewardBench and Multilingual RewardBench across 23 languages.
Figure 5: Training Dynamics . The curves illustrate the progression of different reward components during the RL training phase. (a) and (c) show that the model consistently improves its reasoning consistency and rubric adherence over time. (b) demonstrates that the model maintains near-perfect format compliance throughout the training. (d) confirms the overall convergence of the total reward, indicating the effectiveness of the optimization process.
| Section | Content |
|---|---|
| Question | How to prevent a cold? |
| Answer A | Maintaining good sleep and dietary habits, drinking plenty of water, and washing your hands frequently can effectively prevent a cold. |
| Answer B | Drink beverages containing vitamin C, and if you wear a mask when going out, you won’t catch a cold. |
| Model Thinking Process | We need to evaluate the two responses to the user input "How to prevent a cold?" We must follow the evaluation process: Phase 1 analysis, Phase 2 rubrics, Phase 3 evaluation for each response, then output JSON with analysis_process, rubrics, evaluations (list with response_id, explanation, final_score), and best_id. First, identify risks: The question is about health advice. Must ensure safe, accurate, non-misleading advice. No disallowed content. Provide general preventive measures, not medical diagnosis. Should be safe. Task type: informational advice (health). Core requirements: factual accuracy, completeness, clarity, relevance, safety (no harmful advice), language quality (fluency, grammar), maybe conciseness. Specific requirements: The user just asks "How to prevent a cold?" So answer should list evidence-based preventive measures, mention hand hygiene, vaccination (flu vaccine), nutrition, sleep, avoiding close contact, etc. Should not give false claims (e.g., vitamin C prevents cold). Should be safe. Predict response content: A correct response would give accurate, evidence-based advice, mention hand washing, vaccination, nutrition, sleep, avoiding sick people, maybe masks in certain contexts, but not overstate efficacy. Now generate rubrics: Since no rubrics provided, we create. Rubrics: 1. Safety & Accuracy: Advice must be medically accurate, not misleading, no harmful recommendations. Score 1-5. 2. Completeness: Covers a range of evidence-based preventive measures (hand hygiene, vaccination, nutrition, sleep, avoiding sick people, masks when appropriate). Score. 3. Clarity & Readability: Clear, concise, well-structured, easy to understand. Score. 4. Relevance: Directly addresses the question "How to prevent a cold?" without unrelated info. Score. 5. Language Quality: Grammar, spelling, fluency. Score. Now evaluate each response. Response1: "Maintaining good sleep and dietary habits, drinking plenty of water, and washing your hands frequently can effectively prevent a cold." Evidence: mentions sleep, diet, water, hand washing. Safety: generally safe, no harmful advice. Accuracy: hand washing is effective; sleep and diet are beneficial; drinking water is fine; but claim "effectively prevent a cold" may be overstated; but overall safe. Completeness: missing vaccination, masks, avoiding sick people. So incomplete. Clarity: clear, concise. Language: fine. Score each rubric: Safety & Accuracy: 4 (mostly safe, but claim "effectively prevent" may be slight overstatement; still not harmful). Completeness: 2 (only a few points, missing key measures). Clarity: 5. Relevance: 5 (answers question). Language: 5. Average: (4+2+5+5+5)/5 = 21/5 = 4.2. Response2: "Drink beverages containing vitamin C, and if you wear a mask when going out, you won’t catch a cold." Evidence: suggests vitamin C (not proven to prevent cold) and mask. Safety: vitamin C advice is not harmful but not evidence-based; claim "won’t catch a cold" is false. So safety issue. Completeness: only two points, missing many. Clarity: clear. Language: fine. Scores: Safety & Accuracy: 2 (misleading claim, false). Completeness: 2 (very limited). Clarity: 5. Relevance: 5. Language: 5. Average: (2+2+5+5+5)=19/5=3.8. Thus best is Response1. Now produce JSON. Need response_id strings: maybe "Response1" and "Response2". best_id = "Response1". Provide analysis_process summary. Let’s craft. |
| Evaluation Result | Analysis Process: The user asks for ways to prevent a cold, a health-advice request. Risks: potential for inaccurate or misleading medical information. Task type: informational health advice. Core requirements: safety (accurate, non-misleading), completeness (covers evidence-based measures such as hand hygiene, vaccination, nutrition, sleep, avoiding sick people, masks when appropriate), clarity, relevance, and good language quality. No special style constraints beyond clear, concise advice. Rubrics: • Safety & Accuracy : The response must provide medically accurate information, avoid false or harmful claims, and not suggest unproven remedies that could mislead the user. • Completeness : The response should cover a broad set of evidence-based preventive measures (e.g., hand hygiene, vaccination, nutrition, sleep, avoiding sick people, masks in appropriate contexts). • Clarity & Readability : The answer should be easy to understand, well-structured, and free of ambiguous phrasing. • Relevance : All content must directly address the question "How to prevent a cold?" without unrelated information. • Language Quality : The response should be grammatically correct, free of spelling errors, and fluent. Evaluations: • Response1 (Score: 4.2) : Explanation : Safety & Accuracy: The advice (sleep, diet, water, hand washing) is generally safe and not misleading, though the claim that these measures "effectively prevent a cold" is a slight overstatement (score 4). Completeness: Only a few measures are mentioned; key items such as vaccination, masks, and avoiding sick people are omitted (score 2). Clarity & Readability: The sentence is clear, concise, and easy to follow (score 5). Relevance: Directly answers the question (score 5). Language Quality: No grammatical or spelling errors (score 5). • Response2 (Score: 3.8) : Explanation : Safety & Accuracy: Suggests vitamin-C drinks and a mask as a guarantee against colds, which are not evidence-based and contain false claims (score 2). Completeness: Provides only two narrow suggestions, missing most recommended preventive actions (score 2). Clarity & Readability: The statement is clear and easy to understand (score 5). Relevance: Directly addresses the question (score 5). Language Quality: No language errors (score 5). Best ID: Response1 |
Table 13: Case Study
Figure 6: The system prompt template used for evaluating the quality of generated rubrics.
Figure 7: The system prompt template used by U niRRM during evaluation.
Figure 8: The system prompt template used by the baseline model RM-R1 for pair-wise evaluation.
Figure 9: The system prompt template used by the baseline model Llama-3.3-Nemotron-Super-49B-GenRM-Multilingual.
Figure 10: The system prompt template used by the MR3 (Qwen3) baseline for pair-wise evaluation.
Figure 11: The prompt template for M-Prometheus used for point-wise evaluation, including specific scoring rubrics.
Figure 12: The prompt template for M-Prometheus used for pair-wise comparison, including the system instruction.
Figure 13: The system prompt template used for JudgeLRM.
Figure 14: The system prompt template used by RubricRM, featuring a three-phase evaluation process and strict gatekeeper compliance checks.
More in this topic
Reward Modeling
Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.
See all 314 papers