Policy Optimization With Multiple Verifiers via Pareto Rankings
Organizations: The University of Tokyo · Third Intelligence, Inc.
Abstract
Reinforcement learning with verifiable rewards (RLVR) has become an important approach to improving the reasoning capabilities of large language models. In tasks such as code generation and instruction following, multiple verifiers evaluate whether a response satisfies different requirements. Their outcomes provide detailed feedback about partial success, yet this feedback is commonly aggregated into a scalar reward for policy optimization. Rewarding only complete success discards distinctions among partial solutions, while counts and weighted sums impose comparisons between responses that satisfy different subsets of requirements. We introduce Partial-Order Likelihood Policy Optimization (POLPO), which learns from the comparisons established by verification without first constructing an aggregate reward. Pareto dominance identifies responses that satisfy additional requirements without losing any already satisfied, while leaving responses with conflicting outcomes incomparable. POLPO extends the listwise likelihood underlying Direct Preference Optimization by maximizing the total probability of all complete rankings consistent with these partial orders. This objective provides supervision from verified improvements even before complete solutions are available, without requiring a prescribed ordering of incomparable responses. Experiments on code generation and instruction following demonstrate improved learning efficiency and higher rates of complete requirement satisfaction compared with scalar-reward training. These results support partial-order likelihood as an effective approach to learning from multiple verifiers.
Figures & tables
| Qwen3-0.6B | Qwen3-4B | |||
|---|---|---|---|---|
| Method | All-pass | Avg. pass | All-pass | Avg. pass |
| Initial model | 30.00 | 34.40 | 62.40 | 66.40 |
| GRPO–All | ||||
| GRPO–Rate | ||||
| GRPO–Pareto | ||||
| REINFORCE++–All | ||||
| Method | Overall | |||||
|---|---|---|---|---|---|---|
| Initial model | 56.1 | 29.9 | 18.5 | 7.4 | 3.7 | 27.1 |
| GRPO–All | ||||||
| GRPO–Rate | ||||||
| GDPO | ||||||
| GraphDPO | ||||||
| DPO–Pareto |
| Method | L1 | L2 | L3 | L4 | Overall |
|---|---|---|---|---|---|
| Initial model | 85.3 | 63.7 | 48.0 | 32.3 | 57.3 |
| GRPO–All | |||||
| GRPO–Rate | |||||
| GDPO | |||||
| GraphDPO | |||||
| DPO–Pareto |
| Thinking disabled | Thinking enabled | |||||||
|---|---|---|---|---|---|---|---|---|
| Method | HAcc | SAcc | Acc | Joint | HAcc | SAcc | Acc | Joint |
| Initial model | 31.2 | 55.7 | 13.8 | 5.7 | 25.2 | 40.6 | 25.0 | 6.2 |
| GRPO–All | 55.7 | 74.3 | 31.4 | 20.2 | 41.9 | 46.5 | 27.1 | 22.6 |
| GRPO–Rate | 76.0 | 86.0 | 27.1 | 19.3 | 70.5 | 79.3 | 38.8 | 30.7 |
| GDPO | – | – | – | – | 58.6 | 68.4 | 30.5 | 23.3 |
| GraphDPO | – | – | – | – | 56.4 | 63.4 | 30.0 | 24.8 |
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| 1 | for do |
|---|---|
| 2 | ; hold this rollout policy fixed |
| 3 | Sample prompts |
| 4 | for do |
| 5 | Sample from |
| 6 | Evaluate for |
| 7 |
| Training | Loss (ms) | Overhead | |||
| Setting | update (s) | POLPO | GRPO | (ms) | (%) |
| MBPP, Qwen3-0.6B | 57.5 | 64.2 | 24.2 | 40.0 | 0.07 |
| MBPP, Qwen3-4B | 85.3 | 49.9 | 24.4 | 25.5 | 0.03 |
| APPS Introductory | 155.2 | 72.6 | 23.9 | 48.7 | 0.03 |
| AceCode | 375.1 | 298.3 | 95.4 | 202.9 | 0.05 |
| Training | Loss (ms) | Overhead | |||
|---|---|---|---|---|---|
| Setting | update (s) | POLPO | GRPO | (ms) | (%) |
| IFM5, Qwen3-4B | 50.7 | 43.6 | 19.1 | 24.5 | 0.05 |
| MulDimIF, Qwen3-4B | 60.6 | 93.9 | 19.5 | 74.4 | 0.12 |
| MathIF thinking, Qwen3-0.6B | 75.8 | 42.2 | 19.5 | 22.7 | 0.03 |
| Update (s) | GPU memory (GB) | Loss | ||||
|---|---|---|---|---|---|---|
| GRPO | POLPO | Ratio | GRPO | POLPO | (s/step) | |
| 8 | 10.82 | 10.79 | 1.00 | 16.8 | 19.1 | 0.24 |
| 16 | 5.77 | 6.16 | 1.07 | 19.5 | 18.3 | 0.23 |
| Update (s) | GRPO ratio | Memory (GB) | Loss (ms/group) | ||
|---|---|---|---|---|---|
| 8 | 4 | 10.96 | 1.01 | 19.5 | 9.6 |
| 8 | 7 | 10.89 | 1.01 | 20.4 | 13.5 |
| 16 | 4 | 6.23 | 1.08 | 19.0 | 9.5 |
| 16 | 8 | 5.93 | 1.03 | 17.8 | 14.3 |
| 16 | 12 | 6.51 | 1.13 | 17.7 | 21.0 |
| 16 | 15 | 6.52 | 1.13 | 18.3 | 25.8 |
| Setting | MBPP | APPS Introductory | AceCode |
|---|---|---|---|
| Initial model | Qwen3-0.6B / Qwen3-4B | Qwen2.5-Coder-1.5B-Instruct | |
| Optimizer | AdamW | AdamW | AdamW |
| Learning rate | See Table 12 | ||
| Adam coefficients | |||
| Weight decay | |||
| Learning-rate schedule | Constant | Constant | Constant |
| MBPP | APPS Introductory | AceCode | ||
|---|---|---|---|---|
| Method | Qwen3-0.6B | Qwen3-4B | Qwen2.5-Coder-1.5B-Instruct | |
| GRPO–All | ||||
| GRPO–Rate | ||||
| GRPO–Pareto | – | – | ||
| REINFORCE++–All | – | |||
| REINFORCE++–Rate | – | |||
| Constraint count | 1 | 2 | 3 | 4 | 5 | Total |
|---|---|---|---|---|---|---|
| Before filtering | 478 | 493 | 500 | 394 | 135 | 2,000 |
| Excluded | 2 | 5 | 7 | 3 | 0 | 17 |
| Retained | 476 | 488 | 493 | 391 | 135 | 1,983 |
| Level | Constraint categories | Individual constraints |
|---|---|---|
| I | 1 | 1–2 |
| II | 2 | 2–4 |
| III | 3 | 3–6 |
| IV | 4 | 4–8 |
| Setting | IFM5 | MulDimIF |
|---|---|---|
| Initial model | Qwen3-4B-Instruct-2507 | |
| Optimizer | AdamW | AdamW |
| Learning rate | ||
| Adam coefficients | ||
| Weight decay | ||
| Learning-rate schedule | Constant | Constant |
| Setting | Thinking disabled | Thinking enabled |
|---|---|---|
| Initial model | Qwen3-0.6B | |
| Optimizer | AdamW | |
| Learning rate | ||
| Adam coefficients | ||
| Weight decay | ||
| Learning-rate schedule | Constant | |
| Target | GRPO–All | GRPO–Rate | POLPO–All | POLPO–Rate | POLPO | |
|---|---|---|---|---|---|---|
| 15 | — | — | ||||
| 30 | — | — | — | |||
| 60 | — | — | — | |||
| 60 | — | — | — | |||
| 60 | — | — | — | — |
| Initial signal | Final signal | Solved | |||
| Method | All | Unsolved | All | Unsolved | Initial final |
| IFM5 ( ) | |||||
| POLPO | 47.7 | 57.7 | 14.0 | 32.1 | |
| GRPO–All | 17.7 | 21.4 | 13.1 | 27.2 | |
| GRPO–Rate | 47.7 | 57.7 | 19.1 | 41.5 | |
| MulDimIF ( ) | |||||
| Policy | Identical outcomes | Different, no dominance | Ranking with ties | Genuinely partial | |
|---|---|---|---|---|---|
| IFM5 | |||||
| Initial | 988 | 52.1 | 0.2 | 38.8 | 8.9 |
| POLPO final | 988 | 84.7 | 1.3 | 12.6 | 1.4 |
| MulDimIF | |||||
| Initial | 800 | 25.1 | 0.1 | 43.5 | 31.2 |
| POLPO final | 800 | 88.9 | 0.0 | 10.6 | 0.5 |
| Policy | Dominance groups | Same | Different: tied responses | Different: distinct outcome classes |
|---|---|---|---|---|
| IFM5 | ||||
| Initial | 471 | 7.2 | 80.5 | 12.3 |
| POLPO final | 138 | 5.8 | 88.4 | 5.8 |
| MulDimIF | ||||
| Initial | 598 | 2.2 | 74.9 | 22.9 |
| POLPO final | 89 | 3.4 | 95.5 | 1.1 |
| Policy | Mean | Median | P95 | Max | ||
| IFM5 | ||||||
| Initial | 16 | 471 | 8.9 | 10 | 15 | 16 |
| POLPO final | 16 | 138 | 10.7 | 13 | 15 | 16 |
| MulDimIF | ||||||
| Initial | 16 | 598 | 10.9 | 12 | 15 | 16 |
| POLPO final | 16 | 89 | 11.6 | 13 | 15 | 15 |
| HAcc | Acc | Joint | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Init. | All | Rate | POLPO | Init. | All | Rate | POLPO | Init. | All | Rate | POLPO | |
| 1 | 56.4 | 78.6 | 85.7 | 89.3 | 15.0 | 32.9 | 33.6 | 30.7 | 10.7 | 29.3 | 30.0 | 29.3 |
| 2 | 22.9 | 50.0 | 69.3 | 78.6 | 10.0 | 25.7 | 20.7 | 22.9 | 2.9 | 15.7 | 13.6 | 17.1 |
| 3 | 14.3 | 38.6 | 72.9 | 76.4 | 16.4 | 35.7 | 27.1 | 27.1 | 3.6 | 15.7 | 14.3 | 17.9 |
| Source | Initial | GRPO–All | GRPO–Rate | POLPO | ||
|---|---|---|---|---|---|---|
| GSM8K | 1 | 30 | 26.7 | 56.7 | 60.0 | 63.3 |
| GSM8K | 2 | 30 | 10.0 | 33.3 | 23.3 | 23.3 |
| GSM8K | 3 | 30 | 6.7 | 36.7 | 30.0 | 30.0 |
| GSM8K | All | 90 | 14.4 | 42.2 | 37.8 | 38.9 |
| MATH-500 | 1 | 30 | 16.7 | 50.0 | 60.0 | 53.3 |
| MATH-500 | 2 | 30 | 3.3 | 30.0 | 26.7 | 33.3 |
| HAcc | Acc | Joint | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | |||||||||
| Initial model | 40.0 | 22.9 | 12.9 | 31.4 | 22.1 | 21.4 | 12.9 | 3.6 | 2.1 |
| GRPO–All | 50.7 | 40.0 | 35.0 | 31.4 | 25.7 | 24.3 | 29.3 | 20.0 | 18.6 |
| GRPO–Rate | 81.4 | 70.0 | 60.0 | 42.9 | 36.4 | 37.1 | 37.1 | 27.9 | 27.1 |
| GDPO | 69.3 | 59.3 | 47.1 | 33.6 | 25.7 | 32.1 | 29.3 | 20.0 | 20.7 |
| GraphDPO | 68.6 | 55.0 | 45.7 | 33.6 | 25.7 | 30.7 | 31.4 | 21.4 | 21.4 |
| Source | Initial | GRPO–All | GRPO–Rate | GDPO | GraphDPO | DPO–Pareto | POLPO | ||
|---|---|---|---|---|---|---|---|---|---|
| GSM8K | 1 | 30 | 30.0 | 46.7 | 73.3 | 53.3 | 50.0 | 70.0 | 60.0 |
| GSM8K | 2 | 30 | 10.0 | 33.3 | 50.0 | 33.3 | 40.0 | 53.3 | 46.7 |
| GSM8K | 3 | 30 | 3.3 | 20.0 | 36.7 | 33.3 | 26.7 | 43.3 | 50.0 |
| GSM8K | All | 90 | 14.4 | 33.3 | 53.3 | 40.0 | 38.9 | 55.6 | 52.2 |
| MATH-500 | 1 | 30 | 16.7 | 60.0 | 63.3 | 56.7 | 63.3 | 63.3 | 70.0 |
| MATH-500 | 2 | 30 | 3.3 | 30.0 | 40.0 | 33.3 | 36.7 | 46.7 | 56.7 |
RL for Language Model Reasoning
Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.
See all 1,162 papers