cs.AINov 14, 2026

Policy Optimization With Multiple Verifiers via Pareto Rankings

Authors: Kai Yamashita, Shohei Taniguchi, Hiroki Furuta, Masahiro Suzuki, Yutaka Matsuo

Organizations: The University of Tokyo · Third Intelligence, Inc.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become an important approach to improving the reasoning capabilities of large language models. In tasks such as code generation and instruction following, multiple verifiers evaluate whether a response satisfies different requirements. Their outcomes provide detailed feedback about partial success, yet this feedback is commonly aggregated into a scalar reward for policy optimization. Rewarding only complete success discards distinctions among partial solutions, while counts and weighted sums impose comparisons between responses that satisfy different subsets of requirements. We introduce Partial-Order Likelihood Policy Optimization (POLPO), which learns from the comparisons established by verification without first constructing an aggregate reward. Pareto dominance identifies responses that satisfy additional requirements without losing any already satisfied, while leaving responses with conflicting outcomes incomparable. POLPO extends the listwise likelihood underlying Direct Preference Optimization by maximizing the total probability of all complete rankings consistent with these partial orders. This objective provides supervision from verified improvements even before complete solutions are available, without requiring a prescribed ordering of incomparable responses. Experiments on code generation and instruction following demonstrate improved learning efficiency and higher rates of complete requirement satisfaction compared with scalar-reward training. These results support partial-order likelihood as an effective approach to learning from multiple verifiers.

Figures & tables

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Negative-Only Policy Optimization for One-Sided Verifiable Rewards

    Nov 12, 2026Jiacheng Xu, Shuo He, Fuxiang Zhang +2Reinforcement LearningReinforcement Learning with Verifiable Rewards

  2. An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

    Apr 9, 2026Andreas Plesner, Francisco Guzmán, Anish AthalyeRL for Code GenerationReinforcement Learning