DDO-RM: Distribution-Level Policy Improvement after Reward Learning
Authors: Tiantian Zhang, Jierui Zuo, Michael Chen, Wenping Wang
Organizations: Department of Computer Science Columbia University, New York, NY 10027, USA · Department of Management Science and Engineering Tsinghua University, Beijing, China · Department of Computer Science Carnegie Mellon University, Pittsburgh, PA 15213, USA
Recent theory suggests that reward-model-first methods can be more sample-efficient than direct policy fitting when the reward function is statistically simpler than the induced policy. We propose DDO-RM, a finite-candidate decision-optimization method that converts reward scores into an explicit target distribution. Unlike PPO-based RLHF or DPO, DDO-RM performs a KL-regularized mirror-descent update to project the policy toward a reward-improved distribution over a candidate set. Preliminary experiments on Pythia-410M show that DDO-RM outperforms DPO in pair accuracy (0.52 to 0.56) and mean margin (0.13 to 0.53). Our framework provides a principled connection between reward learning and mirror-descent policy improvement.
Figures & tables
Component
Status in this paper
Reward modeling from human preferences
inherited from the RLHF pipeline
DPO-style direct pairwise policy fitting
baseline / related work
KL mirror descent and exponentiated-gradient update on the simplex
classical optimization tool, not claimed as new
Finite-candidate decision distribution for LLM post-training
proposed formulation
Using a learned reward model to construct a reward-improved target distribution over candidates
proposed DDO-RM policy-improvement step
Distilling the reward-improved distribution into the language model
proposed training interface
Table 1: Novelty boundary. DDO-RM uses a classical KL mirror-descent operator, but applies it as the second-stage policy-improvement map in reward-model-first LLM alignment.
Method
RM?
Candidate structure
Update type
Comparison role
DPO
No
Pair
Direct pairwise policy fitting
Direct-preference baseline
PPO-RLHF
Yes
Sampled responses
Stochastic policy-gradient update
Main reward-model baseline
GRPO
Yes / verifier
Group completions
Group-relative reward update
Multi-candidate or reasoning baseline
DDO-RM
Yes
Finite candidate set
Explicit KL / mirror-descent projection
Proposed reward-model policy-improvement method
Table 2: Comparison of preference-optimization methods. DDO-RM is closest to PPO-RLHF in information structure because both use an explicit reward model. DPO is a direct-preference reference baseline.
Figure 2: Mean metric comparison between DPO and DDO-RM across three seeds. From left to right: pair accuracy, AUC, and mean margin. DDO-RM outperforms DPO on all three metrics, with the largest absolute gain on mean margin.
Figure 3: Per-seed pair accuracy for DPO and DDO-RM across seeds 42, 13, and 3407. DDO-RM consistently exceeds DPO on every seed, with the gap widening on seeds 13 and 3407.
Metric
Method
Seed 42
Seed 13
Seed 3407
Mean
Pair Accuracy
DPO
0.5285
0.5205
0.5225
0.5238
DDO-RM
0.5410
0.5630
0.5765
0.5602
AUC
DPO
0.5335
0.5301
0.5308
0.5315
DDO-RM
0.5335
0.5388
0.5423
0.5382
Mean Margin
DPO
0.1308
0.1384
0.1439
0.1377
DDO-RM
0.2995
0.5196
0.7867
0.5353
Table 4: Held-out benchmark results on test_prefs . Higher is better for all metrics.
Department of Statistics and Operations Research, University of North Carolina · Imperial Business School, Imperial College London · Department of Management Science and Engineering, Stanford University