The effectiveness of Direct Preference Optimization (DPO) depends on preference data that reflect the quality differences that matter in multimodal tasks. Existing pipelines often rely on off-policy perturbations or coarse outcome-based signals, which are not well suited to fine-grained visual reasoning. We propose rDPO, a preference optimization framework based on instance-specific rubrics. For each image-instruction pair, we create a checklist-style rubric of essential and additional criteria to score responses from any possible policies. The instruction-rubric pool is built offline and reused during the construction of on-policy data. On public reward modeling benchmarks, rubric-based prompting massively improves a 30B-A3B judge and brings it close to GPT-5.4. On public downstream benchmarks, rubric-based filtering raises the macro average to 82.69, whereas outcome-based filtering drops it to 75.82 from 81.14. When evaluating scalability on a comprehensive benchmark, rDPO achieves 61.01, markedly outperforming the style-constrained baseline (52.36) and surpassing the 59.48 base model. Together, these results show that visual preference optimization benefits from combining on-policy data construction with instance-specific criterion-level feedback.
Figures & tables
Figure 1: Overview of rDPO. The top lane constructs a model-agnostic instruction-rubric pool by mining challenging seeds via model ensembles and drafting instance-specific rubrics with structured “essential” and “additional” criteria. The bottom lane achieves on-policy data curation and optimization by sampling responses from a target policy, scoring them via rubric-grounded judging, and mining preference pairs to drive iterative DPO.
Model Evaluator
MM-RB
VL-RB
MaaJ
VA-B
WV-B
Macro (Full / - RB / - B)
Proprietary Frontier VLMs
Claude 4.6 Opus
82.23
74.42
71.39
78.83
74.00
76.17 / 74.74 / 76.01
GPT-5.4
81.91
78.27
70.35
74.25
69.75
74.91 / 71.45 / 76.84
Gemini 3.1 Pro
88.79
86.45
66.56
76.25
73.33
- / 72.05 / -
Gemini 3.0 Flash
87.90
85.89
66.60
76.66
73.42
- / 72.23 / -
Gemini 3.1 Flash-Lite
82.25
81.40
64.31
74.33
67.00
- / 68.55 / -
Table 1: Comprehensive evaluation results across diverse preference benchmarks. Accuracy (%) is reported. + Rubric denotes our proposed strategy. Macro presents a triplet: (Full Average / Average excluding MM-RB & VL-RB / Average excluding VA-B & WV-B). Gray numbers indicate potential data contamination. If a model has contaminated entries, the corresponding macro averages are ignored (-). The best performance in each section is bolded , and the second best is underlined .
Policy Model
AI2D
ChartQA
M3CoT
Macro Avg.
Qwen3-VL-30B-A3B-Instruct
84.10
82.32
77.00
81.14
+ Outcome-based Filtering
78.53 −5.57
75.48 −6.84
73.45 −3.55
75.82 −5.32
+ Rubric-based Filtering (Ours)
85.95 +1.85
83.02 +0.70
79.11 +2.11
82.69 +1.55
Table 2: Results in the method-validation setting on AI2D, ChartQA, and M3CoT. Note that for the ChartQA evaluation, we adopt LLM-as-a-Judge, which is stricter than relaxed accuracy. For the latter two rows, each entry reports the metric followed by its absolute change relative to the base model.
Policy Model
Basic Capabilities
Advanced Cognition
Avg.
Perception
Understanding
Knowledge
Reasoning
Qwen3-VL-30B-A3B-Instruct
56.26
74.10
71.94
51.53
59.48
+ Prompting
55.67
68.57
61.02
42.86
52.36
+ rDPO (Ours)
62.82
77.00
71.27
51.62
61.01
Absolute Gain ( Δ )
+7.15
+8.43
+10.25
+8.76
+8.65
Table 3: Results in the scaling-validation setting on the comprehensive benchmark before and after DPO. The initial policy is Qwen3-VL-30B-A3B-Instruct. + Prompting denotes the baseline explicitly instructed to generate concise responses, mitigating the default model’s verbosity and markdown abuse. + rDPO denotes the model aligned using our generated preference dataset under the concise constraint. Absolute performance gains ( Δ ) represent improvements over the Prompting baseline.
Figure 2: Training dynamics and hyperparameter ablations. (a)(b) Convergence curves for three training paradigms on off-policy data, showing loss and reward accuracy. (c) Effect of varying the KL penalty coefficient β . (d) Effect of the SFT loss broken down by task domain.
Ablation Variant
Basic Capabilities
Advanced Cognition
Avg.
Perception
Understanding
Knowledge
Reasoning
rDPO (Full Pipeline)
62.82
77.00
71.27
51.62
61.01
Single-stage DPO
63.02
75.73
71.27
48.62
59.27
w/o essential qualification
60.76
75.20
67.93
48.62
58.36
Random reward margin ( ≥δ )
63.22
77.69
71.27
50.74
60.76
Minimum reward margin ( ≥δ )
62.03
76.22
69.71
50.52
59.96
Table 4: Comprehensive ablation studies on our in-house benchmark. The first row represents our full rDPO pipeline (the baseline for this ablation). Subsequent rows denote variants where specific alignment modules are either removed or altered.