Foundation Pareto Flow Policy for Multi-Objective Reinforcement Learning
Organizations: Shenzhen Technology University · Shenzhen University · Tsinghua University
Abstract
The overarching goal of reinforcement learning (RL) is to efficiently optimize the agent's policy for various environments, tasks, dynamics, and even user preferences, which often consist of multiple and potentially conflicting rewards/objectives. This necessitates the learning of Pareto-optimal policies that can navigate different trade-offs among objectives. However, existing multi-objective RL (MORL) methods, whether trained online or offline, are typically limited to task- or objective-specific policy learning and do not address generalization of learned trade-offs to unseen tasks/objectives. In this work, we propose the Foundation Pareto Flow Policy (FP2) for offline MORL in continuous robotic control, inspired by the pre-mid-post-training paradigm underlying recent foundation models. Specifically, we instantiate FP2 as a multi-modal diffusion transformer equipped with a likelihood-free flow matching loss, enabling unified, scalable, and instructable learning of diverse trade-off trajectories. We further develop a three-stage training pipeline to gradually optimize the policy: (1) pre-training on a large-scale MORL dataset, (2) mid-training on a carefully curated high-quality dataset, and (3) post-training on few collected trajectory data from new tasks and trade-offs. Empirical results demonstrate that FP2 outperforms previous state-of-the-art methods on most tasks of the D4MORL benchmark, achieving superior zero-shot interpolation and extrapolation on complex Pareto manifolds as well as efficient few-shot adaptation of the learned trade-off prior to new continuous-control tasks. Code is available at https://github.com/FibonacciRabbit/FP2.
Figures & tables
| Metric | Method | Ant-S | HalfC.-S | Hopper-S | Walker-S | Swim.-S | Hopper3-S | Ant-N | HalfC.-N | Hopper-N | Walker-N | Swim.-N | Hopper3-N |
| HV | MODT(P) | 5.88 | 5.69 | 1.95 | 5.01 | 3.20 | 2.83 | 5.05 | 5.04 | 1.85 | 4.75 | 3.03 | 3.18 |
| MORvS(P) | 6.38 | 5.73 | 2.06 | 5.14 | 3.24 | 3.28 | 6.06 | 5.46 | 1.98 | 4.85 | 3.10 | 3.32 | |
| BC(P) | 4.76 | 5.66 | 1.49 | 3.47 | 3.19 | 1.95 | 4.90 | 4.88 | 1.29 | 1.88 | 3.06 | 2.32 | |
| MODULI | 6.38 | 5.78 | 2.07 | 5.20 | 3.24 | 3.33 | 6.18 | 5.74 | 2.07 | 5.13 | 3.16 | 3.15 | |
| FP2-Pre | 6.17 | 5.76 | 2.01 | 5.16 | 3.22 | 3.80 | 6.06 | 5.74 | 2.01 | 5.10 | 3.16 | 3.39 | |
| FP2-Mid | 6.35 | 5.79 | 2.08 | 5.22 | 3.26 | 3.70 | 6.24 | 5.79 | 2.10 | 5.22 | 3.26 | 3.81 |
| Environment | Scratch | FP2-Mid (Zero-shot) | FP2-Post (Few-shot) | ||||||
| HV | SP | PAS | HV | SP | PAS | HV | SP | PAS | |
| MO-Ant | 5.04 | 3.53 | 0.36 | 5.70 | 3.37 | 0.85 | 5.83 | 1.69 | 0.91 |
| MO-HalfCheetah | 4.00 | 0.34 | 0.79 | 5.69 | 0.09 | 0.98 | 5.73 | 0.08 | 0.97 |
| MO-Hopper | 0.91 | 0.07 | 0.28 | 1.95 | 0.14 | 0.54 | 1.89 | 0.13 | 0.60 |
| MO-Walker2d | 4.80 | 1.21 | 0.17 | 4.87 | 0.20 | 0.99 | 5.05 | 0.15 | 0.99 |
| MO-Swimmer | 2.79 | 71.33 | 0.67 | 3.03 | 11.13 | 0.99 | 3.09 | 9.15 | 0.99 |
| Method | Ant | HalfCheetah | Hopper | Swimmer | Walker2d | Hopper-3obj | |
| (1) | FP2-Pre | ||||||
| DDIM-Pre | |||||||
| FP2-Mid | |||||||
| DDIM-Mid | |||||||
| (2) | FP2-Mid | ||||||
| w/o Text.Pref. |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| #Objectives | Preference range | #Preference | #Optimal traj. | Total traj. | |
| MO-Ant | 2 | 501 | 102/65 | 50k | |
| MO-HalfCheetah | 2 | 501 | 1547/459 | 50k | |
| MO-Hopper | 2 | 501 | 223/52 | 50k | |
| MO-Walker2d | 2 | 501 | 2055/411 | 50k | |
| MO-Swimmer | 2 | 501 | 1475/58 | 50k | |
| MO-Hopper-3obj | 3 | 501 | 8945/1925 | 50k |
| #Optimal traj. (E/A) | #Suboptimal traj. (E/A) | Total traj. (E/A) | |
| MO-Ant | 98/58 | 166/92 | 264/150 |
| MO-HalfCheetah | 970/393 | 1245/531 | 2215/924 |
| MO-Hopper | 182/47 | 250/71 | 432/118 |
| MO-Walker2d | 1688/360 | 2077/590 | 3765/950 |
| MO-Swimmer | 1032/55 | 1230/92 | 2262/147 |
| MO-Hopper-3obj | 6573/1583 | 5179/1911 | 11752/3494 |
| Parameter | Value | ||
| Pre-train | Mid-train | Post-train | |
| Learning Rate | 2e-4 | 2e-4 | 1e-5 |
| Weight Decay | 1e-5 | 1e-5 | 1e-5 |
| Batch Size/Mini Batch | 2048/256 | 64/64 | 64/64 |
| Training Steps | 400K | 200K | 100K |
| Aspect | FP2 | MODULI |
| Architecture | DiT-1D | DiT-1D |
| Generative backbone | RF | Diffusion |
| Sampler | 10-step Euler | 10-step DDIM |
| Additional modules | Classifier + text encoders | ControlNet + inverse-dynamics MLP |
| Trainable parameters | 12.8M | 23.4M |
| Training steps | 600K | 400K |
| Metric | Min | P25 | Median | Mean | P75 | Max | Win/Tie/Loss |
| HV | -3.75 | +0.35 | +0.98 | +2.96 | +3.62 | +20.95 | 30/3/3 |
| SP | -25.00 | +11.04 | +29.44 | +33.78 | +59.52 | +90.00 | 29/4/3 |
| PAS | -7.07 | +0.00 | +1.04 | +5.51 | +3.17 | +110.71 | 24/9/3 |
| Environment | Metric | B | MODT(P) | MORvS(P) | BC(P) | CQL(P) | MODULI ∗ | MODULI | FP2-Pre | FP2-Mid | |
| Expert | MO-Ant | HV ( ) | |||||||||
| SP ( ) | |||||||||||
| PAS | |||||||||||
| MO-HalfCheetah | HV ( ) | ||||||||||
| SP ( ) | |||||||||||
| PAS |
| Environment | Metric | MODT(P) | MORvS(P) | BC(P) | MODULI ∗ | MODULI | FP2-Pre | FP2-Mid | |
| Expert - Shattered | MO-Ant | HV ( ) | |||||||
| SP ( ) | |||||||||
| PAS | |||||||||
| MO-HalfCheetah | HV ( ) | ||||||||
| SP ( ) | |||||||||
| PAS |
| Environment | Metric | MODT(P) | MORvS(P) | BC(P) | MODULI ∗ | MODULI | FP2-Pre | FP2-Mid | |
| Amateur - Shattered | MO-Ant | HV ( ) | |||||||
| SP ( ) | |||||||||
| PAS | |||||||||
| MO-HalfCheetah | HV ( ) | ||||||||
| SP ( ) | |||||||||
| PAS |
| Dataset | Expert - Shattered | Expert - Narrow | |||||
| Environment | Metric | FP2-Mid (Zero-shot) | Scratch | FP2-Post | FP2-Mid (Zero-shot) | Scratch | FP2-Post |
| MO-Ant | HV ( ) | ||||||
| SP ( ) | |||||||
| PAS | |||||||
| MO-HalfCheetah | HV ( ) | ||||||
| SP ( ) | |||||||
| Method | MO-Ant ( ) | MO-HalfCheetah ( ) | MO-Hopper ( ) | MO-Swimmer ( ) | MO-Walker2d ( ) | MO-Hopper-3obj ( ) |
| Offline-COLA | ||||||
| MO-Q-chunking | ||||||
| FP2 |
| Item | Setting |
| Target environments | Ten Meta-World tasks: reach-v3 , push-v3 , pick-place-v3 , door-open-v3 , drawer-open-v3 , drawer-close-v3 , button-press-topdown-v3 , peg-insert-side-v3 , window-open-v3 , window-close-v3 |
| Source models | FP2 and MODULI trained on all six MO-MuJoCo environments (MO-Ant, MO-HalfCheetah, MO-Hopper, MO-Swimmer, MO-Walker2d, MO-Hopper-3obj) |
| Target-task data | None; all ten tasks are evaluated zero-shot |
| Objectives | Movement speed and control cost |
| Observation | Positions of the end-effector, object, and goal |
| Action | End-effector position |
| Metric | Method | Reach | Push | Pick-Place | Door-Open | Drawer-Open | Drawer-Close | Button-Press | Peg-Insert | Window-Open | Window-Close | Avg |
| HV | MODULI | 1.38 | 1.40 | 1.39 | 1.44 | 1.36 | 1.37 | 1.42 | 1.40 | 1.36 | 1.42 | 1.39 |
| FP2 | 1.65 | 1.53 | 1.52 | 1.50 | 1.46 | 1.49 | 1.50 | 1.55 | 1.47 | 1.51 | 1.52 | |
| SP | MODULI | 28.3 | 33.6 | 4.4 | 4.7 | 10.2 | 14.4 | 12.6 | 11.0 | 4.5 | 8.7 | 13.24 |
| FP2 | 0.8 | 4.1 | 1.7 | 4.1 | 2.2 | 6.4 | 1.6 | 2.4 | 4.0 | 3.7 | 3.10 |
Multi-Objective Reinforcement Learning
Multi-objective reinforcement learning studies how agents can balance competing goals and learn policies for different trade-offs.
See all 161 papers