MAC-v2: Long-horizon model-based value expansion from pixels
Organizations: UC Berkeley
Abstract
We study whether long-horizon model-based value expansion can serve as an effective framework in pixel-based offline reinforcement learning (RL). Long-horizon model-based value expansion from pixels is substantially harder than state-based environments, as the agent must learn both the representation and its dynamics, and errors from either can accumulate recursively over imagined trajectories. We introduce MAC-v2, which enables long-horizon model-based expansion from pixels by training a reconstruction-free latent world model with action chunking. By learning representations from prediction and control objectives, MAC-v2 does not need to encode control-irrelevant visual details that can hinder the dynamics prediction. Action chunking further mitigates the prediction error by reducing the number of recursive model predictions required for value expansion. Across diverse visual benchmarks including visual OGBench, MMBench2, Robomimic, and LIBERO-10, MAC-v2 achieves strong performance against model-free and model-based offline RL baselines. Our analyses show that long-horizon model-based value expansion is critical for the performance of MAC-v2, and that both action chunking and latent learning objectives are essential for reliable long-horizon latent imagination.
Figures & tables
| Model-Free | Model-Based | |||||||||||||||
| Environment | IQL | ReBRAC | IFQL | FQL | QC | LOMPO | OffDV2 | MAC-v2 | ||||||||
| OGBench (25 tasks) | ||||||||||||||||
| visual-cube-single (5 tasks) | 65 | 4 | 74 | 1 | 36 | 5 | 62 | 4 | 78 | 2 | 0 | 0 | 0 | 0 | 74 | 1 |
| visual-cube-double (5 tasks) | 13 | 1 | 4 | 3 | 2 | 1 | 5 | 3 | 3 | 1 | 0 | 0 | 0 | 0 | 53 | 3 |
| visual-scene (5 tasks) | 18 | 8 | 33 | 4 | 21 | 2 | 41 | 2 | 57 | 2 | 2 | 1 | 4 | 2 | 72 | 12 |
| visual-puzzle-3x3 (5 tasks) | 2 | 3 | 19 | 1 | 21 | 1 | 22 | 1 | 20 | 0 | 3 | 2 | 2 | 1 | 95 | 4 |
| Model-Free | Model-Based | |||||||||||
| Environment | BC | QC | QC-IFQL | LOMPO | OffDV2 | MAC-v2 | ||||||
| Robomimic (3 tasks) | ||||||||||||
| lift | 100 | 0 | 100 | 0 | 100 | 0 | 27 | 42 | 51 | 19 | 100 | 0 |
| can | 97 | 3 | 97 | 3 | 92 | 3 | 0 | 0 | 0 | 0 | 95 | 3 |
| square | 68 | 21 | 55 | 13 | 55 | 10 | 0 | 0 | 0 | 0 | 72 | 10 |
| LIBERO (10 tasks) | ||||||||||||
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Resolution | Encoder | Aug. pad |
| OGBench | impala_small | ||
| MMBench2 | 12M Conv. | ||
| Robomimic | 12M Conv. | ||
| LIBERO-10 | ResNet-18 |
| Hyperparameters | Value |
| Gradient steps | K |
| Learning rate | |
| Optimizer | Adam ( Kingma & Ba, 2015 ) |
| Batch size | |
| MLP size | |
| Frame stack |
| Hyperparameters | Value |
| Gradient steps | K |
| Learning rate | |
| Optimizer | Adam ( Kingma & Ba, 2015 ) |
| Batch size | |
| MLP size | |
| Image resolution |
| Hyperparameters | Value |
| Learning rate | |
| Optimizer | Adam |
| Layer normalization | True |
| Target network update rate | |
| Discount factor | |
| -return coefficient |
| Environment | LOMPO | OffDV2 | FQL | ReBRAC | IQL |
| visual-cube-single-play-v0 | |||||
| visual-cube-double-play-v0 | |||||
| visual-scene-play-v0 | |||||
| visual-puzzle-3x3-play-v0 | |||||
| visual-puzzle-4x4-play-v0 |
| Environment | LOMPO | OffDV2 | FQL | ReBRAC | IQL |
| cup-catch-var1 | |||||
| finger-turn-easy-var1 | |||||
| ms-push-banana | |||||
| og-point-var1 | N/A | ||||
| og-point-var2 | |||||
| pygame-point-maze-var4 |
| Task | LOMPO | OffDV2 | BC | QC | QC-IFQL |
| lift | |||||
| can | N/A | N/A | |||
| square | N/A | N/A |
| Task | LOMPO | OffDV2 |
| put the white mug on the left plate and put the yellow and white mug on the right plate | N/A | N/A |
| put the white mug on the plate and put the chocolate pudding to the right of the plate | N/A | N/A |
| put the yellow and white mug in the microwave and close it | N/A | |
| turn on the stove and put the moka pot on it | N/A | N/A |
| put both the alphabet soup and the cream cheese box in the basket | N/A | N/A |
| put both the alphabet soup and the tomato sauce in the basket | N/A |
| Model-Free | Model-Based | |||||||||||||||
| Environment | IQL | ReBRAC | IFQL | FQL | QC | LOMPO | OffDV2 | MAC-v2 | ||||||||
| visual-cube-single-task1 | 73 | 7 | 84 | 4 | 44 | 6 | 67 | 3 | 88 | 5 | 0 | 0 | 0 | 0 | 87 | 2 |
| visual-cube-single-task2 | 57 | 6 | 71 | 6 | 29 | 9 | 70 | 4 | 80 | 5 | 0 | 0 | 0 | 0 | 72 | 3 |
| visual-cube-single-task3 | 79 | 4 | 85 | 1 | 59 | 10 | 68 | 13 | 90 | 4 | 0 | 0 | 0 | 0 | 86 | 3 |
| visual-cube-single-task4 | 62 | 18 | 69 | 4 | 27 | 7 | 51 | 11 | 68 | 5 | 0 | 0 | 0 | 0 | 64 | 3 |
| visual-cube-single-task5 | 54 | 4 | 62 | 4 | 22 | 4 | 57 | 8 | 62 | 6 | 0 | 0 | 0 | 0 | 62 | 3 |
| Model-Free | Model-Based | |||||||||||||||
| Environment | IQL | ReBRAC | IFQL | FQL | QC | LOMPO | OffDV2 | MAC-v2 | ||||||||
| cup-catch-var1 | 13 | 2 | 47 | 6 | 11 | 1 | 71 | 1 | 22 | 4 | 4 | 3 | 45 | 26 | 33 | 4 |
| finger-turn-easy-var1 | 24 | 5 | 32 | 4 | 23 | 4 | 35 | 3 | 23 | 2 | 24 | 9 | 43 | 1 | 29 | 5 |
| ms-push-banana | 5 | 2 | 8 | 3 | 11 | 4 | 3 | 2 | 3 | 2 | 23 | 3 | 0 | 0 | 4 | 2 |
| og-point-var1 | 8 | 5 | 17 | 2 | 11 | 3 | 32 | 6 | 7 | 2 | 0 | 0 | 1 | 0 | 42 | 5 |
| og-point-var2 | 7 | 4 | 14 | 3 | 9 | 3 | 28 | 5 | 6 | 2 | 0 | 0 | 1 | 0 | 24 | 4 |
| Model-Free | Model-Based | |||||||||||
| Environment | BC | QC | QC-IFQL | LOMPO | OffDV2 | MAC-v2 | ||||||
| LIBERO (10 tasks) | ||||||||||||
| turn on the stove and put the moka pot on it | 96 | 3 | 89 | 10 | 85 | 9 | 0 | 0 | 0 | 0 | 97 | 2 |
| put the black bowl in the bottom drawer of the cabinet and close it | 98 | 1 | 86 | 9 | 94 | 3 | 0 | 0 | 0 | 0 | 95 | 3 |
| put the yellow and white mug in the microwave and close it | 84 | 3 | 77 | 7 | 83 | 8 | 9 | 12 | 0 | 0 | 81 | 8 |
| put both moka pots on the stove | 76 | 4 | 69 | 2 | 64 | 10 | 0 | 0 | 0 | 0 | 75 | 16 |
Model-Based RL
Model-based reinforcement learning uses environment models to plan actions and learn policies with fewer real-world interactions.
See all 273 papers