Latent Imagination for Parameterized Action Spaces
Organizations: Turkish Aerospace.
Abstract
Parameterized-action MDPs (PAMDPs) ask an agent to choose a discrete action together with continuous parameters whose meaning depends on that choice. Model-free PAMDP methods act fast but need many interactions; model-based methods are sample-efficient but plan at every decision, which makes them slow to act. We introduce HyLI (Hybrid Latent Imagination), a model-based agent that learns a hybrid policy entirely in the latent imagination of a world model and acts with a single forward pass. To address learning problems that the PAMDP structure creates, HyLI adds counterfactual mode evaluation, which gives every discrete action a learning signal at every imagined state, and success-window replay, which keeps rare successes in training. On seven standard PAMDP benchmarks, HyLI is within one published standard deviation of, or above, the published score of the strongest model-free baseline in our comparison on all 7 tasks, reaches 90% of that score with 2.4x fewer environment steps than that baseline (5.0x when its reward-free pre-training is counted), and decides 67x faster than the model-based planner DLPA at its published planner settings. It is the only method in our comparison that is both more sample-efficient than that baseline and fast enough for real-time control.
Figures & tables
| Task | Budget | HyLI (ours) | DLPA † | HyAR | P-DQN ‡ | HHQN | PA-TD3 | H-PPO |
|---|---|---|---|---|---|---|---|---|
| Platform | 2e5 | 1.00 0.00 | 0.97 0.03 | 0.98 0.01 | 0.84 0.13 | 0.42 0.22 | 0.92 0.03 | 0.81 0.03 |
| Goal | 3e5 | 0.77 0.03 | 0.58 0.29 | 0.74 0.02 | 0.61 0.20 | 0.00 0.00 | 0.00 0.00 | 0.00 0.00 |
| Hard Goal | 3e5 | 0.79 0.04 | 0.34 0.28 | 0.59 0.08 | 0.07 0.08 | 0.02 0.04 | 0.14 0.18 | 0.00 0.00 |
| Catch Point | 5e5 | 0.94 0.00 | 0.92 0.03 | 0.70 0.18 | 0.92 0.05 | 0.33 0.17 | 0.59 0.17 | 0.72 0.06 |
| Hard Move ( ) | 5e5 | 0.91 0.05 | 0.94 0.01 | 0.95 0.01 | 0.30 0.05 | 0.55 0.19 | 0.49 0.18 | 0.64 0.12 |
| Hard Move ( ) | 5e5 | 0.95 0.03 | 0.94 0.01 | 0.93 0.04 | 0.34 0.06 | 0.17 0.05 | 0.07 0.04 | 0.59 0.19 |
| Task | HyLI (ours) | DLPA | HyAR | P-DQN | ratio | |
|---|---|---|---|---|---|---|
| Platform | 3 | 1.29 | 108 | 0.343 | 0.11 | 84 |
| Goal | 3 | 1.3 | 51.6 | 0.347 | 0.112 | 40 |
| H. Goal | 11 | 1.56 | 62 | 0.347 | 0.107 | 40 |
| Catch | 2 | 1.28 | 52.4 | 0.337 | 0.106 | 41 |
| HM4 | 16 | 1.09 | 46.2 | 0.345 | 0.106 | 42 |
| HM6 | 64 | 1.1 | 95.9 | 0.346 | 0.109 | 87 |
| Task | HyLI (ours) | DLPA † | HyAR | HyAR § | P-DQN ‡ | HHQN | PA-TD3 | H-PPO |
|---|---|---|---|---|---|---|---|---|
| Platform (0.88) | 4.5k | 1.5k | 25k | 14k | 9k | 200k (2/5) | 21.5k | 200k (0/5) |
| Goal (0.70) | 45k | 47k (4/5) | 85.8k | 65k | 31.5k (4/5) | 300k (0/5) | 300k (0/5) | 300k (0/5) |
| Hard Goal (0.54) | 29.5k | 29k (3/5) | 105k | 84.5k | 300k (0/5) | 300k (0/5) | 300k (2/5) | 300k (0/5) |
| Catch Point (0.81) | 20k | 10k | 221k | 115k | 105k | 500k (0/5) | 500k (2/5) | 432k (4/5) |
| Hard Move ( ) (0.84) | 27.5k | 15k | 185k | 40k | 500k (2/5) | 55k | 500k (1/5) | 500k (2/5) |
| Hard Move ( ) (0.83) | 35k | 22.5k | 212k | 67.5k | 340k (4/5) | 500k (0/5) | 500k (0/5) | 500k (0/5) |
| Goal | Hard Goal | |||
| score | steps | score | steps | |
| Full method | 0.77 (0.74) [0] | 45k | 0.80 (0.70) [0] | 29.5k |
| SWR | 0.76 (0.69) [0] | 47.5k | 0.80 (0.75) [0] | 42.5k |
| CME | 0.73 (0.63) [0] | 46k | 0.76 (0.62) [0] | 23k |
| -reg | 0.72 (0.62) [0] | 40k | 0.74 (0.23) [1] | 24k (4/5) |
| -mode | 0.64 (0.38) [1] | 55.5k | 0.76 (0.71) [0] | 24k |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | DreamerV3 | Ours (reason) |
| deterministic state | 512 | 256 (4–17-dim observations) |
| stochastic state | (same) | |
| MLP width / embed | 512 / 256 | 256 / 128 (same) |
| imagination horizon | 15 | 10 (short episodes) |
| discount | 0.997 | 0.99 (short episodes) |
| model / actor / critic lr | 1e-4 / 3e-5 / 3e-5 | 3e-4 / 1e-4 / 1e-4 (budgets ) |
| Task | candidates | base | full method | ratio | PA-Dreamer | |
|---|---|---|---|---|---|---|
| Goal | 3 | 3 | 115 | 169 | 1.47 | 103 |
| Hard Goal | 11 | 11 | 122 | 247 | 2.03 | 100 |
| Catch Point | 2 | 2 | 116 | 159 | 1.37 | – |
| Hard Move ( ) | 16 | 16 | 112 | 262 | 2.34 | – |
| Hard Move ( ) | 256 | 16 | 111 | 284 | 2.56 | – |
| Task | H-PPO | PA-TD3 | P-DQN-TD3 | HHQN-TD3 | HyAR-TD3 |
|---|---|---|---|---|---|
| Platform | 0.80 0.02 | 0.94 0.10 | 0.93 0.03 | 0.62 0.23 | 0.98 0.01 |
| Goal | 0.00 0.00 | 0.00 0.00 | 0.71 0.07 | 0.00 0.00 | 0.78 0.03 |
| Hard Goal | 0.00 0.00 | 0.44 0.05 | 0.06 0.07 | 0.01 0.01 | 0.60 0.07 |
| Catch Point | 0.69 0.09 | 0.82 0.10 | 0.89 0.07 | 0.27 0.05 | 0.90 0.03 |
| Hard Move ( ) | 0.09 0.02 | 0.66 0.13 | 0.85 0.10 | 0.52 0.17 | 0.93 0.02 |
| Hard Move ( ) | 0.05 0.01 | 0.04 0.02 | 0.74 0.08 | 0.29 0.13 | 0.92 0.04 |
| Platform | Goal | Hard Goal | Catch Point | HM4 | HM6 | HM8 | HM10 | |
|---|---|---|---|---|---|---|---|---|
| DLPA (published) | 0.92 0.05 | 28.75 6.91 | 28.38 2.88 | 7.56 4.86 | 6.29 5.74 | 8.48 5.45 | 7.80 6.27 | 6.35 9.97 |
| CHDP (published) | 99.7 0.2 | 80.9 4.9 | 79.5 5.0 | 93.8 0.6 | 94.2 1.7 | 93.9 1.0 | 90.6 2.2 | 79.8 5.4 |
| Arm | Task | published | ours, own budget | ours, common window | |
|---|---|---|---|---|---|
| HyAR | Platform | 0.98 0.01 | 0.97 0.03 | 0.98 0.01 | 0.8 |
| Goal | 0.78 0.03 | 0.74 0.03 | 0.74 0.02 | 1.3 | |
| Hard Goal | 0.60 0.07 | 0.57 0.11 | 0.59 0.08 | 0.4 | |
| Catch Point | 0.90 0.03 | 0.85 0.15 | 0.70 0.18 | 1.7 | |
| Hard Move ( ) | 0.93 0.02 | 0.84 0.09 | 0.95 0.01 | 4.6 | |
| Hard Move ( ) | 0.92 0.04 | 0.87 0.06 | 0.93 0.04 | 1.2 |
| budget of Zhang et al. (2024, Fig. 4) | converged budget | ||||||
|---|---|---|---|---|---|---|---|
| Task | published | steps | ours | steps | ours | ||
| Platform | 0.92 0.05 | 2e4 | 0.85 0.28 | 1.5 | 5e4 | 0.98 0.02 | +1.1 |
| Goal | 28.75 6.91 | 2e4 | 16.54 11.51 | 1.8 | 1.5e5 | 25.99 16.09 | 0.4 |
| Hard Goal | 28.38 2.88 | 5e4 | 11.66 14.68 | 5.8 | 9e4 | 13.17 15.82 | 5.3 |
| Catch Point | 7.56 4.86 | 2e5 | 6.90 0.64 | 0.1 | not extended | ||
| Hard Move ( ) | 6.29 5.74 | 1.5e5 | 6.91 0.49 | +0.1 | not extended | ||
| budget (env steps) | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Task | eval every | max steps | metric | reference | ours | DLPA | DLPA (published) | ||
| Platform | 3 | 3 | 500 | 250 | return | 2e5 | 2e5 | 5e4 | 2e4 |
| Goal | 3 | 4 | 500 | 150 | success | 3e5 | 3e5 | 1.5e5 | 2e4 |
| Hard Goal | 11 | 12 | 500 | 150 | success | 3e5 | 3e5 | 9e4 | 5e4 |
| Catch Point | 2 | 2 | 2 500 | 25 | success | 1e6 | 5e5 | 2e5 | 2e5 |
| Hard Move ( ) | 16 | 16 | 2 500 | 30 | success | 1e6 | 5e5 | 1.5e5 | 1.5e5 |
| Task | nominal modes | distinct motions | zero-motion modes |
|---|---|---|---|
| Hard Move ( ) | 16 | 9 | 4 |
| Hard Move ( ) | 64 | 19 | 10 |
| Hard Move ( ) | 256 | 81 | 16 |
| Hard Move ( ) | 1024 | 211 | 34 |
| Arm | Warm-up | Counted in budget |
|---|---|---|
| HyAR | pre-training: 5 000 random episodes (20 000 on Goal/Hard Goal) that train only the action VAE; then 128 random steps before the first policy update | yes, both (the reference code does not count the pre-training; we add it) |
| DLPA | first 50 episodes with random actions, no model update | yes |
| PA-TD3, HHQN | 128 steps with the noisy (not random) policy, no update | yes |
| P-DQN | 128 steps (500 on Platform), -greedy from , no update | yes |
| H-PPO | none (on-policy) | – |
| HyLI | random prefill: 5 000 steps on Goal/Hard Goal, 1 000 elsewhere; then -mode exploration ( 0.1 0 over steps) on Goal/Hard Goal only | yes |
| Task | Budget | HyLI (ours) | DLPA † | HyAR | P-DQN ‡ | HHQN | PA-TD3 | H-PPO |
|---|---|---|---|---|---|---|---|---|
| Platform | 2e5 | 1.00 0.00 | 0.97 0.03 | 0.97 0.03 | 0.84 0.13 | 0.42 0.22 | 0.92 0.03 | 0.81 0.03 |
| Goal | 3e5 | 0.77 0.03 | 0.58 0.29 | 0.74 0.03 | 0.61 0.20 | 0.00 0.00 | 0.00 0.00 | 0.00 0.00 |
| Hard Goal | 3e5 | 0.79 0.04 | 0.34 0.28 | 0.57 0.11 | 0.07 0.08 | 0.02 0.04 | 0.14 0.18 | 0.00 0.00 |
| Catch Point | 5e5 / 1e6 | 0.94 0.00 | 0.92 0.03 | 0.85 0.15 | 0.92 0.03 | 0.24 0.14 | 0.50 0.05 | 0.76 0.06 |
| Hard Move ( ) | 5e5 / 1e6 | 0.91 0.05 | 0.94 0.01 | 0.84 0.09 | 0.43 0.07 | 0.36 0.19 | 0.85 0.07 | 0.64 0.07 |
| Hard Move ( ) | 5e5 / 1e6 | 0.95 0.03 | 0.94 0.01 | 0.87 0.06 | 0.33 0.09 | 0.29 0.17 | 0.04 0.02 | 0.63 0.10 |
| Task | HyLI (ours) | DLPA † | HyAR | P-DQN ‡ | HHQN | PA-TD3 | H-PPO |
|---|---|---|---|---|---|---|---|
| Platform | 1.00 0.00 | 0.98 0.03 | 0.97 0.01 | 0.89 0.03 | 0.44 0.23 | 0.91 0.08 | 0.80 0.01 |
| Goal | 0.76 0.03 | 0.57 0.29 | 0.72 0.02 | 0.61 0.18 | 0.00 0.00 | 0.00 0.00 | 0.00 0.00 |
| Hard Goal | 0.78 0.03 | 0.34 0.28 | 0.58 0.08 | 0.06 0.08 | 0.01 0.02 | 0.18 0.17 | 0.00 0.00 |
| Catch Point | 0.92 0.03 | 0.92 0.01 | 0.72 0.17 | 0.92 0.05 | 0.31 0.14 | 0.61 0.14 | 0.70 0.04 |
| Hard Move ( ) | 0.92 0.02 | 0.95 0.00 | 0.95 0.01 | 0.35 0.05 | 0.63 0.13 | 0.48 0.21 | 0.65 0.12 |
| Hard Move ( ) | 0.95 0.02 | 0.94 0.01 | 0.93 0.03 | 0.33 0.06 | 0.18 0.07 | 0.06 0.03 | 0.57 0.20 |
| Task | HyLI (ours) | DLPA † | HyAR | P-DQN ‡ | HHQN | PA-TD3 | H-PPO |
|---|---|---|---|---|---|---|---|
| Platform | 0.98 0.00 | 0.96 0.06 | 0.90 0.02 | 0.87 0.05 | 0.41 0.10 | 0.88 0.06 | 0.65 0.02 |
| Goal | 0.65 0.02 | 0.54 0.27 | 0.59 0.04 | 0.59 0.15 | 0.00 0.00 | 0.00 0.00 | 0.00 0.00 |
| Hard Goal | 0.69 0.02 | 0.33 0.27 | 0.46 0.11 | 0.02 0.03 | 0.02 0.01 | 0.22 0.15 | 0.00 0.00 |
| Catch Point | 0.90 0.02 | 0.92 0.03 | 0.64 0.12 | 0.84 0.03 | 0.32 0.05 | 0.39 0.13 | 0.45 0.02 |
| Hard Move ( ) | 0.87 0.02 | 0.94 0.02 | 0.88 0.01 | 0.33 0.03 | 0.70 0.08 | 0.24 0.10 | 0.44 0.08 |
| Hard Move ( ) | 0.89 0.03 | 0.93 0.02 | 0.83 0.03 | 0.27 0.03 | 0.22 0.08 | 0.05 0.01 | 0.29 0.09 |
| Task | thr. | HyLI (ours) | DLPA † | HyAR | P-DQN ‡ | HHQN | PA-TD3 | H-PPO |
|---|---|---|---|---|---|---|---|---|
| Platform | 75% | 4k | 1k | 19.5k | 5k | 200k (2/5) | 17k | 93k |
| 90% | 4.5k | 1.5k | 25k | 9k | 200k (2/5) | 21.5k | 200k (0/5) | |
| 100% | 5k | 8.5k | 40k | 47k | 200k (0/5) | 27k | 200k (0/5) | |
| Goal | 75% | 32.5k | 20.5k (4/5) | 60.3k | 20.5k (4/5) | 300k (0/5) | 300k (0/5) | 300k (0/5) |
| 90% | 45k | 47k (4/5) | 85.8k | 31.5k (4/5) | 300k (0/5) | 300k (0/5) | 300k (0/5) | |
| 100% | 105k | 73k (4/5) | 95.8k | 78.5k (4/5) | 300k (0/5) | 300k (0/5) | 300k (0/5) |
| Task | HyLI (ours) | DLPA | HyAR | P-DQN | HHQN | PA-TD3 | H-PPO | |
|---|---|---|---|---|---|---|---|---|
| CPU, 1 thread | ||||||||
| Platform | 3 | 1.29 (1.31) | 108 (109) | 0.343 (0.346) | 0.11 (0.111) | 0.128 (0.13) | 0.0879 (0.0888) | 0.21 (0.212) |
| Goal | 3 | 1.3 (1.32) | 51.6 (51.8) | 0.347 (0.349) | 0.112 (0.113) | 0.128 (0.129) | 0.0886 (0.0921) | 0.213 (0.223) |
| H. Goal | 11 | 1.56 (1.59) | 62 (62.4) | 0.347 (0.35) | 0.107 (0.108) | 0.128 (0.129) | 0.0891 (0.0929) | 0.212 (0.214) |
| Catch | 2 | 1.28 (1.3) | 52.4 (52.6) | 0.337 (0.341) | 0.106 (0.108) | 0.128 (0.13) | 0.0889 (0.0933) | 0.209 (0.211) |
| HM4 | 16 | 1.09 (1.11) | 46.2 (46.5) | 0.345 (0.348) | 0.106 (0.107) | 0.128 (0.129) | 0.0883 (0.0895) | 0.21 (0.212) |
| Task | HyLI (ours) | DLPA † | HyAR | P-DQN ‡ | HHQN | PA-TD3 | H-PPO |
|---|---|---|---|---|---|---|---|
| Platform | 2e5 | 5e4 | 2e5 ∘ | 2e5 ∘ | 2e5 ∘ | 2e5 ∘ | 2e5 ∘ |
| Goal | 3e5 | 1.5e5 | 3e5 | 3e5 | 3e5 | 3e5 | 3e5 |
| Hard Goal | 3e5 | 9e4 | 3e5 | 3e5 | 3e5 | 3e5 | 3e5 |
| Catch Point | 5e5 | 2e5 | 1e6 | 1e6 | 1e6 | 1e6 | 1e6 |
| Hard Move ( ) | 5e5 | 1.5e5 | 1e6 | 1e6 | 1e6 | 1e6 | 1e6 |
| Hard Move ( ) | 5e5 | 8e4 | 1e6 | 1e6 | 1e6 | 1e6 | 1e6 |
| Task | Head | Head width | Score | IQM [95% CI] | Worst | Fail | Steps to thr. |
|---|---|---|---|---|---|---|---|
| Goal | sub ⋆ | default | 0.77 0.03 | 0.77 [0.74, 0.80] | 0.74 | 0/5 | 45k |
| multi | default | 0.73 0.03 | 0.72 [0.69, 0.77] | 0.69 | 0/5 | 57.5k | |
| multi, param-matched | 552 | 0.75 0.10 | 0.78 [0.62, 0.83] | 0.57 | 0/5 | 45.5k (4/5) | |
| Hard Goal | sub ⋆ | default | 0.79 0.04 | 0.80 [0.73, 0.82] | 0.70 | 0/5 | 29.5k |
| multi | default | 0.77 0.03 | 0.77 [0.74, 0.81] | 0.74 | 0/5 | 27.5k | |
| multi, param-matched | 1225 | 0.77 0.04 | 0.76 [0.72, 0.81] | 0.72 | 0/5 | 19.5k |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,835 papers