Learning When to Think: Adaptive Internal Computation for Reinforcement Learning
Organizations: Virginia Polytechnic Institute and State University
Abstract
Iterative neural policies usually spend the same computation on every decision, although the benefit of additional processing varies across states. We introduce Adaptive Internal Computation (AIC), an iterative policy that learns when to stop refining its latent representation and act. In AIC, we train action selection and stopping jointly, an expected-depth penalty values computation and intermediate value predictions provide auxiliary supervision. We show that when only some states benefit from further refinement, an ideal state-dependent allocator outperforms every state-independent stopping schedule at the same expected depth. With at most eight refinement steps per decision, AIC executes 3.3 steps on average across twelve benchmark tasks and lowers per-decision latency by 31--44% relative to a fixed eight-step policy on four control and robotics tasks. Its mean reward exceeds the best fixed-depth policy on seven tasks and trails it on four. Ablating adaptive stopping and value supervision separately shows that stopping accounts for most of the reward gain on PickCube and AntMaze, where it also executes more refinement steps and that supervision accounts for most of it on RareEvent.
Figures & tables
| Task | Fixed ( ) | AIC | Privileged | % | Gap recovery | Best fixed ( ) | ||
|---|---|---|---|---|---|---|---|---|
| MiniGrid-DistShift2 (sat. ctrl.) | ( ) | — | ( ) | |||||
| HalfCheetah-v4 | ( ) | ( ) | ||||||
| GPT-2 on GLUE-RTE | ( ) | ( ) | ||||||
| OGBench cube-double-play-v0 | ( ) | ( ) | ||||||
| AntMaze (low-het.) | ( ) | ( ) | ||||||
| OGBench scene-play-v0 | ( ) | ( ) |
| POGP | AIC | |||
|---|---|---|---|---|
| Environment | Return | Steps | Return | Steps |
| HalfCheetah | ||||
| Walker2d | ||||
| Ant | ||||
| Hopper | ||||
| Task | Normal | Targeted Drop | Random Drop | Targeted Random | Ratio |
|---|---|---|---|---|---|
| MiniGrid-DistShift2 (sat.) | — | ||||
| AntMaze (low-het.) | — | ||||
| HalfCheetah-v4 | |||||
| GPT-2 on GLUE-RTE | |||||
| AntMaze (main) | |||||
| ManiSkill PickCube-v1 |
| ManiSkill PickCube-v1 | ManiSkill StackCube-v1 | |||
|---|---|---|---|---|
| Configuration | Reward | Reward | ||
| Fixed-1 collector Fixed-1 learner | ||||
| Fixed-1 collector AIC learner | ||||
| AIC collector Fixed-1 learner | ||||
| AIC collector AIC learner | ||||
| Neither | Supervision only | Adaptive stopping only | Full AIC | |||||
|---|---|---|---|---|---|---|---|---|
| Task | Reward | Reward | Reward | Reward | ||||
| PickCube | 1.00 | 1.00 | 3.61 | 3.52 | ||||
| AntMaze | 1.00 | 1.00 | 3.28 | 3.19 | ||||
| Memory | 1.00 | 1.00 | 3.42 | 3.33 | ||||
| DoorKey | 1.00 | 1.00 | 3.49 | 3.38 | ||||
| StackCube | 1.00 | 1.00 | 3.94 | 3.81 | ||||
| Trained jointly with the policy | Untrained stopping rules | |||||||
| Task | AIC | ACT-ponder | Lrn non-VoC | Conf. thr. | Action-agr. | Val.-gain thr. | Random | Fixed |
| PickCube | ||||||||
| StackCube | ||||||||
| AntMaze (main) | ||||||||
| GLUE-RTE | ||||||||
| Scene | ||||||||
| Task | DiffusionQL | DPPO | D3P | DSAC-D | IDQL |
|---|---|---|---|---|---|
| HalfCheetah-v4 | |||||
| ManiSkill PickCube-v1 | |||||
| ManiSkill StackCube-v1 | |||||
| OGBench scene-play-v0 | |||||
| AntMaze (main) | |||||
| OGBench puzzle-4x4-play-v0 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Domain | Source | Role |
|---|---|---|---|
| HalfCheetah-v4 | Continuous control | Gym MuJoCo | anchor |
| ManiSkill PickCube-v1 | Robotic manipulation | ManiSkill3 | anchor |
| ManiSkill StackCube-v1 | Sequential manipulation | ManiSkill3 | anchor |
| AntMaze (main) | Maze navigation | D4RL/Gymnasium-Robotics | anchor |
| AntMaze (low-het.) | Maze nav. (low-heterogeneity) | D4RL variant | low-het. |
| GPT-2 on GLUE-RTE | Entailment classification | GLUE-RTE on GPT-2 | anchor |
| AIC | ACT-ponder | Learned non-VoC | |
|---|---|---|---|
| Halting input | stop-gradient copy of | ||
| Halting head | linear, one per depth | linear, shared across depths | two-layer MLP ( ), shared |
| Halting objective | joint PPO ratio over | task loss through the halting-weighted action mixture | cross-entropy toward |
| Stopping in PPO ratio | yes | through the mixture likelihood | no |
| Computation penalty | , | ponder cost , weight | none |
| Per-depth value supervision | yes, | yes, | yes, |
| Trained | Untrained / heuristic | |||||||
|---|---|---|---|---|---|---|---|---|
| Task | AIC | ACT | non-VoC | Conf. | Agr. | Val.-g. | Rand. | |
| HalfCheetah-v4 | ||||||||
| ManiSkill PickCube-v1 | ||||||||
| ManiSkill StackCube-v1 | ||||||||
| AntMaze (main) | ||||||||
| AntMaze (low-het.) | ||||||||
| .001 | ||||
|---|---|---|---|---|
| .003 | ||||
| .010 | ||||
| .030 | ||||
| .100 |
| .001 | ||||
|---|---|---|---|---|
| .003 | ||||
| .010 | ||||
| .030 | ||||
| .100 |
| Setting | Gap recovered |
|---|---|
| Default halting entropy | 6.1% |
| halting entropy | 6.8% |
| halting entropy | 7.4% |
| Privileged future-benefit probe | 29.8% |
| Task | AIC | Fixed ( ) | Ref. | Ref. gap | W/L/T | ||
|---|---|---|---|---|---|---|---|
| MiniGrid-DistShift2 | 1 | ( ) | — | — | 0/0/5 | ||
| 2 | ( ) | — | 0/0/5 | ||||
| 4 | ( ) | — | 0/0/5 | ||||
| 8 | ( ) | — | 0/0/5 | ||||
| HalfCheetah-v4 | 1 | ( ) | — | — | 2/3/0 | ||
| 2 | ( ) | 3/2/0 |
| Task | ||||
|---|---|---|---|---|
| MiniGrid-DistShift2 | ||||
| HalfCheetah-v4 | ||||
| GPT-2 on GLUE-RTE | ||||
| OGBench cube-double-play-v0 | ||||
| AntMaze (low-het.) | ||||
| OGBench scene-play-v0 |
| Selected | Removed refinements | Equal-work | Targeted | ||
| Task | fraction | Targeted | Random | random drop | equal-work random |
| MiniGrid-DistShift2 (sat.) | |||||
| AntMaze (low-het.) | |||||
| HalfCheetah-v4 | |||||
| GPT-2 on GLUE-RTE | |||||
| AntMaze (main) | |||||
| Phase | Share | Mean depth | High-depth states | Deeper-refinement gain | Early-exit change |
|---|---|---|---|---|---|
| Approach | 56.1% | 12% | |||
| Grasp/contact | 14.6% | 68% | |||
| Lift | 14.6% | 54% | |||
| Stabilization/recovery | 14.7% | 74% |
| AIC | ACT-ponder | non-VoC | Fixed | Tgt Rnd | P/R | |||
| Task | Rew. | Rew. | Rew. | Rew. | gap | (crit. states) | ||
| MiniGrid Memory | ||||||||
| MiniGrid DoorKey | ||||||||
| Benchmark tasks at , for scale. | ||||||||
| ManiSkill PickCube-v1 | ||||||||
| AntMaze (main) | ||||||||
| Setting | Fixed-8 | AIC | Mean steps (Fixed-8 / AIC) | Change |
|---|---|---|---|---|
| GPT-2/RTE inference | 9.41 ms | 8.68 ms | 8 / 4.08 | latency |
| HalfCheetah inference | 0.86 ms | 0.48 ms | 8 / 3.04 | latency |
| AntMaze inference | 0.89 ms | 0.53 ms | 8 / 3.15 | latency |
| PickCube inference | 0.92 ms | 0.57 ms | 8 / 3.47 | latency |
| StackCube inference | 0.94 ms | 0.65 ms | 8 / 3.77 | latency |
| PickCube training | 5.9 h | 7.2 h | all eight depths | time |
| Family | Training regime | Halting update | Critic or baseline | Fixed steps |
|---|---|---|---|---|
| DiffusionQL | offline | critic-weighted, score function | DiffusionQL double- critic | 8 |
| IDQL | offline | critic-weighted, score function | IQL -function | 8 |
| DSAC-D | off-policy | critic-weighted, score function | mean of the distributional critic | 8 |
| DPPO | on-policy | joint PPO ratio | GAE with state-only baseline | 8 |
| D3P | on-policy | joint PPO ratio | GAE with state-only baseline | 8 |
| Base Policy | Task / Env | Fixed | AIC | Gain | AIC Steps |
|---|---|---|---|---|---|
| DiffusionQL | HalfCheetah-v4 | ||||
| DiffusionQL | ManiSkill PickCube-v1 | ||||
| DiffusionQL | ManiSkill StackCube-v1 | ||||
| DiffusionQL | OGBench scene-play-v0 | ||||
| DiffusionQL | AntMaze main | ||||
| DiffusionQL | OGBench puzzle-4x4-play-v0 |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,855 papers