cs.AIOct 26, 2025

Toward Agents That Reason About Their Computation

Authors: Adrian Orenstein, Jessica Jiajing Chen, Gwyneth Anne Delos Santos, Bayley Sapara, Michael Bowling

Organizations: Department of Computing Science, University of Alberta, Canada · Alberta Machine Intelligence Institute (Amii) · Department of Computer Science, University of Toronto, Canada · CIFAR AI Chair

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

While reinforcement learning agents can achieve superhuman performance in many complex tasks, they typically do not become more computationally efficient as they improve. In contrast, humans gradually require less cognitive effort as they become more proficient at a task. If agents could reason about their compute as they learn, could they similarly reduce their computation footprint? If they could, we could have more energy efficient agents or free up compute cycles for other processes like planning. In this paper, we experiment with showing agents the cost of their computation and giving them the ability to control when they use compute. We conduct our experiments on the Arcade Learning Environment, and our results demonstrate that with the same training compute budget, agents that reason about their compute perform better on 75% of games. Furthermore, these agents use three times less compute on average. We analyze individual games and show where agents gain these efficiencies. An interactive version of this paper is available at https://adrianorenstein.com/paper/toward-compute.

Figures & tables

Figure 1: Improvement of Compute DQN over DQN in human normalized score (HNS) across 46 Atari games. Compute DQN achieves higher HNS in 75% of games. Amidar shows the largest gain of 487%, while the largest loss is 30% below DQN.
Figure 2: Compute DQN learns game-specific decision rates across the Arcade suite. Each green dot shows the average decisions per second for a single game, measured over 10 independently trained agents. The dashed green line indicates the suite-wide mean of 3.6 Hz. For comparison, Compute DQN uses 3.43.4 times less compute than DQN, which acts at a fixed 12 Hz (red dashed line). These results demonstrate that Compute DQN can adapt its compute use per game while maintaining performance.
Figure 3: Population dynamics of learning: each point is a moment in time during training, reported across 10 seeds per game. Color encodes training progress.
Figure 4: The decision rate within a trajectory differs by game. Pong and Asterix are uni modal, whereas Breakout is bimodal.
Figure 5: Exponentially decayed average of decision rate for Compute DQN on segments of an episode in Pong. Coloured regions indicate when the opponent has just hit the ball , the ball is approaching the player and dotted line indicates when the ball is hit , the player has just hit the ball , and the ball is heading toward the opponent . There are typically rises in decision rate in the red region in anticipation of the ball reaching the player’s paddle.
Figure 6: A sequence of observations (top) with an associated graph of the exponentially decayed average of decision rate at that moment (bottom) for Compute DQN in Breakout. As the game progresses the decision rate increases considerably, in correlation with the ball speeding up as blocks are cleared and a “breakout” occurs. The final image shows an example decision rate dropping while the ball is behind the wall of blocks.
Figure 7: A sequence of observations (top) with an associated graph of the exponentially decayed average of decision rate at that moment (bottom) for Compute DQN in Asterix. The decision rate grows as the waves get faster and have more collectibles. In the first two images, the agent has just finished collecting the objects in a wave, resulting in a spike in decisions followed by a quick decrease awaiting the next wave. The remaining images repeat this structure with very high decision rates as the number and speed of objects rises with dips between waves.
Figure 8: Varying the compute cost, cc , changes the learned decision rate and performance. Higher costs yield fewer decisions per-second; low costs increase the decision rate. Each game shows a performance-compute frontier, showing that agents adapt to their cost parameter.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: We present the human normalized score to validate that (i) DQN and Compute DQN on average perform better than a human (above 100%), and (ii) the performance loss from Compute DQN (seen on the right-hand side) is marginal relative to the performance benefit (seen on the left).
EnvironmentDecisions (M)Updates (M)Emulator Steps (M)
Alien40101030
Amidar4010806
Assault4010590
Asterix4010571
Atlantis4010822
BankHeist4010592
Appendix
Table 1: Compute DQN is trained with the same number of forward passes, decisions, and updates as DQN (40M decisions, 10M updates). However, as Compute DQN skips additional frames to reduce its compute cost, the emulator steps more for an agent to be more efficient.
Figure 10: Improvement of Compute DQN over DQN (at 4 Hz) in human normalized score (HNS) across 44 Atari games. Compute DQN achieves higher HNS in 61% of games. StarGunner shows the largest gain of 349%, while the largest loss of 183% below DQN.
Figure 11: We present the human normalized score to validate that (i) DQN (4 Hz) and Compute DQN on average perform better than a human (above 100%). The performance gains for StarGunner and Breakout (both left), and the performance loss on Centipede are large (right). On average, we see that giving agents control over their compute still results in a performance and efficiency improvement averaged across the suite.
Figure 12: Improvement of Compute Rainbow over Rainbow.
Figure 13: Performance of Compute Rainbow and Rainbow.
Figure 14: Compute cost of Compute Rainbow and Rainbow.
More in this topic

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Explore similar work

CardsList

  1. When Independent Sampling Outperforms Agentic Reasoning

    May 8, 2026Yihe Dong, Boris ShigidaCost-Aware InferenceCompetitive Programming

    Princeton University

  2. Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling

    May 2, 2026Florian Valentin Wunderlich, Lars Benedikt Kaesberg, Jan Philip Wahle +2Self-Consistency DecodingCost-Aware Inference

    University of Göttingen, Germany

  3. On the Role of Computation in Reinforcement Learning

    Feb 5, 2026Raj Ghugare, Michał Bortkiewicz, Alicja Ziarko +1Reinforcement LearningModel-Free RL

    Department of Computer Science, Princeton University · Warsaw University of Technology · University of Warsaw +2