Authors: John D. Martin, Fraser Mince, Esra'a Saleh, Amy Pajak
Organizations: Openmind Research Institute · University of Alberta · Cohere Labs Community · Université de Montréal · Mila – Québec AI Institute · University of Pennsylvania
The situated view of cognition holds that intelligent behavior depends not only on internal memory, but on an agent's active use of environmental resources. Here, we begin formalizing this intuition within Reinforcement Learning (RL). We introduce a mathematical framing for how the environment can functionally serve as an agent's memory, and prove that certain observations, which we call artifacts, can reduce the information needed to represent history. We corroborate our theory with experiments showing that when agents observe spatial paths, the amount of memory required to learn a performant policy is reduced. Interestingly, this effect arises unintentionally, and implicitly through the agent's sensory stream. We discuss the implications of our findings, and show they satisfy qualitative properties previously used to ground accounts of external memory. Moving forward, we anticipate further work on this subject could reveal principled ways to exploit the environment as a substitute for explicit internal memory.
Figures & tables
Figure 1: Page Keeping example: Transitions are labeled with probabilities and observations. Actions are omitted for clarity. Interaction starts from s0 . A is an artifact of B , and is only observed after s1 . Observing A at t=5 implies that B was observed in the past: specifically at t=2 .
Figure 2: The base environment used throughout experiments.
Figure 3: Observing a spatial path reduces the necessary capacity to navigate. Averages of total reward for Linear Q-learning (left) and DQN (right) are shown, along with standard-error bars. We find that learning in presence of a shortest path improves performance for nearly every agent and capacity. Many improvements are statistically significant (see Supplement B.2 ).
Figure 4: Performance improves when agents observe the shortest path: Average reward tends to increase when the shortest path is visible. This can be observed for nearly every capacity of Linear-Q and DQN; it appears most significant for higher capacity systems, but also has a stark affect in the low capacity regime. Averages and standard error regions are computed with 30 seeds.
Figure 5: Environments considered for learning in the presence of other fixed artifacts.
Figure 6: Externalization arises with other fixed artifacts: Average total reward is observed for three paths and one set of geometric landmarks. We find evidence of externalizing memory across all artifacts and with the following number of instances for linear agents: Suboptimal (3), Misleading (2), Random (2), Landmarks (1). For DQN, Suboptimal (2), Landmarks (2), Random (1), Misleading (0). Each bar presents an average and standard-error from the 30 seeds shown.
Figure 7: Externalization with a dynamic path: The left panel illustrates the environment, in which the current policy generates a path vanishing over time. We observe performance uniformly increase over nearly all capacities and section 4.1 satisfied for C=256 .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Experiment Configuration Details. Hyperparameters used for linear agent (above), DQN agents (middle), dynamic experiments (below).
Figure 9: Environments considered in Experiment 1.
Figure 10: Experiment 1. Step-size sweeps. Linear Q-learning (top half), DQN (bottom half). Selected step-size is marked with a star.
Figure 11: Experiment 1. Significance Tests. Let Pi and Pj be the performances associated with capacities of the row and column. The plot should be read row-wise: when the (i,j) -cell is green, Pi is significantly higher than Pj .
Figure 12: Capacity vs performance of Linear-Q in the presence of fixed artifacts. Total reward for each artifact type is shown. Each data point presents an average and standard-error from 30 seeds. Capacity ranges from 12 to 242 (1 to 576).
Figure 13: Environments considered in Experiment 2.: Random (top left), Misleading (top right), Suboptimal (bottom left), Landmarks (bottom right).
Table 1: Average total reward with standard errors over thirty independent seeds, for all settings considered. Bold text indicates satisfaction of our empirical condition.
Figure 16: Experiment 2. Linear Significant Tests. Let Pi and Pj be the performances associated with capacities of the row and column. The plot should be read row-wise: when the (i,j) -cell is green, Pi is significantly higher than Pj .
Figure 17: Experiment 2. DQN Significant Tests. Let Pi and Pj be the performances associated with capacities of the row and column. The plot should be read row-wise: when the (i,j) -cell is green, Pi is significantly higher than Pj .
Figure 18: Experiment 3. Dynamic Path Average Reward and Step-size sweeps. Linear-Q.
Figure 19: Experiment 3. Linear Significance Tests. Let Pi and Pj be the performances associated with capacities of the row and column. The plot should be read row-wise: when the (i,j) -cell is green, Pi is significantly higher than Pj .