Same World, Differently Given: History-Dependent Perceptual Reorganization in Artificial Agents

Hongju Pae

cs.AIApr 6, 2026

Same World, Differently Given: History-Dependent Perceptual Reorganization in Artificial Agents

Authors: Hongju Pae

Organizations: Active Inference Institute, Crescent City, CA, USA

Abstract

What kind of internal organization would allow an artificial agent not only to adapt its behavior, but to sustain a history-sensitive perspective on its world? I present a minimal architecture in which a slow perspective latent $g$ feeds back into perception and is itself updated through perceptual processing. This allows identical observations to be encoded differently depending on the agent's accumulated stance. The model is evaluated in a minimal gridworld with a fixed spatial scaffold and sensory perturbations. Across analyses, three results emerge. First, the perspective latent reorganizes perceptual encoding: identical observations are represented differently depending on prior experience, and this reorganization of salience gating replicates across runs, with five of 16 gating dimensions changing direction consistently across 30 independent runs after correction for multiple comparisons. Second, only adaptive self-modulation yields the characteristic growth-then-stabilization dynamic of the perspective latent, unlike rigid or always-open update regimes. Third, perturbation history is followed by reduced adaptive plasticity after nominal conditions are restored, showing a directionally consistent trend across seeds. Gross behavior remains stable throughout the analysis, suggesting that the dominant reorganization is perceptual rather than behavioral. Together, these findings identify a minimal mechanism for history-dependent perspectival organization in artificial agents.

Figures & tables

Var.	Dim.	Role	Timescale	Stage 2
$x_{t}$	8	local observation patch	step	(input only)
$z_{\mathrm{raw}}$	16	pre-gating percept	step	frozen
$z_{t}$	16	gated perceptual latent	step	FiLM trainable
$p_{t}$	8	action-trace embedding	step	frozen
$g_{t}$	12	perspective latent	cross-episode	trainable
$\alpha_{t}$	1	plasticity rate $\in[0.03,0.30]$	step (state-dep.)	trainable

Table 1: Notation, dimensionality, functional role, characteristic timescale, and second-stage training status of all model variables. “Frozen” components are loaded from first-stage checkpoints and not updated in stage two.

Panel of Figure 1 — Figure 1: Schematic overview of the extended agent architecture. Gray, green, purple, and orange denote the environment, perceptual encoding, perspective dynamics, and action pathway. The main additions are the feedback links between perception and perspective: salience gating (purple arrow), in which $g_{t-1}$ modulates perceptual encoding through the FiLM gate, and self-modulating plasticity (green arrow), in which gated perception influences the plasticity rate $\alpha_{t}$ . Together with the update path $g_{t-1}\rightarrow\alpha_{t}\rightarrow g_{t}$ , these implement the central loop of the model: perspective shapes perception, and perception regulates perspective. The updated $g_{t}$ then biases the fast action pathway through $s_{t}$ . Thin black arrows indicate the main forward flow; gray arrows indicate auxiliary or implicit connections. For clarity, the observation decoder, prediction-error features, and explicit proprioceptive pathway are omitted.

Figure 2: Gridworld environment visualization. This Pygame-based simulator renders a fixed $23\times 7$ gridworld with a left-to-right observation noise gradient: left-side cells are noisier and right-side cells are more reliable. For analysis, the 23 columns are divided into 5 reporting zones (vertical white lines). Background color indicates noise level, from red (high) to teal (low). The agent (white circle) starts at the grid center. The top overlay shows timestep, position, reporting zone, action, perturbation status and trace, number of scheduled perturbations, and local observation noise $\sigma$ . The right panel shows the current 8-neighbor local observation patch $x_{t}$ . This visualization is illustrative only; all simulations are run headless.

Figure 3: Perspective reorganizes perception of the same input. The comparison perspective $g$ was obtained from a single illustrative mixed perturbation run ( $n_{P}=0\!\to\!4\!\to\!0$ ); cross-run statistics are reported in the text. (a) PCA projection of probe encodings under the late post-perturbation perspective state $g_{2}$ and the null condition $g\!=\!0$ , shown with covariance ellipses. Cluster centroids are marked with crosses. Encoding-geometry statistics (variance ratio, participation ratio, principal-axis rotation) are computed across all 30 runs and reported in the text. (b) Signed per-dimension difference $z_{t}(g_{2})-z_{t}(g\!=\!0)$ averaged across probe observations. (c) Per-dimension FiLM salience-gating coefficients $\gamma$ in Block 0 (pre-perturbation, $n_{P}\!=\!0$ ) and Block 2 (post-perturbation, $n_{P}\!=\!0$ ) of the mixed-history run.

Figure 4: Ablation analysis of self-modulating plasticity. (a) Episode-wise adaptive plasticity $\alpha$ with perturbation ( $n_{P}=4$ ) and without perturbation ( $n_{P}=0$ ). In both conditions, $\alpha$ rises rapidly early in training, but only the perturbed condition later declines below baseline. (b) Perspective magnitude $\|g\|$ across four update regimes: Adaptive with perturbation, Adaptive baseline, Rigid ( $\alpha=0.05$ ), and Open ( $\alpha=0.80$ ). The rigid regime shows continued growth, the open regime remains relatively flat, and the adaptive regimes lie between these extremes, with slightly larger late-stage $\|g\|$ under perturbation.

Figure 5: Plasticity under matched three-block histories. Mean adaptive plasticity $\alpha$ is compared between the first and final blocks across three schedules matched in total duration: Baseline ( $n_{P}=0\!\to\!0\!\to\!0$ ), Mixed perturbation ( $n_{P}=0\!\to\!4\!\to\!0$ ), and Persistent perturbation ( $n_{P}=4\!\to\!4\!\to\!4$ ). Bars show hierarchical medians, points show individual seed run values, and error bars indicate IQR. The block-wise decline is smallest under Baseline ( $0.285$ to $0.280$ ), largest under Mixed ( $0.279$ to $0.229$ ), and comparable under Persistent ( $0.293$ to $0.261$ ); see text for inferential statistics.

Cognitive Architectures for AI Agents

Cognitive architectures for AI agents organize perception, memory, goals, reasoning, and action into systems for persistent, adaptive behavior.

See all 85 papers

May 16, 2026

Body-Grounded Perspective Formation and Conative Attunement in Artificial Agents

This paper proposes a minimal architecture for body-grounded perspective formation in artificial agents. Extending prior work, the model introduces an interoceptive viability signal, a Fisher-style metric over fused exteroceptive-interoceptive states, and a conative alignment mechanism linking bodily tendency to action readiness. In a reward-free gridworld, conation converts learned bodily tendency into stable body-directed behavior, while body-to-perspective routing allows bodily perturbations to leave a recoverable geometric residue in the perspective latent. This study shows how minimal structural conditions for artificial subjectivity can be operationalized in the phenomenological sense, through the embodied organization of how a world is given to an agent.

Hongju Pae

Active Inference Institute, CA, USA

Embodied AI Embodied Cognition

Ask PDF

Apr 9, 2026

Artifacts as Memory Beyond the Agent Boundary

The situated view of cognition holds that intelligent behavior depends not only on internal memory, but on an agent's active use of environmental resources. Here, we begin formalizing this intuition within Reinforcement Learning (RL). We introduce a mathematical framing for how the environment can functionally serve as an agent's memory, and prove that certain observations, which we call artifacts, can reduce the information needed to represent history. We corroborate our theory with experiments showing that when agents observe spatial paths, the amount of memory required to learn a performant policy is reduced. Interestingly, this effect arises unintentionally, and implicitly through the agent's sensory stream. We discuss the implications of our findings, and show they satisfy qualitative properties previously used to ground accounts of external memory. Moving forward, we anticipate further work on this subject could reveal principled ways to exploit the environment as a substitute for explicit internal memory.

John D. Martin, Fraser Mince, Esra'a Saleh +1

Openmind Research Institute · University of Alberta · Cohere Labs Community +3

Memory-Augmented Agents Reinforcement Learning

Ask PDF

Aug 31, 2026

Agents in the Large: Perception-Centered Architecture for Persistent Agents

Cognitive language agents have achieved substantial progress by equipping language models with memory, tools, and decision-making procedures, enabling agents to reason and act in interactive environments. Existing frameworks largely cast these agents as systems for solving user-specified, bounded tasks. An increasingly important goal is for language agents to provide persistent assistance in long-lived settings where user needs, context, and service procedures persist and change, and to remain useful across the broad range of tasks that arise over time. Yet we still lack a framework to characterize persistent AI agents, organize existing work, and guide future development. To this end, we propose a Perception-Centered Architecture for Persistent Agents (Pera). Pera describes a persistent agent organized around perception and control components that continually perceive service-relevant signals from episodic task executions, internal context, and changes in the surrounding environment, and use these signals to construct lifecycle tasks. These tasks drive the ongoing operation and adaptation of the agent's service procedures. We use Pera to retrospectively organize recent work, examine a detailed case study, and offer forward-looking insights for building more capable persistent agents. Just as software engineering moved from programming in the small to programming in the large, Pera frames the evolution of language agents as an analogous architectural transition toward long-lived, adaptive intelligence systems.

Shihan Dou, Haoxiang Jia, Shichun Liu +14

NLP Lab, Fudan University · PL Lab, Peking University · CCDS, Nanyang Technological University +1

Cognitive Architectures for AI Agents Lifelong Learning Agents LLM Agents Long-Horizon LLM Agents

Ask PDF

Abstract

Cognitive Architectures for AI Agents

Explore similar work

Body-Grounded Perspective Formation and Conative Attunement in Artificial Agents

Artifacts as Memory Beyond the Agent Boundary

Agents in the Large: Perception-Centered Architecture for Persistent Agents