Factorization Regret mediates compositional generalization in latent space

John Schwarcz

cs.LGMar 28, 2026

Factorization Regret mediates compositional generalization in latent space

Authors: John Schwarcz

Organizations: Edmond & Lily Safra Center for Brain Sciences (ELSC), The Hebrew University of Jerusalem

Abstract

Are there still barriers to generalization once all relevant variables are known? We consider the challenge of generalizing to a novel combination of task-relevant latent variables in the Cognitive Gridworld, a stationary Partially Observable Markov Decision Process where observations are generated by latent variables with parametric interactions. We establish Factorization Regret as an information-theoretic quantity measuring the contribution of latent variable interactions to task performance. Using this metric, we explain the sub-optimality of Echo State Networks and uncover a failure mode whereby confidence becomes decoupled from accuracy. This suggests that utilizing interactions between relevant variables is a non-trivial capability. We next introduce Representation Classification Chains to show that latent variable interactions and how to infer the variables' values can be learned together through Reinforcement Learning. Solving this variational inference problem enables generalization to novel combinations of relevant variables. Lastly, we demonstrate that once interactions are learned, imagined experience is sufficient for zero-shot control. In summary, we develop and motivate a minimal, theoretically grounded setting for research into generalization from experience.

Figures & tables

Figure 1: Environment schematic for $C=2$ . Observations $\mathbf{o}_{t}$ are generated stochastically. Variable interactions $Z$ parameterize likelihood $P_{Z}(\mathbf{o}\mid\mathbf{r})$ and variable realizations $(r_{1},r_{2})$ fix the probability of sampling each observable $o^{i}$ .

Figure 2: The cost of Naive Bayes grows with time and interactions. (a) Example Joint (matrices) and marginalized (vectors) likelihoods. (b) Top: Accuracy of Joint (left) and Naive (right) Bayes across varying context sizes. Bottom: Relative accuracy (left) and Factorization Regret (right). Circles mark four equidistant reference time-points throughout inference.

Figure 3: RNNs align with theoretical predictions . (a) Architecture (left) and gradient flow (right) of the Classifier. Only the goal belief receives a gradient. (b-d) Same as Figure 2 b for

C=1,2

(

C=3

in Supplementary Figure 10 ) with Fully Trained RNNs and ESNs (averaged over

n=5

random seeds). Markers indicate four equidistant reference time-points throughout inference.

Figure 4: Failure to capture FR can induce hallucinations . (a) Sequential updating of marginal beliefs under Joint and Naive inference. (b) Distribution of belief assigned to the correct action at each step of inference, pooled over episodes. Colors progress from blue to red throughout inference.

Figure 5: Compositional generalization schematic. Testing variable embeddings must be learned through their implicit relationships to training variables. Goal embeddings are outlined in green.

Figure 6: A variational architecture learns Compositional Embeddings from reward. (a) Relevant variables interact via learnable embeddings to form interactions. (b) Forward pass and gradient flow of the Classifier and Generator. The Classifier learns from rewards while the Generator uses self-supervised learning (SSL). (c) Testing episode accuracy of the Classifier throughout training.

Figure 7: Latent space navigation enables zero-shot control over observations. Schematic illustration (left) and empirical trajectories (right) of an agent navigating a novel latent space to identify the state of the world which would generate the maximally preferred observation. Heatmaps indicate $\mathcal{R}(\mathbf{r},\mathbf{\Omega},\ell)$ and trajectories show $\arg\max_{\mathbf{r}}\pi(\mathbf{r}\mid\hat{Z})$ throughout training (smoothed for clarity).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: A flexible process for embedding Gridworld structure into latent space. The embeddings of relevant variables are compressed into interactions which are then expanded to a discrete probability distribution over possible realizations of the world. The full process consists of first (i) compressing embedding vectors to their scalar interactions. Then (ii) expanding pairwise interactions to pairs of vectors, expanding the vector pairs to matrices through an outer product, and (for $C\geq 3$ ) expanding the matrices into a tensor by broadcasting each matrix over all other dimensions. Finally, a valid (iii) likelihood is ensured by a sigmoid transformation for normalization.

Figure 9: Results survive additional controls (

C=2

). Excess error above optimality at the final step of inference, plotted against training episode on log–log axes. The mean (

\pm

SEM) training curve of

5

small Fully Trained RNNs is shown in blue. ESNs are colored light to solid red by their number of trainable parameters, where

N=10{,}000

matches the Fully Trained RNN with

N=200

and another ESN (

N=1{,}000

) outputs updates to the joint belief-state.

Figure 10: Results extend to 3 interacting variables. Same as Figure 3 with

C=3

(

n=1

Figure 11: Confidence–accuracy calibration for $C=1,2,3$ . (a) Linear fits between confidence in the selected action and accuracy. (b) Accuracy over confidence as a function of Factorization Regret.

Figure 12: Learning and generalization with Compositional Embeddings depend on $|\mathcal{S}|$ ( $C=2$ ). (a) Learning dynamics for training (top) and testing (bottom) variables, split by the number of latent variables $|\mathcal{S}|$ . (b) All sizes of $\mathcal{S}$ overlaid and split by training progress. (c-d) 3-dimensional visualization of training and testing set accuracy as a function of $|\mathcal{S}|$ and training progress.

Figure 13: Learning Compositional Embeddings through the classifier is unreliable (

C=2

) . Excess error above optimality at the final step of inference (mean

\pm

SEM with

6

seeds), plotted against testing episode (1 test per 500 training episodes) on log–log axes. The embeddings receive gradients from either the Generator (blue) or Classifier (red).

Figure 14: Conditional generative modeling enables optimization in compositional spaces. (a) Schematic illustrating the mapping of preferred observations ( $\mathbf{\Omega}$ ) to their respective likelihoods and the cumulative landscape (accumulated over $i$ in Equation 1 ). (b) An example traversal, from the lowest to the highest point on the landscape, changes observations to best match the agent’s preference. (c) Controller learning curves (mean $\pm$ SEM, $n=20$ initializations of $\mathbf{\Omega}$ ). $\%$ max reward is calculated as $\mathcal{R}(\arg\max_{\mathbf{r}}\pi(\mathbf{r}\mid\hat{Z}),\mathbf{\Omega},\ell)\quad/\quad\max_{\mathbf{r}}\mathcal{R}(\mathbf{r},\mathbf{\Omega},\ell)$ , using the ground truth $\ell$ .

Figure 15: Managing Factorization Regret involves high-dimensional RNN dynamics (

C=2

). (a) Throughout learning, testing accuracy at the final step of inference saturates to the performance of either Joint or Naive Bayes. The vertical dotted line marks the range shown in (b–f). (b) Final step testing accuracy during early training. (c) Difference in Participation Ratio (PR) between the recurrent and read-in activity. (d) Correlation between Factorization Regret and accuracy at the final step of inference. A negative correlation emerges in the beginning of training and persists for the ESN. (e) Correlation versus accuracy. (f) Correlation versus PR difference.

Panel of Figure 16 — Figure 16: Factorization Regret leads to entanglement over time. Top 2 Principal Components of the marginal beliefs after only a single observation (left) and $T=30$ observations (right). Here $c$ and $c^{\prime}$ denote the two variables in context ( $C=2$ ), one of which is the goal. Beliefs are colored by the realization sum ( $r_{c}+r_{c^{\prime}}$ ), difference ( $r_{c}-r_{c^{\prime}}$ ), and belief entropy ( $-\sum_{c}\sum_{r}B_{tcr}\ln B_{tcr}$ ). $R^{2}$ indicates the variance explained of the respective variables by the top 2 components. After only a single observation, representations are still perfectly factorized. By the final observation, ESN and Naive representations become deeply entangled, while the Fully Trained representation remains factorized alongside Optimal Joint Bayes.

Figure 17: Entanglement correlates with learned dynamics of dimensionality. Cross-correlations between Bayesian entanglement and (a) the L2 norm of RNN output

u_{t}

, (b) the effective dimensionality of the projection onto the Principal Components, (c) the portion of the effective dimensionality explained by activity in the band of top Principal Components that explains

25\%

of neural variance and (d) same as (c) for the band of bottom Principal Components. Only data from

t>20

was analyzed in order to isolate the steady-state response profile. Legends give the number of Principal Components in each band.

Generalization in RL

Reinforcement learning generalization studies how agents transfer learned behavior to unseen tasks, environments, dynamics, and observations.

See all 89 papers

Jun 13, 2026

LatentGym: A Testbed For Cross-Task Experiential Learning With Controllable Latent Structure

We envision continually learning agentic systems that become more useful over time: as they encounter sequences of related tasks, they should infer the hidden structure shared across those tasks and use it to improve future decisions. This cross-task experiential learning capability is pivotal in domains such as personalization and interactive assistance, but existing training/evaluation frameworks do not provide shared, controllable latent structures and cannot measure whether or why agents improve. We introduce LatentGym: a controllable suite in which each environment is organized around a ground-truth latent variable governing the structure across tasks. Our construction yields metrics that separate exploration (whether the agent's actions gather information about the latent) from exploitation (whether the agent uses what it has gathered). We demonstrate our suite on empirical studies addressing three questions: how and why frontier models fail to adapt across related tasks; whether post-training on related task sequences improves general cross-task adaptation, and where those gains come from; and how design choices such as inter-task feedback shape training dynamics and generalization. Together, these results establish a controlled foundation for studying how LLM agents learn from experience across tasks, and for designing agents that adapt more reliably in sequential, personalized, and interactive settings.

Daksh Mittal, Tommaso Castellani, Thomson Yen +7

1Columbia University · 2Oumi

Continual Learning for LLM Agents Lifelong Learning Agents Multi-Task Learning AI Agent Benchmarks LLM Agent Training

Ask PDF

Jul 9, 2026

Latent Memory Palace: Reasoning for Control as Autoregressive Variational Inference

Human decision-making is highly flexible -- some actions are taken immediately; others require longer deliberation. Language models have exhibited a similar capacity for adaptive "reasoning." However, transferring this capability to continuous control policies has been challenging, as directly reasoning in language space may lack the granularity for spatial understanding and precise motions. In this work, we show that reasoning for control policies can emerge by organizing information in an autoregressive latent space reminiscent of a memory palace, where retrieval is iterative and adaptive. Our method, Latent Memory Palace (LMP), formulates reasoning as variational inference with an autoregressive latent distribution. We derive a latent-space reinforcement learning technique to tractably optimize its variational lower bound. The resulting policy, LMP-

π

, achieves strong empirical performance in simulation and real-world domains while exhibiting interpretable, adaptive allocation of test-time compute. We further show that the same framework yields a variable-length action tokenizer, LMP-

\texttt{tok}

, which significantly improves the performance of downstream autoregressive policies. Together, these results present a new perspective on latent reasoning for control through the lens of variational inference.

Chuning Zhu, Eva Xu, Jose Barreiros +3

University of Washington · Toyota Research Institute

RL Control Latent Action Learning Variational Inference Policy Learning

Ask PDF

Mar 3, 2026

Temporal Consistency Improves Generalization in Contextual Offline Meta Reinforcement Learning

Offline meta-reinforcement learning seeks to learn a policy that generalizes to new related tasks online. Context-based methods infer a task representation from transition histories, yet learning an effective task representation without supervision remains challenging. Existing methods relying on contrastive learning learn discriminative task representations, but fail to identify task-specific dynamics, while relying on reconstruction can be insufficient to model long-horizon dependencies, limiting generalization to new tasks. We investigate the impact of temporal consistency in latent space on task representation learning, showing that enforcing multi-step predictions in latent space encourages task representations that are able to capture task-dependent dynamics while preventing representation collapse. We provide theoretical analysis characterizing sources of error in value estimation and show through extensive experiments on MuJoCo, Contextual DeepMind Control, and MetaWorld benchmarks that temporal consistency significantly improves both zero-shot and few-shot generalization.

Mohammadreza Nakheai, Aidan Scannell, Kevin Luck +1

Aalto University · Toyota Research Institute · Vrije Universiteit Amsterdam

Representation Learning Offline RL Representation Learning for RL Meta-Reinforcement Learning Generalization in RL Temporal Consistency+2

Ask PDF

Abstract

Generalization in RL

Explore similar work

LatentGym: A Testbed For Cross-Task Experiential Learning With Controllable Latent Structure

Latent Memory Palace: Reasoning for Control as Autoregressive Variational Inference

Temporal Consistency Improves Generalization in Contextual Offline Meta Reinforcement Learning