Contextual Intelligence The Next Leap for Reinforcement Learning

André Biedenkapp

cs.LGFeb 17, 2026

Contextual Intelligence The Next Leap for Reinforcement Learning

Authors: André Biedenkapp

Organizations: Albert-Ludwigs-Universität Freiburg, Germany

Abstract

Reinforcement learning (RL) has produced spectacular results in games, robotics, and continuous control. Yet, despite these successes, learned policies often fail to generalize beyond their training distribution, limiting real-world impact. Recent work on contextual RL (cRL) shows that exposing agents to environment characteristics -- contexts -- can improve zero-shot transfer. So far, the community has treated context as a monolithic, static observable, an approach that constrains the generalization capabilities of RL agents. To achieve contextual intelligence we first propose a novel taxonomy of contexts that separates allogenic (environment-imposed) from autogenic (agent-driven) factors. We identify three fundamental research directions that must be addressed to promote truly contextual intelligence: (1) Learning with heterogeneous contexts to explicitly exploit the taxonomy levels so agents can reason about their influence on the world and vice versa; (2) Multi-time-scale modeling to recognize that allogenic variables evolve slowly or remain static, whereas autogenic variables may change within an episode, potentially requiring different learning mechanisms; (3) Integration of abstract, high-level contexts to incorporate roles, resource & regulatory regimes, uncertainties, and other non-physical descriptors that crucially influence behavior. We envision context as a first-class modeling primitive, empowering agents to reason about who they are, what the world permits, and how both evolve over time. By doing so, we aim to catalyze a new generation of context-aware agents that can be deployed safely and efficiently in the real world.

Figures & tables

Panel of Figure 1 — Figure 1. Schematic of a cMDP with common state & action spaces with transition, reward, and initial-state distributions conditioned on a context $c$ . The diagram shows two learning pipelines: (i) context-oblivious policy $\pi(s)$ receives only the state observation; (ii) context-aware policy $\varpi(s,c)$ additionally observes the context. Both are trained across contexts $c\sim\mathcal{C}$ . Schematic of a Contextual MDP Schematic of a Contextual MDP (cMDP) with a common state-action space whose transition, reward, and initial-state distributions are conditioned on a context. The diagram shows two learning pipelines: (i) a context-oblivious policy – pi of s – that receives only the state observation s, and (ii) a context-aware policy – varpi of s and c – that additionally observes the context c. Both policies are trained across contexts sampled from a distribution.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Taxonomy Level	Influence	Source	Temporal Nature	Type of Change	Examples
Allogenic	Global	Exogeneous, Imposed by the environment, Independent of Agent’s actions	Approximately stationary, Infrequent change	Abrupt ( Gumbsch et al., 2024 , see, e.g., ) , Large scale	Physical constants (gravity ( Benjamins et al., 2023 ) , payload weights, …); hardware variations (motor torque, center of mass ( Ding et al., 2020 ) , …); environment layouts (map topology, wall placement ( Cobbe et al., 2020 ) , lighting conditions, …)
Autogenic	Local	Endogeneous, Arise from agent’s own behavior, internal state, or learning process	Frequent change, Slower change than observations	More smoothly, Continuous	Internal states (battery level, limb failure ( Eimer et al., 2021b ) , fatigue, …); Self-generated task parameters (interaction frequencies ( Biedenkapp et al., 2021 ) , goals set by a higher-level planner ( Eysenbach et al., 2019 ) , curriculum difficulty, …)

Appendix

Table 1. Taxonomy Summary Table

Contextual RL

See all 19 papers

Reflective Context Learning: Studying the Optimization Primitives of Context Space
Apr 3, 2026Nikita Vassilyev, William Berrios, Ruowang Zhang +3LLM Agent Self-Improvement In-Context Learning
Contextual AI
Downstream-Aware Context Selection for Online In-Context Reinforcement Learning
Sep 27, 2026Ruihan A. Li, Shangtong Zhang, Rohan ChandraIn-Context RL In-Context Learning
University of Illinois Urbana-Champaign · University of Virginia
Considering Context: When World Models Need Context Encoders
Oct 5, 2026Oleg Smirnov, Sofiane Ennadir, John Pertoft +2Reinforcement Learning World Model Learning
King AI Labs, Microsoft Gaming

Apr 3, 2026J/K move · Enter open · S save

Reflective Context Learning: Studying the Optimization Primitives of Context Space

Nikita Vassilyev, William Berrios, Ruowang Zhang, Bo Han+2

Contextual AI

LLM Agent Self-Improvement In-Context Learning

Open paper AskPDF arXiv

Generally capable agents must learn from experience in ways that generalize across tasks and environments. The fundamental problems of learning, including credit assignment, overfitting, forgetting, local optima, and high-variance learning signals, persist whether the learned object lies in parameter space or context space. While these challenges are well understood in classical machine learning optimization, they remain underexplored in context space, leading current methods to be fragmented and ad hoc. We present Reflective Context Learning (RCL), a unified framework for agents that learn through repeated interaction, reflection on behavior and failure modes, and iterative updates to context. In RCL, reflection converts trajectories and current context into a directional update signal analogous to gradients, while mutation applies that signal to improve future behavior in context space. We recast recent context-optimization approaches as instances of this shared learning problem and systematically extend them with classical optimization primitives, including batching, improved credit-assignment signal, auxiliary losses, failure replay, and grouped rollouts for variance reduction. On AppWorld, BrowseComp+, and RewardBench2, these primitives improve over strong baselines, with their relative importance shifting across task regimes. We further analyze robustness to initialization, the effects of batch size, sampling and curriculum strategy, optimizer-state variants, and the impact of allocating stronger or weaker models to different optimization components. Our results suggest that learning through context updates should be treated not as a set of isolated algorithms, but as an optimization problem whose mechanisms can be studied systematically and improved through transferable principles.

Abstract

Contextual RL

Explore similar work

Reflective Context Learning: Studying the Optimization Primitives of Context Space

Downstream-Aware Context Selection for Online In-Context Reinforcement Learning

Considering Context: When World Models Need Context Encoders