Organizations: Ebonye Smith (corresponding author), Phoenix Wilson, and Gireeja Ranade are with the Department of Electrical Engineering and Computer Sciences, University of California Berkeley, Berkeley, CA 94720, USA · Aidan Andrews is with the Physics Department, University of Illinois, Urbana-Champaign, Urbana, IL 61801 · Alexis Frias is with the Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089 · Ayush Pandey is with the Department of Electrical Engineering, University of California Merced, Merced, CA 95343
One of the remarkable abilities of Transformer-based neural networks is in-context learning. This motivates us to think about "in-context" control. Can we control a system knowing only (1) that it belongs to a particular family of systems and (2) some data on system evolution (provided as the context trajectory) but no specific parameter information? Specifically, we investigate the ability of Transformer-models to perform "in-context" control of unstable, nonlinear dynamical systems, where context consists only of the unforced system dynamics. We leverage a multi-head, GPT-2-style architecture, and we evaluate performance for two classical nonlinear systems -- the cartpole system and the two-link acrobot system, where the system parameters are unknown to the Transformer. We consider a setting where trajectories from an expert controller are available at training time but not at inference time. Here, we identify training techniques that are essential to train Transformers to perform low-level control. Through an analysis over checkpoints, we show that in-context control is possible, and the learned policy performance generally improves as context increases. Furthermore, the Transformer models can generalize to stabilize systems that are out-of-distribution.
Figures & tables
Fig. 1 : Given a context of zero-torque system evolution u0:k−1=0 and current state xk , the Transformer predicts the next state x^k+1 , controller mode c^k , and action u^k , which is applied to the plant.
Distribution
M (kg
m (kg)
l (m)
In-Distribution ( Ωcart )
[2.0,3.0)
[1.1,2.0)
[1.6,2.1)
OOD Easy ( ΩcartOOD, easy )
[1.5,2.0)
[0.3,0.8)
[1.0,1.5)
OOD Hard ( ΩcartOOD, hard )
[3.0,3.5)
[2.0,2.5)
[2.2,2.7)
TABLE I : Cartpole System Parameter Space Distributions
Fig. 2 : Stabilization success rate vs. context length ( N ) for Cartpole. Left/center: Transformer at an early and late training checkpoint. Right: LSTM baseline at the matched late checkpoint. Dashed lines denote nominal baseline performance. The Transformer’s ability to better exploit long context emerges between checkpoints (a)-(b), while the LSTM (c) outperforms the late-checkpoint Transformer (b) at short context but is overtaken by N=50 on in-distribution and OOD-Easy systems.
Fig. 3 : One example of swing-up for the cartpole system. Comparison between expert reference κω and Transformer roll out given 25 context examples. States [p,p˙,θ,θ˙] and actions are shown over time.
Test Distribution
N=1
N=5
N=10
N=25
N=50
In-Distribution
0.00
0.02
0.02
0.01
0.05
Out-of-Distribution (OOD)
0.00
0.01
0.03
0.04
0.09
TABLE II : Stabilization Success vs. Context Length N for the Cartpole Transformer (Ablation: Noise-Free Data)
Distribution
l1 (m)
l2 (m)
m1 (kg)
m2 (kg)
In-Distr. ( Ωacro )
[0.5,1.0)
[1.0,1.5)
[1.5,2.0)
[1.0,1.5)
OOD ( ΩacroOOD )
[1.0,1.5)
[1.5,2.0)
[1.0,1.5)
[1.0,1.5)
TABLE III : Acrobot System Parameter Space Distributions
Fig. 4 : Stabilization success rate vs. context length ( N ) for Acrobot. (a) Transformer, early checkpoint (step 25k). (b) Transformer, late checkpoint (step 300k). (c) LSTM baseline, matched late checkpoint (step 300k). Dashed lines denote nominal baseline performance. The LSTM matches or exceeds the Transformer on in-distribution systems at all context lengths beyond N=1 , while the Transformer overtakes the LSTM on OOD systems by N=50 .
Fig. 5 : One example of acrobot swing-up rollout comparison: Expert κω (light blue) vs. Transformer (red) given N=50 context steps. Angle θ1 converges to −5π≡π rad upright.
Large-scale foundation models for scientific machine learning adapt to physical settings unseen during training, such as zero-shot transfer between turbulent scales. This phenomenon, in-context learning, challenges conventional understanding of learning and adaptation in physical systems. Here, we study in-context learning of dynamical systems in a minimal setting: we train a small two-layer, single-head transformer to forecast one dynamical system, and then evaluate its ability to forecast a different dynamical system without retraining. We discover an early tradeoff in training between in-distribution and out-of-distribution performance, which manifests as a secondary double descent phenomenon. We discover that attention-based models apply a transfer-operator forecasting strategy in-context. They (1) lift low-dimensional time series using delay embedding, to detect the system's higher-dimensional dynamical manifold, and (2) identify and forecast long-lived invariant sets that characterize the global flow on this manifold. Our results clarify the mechanism enabling large pretrained models to forecast unseen physical systems at test time without retraining, and they illustrate the unique ability of attention-based models to leverage global attractor information in service of short-term forecasts.
Anthony Bao, Jeffrey Lai, William Gilpin
Department of Electrical & Computer Engineering, The University of Texas at Austin, Austin, Texas 78712, USA · The Oden Institute, The University of Texas at Austin, Austin, Texas 78712, USA · Department of Physics, The University of Texas at Austin, Austin, Texas 78712, USA
We investigate the ability of transformers to perform in-context reinforcement learning (ICRL), where a model must infer and execute learning algorithms from trajectory data without parameter updates. We show that a linear self-attention transformer block can provably implement policy-improvement methods, including semi-gradient SARSA and actor-critic, via explicit parameter constructions. Beyond existence, we design a teacher-mimicking training procedure, analyze its gradient-flow dynamics, and establish the first convergence guarantee in the ICRL literature: under suitable richness conditions on the training MDP distribution, gradient flow converges locally and exponentially to an optimal parameter manifold corresponding to the desired RL update. Empirically, training transformers on randomly generated tabular MDPs confirms these predictions: the learned models recover the parameter structure of our explicit constructions and, when deployed on unseen MDPs, deliver strong in-context control performance. Together, these results illuminate how transformer architectures internalize and execute classical reinforcement learning algorithms in context, bridging mechanistic understanding and training dynamics in ICRL.
Haodong Liang, Lifeng Lai
Department of Electrical and Computer Engineering, University of California, Davis
Transformers can implement reinforcement-learning algorithms in context, yet how off-policy control emerges within Transformers remains poorly understood. We study Q-learning, a foundational model-free control algorithm that combines off-policy bootstrapping with greedy policy improvement. We prove that a two-layer attention-only Transformer can implement a finite-temperature form of Q-learning, where the temperature τ>0 controls the approximation to greedy action selection, recovering window Q-learning as τ→0. We train the structured Transformer on randomly generated MDP tasks using output-only supervision. Experiments show that the theoretically predicted parameter structure emerges during training, including the internal greedy action-selection mechanism, and that the trained Transformer achieves performance comparable to Q-learning on unseen tasks. Our theoretical analysis further shows that the equivalent realizations form a solution manifold and that, under suitable task-diversity conditions, the training iterates converge locally to this manifold at an exponential rate. For the random-task model analyzed here, the local worst-case number of gradient steps needed to reduce the distance to the manifold by a factor ε is Θ(τ−5log(1/ε)), quantifying the increasing optimization cost as the learned update approaches window Q-learning.
Yingjun Mo, Yajie Zhao, Fang Kong
Southern University of Science and Technology (SUSTech) · University of Electronic Science and Technology of China (UESTC)