How do Transformers Perform Low-Level Control In-Context?
Organizations: Ebonye Smith (corresponding author), Phoenix Wilson, and Gireeja Ranade are with the Department of Electrical Engineering and Computer Sciences, University of California Berkeley, Berkeley, CA 94720, USA · Aidan Andrews is with the Physics Department, University of Illinois, Urbana-Champaign, Urbana, IL 61801 · Alexis Frias is with the Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089 · Ayush Pandey is with the Department of Electrical Engineering, University of California Merced, Merced, CA 95343
Abstract
One of the remarkable abilities of Transformer-based neural networks is in-context learning. This motivates us to think about "in-context" control. Can we control a system knowing only (1) that it belongs to a particular family of systems and (2) some data on system evolution (provided as the context trajectory) but no specific parameter information? Specifically, we investigate the ability of Transformer-models to perform "in-context" control of unstable, nonlinear dynamical systems, where context consists only of the unforced system dynamics. We leverage a multi-head, GPT-2-style architecture, and we evaluate performance for two classical nonlinear systems -- the cartpole system and the two-link acrobot system, where the system parameters are unknown to the Transformer. We consider a setting where trajectories from an expert controller are available at training time but not at inference time. Here, we identify training techniques that are essential to train Transformers to perform low-level control. Through an analysis over checkpoints, we show that in-context control is possible, and the learned policy performance generally improves as context increases. Furthermore, the Transformer models can generalize to stabilize systems that are out-of-distribution.
Figures & tables
| Distribution | (kg | (kg) | (m) |
|---|---|---|---|
| In-Distribution ( ) | |||
| OOD Easy ( ) | |||
| OOD Hard ( ) |
| Test Distribution | N=1 | N=5 | N=10 | N=25 | N=50 |
|---|---|---|---|---|---|
| In-Distribution | 0.00 | 0.02 | 0.02 | 0.01 | 0.05 |
| Out-of-Distribution (OOD) | 0.00 | 0.01 | 0.03 | 0.04 | 0.09 |
| Distribution | (m) | (m) | (kg) | (kg) |
|---|---|---|---|---|
| In-Distr. ( ) | ||||
| OOD ( ) |
Feedback Control
Feedback control uses ongoing measurements to adjust actions, stabilize dynamical systems, track goals, and respond to disturbances and constraints.
See all 275 papers