We consider stochastic token steering through a deep stack of transformer layers as a mean-field control problem for an interacting particle system on the unit sphere, where interactions arise from the attention mechanism. In the mean-field limit, the token population evolves via a controlled integro-partial differential equation where the feedback control acts as a corrective vector field guiding the population toward an aggregate configuration of tokens in the representation space. We deduce the minimum effort control to accomplish this task over a fixed time horizon interpreted as normalized layer depth. We show that the resulting controlled dynamics for the token population can be understood as a mean-field Schrödinger bridge adapted to the sphere geometry. Building on this, we propose a novel generalization of the dynamic Sinkhorn recursion to compute the consistent token evolution. Numerical experiments confirm that the proposed approach achieves exact steering of the token population.
Figures & tables
q⋅[j][ρ(k)](x)
=∫Sd−1φt[j](y)Zt[ρt(k)](y)eβ⟨Qty,Ktx⟩Rd
×⟨Py⊥(Vtx−bt[ρt(k)](y)),∇φt[j](y)⟩dy.
Algorithm 1 Mean-Field Token Steering Algorithm
Figure 1: Evolution of optimally controlled token distribution and tokens for the S1 experiment in Section 4.1 .
Figure 2: Convergence of Algorithm 1 for the experiment in Section 4.1 .
Figure 3: Evolution of optimally controlled token distribution and tokens for the S2 experiment in Section 4.2 .
Figure 4: Convergence of Algorithm 1 for the experiment in Section 4.2 .
Figure S1: Evolution of uncontrolled versus optimally controlled token distribution for the S1 experiment in Supplementary Section B .
Department of Mathematics Friedrich-Alexander-University Erlangen-Nürnberg · Institute of Mathematics CAIDAS University of Würzburg · Mathematical Institute University of Oxford Reuben College +1