Authors: Jeffrey Lai, Anthony Bao, John Quinn, William Gilpin
Organizations: The Oden Institute, The University of Texas at Austin, Austin, Texas 78712, USA · Department of Electrical Engineering, The University of Texas at Austin, Austin, Texas 78712, USA · Department of Physics, The University of Texas at Austin, Austin, Texas 78712, USA
Reasoning allows artificial intelligence models to revisit and correct their mistakes, enabling recent frontier advances in mathematical theorem solving, software engineering, and autonomous task planning. Reasoning models are widely observed to reason for longer on harder tasks, but the general mechanism responsible for these slowdowns is unknown. Here, we show that reasoning models exhibit transient chaos, a physical consequence of the computational complexity of difficult tasks. As a consequence, we show that diverse leading reasoning models are dynamical systems with fractal basins, with fractality increasing with task difficulty across diverse tasks like Sudoku and maze solving, visual puzzles, and mathematical logic. We show that transient chaos emerges due to reasoning becoming trapped for extended durations near saddle points, which we show correspond to nearly-correct attempted solutions of the underlying problem. Our results show that reasoning slowdowns are an inevitable consequence of problem hardness in modern artificial intelligence models, and establish reasoning traces as a rich new class of dynamical system.
Figures & tables
Figure 1: Recurrent depth reasoning models produce fractal basins. (A) A looped reasoning model encodes a task as a dynamical system evolving in a latent space, and the problem’s solution as a fixed point. (B) Convergence time basins on a difficult Sudoku puzzle not seen during training.
Figure 2: Hard problems produce fractal basins across diverse models and tasks. (A) Examples of Sudoku-Extreme, Maze-Hard, Countdown, and ARC-AGI-1 reasoning tasks, each of which we probe using a different architecture (Equilibrium Reasoners, Fixed Point Reasoning Models, Parcae, and Tiny Recursive Model) [ 4 , 4 , 3 , 6 , 2 , 1 ] (B) Example basins for varying problem difficulties across the four tasks and architectures. (C) Scaling of basin entropy with the number of reasoning iterations required to converge, each across randomly-sampled task instances of varying underlying difficulty.
Figure 3: Scattering from incorrect solutions produces reasoning transients. (A) The “Plinko” model of high-dimensional dynamics. Trajectories that start and end at nearly the same point can take very different routes if one encounters weakly-unstable saddle points. The fast Lyapunov indicator ( λF ) measures their maximum separation. (B) The latent state dynamics projected into two dimensions using PCA, showing two representative initial conditions that start out closely-spaced, but which take direct and indirect routes scattered by a saddle point. (C) The error between the latent state and converged latent state for trajectories originating from the initial conditions in (B). (D) Decoding the latent state near the saddle point produces a nearly-correct solution attempt.
Figure 4: Saddle sets indicate uncertain reasoning trajectories. (A) The fast Lyapunov Indicator λF versus initial condition for a Sudoku-solving model, with two representative initial conditions highlighted (magenta and blue crosses). (B) λF versus the frequency at which reasoning traces switch solutions. We measure the frequency by decoding the latent reasoning dynamics to a candidate solution at each timestep, and measuring the number of changes divided by the total number of iterations before convergence. Each point corresponds to a different puzzle instance.
Figure 5: Fractal basins emerge as bifurcations during training. (A) A looped transformer is trained to solve integer linear systems. Panels correspond to a series of training steps around the bifurcation that occurs 150 k steps into training, when accurate solving first emerges. The top row colors initial conditions based on whether they converge to the correct solution, and the bottom row colors trajectories by their convergence time. (B) Latent reasoning trajectories before and after the bifurcation, projected into two latent components. Open circles denote starting points, and crosses denote final fixed points. (C) The fraction of correct solutions, basin entropy, and number of saddle escape directions along trajectories across the full training procedure. (D) The finite-time Lyapunov exponents separately-calculated for the directly-substitutable variables, and more difficult core variables that require multistep Gaussian elimination to solve.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Hard problems produce fractal basins across alternative models. (A) Examples of Maze-Hard and Countdown reasoning tasks, each of which we probe using a different architecture (Equilibrium Reasoners and a looped transformer) [ 4 , 1 ] (B) Example basins for varying problem difficulties across the two tasks and architectures. (C) Scaling of basin entropy with the number of reasoning iterations required to converge, each across randomly-sampled task instances of varying underlying difficulty.
Figure S2: Alternative metrics of basin complexity. We repeat the basin quantification procedure using three alternative metrics. Each point corresponds to metrics calculated over a random two-dimensional slice through a random problem instance.
Figure S3: Progressive magnifications of a fractal produced by the Equilibrium Reasoning model on a difficult Sudoku task. Top Left: settle-time field (loops to converge); Top Right: fast Lyapunov indicator (FLI) on the settle-time field. Middle Row: zoom sequence on the settle-time field. Bottom Row: zoom sequence on the FLI field.
Figure S4: Scaling of the uncertainty exponent for the fractal in Fig. S3 .
Figure S5: Progressive magnification of a fractal produced by FPRM on the Maze task.
Figure S6: Scaling of the uncertainty exponent for the fractal in Fig. S5 .
Figure S7: Progressive magnification of a fractal produced by the fine-tuned Parcae model on the Countdown mathematical logic task.
Figure S8: Scaling of the uncertainty exponent for the fractal in Fig. S7 .
Department of Information and Communication Engineering, University of Rajshahi, Bangladesh · College of Engineering and Computer Sciences, Marshall University, Huntington, WV, USA · Department of Computer Science and Engineering, University of Rajshahi, Bangladesh