Spherical Fixed-Point Reasoners: Adaptive Loop Halting on the Hypersphere

Arber Zela; Artavazd Maranjyan; Pol Puigdemont; David Berghaus; Valentyn Boreiko; Volkan Cevher

cs.AIOct 19, 2026

Spherical Fixed-Point Reasoners: Adaptive Loop Halting on the Hypersphere

Authors: Arber Zela, Artavazd Maranjyan, Pol Puigdemont, David Berghaus, Valentyn Boreiko, Volkan Cevher

Organizations: EPFL · Lamarr Institute · Fraunhofer IAIS · Amazon AGI

Abstract

Looped (weight-tied) transformers enable adaptive computation by allocating more iterations to harder inputs. Fixed-Point Reasoners (FPRM) achieve this with a damped fixed-point solver, but operate in an unbounded Euclidean residual space with coupled rescaling and relative-norm halting. We introduce Spherical Fixed-Point Reasoners (SFPRM), which perform the same computation on the unit hypersphere of normalized transformers (nGPT). Input injection, damping, and halting become geodesic operations, yielding boundedness by construction and eliminating coefficient coupling. We prove fixed-point existence, local convergence under an explicit step-size condition, and global convergence under strong injection. On Sudoku-Extreme, Maze-Hard, and Mini-ARC, SFPRM achieves 96.6%, 87.9%, and 59.1% test accuracy, respectively, outperforming FPRM and other baselines while allocating test-time compute more effectively. Its converged geometry is also interpretable: token-input angles capture puzzle structure and predict correctness.

Figures & tables

Panel of Figure 1 — Figure 1: SFPRM overview. (a) Each iteration interpolates the current state toward the normalized input ( $\Slerp_{\lambda}$ ), applies the weight-tied spherical nGPT block to obtain the proposal $\widetilde{\mathbf{z}}_{i}$ , and rotates a fraction $\eta_{i}$ of the way toward it ( $\Slerp_{\eta_{i}}$ ); the loop halts once the worst-token chordal residual $r_{\mathrm{c},i}$ falls below $\tau_{c}$ . The eigen-learning-rate retractions $R_{\alpha}$ inside the block keep every token state on unit sphere $\mathbb{S}^{d-1}$ . (b) One token’s iterates rotate along a great circle by shrinking angles $\eta_{i}\Omega_{i,t}$ ; the halting test compares each proposal with the current iterate, so it never needs the fixed point $z_{t}^{\star}$ .

Method	Recurrence	Fix. pt.	State update and input	Halting / output
HRM	Fast $L$ -loop $+$ slow $H$ -loop	Appr.	Euclidean $(z_{L},z_{H})$ ; input drives $L$	ACT head; read $z_{H}$
TRM	Shared net refines $z$ , answer $y$	No	Euclidean; input re-read each step	ACT head; read $y$
URM	UT; fixed inner $+$ ACT outer loop	No	Euclidean; short conv in MLP	ACT head, ${\leq}16$ steps
EqR	Tied loop; $B$ stochastic restarts	Appr.	Euclidean; random init, path noise	ACT; lowest-residual run
Attractor	Backbone $+$ equilibrium loop	Yes	Backbone $\tilde{y}_{0}$ seeds each step	Solver tol.; read $\tilde{y}^{\star}$
FPRM	Tied Transformer loop	Yes	$(\alpha_{1},\beta_{1})$ mix; $(\alpha_{2},\beta_{2})$ injection	Rel.- $\ell_{\infty}$ residual

Table 1: Recent recurrent reasoners: HRM ( Wang et al., 2025 ) , TRM ( Jolicoeur-Martineau, 2025 ) , URM ( Gao et al., 2025 ) , Attractor ( Fein-Ashley and Rashidinejad, 2026 ) , EqR ( Huang et al., 2026 ) and FPRM ( Movahedi et al., 2026 ) . “Appr.” means no numerical fixed-point solve is performed.

Panel of Figure 2 — Figure 2: Proposals accepted by each halting test around a token state $z_{i,t}$ . The chordal test accepts a disk; FPRM’s accepts an axis-aligned $\ell_{\infty}$ section, so two proposals at the same angle can be judged differently.

Figure 3: Exact accuracy and halting on Maze-Hard (Sudoku: fig. 10 ). SFPRM vs. FPRM evaluation on $1000$ test puzzles. Top: exact accuracy; mean halt and number of examples halting. Bottom: per-sequence residual (thin), median (heavy).

	Model	Architecture	# params.	Single Loop (No Hier.)	Sudoku-Ext. Pass@1	Maze-Hard Pass@1	Mini-ARC Pass@1	ARC-1 Pass@2
Larger	HRM	rec. Tf	27M	✗	55.0%	74.5%	44.9% (51.7%)	40.3%
	Attractor Model	Tf $+$ FP loop	27M	✗	91.4%	93.1%	—	—
	URM	rec. Tf $+$ conv	14M	✗	77.6%	—	—	—
$5$ – $7$ M	TRM	rec. Tf	7M	✗	74.7%	85.3%	48.4% (36.9%)	44.6%
	EqR	rec. Mixer $+$ restarts	5-7M	✗	93.0%	—	55.3% (40.3%)	—
	Attractor Model	FP Tf	7M	✗	54.3%	46.7%	—	—

Table 2: Comparison with published recurrent reasoners. Reported numbers are taken from the respective papers; Mini-ARC numbers are from Huang et al. (2026) (Table 14). Numbers in parentheses are our own runs under the same setting as SFPRM. Tf

=

Transformer, Mixer

=

MLP-Mixer, rec.

=

recurrent refinement, FP

=

fixed-point solve. Within the

5

–

7

M group the best result is bold and the second best underlined.

Panel of Figure 4 — Figure 4: Global convergence under input injection (SFPRM, Sudoku). (a) Largest angle between the end states of eight initializations, over $32$ puzzles, against $\lambda$ (teal: all coincide, coral: $\geq 2$ end states); indigo, right axis: token accuracy. (b) Trajectories from the eight $\mathbf{z}_{0}$ (squares) to $\mathbf{z}^{\star}$ (star), coloured by the token accuracy of the decoded state; circles mark the first iteration with $r_{\mathrm{c},i}<\tau_{c}$ , dashed red the exact-match region.

Panel of Figure 5 — Figure 5: Fixed-point solvers (SFPRM, Maze). (a) Accuracy, halting disabled. (b) Radius is the geodesic angle (log-scaled) to $\mathbf{z}^{\star}$ , rings at $0.1/1/10/100^{\circ}$ .

Figure 6: Anchor angles by cell class on Maze : cells (thin lines), class means (heavy), the halt iteration (vertical line), and the final distribution (margin).

Panel of Figure 7 — Figure 7: SFPRM rollout frames (FPRM in Figures 21 and 22 ). Top row : the board decoded from $\mathbf{z}_{i}$ ; bottom row : each cell’s angle to its own input anchor. left : Sudoku-Extreme; right : Maze-Hard.

Panel of Figure 8 — Figure 8: Rollout frames on Mini-ARC for FPRM (left) and SFPRM (right). Input (top) and target (bottom), then iterations $t=0$ to $199$ : decoded grid ( $\times$ : wrong cell) and angle to the input anchor, $\angle(z_{t},\hat{x})$ . Only SFPRM solves this puzzle, and its angle map singles out the cells the task rewrites.

Figure 9: Final anchor angle vs. correctness on

512

Sudoku puzzles: angle (SFPRM) and state norm (FPRM) of clue and blank cells, for solved (filled) and unsolved (hatched) puzzles.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

	Maze-Hard	Sudoku-Extreme	ARC-AGI-1	Mini-ARC
Data and schedule
base puzzles $\times$ augmentations	$1{,}000\times 1$	$5{,}000\times 200$	$960\times 1{,}000$	$149\times 100$
sequence length / vocabulary	$900$ / $6$	$81$ / $11$	$900$ / $12$	$25$ / $12$
test set	$1{,}000$ mazes	$422{,}786$ puzzles	$400$ tasks	$149$ tasks
epochs	$60{,}000$	$15{,}000$ ( $12{,}000$ )	$100{,}000$	$100{,}000$
optimizer steps	$78{,}125$	$97{,}656$ ( $78{,}125$ )	$518{,}226$	$68{,}993$

Appendix

Table 3: Hyperparameters and learned scalars of the SFPRM runs in Section 4 . Sudoku-Extreme values are for the epoch-

15{,}000

run, while values in parentheses are for the

12{,}000

-epoch constant-schedule run. All convolutions are depthwise and bias-free. For the per-channel step sizes

\alpha_{A},\alpha_{M}

and the per-dimension QK scale

s_{qk}

, cells give the mean for layer 1 / layer 2, and the range row gives the minimum and maximum over all channels of both layers. On ARC, a linear Q-head on a detached state copy only breaks ties in Pass@

K

Model	Setting	Sudoku-Ext.
FPRM	Sudoku $1\text{k}\!\times\!1000$	78.5%
FPRM	Sudoku $5\text{k}\!\times\!200$	78.4%
SFPRM	Sudoku $1\text{k}\!\times\!1000$ (best checkpoint)	87.0%
SFPRM	Sudoku $1\text{k}\!\times\!1000$ (final checkpoint)	12.2%
SFPRM	Sudoku $5\text{k}\!\times\!200$	92.8%

Appendix

Table 4: Exact accuracy on Sudoku-Extreme for SFPRM vs. FPRM at matched backbone (

6.83

M trainable parameters), data, global batch (

768

), and optimizer steps/epochs (

78{,}125/12{,}000

). “

1\text{k}\!\times\!1000

” and “

5\text{k}\!\times\!200

” are the number of distinct base puzzles times the augmentations per puzzle. The SFPRM

1\text{k}

run is given at both its best and checkpoint because it collapses between them, which the

5\text{k}

build fixes. For the

5\text{k}

run the final checkpoint is also the best. The FPRM rows fall well below the

94.2\%

reported for FPRM in Table 2 .

Panel of Figure 10 — Figure 10: Accuracy and halting on Sudoku-Extreme and Mini-ARC. Open-loop rollouts of FPRM (amber) and SFPRM (blue). (a) $512$ Sudoku test sequences of the released FPRM checkpoint and of SFPRM with the learned $\lambda=0.47$ (left) and with $\lambda$ fixed at $0.1$ (right, the run in Table 2 ). (b) All $14{,}447$ Mini-ARC test rows ( $149$ tasks, $100$ augmentations each) over $1000$ iterations, for the runs in Table 2 . Top: exact accuracy of the decoded state (a) and task-level Pass@1 (b). Dotted lines and stars mark each model’s mean halt iteration, labelled with how many sequences halt. Bottom: every sequence’s halting residual (thin) and its median (heavy) against its own threshold ( $\tau_{c}{=}0.01$ chordal for SFPRM, $\tau_{\mathrm{E}}{=}0.1$ for FPRM).

Figure 11: Global convergence under input injection on Maze-Hard (SFPRM, learned

\lambda=0.575

), analogue to Figure 4 . (left) The largest angle between the end states of eight initializations of the raw map (teal: all coincide, coral:

\geq 2

end states), and the token accuracy of the end state (indigo, right axis), as a function of

\lambda

. (right) Trajectories at four values of

\lambda

for a single Maze example: the top-2 principal plane of the trajectories from the eight initializations

\mathbf{z}_{0}

(squares) through

\mathbf{z}^{\star}

(star; circles: first iteration with

r_{\mathrm{c},i}<\tau_{c}

). The color shows the token accuracy of the state decoded at each position (dashed red: exact-match region).

Panel of Figure 12 — Figure 12: Learned parameters of SFPRM during training. Per task, top row: injection fraction $\lambda$ , logit scale $s_{z}$ , and QK scale $s_{qk}$ (mean over its 64 dimensions, per layer). Bottom row: the eigen learning rates $\alpha_{A}$ (attention) and $\alpha_{M}$ (MLP) of both layers, as the mean, minimum and maximum over the 512 hidden dimensions.

Figure 13: Final anchor angle on solved and unsolved mazes (Maze-Hard analogue of Figure 9 ). Per-class angle to the anchor at the end of the rollout over

512

test mazes (

438

solved,

74

unsolved), for (a) SFPRM and (b) FPRM’s normalized state. Filled: solved; hatched: unsolved. Unlike on Sudoku, the distributions overlap for every class, so the final angle does not predict correctness.

Panel of Figure 14 — Figure 14: One SFPRM trajectory per cell class, on the sphere (Sudoku left, Maze right). For every cell class we draw the token whose $3$ -frame captures the most trajectory variance. The frame is spanned by the trajectory’s two leading singular directions and the anchor $x_{t}$ , which sits at the pole. $x_{t}$ is nearly orthogonal to the plane of the trajectory ( $|\langle x_{t},\,v_{1,2}\rangle|\approx 0.05$ ). Arcs are great circles between projected iterates. The colour shows the iteration index.

Panel of Figure 15 — Figure 15: Anchor angles by cell class on Sudoku-Extreme . Four test puzzles (Maze in Figures 6 and 16 ). In each pair FPRM is above and SFPRM below. Thin lines are individual cells, thick lines class means, the vertical line the halt iteration, and the margin the distribution at the end of the rollout. SFPRM rotates the clues to ${\approx}52^{\circ}$ from their anchors within about $20$ iterations and leaves the blanks near orthogonal, whereas FPRM’s normalized states settle within the first few iterations with clues and blanks only ${\approx}6^{\circ}$ apart.

Panel of Figure 16 — Figure 16: Anchor angles by cell class on Maze-Hard . Four test puzzles, FPRM above and SFPRM below in each pair, with the conventions of Figure 15 . On every maze SFPRM settles walls at ${\approx}90^{\circ}$ , free cells that stay empty at ${\approx}113^{\circ}$ , free cells that become path at ${\approx}102^{\circ}$ , and start and goal at ${\approx}80^{\circ}$ , whereas FPRM keeps all classes within $70$ – $87^{\circ}$ and separates the two free-cell classes by only a few degrees.

	Halting				Exact at halt (%)			Open loop (%)
Solver	%	med. $t$	mean $t$	$F$ evals	all	$p$	halted	peak ( $t$ )	$t{=}1000$
SFPRM, Maze ( $\lambda{=}0.575$ , $\tau_{c}{=}0.01$ )
Adaptive (default)	99.2	27	96.3	104.5	87.8	–	87.9	87.8 (20)	87.8
Fixed, $\eta{=}1$	63.1	23	25.8	386.3	87.8	1.00	87.3	87.8 (20)	87.8
Anderson, $m{=}3$	100.0	13	13.8	14.8	86.8	0.04	86.8	87.4 (19)	86.7
Broyden, $m{=}5$	99.4	14	14.0	20.9	87.4	0.39	87.9	87.9 (16)	51.5

Appendix

Table 5: Inference-time fixed-point solvers on the trained models. Every example runs

1000

steps from its own random start, shared by the four solvers; each step is one evaluation of

F_{\theta}

, and an example halts at the first iterate with residual below

\tau

. Halting : share of examples that halt (%), median and mean halting step over those, and mean

F

evaluations over all examples. Exact at halt : accuracy over all examples, exact binomial test against the default solver (

p

), and accuracy among halted examples. Open loop : accuracy with halting disabled, at its best step and at

t=1000

. Maze: all

1000

test mazes; Sudoku:

1000

puzzles evenly spaced over the test set. Best within each block: exact accuracy , share halted , fewest halting steps .

Figure 17: Accuracy and halting residual over the iterations for the remaining settings of Table 5 . SFPRM on Sudoku (

\lambda=0.1

, left), FPRM on Maze (middle) and FPRM on Sudoku (right); SFPRM on Maze is Figure 5 . Same

1000

examples and solvers as the table. Top: exact accuracy of the iterate with halting disabled. Bottom: each model’s halting residual, median (solid) and 75th percentile (dotted) over the examples, its threshold (dashed), and the iteration by which half of the examples have halted (circle).

Figure 18: The four solvers in the principal plane through the fixed point , for test example

0

of Maze-Hard (top,

\lambda=0.575

) and of Sudoku-Extreme (bottom,

\lambda=0.1

). Axes and markers as in Figure 4 b, with

\mathbf{z}^{\star}

the end point of Anderson acceleration; every solver runs from the same eight initializations (squares) and six further starts placed on its plane. Each panel uses the top-2 principal directions of that solver’s own iterates, so the background landscapes differ between panels.

Figure 19: SFPRM on Maze-Hard: State trajectories in polar coordinates. The four solvers (rows, with each one’s halting step) on the maze of Figure 5 b, one token per cell class (columns). Radius: geodesic angle to the reference fixed point, log-scaled from $0.1^{\circ}$ at the centre to $180^{\circ}$ at the rim, rings at $1$ , $10$ and $100^{\circ}$ ; azimuth: the top-2 principal plane of that token’s component orthogonal to the reference; square: start, dot: end of the $1000$ -iteration run; colour: iteration. Anderson ends at the centre, the default and fixed steps orbit at a small radius, and Broyden leaves for the rim.

Figure 20: SFPRM on Sudoku-Extreme: State trajectories in polar coordinates. A puzzle on which the default solver halts at its median step (

435

); columns : the middle clue cell and the blank cells whose decode settles earliest, at the median and last. Anderson and Broyden reach the tolerance early, the default solver orbits for hundreds of iterations before halting, fixed

\eta=1

never halts, and Broyden fills the disc after leaving.

Variant	Pass@1	$\Delta$
SFPRM (full)	59.1	–
w/o QK normalization	26.2	$-32.9$
w/o damping ( $\eta=1$ )	47.7	$-11.4$
Fixed injection fraction ( $\lambda=0.1$ )	49.7	$-9.4$
ACT halting instead of chordal residual	54.4	$-4.7$
Fixed eigen learning rates ( $\alpha_{A}=\alpha_{M}=0.5$ )	55.0	$-4.0$

Appendix

Table 6: SFPRM ablations on Mini-ARC. Pass@1 accuracy at an iteration cap of 200. Each row changes one component of the full model.

\Delta

: change relative to the full model.

Panel of Figure 21 — Figure 21: Filmstrip gallery on Sudoku-Extreme. Test puzzles $0$ – $4$ . One test puzzle per row, rolled open-loop for $40$ iterations and rendered at $i=0,2,11,39$ (protocol in Appendix C.4 ), for FPRM (left) and SFPRM (right): decoded board on top, each cell’s angle to its input anchor below.

Panel of Figure 22 — Figure 22: Filmstrip gallery on Maze-Hard. Test puzzles $0$ – $4$ . One $30{\times}30$ test maze per row, rolled open-loop for $40$ iterations and rendered at $i=0,2,11,39$ (protocol in Appendix C.4 ), for FPRM (left) and SFPRM (right): decoded path on top, each cell’s angle to its input anchor below.

Panel of Figure 23 — Figure 23: Filmstrip gallery on Mini-ARC. One test task per row, rolled out open-loop for $200$ iterations. Left: FPRM; right: SFPRM. Each row starts with the test input (top) and its target (bottom), followed by five iterations from $t=0$ to $t=199$ . At each iteration the decoded grid is shown on top, with $\times$ marking cells that differ from the target, and each cell’s angle to its input anchor, $\angle(z_{t},\hat{x})$ , below.

Panel of Figure 24 — Figure 24: Filmstrip gallery on Mini-ARC (continued). Same caption as in Figure 23 .

Adaptive Reasoning in Language Models

See all 31 papers

Jun 16, 2026

Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers

Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The number of effective layers reached by looping determines the quality of the solution these models find. Like deep architectures, looped architectures are prone to a signal propagation problem induced by depth as the halting decision is postponed. In this paper, we address this signal propagation issue using pre-norm layers and residual scaling. Building on these architectural modifications, we propose FPRM, a Transformer-based Fixed-Point Reasoning Model that uses fixed-point convergence as an end-to-end halting mechanism in a looped architecture. We show that fixed-point halting allows FPRM to adapt its compute to task difficulty. FPRM is effective on common reasoning benchmarks, namely Sudoku, Maze, state-tracking, and ARC-AGI.

Sajad Movahedi, Vera Milovanović, Shlomo Libo Feigin +5

ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center · ETH Zurich · Swiss Institute of Bioinformatics +2

Recurrent Transformers Algorithmic Reasoning Adaptive Computation

Ask PDF

Sep 27, 2026

Reasoning on the Simplex: Geometric Fixed-Point Models

Looped reasoners spend test-time compute by iterating a weight-tied map, but a small residual does not mean the state is a fixed point when that map lives in unconstrained latent space. We propose Geometric Fixed-Point Reasoning (GFPR), in which the iterated state is the prediction itself: a field of categorical beliefs on a product of simplices, whose argmax is the answer at every step. Because the state is a belief, task structure can be imposed through compact convex relaxations, either as structured readouts or directly in the recurrent state; in the latter case the update remains a continuous self-map, so a fixed point exists for any parameters. At about 7M parameters, GFPR reaches 95.1% exact match on Sudoku-Extreme, 92.0% on Maze-Hard, and 100% sequence accuracy on S_5 length 128, above the published FPRM numbers at the same scale. The same update also trains a 201M language model on FineWeb-Edu in which each site is a distribution over the vocabulary; with 24 Picard steps it is above GPT-2 small on four zero-shot multiple-choice tasks and above GPT-2 medium on ARC-Easy.

Talgat Daulbaev, Ilya Glazkov, Maxim Rakhuba +1

Applied AI Institute · LigandPro · HSE University +1

Mathematical Reasoning Recursive Language Models

Ask PDF

May 12, 2026

Solve the Loop: Attractor Models for Language and Reasoning

Looped Transformers offer a promising alternative to purely feed-forward computation by iteratively refining latent representations, improving language modeling and reasoning. Yet recurrent architectures remain unstable to train, costly to optimize and deploy, and constrained to small, fixed recurrence depths. We introduce Attractor Models, in which a backbone module first proposes output embeddings, then an attractor module refines them by solving for the fixed point, with gradients obtained through implicit differentiation. Thus, training memory remains constant in effective depth, and iterations are chosen adaptively by convergence. Empirically, Attractor Models outperform existing models across two regimes, large-scale language-model pretraining and reasoning with tiny models. In language modeling, Attractor Models deliver a Pareto improvement over standard Transformers and stable looped models across sizes, improving perplexity by up to 46.6% and downstream accuracy by up to 19.7% while reducing training cost. Notably, a 770M Attractor Model outperforms a 1.3B Transformer trained on twice as many tokens. On challenging reasoning tasks, we show that our model with only 27M parameters and approximately 1000 examples achieves 91.4% accuracy on Sudoku-Extreme and 93.1% on Maze-Hard, scaling favorably where frontier models like Claude and GPT o3, fail completely, and specialized recursive reasoners collapse at larger sizes. Lastly, we show that Attractor Models exhibit a novel phenomenon, which we call equilibrium internalization: fixed-point training places the model's initial output embedding near equilibrium, allowing the solver to be removed at inference time with little degradation. Together, these results suggest that Attractor Models make iterative refinement scalable by turning recurrence into a computation the model can learn to internalize.

Jacob Fein-Ashley, Paria Rashidinejad

University of Southern California

Language Model Pretraining Recurrent Transformers LLM Reasoning

Ask PDF

Abstract

Adaptive Reasoning in Language Models

Explore similar work

Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers

Reasoning on the Simplex: Geometric Fixed-Point Models

Solve the Loop: Attractor Models for Language and Reasoning