From Fuzzy to Exact: The Halo Architecture for Infinite-Depth Reasoning via Rational Arithmetic

Hansheng Ren

cs.LGJan 26, 2026

From Fuzzy to Exact: The Halo Architecture for Infinite-Depth Reasoning via Rational Arithmetic

Authors: Hansheng Ren

Abstract

The prevailing scaling paradigm of Large Language Models (LLMs) rests on a substrate of "Fuzzy" floating-point arithmetic. To mitigate the inherent instability of this approximate foundation, modern architectures have erected a complex scaffolding of structural and numerical heuristics--Complex Residuals, Pre-RMSNorm, Attention Scaling, and Gradient Clipping--consuming significant compute solely to prevent numerical collapse. We propose a paradigm shift to the "Exact". We introduce the Halo Architecture, grounded in the Rational Field (Q) and powered by a custom Exact Inference Unit (EIU). To resolve the exponential bit-width growth of rational arithmetic, Halo employs a Dual-Ring Topology that unifies two complementary control mechanisms: (1) The Micro-Ring (Continuum Maintenance), which strictly bounds memory complexity via Diophantine Approximation; and (2) The Macro-Ring (Symbolic Alignment), which enforces logical consistency via periodic state collapse. This stable dual-ring substrate allows for the "Great Dismantling" of numerical scaffolding, reducing the Transformer block to its "Clean" algebraic form (Tabula Rasa). Furthermore, we verify the "Efficiency Paradox": the elimination of gradient noise (sigma -> 0) allows for Macro-Learning Rates, potentially reducing the Total Time-to-Convergence by orders of magnitude. Halo demonstrates that General Intelligence requires the hybridization of continuous fields and discrete chains under a rigorous mathematical framework.

Figures & tables

Symbol	Meaning
$B_{ring}$	codebook bit-width after Ring collapse
$B_{max}$	hardware representation budget, e.g. $4096$ bits
$D_{max}$	Micro-Ring denominator budget after projection, e.g. $2^{128}$
$D_{lattice}$	Macro-Ring lattice denominator bound, e.g. $2^{16}$
$K$	Ring reset interval in layers
$\alpha$	worst-case bit-growth per layer without reset

Table 1: Unified notation. The Ideal EIU operates over

\mathbb{Q}

without bound; the Implemented EIU operates under the hardware budget

B_{max}

Feature	Micro-Ring (Continuum Maintenance)	Macro-Ring (Logical Anchoring)
Math Object	Continuous Points (Farey Sequence $\mathcal{F}_{N}$ )	Discrete Vectors (Rational Lattice $\mathcal{C}$ )
Operation	Approximation : Find closest rational $p/q$	Quantization : Find closest semantic vector $c_{i}$
Nature	Local, Smooth, Unique	Global, Discontinuous, Many-to-One
Role of Truth	Numerical Truth : Preserves signal magnitude	Logical Truth : Collapses to valid concept
Analogy	Noise Filter : Removes high-freq hiss	Trap : Locks signal into valid state

Table 2: The Core Distinction: Micro-Ring vs. Macro-Ring

Method	Computed Value	Sign Determination
Test 1: $N=40$ , $x=2.01001$ (True $\approx 10^{-79}$ , Max Int. $\approx 10^{24}$ )
FP64	$-8.998199\times 10^{6}$	$<0$ (INVERSION!)
Double-Double	$-3.200329\times 10^{-11}$	$<0$ (INVERSION!)
Halo (Exact)	$\mathbf{1.040789\times 10^{-80}}$	$\mathbf{>0}$ (Correct)
Test 2: $N=65$ , $x=2.01004$ (True $\approx 10^{-129}$ , Max Int. $\approx 10^{39}$ )
FP64	$-1.010299\times 10^{22}$	$<0$ (INVERSION!)

Table 3: Catastrophic Sign Inversion in High-Condition-Number Polynomials. Floating-point formats suffer catastrophic cancellation causing silent logical failures.

Figure 1: Semantic Drift Analysis. Comparison of cumulative error in recursive inference. Standard BF16 (Red) exhibits immediate error accumulation ( $>10^{-4}$ ), rendering deep causal chains unreliable. FP32 (Orange) delays but does not prevent drift. The Halo EIU (Blue Dashed), utilizing Rational Arithmetic ( $\mathbb{Q}$ ), maintains zero error (plotted at $10^{-30}$ ) across 2000 steps.

Figure 2: Logical Survival Limit. Divergence from the analytical truth in a chaotic system. BF16 models (Red) collapse into random noise within 10 steps. FP32 (Orange) survives until $\sim$ 20 steps. The Halo Architecture (Blue), anchored by exact rational arithmetic, maintains perfect trajectory fidelity indefinitely.

Figure 3: Ablation Dynamics. Left: Bit-width growth. Note how the Micro-Ring (Orange) intervenes (blue ’ $\times$ ’) to prevent hardware saturation before the next Macro-reset. Right: Logical Drift. Without the Lattice (Orange), the Micro-Ring accumulates drift. The Lattice (Blue) periodically resets error to zero.

Figure 4: The Derivative of Truth. Gradient magnitude vs. Network Depth. Standard BF16 (Red) gradients explode/vanish beyond 500 layers. Halo (Blue) maintains precise gradient flow indefinitely, enabling the training of arbitrarily deep reasoning chains.

Figure 5: Needle in a Haystack Test. Retrieval error vs. Sequence Length. Due to cumulative “Semantic Drift,” BF16 models (Red) lose the ability to recall specific tokens after $\sim$ 2,000 steps. Halo (Blue) exhibits perfect recall regardless of context length.

Figure 6: The Curse of Dimensionality: 600B Model Instability. Semantic drift accumulation across model scales. Contrary to the belief that scale solves all, the 600B-scale model (Dark Red, $d=24576$ ) degrades faster than the 7B-scale model (Light Red, $d=4096$ ). Wider layers involve larger summation operations, amplifying floating-point rounding errors.

Figure 7: Efficiency of The Ring. Without intervention, rational numbers grow in complexity (Gray Dotted). The Ring mechanism (Green Solid) performs a periodic “Vector-Symbolic Collapse,” bounding the bit-width to a manageable range.

Neural Network Stability

See all 43 papers

Numerical stability analysis of large language models
Mar 13, 2025Stanislav Budzinskiy, Wenyi Fang, Longbin Zeng +1Transformer Inference Transformer Attention
Faculty of Mathematics, University of Vienna, Kolingasse 14-16, 1090, Vienna, Austria · Huawei Technologies
Dynamical Systems Theory Behind a Hierarchical Reasoning Model
Mar 24, 2026Vasiliy A. Es'kin, Mikhail E. SmorkalovSudoku Solving Neural Algorithmic Reasoning
Department of Radiophysics, University of Nizhny Novgorod, Nizhny Novgorod 603950, Russia · Huawei Nizhny Novgorod Research Center, Nizhny Novgorod 603006, Russia · Skolkovo Institute of Science and Technology, Moscow 121205, Russia
Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
Jun 16, 2026Sajad Movahedi, Vera Milovanović, Shlomo Libo Feigin +5Recurrent Transformers Algorithmic Reasoning
ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center · ETH Zurich · Swiss Institute of Bioinformatics +2

Mar 13, 2025J/K move · Enter open · S save

Numerical stability analysis of large language models

Stanislav Budzinskiy, Wenyi Fang, Longbin Zeng, Philipp Petersen

Faculty of Mathematics, University of Vienna, Kolingasse 14-16, 1090, Vienna, Austria · Huawei Technologies

Transformer Inference Transformer Attention

Open paper AskPDF arXiv

Transformers are the state-of-the-art architecture for large language models, and a key to their scalability is the strategic usage of low-precision arithmetic. We develop a mixed-precision analysis of transformer inference, deriving bounds for the condition numbers and forward error of the architecture's constituent parts. Notably, we compare the numerical stability of LayerNorm and RMSNorm in the massive-outlier regime, tighten the error bound of softmax in the presence of attention sinks, and quantify the impact of its shifted evaluation on the sensitivity to perturbations. Furthermore, we derive novel sequence-length-independent bounds on the local Lipschitz constant of self-attention. Our worst-case error bound for transformer inference suggests that its numerical stability is determined by the interplay between weight magnitude and the growth of the residual stream. Crucially, and as validated by experiments with GPT-2, our analysis establishes that the scaling of residual-projection weights preserves the propagation of the relative rounding error unless it forces a qualitative transition in the dynamics of the residual stream.

Abstract

Neural Network Stability

Explore similar work

Numerical stability analysis of large language models

Dynamical Systems Theory Behind a Hierarchical Reasoning Model

Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers