ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoning

Jinsung Jeon; Seung-won Hwang

cs.AISep 30, 2026

ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoning

Authors: Jinsung Jeon, Seung-won Hwang

Organizations: KAIST InnoCORE LLM · Seoul National University · Korea University

Abstract

Precise numerical reasoning with Large Language Models (LLMs) is essential for expanding their applicability to complex real-world tasks. However, text-based tokenization often fragments numbers, significantly hindering precise arithmetic reasoning. Meanwhile, numerical embeddings, despite arithmetic precision, rely on context-agnostic substitution that disregards the semantic role of numbers as identifiers. To combine the complementary strengths, we propose \textbf{ANI (Adaptive Numerical Injection)}, a hybrid framework that governs the selective injection of numerical features based on the semantic context. By employing a context-aware gating mechanism, we selectively inject numerical embeddings (specifically FoNE) into the latent space, explicitly preserving nominal identifiers while enhancing quantitative operands. Through extensive evaluations across various LLMs, we demonstrate that ANI enhances MATH performance by 9.5 points over the official reference model, while maintaining robust performance on general linguistic benchmarks.

Figures & tables

Figure 1: Overview of the proposed adaptive representation. Unlike existing methods that rely exclusively on uniform textual representations, our framework dynamically integrates a unified architecture by explicitly distinguishing between nominal representations and quantitative representations.

Figure 2: The overall architecture of ANI. Numerical embeddings are generated from raw scalar values and mapped to the model’s hidden dimension via a Linear Projection. Simultaneously, the Context-aware Gating Network computes a scalar coefficient ( $\alpha$ ) based on both the hidden state of each numerical token and its numerical embedding. The final representation is obtained by adding the projected numerical embedding, scaled by $\alpha$ , to the original hidden state. Note that non-numerical tokens bypass this module to preserve their linguistic features.

Figure 3: Linear probing analysis for optimal layer selection. We evaluate the degree of numerical disambiguation across layers using a diagnostic dataset. The peak in classification accuracy identifies the layer with maximum contextual maturity, which serves as the optimal point for adaptive numerical injection.

Figure 4: Gating mechanism and last token injection. ANI selectively integrates numerical embeddings into the terminal sub-token’s hidden state via an adaptive gate $\alpha$ . This mechanism ensures that mathematical structures are reinforced only when semantically required, preserving the model’s linguistic integrity.

Reference Models (Official)
Model	Mathematical Reasoning		General
Model	MATH (Acc)	GSM8K (Acc)	MMLU (Acc)
Qwen3-8B	49.28	89.84	73.75
Standard Fine-tuning
Qwen3-8B	55.18	87.79	72.24
Controlled Experiments (Methods)

Table 1: Main Results on Qwen3-8B. We compare our proposed ANI method with an official baseline and controlled baselines. Bold indicates the best performance among models trained on the same dataset.

Figure 5: Layer Sensitivity Analysis on MATH. Performance is measured by relative accuracy (Rel. Acc.) on the MATH benchmark. The linear probe reaches a peak accuracy of 93% at Layer 24.

Method	2-digit	4-digit	6-digit	8-digit	10-digit	12-digit
Pre-trained (Qwen3-8B)	75.0	48.0	6.0	0.0	0.0	0.0
Baseline (SFT)	100.0	95.0	88.0	73.0	44.0	50.0
ANI (Ours)	100.0	98.0	88.0	67.0	68.0	82.0

Table 2: N-Digit Arithmetic Scaling. Exact match accuracy (%) across varying lengths of numerical sequences. The results demonstrate the catastrophic calculation failures of standard subword tokenization on longer sequences (e.g., 6+ digits) and highlight ANI’s effectiveness as a structural stabilizer.

Figure 6: Interpretability Analysis of the Adaptive Gating Mechanism. (a) Selection ratio across benchmarks (y-axis: proportion). (b) Selection ratio across MMLU subjects (x-axis: proportion of

\alpha=1

Method	MATH	GSM8K	MMLU
Static Injection (No Gating)	8.60	62.70	71.41
Digit-wise Injection	58.70	90.37	72.29
ANI (Ours)	58.76	90.14	72.67

Table 3: Ablation Study. Impact of the gating mechanism and injection locus. Static Injection replaces tokens without gating, while Digit-wise Injection applies the gate to every sub-word digit token.

Model Family / Size	Method	MATH	GSM8K
Llama-3.1-8B	Pre-trained	20.5	55.3
	Baseline (SFT)	26.4	63.5
	XVal	21.2	59.5
	ANI (Ours)	27.2	63.5
Mistral-7B-v0.3	Pre-trained	11.4	37.5
	Baseline (SFT)	16.6	52.4

Table 4: Scalability and Generalization Results. Comparison of mathematical reasoning performance across various distinct architectures and model sizes.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Model performance comparison with 95% CI. Bars represent the overall accuracy on the MATH benchmark, with error bars denoting the 95% confidence intervals derived from paired bootstrap resampling.

${}^{*}p<0.05$
Group	Acc. Diff	Paired BS 95% CI	BS $p$ (one-sided)
Level 1, 2, 3	$-0.0049$	$[-0.0150,+0.0057]$	$0.8303$
Level 4, 5	$+0.0134$	$[+0.0004,+0.0268]$	$0.0241^{*}$

Appendix

Table 5: Fine-grained bootstrap statistical analysis. Comparison of ANI’s accuracy gain relative to xVal on the MATH benchmark.

Explore similar work

Numeracy in Large Language Models: Fundamental Limitations and Paths to Improvement
Aug 13, 2026Aoxin NiMathematical Reasoning Benchmarks Fundamental Limits
Department of Computer Science University of Chinese Academy of Sciences
PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models
Jun 2, 2026Zetian Ouyang, Linlin Wang, Gerard de Melo +1Large Language Model Benchmarks
East China Normal University · Hasso Plattner Institute, University of Potsdam
The Effect of Scripts and Formats on LLM Numeracy
Jan 21, 2026Varshini Reddy, Craig W. Schmidt, Seth Ebner +3Large Language Models Fail Impact Analysis
Kensho Technologies · Ben-Gurion University · MIT Cambridge, MA

Aug 13, 2026, cs.AIJ/K move · Enter open · S save

Numeracy in Large Language Models: Fundamental Limitations and Paths to Improvement

Aoxin Ni

Department of Computer Science University of Chinese Academy of Sciences

Mathematical Reasoning Benchmarks Fundamental Limits Improvement Tokenizer Path

Open paper AskPDF arXiv

Large language models (LLMs) achieve strong results on mathematical reasoning benchmarks yet remain unreliable on elementary numerical tasks, including magnitude comparison, large-integer arithmetic, fractions, and scientific notation. This survey examines basic numerical understanding as a capability distinct from high-level mathematical reasoning. We propose the Numerical Grounding Framework (NGF), which decomposes numeracy into Representational Grounding (RG), mapping numeral forms to value, magnitude, and equivalent representations, and Procedural Grounding (PG), executing arithmetic operations in accordance with their mathematical definitions. Using NGF, we organize recent diagnostic benchmarks, failure modes, structural explanations, and mitigation strategies. We review evidence concerning tokenization, positional encoding, embedding geometry, and pretraining-data distribution. We also apply NGF in a coordinated evaluation of three frontier model families across Number Cookbook, NumericBench, and GSM-Symbolic, comparing atomic, contextual, and reasoning-assisted numeracy. Architectural interventions such as digit-aware tokenization and Abacus Embeddings can improve models trained from scratch but are generally unavailable to users of pretrained systems, for whom supervised fine-tuning, reasoning scaffolds, and external tools are more practical. We conclude with deployment recommendations and research directions for more reliable numerical behavior in foundation models.

Abstract

Explore similar work

Numeracy in Large Language Models: Fundamental Limitations and Paths to Improvement

PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models

The Effect of Scripts and Formats on LLM Numeracy