Numerical Reasoning in Language Models

Follow this topic Ask about this topic

Momentum

21 papers in the last four weeks, up 250% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 73

All topics

Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers
Oct 6, 2026Sourabh KasliwalNumerical Reasoning in Language Models Neural Algorithmic Reasoning
FinVector-Market-4B: A Controlled Study of LoRA Adaptation for Structured Financial Tasks
Oct 6, 2026Alina KhaybullinaDomain Adaptation for LLMs LLM Fine-Tuning
Independent Researcher
COMPASS: Finding Where Reasoning Lives in Language Models
Oct 5, 2026Pratyay Dutta, Kowshik Thopalli, Vivek NarayanaswamyEfficient Language Model Reasoning Activation Steering
University of California, Riverside, CA 92521. Work performed during a summer internship at Lawrence Livermore National Laboratory · Lawrence Livermore National Laboratory, Livermore, CA 94550.
Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution
Oct 5, 2026Erfan Baghaei Potraghloo, Seyedarmin Azizi, Arya Fayyazi +4Numerical Reasoning in Language Models Efficient Language Model Reasoning
University of Southern California, Los Angeles, USA · Intel AI, USA
Language models can notice an impossible engineering problem yet still report it as solved
Oct 5, 2026Shaoliang Yang, Jun WangNumerical Reasoning in Language Models Language Model Error Detection
Santa Clara University Santa Clara, CA 95053, USA
Separators Make Carry Propagation Learnable:The Geometry of Latent Carry in a Multiplication Transformer
Oct 5, 2026Sama Satariyan, Raphael Cousin, G{é}rard BiauNumerical Reasoning in Language Models Transformer Interpretability
SCAI · LPSM,IUF,MEGAVOLT
Towards Robust Numerical Claim Verification
Sep 30, 2026Peter Røysland Aarnes, Vinay SettyNumerical Reasoning in Language Models Claim Verification
University of Stavanger · Factiverse AI and University of Stavanger
SCB: SpeechConversationBench for Evaluating Multi-Turn Reasoning in Speech-to-Speech Models
Sep 30, 2026Kanpat Vesessook, Saksorn RuangtanusakNumerical Reasoning in Language Models Mathematical Reasoning
SCBX R&D · SCB DataX, SCBX Group
When the Right Answer Is Missing: An Arithmetic-Dependent Rejection Bottleneck in Jev
Sep 30, 2026Jike Zhong, Ming Li, Yuxiang LaiNumerical Reasoning in Language Models Selective Prediction
University of Southern California · University of Florida · Emory University
ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoning
Sep 30, 2026Jinsung Jeon, Seung-won HwangNumerical Reasoning in Language Models LLM Mathematical Reasoning
KAIST InnoCORE LLM · Seoul National University · Korea University
Targeted Retrieval, Compact Representations: How CoT Reasoning Improves Long-Context Counting
Sep 30, 2026Liang Twist Shan, Tianyu Hu, Hao Yan +1Numerical Reasoning in Language Models Long-Context Retrieval
Department of Statistics, University of Wisconsin-Madison · Department of Computer Sciences, University of Wisconsin-Madison
Reasoning Externalization for Faithful Large Language Model Narratives of Stock Return Predictions
Sep 30, 2026Sujung Kim, Seung Hwan Cho, Sangjin Park +1Numerical Reasoning in Language Models Faithfulness of Language Model Explanations
Department of Industrial Data Engineering, Hanyang University, Republic of Korea · School of Interdisciplinary Industrial Studies, Hanyang University, Republic of Korea
Activation-Conditioned Self-Distillation
Sep 29, 2026Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati +2Numerical Reasoning in Language Models On-Policy Self-Distillation
Rensselaer Polytechnic Institute · IBM Research
When Models Don't Manipulate Manifolds: The Geometry of a Comparison Task
Sep 29, 2026Sai Sumedh R. Hindupur, Hadas Orgad, Thomas Fel +1Numerical Reasoning in Language Models Mechanistic Interpretability
School of Engineering and Applied Science, Harvard University · Kempner Institute, Harvard University · Goodfire AI
Reasoning with Continuous Latent Diffusion
Sep 28, 2026Xiang ChengNumerical Reasoning in Language Models Code Generation
Duke University Department of Electrical and Computer Engineering
When Harness Beats Scale, and When Reading Beats Both
Sep 28, 2026Ivan Bondarenko, Nikolay O. NikitinNumerical Reasoning in Language Models Language Model Scaling
Novosibirsk State University · ITMO University
Next Thoughts Are Distributions: Generative Autoregressive Reasoning in the Latent Space
Sep 27, 2026Yang Li, Yi Wang, Shiyuan Huang +3Numerical Reasoning in Language Models Continuous Latent Reasoning
Rutgers University · Amazon · University of Illinois at Urbana-Champaign
Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning
Sep 27, 2026Safaeid Hossain Arib, Rabeya Akter, Ismam Nur Swapnil +3Numerical Reasoning in Language Models On-Policy Self-Distillation
ACI PLC, Bangladesh · University of Dhaka, Bangladesh · BRAC University, Bangladesh +2
NumericJev: Jev-like LLM Numerical Decoding with Multiway Decision Trees
Sep 23, 2026Weiwei Ye, Hangchen Liu, Renhe JiangNumerical Reasoning in Language Models Language Model Decoding
The University of Tokyo.
Ladders of Thought: A Self-Evolving Curriculum of Progressively Simplified Reasoning Traces
Sep 22, 2026Minghui Liu, Thomas Magelinski, Dehao Yuan +2Numerical Reasoning in Language Models Curriculum Learning
University of Maryland, College Park · Capital One
Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction
Sep 17, 2026Theodore O. CochranNumerical Reasoning in Language Models LLM Reliability
AI for Altruism
Voice of Reason: Reinforcement Learning for Spoken Math
Sep 16, 2026Timothée Weisselberger, Edouard Graves, Alexandre DéfossezNumerical Reasoning in Language Models RL for Language Model Reasoning
Kyutai Paris, France · Kyutai, Gradium Paris, France
Language Is an Insufficient Substrate for Quantitative Reasoning, and Consequential Domains Need Large Quantitative Models
Sep 14, 2026Reuben Vandeventer, David Imrem, David J. WildNumerical Reasoning in Language Models LLM Reliability
Duo Dimensio LLC · Luddy School of Informatics, Computing and Engineering, Indiana University
Towards a Deterministic Math Solver for Clinical Language Models
Sep 12, 2026Felipe Ocampo Osorio, Sebastián Andrés Cajas Ordoñez, Maximin Lange +5Numerical Reasoning in Language Models LLM Reliability
MIT Critical Data · UNC Chapel-Hil · University of Pavia +1
Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning
Sep 9, 2026Yaning Jia, Chunhui Zhang, Wenxuan Xu +3Numerical Reasoning in Language Models Supervised Fine-Tuning
Dartmouth College · Carnegie Mellon University
CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewards
Sep 7, 2026Zhuofan Chen, Ziqian Jiao, Yikai Cui +3Numerical Reasoning in Language Models Reinforcement Learning
School of Computer Science and Engineering, Beihang University, Beijing, China · Beijing Institute for General Artificial Intelligence, China
When Financial Fine-tuning Fails: A Three-Level Detectability Analysis of Numerical Hallucination in Domain-Adapted Language Models
Sep 7, 2026Xiaodong Li, Peiwei LiuNumerical Reasoning in Language Models Hallucination in Language Models
School of Computer Science, Guangzhou College of Applied Science and Technology, Guangzhou, 511370, China · IT Department, COFCO Corporation, Beijing, 100020, China
Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations
Sep 3, 2026Karthika Nhayakkat, Rajat Verma, Maharaj Brahma +4Numerical Reasoning in Language Models Multilingual Language Model Evaluation
Department of CSE, Indian Institute of Technology Bombay · SCEE, Indian Institute of Technology Mandi · Department of CSE, Indian Institute of Technology Hyderabad
Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards
Sep 3, 2026Leqi Zheng, Jinbo Su, Fang Niu +8Numerical Reasoning in Language Models Reward Shaping
Tsinghua University · Renmin University of China · Institute of Information Engineering, CAS +4
Numeracy in Large Language Models: Fundamental Limitations and Paths to Improvement
Aug 13, 2026Aoxin NiNumerical Reasoning in Language Models LLM Grounding
Department of Computer Science University of Chinese Academy of Sciences

Oct 6, 2026, cs.LGJ/K move · Enter open · S save

Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers

Sourabh Kasliwal

Numerical Reasoning in Language Models Neural Algorithmic Reasoning LLM Training Curriculum Learning

Open paper AskPDF arXiv

Autoregressive Large Language Models (LLMs) frequently struggle with deterministic multi-step algorithmic tasks such as multi-digit multiplication and long division. In this paper, we investigate the mechanics of multi-step arithmetic in compact "Tiny" Transformers (~10.6M non-embedding parameters, 49.3M total) trained on synthetic data across four basic operations (+, -, *, /) unrolled as step-by-step scratchpads. First, we establish the necessary training foundations: (1) dataloader sequence padding creates an 83% gradient starvation artifact that collapses accuracy from 40% to 1%, remediated via continuous sequence packing; (2) linguistic pretraining is an essential prerequisite (<= 2.0% without it); and (3) modern architectural primitives (RoPE, RMSNorm, SwiGLU) and Sparse Mixture of Experts (MoE) substantially improve additive reasoning over baseline GPT-2. Second, we demonstrate that algorithmic scratchpad formulation directly dictates success. Introducing a deterministic Digit-by-Digit Long Division scratchpad within a 4-stage Hierarchical Developmental Curriculum dramatically elevates single-digit division from 4.0% to 86.7% accuracy on a 4,000-problem held-out benchmark. In contrast, multi-digit multiplication remained challenging: detailed error analysis revealed that while the model correctly computed single-digit sub-products and place-value zeros, our FOIL scratchpad failed because it forced a simultaneous summation of up to nine multi-digit terms in a single step without pairwise intermediate accumulation. Finally, we identify two key boundaries: performance collapses to 0.00% on unseen 4-digit operands, and unbuffered training induces catastrophic forgetting, collapsing division accuracy from 86.7% down to 0.00%.

Momentum

Related topics

Latest papers 73