Numerical Reasoning in Language Models

Follow this topic Ask about this topic

Numerical reasoning asks whether language models can read quantities, perform arithmetic, compare values, and use numerical evidence in mathematical and practical settings. Researchers test these abilities with math benchmarks such as GSM8K and MATH, as well as table, spreadsheet, financial, scientific, clinical, and multilingual tasks. Approaches include digit-aware representations, chain-of-thought and code execution, calculator and retrieval tools, symbolic solvers, and supervised or reinforcement-learning training with verifiable rewards. Mechanistic analyses examine whether numbers are merely represented or actually used, while tests of formatting, distractors, long contexts, units, and distribution shifts reveal reliability gaps; a central tension is fluent explanations versus consistently correct, auditable computation.

Latest papers

156 papers30 papers in the last four weeks, up 30% on the four weeks before. 0.2% of all new papers.

Latent Path Exploration via Potential-Guided Subtrajectory Balance
Oct 29, 2026Xiaoling Zeng, Qi ZhangLLM Reasoning Continuous Latent Reasoning
Worcester Polytechnic Institute
"Close enough?" An investigation into multiplication errors in LLMs
Oct 28, 2026William Bolton, Bezawit Tilaye, Huw DayNumerical Reasoning in Language Models LLM Reliability
School of Engineering Maths & Technology University of Bristol · School of Computer Science University of Bristol
Calculation and rejection diverge in language models tested at known mechanical limits
Oct 28, 2026Shaoliang Yang, Jun WangLLM Reliability Refusal Behavior in Language Models
Santa Clara University Santa Clara, CA 95053, USA
Assessing Language Models for Scientific Applications: Inertial Confinement Fusion as a Use Case
Oct 27, 2026P. B. Radha, D. D. Meyerhofer, M. Bhattarai +11Scientific Reasoning in Language Models Language Model Reasoning Evaluation
Los Alamos National Laboratory, Los Alamos, NM · NVIDIA Corporation, Santa Clara, CA
Understanding Large Language Models as Numerical Optimizers
Oct 27, 2026Lingyun Guo, Ali BereyhiLarge Language Model-Guided Optimization Numerical Reasoning in Language Models
Edward S. Rogers Sr. Department of Electrical and Computer Engineering (ECE), University of Toronto, Toronto, ON, Canada · Schwartz Reisman Institute (SRI) for Technology and Society
Are Linguistic and Arithmetic Circuits Dissociated in LLMs?
Oct 20, 2026Riku Kisako, Tatsuki Kuribayashi, Ryohei SasanoCircuits in Language Models Numerical Reasoning in Language Models
Nagoya University, Japan · MBZUAI, UAE · Tohoku University, Japan
ProteinGeoBench: Frontier Structural-Biology Agents Are Fluent at Lookup but Lost in Geometry
Oct 19, 2026Matt Turk, Brandon L. Neel, Diana M. Lopez +3AI Agent Benchmarks Numerical Reasoning in Language Models
Protege DataLab · Heimdall Bio
Where RL Breaks: Localizing and Repairing Failures via Targeted Continued Pretraining
Oct 19, 2026Xinyi Wang, Danqi ChenRL for Language Model Reasoning Language Model Pretraining
Princeton Language and Intelligence
TIRE: Top-Down Formula Instantiation with Reliable Execution for Physics Problem Synthesis
Oct 18, 2026Xiaoyu Wu, Zhaoxin Yu, Xinglin Xiao +1Synthetic Data Generation for Language Models Scientific Reasoning in Language Models
ScienceOne AI · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
P2N: Reusing Deep Representations for Greater Effective Depth
Oct 18, 2026Yiqin Hao, Boyi Zeng, Zitong Wang +7Efficient Language Model Inference Recurrent Transformers
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai AI Laboratory
CME-SpectrumBench: Can LLMs Analyze Condensed Matter Spectral Data?
Oct 16, 2026Jin Gene Wong, Anjney Midha, Joseph Mathew Tennyson +3LLM Evaluation Scientific Reasoning in Language Models
Physics Department and Stanford Laboratory for Intelligent Matter, Stanford University, Stanford, CA 94305 · Applied Physics Department, Stanford University, Stanford, CA 94305 · Electrical Engineering & Computer Sciences, University of California, Berkeley, Berkeley, CA 94720 +1
NLData2Opt: Benchmarking Parameter Derivation and Prediction for Modeling Optimization Problems from Natural Language and Data
Oct 16, 2026Lijia Liu, Kyohei Atarashi, Jiyi Li +3Natural Language to Optimization Modeling LLM Evaluation
Kyoto University · Hokkaido University · Recruit Co., Ltd.
Patch where the text is new: a compression view of where byte-level models should spend compute
Oct 15, 2026Nicolás Vera ZúñigaByte-Level Language Model Numerical Reasoning in Language Models
Independent Researcher, Chile
Momentum, Commentary Tone, and Earnings Surprise in LLM Financial Forecasts
Oct 15, 2026Alina KhaybullinaFinancial Forecasting Numerical Reasoning in Language Models
Bellman-Consistent Trajectory RL for LLM Arithmetic Reasoning
Oct 15, 2026Lianfa LiRL for Language Model Reasoning Reinforcement Learning
Department of Computer Science, Stanford University · LREIS, Institute of Geographical Sciences and Natural Resources Research, Chinese Academy of Sciences
INTACT: Inducing Latent Reasoning on a Frozen Backbone
Oct 14, 2026Kutay Tire, Atula Tejaswi, Haris Vikalo +1Efficient Language Model Reasoning Implicit Reasoning in Language Models
The University of Texas at Austin
Dynamic K-Head: Difficulty-Supervised Budget Selection for Continuous Latent Reasoning
Oct 13, 2026Ying Li, Yuqi Chen, Sixuan Li +1Efficient Language Model Reasoning Inference-Time Compute Allocation
Beijing Institute of Mathematical Sciences and Applications, Beijing, China · Tsinghua University, Beijing, China · The University of Hong Kong, Hong Kong, China +1
TPPO: Tracking-Value PPO For Language Model Reasoning
Oct 12, 2026Futing Wang, Chenxi Li, Jianhao Yan +6RL for Language Models RL for Language Model Reasoning
Zhejiang University · Westlake University · Shanghai AI Laboratory +4
MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis
Oct 8, 2026Ximing Lu, Aviral Kumar, Jan Kautz +2Reinforcement Learning with Verifiable Rewards Synthetic Data Generation for Language Models
University of Washington · Carnegie Mellon University · NVIDIA +1
Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers
Oct 6, 2026Sourabh KasliwalNumerical Reasoning in Language Models Neural Algorithmic Reasoning
FinVector-Market-4B: A Controlled Study of LoRA Adaptation for Structured Financial Tasks
Oct 6, 2026Alina KhaybullinaDomain Adaptation for LLMs LLM Fine-Tuning
Independent Researcher
COMPASS: Finding Where Reasoning Lives in Language Models
Oct 5, 2026Pratyay Dutta, Kowshik Thopalli, Vivek NarayanaswamyEfficient Language Model Reasoning Activation Steering
University of California, Riverside, CA 92521. Work performed during a summer internship at Lawrence Livermore National Laboratory · Lawrence Livermore National Laboratory, Livermore, CA 94550.
Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution
Oct 5, 2026Erfan Baghaei Potraghloo, Seyedarmin Azizi, Arya Fayyazi +4Numerical Reasoning in Language Models Efficient Language Model Reasoning
University of Southern California, Los Angeles, USA · Intel AI, USA
Language models can notice an impossible engineering problem yet still report it as solved
Oct 5, 2026Shaoliang Yang, Jun WangNumerical Reasoning in Language Models Language Model Error Detection
Santa Clara University Santa Clara, CA 95053, USA
Separators Make Carry Propagation Learnable:The Geometry of Latent Carry in a Multiplication Transformer
Oct 5, 2026Sama Satariyan, Raphael Cousin, G{é}rard BiauNumerical Reasoning in Language Models Transformer Interpretability
SCAI · LPSM,IUF,MEGAVOLT
Towards Robust Numerical Claim Verification
Sep 30, 2026Peter Røysland Aarnes, Vinay SettyNumerical Reasoning in Language Models Claim Verification
University of Stavanger · Factiverse AI and University of Stavanger
SCB: SpeechConversationBench for Evaluating Multi-Turn Reasoning in Speech-to-Speech Models
Sep 30, 2026Kanpat Vesessook, Saksorn RuangtanusakNumerical Reasoning in Language Models Mathematical Reasoning
SCBX R&D · SCB DataX, SCBX Group
When the Right Answer Is Missing: An Arithmetic-Dependent Rejection Bottleneck in Jev
Sep 30, 2026Jike Zhong, Ming Li, Yuxiang LaiNumerical Reasoning in Language Models Selective Prediction
University of Southern California · University of Florida · Emory University
ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoning
Sep 30, 2026Jinsung Jeon, Seung-won HwangNumerical Reasoning in Language Models LLM Mathematical Reasoning
KAIST InnoCORE LLM · Seoul National University · Korea University
Targeted Retrieval, Compact Representations: How CoT Reasoning Improves Long-Context Counting
Sep 30, 2026Liang Twist Shan, Tianyu Hu, Hao Yan +1Numerical Reasoning in Language Models Long-Context Retrieval
Department of Statistics, University of Wisconsin-Madison · Department of Computer Sciences, University of Wisconsin-Madison

Oct 29, 2026J/K move · Enter open · S save

Latent Path Exploration via Potential-Guided Subtrajectory Balance

Xiaoling Zeng, Qi Zhang

Worcester Polytechnic Institute

LLM Reasoning Continuous Latent Reasoning Numerical Reasoning in Language Models

Open paper AskPDF arXiv

Continuous latent reasoning enables language models to perform intermediate computation without generating explicit reasoning traces. Advances in stochastic latent reasoning allow multiple paths to be sampled in continuous space, but stochasticity alone does not ensure effective exploration. A key challenge is guiding stochastic transitions between intermediate latent states toward useful reasoning paths. We introduce SubLAT, which couples a relative answer potential with Subtrajectory Balance to learn stochastic latent exploration. The relative potential evaluates the answer support of sampled states against replacement states, with a correctness signal at the terminal state. Subtrajectory Balance relates potential differences to forward and backward transition densities across subtrajectories of different lengths. The model is jointly trained with answer supervision, CoT supervision, and representation distillation. Experiments on arithmetic and deductive reasoning show increasing solution coverage with larger sampling budgets across in-domain and out-of-domain benchmarks, as well as improvements on several structural OOD benchmarks. Trajectory analyses reveal progressive divergence among sampled latent paths and higher mean relative potentials for correct trajectories across arithmetic benchmarks.