"Close enough?" An investigation into multiplication errors in LLMs

William Bolton; Bezawit Tilaye; Huw Day

cs.AIOct 28, 2026

"Close enough?" An investigation into multiplication errors in LLMs

Authors: William Bolton, Bezawit Tilaye, Huw Day

Organizations: School of Engineering Maths & Technology University of Bristol · School of Computer Science University of Bristol

Abstract

Large language models (LLMs) frequently fail at multi-digit arithmetic, but the underlying mechanisms governing these failures remain poorly understood. While recent work demonstrates that addition errors follow a "noisy quantisation" model localised around carry boundaries, whether this framework generalises to multiplication remains an open question. In this work, we evaluate multiplication capabilities across several small base LLMs on a dataset of 16,991 problems spanning varied operand lengths. We show that multiplication errors do not conform to the addition-style noisy quantisation model. Instead, we uncover a novel error regime in small Qwen models: errors in lower-order columns predominantly preserve parity. Conversely, higher-order digit errors shift toward off-by-one carry failures. Our findings suggest that arithmetic computation in LLMs relies on distinct, multi-periodic internal representations where parity is tracked more robustly than exact digit identity, providing new mechanistic insights into numerical reasoning in transformers.

Figures & tables

Digit size	Count	Sampling
$1\times 1$	81	Exhaustive
$1\times 2$	810	Exhaustive
$2\times 2$	8,100	Exhaustive
$2\times 3$	3,000	Random sample (of 81,000)
$2\times 4$	1,000	Random sample (of 810,000)
$3\times 3$	3,000	Random sample (of 810,000)

Table 1: Multiplication dataset by operand-size tier. Three smallest tiers are exhaustive over all possible operand pairs while the remaining four are uniform samples (without replacement) from the full set of pairs.

Figure 1: Carry Potential demonstrated for addition

Figure 2: Carry Potential against error percentage for 2-digit $\times$ 2-digit multiplication using Qwen3.5-2B. The dashed line shows the best-fit noisy quantisation model from Wen et al. (2026), with fitted noise standard deviation of $\sigma=0.13$ and goodness-of-fit $R^{2}=-1.74$ . The negative $R^{2}$ indicates that the noisy quantisation fit is worse than a flat baseline, confirming the model does not cleanly transfer from addition to multiplication.

Figure 3: Carry Potential demonstrated for multiplication

Figure 4: Two examples of long multiplication to show an example of no carry columns (left) and one with three carry columns (right). Carries which contribute to those three carry columns are highlighted in yellow. The green arrows indicate those columns with carries.

Figure 5: Comparison of the accuracy of multiplications for different operand sizes (i.e. 2-digit x 2-digit up to 3-digit $\times$ 4-digit) for different carry types (no, low and high).

Figure 6: Error parity preservation rate on a per column basis compared to the null baseline and the ground-truth digit even fraction among incorrect digits in that column.

Model	Accuracy (%)	Tens Parity Preservation (%)
Qwen3.5-2B	56.4	92.7
Qwen3.5-0.8B	60.8	69.5
Qwen2.5-Math-1.5B	50.4	67.9
Llama-3.2-3B	25.0	45.2
gemma-4-E2B	28.8	44.0

Table 2: Multiplication accuracy across similarly sized models tested on the same dataset (n=16,991 and null parity preservation baseline is 44.5-45.2%)

Numerical Reasoning in Language Models

Numerical reasoning in language models studies how systems represent, calculate with, and explain numbers across mathematical and real-world tasks.

See all 156 papers

Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs
Apr 20, 2026Samuel G. Balter, Ethan Jerzak, Connor T. JerzakMultimodal Large Language Models Multimodal Reasoning
University of Texas at Austin · National University of Singapore (NUS)
Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers
Oct 6, 2026Sourabh KasliwalNumerical Reasoning in Language Models Neural Algorithmic Reasoning
Separators Make Carry Propagation Learnable:The Geometry of Latent Carry in a Multiplication Transformer
Oct 5, 2026Sama Satariyan, Raphael Cousin, G{é}rard BiauNumerical Reasoning in Language Models Transformer Interpretability
SCAI · LPSM,IUF,MEGAVOLT

Apr 20, 2026J/K move · Enter open · S save

Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs

Samuel G. Balter, Ethan Jerzak, Connor T. Jerzak

University of Texas at Austin · National University of Singapore (NUS)

Multimodal Large Language Models Multimodal Reasoning Multimodal Foundation Models Multimodal Model Evaluation Multimodal Understanding

Open paper AskPDF arXiv

Multimodal LLMs can accurately perceive numerical content across modalities yet fail to perform exact multi-digit multiplication when the identical underlying arithmetic problem is presented as numerals, number words, images, or in audio form. Because existing benchmarks often lack systematically paired instances across modalities, it remains difficult to compare genuine arithmetic limits within and across model families. We therefore introduce a controlled multimodal multiplication benchmark that factorially varies digit length, digit sparsity, representation (e.g., numerals vs. number words), and modality (text, rendered images, audio), with paired instances from a reproducible generator. We also define arithmetic load, C, as the product of the total and non-zero digit count as a compact, mechanistically motivated proxy for operation count. Across evaluations, accuracy falls sharply as C grows, often nearing zero by C > 100. Indeed, C remains predictive of performance across modalities and models, with R-squared often > 0.5, nearing the value from more complex measures of arithmetic load that count the number of intermediate arithmetic steps. A separate perception-versus-computation decomposition shows that multimodal degradation is primarily computational rather than perceptual: on matched-perception checks, models are near-perfect (> 99%) across modalities, even when multiplication accuracy drops. Beyond measuring when models fail, we ask which procedures they are predisposed to follow. We introduce a forced-completion loss probe that scores heuristic-specific reasoning prefixes--including columnar multiplication, distributive decomposition, and rounding/compensation. Here, decomposition is favored in both text and vision modalities; heuristic-specific LoRA adapters produce near-orthogonal updates yet degrade accuracy, indicating the base model maintains a well-tuned internal router.

Abstract

Numerical Reasoning in Language Models

Explore similar work

Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs

Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers

Separators Make Carry Propagation Learnable:The Geometry of Latent Carry in a Multiplication Transformer