cs.AISep 26, 2026

The Decomposition Tax: LLM Pipelines Lose Up to 40 Accuracy Points at Their Own Interfaces

Authors: Tianqi Bu, YuXuan Peng, Junteng Tu, Henghui Xiao

Organizations: Rutgers University · Nanjing Tech University · Nanjing University of Posts and Telecommunications · New York University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A four-stage LLM pipeline gives up as much as 40.5 accuracy points at its own interfaces (gemma-3-12B on MATH-500, Holm-corrected p = 1.66e-19; the largest tax in the primary family). We hold model, problem, stages, stage prompts and completion budget fixed, vary only whether each stage can still see the original problem, and call the accuracy difference the decomposition tax. Across 21 open-weight models from nine organisations, on GSM-Hard and MATH-500 at n = 200 paired items per cell, 70 of 118 primary-family tests survive Benjamini-Hochberg correction and 54 survive Holm. On GSM-Hard, a placebo recovers nothing: it carries at least 60% of the extra tokens and at most one word of the problem. Builders design a pipeline one stage at a time, and its bill arrives at the interfaces between stages. Rewriting one stage's instruction moves gemma-3-12B's tax from 4.5 to 36.5 points, and adding "every relationship stated between them" to a stage that lists the numerical quantities lowers the tax on 9 of 9 models on MATH-500. Re-grounding, which shows a stage the original problem again, belongs after the loss. With one lossy interface, re-grounding the stage after it beats re-grounding the stage before it on 7 of 7 models on both benchmarks; on MATH-500 the earlier repair is worse than none on 7 of 7. Newer models still pay: gemma-4-12B gives up 37.0 points, and the repair holds on all three of the newest models we test. A sealed held-out test refuted a stronger rule we registered, which predicted the paying stage from the interface and receiver types, so we locate the tax by measuring one stage at a time. The prescription has two parts: re-ground the stage after the lossy interface, and if a stage must list the quantities, tell it to keep the relationships.

Figures & tables

Figure 1: The decomposition tax at a glance. a The measurement: the arms share model, calls, stage prompts and completion budget and differ only in which stages see the original problem. b What is varied at one interface, with the mean tax of each square’s corners. c How large the tax is and the controls behind it. d Where re-grounding helps, the rule that failed its sealed test, and the 2026 cohort.
Figure 2: The tax is large across the corpus, and a condensing edge costs far more than a preserving one. a,b Per-model tax on each benchmark, with bootstrap 95% intervals; a faded monolith tick marks models whose monolith hits the token cap. c The same cells paired within model: points above the diagonal are models where condensing costs more than preserving. d Tax against strict-arm accuracy, part of which the tax contains by definition; hollow markers are monoliths over that same cap.
Figure 3: What the sender keeps, how it writes and what the receiver may do all move the tax. a,b The depth-4 square on MATH-500 and GSM-Hard, across the receiver factor. c The depth-3 square on MATH-500, across output form; all four of its corners use mathematical wording (Section 3.2). In a to c , faint lines are models and bold lines their means. d Each square’s main effects and interaction.
Figure 4: Which stage is re-grounded decides the recovery, and the intuitive choice is the wrong one. All panels MATH-500. a MIX_RL, the loss at the second interface: re-grounding stage 2 recovers more than stage 1 on 7 of 7 models. b MIX_LR, the loss at the first interface: both stages are downstream and neither dominates. c Stage 1 re-grounded alone on MIX_RL, upstream of the loss: negative on all 7 models, worse than no repair. d Total tax with the loss early against late; the registered order test is in Appendix F.
Figure 5: The tax survives into the 2026 cohort; on MATH-500 its make-up shifts. a Tax on MATH-500, A_asset, per model, in the cohorts of Section 3; vertical rules mark cohort means. b Each model’s edge and receiver main effects on the depth-4 square, drawn as a slope; thick lines are cohort means. c Edge against receiver effect, with the diagonal. d The depth-3 square with list output: tax under the condensing against the preserving instruction, both in mathematical wording.
More in this topic

LLM Grounding

Large language model (LLM) grounding studies how models anchor answers and actions in evidence, structured knowledge, and real-world context.

See all 320 papers

Explore similar work

CardsList
Sep 26, 2026

Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning

Large language models (LLMs) can solve every intermediate step of a multi-step math problem on its own and still fail the full problem, even when given a roadmap of the steps and all of their answers. We introduce OracleLadder, a diagnostic evaluation that locates where LLM math reasoning fails by giving the model increasing levels of oracle help. For each problem, a teacher model writes a fixed roadmap of intermediate sub-goals (milestones), and a deterministic symbolic verifier grades every answer. Testing the model with no help, with the roadmap, with the roadmap plus the milestone answers, and on each milestone alone sorts each failure into one of five reasoning gaps. On 354 NuminaMath problems and six models from 8B to 671B parameters (Qwen3, gpt-oss, Llama 3.3, DeepSeek-V3.1), the largest gap for every model is the composition gap, a stricter form of the compositionality gap. It covers 33-48% of problems, and 24-37% after removing problems that an LLM review flags as grading errors. Accuracy and milestone-help recovery rank the two strongest models differently, and two RLVR runs with similar accuracy gains move problems differently. The roadmap effect replicates on MATH500 and AIME 2024/25, per-problem recovery agrees for 83-87% of problems under an independent second teacher, and the help ladder carries over to code generation. We release the data, roadmaps, prompts, and code at https://github.com/slark-prime/OracleLadder.

Zhuohan Wang, Haoran Ma, Tianyu Wu +3

Harvard University · Rutgers University · The Hong Kong University of Science and Technology

LLM Mathematical ReasoningLLM EvaluationMathematical Reasoning Benchmarks
AskPDF
Sep 15, 2026

A Four-Stage Decomposition of Word-Problem Solving and Mechanistic Fragility in LLM Math Reasoning

Large language models solve grade-school math word problems with high accuracy, yet a single irrelevant clause inserted into the problem can collapse it. We reconcile these observations with a mechanistic account. We show that the model's internal computation decomposes into a four-stage sequential pipeline, Schema Abstraction, Operation Planning, Operand Binding, and Computation, each stage producing a distinct intermediate representation in an identifiable band of layers. Using the same scaffold to diagnose distractor-induced failure, we localize the corruption to a single stage, Operation Planning, implemented by a set of attention heads whose causal role we validate bidirectionally. In short, we provide a mechanistic interpretation of math word problem reasoning in LLMs, and their failure when distracted.

Zhongdi Qu, Carla P. Gomes

Cornell University

Mathematical ReasoningCircuits in Language ModelsLanguage Model RobustnessLLM Mathematical Reasoning
AskPDF
Apr 4, 2026

The Format Tax

Asking a large language model to respond in JSON should be a formatting choice, not a capability tax. Yet we find that structured output requirements -- JSON, XML, LaTeX, Markdown -- substantially degrade reasoning and writing performance across open-weight models. The research response has focused on constrained decoding, but sampling bias accounts for only a fraction of the degradation. The dominant cost enters at the prompt: format-requesting instructions alone cause most of the accuracy loss, before any decoder constraint is applied. This diagnosis points to a simple principle: decouple reasoning from formatting. Whether by generating freeform first and reformatting in a second pass, or by enabling extended thinking within a single generation, separating the two concerns substantially recovers lost accuracy. Across six open-weight models, four API models, four formats, and tasks spanning math, science, logic, and writing, decoupling recovers most lost accuracy. Notably, most recent closed-weight models show little to no format tax, suggesting the problem is not inherent to structured generation but a gap that current open-weight models have yet to close. Code is available at https://github.com/ivnle/the-format-tax.

Ivan Yee Lee, Loris D'Antoni, Taylor Berg-Kirkpatrick

UC San Diego

LLM PromptingStructured Output GenerationLanguage Model Generation EvaluationFraming Effects in Language ModelsConstrained Decoding
AskPDF