Modern language models are increasingly asked to handle several tasks within a single prompt. Doing so requires solving each task correctly while integrating the resulting answers into a coherent response. In this paper, we identify a fundamental limitation in this process, where models make substantially more errors when tasks are composed, even when they can solve every constituent task reliably in isolation. This degradation cannot be fully explained by the accumulation of independent errors. Across both open-weight models and the frontier model Claude Opus 5, our findings instead point to an implicit thinking budget underlying this effect. In particular, under composition, models use substantially fewer tokens per task than in isolation, even when the full sequence remains well within the context window. Tasks that require longer solution traces in isolation are consequently more fragile under composition, i.e., models make more errors on prompts containing these tasks. In isolation, non-fragile tasks receive more reasoning than they need and waste compute. However, by grouping tasks according to their fragility, the implicit budgeting can be manipulated to reduce token usage by 43% across a range of math and linguistic tasks while also preserving accuracy.
Figures & tables
Figure 1: Composition reduces every task’s reasoning by the same factor. When solved in isolation (top), each task receives more reasoning than it requires. When composed into a single prompt (bottom), the reasoning allocated to each task shrinks proportionally (yellow). Tasks with sufficient slack absorb this reduction and are still solved correctly, whereas the task that requires long reasoning falls below its requirement and thus fails.
Figure 2: Accuracy degradation on confidently solved GPQA-Diamond tasks. Left: accuracy drop per model at k=10 . Right: accuracy drop as a function of composition size k . All tasks were solved in three out of three isolated attempts, yet accuracy drops once they are composed.
Figure 3: Context usage across datasets. Most responses use under 25% of the context window; in most datasets, under 1% hit the limit.
Figure 4: Token ratio ri(k)=Ti(k)/Ti(1) as a function of composition size k , where Ti(k) is the number of thinking tokens spent on task ti within a composition of size k and Ti(1) the number spent in isolation. Per-task token counts in compositions are obtained by locating task tags in the reasoning trace. Tasks are binned by the quantile of their isolated trace length into Low ( ≤20% ), Medium ( 20 – 80% ), and High ( >80% ). On log-log axes, the ratio decreases approximately linearly in logk , indicating that LLMs shrink the relative token usage of each task under composition by roughly km , with no systematic dependence on Ti(1) . Results are from Qwen3 4B Thinking.
Figure 5: Accuracy drop as a function of the relative thinking token reduction for Qwen3 4B Thinking on MMLU-Redux (left) and MATH-500 (right) below the dashed line, the y-axis is logarithmic for presentation purposes. Tasks are binned by the quantile of their isolated trace length into Low ( ≤20% ), Medium ( 20 – 80% ), and High ( >80% ). As the reduction through composition (comp.) grows, accuracy drops, and tasks with high reasoning requirements degrade substantially more than low-effort tasks. Forcing a fixed budget externally (forced) generally causes stronger degradation, but the degradation follows the same patterns (more details in Appendix A.1.3 ).
Isolated, k=1
Random batching, k=6
Fragility-aware (measured)
Fragility-aware (predictor)
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
GPQA-Diamond
Qwen3 4B T.
66.7%
7663
56.4% (−10.3)
2470 (−67.8%)
66.8% (+0.2)
5200 (−32.1%)
67.3% (+0.7)
7534 (−1.7%)
Qwen3 8B
58.7%
5973
53.4% (−5.3)
1950 (−67.4%)
60.0% (+1.3)
3520 (−41.1%)
56.9% (−1.8)
5106 (−14.5%)
Qwen3 14B
61.5%
4597
51.8% (−9.7)
1500 (−67.4%)
57.9% (−3.7)
2111 (−54.1%)
61.3% (−0.2)
3665 (−20.3%)
Qwen3.8 27B
83.2%
4303
79.2% (−4.0)
2740 (−36.3%)
84.0% (+0.8)
4102 (−4.7%)
85.3% (+2.1)
4324 (+0.5%)
Table 1: Task accuracy (Acc.) and output tokens (Tok., includes thinking and answer) per task under different batching policies. For each policy, the left column reports accuracy and the right column tokens, with the change to solving in isolation ( k=1 ) in parentheses, given as an absolute difference in percentage points for accuracy and as a relative change in % for tokens. The Average row gives the mean change over the models shown. Standard fixed-size random batching causes substantial accuracy degradation, which fragility-aware batching largely mitigates. With a model-specific linear predictor, part of these token savings carries over to unseen tasks while still avoiding compositional degradation.
Figure 6: Effects of implicit budgeting across three post-training stages of OLMo 3 7B on GPQA-Diamond. Left: Token ratio ri(k)=Ti(k)/Ti(1) as a function of composition size k . Right: Accuracy drop as a function of the token ratio.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Accuracy drop as a function of composition size k for all datasets and models. All tasks were solved in three out of three isolated attempts, yet accuracy drops once they are composed, on every dataset except OneWordQA and StrategyQA, whose tasks require little reasoning. The R1-distilled models degrade the most, and the extent of degradation varies strongly between datasets.
Figure 8: Token ratio on Qwen3 4B Thinking. Power-law fits through (1,1) , one per dataset: median slope m=−0.46 (range [−0.79,−0.34] ), mean R2=0.94 .
Figure 9: Token ratio on Qwen3 8B. Power-law fits through (1,1) , one per dataset: median slope m=−0.56 (range [−0.72,−0.45] ), mean R2=0.94 .
Figure 10: Token ratio on Qwen3 14B. Power-law fits through (1,1) , one per dataset: median slope m=−0.55 (range [−0.73,−0.42] ), mean R2=0.87 .
Figure 11: Token ratio on Qwen3.8 27B. Power-law fits through (1,1) , one per dataset: median slope m=−0.15 (range [−0.26,−0.07] ), mean R2=0.72 .
Figure 12: Token ratio on R1 Llama 8B. Power-law fits through (1,1) , one per dataset: median slope m=−0.68 (range [−0.78,−0.35] ), mean R2=0.59 .
Figure 13: Token ratio on R1 Qwen 7B. Power-law fits through (1,1) , one per dataset: median slope m=−0.62 (range [−0.78,−0.04] ), mean R2=0.62 .
Figure 14: Token ratio on R1 Qwen3 8B. Power-law fits through (1,1) , one per dataset: median slope m=−0.61 (range [−0.88,−0.46] ), mean R2=0.55 .
Figure 15: Accuracy drop versus relative thinking token reduction for Qwen3 4B Thinking.
Figure 16: Accuracy drop versus relative thinking token reduction for Qwen3 8B.
Figure 17: Accuracy drop versus relative thinking token reduction for Qwen3 14B.
Figure 18: Accuracy drop versus relative thinking token reduction for Qwen3.8 27B.
Figure 19: Accuracy drop versus relative thinking token reduction for R1 Llama 8B.
Figure 20: Accuracy drop versus relative thinking token reduction for R1 Qwen 7B.
Figure 21: Accuracy drop versus relative thinking token reduction for R1 Qwen3 8B.
Figure 22: Positional dependence of thinking budget allocation and compositional degradation on MATH-500 at composition size k=20 . The upper panels show thinking tokens (yellow-red-purple) and the lower panels show the accuracy drop (yellow-green-purple). The last position in each composition holds the dummy task, which we exclude from the analysis because our method cannot reliably determine its token usage (Appendix A.2.1 ).
Figure 23: Disattenuated Pearson r correlation of accuracy degradation across different conditions, i.e. model and composition size. Each pair of conditions (a cell in the heatmap) averages over datasets and, on the left, over models, on the right, over composition sizes. Left: The correlation is substantially larger when comparing accuracy degradation within the same model across different composition sizes k . Right: By contrast, the correlation between different models is low, especially across model families, suggesting different architectures and training recipes lead to different tasks being fragile.
Isolated, k=1
Random batching, k=6
Fragility-aware (measured)
Fragility-aware (predictor)
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
GPQA-Diamond
Qwen3 4B T.
66.7%
7663
56.4% (−10.3)
2470 (−67.8%)
66.8% (+0.2)
5200 (−32.1%)
67.3% (+0.7)
7534 (−1.7%)
Qwen3 8B
58.7%
5973
53.4% (−5.3)
1950 (−67.4%)
60.0% (+1.3)
3520 (−41.1%)
56.9% (−1.8)
5106 (−14.5%)
Qwen3 14B
61.5%
4597
51.8% (−9.7)
1500 (−67.4%)
57.9% (−3.7)
2111 (−54.1%)
61.3% (−0.2)
3665 (−20.3%)
Qwen3.8 27B
83.2%
4303
79.2% (−4.0)
2740 (−36.3%)
84.0% (+0.8)
4102 (−4.7%)
85.3% (+2.1)
4324 (+0.5%)
Appendix
Table 2: Task accuracy (Acc.) and output tokens (Tok., includes thinking and answer) per task under different batching policies for all models and datasets. The k=1 columns report absolute values. All other columns report absolute values with the change with respect to k=1 in parentheses, as an absolute difference in percentage points for accuracy and as a relative change in % for tokens. Within each policy, the left column is accuracy and the right column is tokens. The Average row reports the mean over all models shown, eight on GPQA-Diamond and seven on the other datasets: absolute values for k=1 and the mean change for all other policies. On OneWordQA, the Average leaves out R1 Qwen3 8B (grey), whose answers in composition do not follow our result contract, which leads to increased parsing errors.
Response at largest k
Responses (%)
Dataset
k
tokens
of budget
hit limit
unparseable
GSM8K
20
2.8k
3%
0.0 [0]
0.2 [2]
MATH-500
20
19.6k
29%
0.9 [10]
2.4 [32]
AIME
10
57.6k
35%
4.8 [39]
10.7 [47]
Integrals
20
7.4k
17%
0.0 [0]
1.4 [15]
MMLU-Redux †
20
1.6k
4%
0.2 [2]
3.9 [44]
Appendix
Table 3: Context limits and parsing failures in the composed sets. Each response has a token budget equal to the context window minus the prompt, capped at the model’s maximum generation length; a response that exhausts it is truncated. Response tokens are the mean at the largest k . Truncation and parsing-failure rates are pooled over all models and k , with the worst model and k in brackets. † Most unparseable responses come from R1 Qwen3 8B, which often breaks the answering contract under composition.
Standard evaluation of large language models assumes stable model rankings across inference conditions. We challenge this assumption by varying the token generation budget, i.e., the maximum tokens a model may produce, across seven levels (64--4,096), evaluating four models on three reasoning benchmarks (56,476 inferences). We report four findings: (i) 3--19% of items exhibit non-monotone behavior (accuracy decreasing with more budget), even after controlling for truncation, and this phenomenon is model-specific (cross-model overlap: 6--14%). (ii) Model rankings reverse across budgets on all benchmarks (p<0.01, McNemar). (iii) Oracle analysis reveals model complementarity up to +27.8pp, most pronounced at constrained budgets. (iv) A budget-aware router captures 14.1% of the oracle gap cross-domain; budget features help within-domain (+1.6 to +5.7pp) but are domain-specific and hurt transfer (−1.2pp). These results argue for budget-conditioned evaluation protocols.
Rodrigo Guedes de Souza, Alison R. Panisson
Federal University of Santa Catarina (UFSC) Brazil
Chain-of-thought reasoning is often treated as a monotone way to improve language-model accuracy by letting a model think longer. We identify a countervailing effect, the coupling tax: when reasoning traces and final answers share one output-token budget, long traces can crowd out the answer they are meant to support. Across GSM8K, MATH-500, and five BIG-Bench Hard tasks with Qwen3 models at three scales, non-thinking mode matches or outperforms thinking mode on GSM8K and MATH-500 at every budget up to 2048 tokens, while harder tasks shift the crossover to larger budgets. We derive a truncation-waste decomposition, Accthink(b)=αcFL(b)+αt(1−FL(b)), that predicts this crossover from chain-length and accuracy statistics and explains inverse scaling within the Qwen family. A DeepSeek-R1-Distill-Llama-8B replication shows the same pattern under a different thinking interface. As a mitigation, split-budget generation decouples reasoning and answer budgets; on full MATH-500, IRIS reaches 74.0% accuracy, a strengthened extraction variant reaches 78.8%, and a fixed non-oracle SC+IRIS gate reaches 83.6%. The results show that test-time reasoning should be evaluated as a budget-allocation problem, not only as a question of whether longer traces are available.
Adding inference structure to a language model lets it search, verify, and revise, but these actions consume the very budget they are supposed to use well. In this paper, we investigate whether there exists a token-budget threshold, below which the overhead of planning and verification hurts performance and above which it helps. We evaluate two systems on FinQA and TAT-QA financial reasoning tasks, using GPT-5.4 mini across 14 budget tiers ranging from 250 to 42,000 output-equivalent tokens. The first system is a monolith, which is a single LLM call. The second is a verified search architecture that adds planning, label-blind checking, and repair capabilities. We run 1,000 cases for a total of 28,000 completed cells. Both systems score 0% at the two lowest tiers, where neither can fit a complete prompt. At 1,000 tokens, the monolith reaches 18% accuracy while verified search scores near 0%, since the planning overhead leaves no room for an answer. From 1,500 tokens onward, verified search surpasses the monolith and maintains a consistent advantage, reaching approximately 44% at the highest tiers while the monolith reaches approximately 40%. The crossover occurs between 1,000 and 1,500 output-equivalent tokens, confirmed by a strict intersection-union test (p≤0.001 at both endpoints).