Explicit reasoning can help language models generate code, but it also introduces a practical risk: the model may spend so much of its output budget reasoning that it never produces usable code. We study this behaviour in relatively small open-weight reasoning models, framing it as reasoning-trace overflow. Across seven models and six code benchmark settings, we measure how much of a fixed 32k-token output budget is spent on reasoning, whether that reasoning improves pass@1, and whether lightweight inference-time controls can mitigate overflow. Our results show that the value of reasoning is highly conditional. Overflow is rare on simpler function-level tasks, but becomes common on harder code-generation benchmarks and for particular model families: on several model--benchmark pairs, more than half of generations exhaust the full 32k-token budget while still reasoning. Most complete overflow is associated with repetitive reasoning loops, although substantial long reasoning also occurs without looping. Disabling reasoning can therefore be competitive for overflow-prone models, simply because it ensures that an answer is produced. At the same time, reasoning remains valuable for stronger models on harder algorithmic and code-understanding tasks. Finally, prompt steering can substantially reduce overflow for some models but is inconsistent, while budgeted transitions provide more direct control and can improve both performance and token efficiency for overflow-prone models. These findings suggest that explicit reasoning should be treated as a controllable resource for code generation, rather than an always-on capability.
Figures & tables
Fig. 1: Results for RQ1 Overflow Dynamics. Distribution of reasoning-token usage under the standard reasoning-enabled setting. Each violin shows the percentage of the 32k output budget consumed by the explicit reasoning trace for a given model and benchmark. The dashed red line marks the full output budget, and the percentage above each violin reports the reasoning-trace overflow rate.
Fig. 2: Reasoning Loop Analysis. Percentage of overflowed generations containing a detected reasoning loop for each model. A loop is identified as a repeated region spanning at least 2,000 characters and three complete cycles.
Fig. 3: Results for RQ2 Reasoning Utility. Model accuracy ( pass@1 ) with and without reasoning across all models and benchmarks. Solid bars show the standard reasoning-enabled setting, while hatched bars show the reasoning-disabled setting, where empty reasoning tags are inserted before generation to force the model to answer directly. Error bars indicate 95% Wilson confidence intervals.
Fig. 4: Results for RQ3 Reasoning Control, Prompt Steering. Effect of prompt-based steering on reasoning behaviour. Bars show macro-average pass@1 and reasoning-trace overflow across all benchmark settings for each model under the reasoning enabled, concise, token-budget, aspects, and Plan-and-Solve prompt-based strategies.
Fig. 5: Results for RQ3 Reasoning Control, Budgeted Transitions. Effect of enforced reasoning-transition budgets on accuracy. Each line shows macro-average pass@1 across all benchmark settings as the transition budget varies from immediate transition to 28k tokens. Dashed lines show each model’s standard reasoning-enabled baseline, and stars mark the best observed transition budget.
Fig. 6: Token Efficiency Analysis. Token efficiency across inference strategies, macro-averaged over all six benchmark settings. The top panel shows mean output tokens , while the bottom panel shows tokens per correct answer on a log scale. For each model, the prompt style and transition budget with the highest macro-average pass@1 are shown.
Reasoning-specialized models like OpenAI's 5.1 and DeepSeek-V3.2 allocate substantial inference compute to extended chain-of-thought (CoT) traces, yet reasoning tokens incur significant costs. How do different reasoning modalities of code, natural language, hybrid, or none do perform under token constraints? We introduce a framework that constrains models to reason exclusively through code, comments, both, or neither, then systematically ablates token budgets to 10%, 30%, 50%, and 70% of optimal. We evaluate four frontier models (GPT-5.1, Gemini 3 Flash, DeepSeek-V3.2, Grok 4.1) across mathematical benchmarks (AIME, GSM8K, HMMT). Our findings reveal: (1) truncated reasoning can hurt as DeepSeek-V3.2 achieves 53% with no reasoning but only 17% with truncated CoT at 50% budget; (2) code degrades gracefully as Gemini's comments collapse to 0% while code maintains 43-47%; (3) hybrid reasoning underperforms single modalities; (4) robustness is model-dependent as Grok maintains 80-90% at 30% budget where OpenAI and DeepSeek collapse to 7-27%. These results suggest incomplete reasoning chains actively mislead models, with implications for deploying reasoning-specialized systems under resource constraints.
Recent advances in large language models (LLMs) have shown that test-time scaling can substantially improve model performance on complex tasks, particularly in the coding domain. Under this paradigm, models use a larger token budget during inference to generate intermediate reasoning traces before producing a final answer. However, current evaluations primarily rely on competitive programming benchmarks, which may not capture the full range of reasoning abilities. In this work, we perform a systematic study of frontier reasoning models to understand their performance on real-world coding benchmarks. To gain more insights into the performance of such models, we devise a programmatic way to {\em automatically generate} coding tasks of arbitrary difficulty and structure from existing benchmarks. Using this framework, our analysis reveals that the structure of a reasoning trace, not just its contents, is a strong predictor of correctness. Motivated by this, we propose structured thought-trees as means to represent reasoning traces. To illustrate their use, we train a lightweight classifier on features extracted from thought-trees to predict trace correctness, and demonstrate that flagging and retrying structurally anomalous traces based on the extracted features yields consistent gains at lower complexity levels.
Recent advances in reasoning Large Language Models (LLMs) have primarily relied on upfront thinking, where reasoning occurs before final answer. However, this approach suffers from critical limitations in code generation, where upfront thinking is often insufficient as problems' full complexity only reveals itself during code implementation. Moreover, it cannot adaptively allocate reasoning effort throughout the code generation process where difficulty varies significantly. In this paper, we propose Think-Anywhere, a novel reasoning mechanism that enables LLMs to invoke thinking on-demand at any token position during code generation. We achieve Think-Anywhere by first teaching LLMs to imitate the reasoning patterns through cold-start training, then leveraging outcome-based RL rewards to drive the model's autonomous exploration of when and where to invoke reasoning. Extensive experiments on four mainstream code generation benchmarks (i.e., LeetCode, LiveCodeBench, HumanEval, and MBPP) show that Think-Anywhere achieves state-of-the-art performance over both existing reasoning methods and recent post-training approaches, while demonstrating consistent generalization across diverse LLMs. Our analysis further reveals that Think-Anywhere enables the model to adaptively invoke reasoning at high-entropy positions, providing enhanced interpretability.
Xue Jiang, Tianyu Zhang, Ge Li +8
School of Computer Science, Peking University · Tongyi Lab, Alibaba Group