Brief Is Better: Non-Monotonic Chain-of-Thought Budget Effects in Function-Calling Language Agents

Xuan Qi

cs.CLApr 2, 2026

Brief Is Better: Non-Monotonic Chain-of-Thought Budget Effects in Function-Calling Language Agents

Authors: Xuan Qi

Organizations: IIIS, Tsinghua University

Abstract

How much should a language agent think before taking action? Chain-of-thought (CoT) reasoning is widely assumed to improve agent performance, but the relationship between reasoning length and accuracy in structured tool-use settings remains poorly understood. We present a systematic study of CoT budget effects on function-calling agents, sweeping six token budgets (0--512) across 200 tasks from the Berkeley Function Calling Leaderboard v3 Multiple benchmark. Our central finding is a striking non-monotonic pattern on Qwen2.5-1.5B-Instruct: brief reasoning (32 tokens) dramatically improves accuracy by 45% relative over direct answers, from 44.0% to 64.0%, while extended reasoning (256 tokens) degrades performance well below the no-CoT baseline, to 25.0% (McNemar p < 0.001). A three-way error decomposition reveals the mechanism. At d = 0, 30.5% of tasks fail because the model selects the wrong function from the candidate set; brief CoT reduces this to 1.5%, effectively acting as a function-routing step, while long CoT reverses the gain, yielding 28.0% wrong selections and 18.0% hallucinated functions at d = 256. Oracle analysis shows that 88.6% of solvable tasks require at most 32 reasoning tokens, with an average of 27.6 tokens, and a finer-grained sweep indicates that the true optimum lies at 8--16 tokens. Motivated by this routing effect, we propose Function-Routing CoT (FR-CoT), a structured brief-CoT method that templates the reasoning phase as "Function: [name] / Key args: [...]," forcing commitment to a valid function name at the start of reasoning. FR-CoT achieves accuracy statistically equivalent to free-form d = 32 CoT while reducing function hallucination to 0.0%, providing a structural reliability guarantee without budget tuning.

Figures & tables

Budget	Accuracy	95% CI	vs. $d{=}0$	vs. $d{=}32$	Validity fail.	Content err.
$d=0$	44.0%	[37.0, 51.0]	—	$-20.0$ pp	6.0%	50.0%
$d=32$	64.0%	[57.5, 70.5]	$+20.0$ pp	—	2.5%	33.5%
$d=64$	58.0%	[51.0, 65.0]	$+14.0$ pp	$-6.0$ pp	0.5%	41.5%
$d=128$	51.5%	[44.5, 58.5]	$+7.5$ pp	$-12.5$ pp	9.0%	39.5%
$d=256$	25.0%	[19.0, 31.0]	$-19.0$ pp	$-39.0$ pp	19.5%	55.5%
$d=512$	22.5%	[17.0, 28.5]	$-21.5$ pp	$-41.5$ pp	21.5%	56.0%

Table 1: Accuracy by CoT token budget on BFCL v3 Multiple (Qwen2.5-1.5B-Instruct,

n{=}200

tasks). 95% CIs from 10,000 bootstrap resamples. Validity failure rate : format failure

\cup

hallucination. Content error rate : valid JSON but incorrect function or arguments.

Panel of Figure 1 — Figure 1: Non-monotonic CoT effect confirmed on both Qwen2.5-1.5B and 7B-Instruct. Both models peak at $d{=}32$ and collapse at $d{\geq}128$ . The 7B model achieves higher peak accuracy (82.5% vs. 64.0%) but suffers a more severe collapse at long budgets (18.0% vs. 25.0% at $d{=}256$ ). All pairwise comparisons ( $d{=}32$ vs. $d{=}256$ ) are McNemar $p{<}0.001$ . Validity failure rate (1.5B only) spikes to 19.5% at $d{=}256$ , driven by function hallucination.

$d^{*}$	Tasks	%	Cumulative %
0 (no CoT)	88	52.7%	52.7%
32 tokens	60	35.9%	88.6%
64 tokens	10	6.0%	94.6%
128 tokens	6	3.6%	98.2%
256 tokens	1	0.6%	98.8%
512 tokens	2	1.2%	100.0%

Table 2: Distribution of oracle optimal budget

d^{*}

across 167 solvable tasks. 16.5% of all 200 tasks are unsolvable (correct at no budget).

Strategy	Accuracy	Gap to oracle	Tokens/task	FLOPs ratio
No CoT ( $d{=}0$ )	44.0%	$-39.5$ pp	0	1.0 $\times$
Fixed $d{=}32$	64.0%	$-19.5$ pp	32	1.1 $\times$
Fixed $d{=}64$	58.0%	$-25.5$ pp	64	1.2 $\times$
Fixed $d{=}256$	25.0%	$-58.5$ pp	256	2.0 $\times$
Fixed $d{=}512$	22.5%	$-61.0$ pp	512	3.0 $\times$
Oracle 2-budget $\{32,128\}$	76.0%	$-7.5$ pp	$\leq 128$	$\leq 1.5\times$

Table 3: Strategy comparison on BFCL v3 Multiple (

n{=}200

tasks). Compute is measured in reasoning tokens per task.

Budget	Correct	Halluc. fn	Wrong valid fn	Wrong args	No JSON
$d{=}0$	44.0%	3.0%	30.5%	19.5%	3.0%
$d{=}32$	64.0%	2.5%	1.5%	32.0%	0.0%
$d{=}64$	58.0%	0.5%	3.0%	38.5%	0.0%
$d{=}128$	51.5%	8.5%	10.5%	29.0%	0.5%
$d{=}256$	25.0%	18.0%	28.0%	27.5%	1.5%
$d{=}512$	22.5%	20.0%	32.5%	23.5%	1.5%

Table 4: Error breakdown as % of all 200 tasks (

n{=}200

, Qwen2.5-1.5B-Instruct). “Halluc. fn” = valid JSON with a function name outside the candidate set; “Wrong valid fn” = a valid candidate selected, but not the ground-truth function; “Wrong args” = correct function selected, but argument values incorrect; “No JSON” = no parseable JSON function call in output. Rows sum to 100%. Bootstrap 95% CIs for Halluc. fn and Wrong valid fn are reported in the text; all key budget comparisons are statistically significant (non-overlapping CIs).

Method	Budget	1.5B Acc.	7B Acc.	Routing valid%	Halluc.%
Free-form $d{=}0$ (no CoT)	0	44.0%	40.5%	—	3.0%
Free-form $d{=}16$	16	71.5%	84.5%	—	$<$ 1%
Free-form $d{=}32$ (default)	32	64.0%	82.5%	—	2.5%
FR-CoT	$\leq$ 30	64.0%	83.0%	99.5%/98.5%	0.0%

Table 5: FR-CoT vs. free-form CoT baselines on BFCL v3 Multiple, matched comparison on the same 200 tasks with identical prompts. “Routing valid%” = fraction of FR-CoT routing traces naming a valid candidate function; “Halluc.%” = fraction of outputs naming a non-candidate function in the final JSON. 1.5B:

n{=}200

(complete); 7B:

n{=}200

(complete). All experiments use greedy decoding.

Method	Budget	1.5B Acc.	7B Acc.	Wrong valid
Free-form	$d{=}0$	44.0%	40.5%	30.5%/16.0%
Free-form	$d{=}32$	64.0%	82.5%	1.5%/ $<$ 1%
Free-form	$d{=}256$	25.0%	18.0%	28.0%/ $\sim$ 26%
Constrained	$d{=}0$	47.5% [40.5,55.0]	58.5% [51.5,65.5]	24.5%/9.5%
Constrained	$d{=}32$	62.5% [55.5,69.0]	63.5% [57.0,70.0]	1.5%/0.0%
Constrained	$d{=}256$	58.0% [51.0,65.0]	33.5% [27.0,40.0]	0.0%/0.0%

Table 6: FR-CoT vs. constrained-decoding baseline on 200 BFCL v3 Multiple tasks (greedy decoding, bfloat16). Constrained decoding forces the final function_name token to be a valid candidate via log-prob scoring; hallucination is 0% by construction. “Wrong valid” = correct JSON format but wrong-but-valid function selected. Brackets show 95% bootstrap CIs.

Phase 1: 32-token reasoning	Phase 2: forced fn	Phase 2: arg. output
...capital of Brazil... {"function_name": "country	country_info .capital	Human: Given the following context... (hallucinated)
...triangle side lengths... {"function_name": "triangle	triangle_prop erties.get	‘‘‘\nTo complete the function... (malformed)
...capacitance formula... {"function_name": "cap	capacitance_ calculator.calc	Human: Given the following... (hallucinated)

Table 7: 7B Qwen2.5-Instruct constrained

d{=}32

traces showing partial-JSON contamination in the reasoning window. The 32-token reasoning window already contains ‘‘{"function_name’’ before phase 2 forces the function name. Despite the doubled context, phase 2 correctly selects the function (0% wrong-valid); failure occurs during argument generation. Traces truncated for display.

Budget	1.5B Acc.	1.5B Halluc.	7B Acc.	7B Halluc.
$d{=}0$	44.0%	3.0%	40.5%	$<$ 1%
$d{=}32$	64.0%	2.5%	82.5%	$<$ 1%
$d{=}64$	58.0%	0.5%	78.5%	$<$ 1%
$d{=}128$	51.5%	8.5%	36.0%	— †
$d{=}256$	25.0%	18.0%	18.0%	— †
$d{=}512$	22.5%	20.0%	14.0%	— †

Table 8: Budget sweep comparison: Qwen2.5-1.5B vs. Qwen2.5-7B (

n{=}200

tasks each). Both models exhibit the non-monotonic CoT budget effect. Halluc. = fraction with hallucinated function name (not in candidate set). † Output storage truncated to 300 chars at

d{\geq}128

; full reasoning traces exceed the storage limit at these budgets so the error-category breakdown (hallucination rate) is not recoverable for 7B. Accuracy figures are unaffected : correctness was evaluated by validate_against_ground_truth on the full generated text before any truncation. All differences

d{=}32

vs.

d{=}256

are McNemar

p{<}0.001

Budget	Phi-3-mini Acc.	Halluc.	EOS rate	Mean tokens
$d{=}0$	29.5%	$<$ 1%	—	—
$d{=}32$	86.0%	0.0%	0.0%	32.0
$d{=}64$	83.5%	$<$ 1%	0.5%	64.0
$d{=}128$	74.5%	— †	24.0%	123.8
$d{=}256$	66.5%	— †	68.0%	184.3
$d{=}512$	57.5%	— †	68.5%	285.8

Table 9: Budget sweep for Phi-3-mini-4k-instruct (3.8B, Phi3ForCausalLM, Microsoft;

n{=}200

tasks). Partial replication : the peak at

d{=}32

is confirmed, but Phi-3 does not drop below its no-CoT baseline at any budget (see text). EOS rate = fraction of reasoning phases that terminated early (model generated EOS token before budget exhausted); Phi-3 uses <|endoftext|> as EOS. Mean tokens = average tokens generated in the reasoning phase. † Accuracy evaluated on full output before storage; error-category breakdown not recoverable from truncated storage at

d{\geq}128

	Qwen2.5-1.5B		Qwen2.5-7B
Budget	EOS rate	Mean tokens	EOS rate	Mean tokens
$d{=}32$	0.0%	32.0/32	0.0%	32.0/32
$d{=}64$	0.0%	64.0/64	0.0%	64.0/64
$d{=}128$	0.0%	128.0/128	0.0%	128.0/128
$d{=}256$	0.0%	256.0/256	0.0%	256.0/256
$d{=}512$	0.0%	512.0/512	0.0%	512.0/512

Table 10: Natural EOS rate during the reasoning phase. EOS rate

=0\%

means the model always fills its entire reasoning budget without generating EOS. Mean generated tokens equals the budget in all cases, confirming complete budget utilization.

Mann-Whitney $p$ (helps vs. hurts)	0.092 (borderline)
CoT-32 effect	Count	Mean $H_{0}$	Std $H_{0}$
Helps (wrong $\to$ right)	60	0.506	0.383
No change	120	—	—
Hurts (right $\to$ wrong)	20	0.634	0.413

Table 11: Pre-reasoning entropy

H_{0}

conditioned on CoT effect (

n{=}200

tasks). The directional pattern (lower

H_{0}

when CoT helps) is consistent but not statistically significant at

\alpha{=}0.05

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Budget	Standard	Format-ctrl
$d=0$	41.0%	48.0%
$d=32$	63.0%	75.0%
$d=64$	59.0%	65.0%
$d=128$	44.0%	46.0%
$d=256$	22.0%	15.0%

Appendix

Table 12: Standard vs. format-control condition accuracy (

n{=}100

tasks). “Format-ctrl” repeats the JSON format reminder at constant distance from generation start, regardless of CoT length. Non-monotonicity (peak at

d{=}32

) is preserved in both conditions.

Budget	Accuracy
$d=0$	41.0%
$d=8$	68.0%
$d=16$	69.0%
$d=24$	61.0%
$d=32$	63.0%
$d=48$	60.0%

Appendix

Table 13: Fine-grained budget sweep accuracy (

n{=}100

tasks). The optimal window is 8–16 tokens, outperforming

d{=}32

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 356 papers

The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits
May 8, 2026Wenhua Nie, Junlin Liu, Jianan Wu +5CoT Evaluation CoT Reasoning
National Taiwan University
AgentWeave: Routing Before Reasoning for Efficient Function Calling in Tool-Rich Language Models
Aug 24, 2026Saurav Singla, Aarav Singla, Advik Gupta +1Tool Selection for Language Model Agents LLM Routing
Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs
Feb 28, 2026Jie Cao, Tianwei Lin, Juekai Lin +7Efficient Language Model Reasoning CoT Reasoning
Zhejiang University · Tencent

May 8, 2026J/K move · Enter open · S save

The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits

Wenhua Nie, Junlin Liu, Jianan Wu, Zijie Meng+4

National Taiwan University

CoT Evaluation CoT Reasoning Efficient Language Model Reasoning Reasoning Trace Analysis

Open paper AskPDF arXiv

Chain-of-thought reasoning is often treated as a monotone way to improve language-model accuracy by letting a model think longer. We identify a countervailing effect, the coupling tax: when reasoning traces and final answers share one output-token budget, long traces can crowd out the answer they are meant to support. Across GSM8K, MATH-500, and five BIG-Bench Hard tasks with Qwen3 models at three scales, non-thinking mode matches or outperforms thinking mode on GSM8K and MATH-500 at every budget up to 2048 tokens, while harder tasks shift the crossover to larger budgets. We derive a truncation-waste decomposition,

\mathrm{Acc}_{\mathrm{think}}(b)=α_c F_L(b)+α_t(1-F_L(b))

, that predicts this crossover from chain-length and accuracy statistics and explains inverse scaling within the Qwen family. A DeepSeek-R1-Distill-Llama-8B replication shows the same pattern under a different thinking interface. As a mitigation, split-budget generation decouples reasoning and answer budgets; on full MATH-500, IRIS reaches 74.0% accuracy, a strengthened extraction variant reaches 78.8%, and a fixed non-oracle SC+IRIS gate reaches 83.6%. The results show that test-time reasoning should be evaluated as a budget-allocation problem, not only as a question of whether longer traces are available.

Abstract

Efficient Language Model Reasoning

Explore similar work

The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits

AgentWeave: Routing Before Reasoning for Efficient Function Calling in Tool-Rich Language Models

Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs