Thinking Costs Tokens: When More Structure is Worth the Price
Organizations: Royal Bank of Canada
Abstract
Adding inference structure to a language model lets it search, verify, and revise, but these actions consume the very budget they are supposed to use well. In this paper, we investigate whether there exists a token-budget threshold, below which the overhead of planning and verification hurts performance and above which it helps. We evaluate two systems on FinQA and TAT-QA financial reasoning tasks, using GPT-5.4 mini across 14 budget tiers ranging from 250 to 42,000 output-equivalent tokens. The first system is a monolith, which is a single LLM call. The second is a verified search architecture that adds planning, label-blind checking, and repair capabilities. We run 1,000 cases for a total of 28,000 completed cells. Both systems score 0% at the two lowest tiers, where neither can fit a complete prompt. At 1,000 tokens, the monolith reaches 18% accuracy while verified search scores near 0%, since the planning overhead leaves no room for an answer. From 1,500 tokens onward, verified search surpasses the monolith and maintains a consistent advantage, reaching approximately 44% at the highest tiers while the monolith reaches approximately 40%. The crossover occurs between 1,000 and 1,500 output-equivalent tokens, confirmed by a strict intersection-union test ( at both endpoints).
Figures & tables
| System | Mechanism |
|---|---|
| Monolith | Issues the question as a direct query, retrieves up to the tier limit, making exactly one answer call capped at 256 output tokens. It never checks or revises. |
| Verified search | Runs a mandatory planner (capped at 128 tokens), performs deterministic query union over planned queries, generates sequential candidates (each capped at 256 tokens), applies label-blind checking, and accepts the first passing candidate. At higher tiers, it may attempt repairs using checker findings only, with the repair limit scaling alongside the token cap (see Table 2 ). |
| Tier | Token cap | Retrieval | Queries | Candidates | Repairs |
|---|---|---|---|---|---|
| t250 | 250 | 1 | 1 | 1 | 0 |
| t500 | 500 | 1 | 1 | 1 | 0 |
| t1000 | 1,000 | 2 | 1 | 1 | 0 |
| t1500 | 1,500 | 2 | 1 | 1 | 0 |
| t2000 | 2,000 | 3 | 1 | 1 | 0 |
| t3000 | 3,000 | 3 | 2 | 2 | 1 |
Efficient Language Model Reasoning
Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.
See all 369 papers