ThinkJev: One Thought, Every Budget for Calibrated Typed Decisions
Organizations: Fujian Bosssoft Co., Ltd., Fuzhou, China
Abstract
Many uses of language models end in a typed decision: a choice among a closed set of options, whose probabilities code can act on. Reading those probabilities from one forward pass is fast but often wrong on decisions that take several steps, such as arithmetic, counting, and date comparison. Letting the model think first fixes most of these errors, but costs seconds per decision instead of milliseconds. To pay for thinking only where it helps, ThinkJev builds a fast and a slow system from one open hybrid-thinking model that can answer directly or after a thought. System 1 reads the model's probabilities for the option letters when it answers directly; System 2 reads the same letters after the model has thought on the same cached prompt, so escalating repeats no work and both systems give probabilities over the same options. Because the letters can be read after any number of thought tokens, one sampled thought also gives the answer under every thinking budget. Without training, ThinkJev lets System 1 answer alone only where a statistical test on labeled dev data guarantees that its accepted answers stay under a target error rate. At a 5% target, Qwen3.5-27B answers 24.5% of 8359 test decisions this way with a realized error of 3.9% (Qwen3.5-4B and 9B: 6.5% and 0.0%). For the rest, stopping each thought as soon as its answer is confident reaches 81.6% accuracy, against 82.7% for always thinking, with 72% of the thought tokens. Choosing when to think from System 1's confidence alone saves less, because the decisions System 1 is least sure of are often not the ones thinking fixes. On the public items of JevBench, an independent benchmark of typed-decision services, Qwen3.5-27B thinks on 29.4% and is not distinguishable in accuracy from the two thinking APIs listed (exploratory). Code and data generators are available at https://github.com/ding-xiaolu/ThinkJev.
Figures & tables
| Work | What it does | What ThinkJev adds or changes |
|---|---|---|
| Jev [ 27 ] | Closed decision model with calibrated probabilities | Open weights, no training, any hybrid-thinking model; a System 2 on the same answer slots |
| SemIf, AnyJev [ 26 , 15 ] | System 1 readout from any LLM; debiasing, temperatures, hidden-state heads | System 2 on the shared prefix, gating, anytime readout, risk guarantee, typed-decision server |
| Adaptive Thinking [ 9 ] | Adapter on prefill hidden states sets the budget before thinking | Not run; our probe router, of the same family, is compared with the gate (Section 5.7 ) |
| DART [ 8 ] | Training-free; agreement of two sampled no-think drafts | One forward gives the calibrated option distribution; closed-set decisions only |
| DEER [ 34 ] | Early exit when a trial answer is confident | Trial answer is a slot read; the curve’s zero point is System 1 |
| When2Think, AdapThink, CAT [ 22 , 33 , 6 ] | Post-training so the model allocates its own reasoning | No weight updates |
| Block | Source | Dev | Test | Language | License |
| A | Escalation (CLINC) | 210 | 290 | en | CC-BY-3.0 |
| A | Every judgments | 70 | 84 | en | unstated |
| A | Intent routing (BANKING77) | 313 | 486 | en | CC-BY-4.0 |
| A | Intent routing (CLINC) | 242 | 358 | en | CC-BY-3.0 |
| A | NLI (WANLI) | 97 | 159 | en | CC-BY-4.0 |
| A | SemIf authored | 60 | 84 | en | MIT |
| Gate | Anytime | ||||
|---|---|---|---|---|---|
| Tolerance | Accuracy | Escalated | Thinking | Accuracy | Thinking |
| 0 points | 82.7% | 89.5% | 97% | 82.7% | 97% |
| 1 point | 81.6% | 62.4% | 81% | 81.6% | 72% |
| 2 points | 80.5% | 53.6% | 72% | 80.6% | 63% |
| Block | Test decisions | Always S1 | Always S2 | Gate | Escalated |
|---|---|---|---|---|---|
| A: routine decisions | 2626 | 77.6% | 80.8% | 80.7% | 39.5% |
| B: public reasoning | 2487 | 58.9% | 78.4% | 78.2% | 67.6% |
| C: multi-step decisions | 3246 | 52.3% | 87.5% | 84.9% | 77.1% |
| All | 8359 | 62.2% | 82.7% | 81.6% | 62.4% |
| Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | Qwen3.8-27B | |||||||||||||
| Benchmark | S1 | S2 | Gate | Esc. | S1 | S2 | Gate | Esc. | S1 | S2 | Gate | Esc. | S1 | S2 | Gate | Esc. |
| BANKING77 (intent, A) | 79.2 | 81.1 | 80.5 | 21.0 | 78.6 | 82.9 | 82.7 | 31.5 | 85.0 | 86.4 | 86.4 | 18.9 | 85.6 | 87.2 | 87.2 | 14.2 |
| CLINC150 (intent, A) | 89.9 | 95.5 | 95.3 | 15.9 | 91.3 | 97.8 | 97.8 | 27.4 | 95.5 | 97.5 | 97.2 | 9.8 | 95.0 | 96.9 | 96.4 | 8.7 |
| WANLI (NLI, A) | 64.2 | 73.6 | 69.8 | 35.2 | 67.3 | 75.5 | 74.8 | 54.7 | 75.5 | 81.1 | 79.2 | 32.7 | 74.2 | 77.4 | 76.7 | 24.5 |
| BIG-Bench Hard (B) | 55.3 | 89.8 | 89.0 | 72.6 | 59.4 | 92.7 | 92.6 | 74.7 | 70.6 | 94.8 | 94.4 | 53.9 | 69.5 | 93.1 | 92.9 | 56.5 |
| MMLU-Pro (B) | 37.8 | 73.6 | 73.1 | 83.2 | 44.5 | 76.2 | 75.9 | 86.7 | 56.9 | 82.9 | 82.8 | 71.9 | 51.3 | 78.7 | 78.3 | 67.1 |
| Target | Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | Qwen3.8-27B | |
|---|---|---|---|---|---|
| 1% | answered by S1 | 0.0% | 0.0% | 0.0% | 6.9% |
| realized risk | – | – | – | 0.5% [0.1, 1.5]% | |
| 2% | answered by S1 | 0.0% | 0.0% | 3.5% | 10.5% |
| realized risk | – | – | 0.0% [0.0, 1.2]% | 1.0% [0.5, 1.9]% | |
| 5% | answered by S1 | 6.5% | 0.0% | 24.5% | 23.2% |
| realized risk | 1.1% [0.4, 2.4]% | – | 3.9% [3.1, 4.8]% | 4.2% [3.3, 5.2]% |
| Quantity | Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | Qwen3.8-27B |
| Test decisions | 8359 | 8359 | 8359 | 8359 |
| Always S1: accuracy | 50.9% | 54.1% | 62.2% | 60.2% |
| Always S2: accuracy | 77.4% | 80.0% | 82.7% | 81.8% |
| Always S2: thought tokens per decision | 2126 | 2088 | 1856 | 722 |
| Oracle accuracy | 81.5% | 83.5% | 85.8% | 86.2% |
| Gate: accuracy | 76.5% | 79.1% | 81.6% | 80.9% |
| Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | Qwen3.8-27B | |
| Gate: normalized area | 0.360 | 0.367 | 0.436 | 0.421 |
| Gate zero-shot router | +0.011 | 0.063 | 0.054 | 0.067 |
| [ 0.017, +0.037] | [ 0.091, 0.038] | [ 0.080, 0.027] | [ 0.092, 0.041] | |
| Gate probe router | +0.043 | 0.014 | +0.040 | +0.021 |
| [+0.018, +0.068] | [ 0.039, +0.009] | [+0.014, +0.068] | [ 0.005, +0.050] | |
| Gate -gram router | +0.161 | +0.159 | +0.121 | +0.110 |
| Policy | Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | Qwen3.8-27B |
| S1 only | 80.1% / 62.2% | 82.3% / 64.9% | 89.6% / 81.1% | 88.7% / 77.5% |
| Gate | 91.3% / 83.8% | 94.4% / 88.3% | 97.0% / 93.7% | 97.8% / 95.5% |
| Anytime | 91.3% / 83.8% | 93.5% / 87.4% | 97.0% / 93.7% | 96.5% / 92.8% |
| Always S2 | 93.1% / 85.6% | 94.8% / 89.2% | 97.8% / 95.5% | 99.6% / 99.1% |
| Escalated by the gate | 32.9% | 42.0% | 29.4% | 24.7% |
| Listed on JevBench (same items, from its per-task file) | ||||
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | State and question (abridged) | Answer |
|---|---|---|
| Dates | Notice given Thursday, June 24, 2027; 20 calendar days of notice, counted from the next day. On what date does the tenancy end? | July 14, 2027 |
| Counting | An eight-row order log (city, status, channel, amount). How many orders have status ‘refunded’? | 1 order |
| Chained rules | A purchasing policy with amount bands and a one-step reduction for pre-approved vendors, except hardware; a $3,630 repair from a vendor not on the list. What approval is needed? | No approval |
| Negation | “It is false that crate A was not sealed.” Was crate A sealed? | Yes |
| Arithmetic | 12 full trays of 17 rolls plus 7 extra rolls; 110 sold. How many rolls are left? | 101 rolls |
| Logic grid | Three friends, three pets, three drinks, five clues. Who owns the turtle? | Jonas |
| Block | Source | Lang. | Level | Rows | S1 | S2 | Gate | Escalated |
|---|---|---|---|---|---|---|---|---|
| A | Escalation (CLINC) | en | – | 290 | 60.0% | 72.4% | 71.0% | 43.1% |
| A | Every judgments | en | – | 84 | 95.2% | 95.2% | 95.2% | 7.1% |
| A | Intent routing (BANKING77) | en | – | 486 | 85.0% | 86.4% | 86.4% | 18.9% |
| A | Intent routing (CLINC) | en | – | 358 | 95.5% | 97.5% | 97.2% | 9.8% |
| A | NLI (WANLI) | en | – | 159 | 75.5% | 81.1% | 79.2% | 32.7% |
| A | SemIf authored | en | – | 84 | 92.9% | 94.0% | 95.2% | 10.7% |
| Block | Source | Lang. | Level | Rows | S1 | S2 | Gate | Escalated |
|---|---|---|---|---|---|---|---|---|
| A | Escalation (CLINC) | en | – | 290 | 69.0% | 84.1% | 77.2% | 34.8% |
| A | Every judgments | en | – | 84 | 96.4% | 95.2% | 94.0% | 4.8% |
| A | Intent routing (BANKING77) | en | – | 486 | 79.2% | 81.1% | 80.5% | 21.0% |
| A | Intent routing (CLINC) | en | – | 358 | 89.9% | 95.5% | 95.3% | 15.9% |
| A | NLI (WANLI) | en | – | 159 | 64.2% | 73.6% | 69.8% | 35.2% |
| A | SemIf authored | en | – | 84 | 82.1% | 95.2% | 94.0% | 23.8% |
| Block | Source | Lang. | Level | Rows | S1 | S2 | Gate | Escalated |
|---|---|---|---|---|---|---|---|---|
| A | Escalation (CLINC) | en | – | 290 | 53.4% | 84.1% | 72.1% | 36.6% |
| A | Every judgments | en | – | 84 | 94.0% | 95.2% | 95.2% | 8.3% |
| A | Intent routing (BANKING77) | en | – | 486 | 78.6% | 82.9% | 82.7% | 31.5% |
| A | Intent routing (CLINC) | en | – | 358 | 91.3% | 97.8% | 97.8% | 27.4% |
| A | NLI (WANLI) | en | – | 159 | 67.3% | 75.5% | 74.8% | 54.7% |
| A | SemIf authored | en | – | 84 | 92.9% | 94.0% | 94.0% | 32.1% |
| Block | Source | Lang. | Level | Rows | S1 | S2 | Gate | Escalated |
|---|---|---|---|---|---|---|---|---|
| A | Escalation (CLINC) | en | – | 290 | 65.2% | 80.7% | 80.7% | 44.5% |
| A | Every judgments | en | – | 84 | 96.4% | 97.6% | 97.6% | 7.1% |
| A | Intent routing (BANKING77) | en | – | 486 | 85.6% | 87.2% | 87.2% | 14.2% |
| A | Intent routing (CLINC) | en | – | 358 | 95.0% | 96.9% | 96.4% | 8.7% |
| A | NLI (WANLI) | en | – | 159 | 74.2% | 77.4% | 76.7% | 24.5% |
| A | SemIf authored | en | – | 84 | 94.0% | 97.6% | 97.6% | 13.1% |
LLM Decision-Making
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
See all 246 papers