cs.AIOct 27, 2026
SaveThinkJev: One Thought, Every Budget for Calibrated Typed Decisions
Organizations: Fujian Bosssoft Co., Ltd., Fuzhou, China
Abstract
Many uses of language models end in a typed decision: a choice among a closed set of options, whose probabilities code can act on. Reading those probabilities from one forward pass is fast but often wrong on decisions that take several steps, such as arithmetic, counting, and date comparison. Letting the model think first fixes most of these errors, but costs seconds per decision instead of milliseconds. To pay for thinking only where it helps, ThinkJev builds a fast and a slow system from one open hybrid-thinking model that can answer directly or after a thought. System 1 reads the model's probabilities for the option letters when it answers directly; System 2 reads the same letters after the model has thought on the same cached prompt, so escalating repeats no work and both systems give probabilities over the same options. Because the letters can be read after any number of thought tokens, one sampled thought also gives the answer under every thinking budget. Without training, ThinkJev lets System 1 answer alone only where a statistical test on labeled dev data guarantees that its accepted answers stay under a target error rate. At a 5% target, Qwen3.5-27B answers 24.5% of 8359 test decisions this way with a realized error of 3.9% (Qwen3.5-4B and 9B: 6.5% and 0.0%). For the rest, stopping each thought as soon as its answer is confident reaches 81.6% accuracy, against 82.7% for always thinking, with 72% of the thought tokens. Choosing when to think from System 1's confidence alone saves less, because the decisions System 1 is least sure of are often not the ones thinking fixes. On the public items of JevBench, an independent benchmark of typed-decision services, Qwen3.5-27B thinks on 29.4% and is not distinguishable in accuracy from the two thinking APIs listed (exploratory). Code and data generators are available at https://github.com/ding-xiaolu/ThinkJev.
Figures & tables
Figure 1: ThinkJev on one test decision, Qwen3.5-27B. Left: the part of the prompt that the two systems share is processed once. Top: System 1, the direct answer, gives calibrated option probabilities; its confidence is below the threshold chosen on dev, so the model thinks. Bottom: one sampled thought, with the same answer slots read after its first tokens (each read calibrated with its own temperature; checkpoints not drawn to scale). Each read is the answer under a thinking budget of , so the policy can stop at the first confident one. Right: accuracy at each budget over all test decisions, with System 1 at . The example was chosen for illustration and is not typical; all values are generated from the released outputs ( example.tex ).
| Work | What it does | What ThinkJev adds or changes |
|---|---|---|
| Jev [ 27 ] | Closed decision model with calibrated probabilities | Open weights, no training, any hybrid-thinking model; a System 2 on the same answer slots |
| SemIf, AnyJev [ 26 , 15 ] | System 1 readout from any LLM; debiasing, temperatures, hidden-state heads | System 2 on the shared prefix, gating, anytime readout, risk guarantee, typed-decision server |
| Adaptive Thinking [ 9 ] | Adapter on prefill hidden states sets the budget before thinking | Not run; our probe router, of the same family, is compared with the gate (Section 5.7 ) |
| DART [ 8 ] | Training-free; agreement of two sampled no-think drafts | One forward gives the calibrated option distribution; closed-set decisions only |
| DEER [ 34 ] | Early exit when a trial answer is confident | Trial answer is a slot read; the curve’s zero point is System 1 |
| When2Think, AdapThink, CAT [ 22 , 33 , 6 ] | Post-training so the model allocates its own reasoning | No weight updates |
Table 1: Closest related work. Descriptions follow each work’s own abstract or documentation.
Figure 2: ThinkJev on one decision: the same answer slots read twice after one processed prompt prefix. System 1 is read on a copy of the key-value cache; if its calibrated confidence reaches , it answers. Otherwise the model thinks, continuing on the original cache, and the same slots are read after the first thought tokens (checkpoints not to scale). The three policies of Sections 3.5 and 3.6 (direct answers under an error guarantee, the gate, and the anytime policy) differ only in how is chosen and where an escalated decision stops thinking.
| Block | Source | Dev | Test | Language | License |
| A | Escalation (CLINC) | 210 | 290 | en | CC-BY-3.0 |
| A | Every judgments | 70 | 84 | en | unstated |
| A | Intent routing (BANKING77) | 313 | 486 | en | CC-BY-4.0 |
| A | Intent routing (CLINC) | 242 | 358 | en | CC-BY-3.0 |
| A | NLI (WANLI) | 97 | 159 | en | CC-BY-4.0 |
| A | SemIf authored | 60 | 84 | en | MIT |
Table 2: Decisions per source and split, from the released split file ( benchmarks/manifests/release-split.jsonl ); licenses as stated upstream ( THIRD_PARTY.md ). Block A: routine decisions; B: public reasoning; C: generated multi-step decisions and chess. Every source is used by all four models.
| Gate | Anytime | ||||
|---|---|---|---|---|---|
| Tolerance | Accuracy | Escalated | Thinking | Accuracy | Thinking |
| 0 points | 82.7% | 89.5% | 97% | 82.7% | 97% |
| 1 point | 81.6% | 62.4% | 81% | 81.6% | 72% |
| 2 points | 80.5% | 53.6% | 72% | 80.6% | 63% |
Table 3: Qwen3.5-27B, test: the gate and the anytime policy at each tolerance . The threshold is chosen on dev to stay at most below always thinking, then applied unchanged to test. “Thinking” is thought tokens per decision as a share of always thinking’s. The 1-point row is the setting used everywhere else. Source: dev_selected_operating_points (gate), anytime_test.dev_selected_operating_points (anytime).
Figure 3: Test accuracy against thought tokens per decision for Qwen3.5-27B. Each curve sweeps the threshold of one policy: the gate, the anytime policy, and the probe router. The oracle escalates exactly the decisions that System 1 gets wrong and System 2 gets right. Always answering directly (always-S1) and always thinking (always-S2) are the endpoints; the marked point is the gate’s threshold chosen on dev and applied unchanged to test. Source fields: cost_curves_test , anytime_test.curves , anytime_test.dev_selected_operating_points .
| Block | Test decisions | Always S1 | Always S2 | Gate | Escalated |
|---|---|---|---|---|---|
| A: routine decisions | 2626 | 77.6% | 80.8% | 80.7% | 39.5% |
| B: public reasoning | 2487 | 58.9% | 78.4% | 78.2% | 67.6% |
| C: multi-step decisions | 3246 | 52.3% | 87.5% | 84.9% | 77.1% |
| All | 8359 | 62.2% | 82.7% | 81.6% | 62.4% |
Table 4: Qwen3.5-27B test accuracy by block, and the share of decisions the gate escalates. The gate’s threshold is the one in per_family_test_gate (at most one point below always thinking on dev, with the most decisions answered directly); values are weighted by the number of decisions in each entry of per_family_test . The numbers in the text use the threshold with the fewest thought tokens on dev ( anytime_test.dev_selected_operating_points ), so the “All” row can differ from them slightly. Appendix C breaks this down per source for every model.
Figure 4: Per task group: accuracy of System 1, System 2, and the gate (left), and the share of decisions the gate sent to System 2 (right), Qwen3.5-27B, test. Source: per_family_test .
Figure 5: Escalation rate and System 1 accuracy by difficulty level, one panel per generated family, Qwen3.5-27B, test. Source: per_family_test .
| Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | Qwen3.8-27B | |||||||||||||
| Benchmark | S1 | S2 | Gate | Esc. | S1 | S2 | Gate | Esc. | S1 | S2 | Gate | Esc. | S1 | S2 | Gate | Esc. |
| BANKING77 (intent, A) | 79.2 | 81.1 | 80.5 | 21.0 | 78.6 | 82.9 | 82.7 | 31.5 | 85.0 | 86.4 | 86.4 | 18.9 | 85.6 | 87.2 | 87.2 | 14.2 |
| CLINC150 (intent, A) | 89.9 | 95.5 | 95.3 | 15.9 | 91.3 | 97.8 | 97.8 | 27.4 | 95.5 | 97.5 | 97.2 | 9.8 | 95.0 | 96.9 | 96.4 | 8.7 |
| WANLI (NLI, A) | 64.2 | 73.6 | 69.8 | 35.2 | 67.3 | 75.5 | 74.8 | 54.7 | 75.5 | 81.1 | 79.2 | 32.7 | 74.2 | 77.4 | 76.7 | 24.5 |
| BIG-Bench Hard (B) | 55.3 | 89.8 | 89.0 | 72.6 | 59.4 | 92.7 | 92.6 | 74.7 | 70.6 | 94.8 | 94.4 | 53.9 | 69.5 | 93.1 | 92.9 | 56.5 |
| MMLU-Pro (B) | 37.8 | 73.6 | 73.1 | 83.2 | 44.5 | 76.2 | 75.9 | 86.7 | 56.9 | 82.9 | 82.8 | 71.9 | 51.3 | 78.7 | 78.3 | 67.1 |
Table 5: Public benchmarks, test split: accuracy (%) of System 1, System 2, and the gate, and the share of decisions the gate escalates (%), for every model. The gate uses the threshold in per_family_test_gate , as in Table 4 ; scores follow our protocol (one option letter, a 4,096-token budget) and are not comparable with published scores. Source: per_family_test .
Figure 6: System 2 accuracy and expected calibration error under each thinking budget , read from one sampled thought per decision, Qwen3.5-27B, test. Source: anytime_test.per_checkpoint , anytime_test.full_thought .
| Target | Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | Qwen3.8-27B | |
|---|---|---|---|---|---|
| 1% | answered by S1 | 0.0% | 0.0% | 0.0% | 6.9% |
| realized risk | – | – | – | 0.5% [0.1, 1.5]% | |
| 2% | answered by S1 | 0.0% | 0.0% | 3.5% | 10.5% |
| realized risk | – | – | 0.0% [0.0, 1.2]% | 1.0% [0.5, 1.9]% | |
| 5% | answered by S1 | 6.5% | 0.0% | 24.5% | 23.2% |
| realized risk | 1.1% [0.4, 2.4]% | – | 3.9% [3.1, 4.8]% | 4.2% [3.3, 5.2]% |
Table 6: System 1 answers accepted by the Learn-then-Test threshold ( ), test: the share of decisions answered by System 1 and the error rate of those answers (realized risk), with an exact binomial 95% interval over the accepted decisions. “0.0%” answered with “–” risk means the threshold accepts no decision. Source: s1_readouts_test.rotation.coverage_at_risk .
Figure 7: Target risk against realized test risk of System 1 answers accepted under the Learn-then-Test threshold ( ), Qwen3.5-27B, with exact binomial 95% intervals. The hollow marker is a target at which no decision is accepted. Source: s1_readouts_test.rotation.coverage_at_risk .
| Quantity | Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | Qwen3.8-27B |
| Test decisions | 8359 | 8359 | 8359 | 8359 |
| Always S1: accuracy | 50.9% | 54.1% | 62.2% | 60.2% |
| Always S2: accuracy | 77.4% | 80.0% | 82.7% | 81.8% |
| Always S2: thought tokens per decision | 2126 | 2088 | 1856 | 722 |
| Oracle accuracy | 81.5% | 83.5% | 85.8% | 86.2% |
| Gate: accuracy | 76.5% | 79.1% | 81.6% | 80.9% |
Table 7: Headline quantities for every model, test. Always S1: always answer directly; always S2: always think. Gate and anytime values use the thresholds chosen on dev ( dev_selected_operating_points , tolerance one point). “Savings” rows are the two halves of the savings criterion; LTT rows are the Learn-then-Test threshold for System 1. Each value’s report field is named in results/figures/<model>/numbers.tex .
| Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | Qwen3.8-27B | |
| Gate: normalized area | 0.360 | 0.367 | 0.436 | 0.421 |
| Gate zero-shot router | +0.011 | 0.063 | 0.054 | 0.067 |
| [ 0.017, +0.037] | [ 0.091, 0.038] | [ 0.080, 0.027] | [ 0.092, 0.041] | |
| Gate probe router | +0.043 | 0.014 | +0.040 | +0.021 |
| [+0.018, +0.068] | [ 0.039, +0.009] | [+0.014, +0.068] | [ 0.005, +0.050] | |
| Gate -gram router | +0.161 | +0.159 | +0.121 | +0.110 |
Table 8: Ranking comparison: normalized area under the test accuracy–coverage curve, , of the gate, and the gate’s difference from each pre-router with a 95% interval from a bootstrap over groups (1,000 resamples); a negative difference means the router ranks better. The last row is the combination of the gate’s confidence and the zero-shot score, fit post hoc (Section 5.7 ). Source: normalized_area_test .
| Policy | Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | Qwen3.8-27B |
| S1 only | 80.1% / 62.2% | 82.3% / 64.9% | 89.6% / 81.1% | 88.7% / 77.5% |
| Gate | 91.3% / 83.8% | 94.4% / 88.3% | 97.0% / 93.7% | 97.8% / 95.5% |
| Anytime | 91.3% / 83.8% | 93.5% / 87.4% | 97.0% / 93.7% | 96.5% / 92.8% |
| Always S2 | 93.1% / 85.6% | 94.8% / 89.2% | 97.8% / 95.5% | 99.6% / 99.1% |
| Escalated by the gate | 32.9% | 42.0% | 29.4% | 24.7% |
| Listed on JevBench (same items, from its per-task file) | ||||
Table 9: JevBench public items: accuracy on all 231 / on the 111 hard ones, under JevBench’s scoring rule, with each model’s release temperatures and thresholds (nothing fit on JevBench). Listed systems’ outcomes are JevBench’s own measurements at the pinned commit (Appendix F ). Exploratory. Source: results/jevbench-<model>-report-v2.json .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | State and question (abridged) | Answer |
|---|---|---|
| Dates | Notice given Thursday, June 24, 2027; 20 calendar days of notice, counted from the next day. On what date does the tenancy end? | July 14, 2027 |
| Counting | An eight-row order log (city, status, channel, amount). How many orders have status ‘refunded’? | 1 order |
| Chained rules | A purchasing policy with amount bands and a one-step reduction for pre-approved vendors, except hardware; a $3,630 repair from a vendor not on the list. What approval is needed? | No approval |
| Negation | “It is false that crate A was not sealed.” Was crate A sealed? | Yes |
| Arithmetic | 12 full trays of 17 rolls plus 7 extra rolls; 110 sold. How many rolls are left? | 101 rolls |
| Logic grid | Three friends, three pets, three drinks, five clues. Who owns the turtle? | Jonas |
Table 10: One generated decision per family (level 1, English), abridged from the released file.
| Block | Source | Lang. | Level | Rows | S1 | S2 | Gate | Escalated |
|---|---|---|---|---|---|---|---|---|
| A | Escalation (CLINC) | en | – | 290 | 60.0% | 72.4% | 71.0% | 43.1% |
| A | Every judgments | en | – | 84 | 95.2% | 95.2% | 95.2% | 7.1% |
| A | Intent routing (BANKING77) | en | – | 486 | 85.0% | 86.4% | 86.4% | 18.9% |
| A | Intent routing (CLINC) | en | – | 358 | 95.5% | 97.5% | 97.2% | 9.8% |
| A | NLI (WANLI) | en | – | 159 | 75.5% | 81.1% | 79.2% | 32.7% |
| A | SemIf authored | en | – | 84 | 92.9% | 94.0% | 95.2% | 10.7% |
Table 11: Qwen3.5-27B: test accuracy and escalation per source, language, and difficulty level.
| Block | Source | Lang. | Level | Rows | S1 | S2 | Gate | Escalated |
|---|---|---|---|---|---|---|---|---|
| A | Escalation (CLINC) | en | – | 290 | 69.0% | 84.1% | 77.2% | 34.8% |
| A | Every judgments | en | – | 84 | 96.4% | 95.2% | 94.0% | 4.8% |
| A | Intent routing (BANKING77) | en | – | 486 | 79.2% | 81.1% | 80.5% | 21.0% |
| A | Intent routing (CLINC) | en | – | 358 | 89.9% | 95.5% | 95.3% | 15.9% |
| A | NLI (WANLI) | en | – | 159 | 64.2% | 73.6% | 69.8% | 35.2% |
| A | SemIf authored | en | – | 84 | 82.1% | 95.2% | 94.0% | 23.8% |
Table 12: Qwen3.5-4B: test accuracy and escalation per source, language, and difficulty level.
| Block | Source | Lang. | Level | Rows | S1 | S2 | Gate | Escalated |
|---|---|---|---|---|---|---|---|---|
| A | Escalation (CLINC) | en | – | 290 | 53.4% | 84.1% | 72.1% | 36.6% |
| A | Every judgments | en | – | 84 | 94.0% | 95.2% | 95.2% | 8.3% |
| A | Intent routing (BANKING77) | en | – | 486 | 78.6% | 82.9% | 82.7% | 31.5% |
| A | Intent routing (CLINC) | en | – | 358 | 91.3% | 97.8% | 97.8% | 27.4% |
| A | NLI (WANLI) | en | – | 159 | 67.3% | 75.5% | 74.8% | 54.7% |
| A | SemIf authored | en | – | 84 | 92.9% | 94.0% | 94.0% | 32.1% |
Table 13: Qwen3.5-9B: test accuracy and escalation per source, language, and difficulty level.
| Block | Source | Lang. | Level | Rows | S1 | S2 | Gate | Escalated |
|---|---|---|---|---|---|---|---|---|
| A | Escalation (CLINC) | en | – | 290 | 65.2% | 80.7% | 80.7% | 44.5% |
| A | Every judgments | en | – | 84 | 96.4% | 97.6% | 97.6% | 7.1% |
| A | Intent routing (BANKING77) | en | – | 486 | 85.6% | 87.2% | 87.2% | 14.2% |
| A | Intent routing (CLINC) | en | – | 358 | 95.0% | 96.9% | 96.4% | 8.7% |
| A | NLI (WANLI) | en | – | 159 | 74.2% | 77.4% | 76.7% | 24.5% |
| A | SemIf authored | en | – | 84 | 94.0% | 97.6% | 97.6% | 13.1% |
Table 14: Qwen3.8-27B: test accuracy and escalation per source, language, and difficulty level.
More in this topic
LLM Decision-Making
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
See all 246 papers