Small Heads, Strong Decisions: A Controlled Study of Typed-Decision Models
Abstract
In September 2026, TypeSafe announced Jev as the first model in its System One category: unstructured state goes in, and typed, calibrated decisions that software can use directly come out. Jev's architecture and training data are unpublished, but many open typed-decision models have followed, including Nimble, the Decision 1.0-3.0 family and Clef. They answer structured questions about a piece of text, such as a yes/no check, a choice among labelled options or a graded score, by reading the text with a language model and scoring the options with a small readout. Each pairs a different readout with a different backbone, training mix and recipe, so it is unclear which design choices matter. We re-implement five readouts and train all of them under one protocol: the same backbone, the same 15 public English training sources, the same recipe with a per-head learning-rate sweep, and the same seed. Our experiments demonstrate that: The complexity of the head does not affect accuracy: on Qwen3-0.6B every readout reaches 81.3-82.0% in-domain accuracy, and a 0.5M-parameter cosine head, which scores each option by its cosine similarity to the question, matches the 27M-parameter joint schema head adapted from Clef in-domain at half its inference time. Reading all of a text's questions from one prompt also matches one prompt per question in accuracy, with 1.6 times faster inference and 2.1-2.5 times faster training on one H100. The backbone's size matters more than either choice: moving from Qwen3-0.6B to Qwen3-4B adds 4 points in-domain and 11 points on held-out sources, while the base, instruct and thinking checkpoints of the same model tie. We recommend spending effort on the backbone and keeping the head minimal.
Figures & tables
| Model | Prompt regime | Readout | New params | Attention |
|---|---|---|---|---|
| Jev [ 1 ] | closed; architecture not published | |||
| Nimble [ 4 ] | one per question | LM letter readout | 0 | causal |
| Decision 1.0/2.0 [ 22 , 34 ] | one per question | bilinear-MLP head | 1.05M | causal |
| Decision 3.0 [ 33 ] | one per question | learned letter readout | 0.26M | mixed † |
| Clef [ 7 ] | all questions jointly | joint schema head | 27.3M | causal |
| Ours | all questions jointly | cosine head | 0.53M | causal |
| Training rows | Evaluation | ||||
|---|---|---|---|---|---|
| Source | Pool | Per epoch | Distinct | Split | Questions |
| BoolQ | 9,427 | 9,427 | 9,427 | val. | 2,616 |
| SST-5 | 8,544 | 8,544 | 8,544 | test | 2,210 |
| Emotion | 16,000 | 10,000 | 16,000 | test | 2,000 |
| SNLI | 549,367 | 10,000 | 20,000 | test | 9,824 |
| MNLI | 392,702 | 10,000 | 20,000 | test | 9,832 |
| In-domain | Held-out | ECE | |||||
|---|---|---|---|---|---|---|---|
| Readout | Params | Acc. | Acc. | ID | OOD | ||
| Cosine | 0.53M | 82.0 | ref. | 78.0 | ref. | 0.034 | 0.069 |
| Joint schema | 27.3M | 81.9 | 0.10 [ 0.33, +0.13] | 76.1 | 1.91 [ 2.65, 1.13] | 0.034 | 0.077 |
| Bilinear-MLP | 1.05M | 82.0 | +0.00 [ 0.23, +0.24] | 76.5 | 1.46 [ 2.27, 0.66] | 0.034 | 0.070 |
| Learned letter | 0.26M | 81.3 | 0.61 [ 0.85, 0.35] | 76.4 | 1.60 [ 2.49, 0.71] | 0.033 | 0.057 |
| LM letter | 0 | 81.6 | 0.31 [ 0.55, 0.07] | 79.4 | +1.41 [+0.54, +2.25] | 0.033 | 0.063 |
| In-domain | Training | Inference | ||||
|---|---|---|---|---|---|---|
| Readout | Design | Prompt | Params | acc. (%) | (GPU-h) | (ms/row) |
| Cosine | ours | joint | 0.53M | 82.0 | 6.9 | 6.4 |
| Joint schema | Clef | joint | 27.3M | 81.9 | 8.7 | 13.6 |
| Bilinear-MLP | Decision 1.0/2.0 | per question | 1.05M | 82.0 | 17.4 | 9.9 |
| Learned letter | Decision 3.0 | per question | 0.26M | 81.3 | 17.6 | 9.4 |
| LM letter | Nimble | per question | 0 | 81.6 | 17.8 | 9.4 |
| Readout | Codes at evaluation | Belebele | Kev MMLU |
|---|---|---|---|
| Cosine | – | 66.3 | 38.8 |
| LM letter | uppercase letters ( A , B , …) | 76.1 | 48.8 |
| LM letter | number words | 72.3 | 47.5 |
| LM letter | lowercase letters ( a , b , …) | 67.4 | 40.0 |
| LM letter | symbols | 59.3 | 32.5 |
| LM letter, trained lowercase | lowercase letters | 69.2 | 41.2 |
| In-domain | Held-out | |||||
|---|---|---|---|---|---|---|
| Question / option | Causal | Bidir. | Causal | Bidir. | ||
| last / last | 82.0 | 77.1 | 78.0 | 66.2 | ||
| mean / last | 81.7 | 77.2 | 76.7 | 71.3 | ||
| mean / mean | 81.8 | 78.3 | 76.8 | 70.7 | ||
| last / mean | 81.8 | 75.8 | 76.2 | 64.4 | ||
| In-domain | Held-out | ||||||
| Backbone | Ep. | Cosine | Joint schema | Cosine | Joint schema | ||
| Qwen3-0.6B | 2 | 82.0 | 81.9 | 0.10 | 78.0 | 76.1 | 1.91 |
| Qwen3-1.7B | 2 | 84.3 | 84.5 | +0.14 | 83.1 | 82.3 | 0.80 |
| Qwen3-4B | 2 | 85.9 | 85.9 ‡ | +0.05 | 88.9 | 89.3 ‡ | +0.43 |
| Qwen3-4B | 1 | 85.2 | – | – | 87.2 | – | – |
| Qwen3-8B | 1 | 85.8 ‡ | 85.9 | +0.07 | 89.8 ‡ | 89.0 | 0.80 |
| In-domain | Held-out | |||||
|---|---|---|---|---|---|---|
| Backbone | Post-trained | Base | Post-trained | Base | ||
| Qwen3-0.6B | 82.0 | 81.7 | 0.23 | 78.0 | 77.7 | 0.29 |
| Qwen3-1.7B | 84.3 | 84.3 | 0.03 | 83.1 | 82.7 | 0.36 |
| Qwen3-4B | 85.9 | 85.9 ‡ | +0.05 | 88.9 | 89.2 ‡ | +0.35 |
| Qwen3-8B | 85.8 ‡ | 85.9 | +0.09 | 89.8 ‡ | 88.5 | 1.33 |
| Qwen3-14B | 86.6 | 86.4 | 0.18 | 90.8 | 90.9 | +0.10 |
| In-domain | Held-out | |||
|---|---|---|---|---|
| Backbone | Acc. | Acc. | ||
| Qwen3-4B (hybrid thinking) | 85.9 | ref. | 88.9 | ref. |
| Qwen3-4B-Instruct-2507 | 85.9 | +0.06 [ 0.16, +0.27] | 89.4 | +0.53 [ 0.03, +1.06] |
| Qwen3-4B-Thinking-2507 | 85.9 | +0.05 [ 0.17, +0.27] | 89.0 | +0.09 [ 0.61, +0.69] |
| Qwen3-4B-Base ‡ | 85.9 | +0.05 [ 0.17, +0.27] | 89.2 | +0.35 [ 0.23, +0.94] |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Precision | bf16, no quantisation |
| LoRA | rank 32, , dropout 0, all attention and MLP projections |
| LoRA learning rate | ( or where marked, section 6.4 ) |
| Head learning rate | cosine , joint schema , bilinear-MLP , learned letter |
| Optimiser | AdamW, , , weight decay 0.01 (0 on norms and biases) |
| Schedule | linear warm-up over 5% of steps, then cosine decay to 0 |
| Head | Learning rate (validation NLL) |
|---|---|
| Cosine | 3e-4 (0.669), 1e-3 (0.660), 3e-3 (0.662) |
| Joint schema | 3e-5 (0.668), 1e-4 (0.671), 3e-4 (0.659), 1e-3 (0.664) |
| Bilinear-MLP | 3e-4 (0.668), 1e-3 (0.668), 3e-3 (0.667) |
| Learned letter | 3e-6 (0.719), 1e-5 (0.721), 3e-5 (0.710), 1e-4 (0.709) |
| Source | Questions | Cosine | Joint schema | Bilin.-MLP | Learned let. | LM let. | Cos. per-q. |
|---|---|---|---|---|---|---|---|
| BoolQ | 2,616 | 84.8 | 84.5 | 84.1 | 84.4 | 84.1 | 84.7 |
| SST-5 | 2,210 | 56.4 | 55.9 | 56.2 | 55.3 | 56.6 | 56.1 |
| Emotion | 2,000 | 92.5 | 93.0 | 92.8 | 92.8 | 93.5 | 92.9 |
| SNLI | 9,824 | 87.4 | 87.7 | 87.7 | 87.7 | 87.2 | 87.8 |
| MNLI | 9,832 | 84.0 | 84.1 | 84.2 | 83.7 | 83.4 | 83.9 |
| MASSIVE | 5,948 | 91.1 | 91.3 | 91.3 | 91.3 | 91.0 | 90.7 |
| Readout | AG News | DBpedia | QASC | Belebele | Kev | Mean |
|---|---|---|---|---|---|---|
| questions | 7,600 | 70,000 | 926 | 900 | 1,046 | |
| Cosine | 82.1 | 96.9 | 88.4 | 66.3 | 56.1 | 78.0 |
| Joint schema | 77.9 | 95.1 | 86.0 | 66.0 | 55.4 | 76.1 |
| Bilinear-MLP | 80.6 | 95.4 | 86.6 | 65.3 | 54.6 | 76.5 |
| Learned letter | 77.4 | 94.6 | 85.6 | 68.1 | 56.2 | 76.4 |
| LM letter | 79.8 | 95.8 | 87.5 | 76.1 | 57.7 | 79.4 |
| Pooling | Attention | AG News | DBpedia | QASC | Belebele | Kev | Mean | ECE |
|---|---|---|---|---|---|---|---|---|
| last / last | causal | 82.1 | 96.9 | 88.4 | 66.3 | 56.1 | 78.0 | 0.069 |
| mean / last | causal | 80.5 | 94.9 | 87.4 | 65.7 | 55.4 | 76.7 | 0.071 |
| mean / mean | causal | 79.6 | 95.5 | 85.3 | 67.3 | 56.3 | 76.8 | 0.068 |
| last / mean | causal | 80.8 | 94.6 | 84.7 | 66.2 | 54.8 | 76.2 | 0.070 |
| last / last | bidir. | 75.1 | 84.6 | 81.7 | 41.2 | 48.4 | 66.2 | 0.107 |
| mean / last | bidir. | 80.2 | 90.9 | 84.8 | 51.8 | 48.9 | 71.3 | 0.074 |
| Source | Causal | Bidirectional |
|---|---|---|
| procedural-typed-decisions | 81.5–81.9 | 49.7–60.2 |
| typed-decisions | 74.9–77.2 | 61.5–68.7 |
| Cosmos QA | 73.5–74.2 | 61.6–66.9 |
| SQuAD 2.0 | 84.4–84.9 | 75.5–79.4 |
| BoolQ | 83.5–84.8 | 80.1–82.5 |
| MNLI | 84.0–84.3 | 80.3–82.9 |