cs.CLNov 16, 2026

Train After You Measure: a Pre-Registered Learning Curve for Fine-Tuning Typed Decision Models on One User's Labels

Authors: Federico Lucca

Organizations: Independent researcher, Trento, Italy

Abstract

Fine-tuning a small typed decision model on one user's email labels did not clear the margin we pre-registered over a logistic regression trained on the same labels. In earlier work, about two hundred labels and a TF-IDF logistic regression had the highest point accuracy on this inbox, though not significantly above the best hosted model; we ask whether the same labels, given to a decision model, close the gap. A first registered study, on a generic 4B base with cross-validation and message type only, stayed below the baseline at every label budget (-1.8 to -7.3 points). The main study was registered before any fine-tuning run: a fixed test set of 135 emails never used for selection, LoRA adapters on the public Tev1-0.8B and Tev1-4B decision models, nested subsets of 50 to 587 labels, two seeds and five decision fields on a free T4 GPU. Four of its five hypotheses are refuted--two in the fine-tunes' favour--and one is confirmed for one model. On message type the 4B model misses the registered three-point margin at 400 labels (+2.6) and passes it with all 587 (+3.3); with all labels both models are above the baseline by +0.7 and +3.3 points, where one email is 0.74 points and a 95% interval is about 15 points wide, and the 4B model is clearly ahead in type macro-F1 (0.581 vs. 0.406). On the three action switches the baseline is ahead at every label budget for both models (0.931 vs. 0.904 and 0.899 mean accuracy with all labels), at under a millisecond per question against 306 and 1681 ms. The 0.8B type curve barely improves after 300 labels and then falls. The baseline never predicts the two rarest classes (7 offers and 8 spam in the training pool); the fine-tuned models find spam in some runs, not consistently across seeds. We list the explanations we did not test and say when a new attempt would be worth making. Scripts are public; weights and emails are not.

Explore similar work

Nov 15, 2026

Do We Still Need Fine-Tuned Classifiers? Swapping Typed Decision Models into 24 ACL-Venue Papers

Typed Decision Models such as TypeSafe's Jev, also called System One or System 1 models, answer typed questions about an input with a probability for each option, and they need no training. Many NLP papers still fine-tune a task-specific classifier, and prompted large language models (LLMs) trail such classifiers on most text classification benchmarks. We ask how many of these fine-tuned classifiers a Typed Decision Model can replace under each paper's own evaluation, and what is needed where it cannot. We swap the predictors of 24 ACL-venue papers for Jev, keep each paper's splits, label definitions and metric, and check the results with Cloudflare's Clef, Microsoft-Decision-1 and OpenAI's GPT-6 Luna Decisions. The resulting 46 English test sets, with their questions and published scores, form the TDM-Gauntlet benchmark. With a zero-shot prompt, Jev directly replaced the paper's predictor on 12 of the 46 test sets, mostly single judgments with plain labels, but rarely where the paper's method was trained on its labels. Where Jev scored lower, we most often attributed the shortfall to an annotation convention that the label name does not state. With 500 training labels plus validation and no user-managed GPU, calibration and GEPA-style prompt optimization closed a median 72% of the gap to the paper's method, about twice what off-the-shelf GEPA closed, for a median $1.66 per search. On the same labels, fine-tuned encoders such as RoBERTa and DeBERTa closed 59%, and LoRA fine-tuning of the open LLM Qwen3.8-27B closed 97%. Fine-tuning remains the most accurate option, but a zero-shot prompt is often enough for plain labels, and prompt optimization closes most of the remaining gap.
Oct 23, 2026

Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox

Personal email agents decide, for every message, its type and urgency and whether to draft a reply, create a task or ignore it. In production these decisions are delegated to a generative LLM or to rules, and their error rate is rarely measured: the stored labels were written by the system itself. We describe Foyer, a self-hosted personal inbox agent, and a measurement protocol (stratified sampling, frozen snapshots, thread-grouped splits, secret redaction, human labels in the review loop) that produces ground truth before any training. On 231 emails in Italian and English we compare eleven systems: the stored production type, a TF-IDF + logistic-regression baseline, local and hosted typed decision models (Tev1, Nimble, Jeva, Jev) and generative LLMs. Four results. The stored production type scores 28% against a 57% majority class. With about two hundred labels a logistic regression has the highest point accuracy on every field (type 0.77 +/- 0.02 over five seeds) at a thousandth of the latency, but on every field at least one hosted model is statistically indistinguishable from it (exact McNemar, Holm-corrected). A generative model recovers part of the sender's relationship to the owner from the text (0.60); inheritance from the contact is right 98.6% of the time where the sender was seen before: it belongs in the contact graph, not in the prompt. Per-email accuracy is not the product metric: composed into actions, the classifier ignores 18% of the emails that needed action, against 4% for the best generative model; a cost-chosen threshold cuts that to 10%, indistinguishable from the hosted decision model's; on the adversarial corpus the thresholded classifier is the cheapest system above a missed-email price of 4. Two blind annotators agree with the reference at kappa 0.81 and 0.86 on type. v4 corrects threshold results computed on a truncated corpus in v1-v3.
Oct 19, 2026

Typed Decision Models under Prevalence Shift and Label Budgets: Decision Training, Calibration, and Implicit Priors

Typed decision models (TDMs) return probabilities over caller-defined options, and Jev's vendor describes it as calibrated. We ask whether vendor decision training improves a model over its backbone, whether a TDM's calibration and thresholds withstand prevalence (label) shift like a fine-tuned classifier's, and how many labels a trained model needs to overtake a TDM. We compare three TDMs (Jev, Tev1-4B, Nimble-9B), their base models and trained classifiers on identical candidate-set decisions from five public datasets (on intent datasets, within the gold label's intent group) under simulated shift, with 35 paired tests frozen before the confirmatory analysis (earlier test-data checks disclosed), and estimate label-budget curves. First, decision training is not always beneficial: Nimble-9B, which differs from Qwen3.5-9B only by a LoRA adapter, is better calibrated on intent tasks but 12 Macro-F1 points worse on financial sentiment, a loss that persists across prompt formats; Tev1-4B's largest gain over its backbone is a prompt-format effect. Second, on 50 fixed moderate shifts, Jev's top-label calibration error and threshold risk grow significantly more than a fine-tuned DeBERTa's only on financial sentiment, sensitive to the prior set; averaged over 2,000 shifts the gap is at most one point. Third, label-free EM correction with the model's implicit prior lowers Jev's shifted calibration error where it is highest, but labelled recalibration lowers it far more. Fourth, DeBERTa needs 100-250 labels to overtake Jev on hate speech and does not clearly overtake it on CLINC150 at any budget; a QLoRA adapter on Qwen3.5-4B overtakes Jev with at most 2,500 labels on every dataset, on CLINC150 only marginally and not under in-budget selection. The TDMs tested are neither uniformly better than their backbones nor uniquely fragile under shift; their lead over trained models depends on the task.