Do We Still Need Fine-Tuned Classifiers? Swapping Typed Decision Models into 24 ACL-Venue Papers
Organizations: Comenius University in Bratislava · Cisco Systems · Kempelen Institute of Intelligent Technologies
Abstract
Typed Decision Models such as TypeSafe's Jev, also called System One or System 1 models, answer typed questions about an input with a probability for each option, and they need no training. Many NLP papers still fine-tune a task-specific classifier, and prompted large language models (LLMs) trail such classifiers on most text classification benchmarks. We ask how many of these fine-tuned classifiers a Typed Decision Model can replace under each paper's own evaluation, and what is needed where it cannot. We swap the predictors of 24 ACL-venue papers for Jev, keep each paper's splits, label definitions and metric, and check the results with Cloudflare's Clef, Microsoft-Decision-1 and OpenAI's GPT-6 Luna Decisions. The resulting 46 English test sets, with their questions and published scores, form the TDM-Gauntlet benchmark. With a zero-shot prompt, Jev directly replaced the paper's predictor on 12 of the 46 test sets, mostly single judgments with plain labels, but rarely where the paper's method was trained on its labels. Where Jev scored lower, we most often attributed the shortfall to an annotation convention that the label name does not state. With 500 training labels plus validation and no user-managed GPU, calibration and GEPA-style prompt optimization closed a median 72% of the gap to the paper's method, about twice what off-the-shelf GEPA closed, for a median $1.66 per search. On the same labels, fine-tuned encoders such as RoBERTa and DeBERTa closed 59%, and LoRA fine-tuning of the open LLM Qwen3.8-27B closed 97%. Fine-tuning remains the most accurate option, but a zero-shot prompt is often enough for plain labels, and prompt optimization closes most of the remaining gap.
Figures & tables
| Paper | Domain | Output | Jev interface | Test |
|---|---|---|---|---|
| PiNose ( 2024b ) | factuality | binary | Noul | R |
| FENICE ( 2024 ) | factuality | binary | Noul per claim | P |
| FactPICO ( 2024 ) | medicine | ordinal 0–4 | Score each | R |
| XSTest ( 2024 ) | safety | 3-way | Choice | R |
| LifeTox ( 2024 ) | safety | binary | Choice | R |
| PRP-Graph ( 2024 ) | biomed. IR | rank 100 | Choice, tournament | E |
| Dataset | Output | Metric | Jev | Paper’s method | LLM | ||
|---|---|---|---|---|---|---|---|
| Jev matches or beats the paper’s method | |||||||
| FENICE ( Scirè et al., 2024 ) | binary | 279 | bal. acc. | 77.4 | 65.7 | – | +11.7 |
| PiNose True-False ( Zhang et al., 2024b ) | binary | 1,000 | AUC | 96.9 | 86.6 | – | +10.3 |
| SciPrompt arXiv ( You et al., 2024 ) | 53-way | 5,295 | acc. | 79.0 | 70.2 | 54.9 | +8.9 |
| XSTest (judge) ( Röttger et al., 2024 ) | 3-way | 2,250 | acc. | 87.0 | 80.4 | 80.4 | +6.6 |
| FactPICO ( Joseph et al., 2024 ) | ordinal | 345 | Kendall | 53.4 | 47.4 | 47.5 | +6.0 |
| Zero-shot | Prompt search (training data) | Optimized q. (ours) | Other model | Same labels | Paper | |||||||||
| GEPA | GEPA | GEPA | Jev- | same q. | ||||||||||
| Configuration | Base q. | Rephr. q. | text | prog. | +d/c | adapted | mean | Clef | Flash | Encoder | Method | LLM | ||
| ConTempo TB-Dense | 1,427 | 54.4 | 57.4 ∘ | 58.7 ∘ | 60.9 | 60.2 | 63.0 | 61.1 1.4 | +6.7 | 56.6 ∘ | 51.7 | 30.8 ∗ 9.5 | 68.6 | 67.6 |
| EZ-STANCE A | 7,666 | 60.7 | 68.6 | 69.5 | 76.0 | 74.9 | 76.3 | 77.0 1.1 | +16.4 | 71.8 | 66.1 | 69.8 ∗ 2.0 | 81.2 | 49.9 |
| SMARTER Implicit Hate | 900 | 41.9 | 53.2 | 58.4 | 61.3 | 61.2 | 57.1 | 61.2 3.2 | +19.3 | 62.9 | 58.9 | 56.2 0.8 | 60.0 | 58.0 |
| Re3-Sci edit intent | 2,312 | 66.3 | 68.8 | 70.5 | 67.7 ∘ | 73.0 | 76.2 | 73.9 1.0 | +7.6 | 74.9 | 68.1 | 75.1 0.3 | 84.3 | 37.0 |
| Gap closed (%) | |||||
|---|---|---|---|---|---|
| Classifier | Labels | Build cost | all 13 | 5 by rule | Reach |
| Jev, prompt | none | none | 0 | 0 | 0 |
| + calibration | 400 | $0.04 | 23 | 23 | 0 |
| + opt. q. | 500+val. | $1.66 | 72 | 72 | 3 |
| Encoder | 500+val. | 0.37 GPU-h | 59 | 58 | 3 |
| LoRA-Qwen | 500+val. | 1.03 GPU-h | 97 | 100 | 6 |
| Gap closed (%) | Median difference | |||
| Model | Cal. | Opt. | vs. Jev | vs. encoder |
| Jev 1.13 | 23 | 75 | – | |
| Clef (27B) | 27 | 77 | ||
| Clef-flash (9B) | 32 | 67 | ||
| GPT-6 Luna Decisions | 18 | 43 | ||
| Microsoft-Decision-1 | 35 | 74 | ||
| Situation | Result in our sample |
|---|---|
| No labels, a prompt only ($0.05 per 1,000 items) | |
| Binary judgment | replaced on 5 of 6 test sets |
| Method not trained on the paper’s labels | replaced on 8 of 8 |
| Method trained on the paper’s labels | replaced on 4 of 38 |
| Over 300 possible labels | replaced on 0 of 5 |
| 500 training labels plus validation | |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Output | Metric | Jev | Paper’s method | LLM | ||
| Jev matches or beats the paper’s method | |||||||
| PiNose TriviaQA ( Zhang et al., 2024b ) | binary | 1,000 | AUC | 96.4 | 83.9 | – | +12.5 |
| FENICE ( Scirè et al., 2024 ) | binary | 279 | bal. acc. | 77.4 | 65.7 | – | +11.7 |
| PiNose True-False ( Zhang et al., 2024b ) | binary | 1,000 | AUC | 96.9 | 86.6 | – | +10.3 |
| SciPrompt arXiv ( You et al., 2024 ) | 53-way | 5,295 | acc. | 79.0 | 70.2 | 54.9 | +8.9 |
| PiNose NQ ( Zhang et al., 2024b ) | binary | 1,000 | AUC | 88.0 | 80.4 | – | +7.6 |
| Paper | Test set | Deviations from the paper’s protocol |
|---|---|---|
| LifeTox | complete | none |
| PiNose | complete | accuracy uses a fixed threshold, where the paper tunes it (AUC, the primary metric, is unaffected) |
| FENICE | partial release | different test rows from the paper. The GPT-3.5 claims behind its best number are not released, so we compare with FENICE on T5 claims. Stories are rebuilt from public editions |
| FactPICO | complete | expert span ratings are not released, so only the per-summary score is compared |
| CICC | complete | the published BANKING77 result cannot be reproduced at the printed coverage level, so we do not compare on it |
| Re3-Sci edit intent | complete | additions and deletions get a three-label question, and scoring still uses all five labels |
| Test set | Gap | Sec. | Conf. |
|---|---|---|---|
| (a) Unstated annotation conventions : 18 sets, 12 papers | |||
| LifeTox | 18.1 | other | low |
| Re3-Sci edit intent | 17.6 | – | high |
| EZ-STANCE A | 20.5 | – | high |
| EZ-STANCE B | 18.8 | – | medium |
| ConTempo MATRES | 10.9 | – | medium |
| Setting | GEPA | GEPA+d/c C/D | Opt. P | Jev-adapt. |
|---|---|---|---|---|
| Fit slice (train examples) | 64 | 400 | 400 | 400 |
| Fit slice held out of reflection | – | yes | no | no |
| Reflection pool (train examples) | – | 100 | – | – |
| Reflection minibatch | 3 | 3 | – | 64 |
| Progressive val subsets | 64/256 | 64/256 | – | 64/256 |
| GEPA merge | yes | yes | – | no |
| Base q. | Opt. | ||
|---|---|---|---|
| + LR | q. | ||
| ConTempo TB-Dense | 65.5 | 68.9 | +3.4 |
| EZ-STANCE A | 64.5 | 79.8 | +15.3 |
| SMARTER Implicit Hate | 57.0 | 62.4 | +5.3 |
| Re3-Sci edit intent | 67.2 | 72.4 | +5.3 |
| INLI | 79.0 | 84.4 | +5.4 |
| Median gap closed (%) | ||||
| Rephrased | Optimized q. + LR | |||
| Labels | Calibration | q. + LR † | all | no demos § |
| 50 | 9 | 8 | 55 ‡ | 2 ‡ |
| 100 | 22 | 22 | 61 | 10 |
| 250 | 26 | 36 | 66 | 42 |
| 400 | 23 | 39 | 75 | 57 |
| Task | Cluster ( ) | 95% CI | Item-level CI | Per seed (0/1/2) | |||
| ConTempo TB-Dense | document (9) | 0.68 | 1.00 | ||||
| EZ-STANCE A | tweet (1,685) | 0.001 | 0.010 | ||||
| SMARTER Implicit Hate | item (900) | 0.97 | 1.00 | – | |||
| Re3-Sci edit intent | document (62) | 0.47 | 1.00 | ||||
| INLI | premise (1,000) | 0.91 | 1.00 | ||||
| DREsS content | prompt (59) | 0.19 | 1.00 |
| Set | Metric | Zero | Cal. | cal. [95% CI] | Opt. | opt. [95% CI] | Paper | LLM | Gap % cal./opt. | Jev $ | LLM $ | Test $ | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| All eligible shortfall sets, final optimizer settings, one run (seed 0) | |||||||||||||
| ConTempo MATRES | micro-F1 | 837 | 72.3 | 79.7 | +7.4 [+4.3, +11.3] | 79.4 | +7.2 [+3.9, +11.6] | 83.2 | 56.7 | 68 / 66 | 0.75 | 0.86 | 0.02 |
| LifeTox | macro-F1 | 3,000 | 78.4 | 82.6 | +4.1 [+2.4, +5.7] | 94.1 | +15.7 [+13.4, +18.1] | 96.5 | 74.4 | 23 / 87 | 0.87 | 0.60 | 0.57 |
| SciPrompt S2ORC | acc. | 5,660 | 52.9 | 53.7 | +0.8 [+0.1, +1.6] | 55.7 | +2.8 [+1.8, +3.9] | 56.7 | 49.6 | 22 / 75 | 1.59 | 1.53 | 0.25 |
| SMARTER HateXplain | macro-F1 | 1,193 | 55.6 | 59.8 | +4.2 [+1.4, +7.0] | 64.6 | +9.0 [+5.8, +11.9] | 64.0 | 56.0 | 50 / 107 | 0.54 | 0.62 | 0.06 |
| SMARTER Latent Hate | macro-F1 | 1,019 | 58.0 | 57.7 | -0.3 [-2.3, +1.8] | 61.3 | +3.2 [+0.4, +6.0] | 69.0 | 60.0 | -3 / 30 | 0.53 | 0.70 | 0.02 |
| Test set | Seed 0 | 1 | 2 | Mean | Paper | Gap % |
|---|---|---|---|---|---|---|
| ConTempo MATRES | 79.4 | 78.9 | 79.6 | 79.3 | 83.2 | 65 |
| LifeTox | 94.1 | 91.3 | 93.8 | 93.1 | 96.5 | 81 |
| SciPrompt S2ORC | 55.7 | 56.6 | 54.5 | 55.6 | 56.7 | 72 |
| SMARTER HateXplain | 64.6 | 64.7 | 64.2 | 64.5 | 64.0 | 106 |
| SMARTER Latent Hate | 61.3 | 61.1 | 62.6 | 61.7 | 69.0 | 33 |
| LoRA | Qwen, opt. q. + LR | Best baseline | Jev, opt. q. | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Set | Score | lr | Score | LoRA (95% CI) | Score | LoRA (95% CI) | Score | LoRA (95% CI) | Paper | GPU-h |
| ConTempo TB-Dense | 68.9 | 2e 4 | 45.3 | +23.6 ∗ (+18.1, +27.2) | 30.8 (ft) | +38.1 ∗ (+27.2, +46.2) | 62.4 | +6.5 (+1.8, +10.8) | 68.6 | 1.1 |
| EZ-STANCE A | 79.5 | 1e 4 | 71.9 | +7.7 ∗ (+6.7, +8.6) | 69.8 (ft) | +9.7 ∗ (+7.8, +11.8) | 77.7 | +1.8 ∗ (+1.0, +2.8) | 81.2 | 1.0 |
| SMARTER Implicit Hate | 63.5 | 1e 4 | 60.4 | +3.1 (+0.2, +6.1) | 56.2 (ft) | +7.3 ∗ (+4.1, +10.6) | 60.4 | +3.2 (+0.3, +5.9) | 60.0 | 0.9 |
| Re3-Sci edit intent | 81.7 | 2e 4 | 71.6 | +10.1 ∗ (+7.5, +13.1) | 75.1 (ft) | +6.6 ∗ (+4.1, +9.7) | 74.0 | +7.7 ∗ (+5.4, +10.6) | 84.3 | 1.1 |
| INLI | 85.9 | 1e 4 | 76.4 | +9.5 ∗ (+8.3, +10.7) | 85.9 (ft) | 0.1 ( 1.7, +1.6) | 82.4 | +3.5 ∗ (+2.4, +4.6) | 92.4 | 0.9 |
| Zero-shot, 45 test sets | With labels, 13 configurations, Jev’s questions | ||||||||||
| Same | Better/within/ | Median | Higher/ | Gap closed (%) | Gain | Better/worse | |||||
| Model | group | worse | Jev | lower | Cal. | Opt. | Opt. (Jev basis) | (points) | vs. Jev | vs. encoder | |
| Clef (27B) | 39 | 11/15/19 | 0.5 | 0.87 | 9/10 | 27 | 77 | 64 | 0/2 | 2/1 | |
| Clef-flash (9B) | 35 | 8/6/31 | 4.4 | – | – | 32 | 67 | 54 | 0/9 | 1/6 | |
| GPT-6 Luna Decisions | 41 | 6/14/25 | 2.0 | 0.96 | 3/16 | 18 | 43 | 28 | 0/7 | 0/7 | |
| Microsoft-Decision-1 | 37 | 12/13/20 | 0.4 | 0.87 | 7/9 | 35 | 74 | 78 | 0/1 | 2/1 | |
| Question set | Spans | A | B | C | D | E | F | Z | A B | C B | D Z | E F | Refit | |
| ConTempo TB-Dense s0 | 1,427 | 1 | 62.4 | 61.9 | 54.4 | 54.7 | 61.3 | 59.4 | 54.4 | +0.5 | 7.5 | +0.4 | +1.9 | 100% |
| ConTempo TB-Dense s1 | 1,427 | 1 | 61.2 | 61.0 | 60.6 | 55.1 | 58.9 | 60.2 | 54.4 | +0.2 | 0.4 | +0.7 | 1.2 | 100% |
| EZ-STANCE A s0 | 7,666 | 3 | 77.7 | 68.6 | 77.0 | 69.3 | 71.4 | 61.8 | 60.7 | +9.1 ∗ | +8.5 ∗ | +8.6 ∗ | +9.6 ∗ | 100% |
| EZ-STANCE A s1 | 7,666 | 7 | 77.6 | 73.2 | 76.2 | 69.5 | 70.4 | 61.2 | 60.7 | +4.5 ∗ | +3.0 ∗ | +8.9 ∗ | +9.2 ∗ | 100% |
| EZ-STANCE A s2 | 7,666 | 4 | 75.8 | 74.8 | 75.5 | 61.8 | 62.3 | 60.7 | 60.7 | +1.0 ∗ | +0.8 | +1.2 ∗ | +1.6 ∗ | 100% |
| SMARTER Implicit Hate s0 | 900 | 3 | 60.4 | 60.3 | 59.9 | 44.0 | 48.6 | 49.4 | 41.9 | +0.1 | 0.4 | +2.1 | 0.8 | 100% |
| Paper and data | Example |
|---|---|
| BIORC800 ( Lan et al., 2024 ) Task. Label each sentence of a biomedical abstract with the role or roles it plays in the abstract, such as stating the background, the methods or the results. Output. One or more of {BACKGROUND, OBJECTIVE, METHODS, RESULTS, CONCLUSIONS, OTHER} per sentence. Test set. abstracts, micro-F1. | Source. Released training split (no licence stated), not a test item. Previous Cytokines were detected by the enzyme-linked immunosorbent assay (ELISA), real-time … Sentence Parenchymal levels of MDSC had a positive correlation with levels of CCL2, IL-17, and IL-18, suggesting these cytokines may attract MDSC into the parenchyma. Gold RESULTS, CONCLUSIONS |
| CICC ( den Hengst et al., 2024 ) Task. Recognise what a user wants from a short request to a virtual assistant. The model’s probabilities are turned into a set of candidate intents: the assistant answers directly if one intent remains, and asks a clarification question if a few remain. Output. A probability for every intent in the dataset’s inventory, e.g. 150 for CLINC150 and 175 for ACID (intent names such as “order checks”), from which conformal prediction sets are built. The paper’s BANKING77 set is not compared (Table 8 ). Test sets. MTOD: , Single; HWU64: , Single; CLINC150: , Single; ACID: , Single; ATIS: , Single. | Source. CLINC150 released training split (CC BY 3.0), not a test item. Utterance send me more checkbooks please Gold order_checks |
| ConTempo ( Niu et al., 2024 ) Task. Given two events mentioned in a news article, decide how they are ordered in time: which starts first (MATRES), or whether one precedes, contains or coincides with the other (TB-Dense), or whether the text leaves it vague. Output. One relation per event pair, from {BEFORE, AFTER, EQUAL, VAGUE} in MATRES and from {BEFORE, AFTER, INCLUDES, IS_INCLUDED, SIMULTANEOUS, VAGUE} in TB-Dense. Test sets. MATRES: event pairs, micro-F1; TB-Dense: event pairs, micro-F1. | Source. MATRES training annotation (TimeBank part, not a test item). The news sentence is LDC text and is not shown. Document ABC19980120.1830.0957 Event 1 predicted Event 2 tried Gold BEFORE |
| DREsS ( Yoo et al., 2025 ) Task. Score an essay written by an English-as-a-foreign-language student in response to a writing prompt, separately for content, organization and language. Output. One score per rubric from {1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0}. The total is their sum. Test set. essays, QWK (total). | Source. No essay shown: the dataset’s consent terms forbid sharing, and the rubric text is from the paper’s Table 2. Essay (not shown) Rubric content: 5.0 out of 5.0 (highest score): Paragraph is well-developed and relevant to the argument, supported with strong reasons and examples. Gold the course instructor’s score for each rubric |
| Entity or Relation Embeddings ( Mtumbuka and Schockaert, 2024 ) Task. Given a sentence in which two entities are marked, decide which relation (if any) the sentence expresses from the first entity to the second. Output. In Wiki-WD, one of 352 Wikidata relation names, such as “no relation”; in NYT-FB, one of 56 Freebase relation paths, such as “NA (no relation)”. Test sets. Wiki-WD: entity pairs, micro-F1; NYT-FB: entity-pair rows, P@10. | Source. Wiki-WD test set (CC BY-SA, item wikiwd-55 ). Sentence She was married to Marc Davis , a Disney animator and Imagineer . Head Marc Davis Tail animator Gold P106 (occupation) |
| Explanation ranking ( Bailleux et al., 2026 ) Task. Given a premise, a hypothesis and the correct inference relation between them, rank candidate explanations of why that relation holds from best to worst. Output. A ranking of the candidate explanations, which come in 5 quality tiers from nonsense to gold. Test set. e-SNLI validation (not comparable: rebuilt candidates): queries, nDCG@5. | Source. Test set (e-SNLI, MIT / CC BY-SA, item esnli-dev-4717112737.jpg#3r1c ). Premise A lady with bright orange hair walking in a crowd. Hypothesis: The woman is asleep. (contradiction) Candidates (a) The statements conflict. (b) These statements contradict because they make incompatible claims. (c) Purple elephants dance backwards when triangles sing opera. (d) Walking is contradictory to sleeping. (e) This is neutral because they’re both statements. Gold (d) gold (b) good (a) fair (e) poor (c) nonsense (rebuilt Algorithm 1 scores) |