UltraBERT-JEV: Encoder models for System One Applications
Organizations: Independent Researcher
Abstract
We study encoder models that pick one of several options in a single forward pass, without generating text. This technical report compares recipes for fine-tuning UltraBERT-380M with one mask per option. The four recipes differ in how an option is scored and how the scores become probabilities; one follows the expert gate of DeepSeek-V3. Trained on the same data as Qwen3-0.6B with the LLM-as-Jev recipe (67.6% test accuracy), the encoder reaches 69.7% to 72.1%. Scoring an option by "yes" minus "no" instead of "yes" alone gives the larger gain, from 69.7% to 71.3% on the test set and from 57.8% to 59.6% on external benchmarks (56.1% for the decoder). Sigmoid normalisation adds 0.8 test points; a square-root softplus adds none. One request at a time on an NVIDIA A100, the encoder takes 29 ms per decision against 68 ms, but uses more peak memory (2.8 against 2.2 GiB). An auxiliary head, trained on human-marked evidence and on labels selected with a reward adapted from SelfCite, predicts in the same pass the sentences annotated as support for the answer.
Figures & tables
| Model | Params | Dev ALL | Test ALL | External |
|---|---|---|---|---|
| UltraBERT-380M, softmax, yes | 380M | 71.2 | 69.7 | 57.8 |
| UltraBERT-380M, softmax, yes no | 380M | 71.7 | 71.3 | 59.6 |
| UltraBERT-380M, sqrt-softplus, yes no | 380M | 71.6 | 71.2 | 59.3 |
| UltraBERT-380M, sigmoid-norm, yes no | 380M | 72.7 | 72.1 | 59.5 |
| Qwen3-0.6B, full FT | 0.6B | 69.0 | 67.6 | 56.1 |
| Qwen3-0.6B, LoRA (reproduced) | 0.6B | 64.5 | 63.0 | 56.9 |
| Softmax, “yes” | Softmax, “yes” “no” | Sqrt-softplus, “yes” “no” | Sigmoid-norm, “yes” “no” | |
| Dev accuracy (%) | 71.4 | 71.5 | 71.1 | 72.9 |
| External accuracy (%) | 57.8 | 59.6 | 59.3 | 59.5 |
| Answer changes, options reordered (%) | 15.6 | 14.6 | 16.6 | 13.7 |
| Answer changes, one wrong option removed (%) | 5.9 | 4.8 | 5.4 | 4.9 |
| Largest gradient norm in training | 842 | 139 | 104 | 190 |
| Label frequencies vs. gold (TV) | 0.167 | 0.213 | 0.154 | 0.139 |
| Model | ms per decision | Weights | Peak, rows 4,096 tok. | Peak, max |
|---|---|---|---|---|
| UltraBERT-380M, fp32 weights | 29 | 1.6 | 2.6 | 2.8 |
| UltraBERT-380M, bf16 weights (lower accuracy) | 26 | 0.8 | 1.5 | 1.9 |
| Qwen3-0.6B | 68 | 1.1 | 2.0 | 2.2 |
| Qwen3-4B | 124 | 7.6 | 8.9 | 9.3 |
| Variant | Dev ALL | External | MuSiQue | ContractNLI | Nimble |
|---|---|---|---|---|---|
| Softmax, “yes” | 71.1 | 57.1 | 57.0 | 75.0 | 55.0 |
| Softmax, “yes” “no” | 71.5 | 59.1 | 64.0 | 78.0 | 48.0 |
| Sigmoid-norm, “yes” “no” | 72.8 | 59.4 | 76.0 | 81.0 | 43.0 |
| Softmax, “correct” | 70.6 | 56.8 | 48.0 | 77.0 | 52.0 |
| Softmax, “correct” “incorrect” | 70.1 | 58.0 | 54.0 | 76.0 | 46.0 |
| Sentence AP | ||||||||
| Method | All | CNLI | MuSiQue | Nimble | PW | Top-1 (%) | LDS | ms per row |
| Average attention | 0.25 | 0.07 | 0.20 | 0.34 | 0.38 | 14 | 0.09 | 69 |
| AT2 | 0.33 | 0.30 | 0.28 | 0.30 | 0.42 | 23 | 0.15 | 71 |
| ContextCite | 0.41 | 0.52 | 0.33 | 0.33 | 0.44 | 44 | 0.64 | 1,860 |
| Evidence head, softmax encoder | 0.55 | 0.28 | 0.32 | 0.96 | 0.67 | 54 | – | 68 |
| Evidence head, sigmoid-norm encoder | 0.68 | 0.55 | 0.52 | 0.96 | 0.68 | 74 | – | 72 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| UltraBERT-380M | Qwen3-0.6B (full) | |
|---|---|---|
| Optimiser | AdamW, , | AdamW |
| Weight decay | 0.1 | 0.01 |
| Schedule | 5% warmup, linear decay to 0 | 100 warmup steps, linear decay to 10% |
| Gradient clipping | 1.0 | 1.0 |
| Precision | fp32 weights, bf16 autocast | fp32 weights, bf16 autocast |
| Attention | PyTorch SDPA | PyTorch SDPA |
| Dataset | Rows | UB yes | UB yes no | UB sqrt-sp | UB sig-norm | Qwen full | Qwen LoRA | Qwen-4B LoRA |
|---|---|---|---|---|---|---|---|---|
| CLINC150 | 400 | 91.8 | 91.5 | 91.7 | 91.8 | 86.3 | 79.8 | 89.8 |
| MASSIVE | 400 | 83.7 | 84.5 | 83.8 | 84.9 | 76.9 | 68.8 | 83.5 |
| Bitext | 400 | 99.3 | 99.3 | 99.3 | 99.3 | 96.0 | 94.8 | 97.2 |
| ReClor | 400 | 58.9 | 60.7 | 61.8 | 59.9 | 57.7 | 51.2 | 72.2 |
| LogiQA 2.0 | 400 | 52.9 | 54.8 | 55.4 | 54.8 | 57.0 | 56.0 | 73.5 |
| CosmosQA | 400 | 74.8 | 74.2 | 75.1 | 75.2 | 59.9 | 55.0 | 76.0 |
Multiple-Choice Question Answering
Multiple-choice question answering studies how AI systems use knowledge and reasoning to select answers from candidate options.
See all 46 papers