Abstention Is Not Threshold Compliance: Stated Scoring Rules in Open Language Models
Organizations: Independent Researcher
Abstract
Kalai et al. propose stating the scoring rule in the prompt with an explicit threshold t, so that a model should answer only when its probability of being correct exceeds t. We probe each requirement of compliance on shared items, in 12 checkpoints from three post-training stages of four fully open 3-8B recipes, with exploratory models up to 32B. At t = 0.75, a line limiting the reply to an answer or "I abstain" (the abstention format) raised the mean abstention rate from 20% to 91%. Yet within this format, raising t from 0.5 to 0.9 changed it by only -0.8 percentage points. Without the line, the higher threshold even lowered abstention, and up to 40% of a checkpoint's items received decisions that no compliant policy with threshold-independent beliefs could make. Given the probability that their answer was correct, the checkpoints barely responded to the threshold in a forced-choice probe. Comparisons with self-assessment signals such as P(True) depended on the grader and on answer drift between prompts. An external threshold on recalibrated P(True) made coverage non-increasing in t by construction. Calibrated on the evaluation distribution (post hoc), it beat the abstention format at t = 0.5; calibrated on TriviaQA and applied to PopQA, however, it fell below that format at 0.75. On OLMo 3 7B, decisions trained on threshold-conditioned targets did not follow unseen thresholds consistently. A post-hoc variant trained at four thresholds answered less as t rose but still failed a pre-specified test, mainly on agreement with its training rule. A high abstention rate is not evidence of compliance: evaluations using stated scoring rules should report how often models abstain, whether decisions respond to the stated threshold, and which items they answer.
Figures & tables
| Hypothesis | Estimate | 95% CI | p (unadjusted) | effect 0 detected | band: point rule / interval rule | |
| H1 | U([email protected]) 0 for all 12 | 12/12 (U 1.08 to 0.23) | – | 0.001 | yes (Holm 0.0083) | – |
| H2 | threshold responsiveness 0 | 0.037 | [0.034, 0.039] | 0.001 | yes (Holm 0.0100) | clear no / clear no |
| H3 | consistency with stated confidence 0 (10 identified) | 0.062 | [0.029, 0.095] | 0.001 | yes (Holm 0.0167) | clear no / clear no |
| H4a | selection regret vs P(True) 0 | 0.068 | [0.012, 0.122] | 0.008 | yes (Holm 0.0250) | clear yes / in between |
| H4b | selection regret vs P(IK) 0 | 0.059 | [ 0.119, 0.005] | 0.968 | no (Holm 0.0500) | clear no / clear no |
| H5 | MC overconfidence PO SFT 0 | 0.093 | [0.084, 0.101] | 0.001 | yes (Holm 0.0125) | – |
| Model | verb. none | verb. .5 | verb. .75 | verb. .9 | ABST none | ABST .5 | ABST .75 | ABST .9 |
| OLMo 3 Final | 12 | 80 | 70 | 68 | 58 | 97 | 97 | 98 |
| Tulu 3 Final | 2 | 8 | 4 | 3 | 23 | 82 | 76 | 78 |
| OLMo 2 Final | 1 | 5 | 6 | 5 | 81 | 95 | 94 | 94 |
| SmolLM3 Final | 5 | 12 | 10 | 8 | 48 | 92 | 93 | 94 |
| Mean of 12 primary | 4 | 24 | 20 | 19 | 54 | 93 | 91 | 92 |
| OLMo 3.1 32B (8-bit) | 9 | 66 | 67 | 66 | 57 | 79 | 80 | 80 |
| t | coverage ABST / I0 | accuracy ABST / I0 | ABST / I0 | perfect selection | R = R(I0) R(ABST) [95% CI] | checkpoints: interval 0 / 0 | reading |
| 0.5 | 0.07 / 0.46 | 0.82 / 0.71 | +0.025 / +0.092 | +0.237 | +0.067 [+0.049, +0.082] | 12 / 0 | I0 higher |
| 0.75 | 0.09 / 0.17 | 0.84 / 0.82 | +0.008 / +0.012 | +0.119 | +0.004 [ 0.004, +0.010] | 3 / 1 | uncertain: the interval includes 0 (not equivalence) |
| 0.9 | 0.08 / 0.04 | 0.81 / 0.89 | 0.005 / +0.001 | +0.047 | +0.006 [+0.001, +0.011] | 3 / 0 | I0 higher (lower bound 0.001); both within 0.01 of always-abstain in R |
| Hypothesis | Seeds | Key parts: value (one-sided 95% bound) | p | Supported (Holm, F1–F7) | |
| F1 | TCAT beats always-abstain at unseen .75 (U and D) | TCAT 3 | U: +0.004 (lower 0.003); D: +0.003 (lower 0.005) | 0.272 | no |
| F2 | TCAT reads unseen thresholds | TCAT 3 | .3 (b): 0.243 (lower 0.269); .6 (b): +0.091 (lower +0.076); .75 (b): 0.077 (lower 0.092) | 1.000 | no |
| F3 | TCAT selects beyond a surface policy | TCAT 3 | vs surface policy: +0.027 (lower +0.017) | 0.001 | yes |
| F4 | I1 calibrates and still ranks | I1 3 | bias gain: +0.499 (lower +0.442); Brier gain: +0.296 (lower +0.266); AUROC(I1) AUROC( ): 0.290 (lower 0.321) | 1.000 | no |
| F5 | no knowledge cost | TCAT 3; I1 seed 0 | TCAT OR@0 acc.: +0.015 (lower +0.001); TCAT MC acc.: +0.002 (lower 0.004); I1 OR@0 acc.: 0.012 (lower 0.027); I1 MC acc.: 0.006 (lower 0.015) | 0.190 | no |
| F6 | no rise in abstention under C and P | TCAT 3 | C: 0.066 (upper 0.051); P: 0.016 (upper 0.004) | 0.001 | yes |
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
| Sample | primary (Qwen3-30B) | gpt-oss-20b | rule | Claude (blind) |
| random [email protected] sample (n=240) | 97.1 (0.95) | 95.4 (0.93) | 83.8 (0.74) | 99.2 (0.99) |
| grader disagreements (n=60) | 86.7 (0.79) | 83.3 (0.75) | 10.0 ( 0.23) | 95.0 (0.92) |
| SFT stage (n=100) | 97.0 (0.95) | 94.0 (0.91) | 71.0 (0.55) | 98.0 (0.97) |
| PO stage (n=100) | 94.0 (0.91) | 91.0 (0.86) | 66.0 (0.47) | 97.0 (0.95) |
| Final stage (n=100) | 94.0 (0.91) | 94.0 (0.91) | 70.0 (0.54) | 100.0 (1.00) |
| Checkpoint | U([email protected]) | abstain .75 (%) | MC overconf. | s .9 | AUROC P(True) | AUROC P(IK) | AUROC |
| OLMo 3 SFT | 0.79 | 48 | 0.184 | 0.98 | 0.74 | 0.69 | 0.57 |
| OLMo 3 DPO | 0.28 | 70 | 0.262 | 0.97 | 0.76 | 0.71 | 0.76 |
| OLMo 3 Final | 0.23 | 70 | 0.271 | 0.93 | 0.81 | 0.71 | 0.79 |
| Tulu 3 SFT | 0.58 | 5 | 0.189 | 0.99 | 0.70 | 0.75 | 0.67 |
| Tulu 3 DPO | 0.42 | 6 | 0.252 | 0.98 | 0.74 | 0.73 | 0.72 |
| Tulu 3 Final | 0.48 | 5 | 0.274 | 0.99 | 0.75 | 0.68 | 0.71 |
| Model | own e2e | own dec. | ABST e2e | ABST dec. | P(True) thr. |
| OLMo 3 Final | 0.692 | 0.682 | 0.753 | 0.752 | 0.761 |
| Tulu 3 Final | 0.631 | 0.639 | 0.763 | 0.775 | 0.771 |
| OLMo 2 Final | 0.550 | 0.552 | 0.756 | 0.756 | 0.755 |
| SmolLM3 Final | 0.531 | 0.529 | 0.751 | 0.760 | 0.781 |
| OLMo 3.1 32B (8-bit) | 0.734 | 0.777 | 0.785 | 0.784 | 0.757 |
| OLMo 2 32B (8-bit) | 0.833 | 0.831 | 0.752 | 0.754 | 0.826 |
| t | m | base | acc | acc | acc [95%] | [95%] | end-to-end gap [95%] | answer-format effect [95%] |
| 0.5 | 37 | 0.47 | 0.849 | 0.887 | +0.038 [ 0.047, +0.108] | 0.0002 [ 0.004, +0.003] | +0.063 [ 0.036, +0.159] | 0.025 [ 0.089, +0.038] |
| 0.75 | 45 | 0.47 | 0.853 | 0.873 | +0.020 [ 0.059, +0.086] | 0.002 [ 0.007, +0.002] | +0.037 [ 0.053, +0.124] | 0.017 [ 0.075, +0.044] |
| 0.9 | 41 | 0.47 | 0.847 | 0.869 | +0.021 [ 0.056, +0.100] | 0.002 [ 0.006, +0.002] | +0.058 [ 0.036, +0.163] | 0.037 [ 0.090, +0.011] |
| Checkpoint | t = .5 | t = .75 | t = .9 |
| A. R = R(I0) R(ABST), end to end | |||
| OLMo 3 SFT | +0.030 [+0.002, +0.049] | 0.007 [ 0.018, +0.006] | +0.002 [ 0.003, +0.008] |
| OLMo 3 DPO | +0.039 [+0.012, +0.060] | 0.004 [ 0.019, +0.011] | 0.001 [ 0.010, +0.006] |
| OLMo 3 Final | +0.047 [+0.018, +0.073] | +0.002 [ 0.016, +0.016] | 0.002 [ 0.012, +0.006] |
| Tulu 3 SFT | +0.088 [+0.060, +0.123] | 0.010 [ 0.032, +0.014] | 0.005 [ 0.015, +0.001] |
| Tulu 3 DPO | +0.079 [+0.046, +0.110] | 0.021 [ 0.052, 0.002] | +0.003 [ 0.011, +0.019] |
| Model | U([email protected]) [95% CI] | ABST .9 .5 (pp) | s .9 | AUROC P(True) | R(.75) own | R(.75) ABST (500 items) | R(.75) P(True) thr. | |
| OLMo 3 Final | 0.23 [ 0.31, 0.16] | +0.2 | 0.09 | 0.93 | 0.81 | 0.692 | 0.753 | 0.761 |
| Tulu 3 Final | 0.48 [ 0.58, 0.37] | 4.2 | 0.05 | 0.99 | 0.75 | 0.631 | 0.763 | 0.771 |
| OLMo 2 Final | 0.80 [ 0.92, 0.69] | 0.4 | 0.01 | 0.84 | 0.66 | 0.550 | 0.756 | 0.755 |
| SmolLM3 Final | 0.88 [ 0.99, 0.76] | +1.8 | 0.03 | 0.64 | 0.83 | 0.531 | 0.751 | 0.781 |
| OLMo 3.1 32B (8-bit) | 0.07 [ 0.16, 0.02] | +0.8 | 0.13 | 0.82 | 0.74 | 0.734 | 0.785 | 0.757 |
| OLMo 2 32B (8-bit) | 0.33 [0.23, 0.42] | +0.6 | 0.25 | 0.96 | 0.82 | 0.833 | 0.752 | 0.826 |
| Arm | t | both ans. | grader abst., rule ans. | grader ans., rule abst. | both abst. | OR@0 correct (col. 4) | U | G | D | G D (coding) | U G (content) |
| REL | .6 | 209 | 105 | 77 | 609 | 0.18 | 0.009 | 0.039 | 0.156 | +0.117 | +0.030 |
| REL | .75 | 234 | 143 | 64 | 559 | 0.18 | 0.234 | 0.270 | 0.543 | +0.273 | +0.036 |
| I1 (seed 0) | .6 | 57 | 79 | 13 | 851 | 0.32 | +0.008 | 0.005 | 0.061 | +0.056 | +0.012 |
| I1 (seed 0) | .75 | 56 | 83 | 23 | 838 | 0.31 | 0.021 | 0.041 | 0.189 | +0.148 | +0.020 |
| Item | Fixed in advance | Applicability check | Status | Observed (descriptive) |
| Prediction (1): ‘applies’ | I1’s own decisions move toward I1+t | OLMo 3 Final 0.093 [0.078, 0.108]: band ‘in between’; needs point 0.30 | not applicable | – |
| Prediction (2): ‘does not apply’ | I1 own REL I1+t | OLMo 3 Final 0.093 [0.078, 0.108]: band ‘in between’; needs upper bound 0.10 | not applicable | description only: t = .6: I1 own REL: U +0.016 [ 0.021, +0.055] (includes 0), D +0.094 [+0.058, +0.131] (above 0); t = .75: I1 own REL: U +0.213 [+0.147, +0.278] (above 0), D +0.354 [+0.285, +0.426] (above 0); t = .6: I1+t I1 own: U 0.009 [ 0.029, +0.011] (includes 0), D +0.060 [+0.031, +0.090] (above 0); t = .75: I1+t I1 own: U +0.021 [ 0.009, +0.054] (includes 0), D +0.189 [+0.136, +0.244] (above 0) |
| Prediction (3): ‘material’ | after TCAT of REL | OLMo 3 Final 0.036; needs 0.05 | not applicable | – |
| Rubric wording | used only if F2 holds | F2 | not used (F2 not supported) | – |
| Self-knowledge wording | used only if both learners’ own labels beat swapped labels | selection gap, lower bound 0 | not used; described as a learned difficulty prior | OLMo 0.0008 [ 0.0028]; Qwen +0.0196 [+0.0106] |
| Recovery ratio U(TCAT)/U(I0) | reported only if U(I0)‘s one-sided lower bound 0, with a Fieller interval | U(I0) bound | condition met at .5, .6 and .75 | t = 0.5: U(I0) lower +0.082, ratio 0.60 [0.34, 0.92]; t = 0.6: U(I0) lower +0.047, ratio 0.20 [0.04, 0.43]; t = 0.75: U(I0) lower +0.012, ratio 0.10 [ 0.15, 0.68] |
| Model | abstain C / .75 / .9 (%) | U(.75) at [email protected] [95%] | U(.9) at [email protected] [95%] | abstain .9 .75 [95%] | correct C (%) | truncated C (%) |
| OLMo 3 Final | 20 / 88 / 86 | 0.36 [ 0.51, 0.24] | 1.17 [ 1.62, 0.76] | 0.015 [ 0.055, +0.025] | 6 | 27 |
| OLMo 2 Final | 2 / 11 / 12 | 2.47 [ 2.63, 2.29] | 7.33 [ 7.80, 6.82] | +0.015 [ 0.005, +0.035] | 6 | 0 |
| Tulu 3 Final | 4 / 24 / 22 | 1.95 [ 2.15, 1.73] | 6.17 [ 6.76, 5.58] | 0.030 [ 0.055, 0.010] | 11 | 1 |
| SmolLM3 Final | 6 / 18 / 20 | 2.27 [ 2.44, 2.07] | 6.65 [ 7.19, 6.10] | +0.015 [ 0.010, +0.040] | 6 | 2 |
| Gemini 3 Pro (Kalai et al.) | – | 0.05 | 1.50 | – | – | – |
| GPT-5 (Kalai et al.) | – | 0.31 | 1.34 | – | – | – |
| Model | abstain (%) greedy / sampled | abstain [95%] | U(.75) greedy / sampled | U [95%] | U per sample (range) | unanimous (%) | greedy = majority (%) |
| OLMo 3 Final | 69 / 66 | 0.028 [ 0.088, +0.036] | 0.29 / 0.40 | 0.108 [ 0.284, +0.074] | 0.58 0.30 | 45 | 82 |
| OLMo 2 Final | 4 / 5 | +0.008 [ 0.024, +0.040] | 0.76 / 1.10 | 0.336 [ 0.586, 0.098] | 1.21 0.93 | 82 | 95 |
| Tulu 3 Final | 3 / 8 | +0.046 [+0.018, +0.072] | 0.55 / 0.59 | 0.038 [ 0.226, +0.140] | 0.68 0.37 | 79 | 97 |
| SmolLM3 Final | 12 / 16 | +0.038 [ 0.014, +0.088] | 0.52 / 0.94 | 0.422 [ 0.652, 0.206] | 1.10 0.74 | 67 | 87 |
| Measure | Think (Final) | Instruct (same items) | RL-Zero General | Instruct (same items) |
| items | 100 | 100 | 200 | 200 |
| unfinished, C / .75 / .9 (%) | 18 / 17 / 22 | – | 1 / 1 / 1 | – |
| abstain, C / .75 / .9 (%) | 21 / 35 / 38 | 11 / 69 / 73 | 18 / 20 / 16 | 14 / 70 / 72 |
| U(.75) under [email protected] [95%] | 0.43 [ 0.73, 0.15] | 0.29 | 0.73 [ 0.96, 0.51] | 0.12 |
| U(.9) under [email protected] [95%] | 1.98 [ 2.78, 1.24] | 0.83 | 3.11 [ 3.74, 2.48] | 0.62 |
| abstain(.9) abstain(.75) [95%] | +0.030 [ 0.060, +0.120] | – | 0.035 [ 0.070, 0] | – |
| Measure | Mid | SFT | SFT Mid [95%] |
| truncated, [email protected] (%) | 99 | 8 | – |
| belief answers attempted (%) | 29 | 96 | – |
| stated confidence missing (%) | 98 | 1 | – |
| P(True) letter mass .5 (%) | 100 | 0 | – |
| abstain [email protected] / [email protected] (%) | 72 / 73 | 10 / 6 | 0.047 [ 0.074, 0.020] (of .9 .5) |
| U([email protected]) | +0.158 [+0.118, +0.198] | 1.083 | 1.241 [ 1.355, 1.139] |
| Checkpoint | c* no rule | c* t = .5 | c* t = .75 | c* t = .9 | slope .5 / .75 / .9 | deference share |
| OLMo 3 SFT | 0.20 [0.18, 0.23] | 0.44 [0.40, 0.49] | 0.51 [0.46, 0.56] | 0.51 [0.47, 0.57] | 0.66 / 0.61 / 0.61 | 0.98 |
| OLMo 3 DPO | 0.32 [0.28, 0.36] | 0.52 [0.48, 0.57] | 0.61 [0.56, 0.65] | 0.65 [0.60, 0.70] | 0.97 / 0.94 / 0.90 | 0.89 |
| OLMo 3 Final | 0.33 [0.29, 0.37] | 0.55 [0.50, 0.61] | 0.64 [0.59, 0.69] | 0.68 [0.62, 0.73] | 0.98 / 0.96 / 0.89 | 0.92 |
| Tulu 3 SFT | 0.06 † [0.04, 0.08] | 0.19 † [0.15, 0.24] | 0.13 † [0.09, 0.17] | 0.10 † [0.07, 0.15] | 0.22 / 0.17 / 0.14 | 0.55 |
| Tulu 3 DPO | 0.02 † [0.01, 0.03] | 0.14 † [0.11, 0.18] | 0.08 † [0.06, 0.11] | 0.06 † [0.04, 0.08] | 0.43 / 0.35 / 0.29 | 0.52 |
| Tulu 3 Final | 0.01 † [0.01, 0.02] | 0.11 † [0.09, 0.15] | 0.07 † [0.05, 0.09] | 0.05 † [0.03, 0.07] | 0.49 / 0.39 / 0.33 | 0.52 |
| Checkpoint | contextual | abstract | share s .9 | (H3) | determ. | abstain [email protected] determ. (%) | non- monotone verbatim (%) | non- monotone ABST (%) |
| OLMo 3 SFT | 0.037 | 0.171 | 0.98 | 0.001 | 0.100 | 35.7 | 30.2 | 0.2 |
| OLMo 3 DPO | 0.085 | 0.139 | 0.97 | 0.008 | 0.080 | 72.5 | 36.8 | 0.8 |
| OLMo 3 Final | 0.093 | 0.155 | 0.93 | 0.130 | 0.187 | 62.3 | 40.1 | 0.8 |
| Tulu 3 SFT | 0.030 | 0.300 | 0.99 | not identified | not identified | 0.3 | 8.5 | 3.6 |
| Tulu 3 DPO | 0.048 | 0.513 | 0.98 | 0.186 | 0.000 | 0.0 | 8.6 | 7.6 |
| Tulu 3 Final | 0.053 | 0.479 | 0.99 | not identified | 0.002 | 0.2 | 8.2 | 6.2 |
| Checkpoint | agree [email protected] (%) | same answer | realized | same answer | realized | determ. | determ. | ||
| OLMo 3 SFT | 35.8 | +0.155 | +0.235 | +0.156 | +0.125 | +0.251 | +0.125 | +0.245 | +0.214 |
| OLMo 3 DPO | 0.7 | +0.071 | +0.017 | +0.072 | +0.042 | 0.006 | +0.042 | +0.107 | +0.095 |
| OLMo 3 Final | 0.0 | +0.076 | +0.083 | +0.050 | +0.036 | +0.064 | +0.010 | +0.136 | +0.118 |
| Tulu 3 SFT | 65.8 | +0.178 | +0.165 | +0.267 | +0.164 | +0.268 | +0.253 | – | – |
| Tulu 3 DPO | 22.7 | +0.049 | 0.143 | +0.186 | 0.131 | +0.143 | +0.006 | – | – |
| Tulu 3 Final | 17.8 | +0.087 | +0.500 | +0.267 | 0.194 | +0.400 | 0.015 | +0.696 | +0.261 |
| Checkpoint | truncated (%) | U([email protected]) as graded | U, truncated as abstention | U, truncated as wrong | MC letter mass .5 (%) |
| OLMo 3 SFT | 1.2 | 0.79 | 0.78 | 0.82 | 10.2 |
| OLMo 3 DPO | 5.5 | 0.28 | 0.21 | 0.37 | 25.2 |
| OLMo 3 Final | 11.2 | 0.23 | 0.14 | 0.48 | 21.6 |
| Tulu 3 SFT | 0.1 | 0.58 | 0.58 | 0.58 | 15.8 |
| Tulu 3 DPO | 0.8 | 0.42 | 0.43 | 0.45 | 37.3 |
| Tulu 3 Final | 0.8 | 0.48 | 0.48 | 0.50 | 24.7 |
| Recipe or lineage | overconfidence PO SFT | temperature SFT PO | ECE raw change | ECE after scaling, change | AUROC change |
| OLMo 3 | +0.078 | 2.19 3.57 | +0.084 | 0.006 | 0.015 |
| OLMo 2 | +0.135 | 1.49 2.66 | +0.129 | 0.004 | 0.008 |
| Tulu 3 | +0.064 | 2.13 3.12 | +0.062 | +0.006 | 0.016 |
| SmolLM3 | +0.094 | 1.95 3.14 | +0.086 | 0.015 | +0.007 |
| OLMo 3 preference data | +0.078 [+0.065, +0.093] | – | – | – | – |
| Tulu 3 preference data | +0.097 [+0.088, +0.107] | – | – | – | – |
| Arm | [email protected] (trained) | [email protected] (unseen) | BRIEF @.75 | PEN-1 (t = .5) | PEN-3 (t = .75) | PEN-4 (t = .8) | PARA @.75A | PARA @.75B | ABST @.75 |
| REL | 21.1 | 37.7 | 96.8 | 40.4 | 42.2 | 45.6 | 2.4 | 92.8 | 3.0 |
| TCAT seed 0 | 31.4 | 1.1 | 94.0 | 37.8 | 36.8 | 40.8 | 1.8 | 35.6 | 7.4 |
| TCAT seed 1 | 29.7 | 1.0 | 92.6 | 31.6 | 31.8 | 35.6 | 1.0 | 46.6 | 6.6 |
| TCAT seed 2 | 33.3 | 1.1 | 94.2 | 37.4 | 38.8 | 41.6 | 2.2 | 29.8 | 5.0 |
| t | t OLMo 3 (%) | t Qwen2.5 (%) | regret vs I0-k (coverage / selection) | regret vs I0 (coverage / selection) | validation answered / target (%) |
| 0.3 | 79.2 | 86.4 | – | – | – |
| 0.5 | 55.9 | 68.5 | +0.003 (+0.000 / +0.003) | +0.051 ( 0.003 / +0.054) | 40.0/48.0; 38.0/44.0; 44.0/60.0; 66.0/60.0 |
| 0.6 | 43.9 | 61.6 | +0.022 (+0.011 / +0.010) | +0.068 (+0.054 / +0.013) | – |
| 0.75 | 23.3 | 50.0 | +0.014 (+0.012 / +0.002) | +0.033 (+0.035 / 0.002) | – |
| 0.9 | 3.3 | 29.0 | – | – | 0.0/2.0; 4.0/2.0; 0.0/2.0; 22.0/26.0 |
| Adapter | answers | distinct values | most common value (share) | validation MAE (25/50/75/100%) |
| OLMo 3 I1 seed 0 | 1000 | 3 | 0.29 (99.7%) | 27.6 17.7 16.1 28.5 |
| OLMo 3 I1 seed 1 | 1000 | 4 | 0.29 (98.9%) | 24.5 23.5 20.4 26.7 |
| OLMo 3 I1 seed 2 | 1000 | 3 | 0.30 (62.8%) | 25.3 19.6 24.4 20.2 |
| Qwen2.5 I1 seed 0 | 500 | 14 | 0.46 (29.0%) | 10.4 10.7 8.7 7.2 |
| part | pooled | per seed (0; 1; 2) | every seed passes | |
| F1 | U | +0.004 (lower 0.003) | +0.003 (lower 0.005); +0.006 (lower 0.001); +0.002 (lower 0.006) | no |
| F1 | D | +0.003 (lower 0.005) | +0.003 (lower 0.005); +0.006 (lower 0.001); 0.001 (lower 0.011) | no |
| F2 | .3 (a) below | +0.819 (lower +0.791) | +0.796 (lower +0.776); +0.843 (lower +0.825); +0.819 (lower +0.799) | yes |
| F2 | .3 (a) above | 0.145 (lower 0.169) | 0.124 (lower 0.141); 0.149 (lower 0.166); 0.163 (lower 0.181) | no |
| F2 | .3 (b) | 0.243 (lower 0.269) | 0.236 (lower 0.261); 0.250 (lower 0.276); 0.244 (lower 0.269) | no |
| F2 | .3 (c) | +0.093 (lower +0.075) | +0.101 (lower +0.083); +0.085 (lower +0.068); +0.093 (lower +0.076) | yes |
| Checkpoint | s, t = .75 agree / over-ans. / over-abst. | s, t = .9 | , t = .75 | , t = .9 | items |
| OLMo 3 SFT | 53 / 1 / 46 | 61 / 1 / 38 | 54 / 1 / 45 | 58 / 33 / 9 | 972 |
| OLMo 3 DPO | 31 / 0 / 68 | 34 / 1 / 65 | 32 / 1 / 68 | 43 / 5 / 51 | 973 |
| OLMo 3 Final | 34 / 1 / 65 | 39 / 1 / 60 | 35 / 1 / 64 | 50 / 6 / 45 | 948 |
| Tulu 3 SFT | 96 / 1 / 3 | 95 / 1 / 4 | 95 / 2 / 3 | 92 / 5 / 3 | 977 |
| Tulu 3 DPO | 96 / 1 / 3 | 95 / 1 / 4 | 96 / 1 / 3 | 94 / 2 / 4 | 964 |
| Tulu 3 Final | 96 / 1 / 3 | 96 / 1 / 3 | 96 / 1 / 3 | 96 / 1 / 3 | 981 |
| Checkpoint | [email protected] decl. & graded | [email protected] graded only (trunc.) | [email protected] decl. only | [email protected] decl. & graded | [email protected] discord. | gpt-oss abst. | gpt-oss correct. | rule abst. | rule correct. |
| OLMo 3 SFT | 35.1 | 12.6 (3.2) | 0.6 | 97.8 | 0.2 | 98.2 | 98.3 | 92.0 | 93.2 |
| OLMo 3 DPO | 60.7 | 9.8 (10.2) | 11.8 | 96.8 | 0.4 | 96.7 | 99.0 | 88.7 | 93.8 |
| OLMo 3 Final | 55.9 | 14.3 (24.5) | 6.4 | 96.8 | 0.4 | 95.3 | 92.2 | 86.9 | 90.7 |
| Tulu 3 SFT | 0.1 | 4.9 (2.0) | 0.2 | 92.4 | 0.2 | 98.2 | 99.5 | 95.6 | 90.4 |
| Tulu 3 DPO | 0.0 | 6.1 (3.3) | 0.0 | 70.0 | 0.6 | 97.5 | 97.2 | 95.6 | 93.1 |
| Tulu 3 Final | 0.1 | 5.1 (3.9) | 0.1 | 75.8 | 0.8 | 97.8 | 99.1 | 94.9 | 94.8 |
| Calibrated on tested on | t | coverage I0 | I0 | ABST | chosen | R = I0 ABST [95% CI] (checkpoints 0 / 0) | R = I0 chosen [95% CI] |
| TriviaQA PopQA | 0.5 | 0.53 | 0.011 | +0.013 | 0.062 | 0.024 [ 0.061, +0.004] (1/3) | +0.051 [+0.029, +0.075] |
| TriviaQA PopQA | 0.75 | 0.29 | 0.066 | +0.005 | +0.005 | 0.071 [ 0.091, 0.033] (0/6) | 0.071 [ 0.086, 0.002] |
| TriviaQA PopQA | 0.9 | 0.03 | 0.002 | 0.002 | +0.0002 | 0.0003 [ 0.023, +0.002] (0/1) | 0.002 [ 0.024, +0.001] |
| PopQA TriviaQA | 0.5 | 0.45 | +0.142 | +0.038 | +0.034 | +0.104 [+0.076, +0.124] (9/0) | +0.107 [+0.078, +0.127] |
| PopQA TriviaQA | 0.75 | 0.11 | +0.015 | +0.012 | +0.011 | +0.003 [ 0.007, +0.015] (2/1) | +0.004 [ 0.004, +0.016] |
| PopQA TriviaQA | 0.9 | 0.06 | +0.002 | 0.009 | 0.005 | +0.011 [+0.0003, +0.020] (4/0) | +0.007 [ 0.002, +0.014] |
| t | coverage TCAT-dense | coverage TCAT | coverage I0 | U TCAT-dense [95% CI] | D TCAT-dense | |
| 0.3 | trained | 63.8 | 16.9 | 67.8 | +0.144 [+0.107, +0.178] | +0.137 |
| 0.4 | unseen | 53.7 | – | 47.0 | +0.084 [+0.047, +0.122] | +0.075 |
| 0.5 | trained | 27.2 | 31.5 | 36.4 | +0.065 [+0.032, +0.095] | +0.058 |
| 0.6 | unseen | 18.9 | 3.9 | 26.6 | +0.048 [+0.017, +0.080] | +0.039 |
| 0.7 | trained | 6.8 | – | 18.2 | +0.021 [ 0.002, +0.043] | +0.011 |
| 0.75 | unseen | 4.7 | 1.1 | 10.0 | 0.001 [ 0.026, +0.021] | 0.001 |
LLM Abstention
Large language model abstention studies when AI systems should answer, ask for more information, defer, or withhold unsupported claims.
See all 110 papers