Learning a calibrated decision head from logged bandit feedback
Organizations: Independent Researcher · Ph.D. in Physics, University College Dublin.
Abstract
A decision head reads a state and a list of options and returns a probability per option in one forward pass. Once deployed, it usually observes only whether the option it chose was right: logged bandit feedback. We adapt such a head, LoRA on Qwen3-0.6B-Base and 1.7B-Base, from that feedback and report three results. First, the Brier or log score of the sampled action's own probability is not a usable REINFORCE reward: every wrong action earns at least gold's reward, with a mean baseline the lowest-probability wrong action gets the largest advantage, and at two options the signal is zero; in one run it cut accuracy on ordered-scale questions from .636 to .111. Scoring the probability as a loss by the pathwise gradient avoids the collapse, as do the REINFORCE rewards r and r minus p. Second, with full labels, none of four tested bandit arms beat cross-entropy plus one temperature on mean expected calibration error (ECE) over four unseen label schemas. Third, from logged feedback alone, Chosen-Action Proper Scoring (CAPS), the direct method with a proper score on the chosen action's probability, beats fine-tuning on confirmed answers only, which discards negatives: on Banking77 by 2.6 accuracy points, .053 lower ECE and lower selective risk (three seeds against one baseline seed), and by 21.7 points on one seed on emotion data with an often-wrong logger. At N 5000, CAPS stays 1.5 points below full-label training at 0.6B and .6 points at 1.7B against one full-label seed, and propensity weights add nothing on this log. Under feedback flipped at a known rate, one seed per rate, CAPS's confidence tracks the noisy channel on average, backward correction recovers most of the clean calibration, and early stopping on held-out logged outcomes prevented memorising noisy draws in all three runs. Code, adapters, logs and per-record results are public.
Figures & tables
| learner | accuracy | ECE | AURC |
|---|---|---|---|
| Banking77, 0.6B | |||
| zero-shot | .851 | .063 | .0333 |
| temperature | .851 | .020 | .0328 |
| positive-only SFT | .905 † | .078 † | .0202 † |
| CAPS-Brier | .931 (.929, .935) | .025 (.019, .031) | .0107 (.0104, .0110) |
| full-label SFT | .946 (.943, .949) | .022 (.016, .030) | .0067 (.0063, .0071) |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| N | metric | zero-shot | temperature | positive-only SFT | full-label SFT | CAPS-Brier |
|---|---|---|---|---|---|---|
| 500 | accuracy | .851 [.839, .863] | .851 | .886 [.874, .896] | .904 [.894, .914] | .888 [.877, .898] |
| 2000 | accuracy | .851 | .851 | .890 [.879, .901] | .924 [.915, .933] | .917 [.907, .926] |
| 5000 | accuracy | .851 | .851 | .905 [.896, .915] | .947 [.938, .955] | .929 [.920, .938] |
| 500 | ECE | .063 [.054, .074] | .023 [.021, .036] | .091 [.082, .102] | .043 [.035, .052] | .031 [.026, .044] |
| 2000 | ECE | .063 | .018 [.017, .032] | .082 [.072, .092] | .017 [.013, .027] | .022 [.017, .031] |
| 5000 | ECE | .063 | .020 [.016, .032] | .078 [.069, .087] | .030 [.024, .038] | .026 [.020, .034] |
| N | vs | accuracy | ECE | Brier | NLL | AURC | [email protected] |
|---|---|---|---|---|---|---|---|
| 500 | positive-only SFT | +.002 [ .006, +.009] | .060 [ .066, .048] | .027 [ .037, .018] | .373 [ .426, .319] | .0035 [ .0066, .0007] | .016 [ .027, +.014] |
| 2000 | positive-only SFT | +.027 [+.019, +.035] | .060 [ .067, .050] | .058 [ .069, .048] | .323 [ .373, .276] | .0099 [ .0131, .0070] | +.065 [+.048, +.093] |
| 5000 | positive-only SFT | +.024 [+.016, +.031] | .052 [ .060, .043] | .058 [ .069, .046] | .389 [ .443, .333] | .0092 [ .0118, .0068] | +.069 [+.046, +.094] |
| 500 | full-label SFT | .017 [ .023, .011] | .011 [ .016, .001] | +.024 [+.018, +.030] | +.052 [+.036, +.070] | +.0038 [+.0018, +.0058] | .048 [ .063, .028] |
| 2000 | full-label SFT | .007 [ .014, +.000] | +.005 [ .002, +.011] | +.013 [+.007, +.019] | +.062 [+.046, +.078] | +.0026 [+.0011, +.0043] | .024 [ .035, .002] |
| 5000 | full-label SFT | .018 [ .025, .011] | .004 [ .010, +.003] | +.029 [+.020, +.038] | +.062 [+.037, +.086] | +.0047 [+.0031, +.0063] | .039 [ .051, .024] |
| run | in-domain accuracy | in-domain ECE | unseen-intents accuracy | unseen-intents ECE | selected step (epochs) | kept at .95 (accuracy) |
|---|---|---|---|---|---|---|
| zero-shot | .956 [.952, .960] | .013 [.011, .017] | .892 [.883, .900] | .068 [.062, .077] | .607 (.978) | |
| temperature | +.000 [+.000, +.000] | .000 [ .005, +.005] | +.000 [+.000, +.000] | .026 [ .029, .024] | .189 (.998) | |
| positive-only SFT | .008 [ .011, .006] | +.017 [+.014, +.019] | +.020 [+.015, +.025] | +.001 [ .004, +.006] | 200 (1.9 of 13.4) | .916 (.942) |
| IPS positive-only SFT | .009 [ .011, .006] | +.017 [+.014, +.019] | +.023 [+.017, +.028] | .001 [ .007, +.004] | 200 (1.9 of 13.4) | .917 (.941) |
| CAPS-Brier seed 0 | .027 [ .031, .023] | .001 [ .004, +.004] | +.025 [+.018, +.032] | .031 [ .037, .023] | 300 (2.1 of 10.0) | .743 (.986) |
| CAPS-Brier seed 1 | .043 [ .048, .039] | +.010 [+.006, +.015] | +.020 [+.012, +.027] | .026 [ .033, .018] | 550 (3.9 of 10.0) | .794 (.984) |
| learner | accuracy | ECE | Brier | NLL | AURC | [email protected] |
|---|---|---|---|---|---|---|
| CAPS-Brier | .931 (.929, .935) | .025 (.019, .031) | .109 (.105, .111) | .2739 (.2737, .2742) | .0107 (.0104, .0110) | .962 (.957, .971) |
| CAPS-log | .933 (.931, .934) | .022 (.019, .027) | .105 (.101, .110) | .254 (.239, .275) | .0099 (.0089, .0113) | .966 (.959, .970) |
| full-label SFT | .946 (.943, .949) | .022 (.016, .030) | .082 (.079, .086) | .192 (.175, .212) | .0067 (.0063, .0071) | .994 (.986, .999) |
| CAPS-Brier, second log draw | .934 [.925, .943] | .026 [.022, .034] | .103 [.090, .115] | .256 [.221, .293] | .0102 [.0076, .0128] | .969 [.949, .989] |
| seed mean minus | accuracy | ECE | Brier | NLL | AURC | [email protected] |
|---|---|---|---|---|---|---|
| CAPS-Brier minus positive-only SFT | +.026 [+.019, +.033] | .053 [ .059, .044] | .060 [ .071, .048] | .389 [ .443, .333] | .0095 [ .0121, .0070] | +.074 [+.052, +.098] |
| per-seed range | (+.024, +.030) | ( .059, .048) | ( .064, .058) | ( .3891, .3886) | ( .0097, .0092) | (+.069, +.083) |
| CAPS-Brier minus full-label SFT | .015 [ .020, .010] | +.003 [ .001, +.008] | +.027 [+.020, +.034] | +.082 [+.061, +.101] | +.0040 [+.0026, +.0053] | .032 [ .041, .019] |
| seed ranges | apart | overlap: unresolved | apart | apart | apart | apart |
| CAPS-log minus CAPS-Brier | +.002 [ .001, +.005] | .003 [ .006, +.000] | .005 [ .008, .001] | .020 [ .029, .010] | .0008 [ .0016, +.0000] | +.004 [ .001, +.010] |
| seed ranges | overlap: unresolved | overlap: unresolved | overlap: unresolved | overlap: unresolved | overlap: unresolved | overlap: unresolved |
| N | learner | accuracy | ECE | Brier | NLL | AURC | [email protected] | accuracy minus CAPS-Brier | weights clipped |
|---|---|---|---|---|---|---|---|---|---|
| 2000 | positive-only SFT | .890 [.879, .901] | .082 [.072, .092] | .185 [.167, .203] | .622 [.551, .692] | .0230 [.0186, .0276] | .857 [.819, .888] | ||
| 2000 | IPS positive-only SFT | .892 [.881, .902] | .077 [.068, .087] | .183 [.165, .200] | .593 [.525, .659] | .0233 [.0189, .0282] | .856 [.819, .891] | .025 [ .032, .018] | 14 of 1332 |
| 2000 | CAPS-log | .907 [.897, .917] | .014 [.012, .025] | .140 [.128, .153] | .319 [.289, .351] | .0161 [.0132, .0194] | .892 [.865, .926] | .010 [ .016, .004] | |
| 2000 | IPS CAPS-log | .906 [.897, .916] | .012 [.010, .024] | .143 [.129, .156] | .323 [.291, .357] | .0168 [.0137, .0202] | .892 [.858, .916] | .011 [ .016, .005] | 216 of 1800 |
| 2000 | REINFORCE, reward | .916 [.906, .925] | .030 [.024, .040] | .132 [.118, .146] | .325 [.287, .365] | .0147 [.0116, .0180] | .919 [.894, .939] | .001 [ .006, +.003] | 132 of 17800 |
| 2000 | CAPS-Brier | .917 [.907, .926] | .022 [.017, .031] | .127 [.113, .140] | .299 [.264, .332] | .0132 [.0104, .0162] | .922 [.900, .946] |
| f | learner | accuracy | minus clean | ECE, uncorrected | ECE, backward-corrected | loss minus positive-only SFT’s |
|---|---|---|---|---|---|---|
| 0 | CAPS-Brier | .929 [.920, .938] | .026 [.020, .034] | |||
| 0.1 | CAPS-Brier | .923 [.914, .932] | .006 [ .013, +.001] | .059 [.055, .069] | .020 [.017, .030] | +.009 [+.000, +.018] |
| 0.2 | CAPS-Brier | .907 [.897, .916] | .022 [ .030, .014] | .164 [.157, .173] | .051 [.042, .059] | +.017 [+.007, +.028] |
| 0.3 | CAPS-Brier | .888 [.877, .898] | .041 [ .050, .032] | .214 [.206, .225] | .068 [.058, .078] | .003 [ .014, +.007] |
| 0 | positive-only SFT | .905 [.896, .915] | .078 [.069, .087] | |||
| 0.1 | positive-only SFT | .890 [.880, .901] | .015 [ .021, .008] | .046 [.039, .057] | .071 [.062, .082] |
| f | learner | channel-scale ECE | reference | mean top-1 | channel line | selected step | final-step accuracy | criterion growth |
|---|---|---|---|---|---|---|---|---|
| 0 | CAPS-Brier | 300 | .933 | .059 | ||||
| 0.1 | CAPS-Brier | .038 [.035, .046] | .021 | .870 | .838 | 450 | .892 | .114 |
| 0.2 | CAPS-Brier | .058 [.052, .063] | .014 | .743 | .744 | 250 | .813 | .284 |
| 0.3 | CAPS-Brier | .069 [.065, .073] | .009 | .674 | .655 | 250 | .734 | .757 |
| 0 | positive-only SFT | 200 | .268 | |||||
| 0.1 | positive-only SFT | .124 [.118, .133] | .063 | .935 | .812 | 100 | 1.244 |
| learner | 1.7B accuracy | 1.7B ECE | 1.7B Brier | 1.7B NLL | 1.7B AURC | 1.7B [email protected] | 0.6B accuracy, seed 0 | 0.6B accuracy, seed mean (range) | 0.6B ECE, seed mean |
|---|---|---|---|---|---|---|---|---|---|
| zero-shot | .848 [.836, .860] | .089 [.078, .100] | .242 | .660 | .0350 [.0301, .0401] | .731 [.699, .763] | .851 | .063 | |
| temperature | .848 | .031 [.026, .044] | .222 | .519 | .0333 [.0285, .0382] | .750 [.719, .774] | .851 | .020 | |
| positive-only SFT | .913 [.902, .923] | .067 [.058, .077] | .154 | .545 | .0177 [.0139, .0218] | .906 [.877, .933] | .905 | (seed 0 only) | .078 |
| CAPS-Brier | .949 [.941, .957] | .021 [.015, .028] | .079 | .188 | .0066 [.0049, .0084] | .998 [.981, 1.000] | .929 | .931 (.929, .935) | .025 |
| full-label SFT | .953 [.945, .960] | .023 [.018, .030] | .072 | .162 | .0053 [.0037, .0071] | 1.000 [.991, 1.000] | .947 | .946 (.943, .949) | .022 |
| 1.7B paired difference | accuracy | ECE | Brier | NLL | AURC | [email protected] |
|---|---|---|---|---|---|---|
| CAPS-Brier seed 0 minus positive-only SFT | +.036 [+.028, +.045] | .046 [ .054, .037] | .075 [ .088, .061] | .357 [ .411, .306] | .0111 [ .0145, .0083] | +.093 [+.065, +.115] |
| CAPS-Brier seed 0 minus full-label SFT | .004 [ .010, +.002] | .002 [ .008, +.004] | +.007 [+.001, +.013] | +.026 [+.013, +.040] | +.0013 [+.0004, +.0022] | .002 [ .015, +.001] |
| CAPS-Brier seed mean minus positive-only SFT | +.034 [+.026, +.041] | .051 [ .057, .040] | .070 [ .083, .057] | .353 [ .409, .303] | .0104 [ .0135, .0078] | +.088 [+.062, +.108] |
| per-seed range | (+.029, +.036) | ( .054, .046) | ( .075, .061) | ( .362, .340) | ( .0111, .0092) | (+.078, +.093) |
| CAPS-Brier seed mean minus full-label SFT | .006 [ .011, .002] | .007 [ .011, +.000] | +.012 [+.006, +.017] | +.030 [+.018, +.043] | +.0021 [+.0012, +.0030] | .007 [ .019, .000] |
| per-seed range | ( .011, .004) | ( .010, .002) | (+.007, +.021) | (+.020, +.042) | (+.0013, +.0032) | ( .017, .002) |
| run | unseen mean ECE (seeds 0, 1, 2) | minus SFT + T | with its own temperature, minus SFT + T |
|---|---|---|---|
| SFT | .152 | ||
| SFT + T | .121 | ||
| continued cross-entropy | .171 (.185, .172, .156) | +.050 [+.044, +.056] (+.034, +.064) | +.002 [ .001, +.009] ( .006, +.011) |
| REINFORCE, outcome | .248 (.262, .244, .239) | +.127 [+.118, +.135] (+.118, +.141) | +.025 [+.018, +.034] (+.019, +.035) |
| REINFORCE, | .153 (.173, .155, .131) | +.031 [+.026, +.038] (+.010, +.051) | .002 [ .006, +.004] ( .010, +.007) |
| CAPS-Brier | .159 (.173, .165, .139) | +.037 [+.032, +.043] (+.017, +.051) | +.001 [ .004, +.007] ( .005, +.006) |
| run | in-domain accuracy | SST-5 score accuracy | Yelp score accuracy | SST-5 score, minus SFT | Yelp score, minus SFT |
|---|---|---|---|---|---|
| SFT | .956 [.952, .960] | .636 [.616, .655] | .505 [.474, .536] | ||
| REINFORCE, detached Brier reward | .948 [.943, .952] | .111 [.098, .124] | .044 [.032, .057] | .525 [ .553, .502] | .461 [ .496, .428] |
| continued cross-entropy | .957 [.953, .960] (.954, .959) | .626 [.608, .644] (.621, .631) | .499 [.471, .528] (.479, .515) | .010 [ .019, .002] ( .015, .005) | .006 [ .018, +.005] ( .026, +.010) |
| REINFORCE, outcome | .953 [.950, .957] (.948, .958) | .623 [.605, .640] (.609, .633) | .494 [.466, .522] (.479, .505) | .014 [ .023, .005] ( .027, .004) | .011 [ .023, +.002] ( .026, +.000) |
| REINFORCE, | .957 [.953, .960] (.952, .959) | .624 [.607, .642] (.617, .635) | .505 [.477, .533] (.486, .521) | .012 [ .021, .003] ( .019, .001) | .000 [ .011, +.011] ( .019, +.016) |
| CAPS-Brier | .957 [.953, .960] (.954, .959) | .622 [.604, .640] (.620, .626) | .506 [.477, .536] (.501, .510) | .014 [ .023, .005] ( .016, .010) | +.001 [ .008, +.012] ( .004, +.005) |
| questions | n | lowered | raised | unchanged | all samples identical | distinct, equal rewards |
|---|---|---|---|---|---|---|
| all | 42422 | 24.0 | 1.8 | 74.2 | 63.7 | 10.4 |
| yes-or-no | 20399 | 0.0 | 0.0 | 100.0 | 78.4 | 21.6 |
| choice | 13482 | 30.8 | 0.9 | 68.4 | 67.9 | 0.0 |
| score | 8541 | 70.5 | 7.7 | 21.8 | 21.8 | 0.0 |
| N | metric | zero-shot | temperature | positive-only SFT | full-label SFT | CAPS-Brier |
|---|---|---|---|---|---|---|
| 500 | accuracy | .540 [.518, .562] | .540 | .578 [.555, .599] | .659 [.640, .681] | .621 [.601, .642] |
| 2000 | accuracy | .540 | .540 | .589 [.567, .611] | .760 [.742, .779] | .715 [.695, .734] |
| 5000 | accuracy | .540 | .540 | .615 [.592, .636] | .871 [.856, .884] | .832 [.816, .848] |
| 500 | ECE | .251 [.232, .272] | .031 [.024, .056] | .363 [.342, .385] | .066 [.057, .091] | .087 [.075, .110] |
| 2000 | ECE | .251 | .027 [.021, .053] | .324 [.304, .344] | .056 [.043, .075] | .066 [.049, .085] |
| 5000 | ECE | .251 | .024 [.020, .051] | .256 [.237, .277] | .041 [.031, .055] | .070 [.059, .087] |
| N | vs | accuracy | ECE | Brier | NLL | AURC | [email protected] |
|---|---|---|---|---|---|---|---|
| 500 | positive-only SFT | +.043 [+.027, +.060] | .276 [ .290, .254] | .230 [ .257, .206] | 1.993 [ 2.138, 1.841] | .050 [ .064, .036] | +.066 [+.019, +.101] |
| 2000 | positive-only SFT | +.126 [+.109, +.144] | .258 [ .282, .232] | .286 [ .313, .261] | 1.379 [ 1.482, 1.273] | .106 [ .121, .093] | +.077 [+.036, +.182] |
| 5000 | positive-only SFT | +.217 [+.197, +.238] | .186 [ .204, .162] | .364 [ .396, .330] | 1.042 [ 1.134, .947] | .174 [ .195, .154] | +.564 [+.439, +.632] |
| 500 | full-label SFT | .038 [ .053, .025] | +.021 [+.001, +.037] | +.053 [+.041, +.064] | +.174 [+.146, +.204] | +.022 [+.014, +.030] | .005 [ .035, +.025] |
| 2000 | full-label SFT | .046 [ .060, .033] | +.010 [ .009, +.027] | +.076 [+.066, +.087] | +.180 [+.154, +.206] | +.047 [+.039, +.054] | .170 [ .242, .030] |
| 5000 | full-label SFT | .039 [ .052, .027] | +.029 [+.017, +.043] | +.070 [+.056, +.088] | +.216 [+.178, +.261] | +.027 [+.020, +.035] | .167 [ .260, .100] |
| run | accuracy | ECE | Brier | NLL | AURC | [email protected] | minus seed 0, accuracy | minus seed 0, ECE |
|---|---|---|---|---|---|---|---|---|
| seed 0 | .832 [.816, .848] | .070 [.059, .087] | .264 | .599 | .058 | .595 | ||
| seed 1 | .829 [.812, .845] | .031 [.024, .049] | .261 | .567 | .059 | .556 | .003 [ .014, +.008] | .039 [ .048, .025] |
| seed 2 | .821 [.803, .837] | .038 [.028, .055] | .266 | .573 | .058 | .572 | .011 [ .022, +.000] | .032 [ .042, .020] |
| mean of 3 training seeds (range) | .827 (.821, .832) | .046 (.031, .070) | .263 (.261, .266) | .580 (.567, .599) | .059 (.058, .059) | .575 (.556, .595) |
| seed mean minus | accuracy | ECE | Brier | NLL | AURC | [email protected] |
|---|---|---|---|---|---|---|
| positive-only SFT | +.213 [+.193, +.233] | .210 [ .227, .186] | .364 [ .395, .334] | 1.061 [ 1.150, .968] | .174 [ .193, .154] | +.543 [+.409, +.618] |
| per-seed range | (+.206, +.217) | ( .225, .186) | ( .367, .362) | ( 1.074, 1.042) | ( .174, .173) | (+.525, +.564) |
| full-label SFT | .043 [ .056, .031] | +.006 [ .005, +.018] | +.070 [+.056, +.085] | +.197 [+.164, +.235] | +.028 [+.022, +.034] | .188 [ .297, .128] |
| per-seed range | ( .050, .039) | ( .010, +.029): unresolved | (+.068, +.072) | (+.184, +.216) | (+.027, +.028) | ( .206, .167) |
| run | in-domain accuracy | in-domain ECE | unseen-intents accuracy | unseen-intents ECE |
|---|---|---|---|---|
| positive-only SFT | .005 [ .007, .002] | +.007 [+.004, +.009] | +.010 [+.006, +.013] | .003 [ .007, +.000] |
| CAPS-Brier seed 0 | .026 [ .031, .023] | +.001 [ .004, +.004] | .020 [ .027, .013] | +.002 [ .005, +.009] |
| CAPS-Brier seed 1 | .019 [ .023, .015] | +.003 [ .002, +.009] | .024 [ .031, .017] | .017 [ .025, .011] |
| CAPS-Brier seed 2 | .024 [ .027, .020] | .003 [ .007, +.004] | .034 [ .042, .026] | +.000 [ .008, +.007] |
| CAPS-log | .022 [ .026, .019] | .001 [ .006, +.003] | .011 [ .018, .004] | .013 [ .020, .007] |
| full-label SFT | .034 [ .038, .029] | .002 [ .007, +.001] | .040 [ .048, .032] | +.011 [+.003, +.018] |
| No. | Item | File: keys | Report |
|---|---|---|---|
| T 1 | main-text summary at N 5000 | S: , L: and E: n_curve.5000.{zero_shot,temperature,positive_sft,full_sft}.{accuracy,ece,aurc}; seeds.training_seeds.{accuracy,ece,aurc}; seeds.full_sft.training_seeds.{accuracy,ece,aurc}; in S: only, seeds.positive_sft.training_seeds.{accuracy,ece,aurc}; each read only where it holds more than one seed ( scripts/main_table.py ) | R3 §§3, 12, 13 |
| T A.1.1 | N-curve | S: n_curve.<N>.<learner> (each metric and its ci); n_curve.<N>.<learner>.coverage_at{0.10,0.20} | R3 §2 |
| T A.1.2 | N-curve differences | S: n_curve.<N>.direct_brier_minus_{positive_sft,full_sft} | R3 §2 |
| T A.1.3 | forgetting, selected steps, .95 threshold | S: forgetting_table.<split>.learners.<run> (accuracy, ece, minus_zero_shot); selected_steps.runs.v3_06b_<run> (selected_step, selected_epochs, last_epochs); coverage.<run>.0.95 | R3 §§5, 6, 11 |
| T A.2.1 | training seeds | S: seeds.training_seeds; seeds.direct_log.training_seeds; seeds.full_sft.training_seeds; seeds.log_seed_1 | R3 §§3, 11 |
| T A.2.2 | seed-mean differences | S: seeds.training_seed_mean_minus_positive_sft; seeds.training_seed_mean_minus_full_sft_seed_mean; seeds.direct_log_seed_mean_minus_direct_brier_seed_mean; per-seed ranges from seeds.seed_<k> and n_curve.5000.positive_sft | R3 §§3, 11 |
Language Model Calibration
Language model calibration studies whether models’ confidence estimates match their actual accuracy across tasks, prompts, and deployment settings.
See all 415 papers