cs.LGOct 26, 2026
SaveConfidence Is Not a Promise: A Reference Audit of Typed Decision Models
Organizations: Department of Mathematics, Federal Institute of Brasília, Brasília, Brazil
Abstract
Typed decision models return a probability distribution over options supplied by the caller, and software tends to act on that number as if it were a guarantee. We measure what it licenses, comparing the open-weight Laya, in five checkpoints, with Jev 1.13, available only through an API. A probability licenses a statement about groups of decisions, not about one decision, and a gate that promises an accuracy among the cases it decides alone keeps that promise only with a lower bound checked on data like its traffic. Across 190 exchangeable evaluations, a threshold chosen by point estimate breaks the promise in 147, Learn then Test keeps it in all of them, and a Beta lower-bound rule decides more items on average than Learn then Test in 128 and fewer in none, without its guarantee. Calibrated in English, Jev keeps a promise of 95% in 8 of 51 locales. Ordinary typing noise breaks a promise calibrated on clean text, and, in exploratory runs, training with noise raises the noisy accuracy with no loss on clean text. In four further domains, a fine-tuned Laya needs from 250 to 8,000 labels to be non-inferior to Jev, and does not get there on LIAR2. Under drift, the preregistered claim for a Bayesian retraining loop is not supported: the loop avoids needless retraining where the change does not degrade the model, at a cost equivalent to recalibration alone, and re-qualifying the threshold in every window keeps the promise more often where it does. On the monthly series of M3, neither decider predicts the direction of the next value better than a constant answer. Confidence separates right from wrong answers with an AUROC from 0.80 to 0.96, but it does not select faithful explanations, and exact coalition games of single decisions show what an explanation can still say through an API that rounds to two decimals. We release the frozen answers with a datasheet, a promise check and the code.
Figures & tables
| Claim | Status | Evidence | Where |
| A point-estimate threshold breaks the promise in 147 of 190 evaluations, Learn then Test in none | Exploratory | two-sided Clopper–Pearson interval per evaluation, 2,000 to 10,000 draws | Section 6.3 |
| The ascending Beta rule decides more items on average than Learn then Test in 128 of 190 | Exploratory | means over draws, and on the static front no gain remains after Holm over 24 comparisons | Section 6.3 |
| Calibrated in English, Jev keeps a promise of 95% in 8 of 51 locales | Registered | per-locale violation rate on report items | Section 8 |
| Typing noise breaks a promise calibrated on clean text, and training with noise helps | Registered | paired accuracy, 95% bootstrap intervals, margin of 2 points | Section 7 |
| A fine-tuned Laya needs 250 to 8,000 labels to be non-inferior to Jev, and no tested count suffices on LIAR2 | Registered | paired non-inferiority over a grid of label counts | Section 9 |
| The retraining loop does not meet the preregistered criterion | Confirmatory | paired cost against the best comparator in three drifts | Section 10 |
Table 1: The claims of the abstract, their status and their evidence. Confirmatory: fixed in a preregistration. Registered: analysis written before the runs and exploratory. Exploratory: decided after seeing data or not registered. Descriptive: a measurement without a test.
| Name in the text | In tables | Base | Fine-tuning |
| Jev 1.13 (and 1.14) | Jev | not published | by the vendor |
| Laya multilingual | Laya, released | mmBERT-base | none |
| Laya English | Laya English, root | ModernBERT-large | none |
| Laya typed-decisions | Laya typed-dec. | ModernBERT-large | by the developer of Laya |
| Laya multilingual, fine-tuned | Laya-ft | mmBERT-base | MASSIVE pt-PT, 11,274 items |
| Laya typed-decisions, fine-tuned | Laya typed-dec., ft | ModernBERT-large | MASSIVE pt-PT, 11,274 items |
Table 2: Deciders and the names used for them. Base: the pre-trained encoder of each Laya checkpoint. Laya-ft is short for the fine-tuned multilingual checkpoint, and the battery and the time-series front refit it on their own data with the drift recipe.
| Symbol | Meaning |
| target of the promise, the accuracy among accepted items (0.95 or 0.99 on MASSIVE) | |
| , | accuracy among accepted items and coverage at threshold |
| credibility level of the Beta lower bound, its -quantile (0.05 unless stated) | |
| level of Learn then Test, the probability over the calibration sample that its threshold breaks the promise (0.05) | |
| tolerance | largest acceptable violation rate of a rule against the plug-in population (0.05) |
| margin of non-inferiority and equivalence ( Section 4.7 ), 2 points for accuracy |
Table 3: Symbols and levels. Several of them take the values 0.05 or 95%, with different roles.
| Lang. | System | Accuracy [95% CI] | Macro-F1 | vs Jev, pp [95% CI] | |
| pt | Jev 1.13 | 0.947 [0.931, 0.959] | 0.945 | ||
| Laya multilingual | 0.731 [0.703, 0.758] | 0.709 | [ , ] | ||
| Laya English | 0.736 [0.708, 0.762] | 0.690 | [ , ] | ||
| Laya typed-decisions | 0.739 [0.711, 0.765] | 0.687 | [ , ] | ||
| Laya multilingual, FT | 0.973 [0.961, 0.981] | 0.953 | [ , ] | ||
| Laya typed-decisions, FT | 0.955 [0.940, 0.966] | 0.928 | [ , ] | 0.40 |
Table 4: Accuracy on the 1,000 reported utterances, macro-F1 over intents, and the paired difference to Jev 1.13 (exact McNemar). FT: fine-tuned on MASSIVE pt-PT. Best per language in bold.
Figure 1: Accuracy of the fine-tuned Laya multilingual checkpoint against the number of labelled pt-PT utterances (three seeds per size, mean in the line), with Jev 1.13 without labels as reference.
| System | Accuracy | Labels | Training | Inference | US$ per 1,000 | State leaves |
| Jev 1.13 | 0.947 | 0 | none | API, not measured | 0.016 | yes |
| Laya multilingual | 0.731 | 0 | none | 12.7 ms, CPU | local | no |
| Laya multilingual, 4,000 labels | 0.947 | 4,000 | 17 s, one GPU | as above | local | no |
| Laya multilingual, fine-tuned | 0.973 | 11,274 | 75 to 105 s, one GPU | 11.5 ms, CPU | local | no |
Table 5: What each route costs, MASSIVE pt-PT, reported half. Price per 1,000 decisions from the mean of 384 input tokens over 1,200 calls, at US$0.042 per million input tokens with output tokens free, as stated by the vendor in October 2026. Inference is CPU time per item for Laya. Jev’s latency was not measured, and a local CPU time is not comparable with an API round trip. Training on one RTX 5090 with three seeds. The full-partition times were measured while the GPU was shared with other runs. In bold, the best of each column where an order exists: the highest accuracy, the fewest labels and the state that does not leave the machine.
| Lang. | System | ECE (as ret.) | ECE ( ) | Brier | NLL | AUROC [95% CI] | ECE scen. med./worst | |
| pt | Jev 1.13 | – | 0.020 | – | 0.082 | 0.264 | 0.905 [0.862, 0.943] | 0.039 / 0.077 |
| Laya multilingual | 1.77 | 0.110 | 0.045 | 0.370 | 0.764 | 0.825 [0.798, 0.852] | 0.142 / 0.312 | |
| Laya English | 1.10 | 0.051 | 0.039 | 0.374 | 0.764 | 0.802 [0.773, 0.833] | 0.127 / 0.236 | |
| Laya typed-decisions | 0.55 | 0.158 | 0.041 | 0.358 | 0.706 | 0.818 [0.790, 0.846] | 0.115 / 0.226 | |
| Laya multilingual, FT | 1.28 | 0.018 | 0.012 | 0.045 | 0.095 | 0.928 [0.890, 0.962] | 0.024 / 0.083 | |
| Laya typed-dec., FT | 0.83 | 0.021 | 0.013 | 0.064 | 0.130 | 0.958 [0.936, 0.977] | 0.037 / 0.057 |
Table 6: Calibration and discrimination on the reported half. ECE with 15 bins as returned and after temperature, Brier score and NLL after temperature, AUROC of confidence as a detector of a correct answer with bootstrap 95% interval, and ECE per MASSIVE scenario (median and worst over scenarios with at least 30 items). Best per language and column in bold (lowest, except AUROC).
Figure 2: Top: reliability diagrams (10 bins with at least 5 items), the diagonal is perfect calibration. Bottom: ROC of confidence as a detector of a correct answer. Arrows point from worse to better.
Figure 3: Cost per item against the gate threshold for a cost ratio of 10:1 (a wrong automatic action costs ten reviews), MASSIVE pt-PT. The dashed line is the Bayes threshold .
| Target | Lang. | System | Point | Beta, ascending | Beta, fixed seq. | LTT | SGR |
| 99% | pt | Jev 1.13 | 0.386 (0.394) / 848 | 0.014 (0.017) / 400 | 0.014 (0.016) / 338 | 0.014 (0.016) / 338 | 0.002 (0.003) / 156 |
| 99% | pt | Laya multilingual, FT | 0.408 (0.416) / 953 | 0.020 (0.023) / 879 | 0.019 (0.022) / 851 | 0.019 (0.022) / 851 | 0.002 (0.003) / 776 |
| 99% | pt | Laya typed-dec., FT | 0.504 (0.513) / 911 | 0.031 (0.034) / 831 | 0.019 (0.022) / 818 | 0.019 (0.021) / 817 | 0.001 (0.001) / 739 |
| 99% | en | Jev 1.13 | 0.487 (0.495) / 880 | 0.021 (0.024) / 448 | 0.017 (0.019) / 447 | 0.017 (0.019) / 447 | 0.004 (0.005) / 128 |
| 99% | en | Laya typed-decisions | 0.534 (0.542) / 631 | 0.034 (0.037) / 344 | 0.012 (0.014) / 216 | 0.012 (0.014) / 214 | 0.000 (0.000) / 1 |
| 99% | en | Laya English | 0.533 (0.541) / 503 | 0.026 (0.028) / 88 | 0.012 (0.013) / 56 | 0.011 (0.013) / 56 | 0.002 (0.003) / 1 |
Table 7: The gate against the plug-in population, , 10,000 calibration draws. Each cell: violation rate (one-sided 95% upper bound) / decisions per 1,000 items. Beta rules at , Learn then Test and SGR at . In bold, per case, the rule with the most decisions among those whose upper bound is at most 0.05.
Figure 4: Violation rate of the promise of 99% against the plug-in population, for the five rules of Table 7 , with the one-sided 95% upper bound as error bar. The dashed line is the tolerance of 0.05, and the shaded band below it is where the promise is kept. The annotations mark the two cases in which the ascending Beta rule decides more items than Learn then Test with a 95% interval above zero that includes the item-level uncertainty ( Section 4.7 ).
| Beta, ascending LTT | Beta, fixed seq. LTT | |||||
| Target | Lang. | System | decisions | violation | decisions | violation |
| 99% | pt | Jev 1.13 | (46.5) | (0.002) | (0.6) | (0.000) |
| 99% | pt | Laya multilingual, FT | (11.1) | (0.003) | (0.2) | (0.000) |
| 99% | pt | Laya typed-dec., FT | (8.0) | (0.004) | (0.2) | (0.000) |
| 99% | en | Jev 1.13 | (4.1) | (0.003) | (0.8) | (0.000) |
| 99% | en | Laya typed-decisions | (52.2) | (0.006) | (1.5) | (0.000) |
Table 8: Paired differences against Learn then Test on the same calibration draws, plug-in population, : decisions per 1,000 items and violation rate, with the item-level standard deviation from 200 bootstrap populations in parentheses. In bold, coverage differences larger than twice their item-level standard deviation.
Figure 5: What is decided alone against what is right, MASSIVE pt-PT, target 99%, split-half evaluation. The line is every threshold on the reported half, and each point is one of 1,000 splits. Splits in which a rule qualifies no threshold decide nothing and are not shown: 608 of 1,000 for the Beta rule with Jev, none for the fine-tuned Laya. Shaded bands: the coverage (5th to 95th percentile) of the Beta-rule splits that kept the promise on the reported half (green) and of those that broke it (red). For the fine-tuned Laya the rule keeps the promise between coverages of about 0.74 and 0.93 and breaks it above, where the curve flattens near the target, the region that Proposition 4 and the measured disagreement of the scans point to.
| Kept | Beta | Mean violation | Decisions per 1,000 | |||||||||
| Decider | Cases | point | Beta | LTT | undet. | broken | point | Beta | LTT | point | Beta | LTT |
| Jev 1.13 | 18 | 5 | 18 | 18 | 0 | 0 | 0.290 | 0.015 | 0.012 | 659 | 494 | 478 |
| Laya as released | 82 | 5 | 78 | 82 | 4 | 0 | 0.477 | 0.027 | 0.010 | 277 | 162 | 89 |
| Laya fine-tuned | 90 | 32 | 87 | 90 | 2 | 1 | 0.321 | 0.024 | 0.017 | 737 | 686 | 652 |
Table 9: The three rules over the 190 exchangeable evaluations of the article, by kind of decider, each case with its tightest measurement (10,000 draws where 2,000 left it undetermined). Kept: upper bound of the violation rate at most 0.05. Undetermined and broken: the Beta cases with the interval straddling 0.05 or above it. Mean violation rate and mean decisions per 1,000 over the cases. In bold, per row, the rule that keeps the promise most often, the lowest mean violation and, between the two bounded rules, the one that decides more.
| Accuracy | Violation at 95% | ||||||
| System | Lang. | clean | typo 10% | mixed 10% | mixed 20% | clean | mixed 10% |
| Jev 1.13 | pt | 0.950 | 0.939 | 0.939 | 0.934 | 0.00 | 0.15 |
| Laya multilingual | pt | 0.732 | 0.686 | 0.646 | 0.636 | 0.18 | 0.98 |
| Laya multilingual, fine-tuned | pt | 0.973 | 0.944 | 0.939 | 0.930 | 0.00 | 1.00 |
| Jev 1.13 | en | 0.961 | 0.947 | 0.948 | 0.943 | 0.00 | 0.75 |
| Laya multilingual | en | 0.816 | 0.752 | 0.744 | 0.726 | 0.01 | 0.18 |
Table 10: Natural noise, report items. Accuracy on clean text and under typos at 10% of the words and the mixed profile at 10% and 20%, and the violation rate of the promise of 95% by the ascending Beta rule, with thresholds chosen on 1,000 calibration samples of clean items and applied to the clean and to the mixed-10% report items. In bold, per language, the highest accuracy and the lowest violation rate of each column.
Figure 6: Greedy typo attack on Laya, 200 report items per language. Left: share of decisions not yet changed after each number of typos, with 95% intervals, for the released and the fine-tuned multilingual checkpoint. Fine-tuning delays the fall but does not prevent it. Right: for the fine-tuned checkpoint, the confidence of each clean decision, in log-odds, grouped by the number of typos that changed it. The groups overlap across the whole range, so a confidence of 0.99 or 0.9999 (dashed lines) does not tell which decisions will survive. Jev was not attacked, only measured under random noise.
| Lang. | Clean utterance | Attacked utterance | Clean decision | Decision after the typos |
| pt | existe um tosquiador só de gatos na cidade | exiate um tosquiador só de gatos na cidade | general_quirky (0.9993) | play_game (0.92) |
| en | open temperature | open temprature | weather_query (0.9993) | play_music (0.33) |
| pt | eliminar uma lista | eiminar uma lisha | lists_remove (0.99997) | iot_hue_lightup (0.90) |
Table 11: Typos that change a decision of the fine-tuned Laya. The examples are not chosen by hand: in each language, the item of median clean confidence among those changed by one typo, and in Portuguese also among those changed by two, as fixed in the implementation notes before they were seen. The gold intent is the clean decision in all three.
Figure 7: The promise of 95% across the 51 locales of MASSIVE other than en-US, report items. Left: violation rate of the Beta rule calibrated on the en-US calibration items, against the accuracy lost relative to en-US on the same items. The green band is where the promise travels (violation at most 0.05). Jev keeps it only where the loss stays below about one point. Right: locales where the promise holds (upper bound of the violation rate at most 0.05), for Jev calibrated in en-US and for Laya calibrated on the calibration items of each locale, with the median decisions per 1,000 over the bars. Calibrating in the locale recovers the promise for Laya with the bounded rules, at the price of fewer decisions.
| Accuracy | Labels for | Promise kept | Decisions | Typos | |||||||||
| Domain | Opt. | Jev | Laya | Laya-ft | NI | sup. | ECE Jev | point | Beta | LTT | Beta | LTT | |
| MASSIVE pt (ref.) | 5 | 0.947 | 0.732 | 0.960 | 4,000 | none | 0.020 | 7 | 8 | 10 | 996 | 996 | 0 of 3 |
| News Category | 5 | 0.817 | 0.585 | 0.831 | 8,000 | none | 0.076 | 0 | 6 | 9 | 517 | 331 | 0 of 3 |
| Financial tweets | 3 | 0.728 | 0.642 | 0.926 | 250 | 250 | 0.171 | 3 | 9 | 10 | 842 | 532 | 0 of 3 |
| LIAR2 | 6 | 0.319 | 0.282 | 0.324 | none | none | 0.162 | 10 | 10 | 10 | 0 | 0 | no decisions |
| Fake.Br (pt) | 2 | 0.716 | 0.567 | 0.981 | 250 | 250 | 0.087 | 7 | 10 | 10 | 999 | 999 | 3 of 3 |
Table 12: The battery, report items. Columns, from the left: accuracy of Jev, of the released Laya and of the largest fine-tune (mean of three seeds), the labels from which the fine-tune is non-inferior (margin of 2 points) and superior to Jev, Jev’s ECE as returned, the cases in which each rule keeps the promise against the plug-in population (of 10, targets 0.90 and 0.95 for five deciders counting each seed) with the tolerance of Section 6.3 , the median decisions per 1,000 of the fine-tunes at 0.95, and the seeds in which the Beta promise of the fine-tune at 0.95, calibrated on clean text, survives typos at 10%. The first row is MASSIVE pt-PT measured with the same code and grid, with Jev’s earlier answers reused. In LIAR2 no rule accepts any item, so every promise is kept by abstention. In bold, per row, the highest accuracy, the rule that keeps the promise most often and the rule that decides most, where they differ.
Figure 8: Label curves of the battery. Accuracy of the fine-tuned Laya on the report items against the labelled items used (three seeds and their mean), with Jev without labels and the released Laya as reference lines. Gold where the fine-tune is more accurate than Jev with a 95% interval above zero, green where it is non-inferior with the margin of 2 points, red where it is neither.
| Cost per item | Windows below | Retrains | ||||||||
| Arm | D1 | D2 | D3 | control | D1 | D2 | D3 | D1 | D3 | control |
| Author’s loop | 0.764 | 0.402 | 0.825 | 0.433 | 0.86 | 0.00 | 0.93 | 0.9 | 1.2 | 0.0 |
| Periodic ( ) | 0.720 | 0.466 | 0.806 | 0.482 | 0.83 | 0.26 | 0.62 | 3.0 | 3.0 | 3.0 |
| DDM | 0.843 | 0.448 | 0.838 | 0.500 | 0.91 | 0.08 | 0.65 | 2.2 | 2.1 | 2.0 |
| ADWIN | 0.795 | 0.427 | 0.883 | 0.475 | 0.88 | 0.03 | 0.96 | 0.9 | 0.0 | 0.0 |
| Page–Hinkley | 0.815 | 0.428 | 0.843 | 0.476 | 0.93 | 0.05 | 0.77 | 1.2 | 1.0 | 0.2 |
Table 13: Drift experiment, main run: mean over the 60 streams of each drift (30 replicates, abrupt and gradual) and the 30 control streams. Cost per item at , share of post-change windows of 500 items whose true selective accuracy is below , and retrains per stream. In bold, the lowest cost and the fewest windows below per column. Retrains are not ranked, because under a drift that degrades the model fewer retrains is not better.
| Method | Against | D1 | D2 | D3 | control |
| Author’s loop | Periodic | – | C | – | C |
| Author’s loop | DDM | C | C | – | C |
| Author’s loop | ADWIN | – | NI | NI | – |
| Author’s loop | Page–Hinkley | T | T | – | T |
| Author’s loop | Recalibration only | – | E | – | E |
| Author’s loop | LTT per window | – | T | – | T |
Table 14: Efficiency at equal promise in the drift experiment (exploratory, 64 comparisons, Section 4.7 ). Outcome: share of windows below (post-change windows, all windows on the control stream), margin 0.02. Resources: labels per 1,000 items, retrains and GPU seconds, paired over 60 streams per drift and 30 control streams. C: clean efficiency gain. T: trade-off, one resource saved and another resource or the error rate larger. E: equivalent, no resource saved. NI: non-inferior, no resource saved. –: non-inferiority not shown, because the method is worse or the interval is too wide. Clean gains, the best outcome of a comparison, in bold.
| S1, noise cycle | S2, language cycle | D1 | D3 | |||||
| Arm | Cost | Below | Cost | Below | Cost | Below | Cost | Below |
| Recalibration only | 0.614 | 0.51 | 0.407 | 0.00 | 0.742 | 0.57 | 0.781 | 0.38 |
| Recalibration, | 0.646 | 0.52 | 0.420 | 0.00 | 0.728 | 0.26 | 0.762 | 0.22 |
| LTT per window | 0.493 | 0.01 | 0.420 | 0.00 | 0.668 | 0.37 | 0.773 | 0.21 |
| Author’s loop | 0.578 | 0.45 | 0.408 | 0.00 | 0.764 | 0.86 | 0.825 | 0.93 |
| Periodic | 0.578 | 0.44 | 0.456 | 0.22 | 0.720 | 0.83 | 0.806 | 0.62 |
Table 15: The dynamic adapter on streams with a pattern and under the degrading drifts: mean cost per item and share of windows below (all windows on S1 and S2, post-change windows on D1 and D3), over 30 streams per cycle and 60 per drift. The seasonal variants were run on S1 and S2 only. In bold, the lowest value per column.
| Decider | Accuracy | Macro-F1 | Rise | Minus ETS | Top 100 |
| ETS | 0.673 | 0.673 | 0.50 | – | 0.88 |
| Jev | 0.540 | 0.510 | 0.23 | 0.76 | |
| Laya as released | 0.476 | 0.324 | 0.00 | 0.56 | |
| Laya fine-tuned | 0.522 | 0.343 | 1.00 | 0.60 |
Table 16: Deciders on the direction of the next value, 150 M3 series at their last 12 origins (1,800 decisions, 52.2% of rises). Accuracy and macro-F1 with 95% intervals from a bootstrap over series, share of decisions that predict a rise, accuracy minus that of ETS on the same decisions, and accuracy of the 100 most confident decisions. Laya fine-tuned averages the probabilities of three seeds. In bold, the highest accuracy.
| Target 0.70 | Target 0.80 | |||
| Gate | Below target | Per 1,000 | Below target | Per 1,000 |
| Point, calibrated once | 0.42 | 795 | 0.29 | 350 |
| Learn then Test, calibrated once | 0.33 | 763 | 0.17 | 311 |
| Point, sliding | 0.50 | 828 | 0.42 | 398 |
| Beta rule, sliding | 0.42 | 792 | 0.38 | 372 |
| Learn then Test, sliding | 0.42 | 790 | 0.17 | 184 |
Table 17: Gates on exponential smoothing over the 1,428 M3 series and the 24 test origins: share of origins whose accuracy among acted decisions is below the target, and decisions acted per 1,000. Sliding gates are calibrated on the 12 preceding origins. Per-bin gates accept a confidence bin when its predicted accuracy reaches the target. In bold, the lowest share below the target per column.
Figure 9: Deletion curves by occlusion, mean over the 1,000 reported utterances: words removed most important first (solid) against random order (dotted), with areas in parentheses. A faster fall means a more faithful explanation.
| Lang. | Checkpoint | AUD, MoRF | AUD, random | gap | Spearman |
| pt | Laya multilingual, FT | 0.631 | 0.790 | 0.159 | |
| pt | Laya multilingual | 0.490 | 0.675 | 0.185 | |
| en | Laya multilingual | 0.514 | 0.693 | 0.179 | |
| en | Laya English | 0.555 | 0.723 | 0.169 |
Table 18: Area under the deletion curve on the 1,000 reported utterances, words removed most important first (MoRF) and in random order, and the Spearman correlation between the probability of the decision and its MoRF area. The gap is random minus MoRF area: a larger gap means the ranking points to the words the decision rests on, beyond what any removal does. The largest gap per language is in bold.
| Question | Measured (criterion) | Useful when |
| Are words and pairs enough? | median order-two share 0.86 to 0.93 (passes, ) | always, for inputs of up to eight units |
| Intervals for the effect of a word under rounding | coverage of 0.55 to 0.59 and of 0.64 to 0.72 , KernelSHAP 0.59 to 0.63 (fails, ) | not available through a two-decimal API, where rounding bounds alone certify a relevant word in 60% to 70% of Jev decisions and the most relevant one in 17% to 21%, and not needed with unrounded probabilities and short inputs |
| Can redundant evidence be detected? | F1 0.300 for the Bayesian model, 0.901 for the Shapley interaction index , 0.889 for HEDGE (fails for the Bayesian model) | with an interaction index or HEDGE, also through the API |
| Does confidence signal a faithful explanation? | positive slope in 4 of 4 Laya cases, small, Jev 1 of 4, planted intents at ranks 2, 3, 6 of 56 (passes) | no, faithfulness has to be measured per decision |
| Does the explanation depend on the options? | approximate invariance in 3 of 4 Laya and 8 of 8 Jev comparisons, 68% to 79% of items change decision or runner-up | only for the option set it was computed on |
| Which ranking deletes fastest? | exact Shapley and HEDGE first, occlusion last, in Laya and in Jev (Kendall 0.6 to 1.0) | exact Shapley or HEDGE, not occlusion |
Table 19: Explanation front on the report items: what was asked, what was measured against the preregistered criterion, and when the answer is useful to a caller. Coverage is the share of exact values inside the 90% interval. Pt-ft is the fine-tuned multilingual checkpoint rounded to two decimals as a stand-in for Jev. Where a row compares methods, the best one is in bold. HEDGE is the hierarchical explanation method of Chen et al. [2020] . G1 to G5 are the criteria of the preregistration.
| Control | What to measure | Measured in |
| Validation | a fixed schema for the state, a token count against the context budget, the language, and the format in which the content is sent, which is part of the input | Sections 7 and 8 |
| Signals | a second, independent decider and the McNemar table of disagreements | Section 6 |
| Logging | state, question, options, model version and probabilities of every call | the released package |
| Drift | a fixed probe set re-run over time, with accuracy, ECE and McNemar between windows, and the accuracy of each confidence bin tracked online | Section 10 |
| Anomaly | shifts in option frequencies, cases with high but low , and the decision on the empty state | Sections 6.2 and 11 |
| Escalation | a gate with a lower-bound rule, Learn then Test where a guarantee is required | Section 6.3 |
Table 20: Seven operational controls turned into measurements, and where the article measures each.
| Situation | What was measured | What to do | Where |
| Promising an accuracy among accepted decisions | the point estimate breaks the promise in 147 of 190 evaluations, Learn then Test keeps it in all, the ascending Beta rule in 183 with more decisions | Learn then Test where a guarantee is required, the ascending Beta rule where more automatic decisions matter more, never the point estimate | Section 6.3 |
| Few calibration labels | a threshold needs at least accepted items to qualify, 58 at 95% | collect at least that many accepted items, or let the gate abstain | Proposition 5 |
| Noisier inputs than the calibration items | a few points of accuracy lost on typed text break the promise | calibrate on items that look like the traffic, noise included, and train with noise | Section 7 |
| Several languages | calibrated in English, Jev keeps the promise in 8 of 51 locales, with a slack of 1.2 points | calibrate in each language, and read the slack as the loss the promise can absorb | Section 8 |
| A new domain | labels for non-inferiority to Jev range from 250 to 8,000, Jev’s ECE from 0.076 to 0.171 | measure the label curve and the calibration in the domain, and test what a fine-tune reads | Section 9 |
| No rule decides anything | the selective ceiling lies below the target | add informative inputs to the state, ask a coarser question or answer with sets | Proposition 6 |
Table 21: What the measurements of the article say to a caller of a typed decider.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: When fine-tuning Laya pays off, for any priced API. Two conditions must hold. The fine-tune must be non-inferior to Jev in accuracy, which the paired test with a margin of 2 points establishes from 4,000 labels on (solid line). Below it, in the red band, no price of a label makes the fine-tune worthwhile. And its labels must cost less than the API spend they replace, with the cost of the decisions served at a price per decision. For a fixed this boundary is a hyperbola, so the number of labels and the price of a label trade off non-linearly. The fine-tune pays off in green, between the solid line and a hyperbola, and the green deepens where it pays off already at a smaller API spend. The right axis gives, at the label counts of the test grid only, the measured difference in accuracy between the fine-tuned Laya and Jev, averaged over three seeds. The curves are labelled with the API spend avoided and, in parentheses, the number of Jev decisions it buys at the measured price. The dashed line marks the 11,274 labels at which Laya becomes more accurate than Jev, and in gold, above it and below a hyperbola, Laya beats Jev on both counts, more accurate and cheaper. Superiority was measured at 11,274 labels, the whole training pool, and the gold band above that count assumes that more labels do not lower the accuracy, which was not tested. Each hyperbola is a lower bound on the volume needed, because local inference is not free, and it does not price that accuracy gain, which would enlarge the region wherever an error has a cost.
| 95% | 99% | 99.5% | ||||||||
| Lang. | System | point | Beta | LTT | point | Beta | LTT | point | Beta | LTT |
| pt | Jev 1.13 | 0.50 / 995 | 0.10 / 976 | 0.10 / 976 | 0.50 / 859 | 0.03 / 314 | 0.03 / 251 | 0.46 / 536 | 0.02 / 15 | 0.02 / 15 |
| pt | Laya multilingual | 0.55 / 443 | 0.10 / 311 | 0.07 / 212 | 0.64 / 142 | none | none | 0.36 / 58 | none | none |
| pt | Laya English | 0.52 / 327 | 0.09 / 182 | 0.02 / 89 | 0.60 / 105 | none | none | 0.47 / 82 | none | none |
| pt | Laya typed-decisions | 0.52 / 397 | 0.09 / 279 | 0.05 / 226 | 0.47 / 160 | none | none | 0.49 / 139 | none | none |
| pt | Laya multilingual, FT | 0.00 / 999 | 0.00 / 999 | 0.00 / 999 | 0.49 / 955 | 0.07 / 883 | 0.06 / 846 | 0.48 / 898 | 0.04 / 709 | 0.04 / 709 |
Table 22: Broken promises per split on the report half and decisions per 1,000, for the point rule, the ascending Beta rule ( ) and Learn then Test ( ), at targets of 95%, 99% and 99.5%, 1,000 splits. “none”: no qualifying threshold in any split, which keeps the promise by abstaining. In bold, per system and target, the rule with the fewest broken promises and, among ties, the most decisions.
| Accuracy | Violation, calibrated in en-US | Violation, in the locale | |||||||
| Locale | Script | Jev | Laya | Laya-ft | Jev | Laya | Laya-ft | Laya | Laya-ft |
| af-ZA | Latin | 0.934 | 0.595 | 0.832 | 0.86 | 1.00 | 1.00 | 0.51 | 0.40 |
| am-ET | Ethiopic | 0.905 | 0.344 | 0.571 | 1.00 | 1.00 | 1.00 | 0.00 | 0.15 |
| ar-SA | Arabic | 0.917 | 0.637 | 0.878 | 0.98 | 1.00 | 1.00 | 0.01 | 0.08 |
| az-AZ | Latin | 0.946 | 0.624 | 0.844 | 0.73 | 1.00 | 1.00 | 0.16 | 0.25 |
| bn-BD | Bengali | 0.938 | 0.555 | 0.788 | 0.83 | 1.00 | 1.00 | 0.03 | 0.19 |
Table 23: Languages without fine-tuning (and Laya fine-tuned in Portuguese), report items. Script: the Unicode script of most letters in the locale’s utterances. Accuracy and violation rate of the Beta rule at 95% (2,000 draws) calibrated in en-US or in the locale. Laya-ft is the multilingual checkpoint fine-tuned on pt-PT. In bold, per row, the highest accuracy and the lowest violation rate of each group, where they differ.
| Promise at 0.90 | Promise at 0.95 | Beta at 0.95 under | Empty | ||||||||
| Decider | Acc. | ECE | point | Beta | LTT | point | Beta | LTT | typos | mixed | state |
| MASSIVE pt-PT (reference) , 5 options, fine-tunes on 8,000 labels | |||||||||||
| Jev 1.13 | 0.947 | 0.020 | 0.00/999 | 0.00/999 | 0.00/999 | 0.41/993 | 0.03/975 | 0.03/974 | 0.48/968 | 0.16/967 | 0.281 |
| Laya | 0.732 | 0.043 | 0.48/635 | 0.05/561 | 0.04/525 | 0.52/438 | 0.05/321 | 0.03/229 | 0.54/319 | 0.98/303 | 0.175 |
| Laya-ft, seed 0 | 0.959 | 0.014 | 0.00/999 | 0.00/999 | 0.00/999 | 0.00/999 | 0.00/994 | 0.00/993 | 0.95/983 | 0.97/983 | 0.395 |
| Laya-ft, seed 1 | 0.963 | 0.019 | 0.00/999 | 0.00/999 | 0.00/999 | 0.00/999 | 0.00/997 | 0.00/997 | 1.00/995 | 0.99/992 | 0.408 |
Table 24: Report cards of the battery, report items. Accuracy, ECE (Jev as returned, Laya after temperature on the calibration half), violation rate over decisions per 1,000 of the point rule, the ascending Beta rule and Learn then Test at 0.90 and 0.95, the Beta promise at 0.95 calibrated on clean text and checked under typos and under the mixed profile at 10%, and accuracy with an empty state. In bold, within each block, the highest accuracy, the lowest ECE and, in each promise column, the decider that keeps the promise with the most decisions.
More in this topic
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 789 papers