Training Negatives Still Matter After Calibrating Answerability Readouts

Jinchen Zhang; Zheyuan Liu; Yihan Zhu; Meng Jiang

cs.CLOct 19, 2026

Training Negatives Still Matter After Calibrating Answerability Readouts

Authors: Jinchen Zhang, Zheyuan Liu, Yihan Zhu, Meng Jiang

Organizations: University of Notre Dame

Abstract

Answerability readouts predict whether a passage supplies an answer, but whether calibration can compensate for removing negative passages during training remains unclear. We present a controlled study of how training-negative construction affects calibrated prediction, ranking, and downstream answer retention. We compare native training, which retains negative passages, with empty training, which removes them, holding questions, labels, positive examples, and model weights fixed. Both are calibrated on the same labeled native examples. On SQuAD 2.0 and RepLiQA, native-trained logistic readouts at selected depths achieve lower Brier loss for Qwen2.5-7B-Instruct and Gemma-2-9B-it, with or without output scores. Native fusion also outperforms calibrated output alone. Post-hoc AUROC analysis shows better ranking under native training, beyond what threshold adjustment alone can explain. On the official SQuAD 2.0 development set, native readout policies improve F1 over empty policies but retain more answers to unanswerable questions. The advantage varies by estimator and depth, while RepLiQA's text shortcuts limit claims about contextual understanding. Unfinished generations prevent complete answer-level evaluation, although worst-case bounds preserve the native-empty F1 gains. Our findings show that training negatives affect ranking and downstream decisions even after common calibration, without establishing a universal advantage for native training.

Figures & tables

Panel of Figure 1 — Figure 1: A SQuAD fitting example with the passage summarized. Native (solid blue) and empty (dashed orange) training share the same unanswerable question and label. The bottom panel concerns calibration on new inputs, not predictions for this example.

Panel of Figure 2 — Figure 2: Controlled comparison of native (solid blue) and empty (dashed orange) fitting, with the model, readout depth, questions, and labels held fixed. The left panel fits readouts on condition-specific states, yielding scores $s$ . Neural and speech icons denote readout and output scores, respectively. Separate maps (gears) use identical labeled native examples (teal) to calibrate readout scores, both signals jointly (fusion), or output scores alone (gray) into probabilities $p$ . Prediction quality uses new SQuAD articles and RepLiQA documents. Answer retention uses official SQuAD 2.0 development questions. Each probability-based veto applies its own threshold to $p$ to keep the same generated candidate within a model or abstain. Primary Brier comparisons use fusion, and primary answer-retention comparisons use readouts alone.

Dataset / model	Native fusion	Empty fusion	Output
SQuAD / Qwen	0.085	0.095	0.097
SQuAD / Gemma	0.095	0.110	0.119
RepLiQA / Qwen	0.029	0.051	0.066
RepLiQA / Gemma	0.043	0.064	0.073

Table 1: Primary Brier losses on new articles and documents at selected depths. Lower is better. Fusion predictors share output scores and labeled calibration examples, differing only in training negatives. Output is each study’s prespecified comparator. Figure 3 shows paired improvements.

Figure 3: Brier reductions from native fusion relative to empty fusion (circles) and output alone (squares). Positive values favor native fusion. Error bars show paired 98.75% article/document

t

intervals, corrected for four comparisons per dataset. Datasets are not pooled. Appendix B , Table 5 , gives exact differences and intervals.

Model	Training	Brier	AUROC	BA
SQuAD
Qwen	native	0.088	0.947	0.888
	empty	0.106	0.923	0.864
Gemma	native	0.099	0.934	0.867
	empty	0.121	0.904	0.823
RepLiQA

Table 2: Logistic readouts without output scores, after common probability calibration, on the same inputs as Table 1 . BA uses a probability threshold of 0.5.

Answer F1
Policy	Qwen	Gemma
Unchanged	81.51	70.65
Empty readout	78.71	76.92
Native readout	82.70	78.85
Selected output	81.67	78.12
Native fusion	83.82	79.69

Table 3: Answer F1 on the official SQuAD 2.0 development set (0–100). Policies share generated candidates within each model. Brackets give paired article-bootstrap intervals, Bonferroni-adjusted for six comparisons at nominal 95% family confidence. F1 values are lower bounds with zero credit for unfinished generations (sensitivity bounds in Appendix C.2 ). Differences use unrounded scores.

Figure 4: Native-minus-empty answer F1, separated into official credit differences on answerable and unanswerable questions. Bars sum to the net gain over all 11,873 questions per model, with zero credit for unfinished generations. These are arithmetic contributions (Appendix C.4 ).

Model	Training	Within scenario AUROC	BA training threshold	BA recalibrated threshold
Q	Logistic / E/Bkg	1.000	0.638	0.898
Q	Logistic / E/I-N	1.000	0.766	0.969
Q	Logistic / E/I-B	1.000	0.770	0.970
G	Logistic / E/Bkg	1.000	0.589	0.920
G	Logistic / E/I-N	1.000	0.682	0.935
G	Logistic / E/I-B	1.000	0.738	0.941

Table 4: Threshold transfer with fixed logistic readouts (Q: Qwen, G: Gemma). Training contrasts are defined in § 6.6 . Both BA columns test held-out scenarios with training thresholds or thresholds calibrated on other scenarios. Metrics average over 24 scenarios and dependent fits. BA includes both irrelevant-wording variants. Appendix E , Table 14 , also reports DiM.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset / model	Native fusion	Empty fusion	Output	Empty fusion $-$ native 98.75% interval	Output $-$ native 98.75% interval
SQuAD / Qwen	0.085	0.095	0.097	0.010 [0.005, 0.015]	0.011 [0.006, 0.017]
SQuAD / Gemma	0.095	0.110	0.119	0.015 [0.010, 0.019]	0.023 [0.015, 0.031]
RepLiQA / Qwen	0.029	0.051	0.066	0.022 [0.017, 0.027]	0.036 [0.028, 0.044]
RepLiQA / Gemma	0.043	0.064	0.073	0.021 [0.017, 0.026]	0.030 [0.024, 0.036]

Appendix

Table 5: Primary Brier comparisons on new SQuAD articles and RepLiQA documents. Lower Brier is better. Difference columns subtract native-fusion Brier from empty-fusion or output Brier, so positive values favor native fusion. Intervals are paired 98.75% article/document

t

intervals, corrected for four comparisons per dataset. The datasets use different fitting protocols and prevalences, so effects are not pooled. Output is the prespecified comparator. Differences are computed before rounding.

Model	$\Delta$ AUROC	95% interval	Adjusted 98.75% interval
SQuAD
Qwen	0.0237	[0.0141, 0.0337]	[0.0115, 0.0368]
Gemma	0.0293	[0.0180, 0.0412]	[0.0153, 0.0450]
RepLiQA
Qwen	0.0119	[0.0061, 0.0188]	[0.0049, 0.0212]
Gemma	0.0149	[0.0090, 0.0217]	[0.0077, 0.0240]

Appendix

Table 6: Post-hoc paired-cluster intervals for the readout-alone AUROCs in Table 2 .

\Delta

is native minus empty. Intervals condition on fixed predictors and resample 164 SQuAD articles or 256 RepLiQA documents. Adjustment covers these four supplementary contrasts, all with positive lower bounds.

Model / training	BS	MCB	DSC	UNC
SQuAD
Qwen / native	0.088	0.006	0.167	0.250
Qwen / empty	0.106	0.008	0.152	0.250
Gemma / native	0.099	0.005	0.156	0.250
Gemma / empty	0.121	0.007	0.136	0.250
RepLiQA

Appendix

Table 7: Empirical CORP decomposition of fixed readout-alone probabilities on new articles and documents. Isotonic PAV is fitted on evaluation labels only for this descriptive analysis. BS = MCB

-

DSC + UNC, up to displayed rounding. No PAV-adjusted prediction is deployed.

Native minus	F1 gain	Adjusted interval	Bounds for unfinished generations
Qwen / empty	3.99	[2.43, 5.77]	[3.93, 4.05]
Qwen / output	1.03	[0.13, 1.99]	[0.97, 1.09]
Qwen / unchanged	1.19	[ $-$ 0.24, 2.62]	[1.13, 1.25]
Gemma / empty	1.93	[0.91, 2.88]	[1.74, 2.13]
Gemma / output	0.72	[ $-$ 0.06, 1.41]	[0.53, 0.92]
Gemma / unchanged	8.20	[6.94, 9.57]	[8.01, 8.39]

Appendix

Table 8: Primary answer-F1 differences in percentage points. Intervals use 20,000 paired article-bootstrap draws and a Bonferroni adjustment for six comparisons (99.1667% per interval for a nominal 95% family level). The last column allows each paired difference with an unfinished generation to range independently from

-1

1

, a conservative point sensitivity bound. It is not a confidence interval or a replacement for the unmet complete-output requirement. Differences are computed before rounding.

Policy	Qwen F1	Gemma F1
Unchanged	81.51	70.65
Always abstain	50.05	50.01
Native readout	82.70	78.85
Native fusion	83.82	79.69
Output (isotonic)	82.41	78.12 sel
Output (logistic)	81.67 sel	78.12

Appendix

Table 9: All evaluated policy F1 scores (0–100), under the same lower-bound convention. Superscript sel marks the output comparator selected by calibration answer F1 for each model and reported as Selected output in Table 3 . Budget denotes the allowed fraction of unanswerable calibration questions whose answers are retained, not a certified test-risk bound. These thresholds are secondary operating points, not alternatives selected after viewing development-set results.

Output calibration / articles	Qwen BS	Gemma BS
Scalar logistic / 16	0.107	0.119
Isotonic / 16	0.097	0.119
Per-marker-token logistic / 16	0.109	0.109
Scalar logistic / 32	0.106	0.119
Isotonic / 32	0.097	0.119
Per-marker-token logistic / 32	0.109	0.110

Appendix

Table 10: Original comparator and post-hoc SQuAD output controls evaluated on the same 164 evaluation articles. Article counts refer to each component’s calibration access, not six disjoint calibration sets. Native fusion Brier is 0.085 for Qwen and 0.095 for Gemma. Per-marker-token logistic uses one feature per token of the prescribed abstention response.

Block / estimator	Native BS	Empty BS	Empty $-$ native	Reported interval
SQuAD / Qwen
20 (selected) / Logistic	0.08535	0.09526	+0.00991	[0.00518, 0.01464] *
20 (selected) / DiM	0.09234	0.09385	+0.00150	—
27 (final) / Logistic	0.09561	0.09484	$-$ 0.00077	—
27 (final) / DiM	0.09534	0.09508	$-$ 0.00027	—
SQuAD / Gemma

Appendix

Table 11: All evaluated estimator–depth combinations on the new articles and documents. BS is fusion Brier loss. Positive empty-minus-native differences favor native training. Intervals are paired article/document

t

intervals, conditional on the fits. Asterisks mark the primary 98.75% intervals corrected for four comparisons per dataset. Other intervals are exploratory 95% intervals without multiplicity correction. A dash means no paired interval was computed, not no effect. Five decimals preserve the signs of small differences.

Dataset / text predictor	Brier	AUROC	BA
SQuAD / question only	—	0.558	0.543
SQuAD / question + passage	—	0.510	0.506
SQuAD / word counts, native	—	0.533	0.527
RepLiQA / question only	0.048	0.982	0.905
RepLiQA / word counts, native	0.041	0.986	0.953
RepLiQA / word counts, empty	0.099	0.898	0.753

Appendix

Table 12: Text-only controls. SQuAD entries replay native-trained predictors under common threshold calibration, for which Brier is not reported (—). RepLiQA entries use probability calibration. The protocols differ across datasets.

Predictor	Positive BS	Negative BS	Class-balanced BS
Word counts	0.038	0.050	0.044
Qwen / readout	0.011	0.098	0.054
Qwen / fusion	0.011	0.102	0.056
Gemma / readout	0.017	0.138	0.078
Gemma / fusion	0.018	0.139	0.079

Appendix

Table 13: RepLiQA class-conditional Brier losses, all with native training. The word-count classifier has lower negative-class and class-balanced loss than both models’ readouts. Under the original class prevalence, Qwen’s readout has lower overall loss than the word-count classifier, whereas Gemma’s does not.

Model	Training	Within scenario AUROC	BA training threshold	BA recalibrated threshold
Q	DiM / E/Bkg	0.986	0.819	0.873
Q	DiM / E/I-N	0.999	0.940	0.950
Q	DiM / E/I-B	1.000	0.959	0.965
Q	Logistic / E/Bkg	1.000	0.638	0.898
Q	Logistic / E/I-N	1.000	0.766	0.969
Q	Logistic / E/I-B	1.000	0.770	0.970

Appendix

Table 14: Complete threshold-transfer results for the constructed factual task (Q: Qwen, G: Gemma). E/Bkg uses bare-background negatives, and E/I-N and E/I-B use notebook and bag additions. Both BA columns evaluate held-out scenarios, before or after threshold calibration on other scenarios. Metrics average over 24 scenarios and dependent fits, with BA including both irrelevant-wording variants.

Model / E/I fit	BA gain	95% interval
Q / DiM / I-N	0.077	[0.031, 0.132]
Q / DiM / I-B	0.092	[0.047, 0.145]
Q / Logistic / I-N	0.071	[0.019, 0.128]
Q / Logistic / I-B	0.072	[0.021, 0.132]
G / DiM / I-N	0.072	[0.033, 0.114]
G / DiM / I-B	0.100	[0.056, 0.150]

Appendix

Table 15: E/I minus E/Bkg BA after common threshold calibration. Q and G denote Qwen and Gemma. I-N and I-B denote notebook and bag training variants. The descriptive paired 95% scenario-bootstrap intervals condition on fitted readouts and calibration halves, without multiplicity correction.

Selective QA

See all 10 papers

Two Axes of LLM Abstention: Answer Correctness and Question Answerability
Jul 9, 2026Benedikt J. WagnerSelective Prediction Confidence Estimation in Language Models
City St George’s, University of London
ABCD: All Biases Come Disguised
Feb 19, 2026Mateusz Nowak, Xavier Cadet, Peter ChinLLM Evaluation Language Model Bias Evaluation
Dartmouth College
Scoring the Wrong Question: Readout Failures in Constrained-Option Evaluation
Sep 27, 2026Jiaxuan Guo, Kejia Zhang, Shuo Xin +3LLM Evaluation Confidence Estimation in Language Models
Stanford University · University of Maryland, College Park

Jul 9, 2026J/K move · Enter open · S save

Two Axes of LLM Abstention: Answer Correctness and Question Answerability

Benedikt J. Wagner

City St George’s, University of London

Selective Prediction Confidence Estimation in Language Models LLM Uncertainty Estimation Refusal Behavior in Language Models

Open paper AskPDF arXiv

A model should refuse two different things: answers it would get wrong, and questions it should not answer at all, such as unanswerable ones or ones resting on a false premise. The usual recipe thresholds a single confidence score, which cannot tell these apart. Across five instruction-tuned models from three families (2B to 14B), we find they are separate axes. Ordinary answer-confidence tracks whether an answer is right but is nearly blind to whether the question is answerable; a linear probe on hidden states does the reverse. The blind spot does not shrink with scale. It is worst on naturally occurring false-premise questions (CREPE). There, answer-confidence, P(IK), P(True), and even asking the model outright whether a premise is false all stay near chance, while a hidden-state probe reaches 0.69 to 0.77 AUROC: the model represents a problem it will not report. This turns out to be fixable. Instructing a model to check premises backfires, because it then disputes sound and false premises alike (57% false challenges), unable to tell them apart; routing the same instruction with the probe roughly triples challenge precision. We turn the two axes into a calibrated policy that answers only when an answerability score and a correctness score each clear a separately certifies behave differently: the unanswerable-answer rate is controllable at every scale, while the wrong-answer rate is capped by model accuracy, so the guarantee tightens as threshold policy certifies both budgets at 0.75 coverage of correct answers, against 0.31 for a single threshold; at 14B it is the only policy that certifies at all.

Abstract

Selective QA

Explore similar work

Two Axes of LLM Abstention: Answer Correctness and Question Answerability

ABCD: All Biases Come Disguised

Scoring the Wrong Question: Readout Failures in Constrained-Option Evaluation