cs.AIAug 17, 2026

Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency

Authors: Parsa Mazaheri, Kasra Mazaheri

Organizations: University of California, Santa Cruz · Massachusetts Institute of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Automated checking pipelines increasingly place one language model as the checker and another (or the same one) as the fixer. We ask whether that wiring changes what the checker reports. Measuring false alarms on human-verified-correct ProcessBench traces with the present task held byte-identical, we find that a completed audit -> repair episode already in the model's context lowers false alarms in 15 of 15 model x wording combinations, by 2.8 to 11.5 percentage points against a length-matched non-audit control, a 9 to 25% reduction relative to that control. The direction contradicts what the accumulated-message literature predicts: an episode whose audit reported an error lowers false alarms further still, at all five wordings on the model where that manipulation lands cleanly, though a negativity asymmetry predicts more flagging. Decomposing the episode finds repair content and audit verdict complementary: different components carry the effect on different model families. Signal-detection analysis locates the change in the threshold rather than in discrimination -- the criterion moves in 15 of 15 combinations and survives correction in 13 while d' survives in none, though the d' test is half as sensitive by construction -- and a hand audit of 50 false alarms finds 82% simply wrong, so at this operating point the shift need not be harmful. With reasoning enabled the effect keeps its relative size on both models tested, and the threshold reading holds there too.

Figures & tables

Figure 1: The question and the instrument. (a) An auditing pipeline: a checker reports, a repairer fixes. We ask whether who repairs next changes what is reported now , on traces human annotators verified correct, so any flag is a false alarm. (b) A false-alarm rate alone cannot distinguish better discrimination from greater reluctance to flag, so every condition is run on both a correct and a labelled-incorrect arm and combined into d′d^{\prime} and the criterion cc . The two arms are disjoint and matched trace for trace on source; the prior-context conditions use a source-proportional half of the correct arm (465 targets), the ladder all 929.
 Qwen3.6-27BQwen3.6-35B-A3BMinistral-3-14B
R0 false-alarm rate0.1850.2320.691
Is it an audit, or merely context?
episode −- AF−4.00-4.00 † [ −5.26-5.26 , −2.80-2.80 ]−3.59-3.59 † [ −5.38-5.38 , −1.92-1.92 ]−8.83-8.83 † [ −11.83-11.83 , −5.91-5.91 ]
AF −- R0 (the filler itself)+1.58+1.58 [ +0.46+0.46 , +2.71+2.71 ]−0.18-0.18 [ −2.12-2.12 , +1.69+1.69 ]−0.30-0.30 [ −1.83-1.83 , +1.18+1.18 ]
Which part of the episode?
AV −- AF (audit alone)−3.13-3.13 † [ −4.38-4.38 , −1.98-1.98 ]−1.62-1.62 [ −3.48-3.48 , +0.19+0.19 ]−3.67-3.67 [ −6.39-6.39 , −0.96-0.96 ]
Table 1: The episode effect and its controls at wording F1, 465 targets, FAR percentage points; negative means fewer false alarms. Every interval is a 95% cluster bootstrap on the frozen episode (§ 3.4 ). “episode” is the mean of the four 2×22\times 2 cells, not AS alone. † marks survival of Holm–Bonferroni within the declared claim family ( k=24k=24 ), the only mark we draw a conclusion from. Ministral’s AV −- AF and AN −- AF are therefore shown with intervals excluding zero that we nonetheless do not claim.
modelcontrastF1F2F3F4F5surv.
Qwen3.6-27BAS −- AN (repair request)−1.31-1.31 †−0.85-0.85 †−0.92-0.92 †−0.39-0.39−0.54-0.543/5
Qwen3.6-35B-A3B −2.35-2.35 †−1.90-1.90 †−2.42-2.42 †−1.80-1.80 †−1.92-1.92 †5/5
Ministral-3-14B −3.13-3.13 †−2.15-2.15−3.10-3.10−4.03-4.03 †−2.11-2.112/5
Qwen3.6-27BAX −- AS (wedge)−0.51-0.51+0.02+0.02+0.99+0.99−0.81-0.81−0.50-0.500/5
Qwen3.6-35B-A3B −3.83-3.83 †−0.80-0.80−1.27-1.27−0.17-0.17−1.18-1.181/5
Ministral-3-14B −5.62-5.62 †−4.05-4.05 †−5.90-5.90 †−5.16-5.16 †−4.33-4.33 †5/5
Table 2: The 4 claim contrasts discussed in the text at every wording, all three models, FAR percentage points; negative means fewer false alarms. Each interval is a 95% cluster bootstrap on the frozen episode, and † marks survival of Holm–Bonferroni within that wording’s claim family ( k=24k=24 ), corrected per wording over the same declared membership. The ‘surv.’ column is the count the body quotes. Intervals are omitted for width; they are in the released artefacts.
Figure 2: Every detection-arm contrast as one point in (threshold move, discrimination change), all three models and all five wordings. Fill encodes survival of the Δd′\Delta d^{\prime} family only , so no prior-context point is filled: that is the 00 of 1515 of § 6 , read off the figure. Quadrants are labelled by what the auditor visibly does, because signal-detection usage and this paper’s usage point opposite ways for the same direction: a higher criterion is “conservative” about asserting an error, which is leniency toward the audited work. The prior-context cell sits right of zero and, in all but two cases, just above the axis; the stars are the same cell with reasoning on (§ 7 ). Of the 14 filled points, 13 are the Qwen prospective rungs of Appendix B , all below zero; the fourteenth is Ministral’s R3u, the one survivor in the upper-right quadrant.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

modelstatusepisode −- AF
Qwen3.6-27Bretained−4.00-4.00 [ −5.26-5.26 , −2.80-2.80 ]
Qwen3.6-35B-A3Bretained−3.59-3.59 [ −5.38-5.38 , −1.92-1.92 ]
Ministral-3-14Bretained−8.83-8.83 [ −11.83-11.83 , −5.91-5.91 ]
Gemma-4-26Bscreened out+0.41+0.41 [ −1.08-1.08 , +1.98+1.98 ]
Gemma-4-31Bscreened out+1.58+1.58 [ +0.45+0.45 , +2.82+2.82 ]
Appendix
Table 3: All five candidates at F1, FAR percentage points. The retained rows are confirmatory and the only ones any claim rests on; the screened-out pair is exploratory, having moved on none of the three framing controls. The selection effect is real and visible here: on Gemma-4-31B the contrast is significant and reversed , which is why reading its other contrasts would be a mistake and why the direction we report is a property of framing-responsive verifiers rather than of every model we tried. Confirmatory rows come from the F1 artefact of record; the exploratory rows exist only in the with-Gemma run, which differs from it in the third decimal.
modelcontrastF1F2F3F4F5surv.
Qwen3.6-27Bepisode −- AF−4.00-4.00 †−3.69-3.69 †−3.66-3.66 †−2.76-2.76 †−3.60-3.60 †5/5
Qwen3.6-35B-A3B −3.59-3.59 †−6.37-6.37 †−6.26-6.26 †−5.44-5.44 †−5.15-5.15 †5/5
Ministral-3-14B −8.83-8.83 †−7.47-7.47 †−6.27-6.27 †−11.50-11.50 †−9.23-9.23 †5/5
Qwen3.6-27BAV −- AF (audit alone)−3.13-3.13 †−3.15-3.15 †−2.85-2.85 †−2.62-2.62 †−2.89-2.89 †5/5
Qwen3.6-35B-A3B −1.62-1.62−4.15-4.15 †−5.09-5.09 †−3.29-3.29 †−3.38-3.38 †4/5
Ministral-3-14B −3.67-3.67−1.59-1.59−3.04-3.04−4.19-4.19 †−2.17-2.171/5
Appendix
Table 4: Every claim contrast at every wording, all three models, FAR percentage points; negative means fewer false alarms. F1–F5 are the five semantically matched wordings of the audit request, F1 being the pre-specified one. Each interval is a 95% cluster bootstrap on the frozen episode, and † marks survival of Holm–Bonferroni within that wording’s claim family ( k=24k=24 ), corrected per wording over the same declared membership. The ‘surv.’ column is the count the body quotes. Intervals are omitted for width; they are in the released artefacts.
More in this topic

Language Model Error Detection

Language model error detection identifies unreliable answers, reasoning steps, and actions so systems can flag, verify, or repair them.

See all 90 papers

Explore similar work

CardsList

  1. Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models

    Jun 8, 2026Vyzantinos Repantis, Ameya Gawde, Harshvardhan SinghLLM EvaluationLanguage Model Error Detection

    The Educational Approaches to Virtual Reality Technologies Lab, University of Ioannina, Greece · Meta Platforms

  2. Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers

    Sep 16, 2026Zihan Chen, Di Zhu, Lei Zheng +1LLM-as-a-JudgeLLM Auditing

    Stevens Institute of Technology Hoboken, NJ, USA · University of Massachusetts Boston Boston, MA, USA · Stony Brook University Stony Brook, NY, USA

  3. Detection Is Not Enough: Reasoning Traces as Testable Program Hypotheses

    Oct 21, 2026Vaidehi HegdeLanguage Model Reasoning EvaluationLanguage Model Error Detection

    Indian Institute of Technology (Indian School of Mines) Dhanbad