cs.AIOct 27, 2026

Verification Learning Theory: A Process Model of Checking and Retention in AI-Assisted Judgment

Authors: Christoph Trattner

Organizations: Department of Information Science and Media Studies, University of Bergen, Norway

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Accurate AI assistance can improve immediate judgments while reducing opportunities to practise the operations needed for independent performance. Verification Learning Theory (VLT) formalises this possibility for claim-evaluation practice followed by correct, independent, rule-specific feedback. The model links perceived assistant reliability to evidence checking and links the history of those checks to retained rule use. Its strict form assumes that task demands and perceived reliability influence learning only through checking, with equal learning per genuine check. It therefore predicts that perceived fallibility can improve delayed unaided judgment even without observed assistant errors, and that this advantage disappears when checking is already required and performed. Rule-specific gains and forecasts from checking histories provide further tests. Simulations examine the consequences and parameter dependence of these commitments; they do not constitute empirical validation. Comparisons with depth-of-processing and feedback-encoding accounts identify where predictions diverge and where the accounts remain observationally equivalent. The framework specifies experiments that can test the proposed mechanism and separates retention of practised rules from the broader problem of recognising which rule applies to a new claim.

Figures & tables

ElementExample
RulePopulation: a finding may not hold for people unlike those studied.
ClaimA sports drink improves the performance of adults who do not exercise regularly.
Evidence that warrants acceptingA randomised trial in adults who do not exercise regularly found that those given the drink performed better than those given a placebo drink. The people studied are those the claim is about, and the evidence supports the claim.
Evidence that warrants rejectingA randomised trial in adults who do not exercise regularly found that those given the drink performed worse than those given a placebo drink; a second trial, in trained athletes, found that it improved their performance. Only the first trial bears on the claim, and it contradicts the claim.
Evidence that warrants withholding judgmentA randomised trial in trained athletes found that those given the drink performed better. No evidence concerns the people the claim is about.
Flip questionWhich of these changes in how the evidence was obtained, each leaving every finding as it is, would change the response the evidence warrants? Options for the withholding version: the trial was run in adults who do not exercise regularly (correct: accept); the trial was run in younger trained athletes (same feature, response unchanged); the trial lasted ten weeks instead of eight; performance was measured on a cycling test instead of a running test. The last two change features that the rules on duration and proxy outcomes concern without breaking them. The accepting and rejecting versions offer the same kinds of options.
Table 1: One family of practice claims on the population rule, with a test claim on a new topic
Figure 1: The process Verification Learning Theory proposes. Perceived reliability starts from the stated error rate and is revised by every explanation, which shows whether the verdict was right. A person checks the evidence for certain when the task requires the check, and otherwise is more likely to check it the less reliable the assistant seems. Each check, successful or not, adds strength to the rule the claim calls for, whatever prompted it, while an explanation read without a check adds only a small fraction ε\varepsilon of that strength. Months later, a new claim is judged as its evidence warrants if the rule it calls for is still strong enough to be recognised.
AccountTask: E-X − E-RBelief: B-R − E-RBelief, check required: B-X − E-XFallibility: F-R − E-RFallibility, check required: F-X − E-XEffort alone: M − E-RUnpractised ruleClaims that warrant acceptance
Strict VLT++0+000no loss
Depth or feedback encoding (nests VLT)+++ small++ small00no loss
Response tendenciesnoneshiftshiftshiftshiftnonesame shiftloss
Answer retrieval0000000no loss
Explanation-driven learning (VLT with ε\varepsilon = 1)0000000no loss
Task-only engagement (nested in VLT)+000000no loss
Table 2: Predicted differences between practice conditions under strict VLT and the alternative accounts
Figure 2: Where strict VLT and its two closest alternatives agree and where they diverge. (A) Strict VLT: with the copyable task, a stated error rate and an assistant that errs improve delayed unaided judgment; with the check-requiring task the three assistants coincide; the effort-matched task equals the copyable task. (B) Depth or feedback encoding: with the copyable task the same order as VLT, with larger gains; with the check-requiring task doubt leaves a residual effect, and without it the account is strict VLT (dashed line). (C) Learning or caution: the change in accuracy from an always-right to an erring assistant, by the response that new claims warrant, on practised rules (filled) and an unpractised rule (open), under VLT and under an account in which the erring assistant leaves learning unchanged and lowers the tendency to accept by half a unit on the log-odds scale. Panels A and B are schematic; panel C shows medians over the 1,000 simulated parameter sets of Section 5.
Figure 3: How the simulated gains depend on the parameters that the model leaves open, in points of balanced accuracy 90 days after practice (medians over 1,000 parameter sets). (A) The task gain, the fallibility gain and the belief gain, with the weight ε\varepsilon of an explanation read without a check fixed at 0, .3, .6 and .9. (B) The belief gain as a share of the fallibility gain, by the weight κ\kappa that people give the statement. (C) The effect of the assistant when the task requires the check, if people who have seen it err take in less from every later explanation (discount ω, left), or if doubt raises the learning value of a check and of the explanation that follows (θ, right); zero is strict VLT. (D) The three gains when the test comes 7 to 365 days after practice.
PredictionContrastResult against strict VLTWhat the result favoursCondition that must hold
1 Task gainE-X − E-RNo gain above the smallest effect of interestAnswer retrieval, explanation-driven or error-driven learningFlip answers attempt the rule
2 Belief gainB-R − E-RNo gain above the smallest effect of interest; or a gain as large as F-R − E-R although B’s rating has recoveredNo gain: error-driven learning or the two-route account. A gain as large: an effect of the statement that does not run through perceived reliabilityB rates the assistant as less reliable than E from the first rating on
3 No belief effect, check requiredB-X − E-XA difference beyond the marginDepth or feedback encoding if positive; a cost of doubt to the processing of explanations (not modelled) if negativeB rates the assistant as less reliable than E; equal shares of guesses and of attempts at another rule in B-X and E-X; measured depth below its ceiling
4 Fallibility gainF-R − E-R, F-R − B-R, F-X − E-XNo gain in F-R − E-R or F-R − B-R; or F-X beyond the marginF-X higher: the two-route account, extra work on error trials or depth. F-X lower: discounting of explanations after errors, or failed checks teaching lessF rates the assistant as less reliable; own judgment before the flip response
5 Effort aloneM − E-RA difference beyond the marginGeneral engagement if M is higherM takes as long as the flip task and leaves checking unchanged
6 SelectivityPractised against unpractised rules; full response matrixGains as large on the unpractised rule; a loss on claims that warrant acceptance; a loss confined to narrowly passing claims on practised rulesGeneral engagement; response tendencies; over-general rule learning, which the response component excludes by assumptionTest claims include claims that warrant acceptance and claims that pass only narrowly
Table 3: The tests of the design, the results that count against strict VLT, and the conditions under which they count
SymbolMeaningValues
1 − ρ0\rho_{0}Stated error rate; when the assistant errs, the actual rate rounds it to a whole number of errors among the 24 practice claimsUniform, .10 to .30
κ\kappaWeight of the statement, in verdicts, the same for everyone within a parameter setUniform, 2 to 30
η0\eta_{0}Log-odds of a check with an assistant believed always right, for an average person and claimUniform, −3 to −1
η1\eta_{1}Rise in that log-odds per unit of perceived unreliability, 1 − ρ^\hat{\rho}Uniform, 5 to 20
ccConfidence in one’s own judgment, for each personNormal, mean 0, SD uniform from .3 to 1
ℓ\ellEffort a check takes, for each person and claimNormal, mean 0, SD .5
Table S1: Parameters of the simulation and the values drawn
More in this topic

Human-in-the-Loop AI

Human-in-the-loop AI combines human feedback, oversight, and decision-making with machine learning and autonomous systems.

See all 290 papers

Explore similar work

CardsList
Aug 9, 2026

Epistemic Transfer in AI-Assisted Verification: A Framework and Evaluation Protocol

AI tools that help people judge online claims are usually evaluated while the tool is present. This paper asks a different question: after using such a tool, what can the user still do on their own? I call this epistemic transfer. It refers to the effect of prior AI-assisted verification on later unassisted performance on new claims. In this paper, I make three contributions. First, I distinguish epistemic transfer from nearby outcomes such as correction effects, trust, reliance, and human--AI team performance. Second, I introduce two simple quantities for studying it: the Epistemic Transfer Effect (ETE), which compares delayed unassisted performance across conditions, and Tool-Removal Cost (TRC), which measures the immediate drop in performance when the tool is taken away. Third, I turn these ideas into a practical evaluation protocol that can be used in online experiments or field studies. The protocol combines answer-first and evidence-first AI conditions with active-practice and no-practice controls, delayed tests on held-out claims, behavioral measures, and participant- and item-level analyses. Putting ETE and TRC together yields a diagnostic space that separates capability building, capability plus tool advantage, epistemic inertness or de-skilling, and verification on loan. The point is not that every AI tool must teach. The point is that when independent judgment matters, we should test not only whether a tool helps now, but also what it leaves behind.

Christoph Trattner

SFI MediaFutures, Research Centre for Responsible Media Technology and Innovation · Department of Information Science and Media Studies, University of Bergen, Norway

Tool-Use EvaluationHuman-in-the-Loop Evaluation
AskPDF
Dec 29, 2025

Althea: The Fact-Checking--Metalearning Tradeoff in AI-Assisted Verification

Fact-checking systems must be scalable and epistemically trustworthy. We introduce Althea, a retrieval-augmented system for user-driven claim evaluation that matches standard pipelines on AVeriTeC while improving supported/refuted discrimination. A longitudinal survey experiment (N=961) treats a ten-day follow-up as a fading test: after modeling a verification procedure, we remove the system and ask whether users reproduce it unaided, testing metalearning rather than one-time accuracy. We compare two AI-assisted treatments, Exploratory (guided reasoning) and Summary (synthesized verdicts), against two baselines, unrelated news and Self-search. The treatments yield the strongest immediate accuracy and confidence gains but do not survive the fading test: on unseen claims they perform no better than news, while Self-search, with no procedure to fade, retains a large advantage. This reveals a factchecking-metalearning tradeoff: conditions that most improve immediate accuracy are least likely to produce metalearning, cautioning against treating AI-delivered verdicts as a source of durable literacy gains.

Svetlana Churina, Kokil Jaidka, Anab Maulana Barik +5

Centre for Trusted Internet & Community (CTIC), National University of Singapore (NUS), Singapore · Department of Communications and New Media, Centre for Trusted Internet & Community (CTIC), National University of Singapore (NUS), Singapore · School of Computing, Centre for Trusted Internet & Community (CTIC), National University of Singapore (NUS), Singapore

Human-in-the-Loop EvaluationAutomated Fact-Checking
AskPDF
Oct 4, 2026

VERA: Verdict-Conditioned Reliability for Adaptive LLM Judges

Accurately estimating judgment reliability is a central challenge in adapting LLM judges to newly verified feedback while preserving previously learned behavior. However, existing approaches often rely on output-level confidence, which can be overconfident and poorly aligned with judgment correctness. We propose VERA, a VErdict-conditioned Reliability Axis that estimates reliability from hidden activations by distinguishing correct from incorrect judgments within each predicted-verdict group. Using VERA as a control signal, we develop a VERA-guided periodic adaptation framework that integrates reliability-ranked corrective updates, reliability-residual replay, and periodic refresh of the reliability directions. After VERA-guided adaptation on Chatbot Arena, 8B- and 14B-parameter judges outperform the strongest baseline on each of four held-out public benchmarks, with relative gains of up to 23.01%. The framework also improves focal-class recall by up to 16.1% relative to the strongest adaptive baselines on a separate proprietary temporal auditing task.

Qiushui Xu, Syamil Mohd Razak, Tao Yuan +1

Penn State University · Amazon

LLM-as-a-JudgeLLM Fine-TuningLLM Reliability
AskPDF