Trust, but verify, audit and attribute: a human-in-the-loop framework for checking AI-produced research results
Authors: György Eigner, Patrik P. Süli, Roland Hollós
Organizations: Biomatics and Applied Artificial Intelligence Institute, John von Neumann Faculty of Informatics, Obuda University, Budapest, Hungary · Physiological Controls Research Center, Obuda University, Budapest, Hungary · Doctoral School of Applied Informatics and Applied Mathematics, Obuda University, Budapest, Hungary · HUN-REN Centre for Agricultural Research, Brunszvik u. 2., 2462 Martonvásár, Hungary
Artificial intelligence (AI) systems now produce research results in bulk: in October 2026 a single laboratory released more than 700 AI-generated mathematics manuscripts, only part of them with machine-checked proofs. Such volumes will be checked mostly by AI agents that call specialised software. A passing check certifies only the artefact that was checked. For proofs translated into the proof assistant Lean, it has been shown that this artefact may state something other than the author's text, and that deciding whether a translation is faithful is undecidable in general. The human researcher therefore stays in the loop, and human and AI contributions must be recorded. We propose a domain-independent, human-in-the-loop framework for checking AI-produced results. Validated, agent-usable tools with different grounds of trust check the results; a fidelity audit asks whether each check addressed the claim as stated; the human decides at fixed points; and a dialog report attributes the work to the human and to the AI. We describe two automatic implementations of the tool layer, for mathematics (Wolfram Language, MATLAB, Lean) and statistics (Python and R), and run the mathematical one on a two-claim example. An ambiguous claim was split into two readings, one refuted and one proved.
Figures & tables
Figure 1: The two gaps in every verification step. A validated tool closes the gap between the artefact and its verdict (solid arrow). Nothing in the tool closes the gap between the author’s claim and the artefact (dashed arrow); this is the gap the fidelity audit examines.
Figure 2: The framework. Double frames (blue): human roles; single frames (orange): AI roles; shaded frames (grey): Stage-1 tools, chosen to rest on different grounds of trust. Filled circles H1–H4: the four human decision points of section 5 . Dashed arrow: deviations returned for a faithful retry, an amended target or a negative verdict. Arrows into Stage 3: records that feed the dialog report.
Class
Examples
A verdict establishes
It does not establish
Formal proof
Lean 4 + Mathlib [ 16 , 17 ] ; Rocq [ 18 ]
The formal statement follows from the axioms
That the formal statement is the author’s claim
Symbolic computation
Wolfram Language [ 19 ]
Identities and decisions under stated assumptions
General results from instance checks; hidden branch choices
Numerical computation
MATLAB [ 20 ] , GNU Octave [ 21 ]
Numbers within stated tolerances
Cases near the tolerance; shared formulation errors
Statistical computation
R [ 22 ] ; SciPy [ 23 ] , statsmodels [ 24 ]
Estimates and tests for the data and model given
That data, model and test are those stated
Physical simulation
COMSOL [ 25 ]
Solutions of the discretised model
That model, parameters and mesh are those stated
Table 1: Classes of validated, agent-usable tools by ground of trust.
Figure 3: Wiring of the two examples. Single frames (orange): the AI agent with the skill and its scripts; shaded frames (grey): Stage-1 tools behind MCP servers, with the version used in this paper; numbers: the fixed order of the tool runs; dashed outline (blue): tools that run on the researcher’s computer; heavy frames (green): the validated record that goes to the human (double frames, decision point H2).
Design choice
Example 1: math-prove-tool
Example 2: stat-prove-tool
1. Tools and grounds of trust
Wolfram Language (symbolic decision), MATLAB (symbolic and numerical), Lean 4 + Mathlib (formal proof)
Python stat-validated server and R: two independent statistical implementations
F(AI)2R is FAIR research with AI in the loop, twice: an AI-assisted authoring pass and a machine-readable audit pass over every artefact. AI systems now draft, refactor, and verify research artefacts, yet their contributions are rarely recorded in a form a later human or machine can audit. Building on the original F(AI)2R experiment, we generalize its provenance model beyond scholarly writing into aiprov, a PROV-O extension covering any AI-in-the-loop artefact, and we package the method as an executable skill that an AI agent operates itself: setup asks the human operator for their ORCID ID, resolves their identity from the public registry, and scaffolds continuous integration that gates every push on graph conformance and publishes the current build of this very paper. The paper is its own case study. Every activity, claim, and source in its production is recorded in the repository's provenance graph under two invariants: no parentless claim, and verification rungs that only humans may grant.
Verifying numerical claims in research written by AI agents is an important but challenging problem. Recent benchmarks show that many agent-written papers fail integrity checks, but such verdicts, often produced by a single LLM judge, identify failure without explaining its nature, and existing checks either test a paper for internal consistency or test whether a reported value appears somewhere in an agent's output. We re-analyze all 540 released judgments from a recent public benchmark of AI research agents (nine agents, fifteen open research problems) and manually label 256 numerical claims from 18 submissions. Failures concentrate on alignment between the paper and the underlying work rather than overt cheating, and about four in five unsupported claims correspond to values in the agent's own files that come from the wrong experiment, metric, or run. We introduce a deterministic claim-to-evidence checker that scores whether the surrounding context identifies the claimed result and calibrates each claim against numerical decoys. With its rules fixed before labeling, it certifies claims with over 90% precision against blinded annotators (78 of 85; Cohen's kappa = 0.89), while verifying only a subset of claims; most claims it cannot verify are nevertheless traceable, so failure to verify is not evidence of falsehood. An LLM verifier given the same evidence recovers more traceable claims but accepts about three times as many misattributed ones, so the two are best used together. Applied to 99 submissions, the checker reveals large differences in traceability across agents, and on 30 submissions a second judge configuration, with a different model and harness, orders the agents' vendor groups as the official judge does while passing twice as many papers. These findings motivate provenance-first auditing of AI-produced research.
Mosab Alfaqeeh, Omar Mohammed Horani, Mou'th Hourani +1
Al-Ahliyya Amman University, Amman, Jordan · Binghamton University, State University of New York, Binghamton, NY, USA
AI-assisted research compresses ideation, implementation, evaluation, and manuscript writing into a single interactive loop. This compression is useful, but it also creates a publication risk: paper claims can become easier to state than to audit. We present ResearchLoop, an evidence-gated control plane for AI-assisted computational research. ResearchLoop treats research questions, task contracts, evidence objects, claim ledgers, closeouts, and paper bindings as durable project state, realized here as a repository-backed runtime. This technical report provides the complete protocol specification, state model, transition rules, claim-admission algorithm, and insight-compounding mechanism. It also reports the full experimental record spanning nine versions (V0--V9), including a self-hosting case study, a controlled task-suite study with component ablations, a mathematical olympiad evaluation, and a supplementary SciCode boundary experiment evaluated with the official generated-code harness. All artifacts, manifests, and verification reports are preserved in the project repository.