Conflicted Records: Typed Adversarial Contracts for Auditable LLM Output Quality Enforcement

Large language model (LLM) data agents answer structured-data questions by generating SQL, invoking tools, executing queries, and summarizing results. In operational settings, reliability depends on both reported answer and computation used to produce it, as errors may remain undetected when the resulting answer matches an expected value. We refer to the mismatch between natural-language intent and operator-level computation required to satisfy that intent as the Structure Gap, and define Trace Integrity as a reliability criterion for assessing whether the computation supporting an answer is explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. We operationalize Trace Integrity through execution contracts that record the schema elements, operator plan, assumptions, executable query, verification status, and final-answer linkage associated with a request. We also define the Correct Answer / Invalid Trace (CAIT) Rate, which measures the proportion of correct answers whose recorded computations fail trace-integrity checks. These provide a way to evaluate whether answer-level success is supported by a valid and inspectable computation. We study this on 100 BIRD Mini-Dev text-to-SQL examples using three prompting conditions: Direct SQL, Operation Summary + SQL, and Contract-First SQL. The three conditions achieve answer accuracies 20.0%, 22.0%, and 24.0%, Trace Integrity Pass Rates 39.0%, 43.0%, and 40.0%, and CAIT Rates 55.0%, 59.1%, and 45.8%, respectively. The results show that answer accuracy, trace validity, and the frequency of correct answers supported by invalid traces capture different aspects of system behavior. Trace-level evaluation therefore complements conventional answer-based evaluation by providing evidence about the computational support, inspectability, and reproducibility of LLM data-agent outputs in real-world systems.

Large Language Models (LLMs) exhibit systematic biases across demographic groups. Auditing is proposed as an accountability tool for black-box LLM applications, but suffers from resource-intensive query access. We conceptualise auditing as uncertainty estimation over a target fairness metric and introduce BAFA, the Bounded Active Fairness Auditor for query-efficient auditing of black-box LLMs. BAFA maintains a version space of surrogate models consistent with queried scores and computes uncertainty intervals for fairness metrics (e.g., $Δ$ AUC) via constrained empirical risk minimisation. Active query selection narrows these intervals to reduce estimation error. We evaluate BAFA on two standard fairness dataset case studies: CivilComments and Bias-in-Bios, comparing against stratified sampling, power sampling, and ablations. BAFA achieves target error thresholds with up to 40$\times$ fewer queries than stratified sampling (e.g., 144 vs 5,956 queries at $\varepsilon=0.02$ for CivilComments) for tight thresholds, demonstrates substantially better performance over time, and shows lower variance across runs. These results suggest that active sampling can reduce resources needed for independent fairness auditing with LLMs, supporting continuous model evaluations.

Confidence scores determine whether an LLM answer is accepted, reviewed, or regenerated, yet scalar recalibration can only correct errors visible through the score itself. We introduce RADAR (Repair, Adopt, and Diagnose After Recalibration), a reliability framework that separates confidence repair from deciding whether to deploy the repair and auditing the errors that remain. Under squared loss, we decompose prediction risk into outcome noise, information hidden by the score, and mapping error; restricting repair to order-preserving maps additionally introduces a pooling gap. For repair, we propose Probe-Anchored Isotonic Calibration (PAIC), which learns a monotone map from correctness-probe predictions on unlabeled target prompts. PAIC's excess risk depends on the probe's score-conditional bias and fitting error rather than its overall accuracy. For adoption, we derive a paired procedure with finite-sample control of false improvements. For auditing, we develop a prompt-clustered kernel test that detects feature-aligned residual errors while accounting for dependent answers from the same prompt. Across multiple LLM checkpoints, benchmarks, and distribution shifts, richer predictors can improve confidence estimation, but projecting their information back to a scalar score can lose these gains, and lower prediction risk need not yield better review decisions. Across 47,400 replayed raw-score allocations, paired betting adopts candidates with lower held-out test risk in 42.95% of allocations, versus 15.09% for one-sided empirical Bernstein. RADAR thus provides a principled workflow to repair what a scalar score can express, adopt a repair only when supported by evidence, and diagnose systematic errors that recalibration leaves behind. Code and data will be released upon acceptance.