cs.LGNov 11, 2026

Anytime-Valid Empirical Likelihood Inference from LLM Annotations with Adaptive Human Auditing

Authors: Hongfu Gao, Jiang Hu, Pengfei Liu, Wang Zhou

Organizations: Department of Statistics and Data Science, National University of Singapore · School of Mathematics and Statistics, Northeast Normal University · School of Mathematics and Statistics, Jiangsu Normal University

Abstract

Large language models (LLMs) are increasingly used to annotate unlabeled data as a fast, low-cost alternative to human annotation. However, their outputs can be biased and miscalibrated relative to human labels, potentially leading to misleading analysis and inference. We propose AAELI (Adaptive Human Auditing and Anytime-Valid Empirical Likelihood Inference), an online framework for estimating the population mean of human labels under a limited auditing budget. AAELI uses LLM outputs to guide adaptive human auditing and combines them with audited labels through predictable empirical-likelihood ee-processes to construct anytime-valid confidence sequences. Our main theoretical result shows that a predictably learned efficient projection preserves the first-order profile empirical-likelihood statistic under adaptive auditing, without requiring a prescribed convergence rate for the projection estimator. This yields an exactly anytime-valid betting construction whose continuous-mixture form incurs a scalar rather than vector-dimensional mixing penalty. Supporting results establish adaptive empirical-likelihood asymptotics and show that a learned audit rule attains the oracle variance within the stated class of augmented estimating-equation designs under the same audit-probability floor and long-run expected budget. Experiments on synthetic data and three real-world datasets demonstrate that AAELI maintains empirical anytime coverage while requiring fewer human audits than the strongest competing baseline to reach a target precision.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing

    Oct 26, 2026Harine Choi, Chenchen Kuai, Zihao Li +5LLM EvaluationLLM Auditing

  2. Quantifying and Auditing LLM Evaluation via Positive--Unlabeled Learning

    Jun 17, 2026Zilong Zhang, Yi-Ting Hung, Lei Ding +1Inverse Optimal TransportLLM-as-a-Judge

  3. AURA: Adaptive Uncertainty-aware Refinement for LLM-as-a-Judge Auditing

    Jun 18, 2026Zilong Zhang, Yi-Ting Hung, Weiyi He +3LLM-as-a-JudgeLLM Auditing