Human-AI interactions are becoming a rich source of data for improving AI systems: while many model responses are already satisfactory, others expose failures that could become valuable supervision if identified and corrected. We study this setting through the notion of an editor: a more capable evaluation process that can inspect a response and either confirm it or modify it, but whose invocation is costly. The editor may be a human expert, a stronger AI system, or the same model equipped with additional computation, context, tools, or information. The key challenge is to decide which interactions to send to the editor before knowing whether an edit will actually be made, since invoking the editor itself consumes limited resources such as expert time, inference cost, latency, or tool-use overhead. We formulate this problem as Edit Discovery Optimization (EDO), which seeks to minimize edit opportunities missed among skipped interactions while controlling unnecessary confirmations and respecting an editor-cost budget. We characterize the population-optimal policy and show that it takes the form of a cost-adjusted threshold on the posterior probability of an edit. Building on this structure, we develop EDIT (Edit Discovery via Iterative Thresholding), an online algorithm for the partial-feedback setting in which edit labels are revealed only for selected interactions. EDIT combines an adaptive edit score, a confirmation-control threshold, and a cost-aware selection rule; it satisfies the editor budget at every round and controls the average confirmation rate up to a vanishing finite-horizon term, without distributional assumptions on the interaction stream. Across four reasoning and factual question-answering benchmarks, EDIT achieves a better trade-off between editor cost, Type-I error, and Type-II error than direct LLM-based batch selection.
Figures & tables
Figure 1: Overview of EDIT . Each HAI interaction Zt=(Xt,Yt) receives an estimated edit score ηθt(Zt) . Based on this score, the policy πt determines whether the interaction is sent to the editor or skipped. Selected responses are either confirmed or edited, yielding validated data and feedback for updating the score and policy online.
Figure 2: Main results with Llama-3.3-70B-Instruct. Columns correspond to MATH-500, MMLU-Pro, PopQA, and GPQA. Rows report running-average Type-II error, Type-I error, and editor cost; dashed horizontal lines mark the configured Type-I error tolerance and per-round editor budget.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Table 3
T
nt
Bt
c(z)
γ
ξ
α1
θ0
d
250
10
5
1
0.1
1.0
0
0.1×1
384
Appendix
Table 3: Hyperparameters shared across the main online experiments.
Figure 3: Human-audit interface. The human annotator inspects the user prompt, candidate response, and reference answer, then decides whether the response can be confirmed as-is or requires a substantive edit. Optional reason categories support later inspection of disagreement cases.
Dataset
N
Benchmark edit
Human edit
GPT-5.6 Sol edit
B–H agree
κBH
F1,BH
H–GPT agree
MATH-500
50
0.280
0.300
0.260
0.980
0.951
0.966
0.960
MMLU-Pro
50
0.460
0.420
0.440
0.960
0.919
0.955
0.980
PopQA
50
0.560
0.520
0.540
0.960
0.920
0.963
0.980
GPQA
50
0.580
0.620
0.600
0.960
0.917
0.967
0.980
Overall
200
0.470
0.465
0.460
0.965
0.930
0.963
0.975
Appendix
Table 4: Validation of the benchmark-derived edit labels. The human annotator is the primary validation target; GPT-5.6 Sol is an independent model annotator. “B–H agree”, κBH , and F1,BH compare the benchmark with the human labels. “H–GPT agree” reports agreement between the human annotator and GPT-5.6 Sol. The positive class is “needs edit”.
Figure 4: Type-I–Type-II trade-offs for EDIT-REP . Panels correspond to MATH-500, MMLU-Pro, PopQA, and GPQA. Colored curves vary the editor budget B , while marker shapes denote the Type-I error tolerance ε . Error bars show 95% confidence intervals across five realized streams. The black cross shows the existing LLM-Select run with B=5 . Lower values on both axes are better.
Figure 5: Editor-cost–edit-recall trade-offs for EDIT-REP . Panels correspond to MATH-500, MMLU-Pro, PopQA, and GPQA. Colored curves vary the editor budget B , while marker shapes denote the Type-I error tolerance ε . Error bars show 95% confidence intervals across five realized streams. The black cross shows the existing LLM-Select run with B=5 . Higher edit recall is better.
Table 9
Dataset
Model
Method
Edit recall ↑
Missed edits ↓
Edit precision ↑
EDIT-REP
0.784
143
0.467
EDIT-LLM
0.691
205
0.473
EDIT-RAND
0.151
563
0.253
Llama-3.3-70B-Instruct
LLM-Select
0.621
251
0.437
EDIT-REP
0.810
96
0.366
EDIT-LLM
0.939
31
0.407
Appendix
Table 8: Complementary metrics on the main Llama-3.3-70B-Instruct and Qwen3-32B runs. Edit recall is the fraction of editable interactions that are selected; missed edits counts editable interactions that are skipped; and edit precision is the fraction of selected interactions that require an edit. Within each model–dataset block, the best value is shown in bold and the second-best is underlined .
Figure 6: Main results with Qwen3-32B. Columns correspond to MATH-500, MMLU-Pro, PopQA, and GPQA; rows show running-average Type-II error, Type-I error, and average editor cost. Dashed lines indicate the configured Type-I error tolerance and per-round editor budget.
Figure 7: Distribution of normalized response-length editor costs. The violin, box, and point layers show the empirical cost distributions for MATH-500 (left) and PopQA (right). The dashed vertical line marks unit cost, corresponding to the median response length within the relevant generation; costs are clipped to the interval [0.25,4.0] .
Figure 8: Edit discovery with response-length-dependent editor cost. Results use Llama-3.3-70B-Instruct on MATH-500 and GPQA with the cost in Equation ( 15 ) and budget Bt=5 . Rows report running-average Type-II error, Type-I error, and average editor cost. Dashed lines denote the configured Type-I error tolerance and per-round editor budget.
Figure 9: Robustness under structured distribution shift. Vertical dotted lines mark phase boundaries in the MMLU-Pro subject stream and the PopQA popularity stream. Rows report running-average Type-II error, Type-I error, and average editor cost. Dashed horizontal lines indicate the configured Type-I error tolerance and per-round editor budget.
Figure 10: Effect of the embedding-model size on EDIT-REP . Results use Llama-3.3-70B-Instruct as the response model and Qwen3-0.6B, Qwen3-1.7B, or Qwen3-4B as the frozen representation model. Columns correspond to MATH-500, MMLU-Pro, PopQA, and GPQA. Rows report running-average Type-II error, Type-I error, and average editor cost per round. Dashed horizontal lines indicate the configured Type-I error tolerance and per-round editor budget.
Despite the promise of AI to assist complex decisions, practitioners still lack ways to detect likely failures and inspect the consequences of model edits before committing them. We present RuleEdit, an interactive, rule-guided human-AI model editing system that (i) surfaces likely failures through interpretable mismatch signals from rule tables and (ii) supports user-authored rule feedback with prospective previews of projected performance changes and embedding shifts. We instantiate RuleEdit in stroke rehabilitation assessment and evaluate it with health professionals and students. Rule-guided failure detection significantly increased Human + AI performance by 14.16% (p<0.001) while improving rejection of incorrect AI and reducing both over- and under- reliance as well as ChangedToWrong decisions. In addition, presenting prospective embedding previews improved participants' feedback for model adaptation, increasing post-update local performance gains from 11.50% to 36.38% after incorporating users' rule-based feedback (p<0.001). Our findings show that mismatch-based failure cues and prospective impact previews can support failure-aware human-AI model editing, while also revealing a local-global tradeoff: edits that help a specific case can degrade performance when transferred globally. We discuss implications of designing failure-aware and controllable human-AI systems.
We study the ability of LLMs to generate edit-inducing questions whose answer will improve a paper draft. On a dataset of paired submission and camera-ready papers from ICLR and NeurIPS, we compare the helpfulness of questions from GPT models with or without full paper context to that of human reviewers. GPT produces more edit-inducing questions and its questions are associated with more extensive edits and cover a broader range of edited content compared to questions from reviewers. However, a much smaller percentage of the GPT questions are edit-inducing. Our analyses confirm that automated questions can be beneficial to authors and highlight an example task where proper attending to long context deteriorates reasoning model ability to produce helpful output.
Sebastian Joseph, Zichao Wang, Jennifer Healey +3
The University of Texas at Austin · Adobe Research
Local factual edits in scientific manuscripts often create non-local revision obligations. If a dataset changes from 215 to 80 documents, claims such as 'medium-scale' or 'a few hundred items' may also become stale, even though they do not repeat the edited number. In an audit of recent arXiv cs.CL benchmark and dataset papers, we find fact-dependent qualitative claims in 37.2% of papers, suggesting that this dependency pattern is common in the target genre. We introduce EditPropBench, a benchmark for measuring whether LLM editors propagate factual edits through dependent manuscript claims. Each item contains an ML/NLP-style synthetic manuscript, a targeted edit, and a controlled fact graph with sentence-level labels for direct targets, required downstream updates, and unrelated text that should remain unchanged. We summarize cascade success with Edit-Ripple Adherence (ERA), the fraction of required downstream updates correctly revised, and validate the metric with adversarial probes and stress-test variants. On the hardest cases, where dependent claims use implicit or free-form wording rather than repeating the edited value, five LLM editing systems span ERA 0.148-0.705. Even the strongest misses roughly 30% of required cascade updates. This advantage persists in a mixed evaluation that includes easy cases solvable by deterministic substitution. EditPropBench shows that current LLM editors can repair many implicit consequences of factual edits, but reliable scientific revision still requires cascade-aware checking.