RuleEdit: Failure-Guided Human-AI Model Editing with Prospective Impact Preview

Min Hun Lee; Justin Yu Feng Teo

cs.HCApr 12, 2026

RuleEdit: Failure-Guided Human-AI Model Editing with Prospective Impact Preview

Authors: Min Hun Lee, Justin Yu Feng Teo

Organizations: Singapore Management University, Singapore

Abstract

Despite the promise of AI to assist complex decisions, practitioners still lack ways to detect likely failures and inspect the consequences of model edits before committing them. We present RuleEdit, an interactive, rule-guided human-AI model editing system that (i) surfaces likely failures through interpretable mismatch signals from rule tables and (ii) supports user-authored rule feedback with prospective previews of projected performance changes and embedding shifts. We instantiate RuleEdit in stroke rehabilitation assessment and evaluate it with health professionals and students. Rule-guided failure detection significantly increased Human + AI performance by 14.16% ( $p<0.001$ ) while improving rejection of incorrect AI and reducing both over- and under- reliance as well as ChangedToWrong decisions. In addition, presenting prospective embedding previews improved participants' feedback for model adaptation, increasing post-update local performance gains from 11.50% to 36.38% after incorporating users' rule-based feedback ( $p<0.001$ ). Our findings show that mismatch-based failure cues and prospective impact previews can support failure-aware human-AI model editing, while also revealing a local-global tradeoff: edits that help a specific case can degrade performance when transferred globally. We discuss implications of designing failure-aware and controllable human-AI systems.

Figures & tables

Figure 1. RuleEdit supports failure inspection, user-authored editing, and prospective preview in AI-assisted decision-making. Given a new case, users first make an initial judgment and then review the system’s AI output together with rule-guided mismatch cues that highlight likely inconsistencies between the model prediction and rule-based evidence. Users then finalize their decision (accept/override). For refinement, users can provide label feedback (relabel) and/or rule feedback (select rules and edit thresholds). The system computes a prospective preview including projected $\Delta$ performance and embedding-shift summaries before users decide whether to commit the update.

Panel of Figure 2 — Figure 2. The system interface includes the video of a post-stroke survivor and the corresponding AI output (prediction, confidence score, and feature-based explanation). (a) The failure-inspection view, where mismatch cues surface likely disagreements between the AI prediction and rule-based evidence to support accept/override decisions, and (b) the editing view, where users select or edit rules and inspect projected performance change together with before/after embedding-shift previews (updated neighborhoods and class centroids) before committing an update.

Figure 3. Two-phase within-subject study. In Phase 1 (AI-assisted decisions), participants complete tasks under Condition A (baseline) and Condition B with rule-based failure detection. In Phase 2 (Label/rule-based feedback), participants review prospective updates under Condition A (projected performance-only preview) and Condition B (projected performance + embedding shifts preview).

	Human	Human–AI Team	Team gain vs. Human	Regret_best
Condition A	67.69	69.71	2.02	16.46
Condition B	67.86	82.02	14.16	4.80
$\Delta$ B-A (p-value)	0.17 (p=0.86)	$\uparrow$ 12.31 (p $<$ 0.001)	$\uparrow$ 12.14 (p $<$ 0.01)	$\downarrow$ -11.66 (p $<$ 0.01)

Table 1. Performance and team-gain metrics. Compared with baseline AI assistance (Condition A), rule-guided mismatch cues (Condition B) significantly improved Human-AI Team accuracy and team gain, while reducing avoidable errors, Regret_best.

	Accept on_correct	Reject on_wrong	Reject on_correct	Accept on_wrong	AI_harm rate
ConditionA	88.09	41.61	11.91	58.39	8.29
ConditionB	95.24	62.14	4.76	37.86	3.70
$\Delta$ B-A (p-value)	$\uparrow$ 7.15 (p $<$ 0.01)	$\uparrow$ 20.53 (p $<$ 0.05)	$\downarrow$ -7.15 (p $<$ 0.01)	$\downarrow$ -20.53 (p $<$ 0.01)	$\downarrow$ -4.59 (p $<$ 0.05)

Table 2. Reliance metrics conditioned on AI correctness. Rule-guided mismatch cues improved reliance calibration by increasing acceptance of correct AI and rejection of incorrect AI, while reducing both over/under-reliance and AI-induced harm.

	Changed	ChangedToRight	ChangedToWrong
ConditionA	34.98	14.74	12.72
ConditionB	26.66	18.88	4.72
$\Delta$ B-A (p-value)	$\downarrow$ -8.32 (p $<$ 0.1)	4.14 (p=0.13)	$\downarrow$ -8.00 (p $<$ 0.01)

Table 3. Decision-change behaviors after viewing AI outputs. Rule-guided mismatch cues (Condition B) reduced overall switching and significantly reduced harmful changes from correct initial judgments to incorrect final decisions.

Metric	A	B	$\Delta$ B–A	$p$
Local update $\Delta$ Perf (overall)	11.50	36.38	24.89	$<0.001$
Global update $\Delta$ Perf (overall)	2.38	-5.05	-7.43	$<0.001$

Table 4. Phase 2 refinement performance for all participants. We report improvement over the baseline model on a held-out evaluation set for local (participant-specific) and global (across-patients) updates.

p

-values are from paired Wilcoxon signed-rank tests across participants.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	ROM				COMP				Overall
Top- $K$	RuleMedian	RuleAvg	RulePercentile	RuleTree	RuleMedian	RuleAvg	RulePercentile	RuleTree	RuleMedian	RuleAvg	RulePercentile	RuleTree
Top10	0.8544	0.8548	0.6505	0.8094	0.8455	0.8503	0.8311	0.8226	0.8500	0.8525	0.7408	0.8160
Top9	0.8701	0.8673	0.6719	0.8311	0.8496	0.8533	0.8369	0.8312	0.8598	0.8603	0.7544	0.8312
Top8	0.8833	0.8774	0.6915	0.8442	0.8522	0.8559	0.8410	0.8407	0.8678	0.8667	0.7663	0.8424
Top7	0.8970	0.8883	0.7154	0.8599	0.8547	0.8587	0.8445	0.8505	0.8759	0.8735	0.7800	0.8552
Top6	0.9077	0.8983	0.7439	0.8769	0.8574	0.8612	0.8471	0.8593	0.8826	0.8797	0.7955	0.8681

Appendix

Table 5. Top-

K

rule performance across rule construction methods for ROM, COMP, and Overall.

ID	Age	Gender	Role	Setting / Others	Experience
TP1	35 - 44 years	Female	Occupational Therapist (OT)	Inpatient Rehabilitation	13
TP2	25 - 34 years	Female	Occupational Therapist (OT)	Outpatient Clinic	8
TP3	25 - 34 years	Female	Occupational Therapist (OT)	Outpatient Clinic	9
TP4	25 - 34 years	Female	Occupational Therapist (OT)	Skilled Nursing Facility	6
TP5	25 - 34 years	Male	Occupational Therapist (OT)	Other (please specify)	7
TP6	55 - 64 years	Female	PhysioTherapist (PT)	Skilled Nursing Facility	30

Appendix

Table 6. Detailed Demographics of Participants: Domain experts, therapists, who have experience in stroke rehabilitation more than one year (TP1 - TP9) and novices (NV1 - NV12) (i.e. other health professionals and students majoring in medicine, therapy)

	All			Experts			Novices
	Human	Human-AI	Changes	Human	Human-AI	Changes	Human	Human-AI	Changes
ConditionA	67.69	69.71	$\uparrow$ 2.02 (p=0.47)	71.35	70.19	$\downarrow$ -1.16 (p=0.56)	64.94	69.35	$\uparrow$ 4.40 (p=0.18)
ConditionB	67.86	82.02	$\uparrow$ 14.16 (p<0.001)	72.22	83.43	$\uparrow$ 11.21 (p<0.05)	64.58	80.95	$\uparrow$ 16.37 (p<0.01)

Appendix

Table 7. Performance of decision-making without/with AI-assistance by all participants (All), domain experts (EXPs), and health professionals and students (NVs)

	All			Experts			Novices
Metrics	CondA	CondB	$\Delta$ B-A (P-Value)	CondA	CondB	$\Delta$ B-A (P-Value)	CondA	CondB	$\Delta$ B-A (P-Value)
Human	67.69	67.86	0.17 (p=0.86)	71.35	72.22	0.87 (p=0.61)	64.94	64.58	-0.36 (p=0.90)
AI	70.76	73.00	2.24 (p=0.14)	70.58	73.31	2.73 (p=0.5)	70.89	72.77	1.88 (p=1.00)
Human-AI Team	69.71	82.02	$\uparrow$ 12.31 (p<0.001)	70.19	83.43	$\uparrow$ 13.25 (p<0.05)	69.35	80.95	$\uparrow$ 11.61 (p<0.05)
Human-AI Team_gain vs Human	2.02	14.16	$\uparrow$ 12.14 (p<0.01)	-1.16	11.21	$\uparrow$ 12.37 (p<0.05)	4.4	16.37	$\uparrow$ 11.96 (p<0.05)
Human-AI Team_gain vs_AI	-1.05	9.01	$\uparrow$ 10.07 (p<0.05)	-0.4	10.12	10.52 (p=0.20)	-1.55	8.18	$\uparrow$ 9.73 (p<0.1)

Appendix

Table 8. Detailed performance and team-gain metrics for all participants (All), domain experts (EXPs), and health professionals and students (NVs)

	All			Experts			Novices
Metrics	CondA	CondB	$\Delta$ B-A (P-Value)	CondA	CondB	$\Delta$ B-A (P-Value)	CondA	CondB	$\Delta$ B-A (P-Value)
Accept_on_correct	88.09	95.24	$\uparrow$ 7.15 (p<0.01)	86.4	98.89	$\uparrow$ 12.49 (p<0.01)	89.36	92.5	3.14 (p=0.34)
Reject_on_wrong	41.61	62.14	$\uparrow$ 20.53 (p<0.05)	42.33	56.11	13.78 (p=0.40)	41.07	66.67	25.6 (p=0.1)
Reject_on_correct	11.91	4.76	$\downarrow$ -7.15 (p<0.01)	13.6	1.11	$\downarrow$ -12.49 (p<0.01)	10.64	7.5	-3.14 (p=0.34)
Accept_on_wrong	58.39	37.86	$\downarrow$ -20.53 (p<0.01)	57.67	43.89	-13.78 (p=0.40)	58.93	33.33	-25.6 (p=0.1)
AI_help_rate	13.76	15.9	2.14 (p=0.58)	10.85	15.08	4.23 (p=0.62)	15.95	16.52	0.57 (p=0.88)

Appendix

Table 9. Detailed acceptance/rejection metrics capturing reliance on AI for all participants (All), domain experts (EXPs), and health professionals and students (NVs)

	All			Experts			Novices
Metrics	CondA	CondB	$\Delta$ B-A (P-Value)	CondA	CondB	$\Delta$ B-A (P-Value)	CondA	CondB	$\Delta$ B-A (P-Value)
Changed_Rate	34.98	26.66	$\downarrow$ -8.32 (p<0.1)	32.41	26.09	-6.32 (p=0.41)	36.9	27.08	-9.82 (p=0.10)
ChangedToRight_Rate	14.74	18.88	4.14 (p=0.13)	12.57	17.46	4.89 (p=0.34)	16.37	19.94	3.57 (p=0.27)
ChangedToWrong_rate	12.72	4.72	$\downarrow$ -8.00 (p<0.01)	13.73	6.25	$\downarrow$ -7.48 (p<0.05)	11.96	3.57	$\downarrow$ -8.39 (p<0.05)

Appendix

Table 10. Ratio of Changed, ChangedToRight, and ChangedToWrong rates after reviewing AI outputs (Human + AI) for all participants (All), domain experts (EXPs), and health professionals and students (NVs)

	All (Experts+Novices)				Experts				Novices
Metric	A	B	$\Delta$ B–A	$p$	A	B	$\Delta$ B–A	$p$	A	B	$\Delta$ B–A	$p$
Local update $\Delta$ Perf (overall)	11.50	36.38	24.89	<0.001	11.20	33.89	22.69	<0.001	11.92	38.88	26.96	<0.05
Global update $\Delta$ Perf (overall)	2.38	-5.05	-7.43	<0.001	5.58	-2.95	-8.52	<0.001	0.00	-6.46	-6.46	<0.01

Appendix

Table 11. Phase 2 refinement performance. We report improvement over the baseline model on a held-out evaluation set for local (participant-specific) and global (across-patients) updates.

p

from paired Wilcoxon signed-rank tests across participants.

Model Editing

Model editing changes targeted facts or behaviors in trained models while aiming to preserve their other capabilities.

See all 64 papers

Edit Discovery in Human-AI Interactions
Oct 15, 2026Alireza Mirrokni, Shayan Kiyani, George Pappas +1Human-in-the-Loop Evaluation Human-AI Interaction
EPFL · University of Pennsylvania
Answer Engineering: Local Trajectory Editing for Protocol-Constrained Decision Making in Large Language Models
Jun 19, 2026Victor Lavrenko, Anastasiia MolodnitskaiaLanguage Model Steering Clinical Decision Support
PeaceTech VC, Israel · Rambam Health Care Campus, Israel
Generalizable Model Editing Could Be More Than Direct Generalization
Oct 11, 2026Xiaotian Ye, Chunyao Yang, Xiaohan Wang +4Model Editing Language Model Editing
University of Cambridge · NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of the Chinese Academy of Sciences +3

Oct 15, 2026J/K move · Enter open · S save

Edit Discovery in Human-AI Interactions

Alireza Mirrokni, Shayan Kiyani, George Pappas, Hamed Hassani

EPFL · University of Pennsylvania

Human-in-the-Loop Evaluation Human-AI Interaction Active Learning

Open paper AskPDF arXiv

Human-AI interactions are becoming a rich source of data for improving AI systems: while many model responses are already satisfactory, others expose failures that could become valuable supervision if identified and corrected. We study this setting through the notion of an editor: a more capable evaluation process that can inspect a response and either confirm it or modify it, but whose invocation is costly. The editor may be a human expert, a stronger AI system, or the same model equipped with additional computation, context, tools, or information. The key challenge is to decide which interactions to send to the editor before knowing whether an edit will actually be made, since invoking the editor itself consumes limited resources such as expert time, inference cost, latency, or tool-use overhead. We formulate this problem as Edit Discovery Optimization (EDO), which seeks to minimize edit opportunities missed among skipped interactions while controlling unnecessary confirmations and respecting an editor-cost budget. We characterize the population-optimal policy and show that it takes the form of a cost-adjusted threshold on the posterior probability of an edit. Building on this structure, we develop EDIT (Edit Discovery via Iterative Thresholding), an online algorithm for the partial-feedback setting in which edit labels are revealed only for selected interactions. EDIT combines an adaptive edit score, a confirmation-control threshold, and a cost-aware selection rule; it satisfies the editor budget at every round and controls the average confirmation rate up to a vanishing finite-horizon term, without distributional assumptions on the interaction stream. Across four reasoning and factual question-answering benchmarks, EDIT achieves a better trade-off between editor cost, Type-I error, and Type-II error than direct LLM-based batch selection.

Abstract

Model Editing

Explore similar work

Edit Discovery in Human-AI Interactions

Answer Engineering: Local Trajectory Editing for Protocol-Constrained Decision Making in Large Language Models

Generalizable Model Editing Could Be More Than Direct Generalization