Is This Evidence Decision-Critical? Learning to Verify Rule-Governed Decisions
Authors: Haoyang Zhang, Jianpeng Zhao, Qi Hao, Pengyang Wang
Organizations: State Key Laboratory of Internet of Things for Smart City and Institute of Smart City Technologies, University of Macau, Macau SAR, China
Rule-based reasoning, as in eligibility checks and contract reviews, requires language models to assess evidence against individual conditions and combine their judgments under explicit rules. Errors in evidence assessment can leave a decision unchanged, but misinterpreting or overlooking decision-critical evidence can reverse it. Identifying such evidence allows more capable models to focus on checking the corresponding condition judgments, supporting accurate and safe decisions. Recognizing the evidence's criticality requires understanding how evidence affects a condition judgment and how that judgment affects the decision. To achieve the goal, we propose a INTERvention-based imPACT learning framework (InterPact), which enables counterfactual verification of evidence criticality in rule-governed decisions. Specifically, its evidence intervention constructor generates training pairs for a propagation verifier by editing case facts with a frozen language model while holding rules and non-target conditions fixed. Human-reviewed labels record the resulting condition and decision changes, while complete state-to-decision mappings supervise consequences beyond the observed edit. During training, the verifier weights learned conditional decision predictions by evidence-based condition probabilities through a fixed composition operation, propagating decision-change supervision into the base model. At inference, the trained base model directly judges criticality from the original case and target evidence, without human or stronger-model supervision. On single-case evidence criticality verification over adapted rule-governed decision cases, InterPact achieves 68.28% accuracy, outperforming all six baselines. These results support learned decision sensitivity as a basis for prioritizing evidence checks.
Figures & tables
Figure 1: The same evidence change can have different decision consequences. A change in the target condition state changes the decision in one case but leaves it unchanged in the other because the remaining condition states differ.
Figure 2: Overview of InterPact . (a) The Evidence Intervention Constructor produces labeled intervention pairs and complete condition-to-decision mappings. (b) The InterPact Verifier learns from these outputs through propagation composition and Hard Warrant. (c) The trained language model directly judges evidence criticality from the original case and target evidence unit.
Table 1: Direct criticality on 207 roots (%). Bold / Underline : best/second-best classification scores.
State Key Laboratory of Internet of Things for Smart City and Institute of Smart City Technologies, University of Macau, Macau SAR, China · ByteDance, Beijing, China