Case-Grounded Evidence Verification: A Framework for Constructing Evidence-Sensitive Supervision

Soroosh Tayebi Arasteh; Mehdi Joodaki; Mahshad Lotfinia; Sven Nebelung; Daniel Truhn

cs.CLApr 10, 2026

Case-Grounded Evidence Verification: A Framework for Constructing Evidence-Sensitive Supervision

Authors: Soroosh Tayebi Arasteh, Mehdi Joodaki, Mahshad Lotfinia, Sven Nebelung, Daniel Truhn

Organizations: RWTH Aachen University, Aachen, Germany

Abstract

Evidence-grounded reasoning requires more than attaching retrieved text to a prediction: a model should make decisions that depend on whether the provided evidence supports the target claim. In practice, this often fails because supervision is weak, evidence is only loosely tied to the claim, and evaluation does not test evidence dependence directly. We introduce case-grounded evidence verification, a general framework in which a model receives a local case context, external evidence, and a structured claim, and must decide whether the evidence supports the claim for that case. Our key contribution is a supervision construction procedure that generates explicit support examples together with semantically controlled non-support examples, including counterfactual wrong-state and topic-related negatives, without manual evidence annotation. We instantiate the framework in radiology and train a standard verifier on the resulting support task. The learned verifier substantially outperforms both case-only and evidence-only baselines, remains strong under correct evidence, and collapses when evidence is removed or swapped, indicating genuine evidence dependence. This behavior transfers across unseen evidence articles and an external case distribution, though performance degrades under evidence-source shift and remains sensitive to backbone choice. Overall, the results suggest that a major bottleneck in evidence grounding is not only model capacity, but the lack of supervision that encodes the causal role of evidence.

Figures & tables

System	AUROC	AUPRC	F1	Bal. Acc.	Brier
S1: Case + Claim	59.90	26.23	32.21	58.40	0.201
S2: Evidence + Claim	82.63	35.11	62.79	87.53	0.255
S3: Case + Evidence + Claim	97.43	87.78	78.72	92.03	0.060

Table 1: Core system comparison on MIMIC-CXR.

Figure 2: Intervention profile of the case-grounded verifier on MIMIC-CXR, showing rapid saturation with additional evidence sentences and sharp degradation when evidence is removed or misaligned.

Figure 3: Representative MIMIC-CXR test example under evidence interventions. The verifier assigns high support under correct evidence, but low support when evidence is swapped or removed.

System / Condition	AUROC	AUPRC	F1	Bal. Acc.	Brier
S1: Case + Claim	52.85	29.16	28.92	53.59	0.266
S2: Evidence + Claim	79.81	42.46	66.45	83.16	0.297
S3: Case + Evidence + Claim	93.46	78.09	75.15	87.45	0.129
Held-Out Articles	88.37	61.23	72.13	86.47	0.193
Swapped Evidence	53.18	28.52	31.69	53.62	0.352

Table 2: External transfer from MIMIC-CXR to CheXpert-Plus.

	Source-Matched Evidence			Held-Out Evidence
Backbone	AUPRC	F1	Brier	AUPRC	F1	Brier
Flan-T5-large (751M) ( 8 )	84.76	75.33	0.071	74.04	77.19	0.108
ModernBERT-large (395M) ( 61 )	87.78	78.72	0.060	74.18	77.94	0.075
RoBERTa-large (355M) ( 37 )	78.18	63.50	0.084	64.98	77.61	0.116
ELECTRA-L-discriminator (335M) ( 9 )	77.07	66.77	0.094	25.52	34.67	0.126
Flan-T5-base (223M) ( 8 )	82.00	76.54	0.064	55.14	67.84	0.125

Table 3: Backbone ablation on MIMIC-CXR under source-matched and held-out evidence.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Pool	Intended Semantics
$\mathcal{E}_{k}^{+}$	Supports that concept $k$ is present
$\mathcal{E}_{k}^{-}$	Supports that concept $k$ is absent
$\mathcal{E}_{k}^{0,h}$	Topically related to concept $k$ but not supportive
$\mathcal{E}_{k}^{0,e}$	Easy non-support evidence for concept $k$

Appendix

Table S1: Support-pool schema for a binary concept

k

. The concrete assignment rules are domain-specific but fixed once as part of the protocol.

Diagnostic	Value
Total verifier examples $\|\mathcal{D}_{\mathrm{ver}}\|$	18,397
Positive fraction $\pi_{+}$	0.17
Wrong-state fraction among negatives $\rho_{D}$	0.25
Mean examples per concept	2628.14
Mean examples per case	3.20
Fraction of present-state claims	0.55

Appendix

Table S2: Suggested diagnostics for a concrete instantiation of the support-structured supervision operator. All statistics are computed after dataset construction and before model training.

Category	Gold State / Claim	Label	Evidence Excerpt
$\mathcal{D}_{C}$	gold=absent / claim=pleural effusion is absent	1	Classically demonstrated in M-mode, the appearance of which the moniker is derived, it is specific for the identification of a pleural effusion, although ins…
$\mathcal{D}_{D}$	gold=present / claim=cardiomegaly is absent	0	There is no associated cardiomegaly, and the radiographic changes often rapidly disappear in patients who survive the acute event.
$\mathcal{D}_{A}$	gold=present / claim=cardiomegaly is present	0	The hematoma itself is typically relatively small, thus fat saturated sequences are crucial by pinpointing the common surrounding soft tissue edema.
$\mathcal{D}_{B}$	gold=present / claim=cardiomegaly is absent	0	It shows a non-uniform distribution around the heart.

Appendix

Table S3: Representative supervision-construction examples. Each row shows a concrete example under the frozen protocol.

Stage	Count
Reports initially ingested from MIMIC-CXR	227,835
Reports retained after deterministic inclusion criteria	227,835
Reports with valid Findings and Impression sections	37,811
Unique patients retained	20,237

Appendix

Table S4: MIMIC-CXR report-level statistics at successive deterministic preprocessing stages.

Statistic	Value
Retained reports yielding at least one valid concept-state claim	31,225
Total retained report-concept pairs before shortcut filtering	55,262
Report-concept pairs excluded by shortcut filtering	46,346
Retained report-concept pairs in the main benchmark	8,916
Mean retained concepts per report	1.2
Distinct pathology concepts in the fixed claim space	10

Appendix

Table S5: Case-source statistics after deterministic claim construction on MIMIC-CXR.

Statistic	Value
Mean tokens per case context	93.0
Standard deviation	31.9
Median	87.0
95th percentile	150.0
Maximum	360.0

Appendix

Table S6: Token-length statistics for MIMIC-CXR case contexts under the verifier tokenizer.

Statistic	Value
Snapshot date	March 1, 2026
Articles in raw snapshot	17,293
Retained articles after initial preprocessing and deduplication	14,391
Articles represented in the cleaned sentence universe	14,199
Total retained evidence sentences $\|\mathcal{U}\|$	55,624
Mean tokens per sentence	25.4

Appendix

Table S7: Radiopaedia snapshot and sentence-universe statistics.

Subset	Articles	Evidence Sentences
Training/validation article subset $\mathcal{A}_{\mathrm{tv}}$	11,361	44,390
Held-out article subset $\mathcal{A}_{\mathrm{ho}}$	2,838	11,234

Appendix

Table S8: Article-level split statistics for the Radiopaedia evidence universe.

Statistic	Value
Mean concept coverage $\mathrm{cov}(k)$	115.0
Median concept coverage	79.0
Minimum concept coverage	7
Maximum concept coverage	326
Concepts with non-empty support-present pool	6
Concepts with non-empty support-absent pool	3

Appendix

Table S9: Concept coverage statistics in the sentence-level evidence universe.

Statistic	Train	Validation	Test
Total verifier examples	14,173	1,659	2,565
Positive examples ( $\mathcal{D}_{C}$ )	2,369	266	446
Negative examples ( $\mathcal{D}_{A}\cup\mathcal{D}_{B}\cup\mathcal{D}_{D}$ )	11,804	1,393	2,119
Positive fraction	0.2	0.2	0.2
Wrong-state negative fraction $\mu_{s}(D)$	0.2	0.2	0.2

Appendix

Table S10: Verifier-dataset statistics after support-structured supervision construction.

Statistic	Value
Training token budget	1024
Mean tokens per verifier input	191.4
Standard deviation	39.0
Median	186.0
95th percentile	262.0
Fraction exceeding 1024 tokens	0.0

Appendix

Table S11: Verifier input length statistics after formatting and before truncation.

	Training	Validation	Test
Patients	15,283	1,923	3,031
Reports	28,683	3,608	5,520
Retained report-concept pairs	6,840	844	1,232
Verifier examples	14,173	1,659	2,565

Appendix

Table S12: Patient-wise split statistics for the MIMIC-CXR instantiation.

Stage	Count
Reports initially ingested from CheXpert-Plus	223,460
Reports retained after deterministic inclusion criteria	55,590
Retained report-concept pairs	23,249
Verifier examples in external evaluation artifact	20,088

Appendix

Table S13: CheXpert-Plus preprocessing and evaluation-asset statistics.

Statistic	Value
Mean tokens per case context	104.7
Standard deviation	50.0
Median	92.0
95th percentile	202.0
Maximum	561.0

Appendix

Table S14: Token-length statistics for CheXpert-Plus case contexts under the verifier tokenizer.

Hyperparameter	Value
Backbone	ModernBERT-large ( 61 )
Training objective	Weighted binary cross-entropy
Class weighting	Inverse-frequency
Training token budget	1024
Evidence sentences used in training	2
Optimizer	AdamW

Appendix

Table S15: Verifier training configuration for the reported primary system.

Metric	S1: Case + Claim	S2: Evidence + Claim	S3: Case + Evidence + Claim
AUROC	59.90 $\pm$ 1.60 [57–63]	82.63 $\pm$ 0.77 [81–84]	97.43 $\pm$ 0.27 [97–98]
AUPRC	26.23 $\pm$ 1.87 [23–30]	35.11 $\pm$ 1.55 [32–38]	87.78 $\pm$ 1.56 [85–91]
F1 score	32.21 $\pm$ 1.83 [28–35]	62.79 $\pm$ 1.54 [60–66]	78.72 $\pm$ 1.94 [74–82]
Accuracy	65.74 $\pm$ 6.61 [56–78]	79.40 $\pm$ 0.80 [78–81]	91.22 $\pm$ 1.01 [88–93]
Bal. Acc.	58.40 $\pm$ 1.20 [56–61]	87.53 $\pm$ 0.46 [87–88]	92.03 $\pm$ 0.64 [91–93]
Sensitivity	47.14 $\pm$ 10.52 [26–63]	100.00 $\pm$ 0.00 [100–100]	93.28 $\pm$ 1.48 [91–97]

Appendix

Table S16: Extended core system comparison on MIMIC-CXR under correct source-matched evidence. Entries report bootstrap mean

\pm

standard deviation with 95% CIs.

Comparison	$\Delta$ AUROC	$\Delta$ AUPRC	$\Delta$ F1	$\Delta$ Bal. Acc.	$\Delta$ Brier
S2 $-$ S1	22.73	8.88	30.58	29.13	0.054
S3 $-$ S1	37.53	61.55	46.51	33.63	-0.141
S3 $-$ S2	14.80	52.67	15.93	4.50	-0.195

Appendix

Table S17: Differences between reported bootstrap mean metrics for the core system comparison on MIMIC-CXR. Positive values indicate improvement for AUROC, AUPRC, F1, and balanced accuracy, whereas negative values indicate improvement for Brier.

Metric	Correct, Source-Matched	Correct, Held-Out Articles	Swapped Evidence
AUROC	97.43 $\pm$ 0.27 [97–98]	97.24 $\pm$ 0.33 [97–98]	55.62 $\pm$ 1.50 [53–59]
AUPRC	87.78 $\pm$ 1.56 [85–91]	74.18 $\pm$ 3.20 [68–80]	21.73 $\pm$ 1.54 [19–25]
F1 score	78.72 $\pm$ 1.94 [74–82]	77.94 $\pm$ 1.91 [74–82]	29.77 $\pm$ 2.84 [23–33]
Accuracy	91.22 $\pm$ 1.01 [88–93]	92.87 $\pm$ 0.65 [92–94]	52.38 $\pm$ 11.86 [39–78]
Bal. Acc.	92.03 $\pm$ 0.64 [91–93]	95.86 $\pm$ 0.33 [95–97]	55.34 $\pm$ 1.06 [53–57]
Sensitivity	93.28 $\pm$ 1.48 [91–97]	99.87 $\pm$ 0.34 [99–100]	59.87 $\pm$ 18.72 [20–80]

Appendix

Table S18: Extended evidence-identity interventions on MIMIC-CXR. Entries report bootstrap mean

\pm

standard deviation with 95% CIs.

Metric	Top-1	Top-2	Top-3	Top-4	Top-5
AUROC	90.70 $\pm$ 0.85	97.43 $\pm$ 0.27	97.49 $\pm$ 0.27	97.50 $\pm$ 0.27	97.50 $\pm$ 0.27
AUPRC	77.08 $\pm$ 1.94	87.78 $\pm$ 1.56	88.03 $\pm$ 1.55	88.07 $\pm$ 1.55	88.07 $\pm$ 1.55
F1 score	72.94 $\pm$ 1.91	78.72 $\pm$ 1.94	78.83 $\pm$ 1.81	78.86 $\pm$ 1.82	78.86 $\pm$ 1.82
Accuracy	89.87 $\pm$ 0.88	91.22 $\pm$ 1.01	91.24 $\pm$ 0.88	91.26 $\pm$ 0.89	91.26 $\pm$ 0.89
Bal. Acc.	85.37 $\pm$ 1.00	92.03 $\pm$ 0.64	92.21 $\pm$ 0.63	92.22 $\pm$ 0.64	92.22 $\pm$ 0.64
Sensitivity	78.48 $\pm$ 2.16	93.28 $\pm$ 1.48	93.70 $\pm$ 1.31	93.68 $\pm$ 1.32	93.68 $\pm$ 1.32

Appendix

Table S19: Evaluation-time evidence-quantity ablations on MIMIC-CXR under correct source-matched evidence for the default verifier trained with top-2 evidence.

Figure S1: Full train-versus-test top- $p$ evidence-quantity ablation on MIMIC-CXR under source-matched and held-out evidence.

Training-time budget	Condition	AUROC	AUPRC	F1	Brier
Random $\{2,3\}$	Source-matched	96.12	84.21	71.95	0.061
Random $\{2,3,4\}$	Source-matched	93.43	78.53	67.17	0.077
Random $\{2,3,4,5\}$	Source-matched	96.10	84.68	76.99	0.067
Random $\{2,3\}$	Held-out	97.44	77.14	77.73	0.073
Random $\{2,3,4\}$	Held-out	96.29	67.76	77.59	0.086
Random $\{2,3,4,5\}$	Held-out	97.96	83.92	77.09	0.067

Appendix

Table S20: Stochastic training-time evidence-budget results on MIMIC-CXR, evaluated at top-2 evidence.

Metric	S1: Case + Claim	S2: Evidence + Claim	S3: Case + Evidence + Claim
AUROC	59.90 $\pm$ 1.60 [57–63]	79.81 $\pm$ 0.75 [78–81]	93.46 $\pm$ 0.41 [93–94]
AUPRC	26.23 $\pm$ 1.87 [23–30]	42.46 $\pm$ 1.35 [40–45]	78.09 $\pm$ 1.70 [75–81]
F1 score	32.21 $\pm$ 1.83 [28–35]	66.45 $\pm$ 1.14 [64–69]	75.15 $\pm$ 1.27 [73–78]
Accuracy	65.74 $\pm$ 6.61 [56–78]	74.75 $\pm$ 0.79 [73–76]	84.57 $\pm$ 1.01 [83–87]
Bal. Acc.	58.40 $\pm$ 1.20 [56–61]	83.16 $\pm$ 0.49 [82–84]	87.45 $\pm$ 0.60 [86–89]
Sensitivity	47.14 $\pm$ 10.52 [26–63]	100.00 $\pm$ 0.00 [100–100]	93.20 $\pm$ 1.69 [90–96]

Appendix

Table S21: External-transfer results from MIMIC-CXR to CheXpert-Plus under correct source-matched evidence. Entries report bootstrap mean

\pm

standard deviation with 95% CIs.

Metric	Correct, Source-Matched	Correct, Held-Out Articles	Swapped Evidence
AUROC	93.46 $\pm$ 0.41 [93–94]	88.37 $\pm$ 0.70 [87–90]	53.18 $\pm$ 1.30 [51–56]
AUPRC	78.09 $\pm$ 1.70 [75–81]	61.23 $\pm$ 2.22 [57–66]	28.52 $\pm$ 1.39 [26–31]
F1 score	75.15 $\pm$ 1.27 [73–78]	72.13 $\pm$ 1.42 [69–75]	31.69 $\pm$ 5.77 [21–41]
Accuracy	84.57 $\pm$ 1.01 [83–87]	81.32 $\pm$ 1.04 [79–83]	60.37 $\pm$ 11.20 [35–72]
Bal. Acc.	87.45 $\pm$ 0.60 [86–89]	86.47 $\pm$ 0.68 [85–88]	53.62 $\pm$ 0.85 [52–55]
Sensitivity	93.20 $\pm$ 1.69 [90–96]	96.76 $\pm$ 1.12 [94–99]	40.13 $\pm$ 22.16 [14–89]

Appendix

Table S22: External-transfer results from MIMIC-CXR to CheXpert-Plus. Entries report bootstrap mean

\pm

standard deviation with 95% CIs.

Comparison	$\Delta$ AUROC	$\Delta$ AUPRC	$\Delta$ F1	$\Delta$ Bal. Acc.	$\Delta$ Brier
S3 Source-Matched $-$ S1	33.56	51.86	42.94	29.05	-0.072
S3 Source-Matched $-$ S2	13.65	35.63	8.70	4.29	-0.168
S3 Held-Out $-$ S3 Source-Matched	-5.09	-16.86	-3.02	-0.98	0.064
S3 Swapped $-$ S3 Source-Matched	-40.28	-49.57	-43.46	-33.83	0.223

Appendix

Table S23: Differences between reported mean metrics for the CheXpert-Plus transfer setting.

Figure S2: External transfer profile on CheXpert-Plus across baselines and evidence conditions.

MIMIC-CXR: Source-Matched Evidence
Metric	Flan-T5-L	ModernBERT-L	RoBERTa-L	ELECTRA-L	Flan-T5-B
Parameters (M)	751	395	355	335	223
AUROC	97.11 $\pm$ 0.29	97.43 $\pm$ 0.27	91.54 $\pm$ 0.94	94.79 $\pm$ 0.41	97.11 $\pm$ 0.29
AUPRC	84.76 $\pm$ 1.89	87.78 $\pm$ 1.56	78.18 $\pm$ 1.90	77.07 $\pm$ 2.11	82.00 $\pm$ 2.28
F1 score	75.33 $\pm$ 2.49	78.72 $\pm$ 1.94	63.50 $\pm$ 2.90	66.77 $\pm$ 1.98	76.54 $\pm$ 2.10
Accuracy	88.98 $\pm$ 1.49	91.22 $\pm$ 1.01	81.06 $\pm$ 2.43	83.18 $\pm$ 1.42	89.86 $\pm$ 1.16

Appendix

Table S24: Extended backbone comparison on MIMIC-CXR under correct evidence and external-transfer results from MIMIC-CXR to CheXpert-Plus for the larger backbones.

MIMIC-CXR: Source-Matched Evidence
Metric	ModernBERT-B	Longformer-B	BiomedBERT
Parameters (M)	149	148	109
AUROC	96.56 $\pm$ 0.33	96.68 $\pm$ 0.37	96.83 $\pm$ 0.31
AUPRC	82.21 $\pm$ 2.17	85.55 $\pm$ 1.65	84.83 $\pm$ 1.82
F1 score	76.40 $\pm$ 3.19	74.30 $\pm$ 1.72	75.97 $\pm$ 1.95
Accuracy	90.23 $\pm$ 2.11	88.46 $\pm$ 0.91	89.72 $\pm$ 1.07

Appendix

Table S25: Extended backbone comparison on MIMIC-CXR under correct evidence and external-transfer results from MIMIC-CXR to CheXpert-Plus for the smaller backbones.

ID	Claim	Cat.	Label	S3	Held-Out	S2	S1	Swapped	Interpretation
P1	atelectasis is present	$\mathcal{D}_{C}$	1	0.992	0.898	0.012	0.041	0.000	Canonical positive support with strong evidence dependence and good transfer to unseen evidence articles.
P2	pleural effusion is present	$\mathcal{D}_{C}$	1	0.986	0.993	0.002	0.001	0.001	Near-ideal positive profile: strong under both correct-evidence conditions and near-zero under all destructive or restricted-input conditions.
P3	pulmonary edema is present	$\mathcal{D}_{C}$	1	0.999	0.000	0.002	0.021	0.000	Positive support under source-matched evidence, but complete collapse under held-out evidence, illustrating source-shift sensitivity at the instance level.

Appendix

Table S26: Selected positive instance-level evidence-sensitivity profiles from the MIMIC-CXR test split. Examples are chosen to illustrate canonical support and source-shift sensitivity.

ID	Claim	Cat.	Label	S3	S2	S1	Swapped	Interpretation
N1	pleural effusion is absent	$\mathcal{D}_{D}$	0	0.000	0.991	0.001	0.000	Wrong-state counterfactual negative: evidence-only verification is highly confident, but the full verifier correctly rejects support once the case is included.
N2	fracture is present	$\mathcal{D}_{A}$	0	0.000	0.000	0.003	0.000	Hard non-support negative with uniformly near-zero support across all conditions.
B1	atelectasis is present	$\mathcal{D}_{C}$	1	0.998	0.057	0.985	0.000	Boundary case with strong residual local solvability despite shortcut filtering. The full verifier remains correct, but the example is not evidence-dependent in the strongest possible sense.
B2	pleural effusion is present	$\mathcal{D}_{C}$	1	1.000	0.002	0.001	1.000	Boundary case in which swapped evidence does not reduce support, showing that aggregate degradation under swapping does not imply collapse on every individual row.

Appendix

Table S27: Selected negative and boundary-case instance-level evidence-sensitivity profiles from the MIMIC-CXR test split. Examples are chosen to illustrate wrong-state counterfactual rejection, hard non-support, residual local solvability, and accidental compatibility under swapped evidence.

Field	Value
Target concept	Pleural effusion
Findings-derived case context	Portable chest radiograph with right pigtail catheter, no pneumothorax, minimal right costophrenic blunting without gross effusion, low lung volumes, bilateral non-specific opacities compatible with atelectasis, and no gross left-sided effusion visible in the imaged field
Impression-derived gold state	absent
Structured question	Does the study show no pleural effusion?
Structured claim	Pleural effusion is absent
Label source text	No gross effusion

Appendix

Table S28: Worked example at the claim-construction stage.

Category	Gold state / Claim state	Evidence source type	Label	Evidence text
$\mathcal{D}_{C}$	absent / absent	support absent	1	Classically demonstrated in M-mode, the appearance of which the moniker is derived, it is specific for the identification of a pleural effusion, although insensitive, as it may be absent with dense or heavily septated collections.
$\mathcal{D}_{D}$	absent / present	support present	0	Refer to the article “pleural effusion volume (ultrasound)” for more information. CT scanning is excellent at detecting small amounts of fluid and is also often able to identify the underlying intrathoracic causes.
$\mathcal{D}_{A}$	absent / present	nonsupport hard	0	Non-specific changes with ground-glass opacity have been the most commonly reported findings with nodular consolidation described in a lesser number of patients. Involvement can often be bilateral.
$\mathcal{D}_{B}$	absent / absent	nonsupport easy	0	On chest radiographs, they are seen to cross normal vascular markings and extend radially from the hilum to the upper lobes.

Appendix

Table S29: Worked example after verifier construction. The same local case context induces one representative row from each supervision category.

Explore similar work

Is This Evidence Decision-Critical? Learning to Verify Rule-Governed Decisions
Sep 30, 2026Haoyang Zhang, Jianpeng Zhao, Qi Hao +1LLM Decision-Making
State Key Laboratory of Internet of Things for Smart City and Institute of Smart City Technologies, University of Macau, Macau SAR, China
Coverage-Controlled Preference Mining from Noisy Claim Verification for Evidence-Grounded Generation
Mar 11, 2026Weixin Liu, Congning Ni, Qingyuan Song +4Claim Verification Healthcare
Vanderbilt University, Nashville, TN, USA · Vanderbilt University Medical Center, Nashville, TN, USA · Virginia Tech, Blacksburg, VA, USA
Evidence-RL: Towards Evidence-intensive Visual Reasoning
Aug 8, 2026Haojie Huang, Xinlei Yu, Chengming Xu +6Vision-Language Grounding Visually Grounded Reasoning
National University of Singapore · Zhejiang University · Fudan University +2

Sep 30, 2026J/K move · Enter open · S save

Is This Evidence Decision-Critical? Learning to Verify Rule-Governed Decisions

Haoyang Zhang, Jianpeng Zhao, Qi Hao, Pengyang Wang

State Key Laboratory of Internet of Things for Smart City and Institute of Smart City Technologies, University of Macau, Macau SAR, China

LLM Decision-Making

Open paper AskPDF arXiv

Rule-based reasoning, as in eligibility checks and contract reviews, requires language models to assess evidence against individual conditions and combine their judgments under explicit rules. Errors in evidence assessment can leave a decision unchanged, but misinterpreting or overlooking decision-critical evidence can reverse it. Identifying such evidence allows more capable models to focus on checking the corresponding condition judgments, supporting accurate and safe decisions. Recognizing the evidence's criticality requires understanding how evidence affects a condition judgment and how that judgment affects the decision. To achieve the goal, we propose a INTERvention-based imPACT learning framework (InterPact), which enables counterfactual verification of evidence criticality in rule-governed decisions. Specifically, its evidence intervention constructor generates training pairs for a propagation verifier by editing case facts with a frozen language model while holding rules and non-target conditions fixed. Human-reviewed labels record the resulting condition and decision changes, while complete state-to-decision mappings supervise consequences beyond the observed edit. During training, the verifier weights learned conditional decision predictions by evidence-based condition probabilities through a fixed composition operation, propagating decision-change supervision into the base model. At inference, the trained base model directly judges criticality from the original case and target evidence, without human or stronger-model supervision. On single-case evidence criticality verification over adapted rule-governed decision cases, InterPact achieves 68.28% accuracy, outperforming all six baselines. These results support learned decision sensitivity as a basis for prioritizing evidence checks.

Abstract

Explore similar work

Is This Evidence Decision-Critical? Learning to Verify Rule-Governed Decisions

Coverage-Controlled Preference Mining from Noisy Claim Verification for Evidence-Grounded Generation

Evidence-RL: Towards Evidence-intensive Visual Reasoning