EviGraph: Proof-Carrying Selective Recommendation over Temporal Public-Service Knowledge Graphs

Yixi Zhou; Sikun Wang; Lei Fan; Fan Zhang

cs.AISep 21, 2026

EviGraph: Proof-Carrying Selective Recommendation over Temporal Public-Service Knowledge Graphs

Authors: Yixi Zhou, Sikun Wang, Lei Fan, Fan Zhang

Organizations: Hong Kong Baptist University · Tokyo University of Science · University of Illinois Urbana-Champaign · The University of Tokyo

Abstract

Public-service recommendations require evidence that matches the requested service, scope, and date. Yet treating every missing detail as decisive can withhold useful recommendations. We introduce EviGraph, which distinguishes critical decision requirements from information that can remain unresolved. A language agent links these requirements to evidence in a temporal knowledge graph, while a deterministic checker establishes whether a recommendation is supported. Evaluation on a bilingual Hong Kong public-service benchmark with executable policy references shows that this distinction reduces unnecessary abstention. Additional verification, however, can withdraw supported recommendations without improving decision quality. These findings suggest that reliable evidence-based navigation depends on specifying what must be established for a decision, rather than simply adding more verification.

Figures & tables

View	Group	Queries	Candidates	R/V/E	S/C/T/M
Partition	Base	1,228	6,140	84/5393/663	110/0/0/6030
	Compositional	92	460	31/391/38	40/0/0/420
	Stress	606	606	133/441/32	178/166/55/207
Domain	Child/family	1,261	4,441	248/4141/52	328/122/55/3936
	Disability	276	1,212	0/892/320	0/0/0/1212
	Elderly	389	1,553	0/1192/361	0/44/0/1509

Table 1: Held-out benchmark support, independent of model success. R/V/E denotes automated-policy RECOMMEND/VERIFY/EXCLUDE; S/C/T/M denotes SUPPORTED/CONFLICTING/STALE/MISSING. The three views describe the same cases and must not be summed together. Compositional includes current/future contrasts. Full validation and construction counts are in the supplementary material.

Figure 1: Proposed 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: pipeline. Query semantics are shared across candidates. Only an initial recommendation can enter verification or be restored by adjudication.

Figure 2: Schematic proof conditions, not an observed prediction. Current kindergarten vacancy can provide a valid witness for that service and time; the scope and date mismatch prevents it from satisfying the future occasional-care obligation. A recommendation additionally requires every other critical obligation.

Method	F1	P/R	FRR	F1	P/R	FRR	F1	P/R	FRR
	Base			Compositional			Stress
Rule-only	0.915	1.00/1.00	0.000	0.816	1.00/1.00	0.000	0.927	1.00/1.00	0.000
Direct	0.618	0.09/1.00	0.914	0.772	0.35/1.00	0.652	0.866	0.83/0.93	0.168
Flat RAG	0.620	0.09/1.00	0.909	0.763	0.34/1.00	0.656	0.850	0.88/0.88	0.120
Critic-and-guard	0.825	0.98/0.67	0.018	0.886	1.00/0.81	0.000	0.708	0.99/0.65	0.011
All-field compiler	0.570	–/0.00	–	0.623	–/0.00	–	0.563	–/0.00	–

Table 2: Held-out results with GPT-5-mini as the common language-model backbone. F1 is fixed-label decision macro-F1; P/R is recommendation precision/recall; FRR is false recommendation rate among issued recommendations. Bold marks the highest F1 in each partition. References are automated research-policy labels.

Split	Baseline	$\Delta$ F1	Pointwise 95% cluster CI	Holm $p$
Base	Rule-only	-0.0148	[-0.0329, 0.0020]	0.7813
	Critic-and-guard	0.0755	[0.0282, 0.1293]	$3.37\times 10^{-22}$
	All-field compiler	0.3303	[0.3059, 0.3562]	$4.74\times 10^{-12}$
	0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier	-0.0133	[-0.0263, -0.0039]	0.1652
Comp.	Rule-only	0.0485	[-0.0028, 0.0931]	0.7813
	Critic-and-guard	-0.0214	[-0.1084, 0.0469]	0.8746

Table 3: 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: minus each baseline. Confidence intervals (CIs) use 10,000 paired entity-cluster resamples. Holm-adjusted candidate-pair McNemar tests form one family of twelve; these tests are diagnostic under repeated entities and queries. Supplementary statistics report scenario-cluster sensitivity and undefined denominators.

Observed pipeline event	Base	Comp.	Stress
Initial 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier recommendations	79	31	128
Final 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: recommendations	73	27	117
Supported recommendations withdrawn	6	4	11
Unsupported recommendations withdrawn	0	0	0
0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier decision errors	233	18	16
0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: decision errors	239	22	27

Table 4: Mechanical blind pipeline diagnostics. Correctness is measured against automated policy references. Rejected proof links and query values are logged events, not independently annotated semantic errors. Verifier/adjudicator counts exclude repeated attempts caused by connection failures.

Method	Base	Comp.	Stress
0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset:	0.898	0.622	0.910
w/o criticality	0.918	0.634	0.901
w/o scope	0.911	0.609	0.919
w/o time	0.905	0.493	0.893
w/o proof checks	0.914	0.569	0.928
w/o verifier	0.924	0.634	0.936

Table 5: Development ablations (decision macro-F1). Each “w/o” row removes the named component; “All gaps blocking” treats every information gap as blocking. The verifier ablation equals 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier. Compiler ablations share query extractions and link proposals.

Figure 3: Blind confidence-ranked risk–coverage curves. The ordinate is overall decision error on retained candidates, not false recommendation rate. Tied confidence values are grouped.

Backbone	Direct	Flat RAG	0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset:	F1	Rec. P/R	FRR
GPT-5 mini	$\checkmark$	$\checkmark$	$\checkmark$	0.900	1.00/0.87	0.000
Claude Opus 4.5	$\checkmark$	$\checkmark$	$\checkmark$	0.944	1.00/0.98	0.000
Claude Sonnet 4.5	$\checkmark$	$\checkmark$	$\checkmark$	0.902	1.00/0.80	0.000

Table 6: Base-partition 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: quality for backbones with complete results. GPT-5-mini is the primary evaluation; Claude results are exploratory.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Partition	Queries	Scored	Rec.
Val.	Base	2,444	12,220	67
	Compositional	133	665	26
	Stress	691	691	86
Blind	Base	1,228	6,140	84
	Compositional	92	460	31
	Stress	606	606	133

Appendix

Table 7: Evaluation support. Rec. counts automated-policy recommendation candidates, not model successes. Compositional includes multi-requirement requests and current/future contrasts.

Method	Calls	In/Out (M tokens)	p95 (s)	Invalid	Retries
Rule-only	0	0.00/0.00	–	0	0
Direct	1941	49.03/5.38	54.9	0	15
Flat RAG	1939	32.35/5.38	55.8	1	13
Critic-and-guard	2337	36.52/6.89	114.2	0	15
All-field compiler	1972	71.26/19.85	351.6	27	46
0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier	3913	40.47/14.48	109.0	29	61

Appendix

Table 8: GPT-5-mini inference resource use. Latency is per observed request, not end-to-end user latency. All language-model methods use Flex; separate execution windows limit latency comparisons. 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier and 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: share extraction and linking. Invalid denotes responses that failed validation.

Method	Base	Comp.	Stress
Rule-only	0.952	0.667	0.978
Direct	0.801	0.434	0.861
Flat RAG	0.815	0.416	0.896
Critic-and-guard	0.801	0.581	0.647
All-field compiler	0.569	0.321	0.580
0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier	0.924	0.634	0.936

Appendix

Table 9: Development decision macro-F1 on the fixed support-enriched subset (100 base, 50 compositional and 100 stress queries). Results use the standard serving tier and seed 20260908 for method selection, separately from held-out testing.

Rule-only variant	Base F1	Comp. F1	Stress F1	Comp. FRR	Stress FRR
Full	0.952	0.667	0.978	0.000	0.000
w/o criticality	0.952	0.667	0.978	0.000	0.000
w/o scope	1.000	0.667	0.978	0.000	0.000
w/o time	0.952	0.515	0.938	0.571	0.143
w/o proof checks	0.952	0.667	0.978	0.000	0.000
w/o verifier	0.952	0.667	0.978	0.000	0.000

Appendix

Table 10: Strongest-baseline development ablations, all deterministic and API-free. The scope and time ablations disable checks in both obligation preparation and compilation. The verifier ablation is an identity because Rule-only has no verifier. The proof-check ablation tests validation of Rule-only’s deterministic proposals. Rule-only extracts fewer noncritical query fields than the LLM extractor, limiting comparisons of criticality and all-gaps-blocking effects across systems.

Split	Group	Queries	Candidates	R/V/E	S/C/T/M
Base	Child/family	742	3710	84/3586/40	110/0/0/3600
	Disability	204	1020	0/737/283	0/0/0/1020
	Elderly	282	1410	0/1070/340	0/0/0/1410
Comp.	Child/family	53	265	31/224/10	40/0/0/225
	Disability	30	150	0/122/28	0/0/0/150
	Elderly	9	45	0/45/0	0/0/0/45

Appendix

Table 11: Blind benchmark support by domain and language. R/V/E denotes automated-policy RECOMMEND/VERIFY/EXCLUDE; S/C/T/M denotes SUPPORTED/CONFLICTING/STALE/MISSING. Domain and language are separate overlapping views and must not be summed together. Construction totals and the full validation split are reported in the main benchmark table.

Method	Split	F1 $\mu\pm\sigma$	Worst F1	Rec. R $\mu\pm\sigma$	Worst R
Rule-only	Base	0.952 $\pm$ 0.000	0.952	1.000 $\pm$ 0.000	1.000
	Comp.	0.667 $\pm$ 0.000	0.667	1.000 $\pm$ 0.000	1.000
	Stress	0.978 $\pm$ 0.000	0.978	1.000 $\pm$ 0.000	1.000
0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset:	Base	0.899 $\pm$ 0.024	0.879	0.833 $\pm$ 0.115	0.688
	Comp.	0.645 $\pm$ 0.016	0.622	0.889 $\pm$ 0.079	0.778
	Stress	0.949 $\pm$ 0.013	0.934	0.833 $\pm$ 0.027	0.800

Appendix

Table 12: Development stability over three fixed seeds, 20260908/09/10. Worst denotes the minimum F1/recall or maximum FRR. 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: uses three independent Flex runs. Rule-only is deterministic: identical repeats measure reproducibility and carry no independent stochastic interpretation. Standard-tier method-selection results are reported separately.

0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: criterion	Declared floor	Observed	Met
Base FRR	$\leq.05$	0.000	Yes
Stress FRR	$\leq.05$	0.000	Yes
Base Rec. R	$\geq.72$	0.869	Yes
Stress Rec. R	$\geq.80$	0.880	Yes
Base V-to-E	$\leq.03$	0.000	Yes
Stress V-to-E	$\leq.05$	0.000	Yes

Appendix

Table 13: Predeclared 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: effectiveness floors on blind data. V-to-E is the fraction of policy-reference VERIFY candidates predicted EXCLUDE. Undefined denominators do not pass. Floor failures are reported as negative results and never trigger test-set tuning. Schema validity and candidate-local citation traceability are checked separately.

Figure 4: Paired entity-cluster 95% intervals for 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: minus each baseline in blind decision macro-F1; 10,000 resamples. Zero marks equal performance.

Split	Method	$N$	Rec. support	Present F1	Elig. F1	Evid. F1	Cite	AURC
Base	Rule-only	6140	84	0.915	0.837	0.497	1.000	0.017
	Direct	6140	84	0.618	0.714	0.209	0.990	0.320
	Flat RAG	6140	84	0.620	0.714	0.212	0.984	0.316
	Critic-and-guard	6140	84	0.825	0.655	0.245	1.000	0.181
	All-field compiler	6140	84	0.570	0.582	0.364	1.000	0.156
	0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier	6140	84	0.914	0.871	0.458	1.000	0.020

Appendix

Table 14: Additional blind metrics. Cite is source traceability for each candidate, which does not establish semantic correctness. Present F1 excludes unsupported reference classes. AURC measures confidence-ordered overall decision error with tied scores grouped. Complete breakdowns by domain, language, service family, and stress type accompany the supplementary data.

Backbone	Method	Decision F1	Rec. P/R	FRR
GPT-5 mini	Direct	0.618	0.09/1.00	0.914
	Flat RAG	0.620	0.09/1.00	0.909
	0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier	0.914	1.00/0.94	0.000
	0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset:	0.900	1.00/0.87	0.000
Claude Opus 4.5	Direct	0.647	0.45/0.15	0.552
	Flat RAG	0.682	0.40/0.32	0.597

Appendix

Table 15: Base: complete-partition cross-backbone results. Rule-only is shared and deterministic. References are automated policy judgments.

Backbone	Method	Decision F1	Rec. P/R	FRR
GPT-5 mini	Direct	0.772	0.35/1.00	0.652
	Flat RAG	0.763	0.34/1.00	0.656
	0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier	0.889	1.00/1.00	0.000
	0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset:	0.864	1.00/0.87	0.000
GPT-4o mini	Flat RAG	0.441	0.21/0.87	0.792
Claude Opus 4.5	Direct	0.674	0.67/0.26	0.333

Appendix

Table 16: Compositional: complete-partition cross-backbone results. Rule-only is shared and deterministic. References are automated policy judgments.

Backbone	Method	Decision F1	Rec. P/R	FRR
GPT-5 mini	Direct	0.866	0.83/0.93	0.168
	Flat RAG	0.850	0.88/0.88	0.120
	0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier	0.918	1.00/0.96	0.000
	0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset:	0.900	1.00/0.88	0.000
GPT-4o mini	Direct	0.440	0.43/0.89	0.574
GPT-4o mini	Flat RAG	0.469	0.44/0.97	0.563

Appendix

Table 17: Controlled stress: complete-partition cross-backbone results. Rule-only is shared and deterministic. References are automated policy judgments.

Comparison	$\Delta$ F1	Entity 95% CI	Holm $p$
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Direct	0.2967	[0.2509, 0.3486]	2.3e-39
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Flat RAG	0.2623	[0.2094, 0.3227]	2.54e-49
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier	0.0000	[0.0000, 0.0000]	1
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. GPT-5 mini 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset:	0.0437	[0.0271, 0.0621]	1.87e-18
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Rule-only	0.0289	[0.0191, 0.0390]	1.22e-21
Claude Sonnet 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Direct	0.2595	[0.2138, 0.3098]	1.17e-35

Appendix

Table 18: Base: paired exploratory cross-backbone comparisons. Intervals use 10,000 entity-cluster bootstrap replicates; McNemar

p

values are diagnostic and Holm-corrected within the 57-comparison family. Only the 42 completed comparisons are reported; correction retains the full family of 57 prespecified comparisons. Scenario-cluster sensitivity results are archived.

Comparison	$\Delta$ F1	Entity 95% CI	Holm $p$
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Direct	0.3000	[0.2167, 0.3914]	2.54e-10
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Flat RAG	0.1576	[0.0975, 0.2296]	1.94e-06
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier	0.0000	[0.0000, 0.0000]	1
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. GPT-5 mini 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset:	0.1100	[0.0571, 0.1910]	0.000565
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Rule-only	0.1585	[0.1073, 0.2407]	3.81e-05
Claude Sonnet 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Direct	0.2215	[0.1220, 0.3116]	3.06e-05

Appendix

Table 19: Compositional: paired exploratory cross-backbone comparisons. Intervals use 10,000 entity-cluster bootstrap replicates; McNemar

p

Comparison	$\Delta$ F1	Entity 95% CI	Holm $p$
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Direct	0.2630	[0.2070, 0.3240]	2.5e-22
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Flat RAG	0.2566	[0.2059, 0.3131]	4.67e-22
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier	0.0000	[0.0000, 0.0000]	1
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. GPT-5 mini 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset:	0.0304	[-0.0177, 0.0869]	1
Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Rule-only	0.0032	[-0.0372, 0.0480]	0.0718
Claude Sonnet 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Direct	0.2921	[0.2217, 0.3707]	2.14e-23

Appendix

Table 20: Controlled stress: paired exploratory cross-backbone comparisons. Intervals use 10,000 entity-cluster bootstrap replicates; McNemar

p

Backbone	Partition	F1 mean $\pm$ SD	Recall mean $\pm$ SD	Min. recall
GPT-5 mini	Base	0.899 $\pm$ 0.024	0.833 $\pm$ 0.115	0.688
	Comp.	0.645 $\pm$ 0.016	0.889 $\pm$ 0.079	0.778
	Stress	0.949 $\pm$ 0.013	0.833 $\pm$ 0.027	0.800
Claude Opus 4.5	Base	0.952 $\pm$ 0.000	1.000 $\pm$ 0.000	1.000
	Comp.	0.667 $\pm$ 0.000	1.000 $\pm$ 0.000	1.000
	Stress	0.972 $\pm$ 0.004	0.889 $\pm$ 0.016	0.867

Appendix

Table 21: 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: development stability on the same 250-scenario subset across three repetitions. SD is population SD. Results include all three repetitions for each reported backbone, including runs with low recall.

Explore similar work

EviReform: Evidence-Guided Query Reformulation for Multi-Hop Graph Retrieval
Aug 13, 2026Xinlong Xu, Yoshua Y. LiMulti-Hop QA Graph-Based Retrieval
Nanjing University of Information Science and Technology · Meituan
GraphDecisionBench: Benchmarking System One Models on Graph Tasks
Oct 5, 2026Xianliang Yang, Yapu Zhang, Li ZhaoLLM Evaluation LLM Reasoning with Graphs
Microsoft · Beijing University of Technology
Keep Your Receipts: Trustworthy Cleanup of Agent-Built Knowledge Graphs
Oct 13, 2026Vishisht ChoudharyKnowledge Graphs Data Provenance
Technical University of Munich

Aug 13, 2026J/K move · Enter open · S save

EviReform: Evidence-Guided Query Reformulation for Multi-Hop Graph Retrieval

Xinlong Xu, Yoshua Y. Li

Nanjing University of Information Science and Technology · Meituan

Multi-Hop QA Graph-Based Retrieval Query Reformulation Evidence Retrieval

Open paper AskPDF arXiv

Multi-hop retrieval must recover passages that provide sufficient evidence together. An initial passage often resolves an entity or relation implicit in the question, making the missing evidence easier to describe only after retrieval begins. Graph retrieval improves access to related evidence through stored corpus structure, but its retrieval signal is commonly derived from the original question. Complementary evidence must then be reached through stored relations even when an observed passage provides a more direct semantic cue. We introduce EviReform, which separates revising the retrieval request from aggregating evidence in the graph. Retrieved source passages formulate residual queries for the unresolved information need. The original and residual retrieval signals are normalized separately, combined, and propagated between propositions that share entities. On 2WikiMultiHopQA, HotpotQA, and MuSiQue, EviReform exceeds the strongest baseline by up to 5.59 Recall@5 points and 4.50 F1 points. These results show that observed evidence can guide graph retrieval toward the part of a supporting chain left underspecified by the original question. Code is available at https://github.com/XrazyMee/EviReform.

Abstract

Explore similar work

EviReform: Evidence-Guided Query Reformulation for Multi-Hop Graph Retrieval

GraphDecisionBench: Benchmarking System One Models on Graph Tasks

Keep Your Receipts: Trustworthy Cleanup of Agent-Built Knowledge Graphs