EviGraph: Proof-Carrying Selective Recommendation over Temporal Public-Service Knowledge Graphs
Organizations: Hong Kong Baptist University · Tokyo University of Science · University of Illinois Urbana-Champaign · The University of Tokyo
Abstract
Public-service recommendations require evidence that matches the requested service, scope, and date. Yet treating every missing detail as decisive can withhold useful recommendations. We introduce EviGraph, which distinguishes critical decision requirements from information that can remain unresolved. A language agent links these requirements to evidence in a temporal knowledge graph, while a deterministic checker establishes whether a recommendation is supported. Evaluation on a bilingual Hong Kong public-service benchmark with executable policy references shows that this distinction reduces unnecessary abstention. Additional verification, however, can withdraw supported recommendations without improving decision quality. These findings suggest that reliable evidence-based navigation depends on specifying what must be established for a decision, rather than simply adding more verification.
Figures & tables
| View | Group | Queries | Candidates | R/V/E | S/C/T/M |
|---|---|---|---|---|---|
| Partition | Base | 1,228 | 6,140 | 84/5393/663 | 110/0/0/6030 |
| Compositional | 92 | 460 | 31/391/38 | 40/0/0/420 | |
| Stress | 606 | 606 | 133/441/32 | 178/166/55/207 | |
| Domain | Child/family | 1,261 | 4,441 | 248/4141/52 | 328/122/55/3936 |
| Disability | 276 | 1,212 | 0/892/320 | 0/0/0/1212 | |
| Elderly | 389 | 1,553 | 0/1192/361 | 0/44/0/1509 |
| Base | Compositional | Stress | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | F1 | P/R | FRR | F1 | P/R | FRR | F1 | P/R | FRR |
| Rule-only | 0.915 | 1.00/1.00 | 0.000 | 0.816 | 1.00/1.00 | 0.000 | 0.927 | 1.00/1.00 | 0.000 |
| Direct | 0.618 | 0.09/1.00 | 0.914 | 0.772 | 0.35/1.00 | 0.652 | 0.866 | 0.83/0.93 | 0.168 |
| Flat RAG | 0.620 | 0.09/1.00 | 0.909 | 0.763 | 0.34/1.00 | 0.656 | 0.850 | 0.88/0.88 | 0.120 |
| Critic-and-guard | 0.825 | 0.98/0.67 | 0.018 | 0.886 | 1.00/0.81 | 0.000 | 0.708 | 0.99/0.65 | 0.011 |
| All-field compiler | 0.570 | –/0.00 | – | 0.623 | –/0.00 | – | 0.563 | –/0.00 | – |
| Split | Baseline | F1 | Pointwise 95% cluster CI | Holm |
|---|---|---|---|---|
| Base | Rule-only | -0.0148 | [-0.0329, 0.0020] | 0.7813 |
| Critic-and-guard | 0.0755 | [0.0282, 0.1293] | ||
| All-field compiler | 0.3303 | [0.3059, 0.3562] | ||
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier | -0.0133 | [-0.0263, -0.0039] | 0.1652 | |
| Comp. | Rule-only | 0.0485 | [-0.0028, 0.0931] | 0.7813 |
| Critic-and-guard | -0.0214 | [-0.1084, 0.0469] | 0.8746 |
| Observed pipeline event | Base | Comp. | Stress |
|---|---|---|---|
| Initial 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier recommendations | 79 | 31 | 128 |
| Final 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: recommendations | 73 | 27 | 117 |
| Supported recommendations withdrawn | 6 | 4 | 11 |
| Unsupported recommendations withdrawn | 0 | 0 | 0 |
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier decision errors | 233 | 18 | 16 |
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: decision errors | 239 | 22 | 27 |
| Method | Base | Comp. | Stress |
|---|---|---|---|
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: | 0.898 | 0.622 | 0.910 |
| w/o criticality | 0.918 | 0.634 | 0.901 |
| w/o scope | 0.911 | 0.609 | 0.919 |
| w/o time | 0.905 | 0.493 | 0.893 |
| w/o proof checks | 0.914 | 0.569 | 0.928 |
| w/o verifier | 0.924 | 0.634 | 0.936 |
| Backbone | Direct | Flat RAG | 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: | F1 | Rec. P/R | FRR |
| GPT-5 mini | 0.900 | 1.00/0.87 | 0.000 | |||
| Claude Opus 4.5 | 0.944 | 1.00/0.98 | 0.000 | |||
| Claude Sonnet 4.5 | 0.902 | 1.00/0.80 | 0.000 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Split | Partition | Queries | Scored | Rec. |
|---|---|---|---|---|
| Val. | Base | 2,444 | 12,220 | 67 |
| Compositional | 133 | 665 | 26 | |
| Stress | 691 | 691 | 86 | |
| Blind | Base | 1,228 | 6,140 | 84 |
| Compositional | 92 | 460 | 31 | |
| Stress | 606 | 606 | 133 |
| Method | Calls | In/Out (M tokens) | p95 (s) | Invalid | Retries |
|---|---|---|---|---|---|
| Rule-only | 0 | 0.00/0.00 | – | 0 | 0 |
| Direct | 1941 | 49.03/5.38 | 54.9 | 0 | 15 |
| Flat RAG | 1939 | 32.35/5.38 | 55.8 | 1 | 13 |
| Critic-and-guard | 2337 | 36.52/6.89 | 114.2 | 0 | 15 |
| All-field compiler | 1972 | 71.26/19.85 | 351.6 | 27 | 46 |
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier | 3913 | 40.47/14.48 | 109.0 | 29 | 61 |
| Method | Base | Comp. | Stress |
|---|---|---|---|
| Rule-only | 0.952 | 0.667 | 0.978 |
| Direct | 0.801 | 0.434 | 0.861 |
| Flat RAG | 0.815 | 0.416 | 0.896 |
| Critic-and-guard | 0.801 | 0.581 | 0.647 |
| All-field compiler | 0.569 | 0.321 | 0.580 |
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier | 0.924 | 0.634 | 0.936 |
| Rule-only variant | Base F1 | Comp. F1 | Stress F1 | Comp. FRR | Stress FRR |
|---|---|---|---|---|---|
| Full | 0.952 | 0.667 | 0.978 | 0.000 | 0.000 |
| w/o criticality | 0.952 | 0.667 | 0.978 | 0.000 | 0.000 |
| w/o scope | 1.000 | 0.667 | 0.978 | 0.000 | 0.000 |
| w/o time | 0.952 | 0.515 | 0.938 | 0.571 | 0.143 |
| w/o proof checks | 0.952 | 0.667 | 0.978 | 0.000 | 0.000 |
| w/o verifier | 0.952 | 0.667 | 0.978 | 0.000 | 0.000 |
| Split | Group | Queries | Candidates | R/V/E | S/C/T/M |
|---|---|---|---|---|---|
| Base | Child/family | 742 | 3710 | 84/3586/40 | 110/0/0/3600 |
| Disability | 204 | 1020 | 0/737/283 | 0/0/0/1020 | |
| Elderly | 282 | 1410 | 0/1070/340 | 0/0/0/1410 | |
| Comp. | Child/family | 53 | 265 | 31/224/10 | 40/0/0/225 |
| Disability | 30 | 150 | 0/122/28 | 0/0/0/150 | |
| Elderly | 9 | 45 | 0/45/0 | 0/0/0/45 |
| Method | Split | F1 | Worst F1 | Rec. R | Worst R | Worst FRR |
|---|---|---|---|---|---|---|
| Rule-only | Base | 0.952 0.000 | 0.952 | 1.000 0.000 | 1.000 | 0.000 |
| Comp. | 0.667 0.000 | 0.667 | 1.000 0.000 | 1.000 | 0.000 | |
| Stress | 0.978 0.000 | 0.978 | 1.000 0.000 | 1.000 | 0.000 | |
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: | Base | 0.899 0.024 | 0.879 | 0.833 0.115 | 0.688 | 0.000 |
| Comp. | 0.645 0.016 | 0.622 | 0.889 0.079 | 0.778 | 0.000 | |
| Stress | 0.949 0.013 | 0.934 | 0.833 0.027 | 0.800 | 0.000 |
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: criterion | Declared floor | Observed | Met |
| Base FRR | 0.000 | Yes | |
| Stress FRR | 0.000 | Yes | |
| Base Rec. R | 0.869 | Yes | |
| Stress Rec. R | 0.880 | Yes | |
| Base V-to-E | 0.000 | Yes | |
| Stress V-to-E | 0.000 | Yes |
| Split | Method | Rec. support | Present F1 | Elig. F1 | Evid. F1 | Cite | AURC | |
|---|---|---|---|---|---|---|---|---|
| Base | Rule-only | 6140 | 84 | 0.915 | 0.837 | 0.497 | 1.000 | 0.017 |
| Direct | 6140 | 84 | 0.618 | 0.714 | 0.209 | 0.990 | 0.320 | |
| Flat RAG | 6140 | 84 | 0.620 | 0.714 | 0.212 | 0.984 | 0.316 | |
| Critic-and-guard | 6140 | 84 | 0.825 | 0.655 | 0.245 | 1.000 | 0.181 | |
| All-field compiler | 6140 | 84 | 0.570 | 0.582 | 0.364 | 1.000 | 0.156 | |
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier | 6140 | 84 | 0.914 | 0.871 | 0.458 | 1.000 | 0.020 |
| Backbone | Method | Decision F1 | Rec. P/R | FRR |
|---|---|---|---|---|
| GPT-5 mini | Direct | 0.618 | 0.09/1.00 | 0.914 |
| Flat RAG | 0.620 | 0.09/1.00 | 0.909 | |
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier | 0.914 | 1.00/0.94 | 0.000 | |
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: | 0.900 | 1.00/0.87 | 0.000 | |
| Claude Opus 4.5 | Direct | 0.647 | 0.45/0.15 | 0.552 |
| Flat RAG | 0.682 | 0.40/0.32 | 0.597 |
| Backbone | Method | Decision F1 | Rec. P/R | FRR |
|---|---|---|---|---|
| GPT-5 mini | Direct | 0.772 | 0.35/1.00 | 0.652 |
| Flat RAG | 0.763 | 0.34/1.00 | 0.656 | |
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier | 0.889 | 1.00/1.00 | 0.000 | |
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: | 0.864 | 1.00/0.87 | 0.000 | |
| GPT-4o mini | Flat RAG | 0.441 | 0.21/0.87 | 0.792 |
| Claude Opus 4.5 | Direct | 0.674 | 0.67/0.26 | 0.333 |
| Backbone | Method | Decision F1 | Rec. P/R | FRR |
|---|---|---|---|---|
| GPT-5 mini | Direct | 0.866 | 0.83/0.93 | 0.168 |
| Flat RAG | 0.850 | 0.88/0.88 | 0.120 | |
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier | 0.918 | 1.00/0.96 | 0.000 | |
| 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: | 0.900 | 1.00/0.88 | 0.000 | |
| GPT-4o mini | Direct | 0.440 | 0.43/0.89 | 0.574 |
| Flat RAG | 0.469 | 0.44/0.97 | 0.563 |
| Comparison | F1 | Entity 95% CI | Holm |
|---|---|---|---|
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Direct | 0.2967 | [0.2509, 0.3486] | 2.3e-39 |
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Flat RAG | 0.2623 | [0.2094, 0.3227] | 2.54e-49 |
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier | 0.0000 | [0.0000, 0.0000] | 1 |
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. GPT-5 mini 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: | 0.0437 | [0.0271, 0.0621] | 1.87e-18 |
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Rule-only | 0.0289 | [0.0191, 0.0390] | 1.22e-21 |
| Claude Sonnet 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Direct | 0.2595 | [0.2138, 0.3098] | 1.17e-35 |
| Comparison | F1 | Entity 95% CI | Holm |
|---|---|---|---|
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Direct | 0.3000 | [0.2167, 0.3914] | 2.54e-10 |
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Flat RAG | 0.1576 | [0.0975, 0.2296] | 1.94e-06 |
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier | 0.0000 | [0.0000, 0.0000] | 1 |
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. GPT-5 mini 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: | 0.1100 | [0.0571, 0.1910] | 0.000565 |
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Rule-only | 0.1585 | [0.1073, 0.2407] | 3.81e-05 |
| Claude Sonnet 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Direct | 0.2215 | [0.1220, 0.3116] | 3.06e-05 |
| Comparison | F1 | Entity 95% CI | Holm |
|---|---|---|---|
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Direct | 0.2630 | [0.2070, 0.3240] | 2.5e-22 |
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Flat RAG | 0.2566 | [0.2059, 0.3131] | 4.67e-22 |
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: w/o verifier | 0.0000 | [0.0000, 0.0000] | 1 |
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. GPT-5 mini 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: | 0.0304 | [-0.0177, 0.0869] | 1 |
| Claude Opus 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Rule-only | 0.0032 | [-0.0372, 0.0480] | 0.0718 |
| Claude Sonnet 4.5 0.31765 0.50196 0.94902E0.27059 0.52549 0.89804v0.22745 0.5451 0.84706i0.18039 0.56863 0.79608G0.13725 0.59216 0.74118r0.0902 0.61569 0.6902a0.04706 0.63529 0.63922p0 0.65882 0.58824h __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: vs. Direct | 0.2921 | [0.2217, 0.3707] | 2.14e-23 |
| Backbone | Partition | F1 mean SD | Recall mean SD | Min. recall | Max. FRR |
|---|---|---|---|---|---|
| GPT-5 mini | Base | 0.899 0.024 | 0.833 0.115 | 0.688 | 0.000 |
| Comp. | 0.645 0.016 | 0.889 0.079 | 0.778 | 0.000 | |
| Stress | 0.949 0.013 | 0.833 0.027 | 0.800 | 0.000 | |
| Claude Opus 4.5 | Base | 0.952 0.000 | 1.000 0.000 | 1.000 | 0.000 |
| Comp. | 0.667 0.000 | 1.000 0.000 | 1.000 | 0.000 | |
| Stress | 0.972 0.004 | 0.889 0.016 | 0.867 | 0.000 |