Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge
Organizations: Sofia University · Indiana University · University of California, San Diego · Northeastern University · Michigan State University
Abstract
Financial reports repeat values across periods, metrics and accounting lines, allowing an LLM-generated calculation to be numerically correct while citing the wrong financial role. We evaluate what probabilistic evidence verification adds beyond number matching using Jev as a source-support verifier for GPT-4.1-mini calculation traces. A signed-number-at-pointer baseline explains most recovery over exact quotation checks. To isolate the remaining role-recognition problem, we hold operands and arithmetic fixed, move citations between same-number cells, and retain controls that express equivalent facts. These contrasts reveal both wrong-role citations that pass and valid alternative citations that are withheld. Explicit column labels improve selected wrong-role decisions while also lowering support for some equivalent evidence. A constructed follow-up on 36 new source pages, labeled by a non-author reviewer, extends this evaluation and exposes the same tradeoff between detecting role errors and retaining valid citations. The contribution is a controlled evaluation that identifies what a probabilistic financial verifier distinguishes when numerical matching is held fixed. For LLM-based financial assistants, it makes numerical correctness, cited-role support and acceptance outcomes separately assessable.
Figures & tables
| Trace draw | Output | Matches / mismatches / unavailable |
|---|---|---|
| 0 | Stated number | 79 / 16 / 1 |
| 0 | Executed number | 84 / 11 / 1 |
| 0 | Execution + lexical/unit acceptance | 58 / 3 / 35 |
| 1 | Stated number | 80 / 16 / 0 |
| 1 | Executed number | 86 / 10 / 0 |
| 1 | Execution + lexical/unit acceptance | 59 / 2 / 35 |
| Acceptance rule | Accepted / 192 | Supported | Unsupported | Unresolved | Joint passes |
|---|---|---|---|---|---|
| Literal/unit rule | 122 | 120 | 2 | 0 | 117 |
| Relaxed pointer rule (post hoc) | 154 | 151 | 2 | 1 | 141 |
| Jev source gate, 0.5 | 159 | 152 | 6 | 1 | 143 |
| Jev source gate, 0.8 | 146 | 145 | 1 | 0 | 142 |
| Jev source gate, 0.9 | 126 | 125 | 1 | 0 | 123 |
| Jev source gate, 0.95 | 87 | 87 | 0 | 0 | 87 |
| Release policy | Trace draw 0 | Trace draw 1 | Mean net match change (pp) | Consulted questions |
|---|---|---|---|---|
| Jev alone | 95 / 1 / 0 | 95 / 1 / 0 | +0.00 | 0 |
| Gate stated number | 93 / 3 / 0 | 94 / 2 / 0 | -1.56 | 12 |
| Gate executed number | 93 / 3 / 0 | 94 / 2 / 0 | -1.56 | 12 |
| Gate execution + lexical/unit acceptance | 88 / 1 / 7 | 87 / 1 / 8 | -7.81 | 12 |