Follow the Policy, Fail the Benchmark: Auditing and Revising Policy-Based Decision Benchmarks
Organizations: Amazon Web Services · Iowa State University
Abstract
Policy-based decision benchmarks evaluate whether systems apply written rules to financial and operational cases. Yet a rule-following response can be marked wrong when the benchmark misstates the required decision. We audit nine public policy-based decision benchmarks through source tracing, program execution, and controlled checks. We find unsupported targets, reference errors that survive replay or propagate as solved steps, and evaluators that reject acceptable answers or credit incomplete decisions. To measure the scoring impact, we rescore unchanged model responses. On the 96 LegalBench SARA numeric cases, correcting the scorer raises acceptance by more than 80 percentage points for each of three models, without changing references or responses. In RuleArena Tax, restoring a required zero floor changes 27 of 300 references and narrows an observed model gap. A controlled federal-exemption task with explicit ownership and eligibility assumptions also shows that errors in supplied intermediate answers can affect final decisions. We present BAR (Benchmark Audit and Revision), a versioned collection of audit evidence and revisions for reevaluation.
Figures & tables
| RuleArena Tax (accepted / 300) | LegalBench SARA numeric (accepted / 96) | ||||||||||
| Model | Original | Revised | Gain | Loss | Floor | Zero only | Original | Parse | Parse + zero | Gain | Loss |
| Haiku 5.5 | 209 | 224 | 16 | 1 | 16 | 0 | 8 | 87 | 91 | 83 | 0 |
| GPT6 Luna | 241 | 261 | 20 | 0 | 10 | 10 | 9 | 83 | 87 | 78 | 0 |
| Gemma 4 31B | 195 | 220 | 25 | 0 | 22 | 3 | 8 | 84 | 88 | 80 | 0 |
| Scored questions | Answer EM (%) | |||
|---|---|---|---|---|
| Model | Original | Revised | Original | Revised |
| Haiku 5.5 | 285 | 285 | 58.63 | 58.63 |
| GPT6 Luna | 261 | 285 | 52.34 | 47.93 |
| Gemma 4 31B | 232 | 285 | 58.02 | 47.23 |
| Accepted / 69 | Changes | ||||
|---|---|---|---|---|---|
| Model | Released | Revised | None | Gain | Loss |
| Haiku 5.5 | 50 (3) | 68 (18) | 69 (19) | 18 | 0 |
| GPT6 Luna | 66 (16) | 68 (18) | 69 (19) | 3 | 1 |
| Gemma 4 31B | 65 (16) | 64 (15) | 69 (19) | 2 | 3 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | What the model receives and produces | How the target is scored |
|---|---|---|
| RuleArena Tax / Airline | Supplied tax instructions or baggage rules and a case; output a signed tax amount or the total travel cost, including ticket and baggage fees. | Extracted numerical answer compared with the reference using numerical closeness. |
| RuleArena NBA | League rules and a transaction scenario; decide whether a violation occurs and identify its operation and team. | Decision and location matching against the stored operation and team. |
| OpenExempt Exemption Valuation (EV) | Supplied exemption statutes and a debtor case; identify applicable clauses and the maximum protectable value for each asset–clause pair independently. | Asset-level F1 and numerical error; the paper describes a 5% amount tolerance and fuzzy asset-name alignment. |
| SARA in DeonticBench | Edited tax statutes and case facts; output a tax amount or the truth of a statutory proposition. A Prolog reference program accompanies each record. | Numerical or boolean answer agreement under the authors’ documented output-scoring rule, which tolerates rounding for amounts and is also used after model inference. |
| USCIS in DeonticBench | Policy text and facts from immigration appeals; output a disposition such as Accepted or Dismissed. | Disposition agreement; reference program execution is an additional artifact to inspect. |
| ShARC | A rule passage, user question, scenario, and dialogue history; output Yes, No, Irrelevant, or a follow-up question. | Four-way decision accuracy (follow-up is the More class) and BLEU for generated follow-up questions. |
| Family | Actual evaluation use | What the use establishes |
|---|---|---|
| RuleArena | Policy-as-Logic evaluates all three domains [ 11 ] . | Task use; our Tax study separately holds its own input and scoring interfaces constant. |
| ConditionalQA / ShARC | Code Prompting reports ConditionalQA results; its experiment repository includes ShARC runs [ 15 ] . | Named task use, with different answer and clarification contracts. |
| LegalBench | Chain of Logic evaluates diversity, personal jurisdiction, and J.Crew tasks [ 17 ] . | Task-specific adoption, not verified use of the SARA scorer audited here. |
| PolicyGuardBench | UniGuard reports external benchmark evaluation [ 14 ] . | Independent use; the evaluated original/strict revision is unresolved. |
| WorkArena++ | TeleTune evaluates a selected L2 cohort [ 3 ] . | Adapted native tasks; exact audited-class membership and revision unresolved. |
| TheAgentCompany | Is Bash All You Need? evaluates workplace agents [ 10 ] . | Independent adapted use; exact selected tasks and evaluator versions unresolved. |
| Component | Inspected population | Check and coverage boundary |
|---|---|---|
| RuleArena Tax | All 300 cases | Reference replay; three-model official-interface rescoring and trace review of all 62 changed judgments; source-dependency checks. |
| RuleArena NBA | 216 annotated cases; selected localization witness | Policy, substring-grader and rule-attribution vocabulary checks; no complete decision-solver replay. |
| RuleArena Airline | 300-case release; allocation convention | Supplied policy, task and reference-code inspection; prior-reported objective ambiguity, not a count of wrong labels. |
| OpenExempt EV1 | 100 test records each in Basic, Intermediate, Advanced | Replay and conditional amount/allocator checks; selected semantic witnesses, not complete semantic coverage. |
| OpenExempt decomposition | 100 test records each in non-exempt-asset stage 4 (NA4) and optimal-exemption stages 4 and 5 (OE4, OE5) | Intermediate-input inspection, source replay, and bounded consistency checks; same source family as EV1. |
| DeonticBench | 100 SARA numeric, 276 SARA binary, 242 USCIS records | Screening of all records; source and execution checks of selected allegations. Airline and Housing excluded. |
| Suite / represented status | Cases | Positive second / item-count pairs |
|---|---|---|
| Basic / single | 100 | 0/24 |
| Intermediate / single | 50 | 0/41 |
| Intermediate / married | 50 | 14/24 |
| Advanced / married | 100 | 84/124 |
| Prediction/reference configuration | Original | Revised |
| Gold echo, all 285 dev references: total EM | 1 | 1 |
| Omit dev-0; echo other 284: total EM | 1 | |
| Wrong but present dev-0: total EM | ||
| No predictions, all 285 references | Division error | EM 0 |
| 143 boolean-only references, gold echo | Division error | EM 1 |
| Release | Population | Rows | Conflicting groups | Involved rows | Minimum edits |
|---|---|---|---|---|---|
| Original | Train | 47,997 | 196 | 531 | 220 |
| Test | 12,000 | 22 | 49 | 22 | |
| Pooled | 59,997 | 279 | 780 | 324 | |
| Strict | Train | 48,452 | 219 | 637 | 263 |
| Test | 11,545 | 60 | 143 | 61 | |
| Pooled | 59,997 | 279 | 780 | 324 |
| Gains | Losses | |||
|---|---|---|---|---|
| Model | Explicit floor | Zero subtotal only | Explicit floor | Zero subtotal only |
| Haiku 5.5 | 16 | 0 | 0 | 1 |
| GPT6 Luna | 10 | 10 | 0 | 0 |
| Gemma 4 31B | 22 | 3 | 0 | 0 |
| Model | Denominator | EM | EM+cond. | F1 | F1+cond. |
|---|---|---|---|---|---|
| Haiku 5.5 | |||||
| GPT6 Luna | |||||
| Gemma 4 31B |
| Control (cases) | Haiku | Luna | Gemma |
|---|---|---|---|
| Unchanged tables (4) | 4/4/4 | 4/4/4 | 4/4/4 |
| Positive residual (20) | 4/19/20 | 17/19/20 | 16/16/20 |
| Zero residual (49) | 46/49/49 | 49/49/49 | 49/48/49 |
| Ceiling matches (19) | 5 | 3 | 0 |
Benchmark Auditing
Benchmark auditing checks whether evaluation datasets, metrics, and infrastructure measure the capabilities and risks they claim to assess.
See all 109 papers