An AI Assistant for Automating Anti-Corruption Expertise of Draft Legislation Based on the TRACE Framework
Organizations: Cherkasy State Business College, State Scientific Research Institute of Armament and Military Equipment Testing and Certification, Uzhhorod University · Finport Technologies Inc. JSC"
Abstract
We present an AI assistant for automating anti-corruption expertise of draft legislation based on the TRACE architecture, designed to screen bills for corruption risks: pinpoint provisions an expert would criticise and name the regulatory defect in each. The pipeline combines a deterministic parsing core, model-guided single-provision qualification, rule-based factor assignment, and external human oversight. To evaluate the assistant, we constructed an open benchmark from the published conclusions of Ukraine's National Agency on Corruption Prevention (NAZK), comprising 132 draft laws with 474 model-extracted findings. A system flag agrees with a finding when it targets a compatible provision and an LLM judge rates the risk as the same (strict) or partly the same (lenient). On 100 draft laws, the pipeline matches 0.28 of the findings within its top ten flags using the local Gemma 4 31B model, 0.30 with Sonnet 5.5, and 0.43-0.46 across all flags (non-blind; the frozen blind baseline achieves 0.21, 95% CI 0.16-0.26). Across all runs, most misses occur on provisions that the system located but qualified as a different risk, and none of the 32 legal-collision findings was matched. On 21 acts held out from tuning, the pipeline outperforms a monolithic prompt containing the full methodology under the strict measure, though not under the lenient one, and Opus 5.5 adds nothing over Sonnet. The reported scores reflect agreement with model-extracted findings rated by an LLM judge whose two access routes agree on 73 of 74 labels, without human expert validation.
Figures & tables
| Split | Acts | Findings |
|---|---|---|
| All pairs | 132 | 474 |
| dev | 8 | 24 |
| test | 12 | 27 |
| holdout | 9 | 29 |
| test100 | 100 | 364 |
| clean15 | 15 | 57 |
| Run | Findings | Judged | Label agreement | |
|---|---|---|---|---|
| Gemma, test | 27 | 16 | 15 | 0.91 |
| Gemma, holdout | 29 | 18 | 18 | 1.00 |
| Sonnet, test | 27 | 19 | 19 | 1.00 |
| Sonnet, holdout | 29 | 21 | 21 | 1.00 |
| Total | 112 | 74 | 73 |
| Question | Design | Positive answer if |
|---|---|---|
| Does pipeline structure gain over a monolithic prompt on the same model? | TRACE v1 vs monolithic prompt, Gemma 4 31B, blind test and holdout acts pooled (21) | the lower bound of the 95 % CI of the paired difference in “same + partial” |
| Does L3 prioritisation narrow the gap between the local and the cloud model? | test100, L1 v1.1 for both models (non-blind) | the gap with prioritisation is smaller than without it and not distinguishable from zero |
| What does structure give a strong model? | Sonnet 5.5, test (12), all flags | recall not lower than the monolithic prompt with at most a third of its flags |
| Where is the ceiling? | breakdown of missed findings by stage, holdout and test100 | the category “location found, risk different” is the largest in every run |
| System | Split | Acts | Findings | Same | Same + partial | Flags / act |
| TRACE v1 · Gemma 4 31B | test100 | 100 | 364 | 0.10 [0.07, 0.13] | 0.21 [0.16, 0.26] | 23.6 |
| TRACE v1 · Gemma 4 31B, all flags | test100 | 100 | 364 | 0.20 [0.15, 0.24] | 0.36 [0.30, 0.42] | 23.6 |
| L1 v1.1 · Gemma 4 31B † | test100 | 100 | 364 | 0.13 [0.09, 0.17] | 0.23 [0.19, 0.29] | 26.3 |
| L1 v1.1 + L3 prioritisation · Gemma 4 31B † | test100 | 100 | 364 | 0.16 [0.12, 0.20] | 0.28 [0.23, 0.33] | 26.3 |
| L1 v1.1 · Sonnet 5.5 † | test100 | 100 | 364 | 0.20 [0.15, 0.24] | 0.30 [0.25, 0.37] | 21.8 |
| L1 v1.1 · Gemma 4 31B, all flags † | test100 | 100 | 364 | 0.24 [0.19, 0.29] | 0.43 [0.37, 0.49] | 26.3 |
| System | Split | Acts | Findings | Same | Same + partial | Flags / act |
| TRACE v1 · Gemma 4 31B | test + holdout | 21 | 56 | 0.13 [0.05, 0.24] | 0.30 [0.18, 0.49] | 14.1 |
| monolithic prompt · Gemma 4 31B | test + holdout | 21 | 56 | 0.00 [0.00, 0.00] | 0.20 [0.10, 0.30] | 4.1 |
| TRACE v1 · Sonnet 5.5 | test + holdout | 21 | 56 | 0.23 [0.14, 0.36] | 0.43 [0.30, 0.61] | 11.1 |
| TRACE v1 · Gemma 4 31B | test | 12 | 27 | 0.19 [0.05, 0.44] | 0.41 [0.18, 0.78] | 9.6 |
| monolithic prompt · Gemma 4 31B | test | 12 | 27 | 0.00 [0.00, 0.00] | 0.26 [0.09, 0.39] | 4.0 |
| TRACE v1 · Sonnet 5.5 | test | 12 | 27 | 0.33 [0.21, 0.53] | 0.63 [0.46, 0.89] | 7.9 |
| Run | Findings | found | location found, risk different | cut by top-10 | L1: unit or address not parsed | candidate without flag | candidate cap | procedural factor F-III-16 |
|---|---|---|---|---|---|---|---|---|
| Gemma, holdout | 29 | 6 | 12 | 3 | 5 | 0 | 1 | 2 |
| Sonnet, holdout | 29 | 7 | 14 | 2 | 3 | 0 | 1 | 2 |
| Gemma, test100 (blind) | 364 | 75 | 110 | 76 | 94 | 8 | 0 | 1 |
| Gemma + L3, test100 † | 364 | 101 | 138 | 73 | 51 | 0 | 0 | 1 |
| Sonnet, test100 † | 364 | 111 | 112 | 81 | 59 | 0 | 0 | 1 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Risk type (and condition) | Factor (alternatives) |
|---|---|
| procedural gap in a power, addressee a private person | F-III-06 (F-III-01) |
| procedural gap in a power, otherwise | F-III-01 (F-III-06) |
| vague rights or duties, addressee a public authority | F-III-01 (F-III-06) |
| vague rights or duties, otherwise | F-III-06 (F-III-01, F-III-11) |
| evaluative words | F-III-11 (F-III-06, F-III-01) |
| collision of norms | F-III-09 (F-III-10) |
| Factor | In benchmark | test100: findings | test100: found | Stage A: findings | Stage A: TRACE | Stage A: monolithic |
|---|---|---|---|---|---|---|
| F-III-01 discretionary powers without criteria | 180 | 133 | 34 | 29 | 9 | 8 |
| F-III-06 vague rights and duties | 136 | 108 | 21 | 14 | 5 | 2 |
| F-III-09 legal collision | 43 | 32 | 0 | 5 | 0 | 0 |
| F-III-12 weakened anti-corruption measures | 23 | 19 | 1 | 5 | 2 | 1 |
| F-III-11 evaluative terms, open lists | 22 | 20 | 7 | 0 | — | — |
| F-III-04 referral provisions | 16 | 12 | 5 | 0 | — | — |
Legal Document Analysis
Legal document analysis uses language technologies to classify, extract, compare, and reason over statutes, contracts, policies, and court decisions.
See all 81 papers