Organizations: School of Artificial Intelligence and Data Science, University of International Business and Economics · Department of Computer Science, University of Chicago · School of Law, University of International Business and Economics · Gaoling School of Artificial Intelligence, Renmin University of China
Legal judgment prediction models may retain a guilty verdict after a legally decisive condition has been altered such that the specified charge is no longer established. We study this guilty bias via selective repair over matched charge-established and charge-not-established cases, where the unsupported verdict should change while its supported counterpart remains stable. We construct CAIL-MC, a legally reviewed dataset of 11,478 matched pairs covering 261 Chinese criminal charges. Held-out activation interventions show that contrasts learned from correctly resolved pairs transfer to false-guilty cases, but target-reaching interventions can also damage matched supported-guilty decisions. Motivated by this observation, we introduce PROSE, which reconstructs case-specific targets from a training-derived residual geometry, aligns them with a local margin-descent signal, attenuates supported-guilty-sensitive components, and realizes bounded targets through a preservation-regularized low-rank parameter update. Across Qwen3-8B and GLM4-9B, PROSE achieves stronger pairwise safe repair and evidence sensitivity, while preserving standard single-pass inference.
Figures & tables
Figure 1: (a) Guilty bias and (b) its correction trade-off.
Figure 2: Qwen3-8B diagnosis. (a) layer-wise NG-margin reduction under forward and control directions, (b) repair feasibility, and (c) paired-G margin damage.
Track
Source
Train
Dev
Test
Pairwise
CAIL2018
9,181
1,147
1,150
Stage distribution: O 35.7%, U 17.8%, C 46.5%
Evidence
LeCaRD
–
–
280
Criminal-S
–
–
800
Table 1: CAIL-MC statistics. Pairwise counts are matched pairs; evidence counts are variants, four per source case (70 LeCaRD; 200 Criminal-S). O/U/C denote Offense/Unlawfulness/Culpability.
CAIL2018 (Pairwise Repair–Preservation)
LeCaRD (Evidence-Sufficiency Generalization)
Criminal-S (Evidence-Sufficiency Generalization)
Case level
Pair level
Case level
Case level
Method
Acc ↑
F1 ↑
FGR ↓
FNR ↓
Rep. ↑
G-Ret. ↑
PSR ↑
Acc ↑
F1 ↑
FGR ↓
FNR ↓
Acc ↑
F1 ↑
FGR ↓
FNR ↓
Qwen3-8B-Base
Base
72.65 ± 1.43
70.58 ± 1.77
53.91 ± 2.82
0.78 ± 0.50
–
–
–
55.71 ± 2.99
46.91 ± 5.22
85.00 ± 6.59
3.57 ± 3.67
66.88 ± 2.58
63.63 ± 3.36
63.00 ± 5.05
3.25 ± 2.00
SFT
98.87 ± 0.44
98.87 ± 0.44
2.00 ± 0.83
0.26 ± 0.30
96.27 ± 1.53
100.00 ± 0.00
96.27 ± 1.53
49.64 ± 2.71
37.60 ± 3.45
94.29 ± 3.79
6.43 ± 5.26
51.00 ± 1.12
36.72 ± 1.83
96.50 ± 1.78
1.50 ± 1.58
Self-Refine
70.35 ± 1.49
68.29 ± 1.79
4.17 ± 1.13
55.13 ± 2.77
95.79 ± 1.63
39.71 ± 3.66
36.79 ± 3.65
54.29 ± 4.56
47.90 ± 6.33
10.71 ± 5.13
80.71 ± 7.98
54.88 ± 1.70
44.86 ± 2.97
2.50 ± 1.53
87.75 ± 3.37
Table 2: Results (%) on both CAIL-MC tracks. Entries are point estimates ± 1.96% bootstrap standard errors (1,000 resamples over pairs for repair–preservation and cases for evidence-sufficiency evaluation). Best and second-best results per backbone, excluding Base, are bold and underlined. Base’s pair-level scores are omitted: Rep. = PSR = 0 and G-Ret. = 100 by construction.
Figure 3: Mean verdict margin as evidence is removed from the 200 Criminal-S cases of Evidence Track, with (a) Qwen3-8B-Base and (b) GLM-4-9B-Base.
Figure 4: Repair and preservation. (a–b) Pair states before and after PROSE. (c) Supported-guilty damage at matched repair; lower is better.
Qwen3-8B-Base
GLM-4-9B-Base
Variant
F1 ↑
Rep. ↑
G-Ret. ↑
PSR ↑
F1 ↑
Rep. ↑
G-Ret. ↑
PSR ↑
PROSE (full)
91.48
85.25
90.76
76.34
87.17
67.37
94.02
62.16
w/o attenuation (Eq. 8 )
76.44
40.84
99.68
40.52
86.82
68.15
93.05
62.36
w/o constraints (Eq. 11 )
57.27
70.83
41.00
21.39
35.40
99.81
2.90
2.70
w/o both
61.16
62.72
50.57
25.12
35.78
99.81
3.67
3.47
Table 3: Preservation ablation on CAIL-MC.
Figure 5: Offline cost on one RTX 4080 SUPER and per-query latency relative to PROSE, both on log scales.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Total
Agree
Disagree
Discarded
Retained
Batch 1
3,141
2,779
362
267
2,874
Batch 2
3,141
2,758
383
274
2,867
Batch 3
3,141
2,791
350
261
2,880
Batch 4
3,141
2,744
397
284
2,857
Total
12,564
11,072
1,492
1,086
11,478
Appendix
Table 4: Human verification results across the four review batches. “Agree” denotes identical decisions independently made by the two junior annotators. Every “Disagree” case was directly referred to the senior criminal-law expert for final adjudication.
Source
Total
Agree
Disagree
Discarded
Retained
LeCaRD
80
65
15
10
70
Criminal-S
200
194
6
0
200
Total
280
259
21
10
270
Appendix
Table 5: Human verification results for the evidence-sufficiency gradient. “Agree” denotes identical retain/discard decisions independently made by the two criminal-law graduate reviewers. Every “Disagree” case is directly referred to the senior criminal-law expert for final adjudication.
Stage
Geometry
Without
With full projection
Offense
Stage-specific
70.6 (108)
17.6 (27)
Shared
70.6 (108)
20.3 (31)
Unlawfulness
Stage-specific
83.2 (154)
15.7 (29)
Shared
76.8 (142)
19.5 (36)
Culpability
Stage-specific
75.9 (217)
10.8 (31)
Shared
76.9 (220)
10.1 (29)
Appendix
Table 6: Repair-target feasibility (%, feasible pairs in parentheses) on Qwen3-8B-Base without and with full projection onto the complement of the supported-guilty gradient subspace ( ρ=0 in Eq. 8 ).
Figure 6: Cross-stage transfer at Layer 22 on Qwen3-8B-Base. Each cell adds the contrast direction of a source stage to cases of a target stage and reports the mean NG-margin reduction; the diagonal (bold) gives each direction’s effect on its own stage.
Figure 9: Repair feasibility and preservation on GLM-4-9B-Base. (a) Repair feasibility under stage-specific and shared contrast geometry for the three legal stages. (b) Paired-G margin damage on matched feasible cases, before and after preservation projection. Lower paired-G damage indicates better preservation of the supported charge-established counterpart.
Symbol
Description
Value / range
ℓ
Edited layer
20, 22
–
Edited modules
attn-out + MLP down
–
Contrast geometry
stage-specific or shared
τ
Target margin endpoint
[−0.02,−0.005]
λ
Contrast–gradient mixture
[0,1]
rq / rp
Contrast / preservation rank
128 / 64
Appendix
Table 7: Hyperparameters of PROSE across both backbones. Ranges give the development-split search space; single values are fixed. Configurations are selected per backbone and legal stage on the development split.
Figure 10: Mean verdict margin on the 1,150 charge-established CAIL-MC test facts rewritten at three evidence levels, with (a) Qwen3-8B-Base and (b) GLM-4-9B-Base. The vertical axis is linear within [−1,1] in (a) and [−0.2,0.2] in (b) (shaded) and logarithmic beyond; error bars are 95% confidence intervals.
Judged established (%)
Method
Zero ↓
Partial
Full ↑
Δs↑
Unedited
93.8
94.8
97.7
0.58
Uniform shift
0.2
34.8
69.4
0.58
SFT
100.0
98.3
99.0
− 0.25
PROSE
10.3
76.0
86.1
0.74
Appendix
Table 8: Share of the rewritten CAIL-MC facts judged as establishing the charge with Qwen3-8B-Base, and the mean margin change from Zero to Full ( Δs ). The uniform shift is the matched-repair control of Sec. 6.3 .
Level
Unedited
SFT
PROSE
Qwen
Zero
0.304 [0.293, 0.315]
4.835 [4.755, 4.921]
− 0.266 [ − 0.281, − 0.251]
Partial
0.568 [0.546, 0.590]
4.130 [4.003, 4.252]
0.314 [0.290, 0.339]
Full
0.888 [0.863, 0.911]
4.589 [4.468, 4.703]
0.478 [0.451, 0.504]
GLM
Zero
0.059 [0.055, 0.063]
4.531 [4.389, 4.674]
− 0.000 [ − 0.004, 0.004]
Partial
0.179 [0.168, 0.191]
6.251 [5.944, 6.564]
0.126 [0.117, 0.135]
Full
0.272 [0.260, 0.284]
6.878 [6.557, 7.205]
0.172 [0.163, 0.181]
Appendix
Table 9: Mean verdict margin [95% bootstrap CI] of the rewritten CAIL-MC facts at each evidence level.
Acc
F1
FGR
FNR
Rep.
G-Ret.
Off.
Unl.
Cul.
All
Qwen3-8B-Base
O
72.30
70.11
54.78
0.61
7.46
100.00
26.02
1.10
3.83
7.46
U
87.96
87.87
20.35
3.74
63.53
96.76
31.71
74.03
63.58
60.29
C
85.91
85.69
26.43
1.74
51.70
99.19
31.71
17.13
77.96
50.89
OU
86.26
86.11
24.00
3.48
59.00
97.08
48.78
67.96
52.40
56.24
OC
82.39
81.94
33.48
1.74
40.52
99.03
44.72
16.57
50.80
39.55
Appendix
Table 10: Stage ablation on CAIL-MC (%). O / U / C denote the Offense, Unlawfulness, and Culpability updates; OUC is PROSE. The right block gives PSR on the strict false-guilty test pairs of each stage and on all of them.
Figure 11: Effect of editing on the ten most frequent charges in the CAIL-MC test split. Beneficial : test predictions that editing flips from incorrect to correct; harmful : predictions it flips from correct to incorrect.
Figure 12: Change in verdict margin of the charge-not-established member ( ΔsNG ) and of its charge-established counterpart ( ΔsG ) for each strict false-guilty CAIL-MC test pair under PROSE, on (a) Qwen3-8B-Base and (b) GLM-4-9B-Base. The diamond marks the medians; a uniform output shift would place every pair at a single point on the dashed diagonal.
Figure 13: Verdict margins of the charge-established member ( sG , horizontal) and the charge-not-established member ( sNG , vertical) for every CAIL-MC test pair under the base model and PROSE, on (a) Qwen3-8B-Base and (b) GLM-4-9B-Base. The dashed lines are the decision boundaries, so the four quadrants are the four joint pair states; the marginal panels show the distribution of each margin.