LexFlip: A Dissociation Diagnostic for Legal Meaning Preservation Metrics

Gaurab Baral

cs.CLSep 4, 2026

LexFlip: A Dissociation Diagnostic for Legal Meaning Preservation Metrics

Authors: Gaurab Baral

Organizations: University of Cincinnati

Abstract

Does a simplified legal clause still say what the original said? The checks in current use cannot establish that it does: requiring an identical pair to score highest and an unrelated pair lowest moves lexical overlap and legal force together, so any monotone function of token overlap satisfies both. Our remedy is a dissociation, an item holding surface form fixed while legal force moves. We release LexFlip, 373 minimal perturbations of Quebec statutory French that reverse legal force while preserving 0.93 of the tokens, with a harness scoring metrics, regressors and prompted judges alike. The seven embedding and BERTScore metrics we test spend only 0.022 to 0.039 of their identical-to-unrelated range on such an edit, against 0.670 for bidirectional NLI, the one family the identical-pair check would disqualify. On FrJudge, against a measured human ceiling of r=0.597, a bare length feature outscores every semantic metric and has the lowest margin we measure.

Figures & tables

Figure 1: One LexFlip item. Replacing doit with peut turns a duty into a discretion and leaves the sentence otherwise untouched, so a metric reading legal meaning must drop and one reading overlap cannot. The judge’s ten-point rating is rescaled to

[0,1]

, as in Table 1 . Over the whole set the edit costs BERTScore 0.022 of its own identical-to-unrelated range and the judge 0.370, against 0.670 for bidirectional NLI.

Metric	identical	LexFlip	unrelated	discrim.%	margin
NLI-bidirectional (min)	0.784	0.265	0.010	93.3	0.670
NLI-bwd (no omission)	0.784	0.432	0.042	77.2	0.473
NLI-fwd (no hallucination)	0.784	0.469	0.043	77.7	0.424
LLM-judge (deepseek-chat, k=5)	1.000	0.632	0.006	86.3	0.370
surface: token Jaccard	1.000	0.928	0.089	99.2	0.079
LaBSE	1.000	0.972	0.292	100.0	0.039

Table 1: LexFlip , min–max normalised per metric. discrim. is the share of items ranked correctly (identical

>

edited), margin the fraction of the identical-to-unrelated range spent on the legal edit.

Perturbation	$n$	mean margin
drop ne …pas	21	0.333
avant $\leftrightarrow$ après	9	0.272
amount $\times$ 10	21	0.256
doit $\rightarrow$ peut	82	0.241
peut $\rightarrow$ doit	111	0.167
tous les $\rightarrow$ certains	42	0.153

Table 2: Margin over the 13 scorers of Table 1 ; selected rules, same-direction pairs pooled. All 12 rules are in the supplementary material.

Unsupervised, French-capable
Metric	$r$	$\rho$	RMSE ${}_{\text{raw}}$	RMSE ${}_{\text{cal}}$	over%
BERTScore-CamemBERTv2	0.291 ±0.09	0.326	2.62	2.36	49.3
BERTScore-FlauBERT	0.482 ±0.08	0.459	2.16	2.12	49.4
LaBSE	0.409 ±0.06	0.402	2.33	2.21	47.2
mE5-base	0.358 ±0.08	0.356	2.52	2.26	48.0
NLI-bidirectional (min)	0.395 ±0.07	0.405	3.74	2.22	46.6

Table 3: Calibrated comparison on the FrJudge test split (

89

items), mean over 10 seeds; all ten unsupervised metrics are in the supplementary material. RMSE

{}_{\text{raw}}

is the decimal-scaling protocol in current use, RMSE

{}_{\text{cal}}

the isotonic map of R2.

Figure 2: The two orderings come apart. Horizontally, calibrated correlation with the human label, the quantity a conventional comparison selects on; vertically, the share of working range spent on an edit that reverses the law.

Explore similar work

Where Experts Disagree, Models Fail: Detecting Implicit Legal Citations in French Court Decisions
Mar 24, 2026Avrile Floro, Tamara Dhorasoo, Soline Pellez +1Annotator Disagreement Legal Citation Verification
UPHF · Télécom Paris, Institut Polytechnique de Paris · Université Polytechnique Hauts-de-France
Which Changes Matter? Towards Trustworthy Legal AI via Relevance-Sensitive Evaluation and Solver-Grounded Reasoning
May 26, 2026Chen Linze, Cai Yufan, Hou Zhe +1Adversarial Robustness of Language Models Language Model Robustness
National University of Singapore · Griffith University
CrossLex: A Source-Grounded Benchmark for Cross-Jurisdictional Legal Reasoning in Large Language Models
Aug 2, 2026Xiaocui Yang, Xican Tan, Shoujie Chen +3LLM Evaluation LLM Grounding
School of Computer Science and Engineering, Northeastern University, Shenyang, China · Institute of Intelligent Computing, University of Electronic Science and Technology of China, Chengdu, China · School of Civil and Commercial Law, Southwest University of Political Science and Law, Chongqing, China +1

Mar 24, 2026J/K move · Enter open · S save

Where Experts Disagree, Models Fail: Detecting Implicit Legal Citations in French Court Decisions

Avrile Floro, Tamara Dhorasoo, Soline Pellez, Nils Holzenberger

UPHF · Télécom Paris, Institut Polytechnique de Paris · Université Polytechnique Hauts-de-France

Annotator Disagreement Legal Citation Verification Legal Document Analysis Classification

Open paper AskPDF arXiv

Applying computational methods to law at scale requires separating genuine legal reasoning from surface similarity. We study this through a concrete task: detecting implicit citations of the French Civil Code, where a court applies a statutory rule without naming it (a post-hoc question about the reasoning a court actually used). We release a benchmark of 1,015 passage-article pairs annotated by three legal experts. Our central finding is that their disagreement is itself informative: the third of cases the experts dispute are where models fail. Our best ensemble reaches an F1 score of 0.70 overall. Yet, two-thirds of its false positives fall on those disputed cases, a concentration that holds across all ten models we evaluate. Expert disagreement thus signals uncertainty in the task and in its gold label, which aggregate metrics hide. This should not block useful tools, however: reframed as top-

k

ranking with multi-model consensus, the same signals reach 76% precision for the top-200 candidates of the benchmark without supervision.

Abstract

Explore similar work

Where Experts Disagree, Models Fail: Detecting Implicit Legal Citations in French Court Decisions

Which Changes Matter? Towards Trustworthy Legal AI via Relevance-Sensitive Evaluation and Solver-Grounded Reasoning

CrossLex: A Source-Grounded Benchmark for Cross-Jurisdictional Legal Reasoning in Large Language Models