Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System
Authors: Rubén Manrique, Michelle Castellanos, Jorge Morales, Juan David Gutiérrez, Antonio Barreto Rozo, Joaquín Vélez Navarro
Organizations: Department of Systems and Computing Engineering, Universidad de los Andes · School of Government, Universidad de los Andes · Faculty of Law, Universidad de los Andes
Large language models (LLMs) are increasingly used to support legal practice, education, and research, yet their reliability in national legal systems outside the United States remains largely undocumented. We introduce an expert-validated benchmark for evaluating LLM reliability on the Colombian legal system. The benchmark comprises 1,042 items spanning ten areas of law and three question formats (closed multiple-choice, semi-open, and open-ended IRAC), built through a human-in-the-loop pipeline with multi-stage expert review. We evaluate 15 contemporary proprietary and open-weight models with format-appropriate metrics. Accuracy on closed questions ranges widely, from 0.905 (Gemini 3.1 Pro) to 0.577, but on free-text legal answers factual correctness never exceeds 0.45 (on a 0-1 scale) for any model. We find a dissociation between answer relevancy and correctness (Spearman rho = -0.46): models reliably sound responsive while frequently being wrong, a pattern of particular concern for non-expert users. Closed-question accuracy and free-text correctness are strongly rank-correlated (rho = 0.94), so cheap multiple-choice screening predicts model ranking but overstates absolute reliability. An independent rubric-based LLM judge and blind human expert scoring both reproduce the free-text ranking (rho >= 0.88). The judge further reveals that only about half of the norms models cite are correct; the rest are wrong or non-existent. Reliability varies systematically by legal area and follows an inverted-U across question complexity. Our results indicate that current LLMs require expert supervision for Colombian legal tasks, and that grounding answers in authoritative sources is a promising path to higher reliability. We release the benchmark construction pipeline to support reproducible evaluation.
Sense of the ruling; literal reproduction of a norm; jurisprudential precedent; temporal validity; conceptual distinction; legal requirements; legal exceptions; impartiality of the judicial order.
High
Normative conflict; factual background; procedural stance; legal problem; central legal basis; balancing of principles and rights; systematic interpretation.
Table 1: Sub-tasks targeted by semi-open questions at each complexity level, adapted from Dahl et al. [4] to the Colombian legal order.
Figure 1: Composition of the evaluated benchmark by legal area and question type. Bars are ordered by total number of items; the value at the end of each bar is the area total.
Legal area
Closed
Semi-open
Open
Total
Constitutional
39
86
9
134
Administrative
53
63
8
124
Criminal
26
93
4
123
Procedural
37
71
3
111
Commercial & corporate
26
70
8
104
Civil
27
70
5
102
Table 2: Evaluated benchmark composition by legal area and question type.
Figure 2: Distribution of semi-open questions across the three complexity levels.
Workshop
Date
Mode
Questions
Participants
W1
10 Nov 2025
In-person
238
16
W2
14 Nov 2025
In-person
64
19
W3
10 Dec 2025
Virtual
26
2
W4
15 Dec 2025
Virtual
7
2
W5
05 Mar 2026
In-person
80
10
Total
415
49
Table 3: Summary of the question-generation workshops.
Figure 3: Questions contributed per workshop, colored by modality.
Figure 4: Qualification-and-approval protocol. Every question—whether generated in a workshop, by the legal team, or by an individual student—is routed through the same four-step protocol (reception, assignment to the legal assistant, expert evaluation, and committee decision) before entering the final bank; at each decision point an item may be edited, sent back for adjustment, or archived.
Provider
Model
Type
Amazon
Nova Lite 1.0
General
Anthropic
Claude Opus 4.5
General
Anthropic
Claude Sonnet 4.5
General
Cohere
Command R (08-2024)
General
DeepSeek
DeepSeek V3.2
General
Google
Gemini 3.1 Pro Preview
General
Table 4: The 15 large language models evaluated on the benchmark.
Metric group
Metrics
Subset ( n )
Accuracy
accuracy ↑
Closed (305)
Answer quality
BERTScore, BLEU, ROUGE-1/L,
Free-text (737)
RAGAS relevancy/correctness ↑
Readability
Fernández-Huerta, Szigriszt-Pazos,
Free-text (737)
(Spanish)
Gutiérrez, μ , Crawford (—)
Table 5: Evaluation metrics by question format and the subset each is computed over. Arrows indicate whether higher ( ↑ ) or lower ( ↓ ) values are preferable; readability indices are descriptive (—).
Closed
Free-text (open + semi-open)
Model
Accuracy
BERTScore
RAGAS rel.
RAGAS corr.
Gemini 3.1 Pro
0.905
0.712
0.758
0.447
Claude Opus 4.5
0.869
0.714
0.742
0.428
GPT-5.4
0.859
0.717
0.792
0.451
GLM 5.2
0.856
0.708
0.745
0.389
Claude Sonnet 4.5
0.803
0.712
0.763
0.408
Table 6: Headline results per model, sorted by closed-question accuracy. Free-text metrics (BERTScore F 1 and RAGAS relevancy/correctness) are computed over the 737 open and semi-open items; accuracy over the 305 closed items. Best value per column in bold .
Figure 5: Accuracy on closed multiple-choice questions. Bars are shaded by performance tier ( ≥0.80 , 0.70 – 0.80 , <0.70 ); the dashed line marks the random-guessing baseline.
Figure 6: Closed-question accuracy by model (rows, sorted by mean accuracy) and legal area (columns, sorted by mean accuracy). Darker cells indicate higher accuracy.
Figure 7: Inter-model agreement on closed questions, showing the number of questions for which a given number of models (0–15) answered correctly. A large consensus spike at 15/15 contrasts with a short tail of near-universally-failed items.
Correct
Area
Topic
Likely cause
0/15
Procedural
Consumer-protection competent authority
Meta-option (“both”)
0/15
Constitutional
Composition of Congress (upper/lower house)
Meta-option (“none”)
0/15
Administrative
Administrative silence (Resolution 368/2014)
External document
0/15
Administrative
Disqualifications in state contracts
Ambiguous
0/15
Criminal
Onset of the protected legal interest (life)
Contested doctrine
1/15
Commercial
Who may grant financial leasing
Nuanced/recent
Table 7: The ten hardest closed questions, by number of models answering correctly (out of 15). “Cause” classifies the likely reason for near-universal failure.
Figure 8: Best single model vs. a 15-model majority vote vs. the oracle (at least one model correct). Majority voting underperforms the best single model because model errors are correlated.
Figure 9: Hierarchical clustering of models by response agreement on closed questions (average linkage; distance =1− agreement). Clusters track capability tier rather than provider.
Figure 10: Pairwise McNemar tests on closed-question outcomes (models sorted by accuracy). Grey cells mark statistically indistinguishable pairs; the block-diagonal grey regions reveal performance tiers.
Figure 11: Closed-question accuracy by model access (open-weight vs. proprietary); reasoning-oriented models highlighted. Horizontal bars are group means.
Figure 12: Item response theory (2PL) item map. Each point is a question, positioned by estimated difficulty (x) and discrimination (y). Orange ticks mark the 15 models’ estimated abilities on the same logit scale.
Figure 13: Distribution of RAGAS answer (a) correctness and (b) relevancy over the 737 free-text items, per model (violins; models sorted by mean correctness). Diamonds mark the mean and horizontal bars the 95 % bootstrap confidence interval. Correctness is low and widely spread for every model (dashed line at 0.3 ), whereas relevancy is uniformly high and concentrated.
Figure 14: RAGAS answer relevancy versus correctness. (a) Across the 15 models (means): higher relevancy does not imply higher correctness ( ρ=−0.46 ). (b) Across all 11055 individual answers (hex-binned density): the two are effectively uncorrelated ( ρ=0.08 ). High relevancy does not imply correctness at either level.
Figure 15: Pairwise paired Wilcoxon signed-rank tests on per-item free-text correctness (Holm-corrected; models sorted by mean correctness). Grey cells mark statistically indistinguishable pairs; the block structure reveals performance tiers.
Corr.
Area
Topic
Likely cause
0.04
Constitutional
Reporting magistrate of ruling T-256/2025 ( tutela vs. Meta)
Exact attribution; post-cutoff
0.07
Constitutional
Deciding judge of ruling T-426/2003
Exact attribution
0.07
Constitutional
Reporting magistrate of ruling T-323/2024 (use of AI)
Exact attribution; recent
0.09
Procedural
Article on the requirements to be a conciliator (Law 2220/2022)
Exact citation; recent statute
0.09
Administrative
Core holding of ruling SU-167/2023 (direct reparation)
Recent case law; reasoning
0.10
Tax
Holding of Council of State ruling 26644/2024
Recent case law
Table 8: The ten hardest free-text items, by mean RAGAS answer correctness across the 15 models. “Likely cause” classifies the dominant reason for near-universal failure. All ten are semi-open items.
Figure 16: RAGAS answer correctness on semi-open items by complexity level. The four highest-scoring models are highlighted; remaining models are shown in grey.
Figure 17: Mean RAGAS correctness by semi-open sub-task (averaged over the 15 models), colored by complexity level; the item count per sub-task is shown at the end of each bar. Low-complexity sub-tasks that hinge on a single exact legal fact (deciding judge, normative existence) are among the worst, alongside the reasoning-heavy high-complexity sub-tasks.
Figure 18: Free-text RAGAS correctness by model (rows, sorted by mean) and legal area (columns, sorted by mean). Darker cells indicate higher correctness. The analogue of the closed-question accuracy heatmap (Figure 6 ).
Figure 19: Reliability of normative citations by model (LLM-judge classification on the 350 -item subset): share of cited norms that are correct, cite the wrong norm, or are non-existent. Dashed line marks the pooled correct rate ( 53.2 %). ∗ GPT-5.4 is also the judge model (Section 7.5 ).
Figure 20: Convergent validity: each model’s mean rubric-based LLM-judge score against its mean RAGAS answer correctness ( 15 models, 350 -item subset). Two methodologically independent instruments agree closely on the ranking. ∗ GPT-5.4 is also the judge model.
Figure 21: Human validation of the LLM judge: confusion matrix of judge versus expert-team scores on the 1 – 5 scale (row-normalized, 323 blind judgments on the 15 -model subset). The extremes are reliable; the mid-scale is shifted down by about one point, a monotone shift that preserves the model ranking.
Figure 22: Answer length and readability. (a) Mean answer length (words) vs. RAGAS correctness across models: longer answers score higher ( ρ=+0.78 ). (b) Fernández-Huerta ease by semi-open complexity level; values sit inside the calibrated 0 – 100 band. Higher ease = easier to read.
Figure 23: Readability of model answers versus the human expert references (Fernández-Huerta ease; higher = easier). (a) Each model’s mean ease (points) against the human-reference mean (solid) and median (dashed): most models write somewhat denser prose than the human references, but within the ordinary range. (b) Ease by semi-open complexity level: the human–model gap is largest at low complexity and closes at high complexity.
As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses has become essential. These tasks require context-sensitive answers and allow little room for error, motivating fine-grained and diagnostic evaluation that can identify specific sources of response quality failures. We introduce LexRubric, a rubric-based benchmark for evaluating open-ended Chinese legal tasks. LexRubric contains 649 instances from legal consultation and judicial examination, which reflect both everyday legal needs and professional legal reasoning and cover 14 legal scenarios. It further includes 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework, enabling accurate evaluation and diagnostic analysis across tasks and evaluation dimensions. To validate the reliability of the evaluation, we test multiple judge models and compare model-based judgments with human judgments. We further evaluate 18 recent general and legal-domain LLMs on LexRubric. Results show that different models exhibit distinct capability profiles, and that open-ended legal tasks remain challenging for current LLMs. Data is available at: https://github.com/foggpoy/LexRubric.
Yifan Chen, Haitao Li, Yiran Hu +6
1Beijing University of Posts and Telecommunications · 2Tsinghua University · University of Waterloo +1
We introduce LegalBench-BR, the first public benchmark for evaluating language models on Brazilian legal text classification. The dataset comprises 3,105 appellate proceedings from the Santa Catarina State Court (TJSC), collected via the DataJud API (CNJ) and annotated across five legal areas through LLM-assisted labeling with heuristic validation. On a class-balanced test set, BERTimbau-LoRA, updating only 0.3% of model parameters, achieves 87.6% accuracy and 0.87 macro-F1 (+22pp over Claude 3.5 Haiku, +28pp over GPT-4o mini). The gap is most striking on administrativo (administrative law): GPT-4o mini scores F1 = 0.00 and Claude 3.5 Haiku scores F1 = 0.08 on this class, while the fine-tuned model reaches F1 = 0.91. Both commercial LLMs exhibit a systematic bias toward civel (civil law), absorbing ambiguous classes rather than discriminating them, a failure mode that domain-adapted fine-tuning eliminates. These results demonstrate that general-purpose LLMs cannot substitute for domain-adapted models in Brazilian legal classification, even when the task is a simple 5-class problem, and that LoRA fine-tuning on a consumer GPU closes the gap at zero marginal inference cost. We release the full dataset, model, and pipeline to enable reproducible research in Portuguese legal NLP.
Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal reasoning remains underexplored. We present TW-LegalBench that utilizes Taiwanese legal system's rich official corpus open to the public to fill the gap in evaluating LLMs on Taiwanese law, among common-law benchmarks that focus on English sources and civil-law benchmarks focusing on sources of Simplified Chinese. TW-LegalBench comprises three task types: (1) over 16,000 multiple-choice questions (MCQs) across five years of official examinations in 18 professional domains; (2) 117 open-ended essay questions (OEQs) from examinations for legal professionals with official scoring rubrics; and (3) more than 14,000 legal judgment prediction (LJP) instances covering hundreds of crime categories. We evaluate 13 LLMs using accuracy for MCQs, a decomposed LLM-as-Judge framework based on the scoring rubric points for OEQs, and metrics for sentencing accuracy and statute citation for LJP. Our results reveal that top-performing models exceed the passing threshold for qualified lawyers (passing rate: 11%) but fall short of that for judges and prosecutors (passing rate: 1~2%). For LJP, while models demonstrate reasonable verdict type accuracy and sentence prediction capability, they struggle to cite exact legal articles. These findings highlight that reliable legal text generation remains challenging for LLMs, even though their performance on qualification examinations approaches human level.
Fei-Yueh Chen, Chun Huang Lin, Chan Wei Hsu +4
University of Rochester Rochester, NY, USA · National Taiwan University Taipei, Taiwan · NVIDIA Santa Clara, CA, USA