cs.AIOct 2, 2026
SaveDo Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System
Organizations: Department of Systems and Computing Engineering, Universidad de los Andes · School of Government, Universidad de los Andes · Faculty of Law, Universidad de los Andes
Abstract
Large language models (LLMs) are increasingly used to support legal practice, education, and research, yet their reliability in national legal systems outside the United States remains largely undocumented. We introduce an expert-validated benchmark for evaluating LLM reliability on the Colombian legal system. The benchmark comprises 1,042 items spanning ten areas of law and three question formats (closed multiple-choice, semi-open, and open-ended IRAC), built through a human-in-the-loop pipeline with multi-stage expert review. We evaluate 15 contemporary proprietary and open-weight models with format-appropriate metrics. Accuracy on closed questions ranges widely, from 0.905 (Gemini 3.1 Pro) to 0.577, but on free-text legal answers factual correctness never exceeds 0.45 (on a 0-1 scale) for any model. We find a dissociation between answer relevancy and correctness (Spearman rho = -0.46): models reliably sound responsive while frequently being wrong, a pattern of particular concern for non-expert users. Closed-question accuracy and free-text correctness are strongly rank-correlated (rho = 0.94), so cheap multiple-choice screening predicts model ranking but overstates absolute reliability. An independent rubric-based LLM judge and blind human expert scoring both reproduce the free-text ranking (rho >= 0.88). The judge further reveals that only about half of the norms models cite are correct; the rest are wrong or non-existent. Reliability varies systematically by legal area and follows an inverted-U across question complexity. Our results indicate that current LLMs require expert supervision for Colombian legal tasks, and that grounding answers in authoritative sources is a promising path to higher reliability. We release the benchmark construction pipeline to support reproducible evaluation.
Figures & tables
| Level | Sub-tasks |
|---|---|
| Low | Normative existence; competent authority; deciding judge; legal hierarchy; basic definition; legal classification; essential elements. |
| Medium | Sense of the ruling; literal reproduction of a norm; jurisprudential precedent; temporal validity; conceptual distinction; legal requirements; legal exceptions; impartiality of the judicial order. |
| High | Normative conflict; factual background; procedural stance; legal problem; central legal basis; balancing of principles and rights; systematic interpretation. |
Table 1: Sub-tasks targeted by semi-open questions at each complexity level, adapted from Dahl et al. [4] to the Colombian legal order.
Figure 1: Composition of the evaluated benchmark by legal area and question type. Bars are ordered by total number of items; the value at the end of each bar is the area total.
| Legal area | Closed | Semi-open | Open | Total |
|---|---|---|---|---|
| Constitutional | 39 | 86 | 9 | 134 |
| Administrative | 53 | 63 | 8 | 124 |
| Criminal | 26 | 93 | 4 | 123 |
| Procedural | 37 | 71 | 3 | 111 |
| Commercial & corporate | 26 | 70 | 8 | 104 |
| Civil | 27 | 70 | 5 | 102 |
Table 2: Evaluated benchmark composition by legal area and question type.
Figure 2: Distribution of semi-open questions across the three complexity levels.
| Workshop | Date | Mode | Questions | Participants |
|---|---|---|---|---|
| W1 | 10 Nov 2025 | In-person | 238 | 16 |
| W2 | 14 Nov 2025 | In-person | 64 | 19 |
| W3 | 10 Dec 2025 | Virtual | 26 | 2 |
| W4 | 15 Dec 2025 | Virtual | 7 | 2 |
| W5 | 05 Mar 2026 | In-person | 80 | 10 |
| Total | 415 | 49 |
Table 3: Summary of the question-generation workshops.
Figure 3: Questions contributed per workshop, colored by modality.
Figure 4: Qualification-and-approval protocol. Every question—whether generated in a workshop, by the legal team, or by an individual student—is routed through the same four-step protocol (reception, assignment to the legal assistant, expert evaluation, and committee decision) before entering the final bank; at each decision point an item may be edited, sent back for adjustment, or archived.
| Provider | Model | Type |
|---|---|---|
| Amazon | Nova Lite 1.0 | General |
| Anthropic | Claude Opus 4.5 | General |
| Anthropic | Claude Sonnet 4.5 | General |
| Cohere | Command R (08-2024) | General |
| DeepSeek | DeepSeek V3.2 | General |
| Gemini 3.1 Pro Preview | General |
Table 4: The large language models evaluated on the benchmark.
| Metric group | Metrics | Subset ( ) |
|---|---|---|
| Accuracy | accuracy | Closed (305) |
| Answer quality | BERTScore, BLEU, ROUGE-1/L, | Free-text (737) |
| RAGAS relevancy/correctness | ||
| Readability | Fernández-Huerta, Szigriszt-Pazos, | Free-text (737) |
| (Spanish) | Gutiérrez, , Crawford (—) |
Table 5: Evaluation metrics by question format and the subset each is computed over. Arrows indicate whether higher ( ) or lower ( ) values are preferable; readability indices are descriptive (—).
| Closed | Free-text (open + semi-open) | |||
|---|---|---|---|---|
| Model | Accuracy | BERTScore | RAGAS rel. | RAGAS corr. |
| Gemini 3.1 Pro | 0.905 | 0.712 | 0.758 | 0.447 |
| Claude Opus 4.5 | 0.869 | 0.714 | 0.742 | 0.428 |
| GPT-5.4 | 0.859 | 0.717 | 0.792 | 0.451 |
| GLM 5.2 | 0.856 | 0.708 | 0.745 | 0.389 |
| Claude Sonnet 4.5 | 0.803 | 0.712 | 0.763 | 0.408 |
Table 6: Headline results per model, sorted by closed-question accuracy. Free-text metrics (BERTScore F 1 and RAGAS relevancy/correctness) are computed over the open and semi-open items; accuracy over the closed items. Best value per column in bold .
Figure 5: Accuracy on closed multiple-choice questions. Bars are shaded by performance tier ( , – , ); the dashed line marks the random-guessing baseline.
Figure 6: Closed-question accuracy by model (rows, sorted by mean accuracy) and legal area (columns, sorted by mean accuracy). Darker cells indicate higher accuracy.
Figure 7: Inter-model agreement on closed questions, showing the number of questions for which a given number of models (0–15) answered correctly. A large consensus spike at 15/15 contrasts with a short tail of near-universally-failed items.
| Correct | Area | Topic | Likely cause |
|---|---|---|---|
| 0/15 | Procedural | Consumer-protection competent authority | Meta-option (“both”) |
| 0/15 | Constitutional | Composition of Congress (upper/lower house) | Meta-option (“none”) |
| 0/15 | Administrative | Administrative silence (Resolution 368/2014) | External document |
| 0/15 | Administrative | Disqualifications in state contracts | Ambiguous |
| 0/15 | Criminal | Onset of the protected legal interest (life) | Contested doctrine |
| 1/15 | Commercial | Who may grant financial leasing | Nuanced/recent |
Table 7: The ten hardest closed questions, by number of models answering correctly (out of 15). “Cause” classifies the likely reason for near-universal failure.
Figure 8: Best single model vs. a 15-model majority vote vs. the oracle (at least one model correct). Majority voting underperforms the best single model because model errors are correlated.
Figure 9: Hierarchical clustering of models by response agreement on closed questions (average linkage; distance agreement). Clusters track capability tier rather than provider.
Figure 10: Pairwise McNemar tests on closed-question outcomes (models sorted by accuracy). Grey cells mark statistically indistinguishable pairs; the block-diagonal grey regions reveal performance tiers.
Figure 11: Closed-question accuracy by model access (open-weight vs. proprietary); reasoning-oriented models highlighted. Horizontal bars are group means.
Figure 12: Item response theory (2PL) item map. Each point is a question, positioned by estimated difficulty (x) and discrimination (y). Orange ticks mark the models’ estimated abilities on the same logit scale.
Figure 13: Distribution of RAGAS answer (a) correctness and (b) relevancy over the free-text items, per model (violins; models sorted by mean correctness). Diamonds mark the mean and horizontal bars the % bootstrap confidence interval. Correctness is low and widely spread for every model (dashed line at ), whereas relevancy is uniformly high and concentrated.
Figure 14: RAGAS answer relevancy versus correctness. (a) Across the models (means): higher relevancy does not imply higher correctness ( ). (b) Across all individual answers (hex-binned density): the two are effectively uncorrelated ( ). High relevancy does not imply correctness at either level.
Figure 15: Pairwise paired Wilcoxon signed-rank tests on per-item free-text correctness (Holm-corrected; models sorted by mean correctness). Grey cells mark statistically indistinguishable pairs; the block structure reveals performance tiers.
| Corr. | Area | Topic | Likely cause |
|---|---|---|---|
| 0.04 | Constitutional | Reporting magistrate of ruling T-256/2025 ( tutela vs. Meta) | Exact attribution; post-cutoff |
| 0.07 | Constitutional | Deciding judge of ruling T-426/2003 | Exact attribution |
| 0.07 | Constitutional | Reporting magistrate of ruling T-323/2024 (use of AI) | Exact attribution; recent |
| 0.09 | Procedural | Article on the requirements to be a conciliator (Law 2220/2022) | Exact citation; recent statute |
| 0.09 | Administrative | Core holding of ruling SU-167/2023 (direct reparation) | Recent case law; reasoning |
| 0.10 | Tax | Holding of Council of State ruling 26644/2024 | Recent case law |
Table 8: The ten hardest free-text items, by mean RAGAS answer correctness across the models. “Likely cause” classifies the dominant reason for near-universal failure. All ten are semi-open items.
Figure 16: RAGAS answer correctness on semi-open items by complexity level. The four highest-scoring models are highlighted; remaining models are shown in grey.
Figure 17: Mean RAGAS correctness by semi-open sub-task (averaged over the models), colored by complexity level; the item count per sub-task is shown at the end of each bar. Low-complexity sub-tasks that hinge on a single exact legal fact (deciding judge, normative existence) are among the worst, alongside the reasoning-heavy high-complexity sub-tasks.
Figure 18: Free-text RAGAS correctness by model (rows, sorted by mean) and legal area (columns, sorted by mean). Darker cells indicate higher correctness. The analogue of the closed-question accuracy heatmap (Figure 6 ).
Figure 19: Reliability of normative citations by model (LLM-judge classification on the -item subset): share of cited norms that are correct, cite the wrong norm, or are non-existent. Dashed line marks the pooled correct rate ( %). ∗ GPT-5.4 is also the judge model (Section 7.5 ).
Figure 20: Convergent validity: each model’s mean rubric-based LLM-judge score against its mean RAGAS answer correctness ( models, -item subset). Two methodologically independent instruments agree closely on the ranking. ∗ GPT-5.4 is also the judge model.
Figure 21: Human validation of the LLM judge: confusion matrix of judge versus expert-team scores on the – scale (row-normalized, blind judgments on the -model subset). The extremes are reliable; the mid-scale is shifted down by about one point, a monotone shift that preserves the model ranking.
Figure 22: Answer length and readability. (a) Mean answer length (words) vs. RAGAS correctness across models: longer answers score higher ( ). (b) Fernández-Huerta ease by semi-open complexity level; values sit inside the calibrated – band. Higher ease easier to read.
Figure 23: Readability of model answers versus the human expert references (Fernández-Huerta ease; higher easier). (a) Each model’s mean ease (points) against the human-reference mean (solid) and median (dashed): most models write somewhat denser prose than the human references, but within the ordinary range. (b) Ease by semi-open complexity level: the human–model gap is largest at low complexity and closes at high complexity.