Legal QA

QA: Question Answering

Legal question answering studies how AI systems can answer questions about laws, cases, and regulations while accounting for the facts and jurisdiction involved. Common approaches retrieve authoritative passages with keyword, dense, or hybrid search, then generate answers; newer systems add legal-structure-aware indexing, multi-agent reasoning, and claim-by-claim citation checks. Benchmarks increasingly test realistic consultations, citation support, missing facts, and changes in the law over time, rather than only multiple-choice accuracy. A central challenge is that retrieval can improve grounding but does not guarantee correct answers: systems may misuse supplied law, cite unsupported claims, or fail to ask for clarification or abstain when evidence is insufficient.

Latest papers

56 papers6 papers in the last four weeks, down 45% on the four weeks before. 0.0% of all new papers.

norregel: a preregistered benchmark of dated numeric values in Norwegian law
Oct 21, 2026Eirik Botten NicolaysenLLM Reliability Factual Knowledge in Language Models
EcoDeco AS
Evidence Sufficiency and Model Comparisons in MexRegQA: An Item-Level Audit of Mexican-Spanish Regulatory QA
Oct 19, 2026Fred Hu, Yeqi Wang, Xing ZhouBenchmark Auditing Legal QA
Huawei Technologies de México, Mexico · Northeastern University, USA
Legal-R1: Agentic Retrieval for Evidence-Based Legal Reasoning
Oct 13, 2026Huaiwen Pang, Jiale Han, Junying Chen +2Legal QA Legal Reasoning
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · Shenzhen Shenten Digital Technology Co., Ltd.
Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System
Oct 2, 2026Rubén Manrique, Michelle Castellanos, Jorge Morales +3LLM Reliability LLM Evaluation
Department of Systems and Computing Engineering, Universidad de los Andes · School of Government, Universidad de los Andes · Faculty of Law, Universidad de los Andes
LawCompass: Navigating from Legal QA to Multi-Agent Deep Research with Grounded Evidence
Oct 1, 2026Xiaoxia Cheng, Linnan Wang, Jiahao Ma +5Multi-Agent LLM Systems Legal IR
Anhui University · Tsinghua University
ViLegalExpert: A Large-Scale Benchmark for Vietnamese Legal Retrieval and Question Answering from Real-World Consultations
Sep 30, 2026Dat Tien Nguyen, Nghia Hieu Nguyen, Anh Thi-Hoang Nguyen +3Legal QA Legal NLP
University of Information Technology · Vietnam National University, Ho Chi Minh city, Viet Nam
ARGUS: Role-Aware Event Knowledge Graphs for U.S. Employment-Discrimination Complaints
Sep 24, 2026Sriram Kannan, Swetha Saseendran, Vishnu Vardhan Reddy Kandi +3KG Construction Document Information Extraction
University of Massachusetts Amherst · Bloomberg
Automated Regulatory Compliance Question Answering in Financial Services with Domain-Adapted Retrieval-Augmented Generation
Sep 24, 2026Tobias Deußer, Abhishek Pillai, Aurelio F. Bariviera +5Retrieval-Augmented Generation Legal QA
University of Bonn, Bonn, Germany · Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany · Universitat Rovira i Virgili, Reus, Spain +1
Cross-Lingual Legal QA for Vietnamese Labour Law: Retrieval, Translation, and Verifier-Guided Correction
Sep 23, 2026Nguyen Minh Chi, Mo El-Haj, Nguyen Ha Thanh +2LLM Answer Verification Legal Citation Verification
VinUniversity, Vietnam · Lancaster University, UK · Cardiff University, UK
LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoning
Sep 22, 2026Qingjing Chen, Junkai Zhang, Shaochun Wang +7LLM Reasoning with Graphs Graph Retrieval-Augmented Generation
Alma AI, University of Bologna, Italy · Tsinghua University, China · Modelbest Inc. +2
Efficient LLM Distillation for Bangladesh Legal Context: A Smartphone-Compatible Retrieval-Augmented Generation Model
Sep 21, 2026MD. Nafis Kamal, Mahadi Hasan Fahim, Talha Ridwan +4Retrieval-Augmented Generation Legal NLP
Computer Science and Engineering BRAC University Dhaka, Bangladesh
PRISM-RAG: Multimodal Hypergraph Retrieval-Augmented Generation for Tobacco Product and Legislative Policy Reasoning
Sep 20, 2026Manuel Serna-Aguilera, Raegan Anderes, Page Dobbs +1Multimodal RAG Legal IR
Fay W. Boozman College of Public Health, University of Arkansas for Medical Sciences, Little Rock, Arkansas, 72205
Parser, Chunking, and Embedding Interactions in Retrieval-Augmented Generation over Indian Government Regulatory Documents
Sep 15, 2026Shubham Kumar SinghRAG Evaluation IR Evaluation
Building Legal Reward Models for Grounding and Abstention
Sep 13, 2026Rilton Franzone, Valentin Noël, Puyu Wang +2Reward Modeling LLM Grounding
Devoteam
GANDR: Claim Auditing for Verifiable Legal Answer Generation
Sep 9, 2026Chen Qian, Yimeng Wang, Yu Chen +2LLM Answer Verification Legal Citation Verification
William & Mary · Anytime AI
KhatianDoc: A Human-Verified Benchmark Diagnosing Multimodal LLM Failure on Bengali Legal Land Records
Sep 3, 2026Tasmiad Hasan, Arafat Zaman Ratul, Sarker Sadman Saalim +3Legal Document Analysis Legal QA
North South University, Dhaka, Bangladesh
Do Small Models Use the Law You Give Them? Measuring Context Use on a Bilingual Bangladesh Legal Benchmark
Aug 31, 2026Moniruzzaman Mahadi, Abrar Mohammed Tanzim Alam, Sayma Siddika Monalisa +3Multilingual Language Model Evaluation LLM Evaluation
North South University · BRAC University
Cloud and On-Premises Deployment of Uzbek Legal RAG via Targeted Retriever Fine-Tuning
Aug 29, 2026Tatul Danielyan, Mariam Avetisyan, Hrant DavtyanLegal QA Legal NLP
Metric AI Lab
InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries
Aug 20, 2026Samuel J. Vincent, Daniel Calloway, Fangyi Yu +2Legal QA LLM Evaluation
Thomson Reuters Foundational Research, 5 Canada Square, London · Imperial College London
LabourCrew: A Multi-Agent RAG Framework for Trustworthy Adversarial Deliberation and Statutory Reasoning over Labour Law
Aug 18, 2026Fatema Tuj Johora Faria, Mukaffi Bin Moin, Jubayer Al Mahmud +2Legal QA Agentic RAG
Ahsanullah University of Science and Technology, Bangladesh · Jashore University of Science and Technology, Bangladesh · American International University - Bangladesh
CoAL-RAG: A Complexity-Aware Legal Retrieval-Augmented Generation Method
Aug 18, 2026Jin Su, Zhuofeng Zhao, Huanhuan Wang +1Retrieval-Augmented Generation Legal QA
North China University of Technology, Beijing, 100144, China · Beijing Key Laboratory of Key Technologies for AI+ Domain Applications, Beijing, 100144, China · Beijing Key Laboratory on Integration and Analysis of Large-scale Stream Data, Beijing 100144, China
Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks
Aug 15, 2026Volodymyr OvcharovLLM Evaluation Multiple-Choice Question Answering
LEX AI
How Much Do Legal RAG Systems Still Hallucinate?
Aug 14, 2026Souvick Das, Sallam Abualhaija, Domenico BianculliHallucination in Language Models Retrieval-Augmented Generation
SnT, University of Luxembourg Luxembourg
Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law
Aug 10, 2026Rose Cymbler, Daniel Guez, Laurent FabreTemporal IR Legal QA
Talia, Paris, France · Databricks
Time Present and Time Past: Benchmarking Large Language Models on Temporally Evolving Document Understanding
Aug 9, 2026Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Fahmid Hasan Chowdhury +4LLM Evaluation Document QA
1BRAC University · 2United International University · 3North South University +1
Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh
Jul 26, 2026Moniruzzaman Mahadi, Abrar Mohammed Tanzim Alam, Sayma Siddika Monalisa +3Multilingual Language Model Evaluation LLM Grounding
Department of Electrical and Computer Engineering, North South University Dhaka, Bangladesh · Department of Computer Science and Engineering, BRAC University Dhaka, Bangladesh
AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal System
Jul 21, 2026Shubham Kumar Nigam, Shubham Kumar Mishra, Noel Shallum +2Retrieval-Augmented Generation Legal NLP
Computer Science and Engineering, Indian Institute of Technology, Kanpur, 208016, Uttar Pradesh, India. · School of Computer Science, University of Birmingham, Dubai, UAE. · Law, Symbiosis Law School, Pune, 411014, Maharashtra, India. +1
Beyond Accuracy: How Humans Evaluate Legally Correct but Socially Controversial Legal Advice from Machines
Jul 6, 2026Benjamin Minhao Chen, Zhiyu LiAI-Assisted Decision Making Legal QA
a Associate Professor, Faculty of Law, University of Hong Kong · b Associate Professor in Law and Policy, Durham Law School, Durham University
Train, Retrieve, or Both? A Four-Arm Head-to-Head for Correct Statutory Citation on the Ontario Residential Tenancies Act
Jun 18, 2026Ali Asaria, Tony Salomone, Deep GandhiSupervised Fine-Tuning Retrieval-Augmented Generation
Transformer Lab
DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation
Jun 11, 2026Li Zhang, Yuzhen Shi, Yiran Hu +15LLM Evaluation Legal QA
University of Pittsburgh · Alibaba Group · University of Waterloo +8

Oct 21, 2026J/K move · Enter open · S save

norregel: a preregistered benchmark of dated numeric values in Norwegian law

Eirik Botten Nicolaysen

EcoDeco AS

LLM Reliability Factual Knowledge in Language Models Legal QA

Open paper AskPDF arXiv

norregel asks language models for numeric values in Norwegian statutes, regulations and parliamentary decisions (NOK amounts, percentages, multiples of the National Insurance basic amount (G) and age limits), each at a stated date. The 916 items were extracted mechanically from Lovdata's open data, and each carries the dated history of its provision, so an answer can be classed as correct, outdated, wrong or uninterpretable. We report a first and a second preregistered run on the same items. In the first run (five models, listed publicly only after local analysis), H1, that outdated answers are more frequent for values that took effect after a model's cutoff, could not be tested in substance, and H2 and H3 were inconclusive; Opus 5.5 and Sonnet 4.6 answered 46.7% and 40.2% of the 880 primary items correctly. The second run was preregistered and published before any of its calls, and eight models were analysed. Its pooled primary test supported H1 under the preregistered decision rule (d = 17.8 percentage points, 95% interval [7.7, 28.9], p = 0.0001). In the preregistered sensitivity analysis S8, which counts as outdated only the most recent value that differs from the key, d was 3.9 [-3.6, 12.7] (p = 0.1754), an interval that contains 0. H1 was supported in every other preregistered sensitivity analysis, but S8 is the only one that gives both groups one target value, so we do not regard the support for H1 as robust. The share of outdated answers equal to that most recent value (H2) was 56.1% [46.6, 65.0], an interval that contains 0.5, and the pooled test of a lower share correct in Nynorsk than in Bokmål (H3) was inconclusive. Several models, among them Haiku 4.5, gave mostly uninterpretable answers. Items and keys are withheld; item IDs, per-answer classes, code and results are on Zenodo.