cs.CLNov 14, 2026

SG-LegalClarify: A Large-Scale Benchmark for Legal Reasoning and Ambiguity Recognition in Singapore Law

Authors: Ziwei Gu, Michael S. Lee, Drew Rein, Yash Maurya, Vikneswaran Kumaran, Darshini Ramiah, Chenguang Wang, Udari Madhushani Sehwag

Organizations: Human Frontier Collective, Scale AI · Harvard University · Preparedness and Policy, Scale AI · Scale AI · 

Abstract

Singapore offers a distinctive testbed for legal reasoning by large language models (LLMs). Its legal system combines inherited English common law with jurisdiction-specific statutes, requiring models to distinguish local doctrine from generic common-law intuition. This setting also highlights the challenge of recognizing underspecification in lay legal queries, because the facts material to a legal outcome depend on the governing doctrine. Yet legal benchmarks rarely cover Singapore law or test this form of ambiguity recognition. To address these gaps, we introduce SG-LegalClarify, a large-scale benchmark for open-ended legal reasoning and ambiguity recognition in Singapore law. SG-LegalClarify contains 1,100 lawyer-authored source questions spanning 10 legal domains, including 340 fully specified and 760 underspecified questions. Each underspecified question has a fixed hidden-fact assignment and a fully specified counterpart, yielding 1,860 evaluation instances. Responses are graded with lawyer-authored Issue, Rule, Application, and Conclusion (IRAC) rubrics. We evaluate 19 LLMs, closed-book and with web search, using a multi-turn pipeline in which models decide whether to answer or clarify, and a user simulator reveals hidden facts only when asked. We find the benchmark far from saturated: the best models satisfy only 69.8% of mandatory rubric items closed-book (average 53.8%) and 77.8% with web search (average 63.0%). Models also struggle to recognize when clarification is necessary. We will release the public split of our benchmark, the model outputs on it, and the evaluation pipeline to support future research.

Explore similar work

CardsList
  1. TW-LegalBench: Measuring Taiwanese Legal Understanding

    Jun 17, 2026Fei-Yueh Chen, Chun Huang Lin, Chan Wei Hsu +4LLM EvaluationLegal Judgment Prediction

  2. CrossLex: A Source-Grounded Benchmark for Cross-Jurisdictional Legal Reasoning in Large Language Models

    Aug 2, 2026Xiaocui Yang, Xican Tan, Shoujie Chen +3LLM EvaluationLLM Grounding

  3. InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries

    Aug 20, 2026Samuel J. Vincent, Daniel Calloway, Fangyi Yu +2Legal QALLM Evaluation