Organizations: Centro de Ciências e Tecnologia, Universidade Federal do Cariri (UFCA), Juazeiro do Norte, Ceará, Brazil. · Faculdade de Engenharia Elétrica e da Computação, Universidade Estadual de Campinas (UNICAMP), Campinas, São Paulo, Brazil. · Câmara dos Deputados, Brasília, Brazil. · NeuralMind.ai, Campinas, São Paulo, Brazil.
Legal information retrieval in Portuguese remains difficult to evaluate systematically because available datasets differ widely in document type, query style, and relevance definition. We present JUÁ, a public benchmark for Brazilian legal retrieval designed to support more reproducible and comparable evaluation across heterogeneous legal collections. More broadly, JUÁ is intended not only as a benchmark, but as a continuous evaluation infrastructure for Brazilian legal IR, combining shared protocols, common ranking metrics, fixed splits when applicable, and a public leaderboard. The benchmark covers jurisprudence retrieval as well as broader legislative, regulatory, and question-driven legal search. We evaluate lexical, dense, and BM25-based reranking pipelines, including a domain-adapted Qwen embedding model fine-tuned on JUÁ-aligned supervision. Results show that the benchmark is sufficiently heterogeneous to distinguish retrieval paradigms and reveal substantial cross-dataset trade-offs. Domain adaptation yields its clearest gains on the supervision-aligned JUÁ-Juris subset, while BM25 remains highly competitive on other collections, especially in settings with strong lexical and institutional phrasing cues. Overall, JUÁ provides a practical evaluation framework for studying legal retrieval across multiple Brazilian legal domains under a common benchmark design.
Figures & tables
Figure 1: Complementary comparison of representative models using MRR@10 across the JUÁ datasets. This plot emphasizes the quality of early-hit ranking and makes the strong effect of domain adaptation on JUÁ-Juris more visible.
Figure 2: Complementary comparison of representative models using MAP@10 across the JUÁ datasets. MAP@10 highlights quality across the full top-10 ranking rather than only the position of the first relevant document.
Figure 3: Complementary comparison of representative models using Recall@10 across the JUÁ datasets. Recall@10 highlights differences in top-10 coverage, especially on the legislative and normative subsets.
Department of Informatics, Athens University of Economics and Business, Greece · Archimedes, Athena Research Center, Greece · Institute for Language and Speech Processing, Athena Research Center, Greece