LabourCrew: A Multi-Agent RAG Framework for Trustworthy Adversarial Deliberation and Statutory Reasoning over Labour Law

Fatema Tuj Johora Faria; Mukaffi Bin Moin; Jubayer Al Mahmud; M. F. Mridha; Md. Alam Hossain

cs.CLAug 18, 2026

LabourCrew: A Multi-Agent RAG Framework for Trustworthy Adversarial Deliberation and Statutory Reasoning over Labour Law

Authors: Fatema Tuj Johora Faria, Mukaffi Bin Moin, Jubayer Al Mahmud, M. F. Mridha, Md. Alam Hossain

Organizations: Ahsanullah University of Science and Technology, Bangladesh · Jashore University of Science and Technology, Bangladesh · American International University - Bangladesh

Abstract

In statutory question answering, every claim must be traceable to evidence, not merely relevant, since unverifiable labour-rights answers carry serious legal consequences. Current systems fall short: single-pass RAG cannot detect insufficient evidence, while multi-agent legal-debate systems treat grounding as a prompting convention, letting agents cite unretrieved evidence. To address this gap, we introduce LabourCrew, a multi-agent RAG framework built around three grounding mechanisms: StatuteGraph, a graph index that explicitly links chapter, section, proviso, and cross-reference structure rather than fixed-length spans; an Evidence Exchange Protocol that confines advocates and an interpreter to an evidence ledger, making citation to unretrieved text impossible, while a fault-tolerant supervisor board runs advocates in parallel so individual failures degrade rather than crash the system; and a Calibrated Trust Gate that replaces categorical accept/reject decisions with a trust score, thresholded via conformal risk control for a distribution-free bound on the false-accept rate. We evaluate on LabourActQA, a 500-item Bangla question set from the Bangladesh Labour Act, 2006, spanning seven reasoning categories and three difficulty tiers. The framework drives the empirical false-accept rate to 0.081, within the target level ( $α= 0.10$ ), achieves the highest Answer Relevancy among HyDE RAG, Graph-RAG, and Hierarchical RAG (0.862 $>$ 0.839, 0.815, 0.828), and degrades gradually rather than catastrophically as question difficulty increases. These results show that calibrated abstention, not retrieval quality alone, is what makes legal question answering auditable in low-resource statutory domains.

Figures & tables

Figure 1: How LabourCrew answers a query: Retrieve $\rightarrow$ Deliberate $\rightarrow$ Audit & Calibrate $\rightarrow$ a grounded, degraded, or withheld answer.

Figure 2: The LabourCrew architecture: Phase I builds StatuteGraph offline; Phases II–V run per query, retrieving evidence, deliberating over it, auditing and retrying failed roles, then composing and gating release as TGLO , DEGRADED , or UNDECIDED . All agents communicate only through the shared Formal Case Ledger (FCL) .

Method / Configuration	N	Answer Relevancy $\uparrow$	Context Relevancy $\uparrow$	Context Precision $\uparrow$	Context Recall $\uparrow$	Noise Sensitivity $\downarrow$
LabourCrew	500	0.862	0.891	0.914	0.913	0.081
w/o Trust Auditor	500	0.839	0.880	0.879	0.904	0.121
w/o Link Hopper	500	0.842	0.829	0.844	0.844	0.129
w/o Worker Counsel	500	0.833	0.885	0.891	0.905	0.095
w/o Employer Counsel	500	0.846	0.887	0.902	0.908	0.089
w/o Legal Interpreter	500	0.838	0.882	0.894	0.904	0.094

Table 1: RAGAS evaluation (GPT-4.1 judge) of the full LabourCrew framework against six leave-one-out agent ablations and three external retrieval baselines, on the same 500 LabourActQA questions.

N

is the number of questions evaluated per configuration. Arrows indicate whether higher or lower is better.

Category	N	Lat. (s)	Rounds	EC	PC	GVCP	TGLO	CBP
Direct Factual Retrieval	78	19.7	1.00	0.949	0.979	1.000	100.0%	0.763
Definitional and Classification	76	20.5	1.00	0.912	0.965	1.000	100.0%	0.679
Procedural Reasoning	61	24.5	1.01	0.894	0.972	0.998	100.0%	0.630
Conditional Reasoning	74	22.3	1.02	0.905	0.988	0.985	99.0%	0.641
Comparative Reasoning	70	23.0	1.01	0.930	0.964	1.000	100.0%	0.680
Multi-hop Reasoning	74	30.2	1.14	0.804	0.958	0.975	97.0%	0.412

Table 2: Deterministic, execution-trace evaluation of the full LabourCrew system, broken down by the seven LabourActQA reasoning categories, ordered easiest to hardest; metrics defined in Section B.6 . EC: Evidence Coverage; PC: Path Completeness; GVCP: Gate-Verified Citation Precision; TGLO: TGLO Rate; CBP: Citation-Based Precision. All columns are higher-is-better except Latency.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Composition of the 500-item LabourActQA benchmark by reasoning category and difficulty tier (Section A.5 ).

Evaluation Criterion	Description	Mean $±$ SD
Correctness	The legal interpretation is factually correct according to the Labour Act.	4.61 $±$ 0.54
Faithfulness	All claims are supported by the cited legal evidence without hallucination.	4.74 $±$ 0.46
Completeness	The response addresses all important aspects of the user’s query.	4.52 $±$ 0.61
Clarity	The explanation is coherent, well-structured, and easy to understand.	4.67 $±$ 0.49
Citation Quality	The cited legal provisions are accurate, relevant, and sufficient to justify the conclusions.	4.79 $±$ 0.42
Overall Quality	Overall usefulness, reliability, and practical value of the generated legal opinion.	4.68 $±$ 0.45

Appendix

Table 3: Human evaluation of a sample of LabourCrew -generated legal opinions, rated independently by two legal-domain evaluators on a five-point Likert scale across six quality dimensions. Values are mean

\pm

SD across rated opinions and both evaluators.

Reasoning Type	Difficulty Level	Sample Size
Direct Factual Retrieval	Easy	78
Definitional and Classification	Easy	76
Procedural Reasoning	Medium	61
Conditional Reasoning	Medium	74
Comparative Reasoning	Medium	70
Multi-hop Reasoning	Hard	74

Appendix

Table 4: Number of LabourActQA items per reasoning category and difficulty tier.

Figure 4: Internal Faithfulness across five statute-chunking strategies feeding an otherwise identical retrieval pipeline: fixed 512-token windows, a recursive character splitter, whole-chapter chunks, section-only chunks, and the proposed structure-preserving parser.

Retrieval Strategy	Initial Retrieval Precision	TGLO Rate	Latency (s)
Dense only	0.52	82.3%	18.2
Dense + BM25	0.69	91.6%	21.5
Dense + Link Hops	0.74	96.2%	25.8
StatuteGraph (Full)	0.78	99.0%	24.3

Appendix

Table 5: Ablation over four retrieval strategies, each substituted into the full pipeline with everything else held fixed, evaluated on all 500 LabourActQA questions. Precision and TGLO Rate are higher-is-better; Latency is lower-is-better.

Chunking Strategy	Internal Faithfulness
Fixed 512 Token	0.71
Recursive Splitter	0.76
Whole Chapter	0.63
Section-only	0.74
Structure-aware (Ours)	0.82

Appendix

Table 6: Ablation over five statute-chunking strategies feeding an otherwise identical retrieval pipeline. Internal Faithfulness is the fraction of a claim’s cited spans found verbatim in their cited chunk’s text; higher is better.

Planner Config	Planner-Isolated Precision	Avg Retrieval Rounds	Latency (s)
Static Query	0.71	1.89	28.4
Query Rewrite	0.81	1.61	26.1
Planner Only	0.89	1.18	25.2
Planner + Replanning	0.91	1.05	24.3

Appendix

Table 7: Ablation over four query-planning configurations, holding retrieval execution, deliberation, and gating fixed. Precision is higher-is-better; rounds and latency are lower-is-better.

Max Hop Depth	Internal Faithfulness	Latency (s)
0 (no hops)	0.72	22.8
1	0.78	23.6
2	0.81	25.1
3 (ours)	0.82	24.3

Appendix

Table 8: Internal Faithfulness and end-to-end latency as the Link Hopper’s maximum graph-traversal depth

h

increases from 0 to 3, holding every other pipeline component fixed.

Configuration	Failed Runs	Recovery Rate (of prior failures)	TGLO Rate	Latency (s)
Sequential Pipeline	71/500 (14.2%)	N/A	85.8%	31.2
Supervisor Only	24/500 (4.8%)	66.2% (47/71)	95.2%	28.1
Supervisor + Retry	6/500 (1.2%)	75.0% (18/24)	98.8%	25.8
Full Supervisor	5/500 (1.0%)	16.7% (1/6)	99.0%	24.3

Appendix

Table 9: Ablation over four orchestration configurations, evaluated on all 500 questions. Recovery Rate is the share of the immediately preceding row’s failures resolved by that configuration. Failed Runs and Latency are lower-is-better; Recovery Rate and TGLO Rate are higher-is-better.

Figure 5: Three worked LabourCrew runs, each showing the Bangla query, ground-truth answer, generated answer, and cited statute nodes: Left (Conditional) correctly finds Section 15 exempts small establishments from Sections 12–18; Middle (Comparative) correctly contrasts juveniles’ working-hour limits across Sections 41(1)/(2); Right (Multi-hop) correctly chains Sections 39(2) and 40 to show the hazardous-work ban still holds despite Section 40’s training exception.

Metric	Aggregate
Latency (s)	24.3
Avg Retrieval Rounds	1.05
Evidence Coverage	0.890
Path Completeness	0.962
Gate-Verified Citation Precision	0.991 (99.1%)
Citation-Checker Pass Rate	99.2%

Appendix

Table 10: Aggregate deterministic evaluation of the full LabourCrew framework over all 500 LabourActQA questions. This is the source aggregate for the per-category breakdown in Table 2 .

Question Type	k Ceiling	h Ceiling	Query Ceiling
Direct Factual Retrieval / Definitional & Classification	5	1	1
Procedural / Conditional Reasoning	8	2	2
Comparative Reasoning	8	2	4
Multi-hop Reasoning	10	3	4
Hypothetical Legal Reasoning	8	2	2

Appendix

Table 11: Per-question-type ceilings on the Retrieval Planner’s dense-hit count (

k

), Link Hopper hop depth (

h

), and query count, enforced deterministically at execution time.

Configuration	Internal Faithfulness	Gate-Verified Citation Precision	TGLO Rate
LabourCrew	0.82	0.991	99.0%
w/o Trust Auditor	0.76	0.842	96.4%
w/o Link Hopper	0.77	0.923	97.1%
w/o Worker Counsel	0.79	0.964	98.0%
w/o Employer Counsel	0.81	0.978	98.6%
w/o Legal Interpreter	0.80	0.982	98.8%

Appendix

Table 12: Leave-one-out ablation removing one of six reasoning/gating agents at a time from the full eleven-agent pipeline, evaluated on all 500 questions; all columns are higher-is-better.

Configuration	Internal Faithfulness	Citation Error Rate	Unsupported Claims %	Gate-Verified Citation Precision
No Trust Gate	0.71	0.441	8.9%	0.559
Citation Checker Only	0.76	0.158	7.5%	0.842
Trust Auditor Only	0.79	0.088	4.8%	0.912
Full Trust Gate	0.82	0.009	1.2%	0.991

Appendix

Table 13: Decomposition of the Trust Gate’s contribution to citation reliability across four configurations, evaluated on the same 500 questions. Faithfulness and Precision are higher-is-better; Citation Error Rate and Unsupported Claims are lower-is-better.

Configuration	Unsupported Claims %	Internal Faithfulness	Gate-Verified Citation Precision	TGLO Rate
Unrestricted Debate-Agent Access	6.4%	0.77	0.947	97.8%
Role-Isolated (EEP, ours)	1.2%	0.82	0.991	99.0%

Appendix

Table 14: Effect of the Evidence Exchange Protocol on grounding, evaluated on all 500 questions: unrestricted retrieval access versus the proposed role-isolated protocol. Unsupported Claims is lower-is-better; remaining columns are higher-is-better.

Gate Mode	Gate-Verified Citation Precision	Empirical False-Accept Rate	Target α	Guarantee Holds?	TGLO Rate
Categorical (LLM-decided)	0.938	0.147	N/A	No	97.6%
Calibrated (CRC, ours)	0.991	0.081	0.10	Yes	99.0%

Appendix

Table 15: Comparison of the Trust Gate’s release-decision mode at a fixed target risk level

\alpha=0.10

. Precision and TGLO Rate are higher-is-better; Empirical False-Accept Rate is lower-is-better and is the quantity

\alpha

bounds.

Figure 6: LabourCrew against three external retrieval baselines (Hierarchical RAG, HyDE RAG, Graph-RAG) on five RAGAS metrics, scored on the same 500 LabourActQA questions (Table 1 ). LabourCrew

>

all baselines on every metric.

Figure 7: Deterministic metrics (Table 2 ) across the seven LabourActQA reasoning categories, easiest to hardest. Gate-Verified Citation Precision and TGLO Rate stay nearly flat with difficulty, while Citation-Based Precision drops sharply on the hardest categories: harder questions retrieve a wider evidence pack and cite less of it, rather than relaxing what counts as a verified citation.

Figure 8: Mean human Likert ratings (1–5 scale) across the six quality dimensions defined in Table 3 , from two independent legal-domain evaluators; error bars show standard deviation. Citation Quality rates highest (

4.79\pm 0.42

) and Completeness rates lowest (

4.52\pm 0.61

), though all six dimensions exceed 4.5.

Legal QA

Legal question answering uses language models and legal sources to provide accurate, jurisdiction- and time-specific, verifiable answers.

See all 56 papers

Oct 13, 2026

Legal-R1: Agentic Retrieval for Evidence-Based Legal Reasoning

Legal question answering requires answers that are not only plausible, but also grounded in legally valid and citable evidence. Standard retrieval-augmented generation often relies on one-shot semantic matching, which is insufficient when legal evidence must be selected according to authority, applicability, and diversity across heterogeneous sources. We introduce Legal-R1, an agentic retrieval framework for evidence-based legal reasoning. Legal-R1 decomposes legal reasoning into a multi-agent workflow, where a Lawyer agent learns when to retrieve, which legal source to query, what evidence to seek, and when the collected evidence is sufficient. A Legal Assistant agent retrieves from normative legal sources and precedent cases with authority-aware and diversity-aware selection, while a Judge agent generates the final answer from the complete evidence trajectory. We post-train only the Lawyer agent with masked supervised fine-tuning and trajectory-aware reinforcement learning. Experiments on multiple Chinese legal benchmarks show that Legal-R1 improves over general-purpose, legal-specific, and retrieval-augmented baselines.

Huaiwen Pang, Jiale Han, Junying Chen +2

The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · Shenzhen Shenten Digital Technology Co., Ltd.

Legal QA Legal Reasoning Agentic Retrieval Evidence-Grounded Generation Multi-Agent Reasoning

Ask PDF

Mar 2, 2026

Legal RAG Bench: an end-to-end benchmark for legal RAG

We introduce Legal RAG Bench, a benchmark and evaluation methodology for assessing the end-to-end performance of legal RAG systems. As a benchmark, Legal RAG Bench consists of 4,876 passages from the Victorian Criminal Charge Book alongside 100 complex, hand-crafted questions demanding expert knowledge of criminal law and procedure. Both long-form answers and supporting passages are provided. As an evaluation methodology, Legal RAG Bench leverages a full factorial design and novel hierarchical error decomposition framework, enabling apples-to-apples comparisons of the contributions of retrieval and reasoning models in RAG. We evaluate three state-of-the-art embedding models (Isaacus' Kanon 2 Embedder, Google's Gemini Embedding 001, and OpenAI's Text Embedding 3 Large) and two frontier LLMs (Gemini 3.1 Pro and GPT-5.2), finding that information retrieval is the primary driver of legal RAG performance, with LLMs exerting a more moderate effect on correctness and groundedness. Kanon 2 Embedder, in particular, had the largest positive impact on performance, improving average correctness by 17.5 points, groundedness by 4.5 points, and retrieval accuracy by 34 points. We observe that many errors attributed to hallucinations in legal RAG systems are in fact triggered by retrieval failures, concluding that retrieval sets the ceiling for the performance of many modern legal RAG systems. We document why and how we built Legal RAG Bench alongside the results of our evaluations. We also openly release our code and data to assist with reproduction of our findings.

Abdur-Rahman Butler, Umar Butler

Isaacus

Retrieval-Augmented Generation RAG Evaluation Legal QA IR Evaluation

Ask PDF

Mar 13, 2026

HR-Agents: Using Multiple LLM-based Agents to Improve Q&A about Brazilian Labor Legislation

The Consolidation of Labor Laws (CLT) serves as the primary legal framework governing labor relations in Brazil, ensuring essential protections for workers. However, its complexity creates challenges for Human Resources (HR) professionals in navigating regulations and ensuring compliance. Traditional methods for addressing labor law inquiries often lead to inefficiencies, delays, and inconsistencies. To enhance the accuracy and efficiency of legal question-answering (Q&A), a multi-agent system powered by Large Language Models (LLMs) is introduced. This approach employs specialized agents to address distinct aspects of employment law while integrating Retrieval-Augmented Generation (RAG) to enhance contextual relevance. Implemented using CrewAI, the system enables cooperative agent interactions, ensuring response validation and reducing misinformation. The effectiveness of this framework is evaluated through a comparison with a baseline RAG pipeline utilizing a single LLM, using automated metrics such as BLEU, LLM-as-judge evaluations, and expert human assessments. Results indicate that the multi-agent approach improves response coherence and correctness, providing a more reliable and efficient solution for HR professionals. This study contributes to AI-driven legal assistance by demonstrating the potential of multi-agent LLM architectures in improving labor law compliance and streamlining HR operations.

Abriel K. Moraes, Gabriel S. M. Dias, Vitor L. Fabris +14

Eldorado Institute of Research – Av. Alan Turing, 275, Cidade Universitária, 13.083-898 – Campinas – SP – Brazil

Legal QA Multi-Agent LLM Systems Legal NLP

Ask PDF

Abstract

Legal QA

Explore similar work

Legal-R1: Agentic Retrieval for Evidence-Based Legal Reasoning

Legal RAG Bench: an end-to-end benchmark for legal RAG

HR-Agents: Using Multiple LLM-based Agents to Improve Q&A about Brazilian Labor Legislation