cs.CLApr 12, 2026
SaveFrom Query to Counsel: Structured Reasoning with a Multi-Agent Framework and Dataset for Legal Consultation
Organizations: Australian Artificial Intelligence Institute (AAII), University of Technology Sydney · School of Computer Science, University of Birmingham
Abstract
Legal consultation question answering (Legal CQA) presents unique challenges compared to traditional legal QA tasks, including the scarcity of high-quality training data, complex task composition, and strong contextual dependencies. To address these, we construct JurisCQAD, a large-scale dataset of over 43,000 real-world Chinese legal queries annotated with expert-validated positive and negative responses, and design a structured task decomposition that converts each query into a legal element graph integrating entities, events, intents, and legal issues. We further propose JurisMA, a modular multi-agent framework supporting dynamic routing, statutory grounding, and stylistic optimization. Combined with the element graph, the framework enables strong context-aware reasoning, effectively capturing dependencies across legal facts, norms, and procedural logic. Trained on JurisCQAD and evaluated on a refined LawBench, our system significantly outperforms both general-purpose and legal-domain LLMs across multiple lexical and semantic metrics, demonstrating the benefits of interpretable decomposition and modular collaboration in Legal CQA.
Figures & tables
Figure 1: An illustrative example of legal consultation task decomposition, highlighting key challenges, limitations of prior approaches
Figure 2: Overview of JurisMA , a multi-agent framework that parses legal queries into element graphs, refines drafts via agent collaboration, and outputs a final legal opinion with supporting statutes.
| Dataset | QA Pairs | QLength | ALength (Pos./Neg.) |
|---|---|---|---|
| # Train | 39,163 | 15.11 | 263.5 / 193.7 |
| # Val | 2,176 | 15.57 | 285.7 / 211.3 |
| # Test | 2,176 | 15.22 | 270.8 / 197.5 |
Table 1: Statistics of JurisCQAD .
| Models | Rouge (%) | Bleu (%) | BertScore (%) | Bleurt (%) | LLM Score | ||||
|---|---|---|---|---|---|---|---|---|---|
| Rouge-1 | Rouge-2 | Rouge-L | Bleu-1 | Bleu-2 | Bleu-N | ||||
| General LLM | |||||||||
| GPT4o | 40.24 | 15.27 | 24.50 | 34.94 | 12.34 | 8.89 | 73.22 | 55.16 | 3.36 |
| Qwen3-14B | 42.55 | 19.24 | 27.27 | 21.93 | 10.25 | 8.10 | 74.64 | 62.38 | 3.43 |
| Qwen2.5-14B | 40.96 | 16.10 | 23.70 | 34.24 | 13.52 | 9.90 | 73.15 | 56.22 | – |
| Legal LLM | |||||||||
Table 2: Main results on the revised LawBench. “ ” indicates statistically significant improvement over all baselines under a paired t-test with . Bold numbers denote the best performance. Underlined numbers indicate the second-best results.
| Models | Legal Soundness | Reasoning | Completeness |
|---|---|---|---|
| GPT4o | 3.54 | 3.38 | 2.88 |
| Lawyer-LLaMA | 2.72 | 2.64 | 2.86 |
| JurisMA | 4.14 | 3.58 | 3.66 |
Table 3: Human evaluation results on three key criteria.
Figure 3: Rouge-L and BertScore comparison before and after DPO across Qwen2.5 models (3B/7B/14B).
Figure 4: Case study comparing model-generated responses to a time-sensitive legal query. The example illustrates differences in factual interpretation, statutory grounding, and reasoning structure. To aid analysis, we use the following inline annotations: bold text for key legal focus points, underlined text for the conclusion of the response, light red highlighting for cited statutes, and light green highlighting for underlying legal principles.
| Model | Rouge-L | Change | BertScore | Change |
|---|---|---|---|---|
| Qwen2.5-3B (Base) | 21.72 | – | 70.92 | – |
| + JurisCQAD | 26.19 | +4.47 | 73.43 | +2.51 |
| + LawGPT | 20.50 | –1.22 | 68.90 | –2.02 |
Table 4: Performance comparison of Qwen2.5-3B fine-tuned on different training corpora.
| Models | Rouge-L (%) | Bleu-N (%) | BertScore (%) |
|---|---|---|---|
| w/o KG | |||
| Qwen2.5:7b | 21.33 (-6.68) | 7.72 (-3.6) | 71.23 (-2.59) |
| Qwen2.5:14b | 21.57 (-9.57) | 7.21 (-7.04) | 71.74 (-3.31) |
| w/o Manager | |||
| Qwen2.5:7b | 21.41 (-6.6) | 7.81 (-3.51) | 71.62 (-2.2) |
| Qwen2.5:14b | 21.30 (-9.84) | 6.89 (-7.36) | 71.68 (-3.37) |
Table 5: Ablation study showing performance drops when removing key modules. Results confirm the importance of structured graph input, dynamic task routing, and iterative refinement.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Property | Description |
|---|---|
| Source | Real-world legal consultation platforms |
| Language | Chinese |
| Size | 43,126 triplets |
| Data Format | (query, positive answer, negative answer) |
| Annotation Method | LLM-assisted generation + expert verification |
| Negative Sample Strategy | LLM-generated distractors with legal/semantic flaws |
Table 6: Summary of JurisCQAD Dataset
| Comparison Dimension | Legal QA | Legal CQA |
|---|---|---|
| Task Goal | Answer exam questions or legal provisions | Respond to real-world legal concerns from users |
| Task Type | Mostly multiple-choice or extraction tasks | Requires generation of context-relevant legal suggestions |
| Data Source | Legal exams, statutory texts | Legal forums and Q&A communities |
| Data Characteristics | Standardized answers, concise questions | Long, complex questions with diverse factual scenarios |
| Question Structure | Short, standardized text | Long, unstructured, and often informal expressions |
| Legal Context | Involves a single legal provision | Involves multiple statutes and factual elements |
Table 7: Comparison between Legal QA and Legal Consultation QA (Legal CQA)
| Agent | Prompt Summary |
|---|---|
| Negative Example Generation Agent | Please generate a plausible but flawed response to the following legal consultation question. The response should appear reasonable on the surface but contain either incorrect legal references, flawed logical reasoning, or lack critical statutory support. Requirements: • The response must be grammatically fluent and professionally styled; • No social biases are allowed (e.g., based on gender, age, ethnicity, or region); • No fabricated or non-existent statute numbers are permitted; • The flaw should be mild—such as subtle legal or factual inaccuracies—not absurd or completely irrelevant; Instruction: Answer the following question. First, provide the flawed response, then specify the legal basis. |
| Draft Agent | You are a legal consultation draft generation agent. When a user submits a legal-related question, your task is to generate a professional response based on existing legal knowledge. |
| Element Agent | You are a professional legal element extraction expert. Your tasks include: • Extract key case elements from the user’s legal consultation; • Identify legal relationships and entities; • Clarify the user’s legal demands; • Output a structured element graph in JSON format. Element Graph Format: { "entities": [{ "name": "...", "type": "...", "attributes": {...} }], "events": [{ "description": "...", "time": "..." }], "relationships": [{ "type": "...", "source": ..., "target": ... }], "user_claims": ["..."], "key_facts": ["..."], "legal_questions": ["..."] } |
| Manager Agent | You are the decision-making agent in a multi-agent legal consultation system. Your task is to determine, based on the content of a draft legal response, whether it requires format refinement or legal citation supplementation. Decision Criteria: • If the response is not concise, lacks clear logic, or contains redundancy: Call: FormatCheckAgent ; • If the response lacks statutory references: Call: LawSearchAgent ; • If both issues apply: Call: FormatCheckAgent then LawSearchAgent ; • If the response is acceptable: Pass |
| FormatCheck Agent | Review the draft for clarity, redundancy, and stylistic issues. Output concrete editing suggestions without changing the legal meaning. |
| LawSearch Agent | Retrieve authoritative legal provisions from Chinese law based on the question and draft response. Output only relevant statute texts. |
Table 8: Prompts used by different agents.
| Section | Content |
|---|---|
| Entities | • User (Person) : Drunk driving record in 2011; the user themself committed the act. • Child (Person) : Child of the user; intends to apply for an aviation school. • Drunk Driving (Illegal Act) : Occurred in 2011; not criminalized at the time (before May 1, 2011). • Aviation School (Institution) : Has specific eligibility requirements for applicants. |
| Event | • Description : The user committed drunk driving in 2011. • Time : 2011 • Legal Context : Drunk driving was not yet criminalized before May 1, 2011. • Change Effect : Criminalization started after May 1, 2011, but the user’s act occurred earlier. |
| Relationships | • Kinship : User → Child • Application Target : Child → Aviation School • Legal Involvement : User → Drunk Driving |
| User Claims | • What should I do? • Is it illegal? • Can my child apply to an aviation school? |
| Key Facts | • The user committed drunk driving in 2011. • Drunk driving was not criminalized before May 1, 2011. • The user’s child intends to apply for an aviation school. • Aviation schools have specific background requirements. |
| Legal Questions | • Will the drunk driving record affect the child’s application to aviation school? • Was drunk driving a criminal offense in 2011? • What are the background screening standards for aviation school applicants? |
Table 9: Example of Element Graph
| Case ID | Revision Reason | Key Correction |
|---|---|---|
| 1 | Original answer failed to distinguish pre-/post-May 1, 2011 legal status of drunk driving and omitted aviation-specific background check regulations. | Added analysis of non-criminal administrative penalty and cited Article 133-1 of the Criminal Law and aviation review guidelines. |
| 2 | Original answer discussed unrelated payment default topic and lacked any applicable law to the real estate recovery dispute. | Rewritten answer clarified legal ownership transfer, invoked Civil Code articles on registration, good faith acquisition, inheritance, and statute of limitations. |
| 3 | Original answer incorrectly stated that all owners must sign service contracts. It misunderstood the legal effect of contracts signed by the owners’ committee and confused public and private contracting rules. | Clarified that a legally signed service contract by the owners’ committee is binding for all owners under Civil Code Article 939 and Property Management Regulations . |
| 4 | Original answer cited outdated or inaccurate medical insurance provisions and failed to reflect local retirement policies. | Updated answer clarified retirement exemption from further payments, citing Social Insurance Law and regional cumulative contribution rules. |
| 5 | Original answer misunderstood liability in sublease and construction. Misapplied contract law and omitted tenant’s liability for subtenants’ actions. | Added correct explanation using Articles 714, 716, 577 of Civil Code , showing tenant’s liability for third-party damages and breach of duty to maintain the property. |
| 6 | Original answer failed to cite core law on execution exemption and missed user’s intent to reserve minimum livelihood funds. | Correction referenced Civil Procedure Law Article 243 and Supreme Court regulations on exempt property and basic living standards. |
Table 10: Representative Examples of Test Set Corrections
| Models | Rouge (%) | Bleu (%) | BertScore (%) | ||||
|---|---|---|---|---|---|---|---|
| Rouge-1 | Rouge-2 | Rouge-L | Bleu-1 | Bleu-2 | Bleu-N | ||
| General LLM | |||||||
| GPT4o | 10.10 | 1.40 | 8.11 | 4.66 | 0.49 | 0.21 | 57.88 |
| Legal LLM | |||||||
| Lawyer-llama | 8.42 | 0.07 | 7.70 | 3.70 | 0.03 | 0.97 | 57.13 |
| Our Method | |||||||
Table 13: Generalization results on the LegalBench -RuleQA subset (50 samples, averaged over 5 seeds). “†” indicates statistically significant improvement over all baselines under a paired t-test with . Bold numbers denote the best performance. Underlined numbers indicate the second-best results.