LexPilot: Benchmarking the Proactive Interaction Ability of Legal LLMs with Laypeople in Chinese Civil Case Consultation

Wenbin Zhang; ShangZhan Li; Bingxiang He; Ye He; Yuxin Zhou; Muzhen Cai; Qi Shi; Wanxiang Che

cs.AIOct 16, 2026

LexPilot: Benchmarking the Proactive Interaction Ability of Legal LLMs with Laypeople in Chinese Civil Case Consultation

Authors: Wenbin Zhang, ShangZhan Li, Bingxiang He, Ye He, Yuxin Zhou, Muzhen Cai, Qi Shi, Wanxiang Che

Organizations: Harbin Institute of Technology · Tsinghua University

Abstract

Large Language Models (LLMs) have greatly assisted legal professionals. However, our empirical studies reveal that, when faced with underspecified consultations from non-experts, current LLMs often produce irresponsible responses with limited utility despite lacking sufficient case details for definitive conclusions. To better assist laypeople, legal LLMs must identify focal points of dispute and proactively elicit crucial context from users. Unfortunately, existing data resources often focus on interaction and response quality, offering limited insight into how proactive interaction affects LLMs' downstream claim-level judgment prediction in actual civil cases. To address this gap, we introduce LexPilot, a benchmark linking proactive legal interaction to claim-level verdict prediction using authentic Chinese civil judgments. Constructed from 432 real-world judgments across 36 common causes of action, LexPilot comprises expert-reviewed pairs of complete and underspecified consultation histories with corresponding court verdicts. Experiments reveal that current LLMs struggle with proactive legal interactions in underspecified consultations, with downstream judgment prediction substantially underperforming that in the fully specified consultation setup (-23.84% on average). Further analysis identifies critical bottlenecks hindering these capabilities. Our work highlights the urgent need for more user-centered legal AI to better serve the public.

Figures & tables

Figure 1: LLMs deliver more responsible judgment predictions via proactive interaction than direct responses.

Figure 2: The construction of LexPilot involves judgment reorganization ( Left ); key information extraction alongside seed data collection ( Mid ); and consultation generation followed by anonymization and expert validation ( Right ).

Category	Type	Value
Consultation Counts by Type	Total	864
	Underspecified	432
	Complete	432
Dataset Profile	Avg. Length (chars)	398.75
	Avg. Turns	5.82
	Causes of Action	36

Table 1: The statistics of the LexPilot benchmark.

Category	Model	Family	Real Rights	Torts	Service Contract	Finance Contract	Misc.	Overall
Legal Completeness Assessment (LCA) Task – Macro-F1
Legal LLMs	LawLLM-7B	0.4240	0.3987	0.4116	0.4081	0.4125	0.4653	0.4186
Legal LLMs	LegalOne-8B	0.7136	0.6979	0.7110	0.6332	0.6674	0.6455	0.6768
Open-Source General LLMs	GLM-5.3	0.6621	0.6114	0.6610	0.6522	0.6421	0.6933	0.6496
	DeepSeek-V4-Pro	0.6667	0.6471	0.6555	0.5698	0.5946	0.5616	0.6149
	Kimi-K3	0.6560	0.6636	0.7091	0.5185	0.6265	0.6935	0.6395

Table 2: Evaluation results of the Legal Completeness Assessment ( Macro-F1 ) and Detail Completion Guidance ( Accuracy ) tasks across 10 leading legal and general LLMs.

Category	Model	Direct	Interact- Based	Gold- Stan.
Legal	LawLLM-7B	0.161	0.153	0.218
Legal	LegalOne-8B	0.337	0.366	0.492
Open- Source	GLM-5.3	0.382	0.436	0.553
	DeepSeek-V4-Pro	0.350	0.398	0.504
	Kimi-K3	0.321	0.369	0.484
	Qwen3.8-Max	0.348	0.386	0.469

Table 3: Evaluation results ( QWK Score) of Legal Judgment Prediction task under three setups across 10 LLMs.

Figure 3: Impact of Case Complexity on LCA and DCG tasks. Each point represents a specific Cause of Action.

Figure 4: Average DCG accuracy of the three closed-source LLMs categorized by focal point types (a) and cause of action classes (b) , accompanied by the distribution of focal point types across different cause of action classes (c) .

Model	Direct	Targeted	Off-target
Claude-Opus-5	0.431	0.557	0.386
GPT-5.6-Sol	0.428	0.532	0.373
Gemini-3.8-Flash	0.357	0.488	0.303

Table 4: LJP results ( QWK ) under different conditions.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

	Laypeople	Law majors
Queries w/o. key detail	79	24
Queries w. Single key detail	97	59
Queries w. Two key details	224	317
Key detail Coverage (%)	68.1%	86.6%

Appendix

Table 5: Key detail Coverage in civil case consultations across different groups, which is computed as the ratio of key details included by each group to total key details.

	GPT-5.4	GLM-5	Kimi-K2.5
Deficiency Recognition	7.3%	5.8%	4.2%
Cond. Generic Advice	31.7%	34.7%	27.4%
Uncond. Generic Advice	19.7%	25.1%	32.0%
Premature Prediction	41.3%	34.4%	36.3%

Appendix

Table 6: Distribution of response types to underspecified legal consultations. The results highlight a tendency across models to frequently provide unconditional generic advice or prematurely predict the verdicts.

Consultation & Missing Details	Model Response
Response Type: Deficiency Recognition
Consultation: Hello, I need legal advice regarding property rights. My ex-wife and I divorced in 2016, but our shared home was never divided. The house was purchased by me, my ex-wife, and her sister, but the deed only lists the sisters’ names. However, my family made significant financial contributions: my father helped with the deposit and down payment, and I paid the deed tax, other fees, and renovation costs. We also paid the mortgage together and lived there as a family. I believe this should be considered joint marital property, and I am entitled to half, but my ex-wife disagrees. Can I legally claim half of the property rights? Thank you! Missing details surrounding the focal point: Since the plaintiff did not specify who paid for the disputed property, it is impossible to determine the share of ownership they are entitled to.	Response by GPT-5.4: I understand this property dispute is a stressful situation for you, but please note that the information provided in your consultation is currently insufficient to give you an accurate legal assessment regarding your claim to the property rights. Because you did not clearly specify the exact financial details and proportions of who provided the funds for the property purchase, it is impossible to determine the legal distribution of the assets at this stage… Therefore, please supplement your consultation with detailed information regarding the exact source of all purchase funds, including the specific amounts contributed by you, your family, your ex-wife, and her sister for the down payment, taxes, and mortgage payments.
Response Type: Premature Prediction
Consultation: Last year, after Chen’s sheep ate my crops, I took one of her sheep in retaliation. The police mediated the dispute, and we signed a document that I thought settled the matter. However, days later, Chen took 500 bundles of my cornstalks from my field and refuses to return them, claiming they are hers. I was going to sell them for 9 RMB a bundle, totaling a 4,500 RMB loss. Can I sue her for this 4,500 RMB compensation? Thank you! Missing details surrounding the focal point: What document was signed between the plaintiff and the defendant? Did it stipulate that the cornstalks in the field were given to the defendant?	Response by Kimi-K2.5: "It is incredibly frustrating to have your bundled cornstalks taken without permission, and your anger and demand for compensation are completely justified. To answer your questions directly: Yes, you can absolutely sue her for the 4,500 RMB loss. If you win the case, she will also have to cover the court fees. …… Next steps: take the cooperative’s purchase proof, your ID card, and the land lease contract directly to your local court’s service window to file a case. Best of luck recovering your losses quickly!"

Appendix

Table 7: Examples of model responses attempting a direct conclusion ( Deficiency Recognition and Premature Prediction ) when confronted with underspecified consultations.

Consultation & Missing Details	Model Response
Response Type: Conditional Generic Advice
Consultation: I worked for a boss named Shi from October to December 2013, and he still owes me 4,380 RMB in unpaid wages, along with money owed to about a dozen other workers. I have timesheets proving my workdays (12 days in Oct, 17.5 in Nov, and 7 in Dec). Shi initially acknowledged the debt but now makes excuses, claiming we were hired by a foreman named Wang X. However, it is clear to all of us workers that Shi hired us and should pay us. We did sign some sort of agreement back then, but I don’t recall the details as we just signed what we were told to. Can I still recover my wages, and who exactly should I hold responsible? Missing details surrounding the focal point: It cannot be definitively determined from the user’s consultation whether the employer is Shi or Wang.	Response by Gemini-3.1-Pro-Preview: To successfully recover your unpaid wages, you must preserve your work records, written agreements, and evidence of continuous debt collection over the past decade to prevent the statute of limitations from expiring. Depending on who is legally identified as the direct employer, there are two possible scenarios: Scenario 1: Boss Shi is the direct employer. If evidence (e.g., work records signed by Shi, agreements that do not transfer the debt) proves Shi directly hired and managed you, he is solely responsible. A court would order Shi directly to pay the owed wages. Scenario 2: Foreman Wang A is the direct employer. If Shi subcontracted the work to Wang A, who then hired you, Wang A holds the primary obligation to pay your wages. However, if Wang A lacks the legal qualifications to act as an employer, Shi (the contractor) cannot escape liability and must legally bear joint and several liability or advance payment responsibility. In this case, it is strongly recommended to list both men as co-defendants when filing a lawsuit, demanding payment from Wang A and joint liability from Shi, to ensure you are paid.
Response Type: Unconditional Generic Advice
Consultation: Last July, I signed up for a two-day tour group to Zhangbei Grassland, paying 278 RMB per person. However, the day after returning, I started experiencing a fever and diarrhea. I couldn’t bear it anymore, so I went to Hospital A. The doctor diagnosed it as bacillary dysentery and prescribed over 300 RMB worth of medication. It took me half a month to recover from this illness, which even delayed me from handling my spouse’s retirement procedures. Should the travel agency refund my tour fee? Also, can I get them to compensate for the medical expenses and the losses incurred during this time? Missing details surrounding the focal point: Whether the resulting harm of the plaintiff being diagnosed with bacillary dysentery following diarrhea was caused by the consumption of unsanitary meals arranged by the defendant during the tour.	Response by MiniMax-M2.5: Legally, claiming a refund, medical expenses, and indirect losses depends on whether the agency fulfilled the contract. Since the trip was completed, refunds are difficult; for damages, you will need strong evidence, such as medical records or proof that other group members also fell ill. …… Given the time passed, administrative complaints may be time-barred. Please preserve all your evidence (receipts, contract) and try negotiating a settlement with the travel agency first. If friendly mediation fails, you can then assess whether your evidence is strong enough to pursue a civil lawsuit.

Appendix

Table 8: Examples of model responses withholding a definitive conclusion and providing generic advice ( Conditional Generic Advice and Unconditional Generic Advice ) when confronted with underspecified consultations.

Province / Region	Count	Province / Region	Count
Guangdong	48	Hebei	11
Hunan	35	Tianjin	11
Shandong	28	Liaoning	10
Sichuan	25	Jilin	10
Hubei	24	Jiangxi	10
Shaanxi	21	Gansu	10

Appendix

Table 9: The geographical distribution of legal judgment documents used to construct LexPilot, organized by province.

Cause of Action (Chinese)	Cause of Action (English Translation)
gkai婚姻家庭、继承纠纷	Marriage, Family, and Inheritance Disputes ( abbr . Family )
离婚纠纷	Divorce Dispute
离婚后财产纠纷	Property Dispute after Divorce
婚约财产纠纷	Property Dispute involving Engagement
抚养纠纷	Child Support/Maintenance Dispute
法定继承纠纷	Statutory Succession Dispute

Appendix

Table 10: The 36 causes of action of civil judgments used to construct LexPilot, grouped by 6 major classes.

Category	Model	LCA ( Macro-F1 )	DCG ( Acc. )	LJP ( QWK )
Category	Model	LCA ( Macro-F1 )	DCG ( Acc. )	Direct	Interaction-Based	Gold-Standard
Legal LLMs	LawLLM-7B	$0.419\pm 0.015$	$0.392\pm 0.012$	$0.161\pm 0.018$	$0.153\pm 0.013$	$0.218\pm 0.031$
Legal LLMs	LegalOne-8B	$0.677\pm 0.018$	$0.617\pm 0.026$	$0.337\pm 0.014$	$0.366\pm 0.019$	$0.492\pm 0.010$
Open-Source General LLMs	GLM-5.3	$0.650\pm 0.032$	$0.692\pm 0.002$	$0.382\pm 0.029$	$0.436\pm 0.025$	$0.553\pm 0.034$
	DeepSeek-V4-Pro	$0.615\pm 0.020$	$0.664\pm 0.030$	$0.350\pm 0.037$	$0.398\pm 0.045$	$0.504\pm 0.009$
	Kimi-K3	$0.640\pm 0.011$	$0.662\pm 0.013$	$0.321\pm 0.021$	$0.369\pm 0.024$	$0.484\pm 0.016$

Appendix

Table 11: Overall performance of the 10 evaluated models on three tasks. Entries report the Mean

\pm

Standard Deviation across five repeated runs under identical experimental settings. LJP results are reported separately for the Direct, Interaction-Based, and Gold-Standard prediction setups.

Evaluation Task	Human-Human		LLM-Human
Evaluation Task	Agree.	$\kappa$	Agree.	$\kappa$
LCA	96.0%	0.92	93.0%	0.86
DCG	92.0%	0.84	89.0%	0.78
LJP	98.0%	0.96	96.0%	0.92
Average	95.3%	0.91	92.7%	0.85

Appendix

Table 12: Human Inter-Annotator Agreement (IAA) and LLM-to-Human alignment across the three evaluation tasks, based on 100 sampled instances per task.

Condition	Answer	OD	Error	DCG+
A: Direct	94.4%	2.8%	1.4%	95.8%
B: Indirect	91.7%	4.2%	2.8%	93.1%
C: Broad	—	5.6%	1.4%	5.6%
D: Withheld	93.1%	2.8%	4.2%	—

Appendix

Table 13: User simulator fidelity and DCG judge acceptance across probe conditions. OD: over-disclosure beyond the permitted facts; Error: unsupported assertions, contradictions, or entity confusion; DCG+: judge acceptance rate. OD and Error are non-exclusive.

Model	Macro-F1	$\mathrm{Acc}_{C}$	$\mathrm{Acc}_{U}$
LawLLM-7B	0.4186	0.5278	0.3218
LegalOne-8B	0.6768	0.5903	0.7685
GLM-5.3	0.6496	0.5301	0.7801
DeepSeek-V4-Pro	0.6149	0.5278	0.7083
Kimi-K3	0.6395	0.5532	0.7315
Qwen3.8-Max	0.5960	0.4815	0.7222

Appendix

Table 14: LCA metric breakdown for the evaluated models. Macro-F1 equally weights the two class-specific F1 scores;

\mathrm{Acc}_{C}

and

\mathrm{Acc}_{U}

denote accuracy within complete and underspecified consultations, respectively. Errors on these two consultation types correspond to over-asking and premature legal judgments , respectively.

Model	Standard	Delayed Disclosure	Self- Correction
Claude-Opus-5	0.5183	0.4775	0.4504
GPT-5.6-Sol	0.4788	0.4469	0.4349
Gemini-3.8-Flash	0.4278	0.4020	0.3881

Appendix

Table 15: Interaction-Based LJP performance ( QWK ) under non-ideal user settings compared with the Standard setup, averaged over five runs per condition. Standard scores reuse the main-experiment runs.

Category	Model	Family	Real Rights	Torts	Service Contract	Finance Contract	Misc.	Overall
Legal LLMs	LawLLM-7B	0.3581	0.0416	0.1456	0.3936	0.1082	0.2407	0.2180
Legal LLMs	LegalOne-8B	0.7386	0.4734	0.2223	0.4191	0.4136	0.6250	0.4915
Open-Source General LLMs	GLM-5.3	0.6172	0.5202	0.5975	0.4596	0.3652	0.7291	0.5526
	DeepSeek-V4-Pro	0.6114	0.5585	0.2925	0.4661	0.3841	0.5972	0.5040
	Kimi-K3	0.5861	0.5715	0.3828	0.4977	0.2114	0.5503	0.4837
	Qwen3.8-Max	0.6410	0.4462	0.4586	0.4044	0.2947	0.5128	0.4690

Appendix

Table 16: Evaluation results ( QWK Score) of Legal Judgment Prediction task under the Gold-Standard Prediction setup across various cause of action classes. All reported results are averaged over five repeated runs.

Category	Model	Family	Real Rights	Torts	Service Contract	Finance Contract	Misc.	Overall
Legal LLMs	LawLLM-7B	0.0545	0.1718	0.1527	0.1606	0.0879	0.2602	0.1530
Legal LLMs	LegalOne-8B	0.4492	0.3409	0.3322	0.4207	0.1406	0.4502	0.3655
Open-Source General LLMs	GLM-5.3	0.5398	0.5159	0.5267	0.3541	0.1153	0.6340	0.4356
	DeepSeek-V4-Pro	0.4827	0.3513	0.4412	0.3970	0.1095	0.6001	0.3980
	Kimi-K3	0.5488	0.2532	0.3881	0.4160	0.1387	0.5016	0.3688
	Qwen3.8-Max	0.4859	0.4575	0.4570	0.3310	0.1862	0.4068	0.3860

Appendix

Table 17: Evaluation results ( QWK Score) of Legal Judgment Prediction task under the Interaction-Based Prediction setup across various cause of action classes. All reported results are averaged over five repeated runs.

Category	Explanation	Examples
Subject Identity (gkai主体身份类)	Examines whether the parties possess specific legal standing, status relationships, or appropriate qualifications (e.g., confirming labor/employment relationships, determining membership in collective economic organizations, or identifying contract parties).	$\bullet$ Whether the plaintiff qualifies as a member of the defendant’s collective economic organization. $\bullet$ Whether a labor relationship exists between the plaintiff and the defendant. $\bullet$ Identification of the lessee in the disputed “Tower Crane Lease Contract.”
Rights Allocation (gkai权属界定类)	Confirms the ultimate ownership and attribution status of substantive rights such as property rights, creditor’s rights, intellectual property, or inherited assets (e.g., house ownership, scope of inheritance, land contracting rights).	$\bullet$ Does the house belong to He XX, is it the joint property of He XX and Lai XX, or is it family property jointly owned by family members? $\bullet$ Whether the disputed house located at No. XX North, XX Village constitutes the inheritance of Chen X and Guo X. $\bullet$ The ownership of the house in the courtyard at No.XX North Street, a village in XX District.
Contract Validity (gkai契约效力类)	Evaluates the legal establishment, effectiveness, invalidity, or revocability of legal actions or documents, such as contracts, agreements, and wills.	$\bullet$ The validity of the provision in the divorce agreement requiring the man to pay the woman 400,000 RMB. $\bullet$ Whether the “Land Transfer Agreement” signed by both parties is valid. $\bullet$ The validity of the will made by Zhao XX.
Factual Conduct (gkai客观行为类)	Investigates whether the parties have actually performed specific objective actions or fulfilled obligations (e.g., whether specific payments were made, delivery obligations were fulfilled, or property was actually returned).	$\bullet$ Whether the plaintiff has fulfilled the signboard renovation obligation for “XX Bakery” as stipulated in the agreement. $\bullet$ Whether the 240,000 RMB paid by Chen to Zhu includes the 200,000 RMB living expenses stipulated in the divorce agreement. $\bullet$ Whether the defendant has returned 50,000 RMB in cash to the plaintiff.
Amount Assessment (gkai数额界定类)	Objectively calculates, verifies, and confirms specific amounts, quantities, proportions, and values.	$\bullet$ The principal amount of the loan claimed by the plaintiff. $\bullet$ Determination of the principal and interest of the loan owed by the defendant to the plaintiff. $\bullet$ The value of the plaintiff’s tree loss.
Liability Allocation (gkai归责逻辑类)	Determines the causal relationship between the resulting damage and the actions taken, the degree of fault, liability for breach of contract, and the legal subject responsible for bearing the liability (who compensates and why).	$\bullet$ Whether the vehicle malfunction on April 5, 2018, was caused by inadequate oil replacement during the defendant’s maintenance or by other reasons. $\bullet$ Should Wang XX bear compensation liability for Zhang’s death? $\bullet$ Who is at fault for the fire accident.

Appendix

Table 18: The seven categories of focal points of dispute in civil case consultations.

Original Chinese Prompt for LexAwN

你是一个专业的AI法律顾问。你的任务是评估原告提交的民事案件法律咨询，并预测法院是否会支持其诉讼请求。用户的案件陈述可能是清晰、完备的，也可能缺失关键的法律点。在决定如何回复之前，请你首先对照该类纠纷的构成要件，对<对话历史>进行评估。如果当前事实能够形成法律推演链条，则直接预测判决结果；如果存在法定要件缺失，导致法律推演难以进行，则按需提问。请根据评估结果，严格选择并执行以下两种策略之一： gkai【策略一】：事实脉络大体清晰，则直接预测如果<对话历史>中的事实已经能够形成法律推演链条，具备了裁决基础，直接执行： 1. 法律分析：结合案件事实与适用的法律规则进行专业分析，指出哪些事实支持或削弱了原告的诉请。 2. 预测结论：对于用户的每一条诉讼请求，分别给出明确的判决预测结果（必须明确分类为：支持、部分支持，被驳回其中之一）。 gkai【策略二】：存在法律要件缺失导致难以推演，则按需提问 (Ask-when-Needed) 如果<对话历史>中存在法定要件缺失，导致难以进行合理的法律推演时，执行： 1. 指出缺失：明确告知用户我们需要核实哪个案件关键点。（只能核实【一个】关键点，严禁一次核实多个） 2. 解释原因：简明扼要地说明为什么该关键点对法院的最终判决至关重要。 3. 提出问题：提出一个清晰、具体的单点问题，用于引导用户补充相关事实。 # Input Data <对话历史>{history} </对话历史> # Output Constraints 仅输出回复内容，不要包含任何标题、前言、后语、Markdown标记、标签、元数据或自我解释。

English Translation

You are a professional AI legal advisor. Your task is to evaluate civil case legal consultations submitted by plaintiffs and predict whether the court will support their litigation claims. The user’s case statement may be clear and complete, or it may lack crucial legal points. Before deciding how to respond, please first evaluate the <Dialogue History> against the constitutive requirements of such disputes. If the current facts can form a chain of legal reasoning, directly predict the judgment result; if there is a lack of statutory requirements that makes legal reasoning difficult, ask questions as needed. Based on the evaluation results, strictly select and execute one of the following two strategies: [Strategy 1]: The factual context is generally clear, predict directly If the facts in the <Dialogue History> can already form a chain of legal reasoning and provide a basis for adjudication, directly execute: 1. Legal Analysis: Conduct a professional analysis combining the case facts and applicable legal rules, pointing out which facts support or weaken the plaintiff’s claims. 2. Prediction Conclusion: For each of the user’s litigation claims, provide a clear prediction of the judgment (must be explicitly categorized as one of the following: Supported, Partially Supported, or Dismissed). [Strategy 2]: Missing legal requirements make reasoning difficult, ask when needed (Ask-when-Needed) If the <Dialogue History> lacks statutory requirements, making it difficult to conduct reasonable legal reasoning, execute: 1. Point out the missing information: Explicitly inform the user which key case point needs to be verified. (You can only verify [ONE] key point; verifying multiple at once is strictly prohibited). 2. Explain the reason: Briefly explain why this key point is crucial to the court’s final judgment. 3. Ask a question: Pose a clear, specific, single-point question to guide the user in supplementing the relevant facts. # Input Data <Dialogue History>{history} </Dialogue History> # Output Constraints Output only the response content. Do not include any titles, prefaces, epilogues, Markdown tags, labels, metadata, or self-explanations.

Appendix

Table 19: The LexAwN prompt, presenting the original Chinese version which was actually used in our experiments, as well as its English translation.

Original Chinese Prompt for DCG Task

你是一个专业的AI法律顾问。你的任务是评估原告提交的民事案件法律咨询。用户的案件陈述中缺失了关键的法律点。请你首先对照该类纠纷的构成要件，对<对话历史>进行评估，找出存在法定要件缺失、导致法律推演难以进行的地方，并直接执行以下提问策略： 1. 指出缺失：明确告知用户我们需要核实哪个案件关键点。（注意：只能核实【一个】关键点，严禁一次核实多个） 2. 解释原因：简明扼要地说明为什么该关键点对法院的最终判决至关重要。 3. 提出问题：提出一个清晰、具体的单点问题，用于引导用户补充相关事实。 # Input Data <对话历史>{history} </对话历史> # Output Constraints 仅输出回复内容，不要包含任何标题、前言、后语、Markdown标记、标签、元数据或自我解释。

English Translation

You are a professional AI legal advisor. Your task is to evaluate civil case legal consultations submitted by plaintiffs. The user’s case statement lacks crucial legal points. Please first evaluate the <Dialogue History> against the constitutive requirements of such disputes, identify where the lack of statutory requirements makes legal reasoning difficult, and directly execute the following questioning strategy: 1. Point out the missing information: Explicitly inform the user which key case point needs to be verified. (Note: You can only verify [ONE] key point; verifying multiple at once is strictly prohibited). 2. Explain the reason: Briefly explain why this key point is crucial to the court’s final judgment. 3. Ask a question: Pose a clear, specific, single-point question to guide the user in supplementing the relevant facts. # Input Data <Dialogue History>{history} </Dialogue History> # Output Constraints Output only the response content. Do not include any titles, prefaces, epilogues, Markdown tags, labels, metadata, or self-explanations.

Appendix

Table 20: The modified version of the LexAwN prompt used in the Detail Completion Guidance task, presenting both the original Chinese version and its English translation.

Original Chinese LJP Prompt Shared by Direct and Gold-Standard

你是一个专业的AI法律顾问。你的任务是评估原告提交的民事案件法律咨询，并预测法院是否会支持其诉讼请求。请你对照该类纠纷的构成要件，根据<对话历史>中当前提供的事实进行法律分析，并直接预测各项诉讼请求的判决结果。本次不进行进一步问答，请勿向用户提出补充问题。请严格执行以下步骤： 1. 法律分析：结合案件事实与适用的法律规则进行专业分析，指出哪些事实支持或削弱了原告的诉请。 2. 预测结论：对于用户的每一条诉讼请求，分别给出明确的判决预测结果（必须明确分类为：支持、部分支持，被驳回其中之一）。 # Input Data <对话历史>{history} </对话历史> # Output Constraints 仅输出回复内容，不要包含任何标题、前言、后语、Markdown标记、标签、元数据或自我解释。

English Translation

You are a professional AI legal advisor. Your task is to evaluate civil case legal consultations submitted by plaintiffs and predict whether the court will support their litigation claims. Evaluate the case against the constitutive requirements of the dispute using the facts currently provided in the <Dialogue History>Ḋirectly predict the outcome of each claim without asking follow-up questions. Please strictly execute the following steps: 1. Legal Analysis: Conduct a professional analysis combining the case facts and applicable legal rules, pointing out which facts support or weaken the plaintiff’s claims. 2. Prediction Conclusion: For each of the user’s litigation claims, provide a clear prediction of the judgment (must be explicitly categorized as one of the following: Supported, Partially Supported, or Dismissed). # Input Data <Dialogue History>{history} </Dialogue History> # Output Constraints Output only the response content. Do not include any titles, prefaces, epilogues, Markdown tags, labels, metadata, or self-explanations.

Appendix

Table 21: The direct judgment prediction prompt shared by the Direct and Gold-Standard LJP setups, in Chinese and English. The input is an underspecified consultation for Direct setup and its complete counterpart for Gold-Standard setup.

Original Chinese Prompt for the Standard User Model

你现在需要扮演一名正在进行民事案件法律咨询的原告当事人。你的任务是根据<对话历史>中用户已提供的事实和<案件客观事实>，对AI法律顾问的最新追问做出自然、恰当的回复。 # Input Data <对话历史>{history} </对话历史> <案件客观事实> {key_facts} </案件客观事实> gkai# 逻辑与回复策略请根据最新追问的具体含义决定回复，不要把AI的假设或推测当作已确认事实。 1. 事实命中（有效且可答）： - 评估标准：问题是在核实具体事实，且用户已提供的事实或<案件客观事实>足以支持回答。日常表达、间接提问或证据线索均可，不要求与事实原文逐字匹配。 - 回复策略：准确、充分地回答所问事实，可补充理解回答所必需的解释；不要披露超出当前问题范围的其他隐藏细节。若只能回答部分内容，回答有依据的部分，并明确说明其余部分不清楚。绝不能编造未提供的细节。 2. 事实未命中（偏离或不可答）： - 评估标准：问题仅宽泛提及法律概念，没有核实具体事实；或者所提供的信息不足以回答。 - 回复策略：宽泛提问时不要主动披露隐藏事实，可请AI明确想了解的情况。信息不足时，自然地说明“不清楚”或“无法确认”；不要把未提及的事情说成没有发生，也不要仅因缺少信息而断言问题与案件无关。 # Output Constraints - 必须以原告（缺乏专业法律背景的咨询者）的第一人称口吻进行回复，语气和你在<对话历史>中的咨询一致，符合上下文情境。 - 仅输出你作为用户的回复文本，绝对不要包含任何标题、Markdown代码块标记、前缀标签（例如“原告：”、“回复：”）、元数据或自我解释。

English Translation

You are a plaintiff seeking legal advice about a civil case. Respond naturally and appropriately to the AI legal advisor’s latest follow-up question using the facts already provided by the user in <Dialogue History> and <Objective Case Facts>. # Input Data <Dialogue History>{history} </Dialogue History> <Objective Case Facts> {key_facts} </Objective Case Facts> # Reasoning and Response Strategy Respond according to the specific meaning of the latest question. Do not treat the AI’s assumptions or speculation as confirmed facts. 1. Factual match (relevant and answerable): - Criterion: The question seeks specific facts, and the facts already provided by the user or <Objective Case Facts> support an answer. Everyday wording, indirect questions, and evidence cues are acceptable; verbatim matching is unnecessary. - Response: Answer the requested facts accurately and sufficiently, including explanations necessary to understand the answer. Withhold other hidden details beyond the current question’s scope. If only part is answerable, give the supported information and explicitly state uncertainty about the rest. Never invent details that were not supplied. 2. No factual match (off-target or unanswerable): - Criterion: The question only mentions a broad legal concept without requesting specific facts, or the supplied information is insufficient to answer. - Response: For broad questions, do not volunteer hidden facts; you may ask the AI to specify what it wants to know. If information is insufficient, naturally say “I don’t know” or “I cannot confirm.” Do not treat an unmentioned event as having never happened or assert that a question is irrelevant merely because information is missing. # Output Constraints - Reply in the first person as the plaintiff, a layperson without professional legal training. Match the user’s tone in <Dialogue History> and the conversational context. - Output only the user’s reply. Do not include titles, Markdown code fences, prefix labels (e.g., “Plaintiff:” or “Reply:”), metadata, or self-explanations.

Appendix

Table 22: The Standard User Model prompt shared by DCG and Interaction-Based LJP, with the original Chinese text and its English translation.

Original Chinese Evaluation Prompt for LCA

你是一个严谨的文本分类器与指令遵循评估器。你的任务是阅读一段由“AI法律顾问”生成的回复，并判断该回复执行的是【策略一】（直接预测）还是【策略二】（主动追问）。仅根据待评估的新回复识别策略，不独立判断案件是否完备，也不评价判决预测是否正确。策略一对应模型将咨询视为完备，策略二对应模型将咨询视为信息不足；分类结果由评分程序与真实类别比较。 gkai【背景与策略定义】生成待评估回复的AI模型，其系统指令要求它严格在以下两种策略中选择其一执行： - 【策略一】（直接预测）：当用户的案件陈述法律要件基本齐备时触发。该策略的回复特征是： 1. 包含法律分析：结合案件事实与适用的法律规则进行分析。 2. 包含预测结论：会对用户的每一条诉讼请求，分别给出明确的预测结果（明确带有“支持”、“部分支持”或“被驳回”等字眼）。 - 【策略二】（按需提问/Ask-when-Needed）：当案件陈述明显缺失核心法律要件时触发。该策略的回复特征是： 1. 指出缺失：明确告知用户需要核实哪个单一的案件关键点。 2. 解释原因：说明为什么该关键点对法院的最终判决至关重要。 3. 提出问题：没有给出最终的判决预测，而是向用户提出一个具体的单点问题，引导其补充事实。 gkai【判断标准】 - 如果待评估回复执行的是【策略一】（即：给出了明确的判决预测结论），请输出：策略一 - 如果待评估回复执行的是【策略二】（即：指出了缺失要件并向用户提出了追问），请输出：策略二 - 【特殊情况处理】（混合特征）：如果待评估回复中既给出了明确的判决结果，又指出了缺失要件并向用户提出了追问，请根据回复的核心侧重点（主导地位）进行判定： - 若“明确的预测结论”占主导地位（例如：主体已经完成了明确的判决预测，追问仅作为附带的补充建议或针对非核心信息的询问），请输出：策略一 - 若“指出缺失并追问”占主导地位（例如：预测仅仅是基于假设的模糊推测，回复的主体目的是要求用户补充关键事实才能真正得出结论），请输出：策略二 gkai【输入数据】 <待评估回复> {history} </待评估回复> gkai【输出限制】严格只输出“策略一”或“策略二”三个字(不带引号)。绝对不要包含任何前言后语、解释说明、标点符号、Markdown标记或换行符。

English Translation

You are a rigorous text classifier and instruction-following evaluator. Read a reply generated by an AI legal advisor and identify whether it follows [Strategy 1] (direct prediction) or [Strategy 2] (proactive clarification). Classify only the new reply. Do not independently assess case completeness or prediction correctness. Strategy 1 means the model treats the consultation as complete; Strategy 2 means it treats it as underspecified. The scoring code compares these labels with the reference categories. [Background and Strategy Definitions] The model generating the reply was instructed to choose exactly one of these strategies: - [Strategy 1] (direct prediction): Triggered when the user’s case statement contains the essential legal elements. Its response features are: 1. Legal analysis combining case facts and applicable legal rules. 2. A clear prediction for each claim, explicitly using terms such as “Supported,” “Partially Supported,” or “Dismissed.” - [Strategy 2] (Ask-when-Needed): Triggered when core legal elements are clearly missing. Its response features are: 1. Identify the single key case point requiring clarification. 2. Explain why it is crucial to the court’s final judgment. 3. Ask a specific, single-point question to elicit facts instead of giving a final verdict prediction. [Classification Criteria] - If the reply follows [Strategy 1], i.e., gives a clear verdict prediction, output: Strategy 1 - If it follows [Strategy 2], i.e., identifies missing elements and asks for clarification, output: Strategy 2 - [Mixed features]: If the reply both gives a clear verdict prediction and identifies missing elements with a follow-up question, classify by its dominant focus: - If the clear prediction dominates (e.g., a definitive prediction is already given and clarification is merely supplementary advice or concerns non-essential information), output: Strategy 1 - If identifying missing information and seeking clarification dominates (e.g., the prediction is only vague, hypothetical speculation and the reply mainly seeks crucial facts before reaching a conclusion), output: Strategy 2 [Input Data] <Reply to Evaluate> {history} </Reply to Evaluate> [Output Constraints] Output only “Strategy 1” or “Strategy 2” without quotation marks. Do not include any preface, closing remarks, explanation, punctuation, Markdown markup, or line breaks.

Appendix

Table 23: The LCA evaluation prompt, with the original Chinese text and its English translation.

Original Chinese Evaluation Prompt for DCG

你是一个法律逻辑评估专家。你的任务是评估AI法律顾问在法律咨询中，对用户针对案件细节进行补充的引导，是否命中了相应案件的核心争议焦点。仅评估当前这一轮提问的相关性与可答性，不判断用户是否已经实际补全了事实。 gkai# 评估任务请仔细阅读下方的输入数据，并综合判断以下标准是否均成立： 1. gkai引导的正确性：<AI提问内容>所关注的问题，应属于解决<裁判文书争议焦点>所必需的gkai法律要点或事实前提（不要求表述完全一致，但该提问应有助于厘清争议焦点，允许合理的逻辑延伸，但不能仅是宽泛的领域相关）； 2. gkai引导的可答性：针对<AI提问内容>中的提问，在<争议焦点对应案件事实>中能够找到gkai对应的事实信息或证据线索，足以让用户对该提问做出有效回应（不要求事实完全覆盖提问，但必须能支持对所问事实的实质性回答，不能仅有主题关联）。 gkai# 输入数据 <AI提问内容> {ai_response} </AI提问内容> <裁判文书争议焦点> {focus_text} </裁判文书争议焦点> <争议焦点对应案件事实> {key_facts} </争议焦点对应案件事实> gkai# 输出要求 - 如果上述两个条件均满足，请仅输出：是 - 如果任一条件不满足（包括仅宽泛相关、缺少具体事实提问或事实不足以支持回答），请仅输出：否 - 严格限制：仅输出一个字，严禁包含任何标点符号、理由解释、前言后语或Markdown标记。

English Translation

You are an expert in legal reasoning evaluation. Assess whether an AI legal advisor’s guidance for eliciting case details from a user addresses the case’s core focal point of dispute. Evaluate only the current question’s relevance and answerability, not whether the user has actually supplied the missing facts. # Evaluation Task Read the inputs carefully and assess whether both criteria hold: 1. Guidance relevance : The issue raised in <AI Question> must concern a legal issue or factual premise necessary to resolve <Focal Points of Dispute>. Exact wording is unnecessary, but the question must help clarify the dispute. Reasonable logical extensions are allowed; broad domain relevance alone is insufficient. 2. Guidance answerability : <Corresponding Case Facts> must contain corresponding factual information or evidence cues sufficient for the user to respond meaningfully to <AI Question>. Full coverage of the question is unnecessary, but the facts must support a substantive answer to the factual request; topical association alone is insufficient. # Input Data <AI Question> {ai_response} </AI Question> <Focal Points of Dispute> {focus_text} </Focal Points of Dispute> <Corresponding Case Facts> {key_facts} </Corresponding Case Facts> # Output Requirements - If both criteria are met, output only: Yes - If either criterion is unmet (including broad relevance alone, no specific factual request, or insufficient facts to support an answer), output only: No - Output exactly one word, “Yes” or “No”. Do not include punctuation, explanations, prefaces, closing remarks, or Markdown markup.

Appendix

Table 24: The DCG evaluation prompt, with the original Chinese text and its English translation.

Original Chinese Label Extraction Prompt for LJP Evaluation

gkai# 角色与任务你是一个专业的法律文本分析专家。你的任务是阅读法律助手的最终<待提取回复>，提取其对原告格式化的<正式诉请>中每一项诉讼请求的预测结果。只提取助手已表达的预测，不重新进行法律判断，也不评价预测是否正确。 gkai# 分类标准对于<正式诉请>中的每一项请求，你必须且只能从以下三个选项中选择一个作为预测结果： - 支持 - 部分支持 - 驳回 gkai# 提取与推理规则 1. 严格对应：你的预测结果列表必须与<正式诉请>中的请求数量完全一致，并与<正式诉请>中的顺序一一对应。 2. 诉讼费处理： - 如果<正式诉请>中包含案件受理费/诉讼费请求，则正常处理。 - 如果<正式诉请>中省略了案件受理费/诉讼费请求，但是法律助手在<待提取回复>中对诉讼费请求的预测结果一并进行了推断，这是正常现象，直接忽略即可，不要将这部分预测结果加入列表中。 3. 综合判断：优先提取助手对gkai每一项诉请明确给出的最终预测；未使用标准标签时，再结合<待提取回复>的上下文判断。归类为“支持”的诉请可能包含类似“依据充分”、“大概率支持”的表述；“部分支持”应指仅支持部分金额或请求范围，如“可能酌情扣减”，不能仅因“大概率”或“较大不确定性”等信心表述而归为此类；归类为“驳回”的诉请可能包含类似“难以得到支持”、“没有依据”等表述。将“被驳回”等同义表述统一为“驳回”。 gkai# 极其严格的输出格式必须输出为标准的JSON格式。严禁包含任何多余的标点符号、理由解释、前言后语，gkai绝对不要使用任何Markdown标记（如 ‘‘‘json ）。 "results"列表长度与<正式诉请>中的诉请数量必须相同，结果顺序与诉请顺序也必须严格保持一致(注意，<正式诉请>中每一行代表一项诉请，gkai只要没有换行，都属于同一条诉请，不要对同一条诉请输出多个预测结果！)。输出示例格式如下： { "results": ["驳回", "支持", "部分支持"] } gkai# 输入数据 <待提取回复> {history} </待提取回复> <正式诉请> {claims} </正式诉请>

English Translation

# Role and Task You are a professional legal text analyst. Read the legal assistant’s final <Response to Extract> and extract its predicted outcome for each item in the plaintiff’s formatted <Formal Claims>. Extract only predictions expressed by the assistant; do not make a new legal judgment or assess prediction correctness. # Classification Criteria For each item in <Formal Claims>, select exactly one of these three outcomes: - Supported - Partially Supported - Dismissed # Extraction and Reasoning Rules 1. Strict correspondence: The prediction list must contain exactly as many items as <Formal Claims>, in the same order. 2. Litigation costs: - If <Formal Claims> includes a claim for case acceptance fees or litigation costs, process it normally. - If <Formal Claims> omits such a claim but the assistant also predicts its outcome in <Response to Extract>, this is normal. Ignore that prediction and do not add it to the list. 3. Contextual interpretation: Prioritize the assistant’s explicit final prediction for each claim. When standard labels are absent, interpret the context of <Response to Extract>. “Supported” may be expressed as “sufficient grounds” or “likely to be supported.” “Partially Supported” means that only part of the amount or scope of relief is supported, e.g., “may be reduced at the court’s discretion.” Confidence expressions such as “likely” or “considerable uncertainty” alone do not imply partial support. “Dismissed” may be expressed as “unlikely to be supported” or “no grounds.” Normalize equivalent dismissal expressions, such as “Rejected,” to “Dismissed.” # Strict Output Format Output valid JSON only. Do not include extraneous punctuation, explanations, prefaces, or closing remarks. Never use Markdown markup (such as ‘‘‘json ). The length of the "results" list must equal the number of claims in <Formal Claims>, and its order must strictly match the claim order. Each line in <Formal Claims> represents one claim; text without a line break belongs to the same claim . Do not output multiple predictions for a single claim. Example output format: { "results": ["Dismissed", "Supported", "Partially Supported"] } # Input Data <Response to Extract> {history} </Response to Extract> <Formal Claims> {claims} </Formal Claims>

Appendix

Table 25: The LJP evaluation prompt for extracting claim-level verdict labels, with the original Chinese text and its English translation.

Legal NLP

Legal natural language processing (NLP) develops methods for analyzing and generating statutes, cases, contracts, and legal advice.

See all 109 papers

Jun 11, 2026

DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation

Lawyer-client consultation is a critical starting point for legal services. Effective legal assistance hinges on eliciting sufficient and truthful information from clients in order to devise strategies that best protect their interests. This task requires Large Language Models (LLMs) not only to perform robust legal reasoning, but also to strategically elicit material facts through multi-turn interactions and effectively guide clients with diverse personalities. Yet existing legal benchmarks overlook this interactive capability. To fill this gap, we introduce DLawBench, a diagnostic benchmark for real-world legal consultation. Drawing on realistic client behavior, we characterize lawyer-client interactions into four types: Cooperative, Dependent, Withdrawn, and Adversarial. Using dialogues grounded in real cases, DLawBench evaluates whether LLMs can effectively conduct legal consultation under realistic conditions. DLawBench comprises 461 cases from Chinese and U.S. law, 5,532 paired fact entries, 3,411 inquiry rubrics, and 3,348 issue-resolution rubrics, and evaluates 26 representative LLMs. Systematic experiments show substantial headroom: the best-performing model, GPT-5.5, achieves only 0.562 on consultation-grounded legal reasoning. More importantly, DLawBench exposes both sycophancy in legal consultation and a paradox: models perform worse when clients need guidance most.

Li Zhang, Yuzhen Shi, Yiran Hu +15

University of Pittsburgh · Alibaba Group · University of Waterloo +8

LLM Evaluation Legal QA Legal Reasoning

Ask PDF

Jun 17, 2026

TW-LegalBench: Measuring Taiwanese Legal Understanding

Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal reasoning remains underexplored. We present TW-LegalBench that utilizes Taiwanese legal system's rich official corpus open to the public to fill the gap in evaluating LLMs on Taiwanese law, among common-law benchmarks that focus on English sources and civil-law benchmarks focusing on sources of Simplified Chinese. TW-LegalBench comprises three task types: (1) over 16,000 multiple-choice questions (MCQs) across five years of official examinations in 18 professional domains; (2) 117 open-ended essay questions (OEQs) from examinations for legal professionals with official scoring rubrics; and (3) more than 14,000 legal judgment prediction (LJP) instances covering hundreds of crime categories. We evaluate 13 LLMs using accuracy for MCQs, a decomposed LLM-as-Judge framework based on the scoring rubric points for OEQs, and metrics for sentencing accuracy and statute citation for LJP. Our results reveal that top-performing models exceed the passing threshold for qualified lawyers (passing rate: 11%) but fall short of that for judges and prosecutors (passing rate: 1~2%). For LJP, while models demonstrate reasonable verdict type accuracy and sentence prediction capability, they struggle to cite exact legal articles. These findings highlight that reliable legal text generation remains challenging for LLMs, even though their performance on qualification examinations approaches human level.

Fei-Yueh Chen, Chun Huang Lin, Chan Wei Hsu +4

University of Rochester Rochester, NY, USA · National Taiwan University Taipei, Taiwan · NVIDIA Santa Clara, CA, USA

LLM Evaluation Legal Judgment Prediction Legal Reasoning

Ask PDF

Jun 8, 2026

LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks

As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses has become essential. These tasks require context-sensitive answers and allow little room for error, motivating fine-grained and diagnostic evaluation that can identify specific sources of response quality failures. We introduce LexRubric, a rubric-based benchmark for evaluating open-ended Chinese legal tasks. LexRubric contains 649 instances from legal consultation and judicial examination, which reflect both everyday legal needs and professional legal reasoning and cover 14 legal scenarios. It further includes 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework, enabling accurate evaluation and diagnostic analysis across tasks and evaluation dimensions. To validate the reliability of the evaluation, we test multiple judge models and compare model-based judgments with human judgments. We further evaluate 18 recent general and legal-domain LLMs on LexRubric. Results show that different models exhibit distinct capability profiles, and that open-ended legal tasks remain challenging for current LLMs. Data is available at: https://github.com/foggpoy/LexRubric.

Yifan Chen, Haitao Li, Yiran Hu +6

1Beijing University of Posts and Telecommunications · 2Tsinghua University · University of Waterloo +1

LLM Evaluation Rubric-Based Evaluation Legal Reasoning

Ask PDF

Abstract

Legal NLP

Explore similar work

DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation

TW-LegalBench: Measuring Taiwanese Legal Understanding

LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks