cs.CLOct 18, 2026

Legal Guidance in the Wild: How Users Seek Legal Help in Real-World LLM Conversations

Authors: Amirreza Mohseni, Gerasimos Spanakis

Organizations: Maastricht University, Maastricht, The Netherlands

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

People ask conversational AI models about their legal rights, obligations, procedures, and personal situations that may have legal consequences. To study this behavior, we first need to distinguish requests for legal guidance from other legally themed discussions and unrelated conversations. We contribute a multi-axis taxonomy of legal queries capturing whether legal guidance is sought, how a request is formulated, and which legal domains it concerns; apply it to 2,122 naturally occurring WildChat-1M conversations; and create a human-reviewed gold evaluation set of 200 conversations. All annotations, model checkpoints, and evaluation code are publicly available. In this enriched sample, 71.1% of conversations containing legal content seek personalized or practical guidance. Rights-and-options advice is the most common primary request type among legal-content conversations, and 25.3% of guidance-positive conversations span multiple legal topics. The gold evaluation set supports a two-stage comparison of ModernBERT-base and ModernBERT-large against a TF-IDF cascade. Our framework supports large-scale studies of legal guidance in conversational AI, and the public release of the annotated data facilitates reproducible research and future model development.

Figures & tables

SplitnnGuidanceLegal contentHuman guidance
Silver-labeled dataset2,122972 (45.8%)1,367 (64.4%)–
Silver development set1,922884 (46.0%)1,232 (64.1%)–
Train1,632750 (46.0%)1,048 (64.2%)–
Validation290134 (46.2%)184 (63.4%)–
Gold evaluation set20088 (44.0%)135 (67.5%)84 (42.0%)
Table 1: Dataset and label summary. The Guidance and Legal content columns report silver labels; the last column reports the final human-reviewed guidance labels, which exist only for the gold evaluation set and replace the corresponding silver guidance labels during evaluation. The dataset was enriched during screening, so these percentages are not WildChat prevalence estimates.
TopicPrimary nnPrimary %Any %
Immigration & Citizenship13714.114.6
Criminal Law11111.411.9
Intellectual Property11011.312.0
Business & Commercial10711.017.9
Tax Law10510.813.3
Employment & Labor879.010.0
Table 2: Legal-topic distribution among the 972 silver-labeled guidance-positive conversations. “Primary” counts the predominant topic; “Any” also counts secondary-topic assignments and therefore does not sum to 100%.
SystemInput and representationTraining configurationValidation selection
Full TF–IDFComplete user turns; word 1–2-grams (150k) and character 3–5-grams (200k)Class-balanced logistic regression; selected C=1C=1 from {0.25,1,4}\{0.25,1,4\}Guidance threshold 0.51 selected on validation data
Full ModernBERTComplete user turns; base and large encoders; maximum length 4,096Batch size 32, weight decay 0.01, seed 42; guidance: 3 epochs at 5×10−55\times 10^{-5} ; class-weighted topic: 10 epochs at 6×10−56\times 10^{-5}Checkpoint selected by macro-F1; guidance thresholds: 0.26 (base), 0.38 (large)
Prefix TF–IDFUser-turn prefixes; word 1–2-grams and character-within-word 3–5-gramsLogistic regression with C=0.25C=0.25 ; conversation-level weightingWeighted macro-F1; threshold 0.42
Prefix ModernBERTUser-turn prefixes; base and large encoders; maximum length 4,096Batch size 32, weight decay 0.01, seed 42; 3 epochs at 5×10−55\times 10^{-5} ; conversation-level weightingWeighted validation loss; thresholds: 0.51 (base), 0.36 (large)
Table 3: Training configurations. Thresholds apply to guidance detection. Prefix thresholds maximize conversation-weighted macro-F1, using positive-class F1 as a tie-breaker.
ModelAcc.Macro-F1
Full conversation
TF–IDF + logistic regression.840.839
ModernBERT-base.815.813
ModernBERT-large.845.845
Prefix models (final turn)
TF–IDF + logistic regression.825.823
Table 4: Guidance detection on the 200-conversation gold evaluation set. Prefix models are scored after the final user turn. Full metrics and bootstrap intervals are provided in the supplementary material. Bold marks the best value within each block.
 Turn 1 ( n=200n=200 )Turn 2 ( n=104n=104 )
ModelAcc.Macro-F1Acc.Macro-F1
TF–IDF + logistic regression.830.826.798.798
ModernBERT-base.800.792.808.808
ModernBERT-large.825.819.865.865
Table 5: Exploratory initial-turn prediction results. Turn 1 uses all conversations in the gold evaluation set; Turn 2 uses the 104 conversations containing at least two user turns. Labels indicate whether the complete conversation eventually seeks guidance, not whether guidance has begun by that turn. The columns use different evaluation sets and should not be interpreted as within-model improvement.
 Topic only †End-to-endCascade errors
ModelAcc.Macro-F195% CIAcc.Macro-F1FPFNTopicAll
TF–IDF + logistic regression.679.559[.441, .632].740.5272392052
ModernBERT-base.750.646[.537, .715].725.54424131855
ModernBERT-large.762.671[.538, .750].755.5932651849
Table 6: Topic-only classification and end-to-end routing on the gold evaluation set. † Topic-only evaluation uses gold guidance labels for the 84 guidance-positive conversations, which represent only 12 of the 14 classes; that fixed set is used for macro-F1 and its bootstrap intervals. End-to-end scores use all 200 conversations and the 13 classification outcomes observed in the evaluation set. Cascade errors comprise false-positive (FP) routing, false-negative (FN) routing, and wrong-topic predictions after correct positive routing (Topic).
More in this topic

Conversational Agents

Conversational agents are AI systems that interact with people through multi-turn dialogue to answer questions, provide support, or complete tasks.

See all 291 papers

Explore similar work

CardsList
Aug 20, 2026

InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries

Legal AI systems are increasingly used to answer legal questions, yet existing benchmarks assume queries arrive fully specified. In practice, users omit facts that materially determine the legal outcome. We introduce InsufficiencyBench, the first legal benchmark targeting query-side insufficiency: whether a model recognizes when a query lacks legally material information, identifies what is missing, and refrains from premature conclusions. We formalize a taxonomy of eight canonical missing-element categories across three structural failure modes---switch, gating, and fatal prerequisite--- and construct 202 benchmark items (58 base queries, 144 deficient variants) spanning six legal domains and 24 US jurisdictions and annotated by practising attorneys. Evaluating ten frontier models, we find that no model exceeds F2 = 0.46 on missing-element identification and that the median recall is 0.44. Models either hedge indiscriminately or answer silently under fabricated presumptions. No model both identifies and qualifies responses to deficient queries while directly addressing complete ones.

Samuel J. Vincent, Daniel Calloway, Fangyi Yu +2

Thomson Reuters Foundational Research, 5 Canada Square, London · Imperial College London

Legal QALLM Evaluation
AskPDF
Sep 30, 2026

ViLegalExpert: A Large-Scale Benchmark for Vietnamese Legal Retrieval and Question Answering from Real-World Consultations

Trustworthy Legal AI requires systems that can answer legal questions while grounding their responses in authoritative sources. However, existing Vietnamese legal benchmarks provide limited coverage of real-world legal consultations. We introduce ViLegalExpert, a large-scale benchmark constructed from authentic citizen--lawyer consultations, containing over 172K questions across 34 legal domains, together with professional answers and expert-verified legal evidence. ViLegalExpert supports legal information retrieval, extractive QA, and abstractive QA. Experiments with representative retrieval methods and language models reveal substantial challenges in evidence retrieval and grounded answer generation. While pretrained models perform strongly on QA, hybrid retrieval achieves the best retrieval performance. These results demonstrate the difficulty of mapping naturally expressed legal questions to authoritative provisions and establish ViLegalExpert as a challenging benchmark for reliable Vietnamese Legal AI.

Dat Tien Nguyen, Nghia Hieu Nguyen, Anh Thi-Hoang Nguyen +3

University of Information Technology · Vietnam National University, Ho Chi Minh city, Viet Nam

Legal QALegal NLPLegal IR
AskPDF
Jun 11, 2026

DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation

Lawyer-client consultation is a critical starting point for legal services. Effective legal assistance hinges on eliciting sufficient and truthful information from clients in order to devise strategies that best protect their interests. This task requires Large Language Models (LLMs) not only to perform robust legal reasoning, but also to strategically elicit material facts through multi-turn interactions and effectively guide clients with diverse personalities. Yet existing legal benchmarks overlook this interactive capability. To fill this gap, we introduce DLawBench, a diagnostic benchmark for real-world legal consultation. Drawing on realistic client behavior, we characterize lawyer-client interactions into four types: Cooperative, Dependent, Withdrawn, and Adversarial. Using dialogues grounded in real cases, DLawBench evaluates whether LLMs can effectively conduct legal consultation under realistic conditions. DLawBench comprises 461 cases from Chinese and U.S. law, 5,532 paired fact entries, 3,411 inquiry rubrics, and 3,348 issue-resolution rubrics, and evaluates 26 representative LLMs. Systematic experiments show substantial headroom: the best-performing model, GPT-5.5, achieves only 0.562 on consultation-grounded legal reasoning. More importantly, DLawBench exposes both sycophancy in legal consultation and a paradox: models perform worse when clients need guidance most.

Li Zhang, Yuzhen Shi, Yiran Hu +15

University of Pittsburgh · Alibaba Group · University of Waterloo +8

LLM EvaluationLegal QALegal Reasoning
AskPDF