People ask conversational AI models about their legal rights, obligations, procedures, and personal situations that may have legal consequences. To study this behavior, we first need to distinguish requests for legal guidance from other legally themed discussions and unrelated conversations. We contribute a multi-axis taxonomy of legal queries capturing whether legal guidance is sought, how a request is formulated, and which legal domains it concerns; apply it to 2,122 naturally occurring WildChat-1M conversations; and create a human-reviewed gold evaluation set of 200 conversations. All annotations, model checkpoints, and evaluation code are publicly available. In this enriched sample, 71.1% of conversations containing legal content seek personalized or practical guidance. Rights-and-options advice is the most common primary request type among legal-content conversations, and 25.3% of guidance-positive conversations span multiple legal topics. The gold evaluation set supports a two-stage comparison of ModernBERT-base and ModernBERT-large against a TF-IDF cascade. Our framework supports large-scale studies of legal guidance in conversational AI, and the public release of the annotated data facilitates reproducible research and future model development.
Figures & tables
Split
n
Guidance
Legal content
Human guidance
Silver-labeled dataset
2,122
972 (45.8%)
1,367 (64.4%)
–
Silver development set
1,922
884 (46.0%)
1,232 (64.1%)
–
Train
1,632
750 (46.0%)
1,048 (64.2%)
–
Validation
290
134 (46.2%)
184 (63.4%)
–
Gold evaluation set
200
88 (44.0%)
135 (67.5%)
84 (42.0%)
Table 1: Dataset and label summary. The Guidance and Legal content columns report silver labels; the last column reports the final human-reviewed guidance labels, which exist only for the gold evaluation set and replace the corresponding silver guidance labels during evaluation. The dataset was enriched during screening, so these percentages are not WildChat prevalence estimates.
Topic
Primary n
Primary %
Any %
Immigration & Citizenship
137
14.1
14.6
Criminal Law
111
11.4
11.9
Intellectual Property
110
11.3
12.0
Business & Commercial
107
11.0
17.9
Tax Law
105
10.8
13.3
Employment & Labor
87
9.0
10.0
Table 2: Legal-topic distribution among the 972 silver-labeled guidance-positive conversations. “Primary” counts the predominant topic; “Any” also counts secondary-topic assignments and therefore does not sum to 100%.
System
Input and representation
Training configuration
Validation selection
Full TF–IDF
Complete user turns; word 1–2-grams (150k) and character 3–5-grams (200k)
Class-balanced logistic regression; selected C=1 from {0.25,1,4}
Guidance threshold 0.51 selected on validation data
Full ModernBERT
Complete user turns; base and large encoders; maximum length 4,096
Batch size 32, weight decay 0.01, seed 42; guidance: 3 epochs at 5×10−5 ; class-weighted topic: 10 epochs at 6×10−5
Checkpoint selected by macro-F1; guidance thresholds: 0.26 (base), 0.38 (large)
Prefix TF–IDF
User-turn prefixes; word 1–2-grams and character-within-word 3–5-grams
Logistic regression with C=0.25 ; conversation-level weighting
Weighted macro-F1; threshold 0.42
Prefix ModernBERT
User-turn prefixes; base and large encoders; maximum length 4,096
Table 3: Training configurations. Thresholds apply to guidance detection. Prefix thresholds maximize conversation-weighted macro-F1, using positive-class F1 as a tie-breaker.
Model
Acc.
Macro-F1
Full conversation
TF–IDF + logistic regression
.840
.839
ModernBERT-base
.815
.813
ModernBERT-large
.845
.845
Prefix models (final turn)
TF–IDF + logistic regression
.825
.823
Table 4: Guidance detection on the 200-conversation gold evaluation set. Prefix models are scored after the final user turn. Full metrics and bootstrap intervals are provided in the supplementary material. Bold marks the best value within each block.
Turn 1 ( n=200 )
Turn 2 ( n=104 )
Model
Acc.
Macro-F1
Acc.
Macro-F1
TF–IDF + logistic regression
.830
.826
.798
.798
ModernBERT-base
.800
.792
.808
.808
ModernBERT-large
.825
.819
.865
.865
Table 5: Exploratory initial-turn prediction results. Turn 1 uses all conversations in the gold evaluation set; Turn 2 uses the 104 conversations containing at least two user turns. Labels indicate whether the complete conversation eventually seeks guidance, not whether guidance has begun by that turn. The columns use different evaluation sets and should not be interpreted as within-model improvement.
Topic only †
End-to-end
Cascade errors
Model
Acc.
Macro-F1
95% CI
Acc.
Macro-F1
FP
FN
Topic
All
TF–IDF + logistic regression
.679
.559
[.441, .632]
.740
.527
23
9
20
52
ModernBERT-base
.750
.646
[.537, .715]
.725
.544
24
13
18
55
ModernBERT-large
.762
.671
[.538, .750]
.755
.593
26
5
18
49
Table 6: Topic-only classification and end-to-end routing on the gold evaluation set. † Topic-only evaluation uses gold guidance labels for the 84 guidance-positive conversations, which represent only 12 of the 14 classes; that fixed set is used for macro-F1 and its bootstrap intervals. End-to-end scores use all 200 conversations and the 13 classification outcomes observed in the evaluation set. Cascade errors comprise false-positive (FP) routing, false-negative (FN) routing, and wrong-topic predictions after correct positive routing (Topic).