How You Ask Shapes What You Get: A Theory-Seeded Measurement of Articulation in Advice-Seeking LLM Conversations

Juneha Baek; Suhyeon Lee; Donghyuk Shin

cs.CLAug 30, 2026

How You Ask Shapes What You Get: A Theory-Seeded Measurement of Articulation in Advice-Seeking LLM Conversations

Authors: Juneha Baek, Suhyeon Lee, Donghyuk Shin

Organizations: KAIST

Abstract

Users articulate the same advice-seeking request in different ways: some specify detailed constraints, others gesture at a vague need. Prior work treats this variation as noise to be averaged away; we instead treat it as a stable, measurable structure in the input distribution. We ask whether articulation (how people ask) forms latent dimensions separable from topic (what they ask about), and whether it is associated with how language models respond. We extract interpretable features from 16,447 advice-seeking prompts pooled from public chat corpora (WildChat, LMSYS, and ShareChat) and recover a small set of latent articulation factors that replicate across train/test splits and across corpora. Because this structure is largely separable from topic, the populations it defines cut across topics and stay invisible to topic- or task-based evaluation. The factors define a handful of recurring articulation styles, one of which stands out: a long-form but information-poor style, roughly one in six prompts in the largest corpus, where models return shorter, vaguer answers and do not ask for clarification even though under-specification is exactly the condition that warrants it. The contrast holds within every topic group and length quintile, and is not under-specification alone -- a second, equally under-specified style does draw clarifying questions. Two independent human annotators reproduce this contrast. We argue that benchmarks should stratify on articulation, and we offer the extracted structure as a measurement instrument for doing so.

Figures & tables

Dataset	Loaded	Cand.	LLM+	Prev.
WildChat	1,211,454	414,406	7,225	0.60%
LMSYS	479,413	104,547	7,720	1.61%
SC-ChatGPT	56,078	12,907	1,163	2.07%
SC-Grok	8,594	1,758	165	1.92%
SC-Perplexity	9,410	1,236	111	1.18%
SC-Gemini	2,990	603	51	1.71%

Table 1: Stage 0 prevalence and the role of each slice. SC = ShareChat; Loaded = English first-turn prompts retained after de-duplication and a minimum-length filter; Cand. = cascade candidates; LLM+ = surviving the LLM intent classifier. WildChat is the sole training corpus: the factor model and the GMM are fit there and only there. LMSYS is the primary replication corpus, held out and receiving the fitted model transform-only, and cross-corpus congruence (Table 5 ) is measured against it. The five SC slices, pooled, give cross-assistant replication under the same transform-only application; SC-Claude (

n{=}12

) is too small for segment replication and appears here only.

Figure 1: Segment centroids on the WildChat training set ( $n{=}7{,}225$ ): $6$ segments $\times$ $6$ factors, each cell the segment’s mean standardised factor score, centered at the corpus mean.

Corpus	$n$	$s_{0}$	$s_{1}$	$s_{2}$	$s_{3}$	$s_{4}$	$s_{5}$
WildChat (train)	7,225	16%	15%	26%	14%	16%	14%
LMSYS	7,720	13%	14%	21%	23%	0 8%	21%
ShareChat (ChatGPT)	1,163	15%	17%	34%	12%	0 7%	15%
ShareChat (Gemini)	51	18%	16%	33%	20%	0 2%	12%
ShareChat (Grok)	165	24%	13%	23%	24%	0 4%	12%
ShareChat (Perplexity)	111	20%	15%	31%	21%	0 3%	11%

Table 2: Segment shares under transform-only prediction with the WildChat-trained GMM. ShareChat is analysed pooled throughout; the four platform rows appear here as a transparency note and to vary the responding assistant with corpus construction held fixed. ShareChat-Claude (

n{=}12

) is below the minimum reliable sample. Row sums can deviate by

\pm 1

pp from independent rounding.

	$s_{0}$	$s_{1}$	$s_{2}$	$s_{3}$	$s_{4}$	$s_{5}$
	baseline	open-asker	specifier	named-comp.	long-form/low-d.	vague/situated
$n$	$2{,}418$	$2{,}408$	$4{,}016$	$2{,}975$	$1{,}852$	$2{,}778$
Response length (tokens)	309	299	369	246	260	240
Recommendation specificity ( $1$ – $5$ )	3.47	3.18	3.55	3.33	3.33	2.76
$n$ options recommended	3.98	3.66	3.65	3.58	0.82	2.67
Asked for clarification	$6.7\%$	$7.4\%$	$6.8\%$	$7.4\%$	$8.1\%$	11.9%

Table 3: Per-segment means of six response outcomes, pooled across corpora. Raw means without controls; the regression-controlled analogue is in Table 9 and Appendix I . Bold marks the most extreme cell per row.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Feature	F1	F2	F3	F4	F5	F6
ilr_attr_1	$+0.05$	$+0.01$	$-0.03$	$+0.58$	$-0.02$	$+0.02$
ilr_attr_2	$+0.54$	$+0.14$	$-0.05$	$-0.10$	$-0.10$	$-0.09$
ilr_hard_1	$-0.11$	$-0.09$	$-0.04$	$+0.86$	$+0.07$	$+0.02$
ilr_hard_2	$+0.72$	$+0.18$	$-0.41$	$-0.05$	$-0.02$	$+0.15$
ilr_hard_3	$+0.62$	$-0.16$	$+0.15$	$-0.05$	$+0.10$	$+0.01$
rate_n_spans	$+0.15$	$+0.90$	$+0.18$	$+0.00$	$+0.09$	$-0.08$

Appendix

Table 4: Full Promax-rotated factor loading matrix (

19

features

\times

6

factors). Bold:

|\lambda|>0.30

. Pattern matrix from factor_analyzer with Promax rotation; sample size

n{=}16{,}447

. F1 = specificity/structure, F2 = density, F3 = named-alternative, F4 = search-attribute hardness, F5 = self-situation, F6 = length vs imperative. Note: rate_hedge_count has no factor loading

|\lambda|>0.3

on its own: hedging covaries with self-situation (F5) and is captured at the feature level rather than as a standalone factor under the corrected preprocessing pipeline (see Section 3.4 on hedge feature deduplication).

Factor	Train/test $\phi$	WC $\leftrightarrow$ LM $\phi$
F1 (specificity)	0.986	0.973
F2 (density)	0.995	0.982
F3 (named-alt)	0.993	0.985
F4 (search-hard)	0.990	0.968
F5 (self-situation)	0.904	0.819
F6 (length-imp.)	0.890	0.811

Appendix

Table 5: Tucker’s congruence

\phi

per factor, summarised in Section 4.1 . Train/test : single 50/50 split. WC

\leftrightarrow

LM : WildChat vs. LMSYS. The four core factors clear the conventional

0.95

“equal” threshold ( Lorenzo-Seva and ten Berge, 2006 ) ; F5 and F6 sit in the “fair” band. Appendix O calibrates these values against a permutation null and reports them under a stricter extractor.

seg	F1	F2	F3	F4	F5	F6
$s_{0}$ baseline	$+0.34$	$+0.47$	$+0.29$	$+0.11$	$+0.13$	$+0.10$
$s_{1}$ open-asker	$+0.21$	$-0.04$	$-0.26$	$-1.20$	$+0.42$	$+0.11$
$s_{2}$ specifier	$+1.00$	$-0.32$	$-0.29$	$+0.40$	$+0.23$	$+0.35$
$s_{3}$ named-comp.	$-0.12$	$+0.76$	$+0.87$	$+0.85$	$-0.89$	$-0.58$
$s_{4}$ long-form	$-0.95$	$-1.19$	$-0.17$	$-0.28$	$-0.93$	$+1.24$
$s_{5}$ vague/sit.	$-1.05$	$-0.27$	$-0.23$	$-0.33$	$+0.51$	$-0.02$

Appendix

Table 6: Segment centroids on the WildChat training set under the corrected

k{=}6

factor model. Bold:

|\text{score}|>0.5

. Compare with Figure 1 .

Topic group	$n$	$\overline{\mathrm{ARI}}$	sd
H7 (Other & General)	1,935	0.684	0.025
H5 (Creative & Media)	1,015	0.582	0.028
H6 (Relational & Emotional)	2,029	0.553	0.096
H4 (Technical / STEM)	622	0.531	0.041
H3 (Education)	1,257	0.518	0.071
H1 (Information & News)	859	0.494	0.039

Appendix

Table 7: Within-topic re-clustering. The GMM is re-fit on each topic subset and its labels compared to the pooled-fit labels by ARI; we report the mean and sd over

10

random restarts, since a single restart is not stable (H3, for instance, ranges

0.40

–

0.66

across seeds). Values near

1

mean the pooled segment structure recovers inside the topic. Six topics recover at

\overline{\mathrm{ARI}}{\geq}0.49

; only H2, the bulk-advice topic and

53\%

of the sample, is substantially redrawable, and Table 8 explains why: H2 is the one topic with no dominant articulation mode.

Topic	$n$	$s_{0}$	$s_{1}$	$s_{2}$	$s_{3}$	$s_{4}$	$s_{5}$	top seg
		baseline	open-asker	specifier	named-comp.	long-form	vague/sit.
H1 (Information & News)	859	15%	11%	18%	18%	29%	0 9%	$s_{4}$
H2 (Personal Decisions)	8,730	18%	14%	28%	24%	0 3%	14%	$s_{2}$
H3 (Education & Tutoring)	1,257	13%	18%	32%	17%	0 8%	11%	$s_{2}$
H4 (Technical / STEM)	622	13%	10%	37%	10%	20%	10%	$s_{2}$
H5 (Creative & Media)	1,015	15%	18%	26%	11%	24%	0 6%	$s_{2}$

Appendix

Table 8: Topic

\times

segment cross-tab (row percentages within each topic); the last row gives the segment marginals, which the row percentages reproduce exactly. The specifier

s_{2}

is modal in four topics, including the bulk-advice topic H2. Two topics have a strongly dominant, topic-specific mode: H6 is

46\%

s_{5}

(vague and self-disclosing, unsurprising for relational and emotional problems) and H7 is

40\%

s_{4}

(long-form/low-density). That concentration is what makes the within-topic GMM in Table 7 easy to recover: H7 recovers at

\overline{\mathrm{ARI}}{=}0.68

and H6 at

0.55

. H2, conversely, has no dominant mode — its mass spreads over five segments, none above

28\%

— and it is exactly the topic whose within-topic partition is redrawable (

0.31

Outcome	$R^{2}$	seg. LR $p$
Response length	0.257	<0.001
Coverage (search)	0.201	<0.001
Recommendation specificity †	0.186	<0.001
Coverage (count)	0.159	<0.001
Response hedge count	0.129	<0.001
Asked for clarification	0.114	<0.001

Appendix

Table 9: Covariation results (overall).

R^{2}

is McFadden’s pseudo-

R^{2}

for binary outcomes, OLS

R^{2}

for continuous.

p

from likelihood-ratio test of joint segment dummies in the full regression.

n{=}16{,}447

except where noted. † Recommendation specificity is restricted to responses with at least one recommended item (

n{=}8{,}673

), since the outcome is undefined for replies that recommend nothing. The segment block is jointly significant on all twelve outcomes, which at this

n

is a weak statement; the informative quantities are the per-segment coefficients in Table 10 , and in particular the

s_{4}

clarification coefficient, which is not significant (

p{=}0.11

) even though the block containing it is.

	Resp. length	Recomm. specif.	Coverage(search)	Asked clarify	Gave direct ans.	$n_{\text{options}}$
	OLS coef	OLS coef	logit coef	logit coef	logit coef	OLS coef
$s_{1}$ open-asker	$-0.03$	$-0.19^{***}$	$-0.70^{***}$	$+0.16$	$-0.37^{***}$	$-0.06$
$s_{2}$ specifier	$+0.01$	$+0.08^{**}$	$-0.04$	$-0.00$	$-0.18^{**}$	$+0.03$
$s_{3}$ named-comparer	$-0.04$	$-0.02$	$-0.25^{***}$	$+0.25^{*}$	$-0.59^{***}$	$-0.70^{***}$
$s_{4}$ long-form	$-0.69^{***}$	$-0.16^{**}$	$-1.23^{***}$	$+0.23$	$-1.48^{***}$	$-1.86^{***}$
$s_{5}$ vague/situated	$-0.13^{***}$	$-0.44^{***}$	$-1.26^{***}$	$+0.77^{***}$	$-0.84^{***}$	$-0.96^{***}$

Appendix

Table 10: Segment dummy coefficients relative to

s_{0}

baseline. Significance:

{}^{*}p{<}0.05

{}^{**}p{<}0.01

{}^{***}p{<}0.001

(Wald). For logistic outcomes the coefficient is a log-odds difference; for

\log(\text{response length}+1)

a coefficient of

-0.69

implies

\approx

50\%

shorter responses than baseline. Source corpus, model, topic, and prompt-length controls are not shown.

Class	Prompt (excerpt)
spec_heavy	“Looking for a 15-inch laptop, must have USB-C, 32GB RAM, dedicated GPU with at least 8GB VRAM, under $1800, weight under 2.2kg. Should support Linux out of the box.”
spec_heavy	“Recommend an SUV with at least 7 seats, AWD, towing capacity 5000lb+, MSRP under $50k.”
spec_heavy	“Need a wireless mouse, 800 DPI minimum, USB-C charging, must work with macOS and Linux, under $80.”
experience_heavy	“I want a cozy cafe in Brooklyn for journaling on a rainy Saturday — something with a warm aesthetic, soft lighting, maybe vintage.”
experience_heavy	“Looking for a movie that feels like late autumn, melancholic but hopeful, with a slow burn.”
experience_heavy	“Recommend a perfume that smells warm and grounded, like an old library with leather chairs.”

Appendix

Table 11: Hand-authored known-groups prompts.

Feature	A–B	A–LLM	B–LLM
asked_clarify ( $\kappa$ )	$0.74$	$0.65$	$0.52$
coverage_search ( $\kappa$ )	$0.35$	$0.34$	$\mathbf{0.45}$
gave_direct_answer ( $\kappa$ )	$0.34$	$-0.02$	$0.18$
strict-rubric †	—	$\mathbf{0.33}$	—
rec. specificity ( $\rho$ )	$0.67$	$\mathbf{0.85}$	$0.61$
n_options ( $\rho$ )	$0.50$	$0.43$	$\mathbf{0.60}$

Appendix

Table 12: Pairwise agreement on the

N{=}100

audit. A, B = two human annotators (one author + one paid external); LLM = Stage 2 extractor. Cohen’s

\kappa

(binary) / Spearman’s

\rho

(ordinal). Bold : extractor agreement with at least one annotator

\geq

A–B baseline. † Annotator A re-coded under the spec’s strict rule; A–LLM

\kappa

rises to

0.58

within

s_{4}

$s_{0}\to s_{4}$ rate	Annot. A	LLM	Annot. B
gave_direct_answer	$92{\to}93$	$40{\to}20$	$\mathbf{100{\to}43}$
( $\Delta$ pp)	( $+1$ )	( $-20$ )	( $\mathbf{-57}$ )
coverage_search	$24{\to}13$	$48{\to}7$	$60{\to}30$
( $\Delta$ pp)	( $-11$ )	( $-41$ )	( $-30$ )
asked_clarify	$0{\to}3$	$4{\to}3$	$0{\to}3$

Appendix

Table 13: Per-segment binary rates (%) on the audit (

n_{s_{0}}{=}25

n_{s_{4}}{=}30

). The

s_{4}

deficit on direct-answer and search-coverage reproduces under both human annotators and the extractor.

Feature	Pairwise $r$
specificity_mean	0.993
n_experience	0.990
crystallization	0.981
n_brand	0.980
imperative_count	0.975
n_spans	0.948

Appendix

Table 14: Self-consistency reliability (

n{=}80

, two independent runs).

Outcome	WildChat coef	LMSYS coef
	( $n{=}7{,}225$ )	( $n{=}7{,}720$ )
Response length (log)	$-0.91^{***}$	$-1.11^{***}$
Recommendation specificity	$-0.32^{***}$	$-0.25^{**}$
Gave direct answer (logit)	$-1.59^{***}$	$-1.91^{***}$
Search coverage (logit)	$-2.07^{***}$	$-1.68^{***}$
Asked clarification (logit)	$+0.09$	$+0.83^{**}$

Appendix

Table 15:

s_{4}

coefficient on each outcome, fit separately within each corpus (reference

s_{0}

;

\log

prompt tokens is the only control, since a per-corpus fit removes the corpus and model confounds by construction). Significance:

{}^{**}p{<}0.01

{}^{***}p{<}0.001

. The four substantive deficits replicate independently in both corpora with the same sign and comparable magnitude, so the finding is not a pooling artefact. Clarification is the honest caveat and we separate it out: it is flat in WildChat (

p{=}0.67

), matching the pooled null, but in LMSYS

s_{4}

does draw somewhat more clarification (

p{=}0.001

), and the pooled estimate (

+0.23

p{=}0.11

) sits between them. We therefore state the claim as:

s_{4}

shows no reliable increase in clarification, in sharp contrast to

s_{5}

, whose increase is large and replicates everywhere. We do not claim clarification never rises for

s_{4}

in any corpus.

Outcome	$s_{0}$	$s_{4}$ (all)	$s_{4}$ (interrog.)
	$n{=}2{,}418$	$n{=}1{,}852$	$n{=}1{,}097$
Response length (tokens)	$309$	$260$	$\mathbf{231}$
$n$ options	$3.98$	$0.82$	$\mathbf{0.80}$
Asked clarify	$6.7\%$	$8.1\%$	$\mathbf{9.3\%}$
Direct answer	$66.7\%$	$19.0\%$	$\mathbf{20.3\%}$
Search coverage	$57.2\%$	$14.0\%$	$\mathbf{14.2\%}$

Appendix

Table 16: Restricting

s_{4}

to prompts containing a question mark ( interrog. ) preserves or intensifies every response-side effect. The long-form/low-density failure mode is not driven by task-like contamination.

Proxy	LLM?	AUC	FPR@TPR $=0.80$
token_count	no	$0.874$	$0.116$
avg_sentence_length	no	$0.586$	$0.712$
question_density	no	$0.549$	$0.563$
combined (logit)	no	$0.837$	$0.219$
specificity_mean	yes	$0.908$	$0.184$
spans_per_token	yes	$0.986$	$0.027$

Appendix

Table 17: Single-feature ROC AUC for

s_{4}

detection. spans_per_token (

0.986

) is essentially definitional — F2 density is dominated by this feature — so its AUC reflects how the segment was built, not a discovery. The deployment-relevant rows are the others: an LLM-free proxy, raw word count, reaches

0.874

, and a single Stage-1 feature, specificity_mean , reaches

0.908

with no factor model and no clustering. This is also our answer to “could a simpler method do this?” For the detection task, yes — one threshold on one feature, and we recommend it. What the factor analysis and clustering buy is what a threshold cannot: evidence that the flagged prompts form a reproducible population, an estimate of how much of real traffic they are, and the

s_{4}

s_{5}

contrast showing the deficit is not plain under-specification (Section 4.4 ). The word-count AUC is high enough (

0.874

) that it is worth restating why length is a cue and not the cause: L.5 shows the response deficit persists inside every length quintile, and L.7 inside every topic.

Quintile	$n$ ( $s_{0}$ / $s_{4}$ )	Direct (%)		$\Delta$	Search (%)
(tokens)		$s_{0}$	$s_{4}$	(pp)	$s_{0}$ / $s_{4}$
Q1 (3–14)	807/143	72.6	25.2	$-47.4$	61.7/25.2
Q2 (14–26)	732/34	65.7	17.6	$-48.1$	57.8/8.8
Q3 (26–96)	661/189	62.9	16.9	$-46.0$	53.0/7.4
Q4 (96–309)	172/680	62.8	21.6	$-41.2$	54.1/16.6
Q5 (309+)	46/806	45.7	16.1	$-29.6$	43.5/11.7

Appendix

Table 18: Direct-answer rate and search-attribute coverage for

s_{0}

vs.

s_{4}

within token-count quintiles. The

s_{4}

direct-answer deficit persists at

29

–

48

percentage points in every length stratum, including the shortest quintile (Q1,

3

–

14

tokens) where a long monologue is structurally impossible. Length therefore does not explain the deficit: the gap is a property of articulation density, not prompt length. The crossover in

s_{0}

s_{4}

sample sizes across quintiles also confirms the two segments are not simply long vs. short: both span the full length range.

Comparison	raw %	Cohen $\kappa$
A1 vs. A2 (human–human)	$76.0$	$\mathbf{+0.715}$
A1 vs. LLM	$67.0$	$+0.615$
A2 vs. LLM	$67.0$	$+0.614$
three-way:
all three unanimous	$57/100$
two humans agree (A1=A2)	$76/100$

Appendix

Table 19: Pairwise agreement on the

7

-way high-level topic decision. Human–human

\kappa{=}0.72

is the inter-annotator ceiling; the LLM tagger sits

\approx 0.1\,\kappa

below it on each pairwise comparison.

H class	support	F1 A1	F1 A2
H4 Tech/STEM	$14$ / $17$	$0.86$	$\mathbf{0.90}$
H6 Relational	$13$ / $14$	$\mathbf{0.96}$	$\mathbf{1.00}$
H3 Education	$14$ / $17$	$0.64$	$0.77$
H5 Creative	$27$ / $19$	$0.68$	$0.73$
H2 Personal	$12$ / $23$	$0.64$	$0.51$
H7 Other	$15$ / $\phantom{0}9$	$0.41$	$0.35$

Appendix

Table 20: Per-class F1 with the LLM tag treated as a predictor and each human annotator’s label treated as gold. support = number of prompts (A1/A2); F1 A1 /F1 A2 use annotator A1/A2 as gold. H4 and H6 are nearly perfectly recovered; H7 (Other) is the weakest class, reflecting that “other-than-everything-else” is itself an underdefined target. † H1 support of

1

on A2 makes the F1 unstable.

A1 confidence band	$n$	A1 $=$ LLM (raw %)
Low ( $1$ – $2$ )	$29$	$41.4$
Mid ( $3$ )	$35$	$65.7$
High ( $4$ – $5$ )	$36$	$\mathbf{88.9}$

Appendix

Table 21: Agreement between annotator A1 and the LLM tag stratified by A1’s self-reported confidence. When A1 finds the prompt unambiguous, LLM agreement reaches

89\%

; disagreement is concentrated on prompts where the annotator’s own confidence is low.

A1 $\downarrow$ LLM $\to$	H1	H2	H3	H4	H5	H6	H7
H1	4	0	0	0	0	0	1
H2	2	9	0	1	0	0	0
H3	3	0	9	1	0	0	1
H4	0	1	1	12	0	0	0
H5	2	4	1	0	14	0	6
H6	0	0	0	0	0	13	0

Appendix

Table 22: Confusion matrix, annotator A1 (rows) vs. LLM tag (columns). The dominant off-diagonal pattern is the LLM moving prompts from A1’s H5 (Creative & Media) and H7 (Other) into adjacent classes: the LLM under-recalls H5 (

14/27

retained) and over-conflates H7 with H1/H2/H3. Both error modes are at the boundaries humans also find difficult.

Topic	$n$ ( $s_{0}$ / $s_{4}$ )	Direct (%)		$\Delta$	Search
		$s_{0}$	$s_{4}$	(pp)	$\Delta$ (pp)
H1 Info/News	125/250	61.6	0 9.2	$-52.4$	$-41.6$
H2 Personal	1600/225	70.8	44.9	$-25.9$	$-19.6$
H3 Education	164/ 0 99	64.0	19.2	$-44.8$	$-41.7$
H4 Tech/STEM	0 82/127	56.1	33.9	$-22.2$	$-44.0$
H5 Creative	150/244	64.7	10.2	$-54.4$	$-27.2$

Appendix

Table 23:

s_{4}

against the

s_{0}

baseline, refit inside each topic group: direct-answer rates and their gap, with the same gap on search-attribute coverage in the last column. Every direct-answer gap is negative and significant (Fisher exact,

p{<}0.002

in all seven; the smallest effect is H6 at

-19.0

pp,

p{=}1.1\times 10^{-3}

). The deficit is therefore not carried by any single topic and does not disappear when topic is held fixed by construction rather than by a regression term. The one exception anywhere in the table is search coverage in H6 (Relational & Emotional), which is flat — but H6 is the topic where search attributes barely apply at all (

s_{0}

itself reaches only

10.3\%

), so there is no headroom for a gap.

$k$	BIC	Silhouette
2	0 54,592	0.197
3	0 38,280	0.192
4	0 35,313	0.215
5	0 27,611	0.190
6	21,645	0.191
beyond the factor-count cap:

Appendix

Table 24: GMM

k

-selection curves on the WildChat training set (

n{=}7{,}225

6

-factor scores). Lower BIC is better; higher silhouette is better. Within the factor-count cap (

k\leq 6

) BIC decreases monotonically and selects

k{=}6

, while silhouette peaks at

k{=}4

. If the cap is lifted, BIC keeps decreasing through

k{=}10

(no interior minimum), but silhouette drops sharply from

0.19

k{=}6

0.14

k{=}7

and stays flat thereafter. We read the post-

6

BIC gains as statistical refinement of existing modes rather than the emergence of new interpretive types: the components added beyond

k{=}6

do not separate cleanly (silhouette collapse), so they buy likelihood without distinctness. We therefore cap

k

at the factor count (

6

), the largest value at which segments remain distinct enough for stable semantic labels. We follow BIC over silhouette within the capped range for the reason in Section 3.5 : silhouette assumes spherical equal-variance clusters, violated in the oblique-rotated factor space, while BIC is the likelihood-based criterion appropriate for Gaussian mixtures ( Fraley and Raftery, 2002 ) . The bootstrap ARI for

k{=}6

(

\overline{\,}{=}0.55

, Section 4.2 ) confirms the partition is reproducible at the moderate level.

Figure 2: Parallel-analysis scree plot. Solid: data eigenvalues. Dashed: mean random eigenvalues from standard-normal draws. The crossover at

k{=}6

retains all factors with non-random variance.

	main text	re-run
Latent factors (parallel analysis)	6	6
Train/test Tucker $\phi$	0.890–0.995	0.993–0.9995
WildChat $\leftrightarrow$ LMSYS $\phi$	0.811–0.985	0.970–0.997
Bootstrap ARI (segments)	0.549	0.785
Cramér’s $V$ (segment $\times$ topic)	0.243	0.150
Within-topic ARI, H2	0.312	0.710

Appendix

Table 25: Every structural statistic improves under a stricter extraction standard, and the factor count is unchanged. In the main-text run, F5 and F6 sat at

\phi\approx 0.82

cross-corpus and were hedged in Section 4.1 as partly dataset-specific; in the re-run all six factors clear

0.97

. A structure manufactured by extraction noise should degrade, not sharpen, when the extractor is made stricter.

Manipulation	target	$\Delta$	$d$	verdict
pad (filler only)	density	$-2.12$	1.55	pass
compress	density	$+0.93$	1.08	pass
specify (checkable)	verif. spec.	$+1.09$	1.52	pass
exemplify	exemplar	$-2.20$	4.59	pass
name (3 brands)	named-alt	$+1.41$	1.38	pass
disclose	self-disc.	$+0.84$	1.01	pass

Appendix

Table 26: Pre-registered known-groups results.

\Delta

is the mean shift in the target factor’s standardised score;

d

is Cohen’s

d

. Twenty of the twenty-four frozen criteria hold. The six passing manipulations move their named factor at

d{=}1.0

4.6

, move it further than any other factor, and clear the extractor’s test–retest noise floor by

9\times

49\times

. Factor numbering here is that of the re-extracted solution (Appendix O ); we give construct names rather than indices for that reason.

Outcome	$\Delta R^{2}$	segment block
Recommendation specificity	$+0.0219$	$F{=}43.2$ , $p{\approx}5\mathrm{e}{-}44$
Coverage (search)	$+0.0138$	$\chi^{2}{=}258$ , $p{\approx}1\mathrm{e}{-}53$
Gave direct answer	$+0.0103$	$\chi^{2}{=}191$ , $p{\approx}2\mathrm{e}{-}39$
Number of options	$+0.0062$	$F{=}17.7$ , $p{\approx}2\mathrm{e}{-}17$
Asked for clarification	$+0.0037$	$\chi^{2}{=}27.6$ , $p{\approx}4\mathrm{e}{-}05$
Response length	$+0.0024$	$F{=}9.2$ , $p{\approx}1\mathrm{e}{-}08$

Appendix

Table 27: Incremental contribution of the segment block over

M_{0}

(topic

+

assistant model

+

source corpus

+

\log

prompt tokens). McFadden’s pseudo-

R^{2}

for binary outcomes, ordinary

R^{2}

for continuous. The increments are modest in absolute terms, which is what one should expect from a single categorical predictor added on top of strong controls, but they are systematic and jointly significant on every outcome: articulation type predicts response characteristics over and above what the prompt is about, who answered it, where it came from, and how long it was.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering
Apr 16, 2026Neha Srikanth, Jordan Boyd-Graber, Rachel RudingerQuestion Answering
University of Maryland
Less Back-and-Forth: A Comparative Study of Structured Prompting
May 19, 2026Saurav Ghosh, Gabriella Polach, Abdou SowLLM Evaluation Structured Prompting
Washington University in St. Louis Saint Louis, Missouri, USA
Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions
May 24, 2026Jinyan Su, Claire CardieLLM Evaluation Clarification Question Generation
1Cornell University

Apr 16, 2026J/K move · Enter open · S save

DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering

Neha Srikanth, Jordan Boyd-Graber, Rachel Rudinger

University of Maryland

Question Answering

Open paper AskPDF arXiv

We introduce DiscoTrace, a method to identify the rhetorical strategies answerers use when responding to information-seeking questions. DiscoTrace represents answers as a sequence of question-related discourse acts paired with interpretations of the original question, annotated on top of rhetorical structure theory parses. Applying DiscoTrace to answers from nine different communities reveals that communities have diverse preferences for answer construction. In contrast, LLMs do not exhibit rhetorical diversity in their answers, even when prompted to mimic specific human community answering guidelines. LLMs also systematically opt for breadth, addressing interpretations of questions that human answerers choose not to address. The rich, community-sensitive answering behavior structurally revealed by DiscoTrace can guide the development of pragmatic LLM answerers that are more attuned to contextual information needs.

Abstract

LLM Evaluation

Explore similar work

DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering

Less Back-and-Forth: A Comparative Study of Structured Prompting

Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions