Useful Diversity or Hallucinated Variety? Persona-Conditioned Synthetic Query Generation for E-Commerce Search

Filippos Ventirozos; Tom Armitage; Mahmoud Oshagh; Peter A. Appleby; Matthew Shardlow

cs.CLOct 26, 2026

Useful Diversity or Hallucinated Variety? Persona-Conditioned Synthetic Query Generation for E-Commerce Search

Authors: Filippos Ventirozos, Tom Armitage, Mahmoud Oshagh, Peter A. Appleby, Matthew Shardlow

Organizations: Manchester Metropolitan University, Manchester, UK · Autotrader UK, Manchester, UK

Abstract

Synthetic supervision can bootstrap semantic parsers in new search domains, but prompt-induced diversity is useful only when generations are faithful to the structured intent. We explore this for cold-start e-commerce search: 118,098 synthetic pairs produced by 6 LLMs with 9 persona-conditioned prompt families on a 47-field UK automotive schema, against 1,214 human-annotated and 15,304 production queries. We show that the IPIP personality bundle improves parser F1 in all four matched contrasts (Holm-corrected $p<.001$ ), and all three bundles have a positive main effect over the complete $2^3$ grid. Syntax conditioning is the axis that moves generations towards real traffic. It also induces lexical diversity. Alignment sKL, the divergence between von Mises-Fisher distributions fitted to synthetic and human-annotated queries, is an unlabelled proxy for parser F1 ( $ρ=-0.917$ against human queries written for simulated shopper scenarios, lower is closer). An LLM committee, checked against a blinded author pass, triages the 927 queries missed by every parser and labels 37% as hallucinated. Dropped constraints dominate. The four syntax-bearing families have the largest hallucinated shares, 38 to 42%, while IPIP alone has the lowest at 28%.

Figures & tables

Figure 1: Internal diversity per prompt configuration, ordered by Self-BLEU-4 (more diverse on the left). Coloured bars are the mean across 6 generators; the light-grey shadow behind each bar spans

\pm 1

std across LLMs. Tall shadows flag configurations whose diversity depends heavily on the generator; short shadows indicate that the prompt itself, not the LLM, is doing the work.

			In-house human reference			Real users
Cfg	F1 $\uparrow$	Spread $\kappa$ $\downarrow$	Persona-card	Website-front	Both protocols	Production
			$n{=}1{,}058$	$n{=}156$	$n{=}1{,}214$	$n{=}15{,}304$
pas	$79.62_{\text{1st}}$	3641	$\mathbf{264.6}_{\text{1st}}$	$\mathbf{438.5}_{\text{1st}}$	$\mathbf{252.5}_{\text{1st}}$	$\mathbf{812.5}_{\text{1st}}$
pa	$79.40_{\text{2nd}}$	3786	$279.5_{\text{4th}}$	$497.5_{\text{7th}}$	$273.5_{\text{5th}}$	$928.7_{\text{8th}}$
u	$79.19_{\text{3rd}}$	3756	$277.0_{\text{3rd}}$	$483.7_{\text{5th}}$	$269.5_{\text{3rd}}$	$894.1_{\text{5th}}$
ps	$79.05_{\text{4th}}$	3653	$270.1_{\text{2nd}}$	$447.2_{\text{2nd}}$	$258.5_{\text{2nd}}$	$821.6_{\text{2nd}}$

Table 1: Alignment sKL (von Mises–Fisher symmetric KL, lower is closer) from each prompt configuration to each reference, with every value’s within-column rank as a subscript and cell shading on each column separately (darker is better: higher F1, lower sKL). Spread

\kappa

is the synthetic cloud’s own concentration (lower is broader). Rows are ordered by tuple F1 on the in-house test set, so a proxy that tracked utility perfectly would read

1\text{st},2\text{nd},\dots

down each column; F1 is identical across columns because no parser was scored on production text. Absolute sKL is not comparable across columns (real traffic is more dispersed); only the within-column ranking is. All three production views give identical rankings (

\tau{=}1.000

Cfg	F1	Prec.	Rec.
n	75.98 $\pm$ 0.22	73.57 $\pm$ 0.40	80.94 $\pm$ 0.14
s	76.91 $\pm$ 0.20	74.67 $\pm$ 0.24	81.71 $\pm$ 0.29
a	77.70 $\pm$ 0.20	76.55 $\pm$ 0.33	80.87 $\pm$ 0.18
p	77.99 $\pm$ 0.23	76.40 $\pm$ 0.39	81.69 $\pm$ 0.07
as	78.55 $\pm$ 0.27	77.07 $\pm$ 0.17	82.21 $\pm$ 0.45
ps	79.05 $\pm$ 0.40	77.74 $\pm$ 0.57	82.50 $\pm$ 0.11

Table 2: Per-configuration parser utility on the held-out human reference, tuple-level F1 with field–value precision and recall, mean

\pm

std across three train/val resamples (test set held fixed, all numbers

\times 100

). mixed is the uniform-sample baseline pooling all 9 configurations at matched training volume.

LLM	n	u	p	a	s	pa	ps	as	pas	All
DeepSeek	2/9	8/13	7/13	7/13	4/10	11/17	8/13	9/13	9/15	65/116
Gemini-3.1	3/9	14/20	2/6	2/10	14/28	1/7	8/17	13/31	17/27	74/155
GPT-5.4	2/7	2/6	3/8	2/5	2/7	2/6	2/8	2/6	2/9	19/62
Hunyuan	10/16	10/14	3/7	5/8	13/16	7/11	5/10	4/9	6/10	63/101
Opus-4.6	6/20	14/65	9/43	8/29	15/57	9/33	27/64	18/50	18/54	124/415
Qwen3	0/12	0/11	0/8	0/7	0/9	0/8	0/8	0/7	0/8	0/78

Table 3: Committee majority-vote labels within the conditional universal-fail subset, under the rubric of § 5.2 . Each cell shows

n_{\text{hallu}}/n_{\text{majority}}

among the

927

universal failures after the scoring rule of Appendix K , shaded by absolute

n_{\text{hallu}}

. Configurations enter this pool at different rates, so these composition counts estimate neither configuration-level nor corpus-level hallucination rates.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Code	UK	IPIP	Adj	Syn	Active conditioning
n	–	–	–	–	no persona conditioning
u	✓	–	–	–	UK demographic descriptors only
p	–	✓	–	–	IPIP personality only
a	–	–	✓	–	attribute-style adjectives only
s	–	–	–	✓	syntactic mode only
pa	–	✓	✓	–	IPIP + adjectives

Appendix

Table 4: Legend for the 9 prompt families. Columns indicate which of the four prompt axes are active: UK = UK demographic descriptors, IPIP = IPIP Big Five personality cues, Adj = attribute-style adjective injection, Syn = syntactic mode. u is the only family that uses the UK demographic axis; the other eight families combine subsets of the IPIP, adjective and syntax axes.

Trait	Intended linguistic cues
Openness	exploratory wording, broader product framing, more descriptive modifiers
Conscientiousness	explicit constraint listing, precise budgeting, careful slot marking
Extraversion	energetic wording, higher affect, more direct engagement with the request
Agreeableness	polite requests, softened directives, cooperative framing
Neuroticism	hedging, uncertainty markers, reassurance-seeking language

Appendix

Table 5: Operational mapping from IPIP personality axes to prompt-level linguistic cues.

Mode	$n$	Hallucinated	Rate
Noun phrase	228	99	0.434
Telegraphic	134	56	0.418
Imperative	32	11	0.344
Natural question	57	21	0.368
Post-modified NP	35	9	0.257

Appendix

Table 6: Committee positive-label share by syntactic mode within the selected universal-fail subset. Function-word-suppressing modes (noun phrase, telegraphic) carry the largest share; the post-modified noun phrase, which preserves relational prepositions, the smallest. These conditional shares are not mode-level risk estimates.

Dimension	$\Pr$	Semantic gate (illustrative)
hard_specs	0.20	always eligible; no adjective injection
economic	0.18	price field or sort_by=PRICE_ASC present
affordance	0.15	body type / seat count compatible
evaluative	0.15	mileage / condition / certification present
vague	0.12	always eligible
aesthetic	0.08	colour / body type present

Appendix

Table 7: Configured probabilities and example semantic gates for the eight attribute-style adjective dimensions. Gates ensure adjective injection is consistent with the recoverable structured intent.

Model	Provider	Exact identifier
Closed-source frontier APIs
GPT-5.4	OpenAI	gpt-5.4
Claude Opus 4.6	Anthropic	claude-opus-4-6
Gemini 3.1 Pro	Google DeepMind	gemini-3.1-pro-preview
Open-weight reasoning models (Hugging Face Hub)
DeepSeek-R1-Distill-Llama-70B	DeepSeek-AI	deepseek-ai/DeepSeek-R1-Distill-Llama-70B

Appendix

Table 8: Generator pool used to build the synthetic corpus. The top block lists closed-source frontier APIs; the bottom block lists open-weight reasoning models, with Hugging Face Hub identifiers shown verbatim.

LLM	n	u	p	a	s	pa	ps	as	pas	Range
DeepSeek	0.606	0.580	0.580	0.582	0.587	0.584	0.582	0.586	0.585	0.026
Gemini-3.1	0.591	0.620	0.610	0.572	0.440	0.608	0.376	0.409	0.368	0.251
GPT-5.4	0.662	0.655	0.658	0.655	0.649	0.651	0.656	0.657	0.654	0.013
Hunyuan	0.582	0.626	0.626	0.628	0.620	0.624	0.623	0.628	0.623	0.046
Opus-4.6	0.582	0.489	0.547	0.537	0.469	0.523	0.436	0.440	0.422	0.160
Qwen3	0.524	0.637	0.641	0.643	0.645	0.641	0.647	0.642	0.645	0.123

Appendix

Table 9: Self-BLEU-4 (lower is more diverse) per (LLM, configuration) cell on split_0 . Cell shading is on a global green scale (deeper green = more diverse); the Range column gives each LLM’s max

-

min Self-BLEU across the nine configurations.

	Synth.	Human	Prod.
$n$	109,098	1,214	15,304
Fields, 47-basis (mean)	5.96	3.17	—
Fields, 23-basis (mean)	5.10	2.66	2.36
Fields, 23-basis (median)	5	3	2
Words (mean)	17.64	14.31	4.85
Words (median)	15	12	3

Appendix

Table 10: Corpus comparison. Synthetic counts are the nine per-configuration train+val pools used for the alignment fits; production field counts are computed on the

12{,}610

rows where the deployed engine returned a parse, while length statistics use every retained row.

Figure 2: Corpus distributions behind Table 10 . Left: share of each corpus by number of structured fields per query on the 23-field parity basis (axis cut at 12 fields). Right: share by query length in words, axis cut at 30 words, which covers 90%, 93% and 99% of the synthetic, human and production corpora respectively. Production field counts as in Table 10 .

Category	Representative examples
Vehicle attributes	manufacturer, model, transmission
Vehicle specifications	mileage, registration year, engine size
Purchase price	minimum price, maximum price, total-price band
Finance/leasing	deposit, annual mileage allowance, contract length
Search controls	search radius, newest-first sorting, price sorting
Listing filters	manufacturer-approved listings, Northern Ireland only, part exchange available

Appendix

Table 11: The 47-field schema by category. Examples are representative schema constraints rather than exhaustive field lists.

By collection protocol
Subset	Contrast	$\Delta$ F1	95% CI
Persona-card	n $\rightarrow$ p	2.019	[1.425, 2.617]
Persona-card	a $\rightarrow$ pa	1.568	[1.071, 2.073]
Persona-card	s $\rightarrow$ ps	2.082	[1.562, 2.620]
Persona-card	as $\rightarrow$ pas	1.204	[0.740, 1.685]
Website-front	n $\rightarrow$ p	1.931	[0.271, 3.608]

Appendix

Table 12: Paired IPIP contrasts within subsets of the human reference: by collection protocol (persona-card

n=1{,}058

, website-front

n=156

) and by gold-intent field count (

n=174

with

0

–

2

fields,

608

with

3

432

with

4

+). The absolute protocol gap (persona-card F1

79.6

–

83.2

against website-front

51.7

–

55.2

across the nine configurations) motivates reporting paired effects within, rather than pooling away, protocol. Overall-to-stratum rank agreement across the field-count strata is

\rho=0.933

0.983

and

0.950

. All nine training configurations contain exactly the same

12{,}122

-intent multiset and field-count distribution, so the field-count rows test persistence across human intent complexity, not reweighted training.

Human reference	$n$	$\rho$	95% interval
Both protocols	1,214	$-0.800$	[ $-0.917$ , $-0.633$ ]
Persona-card	1,058	$-0.917$	[ $-0.967$ , $-0.783$ ]
Website-front	156	$-0.500$	[ $-0.800$ , $-0.200$ ]

Appendix

Table 13: Spearman association between vMF sKL (lower is closer) and F1 (higher is better) across nine prompt configurations. F1 is computed on each reference’s own rows, whereas Table 1 ranks every column against the pooled F1 it displays, which is why the website-front

\rho

-0.500

here and

-0.417

there. Intervals jointly bootstrap human rows

2{,}000

times, recomputing both alignment and F1 ranks; the proportions below zero are 1.000, 1.000, and 0.999. This is an unlabelled alignment proxy, not an absolute-F1 predictor.

Cfg	Words	Func.%	1st-p.%	Oper.%	Types	Cov.%
n	15.26	29.7	10.6	92.1	1,799	60.9
u	18.90	33.1	17.4	92.3	2,615	73.6
p	20.84	34.1	19.0	92.3	2,582	69.6
a	16.50	29.4	10.8	92.2	1,801	63.6
s	15.74	28.8	10.9	88.4	1,858	61.4
pa	21.47	34.4	20.0	92.4	2,649	71.0

Appendix

Table 14: Surface anatomy of the prompt axes. Words : mean query length (whitespace tokens, as in Table 10 ). Func.% : function-word share of tokens, from a fixed closed-class list released with the code. 1st-p.% : share of queries containing a first-person form. Oper.% : share of queries containing an operator-bearing word ( under , within , from , newest , …); the operator list overlaps the function-word list, so the two columns are related signals, not independent ones. Types : distinct word types in a fixed

100

k-token sample (seed

20260801

) — vocabulary breadth at matched budget; the nine banks are row-matched rather than token-matched, and the anchor corpora are too small for the budget, so their value is withheld. Cov.% : share of the

1{,}000

most frequent production content words (glue and bare numbers excluded) present in the corpus vocabulary — lexical coverage, not distributional match. Adding s deletes grammatical glue, operator words included (the realism direction); adding p adds glue of the annotator-register kind (pronouns, politeness) while leaving operator presence untouched, and widens the vocabulary at fixed intent; adding a moves no surface channel. Real traffic is nearly glue- and first-person-free, while the in-house reference is majority first-person. All values are descriptive statistics over nine configurations.

Adding s
	Real traffic		In-house, both protocols
Contrast	$\Delta$ sKL	95% CI	$\Delta$ sKL	95% CI
n $\rightarrow$ s	$-60.7$	$[-61.4,-60.0]$	$-33.9$	$[-35.9,-31.9]$
p $\rightarrow$ ps	$-105.5$	$[-106.1,-104.9]$	$-17.9$	$[-21.1,-14.7]$
a $\rightarrow$ as	$-114.1$	$[-114.7,-113.6]$	$-41.2$	$[-43.9,-38.5]$
pa $\rightarrow$ pas	$-116.1$	$[-116.8,-115.4]$	$-21.0$	$[-24.5,-17.5]$

Appendix

Table 15: Matched-contrast change in Alignment sKL (negative is closer) from adding each prompt axis, against real production traffic and the in-house reference (both protocols). Intervals are

2{,}000

-replicate paired bootstraps over reference rows (seed

20260801

): each replicate resamples the reference once and evaluates all configurations against it, so the four contrasts of an axis are paired; synthetic fits are held at their full-corpus values, as in every other interval reported for the alignment analysis. Adding s moves generations toward real traffic in all four contrasts; adding p moves them toward the in-house reference in all four but not toward real traffic ( n

\rightarrow

p is positive), and a helps only when s is present.

Comparator	$n$	Agreement [95% CI]	$\kappa$	Precision	Recall	F1	TP/FP/FN/TN
Opus-4.7	100	91.0 [83.8, 95.2]	.775	.767	.920	.836	23/7/2/68
GPT-5.5	100	80.0 [71.1, 86.7]	.570	.558	.960	.706	24/19/1/56
Gemini-3.1	100	85.0 [76.7, 90.7]	.659	.632	.960	.762	24/14/1/61
Committee majority	100	83.0 [74.5, 89.1]	.622	.600	.960	.738	24/16/1/59

Appendix

Table 16: Author–committee agreement within the universal-fail subset after the author’s targeted correction pass, under the rubric as run and the scoring rule of this appendix. Agreement is adjudicated rather than independent: the blinded first pass agreed on

72\%

. Hallucination is the positive class, and the final single-author label is the reference. Agreement intervals are 95% Wilson intervals.

E-Commerce Search

E-commerce search studies how to match shoppers’ queries, preferences, and product signals with useful items in online marketplaces.

See all 42 papers

Demand-Side Measurement for Generative Engine Optimization: Constructing and Validating a Million-Persona, Intent-Annotated Buyer Corpus
Aug 30, 2026Dmitrij Żatuchin, Daniil DzemesjukGenerative Engine Optimization Synthetic Data Generation
Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia. · Rankfor.AI OÜ, Tallinn, Estonia.
Beyond Relevance: Structured Semantic Supervision for Product Search with LLM-Augmented Annotations
Sep 20, 2026Girish A. Koushik, Swapnil Bhosale, Samarth Agrawal +4Learning to Rank LLM-Assisted Annotation
Computer Science, University of Surrey, Guildford, United Kingdom · People-Centred AI, University of Surrey, Guildford, United Kingdom · eBay USA +2
SimPersona: Learning Discrete Buyer Personas from Raw Clickstreams for Grounded E-Commerce Agents
May 14, 2026Zahra Zanjani Foumani, Alberto Castelo, Shuang Xie +3LLM Personalization Vector-Quantized Variational Autoencoder
Shopify · Bellevue, Washington, USA

Aug 30, 2026J/K move · Enter open · S save

Demand-Side Measurement for Generative Engine Optimization: Constructing and Validating a Million-Persona, Intent-Annotated Buyer Corpus

Dmitrij Żatuchin, Daniil Dzemesjuk

Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia. · Rankfor.AI OÜ, Tallinn, Estonia.

Generative Engine Optimization Synthetic Data Generation User Preference Modeling

Open paper AskPDF arXiv

Generative engines such as ChatGPT, Gemini, and Perplexity answer buyer questions directly and name a shortlist of brands inside the answer. Studying how brands enter or fail to enter that shortlist requires demand-side data: what buyers in a category ask, what information they need, and which sources they trust. Existing large persona corpora are built for training-data diversity and carry neither a staged search-intent label nor a preferred-sources field, so they cannot be joined to supply-side recommendation measurements. We built and validated PersonaGen-1M, a corpus of 1,031,732 synthetic buyer personas spanning 511 industry labels and 4 market contexts, carrying 19,416,821 structured behavioral attributes, 5,160,046 of them search queries. Each persona carries a single primary_intent label covering its query set (78.3% informational, 17.4% commercial, 4.3% transactional) and a preferred_sources field naming the source types that buyer would trust. The corpus was built from roughly 40 million raw persona descriptions drawn from four public datasets through GPU-accelerated MinHash LSH plus semantic deduplication, then enriched to a fixed schema. The intent field selects the commercial-evaluation personas whose queries drive recommendation, and the preferred_sources field pairs against citation-provenance data; that join is the primary intended use, and its controlled empirical estimate is future work. Among million-scale persona corpora surveyed in August 2026, one other carries a source-preference attribute, as a six-value media-channel enum; PersonaGen-1M pairs named per-persona source lists with a staged commercial search-intent label and an attached query set. The full corpus is shared on request for non-commercial research; a stratified subset is published openly so the protocol, the schema and the validation can be inspected and reused without asking us.

Abstract

E-Commerce Search

Explore similar work

Demand-Side Measurement for Generative Engine Optimization: Constructing and Validating a Million-Persona, Intent-Annotated Buyer Corpus

Beyond Relevance: Structured Semantic Supervision for Product Search with LLM-Augmented Annotations

SimPersona: Learning Discrete Buyer Personas from Raw Clickstreams for Grounded E-Commerce Agents