What the Query Remembers: Query Abstraction Governs Lexical and Dense Retrieval of Classical Arabic Poetry

Omar Farouk Zouak; Selma Mani; Larbi Said Chikh; Raid Abderrezak Ouahioune

cs.CLOct 22, 2026

What the Query Remembers: Query Abstraction Governs Lexical and Dense Retrieval of Classical Arabic Poetry

Authors: Omar Farouk Zouak, Selma Mani, Larbi Said Chikh, Raid Abderrezak Ouahioune

Organizations: National School of Artificial Intelligence

Abstract

How well a poetry archive can be searched depends on what the user still remembers, not only on what the model understands. We show this for Classical Arabic poetry, benchmarking nine dense encoders and BM25 over two corpora, three indexing modes and two evaluation protocols. Which retriever wins is governed by how much of a poem's wording the query still carries. On tip-of-the-tongue queries BM25 leads every dense encoder, including an 8B state-of-the-art model (0.746 vs. 0.717 Recall@5); on abstract queries it falls to eighth out of ten (0.392 vs. 0.748). Editing the wording of the same queries shows that this ranking largely reflects query composition: adding one remembered half-verse raises BM25 by 38 points, rewriting the quoted words by hand leaves BM25 level with the 8B encoder, and BM25's sensitivity to remembered wording is significantly steeper than that of the strongest dense encoder. The same axis controls explanation augmentation, whose effect swings from +37 to -39 points. The mechanism splits by system type: lexical containment predicts BM25's behaviour, while embedding proximity predicts dense encoders. Finally, the field's two standard protocols, known-item retrieval and thematic relevance, dissociate on the same systems, a result reproduced by two blind human annotators and by two open-weights judges, one of them under a rubric with no literal-match criterion. A benchmark that reports one protocol, or one query register, can give a materially wrong picture. Code and data are available at https://github.com/Ellzo/what-the-query-remembers.

Figures & tables

Query set	$n$	len	% quote	$C_{\mathrm{vrs}}$	$C_{\mathrm{ai}}$	$C_{\mathrm{ff}}$
FannOrFlop	244	10.5	32.8	0.368	0.254	0.461
Diwan	258	12.8	2.7	0.151	0.360	n/a

Table 1: The query-abstraction spectrum. % quote is the share of queries sharing a verbatim three-token span with the target poem;

C

is mean query-document containment against verses, AI explanations and FoF explanations. Thematic queries (

n{=}200

) have no target poem and are used only in Track B.

	FannOrFlop (tip-of-the-tongue)			Diwan (abstract)
System	verses	AI expl.	FoF expl.	verses	AI expl.
Qwen3-8B	0.717	0.656	0.816 ∗	0.748	0.930 ∗∗∗
BGE-M3	0.693	0.598	0.725	0.574	0.845 ∗∗∗
mE5-large	0.578	0.492	0.717 ∗∗	0.585	0.802 ∗∗∗
jina-v3	0.647	0.504 ∗∗	0.676	0.523	0.756 ∗∗∗
silma-sts	0.566	0.508	0.717 ∗∗∗	0.558	0.810 ∗∗∗

Table 2: Track A, Recall@5, one pipeline throughout. Stars mark a significant change from that system’s verse baseline (paired exact McNemar, Holm-corrected across the ten systems within each corpus and contrast):

{}^{*}p{<}.05

{}^{**}p{<}.01

{}^{***}p{<}.001

. Confidence intervals in Appendix H . BM25 ranks first on Fann or Flop and eighth on Diwan ; augmentation reverses sign between the two.

	half-verse added, $n{=}164$			rewritten
System	opening	middle	final	$n{=}79$
Qwen3-8B	${+}.232$	${+}.207$	${+}.128$	${-}.190$
BGE-M3	${+}.287$	${+}.207$	${+}.055$	${-}.266$
mE5-large	${+}.348$	${+}.183$	${+}.079$	${-}.203$
jina-v3	${+}.262$	${+}.140$	${-}.018$	${-}.253$
silma-sts	${+}.274$	${+}.256$	${+}.146$	${-}.139$

Table 3: Change in Recall@5 on Fann or Flop when the same queries are edited. Left: one verbatim half-verse of the target poem appended to each non-quoting query, taken from the opening, a middle or the final verse. Right: the quoted words of the quoting queries rewritten by hand at equal length. Last row: BM25’s change minus Qwen3-8B’s (paired bootstrap;

{}^{**}p{<}.01

{}^{***}p{<}.001

	FannOrFlop R@5		slope		$\rho$
System	quote	no-qt.	$\Delta$	vs. ref	lex.	emb.
	$n{=}80$	$n{=}164$
Qwen3-8B	0.863	0.646	0.22	ref.	0.17	0.32
BGE-M3	0.875	0.604	0.27	$+0.06$	0.22	0.35
mE5-large	0.713	0.512	0.20	$-0.02$	0.25	0.46
jina-v3	0.787	0.579	0.21	$-0.01$	0.29	0.45

Table 4: Left: verse-mode Recall@5 on Fann or Flop , stratified by verbatim quotation, with slope

\Delta

and its difference against Qwen3-8B (paired bootstrap). Right: within-query Spearman

\rho

between overlap change and retrieval change ( Diwan , AI explanations); bold marks the better predictor.

\dagger

gte’s

\rho_{\mathrm{emb}}

is not independent of the measure (Appendix C ).

Figure 1: Top: Verse-mode Recall@5 rises monotonically with query-verse containment on both corpora; AI explanations flatten the curve on Diwan . Bottom: The same systems on Fann or Flop , split by whether the query quotes the poem.

	FannOrFlop		Diwan
System	Track A	Track B	Track A	Track B
Qwen3-8B	0.717	2.94	0.748	2.83
BGE-M3	0.693	2.47	0.574	2.29
mE5-large	0.578	2.52	0.585	2.35
jina-v3	0.647	2.64	0.523	2.53
silma-sts	0.566	2.75	0.558	2.63

Table 5: Track A (known-item Recall@5) beside Track B (thematic judge score, 1 to 5), verse mode,

n{=}200

queries per cell with no judge failures. The lower block gives the first rater’s blind human relevance of the top-3 (0 to 2) for the three annotated systems.

Figure 2: The dissociation. Each point is a system;

x

is known-item Recall@5,

y

is the thematic judge score. Top: on Fann or Flop , BM25 (red diamond) is the best system on one axis and the worst on the other. Bottom: on Diwan , where no query carries remembered wording, the two protocols broadly agree.

	FannOrFlop			Diwan
Index configuration	all	quote	no-qt.	all
verses only	0.717	0.863	0.646	0.748
AI expl. only	0.656	0.713	0.628	0.930
FoF expl. only	0.816	0.812	0.817	n/a
vrs. $\oplus$ AI expl.	0.730	0.800	0.695	0.857
vrs. $\oplus$ FoF expl.	0.824	0.912	0.780	n/a

Table 6: Reciprocal-rank fusion over index configurations, Qwen3-8B.

\oplus

denotes RRF (

k{=}60

). Fusing explanations is safe everywhere; adding a lexical arm helps only where queries carry remembered wording.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Within-query

\rho

per system under a lexical and an embedding overlap measure.

	$\geq 2$		$\geq 3$		$\geq 4$		$\geq 5$
System	qt.	no-qt.	qt.	no-qt.	qt.	no-qt.	qt.	no-qt.
	146	98	80	164	40	204	24	220
Qwen3-8B	0.815	0.571	0.863	0.646	0.825	0.696	0.833	0.705
BGE-M3	0.856	0.449	0.875	0.604	0.950	0.642	0.958	0.664
mE5-large	0.678	0.429	0.713	0.512	0.775	0.539	0.917	0.541
jina-v3	0.801	0.418	0.787	0.579	0.850	0.608	0.917	0.618

Appendix

Table 7: Verse-mode Recall@5 on Fann or Flop , split at verbatim spans of

\geq

2 to

\geq

5 tokens. Subset sizes are given under each threshold.

Metric	BM25	Qwen3-8B	difference	95% CI	$p$	$p_{\mathrm{Holm}}$
	slope $\Delta$
R@1	0.419	0.237	$+0.182$	[0.040, 0.325]	0.009	0.085
R@5	0.378	0.216	$+0.162$	[0.046, 0.285]	0.005	0.045
MRR	0.389	0.229	$+0.160$	[0.045, 0.277]	0.006	0.050
nDCG@10	0.371	0.231	$+0.140$	[0.035, 0.247]	0.009	0.079

Appendix

Table 8: Quotation slope (quoting minus non-quoting subset) on Fann or Flop , verse mode, and BM25’s difference against Qwen3-8B (paired bootstrap,

10^{4}

resamples).

Fann or Flop
System	Mode	R@1	R@5	MRR	nDCG@10
Qwen3-8B	verses	0.566 [0.504,0.627]	0.717 [0.660,0.770]	0.635 [0.581,0.688]	0.667 [0.615,0.718]
Qwen3-8B	AI expl.	0.426 [0.365,0.488]	0.656 [0.594,0.713]	0.530 [0.475,0.584]	0.570 [0.517,0.622]
Qwen3-8B	FoF expl.	0.639 [0.578,0.701]	0.816 [0.766,0.865]	0.725 [0.677,0.773]	0.754 [0.710,0.800]
BGE-M3	verses	0.574 [0.512,0.635]	0.693 [0.635,0.750]	0.634 [0.578,0.689]	0.657 [0.603,0.712]
BGE-M3	AI expl.	0.406 [0.344,0.467]	0.598 [0.537,0.660]	0.489 [0.434,0.544]	0.515 [0.459,0.570]

Appendix

Table 9: Track A in full, with bootstrap 95% confidence intervals.

	FannOrFlop				Diwan
System	Gem.	Gem-3	Q-orig	Q-neut	Gem.	Gem-3	Q-orig	Q-neut
Qwen3-8B	2.83	3.36	3.17	3.35	2.66	3.29	3.12	3.32
BGE-M3	2.49	3.09	2.80	2.99	2.33	3.00	2.77	2.98
mE5-large	2.48	3.23	3.03	3.11	2.36	3.18	2.91	3.06
jina-v3	2.51	3.19	2.86	3.10	2.48	3.18	3.00	3.11
silma-sts	2.73	3.23	2.99	3.24	2.55	3.17	3.00	3.28

Appendix

Table 10: Track B overall quality on the 100 thematic queries scored by every judge, verse mode: Gemini 3.5 Flash (Gem.), Gemma-3-27B-it (Gem-3), and Qwen3-32B under the original (Q-orig) and neutral (Q-neut) rubrics. The last row gives Kendall

\tau

of each system ordering with Gemini’s.

Corpus	System	sem. rel.	them. coh.	metaph.	coverage	lit. match	prec.	overall
FoF	Qwen3-8B	3.00	2.70	2.94	3.08	3.27	2.80	2.94
FoF	BGE-M3	2.54	2.33	2.55	2.58	3.25	2.37	2.47
FoF	mE5-large	2.60	2.46	2.66	2.58	3.28	2.37	2.52
FoF	jina-v3	2.69	2.48	2.73	2.75	3.15	2.52	2.64
FoF	silma-sts	2.85	2.57	2.84	2.85	2.90	2.62	2.75
FoF	gte	2.42	2.21	2.38	2.46	3.40	2.22	2.37

Appendix

Table 11: Track B by judge aspect, verse mode,

n{=}200

queries per cell. All aspects are scored from 1 to 5 with 5 highest; on lit. match a high score means the results repeat the query’s wording, which the rubric treats as evidence against implicit understanding. Lexical overlap can coexist with genuine relevance, and no claim in the paper rests on this aspect: the ordering from the aboutness aspects alone matches the overall one.

	FannOrFlop		Diwan
System	judge 1	judge 2	judge 1	judge 2
Qwen3-8B	2.83	3.36	2.68	3.29
BGE-M3	2.49	3.09	2.34	3.00
mE5-large	2.48	3.23	2.36	3.18
jina-v3	2.51	3.19	2.48	3.18
silma-sts	2.73	3.23	2.56	3.17

Appendix

Table 12: Track B overall quality under the primary judge (Gemini 3.5 Flash,

n{=}200

) and the open-weights second judge (Gemma-3-27B-it,

n{=}100

). Scales differ; the orderings do not (Kendall

\tau{=}{+}0.85

and

{+}0.81

Aspect	cell-level $\rho$
semantic relevance	+0.71
thematic coherence	+0.64
metaphor understanding	+0.26
coverage	+0.68
literal match	+0.29
precision	+0.65

Appendix

Table 13: Cell-level Spearman agreement between the two judges,

n{=}2{,}000

paired cells.

Lexical Retrieval

Lexical retrieval ranks documents by query-term overlap using methods such as TF-IDF and BM25.

See all 27 papers

Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies
May 28, 2026Benjamin Clavié, Sean Lee, Aamir Shakir +1Learned Sparse Retrieval Information Retrieval
Mixedbread AI · National Institute of Informatics (NII) · University of Tsukuba
CeQe: Grounding Lexical Retrieval in Semantic Evidence
Aug 1, 2026Adam Kahirov, Umesh Deshpande, Swaminathan SundararamanLexical Retrieval Information Retrieval
IBM Research
STORM: Stepwise Token Optimization with Reward-Guided Beam Search
Jun 9, 2026Arthur Satouf, Giulio D'Erasmo, Yuxuan Zong +3Language Model Decoding Lexical Retrieval
MILA – Quebec AI Institute & ILLS, Canada · 5Sapienza, University of Rome, Italy · 4Sorbonne Université & ISIR & CNRS, France +2

May 28, 2026J/K move · Enter open · S save

Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies

Benjamin Clavié, Sean Lee, Aamir Shakir, Makoto P. Kato

Mixedbread AI · National Institute of Informatics (NII) · University of Tsukuba

Learned Sparse Retrieval Information Retrieval Dense Retrieval

Open paper AskPDF arXiv

We propose Latent Terms, a method revealing that models trained for dense retrieval, whether single- or multi-vector, learn representations that can trivially be decomposed into retrieval-ready sparse features. When trained on frozen retrievers, Sparse Autoencoders without any retrieval-specific adjustments extract a latent vocabulary with approximately Zipfian collection statistics, directly suitable for classical sparse retrieval scoring via BM25. This approach enables sparse retrieval while requiring no learned expansion objective or sparse retrieval supervision whatsoever, and can be readily applied to any dense retriever. Latent Terms is able to match or outperform single-vector scoring methods from its own base model as well as comparable SPLADE variants. In addition, it substantially outperforms its base model on LIMIT, a task specifically designed to highlight the failures of single-vector retrieval. Overall, our results highlight that neural retrievers contain more expressive and indexable structure than their default scoring functions expose, but that other methods can nonetheless be leveraged.

Abstract

Lexical Retrieval

Explore similar work

Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies

CeQe: Grounding Lexical Retrieval in Semantic Evidence

STORM: Stepwise Token Optimization with Reward-Guided Beam Search