Peeling the Matryoshka: Extracting Ontological Structure from Retrieval Embeddings

Eunchan Kim; Inwhee Joe

cs.CLOct 30, 2026

Peeling the Matryoshka: Extracting Ontological Structure from Retrieval Embeddings

Authors: Eunchan Kim, Inwhee Joe

Organizations: Hanyang University

Abstract

Taxonomy construction is essential for knowledge-intensive NLP yet remains costly: clustering-based methods (i) need careful per-corpus tuning and yield unstable hierarchies, while LLM-based pipelines (ii) depend on expensive task-specific training that does not transfer across domains. We report an empirical finding that bears on both: in Matryoshka Representation Learning (MRL) embeddings, where each leading prefix is itself a usable, coarser vector, prefix length aligns with the coarse-to-fine granularity of gold taxonomies; shuffled- and reverse-dimension controls isolate the alignment to the learned early prefix dimensions. The pattern holds across three architecturally distinct MRL models. To test whether this regularity is usable, we introduce PEEL (Peeling Embeddings to Extract Latent ontologies): a minimal probe over these frozen embeddings, trained once with an order-embedding objective that represents hierarchy as vector containment; a minimum spanning arborescence then decodes taxonomic DAGs. Using only a concept vocabulary (no documents), PEEL recovers taxonomic DAGs with high structural fidelity on two OLLM benchmarks, transferring to a new domain with no target-edge supervision. The end-to-end OLLM pipeline, which jointly discovers concepts and induces structure from document corpora, is reported as a cross-task reference. Pre-trained retrieval embeddings encode a decodable granularity gradient; a minimal probe renders it as explicit parent-child (is-a) structure.

Figures & tables

Model	Data	Depth 1	Depth 2	Depth 3
Nomic (137M)	wiki	.058	.159	.364
jina-v3 (572M)	wiki	.067	.181	.400
Qwen3 (600M)	wiki	.043	.131	.337
Nomic (137M)	arXiv	.039	.122	—
jina-v3 (572M)	arXiv	.078	.205	—
Qwen3 (600M)	arXiv	.039	.131	—

Table 1: ANOVA effect sizes (

\eta^{2}

) for kNN label purity by taxonomy depth, reported as a supporting cross-model consistency check. arXiv has max depth 2. The depth ordering holds in every row. Because gold class counts differ sharply across depths (Wikipedia: 33/875/7124 at Depth 1/2/3), these values index consistency, not an absolute cross-depth magnitude. Cohen’s benchmarks:

.01

small,

.06

medium,

.14

large ( Cohen, 1988 ) .

Figure 1: Relative advantage of MRL prefix truncation over two controls, by embedding dimension, on Wikipedia and arXiv at the deepest gold taxonomic level.

y=\log_{2}(\text{MRL}\div\text{control kNN purity})

;

y{=}0

is parity with the control. Solid

=

vs. shuffled control (dimension ordering destroyed); dotted

=

vs. reverse control (last-

k

tail dimensions). Colors denote three architecturally distinct MRL models (Nomic, jina-v3, Qwen3). All curves rise above parity and peak within the early dimensions; at full dimensionality the prefix equals the complete embedding, so the controls coincide by construction.

Figure 2: The PEEL pipeline. Training (top): frozen MRL embeddings are projected through a probe (98K parameters) trained with order embedding loss under random dimensional truncation. Inference (bottom): the trained probe maps

N

concept embeddings to a 128-d order-embedding space; a pairwise violation matrix is constructed and resolved by Chu-Liu/Edmonds’ algorithm into a threshold-free tree, then expanded into a directed acyclic graph via near-miss edges.

Concept-only methods (no document access)
Dataset	Method	Input	Lit.F1 $\uparrow$	Fuz.F1 $\uparrow$	Cont.F1 $\uparrow$	Grph.F1 $\uparrow$	Motif.D $\downarrow$
wiki-ol	Memorisation	training edges	.134	.837	.314	.419	.063
	PEEL (ours)	concepts only	.111	.877	.551	.885	.000
Document-using methods (corpus + concepts)
wiki-ol	Hearst Patterns	corpus + concepts	.003	.538	.350	.544	.163
	Zero-shot (M-7B)	title + abstract	.007	.871	.455	.639	.341

Table 2: Main results on OLLM benchmarks, grouped by input modality. LLM-based baselines additionally consume each article’s title and abstract per the OLLM protocol ( Lo et al., 2024 , Figs. 5–6); OLLM (transfer) fine-tunes Mistral-7B on 2,048 additional arXiv samples. PEEL rows use a gold concept vocabulary and a probe trained on wiki-ol gold edges (§ 5.1 ), the arXiv-ol row zero-shot; OLLM instead discovers concepts and structure from documents, so the OLLM scores are a cross-task reference, not a ranking. Motif Dist. values below

10^{-4}

are reported as

.000

Figure 3: Mathematics subtree extracted at three prefix dimensions from the trained probe: at $d{=}64$ , only one child is recovered; at $d{=}256$ , 19 children emerge across depth 2; at $d{=}768$ , 68 children span 12 levels of depth. Increasing prefix dimension progressively unfolds fine-grained hierarchical structure.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Input prefix truncation $d$ ( $d_{\text{out}}{=}128$ fixed)
Configuration	Cont.F1 $\uparrow$	Grph.F1 $\uparrow$
$d=64$	.463	.879
$d=128$	.485	.880
$d=256$	.519	.879
$d=384$	.537	.880
$d=512$ ⋆	.551	.885

Appendix

Table 3: Input prefix truncation

d

ablation on wiki-ol. ⋆ denotes the Table 2 configuration. OLLM (reference) included for the same metrics.

$d_{\text{out}}$	Params	Cont. F1 $\uparrow$	Graph F1 $\uparrow$
32	24K	.517	.870
64	49K	.544	.878
128 ⋆	98K	.551	.885
OLLM (ref)	7B	.500	.644

Appendix

Table 4: Probe output dimension sweep on wiki-ol (

d{=}512

fixed). ⋆ denotes the Table 2 configuration.

$n_{\text{expand}}$	Cont. F1 $\uparrow$	Graph F1 $\uparrow$	Motif Dist. $\downarrow$
10,000	.503	.899	.000
12,000	.533	.893	.000
15,000 ⋆	.551	.885	.000
20,000	.500	.876	.001
25,000	.451	.868	.001
OLLM (ref)	.500	.644	.080

Appendix

Table 5: DAG expansion budget (

n_{\text{expand}}

) sensitivity on wiki-ol. ⋆ denotes the Table 2 configuration. Graph F1 is stable across all tested budgets.

$n_{\text{expand}}$	Lit-P $\uparrow$	Lit-R $\uparrow$
tree only	16.9	9.3
10,000	14.6	10.0
12,000	12.8	10.5
15,000 ⋆	11.0	11.2
20,000	9.2	12.6
25,000	8.0	13.7

Appendix

Table 6: Literal edge precision and recall by expansion budget on wiki-ol (%). ⋆ denotes the Table 2 configuration; the reported setting is balanced rather than recall-maximizing. See § 5.1 for the primary structure-sensitive metrics.

Model	Wiki	arXiv	Comb.	one-sided $p$
Nomic	26/30	16/20	42/50	$5.8{\times}10^{-7}$
jina-v3	30/33	20/22	50/55	$1.1{\times}10^{-10}$
Qwen3	29/33	20/22	49/55	$9.1{\times}10^{-10}$

Appendix

Table 7: Per-condition MRL

>

shuffled win counts (prefix

\times

depth) and combined one-sided sign test, by model and benchmark. All per-model

p<10^{-6}

Figure 4: PEEL performance by embedding dimension on wiki-ol (

d_{\text{out}}{=}128

fixed). ⋆ marks the Table 2 configuration (dim=512). Dashed lines show OLLM baselines.

Figure 5: Geography subtree at three prefix dimensions. At $d{=}64$ , only 1 child is recovered at depth 1; at $d{=}256$ , 2 children remain at depth 1; at $d{=}768$ , 38 children span depth 10. Compare with Mathematics (Figure 3 ).

Figure 6: Science subtree at three prefix dimensions. At $d{=}64$ , 3 children appear at depth 1; at $d{=}256$ , 27 children span depth 8; at $d{=}768$ , 51 children span depth 17. Together with Geography (Figure 5 ) and Mathematics (Figure 3 ), these examples confirm that prefix truncation induces qualitatively distinct hierarchical decompositions across diverse categories, consistent with the low inter-dimension Jaccard similarity ( $<.18$ ) reported in § 5.4 .

Taxonomy Induction

See all 20 papers

Apr 30, 2026

Polaris: Coupled Orbital Polar Embeddings for Hierarchical Concept Learning

Real-world knowledge is often organized as hierarchies such as product taxonomies, medical ontologies, and label trees, yet learning hierarchical representations is challenging due to asymmetric structure and noisy semantics. We introduce Polaris, a polar hyperspherical embedding framework that separates semanticity from hierarchy using angular geometry and radius, enabling the learning of meaning and structure without interference. To map latent representation onto the sphere, we project it to the tangent space at the north pole, apply the exponential map, and learn unit-norm representations using spherical linear layers. Polaris then combines robust local constraints, global regularization that prevents geometric collapse, and uncertainty-aware asymmetric objectives that encourage directional containment. At inference time, Polaris uses structure-guided retrieval to efficiently narrow down candidate parents before final ranking. We evaluate Polaris on different settings of taxonomy expansion - spanning trees, multi-parent DAGs, and multimodal hierarchies, showing consistent improvements of up to ~19 points in top-K retrieval and up to ~60% reduction in mean rank over fourteen strong baselines.

Sahil Mishra, Srinitish Srinivasan, Sourish Dasgupta +1

Indian Institute of Technology Delhi, New Delhi, India · KDM Lab, Dhirubhai Ambani University, Gandhinagar, Gujarat, India · Indian Institute of Technology Delhi, Abu Dhabi, UAE.

Hierarchical Classification Hierarchical Representation Learning Hyperspherical Representation Learning Geometric Representation Learning Knowledge Representation

Ask PDF

Sep 8, 2026

OntologyBench: Can Dense Retrieval Satisfy Structured Biomedical Constraints?

We introduce OntologyBench, a tiered biomedical retrieval benchmark comprising 471,854 training and 125,744 evaluation query-document relevance pairs across concept grounding, relational retrieval, and compositional phenotype-based retrieval. Although these tasks can be tractable using ontology-aware reference methods, across task tiers, embedding performance is generally lower on relational and compositional tasks than on concept-grounding tasks. Fine-tuning on ontology-derived supervision improves performance on several relational and compositional tasks, whereas the evaluated reranking and LLM-based candidate-scoring methods provide little or no end-to-end improvement. Errors frequently reflect diseases matching only subsets of the phenotype evidence. These findings indicate that the evaluated embedding and reranking configurations do not reliably recover the compatibility encoded by the selected ontology relations and phenotype combinations and motivate retrieval systems that better integrate learned representations with structured biomedical knowledge.

Xiao Yu Cindy Zhang, Wyeth Wasserman, Jian Zhu

Graduate Program in Bioinformatics, The University of British Columbia · Centre for Molecular Medicine and Therapeutics, BC Children’s Hospital Research Institute · Department of Medical Genetics, The University of British Columbia +1

Information Retrieval Biomedical IR IR Evaluation Dense Retrieval

Ask PDF

Aug 27, 2026

pro-team at LLMs4OL 2026 Tasks Flagship and Reuse: Retrieval-Augmented Generation and Vocabulary-Constrained Filtering for Ontology Learning

Ontology learning from text remains challenging despite significant progress in Large Language Models (LLMs), which can hallucinate domain terms, produce inconsistent formats, and favor hierarchical over associative relations. In the LLMs4OL 2026 Challenge, we address both the End-to-End Flagship Task (Task A) and Ontology Extension Reuse Task (Task B) using an offline retrieval-augmented few-shot prompting pipeline. Our system employs Qwen2.5-14B-Instruct with all-MiniLM-L6-v2 for demonstration retrieval, selecting the top-5 examples for Task A and top-2 for Task B. A left-truncated context-windowing strategy preserves task instructions within long prompts. For Task B, generated triples undergo deterministic vocabulary-constrained filtering, retaining triples when at least one endpoint belongs to the sample's closed term/type vocabulary and removing duplicates of the initial ontology. The approach achieves Semantic Graph Similarity of 0.8692, Term-Typing F1 of 0.9200, and Taxonomy Discovery F1 of 0.8540 on Task B, while Task A achieves 0.7416 Semantic Graph Similarity. However, no non-taxonomic relations are extracted, highlighting limitations of closed, taxonomy-oriented relation vocabularies.

Shivam Mishra, Dhannu Ram Meena, Muneendra Ojha +2

1AIMS Lab, Indian Institute of Information Technology Allahabad, Prayagraj, UP, India

Ontology Learning

Ask PDF

Abstract

Taxonomy Induction

Explore similar work

Polaris: Coupled Orbital Polar Embeddings for Hierarchical Concept Learning

OntologyBench: Can Dense Retrieval Satisfy Structured Biomedical Constraints?

pro-team at LLMs4OL 2026 Tasks Flagship and Reuse: Retrieval-Augmented Generation and Vocabulary-Constrained Filtering for Ontology Learning