Mycelium-Index: A Streaming Approximate Nearest Neighbor Index with Myelial Edge Decay, Traffic-Driven Reinforcement, and Adaptive Living Hierarchy

Anton Pakhunov

cs.LGApr 13, 2026

Mycelium-Index: A Streaming Approximate Nearest Neighbor Index with Myelial Edge Decay, Traffic-Driven Reinforcement, and Adaptive Living Hierarchy

Authors: Anton Pakhunov

Organizations: Independent Researcher, Haifa, Israel

Abstract

We present mycelium-index, a streaming approximate nearest neighbor (ANN) index for high-dimensional vector spaces, inspired by the adaptive growth patterns of biological mycelium. The system continuously adapts its topology through myelial edge decay and reinforcement, a traffic-driven living hierarchy, and hybrid deletion combining O(1) bypass for cold nodes with O(k) beam-search repair for hub nodes. Experimental evaluation on SIFT-1M demonstrates that mycelium achieves 0.927 +/- 0.028 recall@5 under FreshDiskANN's 100%-turnover benchmark protocol -- within the measurement confidence interval of FreshDiskANN's ~0.95 -- while using 5.7x less RAM (88 MB vs. >500 MB) and achieving 4.7x higher QPS (2,795 vs. ~600). On the static index, at ef=192, mycelium matches HNSW M=16 recall (0.962 vs. 0.965) at 5.2x less RAM (163 MB vs. 854 MB). Performance optimizations including NEON SIMD distance computation, Vec-backed node storage, and bitset visited tracking yield a cumulative 2.7x QPS improvement. A systematic study of ten streaming repair mechanisms finds that geometric heuristics universally fail in high dimensions, while topological mechanisms succeed -- a principle we term the topological repair invariance of high-dimensional ANN graphs.

Figures & tables

Method	ef	Recall@10	QPS	RAM (MB)	Streaming
HNSW M=16	64	0.942	1,941	854	No
HNSW M=16	128	0.965	1,170	854	No
HNSW M=8	128	0.876	1,890	671	No
HNSW M=8	200	0.896	1,170	671	No
HNSW M=6	200	0.813	1,732	626	No
HNSW M=4	200	0.646	2,281	580	No

Table 1: Recall–throughput–memory comparison on SIFT-1M. RAM for HNSW:

n\times d\times 4+n\times M\times 2\times 12

bytes. All benchmarks single-threaded.

Configuration	Recall@10	QPS
Base graph, random entry	0.774	843
+ Living hierarchy	0.938	843
+ Bypass reconnection	0.938	843
+ Hybrid deletion	0.938	843

Table 2: Component ablation on SIFT-1M at ef=128.

Optimization	QPS	Cumul.
Baseline (HashSet, HashMap, scalar)	742	$1.0\times$
+ Remove .collect() + #[inline]	1,287	$1.7\times$
+ Vec-backed NodeStore	1,414	$1.9\times$
+ NEON SIMD + compact NodeMeta	2,006	$\mathbf{2.7\times}$

Table 3: Performance optimization progression, ef=128. Recall unchanged (0.938) across all steps.

Iter	Mycelium	Tombstone	$\Delta$	Edge store
0	0.858	0.912	$-$ 0.054	—
10	0.908	0.840	+0.068	—
50	0.838	0.640	+0.198	2.3 $\to$ 112 MB
100	0.704	0.608	+0.096	1.2 $\to$ 105 MB
150	0.608	0.526	+0.082	972 $\to$ 102 MB
200	0.684	0.442	+0.242	732 $\to$ 101 MB

Table 4: Streaming deletion comparison, 500K active vectors.

Size	Before ( $\mu$ s)	After ( $\mu$ s)	Speedup
100K	386	467	$0.8\times$
200K	1,119	731	$1.5\times$
500K	3,726	1,390	$2.7\times$
1M	5,290	1,835	$\mathbf{2.9\times}$

Table 5: Mean query latency (

\mu

s) vs. dataset size.

#	Mechanism	Recall@5 (cycle 20)	Degree	Verdict
1	Baseline (no hierarchy refresh)	0.122	13.4	Hierarchy staleness
2	+ Per-cycle hierarchy refresh	0.738	13.4	Best—fixes navigation
3	+ Adaptive ef (auto-tuned beam)	0.682	13.3	Marginal improvement
4	+ Physarum flow-weighted bypass	0.020	13.2	Hot edges dominate beam
5	+ Naive anastomosis (edge sharing)	0.496	14.3	Geometrically wrong edges
6	+ Greedy multi-bypass (all neighbors)	0.280	15.8	More bad edges = worse

Table 6: Systematic evaluation of streaming repair mechanisms. SIFT-1M, 20-cycle FreshDiskANN protocol, 100% total turnover.

Cycle	Mycelium	FreshDiskANN	Gap	Hub%	RAM
0	0.966	${\sim}$ 0.95	+0.02	—	84
5	0.921	${\sim}$ 0.95	$-$ 0.03	26%	92
10	0.926	${\sim}$ 0.95	$-$ 0.02	29%	97
15	0.924	${\sim}$ 0.95	$-$ 0.03	31%	101
20	0.927	${\sim}$ 0.95	$\mathbf{-0.02}$	40%	88

Table 7: Final streaming recall trajectory (

\pm

0.028 CI).

Constraint removed	Recall@5 (cycle 10)	$\Delta$ from unconstrained
Unconstrained (k=16, ef=200, $O(k)$ all, f32, $\infty$ RAM)	0.988	—
k=8 (from 16)	0.975	$-$ 0.013
+ ef=128 (from 200)	0.955	$-$ 0.033
+ Hybrid delete (from $O(k)$ all)	0.927	$-$ 0.061
+ Scalar quantization (u8)	0.927	$-$ 0.061
+ 163 MB RAM limit	0.927	$-$ 0.061

Table 8: Constraint ablation on SIFT-1M, cycle 10. Starting from unconstrained recall (k=16, ef=200,

O(k)

all deletions, f32 vectors, unlimited RAM), constraints are added one by one.

	Mycelium (streaming)	Mycelium (static)	FreshDiskANN	HNSW M=16
Recall@5 (100% turnover)	$\mathbf{0.927\pm 0.028}$	—	${\sim}0.95$	N/A
Recall@10 (static)	—	0.962	—	0.965
RAM	88 MB	163 MB	$>$ 500 MB	854 MB
QPS	2,795	2,006	${\sim}600$	1,170
Delete cost	$O(1)$ 90% / $O(k)$ 10%	—	$O(k)$ all	N/A
Streaming	Yes	—	Yes	No

Table 9: Unified comparison across systems and workloads.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value	Default	Sensitivity
Graph construction
k_grow	8	8	8–16; higher improves recall, costs memory
k_local	6	6	Subset of k_grow
k_long_range	2	2	Kleinberg shortcuts; 0 in streaming
long_range_alpha	2.0	2.0	Optimal for high-dim data
max_edges_per_node	64	64	Effectively unlimited

Appendix

Table 10: Full parameter table for SIFT-1M experiments.

Approximate Nearest Neighbor Search

Approximate nearest neighbor search finds close vectors without exhaustive comparisons, enabling scalable retrieval in high-dimensional datasets.

See all 42 papers

Panorama: Fast-Track Nearest Neighbors
Oct 1, 2025Vansh Ramani, Alexis Schlomer, Akash Nayar +3Product Quantization Approximate Nearest Neighbor Search
IIT Delhi & CMU & UCPH · Carnegie Mellon University · Indian Institute of Technology Delhi +1
Aperon Technical Report: Hierarchical No-Pointer Tangent-Local Search for High-Dimensional Approximate Nearest Neighbors
Jun 7, 2026Yong FuLocal Search Approximate Nearest Neighbor Search
Substratum Labs
Approximate Nearest Neighbor Search for Modern AI: A Projection-Augmented Graph Approach
Mar 1, 2026Kejing Lu, Zhenpeng Pan, Jianbin Qin +2Approximate Nearest Neighbor Search Graph-Based Retrieval
University of Yamanashi · Beijing Institute of Technology, Zhuhai · Shenzhen University +2

Oct 1, 2025J/K move · Enter open · S save

Panorama: Fast-Track Nearest Neighbors

Vansh Ramani, Alexis Schlomer, Akash Nayar, Sayan Ranu+2

IIT Delhi & CMU & UCPH · Carnegie Mellon University · Indian Institute of Technology Delhi+1

Product Quantization Approximate Nearest Neighbor Search

Open paper AskPDF arXiv

Approximate Nearest-Neighbor Search (ANNS) pipelines for high-dimensional neural embeddings spend the bulk of their query time in candidate verification, making it the primary bottleneck in the search process. In this paper, we present PANORAMA, a state-of-the-art refinement technique that accelerates verification by exploiting the inherent spectral decay of these embeddings. Using PCA to compact signal energy, PANORAMA evaluates candidate distances incrementally, computing at each step a strict lower bound on the full-vector distance and dynamically pruning candidates the moment this bound exceeds the running k-th nearest neighbor distance. While PCA's concentration of variance facilitates pruning, it breaks the uniform-variance assumption required by Product Quantization (PQ); we resolve this with a variance-shaping step that redistributes energy across subvectors, rendering accretive refinement compatible with quantized indexes. Optimized for modern memory hierarchies via vectorized bulk-pruning and cache-conscious data layouts, PANORAMA has been upstreamed into the FAISS library across major index families (IVFPQ/Flat, HNSW, and Refine). PANORAMA achieves higher QPS at any target recall with a cost that provably scales inversely with dataset spectral decay, delivering end-to-end speedups of up to 28.9x and outperforming probabilistic methods across all recall bands.

Abstract

Approximate Nearest Neighbor Search

Explore similar work

Panorama: Fast-Track Nearest Neighbors

Aperon Technical Report: Hierarchical No-Pointer Tangent-Local Search for High-Dimensional Approximate Nearest Neighbors

Approximate Nearest Neighbor Search for Modern AI: A Projection-Augmented Graph Approach