An In-Depth Study of Filter-Agnostic Vector Search on a PostgreSQL Database System: [Experiments and Analysis]

Duo Lu; Helena Caminal; Manos Chatzakis; Yannis Papakonstantinou; Yannis Chronis; Vaibhav Jain; Fatma Özcan

cs.DBMar 24, 2026

An In-Depth Study of Filter-Agnostic Vector Search on a PostgreSQL Database System: [Experiments and Analysis]

Authors: Duo Lu, Helena Caminal, Manos Chatzakis, Yannis Papakonstantinou, Yannis Chronis, Vaibhav Jain, Fatma Özcan

Organizations: Brown University, USA · Google, USA · Université Paris Cité, LIPADE, France · ETH Zurich & Google, Switzerland ·

Abstract

Filtered Vector Search (FVS) is critical for supporting semantic search and GenAI applications in modern database systems. However, existing research most often evaluates algorithms in specialized libraries, making optimistic assumptions that do not align with enterprise-grade database systems. Our work challenges this premise by demonstrating that in a production-grade database system, commonly made assumptions do not hold, leading to performance characteristics and algorithmic trade-offs that are fundamentally different from those observed in isolated library settings. This paper presents the first in-depth analysis of filter-agnostic FVS algorithms within a production PostgreSQL-compatible system. We systematically evaluate post-filtering and inline-filtering strategies across a wide range of selectivities and correlations. Our central finding is that the optimal algorithm is not dictated by the cost of distance computations alone, but that system-level overheads that come from both distance computations and filter operations (like page accesses and data retrieval) play a significant role. We demonstrate that graph-based approaches (such as NaviX/ACORN) can incur prohibitive numbers of filter checks and system-level overheads, compared with clustering-based indexes such as ScaNN, often canceling out their theoretical benefits in real-world database environments. Ultimately, our findings provide the database community with crucial insights and practical guidelines, demonstrating that the optimal choice for a filter-agnostic FVS algorithm is not absolute, but rather a system-aware decision contingent on the interplay between workload characteristics and the underlying costs of data access in a real-world database architecture.

Figures & tables

Figure 1. Average latency for 100 filtered vector search queries on OpenAI-1M: HNSWLib (solid) vs. PGVector (dashed). System latency can be up to 10× higher, and the gap varies with selectivity, motivating an in-depth comparison of FVS algorithms in a real DBMS.

Arch. Aspect	Library	System (PGVector)	System overheads
Index Storage	Compact representation: Neighbors stored as raw pointers. Nodes are packed contiguously with no alignment constraints.	Tuple-based storage: Each neighbor reference is a 6-byte TID. Nodes are stored as tuples, with header overhead, in 8KB pages.	Space amplification: Page-level overhead (headers, alignment) reduces effective storage density, impacting cache efficiency.
Node Access	Direct deference: The neighbor access is a CPU-level memory dereference.	Scattered page access: The neighbors are stored as TIDs pointing to arbitrary pages.	Random I/O amplification: Each hop multiplies page overhead by neighbor fanout.
Filter Evaluation	Unified identifier: A single ID/offset indexes both vector data and attribute data in memory.	Indirection: Index page $\rightarrow$ HeapTID $\rightarrow$ Heap page. Filter columns reside in the heap. Evaluating predicates needs HeapTID.	Double lookup: Every filtered candidate requires: (1) index access for vector+HeapTID, (2) heap access for filter columns.
Parallelism	Flexible: Intra-query parallelism via OpenMP, std::thread , or SIMD.	Constrained: One query per connection. No batch query primitive.	Arch. limit: Executor/storage hinders fine-grained parallelism and batching.

Table 1. System breakdown for HNSW: architectural differences between standalone libraries and database systems.

Figure 2. Execution strategies for an FVS query.

Figure 3. Filter-first execution methods for an FVS query.

Figure 4. Traversal-first execution methods for an FVS query.

Figure 5. Execution methods for an FVS query on ScaNN.

Figure 6. NaviX, ACORN, Sweeping, and Iterative Scan implementations in PGVector.

Figure 7. Filtered ScaNN implementation in the system.

Figure 8. Illustration of the filter generator methodology. A Positive Correlation workload is then generated by preferentially sampling vectors from the closest neighbors, conversely, for the Negative Correlation.

Dataset	# Vec	Dims	Metric	Source	Dist.-Filt. Rel. Cost	LID/LRC
sift10M ( Jégou et al., 2011 )	10M	128	L2	Image Descriptors	0.8x	19.11/0.82
openai5M ( , 2025 )	5M	1536	IP	Text Embeddings	6x	33.34/0.97
cohere10M ( , 2022 )	10M	768	L2	Text Embeddings	3.3x	38.63/0.97
text2image10M ( Simhadri et al., 2022 )	10M	200	L2	Multimodal Emb.	1.3x	52.47/0.98

Table 2. Comparison and characteristics of benchmark datasets.

	Time (min)		Size (GB)
Dataset	HNSW	ScaNN	HNSW	ScaNN
sift10M	24.80	1.87	9.58	1.73
openai5M	49.10	10.36	38.15	1.89
cohere10M	72.83	18.24	38.14	1.95
text2image10M	20.35	2.84	12.72	2.35

Table 3. Index build time and size comparison.

Dataset	Method	QPS	Index Size	Build Time
sift10M	Halfvec	0.97 $\times$	1.34 $\times$	1.01 $\times$
sift10M	BQ+rerank	–*	2.00 $\times$	1.02 $\times$
openai5M	Halfvec	1.03 $\times$	2.00 $\times$	1.11 $\times$
openai5M	BQ+rerank	0.75 $\times$	11.63 $\times$	2.86 $\times$
cohere10M	Halfvec	1.04 $\times$	2.00 $\times$	1.24 $\times$
cohere10M	BQ+rerank	0.96 $\times$	11.83 $\times$	2.33 $\times$

Table 4. QPS speedup and reduction in index size and build time of 1,000 unfiltered vector search queries in PGVector-HNSW with 16 threads (Microbenchmark 95% Recall@10).

Dataset	Method	1% sel.	5% sel.	20% sel.	50% sel.	80% sel.
sift10M	Quant.	1.40 $\times$	1.42 $\times$	1.56 $\times$	1.49 $\times$	1.58 $\times$
openai5M	PCA	29.39 $\times$	5.79 $\times$	3.92 $\times$	3.22 $\times$	3.62 $\times$
openai5M	PCA+Quant.	50.21 $\times$	8.61 $\times$	6.06 $\times$	5.02 $\times$	5.17 $\times$
cohere10M	PCA	5.24 $\times$	4.47 $\times$	3.48 $\times$	3.76 $\times$	3.83 $\times$
cohere10M	PCA+Quant.	9.60 $\times$	7.27 $\times$	6.55 $\times$	6.48 $\times$	5.52 $\times$
text2image10M	Quant.	2.17 $\times$	1.51 $\times$	1.45 $\times$	1.87 $\times$	1.79 $\times$

Table 5. Latency speedup of different settings w.r.t. non-quantized non-PCAed ScaNN on 95% recall@10 with 16 threads (no correlation). Columns show increasing selectivity. Quantization uses SQ8.

Figure 9. QPS at 95% Recall@10 for methods across various selectivities on four datasets with no vector-predicate correlation.

Distance Computations

Filter Checks

Hops / (Leaves & Reordering)

Page Access

Sel.

NaviX

ACORN

Sweep.

ScaNN

NaviX

ACORN

Sweep.

ScaNN

NaviX

ACORN

Sweep.

ScaNN

NaviX

ACORN

Sweep.

ScaNN

246

238

23.0K

1.4K

71.8K

71.6K

2.6K

143.9K

17.8

17.7

1.1K

150/62

1.2K

24.0K

4.5K

357

408

9.4K

1.9K

66.6K

59.6K

1.4K

96.0K

19.8

20.9

398

100/60

1.5K

1.6K

9.8K

3.2K

627

614

5.0K

3.8K

36.2K

34.9K

910

76.9K

13.0

23.0

197

80/60

1.1K

1.4K

5.2K

2.8K

10%

886

1.1K

3.3K

4.8K

24.5K

40.4K

359

48.2K

12.8

21.9

107

50/95

1.2K

1.9K

3.4K

2.2K

20%

1.2K

2.2K

9.7K

12.6K

14.6K

219

48.2K

13.2

32.3

64.3

50/80

1.4K

1.8K

2.3K

2.1K

Table 6. Breakdown of internal index statistics across varying selectivities on OpenAI-5M (no correlation). Colors indicate relative performance within each metric group (green=lower/better, red=higher/worse). Bold indicates the best value per row.

Figure 10. Latency breakdown (CPU cycles) of a query running for 1, 10, 50, and 80% selectivities on the OpenAI-5M dataset without correlation. Bars (1% sel. for Sweeping) are clipped at the plot’s y-axis limit for readability; annotations “True: X” report the full (unclipped) cycle count.

Figure 11. Varying LIMIT

k

on all datasets (no correlation).

Figure 12. QPS v.s. Selectivity on OpenAI-5M, showing how vector-predicate correlation alters the performance and crossover point of each FVS algorithm.

Figure 13. Latency breakdowns with/without a translation map on the OpenAI-5M dataset (no correlation).

Method	Threads	Total Cycles	SysOH%	DistComp%	Filter%
NaviX	1T	24.1M	55.9%	13.0%	12.3%
NaviX	16T	35.6M (+48%)	73.5%	7.5%	9.5%
Sweep.	1T	45.0M	81.8%	18.6%	0.2%
Sweep.	16T	75.7M (+68%)	91.0%	11.9%	0.2%
ScaNN	1T	20.7M	84.4%	3.1%	12.5%
ScaNN	16T	32.9M (+59%)	86.9%	2.5%	10.5%

Table 7. Comparison of average CPU cycles breakdown between single- v.s. multiple- threads (1T v.s. 16T) at 10% selectivity on OpenAI-5M with 95% recall@10 (no correlation).

Approximate Nearest Neighbor Search

Approximate nearest neighbor search finds close vectors without exhaustive comparisons, enabling scalable retrieval in high-dimensional datasets.

See all 42 papers

Policy-aware Vector Search: A Vision for Fine Grained Access Control in Vector Databases
Jun 18, 2026Lakshmi Sahithi Yalamarthi, Primal PappachanAccess Control
Portland State University Portland, Oregon, USA
QASP: Query-Adaptive Robust Vector Search Policy
Jul 31, 2026Hakan Ferhatosmanoglu, Kushal Kumar, Tal Wagner +1Approximate Nearest Neighbor Search
Amazon, London, UK · Amazon, New York, USA · Amazon and Tel-Aviv University, Tel Aviv, Israel +1
TurboVec: A Case Study in Cost-Efficient Private Retrieval for Enterprise RAG via Codebook-Oblivious Quantization
Jul 18, 2026Navnit Shukla, Kamal Pandey, Omsankar TiwariRetrieval-Augmented Generation Vector Quantization
Snowflake Inc. Tustin, California, USA · Rivian Automotive Inc. Irvine, United States of America · Google Deepmind Sunnyvale, California, USA

Jun 18, 2026J/K move · Enter open · S save

Policy-aware Vector Search: A Vision for Fine Grained Access Control in Vector Databases

Lakshmi Sahithi Yalamarthi, Primal Pappachan

Portland State University Portland, Oregon, USA

Access Control

Open paper AskPDF arXiv

Vector databases are increasingly used in security sensitive contexts with Retrieval Augmented Generation and organizational AI pipelines; however, their security capabilities remain limited. Specifically, Fine-grained Access Control (FGAC) which is required to ensure that data access adheres to user-specific policies is not fully supported in modern vector databases. Unlike relational databases, vector databases combine structured and unstructured attributes to provide semantic, approximate query results, which complicates FGAC implementation. This creates an inherent tension between enforcing FGAC policies correctly, achieving high ANN search recall and maintaining low query latency. In this paper, we present a vision for Policy-aware Vector Search by formalizing the FGAC policy model in vector databases as well as the enforcement problem. We compare various enforcement strategies, present preliminary findings, and identify key open challenges for future research in policy-aware vector search.

Abstract

Approximate Nearest Neighbor Search

Explore similar work

Policy-aware Vector Search: A Vision for Fine Grained Access Control in Vector Databases

QASP: Query-Adaptive Robust Vector Search Policy

TurboVec: A Case Study in Cost-Efficient Private Retrieval for Enterprise RAG via Codebook-Oblivious Quantization