Hybrid Product Search Without the Copy: Lexical, Semantic and Filtered Retrieval Served Directly from a Data Lake
Organizations: StarTree Inc., Mountain View, CA, USA
Abstract
E-commerce catalogs increasingly live in lakehouse tables: Apache Iceberg metadata over Parquet files in object storage. Product search, however, still runs on copies of that catalog, in a lexical engine and a vector database, each with its own ingestion pipeline, schema, staleness window and bill. We show that search applications can be hosted directly on the data lake, with the lakehouse table as the single system of record. We extend Apache Pinot, a real-time OLAP engine that already serves Iceberg tables in place, with BM25 relevance scoring over its Lucene text index, corpus-wide term statistics so that scores are comparable across files, and reciprocal rank fusion in plain SQL; a real-time table fed by a stream can sit behind the same table, merged by the broker. On the Amazon ESCI benchmark (1.2M US products, 22,458 test queries), hybrid search served from the lake matches exact retrieval (nDCG@10 .361). The lake table is queryable 149 s after its files are registered, and a lake commit is searchable in 3.8 s, close to indexing directly into a search engine. On a brand-sorted lake of 277M products (0.8 TB), one 16-vCPU node answers brand-scoped vector, BM25 and client-fused hybrid searches in 6.5-9 ms (p50), sustains 1,100 vector and 635 BM25 searches per second at under 180 ms p99, and makes a commit searchable in 4.3 s. Search adds only its indexes: no copy of the catalog, no sync pipeline and no second schema, which cuts storage, rebuild time and the operational work of keeping copies consistent.
Figures & tables
| BM25 | Vector (HNSW) | Hybrid RRF | Filtered hybrid | |||||
| System | nDCG@10 | R@100 | nDCG@10 | R@100 | nDCG@10 | R@100 | nDCG@10 | R@100 |
| Reference (exact BM25 / exact cosine) | .308 | .530 | .350 | .609 | .361 | .621 | .342 | .588 |
| Pinot lake-native (global BM25 statistics) | .312 | .533 | .348 | .603 | .361 | .622 | .345 † | .595 † |
| with per-segment IDF | .271 | .506 | .341 † | .612 † | ||||
| Elasticsearch 9.5.5 (1,000 candidates) | .312 | .533 | .339 | .588 | .354 | .620 | – | – |
| OpenSearch 3.9.0 (ef_search 1,000) | .312 | .533 | .342 | .593 | .356 | .621 | – | – |
| 1 client | 32 clients | Files | ||
|---|---|---|---|---|
| Mode | p50/p99 | QPS | p50/p99 | searched |
| Brand vector | 7.5/9.5 | 1,103 | 27/53 | 1 |
| Brand BM25 | 6.5/47 | 635 | 40/176 | 1 |
| Brand hybrid, client RRF | 8.9/48 | 375 | 75/246 | 1 |
| Brand hybrid, SQL RRF | 33/74 | 122 | 252/389 | 1 |
| System | Load path | Time to queryable | Text index | Vector index | Row data | Extra bytes |
| Pinot lake-native | register 33 files (3.4 s), index at load | 149 s | 0.27 GB | 3.78 GB | 0 | 4.06 GB |
| Elasticsearch 9.5.5 | bulk load (4 clients), refresh | 447 s | 0.24 GB | 3.78 GB | 0 ∗ | 4.10 GB |
| OpenSearch 3.9.0 | bulk load (4 clients), refresh | 435 s | 0.50 GB | 3.78 GB | 0 ∗ | 4.29 GB |
| ∗ _source disabled for a like-for-like comparison; keeping it stores the rows again. | ||||||
| 1.2M products | 277M (0.8 TB) | |||
| Copy | Pinot | Copy | Pinot | |
| Rows copied out of the lake | 8.8 GB | 0 | 1.6 TB | 0 |
| Indexes | 8.2 GB | 8.1 GB | 1.9 TB ∗ | 2.2 TB |
| Sync pipelines | 1 | 0 | 1 | 0 |
| Full build, one node | 707 s | 149 s | 45 h ∗ | 2.7 h |
| Nodes | 2 | 2 | 2 ‡ | 2 ‡ |
E-Commerce Search
E-commerce search studies how to match shoppers’ queries, preferences, and product signals with useful items in online marketplaces.
See all 43 papers