Hybrid GPU-CPU Retrieval for Personalized Search at Ultra-Large Scale

Hao Fu; Jichao Sun; Baiting Zhu; Qiaoling Liu; Yan Shi; Cheng Lu; Liu Liu; Yubo Wang

cs.IRSep 18, 2026

Hybrid GPU-CPU Retrieval for Personalized Search at Ultra-Large Scale

Authors: Hao Fu, Jichao Sun, Baiting Zhu, Qiaoling Liu, Yan Shi, Cheng Lu, Liu Liu, Yubo Wang,

Organizations: Meta Platforms, Inc. Menlo Park, California, USA

Abstract

Embedding-based retrieval on user-generated content at the trillion-document scale exposes a sharp conflict between two production demands: deep, expressive personalization for queries with rich user intent, and broad coverage of a massive inventory under fixed latency and resource budgets. We characterize this as the personalization-scale paradox: hosting the full serving inventory in GPU memory is too resource intensive, while CPU compute cannot execute the same interaction-heavy model on the latency-critical path. We present a hybrid GPU-CPU co-serving system that resolves the paradox through orchestration rather than a new model class. A high-depth GPU pathway fuses retrieval and interaction pre-ranking over a curated online pool on the order of a billion documents, while a high-breadth CPU pathway searches an independently selected online inventory roughly twenty times larger with lightweight personalized scoring. Either or both pathways can run per request; candidates are deduplicated before shared downstream ranking. The system is deployed in production. A full-system A/B test against the legacy CPU-only configuration improves model-scored relevance and substantive engagement, while separate pathway experiments show positive value at their own deployment scopes. Retrieval logs show that the pathways contribute structurally distinct candidates, production serving measurements characterize their latency, and a matched capacity plan quantifies the economic rationale for assigning modeling depth to GPUs and inventory breadth to CPUs. Together, these results validate a practical, independently evolvable depth-breadth architecture for ultra-large-scale personalized search.

Figures & tables

System	Scale	Hardware	Fused pre-rank
Facebook search ( Huang et al., 2020 )	$100$ M-scale index	CPU	No
DiskANN ( Subramanya et al., 2019 )	$1$ B points	CPU	No
SPANN ( Chen et al., 2021 )	$1$ B vectors	CPU	No
SVFusion ( Peng et al., 2026 )	Up to $1$ B vectors	CPU+GPU	No
FusionANNS ( Tian et al., 2024 )	$1$ B vectors	CPU+GPU	No
This work	Billion + tens-of-billions scale	CPU+GPU	Yes

Table 1. Architectural comparison at each paper’s reported evaluation scale. Fused pre-rank denotes learned interaction scoring within the retrieval pass, before shared aggregation.

Figure 1. GPU–CPU co-serving over independently selected billion- and tens-of-billions-scale inventories from a multi-trillion-document source corpus. An input query and user context feed a billion-scale GPU retrieval branch and a tens-of-billions-scale CPU retrieval branch selected from a multi-trillion-document source corpus. Their candidates are aggregated before downstream ranking.

Figure 2. Joint training combines InfoNCE retrieval with engagement and relevance objectives. User-query and document towers are jointly trained with an interaction module using retrieval, relevance, and engagement losses.

Figure 3. GPU inference; sparse tables remain on CPU. A GPU-resident pipeline performs query encoding, approximate nearest-neighbor retrieval, interaction pre-ranking, and value scoring while fetching sparse tables from host memory.

Figure 4. Eager retrieval for three nearest neighbors with prefetch multiplier $2$ scans clusters before applying other rules to the top- $6$ . Three nearest-neighbor clusters are scanned sequentially to produce six prefetched candidates before other query constraints are evaluated.

Configuration	DCG@20	GSRR
High-depth GPU	$+0.73\%\pm 0.15\%$	$+1.04\%\pm 0.13\%$
Broad-inventory CPU	$+3.16\%\pm 0.15\%$	$+1.20\%\pm 0.11\%$
Hybrid architecture	$+4.51\%$ $[+3.99\%,+5.04\%]$	$+2.01\%$ $[+1.71\%,+2.32\%]$

Table 2. Production evaluation. Relative lifts use contemporaneous controls; experiments ran in separate windows and should not be compared across rows. For the hybrid row, both intervals are the conservative nine-day interval described in Appendix A.5 .

Figure 5. Recall–candidate frontier for 64k and 512k centroid indices as the number of probed clusters varies from 1 to 1024. The 512k index reaches comparable recall with fewer scanned candidates. Recall at 20 versus average scanned candidates for 64k- and 512k-centroid indices. Point color encodes the number of probed clusters. The 512k curve reaches the same recall with fewer candidates over their overlapping range.

Branch	Calls	P50 (ms)	P95 (ms)	P99 (ms)
Lexical	3.43M	196	686	1,184
CPU pathway	2.11M	41	439	859
GPU pathway	0.94M	151	675	1,376

Table 3. Production retrieval-branch wall time for posts over seven complete days. Call counts and latency percentiles are sampling-weighted and measured at the retrieval-branch boundary.

Segment (queries)	GPU-side	CPU-side	Jaccard
Head ( $85{,}617$ )	$2.41\%$	$1.15\%$	$0.75\%$
Torso ( $17{,}057$ )	$2.08\%$	$0.60\%$	$0.44\%$
Tail ( $11{,}405$ )	$2.20\%$	$0.52\%$	$0.41\%$

Table 4. Macro-averaged GPU–CPU dense-retrieval overlap over three days. GPU-side is

|G\cap C|/|G|

; CPU-side is

|G\cap C|/|C|

Pathway	Retrieved	Ranker input	Reach rate
GPU pathway	594	224	$37.6\%$
Dense CPU union	1,352	59	$4.4\%$

Table 5. Dense-pathway candidate flow over three days. Counts are rounded averages over eligible content-search requests for which the GPU pathway ran; reach rates use unrounded counts. The CPU row is the deduplicated union of all dense CPU sources.

Planning input	CPU host	Accelerator
Unit capacity cost	$1.0\times$	$\sim 12\times$
Query throughput/unit	$1.0\times$	$\sim 3$ – $4\times$
Vector capacity/unit	$1.0\times$	$\sim 2$ – $3\times$
Units/region	$1.0\times$	$\sim 1/3\times$
3-region capacity cost	$1.0\times$	$\sim 4\times$

Table 6. Normalized capacity-plan inputs for a matched

d=256

SQ8 workload with three regional replicas. Capacity-cost values use the CPU plan as the reference.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	GPU pathway	CPU pathway
Hardware	MI300X accelerator	Commodity x86 DRAM hosts
Inventory	Billion-scale selected pool	Tens-of-billions-scale inventory
Refresh	Daily model + snapshot	Base backfill + live updates
Measurement	Roughly 150–200 queries/s/card; 99th-percentile latency $\sim$ 30–40 ms	Online CPU-use measurements
Excludes	Network, aggregation, downstream ranker	No comparable per-host latency reported

Appendix

Table 7. Serving scopes. The GPU benchmark is per accelerator and excludes end-to-end search stages.

Exclusive source group	Candidates	Share
GPU only	209	$27\%$
Dense CPU only	47	$6\%$
Lexical CPU only	500	$64\%$
Cross-group overlap	26	$3\%$
Total	$\sim$ 781	$100\%$

Appendix

Table 8. Deduplicated composition at the shared-ranker input per eligible search request. Counts are independently rounded; “Overlap” contains documents attributed to more than one group.

Path	Objective	Supervision and serving role
GPU	Cross-session InfoNCE	In-batch documents from other sessions; learns the ANN space.
GPU	Within-session InfoNCE	Engaged positives and relevant non-engaged negatives.
GPU	Binary cross-entropy + Smooth L1	Engagement and relevance targets for interaction scoring.
GPU	Weighted value model	Task weights select an operating point.
CPU	Bidirectional InfoNCE	Produces reusable 384-dimensional document vectors.

Appendix

Table 9. Training objectives and serving roles.

Information Retrieval

Information Retrieval studies how to find, rank, and organize relevant information in large collections of data.

See all 474 papers

Efficient GPU Retrieval for Semantic Search
Aug 29, 2026Dhritiman Das, Chujie Zheng, Ronak Kaoshik +18Semantic Search Dense Retrieval
LinkedIn Mountain View, CA, USA
Efficient Iterative Retrieval with Heterogeneous Batching
Sep 21, 2026Dohyun Park, Hubertus Franke, Daniel G. Waddington +2Retrieval-Augmented Generation LLM Serving
University of Illinois Urbana-Champaign · IBM Research
MTServe: Efficient Serving for Generative Recommendation Models with Hierarchical Caches
Apr 24, 2026Xin Wang, Chi Ma, Shaobin Chen +14KV-Cache Offloading KV Caching
School of Computer Science, Wuhan University · Meituan · Nvidia

Aug 29, 2026J/K move · Enter open · S save

Efficient GPU Retrieval for Semantic Search

Dhritiman Das, Chujie Zheng, Ronak Kaoshik, Pratik Dixit+17

LinkedIn Mountain View, CA, USA

Semantic Search Dense Retrieval

Open paper AskPDF arXiv

Semantic Search on LinkedIn must retrieve relevant profiles from a corpus of hundreds of millions in response to natural-language queries such as "a fintech founder in Berlin who worked in payments." The deployed relevance policy is bottleneck-oriented: every active non-negotiable facet must be satisfied, and a pre-existing LLM Graded Relevance (GR) judge operationalizes this through a fixed min/median aggregation over facet grades. Cosine similarity instead averages evidence, letting a strong match on one facet mask failure on another, capping the recall of the first-stage (L0) retriever. We present a policy-aligned retrieval framework: embeddings are partitioned into eight category-supervised segments whose scores follow the same min/median rule at serving time; for multi-vector retrieval, this segment score is computed independently per tagged document slot and maximized across slots. A lightweight single-slot Stage-1 scorer generates high-recall candidates, while scale-invariant relative-norm gating keeps category activation consistent across training, evaluation, and serving. On 21K held-out queries, this representation improves offline relevance over a matched-capacity baseline, with gains broadly distributed across facet combinations. We serve this framework with a two-stage GPU architecture: an FP8 coarse ranker scores the full corpus, increasing per-shard capacity by 71% and Stage-1 matmul throughput by 36%, then an FP16 stage exactly re-ranks an oversampled candidate set, recovering 99.6-99.8% of full-FP16 recall at over 500 QPS per shard replica. In a member-randomized A/B test, exploratory-query Precision@10 under the unchanged GR judge rises from 63.7% to 79.0% and navigational Precision@1 from 65.5% to 74.7%, with a blinded human evaluation independently confirming the Precision@10 gain.

Abstract

Information Retrieval

Explore similar work

Efficient GPU Retrieval for Semantic Search

Efficient Iterative Retrieval with Heterogeneous Batching

MTServe: Efficient Serving for Generative Recommendation Models with Hierarchical Caches