Efficient GPU Retrieval for Semantic Search

Dhritiman Das; Chujie Zheng; Ronak Kaoshik; Pratik Dixit; Vishal Shah; Yanbo Li; Jiahao Xu; Manika Agarwal

cs.AIAug 29, 2026

Efficient GPU Retrieval for Semantic Search

Authors: Dhritiman Das, Chujie Zheng, Ronak Kaoshik, Pratik Dixit, Vishal Shah, Yanbo Li, Jiahao Xu, Manika Agarwal,

Organizations: LinkedIn Mountain View, CA, USA

Abstract

Semantic Search on LinkedIn must retrieve relevant profiles from a corpus of hundreds of millions in response to natural-language queries such as "a fintech founder in Berlin who worked in payments." The deployed relevance policy is bottleneck-oriented: every active non-negotiable facet must be satisfied, and a pre-existing LLM Graded Relevance (GR) judge operationalizes this through a fixed min/median aggregation over facet grades. Cosine similarity instead averages evidence, letting a strong match on one facet mask failure on another, capping the recall of the first-stage (L0) retriever. We present a policy-aligned retrieval framework: embeddings are partitioned into eight category-supervised segments whose scores follow the same min/median rule at serving time; for multi-vector retrieval, this segment score is computed independently per tagged document slot and maximized across slots. A lightweight single-slot Stage-1 scorer generates high-recall candidates, while scale-invariant relative-norm gating keeps category activation consistent across training, evaluation, and serving. On 21K held-out queries, this representation improves offline relevance over a matched-capacity baseline, with gains broadly distributed across facet combinations. We serve this framework with a two-stage GPU architecture: an FP8 coarse ranker scores the full corpus, increasing per-shard capacity by 71% and Stage-1 matmul throughput by 36%, then an FP16 stage exactly re-ranks an oversampled candidate set, recovering 99.6-99.8% of full-FP16 recall at over 500 QPS per shard replica. In a member-randomized A/B test, exploratory-query Precision@10 under the unchanged GR judge rises from 63.7% to 79.0% and navigational Precision@1 from 65.5% to 74.7%, with a blinded human evaluation independently confirming the Precision@10 gain.

Figures & tables

Seg.	GR Category	Dims	Seg.	GR Category	Dims
0	Temporal	$0{:}64$	4	Education	$256{:}320$
1	Person Name	$64{:}128$	5	Title/Role	$320{:}384$
2	Company/Org	$128{:}192$	6	Expertise (negotiable)	$384{:}448$
3	Location	$192{:}256$	7	Industry (negotiable)	$448{:}512$

Table 1. Per-slot GR segment layout:

64

dimensions per segment at

512

-d and

32

256

-d.

Stage-1 depth $M$	Flat 256-dim dot product	Non-neg. $6{\times}32$ -d dot product	Non-neg. segment minimum
1000	44.1%	42.0%	78.3%
1100	47.0%	–	86.1%
1200	49.8%	–	94.0%
1500	57.5%	–	100%
2000	68.0%	65.1%	100%
3000	82.7%	81.7%	100%

Table 2. Stage-1 candidate recall for

d_{s}{=}256

K{=}3

. Fraction of the final FP16 top-

1000

retained in the Stage-1 top-

M

; higher is better.

Objective	RS-NDCG@10	Capped R@10	P@1	PMR@10
Contrastive objectives	0.8800	0.8777	0.7609	0.2787
$+$ Null regularization	0.8918	0.8855	0.7885	0.2595
$+$ Score calibration	0.8917	0.8856	0.7895	0.2583

Table 3. Segment-loss ablation. All rows include per-segment and full-vector InfoNCE. Capped R@10 uses relevant profiles found within the model’s top-

100

as the reference set, with its denominator capped at

10

; lower PMR@10 is better.

Model (per-slot dim. $\times K$ )	RS-N	C-R	P1	P10	PMR
Single L0 ( $512\!\times\!1$ )	0.8704	0.8709	0.7093	0.5880	0.2987
GR segment ( $512\!\times\!1$ )	0.8820	0.8795	0.7718	0.6377	0.2809
Tagged MV ( $512\!\times\!4$ ) †	0.8972	0.8871	0.7952	0.6659	0.2565
Tagged MV ( $512\!\times\!3$ ) †	0.8890	0.8817	0.7820	0.6474	0.2699
Tagged MV ( $256\!\times\!3$ )	0.8824	0.8794	0.7724	0.6379	0.2806
MV+GR ( $512\!\times\!3$ ) †	0.8917	0.8856	0.7895	0.6622	0.2583

Table 4. Capacity–quality trade-offs on the full corpus. RS-N and C-R abbreviate RS-NDCG@10 and Capped R@10; P1/P10 denote P@1/P@10, and lower PMR is better. † denotes an offline-only configuration that exceeds the co-resident control/treatment serving memory budget; ∗ denotes the configuration that fits within the co-resident control/treatment memory budget.

Panel of Figure 1 — Figure 1. A sample configuration of SPS GPU-RAR architecture. A broker scatters requests to all shards in parallel, gathers per-shard top- $K$ lists, and merges them. A two-tier retrieval architecture in which an SPS mid-tier calls a replicated broker that scatters requests to 13 H200 GPU shards and merges their top- $K$ results.

Metric	FP16	FP8	Two-pass FP8+FP16
GPU bytes/scalar	2 B	1 B	1 B (GPU)
Recall@1000 vs. FP16	100%	89–93%	99.6–99.8%
Stage-1 matmul	$1.0\times$	$1.36\times$	$\approx 1.36\times$
Host RAM/shard	30 GB	30 GB	150 GB
Documents/shard	35M	60M	60M

Table 5. Single-pass vs. two-pass FP8 retrieval on a sample evaluation configuration (

K{=}1000

; recall relative to exact FP16).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	$\mathcal{L}_{\mathrm{div}}$	$\mathcal{L}_{\mathrm{bal}}$	RS-NDCG@10	Capped R@10	P@1	P@10	PMR@10
Base objectives	–	–	0.8804	0.8775	0.7558	0.6244	0.278
Base $+$ diversity	Y	–	0.8832	0.8777	0.7730	0.6462	0.2776
Base $+$ balancing	–	Y	0.8820	0.8785	0.7718	0.6377	0.2809
Full model	Y	Y	0.8835	0.8784	0.7731	0.6462	0.2717

Appendix

Table 6. Training-objective component analysis for the

256\times 3

configuration. Every row includes all shared base objectives listed above, including the contrastive objectives and slot-0 recall loss. Only

\mathcal{L}_{\mathrm{div}}

and

\mathcal{L}_{\mathrm{bal}}

are ablated; Y indicates an enabled regularizer. Lower PMR@10 is better.

Query group	$k_{g}$	Single	Segment	MV	MV+Seg	$\Delta_{1}$	$\Delta_{3}$
education	813	2.8220	3.1570	3.3137	3.3530	+0.3350	+0.0393
education, location	737	2.6709	2.9511	3.1377	3.2176	+0.2802	+0.0799
education, expertise	728	2.7708	3.0494	3.1368	3.1941	+0.2786	+0.0573
education, name	746	1.7058	1.8401	1.8510	1.9211	+0.1343	+0.0701
location, name, title	698	2.0010	1.9452	1.8964	1.9487	-0.0558	+0.0523
expertise, location	758	2.8825	3.0791	3.1122	3.1601	+0.1966	+0.0479

Appendix

Table 7. Average final GR score (0–4) by query group under the same category-uniform retriever-quality protocol.

\Delta_{1}

is Segment only minus Single-vector L0;

\Delta_{3}

is MV

+

segment minus plain MV.

Second-degree	Bitmap (KB)
connections	1024	2048	3072
100K	0.0%	0.0%	0.0%
200K	0.0%	0.0%	0.0%
400K	0.01%	0.00%	0.00%
800K	0.57%	0.01%	0.00%
1.6M	14.0%	0.66%	0.05%

Appendix

Table 8. Bloom filter false-positive probability (FPP) and expected candidate volume needed to obtain 500 true matches.

Depth $S$	P95@400	Max QPS	P95@Max
2,000	72.8	600	104.9
4,000	80.8	550	112.6
8,000	104	500	124.36
10,000	116.5	450	134.1
16,000	151.1	350	122.1
20,000	158.74	300	107.24

Appendix

Table 9. Representative multi-vector operating points on a infra shards. Latency values are in ms; P90 is measured at 400 QPS, and Max QPS satisfies P95

<150

ms with an error rate below

0.5\%

Re-rank depth $S$	P50 (ms)	P95 (ms)	P99 (ms)
2,000	54.2	72.8	75.4
4,000	60.4	80.8	84.7
8,000	78.3	104.0	107.7
10,000	87.9	116.5	120.4
16,000	115.5	151.5	154.5
20,000	122.7	158.7	162.4

Appendix

Table 10. Multi-vector performance benchmarks for second-pass reranking on a infra shard (concurrency 512,

K{=}1000

). All latency values are in ms.

Batch $\times N$	torch.topk	Chunked	Blitz	Speedup
$\star\ 1\times 50\mathrm{M}$	0.59	0.57	0.20	$2.9/2.8\times$
$\star\ 1\times 100\mathrm{M}$	1.11	0.93	0.27	$4.1/3.4\times$
$1\times 200\mathrm{M}$	2.27	1.65	0.40	$5.7/4.1\times$
$1\times 500\mathrm{M}$	5.21	3.82	0.80	$6.5/4.8\times$
$1\times 1\mathrm{B}$	11.21	7.22	1.48	$7.6/4.9\times$
$\star\ 32\times 50\mathrm{M}$	11.51	11.19	3.19	$3.6/3.5\times$

Appendix

Table 11. Sorted top-

K

selection latency (

K{=}1000

, FP16, H200 GPU, PyTorch 2.10.0.6, CUDA 12.8). Values are medians of 10 runs after 3 warm-up iterations. The final column reports speedup over torch.topk /Chunked, respectively;

\star

marks the four target configurations.

Query Mix	QPS	P90	P95	P99
100% attr. filter; 20% network filter	500	120	123.1	126.1
20% no-filter; 16% net.-filter; 64% other filters	400	120.6	124.2	128.9

Appendix

Table 12. Representative derived-features latency under query mixes (33.6M-doc index, model plus two derived features). Latencies are in ms.

Statistic	Human P@1	Human P@10
$N$ queries	50	50
Control	0.68	0.61
Exp	0.76	0.67
Absolute lift	+8.0 pp	+6.0 pp
Query outcomes (E/C/T)	9 / 5 / 36	31 / 14 / 5
95% CI for lift	$[-2.1,+18.0]$ pp	$[+2.0,+10.0]$ pp

Appendix

Table 13. Blinded evaluation on 50 paired queries. E/C/T denotes Exp wins, Control wins, and ties. Confidence intervals use paired query-level bootstrap; tests are exact McNemar for P@1 and paired permutation for P@10. P@10 is confirmatory here; P@1 is directional at this sample size.

Semantic Search

Semantic search retrieves information by meaning, matching queries and documents even when they use different words.

See all 42 papers

PAIR: Preference-Aligned Intent Retrieval for Premium People Recommendation at LinkedIn
Oct 22, 2026Zhongkai Shangguan, Yanbin Kang, Yihan Cao +7Recommender Systems Preference Modeling
LinkedIn, Sunnyvale, CA, USA
Mine and Refine: Optimizing Graded Relevance in E-commerce Semantic Search Retrieval
Feb 19, 2026Jiaqi Xi, Raghav Saboo, Luming Chen +8Semantic Search E-Commerce Search
DoorDash Inc. New York City, New York, USA · DoorDash Inc. Sunnyvale, California, USA · DoorDash Inc. USA +1
Connected Content Retriever: Dense Graph Edge Features Powering Pre-Ranking at LinkedIn
Sep 18, 2026Akhilesh Gupta, Sudarshan Srinivasa Ramanujam, Chirag Bhanuprasad Mehta +6Recommender Systems Learning to Rank
LinkedIn Corporation Mountain View, CA, USA

Oct 22, 2026J/K move · Enter open · S save

PAIR: Preference-Aligned Intent Retrieval for Premium People Recommendation at LinkedIn

Zhongkai Shangguan, Yanbin Kang, Yihan Cao, Ruijie Jiang+6

LinkedIn, Sunnyvale, CA, USA

Recommender Systems Preference Modeling Job-Candidate Matching Contrastive Learning

Open paper AskPDF arXiv

General-purpose embeddings trained on profile similarity often misrank candidates in professional networks, where relevance is inherently complementary rather than symmetric. For example, job seekers usually engage with recruiters rather than peers with similar credentials. Current user recommendation systems are often built on profile similarity or social signals such as shared connections, which are effective in friend-oriented social networks where users tend to connect with similar people or within existing social circles. However, this does not hold in professional platforms like LinkedIn, where interactions are often goal-driven and cross-role interactions, requiring matching based on complementary intents rather than similarity or social proximity. In this paper, we propose Preference-Aligned Intent Retrieval, a retrieval framework that is designed to model user intent preference instead of profile similarity. Our approach is inspired by the preference-learning perspective of Direct Preference Optimization, but instantiates preference learning as an exposure-conditioned multi-negative contrastive objective over embeddings. This formulation simplifies training and scales efficiently to large candidate sets without requiring an explicit reference model. Furthermore, we incorporate user interaction history by aggregating click histories with positional encoding to capture temporal order, which enables constant-cost updates of user representations over a fixed history window, allowing new interactions to be incorporated without re-encoding history. As a result, personalization can be updated efficiently using lightweight CPU operations without repeated LLM inference during serving. We evaluate PAIR on LinkedIn's Premium retrieval task. A 0.6B parameter model achieves a MAR@100 of 0.45.

Abstract

Semantic Search

Explore similar work

PAIR: Preference-Aligned Intent Retrieval for Premium People Recommendation at LinkedIn

Mine and Refine: Optimizing Graded Relevance in E-commerce Semantic Search Retrieval

Connected Content Retriever: Dense Graph Edge Features Powering Pre-Ranking at LinkedIn