From Generation to Scoring: Scaling LLM-Based Semantic Labeling at LinkedIn

Jian Sheng; Sriram Vasudevan; Sundara Raman Ramachandran; Yang Hu; Junyu Zhou; Xiaojing Ma; Haolan Qin; Tiffany Zhou

cs.LGOct 26, 2026

From Generation to Scoring: Scaling LLM-Based Semantic Labeling at LinkedIn

Authors: Jian Sheng, Sriram Vasudevan, Sundara Raman Ramachandran, Yang Hu, Junyu Zhou, Xiaojing Ma, Haolan Qin, Tiffany Zhou,

Organizations: LinkedIn.

Abstract

For efficiency, industrial large language model (LLM) labelers often pack many candidate labels into one prompt and generate a YES/NO answer for each, even when each decision depends only on the entity and one candidate. Generating these independent decisions as one answer list couples them, spends decoding on the list, and relies on the list being well-formed. In this paper, we present an end-to-end recipe that reformulates the labeler as semantic scoring, fine-tuning it on single-candidate prompts to return each decision as one answer-token score from a prefill-only pass. We also adapt Multiple-Item Scoring (MIS) to score hundreds of candidates in one sequence with attention masked between them. Deployed for profile qualification labeling at LinkedIn, semantic scoring changes at most 0.01% of decisions when candidates are reordered and matches the agreement with reference labels that packed generation reaches only with about 4.7x as many training pairs. The MIS-based stack raises throughput over the deployed stack by 1.24x offline and 1.9x in model serving under a one-second p99 latency target. For offline batch inference, we implement MIS-based scoring as a reusable, model-agnostic Ray Data API proposed for upstream adoption. Model serving uses MIS support contributed to SGLang.

Figures & tables

Figure 1: Profile qualification labeling against a supplied catalog. The baseline uses packed generation for the highlighted labeling stage. Sections 3 and 4 replace that stage with semantic scoring and MIS. Stored predictions are reused by downstream consumers.

Stage	Change	Check before the next stage
Prompt (§ 3.1 )	One decision per prompt, shared text in $q$	Single-token answers, split tokens match
Training (§ 3.2 )	Reference label per item, label-only loss	Held-out quality on the served prompt
Readout (§ 3.3 )	Score from yes / no logits, tuned threshold $\tau$	Held-out FP/FN against references at chosen $\tau$
Inference (§ 4 )	MIS scores a round in one request	Matches single-item scores, $N$ fits limits
Integration (§ 5 )	Only the labeler changes, on both paths	Same checkpoint, answer tokens, and $\tau$ on all paths

Table 1: Recipe stages: the change each makes to the packed-generation labeler and the check the recipe calls for before the next stage.

Figure 2: MIS attention mask for a synthetic packed sequence

x=[q;c_{1};c_{2};c_{3}]

: each item token attends only to

q

and its own causal prefix, and every item reuses the position IDs that follow

q

	Packed gen.	Independent	+ prefix cache	MIS
Requests	$\lceil N/g\rceil$	$N$	$N$	$1$
Query prefill	$\lceil N/g\rceil L_{q}$	$NL_{q}$	$L_{q}$ to $NL_{q}$	$L_{q}$
Decoded tokens	$\Theta(N)$	$0$	$0$	$0$
Cross-item attn.	yes	no	no	no
Output	text	score	score	score

Table 2: Work to score the

N

items of one query (

g

items per packed-generation prompt; delimiters and template tokens omitted).

Figure 3: MIS changes only the scoring stage of both serving paths. (a) Offline: tokenization and scoring (blue) run in separate autoscaling actor pools and bypass the generation-only stages (dashed). (b) Profile updates: a multi-process gRPC server ( Li et al., 2026 ) sends each packed request to a prefill-only SGLang engine.

Figure 4: Agreement with reference labels against training-data size (profile–qualification pairs) for semantic scoring and packed generation.

Method	Offline	Profile updates
Packed generation ( $g=20$ )	5.1K (1.0 $\times$ )	2.5K (1.0 $\times$ )
MIS ( $N=200$ )	6.3K (1.24 $\times$ )	4.8K (1.9 $\times$ )

Table 3: Throughput of the deployed inference stacks, in labels per second per GPU, with the ratio to packed generation in parentheses. Offline (H100): steady state at the best batch size and concurrency; profile updates (H200): highest arrival rate with p99 latency

\leq

1 s. The stacks differ in engine, checkpoint, prompt format, and runtime tuning as well as method.

Figure 6: Model-serving speedup over 1.5B packed generation versus items per request for MIS with the 1.5B checkpoint (chat format) and the exploratory 0.6B checkpoint (readout-token format, Section 6.5 ), on one H200 at the one-second p99 knee without cross-request prefix caching. The dotted curve is

R(N)

(Equation ( 5 )) scaled to the 1.5B

N=1

speedup. The two curves differ in model, checkpoint, and layout, so they do not isolate model size.

Efficient Language Model Inference

Efficient language model inference reduces the compute, memory, latency, and energy needed to generate useful outputs.

See all 318 papers

Auditing LLM Benchmarks with Item Response Theory
May 28, 2026Sander Land, Daniel M. BikelBenchmark Auditing LLM Auditing
Writer, Inc.
SHALA-LLM: Smartly Handling Ambiguous Labels in Aligning LLMs
Jun 3, 2026Jingyao Wu, Ashley Wang, Keane Ong +2RL for Language Models LLM Alignment
MIT Media Lab, Massachusetts Institute of Technology · National University of Singapore
Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation
Apr 6, 2026Firoj Alam, Gagan Bhatia, Sahinur Rahman Laskar +1Language Model Generation Evaluation Multilingual Language Model Evaluation
Qatar Computing Research Institute, HBKU, Qatar · †UPES, India

May 28, 2026J/K move · Enter open · S save

Auditing LLM Benchmarks with Item Response Theory

Sander Land, Daniel M. Bikel

Writer, Inc.

Benchmark Auditing LLM Auditing Benchmark Validity

Open paper AskPDF arXiv

LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all. We introduce an Item Response Theory-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples across seven preference and multiple-choice benchmarks using responses from 114 models, outperforming a supervised classifier. We trace these errors to mechanical labeling heuristics, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single label. The same model fit reveals that reward models specialize in stylistic preference rather than factual knowledge, and identifies one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with benchmark contamination or benchmark-specific over-optimization.

Abstract

Efficient Language Model Inference

Explore similar work

Auditing LLM Benchmarks with Item Response Theory

SHALA-LLM: Smartly Handling Ambiguous Labels in Aligning LLMs

Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation