The Generator-Evaluator (G-E) framework generates K candidate sequences and uses an evaluator to select the highest-scoring one, which is widely used in recommender systems (RecSys) and natural language processing (NLP). Existing evaluators commonly score candidates independently. Although such evaluations can be batched, independent scoring neither models interactions among candidates nor eliminates repeated computation of request-level context and recurring candidate elements, causing the total evaluation work to grow approximately linearly with K. To handle with, we propose FlashEvaluator, a joint evaluator that scores all candidate sequences in a single forward pass. FlashEvaluator factorizes evaluation into shared request-level encoding, reusable candidate-side computation, sequence assembly by indexing, and cross-sequence interaction for setwise comparison. We call this request-local reuse scheme QKV-Cache: inspired by autoregressive KV caching, it reuses context-side key/value representations across candidate sequences and, when candidate elements recur, reuses their request-conditioned representations on the query side. In repeated-item settings, the dominant item-encoding cost therefore depends on the number of distinct items rather than their total occurrences across sequences, reducing the marginal cost of evaluating additional candidates. We provide a computational analysis and evaluate FlashEvaluator on recommendation and text summarization. The results show lower latency and higher throughput with competitive recommendation and summarization quality. In an online deployment at Kuaishou with K=50, FlashEvaluator reduces inference latency by 44% and increases QPS by 114% relative to the production baseline, while yielding statistically significant gains in retention, engagement, and ecosystem metrics.
Figures & tables
Figure 1. Comparison between a conventional candidate-factorized evaluator and the proposed joint evaluator.
Dataset
Domain
# Requests
# Items
Candidate Pool Size
Target Length
ML-1M
RecSys
161,646
3,043
50
6
Amazon-Books
RecSys
309,917
38,121
50
6
RecFlow
RecSys
3,308,233
14,181,768
120
6
CNN/DM
NLP
30,000
–
–
1 sentence
Table 1. Statistics of the datasets used in the experiments.
Category
Model
ML-1M
Amazon-Books
RecFlow
N@6
P@6
R@6
F1@6
N@6
P@6
R@6
F1@6
N@6
P@6
R@6
F1@6
G-Only
DNN
0.5950
0.4539
0.5542
0.4876
0.6448
0.5072
0.6125
0.5472
0.1584
0.0793
0.2069
0.1084
DCN
0.5981
0.4561
0.5573
0.4901
0.6683
0.5298
0.6461
0.5701
0.1597
0.0795
0.2083
0.1088
Seq2Slate
0.6222
0.4867
0.5927
0.5225
0.6952
0.5654
0.6871
0.6078
0.1693
0.0821
0.2134
0.1130
DLCM
0.6061
0.4643
0.5667
0.4988
0.6597
0.5242
0.6396
0.5641
0.1747
0.0861
0.2240
0.1169
SetRank
0.7154
0.5720
0.6933
0.6132
0.8014
0.6635
0.8145
0.7156
0.1823
0.0896
0.2344
0.1225
Table 2. Performance comparison on three recommendation datasets. N@6, P@6, R@6, and F1@6 denote NDCG@6, Precision@6, Recall@6, and F1@6, respectively. The best and second-best results are highlighted in bold and underlined , respectively.
Method
T5-base
BART-Large
Llama-3.1-8B (3-shot)
R-1
R-2
R-L
BS
R-1
R-2
R-L
BS
R-1
R-2
R-L
BS
Bounds
Base (Beam 1)
40.81
18.37
37.81
87.23
43.63
20.66
40.47
87.88
38.78
13.44
35.08
87.53
Oracle ( K=16 )
48.21
24.95
45.29
88.32
52.43
28.73
49.42
89.30
46.35
19.72
42.75
88.49
Rerankers
RankGPT (Qwen2.5-32B)
41.09
18.35
37.99
87.32
44.54
21.17
41.33
88.04
39.00
13.82
35.33
87.51
Table 3. Performance comparison of different reranking methods on CNN/DailyMail using three generator backbones. The K=64 expands the candidate set under a lower latency to SimCLS K=16 . The best and second-best results excluding Oracle are highlighted in bold and underlined , respectively.
Figure 2. Comparison of efficiency
Category
Metric
Relative Lift
Retention
7-day Lifetime (LT7)
+0.039%
Engagement
Duration Time(per User)
+0.142%
Active User
+0.077%
Ecosystem
Cold-start Exposure
+2.507%
Table 4. Online A/B testing results over a one-week period on Kuaishou APP. The proposed FlashEvaluator is compared against the production baseline. All reported improvements are statistically significant ( p<0.05 ).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
N@6
P@6
R@6
F1@6
PIER + Flash − CrossSeq
0.1919
0.0936
0.2438
0.1280
PIER + Flash + Pointwise
0.1912
0.0933
0.2427
0.1275
PIER + Flash + Listwise
0.1925
0.0938
0.2446
0.1284
Appendix
Table 5. Ablation study of FlashEvaluator on RecFlow.
Method
K
N@6
P@6
R@6
F1@6
PIER
10
0.1910
0.0935
0.2431
0.1277
PIER+Flash
10
0.1925
0.0938
0.2446
0.1284
PIER+Flash
20
0.1952
0.0952
0.2483
0.1300
PIER+Flash
30
0.1978
0.0963
0.2506
0.1313
Appendix
Table 6. Effect of increasing the number of candidate sequences K on RecFlow.
First-stage recommenders in multi-stage systems produce a ranked candidate list from which a limited prefix is forwarded to downstream rankers. Because each forwarded item must be processed by more expensive ranking stages, this shortlist cannot be arbitrarily large. The first-stage objective is therefore high coverage of relevant items within the forwarded prefix, commonly measured by Recall@k. A relevant item may be available deeper in the retrieved list but absent from the shorter prefix that is actually consumed. This paper studies post-hoc verification for promoting such candidates into the consumed shortlist without retraining or replacing the retriever. We introduce a lightweight generative verifier for retrieval models. Given a retriever state and a candidate item, the verifier scores the item through the likelihood of its identifier tokens. It is trained post hoc with next-token cross entropy, requires no sampled negatives or candidate pool during training, and scores only the retriever's top-K candidates at inference. The interface is minimal: the retriever supplies a query state and candidate items, and the item representation can use any fixed tokenization. Across Amazon product recommendation and YaMBDa music recommendation, the same verifier training recipe improves Recall@10 for SASRec, GRU4Rec, NextItNet, and MiniOneRec. Ablations show that the improvements are not explained solely by injecting item-content features into the retriever, supporting verification as a post-hoc output-side adaptation mechanism.
Early ranking stages in recommendation systems precompute item embeddings and cache them in-model for scoring within strict latency constraints. Because this cache exists only at serving time, outside the training loop, training and serving use different item representations, a structural discrepancy that limits quality and adds operational fragility. We show that co-designing the training and serving paths removes this representation discrepancy at its source. We introduce the memory layer, an in-model key-value embedding cache co-trained with the model: the item tower writes embeddings during training and the model reads them at serving, one source of truth for item representations by construction. Always-on embeddings cover items not yet cached, so every item receives a prediction, and the design consolidates three separate trainer-to-predictor update paths into a single self-contained pipeline. Deployed in production on Instagram Reels, the memory layer raises prediction coverage from 96% to 100%, improves embedding freshness from O(5 min) to O(20 s), and narrows the training-serving Normalized Entropy (NE) gap by up to 86%, yielding over 2× recall for the freshest content and a 5-6% cold start engagement lift. Because embeddings are produced during training, the system needs no separate bulk-evaluation or publish-time recomputation, cutting training-and-publish computational cost by 30% at neutral serving computational cost.
Position-independent KV cache reuse aims to reduce latency in retrieval-augmented generation by reusing chunk-level KV caches across prompts. We show that current evaluations of KV cache reuse techniques rely on measurements that fail to faithfully capture the loss of accuracy attributable to reuse, often artificially inflating the reported effectiveness. We also show that existing datasets do not exhibit the reuse dynamics needed to thoroughly evaluate such techniques. To address these issues, we propose an evaluation methodology that measures this accuracy loss without ambiguity and we introduce Boxoffice, a tool that programmatically generates evaluation datasets that exercise challenging KV cache reuse patterns.
Samuel Cestola, Tianxiang Xia, Pengfei Zheng +4
Huawei Technologies Ltd. · Department of Computer Science, ETH Zurich