Open rerankers trained for general web retrieval transfer imperfectly to e-commerce, where ranking decisions depend not only on topical relevance but also on user preferences, product constraints, and comparative product fit. These preference signals are difficult to supervise at scale: real search traffic provides authentic queries and candidates but no clean pairwise labels. We present ZooWork-ShopRanker, a family of e-commerce rerankers (0.6B, 4B, and 8B) aligned to judge-labeled shopping preference. Training pairs are labeled by a panel of reasoning large language models (LLMs) from different families acting as a preference oracle, with position-debiased judgments and agreement tiers, and the rerankers are trained on these labels. The aligned 8B flagship then serves as a distillation teacher for the efficient 4B and 0.6B models, which are fit to its scores and sharpened on judged pairs. To measure progress, we introduce ShopRank-Bench, a contamination-limited benchmark of ~10,000 private-traffic preference pairs in both text formats, tiered by how many judge families committed to each label. ZooWork-ShopRanker-8B and -4B significantly outperform the strongest open reranker baseline, every model significantly beats its own un-aligned base, and ZooWork-ShopRanker-0.6B beats its size peer; the gains hold in both formats and extend to common MTEB benchmarks. We release the models and the dual-format ShopRank-Bench to facilitate further research.
Figures & tables
Query
Decisive constraint
Candidate products
“shape sorter”
product type
✓ Fisher-Price shape-sorter toy ✗ office mail sorter
“jumpsuit under $100 boho”
explicit budget
✓ 44jumpsuit✗239 “bohemian” one
“jacket for dad vintage”
intended recipient
✓ men’s jacket ✗ women’s “vintage” jacket
“swim trunks without mesh lining”
explicit exclusion
✓ non-mesh trunk ✗ mesh -lined trunk
Table 1: Open rerankers rank by topical relevance, not shopper preference. Four gold-tier ShopRank-Bench pairs in which one explicit constraint decides the winner ( ✓ preferred, ✗ rejected). All four open baselines—BM25, BGE-Reranker-large, BGE-Reranker-v2-m3, Jina-m0—pick the violating product on every pair, each lured by a surface token match; ZooWork-ShopRanker -8B and -4B honor the constraint on all four, ZooWork-ShopRanker -0.6B on two.
Figure 1: Open rerankers score far lower on pairs whose query states a hard constraint. Accuracy on the 1,843 gold-tier pairs ( Section 3.2 ), split by whether the query states an explicit constraint; each segment is the accuracy on that subset. These are two different sets of pairs, not the same pairs with a constraint added and removed, so the difference is an association rather than a measured effect of the constraint. The dashed line marks chance (50%), since every pair is a two-way choice. Baselines are 14–20 points lower, the un-aligned base is more robust, and ZooWork-ShopRanker is flat. Per-subset numbers, including the budget slice where BM25 falls below chance, are in Table 6 .
Avg. tokens
Track
Pairs
query
doc
Label source
Purpose
Preference
10,511
6.7
99
Cross-family LLM judges, no conflict at any tier: 1,843 gold (3/3 committed), 4,445 silver (2/3), 4,223 bronze (1/3)
Table 2: Overview of the three ShopRank-Bench tracks; lengths are mean query and product-text token counts. The preference track is the headline metric (released in structured and natural-language formats); the attribute-hierarchy (AHP) and explicit-budget tracks are diagnostics, scored separately.
Figure 2: ShopRank-Bench pairwise accuracy ( n=10,511 ) by product-text format. Bars show overall accuracy ( ZooWork-ShopRanker variants in green; the un-fine-tuned Qwen3-Reranker bases and open baselines in gray); the diamond, square, and triangle markers show the gold, silver, and bronze agreement tiers ( Section 3.2 ). The axis starts at zero and the dashed line marks chance (50%), since every pair is a two-way choice. Full numbers with 95% query-clustered bootstrap CIs are in Section B.13 .
Figure 3: Preference-track accuracy versus parameter count (log scale; n=10,511 ; error bars are 95% query-clustered bootstrap CIs). ZooWork-ShopRanker beats its base at every size in both formats (paired McNemar p≤1.3×10−4 ); BM25 (56.2/58.4) is omitted for axis readability. The size-scaling view of the two diagnostic tracks is in Figure 5 ( Appendix B ).
Model
ESCIReranking
SciDocsRR
AskUbuntuDupQ
Avg.
ZooWork-ShopRanker-4B
0.848
0.882
0.667
0.799
ZooWork-ShopRanker-8B
0.844
0.885
0.667
0.799
Qwen3-Rnk-8B
0.835
0.885
0.659
0.793
Qwen3-Rnk-4B
0.836
0.890
0.641
0.789
Jina-m0 (2.4B)
0.837
0.851
0.632
0.773
ZooWork-ShopRanker-0.6B
0.823
0.835
0.624
0.761
Table 3: Evaluation-only results on three common MTEB reranking tasks, each reported at MTEB’s main score for that task. No confidence intervals or significance tests are computed for these numbers.
Figure 4: Diagnostic-track accuracy ( ZooWork-ShopRanker in green; base Qwen3-Reranker and open baselines in gray; the 0.6B budget specialist in gold). The dashed line is chance (50%): each item is a two-way choice, so scores below it mean systematically wrong choices. On the budget track, the diamond and triangle markers give the threshold and control slices ( Section 3.3 ); a pick-cheaper policy would score 100/0 on them, so only genuine threshold behavior is high on both.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Non-gold pairs
Model
Gold ( n=1,843 )
DeepSeek voted ( n=5,100 )
Training panel only ( n=3,568 )
ZooWork-ShopRanker-8B
96.5
83.6
76.2
ZooWork-ShopRanker-4B
96.3
81.6
72.8
Qwen3-Rnk-8B
93.2
81.0
69.3
Jina-m0 (2.4B)
90.8
80.7
71.0
ZooWork-ShopRanker-0.6B
91.2
77.4
68.1
Appendix
Table 4: Accuracy on ShopRank-Bench structured pairs, split by whether the training-disjoint judge family committed a verdict. The ZooWork-ShopRanker advantage survives on pairs that DeepSeek also decided ( +2.9 and +0.9 ), which is the subset that bears on circularity. The “DeepSeek voted” column is not training-disjoint—the training families may also have voted on those pairs—so this bounds a training-family effect rather than removing it.
Field
Example
style
casual, sporty
color
black
material
cotton
audience
women
product type
t-shirt
brand
Zara
Appendix
Table 5: Product text schema fields.
Figure 10
Figure 5: Diagnostic-track accuracy versus parameter count (log scale). The dashed line is chance (50%).
Subset
n
BM25
BGE lg
BGE v2-m3
Jina m0
Base 8B
Ours 0.6B
Ours 4B
Ours 8B
All gold
1,843
69.0
82.4
84.9
90.8
93.2
91.2
96.3
96.5
No stated constraint
1,159
75.1
89.6
90.3
96.1
95.3
91.1
95.9
96.3
Any hard constraint
684
58.6
70.0
75.6
81.9
89.8
91.4
96.9
96.8
budget
338
47.9
55.6
63.9
70.4
89.9
92.3
97.9
97.9
recipient
329
71.4
82.4
86.3
92.4
91.5
91.2
96.0
96.0
exclusion
48
45.8
64.6
70.8
72.9
68.8
83.3
97.9
91.7
Appendix
Table 6: Gold-tier accuracy by constraint subset. Shaded cells are at or below the 50% chance line. “Base-8B” is the un-fine-tuned Qwen3-Reranker-8B. The three indented rows are subsets of “any hard constraint” and overlap.
Size
Stage
Easy
Hard
Hard-NL
0.6B
Base
77.7
72.6
71.2
+ judged pairs
78.3
73.9
71.6
+ distill (4B teacher)
82.5
78.3
73.8
+ mixed format
82.5
78.3
73.6
4B
Base
82.1
76.4
75.8
+ judged pairs
83.6
78.7
75.7
Appendix
Table 7: Ablation of the alignment-from-base pipeline on the two-judge development slices of Section 4.1 (easy, hard, hard-NL). This is the route the released 0.6B and 4B do not take—they are distilled from the aligned 8B instead ( Section 4.2 )—so the table motivates the recipe rather than decomposing the shipped models. Rows are consistent within this table and are not directly comparable to the three-judge release benchmark. The 0.6B distillation rows here use an earlier 4B teacher, superseded by the 8B teacher of the released models.
Model
Params
Batch-1 p50 (ms)
Batch-1 p95 (ms)
Docs/s (batch 32)
Peak mem (GB)
ZooWork-ShopRanker-0.6B
0.60B
17.6
25.1
433.1
4.5
ZooWork-ShopRanker-4B
4.02B
25.2
33.3
154.5
11.2
ZooWork-ShopRanker-8B
8.19B
25.8
30.4
102.7
18.9
Qwen3-Rnk-0.6B
0.60B
18.6
20.3
431.9
4.5
Qwen3-Rnk-4B
4.02B
24.5
26.6
154.9
11.2
Qwen3-Rnk-8B
8.19B
25.4
28.4
102.8
18.9
Appendix
Table 8: Serving cost on one NVIDIA H200, fp16, 1024-token limit, measured over 256 real ShopRank-Bench (query, document) pairs after 8 untimed warmup batches, with the device synchronised around every timed region. Batch-1 latency is the serving-budget number and is measured identically for every system; throughput is reported only where a batched forward pass exists. † Jina-m0 is driven through its upstream per-pair API, so it has no comparable batched figure and is not given one. Parameter counts are of the merged serving model as loaded; Jina-m0’s nominal size is 2.4B, the figure we use to label it elsewhere, and 2.21B is the count of its loaded weights. The GPU was exclusive to this job but the host was shared with an unrelated CPU-only workload, which widens the p95 tail; the p50 column is the stable comparison. BGE-Reranker-large was not evaluated here.
Model
Overall
Gold
Silver
Bronze
p50 per call
$ per 1k pairs
GLM-5.3
91.9
99.6
95.9
84.3
3.4 s
3.56
Qwen3.5-27B
92.1
99.9
96.0
84.6
11.8 s
3.67
ZooWork-ShopRanker-8B
83.4
96.5
86.1
74.8
25.8 ms
—
Appendix
Table 9: Zero-shot LLM reference on structured ShopRank-Bench ( n=10,511 ). LLM latency is API wall-clock per call—GLM-5.3 through Fireworks, Qwen3.5-27B through OpenRouter—with mean output of 315 (GLM) and 1,140 (Qwen) tokens, almost all reasoning; cost covers both orders at list price. Both timing and cost are route-dependent and are not properties of the models themselves. The ZooWork-ShopRanker -8B latency is batch-1 p50 on one H200 ( Table 8 ). McNemar against ZooWork-ShopRanker -8B: χ2=416,p=1.6×10−92 (GLM) and χ2=453,p=1.6×10−100 (Qwen); the two LLMs do not differ ( p=0.46 ).
Method
Easy
Hard
Hard-NL
LoRA (0.6B, controlled)
78.3
73.9
71.6
Full FT (0.6B, controlled)
77.7
73.0
71.0
Appendix
Table 10: Controlled LoRA versus full fine-tuning at 0.6B: identical data, recipe, and evaluation, differing only in whether the update is low-rank. Measured on the development slices of Section 4.1 .
Accuracy by hierarchy level
Model
Overall
audience
prod. type
price
style
color
ZooWork-ShopRanker-8B
0.543
0.458
0.595
0.917
0.100
0.530
ZooWork-ShopRanker-4B
0.475
0.286
0.485
0.973
0.030
0.780
ZooWork-ShopRanker-0.6B
0.317
0.112
0.480
0.550
0.015
0.600
Jina-m0 (2.4B)
0.299
0.286
0.635
0.003
0.215
0.070
Qwen3-Rnk-4B
0.278
0.148
0.620
0.177
0.200
0.020
Appendix
Table 11: AHP diagnostic by hierarchy level ( n=1,500 ), under the same evaluation harness as Figure 4(a) . Shading in the overall column marks accuracy below the 0.50 chance line, i.e. systematic violation of the proposed hierarchy; only ZooWork-ShopRanker -8B clears it (green). Per-level cells are unshaded, but almost all of them are below chance too. Preference alignment does not lift the hierarchy uniformly—it transforms price and color while degrading style and product type —which is why we treat the aggregate AHP gain as a side effect rather than evidence of hierarchy-following.
Model
Overall [95% CI]
Gold
Silver (2/3)
Bronze (1/3)
ZooWork-ShopRanker-8B
83.4 [82.5, 84.2]
96.5
86.1
74.8
ZooWork-ShopRanker-4B
81.2 [80.3, 82.0]
96.3
84.3
71.4
Qwen3-Rnk-8B
79.2 [78.2, 80.1]
93.2
82.6
69.4
Jina-m0 (2.4B)
79.2 [78.3, 80.1]
90.8
82.9
70.2
Qwen3-Rnk-4B
77.5 [76.5, 78.5]
92.7
81.4
66.7
ZooWork-ShopRanker-0.6B
76.6 [75.7, 77.6]
91.2
79.7
67.1
Appendix
Table 12: ShopRank-Bench structured format ( n=10,511 ). Accuracy is in percent; intervals are 95% query-clustered bootstrap CIs. Tiers are by the number of judge families that committed ( Section 3.2 ). Qwen3-Rnk rows are the un-fine-tuned Qwen3-Reranker bases evaluated zero-shot under ; each ZooWork-ShopRanker row is the LoRA-aligned counterpart of the base of the same size. Every reranker loses 20–26 points from gold to bronze, which is why the aggregate column alone is a poor summary. Bold marks the best reranker; the LLM rows count a pair correct only when both presentation orders pick the preferred product.
Model
Overall [95% CI]
Gold
Silver (2/3)
Bronze (1/3)
ZooWork-ShopRanker-8B
81.8 [80.9, 82.7]
95.0
85.4
72.2
ZooWork-ShopRanker-4B
79.5 [78.6, 80.4]
95.4
82.1
69.8
Qwen3-Rnk-8B
78.4 [77.5, 79.4]
91.9
82.3
68.4
Jina-m0 (2.4B)
78.1 [77.1, 79.0]
88.8
82.7
68.5
Qwen3-Rnk-4B
77.9 [76.9, 78.9]
92.1
81.0
68.5
ZooWork-ShopRanker-0.6B
75.3 [74.2, 76.3]
91.2
78.1
65.4
Appendix
Table 13: Natural-language rendering of the same ShopRank-Bench pairs, under the same tiers. Note the ordering change against Table 12 : base-8B overtakes Jina-m0 on prose, while base-0.6B falls below both BGE baselines. BM25 is below chance on the bronze tier in both formats.
Track (format)
Comparison
Paired test
Preference (structured)
4B vs. Jina-m0
χ2=18.2,p=1.9×10−5
Preference (structured)
8B vs. Jina-m0
χ2=85.0,p=3.0×10−20
Preference (structured)
8B vs. 4B
χ2=29.7,p=5.0×10−8
Preference (structured)
0.6B vs. BGE-v2-m3
χ2=47.4,p=5.8×10−12
Preference (structured)
0.6B vs. base-0.6B
χ2=68.9,p=1.0×10−16
Preference (structured)
4B vs. base-4B
χ2=84.0,p=4.8×10−20
Appendix
Table 14: Continuity-corrected paired McNemar tests across all ShopRank-Bench tracks. Unqualified model names denote ZooWork-ShopRanker variants; “base” denotes the corresponding un-fine-tuned Qwen3-Reranker. † Significant in favor of base-4B: the absence of a detected 0.6B/4B difference is specific to the structured format.
LLM-based shopping agents increasingly rely on long purchase histories and multi-turn interactions for personalization, yet naively appending raw history to prompts is often ineffective due to noise, length, and relevance mismatch. We propose MemRerank, a preference memory framework that distills user purchase history into concise, query-independent signals for personalized product reranking. To study this problem, we build an end-to-end benchmark and evaluation framework centered on an LLM-based 1-in-5 selection task, which measures both memory quality and downstream reranking utility. We further train the memory extractor with reinforcement learning (RL), using downstream reranking performance as supervision. Experiments with two LLM-based rerankers show that MemRerank consistently outperforms no-memory, raw-history, and off-the-shelf memory baselines, yielding up to +10.61 absolute points in 1-in-5 accuracy. These results suggest that explicit preference memory is a practical and effective building block for personalization in agentic e-commerce systems.
Large language models (LLMs) have shown promise for recommendation reranking, but their use introduces an important trade-off between recommendation quality and serving efficiency. We investigate whether decision-oriented systems provide a useful alternative when the reranking task is fundamentally a structured choice among predefined candidate items. Specifically, we conduct a controlled empirical study of two decision-oriented systems, TypeSafe Jev and OpenAI Decisions, for recommendation reranking. Across three Amazon Reviews domains and four candidate-set sizes, we compare these systems against recommendation-specific models and pointwise and listwise LLM-based rerankers, systematically evaluating recommendation effectiveness, candidate-order robustness, and efficiency and scalability. Our results reveal that Jev frequently achieves a favorable quality--latency trade-off, offering competitive recommendation effectiveness at substantially lower observed latency than stronger LLM-based alternatives. Overall, our findings suggest that specialization for decision-making does not inherently guarantee superior recommendation quality or robustness. Instead, the practical value of decision-oriented inference depends on implementation-specific characteristics and application requirements. These findings motivate further investigation of decision-oriented systems for recommendation and other ranking tasks with structured output spaces.
Decoding-free reranking methods that read relevance signals directly from LLM attention weights offer significant latency advantages over autoregressive approaches, yet suffer from attention score homogenization: middle-context documents receive near-identical scores, destroying the fine-grained distinctions required for ranking. We propose HeadRank, a framework that lifts preference optimization from discrete token space into the continuous attention domain through entropy-regularized head selection, hard adjacent-level preference pairs, and a distribution regularizer that jointly sharpen discriminability in the homogenized middle zone. Depth truncation at the deepest selected layer further reduces inference to O(1) forward passes. Across 14 benchmarks on three Qwen3 scales (0.6B--4B) using only 211 training queries, HeadRank achieves the highest average NDCG@10 at every scale, outperforming both generative and decoding-free baselines on the majority of benchmarks with 100% formatting success. At 4B, 57.4% of relevant middle-zone documents reach the top quartile versus 14.2% for irrelevant ones -- a 43-percentage-point selectivity gap that demonstrates the effectiveness of attention-space preference alignment for listwise reranking.
Juyuan Wang, Chenxing Wang, Yuchen Fang +8
Weixin Group, Tencent, China · 2South China University of Technology, Guangzhou, China