Decision-Oriented Recommendation Reranking: An Empirical Study of Its Promise and Limitations

Hanjia Lyu; Yinglong Xia

cs.IRSep 30, 2026

Decision-Oriented Recommendation Reranking: An Empirical Study of Its Promise and Limitations

Authors: Hanjia Lyu, Yinglong Xia

Organizations: Singapore Management University · Meta AI

Abstract

Large language models (LLMs) have shown promise for recommendation reranking, but their use introduces an important trade-off between recommendation quality and serving efficiency. We investigate whether decision-oriented systems provide a useful alternative when the reranking task is fundamentally a structured choice among predefined candidate items. Specifically, we conduct a controlled empirical study of two decision-oriented systems, TypeSafe Jev and OpenAI Decisions, for recommendation reranking. Across three Amazon Reviews domains and four candidate-set sizes, we compare these systems against recommendation-specific models and pointwise and listwise LLM-based rerankers, systematically evaluating recommendation effectiveness, candidate-order robustness, and efficiency and scalability. Our results reveal that Jev frequently achieves a favorable quality--latency trade-off, offering competitive recommendation effectiveness at substantially lower observed latency than stronger LLM-based alternatives. Overall, our findings suggest that specialization for decision-making does not inherently guarantee superior recommendation quality or robustness. Instead, the practical value of decision-oriented inference depends on implementation-specific characteristics and application requirements. These findings motivate further investigation of decision-oriented systems for recommendation and other ranking tasks with structured output spaces.

Figures & tables

	Movies and TV				Video Games				Books
Method	$K=20$	$K=50$	$K=100$	$K=200$	$K=20$	$K=50$	$K=100$	$K=200$	$K=20$	$K=50$	$K=100$	$K=200$
SASRec	0.096	0.096	0.095	0.096	0.081	0.081	0.081	0.081	0.082	0.082	0.082	0.082
DCNv2	0.172	0.107	0.084	0.075	0.157	0.090	0.065	0.054	0.187	0.102	0.076	0.062
Qwen2.5 7B (P)	0.289	0.151	0.093	0.059	0.321	0.175	0.119	0.078	0.347	0.189	0.115	0.080
Qwen2.5 7B (L)	0.247	0.118	0.058	0.034	0.263	0.106	0.059	0.025	0.293	0.138	0.062	0.030
Qwen2.5 32B (P)	0.267	0.117	0.055	0.037	0.283	0.117	0.064	0.040	0.295	0.136	0.072	0.041

Table 2: Recommendation effectiveness on three Amazon Reviews 2023 domains, measured by NDCG@10 (

\uparrow

) across different candidate-set sizes

K

. (P) and (L) denote pointwise and listwise reranking, respectively. Listwise Qwen uses locally accessible logits, whereas GPT exposes only the top 20 token log probabilities, requiring different prompting and scoring procedures (Appendices A.2 and A.3 ). The best and second-best results in each setting are highlighted in bold and underlined , respectively.

Retriever	Reranker	Movies and TV	Video Games	Books
SASRec	SASRec	0.150	0.119	0.203
	Decisions	0.307	0.389	0.366
	Jev	0.288	0.355	0.338
LightGCN	LightGCN	0.089	0.114	0.068
	Decisions	0.360	0.367	0.487
	Jev	0.305	0.324	0.382

Table 3: Generalizability of decision-oriented recommendation reranking across SASRec and LightGCN retrievers, measured by NDCG@10 (

\uparrow

) at candidate-set size

K=20

on three Amazon Reviews 2023 domains. All methods are evaluated on the same overlapping set of eligible users to ensure a fair cross-retriever comparison. The original retriever rankings are included as baselines.

	Jev				Decisions
Metric	$K=20$	$K=50$	$K=100$	$K=200$	$K=20$	$K=50$	$K=100$	$K=200$
Mean NDCG@10	0.288	0.162	0.113	0.077	0.265	0.134	0.086	0.054
Mean Absolute $\Delta$ NDCG@10	0.138	0.074	0.051	0.027	0.176	0.094	0.056	0.042
Hit@10 Flip Rate	0.268	0.149	0.095	0.048	0.349	0.172	0.094	0.075
Informative Mean Absolute $\Delta$ NDCG@10	0.176	0.172	0.177	0.157	0.221	0.238	0.236	0.260
Informative-User Unchanged Rate	0.139	0.161	0.170	0.202	0.072	0.069	0.088	0.059

Table 4: Candidate-order robustness of Jev and Decisions across different candidate-set sizes on Movies and TV. For each user, we evaluate three permutations of the same candidate set (one original and two shuffled). Mean NDCG@10 is averaged across permutations, while Mean Absolute

\Delta

NDCG@10 and Hit@10 Flip Rate are averaged over all permutation pairs. Informative users are those achieving nonzero NDCG@10 under at least one permutation. The informative-user metrics are computed on this subset, with its size reported as N Informative. Lower absolute changes and flip rates, and higher unchanged rates, indicate greater robustness.

Panel of Figure 1 — Figure 1: Quality–latency trade-off across domains and candidate-set sizes. Each panel shows NDCG@10 against median observed per-user inference latency (seconds), with rows representing recommendation domains and columns representing candidate-set sizes $K\in\{20,50,100,200\}$ . Black lines connect the empirical Pareto-optimal methods, for which no other evaluated method achieves both lower latency and higher NDCG@10. Locally executed methods are evaluated on a single NVIDIA A800-SXM4-80GB GPU, whereas hosted methods are measured using observed API latency, including network communication and provider-side processing. Consequently, hosted and local latencies are not hardware-normalized. Detailed latency measurement protocols are provided in Appendix B .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Movies and TV				Video Games				Books
Method	$K=20$	$K=50$	$K=100$	$K=200$	$K=20$	$K=50$	$K=100$	$K=200$	$K=20$	$K=50$	$K=100$	$K=200$
SASRec	0.117	0.098	0.093	0.093	0.102	0.082	0.078	0.077	0.101	0.080	0.075	0.074
DCNv2	0.152	0.104	0.085	0.074	0.143	0.091	0.069	0.056	0.159	0.099	0.074	0.061
Qwen2.5 7B (P)	0.232	0.136	0.089	0.061	0.259	0.150	0.104	0.071	0.282	0.164	0.100	0.074
Qwen2.5 7B (L)	0.199	0.107	0.061	0.040	0.213	0.102	0.063	0.033	0.245	0.134	0.063	0.035
Qwen2.5 32B (P)	0.208	0.107	0.059	0.044	0.212	0.106	0.067	0.043	0.225	0.125	0.068	0.044

Appendix

Table 5: Recommendation effectiveness on three Amazon Reviews 2023 domains, measured by MRR (

\uparrow

) across different candidate-set sizes

K

	Movies and TV				Video Games				Books
Method	$K=20$	$K=50$	$K=100$	$K=200$	$K=20$	$K=50$	$K=100$	$K=200$	$K=20$	$K=50$	$K=100$	$K=200$
SASRec	0.170	0.169	0.168	0.169	0.161	0.161	0.161	0.161	0.166	0.166	0.165	0.166
DCNv2	0.373	0.217	0.160	0.143	0.346	0.189	0.129	0.108	0.411	0.216	0.159	0.126
Qwen2.5 7B (P)	0.575	0.300	0.183	0.109	0.612	0.346	0.244	0.164	0.644	0.361	0.240	0.161
Qwen2.5 7B (L)	0.519	0.250	0.122	0.065	0.535	0.220	0.121	0.051	0.551	0.251	0.130	0.062
Qwen2.5 32B (P)	0.566	0.255	0.121	0.072	0.609	0.265	0.141	0.093	0.617	0.278	0.170	0.094

Appendix

Table 6: Recommendation effectiveness on three Amazon Reviews 2023 domains, measured by HR@10 (

\uparrow

) across different candidate-set sizes

K

Dataset	$K$	Comparison	$\Delta$ NDCG@10	95% CI	Adjusted p-value
Movies and TV	20	Jev vs GPT-6-Sol	-0.014	[-0.033, 0.004]	0.2504
Movies and TV	20	Decisions vs GPT-6-Sol	-0.046	[-0.065, -0.025]	0.0024
Movies and TV	50	Jev vs GPT-6-Sol	-0.016	[-0.031, -0.002]	0.1156
Movies and TV	50	Decisions vs GPT-6-Sol	-0.046	[-0.062, -0.030]	0.0024
Movies and TV	100	Jev vs GPT-6-Sol	-0.023	[-0.035, -0.010]	0.0070
Movies and TV	100	Decisions vs GPT-6-Sol	-0.053	[-0.068, -0.038]	0.0024

Appendix

Table 7: Statistical significance of NDCG@10 differences between decision-oriented models (Jev and Decisions) and GPT-6-Sol (Listwise) across three datasets and four candidate-set sizes

K

\Delta

NDCG@10 denotes the mean paired difference (decision-oriented model minus GPT-6-Sol). We report unadjusted 95% paired bootstrap confidence intervals (CIs) and p-values from two-sided paired permutation tests with Holm–Bonferroni correction across 24 comparisons. Statistically significant differences (

p<0.05

) are highlighted in bold.

	Jev				Decisions
Metric	$K=20$	$K=50$	$K=100$	$K=200$	$K=20$	$K=50$	$K=100$	$K=200$
Mean NDCG@10	0.295	0.165	0.113	0.078	0.270	0.138	0.089	0.056
Mean Absolute $\Delta$ NDCG@10	0.139	0.077	0.054	0.028	0.173	0.097	0.059	0.043
Mean Absolute $\Delta$ Expected Hit@10	0.267	0.159	0.102	0.049	0.335	0.184	0.100	0.076
Informative Mean Absolute $\Delta$ NDCG@10	0.151	0.111	0.108	0.088	0.187	0.134	0.107	0.087
Informative-User Unchanged Rate	0.106	0.099	0.093	0.110	0.058	0.033	0.038	0.017

Appendix

Table 8: Tie-aware candidate-order robustness of Jev and Decisions on Movies and TV across candidate-set sizes. To account for tied candidate probabilities, we compute expected NDCG@10 and Hit@10 under uniformly random tie-breaking. For each user, we evaluate three permutations of the same candidate set (one original and two shuffled). Mean NDCG@10 is averaged across permutations, while Mean Absolute

\Delta

NDCG@10 and Mean Absolute

\Delta

Expected Hit@10 are averaged over all permutation pairs. Informative users are those achieving positive expected NDCG@10 under at least one permutation. The informative-user metrics are computed on this subset, with its size reported as

N

Informative. Lower absolute changes and higher unchanged rates indicate greater robustness.

Method	$K=20$	$K=50$	$K=100$	$K=200$
Movies and TV
SASRec	$1.165$ [ $1.152$ , $1.183$ ]	$1.205$ [ $1.194$ , $1.223$ ]	$1.184$ [ $1.174$ , $1.200$ ]	$1.189$ [ $1.173$ , $1.216$ ]
DCNv2	$0.565$ [ $0.556$ , $0.569$ ]	$0.557$ [ $0.552$ , $0.565$ ]	$0.548$ [ $0.544$ , $0.555$ ]	$0.554$ [ $0.549$ , $0.561$ ]
Qwen2.5 7B (P)	$606.545$ [ $524.313$ , $1305.220$ ]	$1526.348$ [ $1310.938$ , $3233.704$ ]	$3044.906$ [ $2597.272$ , $6449.878$ ]	$6072.933$ [ $5200.285$ , $12\,840.475$ ]
Qwen2.5 7B (L)	$63.604$ [ $61.785$ , $173.579$ ]	$120.303$ [ $116.140$ , $363.540$ ]	$202.384$ [ $196.555$ , $680.471$ ]	$398.461$ [ $382.391$ , $1413.075$ ]
Qwen2.5 32B (P)	$2520.937$ [ $2163.036$ , $5395.480$ ]	$6252.235$ [ $5365.620$ , $13\,302.156$ ]	$12\,465.888$ [ $10\,672.714$ , $26\,490.238$ ]	$24\,925.080$ [ $21\,354.697$ , $52\,958.626$ ]

Appendix

Table 9: Distribution of observed per-user inference latency across three recommendation domains and four candidate-set sizes. Each entry reports median [P25, P75] latency in milliseconds. (P) and (L) denote pointwise and listwise reranking, respectively. Qwen2.5 7B and 32B refer to the corresponding instruction-tuned models. Hosted API latencies include network communication and provider-side processing and are not hardware-normalized against locally executed methods.

LLM Reranking

Large language model reranking uses LLMs to reorder retrieved documents or candidate outputs by relevance, usefulness, or task-specific criteria.

See all 126 papers

One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation
Apr 30, 2026Ethan Bito, Yongli Ren, Estrid HeLearning to Rank LLM Reranking
RMIT University, Melbourne, Australia
Decision4Rec: A Jev-Inspired Framework of Decision Models on Knowledge and Active Graphs for Multi-Agent Recommender Systems
Oct 19, 2026Marco Polignano, Het Darshan Mehta, Pasquale Lops +2Recommender Systems Large Language Model-Based Recommendation
Università degli studi di Bari Aldo Moro, Piazza Umberto I, 70121 Bari, Italy · Otto-von-Guericke University, Universitätsplatz 2, 39106 Magdeburg, Germany · Leibniz Institute for Educational Media | Georg Eckert Institute, Freisestr. 1, 38118 Braunschweig, Germany
GR2: Generative Reasoning Re-ranker
Feb 8, 2026Mingfu Liang, Yufei Li, Jay Xu +20RL for Language Models RL for Recommendation
Meta AI

Apr 30, 2026J/K move · Enter open · S save

One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation

Ethan Bito, Yongli Ren, Estrid He

RMIT University, Melbourne, Australia

Learning to Rank LLM Reranking Large Language Model-Based Recommendation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used for recommendation reranking, but their listwise predictions can depend on the order in which candidates are presented. This creates a mismatch between the set-based nature of recommendation and the sequence-based computation of decoder-only LLMs, where permuting an otherwise identical candidate set can change item scores and final rankings. Such order sensitivity makes LLM-based rerankers difficult to rely on, since rankings may reflect prompt serialization rather than user preference. We propose InvariRank, a permutation-invariant listwise reranking framework that addresses this dependence at the architectural level. InvariRank blocks cross-candidate attention with a structured attention mask and negates position-induced scoring changes through shared positional framing under Rotary Positional Embeddings (RoPE). Combined with a listwise learning-to-rank objective, the model scores all candidates in a single forward pass, avoiding permutation-based invariance training objectives that require multiple permutations of a candidate set. Experiments on recommendation benchmarks show that InvariRank maintains competitive ranking effectiveness while producing stable rankings across candidate permutations. The results suggest that architectural invariance is a practical route to reliable and efficient LLM-based recommendation reranking. The source code is at https://github.com/ejbito/InvariRank.

Abstract

LLM Reranking

Explore similar work

One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation

Decision4Rec: A Jev-Inspired Framework of Decision Models on Knowledge and Active Graphs for Multi-Agent Recommender Systems

GR2: Generative Reasoning Re-ranker