Large language models (LLMs) are attractive rerankers for recommendation. Running an expensive model on every request is costly, and for users with sparse interaction histories it often lowers quality. We present CascadeRec, a two-tier LLM reranking cascade deployed in production at a large financial planning and analysis (FP&A) SaaS provider. It serves content recommendations to enterprise tenants on AWS EKS and GKE. A lightweight Tier-1 model reranks every request. A router escalates only the requests where extra inference-time compute is likely to help, and sends them to a more expensive Tier-2 procedure (self-consistency sampling, multi-path reasoning, or a larger model). The system has run in a production A/B experiment for over a month. Compared with uniform use of the expensive tier, CascadeRec has reduced inference cost by about 10%, with minimal latency impact and no quality regression observed to date. Tenant data cannot be released, so we also evaluate the pipeline on public benchmarks (MovieLens-1M and Amazon-Electronics). Segment-level analysis with paired-bootstrap confidence intervals shows that uniform LLM reranking lowers NDCG@10 for sparse-history users by up to 30%, and its only statistically significant gain appears on the segment with the longest histories. Escalating cold users can make the loss larger instead of repairing it. Which segments benefit from escalation depends on the domain, and this motivates the segment-gated rollout we use in production. We describe the deployment architecture, the measured benefits, and lessons learned, including inter-tenant load imbalance as a driver for cascading and the handling of LLM output-parsing failures at the serving layer.
Figures & tables
Figure 1: CascadeRec architecture. Tier-1 reranks every request and emits a confidence signal. The router accepts the Tier-1 ranking or escalates the request to Tier-2.
Figure 2: Production request path. The router escalates a minority of requests to the Tier-2 pool. Any LLM failure falls back to the best earlier ranking (dashed red). Orange boxes are LLM inference.
Model
Method
k=1
k=3
k=5
k=10
k=20
MovieLens-1M
–
B0
.059
.076
.074
.071
.038
Mistral
B1
.053
.070
.066
.070
.048 ∗
CR+SC
.054
.066
.069
.070
.048 ∗
CR+MP
.053
.074
.071
.070
.048 ∗
Qwen
B1
.061
.070
.073
.069
.041
CR+SC
.059
.062 ∗
.076
.069
.041
Table 1: NDCG@10 by cold-start bin (200 users per bin). ∗ : the 95% paired-bootstrap CI of the delta vs. B0 excludes zero. CR uses τ=5 , so bins with k≤5 are escalated.
Model
Method
NDCG@10
MRR
Esc.
Tok
MovieLens-1M
–
B0 No-Rerank
0.0636
0.0447
0%
0
Mistral
B1 Uniform
0.0612
0.0414
0%
1335
Mistral
CR( τ =5)+SC
0.0614
0.0415
60%
3313
Mistral
CR( τ =5)+MP
0.0633
0.0440
60%
3376
Qwen
B1 Uniform
0.0629
0.0436
0%
668
Table 2: Aggregate results (1,000 users per dataset). Bold : best NDCG@10 per dataset. Underline : best within each LLM group. Esc.: share of escalated users. Tok: average tokens per user.
Figure 3: Cost and quality on MovieLens-1M: aggregate NDCG@10 against average tokens per user, with values from Table 2 . The dashed line marks B0.
k
NDCG@10
R@10
R@20
R@100
1
0.059
0.115
0.205
0.400
3
0.076
0.140
0.200
0.440
5
0.074
0.140
0.240
0.480
10
0.071
0.155
0.265
0.500
20
0.038
0.085
0.155
0.440
Table 3: First-stage retrieval quality on MovieLens-1M (iALS fold-in). R@10 bounds what head-10 reranking can change. R@100 bounds any reranker on this candidate set.
Dataset
Model
Tier-1 fail
Tier-2 fail
MovieLens
Mistral
0.8–6.5%
2.0–3.7%
MovieLens
Qwen (freeform)
96.9%
–
MovieLens
Qwen (constrained)
10.4%
5.8–7.0%
Amazon
Mistral
0.2%
0.2–0.3%
Amazon
Qwen (constrained)
0.1%
0.0–0.2%
Table 4: Output-compliance failure rates in the offline matrix: the share of requests whose LLM output had no parseable ranking. Such requests fall back silently.
Large language model-based recommendation uses language models to personalize ranking, retrieval, and conversational suggestions from text and behavior.
Serving open-weight language models on premises forces a trade-off between accuracy and computational cost. Hosting models of different sizes lets easy queries go to a small, efficient model and hard ones to a large model. We present CRE-Router (Cluster, Route, Escalate), a two-stage method that decides which of these models answers each query. It clusters queries and routes each cluster to the model that best trades error against cost, under a budget on the time per output token (TPOT) or the end-to-end latency per request (E2EL). It then either accepts the answer or, when a quality estimator rejects it, escalates the query to the strong model. Routing decides before generation, so a hard query goes straight to the strong model; escalation decides after, so it can catch the errors routing cannot predict. It needs no annotation beyond task-correctness labels. We evaluate CRE-Router on TeleMath, TeleQnA and AIME, with pools of two to nine models served on A100 GPUs. CRE-Router is more accurate than the efficient models in its pool and comparable to the strong model it escalates to, at a lower cost per token. We also find that the choice of cost term decides which models serve the workload, that pricing each model per cluster shortens the wait per request, and that a change of hardware changes the best routing.
LLM cascades, in which a cheap model defers to an expensive one on low-confidence queries, are widely used to reduce inference cost. Given a pool of models, a practitioner must decide how many models to include and where to set each deferral threshold. We derive first-order optimality conditions showing that, at an optimum, the ratio of expected accuracy gain to expected downstream cost is equal across deferral boundaries. A local search based on these conditions closely matches exhaustive search. We also derive an identity that decomposes the accuracy gain of score-based escalation over random escalation into two AUROC terms. Across five benchmarks and nine deferral scores, with model sequences and thresholds optimized from a pool of eight models, two-model cascades improve mean test-set accuracy over single-model selection by 2.1 to 8.2 percentage points. However, allowing more than two models does not improve mean test-set accuracy in 118 of 135 comparisons across scorers, datasets, and depth caps, and adds at most 0.43 percentage points. To understand the role of deferral scores in depth gains, we conduct counterfactual experiments with simulated confidence scores. When these scores have high AUROC and reflect only whether the current model answered correctly, allowing more than two models improves test-set accuracy on four of five benchmarks. However, these gains do not persist when the scores also reflect query difficulty shared across models, even at the same AUROC. These results suggest that gains from additional depth depend on how well the confidence score separates correct from incorrect answers for the current model compared with later models.
Large language models (LLMs) have shown promise for recommendation reranking, but their use introduces an important trade-off between recommendation quality and serving efficiency. We investigate whether decision-oriented systems provide a useful alternative when the reranking task is fundamentally a structured choice among predefined candidate items. Specifically, we conduct a controlled empirical study of two decision-oriented systems, TypeSafe Jev and OpenAI Decisions, for recommendation reranking. Across three Amazon Reviews domains and four candidate-set sizes, we compare these systems against recommendation-specific models and pointwise and listwise LLM-based rerankers, systematically evaluating recommendation effectiveness, candidate-order robustness, and efficiency and scalability. Our results reveal that Jev frequently achieves a favorable quality--latency trade-off, offering competitive recommendation effectiveness at substantially lower observed latency than stronger LLM-based alternatives. Overall, our findings suggest that specialization for decision-making does not inherently guarantee superior recommendation quality or robustness. Instead, the practical value of decision-oriented inference depends on implementation-specific characteristics and application requirements. These findings motivate further investigation of decision-oriented systems for recommendation and other ranking tasks with structured output spaces.