Cascaded Inference

Follow this topic Ask about this topic

Latest papers

22 papers3 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

CascadeRec: Deployed Adaptive LLM Compute Allocation for Multi-Tenant Content Recommendation
Nov 5, 2026Kai Zhao, Guilin Zhang, Xiang Cheng +4Large Language Model-Based Recommendation LLM Reranking
Workday AI Research
Generative Forecasting from Coarse-Grained Observations: When Does Cascading Help?
Oct 31, 2026Alexandra Spitzer, Giovanni Abati, Lorenzo Baldassari +3Probabilistic Forecasting Cascaded Inference
Department of Mathematics and Computer Science, University of Basel, Switzerland · Département d’Informatique, École Normale Supérieure, PSL University, Paris, France
Bayes-Consistent Losses for Adaptive Inference in Classification
Oct 29, 2026Shirley Xiaoqi Liu, Florence Regol, Antonios Valkanas +2Adaptive Model Routing Adaptive Inference
University of Oxford · McGill University
System Switch: When Should a Fast Decision Model Stop and Think?
Oct 7, 2026Gian Luca BailoCascaded Inference LLM Decision-Making
Independent Researcher, Recco (Genoa), Italy
Safety-Constrained Cascade Inference for Robust Malaria Cell Classification Under Field Corruptions
Sep 26, 2026J. T. Hagbe, Michel EmelCascaded Inference Medical Image Classification
African School of Economics, Abomey-Calavi, Benin · African Institute for Mathematical Sciences (AIMS), Cameroon
Calibration-Aware Uncertainty Cascades for Efficient Heterogeneous Model Collaboration
Sep 10, 2026Yilin Zhang, Han Jiang, Cai Xu +2Selective Prediction Cost-Aware Inference
School of Computer Science and Technology, Xidian University Xi’an, China
Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference
Sep 7, 2026Zheyuan Wang, Siyu Li, Peiqiao Song +3LLM Inference Efficiency LLM Routing
School of Artificial Intelligence, Beijing Normal University · School of Information Science and Engineering, Chongqing Jiaotong University · School of Computer Science and Technology, Dalian University of Technology +1
Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades
Sep 1, 2026Dushyant Rajput, Nirdesh Chauhan, Siddharth KosarajuLLM Inference Efficiency LLM Answer Verification
AltSlate Labs LLP
RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning
Aug 16, 2026Shihong Huang, Shengjie Wang, Hong Ma +1LLM Routing Language Model Cascading
Polytechnic Institute, Zhejiang University, Hangzhou, China · Department of Logistics and Maritime Studies, Faculty of Business, The Hong Kong Polytechnic University, Hong Kong, China
CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing
Aug 7, 2026Yan Zhou, Yue Ouyang, Kaiyang Zheng +1Cost-Aware Inference Test-Time Scaling
School of Mathematics and Statistics, Changsha University of Science and Technology, Changsha, China · School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China.
CARE: A Cascaded Framework for Efficient and Reliable Time Series Anomaly Detection
Aug 3, 2026Zemin Chao, Qianhui Xu, Jianhe Cen +3Efficient Neural Network Inference Cost-Aware Inference
Massive Data Computing Lab, Harbin Institute of Technology
Safety-Aware Cascaded Inference for Crop Damage Assessment with Controlled Error Trade-offs
Jul 28, 2026José Thiéry Messigbédé Hagbe, Gani Kawsar Gounou, Songbian Karim ZiméAgricultural Image Analysis Agricultural Remote Sensing
aAfrican School of Economics, www.africanschoolofeconomics.com, Abomey Calavi, Benin
Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference
Jul 27, 2026Yifan Dou, Shikan Lian, Shibo LiLLM Inference Efficiency LLM Inference Scheduling
Department of Computer Science, Florida State University
Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving
Jun 25, 2026Yasmin Moslem, Magdalena Kacmajor, Vasudevan Nedumpozhimana +11LLM Serving Cost-Aware Inference
ADAPT Centre, Trinity College Dublin · Huawei Research
SAFE-Cascade: Cost-Adaptive Vision-Language Routing for Chart Question Answering
Jun 17, 2026Ayush Dwivedi, Qixin Wang, Ashvi Soni +5Multimodal Understanding Efficient VLM Inference
Walmart USA · University of Arkansas USA
Forced Deferral: Manipulating Routing Decisions in Multimodal LLM Cascades
Jun 13, 2026Zhongye Liu, Yaopei Zeng, Yurui Chang +1Adversarial Perturbations Multimodal Robustness
Pennsylvania State University
CADS: Conformal Adaptive Decision System for Cost-Efficient Image Classification
May 12, 2026Mikael Turkoglu, Tim Bary, Vincent Thielens +2Adaptive Inference Cost-Aware Inference
1ICTEAM, UCLouvain, Belgium · 2SAFiR Lab, Univ. of Sherbrooke, Canada · 3Univ. of Mons, Belgium
UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing
May 11, 2026Varun KotteCost-Aware Inference Language Model Calibration
Independent Researcher.
Is Escalation Worth It? On the Depth of LLM Cascades
May 7, 2026Dylan BouchardCost-Aware Inference LLM Routing
Streaming Model Cascades for Semantic SQL
Apr 1, 2026Paweł Liskowski, Kyle SchmausCost-Aware Inference LLM Routing
Snowflake Inc. · Poznań, Poland · San Francisco, CA, USA
Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference
Feb 23, 2026Arindam KhaledAnchoring Bias in Language Models LLM Routing
Independent Researcher
FlowWise: Stateful Fast-Slow Model Serving for Streaming Traffic Intelligence
Feb 6, 2024Shinan Liu, Ted Shaowang, Gerry Wan +4Cascaded Inference Efficient Neural Network Inference
University of Chicago, Chicago, IL, USA · Stanford University, Stanford, CA, USA

Nov 5, 2026J/K move · Enter open · S save

CascadeRec: Deployed Adaptive LLM Compute Allocation for Multi-Tenant Content Recommendation

Kai Zhao, Guilin Zhang, Xiang Cheng, Xiquan Cui+3

Workday AI Research

Large Language Model-Based Recommendation LLM Reranking LLM Routing LLM Serving Cost-Aware Inference Cascaded Inference

Open paper AskPDF arXiv

Large language models (LLMs) are attractive rerankers for recommendation. Running an expensive model on every request is costly, and for users with sparse interaction histories it often lowers quality. We present CascadeRec, a two-tier LLM reranking cascade deployed in production at a large financial planning and analysis (FP&A) SaaS provider. It serves content recommendations to enterprise tenants on AWS EKS and GKE. A lightweight Tier-1 model reranks every request. A router escalates only the requests where extra inference-time compute is likely to help, and sends them to a more expensive Tier-2 procedure (self-consistency sampling, multi-path reasoning, or a larger model). The system has run in a production A/B experiment for over a month. Compared with uniform use of the expensive tier, CascadeRec has reduced inference cost by about 10%, with minimal latency impact and no quality regression observed to date. Tenant data cannot be released, so we also evaluate the pipeline on public benchmarks (MovieLens-1M and Amazon-Electronics). Segment-level analysis with paired-bootstrap confidence intervals shows that uniform LLM reranking lowers NDCG@10 for sparse-history users by up to 30%, and its only statistically significant gain appears on the segment with the longest histories. Escalating cold users can make the loss larger instead of repairing it. Which segments benefit from escalation depends on the domain, and this motivates the segment-gated rollout we use in production. We describe the deployment architecture, the measured benefits, and lessons learned, including inter-tenant load imbalance as a driver for cascading and the handling of LLM output-parsing failures at the serving layer.