MUSE: Multi-Tenant Model Serving With Seamless Model Updates
Authors: Cláudio Correia, Alberto E. A. Ferreira, Lucas Martins, Miguel P. Bento, Sofia Guerreiro, Ricardo Ribeiro Pereira, Ana Sofia Gomes, Jacopo Bono, Hugo Ferreira, Pedro Bizarro
In binary classification systems, decision thresholds translate model scores into actions. Choosing suitable thresholds relies on the specific distribution of the underlying model scores but also on the specific business decisions of each client using that model. However, retraining models inevitably shifts score distributions, invalidating existing thresholds. In multi-tenant Score-as-a-Service environments, where decision boundaries reside in client-managed infrastructure, this creates a severe bottleneck: recalibration requires coordinating threshold updates across hundreds of clients, consuming excessive human hours and leading to model stagnation. We introduce MUSE, a model serving framework that enables seamless model updates by decoupling model scores from client decision boundaries. Designed for multi-tenancy, MUSE optimizes infrastructure re-use by sharing models via dynamic intent-based routing, combined with a two-level score transformation that maps model outputs to a stable, reference distribution. Deployed at scale by Feedzai, MUSE processes over a thousand events per second, and over 55 billion events in the last 12 months, across several dozens of tenants, while maintaining high-availability and low-latency guarantees. By reducing model lead time from weeks to minutes, MUSE promotes model resilience against shifting attacks, saving millions of dollars in fraud losses and operational costs.
Figures & tables
Figure 1. MUSE infrastructure overview with three predictors ( p1 , p2 , and p3 ) serving scores. Both p1 and p2 are ensembles composed by the individual models m1 , m2 and m1 , m2 , m3 , respectively, while p3 is an individual model. The upper section details the data pipeline for predictor p2 during scoring.
Figure 3. The model lifecycle: from training to shadow validation, and finally to live promotion.
Figure 4. Quantile Transformation update for a “cold-start” deployment. Comparison of the relative error against the target distribution for predictor v0 (Default Transformation), predictor v1 (Custom Transformation), and predictor raw (No Quantile Transformation).
Figure 5. System performance during the update from Tv0Q to Tv1Q . Despite K8s pod restarts, the warm-up process ensures strict adherence to latency targets throughout the transition.
Figure 6. Expert Update - Relative error to target distribution. Replacing an expert without updating the transformation (squares) leads to significant over-alerting for thresholds in the [0.2,0.4] range. Pairing the new predictor with an updated transformation (circles) maintains target alignment.
Dataset
Predictor
PC β
Error
Without PC
With PC
Change
Validation Data m1
Expert m1
≈18%
ECE
4.97\times10−3
7.67\times10−4
−84.6%
Brier
1.46\times10−4
9.61\times10−5
−34.2%
Validation Data m2
Expert m2
≈18%
ECE
4.83\times10−2
8.63\times10−3
−82.2%
Brier
2.70\times10−3
4.36\times10−4
−83.9%
Table 1. Brier score and ECE calibration errors (lower is better) before and after Posterior Correction (PC), for each expert in their respective validation dataset; and for live client data, for each expert and the predictor p2 from Section 3.2 .