FlowWise: Stateful Fast-Slow Model Serving for Streaming Traffic Intelligence

Shinan Liu; Ted Shaowang; Gerry Wan; Jeewon Chae; Jonatas Marques; Sanjay Krishnan; Nick Feamster

cs.NIFeb 6, 2024

FlowWise: Stateful Fast-Slow Model Serving for Streaming Traffic Intelligence

Authors: Shinan Liu, Ted Shaowang, Gerry Wan, Jeewon Chae, Jonatas Marques, Sanjay Krishnan, Nick Feamster

Organizations: University of Chicago, Chicago, IL, USA · Stanford University, Stanford, CA, USA

Abstract

Network traffic analysis in cloud and internet service provider (ISP) networks increasingly relies on machine learning (ML) models, as encryption and service consolidation make rule-based classification unreliable. At high flow-arrival rates, however, new flows can arrive faster than end-to-end analysis pipelines can process. Unlike typical cloud inference workloads, packets from the same flow arrive over time and require per-flow state. This streaming, stateful setting limits naive scale-out with stateless workers. This paper presents FlowWise, a serving system for streaming, stateful flow intelligence on commodity CPU/GPU servers that jointly selects (i) how many packets to collect and (ii) which model(s) to apply for each flow to balance latency, service rate, and accuracy. Across three representative tasks, inference time across candidate models differs by 1.8x-141.3x, while packet waiting time can exceed feature computation and inference by 6-8 orders of magnitude at the tail. FlowWise exploits these gaps with an uncertainty-driven fast-slow cascade: it issues early predictions from a one-packet model and escalates only uncertain flows to slower models that wait for additional packets and use richer features. A queue-centric runtime maintains per-flow state, performs asynchronous feature joining, and uses brokered, flow-ID-keyed streams to distribute workload across heterogeneous workers. FlowWise serves 76.3% of flows within 16 ms, improving median end-to-end serving latency by 40.5x while increasing service rate and maintaining similar accuracy to always running the slow model. With thousands of packet-header features per flow, our scale-up benchmark sustains over 48.5k new flows per second on a single 16-core commodity CPU server, comparable to reported flow-arrival rates in city-backbone traces. FlowWise is open sourced at https://github.com/noise-lab/FlowWise.

Figures & tables

Figure 1. Traffic analysis holds a tradeoff between latency and accuracy. FlowWise dynamically assigns predictions to either a slower, more accurate model (waiting for more packets or using a sophisticated design) or a faster, less accurate model (predicting on the first packet). The chart compares its request assignment efficacy against an oracle (using ground-truth classification correctness) and random assignment, based on the service recognition dataset.

Panel of Figure 2 — Figure 2. Model serving for network traffic analysis.

Application	Model	F1 (1pkt)	F1(5pkts)	F1(10pkts)
Service Recognition	DT	$0.845$	$0.907$	$0.909$
	LGBM	$0.921$	$0.967$	$0.973$
	CNN	$0.887$	$0.929$	$0.930$
Device Identification	DT	$0.812$	$0.824$	$0.827$
	LGBM	$0.913$	$0.918$	$0.919$
	CNN	$0.881$	$0.910$	$0.911$

Table 1. Empirically, adding more packets increases the (weighted) F1 score. DT: Decision Tree, LGBM: LightGBM, CNN: Convolutional Neural Networks.

	Serv. Rec.		Dev. Id.		QoE Inf.
Model	F1	Inf. Time	F1	Inf. Time	F1	Inf. Time
DT	0.845	0.052	0.812	0.053	0.715	0.053
RF	0.859	0.107	0.874	0.106	0.709	0.098
LGBM	0.921	0.139	0.913	0.210	0.723	0.162
XGBoost	0.672	0.117	0.507	0.152	0.646	0.127
CNN	0.887	0.481	0.881	0.491	0.710	0.385

Table 3. Different model performance for the 1st packet. Deeper green means higher F1 score, and deeper blue means lower model inference latency (in ms).

Figure 4. FlowWise , which supports the fast-slow model serving architecture, is tailored to the discrete arrival of network flow packets. The arrow width represents the proportion of flows processed at each stage.

Figure 5. The F1 score vs. end-to-end latency (in ms) across 5 models for the service recognition dataset. We use the Pareto Front to determine the placement of models in FlowWise .

Figure 6. FlowWise supports parallelism across heterogeneous hardware.

Method	Packet Depth	Feature	Model(s)	Queue
LEXNet ( Fauvel et al., 2023 )	N	Pkt Size & Direction	LEXNet (CNN)	No
FastTraffic ( Xu et al., 2023 )	N	N-gram of Raw Pkts	3-layer MLP	No
Best Effort	N	nPrint	LGBM	No
Queueing	N	nPrint	LGBM	Yes
FlowWise	1+N	nPrint	DT+LGBM+LGBM	Yes

Table 4. Comparison of evaluated traffic analysis frameworks. N is the best packet depth for an application (10 for service recognition, 3 for device identification, 20 for QoE inference.)

Panel of Figure 7 — Figure 7. Performance of different approaches for the service recognition task (other tasks are in Appendix D ). We compare FlowWise against four baselines, all considering 10-packet inputs. The F1 score is computed on the non-missed flows.

Panel of Figure 8 — Figure 8. End-to-end latency analysis of 4k flows per second on a single consumer ( i.e. , 1 CPU core) for service recognition : (a) The CDF of single-consumer latency demonstrates that FlowWise can accelerate 76% of flows to millisecond-level service. (b) A breakdown of latency at different stages illustrates the reduction of packet waiting time, showing the efficiency of FlowWise in minimizing overall latency.

Panel of Figure 9 — Figure 9. Assigned portion of requests (#assigned flows / #all flows) vs. assigned incorrect portion (#assigned incorrect predictions / #all incorrect predictions) using different approaches on the fastest model. T: Uncertainty threshold for the Uncertainty and Per-class Uncertainty approaches.

Panel of Figure 10 — Figure 10. Assigned portion of requests vs. F1 score for service recognition . Based on models, FlowWise is best configured to use Uncertainty for the first assignment, and Per-Class Uncertainty for the second.

Table 5. Comparison of Normalized AUC (assigned portion vs. improved F1 score normalized on the Oracle results) across different assignment methods. We vary models in a fastest-fast-slow (DT 1 pkt + LGBM 1 pkt + LGBM 10 pkts) architecture. Highlighted cells indicate the best performance in each. This is evaluated using service recognition dataset, but similar trends are observed with other datasets.

Figure 11. FlowWise service rate with the # of consumers.

Packet Number	2	4	6	8	10
Final F1 Score	0.945	0.956	0.960	0.962	0.963
Average Latency	132.7	188.5	323.0	358.8	372.2
Median Latency	11.2	11.1	11.3	11.2	11.2
Max Service Rate	4285	4210	4234	4226	4319

Table 6. FlowWise performance when varying the number of packets on the slow model, for the service recognition task. The unit of latency is ms, and the unit of service rate is classified flows per second.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

function GetPerClassSlope( $D,U$ )
	Initialize Results $R$
	for each Class Label $L$ in $D$ do
		Partition $D$ into Correct $C$ and Incorrect $I$ for $L$ $\triangleright$ Classify predictions
		Get Thresholds $T$ from $U$ quantiles for $L$
		Update $R$ with assigned portions for $C$ and $I$ $\triangleright$ Track portions
	end for

Appendix

Algorithm 1 Slope-based Per-Class Uncertainty Thresholds

function GetUniversalThresholds( $D$ )
	Initialize Uncertainty List $U$
	for each $DataPoint$ in Dataset $D$ do
		Compute Uncertainty $u$ for $DataPoint$ $\triangleright$ $D$ is validation data
		Append $u$ to $U$
	end for
	Sort $U$ in ascending order

Appendix

Algorithm 2 Universal Uncertainty Thresholds

#Cons.	All CPU	Half CPU+Half GPU
1	4.319k	-
2	8.805k	8.620k
4	16.692k	16.484k
8	31.114k	30.651k
12	40.238k	41.009k
16	48.505k	47.040k

Appendix

Table 7. Maximum service rate (new classified flows per second) when we increase the number of consumers.

Service	Application	# of Flows
Video Conferencing ( MacMillan et al., 2021 )	Zoom	1312
	Google Meet	1313
	MS Teams	3886
Video Streaming ( Bronzino et al., 2019 )	Twitch	1150
	Amazon	1509
	YouTube	2702

Appendix

Table 8. Service recognition dataset.

Device	# of Flows
Samsung Fridge	3770
Nvidia Jetson Nano	3770
Amazon Echo 3rd Gen	3770
Google Home	3770
Raspberry Pi 3	3770
SmartThings Dishwasher	3770

Appendix

Table 9. Device identification dataset.

VCA	Total # of Seconds
Meet	10189
Webex	12331
Teams	14408
Total	36928

Appendix

Table 10. Video conferencing application (VCA) QoE measurement dataset.

Panel of Figure 12 — Figure 12. Performance of different approaches for the device identification task. We compare FlowWise against four baselines, all considering three-packet inputs. The F1 score is computed on the flows that are not missed.

Panel of Figure 13 — Figure 13. Assigned portion of requests vs. F1 score at each stage for device identification task.

Panel of Figure 14 — Figure 14. Assigned portion of requests vs. F1 score at each stage for QoE measurement task.

Cascaded Inference

See all 19 papers

FlexiFlow: Bandit-based Model Switching in ML Workflows
Oct 5, 2026Abhilash Jindal, Todd Nief, Bhanu Prakash Vangala +6Multi-Armed Bandits Inference-Time Optimization
IIT Delhi India · University of Chicago USA · University of Missouri USA +1
A First Glance at Jev for Network Traffic Classification: Accuracy, Processing Time, and Cost
Sep 30, 2026Shenghe Xu, Lifan MeiCybersecurity Few-Shot Learning
Amazon.com, Inc. · Xi’an Jiaotong-Liverpool University
Network-Aware Forecasting on Wireless Access Points
Sep 2, 2026Niloo Bahadori, Swadhin Pradhan, Peiman AminiEfficient Neural Network Inference Edge Computing
Cisco Systems

Oct 5, 2026J/K move · Enter open · S save

FlexiFlow: Bandit-based Model Switching in ML Workflows

Abhilash Jindal, Todd Nief, Bhanu Prakash Vangala, Shankaradithyaa V+5

IIT Delhi India · University of Chicago USA · University of Missouri USA+1

Multi-Armed Bandits Inference-Time Optimization Automated Machine Learning

Open paper AskPDF arXiv

Model optimizations help improve inference performance and accuracy of ML workflows. However, relying on a single model to perform inference across all data batches often fails to maximize accuracy and thus overall performance. In many cases, alternate models could perform better on specific subsets of data where a primary model underperforms. Our experiments with real ML workflows indeed show that switching models improves workflow accuracy by up to 23%. Yet, current systems lack the ability to adaptively switch between models based on performance, forcing users to manually test models in sequence. We present FlexiFlow, a dataflow system that dynamically switches between alternate models when the current model exhibits low accuracy. FlexiFlow learns to rank models using a novel multi-armed bandit approach that accounts for model runtimes, probability of passing user-defined assertions, and the computational structure of the ML workflow. We show that the standard Thompson sampling approach is insufficient for switching models in ML workflows. In contrast, our proposed approaches are effective and scales to complex real-world ML workflows. Experiments show that switching models at runtime while reusing intermediate results provides higher accuracy, but also 48% efficiency gain compared to sequential workflow runs.

Abstract

Cascaded Inference

Explore similar work

FlexiFlow: Bandit-based Model Switching in ML Workflows

A First Glance at Jev for Network Traffic Classification: Accuracy, Processing Time, and Cost

Network-Aware Forecasting on Wireless Access Points