Network traffic analysis in cloud and internet service provider (ISP) networks increasingly relies on machine learning (ML) models, as encryption and service consolidation make rule-based classification unreliable. At high flow-arrival rates, however, new flows can arrive faster than end-to-end analysis pipelines can process. Unlike typical cloud inference workloads, packets from the same flow arrive over time and require per-flow state. This streaming, stateful setting limits naive scale-out with stateless workers. This paper presents FlowWise, a serving system for streaming, stateful flow intelligence on commodity CPU/GPU servers that jointly selects (i) how many packets to collect and (ii) which model(s) to apply for each flow to balance latency, service rate, and accuracy. Across three representative tasks, inference time across candidate models differs by 1.8x-141.3x, while packet waiting time can exceed feature computation and inference by 6-8 orders of magnitude at the tail. FlowWise exploits these gaps with an uncertainty-driven fast-slow cascade: it issues early predictions from a one-packet model and escalates only uncertain flows to slower models that wait for additional packets and use richer features. A queue-centric runtime maintains per-flow state, performs asynchronous feature joining, and uses brokered, flow-ID-keyed streams to distribute workload across heterogeneous workers. FlowWise serves 76.3% of flows within 16 ms, improving median end-to-end serving latency by 40.5x while increasing service rate and maintaining similar accuracy to always running the slow model. With thousands of packet-header features per flow, our scale-up benchmark sustains over 48.5k new flows per second on a single 16-core commodity CPU server, comparable to reported flow-arrival rates in city-backbone traces. FlowWise is open sourced at https://github.com/noise-lab/FlowWise.
Figures & tables
Figure 1. Traffic analysis holds a tradeoff between latency and accuracy. FlowWise dynamically assigns predictions to either a slower, more accurate model (waiting for more packets or using a sophisticated design) or a faster, less accurate model (predicting on the first packet). The chart compares its request assignment efficacy against an oracle (using ground-truth classification correctness) and random assignment, based on the service recognition dataset.
Figure 2. Model serving for network traffic analysis.
Application
Model
F1 (1pkt)
F1(5pkts)
F1(10pkts)
Service Recognition
DT
0.845
0.907
0.909
LGBM
0.921
0.967
0.973
CNN
0.887
0.929
0.930
Device Identification
DT
0.812
0.824
0.827
LGBM
0.913
0.918
0.919
CNN
0.881
0.910
0.911
Table 1. Empirically, adding more packets increases the (weighted) F1 score. DT: Decision Tree, LGBM: LightGBM, CNN: Convolutional Neural Networks.
Figure 4
Serv. Rec.
Dev. Id.
QoE Inf.
Model
F1
Inf. Time
F1
Inf. Time
F1
Inf. Time
DT
0.845
0.052
0.812
0.053
0.715
0.053
RF
0.859
0.107
0.874
0.106
0.709
0.098
LGBM
0.921
0.139
0.913
0.210
0.723
0.162
XGBoost
0.672
0.117
0.507
0.152
0.646
0.127
CNN
0.887
0.481
0.881
0.491
0.710
0.385
Table 3. Different model performance for the 1st packet. Deeper green means higher F1 score, and deeper blue means lower model inference latency (in ms).
Figure 4. FlowWise , which supports the fast-slow model serving architecture, is tailored to the discrete arrival of network flow packets. The arrow width represents the proportion of flows processed at each stage.
Figure 5. The F1 score vs. end-to-end latency (in ms) across 5 models for the service recognition dataset. We use the Pareto Front to determine the placement of models in FlowWise .
Figure 6. FlowWise supports parallelism across heterogeneous hardware.
Method
Packet Depth
Feature
Model(s)
Queue
LEXNet ( Fauvel et al., 2023 )
N
Pkt Size & Direction
LEXNet (CNN)
No
FastTraffic ( Xu et al., 2023 )
N
N-gram of Raw Pkts
3-layer MLP
No
Best Effort
N
nPrint
LGBM
No
Queueing
N
nPrint
LGBM
Yes
FlowWise
1+N
nPrint
DT+LGBM+LGBM
Yes
Table 4. Comparison of evaluated traffic analysis frameworks. N is the best packet depth for an application (10 for service recognition, 3 for device identification, 20 for QoE inference.)
Figure 7. Performance of different approaches for the service recognition task (other tasks are in Appendix D ). We compare FlowWise against four baselines, all considering 10-packet inputs. The F1 score is computed on the non-missed flows.
Figure 8. End-to-end latency analysis of 4k flows per second on a single consumer ( i.e. , 1 CPU core) for service recognition : (a) The CDF of single-consumer latency demonstrates that FlowWise can accelerate 76% of flows to millisecond-level service. (b) A breakdown of latency at different stages illustrates the reduction of packet waiting time, showing the efficiency of FlowWise in minimizing overall latency.
Figure 9. Assigned portion of requests (#assigned flows / #all flows) vs. assigned incorrect portion (#assigned incorrect predictions / #all incorrect predictions) using different approaches on the fastest model. T: Uncertainty threshold for the Uncertainty and Per-class Uncertainty approaches.
Figure 10. Assigned portion of requests vs. F1 score for service recognition . Based on models, FlowWise is best configured to use Uncertainty for the first assignment, and Per-Class Uncertainty for the second.
Table 5. Comparison of Normalized AUC (assigned portion vs. improved F1 score normalized on the Oracle results) across different assignment methods. We vary models in a fastest-fast-slow (DT 1 pkt + LGBM 1 pkt + LGBM 10 pkts) architecture. Highlighted cells indicate the best performance in each. This is evaluated using service recognition dataset, but similar trends are observed with other datasets.
Figure 11. FlowWise service rate with the # of consumers.
Packet Number
2
4
6
8
10
Final F1 Score
0.945
0.956
0.960
0.962
0.963
Average Latency
132.7
188.5
323.0
358.8
372.2
Median Latency
11.2
11.1
11.3
11.2
11.2
Max Service Rate
4285
4210
4234
4226
4319
Table 6. FlowWise performance when varying the number of packets on the slow model, for the service recognition task. The unit of latency is ms, and the unit of service rate is classified flows per second.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
function GetPerClassSlope( D,U )
Initialize Results R
for each Class Label L in D do
Partition D into Correct C and Incorrect I for L▹ Classify predictions
Get Thresholds T from U quantiles for L
Update R with assigned portions for C and I▹ Track portions
Compute Uncertainty u for DataPoint▹D is validation data
Append u to U
end for
Sort U in ascending order
Appendix
Algorithm 2 Universal Uncertainty Thresholds
#Cons.
All CPU
Half CPU+Half GPU
1
4.319k
-
2
8.805k
8.620k
4
16.692k
16.484k
8
31.114k
30.651k
12
40.238k
41.009k
16
48.505k
47.040k
Appendix
Table 7. Maximum service rate (new classified flows per second) when we increase the number of consumers.
Service
Application
# of Flows
Video Conferencing ( MacMillan et al., 2021 )
Zoom
1312
Google Meet
1313
MS Teams
3886
Video Streaming ( Bronzino et al., 2019 )
Twitch
1150
Amazon
1509
YouTube
2702
Appendix
Table 8. Service recognition dataset.
Device
# of Flows
Samsung Fridge
3770
Nvidia Jetson Nano
3770
Amazon Echo 3rd Gen
3770
Google Home
3770
Raspberry Pi 3
3770
SmartThings Dishwasher
3770
Appendix
Table 9. Device identification dataset.
VCA
Total # of Seconds
Meet
10189
Webex
12331
Teams
14408
Total
36928
Appendix
Table 10. Video conferencing application (VCA) QoE measurement dataset.
Figure 12. Performance of different approaches for the device identification task. We compare FlowWise against four baselines, all considering three-packet inputs. The F1 score is computed on the flows that are not missed.
Figure 13. Assigned portion of requests vs. F1 score at each stage for device identification task.
Figure 14. Assigned portion of requests vs. F1 score at each stage for QoE measurement task.
Model optimizations help improve inference performance and accuracy of ML workflows. However, relying on a single model to perform inference across all data batches often fails to maximize accuracy and thus overall performance. In many cases, alternate models could perform better on specific subsets of data where a primary model underperforms. Our experiments with real ML workflows indeed show that switching models improves workflow accuracy by up to 23%. Yet, current systems lack the ability to adaptively switch between models based on performance, forcing users to manually test models in sequence. We present FlexiFlow, a dataflow system that dynamically switches between alternate models when the current model exhibits low accuracy. FlexiFlow learns to rank models using a novel multi-armed bandit approach that accounts for model runtimes, probability of passing user-defined assertions, and the computational structure of the ML workflow. We show that the standard Thompson sampling approach is insufficient for switching models in ML workflows. In contrast, our proposed approaches are effective and scales to complex real-world ML workflows. Experiments show that switching models at runtime while reusing intermediate results provides higher accuracy, but also 48% efficiency gain compared to sequential workflow runs.
We evaluate Jev on ten dataset-defined application labels in CESNET-QUICEXT-25 using only the first ten packets' sizes, directions, and inter-packet times. To the best of our knowledge, this is the first empirical study of general-purpose decision models, represented here by Jev, for application classification of network flows. Across 52,000 records from 26 collection weeks following the training period, 40 fixed labeled examples raise Jev's accuracy from 9.80% to 28.42%. Random Forest and Extra Trees trained on 8,000 records achieve 69.95% and 66.80% and outperform Jev in every week. Increasing Jev's context to 150 examples yields 34.50% on the first test week. On a paired 100-record subset, Jev with 40 examples achieves 29% accuracy at a median request time of 0.750 s, versus 37% and 6.036 s for the generative language model OpenAI GPT-5.6 Sol with high reasoning effort through Azure; Jev also incurs lower API charges. The paired subset does not establish an accuracy advantage for either service, and the timing reflects different service configurations. Thus, labeled examples substantially improve Jev, but the tested Jev configurations remain less accurate than trained tree ensembles; unequal supervision budgets and fixed configurations prevent attributing the gap to a single cause.
Shenghe Xu, Lifan Mei
Amazon.com, Inc. · Xi’an Jiaotong-Liverpool University
Enterprise wireless access points (APs) are promising platforms for predictive machine learning (ML), but their primary responsibility remains providing wireless connectivity and network services. Predictive inference must therefore share an AP's CPU and memory with packet processing, Wi-Fi and IoT radio operations, and client management. This resource contention creates two risks: a model that performs well on proxy hardware may be too slow on the target AP, while a model that fits in isolation may still degrade network services under load. We define network-aware deployability using two gates: qualification of the model and its execution path on the target AP, followed by validation of its execution profile under packet-service and forecasting constraints. Our benchmarks show that edge testbeds do not reliably capture target behavior. Across matched artifacts and serving settings, five model implementations run 6.1--19.1× slower on an AP than on a Raspberry Pi5, while peak memory usage differs by up to 22%. Moreover, two forecasting foundation models of similar size differ in AP latency by 19×. When serving a smaller model across 13 parallel streams at a 30s cadence under network saturation, default execution increases p99 round-trip time (RTT) by 76% and reduces throughput by 7.06%. Understanding these trade-offs is essential for live deployment if we aim to use APs for both networking and ML workloads.