Evaluating Near-Perfect Accuracy Claims in Machine-Learning-Based Leak Detection for Two-Phase Pipelines: A Condition-Blocked Audit
Authors: Issah N. Suleiman, Sormeh Serpoosh, Nadine Elkholy, Hicham Ferroudji, Mohammad Azizur Rahman, Matthew Hamilton
Organizations: Memorial University of Newfoundland, St. John’s, Newfoundland and Labrador, Canada · College of Science and Engineering, Hamad Bin Khalifa University, Education City, Doha, Qatar · Laboratory of Petroleum Equipment’s Reliability and Materials, Faculty of Hydrocarbons and Chemistry, University M’Hamed Bougara of Boumerdes, Boumerdes, Algeria
Reliable leak detection in gas-liquid two-phase pipelines is important for process safety, yet machine-learning studies frequently report near-perfect performance using evaluation protocols that may not represent deployment to unseen operating conditions. This study audits a pressure-based leak-detection model using condition-blocked leave-one-condition-out validation, in which the classifier is tested on a flow condition excluded from training. Accuracy decreased from 93.3% under within-condition evaluation to 59.1% across fifteen unseen-condition tests. Ablation analysis showed that most of this difference arose from the data-splitting protocol, with a smaller contribution from encoding operating conditions as model inputs. Under condition-blocked evaluation, the false-alarm and missed-detection rates increased to 34.1% and 10.2%, respectively, demonstrating that near-perfect reported accuracy did not translate into reliable operation at unseen flow conditions. The reduction persisted across all evaluated classifiers, indicating that it was caused by the evaluation design rather than a particular model architecture. Grouped evaluation of an independent acoustic pipeline dataset produced a comparable reduction toward chance-level performance. These findings show that validation units must represent the conditions encountered during deployment. Condition-blocked evaluation and explicit reporting of operational error rates are therefore necessary before machine-learning leak detectors can support process-safety decisions.
Figures & tables
Figure 1: Schematic of the experimental two-phase flow loop (50.8 mm ID, 6 m): pressure sensors S1 (2.4 m from inlet) and S2 (5.9 m from inlet), and the two-orifice leak zone (1.8 mm and 3 mm) at 3.85 m. Dynamic pressure sampled at 2 Hz. Only S1 and S2 pressure signals were used in this study.
Figure 2: Corrected ML audit pipeline for multiphase leak classification (C0: no leak; C1: single-point; C2: two-point/multiple leakage). Two protocol changes are enforced relative to [ 1 ] : windows are extracted within individual 30-second recordings only, and LOCO withholds each (VSG,VSL) condition from training in turn. StandardScaler is fitted on the training split of each fold only.
Figure 3: Two-stage cascaded classifier architecture (MC4). Stage 1 performs binary detection (C0 versus C1 or C2); windows classified as leaking are forwarded to Stage 2, which discriminates single-point (C1) from two-point/multiple leakage (C2). FAR is measured at Stage 1; MDR is measured across Stage 2. The LOCO loop repeats both stages for each of the fifteen held-out conditions, aggregating results across ten independently seeded trials.
Protocol
Wk =6
Wk =8
Wk =10
A : Time-based + VSL / VSG
89.3±1.0 %
91.8±0.9 %
93.3±0.7 %
B : LOCO + VSL / VSG
62.1±0.7 %
61.7±0.7 %
62.6±0.5 %
C ⋆ : LOCO, no VSL / VSG (15 cond.)
55.4±0.7 %
56.0±0.8 %
59.1±0.6 %
D : Time-based, no VSL / VSG
84.8±1.2 %
88.8±0.6 %
88.4±0.9 %
C-9 sensitivity (9 conditions)
49.9±0.3 %
49.8±0.8 %
50.9±0.5 %
Δsplit (A → B)
−27.2 pp
−30.1 pp
−30.7 pp
Table 1: Complete 2×2 ablation: accuracy (mean ± std across 10 seeds). The interaction contrast I=A−B−D+C quantifies approximate additivity; values ∣I∣≤3 pp at all window sizes indicate the two components combine with negligible interaction.
Figure 4: Accuracy across evaluation protocols at three window sizes ( Wk=6 , 8, 10). Error bars: ±1 standard deviation across ten seeds, reflecting model-training instability. The dashed line marks the accuracy reported in [ 1 ] . Condition A uses a time-based 80/20 split retaining all conditions in training and test.
Class
F1 (%)
Function
C0: No leak
70.8±0.8
Binary alarm
C1: Single-point
56.6±0.8
Count discrimination
C2: Two-point/mult.
50.3±0.7
Count discrimination
Macro F1
59.2±0.6
Accuracy
59.1±0.6
FAR (laboratory)
34.1±1.0
C0 classified as leak
Table 2: Per-class metrics: Wk=10 , LOCO, no VSL / VSG , 15 conditions, 10 seeds. FAR and MDR are laboratory estimates from a single rig; they should not be used as deployment readiness criteria without a prospectively designed field study at the intended operating conditions.
Figure 5: Normalised confusion matrix at Wk=10 , LOCO, no VSL / VSG , 15 conditions, averaged over ten seeds. Class C2 is two-point or multiple leakage, consistent with [ 1 ] . Overall accuracy: 59.1%; FAR: 34.1%; MDR: 10.2%.
Model
Accuracy (%)
Macro F1
FAR (%)
MDR (%)
MC4
Cascade CatBoost (primary) ⋆
59.1±0.6
0.592±0.006
34.1±1.0
10.2±0.4
MS5
Single-stage CatBoost
61.6±0.7
0.617±0.008
27.9±0.6
10.3±0.9
MS2
SVM
57.9±0.0
0.583±0.000
29.4±0.0
10.3±0.0
MS6
Max-vote ensemble
57.7±1.0
0.581±0.009
29.0±0.8
9.8±0.5
MS7
Stacking ensemble
56.4±1.2
0.564±0.013
35.1±1.2
15.8±1.9
MS4
Random forest
55.8±0.6
0.558±0.006
33.0±0.8
14.4±0.4
Table 3: All classifiers at Wk=10 , LOCO, no VSL / VSG , POOL 15 , 10 seeds. No model exceeds 61.6% accuracy; the finding is not specific to any architecture. MS1 (k-NN) and MS2 (SVM) are deterministic, hence ±0.0 % across seeds.
Figure 6: Accuracy of all nine classifiers at Wk=10 , LOCO, no VSL / VSG , POOL 15 , 10 seeds. The range 39.4–61.6% confirms the within-condition versus unseen-condition accuracy gap is not specific to the MC4 cascade architecture.
Wk
Duration
Accuracy
Macro F1
FAR
MDR
6
3.0 s
55.4±0.7 %
55.3±0.7 %
37.7±0.5 %
10.1±0.2 %
8
4.0 s
56.0±0.8 %
56.1±0.7 %
35.3±1.1 %
9.5±0.1 %
10
5.0 s
59.1±0.6 %
59.2±0.6 %
34.1±1.0 %
10.2±0.4 %
Table 4: Window-size ablation: Condition C, 15 conditions, 10 seeds. Accuracy does not increase monotonically with Wk , in contrast to the within-condition results in [ 1 ] .
Noise (%FSO)
Paper (asymm.)
Accuracy (%)
FAR (%)
MDR (%)
0.0
100%
59.1±0.6
34.1±1.0
10.2±0.4
0.1
100%
58.5±0.8
31.9±1.0
10.6±0.5
0.2
99%
58.9±1.1
31.1±1.7
10.5±0.4
0.5
N/A
59.0±0.7
30.4±0.9
10.5±0.3
1.0
N/A
59.0±0.4
29.8±0.8
10.1±0.4
Table 5: Noise robustness under symmetric contamination: Wk=10 , LOCO, no VSL / VSG , 10 seeds. Accuracy is essentially invariant across the tested range (58.5–59.1%); the performance floor is set by the evaluation protocol, not sensor noise at these FSO levels. The 0% FSO entry matches Condition C in Table 1 .
Figure 7: Noise robustness under the corrected symmetric protocol: accuracy (left) and FAR/MDR (right) versus noise level. Accuracy remains within 0.6 pp of the noise-free baseline across all five FSO levels, confirming the performance floor is set by the condition-blocked evaluation, not by additive sensor noise. FAR decreases from 34.1% to 29.8% as noise increases; MDR is approximately constant at ∼ 10.3%.
Rank
Feature
Importance (%)
1
P2_max
41.19
2
P1_min
8.12
3
P2_range
7.81
4
P2_meandiff
6.38
5
P1_max
6.20
6
P2_std
4.91
Table 6: Top-10 CatBoost feature importances at Wk=10 , LOCO, no VSL / VSG . P2_max (downstream sensor maximum) dominates at 41.2%.
Figure 8: CatBoost feature importances at Wk=10 , LOCO, no VSL / VSG . Green: upstream sensor S1 features (P1); red: downstream sensor S2 features (P2). P2_max alone accounts for 41.2% of total importance.
Condition
Accuracy (%)
A: random split, with domain features
96.8±1.3
B: LOGO (4 groups), with features
35.4±2.5
C: LOGO (4 groups), no features
33.7±1.1
Gap (A − B)
−61.4 pp
Feature-encoding gap (B − C)
−1.7 pp
Table 7: External evaluation on GPLA-12 (ten seeds, four leave-one-group-out folds). Only four domain groups provide cross-domain evidence; group-level uncertainty is substantial.
Authors
Method / Parameters
Configuration
d / L
Leak
Experimental modelling
Siebenaler et al. (2017)
Gamma densitometer, hydrophones, high-speed camera
Horiz. slug, air–water
127 mm / 6 m
3.2–9.5 mm
Stajanca et al. (2018)
Fibre optic, acoustics, accelerometer
Horiz. air
114 mm / 38 m
1–8 mm
Ji et al. (2018)
Acoustics, dynamic pressure
Horiz. strat./wavy/slug
80 mm / 14.3 m
3–6 mm
Li et al. (2022)
Dynamic pressure, acoustics
Horiz. stratified
40 mm / 48 m
8 mm
Meng et al. (2024)
Ultrasonic, pressure, mass flow
Horiz. strat./slug/annular
32 mm / 4.8 m
3 mm
Table 8: Contributions to leak detection research in multiphase pipeline systems. † marks studies subject to the evaluation-protocol audit in this paper.
Center for Molecular Cardiology, University of Zurich, Zurich, Switzerland · Center for Complexity Sciences, National University of Mexico, Mexico City, Mexico
Medical Informatics Center, Institute of Advanced Clinical Medicine, Peking University, Beijing, China · Department of Ultrasound, Peking University Third Hospital, Beijing, China · Department of Hospital-Acquired Infection Control, Beijing Jishuitan Hospital, Beijing, China +4