Data Quality Audit and Bivariate Temperature-Humidity Analysis of a Long-Term LoRaWAN Network: Anomaly Detection, PCA, Clustering and Machine Learning

Mamadou Sow

cs.LGOct 23, 2026

Data Quality Audit and Bivariate Temperature-Humidity Analysis of a Long-Term LoRaWAN Network: Anomaly Detection, PCA, Clustering and Machine Learning

Authors: Mamadou Sow

Organizations: LIPN-CNRS7030, Université Sorbonne Paris Nord, 99 avenue JB Clément, F-93430 Villetaneuse, France

Abstract

Long-term environmental IoT/LPWAN deployments often aggregate heterogeneous data streams from diverse devices without systematic auditing of each variable's actual coverage prior to analysis-an increased risk when multiple sensor types coexist within the same network. This work proposes a C++17 processing pipeline applied to the Tour Perret LoRaWAN dataset (421 937 frames, June 2021--June 2023, 5 devices). A systematic coverage audit integrated into the extraction phase reveals that CO2 is completely missing (0% coverage) and pressure is reported only by an isolated base module (Wyres, 4.24% of the corpus), invalidating the initial hypothesis of a joint four-variable analysis. The analysis is refocused on the Temperature--Humidity pair, jointly available across 205{,}005 frames (95.76%) from 4 Elsys EMS sensors. The pipeline sequentially performs anomaly detection (modified Z-score, Isolation Forest), bivariate Principal Component Analysis, clustering (K-means, DBSCAN), and a multilayer perceptron predicting humidity from temperature and time, whose residuals serve as a detector for slow drift. On real-world data: Temperature--Humidity correlation r=-0.662, two-component PCA (PC1 =83.1% of variance), Isolation Forest detects 2.12% multivariate anomalies while univariate Z-score detects none, calibrated DBSCAN isolates 0.17% noise across 13 clusters, and the MLP achieves $R^2=0.672$ (RMSE =10.54%) in cross-validation. This work demonstrates that per-variable coverage auditing prior to modeling is a necessary methodological step for heterogeneous multi-sensor IoT datasets.

Figures & tables

Panel of Figure 1 — Figure 1: LoRaWAN telemetry ingestion and cleaning pipeline: the coverage audit (Step 1) is executed before any downstream analysis, separating structurally unusable variables (CO 2 , pressure) from the retained bivariate (T,H) core dataset.

Panel of Figure 2 — Figure 2: Four-tier complementary detection framework: each tier targets a distinct, orthogonal anomaly signature within the clean bivariate (T,H) dataset.

Detection Tier	Underlying Principle	Primary Target	Insensitive To
Modified Z-Score	Marginal distance from median, in units of MAD	Extreme single-variable spikes ( $T$ or $H$ )	Multivariate correlation violations within normal marginal bounds
Isolation Forest	Random axis-aligned partitioning path length	Unusual joint combinations of $(T,H)$	Slow, continuous calibration drifts over long horizons
PCA Reconstruction	Distance to the primary variance subspace	Departure from T–H anti-correlation ( $e_{i}=\lvert y_{\mathrm{PC2}}\rvert$ )	Anomalies aligned along the primary variance axis (PC1)
DBSCAN Noise Label	Local density threshold ( $N_{\varepsilon}(p)<\mathrm{minPts}$ )	Sparse spatial outliers, isolated noise points	Uniform dense clusters from sensor quantization
MLP Cross-Validation	Deviation from learned physical/temporal dependency	Slow sensor drift, temporal micro-climate anomalies	Sudden point anomalies uncoupled from diurnal/seasonal trends

Table I : Methodological orthogonality of the four detection tiers.

Panel of Figure 3 — Figure 3: Data processing and analytical pipeline: from raw LoRaWAN logs to the four output reports (anomaly, MLP, PCA/cluster) that feed the Results section.

Variable	Raw	Retained	Cov.	Cov.	Scope	Nodes
	Frames		(%Tot.)	(%Ret.)
Temperature ( $T$ )	393,685	214,076	93.30	100.00	Elsys + Wyres	5
Humidity ( $H$ )	384,614	205,005	91.15	95.76	Elsys only	4
Pressure ( $P$ )	9,071	0 (isolated)	2.15	0.00	Wyres only	1
Carbon Dioxide (CO 2 )	0	0 (absent)	0.00	0.00	—	0

Table II : Coverage per variable and per device (audit on the 421,937 raw frames).

Figure 4: Uniform grid spatial index (2D), cell size $=\varepsilon$ : DBSCAN’s $\varepsilon$ -neighborhood query is restricted to a $3\times 3$ neighborhood of grid cells around the query point’s cell, avoiding an $O(n^{2})$ full scan.

Figure 5: $k$ -distance plot ( $k=\mathrm{minPts}=10$ ) used to objectively select the DBSCAN neighborhood radius $\varepsilon$ on real-world data. Candidate thresholds are overlaid; the knee of the curve, identified via maximum perpendicular distance to the chord, is retained at $\varepsilon\approx 0.071$ .

Variable	Mean	Std. Dev.	Min	Q1 (25%)	Median	Q3 (75%)	Max
Temperature $T$ (°C)	14.36	8.72	$-4.30$	7.00	14.80	20.70	46.10
Humidity $H$ (%)	72.14	18.41	19.00	58.00	74.00	88.00	100.00

Table III : Statistical summary of the

n=205{,}005

synchronized Elsys EMS bivariate telemetry corpus.

Figure 6: Bivariate PCA eigenvector axes in standardized space. Variance PC1 $=83.10\%$ , PC2 $=16.90\%$ . Points are a schematic reconstruction of the standardized $(T^{\prime},H^{\prime})$ cloud preserving the reported correlation $r=-0.662$ ; the plotted axes are exact ( $\pm 1/\sqrt{2}$ loadings, Eq. ( 10 )).

Component	Eigenvalue $\lambda_{j}$	Explained Var. (%)	Cumulative Var. (%)	$T$ Loading	$H$ Loading
PC1	1.6620	83.10%	83.10%	$-0.7071$	$+0.7071$
PC2	0.3380	16.90%	100.00%	$-0.7071$	$-0.7071$

Table IV : Eigenvalue decomposition of the

2\times 2

correlation matrix (cf. Figure 8 ).

Figure 7: Actual Pearson correlation matrix, Temperature–Humidity ( $n=205{,}005$ ).

Figure 8: Scree plot of eigenvalues (bivariate PCA: T, H, real data). PC1 alone captures $83.10\%$ of total variance.

Anomaly Detector	Primary Target Anomaly Type	Flagged ( $n$ )	Rate (%)
Modified Z-Score ( $\lvert M_{i}\rvert>3.5$ )	Extreme single-variable spikes	0	0.00
Isolation Forest ( $s>0.62$ )	Atypical multivariate combinations	4,346	2.12
Joint Consensus (both methods)	Simultaneous univariate & multivariate	0	0.00

Table V : Comparative performance across the primary point-based anomaly detectors (

n=205{,}005

Figure 9: Schematic reconstruction of the temperature/humidity time series for the Perret Tower (2021–2023) with Isolation Forest-flagged anomalies overlaid (Modified Z-score set to zero anomalies on this dataset). Curve shape preserves the reported seasonal range and anomaly rate (2.12%); exact daily values are not reproduced from the original data.

Cluster Count $K$	Inertia $J$	Silhouette $s$	Structural Interpretation
$K=2$	165,530	0.4910	Optimal partition (Warm/Dry vs. Cool/Humid)
$K=3$	114,930	0.4040	Fragmented intermediate transition zone
$K=4$	88,253	0.3760	Arbitrary seasonal sub-division
$K=5$	68,978	0.3910	Sub-cluster fragmentation
$K=6$	58,310	0.3740	Over-segmented clusters
$K=7$	49,814	0.3780	Over-segmented clusters

Table VI : Evaluating K-means++ clustering across

K\in\{2,\dots,8\}

using the silhouette index.

Figure 10: DBSCAN density clustering in standardized PCA space ( $\varepsilon=0.071$ , minPts $=10$ ), accelerated by the uniform grid spatial index of Figure 4 . Point cloud is a schematic reconstruction preserving the reported cluster/noise proportions.

Figure 11: Actual PCA projection (PC1–PC2, $n=205{,}005$ ) and partitioning: (a) K-means bipartition along PC1, colored by cluster label; (b) DBSCAN partition, dominated by a single continuous cluster (label $-1$ = noise), with the 12 quantization micro-clusters and noise points overlaid. Schematic reconstruction preserving reported silhouette and cluster statistics.

Figure 12: MLP regression residual distribution ( $|y_{i}-\hat{y}_{i}|$ ), predictive cross-validation (humidity estimation). A subset of $n=1{,}271$ frames ( $0.62\%$ ) exceeds the $3\times$ RMSE drift threshold. Density curve is a schematic reconstruction matching the reported median ( $-1.68\%$ ) and standard deviation ( $10.48\%$ ).

Figure 13: Actual performance of the cross-validated MLP (humidity prediction). Panel (a): observed vs. predicted humidity. Panel (b): residual histogram. Schematic reconstruction preserving reported RMSE and $R^{2}$ .

Panel of Figure 14 — Figure 14: Sequential execution flow of the five C++ modules and their intermediate CSV reports.

Time-Series Anomaly Detection

Time-Series Anomaly Detection identifies unusual patterns or events in sequential data, from sensor readings and network traffic to health and industrial telemetry.

See all 208 papers

Real-World Deployment and Performance Characterisation of Fog-Based Deep Learning for Cold-Chain Temperature Prediction over LoRaWAN
Sep 12, 2026Jeremiah Taguta, Jean Frederic Isingizwe Nturambirwe, Clement Nthambazale NyirendaTime Series Forecasting Edge Computing
Department of Computer Science University of the Western Cape Cape Town, South Africa · eResearch Office University of the Western Cape Cape Town, South Africa
FLoRa: Flight-Assisted Data Collection from Duty Cycling LoRa Nodes under Energy Constraints
Oct 6, 2026Naresh Babu Kakarla, V. MahendranRobotic Data Collection Internet of Things
Department of Computer Science and Engineering, Indian Institute of Technology Tirupati, Andhra Pradesh, 517619, India
Thermal Anomaly Detection using Physics Aware Neuromorphic Networks: Comparison between Raw and L1C Sentinel-2 Data
Apr 10, 2026Stephen Smith, Cormac Purcell, Gabriele Meoni +2Anomaly Detection Physics-Informed ML
School of Physics, University of Sydney, Sydney, Australia · BolgianTen Ltd., Liverpool, United Kingdom · School of Computer Science, University of New South Wales, Sydney, Australia +3

Sep 12, 2026J/K move · Enter open · S save

Real-World Deployment and Performance Characterisation of Fog-Based Deep Learning for Cold-Chain Temperature Prediction over LoRaWAN

Jeremiah Taguta, Jean Frederic Isingizwe Nturambirwe, Clement Nthambazale Nyirenda

Department of Computer Science University of the Western Cape Cape Town, South Africa · eResearch Office University of the Western Cape Cape Town, South Africa

Time Series Forecasting Edge Computing Internet of Things

Open paper AskPDF arXiv

Fresh fruits and vegetables (FFVs) are highly perishable, and cold-chain breaks contribute significantly to global food waste. While Machine Learning (ML) can enable proactive intervention, cloud-based inference faces challenges such as latency and data loss. Fog computing addresses these issues but has been tested only in simulation for FFV cold-chain temperature prediction. To the best of the authors' knowledge, this paper presents its first real-world deployment. A fog-deployed LSTM-GRU model predicted cold-room temperature using LoRaWAN sensor data collected from a South African apple cold-storage facility with induced cold-chain breaks. Running entirely on a Raspberry Pi 4 with no cloud dependency, the system generated conditional SHAP explanations only when a break is predicted. The deployed system predicts cold-room temperature with an MAE of 0.2°C at roughly 0.2 kWh per day (

\approx 0.7

Wh per prediction). Predictions were delivered in under one second (555 ms), dominated by network and messaging rather than computation, with conditional explanations adding modest cost. SHAP consumes 28% more CPU but is well within the hardware's capacity. The model attributes its predictions primarily to temperature, humidity, and their interaction. Critically, the deployment surfaced what simulation cannot: a sensor-triggered single point of failure, alongside genuine resilience, autonomous recovery from infrastructure faults and continued operation through internet loss. These are the first published deployment benchmarks for fog-based temperature prediction in FFV cold chains, establishing that explainable temperature forecasting is feasible on resource-constrained edge hardware. Future work includes asynchronous sensor fusion, commercial cold chain deployment, alternative model architectures, and causal analysis.

Abstract

Time-Series Anomaly Detection

Explore similar work

Real-World Deployment and Performance Characterisation of Fog-Based Deep Learning for Cold-Chain Temperature Prediction over LoRaWAN

FLoRa: Flight-Assisted Data Collection from Duty Cycling LoRa Nodes under Energy Constraints

Thermal Anomaly Detection using Physics Aware Neuromorphic Networks: Comparison between Raw and L1C Sentinel-2 Data