Data Quality Audit and Bivariate Temperature-Humidity Analysis of a Long-Term LoRaWAN Network: Anomaly Detection, PCA, Clustering and Machine Learning

Mamadou Sow

cs.LGOct 23, 2026

Data Quality Audit and Bivariate Temperature-Humidity Analysis of a Long-Term LoRaWAN Network: Anomaly Detection, PCA, Clustering and Machine Learning

Authors: Mamadou Sow

Organizations: LIPN-CNRS7030, Université Sorbonne Paris Nord, 99 avenue JB Clément, F-93430 Villetaneuse, France

Abstract

Long-term environmental IoT/LPWAN deployments often aggregate heterogeneous data streams from diverse devices without systematic auditing of each variable's actual coverage prior to analysis-an increased risk when multiple sensor types coexist within the same network. This work proposes a C++17 processing pipeline applied to the Tour Perret LoRaWAN dataset (421 937 frames, June 2021--June 2023, 5 devices). A systematic coverage audit integrated into the extraction phase reveals that CO2 is completely missing (0% coverage) and pressure is reported only by an isolated base module (Wyres, 4.24% of the corpus), invalidating the initial hypothesis of a joint four-variable analysis. The analysis is refocused on the Temperature--Humidity pair, jointly available across 205{,}005 frames (95.76%) from 4 Elsys EMS sensors. The pipeline sequentially performs anomaly detection (modified Z-score, Isolation Forest), bivariate Principal Component Analysis, clustering (K-means, DBSCAN), and a multilayer perceptron predicting humidity from temperature and time, whose residuals serve as a detector for slow drift. On real-world data: Temperature--Humidity correlation r=-0.662, two-component PCA (PC1 =83.1% of variance), Isolation Forest detects 2.12% multivariate anomalies while univariate Z-score detects none, calibrated DBSCAN isolates 0.17% noise across 13 clusters, and the MLP achieves $R^2=0.672$ (RMSE =10.54%) in cross-validation. This work demonstrates that per-variable coverage auditing prior to modeling is a necessary methodological step for heterogeneous multi-sensor IoT datasets.

Figures & tables

Panel of Figure 1 — Figure 1: LoRaWAN telemetry ingestion and cleaning pipeline: the coverage audit (Step 1) is executed before any downstream analysis, separating structurally unusable variables (CO 2 , pressure) from the retained bivariate (T,H) core dataset.

Panel of Figure 2 — Figure 2: Four-tier complementary detection framework: each tier targets a distinct, orthogonal anomaly signature within the clean bivariate (T,H) dataset.

Detection Tier	Underlying Principle	Primary Target	Insensitive To
Modified Z-Score	Marginal distance from median, in units of MAD	Extreme single-variable spikes ( $T$ or $H$ )	Multivariate correlation violations within normal marginal bounds
Isolation Forest	Random axis-aligned partitioning path length	Unusual joint combinations of $(T,H)$	Slow, continuous calibration drifts over long horizons
PCA Reconstruction	Distance to the primary variance subspace	Departure from T–H anti-correlation ( $e_{i}=\lvert y_{\mathrm{PC2}}\rvert$ )	Anomalies aligned along the primary variance axis (PC1)
DBSCAN Noise Label	Local density threshold ( $N_{\varepsilon}(p)<\mathrm{minPts}$ )	Sparse spatial outliers, isolated noise points	Uniform dense clusters from sensor quantization
MLP Cross-Validation	Deviation from learned physical/temporal dependency	Slow sensor drift, temporal micro-climate anomalies	Sudden point anomalies uncoupled from diurnal/seasonal trends

Table I : Methodological orthogonality of the four detection tiers.

Panel of Figure 3 — Figure 3: Data processing and analytical pipeline: from raw LoRaWAN logs to the four output reports (anomaly, MLP, PCA/cluster) that feed the Results section.

Variable	Raw	Retained	Cov.	Cov.	Scope	Nodes
	Frames		(%Tot.)	(%Ret.)
Temperature ( $T$ )	393,685	214,076	93.30	100.00	Elsys + Wyres	5
Humidity ( $H$ )	384,614	205,005	91.15	95.76	Elsys only	4
Pressure ( $P$ )	9,071	0 (isolated)	2.15	0.00	Wyres only	1
Carbon Dioxide (CO 2 )	0	0 (absent)	0.00	0.00	—	0

Table II : Coverage per variable and per device (audit on the 421,937 raw frames).

Figure 4: Uniform grid spatial index (2D), cell size $=\varepsilon$ : DBSCAN’s $\varepsilon$ -neighborhood query is restricted to a $3\times 3$ neighborhood of grid cells around the query point’s cell, avoiding an $O(n^{2})$ full scan.

Figure 5: $k$ -distance plot ( $k=\mathrm{minPts}=10$ ) used to objectively select the DBSCAN neighborhood radius $\varepsilon$ on real-world data. Candidate thresholds are overlaid; the knee of the curve, identified via maximum perpendicular distance to the chord, is retained at $\varepsilon\approx 0.071$ .

Variable	Mean	Std. Dev.	Min	Q1 (25%)	Median	Q3 (75%)	Max
Temperature $T$ (°C)	14.36	8.72	$-4.30$	7.00	14.80	20.70	46.10
Humidity $H$ (%)	72.14	18.41	19.00	58.00	74.00	88.00	100.00

Table III : Statistical summary of the

n=205{,}005

synchronized Elsys EMS bivariate telemetry corpus.

Figure 6: Bivariate PCA eigenvector axes in standardized space. Variance PC1 $=83.10\%$ , PC2 $=16.90\%$ . Points are a schematic reconstruction of the standardized $(T^{\prime},H^{\prime})$ cloud preserving the reported correlation $r=-0.662$ ; the plotted axes are exact ( $\pm 1/\sqrt{2}$ loadings, Eq. ( 10 )).

Component	Eigenvalue $\lambda_{j}$	Explained Var. (%)	Cumulative Var. (%)	$T$ Loading	$H$ Loading
PC1	1.6620	83.10%	83.10%	$-0.7071$	$+0.7071$
PC2	0.3380	16.90%	100.00%	$-0.7071$	$-0.7071$

Table IV : Eigenvalue decomposition of the

2\times 2

correlation matrix (cf. Figure 8 ).

Figure 7: Actual Pearson correlation matrix, Temperature–Humidity ( $n=205{,}005$ ).

Figure 8: Scree plot of eigenvalues (bivariate PCA: T, H, real data). PC1 alone captures $83.10\%$ of total variance.

Anomaly Detector	Primary Target Anomaly Type	Flagged ( $n$ )	Rate (%)
Modified Z-Score ( $\lvert M_{i}\rvert>3.5$ )	Extreme single-variable spikes	0	0.00
Isolation Forest ( $s>0.62$ )	Atypical multivariate combinations	4,346	2.12
Joint Consensus (both methods)	Simultaneous univariate & multivariate	0	0.00

Table V : Comparative performance across the primary point-based anomaly detectors (

n=205{,}005

Figure 9: Schematic reconstruction of the temperature/humidity time series for the Perret Tower (2021–2023) with Isolation Forest-flagged anomalies overlaid (Modified Z-score set to zero anomalies on this dataset). Curve shape preserves the reported seasonal range and anomaly rate (2.12%); exact daily values are not reproduced from the original data.

Cluster Count $K$	Inertia $J$	Silhouette $s$	Structural Interpretation
$K=2$	165,530	0.4910	Optimal partition (Warm/Dry vs. Cool/Humid)
$K=3$	114,930	0.4040	Fragmented intermediate transition zone
$K=4$	88,253	0.3760	Arbitrary seasonal sub-division
$K=5$	68,978	0.3910	Sub-cluster fragmentation
$K=6$	58,310	0.3740	Over-segmented clusters
$K=7$	49,814	0.3780	Over-segmented clusters

Table VI : Evaluating K-means++ clustering across

K\in\{2,\dots,8\}

using the silhouette index.

Figure 10: DBSCAN density clustering in standardized PCA space ( $\varepsilon=0.071$ , minPts $=10$ ), accelerated by the uniform grid spatial index of Figure 4 . Point cloud is a schematic reconstruction preserving the reported cluster/noise proportions.

Figure 11: Actual PCA projection (PC1–PC2, $n=205{,}005$ ) and partitioning: (a) K-means bipartition along PC1, colored by cluster label; (b) DBSCAN partition, dominated by a single continuous cluster (label $-1$ = noise), with the 12 quantization micro-clusters and noise points overlaid. Schematic reconstruction preserving reported silhouette and cluster statistics.

Figure 12: MLP regression residual distribution ( $|y_{i}-\hat{y}_{i}|$ ), predictive cross-validation (humidity estimation). A subset of $n=1{,}271$ frames ( $0.62\%$ ) exceeds the $3\times$ RMSE drift threshold. Density curve is a schematic reconstruction matching the reported median ( $-1.68\%$ ) and standard deviation ( $10.48\%$ ).

Figure 13: Actual performance of the cross-validated MLP (humidity prediction). Panel (a): observed vs. predicted humidity. Panel (b): residual histogram. Schematic reconstruction preserving reported RMSE and $R^{2}$ .

Panel of Figure 14 — Figure 14: Sequential execution flow of the five C++ modules and their intermediate CSV reports.

Time-Series Anomaly Detection

Time-Series Anomaly Detection identifies unusual patterns or events in sequential data, from sensor readings and network traffic to health and industrial telemetry.

See all 208 papers

Sep 12, 2026

Real-World Deployment and Performance Characterisation of Fog-Based Deep Learning for Cold-Chain Temperature Prediction over LoRaWAN

Fresh fruits and vegetables (FFVs) are highly perishable, and cold-chain breaks contribute significantly to global food waste. While Machine Learning (ML) can enable proactive intervention, cloud-based inference faces challenges such as latency and data loss. Fog computing addresses these issues but has been tested only in simulation for FFV cold-chain temperature prediction. To the best of the authors' knowledge, this paper presents its first real-world deployment. A fog-deployed LSTM-GRU model predicted cold-room temperature using LoRaWAN sensor data collected from a South African apple cold-storage facility with induced cold-chain breaks. Running entirely on a Raspberry Pi 4 with no cloud dependency, the system generated conditional SHAP explanations only when a break is predicted. The deployed system predicts cold-room temperature with an MAE of 0.2°C at roughly 0.2 kWh per day (

\approx 0.7

Wh per prediction). Predictions were delivered in under one second (555 ms), dominated by network and messaging rather than computation, with conditional explanations adding modest cost. SHAP consumes 28% more CPU but is well within the hardware's capacity. The model attributes its predictions primarily to temperature, humidity, and their interaction. Critically, the deployment surfaced what simulation cannot: a sensor-triggered single point of failure, alongside genuine resilience, autonomous recovery from infrastructure faults and continued operation through internet loss. These are the first published deployment benchmarks for fog-based temperature prediction in FFV cold chains, establishing that explainable temperature forecasting is feasible on resource-constrained edge hardware. Future work includes asynchronous sensor fusion, commercial cold chain deployment, alternative model architectures, and causal analysis.

Jeremiah Taguta, Jean Frederic Isingizwe Nturambirwe, Clement Nthambazale Nyirenda

Department of Computer Science University of the Western Cape Cape Town, South Africa · eResearch Office University of the Western Cape Cape Town, South Africa

Time Series Forecasting Edge Computing Internet of Things

Ask PDF

Oct 6, 2026

FLoRa: Flight-Assisted Data Collection from Duty Cycling LoRa Nodes under Energy Constraints

Data collection using Unmanned Aerial Vehicles (UAVs) is challenging when LoRa IoT Devices (IoTDs) duty-cycle to conserve battery. Under energy constraints, a UAV must decide which IoTDs to visit, in what order, where to hover, and how many times to probe each node, while time-based data freshness decays. Tractably solving this problem requires a multi-level optimization architecture: discrete combinatorial optimization for routing, continuous global optimization for spatial positioning, and sequential decision-making under uncertainty. We propose FLoRa, a Flight-assisted LoRa data collection architecture using Simulated Annealing (SA) for path planning, Covariance Matrix Adaptation Evolution Strategy (CMA-ES) for hover positioning, and Partially Observable Markov Decision Processes (POMDPs) for probing IoTDs. To quantify collection utility from duty-cycling nodes, we introduce the Value of Information for Pull-based systems (VIP), a metric that rewards fresh data and penalizes failed probes, imposing well-posedness and preventing indefinite probing when an IoTD is off. Tracking hard battery constraints on every POMDP sample path requires state augmentation, worsening the curse of dimensionality. For tractability, SA and CMA-ES work on the hard battery constraints, while at the POMDP layer we relax them into soft average constraints via Lagrangian relaxation. Since solving the network-wide POMDP is computationally complex, we decompose it into node-level POMDPs by approximating inter-node time dependency using a forward-decomposition technique. Evaluation shows FLoRa outperforms metaheuristic, greedy, and deep reinforcement learning baselines by 30.6%, 27.8%, and 15.2% in total expected VIP, while increasing node coverage by 24.5%, 29.2%, and 8.8%, and successful collections by 24.3%, 25.6%, and 15.2%, respectively.

Naresh Babu Kakarla, V. Mahendran

Department of Computer Science and Engineering, Indian Institute of Technology Tirupati, Andhra Pradesh, 517619, India

Robotic Data Collection Internet of Things UAV Trajectory Optimization Wireless Communications

Ask PDF

Apr 10, 2026

Thermal Anomaly Detection using Physics Aware Neuromorphic Networks: Comparison between Raw and L1C Sentinel-2 Data

Damage caused by bushfires and volcanic eruptions escalates rapidly when detection is delayed, making fast and reliable early warning capabilities essential. Recent Earth Observation (EO) approaches have shown that thermal anomaly detection can be performed directly on decompressed Level-0 (L0) sensor data, avoiding computationally expensive preprocessing chains. However, direct exploitation of raw data remains challenging due to domain shift, sensor drift, radiometric inconsistencies, and the scarcity of labelled training samples. To address these challenges, this work proposes a Physics-Aware Neuromorphic Network (PANN) framework for onboard thermal anomaly detection. The proposed lightweight architecture, inspired by physical neural network principles and neuromorphic computing paradigms, is evaluated using two Sentinel-2 datasets: decompressed L0 with additional metadata (i.e. raw) and Level-1C (L1C). The PANN achieves a Matthews Correlation Coefficient (MCC) of

0.809

on raw measurements, compared to

0.875

when using ground-processed L1C products. The mean processing latency per L0 granule is

2.44 \pm 0.09~\mathrm{s}

, which is below the Sentinel-2 acquisition time of

3.6~\mathrm{s}

, demonstrating the feasibility of real-time, onboard processing. Furthermore, the projected execution time for the corresponding neuromorphic hardware instantiation is substantially lower at

0.1290 \pm 0.0002~\mathrm{s}

. Memory usage, including all necessary programs and packages, remains within realistic onboard constraints, with requirements of

0.673 \pm 0.007~\mathrm{Gb}

for the software PANN and

0.393 \pm 0.004~\mathrm{Gb}

for the estimated hardware realisation. Overall, these results indicate that PANN offers a promising pathway toward low-latency and resource-efficient onboard EO processing for thermal event detection.

Stephen Smith, Cormac Purcell, Gabriele Meoni +2

School of Physics, University of Sydney, Sydney, Australia · BolgianTen Ltd., Liverpool, United Kingdom · School of Computer Science, University of New South Wales, Sydney, Australia +3

Anomaly Detection Physics-Informed ML Neuromorphic Computing Wildfire Detection Efficient Neural Network Inference

Ask PDF

Abstract

Time-Series Anomaly Detection

Explore similar work

Real-World Deployment and Performance Characterisation of Fog-Based Deep Learning for Cold-Chain Temperature Prediction over LoRaWAN

FLoRa: Flight-Assisted Data Collection from Duty Cycling LoRa Nodes under Energy Constraints

Thermal Anomaly Detection using Physics Aware Neuromorphic Networks: Comparison between Raw and L1C Sentinel-2 Data