A reconfigurable smart camera implementation for jet flames characterization based on an optimized segmentation model
Authors: Gerardo Valente Vazquez-Garcia, Carmina Perez Guerrero, Eduardo Garduño, Miguel Gonzalez-Mendoza, Adriana Palacios, Gerardo Rodriguez-Hernandez, Vahid Foroughi, Alba Àgueda, Elsa Pastor, Gilberto Ochoa-Ruiz
Organizations: Universidad Autónoma de Guadalajara, Maestria en Ciencias Computacionales, Guadalajara, 45017, Jalisco, Mexico · Escuela de Ingeniería y Ciencias, Tecnológico de Monterrey, Monterrey, 64849, N.L., Mexico · Universidad de las Américas Puebla, Department of Chemical, Food and Environmental Engineering, Puebla, 72810, Mexico · Universitat Politècnica de Catalunya - BarcelonaTech. EEBE, Eduard Maristany 16, 08019 Barcelona. Catalonia, Spain
In this work we present a novel framework for fire safety management in industrial settings through the implementation of a smart camera platform for jet flames characterization. The approach seeks to alleviate the lack of real-time solutions for industrial early fire segmentation and characterization. As a case study, we demonstrate how a SoC FPGA, running optimized Artificial Intelligence (AI) models can be leveraged to implement a full edge processing pipeline for jet flames analysis. In this paper we extend previous work on computer-vision jet fire segmentation by creating a novel experimental set-up and system implementation for addressing this issue, which can be replicated to other fire safety applications. The proposed platform is designed to carry out image processing tasks in real-time and on device, reducing video processing overheads, and thus the overall latency. This is achieved by optimizing a UNet segmentation model to make it amenable for an SoC FPGAs implementation; the optimized model can then be efficiently mapped onto the SoC reconfigurable logic for massively parallel execution. For our experiments, we have chosen the Ultra96 platform, as it also provides the means for implementing full-fledged intelligent systems using the SoC peripherals, as well as other Operating System (OS) capabilities (i.e., multi-threading) for systems management. For optimizing the model we made use of the Vitis (Xilinx) framework, which enabled us to optimize the full precision model from 7.5 million parameters to 59,095 parameters (125x less), which translated into a reduction of the processing latency of 2.9x. Further optimization (multi-threading and batch normalization) led to an improvement of 7.5x in terms of latency, yielding a performance of 30 Frames Per Second (FPS) without sacrificing accuracy in terms of the evaluated metrics (Dice Score).
Figures & tables
Figure 1: Different types of flames patterns in industrial settings: pool, ball and jet fires. Modified from Palacios (2011).
Figure 2: Visual example of jet fire characterization based on segmentation of three temperature zones.
Figure 3: Overall flow followed in this work: a full precision model is trained using an IR dataset of jet flames. The model is then optimized using Vitis and the resulting moddel is ported on an FPGAs -based SoC architecture for evaluation (adapted from Garduño et al. (2024) ).
Figure 4: Edge computing vs cloud computing.
Smart Camera type
Main Applications
Single-chip
Information sensors.
Embedded
Optical mice, fingerprint readers, smartphones
Stand-alone
Industrial machine vision, human computer interfaces
Compact system
Security, traffic surveillance and machine vision
Distributed
Intelligent video surveillance, industrial machine vision, information gathering systems
Table 1: Types of smart cameras and their main application types.
Figure 5: Comparison in terms of NRE , flexibility, processing performance, power consumption and programmability of different technological choices for implementing smart cameras : a) ASICs ; b) Processor (i.e., SoC ) and Digital Signal Processors ( DSPs ); c) GPU and d) FPGAs (figure adapted from Real and Berry (2010) )
Figure 6: Base hardware architecture used for this project. Images are fed through a USB 3.0 and then an MPSoC device (Ultra96v2) is in charge of processing the video stream by using the integrated Application Processing Unit (or PS, processing system) to run the Operative System (OS), whereas the Programmable Logic section of the SoC implements a neural network accelerator (custom logic).
Figure 7: Proposed solution model for implementing a smart camera for jet fire risk assessment. The Ultra96 board is connected to an IR camera via an USB peripheral within the SoC. The input image is processed by the PS block (an ARM processor) and fed to the PL section where our binarized UNet model resides. After processing the image, this is fed back to the PS to perform feature extraction. These features can be used for visualization purposes on a TFT screen (in the lab) or can be sent via an communication protocol (i.e., LORA) to a cloud provider to implement a full-fledged IIoT solution. Herein, we haven’t implemented these functionalities yet and is left for future work. All the communication between the PS and the PL and peripheral is done through an AXI interconnect, not shown here in order to simplify the diagram.
Tool-chain
Developer
Devices
ARM NN
ARM
ARM-Cortex-A CPU, Mali GPU, Ethos Neural Processor Unit (NPU)
CMSIS NN
ARM
ARM-Cortex-A and ARM-Cortex-M CPUs
Vitis AI
XILINX (AMD)
Deep Learning Processor Unit (DPU) implemented in a Field Programmable Gate Array (FPGAs), Multiprocessing SOC (MPSOC), System on module (SOM)
FINN
XILINX (AMD)
It’s an experimental framework focused on quantized neural networks, which creates custom architectures for each model
OpenVino
Intel
Intel Core, Atom and Xeon CPUs, Integrated GPUs, Vision Processing Units (VPUs) and, FPGAs
Intel FPGA AI Suite
Intel
Agilex, Cyclone 10 GX and, Arria 10 FPGAs
Table 2: Mapping tool-chains developed by hardware vendors to deploy neural networks on their devices.
Feature
Supported
Clock Frequency
300MHz
Double Data Rate
Yes
Throughput
Up to 2304 per clock cycle
Channel Augmentation
Yes
Convolution Layers
Yes
Depthwise Convolution Layers
Yes
Table 3: DPU configuration.
Figure 8: Software stack of Vitis AI .
Figure 9: Process used to port a neural network to an embedded system extending from Vitis AI flow.
Operation
Probability
Range of Values
Vertical Flip
0.5
Horizontal Flip
0.5
Horizontal shift
0.66
± 10% of the pixels
Vertical shift
0.66
± 10% of the pixels
Rotation
0.66
± 30 degrees
Table 4: Data augmentation pipeline.
Figure 10: Flow chart for single- and multi-threading inference approaches.
Model (Metric)
Parameters Count
DI - Floating Point Model
DI - Quantized model
Performance drop
UNet (Dice)
7’481,028
0.953
0.950
N/A
Pruned UNet (Dice)
224,118
0.940
0.942
-1.05 %
UNet (Jaccard)
7’481,028
0.918
0.917
-3.47 %
Pruned UNet (Jaccard)
234,081
0.893
0.887
-6.63 %
UNet Batch Normalization (Cross Entropy)
6’151,748
0.975
0.974
+2.53%
Pruned UNet Batch Normalization (Cross Entropy)
59,095
0.974
0.973
+2.42%
Table 5: Performance drop observed in different models after quantization. Performance values refer to Dice Index (DI). The perceptual drops in the last column are given for the quantized models against the base model, UNet (Dice)
Table 6: Visual comparison of the results obtained with different models for four different images from the jet flame case study and the corresponding ground truth. It is important to note that we are interested to maintain a similar performance across the models.
Model (Metric)
MACs
FLOPs
UNet (Dice)
48,587 M
97,262 M
Pruned UNet (Dice)
7,853 M
15,752 M
UNet (Jaccard)
48,587 M
97,262 M
Pruned UNet (Jaccard)
8,022M
16,091 M
UNet Batch Normalization (Cross Entropy)
41,394 M
82,871 M
Pruned UNet Batch Normalization (Cross Entropy)
401 M
811 M
Table 7: Computational complexity analysis. M refers to millions.
Model (Metric)
DPU latency
Single-thread frame rate
Multi-thread frame rate
UNet (Dice)
0.168 s
5.183 fps
5.83 fps
Pruned UNet (Dice)
0.059 s
11.44 fps
14.98 fps
UNet (Jaccard)
0.164 s
5.20 fps
5.91 fps
Pruned UNet (Jaccard)
0.065 s
10.98 fps
14.01 fps
UNet Batch Normalization (Cross Entropy)
0.185 s
4.51 fps
1.45 fps
Pruned UNet Batch Normalization (Cross Entropy)
0.023 s
14.39 fps
29.94 fps
Table 8: DPU latency and experimental frame rate measured for every model using single- and multi-threading approaches.
Model
Metric
Height
Area
Lift-Off
UNet
MAPE
2.9%
7.9%
5.4%
RMSPE
3.7%
9.2%
6.6%
Attention UNet
MAPE
2.9%
6.5%
5.2%
RMSPE
3.4%
8.2%
7.2%
UNet++
MAPE
2.9%
7.1%
5.8%
RMSPE
3.6%
8.6%
7.7%
Table 9: MAPE and RMSPE errors for 3 full precision segmentation models (UNet, Attention UNet, UNet++) and the best performing optimized model (UNet with Batch Normalization (BN)). Results are reported for a nozzle of 0.02 m in diameter.
Figure 11: Four examples of jet flame measurements using the proposed approach.
Wildfire smoke detection from satellite imagery is critical for early warning and rapid response. For onboard satellite deployment, detection systems must operate under strict memory and latency constraints while providing spatially informative outputs for downstream decision-making. Existing tile-level classification methods are computationally efficient but lack spatial localization, whereas pixel-level segmentation approaches provide detailed masks yet are typically too computationally demanding for real-time onboard execution. To address this gap, we propose WISE (Weakly-supervised Inference-efficient Smoke Extraction), a deployment-oriented framework for onboard fire smoke detection and localization. WISE leverages only tile-level annotations through a teacher-student distillation strategy, where an offline teacher provides soft spatial supervision to a lightweight WISE-Student optimized for efficient onboard inference. The student jointly predicts tile-level smoke presence and smoke probability maps within a single forward pass, enabling spatially informative detection under strict computational constraints. WISE was evaluated through in-orbit execution aboard the ISS-mounted IMAGIN-e payload. Three model variants achieve average inference times of 0.10 s, 0.14 s, and 0.26 s per tile, indicating near-real-time per-tile inference within onboard resource limits. Ground-based experiments on Landsat 5 and Landsat 8 imagery further indicate effective detection and spatially informative localization. The best-performing variant achieves a mean tile-level F1 score of 0.964 and a mean pixel-level F1 score of 0.750 across 10 runs, while containing only 0.12M parameters and requiring approximately 3 GFLOPs. Together, these results indicate that WISE is a practical candidate for low-latency wildfire smoke monitoring from space under onboard resource constraints.
Sha Lu, Yu Sun, Liang Zhao +6
Adelaide University, Adelaide, SA 5000, Australia · Swinburne University of Technology, Hawthorn, VIC 3122, Australia · Thales Alenia Space, Cannes, 06150, France
We present a deployed system for on-orbit wildfire detection aboard a nine-satellite commercial thermal infrared constellation, operating under demanding joint constraints: sub-megabyte model footprint, sub-150 ms per-batch TensorRT FP16 inference on an NVIDIA Jetson Xavier NX, and an end-to-end alert pipeline targeting under 10 minutes from satellite overpass to fire event communication. The system operates on uncalibrated mid-wave infrared (MWIR) single-band imagery at 200 m ground sampling distance, where fires frequently appear as sub-pixel or single-pixel thermal anomalies under extreme class imbalance -- challenges not addressed by the contextual thermal-thresholding pipelines (MODIS, VIIRS) that currently dominate operational fire monitoring. We present an empirical study of lightweight dense representation learning for this regime using a proprietary nine-satellite MWIR dataset. We compare dense masked autoencoding (DenseMAE) and a hybrid DenseMAE+EMA (exponential moving average) distillation variant, and evaluate representations via linear probing and full-distribution pixel-level average precision (AP) under extreme class imbalance. DenseMAE pretraining enables compact downstream models on the latency-accuracy Pareto frontier: our fastest SSL-pretrained model achieves 0.640 test AP and 0.69 event-level Fire-F1 with 65.34 ms latency per batch and a 0.52 MB engine, without pruning or compression. The best configuration reaches 0.699 AP and 0.744 Fire-F1 below 1 MB, outperforming a supervised baseline (0.650 AP) under comparable constraints.
Matthias Rötzer, Veronika Pörtge, Martin Ickerott +6
Building upon our previous Ancilia framework, this work presents YUTELA, an edge--server--cloud AIoT system for long-term, multi-site video anomaly detection. YUTELA replaces workstation-based on-site processing with NVIDIA Jetson Thor edge devices and redesigns the multi-camera pipeline through camera batching and redistribution, shared memory, reduced inter-stage data transfer, task integration, and TensorRT-accelerated inference. It maintains a separate anomaly model per camera, retrained on a centralized GPU server once enough recent pseudo-normal data are collected, with updated weights returned to the edge. With eight cameras, TensorRT increases throughput from 14.17 to 29.94 FPS under normal density, and the optimized edge node exceeds the throughput of the workstation-based Ancilia system in seven of nine workload settings, reaching up to 2.03 times its throughput at extreme density. This gain comes at the cost of higher latency under lighter loads with four or eight streams. YUTELA was then operated continuously for 170 hours at four public sites using 31 existing CCTV cameras. The deployment shows that runtime load is driven by scene activity and input frame rate rather than by camera count, and that retraining frequency varies by about three times across sites. The 101 automatically triggered training events occupied only about 6.6% of a single GPU, indicating that one global node can support adaptation for tens of sites. Mean end-to-end alert latency ranged from 6.8 to 15.4~s depending on site traffic and anomaly type. These results show that YUTELA enables practical multi-camera edge processing with low-cost, camera-specific adaptation across heterogeneous sites.
Shanle Yao, Babak Rahimi Ardabili, Amir Kimiyaie +1
The authors are with the Electrical and Computer Engineering Department, The University of North Carolina at Charlotte, Charlotte, NC, 28223 USA.