Event-based cameras provide high temporal resolution, low power consumption, and a high dynamic range, but progress in event-based object detection remains constrained by the scarcity of annotated training data. To solve this issue, this paper introduces the tool UnrealEvent, a synthetic dataset-generation pipeline implemented in Unreal Engine 5.5. The pipeline combines a V2E-inspired event-frame generator with EasySynth semantic-rendering passes, allowing rendered level sequences to be converted into paired event representations and YOLO-format bounding-box annotations without manual labeling. Events are generated from per-pixel log-luminance differences between consecutive frames, with contrast thresholds sampled independently from a normal distribution to emulate the fabrication-induced threshold mismatch of real sensors. Using this pipeline, we provide a synthetic dataset and evaluate YOLOv8 detectors under synthetic, zero-shot real, and real fine-tuning training on eTraM. The results show strong in-domain synthetic performance but a substantial synthetic-to-real gap, especially for pedestrians. Fine-tuning on a small real subset improves real-domain performance, while the remaining gap highlights the importance of asset diversity and sensor calibration.
Figures & tables
Event-based data simulators
Simulator
Noise
Method
Defining feature
Year
DAVIS Simulator [ 18 ]
Ideal Pixel
Based on Blender
Linear Time Interpolation Sampling
2017
ESIM [ 24 ]
Standard Noise Model
Based on Rendering Engine
Adaptive Sampling
2018
V2E [ 12 ]
Temporal Noise, Leak Events, Finite Bandwidth, Gaussian Threshold Distribution
RGB-video to Event Sequence
Inclusion of Motion Blur Effects
2021
DVS-Voltmeter [ 17 ]
Stochastic Process
Stochastic Process-based Event Simulator
Incorporating Sensor Physics
2022
V2CE [ 34 ]
Learned by Training on Real Event Voxels
RGB-video to Event Sequence
Local Dynamics-Aware Timestamp Inference
2024
Table 1: Overview of simulators
Figure 1: Full simulation pipeline
Figure 2: Example pipeline output: a rendered RGB frame (left) and its corresponding event representation (right). Blue and red denote ON ( ΔL>0 ) and OFF ( ΔL<0 ) events.
YOLOv8 AP50 Results
Data Regime
Pedestrian
Vehicle
eTraM †
0.134
0.314
Synthetic data
0.888±0.009
0.991±0.002
Zero-shot on eTraM
0.059±0.019
0.074±0.010
Fine-tuned on eTraM
0.356±0.012
0.606±0.017
Table 2: YOLOv8 detection performance across data regimes, reported as per-class AP50 (average precision at IoU 0.5 ), mean ± std over three seeds. † Values from the three-class (Pedestrian, Vehicle, Micro-mobility) real-data YOLOv8 baseline published by eTraM [ 29 ] , its Micro-mobility class omitted.
Figure 3: Default pose of the pedestrian model in UE.
Figure 4: Predicted eTraM test data using fine-tuned model.
Configuration
AP50
mAP@ [.5:.95]
Δ AP50 [95% CI]
Baseline
Real-only (from scratch)
0.501±0.009
0.216±0.004
—
Synthetic pre-training (ours)
Frozen backbone
0.481±0.014
0.204±0.005
−0.020 [ −0.031,−0.001 ]
Full fine-tune
0.496±0.024
0.217±0.010
−0.005 [ −0.040,+0.024 ]
Table 3: Controlled comparison on the real eTraM test set (mean ± std over three seeds). AP50 here denotes the two-class mean; Δ is the per-seed paired difference in this quantity against the real-only baseline, with a 95% bootstrap CI.
Synthetic images are increasingly used to augment scarce real data for object detection. However, not all synthetic sets help equally, and the only way to know a set's value is to train a detector on it, which is slow and demands dense annotation. We ask whether a training-free metric can instead rank candidate synthetic training sets by their downstream utility. Existing image-set metrics such as FID, KID, and MMD compare two feature distributions with a single global statistic, which we show is mis-specified for detection-data selection in two ways: it is blind to per-image composition (object count, box scale, class mix), and even at fixed composition its global averaging washes out the appearance differences that separate high-mAP pools from low-mAP ones. We propose Conditional-Composition Domain Match (CCDM), which converts any feature-space distance into a composition-stratified comparison, matching candidate and target within metadata-defined strata without training a detector. On COCO and VisDrone-DET, the best CCDM variant ranks 19 candidate training sets in strong agreement with YOLOv8 mAP (Spearman \r{ho} = 0.97 and 0.96), outperforming FID, KID, and MMD. Furthermore, CCDM holds when reference metadata comes from detector pseudo-labels rather than ground-truth boxes.
Event cameras are increasingly adopted in embodied perception for their microsecond temporal resolution, high dynamic range, and resilience to motion blur. However, training event-based models for robotics requires large-scale, action-conditioned data with dense physical annotations that are difficult to collect in the real world. We introduce EVIS, an open-source physics-grounded event simulator integrated into NVIDIA Isaac Sim that generates events from linear-HDR radiance from a closed-loop robot training episode. Rather than relying on learning-based video interpolation or expensive dense rendering, EVIS exploits renderer-provided motion vectors and depth maps through bi-directional warping and depth-based splatting. This enables high throughput, real-time, and high-fidelity event generation. We evaluate EVIS across runtime efficiency, sim-to-real transfer, and zero-shot model compatibility. A rotation-speed estimator trained solely on EVIS events achieves 2.75 rad/s MAE on real sensor data. Pretrained models for reconstruction, matching, and tracking perform competitively on EVIS events without any fine-tuning. EVIS sustains real-time generation across GPU-parallel environments on a single GPU. Code repository: https://github.com/spikelab-jhu/isaac-sim-event-camera-plugin.
Existing hand detection algorithms work on images and the detection rate is restricted by the frame rate of the camera. In hand detection applications for moving robotic systems, conventional cameras cause motion blur, especially in darker lighting conditions. We can leverage the use of event-based cameras which possess a high dynamic range, high temporal resolution, and low power consumption. Recent work has shown that using a stereo setup of an event-based and a frame-based camera improves detection accuracy and the bandwidth-latency tradeoff. The main bottleneck in using event-based cameras in object detection and recognition tasks is a relatively low amount of training data. In this work, we propose a methodology and an exemplary synthetic event-based hand dataset from an egocentric, first-person view perspective. The data is synthesized from the existing RGB Egohands dataset with the v2e toolbox. Parameters of the v2e toolbox are varied to provide versions of the dataset with different lighting conditions and scales. Ground truth detections are generated with a fine-tuned YOLOv8 model which is applied to the RGB images in the Egohands dataset and interpolated on the high-temporal resolution events. We use the multi-modal dataset to perform hand detection with existing object detection algorithms which use a multi-modal setup of event and RGB cameras and demonstrate performance comparable to the state-of-the-art.
Bharghav Kota, Yulia Sandamirskaya
Zurich University of Applied Sciences, Wädenswil, Switzerland · Zurich University of Applied Sciences Wädenswil, Switzerland