Field Tracking of Insects Using a Stereoscopic Event-Based Camera Setup
Authors: Pratham G. Shenwai, Martin J. Lankheet, John T. Hrynuk, Mandiyam Y. Mahadeeswara, Mandyam V. Srinivasan, Sridhar Ravi
Organizations: School of Engineering and Technology University of New South Wales Canberra, Australia · Experimental Zoology Group Wageningen University & Research Wageningen, The Netherlands. · DEVCOM Army Research Lab Aberdeen Proving Ground, MD, 21005, USA · Queensland Brain Institute University of Queensland St Lucia, QLD, Australia
High-speed tracking of small, fast-moving organisms in their natural environments is important to better understand their behavior and ecology. Traditional frame-based imaging suffers from motion blur due to low temporal resolution, and data storage limitations, propelling a search for more adaptive solutions. Event cameras, which capture changes in brightness at pixel level instead of entire frames, have emerged as a promising solution by increasing temporal resolution and data efficiency. Here, we demonstrate the use of event-based imaging with standard video-based processing methods by converting the asynchronous events into conventional video formats, allowing us to leverage the event camera's enhanced temporal detail to capture intricate insect flight movements and apply established image analysis techniques. Coupling this conversion process with a stereoscopic configuration provides continuous, low-latency, three-dimensional tracking of fast-moving subjects in field conditions. As a result, we substantially mitigate motion artifacts and achieve more accurate representations of animal movements. By making event-based imaging more readily applicable in natural field settings, our method support broader applications across animal behavior and ecological research, agricultural management, and other fields requiring high-fidelity object tracking in the wild.
Figures & tables
Figure 1: Parameters of an event camera capturing the motion of a palm (moving right to left), with maximum threshold in each case: (a) Palm recorded by a frame-based camera. (b) Raw frame from an event camera, where white pixels represent ON events and blue pixels represent OFF events. (c) BCon_ON : Sets the ON contrast threshold, determining how much a pixel’s brightness must increase to trigger an ON event. Since the threshold is set to maximum, no ON events are registered. (d) BCon_OFF : Sets the OFF contrast threshold, determining how much a pixel’s brightness must decrease to trigger an OFF event. Since the threshold is set to maximum, no OFF events are registered. (e) BLPF : Adjusts the low-pass filter to smooth out rapid light fluctuations. (f) BHPF : Adjusts the high-pass filter to filter out slow changes in illumination.
Figure 2: Flowchart of the 3D tracking system: The process begins with camera parameter adjustments, camera focusing, and stereo calibration, followed by data acquisition and loading of calibration data. In the detection phase, rectified frames from both the left and right cameras are processed independently to detect blobs, and epipolar geometry is computed to match the blobs between the two views. In the tracking phase, these matched blobs are first used to triangulate 3D points, and then each set of points is assigned an ID. A Kalman filter is subsequently updated to track the objects in both 2D and 3D space. Throughout this process, data from each tracker is continuously logged to facilitate 3D trajectory plotting.
Figure 3: Stereo Calibration and Bee Tracking Using Epipolar Geometry: (a) Stereo calibration using a flashing checkerboard to obtain intrinsic and extrinsic parameters. (b) Data acquisition by observing the hive from a distance of approximately 3.4 meters using a stereo setup of event camera, with a blue synchronization cable connecting both cameras. (c) Rectified frame from the left camera detecting and tracking 3 bees in the scene. (d) A rectified frame from the right camera shows detections with epipolar lines overlaid. A red dot on the bee confirms a successful match across both frames. Each colored line represents the 2D trajectory of an individual bee. Once matched, each bee is assigned a consistent ID, represented by the same color used for its trajectory across all frames.
Figure 4: Illustration of honeybee trajectories observed in a 40-second recording session: (a) Displays all trajectories of honeybees during a session. There are two visualizations: the full axis plot, including the depth data for all the movements of bees in relationship to the hive (yellow box) and; a magnified hive entrance view capturing highly specific detail around the hive. (b) a corresponding side view that contextualizes the relative view of the camera with trajectories of bees, with altitude data. (c) A top-down view of the reconstructed trajectories, providing an overview of movement patterns.
Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.
This work introduces and demonstrates the first system capable of imaging fast-moving extended non-rigid objects through strong atmospheric turbulence at high frame rate. Event cameras are a novel sensing architecture capable of estimating high-speed imagery at thousands of frames per second. However, on their own event cameras are unable to disambiguate scene motion from turbulence. In this work, we overcome this limitation using event-based light field cameras: By simultaneously capturing multiple views of a scene, event-based light field cameras and machine learning-based reconstruction algorithms are able to disambiguate motion-induced dynamics, which produce events that are strongly correlated across views, from turbulence-induced dynamics, which produce events that are weakly correlated across view. Tabletop experiments demonstrate event-based light field can overcome strong turbulence while imaging high-speed objects traveling at up to 16,000 pixels per second.
Yu-Hsiang Huang, Levi Burner, Sachin Shah +3
University of Maryland, College Park MD 20742, USA · Dartmouth College, Hanover NH 03755, USA
Object pose tracking is a fundamental and essential task for robotics to perform tasks in the home and industrial settings. The most commonly used sensors to do so are RGB-D cameras, which can hit limitations in highly dynamic environments due to motion blur and frame-rate constraints. Event cameras have remarkable features such as high temporal resolution and low latency, which make them a potentially ideal vision sensors for object pose tracking at high speed. Even so, there are still only few works on 6D pose tracking with event cameras. In this work, we take advantage of the high temporal resolution and propose a method that uses both a propagation step fused with a pose correction strategy. Specifically, we use 6D object velocity obtained from event-based optical flow for pose propagation, after which, a template-based local pose correction module is utilized for pose correction. Our learning-free method has comparable performance to the state-of-the-art algorithms, and in some cases out performs them for fast-moving objects. The results indicate the potential for using event cameras in highly-dynamic scenarios where the use of deep network approaches are limited by low update rates.
Zhichao Li, Chiara Bartolozzi, Lorenzo Natale +1
Event-driven Perception for Robotics, Istituto Italiano di Tecnologia, Italy · Humanoid Sensing and Perception, Istituto Italiano di Tecnologia, Italy · University of Genoa, Genoa, Italy
Event cameras provide low-latency, high temporal resolution perception for real-time vision tasks such as robotics.The novel circuitry (i.e. asynchronous, independent pixels) that enables these advantages also introduces new algorithmic challenges. Velocity-invariant representations alleviate missing observations under slow motion and motion blur under fast motion, but most discard temporal information by converting events into image-like representations. We propose Set of Centre Active Receptive Fields (SCARF), a real-time velocity-invariant representation that preserves raw events while consistently handling fast motion, stationary scenes, and independently moving objects. SCARF achieves state-of-the-art performance in both computational efficiency and representation quality.
Mikihiro Ikura, Luna Gava, Jiahang Wu +2
Istituto Italiano di Tecnologia, Via Morego 30 16163 Genova, Italy