Inter-robot communication is a key enabling technology in cooperative robotics applications. While wireless communication is ubiquitous, simple, and effective, it has inherent limitations: the receiving robot cannot easily localise the source of an incoming signal, clock synchronisation is challenging, and signals are broadcast to all nearby devices rather than targeted recipients. Optical communication provides a complementary channel that is inherently directional and spatially localised, directly addressing these limitations. Event cameras are bio-inspired dynamic vision sensors that respond to changes in image intensity with high temporal resolution, high dynamic range, and low latency, making them well-suited as receivers for high-rate optical communication in cooperative robotic systems. In this paper, we propose the Asynchronous Tracking and Optical Communication (ATOC) system, which integrates LED smart-beacon modulation, event-based detection, optical tracking, and event-data demodulation in a single pipeline. By simultaneously tracking and demodulating multiple LED smart beacons, ATOC transforms a conventional visual marker into a robust communication channel suitable for a wide range of high-impact robotics applications. We validate ATOC in a suite of laboratory studies and two 'applications': a smart-city demonstration and a 3D motion-capture system.
Figures & tables
Fig. 1: Asynchronous Detection, Tracking and Optical Communication Architecture.
Fig. 2: Communications pipeline. In the transmitter, a message is divided into data packets X that are first encoded with parity bits via channel coding, producing the encoded packet Xe . This is converted into LED high-low signals L through line coding and transmitted. In the receiver, an event camera captures the optical signals and a detection and tracking algorithm is used for data association to discard background visual cues and recover the binary stream L′ of the observed LED. Line decoding reconstructs the received encoded packet Xe′ , followed by channel decoding to correct errors and recover the data packet X′ . These data packets are then reconstructed into the received message.
Data Bit
Previous Bit
Encoded LED States
1
x
L H
0
1
L L
0
0
H L
TABLE I: Modified Frequency Modulation encoding scheme. A logical ‘1’ is always encoded as ‘LH’. A logical ‘0’ is encoded as ‘LL’ if it follows a ‘1’, or ‘HL’ if it follows a ‘0’.
Fig. 3: Encoded communication packet using extended Hamming (32,26). Pink bits are associated with the start delimiter {LHLLLLH} and are not part of the Hamming codeword. Two bits (blue) are used as a message type tag and then 24 bits (3 bytes) (white) contain the primary data packet. Five parity bits (orange) and one overall parity bit (purple) are distributed throughout the message following the Hamming code structure [ 44 ] .
Fig. 4: Example of modified high-pass filter intensity signals from a smart beacon at different transmission rates (10 kb/s and 16.67 kb/s), shown in both static and moving states (9.2 kpx/s). Detected onset points (dots) and the ground truth MFM (square wave) are overlaid as red. Note that at 16.67 kb/s the data transmission is highly compromised with only ∼ 60–65% bit accuracy and only ∼ 40–45% message accuracy when static (cf. Tables II and III ). Conversely at 10 kb/s transmission the bit and message accuracy are 100% even when the LED is moving in the image.
Fig. 5: Demonstration of our designed smart beacons, controlled by an STM32F103 microcontroller. We also show the size comparison to the off-the-shelf smart beacon Adafruit FLORA in (b). It is controlled by an ATmega32U4, which is only suitable for lower frequency optical communication applications.
Fig. 6: Experimental setup for spinning disk experiments. The beacon is attached at the perimeter of a 700 mm diameter disk at a distance of 2 m from the camera. The speed of the disk is controlled through the input frequency to a stepper motor. The event camera is mounted alongside a classical machine vision camera (with a signal conditioning circuit to capture frame trigger signals of the machine vision cameras in the event stream) although only the event camera was used in the experiments documented in this paper.
LED Pulse Duration ( μs )
Transmission Rate (kb/s)
Bit Accuracy, 1-BER, (%)
Message Accuracy, 1-MER, (%)
60
8.33
100.00
100.00
55
9.09
100.00
100.00
50
10.00
100.00
100.00
45
11.11
100.00
100.00
40
12.50
100.00
100.00
35
14.29
94.86
92.23
TABLE II: Bit and message accuracy for a static beacon. The LED pulse duration is dwell time for illumination of the beacon programmed in the STM32 for each LED voltage state (L or H) in the line encoded signal.
LED Pulse Duration ( μs )
Transmission Rate (kb/s)
Speed (px/s)
Bit Accuracy, 1-BER, (%)
Message Accuracy, 1-MER, (%)
50
10
3.5k
100.00
100.00
6.5k
100.00
100.00
9.2k
100.00
100.00
45
11.11
3.5k
100.00
100.00
6.5k
100.00
100.00
9.2k
100.00
100.00
TABLE III: Bit and message accuracy for a moving beacon. The speed (pixels per second) is that of the beacon on the image plane. For each transmission rate, three speeds were evaluated.
Fig. 7: Detection and tracking experiment. The beacon is moving at 12.5 kpx/s and enters and leaves the camera Field of View (FoV). (a) On entering the FoV, the beacon is identified as a candidate within 4 pixels (blue star), validated within 5 ms (red star), and tracked reliably through its trajectory. (b) Packets used to reconstruct the full message are shown (green stars) with the corresponding group ID and ASCII characters. The group ID allows the message to be reassembled even when the first packet decoded is partway through the message. Red stars at the top of the image indicate repeated data. To read the encoded message, start reading from group (0): ‘Sha’ and read upwards in the direction of motion, then return to the first packet decoded once repeated data is received.
Fig. 8: Example of the smart city application. (a) shows the experimental environment, where a bike rider passes by an event camera positioned at the exit of the university campus. (b) illustrates a smart beacon mounted on the bike’s handlebar, transmitting the rider’s information as they pass the camera. (c)–(d) show the event-based reconstruction and the tracked LED beacon marked in red. The beacon first enters the frame at t=6.011 s and is validated by t=6.019 s (within 8 ms).
Fig. 9: Cross-camera clock offset residuals. Left: residuals of a linear fit to the measured time offset between the two event cameras, computed from beacon timestamps decoded simultaneously by both cameras, plotted against sample index. Right: histogram of the residuals with a fitted Gaussian distribution ( σ=4.54μ s), indicating the precision of the cross-camera temporal alignment.
Fig. 10: LED Calibration Board. (a) Our calibration board with 24 LEDs, each transmitting a unique ID. (b) Feature point correspondence is simplified using the unique beacon IDs and the packet counter for temporal synchronisation. The image reconstruction serves only as a visualisation, camera calibration is performed directly using the AEB tracker coordinates.
Beacons
Distance
Error ( ±1σ )
Color
ID 0 and ID 1
194.69 mm
0.194 mm
Blue
ID 0 and ID 2
192.17 mm
0.116 mm
Orange
ID 0 and ID 3
194.99 mm
0.135 mm
Yellow
TABLE IV: Inter-beacon distances
Fig. 11: Calibration of the pyramid baseline distances. Because the beacons were hand-mounted, their true separations are measured before the dynamic experiments, with the pyramid held stationary in the centre of both camera fields of view. This calibrates the geometry of the rig; the intrinsic and extrinsic camera calibration is a separate procedure, described in § VI-B . Left: the three distances from the apex beacon ID 0 to the face beacons ID 1 , ID 2 and ID 3 over time. Right: the corresponding histograms, with the mean indicated by a dashed line. The sample means and standard deviations are listed in Table IV and serve as the reference distances for Figure 13 .
Fig. 12: Rigid 3D structure used to evaluate the precision of our 3D reconstruction. (a) The beacons and pyramid were modelled to the specified dimensions in CAD software, with the beacon IDs as shown. (b) The frame was 3D printed and the beacons were rigidly attached. The frame was printed in sections due to limitations in the 3D printer bed size, which introduced some structural offsets.
Fig. 13: Evaluation of the spatial precision of 3D reconstruction using ATOC for beacon tracking and wireless time synchronisation. Four beacons are programmed to transmit an ID and a packet counter that are used to synchronise the two cameras. The distance between the apex beacon (ID 0 ) and each of the three face beacons (ID 1 , ID 2 , ID 3 ) of the pyramid structure (see Figure 12 ) is compared to the measured baseline distances (194.7, 192.2 and 195.0 mm; see Table IV ). The distances are reconstructed when a full packet is received (packet rate, left) and at each matching onset point (onset rate, right). The two central plots show 500 ms and 5 ms time windows, respectively, highlighting the high temporal resolution our algorithm achieves. The residuals of the distance around the moving mean are shown in each histogram.
Fig. 14: An example of 3D reconstruction results. The two event cameras are shown as blue and red camera frusta in the 3D scene. A reference RGB frame is displayed to the right, showing a person playing the fiddle with nine LED beacons attached to the instrument and body. In the 3D reconstruction, the beacons are colour-coded by the part they are attached to. The 3D models for the violin, bow and boots were sourced from [ 52 , 53 , 54 ] , respectively.
Fig. 15: Experimental setup for 3D reconstruction using the ATOC algorithm. Left: A fiddle, bow, and boots, attached with 9 programmed LED beacons. Right: Stereo event cameras are used to capture data for 3D reconstruction. An RGB camera records reference video, which is not used in the ATOC pipeline.
Fig. 16: High-resolution 3D trajectories of the bow tip and right toe beacons. For each beacon, the reconstructed X (red), Y (green), and Z (blue) positions are plotted relative to their mean. The top two plots show the trace of position over a 3 s trajectory, while the bottom plots are a 300 × zoom in the temporal axis, showing 10 ms. This plot shows data at the temporal resolution of the underlying AEB tracker, ∼ 1 MHz.
In multi-robot systems, traditional radio frequency (RF) communication struggles with contention and jamming. Optical communication offers a strong alternative. However, conventional frame-based cameras suffer from limited frame rates, motion blur, and reduced robustness under high dynamic range lighting. Event cameras support microsecond temporal resolution and high dynamic range, making them extremely sensitive to scene changes under fast relative motion with an optical transmitter. Leveraging these strengths, we develop a complete optical communication system capable of tracking moving transmitters and decoding messages in real time. Our system achieves over 95% decoding accuracy for text transmission during motion by implementing a Geometry-Aware Unscented Kalman Filter (GA-UKF), achieving 7x faster processing speed compared to the previous state-of-the-art method, while maintaining equivalent tracking accuracy at transmitting frequencies ≥ 1 kHz.
Harmeet Dhillon, Pranay Katyal, Brendan Long +3
Robotics Engineering Department, Worcester Polytechnic Institute, Worcester, MA, USA. · MIT Lincoln Laboratory, Lexington, MA, USA.
Event cameras offer microsecond-level temporal resolution and high dynamic range for low-altitude UAV perception. However, long-range UAVs often produce sparse, fragmented, and noise-contaminated event responses, where one semantic target may appear as multiple spatially separated blobs. Direct blob-level asynchronous tracking therefore suffers from duplicate trajectories and unstable identities. We propose ASUMOT, a motion-consistency-based asynchronous UAV detection and tracking framework operating directly on raw events. ASUMOT models each UAV as a set of motion-consistent event blobs. A local motion-consistency estimator triggers reliable candidates, a lightweight multi-task verifier provides UAV confidence and motion-direction cues, and motion-consistency clustering aggregates fragmented blobs into identity-consistent UAV tracks. We also introduce ES-UAV, a high-definition event-level UAV benchmark with dense semantic annotations. Experiments on public UAV tracking data and ES-UAV show that ASUMOT improves the accuracy--efficiency trade-off while preserving asynchronous event processing. Code and Dataset will be released.
Baofeng Jia, Xiaoyu Chen, Jingyuan Zhang +4
State key Laboratory of Extreme Environment Optoelectronic Dynamic Testing Technology and Instrument, Nanjing University of Science and Technology, Nanjing 210094, China · Jiangsu Key Laboratory of Visual Sensing and Intelligent Perception, Nanjing University of Science and Technology, Nanjing 210094, China
Tracking any point (TAP) is a fundamental yet challenging task in computer vision, requiring high precision and long-term motion reasoning. Recent attempts to combine RGB frames and event streams have shown promise, yet they typically rely on synchronous or non-adaptive fusion, leading to temporal misalignment and severe degradation when one modality fails. We introduce TAPFormer, a transformer-based framework that performs asynchronous temporal-consistent fusion of frames and events for robust and high-frequency arbitrary point tracking. Our key innovation is a Transient Asynchronous Fusion (TAF) mechanism, which explicitly models the temporal evolution between discrete frames through continuous event updates, bridging the gap between low-rate frames and high-rate events. In addition, a Cross-modal Locally Weighted Fusion (CLWF) module adaptively adjusts spatial attention according to modality reliability, yielding stable and discriminative features even under blur or low light. To evaluate our approach under realistic conditions, we construct a novel real-world frame-event TAP dataset under diverse illumination and motion conditions. Our method outperforms existing point trackers, achieving a 28.2% improvement in average pixel error within threshold. Moreover, on standard point tracking benchmarks, our tracker consistently achieves the best performance. Project website: tapformer.github.io
Jiaxiong Liu, Zhen Tan, Jinpu Zhang +4
National University of Defense Technology · Hunan University