cs.ROOct 3, 2026
SaveAsynchronous Tracking, Optical Communication and 3D Motion Capture using Event-based Sensors
Organizations: Systems Theory and Robotics Group, College of Engineering and Computer Science, Australian National University
Abstract
Inter-robot communication is a key enabling technology in cooperative robotics applications. While wireless communication is ubiquitous, simple, and effective, it has inherent limitations: the receiving robot cannot easily localise the source of an incoming signal, clock synchronisation is challenging, and signals are broadcast to all nearby devices rather than targeted recipients. Optical communication provides a complementary channel that is inherently directional and spatially localised, directly addressing these limitations. Event cameras are bio-inspired dynamic vision sensors that respond to changes in image intensity with high temporal resolution, high dynamic range, and low latency, making them well-suited as receivers for high-rate optical communication in cooperative robotic systems. In this paper, we propose the Asynchronous Tracking and Optical Communication (ATOC) system, which integrates LED smart-beacon modulation, event-based detection, optical tracking, and event-data demodulation in a single pipeline. By simultaneously tracking and demodulating multiple LED smart beacons, ATOC transforms a conventional visual marker into a robust communication channel suitable for a wide range of high-impact robotics applications. We validate ATOC in a suite of laboratory studies and two 'applications': a smart-city demonstration and a 3D motion-capture system.
Figures & tables
Fig. 1: Asynchronous Detection, Tracking and Optical Communication Architecture.
Fig. 2: Communications pipeline. In the transmitter, a message is divided into data packets that are first encoded with parity bits via channel coding, producing the encoded packet . This is converted into LED high-low signals through line coding and transmitted. In the receiver, an event camera captures the optical signals and a detection and tracking algorithm is used for data association to discard background visual cues and recover the binary stream of the observed LED. Line decoding reconstructs the received encoded packet , followed by channel decoding to correct errors and recover the data packet . These data packets are then reconstructed into the received message.
| Data Bit | Previous Bit | Encoded LED States |
| 1 | x | L H |
| 0 | 1 | L L |
| 0 | 0 | H L |
TABLE I: Modified Frequency Modulation encoding scheme. A logical ‘1’ is always encoded as ‘LH’. A logical ‘0’ is encoded as ‘LL’ if it follows a ‘1’, or ‘HL’ if it follows a ‘0’.
Fig. 3: Encoded communication packet using extended Hamming (32,26). Pink bits are associated with the start delimiter {LHLLLLH} and are not part of the Hamming codeword. Two bits (blue) are used as a message type tag and then 24 bits (3 bytes) (white) contain the primary data packet. Five parity bits (orange) and one overall parity bit (purple) are distributed throughout the message following the Hamming code structure [ 44 ] .
Fig. 4: Example of modified high-pass filter intensity signals from a smart beacon at different transmission rates (10 kb/s and 16.67 kb/s), shown in both static and moving states (9.2 kpx/s). Detected onset points (dots) and the ground truth MFM (square wave) are overlaid as red. Note that at 16.67 kb/s the data transmission is highly compromised with only 60–65% bit accuracy and only 40–45% message accuracy when static (cf. Tables II and III ). Conversely at 10 kb/s transmission the bit and message accuracy are 100% even when the LED is moving in the image.
Fig. 5: Demonstration of our designed smart beacons, controlled by an STM32F103 microcontroller. We also show the size comparison to the off-the-shelf smart beacon Adafruit FLORA in (b). It is controlled by an ATmega32U4, which is only suitable for lower frequency optical communication applications.
Fig. 6: Experimental setup for spinning disk experiments. The beacon is attached at the perimeter of a 700 mm diameter disk at a distance of 2 m from the camera. The speed of the disk is controlled through the input frequency to a stepper motor. The event camera is mounted alongside a classical machine vision camera (with a signal conditioning circuit to capture frame trigger signals of the machine vision cameras in the event stream) although only the event camera was used in the experiments documented in this paper.
| LED Pulse Duration ( ) | Transmission Rate (kb/s) | Bit Accuracy, 1-BER, (%) | Message Accuracy, 1-MER, (%) |
| 60 | 8.33 | 100.00 | 100.00 |
| 55 | 9.09 | 100.00 | 100.00 |
| 50 | 10.00 | 100.00 | 100.00 |
| 45 | 11.11 | 100.00 | 100.00 |
| 40 | 12.50 | 100.00 | 100.00 |
| 35 | 14.29 | 94.86 | 92.23 |
TABLE II: Bit and message accuracy for a static beacon. The LED pulse duration is dwell time for illumination of the beacon programmed in the STM32 for each LED voltage state (L or H) in the line encoded signal.
| LED Pulse Duration ( ) | Transmission Rate (kb/s) | Speed (px/s) | Bit Accuracy, 1-BER, (%) | Message Accuracy, 1-MER, (%) |
| 50 | 10 | 3.5k | 100.00 | 100.00 |
| 6.5k | 100.00 | 100.00 | ||
| 9.2k | 100.00 | 100.00 | ||
| 45 | 11.11 | 3.5k | 100.00 | 100.00 |
| 6.5k | 100.00 | 100.00 | ||
| 9.2k | 100.00 | 100.00 |
TABLE III: Bit and message accuracy for a moving beacon. The speed (pixels per second) is that of the beacon on the image plane. For each transmission rate, three speeds were evaluated.
Fig. 7: Detection and tracking experiment. The beacon is moving at 12.5 kpx/s and enters and leaves the camera Field of View (FoV). (a) On entering the FoV, the beacon is identified as a candidate within 4 pixels (blue star), validated within 5 ms (red star), and tracked reliably through its trajectory. (b) Packets used to reconstruct the full message are shown (green stars) with the corresponding group ID and ASCII characters. The group ID allows the message to be reassembled even when the first packet decoded is partway through the message. Red stars at the top of the image indicate repeated data. To read the encoded message, start reading from group (0): ‘Sha’ and read upwards in the direction of motion, then return to the first packet decoded once repeated data is received.
Fig. 8: Example of the smart city application. (a) shows the experimental environment, where a bike rider passes by an event camera positioned at the exit of the university campus. (b) illustrates a smart beacon mounted on the bike’s handlebar, transmitting the rider’s information as they pass the camera. (c)–(d) show the event-based reconstruction and the tracked LED beacon marked in red. The beacon first enters the frame at s and is validated by s (within 8 ms).
Fig. 9: Cross-camera clock offset residuals. Left: residuals of a linear fit to the measured time offset between the two event cameras, computed from beacon timestamps decoded simultaneously by both cameras, plotted against sample index. Right: histogram of the residuals with a fitted Gaussian distribution ( s), indicating the precision of the cross-camera temporal alignment.
Fig. 10: LED Calibration Board. (a) Our calibration board with 24 LEDs, each transmitting a unique ID. (b) Feature point correspondence is simplified using the unique beacon IDs and the packet counter for temporal synchronisation. The image reconstruction serves only as a visualisation, camera calibration is performed directly using the AEB tracker coordinates.
| Beacons | Distance | Error ( ) | Color |
|---|---|---|---|
| ID 0 and ID 1 | 194.69 mm | 0.194 mm | Blue |
| ID 0 and ID 2 | 192.17 mm | 0.116 mm | Orange |
| ID 0 and ID 3 | 194.99 mm | 0.135 mm | Yellow |
TABLE IV: Inter-beacon distances
Fig. 11: Calibration of the pyramid baseline distances. Because the beacons were hand-mounted, their true separations are measured before the dynamic experiments, with the pyramid held stationary in the centre of both camera fields of view. This calibrates the geometry of the rig; the intrinsic and extrinsic camera calibration is a separate procedure, described in § VI-B . Left: the three distances from the apex beacon ID 0 to the face beacons ID 1 , ID 2 and ID 3 over time. Right: the corresponding histograms, with the mean indicated by a dashed line. The sample means and standard deviations are listed in Table IV and serve as the reference distances for Figure 13 .
Fig. 12: Rigid 3D structure used to evaluate the precision of our 3D reconstruction. (a) The beacons and pyramid were modelled to the specified dimensions in CAD software, with the beacon IDs as shown. (b) The frame was 3D printed and the beacons were rigidly attached. The frame was printed in sections due to limitations in the 3D printer bed size, which introduced some structural offsets.
Fig. 13: Evaluation of the spatial precision of 3D reconstruction using ATOC for beacon tracking and wireless time synchronisation. Four beacons are programmed to transmit an ID and a packet counter that are used to synchronise the two cameras. The distance between the apex beacon (ID 0 ) and each of the three face beacons (ID 1 , ID 2 , ID 3 ) of the pyramid structure (see Figure 12 ) is compared to the measured baseline distances (194.7, 192.2 and 195.0 mm; see Table IV ). The distances are reconstructed when a full packet is received (packet rate, left) and at each matching onset point (onset rate, right). The two central plots show 500 ms and 5 ms time windows, respectively, highlighting the high temporal resolution our algorithm achieves. The residuals of the distance around the moving mean are shown in each histogram.
Fig. 14: An example of 3D reconstruction results. The two event cameras are shown as blue and red camera frusta in the 3D scene. A reference RGB frame is displayed to the right, showing a person playing the fiddle with nine LED beacons attached to the instrument and body. In the 3D reconstruction, the beacons are colour-coded by the part they are attached to. The 3D models for the violin, bow and boots were sourced from [ 52 , 53 , 54 ] , respectively.
Fig. 15: Experimental setup for 3D reconstruction using the ATOC algorithm. Left: A fiddle, bow, and boots, attached with 9 programmed LED beacons. Right: Stereo event cameras are used to capture data for 3D reconstruction. An RGB camera records reference video, which is not used in the ATOC pipeline.
Fig. 16: High-resolution 3D trajectories of the bow tip and right toe beacons. For each beacon, the reconstructed (red), (green), and (blue) positions are plotted relative to their mean. The top two plots show the trace of position over a s trajectory, while the bottom plots are a 300 zoom in the temporal axis, showing 10 ms. This plot shows data at the temporal resolution of the underlying AEB tracker, 1 MHz.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Fig. 17: Custom smart beacon PCB. Top: 3D rendering of the assembled board; bottom: PCB layout.
More in this topic
Event-Based Vision
Event-based vision uses asynchronous camera measurements to perceive motion and scene changes with high temporal resolution and dynamic range.
See all 227 papers