Accurate observation of dynamic environments traditionally relies on synthesizing raw, signal-level information from multiple distributed sensors. This work investigates an alternative approach: performing geospatial inference using only encrypted packet-level information, without access to the raw sensory data. We further explore how this indirect information can be fused with directly available sensory data to extend overall inference capabilities. We introduce GraySense, a learning-based framework that performs geospatial object tracking by analyzing encrypted wireless video transmission traffic, such as packet sizes, from cameras with inaccessible streams. GraySense leverages the inherent relationship between scene dynamics and transmitted packet sizes to infer object motion. The framework consists of two stages: (1) a Packet Grouping module that identifies frame boundaries and estimates frame sizes from encrypted network traffic, and (2) a Tracker module, based on a Transformer encoder with a recurrent state, which fuses indirect packet-based inputs with optional direct camera-based inputs to estimate the object's position. Extensive experiments with realistic videos from the CARLA simulator and emulated networks under varying conditions show that GraySense achieves 2.33 meters tracking error (Euclidean distance) without raw signal access, within the dimensions of tracked objects (4.61m x 1.93m). To our knowledge, this capability has not been previously demonstrated, expanding the use of latent signals for sensing.
Figures & tables
Figure 1. Application scenario of GraySense. The scene illustrates a distributed sensing environment consisting of accessible cameras (blue nodes) and inaccessible cameras (gray nodes) whose video streams are encrypted. While blue nodes provide direct visual input, gray nodes contribute only encrypted network traffic, which indirectly reflects scene dynamics through variations in packet sizes.
Figure 2. Group of Pictures (GOP) in H.264. Each group starts with an I-frame which is encoded independently. P and B frames are encoded based on their difference from the reference I or P frames. Due to this differential encoding, the packet-size variations are informative regarding the total change in the scene.
Figure 3. System overview of GraySense. Information on frame sizes extracted by Packet Grouping along with the extracted image information is fed to the Tracker module which estimates object’s visibility by the sensors and its position when it is visible.
Figure 4. The Genesis setup used for geometric analysis. Four identical cameras are looking towards the sphere. The blue polygon shows the area in which the sphere moves, a region always visible to all cameras. In each experiment, the sphere moves at a random but constant velocity between randomly sampled initial and final points within this area.
Figure 5. Geometric problem. The bold lines denote rays emanating from the camera center. Those rays that are tangent to the sphere intersect the image plane along an ellipse, forming the sphere’s silhouette in the image plane.
Frame-to-Area (Rel. Err.)
Area-to-Pos. (m)
Pos.-from-Area (m)
End-to-End (m)
0.030±0.032
0.123±0.041
0.433±0.258
0.368±0.217
Table 1. Genesis simulation results.
Figure 6. Tracker diagram with packet and image inputs. The extracted image and frame size vectors are tokenized and passed to the transformer along with the recurrent state token from the previous timestep. The output is retained as the next state token, and passed to the FoV and Position predictor heads.
Figure 7. CARLA setup for synthetic data collection. The environment consists of an intersection monitored by four cameras (green), which can serve as either blue nodes or gray nodes depending on the experiment configuration. Vehicles are spawned from four starting areas (blue) and follow one of four possible trajectories or movements (red) per direction . For clarity, only the four trajectories originating from Spawn Area 1 are illustrated.
Data
Network Params (BW, Delay, Jitter)
Stage 1 Pred. Error
Stage 2 FoV Error (%)
Stage 2 Pos Error (m)
Raw Video Access
(Highest Upper Bound)
-
-
0.881 ± 1.266
0.780 ± 0.414
Raw Frame Size
(Network Upper Bound)
-
-
2.669 ± 2.931
2.335 ± 3.487
Packet size
100Mbps, 20ms, 5ms
(6.58 ± 8.10) ×10−5
2.661 ± 3.150
2.961 ± 3.428
50Mbps, 20ms, 5ms
(5.62 ± 7.92) ×10−5
2.533 ± 2.812
2.334 ± 3.180
30Mbps, 20ms, 5ms
(6.34 ± 8.79) ×10−5
2.604 ± 2.861
2.597 ± 3.247
10Mbps, 20ms, 5ms
(6.70 ± 9.42) ×10−5
2.815 ± 3.056
2.907 ± 3.033
Table 2. Main results for reference upper bound settings and different network configurations
Grouping Method
Network Params (BW, Delay, Jitter)
Boundary Pred. Err. (1e-5)
DTW Dist. to True Frame Sizes
Learning -based (Ours)
100Mbps, 20ms, 5ms
6.58 ± 8.10
0.34 ± 1.48
50Mbps, 20ms, 5ms
5.62 ± 7.92
0.32 ± 1.48
30Mbps, 20ms, 5ms
6.34 ± 8.79
0.19 ± 0.92
10Mbps, 20ms, 5ms
6.70 ± 9.42
0.16 ± 0.76
Time window
100Mbps, 20ms, 5ms
-
9.83 ± 5.51
50Mbps, 20ms, 5ms
-
70.87 ± 24.66
Table 3. Stage 1 Methods Comparison.
Tin
Tstride
( Tavg , τ )
State Token
FoV Error (%)
Position Error (m)
20
10
(10, 0.5)
w/
4.62 ± 3.34
4.05 ± 3.49
10
10
(10, 0.5)
w/
3.97 ± 3.12
3.80 ± 3.11
40
10
(10, 0.5)
w/
4.73 ± 3.31
4.76 ± 3.46
20
5
(10, 0.5)
w/
5.12 ± 2.75
5.39 ± 3.73
20
10
(6, 5/6)
w/
2.25 ± 2.70
3.79 ± 3.19
20
10
(20,0.25)
w/
11.89 ± 4.91
5.79 ± 4.07
Table 4. Parameter Search for Stage 2 DNN.
Noise Level
( σH,ν,τGPS )
FoV Error (%)
Pos Error (m)
None
-
1.50 ± 2.20
1.86 ± 2.10
Low
(0.64, 9, 60)
1.87 ± 2.51
2.36 ± 2.38
Medium
(1.00, 5, 300)
2.53 ± 2.81
2.33 ± 3.18
High
(3.70, 5, 300)
6.83 ± 4.70
5.13 ± 3.86
Table 5. Impacts of GPS noise on tracking performance.
Training Set
Error Metric
08:00
09:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
All times
FoV Error (%)
3.87 ± 3.68
4.05 ± 3.76
4.46 ± 4.17
3.60 ± 5.61
3.72 ± 3.60
3.31 ± 2.97
4.07 ± 4.32
3.57 ± 3.27
3.29 ± 3.46
2.82 ± 3.33
Pos. Error (m)
3.44 ± 2.91
4.56 ± 5.65
3.52 ± 3.81
2.86 ± 2.13
3.10 ± 2.36
3.57 ± 3.85
2.84 ± 2.11
3.22 ± 3.39
2.99 ± 3.54
2.97 ± 3.55
Table 6. Affect of time of day and lighting conditions on GraySense tracking performance.
Error Metric
1 Blue
1 Blue + 3 Gray
Visible to Gray Nodes Only
FoV Error (%)
95.11±15.15
7.44±18.35
Pos. Error (m)
27.28±9.23
3.70±4.91
Visible by the Blue Node
FoV Error (%)
2.92±3.84
0.90±2.39
Pos. Error (m)
1.86±0.95
1.44±0.726
Table 7. Comparison of single blue node and its fusion with three gray nodes. Average over the windows of 200 validation experiments is reported.
Vehicle tracking is fundamental to applications ranging from urban mobility and public safety to security and defense. Conventional tracking relies on direct access to sensors that provide strong observations such as vehicle identity and location. In practice, however, factors such as ownership, privacy, cost, and operational constraints may limit directly accessible sensors, leaving sparse observations and long tracking gaps. Meanwhile, many additional third-party sensing assets may be present across the environment but remain inaccessible at the raw-data level, preventing their direct integration into the tracking system. In this work, we investigate whether weak, indirect observations with uncertain spatial and temporal cues can complement sparse direct sensing for vehicle tracking. Specifically, we propose GrayTrack, which fuses weak anonymous events with sparse direct observations using a road-constrained particle filter. We build a CARLA-Mininet-WiFi pipeline to evaluate the system under controlled conditions, generating direct observations from accessible cameras and indirect observations from third-party cameras. Our learning-based detector achieves an F1 score of 0.989 for anonymous vehicle passages. Further, incorporating indirect third-party observations reduces trajectory RMSE by 60.1% and catastrophic track loss from 35.8% to 0.3%. These results demonstrate that GrayTrack can effectively exploit weak indirect observations to extend tracking capabilities.
Gaofeng Dong, Vamsi Eyunni, Pragya Sharma +2
Department of Electrical and Computer Engineering University of California, Los Angeles Los Angeles, CA, USA
Dashcam videos collected by autonomous or assisted-driving systems are increasingly shared for safety auditing and model improvement. Even when explicit GPS metadata are removed, an attacker can still infer the recording location by matching background visual cues (e.g., buildings and road layouts) against large-scale street-view imagery. This paper studies location-privacy leakage under a background-based retrieval attacker, and proposes PPEDCRF, a privacy-preserving enhanced dynamic conditional random field framework that injects calibrated perturbations only into inferred location-sensitive background regions while preserving foreground detection utility. PPEDCRF consists of three components: (i) a dynamic CRF that enforces temporal consistency to discover and track location sensitive regions across frames, (ii) a normalized control penalty (NCP) that allocates perturbation strength according to a hierarchical sensitivity model, and (iii) a utility-preserving noise injection module that minimizes interference to object detection and segmentation. Experiments on public driving datasets demonstrate that PPEDCRF significantly reduces location-retrieval attack success (e.g., Top-k retrieval accuracy) while maintaining competitive detection performance (e.g., mAP and segmentation metrics) compared with common baselines such as global noise, white-noise masking, and feature-based anonymization. The source code is in https://github.com/mabo1215/PPEDCRF.git
Bo Ma, Jinsong Wu, Weiqi Yan +2
Auckland University of Technology, New Zealand. · the Department of Electrical Engineering, Universidad de Chile, Chile.
The recently established 'Out of Sight, Not out of Mind' (OSNOM) task for egocentric videos focuses on tracking objects that are moved by the camera wearer, online, maintaining knowledge of instance locations throughout the video even when they leave the field of view or become heavily occluded. In this paper, we propose the first learning-based solution to the OSNOM task: Whareformer, a transformer-based model with two components: an updatable memory of established tracks and a track assignment module that associates observations with existing tracks in a feed-forward manner. Whareformer jointly reasons over evolving object appearance (what) and updated 3D location (where), and employs a dedicated New Track token to reason about novel objects. Thanks to its design choices of using relative distances and evolving track representations, Whareformer is trained on a small set of 56 videos but achieves SOTA performance on 260 long test videos from three datasets: EPIC-KITCHENS-100 (unseen videos), IT3DEgo, and HD-EPIC, with significant absolute improvements over prior work.
Jacob Chalk, Saptarshi Sinha, Dima Damen +2
University of Bristol, UK · NAVER LABS Europe, France