Tracking without Seeing: Geospatial Inference using Encrypted Traffic from Distributed Nodes

Sadik Yagiz Yetim; Gaofeng Dong; Isaac-Neil Zanoria; Ronit Barman; Maggie Wigness; Tarek Abdelzaher; Mani Srivastava; Suhas Diggavi

cs.CVMar 29, 2026

Tracking without Seeing: Geospatial Inference using Encrypted Traffic from Distributed Nodes

Authors: Sadik Yagiz Yetim, Gaofeng Dong, Isaac-Neil Zanoria, Ronit Barman, Maggie Wigness, Tarek Abdelzaher, Mani Srivastava, Suhas Diggavi

Organizations: University of California, Los Angeles · DEVCOM Army Research Laboratory · University of Illinois at Urbana-Champaign

Abstract

Accurate observation of dynamic environments traditionally relies on synthesizing raw, signal-level information from multiple distributed sensors. This work investigates an alternative approach: performing geospatial inference using only encrypted packet-level information, without access to the raw sensory data. We further explore how this indirect information can be fused with directly available sensory data to extend overall inference capabilities. We introduce GraySense, a learning-based framework that performs geospatial object tracking by analyzing encrypted wireless video transmission traffic, such as packet sizes, from cameras with inaccessible streams. GraySense leverages the inherent relationship between scene dynamics and transmitted packet sizes to infer object motion. The framework consists of two stages: (1) a Packet Grouping module that identifies frame boundaries and estimates frame sizes from encrypted network traffic, and (2) a Tracker module, based on a Transformer encoder with a recurrent state, which fuses indirect packet-based inputs with optional direct camera-based inputs to estimate the object's position. Extensive experiments with realistic videos from the CARLA simulator and emulated networks under varying conditions show that GraySense achieves 2.33 meters tracking error (Euclidean distance) without raw signal access, within the dimensions of tracked objects (4.61m x 1.93m). To our knowledge, this capability has not been previously demonstrated, expanding the use of latent signals for sensing.

Figures & tables

Figure 1. Application scenario of GraySense. The scene illustrates a distributed sensing environment consisting of accessible cameras (blue nodes) and inaccessible cameras (gray nodes) whose video streams are encrypted. While blue nodes provide direct visual input, gray nodes contribute only encrypted network traffic, which indirectly reflects scene dynamics through variations in packet sizes.

Figure 2. Group of Pictures (GOP) in H.264. Each group starts with an I-frame which is encoded independently. P and B frames are encoded based on their difference from the reference I or P frames. Due to this differential encoding, the packet-size variations are informative regarding the total change in the scene.

Figure 3. System overview of GraySense. Information on frame sizes extracted by Packet Grouping along with the extracted image information is fed to the Tracker module which estimates object’s visibility by the sensors and its position when it is visible.

Figure 4. The Genesis setup used for geometric analysis. Four identical cameras are looking towards the sphere. The blue polygon shows the area in which the sphere moves, a region always visible to all cameras. In each experiment, the sphere moves at a random but constant velocity between randomly sampled initial and final points within this area.

Figure 5. Geometric problem. The bold lines denote rays emanating from the camera center. Those rays that are tangent to the sphere intersect the image plane along an ellipse, forming the sphere’s silhouette in the image plane.

Frame-to-Area (Rel. Err.)	Area-to-Pos. (m)	Pos.-from-Area (m)	End-to-End (m)
$0.030\pm 0.032$	$0.123\pm 0.041$	$0.433\pm 0.258$	$0.368\pm 0.217$

Table 1. Genesis simulation results.

Figure 6. Tracker diagram with packet and image inputs. The extracted image and frame size vectors are tokenized and passed to the transformer along with the recurrent state token from the previous timestep. The output is retained as the next state token, and passed to the FoV and Position predictor heads.

Figure 7. CARLA setup for synthetic data collection. The environment consists of an intersection monitored by four cameras (green), which can serve as either blue nodes or gray nodes depending on the experiment configuration. Vehicles are spawned from four starting areas (blue) and follow one of four possible trajectories or movements (red) per direction . For clarity, only the four trajectories originating from Spawn Area 1 are illustrated.

Data		Network Params (BW, Delay, Jitter)	Stage 1 Pred. Error	Stage 2 FoV Error (%)	Stage 2 Pos Error (m)
Raw Video Access	(Highest Upper Bound)	-	-	0.881 $\pm$ 1.266	0.780 $\pm$ 0.414
Raw Frame Size	(Network Upper Bound)	-	-	2.669 $\pm$ 2.931	2.335 $\pm$ 3.487
Packet size		100Mbps, 20ms, 5ms	(6.58 $\pm$ 8.10) $\,\times\,10^{-5}$	2.661 $\pm$ 3.150	2.961 $\pm$ 3.428
		50Mbps, 20ms, 5ms	(5.62 $\pm$ 7.92) $\,\times\,10^{-5}$	2.533 $\pm$ 2.812	2.334 $\pm$ 3.180
		30Mbps, 20ms, 5ms	(6.34 $\pm$ 8.79) $\,\times\,10^{-5}$	2.604 $\pm$ 2.861	2.597 $\pm$ 3.247
		10Mbps, 20ms, 5ms	(6.70 $\pm$ 9.42) $\,\times\,10^{-5}$	2.815 $\pm$ 3.056	2.907 $\pm$ 3.033

Table 2. Main results for reference upper bound settings and different network configurations

Grouping Method	Network Params (BW, Delay, Jitter)	Boundary Pred. Err. (1e-5)	DTW Dist. to True Frame Sizes
Learning -based (Ours)	100Mbps, 20ms, 5ms	6.58 $\pm$ 8.10	0.34 $\pm$ 1.48
	50Mbps, 20ms, 5ms	5.62 $\pm$ 7.92	0.32 $\pm$ 1.48
	30Mbps, 20ms, 5ms	6.34 $\pm$ 8.79	0.19 $\pm$ 0.92
	10Mbps, 20ms, 5ms	6.70 $\pm$ 9.42	0.16 $\pm$ 0.76
Time window	100Mbps, 20ms, 5ms	-	9.83 $\pm$ 5.51
	50Mbps, 20ms, 5ms	-	70.87 $\pm$ 24.66

Table 3. Stage 1 Methods Comparison.

$T_{\text{in}}$	$T_{\text{stride}}$	( $T_{\text{avg}}$ , $\tau$ )	State Token	FoV Error (%)	Position Error (m)
20	10	(10, 0.5)	w/	4.62 $\pm$ 3.34	4.05 $\pm$ 3.49
10	10	(10, 0.5)	w/	3.97 $\pm$ 3.12	3.80 $\pm$ 3.11
40	10	(10, 0.5)	w/	4.73 $\pm$ 3.31	4.76 $\pm$ 3.46
20	5	(10, 0.5)	w/	5.12 $\pm$ 2.75	5.39 $\pm$ 3.73
20	10	(6, 5/6)	w/	2.25 $\pm$ 2.70	3.79 $\pm$ 3.19
20	10	(20,0.25)	w/	11.89 $\pm$ 4.91	5.79 $\pm$ 4.07

Table 4. Parameter Search for Stage 2 DNN.

Noise Level	( $\sigma_{H},\nu,\tau_{GPS}$ )	FoV Error (%)	Pos Error (m)
None	-	1.50 $\pm$ 2.20	1.86 $\pm$ 2.10
Low	(0.64, 9, 60)	1.87 $\pm$ 2.51	2.36 $\pm$ 2.38
Medium	(1.00, 5, 300)	2.53 $\pm$ 2.81	2.33 $\pm$ 3.18
High	(3.70, 5, 300)	6.83 $\pm$ 4.70	5.13 $\pm$ 3.86

Table 5. Impacts of GPS noise on tracking performance.

Training Set	Error Metric	08:00	09:00	10:00	11:00	12:00	13:00	14:00	15:00	16:00	17:00
All times	FoV Error (%)	3.87 $\pm$ 3.68	4.05 $\pm$ 3.76	4.46 $\pm$ 4.17	3.60 $\pm$ 5.61	3.72 $\pm$ 3.60	3.31 $\pm$ 2.97	4.07 $\pm$ 4.32	3.57 $\pm$ 3.27	3.29 $\pm$ 3.46	2.82 $\pm$ 3.33
All times	Pos. Error (m)	3.44 $\pm$ 2.91	4.56 $\pm$ 5.65	3.52 $\pm$ 3.81	2.86 $\pm$ 2.13	3.10 $\pm$ 2.36	3.57 $\pm$ 3.85	2.84 $\pm$ 2.11	3.22 $\pm$ 3.39	2.99 $\pm$ 3.54	2.97 $\pm$ 3.55

Table 6. Affect of time of day and lighting conditions on GraySense tracking performance.

Error Metric	1 Blue	1 Blue + 3 Gray
Visible to Gray Nodes Only
FoV Error (%)	$95.11\pm 15.15$	$\textbf{7.44}\pm 18.35$
Pos. Error (m)	$27.28\pm 9.23$	$\textbf{3.70}\pm 4.91$
Visible by the Blue Node
FoV Error (%)	$2.92\pm 3.84$	$\textbf{0.90}\pm 2.39$
Pos. Error (m)	$1.86\pm 0.95$	$\textbf{1.44}\pm 0.726$

Table 7. Comparison of single blue node and its fusion with three gray nodes. Average over the windows of

200

validation experiments is reported.

Encrypted Network Traffic Analysis

Encrypted network traffic analysis infers applications, threats, and network conditions from traffic patterns without decrypting payloads.

See all 29 papers

Beyond Direct Sensing: Harnessing Indirect Observations from Third-Party Sensors in Vehicle Tracking
Sep 16, 2026Gaofeng Dong, Vamsi Eyunni, Pragya Sharma +2Visual Object Tracking Multimodal Sensor Fusion
Department of Electrical and Computer Engineering University of California, Los Angeles Los Angeles, CA, USA
PPEDCRF: Privacy-Preserving Enhanced Dynamic CRF for Location-Privacy Protection for Sequence Videos with Minimal Detection Degradation
Mar 2, 2026Bo Ma, Jinsong Wu, Weiqi Yan +2Conditional Random Fields
Auckland University of Technology, New Zealand. · the Department of Electrical Engineering, Universidad de Chile, Chile.
Whareformer: Learning to Track What is Where in Long Egocentric Videos
Jul 9, 2026Jacob Chalk, Saptarshi Sinha, Dima Damen +2Visual Object Tracking Egocentric Video Understanding
University of Bristol, UK · NAVER LABS Europe, France

Sep 16, 2026J/K move · Enter open · S save

Beyond Direct Sensing: Harnessing Indirect Observations from Third-Party Sensors in Vehicle Tracking

Gaofeng Dong, Vamsi Eyunni, Pragya Sharma, Kang Yang+1

Department of Electrical and Computer Engineering University of California, Los Angeles Los Angeles, CA, USA

Visual Object Tracking Multimodal Sensor Fusion Multi-Object Tracking

Open paper AskPDF arXiv

Vehicle tracking is fundamental to applications ranging from urban mobility and public safety to security and defense. Conventional tracking relies on direct access to sensors that provide strong observations such as vehicle identity and location. In practice, however, factors such as ownership, privacy, cost, and operational constraints may limit directly accessible sensors, leaving sparse observations and long tracking gaps. Meanwhile, many additional third-party sensing assets may be present across the environment but remain inaccessible at the raw-data level, preventing their direct integration into the tracking system. In this work, we investigate whether weak, indirect observations with uncertain spatial and temporal cues can complement sparse direct sensing for vehicle tracking. Specifically, we propose GrayTrack, which fuses weak anonymous events with sparse direct observations using a road-constrained particle filter. We build a CARLA-Mininet-WiFi pipeline to evaluate the system under controlled conditions, generating direct observations from accessible cameras and indirect observations from third-party cameras. Our learning-based detector achieves an F1 score of 0.989 for anonymous vehicle passages. Further, incorporating indirect third-party observations reduces trajectory RMSE by 60.1% and catastrophic track loss from 35.8% to 0.3%. These results demonstrate that GrayTrack can effectively exploit weak indirect observations to extend tracking capabilities.

Abstract

Encrypted Network Traffic Analysis

Explore similar work

Beyond Direct Sensing: Harnessing Indirect Observations from Third-Party Sensors in Vehicle Tracking

PPEDCRF: Privacy-Preserving Enhanced Dynamic CRF for Location-Privacy Protection for Sequence Videos with Minimal Detection Degradation

Whareformer: Learning to Track What is Where in Long Egocentric Videos