Probabilistic Pedestrian Forecasts from a Handheld Phone: World-Frame Heat Maps, Visual-Inertial Height Drift, and Evaluation without Ground Truth

Danial Safaei

cs.CVOct 3, 2026

Probabilistic Pedestrian Forecasts from a Handheld Phone: World-Frame Heat Maps, Visual-Inertial Height Drift, and Evaluation without Ground Truth

Authors: Danial Safaei

Organizations: WMG, University of Warwick, Coventry, United Kingdom

Abstract

A pedestrian with a phone could be shown where nearby people will be in the next few seconds, if the forecast stays on the ground while the phone moves, is calibrated, and needs only a monocular camera and visual-inertial odometry (VIO). We build and evaluate such a system. People are detected, lifted onto the floor by ray-plane intersection, tracked in a gravity-aligned metric frame, and forecast as per-step probability maps by a small U-Net trained with a negative log-likelihood (NLL) loss on bird's-eye (SDD) and first-person (EgoTraj-Bench) trajectories. On handheld ADVIO recordings, vertical VIO drift and the user's own changes of level silently rescale monocular ground positions (by 87% within 90 s on one clip; on another, all tracks are lost for the last 31% of the clip); keeping the camera's height above the floor constant under a low-pass-filtered altitude avoids this, though it lags on escalators. On the SDD and EgoTraj-Bench test splits, the final forecaster lowers the NLL at 4.8 s by 1.51 and 1.37 nats relative to a fitted constant-velocity Gaussian. Lacking ground truth for people in handheld video, we score forecasts against the tracker's own later raw measurements. In an internally pre-registered evaluation on seven held-out clips, the final forecaster's NLL is lower than the benchmark-fitted baseline's at 1.2, 2.4 and 4.8 s (by 0.17, 0.23 and 0.47 nats; 95% intervals over people exclude zero), but by less than half as much as on the development clips. Exploratory analyses cut both ways: resampling clips instead of people widens the intervals to include zero at 1.2 and 2.4 s, and once both forecasters are recalibrated on the development clips the network is significantly better only at 1.2 s; but two clips run with ADVIO's reference poses favour the network much more when re-run with the phone's own poses. We discuss what such self-consistency scores can and cannot show.

Figures & tables

Panel of Figure 1 — Figure 1: System overview. Per frame, people are detected, their feet are lifted onto the floor by intersecting pixel rays with a floor plane that follows the phone’s smoothed height ( Section 3.3 ), and tracked in metres. A bird’s-eye map collects where tracked people walked and, at 2 Hz, image segmentation projected onto the floor (the RGB-to-map path is not drawn). Each confirmed track is re-forecast at up to 2.5 Hz as a probability distribution over the ground for 12 future steps of 0.4 s; the stored forecasts are re-projected into every new frame with the current camera pose.

	NLL (nats) $\downarrow$			coverage @4.8 s		area 90	best of 20 (m) $\downarrow$
Method	1.2 s	2.4 s	4.8 s	50 %	90 %	(m 2 )	ADE	FDE
Stanford Drone, TrajNet test (2,829 windows)
CV Gaussian	0.72	2.03	3.68	0.85	0.95	56.6	0.59 ∗	1.19 ∗
Tier A (physics prior)	0.35	1.57	3.09	0.60	0.90	26.7	0.49	0.75
Tier B goals + physics	0.25	1.35	2.54	0.59	0.89	14.7	0.47	0.68
Tier B heat map, no hedge	0.37	1.07	2.20	0.59	0.86	12.3	–	–

Table 1: Test-set results. NLL of the true position (nats; lower is better), coverage of the 50 % and 90 % highest-density regions at 4.8 s (calibrated forecasts reach the nominal level), area of the 90 % region at 4.8 s, and best-of-20 errors over the 4.8 s horizon (only for sample-based forecasters; ∗ single path). Tier A, Tier B goals and the final configuration, the Tier B heat map, include the constant-velocity hedge of Equation 4 (

\varepsilon=0.15

on SDD,

0.25

on EgoTraj-Bench). Bold: best NLL. The last row of each block is the paired difference between the final configuration and the CV baseline with a 95 % bootstrap interval.

				SDD test NLL			EgoTraj test NLL
Seed	$w$ (SDD / Ego)	best epoch	val. score	1.2 s	2.4 s	4.8 s	1.2 s	2.4 s	4.8 s
0	1.0 / 1.0	46	1.338	0.28	1.02	2.17	0.78	1.60	2.94
1	1.0 / 1.0	30	1.358	0.27	1.04	2.20	0.82	1.65	2.96
2	1.0 / 1.0	34	1.343	0.24	1.02	2.17	0.80	1.63	2.95
mean			1.346	0.26	1.03	2.18	0.80	1.63	2.95
s.d.			0.010	0.02	0.01	0.02	0.02	0.03	0.01

Table 2: Training variance: the final configuration trained with three seeds (seed 0 is the network used everywhere else). For each seed, the fusion weight

w

was chosen on its own validation runs and the test sets were scored once with it. “val. score”: the checkpoint-selection score at the best epoch (mean of the SDD and EgoTraj-Bench validation NLLs, Appendix A ). Test NLL of the final configuration (with the hedge), in nats.

Figure 2: Left four panels: change of the phone’s height over the 90 s clip according to ARKit, ADVIO’s reference poses (“GT”) and the phone’s barometer. advio-06: ARKit drifts upward by 1.23 m while the reference poses stay level (barometer +0.33 m, within its noise). advio-22: ARKit and the barometer agree on a descent of about 1.5–1.8 m along a sloping street that the reference poses do not show. advio-23: ARKit fails; the barometer indicates a descent of about 4 m that the reference poses do not show. advio-01 (held out): two escalator rides. Right: confirmed tracks on advio-22 in the development run of Tier A with the floor held at its first-frame height and with the floor following the phone ( Section 3.3 ).

	forecasts scored		within 8 m		time with no track		NLL $-$ CV @1.2 s
Clip	fixed	follows	fixed	follows	fixed	follows	fixed	follows
advio-06	717	770	7 %	20 %	9 %	7 %	$-0.18$	$-0.08$
advio-11	583	568	32 %	20 %	20 %	20 %	$-0.09$	$-0.20$
advio-12	533	543	1 %	1 %	11 %	11 %	$-0.16$	$-0.12$
advio-22	803	1,107	57 %	7 %	31 %	0 %	$+0.42$	$-0.04$
advio-23	147	147	7 %	7 %	20 %	20 %	$+0.04$	$0.00$

Table 3: Floor held fixed at the first frame (as before the fix) versus following the phone, on the development clips (Tier A). Forecasts scored at 1.2 s, the share of them made for people within 8 m, the share of the clip (sampled every 2 s) with no confirmed track, and the mean NLL difference to the CV baseline at 1.2 s (raw truth). The NLL differences of the two variants are not directly comparable, because a wrong floor rescales positions and truth alike.

Ahead	Distance	$n$ (people)	final	Tier A	CV	final $-$ CV	final $-$ Tier A
1.2 s	all	3,135 (462)	2.59	2.93	3.02	$-0.43$ [ $-0.49$ , $-0.38$ ]	$-0.34$ [ $-0.40$ , $-0.30$ ]
	<8 m	361 (106)	0.95	1.54	1.54	$-0.59$ [ $-0.72$ , $-0.49$ ]	$-0.59$ [ $-0.76$ , $-0.44$ ]
	8–15 m	1,069 (222)	1.67	1.98	2.04	$-0.37$ [ $-0.44$ , $-0.30$ ]	$-0.32$ [ $-0.39$ , $-0.25$ ]
	>15 m	1,705 (298)	3.51	3.82	3.95	$-0.44$ [ $-0.52$ , $-0.37$ ]	$-0.31$ [ $-0.39$ , $-0.24$ ]
2.4 s	all	2,137 (311)	3.45	4.17	4.32	$-0.86$ [ $-0.96$ , $-0.77$ ]	$-0.72$ [ $-0.81$ , $-0.63$ ]
	<8 m	158 (42)	1.74	2.65	2.61	$-0.86$ [ $-1.10$ , $-0.64$ ]	$-0.91$ [ $-1.15$ , $-0.66$ ]

Table 4: Development clips (advio-06, 11, 12, 22, 23), pooled. NLL against the raw measurement nearest each horizon, by the person’s distance when the forecast was made;

n

forecasts from the given number of people. Differences are paired, with 95 % intervals from resampling people.

Figure 3: The final forecaster on three held-out clips at the middle of each clip ( $t=45$ s); the clips and the time were fixed in the protocol before the runs, so the frames are not selected. Coloured regions on the floor are the 50 % highest-density regions of each person’s forecast, from 0.4 s ahead (yellow) through green and blue to 4.8 s ahead (magenta), drawn fainter where the forecast is spread out; the inset is the bird’s-eye view (30 m). At this instant advio-03 and advio-20 have nobody close by, and in advio-05 shop-window mannequins are detected as people and forecast. Forecasts of people whose position is uncertain by more than 0.6 m are not drawn (display only; 9 % of the scored forecasts). Selected frames with more people are in Figure 5 . Frames from ADVIO (CC BY-NC 4.0); heads of detected people blurred and the run’s timing line removed.

Ahead	Distance	$n$ (people)	final	Tier A	CV	final $-$ CV	final $-$ Tier A
1.2 s	all	1,870 (337)	2.18	2.96	2.35	$-0.17$ [ $-0.27$ , $-0.06$ ]	$-0.79$ [ $-0.97$ , $-0.59$ ]
	<8 m	1,053 (185)	1.24	2.36	1.22	$+0.01$ [ $-0.13$ , $+0.18$ ]	$-1.13$ [ $-1.37$ , $-0.84$ ]
	8–15 m	347 (103)	2.12	2.55	2.53	$-0.41$ [ $-0.58$ , $-0.25$ ]	$-0.43$ [ $-0.60$ , $-0.28$ ]
	>15 m	470 (107)	4.33	4.62	4.75	$-0.42$ [ $-0.63$ , $-0.24$ ]	$-0.29$ [ $-0.44$ , $-0.14$ ]
2.4 s	all	1,189 (182)	3.18	4.10	3.40	$-0.23$ [ $-0.40$ , $-0.04$ ]	$-0.93$ [ $-1.20$ , $-0.64$ ]
	<8 m	686 (96)	2.31	3.58	2.36	$-0.06$ [ $-0.30$ , $+0.28$ ]	$-1.27$ [ $-1.61$ , $-0.82$ ]

Table 5: Held-out clips (advio-01–05, 20, 21), pooled, as in Table 4 . The pre-registered primary comparison is the first column of differences on the “all” rows.

Comparison (raw truth, all distances)	$n$ @1.2 s	1.2 s	2.4 s	4.8 s
final $-$ CV (primary)	1,870	$-0.17$ [ $-0.27$ , $-0.06$ ]	$-0.23$ [ $-0.40$ , $-0.04$ ]	$-0.47$ [ $-0.75$ , $-0.11$ ]
final $-$ Tier A	1,870	$-0.79$ [ $-0.97$ , $-0.59$ ]	$-0.93$ [ $-1.20$ , $-0.64$ ]	$-0.83$ [ $-1.18$ , $-0.36$ ]
final $-$ final without segmentation	1,870	$+0.06$ [ $-0.02$ , $+0.15$ ]	$+0.10$ [ $-0.04$ , $+0.26$ ]	$+0.07$ [ $-0.16$ , $+0.36$ ]
Tier B goals + physics $-$ CV	1,870	$+0.58$ [ $+0.41$ , $+0.74$ ]	$+0.70$ [ $+0.46$ , $+0.92$ ]	$+0.49$ [ $+0.23$ , $+0.72$ ]
Tier A $-$ CV	1,870	$+0.61$ [ $+0.45$ , $+0.76$ ]	$+0.70$ [ $+0.50$ , $+0.88$ ]	$+0.36$ [ $+0.12$ , $+0.59$ ]
final $-$ CV, smoothed truth	2,249	$-0.16$ [ $-0.27$ , $-0.04$ ]	$-0.13$ [ $-0.34$ , $+0.08$ ]	$-0.32$ [ $-0.63$ , $+0.09$ ]

Table 6: Held-out clips: primary, secondary and sensitivity comparisons (raw truth, all distances, 95 % intervals from resampling people; negative favours the first method). Rows 1–3 and 6–8 are specified by the protocol; rows 4 and 5 compare pre-registered variants with CV, which the protocol did not specify. The last two rows change the geometry, so each compares a variant with its own CV baseline on its own scored forecasts.

	as configured		one width (CV) / blur (final)			mixtures
Ahead	final	CV	final	CV	final $-$ CV	final	CV	final $-$ CV
1.2 s	2.18	2.35	2.18	2.36	$-0.18$ [ $-0.28$ , $-0.07$ ]	1.91	2.05	$-0.14$ [ $-0.20$ , $-0.07$ ]
2.4 s	3.18	3.40	3.17	3.27	$-0.10$ [ $-0.26$ , $+0.07$ ]	2.87	2.90	$-0.04$ [ $-0.15$ , $+0.07$ ]
4.8 s	4.40	4.87	4.36	4.29	$+0.07$ [ $-0.21$ , $+0.41$ ]	3.98	3.93	$+0.05$ [ $-0.10$ , $+0.21$ ]

Table 7: Exploratory (not pre-registered): held-out clips, raw truth, mean NLL (nats) of the final forecaster and the CV baseline as configured and after recalibrating both on the development clips (per horizon and distance band), with one width (our forecast: an extra blur) or with mixtures (CV: two Gaussians with a common mean; our forecast: mixed with a Gaussian around the constant-velocity mean), and the paired difference between the recalibrated forecasters with a 95 % interval from resampling people. All are scored on each forecast’s own grid; the fitted values are in Table 16 .

	detection + tracking	map	forecast	drawing	total	frames/s
GPU (RTX 4000 Ada) + CPU	20	14	15	34	83	12.0
CPU only (Xeon w7-2495X)	135	26	36	34	231	4.3

Table 8: Mean processing time per frame (ms) of the final configuration (segmentation in the map) over the first 30 s (900 frames, 720

\times

1280) of the advio-06 window, a crowded mall clip, with no other experiment running; all frames are included and reading the video is not. “Detection + tracking” also includes lifting and the floor update; “map” is every map update, including segmentation at 2 Hz; “forecast” is at most one person’s forecast per frame, including the preparation of its display layer. The detector runs on ONNX Runtime (CUDA) in the GPU configuration and on OpenCV’s DNN module in the CPU one; the evaluation runs used the latter. Both configurations tracked the same number of people throughout.

		final			CV
Clips	Ahead	50 %	80 %	90 %	50 %	80 %	90 %
development	1.2 s	0.65	0.84	0.89	0.60	0.78	0.84
	2.4 s	0.51	0.74	0.83	0.47	0.65	0.71
	4.8 s	0.39	0.63	0.73	0.40	0.54	0.61
held-out	1.2 s	0.54	0.75	0.86	0.64	0.80	0.86
	2.4 s	0.41	0.65	0.79	0.62	0.75	0.80

Table 9: Phone clips, raw truth, all distances: share of the raw measurements inside the 50, 80 and 90 % highest-density regions of the final forecaster and of the CV baseline (calibrated forecasts reach the nominal level).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	Params	EgoTraj windows	Epochs	Best epoch	SDD val	EgoTraj val	Score
base	1.02 M	10,000	60	36	0.995	2.126	1.561
wide	4.06 M	10,000	60	31	0.970	2.036	1.503
long	1.02 M	10,000	120	29	1.015	1.989	1.502
wide + long	4.06 M	10,000	120	29	1.028	2.061	1.544
all EgoTraj windows	1.02 M	all	60	46	0.979	1.698	1.338

Appendix

Table 10: The five variants trained, the last being the final configuration (one run each, seed 0; NLL on the validation sets at the best epoch, mean over 1.2, 2.4 and 4.8 s; the selection score is the mean of the two).

Clip	Role	Venue	Crowd	Path	Poses	Camera height			Height change (m)
				(m)		(m)	source	horizon	ARKit	GT	barom.
advio-06	dev	Mall	high	127	ARKit	1.42	people (283)	$-0.7^{\circ}$	$+1.23$	$-0.04$	$+0.33$
advio-11	dev	Metro	high	98	ARKit	1.48	people (193)	$+0.4^{\circ}$	$-0.48$	$-0.03$	$-0.10$
advio-12	dev	Metro	high	111	ARKit	1.47	people (61)	$0.0^{\circ}$	$+0.28$	$+0.39$	$+0.12$
advio-22	dev	Outdoor	high	106	ARKit	1.47	people (220)	$+0.4^{\circ}$	$-1.79$	$-0.04$	$-1.54$
advio-23	dev	Outdoor	high	98	ref.	1.53	people (48)	$0.0^{\circ}$	failed	$-0.01$	$-4.23$

Appendix

Table 11: The twelve ADVIO clips (90 s windows with the most people). Venue and crowd level from the ADVIO documentation; horizontal path length of the phone from ADVIO’s reference poses; poses used by the pipeline (ARKit unless it fails a plausibility check over the whole sequence, else ADVIO’s reference poses, “ref.”; † ARKit plausible within the window but not over the whole sequence); camera height above the floor, its source (people as the ruler, with the number of person observations used, or the 1.40 m fallback when the estimate was rejected as implausible) and the horizon correction fitted with it ( Section 3.3 ); change of the phone’s height from the start to the end of the window according to ARKit, ADVIO’s reference poses (GT) and the phone’s barometer.

Method	minADE 20	minFDE 20
SDD TrajNet test, pixels
S-GAN [ 14 ]	27.23	41.44
SoPhie [ 46 ]	16.27	29.38
PECNet [ 33 ]	9.96	15.88
Y-Net [ 32 ]	7.85	11.85
Tier A (ours)	13.97	21.58

Appendix

Table 12: Best-of-20 errors next to published results (as reported by their authors; SDD in pixels of the original images, EgoTraj-Bench in metres). Our values use 20 random samples of the goal-conditioned rollouts; the final heat-map forecaster produces no samples. On EgoTraj-Bench, only BiFlow has been reported on the current data release (by its authors, about 0.30/0.44 m); the paper’s other baselines were run on an earlier release [ 29 ] . We evaluate all 9,828 test agents of the release (our loader skips none).

Figure 4: Choosing the weight

w

of the network’s own heat map against the physics rollout steered by the network’s goals, on validation data only: mean NLL over 1.2, 2.4 and 4.8 s, and coverage of the 90 % region at 4.8 s.

Clip	poses	$n$	final $-$ CV			Tier A $-$ CV
			1.2 s	2.4 s	4.8 s	1.2 s
advio-06	ARKit	770	$-0.52$ [ $-0.63$ , $-0.42$ ]	$-1.06$ [ $-1.23$ , $-0.86$ ]	$-1.32$ [ $-1.71$ , $-0.93$ ]	$-0.08$ [ $-0.15$ , $-0.00$ ]
advio-11	ARKit	568	$-0.57$ [ $-0.70$ , $-0.47$ ]	$-1.06$ [ $-1.28$ , $-0.86$ ]	$-1.63$ [ $-1.98$ , $-1.27$ ]	$-0.20$ [ $-0.29$ , $-0.11$ ]
advio-12	ARKit	543	$-0.40$ [ $-0.51$ , $-0.30$ ]	$-0.89$ [ $-1.12$ , $-0.69$ ]	$-1.09$ [ $-1.61$ , $-0.62$ ]	$-0.12$ [ $-0.16$ , $-0.07$ ]
advio-22	ARKit	1,107	$-0.36$ [ $-0.45$ , $-0.27$ ]	$-0.70$ [ $-0.84$ , $-0.56$ ]	$-1.01$ [ $-1.28$ , $-0.74$ ]	$-0.04$ [ $-0.09$ , $+0.01$ ]
advio-23	reference	147	$-0.12$ [ $-0.35$ , $+0.09$ ]	$-0.12$ [ $-0.66$ , $+0.37$ ]	$-0.19$ [ $-1.29$ , $+0.47$ ]	$0.00$ [ $-0.10$ , $+0.13$ ]

Appendix

Table 13: Development clips, per clip: poses used, forecasts scored at 1.2 s, and paired raw-truth NLL differences (nats; negative favours the first method) of the final forecaster to CV at 1.2, 2.4 and 4.8 s and of Tier A to CV at 1.2 s, with 95 % intervals from resampling people.

Clip	poses	$n$	final $-$ CV			Tier A $-$ CV
			1.2 s	2.4 s	4.8 s	1.2 s
advio-01	ARKit	130	$-0.30$ [ $-0.48$ , $-0.11$ ]	$-0.59$ [ $-0.89$ , $-0.23$ ]	$-0.48$ [ $-1.03$ , $+0.55$ ]	$+0.14$ [ $-0.06$ , $+0.35$ ]
advio-02	reference	267	$-0.92$ [ $-1.24$ , $-0.63$ ]	$-0.72$ [ $-1.33$ , $-0.24$ ]	$+0.10$ [ $-1.42$ , $+0.92$ ]	$-0.34$ [ $-0.56$ , $-0.15$ ]
advio-03	ARKit	178	$-0.19$ [ $-0.41$ , $+0.00$ ]	$-0.03$ [ $-0.66$ , $+0.52$ ]	$+0.24$ [ $-1.65$ , $+0.75$ ]	$+0.22$ [ $+0.06$ , $+0.40$ ]
advio-04	reference	729	$-0.15$ [ $-0.31$ , $+0.06$ ]	$-0.28$ [ $-0.54$ , $+0.11$ ]	$-0.49$ [ $-0.90$ , $+0.14$ ]	$+1.42$ [ $+1.28$ , $+1.53$ ]
advio-05	reference	301	$+0.42$ [ $+0.18$ , $+0.68$ ]	$+0.46$ [ $+0.12$ , $+0.79$ ]	$+0.24$ [ $-0.15$ , $+0.75$ ]	$+0.42$ [ $+0.25$ , $+0.59$ ]

Appendix

Table 14: Held-out clips, per clip, as in Table 13 . The protocol specified the per-clip results of the final forecaster; Tier A

-

CV was not specified.

	forecasts scored		within 8 m		time with no track		NLL $-$ CV @1.2 s
Clip	fixed	follows	fixed	follows	fixed	follows	fixed	follows
advio-01	124	130	0 %	19 %	40 %	38 %	$-0.31$	$-0.30$
advio-02	276	267	9 %	10 %	0 %	0 %	$-0.82$	$-0.92$
advio-03	175	178	37 %	40 %	11 %	11 %	$-0.12$	$-0.19$
advio-04	28	729	82 %	98 %	93 %	2 %	$+0.09$	$-0.15$
advio-05	299	301	68 %	68 %	7 %	7 %	$+0.43$	$+0.42$

Appendix

Table 15: Held-out clips: the final forecaster with the floor held fixed at the first frame versus following the phone (pre-registered secondary analysis); columns as in Table 3 .

Figure 5: Frames selected by us after the runs from the final forecaster’s held-out videos, to show typical output with several people and two failure modes. (a, b) advio-02: forecasts of people walking towards and past the phone; near the camera the 4.8 s regions cover much of the floor. (c) advio-04: a person at the top of an escalator, whose forecast is drawn on the escalator steps. (d) advio-04: seen from an escalator, people on the floor below; the assumption of one level floor a camera height below the phone does not hold here, so their metric positions are wrong even where the overlay, drawn with the same floor, looks plausible. Colours and anonymisation as in Figure 3 .

			one width		CV mixture			final + CV component
Ahead	Distance	$n$ (dev)	CV $s^{\prime}$	blur	$s_{1}$	$s_{2}$	$a$	blur	$s$	$a$
1.2 s	<8 m	361	0.44	0.00	0.17	0.79	0.33	0.00	0.09	0.39
	8–15 m	1,069	0.46	0.00	0.12	1.06	0.29	0.00	0.00	0.41
	>15 m	1,705	1.02	0.00	0.00	2.17	0.35	0.00	2.59	0.03
2.4 s	<8 m	158	0.71	0.00	0.30	1.31	0.36	0.00	0.12	0.20
	8–15 m	703	0.92	0.00	0.37	1.52	0.41	0.25	0.11	0.19

Appendix

Table 16: Exploratory recalibration ( Section 5.6 ): values fitted on the development clips per horizon and distance band (

n

: development forecasts). One width: the CV baseline’s width

s^{\prime}

(m, added in quadrature to the measurement noise) and the extra blur of the final forecaster (m). Mixtures: the CV baseline as two Gaussians with widths

s_{1}<s_{2}

(m, each added in quadrature to the measurement noise) and weight

a

on the second; the final forecaster after its blur, mixed with weight

a

with a Gaussian around the constant-velocity mean of width

s

(m, likewise).

Figure 6: Calibration: share of the truths inside the

\alpha

-highest-density region against

\alpha

, at 1.2, 2.4 and 4.8 s (lighter to darker). Left and middle: the benchmark test splits (true positions), the final forecaster, Tier A and CV,

\alpha=0.5,0.8,0.9,0.95

. Right: the held-out phone clips (raw measurements), the final forecaster and CV only,

\alpha=0.5,0.8,0.9

. Perfect calibration lies on the diagonal; above it, the regions are too wide.

Figure 7: Training curves: the checkpoint-selection score (mean of the SDD and EgoTraj-Bench validation NLLs) of the five variants, the last of which is the final configuration (a); of the final configuration with three seeds and, dashed, retrained with the mirroring fix of Appendix A (b); and the coverage of the 90 % region on EgoTraj-Bench validation (c).

Pedestrian Trajectory Prediction

Pedestrian trajectory prediction forecasts where people will move from their observed motion and surrounding context.

See all 36 papers

Jun 17, 2026

Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-centric Videos

Pedestrian trajectory prediction from an ego-centric camera is challenging since it depends on complex interactions with vehicles and scene context, as well as the intention of the pedestrian. By modelling correlation and intent from the historical and future trajectories of the pedestrian, it will usually result in a multimodal (i.e. multiple modes) distribution. Existing stochastic predictors often sample multiple futures from a single unimodal distribution, which can yield sub-optimal 'mixed-mode' trajectories that lie between distinct motion patterns and become implausible in real scenes. In this paper, we propose MMPM, a mode-aware framework that separately models future trajectory distributions into semantically meaningful modes based on the pedestrian's crossing behavior. MMPM consists of two modules: behavior-aware Pedestrian Interaction Module (PIM) that jointly captures pedestrian-vehicle and pedestrian-environment interactions by introducing gaze, head and hand gesture, and a CVAE-based Mode-aware Trajectory Predictor (MTP) module to model the future trajectory distributions on two modes, crossing and non-crossing the road, separately. A query-based decoder further enforces mode consistency during decoding. Experiments on PIE and JAAD datasets show that our method surpasses state-of-the-art baselines. Our proposed MTP is model-agnostic, which can be integrated into existing frameworks such as BiTrap-NP and SGNet-ED to further improve future trajectory prediction performance. We additionally introduce a data-driven validation protocol that matches predictions to spatio-temporally consistent ground-truth trajectories, demonstrating improved frame-wise displacement errors over previous work.

Yuxuan Xie, Nicolas Pugeault, Chongfeng Wei +2

School of Computing Science, University of Glasgow, Glasgow, United Kingdom · James Watt School of Engineering, University of Glasgow, Glasgow, United Kingdom · Department of Computer Science, Durham University, Durham, United Kingdom

Pedestrian Intention Prediction Pedestrian Behavior Modeling Pedestrian Trajectory Prediction Autonomous Driving Perception Multimodal Trajectory Prediction Trajectory Prediction

Ask PDF

Sep 23, 2026

Kairos: Grounded Forecasting of Presence and Directional Flow in 4D Scene Graphs

Long-term autonomy in human-populated environments requires anticipating whether and how people will move at times a robot has not yet observed. Existing representations of pedestrian motion face a tradeoff: they either forecast future activity, reducing each location to a scalar rate, or model the full directional distribution, holding it fixed in time. We present Kairos, a predictive directional-flow memory that extends a hierarchical 3D scene graph (3DSG) to a 4D scene graph (4DSG). Every observed voxel of the reconstructed geometry stores a directional mixture and a presence rate, and spectral predictors forecast, for any future query time, both the probability that people are present and the full directional distribution of their motion. Pairwise flow dependence between adjacent voxels supports conditional queries, and per-voxel predictive variances yield calibrated credible intervals that tighten as observations accumulate. We evaluate Kairos on three real pedestrian environments: a robot-collected campus dataset, a shopping mall, and a station concourse recorded continuously for eleven months. Its learned state remains consistent under loop-closure corrections, and its forecasts are competitive with dedicated occupancy and flow models trained on the full detection stream, although Kairos learns from only the small fraction available to a patrolling robot. Finally, we validate the representation on a downstream encounter-probability planning task, where plans computed over the Kairos forecasts encounter more people than plans computed over any time-invariant map at an equal success rate. We provide the code at https://github.com/IacopomC/kairos.

Iacopo Catalano, Julio A. Placed, Javier Civera +1

University of Turku, Finland · Instituto Tecnol´ogico de Arag´on (ITA) and the University of Zaragoza, Spain · University of Zaragoza, Spain +1

Spatiotemporal Forecasting Probabilistic Forecasting 4D Scene Understanding Flow Map Learning

Ask PDF

Mar 16, 2026

TrajMamba: An Ego-Motion-Guided Mamba Model for Pedestrian Trajectory Prediction from an Egocentric Perspective

Future trajectory prediction of a tracked pedestrian from an egocentric perspective is a key task in areas such as autonomous driving and robot navigation. The challenge of this task lies in the complex dynamic relative motion between the ego-camera and the tracked pedestrian. To address this challenge, we propose an ego-motion-guided trajectory prediction network based on the Mamba model. Firstly, two Mamba models are used as encoders to extract pedestrian motion and ego-motion features from pedestrian movement and ego-vehicle movement, respectively. Then, an ego-motion guided Mamba decoder that explicitly models the relative motion between the pedestrian and the vehicle by integrating pedestrian motion features as historical context with ego-motion features as guiding cues to capture decoded features. Finally, the future trajectory is generated from the decoded features corresponding to the future timestamps. Extensive experiments demonstrate the effectiveness of the proposed model, which achieves state-of-the-art performance on the PIE and JAAD datasets.

Yusheng Peng, Gaofeng Zhang, Liping Zheng

School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China · School of Software, Hefei University of Technology, Hefei 230601, China

Pedestrian Trajectory Prediction Trajectory Prediction Human Trajectory Prediction

Ask PDF

Abstract

Pedestrian Trajectory Prediction

Explore similar work

Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-centric Videos

Kairos: Grounded Forecasting of Presence and Directional Flow in 4D Scene Graphs

TrajMamba: An Ego-Motion-Guided Mamba Model for Pedestrian Trajectory Prediction from an Egocentric Perspective