Revisiting Numerical Forecasting Models for Language-Based Trajectory Prediction

JunGyu Lee; Inhwan Bae; Hae-Gon Jeon

cs.CVOct 6, 2026

Revisiting Numerical Forecasting Models for Language-Based Trajectory Prediction

Authors: JunGyu Lee, Inhwan Bae, Hae-Gon Jeon

Organizations: Yonsei University · DGIST

Abstract

Language-based trajectory predictors represent coordinates as discrete tokens and learn auxiliary tasks such as destination and group reasoning. This formulation enables the model to capture behavioral intent and social context beyond coordinate dynamics alone. However, token-level objectives provide only indirect guidance for continuous coordinate-space dynamics. To address this limitation, we introduce MoRE (Mixture of Reward Experts), a refinement framework that transfers numerical forecasting priors into a pretrained language-based predictor through reinforcement learning. Five frozen numerical predictors provide complementary coordinate-level knowledge of motion and interactions. Their predictions are converted into expert rewards and combined through an uncertainty-weighted consensus that penalizes disagreement. A ground-truth reward anchors the prediction to the target trajectory. To focus refinement on difficult cases, MoRE refines the policy using the top 1% of training samples ranked by predictive entropy. Expert predictions are computed once and cached before PPO training, so the experts are not run during policy updates or inference. In this way, MoRE combines the contextual modeling of the language-based predictor with coordinate-level feedback from numerical experts. On ETH-UCY, MoRE reduces ADE from 0.22 to 0.20 m and FDE from 0.32 to 0.29 m. Relative to the base policy, ADE decreases by 17.9% on SDD and 12.7% on NBA. On ETH-UCY, MoRE also reduces collision rates and better matches ground-truth pedestrian spacing, without increasing measured inference memory or latency. The project page is available at https://jungyu0413.github.io/MoRE/.

Figures & tables

Figure 1: Prediction examples on ETH-UCY. “Multimodal” shows 20 samples, and “Best” shows the closest sample. (a) Numerical baselines and (b) LMTraj exhibit different spatial patterns. (c) MoRE refines the language-based predictions using numerical feedback.

Figure 2: MoRE training pipeline. A frozen base policy ranks training samples by predictive entropy. Numerical expert predictions for the selected subset are cached once. Decoded policy samples receive expert-consensus and ground-truth rewards for PPO refinement, interleaved with supervised learning. Only the refined forecasting policy is needed at inference.

Model	ETH	HOTEL	UNIV	ZARA1	ZARA2	AVG	SDD	GCS
Numerical models
Social GAN	0.77/1.40	0.43/0.88	0.75/1.50	0.35/0.69	0.36/0.72	0.53/1.04	13.6/24.6	15.9/32.6
Social-STGCNN	0.65/1.10	0.50/0.86	0.44/0.80	0.34/0.53	0.31/0.48	0.45/0.75	20.8/33.2	14.7/23.9
PECNet	0.61/1.07	0.22/0.39	0.34/0.56	0.25/0.45	0.19/0.33	0.32/0.56	10.0/15.9	17.1/29.3
Trajectron++	0.61/1.03	0.20/0.28	0.30/0.55	0.24/0.41	0.18/0.32	0.31/0.52	11.4/20.1	12.8/24.2
AgentFormer	0.46/0.80	0.14/0.22	0.25/0.45	0.18/0.30	0.14/0.24	0.23/0.40	8.7/14.9	10.2/16.9

Table 1: Best-of-20 ADE / FDE (ETH-UCY in meters, with SDD and GCS in pixels). Bold and underline indicate best and second-best values, including ties. Model references are in Sec. C.1 .

Table 4

Figure 3: Sampled predictions on ZARA1 and ZARA2 at $t=6$ and $t=12$ , with the best-of-20 trajectory. Rows show (a) numerical baselines, (b) LMTraj, and (c) MoRE. Additional qualitative results are provided in Appendix E .

Table 6

Table 7

Evaluation	Reference	MoRE (Ours)
Non-linear group walking	0.297 / 0.323	0.211 / 0.274
Non-linear mimicry walking	0.195 / 0.277	0.151 / 0.217
Non-linear collision avoidance	0.251 / 0.288	0.198 / 0.264
Non-linear subset average	0.248 / 0.296	0.187 / 0.252
ETH-UCY $\rightarrow$ SDD	10.03 / 17.93	9.14 / 15.32
Inference memory (MB)	1,401	1,401

Table 8: Additional evaluation. Forecasting entries are ADE / FDE. Non-linear errors are in meters and ETH-UCY-to-SDD transfer errors are in pixels. Inference is measured on an RTX 4090. The reference is the base policy for non-linear subsets and LMTraj-SUP for transfer and cost.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Single-expert alignment
Configuration	ADE / FDE
SingularTrajectory	0.211 / 0.315
MART	0.207 / 0.314
MoFlow	0.206 / 0.307
(b) Leave-one-out removal
MoRE without Social-STGCNN	0.207 / 0.291

Appendix

Table 9: Numerical expert configurations on ETH-UCY (ADE / FDE, meters). The proposed five-expert model is the reference for leave-one-out, replacement, and addition tests.

Expert	Group	Mimicry	Non-linear	Collision	Reported avg.
None	0.206/0.256	0.164/0.232	0.322/0.474	0.200/0.275	0.223/0.309
STGCNN	0.202/0.252	0.158/0.212	0.312/0.455	0.215/0.292	0.221/0.301
DMRGCN	0.198/0.248	0.148/ 0.195	0.318/0.462	0.205/0.282	0.224/0.305
GP-Graph	0.192 / 0.242	0.152/0.202	0.310/0.450	0.192 /0.268	0.216 /0.298
SingularTraj.	0.212/0.268	0.145 /0.200	0.292 / 0.428	0.202/0.278	0.218/0.296
ExpertTraj.	0.210/0.266	0.160/0.216	0.308/0.448	0.198/ 0.260	0.220/ 0.290

Appendix

Table 10: Refinement with individual experts on ETH-UCY behavioral subsets (ADE / FDE, meters). Behavioral subsets may overlap, and the reported aggregate is evaluated separately from the four displayed subsets.

Baseline	COL	Integration	COL
GP-Graph	2.44	Ensemble	3.67
LED	2.62	Knowledge distillation	2.92
MART	2.23	Maximum aggregation	2.79
SingularTrajectory	2.96	Mean aggregation	2.37
LMTraj-SUP	2.54	UWO (MoRE)	2.11

Appendix

Table 11: Collision rates on ETH-UCY (%). Lower is better. The baseline values also appear in Tab. 3 .

Method	ETH	HOTEL	UNIV	ZARA1	ZARA2	Reported avg.
Base policy	0.409/0.501	0.120/0.159	0.218/0.344	0.199/0.318	0.175/0.272	0.224/0.318
Supervised FT	0.401/0.514	0.108/0.148	0.247/0.353	0.211/0.319	0.168/0.271	0.227/0.321
Ensemble	0.421/0.612	0.173/0.261	0.292/0.514	0.231/0.382	0.203/0.331	0.264/0.420
Distillation	0.382/0.492	0.122/0.155	0.211/0.338	0.196/0.301	0.177/0.277	0.218/0.313
Reward-weighted reg.	0.388/0.499	0.121/0.158	0.215/0.342	0.198/0.305	0.173/0.283	0.219/0.317
Minimum-risk training	0.412/0.515	0.135/0.172	0.230/0.355	0.210/0.315	0.188/0.298	0.235/0.331

Appendix

Table 12: Per-scene expert integration on ETH-UCY (ADE / FDE, meters). Aggregate values are reproduced at their reported precision.

$\lambda_{\mathrm{exp}}$	0	0.01	0.05	0.1	0.5
0.1	0.223/0.316	0.218/0.309	0.211/0.297	0.213/0.300	0.222/0.315
0.3	0.222/0.312	0.212/0.298	0.208/0.289	0.214/0.292	0.217/0.306
0.5	0.217/0.307	0.210/0.295	0.204/0.285	0.206/0.288	0.211/0.299
0.7	0.219/0.310	0.217/0.302	0.207/0.289	0.211/0.306	0.219/0.316
1.0	0.221/0.309	0.216/0.300	0.210/0.294	0.215/0.297	0.221/0.312

Appendix

Table 13: Reward-coefficient sensitivity on ETH-UCY (ADE / FDE, meters), with

\lambda_{\mathrm{RL}}=0.5

. Columns vary

\lambda_{\mathrm{uwo}}

. Zero is mean aggregation.

$\lambda_{\mathrm{RL}}$	0 (SFT)	0.1	0.3	0.5	0.7	1.0
ADE / FDE	0.227/0.321	0.213/0.301	0.207/0.290	.204/.285	.206/.289	.213/.303

Appendix

Table 14: Sensitivity to

\lambda_{\mathrm{RL}}

on ETH-UCY (ADE / FDE, meters), with

\lambda_{\mathrm{exp}}=0.5

and

\lambda_{\mathrm{uwo}}=0.05

Tuning	Parameters (%)	Data (%)	ADE	FDE
Full FT	100	1	0.213	0.297
LoRA	0.3	100	0.221	0.303
LoRA	0.3	50	0.220	0.312
LoRA	0.3	10	0.219	0.299
LoRA	0.3	0.1	0.223	0.288
LoRA (Ours)	0.3	1	0.204	0.285

Appendix

Table 15: Complete tuning and data-fraction comparison on ETH-UCY (ADE / FDE, meters). Parameter fractions refer to the forecasting backbone and exclude the critic.

Metric	Top 1%	Random 1%	Bottom 1%	All
Mean entropy (nats)	2.8	1.3	0.4	1.3
Base-policy ADE / FDE (m)	0.40/0.60	0.24/0.35	0.05/0.08	0.22/0.32
Constant-velocity error (px)	11.0	4.8	0.8	4.5
Non-linear (%)	82	45	2	41
Collision-involved (%)	25	13	1	9

Appendix

Table 16: Entropy-partitioned training samples on ETH-UCY. ADE / FDE is measured with the base policy. Constant-velocity error uses pixel coordinates. Behavioral labels follow Bae et al. (2024a) .

Model	ADE	FDE
LMTraj-SUP	10.03	17.93
MoRE	9.14	15.32

Appendix

Table 17: Cross-dataset transfer from ETH-UCY to SDD without fine-tuning on SDD. ADE / FDE is measured in pixels.

Model	Deterministic	Stochastic
SocialVAE	0.54 / 1.12	0.21 / 0.33
EigenTrajectory	0.51 / 1.11	0.21 / 0.34
LMTraj-SUP	0.477 / 0.880	0.22 / 0.32
MoRE	0.476 / 0.877	0.20 / 0.29
MoRE + deterministic experts	0.477 / 0.878	0.20 / 0.29

Appendix

Table 18: Deterministic and best-of-20 results on ETH-UCY (ADE / FDE, meters).

Method	Non-linear group walking	Non-linear mimicry walking	Non-linear collision avoidance	Subset average
Base policy	0.297/0.323	0.195/0.277	0.251/0.288	0.248/0.296
MoRE	0.211/0.274	0.151/0.217	0.198/0.264	0.187/0.252

Appendix

Table 19: Non-linear behavioral subsets of ETH-UCY (ADE / FDE, meters). These subsets differ from the four categories in Tab. 5 .

Model	Memory (MB)	Training (h)	Inference (ms)
PECNet	1,733	0.3	57.0
MID	2,929	6.9	35.0
AgentFormer	9,639	22.0	8.2
SocialVAE	1,762	2.1	73.0
LMTraj-SUP	1,401	3.8	18.3
MoRE (Ours)	1,401	3.8 + 1	18.3

Appendix

Table 20: Computational cost on ETH-UCY. Inference memory and latency are measured on an RTX 4090. Training times follow the reported setups and are not hardware-matched.

Platform	Latency (ms)	Rate (Hz)
NVIDIA DGX Spark	55	18.2
NVIDIA AGX Orin	90	11.1

Appendix

Table 21: Additional inference measurements in FP32 with PyTorch and without deployment-specific optimization. These are predictor latencies, not end-to-end navigation measurements.

Figure 5: Visualization of prediction results on the ETH and HOTEL scenes. To aid visualization, selected agents are shown.

Figure 6: Visualization of prediction results on the UNIV and ZARA1 scenes. To aid visualization, selected agents are shown.

Figure 7: Visualization of prediction results on the ZARA2 scene. To aid visualization, selected agents are shown.

Figure 8: Visualization of trajectory predictions on nexus_6 scene. To aid visualization, selected agents are shown.

Figure 9: Visualization of trajectory predictions on gates_2, hyang_0, 1 scenes. To aid visualization, selected agents are shown.

Figure 10: Visualization of prediction results on the coupa_0 and coupa_1 scenes. To aid visualization, selected agents are shown.

Figure 11: Visualization of trajectory predictions on hyang_3 and quad_0 scenes. To aid visualization, selected agents are shown.

Figure 12: Visualization of prediction results on the quad_1, 2, 3 scenes. To aid visualization, selected agents are shown.

Figure 13: Visualization of prediction results on the hyang_8 and little_0, 1 scenes. To aid visualization, selected agents are shown.

Figure 14: Visualization of prediction results on the little_2, 3 and nexus_5 scenes. To aid visualization, selected agents are shown.

Figure 15: Visualization of prediction results on the GCS dataset. To aid visualization, selected agents are shown.

Explore similar work

Oct 5, 2026cs.CV

MoCAR: Motion-code Coordinate-aware AutoRegression for Continuous Trajectory Forecasting

Autoregressive generation is natural for language, where predicted tokens can be directly reused as the next prediction state, but trajectory forecasting lacks such a clean token: motion is continuous, multimodal, and expressed in local coordinate frames that evolve with the predicted trajectory. We present MoCAR (Motion-code Coordinate-aware AutoRegression), a decoder-only framework that casts trajectory forecasting as next-code prediction in a coordinate-aware continuous latent space. MoCAR learns a continuous motion-code space from endpoint-normalized trajectory segments, where each code jointly captures local trajectory geometry and the reference-frame transition induced by that segment. Historical motion codes are used as a teacher-forced prefix, future codes are generated autoregressively under temporal, map, agent, and mode interactions, and predicted codes persist in latent memory while decoded endpoints update the local scene context. This enables rollout without trajectory-space re-tokenization, trajectory queries, goal candidates, or proposal-and-refinement pipelines. On Argoverse (AV) benchmarks, MoCAR achieves top-tier performance with a simple single-stage architecture, transfers strongly from AV2 to AV1 in zero-shot evaluation, and improves on turn-heavy scenarios. Ablations confirm that the learned continuous motion-code space, latent alignment, weak KL regularization, and joint tokenizer-predictor optimization are essential for stable latent autoregression.

Yiming Xu, Hao Cheng, Monika Sester

Institute of Cartography and Geoinformatics Leibniz University Hannover, 30167 Hannover, Germany · Faculty of Geo-Information Science and Earth Observation University of Twente, 7522 NH Enschede, The Netherlands

Topic pending

Ask PDF

May 25, 2026cs.AI

Agent-Centric Social Trajectory Prediction: A Free Energy Principle Perspective

Trajectory prediction methods have demonstrated remarkable capabilities in capturing complex motion patterns. However, existing methods rely on global state assumptions, suffer from insufficient belief inference under partial observability, and lack cognitive behavioral constraints in prediction. These limitations severely compromise both deployment feasibility and physical plausibility in real-world settings. In this work, we propose FEP-Diff, an agent-centric trajectory prediction framework grounded in the Free Energy Principle, aimed at achieving cognitively plausible predictions under realistic constraints. Specifically, a dual-branch spatiotemporal encoder extracts ego-motion dynamics and social interaction cues from local observations. Building upon this, a goal-conditioned belief learner infers multimodal latent belief distributions optimized via a free-energy objective, with a social consistency constraint on the local neighborhood graph to promote cognitive alignment among neighboring agents. Finally, a residual diffusion trajectory generator is conditioned on the learned belief representations with token-level proxy conditioning, producing precise and diverse future predictions. Extensive experiments on five public benchmarks demonstrate that FEP-Diff consistently outperforms state-of-the-art methods under restricted observability. Code: https://anonymous.4open.science/r/FEP-Diff-8876.

Yanping Wu, Ji Zhang, Hao Chen +2

University of Glasgow Glasgow, UK · Southwest Jiaotong University Chengdu, China

Ego-Trajectory Prediction Spatial Encoding Residual Conditional Diffusion Model Free Energy Principle

Ask PDF

May 30, 2026cs.CV

MUSCLE-NET: Predicted-Multiscale-Aware Network for Pedestrian Trajectory Forecasting

Accurate pedestrian trajectory prediction is essential for safe navigation in autonomous driving and intelligent transportation systems. Despite substantial progress made by recent methods, most existing approaches are limited in fully exploiting diverse observations and often overlook the scale dependency of future motion, treating multiscale features uniformly regardless of underlying motion dynamics. This limits their robustness across diverse pedestrian behaviors. To address these challenges, we propose a Predicted-MUltiSCale-Aware Network (MUSCLE-NET) for Pedestrian Trajectory Forecasting that integrates complementary multimodal cues with scale-adaptive prediction mechanisms. The proposed framework is built upon a Multiscale Multimodal Feature Extraction (MMFE) module, which combines multiscale representation, modality-aware recalibration, and directional cross-modal fusion to construct semantically aligned representations from bounding boxes, velocities, and pose information. Building on these features, a Multiscale Enhanced Hierarchical Prediction (MEHP) module performs prediction-aware future-motion refinement via a probabilistic coarse predictor, scale-aligned fusion, and progressive refinement, adaptively selecting scale-relevant cues to mitigate spatial drift. Extensive experiments on the JAAD and PIE benchmarks demonstrate that the proposed MUSCLE-Net achieves competitive performance and consistent gains compared with state-of-the-art trajectory prediction methods.

Yu Liu, Ming Huang, Xiao Ren +3

Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, School of Automation and Intelligent Manufacturing, Southern University of Science and Technology (SUSTech), Shenzhen 518055, China · Department of Mechanical Engineering, City University of Hong Kong, Hong Kong SAR, China

Pedestrian Trajectory Prediction Human Motion Prediction Multi-Scale Convolution Multiscale

Ask PDF