In highly interactive driving scenes, trajectory prediction is conditioned on information from surrounding traffic participants such as cars and pedestrians. Our main contribution is a comprehensive analysis of state-of-the-art trajectory predictors, which reveals a surprising and critical flaw: many surrounding agents degrade prediction accuracy rather than improve it. Using Shapley-based attribution, we rigorously demonstrate that models learn unstable and non-causal decision-making schemes that vary significantly across training runs. Building on these insights, we propose to integrate a Conditional Information Bottleneck (CIB), which does not require additional supervision and is trained to effectively compress agent features as well as ignore those that are not beneficial for the prediction task. Comprehensive experiments using multiple datasets and model architectures demonstrate that this simple yet effective approach not only improves overall trajectory prediction performance in many cases but also increases robustness to different perturbations. Our results highlight the importance of selectively integrating contextual information, which can often contain spurious or misleading signals, in trajectory prediction. Moreover, we provide interpretable metrics for identifying non-robust behavior and present a promising avenue towards a solution.
Figures & tables
Fig. 1 : (a) Removing a Confounding Agent can markedly improve prediction accuracy. (b) The Insertion Test reveals a U-shaped curve: using only the most helpful agents yields the best results.
Fig. 2 : Proposed architecture. The CIB module compresses surrounding-agent features while conditioning on the target agent’s state; the result is fused with lane geometry and target embeddings in the Interactor before decoding.
Fig. 3 : Insertion test on the nuScenes train and validation sets. On validation, Confounding Agents have a much stronger effect, almost canceling out the Super Agents.
Fig. 4 : Consistency of agent influence for QEANet on nuScenes. (a) Intra-model: a single model is highly consistent across five inference seeds. (b) Inter-model: models trained with different seeds vary significantly, following the random baseline, indicating unstable learned decision-making.
Fig. 5 : Consistency of improving agent influence for the MTR model and the MTR+CIB model compared to human labels. (a) Causal Agents are only slightly more likely to be performance-improving. (b) Non-Causal Agents follow the random baseline, showing the model cannot identify them. For MTR+CIB: (c) the number of improving Causal Agents does not rise, but the mean attribution at rNLL=55 drops significantly; (d) Non-Causal usage still matches the random baseline.
Model
minADE10↓
minADE5↓
MissRate10↓
MissRate5↓
NLL↓
QEANet
1.053±0.010
1.203±0.012
0.479±0.009
0.511±0.009
20.125±0.210
- Super Agents
0.961±0.012
1.106±0.012
0.443±0.009
0.475±0.009
18.336±0.276
- No Agents
1.116±0.014
1.317±0.012
0.542±0.010
0.574±0.008
21.025±0.253
QEANet + CIB
1.042±0.016
1.191±0.017
0.471±0.008
0.504±0.008
19.820±0.341
- Super Agents
0.977±0.017
1.122±0.013
0.448±0.010
0.479±0.006
18.629±0.338
- No Agents
1.109±0.016
1.316±0.018
0.538±0.005
0.571±0.002
20.868±0.328
TABLE I : QEANet and LAformer trained and evaluated on nuScenes. No Agents is the baseline without social context; Super Agents uses only improving-influence agents. The three rows correspond to the right endpoint, minimum, and left endpoint of the insertion curve (Fig. 1b ), respectively. Note that Super Agent identification is a post-hoc, model-dependent analysis relative to a specific metric (NLL here) and requires ground truth. Results are averaged over five runs (mean ± std).
Model
minADE↓
minFDE↓
MissRate↓
MTR
0.673±0.003
1.377±0.009
0.168±0.001
MTR + CIB
0.664±0.003
1.354±0.004
0.164±0.001
EDA
0.654±0.004
1.365±0.011
0.151±0.001
EDA + CIB
0.640±0.006
1.334±0.015
0.147±0.002
TABLE II : Methods trained on a 20% subset of WOMD and evaluated on the validation split; the subsampling of Shi et al. [ 13 ] yields a distribution close to the full dataset. The CIB consistently improves benchmark performance.
Model
minADE↓
minFDE↓
MissRate↓
MTR
0.700±0.026
1.440±0.042
0.164±0.003
- Super Agents
0.695±0.026
1.436±0.040
0.164±0.003
- No Agents
0.893±0.085
1.919±0.220
0.248±0.059
MTR + CIB
0.683±0.009
1.397±0.004
0.160±0.004
- Super Agents
0.683±0.007
1.405±0.014
0.161±0.004
- No Agents
0.855±0.036
1.819±0.106
0.227±0.019
TABLE III : Shapley-value analysis (scenes with at most 24 agents, for compute reasons). The Information Bottleneck decreases the gap between Super Agents and the default (All-agent) performance.
Model
minADE↓
minFDE↓
MissRate↓
BeTop
0.988±0.002
2.267±0.016
0.298±0.012
- Super Agents
0.912±0.016
2.069±0.047
0.272±0.017
- No Agents
1.221±0.019
2.901±0.047
0.373±0.009
BeTop + CIB
1.018±0.023
2.335±0.042
0.318±0.008
- Super Agents
0.937±0.043
2.143±0.113
0.288±0.005
- No Agents
1.241±0.035
2.939±0.067
0.383±0.004
TABLE IV : Joint motion prediction. The marginal contribution of surrounding agents is estimated by fixing the two target trajectories and removing the remaining actors. Both variants (BeTop, BeTop+IB) were trained on the WOMD split over three seeds and evaluated on the Waymo Interaction Challenge validation split.
%Abs( Δ ) Noise,0.2↓
%Abs( Δ ) Noise,0.4↓
QEANet
0.75%±0.07%
1.48%±0.15%
QEANet + CIB
0.72%±0.05%
1.33%±0.15%
LAformer
1.06%±0.06%
2.06%±0.10%
LAformer + CIB
0.35%±0.12%
0.71%±0.24%
TABLE V : Noise-based perturbation robustness ( minADE10 ).
% Abs(Δ)Causal↓
% Abs(Δ)Non-Causal↓
MTR
61.7%±6.3%
12.7%±1.7%
MTR + CIB
58.1%±1.0%
11.6%±0.3%
EDA
72.7%±1.5%
17.0%±0.4%
EDA + CIB
65.6%±4.7%
14.8%±0.7%
TABLE VI : Removal-based perturbation on causal and non-causal agents ( minADE ).
Trajectory prediction methods have demonstrated remarkable capabilities in capturing complex motion patterns. However, existing methods rely on global state assumptions, suffer from insufficient belief inference under partial observability, and lack cognitive behavioral constraints in prediction. These limitations severely compromise both deployment feasibility and physical plausibility in real-world settings. In this work, we propose FEP-Diff, an agent-centric trajectory prediction framework grounded in the Free Energy Principle, aimed at achieving cognitively plausible predictions under realistic constraints. Specifically, a dual-branch spatiotemporal encoder extracts ego-motion dynamics and social interaction cues from local observations. Building upon this, a goal-conditioned belief learner infers multimodal latent belief distributions optimized via a free-energy objective, with a social consistency constraint on the local neighborhood graph to promote cognitive alignment among neighboring agents. Finally, a residual diffusion trajectory generator is conditioned on the learned belief representations with token-level proxy conditioning, producing precise and diverse future predictions. Extensive experiments on five public benchmarks demonstrate that FEP-Diff consistently outperforms state-of-the-art methods under restricted observability. Code: https://anonymous.4open.science/r/FEP-Diff-8876.
Yanping Wu, Ji Zhang, Hao Chen +2
University of Glasgow Glasgow, UK · Southwest Jiaotong University Chengdu, China
Predicting traffic agent trajectories plays an important role in autonomous driving, traffic operations, transportation safety analysis, etc. Although many deep learning algorithms are devised to predict future agent trajectories, the trajectory prediction problem is still challenging due to the complexity of decision-making process, interactions with surrounding vehicles, and the existence of multiple possible intentions for the traveling agents even under similar scenarios. Most existing methods are limited by the requirement of graph structures (e.g., Graph Neural Network) or the requirement of manually labeled intentions. In this study, we propose a pure Transformer-based deep learning model for multi-modal trajectory prediction considering temporal dependencies and agent-agent spatial interactions. After encoding the historical trajectories, two parallel decoders are employed to generate trajectories and probabilities on separate decoder tracks. The model is evaluated on two real-world datasets, one highway dataset and the other pedestrian dataset with solid performance. One important insight is that following a ``propose-then-select'' strategy, the agent-agent spatial interactions are only considered for probability estimation instead of trajectory generation. In summary, the proposed model provides a potential direction to design more robust and effective multi-modal trajectory prediction models.
Diyi Liu, Zihan Niu, Tu Xu +2
College of Metropolitan Transportation, Beijing University of Technology, Beijing, China · Zhejiang Police College, Zhejiang, China
Human behavior has the nature of mutual dependencies, which requires human-robot interactive systems to predict surrounding agents trajectories by modeling complex social interactions, avoiding collisions and executing safe path planning. While there exist many trajectory prediction methods, most of them do not incorporate the own motion of the ego agent and only model interactions based on static information. We are inspired by the humans theory of mind during trajectory selection and propose a Cross time domain intention-interactive method for conditional Trajectory prediction(CiT). Our proposed CiT conducts joint analysis of behavior intentions over time, and achieves information complementarity and integration across different time domains. The intention in its own time domain can be corrected by the social interaction information from the other time domain to obtain a more precise intention representation. In addition, CiT is designed to closely integrate with robotic motion planning and control modules, capable of generating a set of optional trajectory prediction results for all surrounding agents based on potential motions of the ego agent. Extensive experiments demonstrate that the proposed CiT significantly outperforms the existing methods, achieving state-of-the-art performance in the benchmarks.