Super Agents and Confounders: Influence of surrounding agents on vehicle trajectory prediction

Daniel Jost; Luca Paparusso; Martin Stoll; Jörg Wagner; Raghu Rajan; Joschka Bödecker

cs.LGApr 3, 2026

Super Agents and Confounders: Influence of surrounding agents on vehicle trajectory prediction

Authors: Daniel Jost, Luca Paparusso, Martin Stoll, Jörg Wagner, Raghu Rajan, Joschka Bödecker

Organizations: Department of Computer Science, University of Freiburg, Germany. · Bosch Center for Artificial Intelligence, Renningen, Germany.

Abstract

In highly interactive driving scenes, trajectory prediction is conditioned on information from surrounding traffic participants such as cars and pedestrians. Our main contribution is a comprehensive analysis of state-of-the-art trajectory predictors, which reveals a surprising and critical flaw: many surrounding agents degrade prediction accuracy rather than improve it. Using Shapley-based attribution, we rigorously demonstrate that models learn unstable and non-causal decision-making schemes that vary significantly across training runs. Building on these insights, we propose to integrate a Conditional Information Bottleneck (CIB), which does not require additional supervision and is trained to effectively compress agent features as well as ignore those that are not beneficial for the prediction task. Comprehensive experiments using multiple datasets and model architectures demonstrate that this simple yet effective approach not only improves overall trajectory prediction performance in many cases but also increases robustness to different perturbations. Our results highlight the importance of selectively integrating contextual information, which can often contain spurious or misleading signals, in trajectory prediction. Moreover, we provide interpretable metrics for identifying non-robust behavior and present a promising avenue towards a solution.

Figures & tables

Panel of Figure 1 — Fig. 1 : (a) Removing a Confounding Agent can markedly improve prediction accuracy. (b) The Insertion Test reveals a U-shaped curve: using only the most helpful agents yields the best results.

Panel of Figure 2 — Fig. 2 : Proposed architecture. The CIB module compresses surrounding-agent features while conditioning on the target agent’s state; the result is fused with lane geometry and target embeddings in the Interactor before decoding.

Fig. 3 : Insertion test on the nuScenes train and validation sets. On validation, Confounding Agents have a much stronger effect, almost canceling out the Super Agents.

Panel of Figure 4 — Fig. 4 : Consistency of agent influence for QEANet on nuScenes. (a) Intra-model: a single model is highly consistent across five inference seeds. (b) Inter-model: models trained with different seeds vary significantly, following the random baseline, indicating unstable learned decision-making.

Panel of Figure 5 — Fig. 5 : Consistency of improving agent influence for the MTR model and the MTR+CIB model compared to human labels. (a) Causal Agents are only slightly more likely to be performance-improving. (b) Non-Causal Agents follow the random baseline, showing the model cannot identify them. For MTR+CIB: (c) the number of improving Causal Agents does not rise, but the mean attribution at $r_{\text{NLL}}=\frac{5}{5}$ drops significantly; (d) Non-Causal usage still matches the random baseline.

Model	$\mathrm{minADE}_{10}\downarrow$	$\mathrm{minADE}_{5}\downarrow$	$\mathrm{MissRate}_{10}\downarrow$	$\mathrm{MissRate}_{5}\downarrow$	$\mathrm{NLL}\downarrow$
QEANet	$1.053\pm 0.010$	$1.203\pm 0.012$	$0.479\pm 0.009$	$0.511\pm 0.009$	$20.125\pm 0.210$
- Super Agents	$0.961\pm 0.012$	$1.106\pm 0.012$	$0.443\pm 0.009$	$0.475\pm 0.009$	$18.336\pm 0.276$
- No Agents	$1.116\pm 0.014$	$1.317\pm 0.012$	$0.542\pm 0.010$	$0.574\pm 0.008$	$21.025\pm 0.253$
QEANet + CIB	$\mathbf{1.042\pm 0.016}$	$\mathbf{1.191\pm 0.017}$	$\mathbf{0.471\pm 0.008}$	$\mathbf{0.504\pm 0.008}$	$\mathbf{19.820\pm 0.341}$
- Super Agents	$0.977\pm 0.017$	$1.122\pm 0.013$	$0.448\pm 0.010$	$0.479\pm 0.006$	$18.629\pm 0.338$
- No Agents	$1.109\pm 0.016$	$1.316\pm 0.018$	$0.538\pm 0.005$	$0.571\pm 0.002$	$20.868\pm 0.328$

TABLE I : QEANet and LAformer trained and evaluated on nuScenes. No Agents is the baseline without social context; Super Agents uses only improving-influence agents. The three rows correspond to the right endpoint, minimum, and left endpoint of the insertion curve (Fig. 1b ), respectively. Note that Super Agent identification is a post-hoc, model-dependent analysis relative to a specific metric (NLL here) and requires ground truth. Results are averaged over five runs (mean

\pm

std).

Model	$\text{minADE}\downarrow$	$\text{minFDE}\downarrow$	$\text{MissRate}\downarrow$
MTR	$0.673\pm 0.003$	$1.377\pm 0.009$	$0.168\pm 0.001$
MTR + CIB	$\mathbf{0.664\pm 0.003}$	$\mathbf{1.354\pm 0.004}$	$\mathbf{0.164\pm 0.001}$
EDA	$0.654\pm 0.004$	$1.365\pm 0.011$	$0.151\pm 0.001$
EDA + CIB	$\mathbf{0.640\pm 0.006}$	$\mathbf{1.334\pm 0.015}$	$\mathbf{0.147\pm 0.002}$

TABLE II : Methods trained on a 20% subset of WOMD and evaluated on the validation split; the subsampling of Shi et al. [ 13 ] yields a distribution close to the full dataset. The CIB consistently improves benchmark performance.

Model	$\text{minADE}\downarrow$	$\text{minFDE}\downarrow$	$\text{MissRate}\downarrow$
MTR	$0.700\pm 0.026$	$1.440\pm 0.042$	$0.164\pm 0.003$
- Super Agents	$0.695\pm 0.026$	$1.436\pm 0.040$	$0.164\pm 0.003$
- No Agents	$0.893\pm 0.085$	$1.919\pm 0.220$	$0.248\pm 0.059$
MTR + CIB	$\mathbf{0.683\pm 0.009}$	$\mathbf{1.397\pm 0.004}$	$\mathbf{0.160\pm 0.004}$
- Super Agents	$0.683\pm 0.007$	$1.405\pm 0.014$	$0.161\pm 0.004$
- No Agents	$0.855\pm 0.036$	$1.819\pm 0.106$	$0.227\pm 0.019$

TABLE III : Shapley-value analysis (scenes with at most 24 agents, for compute reasons). The Information Bottleneck decreases the gap between Super Agents and the default (All-agent) performance.

Model	$\text{minADE}\downarrow$	$\text{minFDE}\downarrow$	$\text{MissRate}\downarrow$
BeTop	$\mathbf{0.988\pm 0.002}$	$\mathbf{2.267\pm 0.016}$	$\mathbf{0.298\pm 0.012}$
- Super Agents	$0.912\pm 0.016$	$2.069\pm 0.047$	$0.272\pm 0.017$
- No Agents	$1.221\pm 0.019$	$2.901\pm 0.047$	$0.373\pm 0.009$
BeTop + CIB	$1.018\pm 0.023$	$2.335\pm 0.042$	$0.318\pm 0.008$
- Super Agents	$0.937\pm 0.043$	$2.143\pm 0.113$	$0.288\pm 0.005$
- No Agents	$1.241\pm 0.035$	$2.939\pm 0.067$	$0.383\pm 0.004$

TABLE IV : Joint motion prediction. The marginal contribution of surrounding agents is estimated by fixing the two target trajectories and removing the remaining actors. Both variants (BeTop, BeTop+IB) were trained on the WOMD split over three seeds and evaluated on the Waymo Interaction Challenge validation split.

	%Abs( $\Delta$ ) ${}_{\text{Noise,0.2}}$ $\downarrow$	%Abs( $\Delta$ ) ${}_{\text{Noise,0.4}}$ $\downarrow$
QEANet	$0.75\%\pm 0.07\%$	$1.48\%\pm 0.15\%$
QEANet + CIB	$\mathbf{0.72\%\pm 0.05\%}$	$\mathbf{1.33\%\pm 0.15\%}$
LAformer	$1.06\%\pm 0.06\%$	$2.06\%\pm 0.10\%$
LAformer + CIB	$\mathbf{0.35\%\pm 0.12\%}$	$\mathbf{0.71\%\pm 0.24\%}$

TABLE V : Noise-based perturbation robustness (

\mathrm{minADE}_{10}

	% $\text{Abs}(\Delta)_{\text{Causal}}$ $\downarrow$	% $\text{Abs}(\Delta)_{\text{Non-Causal}}$ $\downarrow$
MTR	$61.7\%\pm 6.3\%$	$12.7\%\pm 1.7\%$
MTR + CIB	$\mathbf{58.1\%\pm 1.0\%}$	$\mathbf{11.6\%\pm 0.3\%}$
EDA	$72.7\%\pm 1.5\%$	$17.0\%\pm 0.4\%$
EDA + CIB	$\mathbf{65.6\%\pm 4.7\%}$	$\mathbf{14.8\%\pm 0.7\%}$

TABLE VI : Removal-based perturbation on causal and non-causal agents (

\mathrm{minADE}

Vehicle Trajectory Prediction

Vehicle trajectory prediction forecasts how vehicles will move from their histories, road maps, traffic interactions, and driving intentions.

See all 64 papers

Agent-Centric Social Trajectory Prediction: A Free Energy Principle Perspective
May 25, 2026Yanping Wu, Ji Zhang, Hao Chen +2Multi-Agent Trajectory Prediction Probabilistic Forecasting
University of Glasgow Glasgow, UK · Southwest Jiaotong University Chengdu, China
RankFormer: A Propose-then-Select Transformer for Multi-Agent Multimodal Trajectory Prediction
Apr 8, 2026Diyi Liu, Zihan Niu, Tu Xu +2Multi-Agent Trajectory Prediction Multimodal Trajectory Prediction
College of Metropolitan Transportation, Beijing University of Technology, Beijing, China · Zhejiang Police College, Zhejiang, China
Chatting about Conditional Trajectory Prediction
Apr 20, 2026Yuxiang Zhao, Wei Huang, Haipeng Zeng +2Multi-Agent Trajectory Prediction Human Behavior Prediction
Sun Yat-sen University · Alibaba Group

May 25, 2026J/K move · Enter open · S save

Agent-Centric Social Trajectory Prediction: A Free Energy Principle Perspective

Yanping Wu, Ji Zhang, Hao Chen, Edmond S. L. Ho+1

University of Glasgow Glasgow, UK · Southwest Jiaotong University Chengdu, China

Multi-Agent Trajectory Prediction Probabilistic Forecasting Free Energy Principle Trajectory Prediction Probabilistic Trajectory Prediction

Open paper AskPDF arXiv

Trajectory prediction methods have demonstrated remarkable capabilities in capturing complex motion patterns. However, existing methods rely on global state assumptions, suffer from insufficient belief inference under partial observability, and lack cognitive behavioral constraints in prediction. These limitations severely compromise both deployment feasibility and physical plausibility in real-world settings. In this work, we propose FEP-Diff, an agent-centric trajectory prediction framework grounded in the Free Energy Principle, aimed at achieving cognitively plausible predictions under realistic constraints. Specifically, a dual-branch spatiotemporal encoder extracts ego-motion dynamics and social interaction cues from local observations. Building upon this, a goal-conditioned belief learner infers multimodal latent belief distributions optimized via a free-energy objective, with a social consistency constraint on the local neighborhood graph to promote cognitive alignment among neighboring agents. Finally, a residual diffusion trajectory generator is conditioned on the learned belief representations with token-level proxy conditioning, producing precise and diverse future predictions. Extensive experiments on five public benchmarks demonstrate that FEP-Diff consistently outperforms state-of-the-art methods under restricted observability. Code: https://anonymous.4open.science/r/FEP-Diff-8876.

Abstract

Vehicle Trajectory Prediction

Explore similar work

Agent-Centric Social Trajectory Prediction: A Free Energy Principle Perspective

RankFormer: A Propose-then-Select Transformer for Multi-Agent Multimodal Trajectory Prediction

Chatting about Conditional Trajectory Prediction