SymAnt: Grounding Long-Horizon Action Anticipation with Adaptable Symbolic Priors

Fnu Aryan; Yaqi Xie; Joseph Campbell; Woojun Kim; Kwonjoon Lee; Hossein Nourkhiz Mahjoub; Ehsan Moradi-Pari; Katia Sycara

cs.ROOct 29, 2026

SymAnt: Grounding Long-Horizon Action Anticipation with Adaptable Symbolic Priors

Authors: Fnu Aryan, Yaqi Xie, Joseph Campbell, Woojun Kim, Kwonjoon Lee, Hossein Nourkhiz Mahjoub, Ehsan Moradi-Pari, Katia Sycara,

Organizations: Carnegie Mellon University · Purdue University · Honda Research Institute USA, Inc. · Virginia Tech

Abstract

As robots and intelligent systems increasingly interact with humans, they must anticipate long sequences of future actions from only brief observations. While purely neural models typically require extensive observations to predict future actions and remain difficult to interpret and adjust, symbolic priors offer a structured alternative. To this end, we introduce SymAnt, a novel neuro-symbolic approach that unifies the expressiveness of neural networks with the structured reasoning of scene and knowledge graphs. Given a short video, SymAnt performs a joint-search over both graphs to construct a symbolic context, enabling unified reasoning over spatial relationships, object attributes, affordances, and associated actions. This symbolic context serves as a prior for an iterative refinement process that progressively generates and improves the sequence of future actions. By grounding predictions in symbolic context, SymAnt anticipates long-horizon actions from short observation windows in a parameter-efficient manner, while enabling users to directly edit the underlying knowledge to adjust predictions without further training. Through our experiments, we demonstrate the effectiveness of SymAnt, particularly when only short observation contexts are available, across five diverse datasets: Breakfast, 50 Salads, EPIC-KITCHENS, EGTEA Gaze+, and Assembly101.

Figures & tables

Figure 1: SymAnt architecture: Utilizing a short video of prior observations, we extract a relevant scene graph and utilize a knowledge graph as additional context to predict a sequence of future actions. We propose a joint-graph search to capture additional context, utilizing it as a prior for an iterative action refinement.

	Breakfast ( $\alpha=0.05$ )				Breakfast ( $\alpha=0.1$ )				50 Salads ( $\alpha=0.05$ )				50 Salads ( $\alpha=0.1$ )
Method	$\beta=0.1$	0.2	0.3	0.5	0.1	0.2	0.3	0.5	0.1	0.2	0.3	0.5	0.1	0.2	0.3	0.5
KG [ 8 ]	5.44	4.95	4.22	3.98	6.02	5.15	4.86	4.51	6.92	6.21	6.01	5.58	7.13	6.48	6.07	5.78
RNN [ 2 ]	6.16	5.60	5.53	4.96	7.67	6.73	6.15	5.22	7.98	6.90	6.48	6.42	8.78	7.92	7.57	7.26
DS-r1 [ 36 ]	5.34	8.25	5.00	6.23	7.41	10.20	7.06	8.09	6.67	5.36	4.04	2.80	10.20	6.92	4.95	3.30
FUTR [ 32 ]	9.54	7.24	6.42	5.58	14.70	12.55	12.10	11.71	8.90	7.46	7.29	8.63	15.17	11.34	11.31	11.36
GPT-o1 [ 61 ]	7.61	6.67	7.89	5.15	$-$	$-$	$-$	$-$	15.64	13.12	13.38	11.18	15.18	15.07	13.47	12.29

Table 1: Comparison of anticipation performance with short observations (

\alpha=0.05,0.1

) on Breakfast and 50 Salads datasets. SymAnt is best in every setting, by

3.2\%

on average over NeSCA. Cells marked

-

for the LLM baselines were not evaluated under our inference budget. The last row reports the standard deviation of our results over the official splits.

	Breakfast ( $\alpha=0.2$ )				Breakfast ( $\alpha=0.3$ )				50 Salads ( $\alpha=0.2$ )				50 Salads ( $\alpha=0.3$ )
Method	$\beta=0.1$	0.2	0.3	0.5	0.1	0.2	0.3	0.5	0.1	0.2	0.3	0.5	0.1	0.2	0.3	0.5
DS-r1 [ 36 ]	15.77	11.94	10.61	6.60	16.40	13.13	9.87	6.08	7.83	5.87	3.76	2.30	6.00	4.99	3.56	2.39
RNN [ 2 ]	18.11	17.20	15.94	15.81	21.64	20.02	19.73	19.21	30.06	25.43	18.74	13.49	30.77	17.19	14.79	09.77
GPT-o1 [ 61 ]	$-$	$-$	$-$	$-$	$-$	$-$	$-$	$-$	21.60	21.75	17.86	12.93	28.38	25.08	20.44	15.99
UAAA [ 1 ]	16.71	15.40	14.47	14.20	20.73	18.27	18.42	16.86	24.86	22.37	19.88	12.82	29.10	20.50	15.28	12.31
TimeCond. [ 44 ]	18.41	17.21	16.42	15.84	22.75	20.44	19.64	19.75	32.51	27.61	21.26	15.99	35.12	27.05	22.05	15.59

Table 2: Comparison of long-term anticipation performance with observation horizons

\alpha=0.2,0.3

on the Breakfast and 50 Salads datasets. Results show that SymAnt excels at longer prediction horizons (

\beta=0.5

), with largest margins on 50 Salads.

Figure 2: Qualitative evaluation of scene and knowledge graph integration in a real-world kitchen scenario. Observed video and actions on the left, with predicted actions on the right, demonstrating the relevant portions of the expanded graphs (SG in green, KG in blue).

Method	Params	EPIC-KITCHENS			EGTEA Gaze+
		All	Freq.	Rare	All	Freq.	Rare
I3D [ 14 ]	NA	37.7	53.5	23.0	72.1	79.3	53.3
ActionVLAD [ 31 ]	NA	29.2	53.5	18.6	73.3	79.0	58.6
TimeCeption [ 40 ]	NA	35.6	55.9	26.1	74.1	79.7	59.7
VideoGraph [ 41 ]	NA	22.5	49.4	14.0	67.7	77.1	47.2
EGO-TOPO [ 59 ]	NA	38.0	56.9	29.2	73.5	80.7	54.7

Table 3: Performance (mAP) on EPIC-KITCHENS-55 and EGTEA Gaze+ across all, frequent, and rare splits. Parameter counts are approximated and report trainable parameters only; for SymAnt, the frozen object detector, ViT node encoder, and I3D backbone are excluded.

	$\beta$ ( $\alpha=0.2$ )				$\beta$ ( $\alpha=0.3$ )
Method	0.1	0.2	0.3	0.5	0.1	0.2	0.3	0.5
Tri-gram [ 1 ]	2.8	2.2	1.9	1.5	3.5	2.7	2.3	1.8
UAAA ∗ [ 1 ]	2.7	2.1	1.8	1.6	2.4	2.1	1.9	1.7
FUTR ∗ [ 32 ]	7.5	5.5	4.7	4.1	7.8	6.0	5.2	4.0
GTDA [ 78 ]	6.4	4.5	3.5	2.8	5.9	4.2	3.5	2.9
MANTA [ 79 ]	6.7	5.3	4.2	3.5	6.6	4.7	4.2	3.5

Table 4: Long-term dense anticipation on Assembly101 (MoC). ∗ : retrained on Assembly101 [ 78 ] .

			Breakfast		50 Salads
			$\beta$ ( $\alpha=0.2$ )		$\beta$ ( $\alpha=0.2$ )
Enc.	Dec.	KG	0.1	0.5	0.1	0.5
T	T	$-$	27.03 ±1.4	20.82 ±0.3	36.77 ±5.3	20.12 ±6.9
M	T	$-$	26.70 ±1.6	20.25 ±0.8	36.20 ±8.3	21.60 ±2.7
M	R	$-$	27.20 ±1.1	23.60 ±0.4	37.20 ±4.2	27.70 ±5.6
$-$	R	I	17.81 ±1.6	11.86 ±0.7	25.20 ±7.3	12.41 ±5.6

Table 5: Ablation on model configurations. Enc. : M (Mamba), T (Transformer). Dec. : R (Iterative Refinement), D (Diffusion), S (DDIM sampling [ 68 ] ). KG (symbolic context): I (init. from joint SG+KG search), C (cond. on joint SG+KG search), K (init. from KG-only search, scene graph disabled), O (init. from max-pooled CLIP embeddings, no graph), A (init. from a GAT, no graph search), G (init. from a Visual Genome KG [ 46 ] ).

Figure 3: Quantitative analysis of anticipation with manual graph edits: accuracy drops on deactivation and is

100\%

on activation.

Enc.	Dec.	Enc.	Dec.	Cont.	0.1	0.5	0.1	0.5
		Inference Time			Performance
		(ms)			$\beta$ ( $\alpha=0.2$ )		$\beta$ ( $\alpha=0.3$ )
T	T	1.5	1.6	0.9	39.48	23.60	34.65	19.50
M	T	1.1	1.6	0.9	39.16	23.84	34.72	19.46
M	R(10)	1.1	14.5	0.9	40.78	28.19	35.23	25.50
M	R(10) †	1.1	14.5	0.9	40.41	27.55	35.13	25.01

Table 6: Module-wise inference time (ms) on 50 Salads; all rows use KG initialization (I). Cont.: symbolic context aggregation.

\dagger

: SG built with Faster R-CNN.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

			EPIC Kitchens			EGTEA Gaze+
Enc.	Dec.	KG	All	Frequent	Rare	All	Frequent	Rare
T	T	$-$	37.9	55.3	29.3	72.1	75.4	59.2
M	T	$-$	38.6	56.1	29.7	72.1	76.8	59.4
M	D	C	38.5	55.3	29.8	73.6	78.4	61.6
M	R	$-$	39.7	56.4	31.8	74.4	79.7	63.8
M	R	I	42.4	59.3	32.2	78.7	84.9	65.5

Appendix

Table 7: Comparison of anticipation accuracy across different model configurations for EPIC Kitchens and EGTEA Gaze+ dataset. Enc. (Encoder): M (Mamba), T (Transformer). Dec. (Decoder): R (Iterative Refinement), T (Transformer), D (Diffusion model with explicit forward process [ 86 , 33 ] ). KG (Knowledge Graph Integration): I (Initialization), C (Conditioning),

-

(None).

Figure 4: Analysis of intermediate refinement steps. We report mean accuracy and std. deviation over intermediate inference steps for a model trained for 10 steps.

Figure 5: MoC accuracy vs. number of steps in the refinement process on the Breakfast dataset, showing that accuracy at 50 and 100 steps does not significantly improve compared to 10 steps.

AR	Rec.	Init.	0.1	0.5	0.1	0.5	0.1	0.5	0.1	0.5
			Breakfast				50 Salads
			$\beta$ ( $\alpha=0.2$ )		$\beta$ ( $\alpha=0.3$ )		$\beta$ ( $\alpha=0.2$ )		$\beta$ ( $\alpha=0.3$ )
✓			23.75	19.21	26.64	23.28	32.19	22.05	30.61	17.52
✓		✓	24.02	19.44	26.21	23.75	32.78	22.70	31.74	17.26
✓	✓		28.07	25.81	34.12	29.51	40.69	27.98	35.51	25.31
✓	✓	✓	28.22	25.67	34.22	30.11	40.78	28.19	35.23	25.50

Appendix

Table 8: Results for different loss terms on the Breakfast and 50 Salads datasets, highlighting the significant impact of the recognition loss on model performance.

		$\beta$ ( $\alpha=0.2$ )		$\beta$ ( $\alpha=0.3$ )
Method	Params	0.2	0.5	0.2	0.5
Ours	400k	23.88	22.87	28.60	25.00
Ours	800k	24.54	21.58	30.61	27.63
FUTR	1M	24.55	22.04	29.88	25.87
Ours	1.2M	27.54	25.05	31.56	29.78
Ours	3M	27.55	25.67	32.06	30.11

Appendix

Table 9: Parameter efficiency on the Breakfast dataset across different model size variants, showing high performance even with

<1M

parameters. Counts are trainable parameters of the anticipation model; the frozen I3D backbone (shared across all baselines) and the swappable SG detector (see the main paper) are excluded as common input pipeline.

Figure 6: Qualitative anticipation results on 50 Salads with symbolic edits applied at inference. For each example we show the observed segment with two sample frames (left), and the ground-truth future (GT), our prediction (Pred.), and the prediction after editing the tomato nodes of the expanded joint-graph (Edited). (a) the tomato is occluded during the observation and missed by the detector, and activating the nodes recovers the cut tomato actions; (b) deactivating them removes cut tomato actions; (c) a case where the prediction persists despite deactivation, i.e. the visual stream overrides the symbolic prior.

Next-Action Prediction

Next-action prediction estimates what a person, process, or agent will do next from observations and prior events.

See all 35 papers

Neuro-Symbolic Hierarchical Intention Anticipation in Human Behavior
Sep 15, 2026Farnaz Soleimani, Abdelghani Chibani, Yacine Amirat +1Human Behavior Prediction Neuro-Symbolic Reasoning
LISSI
HiFOR: Hierarchical Foresight with Action-Guided World Grounding for Vision-Language-Action Model
Oct 29, 2026Rongxing Ding, Hongyu Qu, Kehong Wu +4Vision-Language-Action Models Temporal Modeling for VLA Models
Nanjing University of Science and Technology · National University of Singapore · Shenyang Institute of Automation, Chinese Academy of Sciences +1
FactCheck: Feasibility-aware Long-term Action Anticipation with Multi-agent Collaboration
Jun 10, 2026Rui Cao, Jiannong Cao, Bo Yuan +2Temporal Video Understanding
Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China · JIUTIAN Research, China Mobile, Beijing, China

Sep 15, 2026J/K move · Enter open · S save

Neuro-Symbolic Hierarchical Intention Anticipation in Human Behavior

Farnaz Soleimani, Abdelghani Chibani, Yacine Amirat, Ghazaleh Khodabandelou

LISSI

Human Behavior Prediction Neuro-Symbolic Reasoning Neuro-Symbolic Planning

Open paper AskPDF arXiv

Assistive autonomous systems must anticipate human goals before an observed behavior is complete. This article formulates anticipation as goal inference from a partially observed multimodal episode together with structured prediction of the remaining behavior, rather than exact motor forecasting. A compact Hierarchical Planning Decoder (HPD) is attached to a frozen neuro-symbolic recognition encoder and predicts, at four ontological levels, the next actions, the remaining activities and low-level intentions, and the episode high-level intention(HLI). The decoder is trained with soft neuro-symbolic regularization combining transition-coherence and hierarchical continuity losses, and is decoded with hard reachability masks that enforce ontological validity at inference. On a compositional four-level benchmark of 15,002 multimodal episodes built over NTU RGB+D 120 features, three headline properties are observed together. The advantage over the strongest sequential baseline grows with the anticipation horizon, from +1.7 points at step 1 to +7.3 points at step 3 (top-5). Under compositional generalization, where one parent association per multi-parent low level intention is held out, this advantage widens to +4.9 points at step 1. At the episode level, 96.8% of anticipated trajectories satisfy the joint logic constraints, above the 88.1% strongest-baseline value and the 73.9% ground-truth floor; soft logic terms alone account for a 59.8 to 71.1% relative reduction of HLI-reachability violations, and the hard masks then eliminate them entirely. Neural generation supplies predictive ranking, symbolic constraints supply onto logical validity, and their combination yields coherent hierarchical anticipation while exposing remaining challenges in compositional goal generalization and unordered set prediction.

Abstract

Next-Action Prediction

Explore similar work

Neuro-Symbolic Hierarchical Intention Anticipation in Human Behavior

HiFOR: Hierarchical Foresight with Action-Guided World Grounding for Vision-Language-Action Model

FactCheck: Feasibility-aware Long-term Action Anticipation with Multi-agent Collaboration