Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challenges

Mahsa Mohammadi; Zeyu Fu; Sareh Rowlands

cs.CVSep 15, 2026

Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challenges

Authors: Mahsa Mohammadi, Zeyu Fu, Sareh Rowlands

Organizations: Department of Computer Science, University of Exeter, Harrison Building, Streatham Campus, North Park Road, Exeter, EX4 4QF, Devon, United Kingdom

Abstract

Action anticipation predicts future human actions from partial video under incomplete context and temporal uncertainty. Recent systems introduce large language models (LLMs), vision-language models (VLMs), or language-derived semantics at different stages, but reported gains are difficult to interpret when task formulation, visual pretraining, supervision, decoder design, and evaluation code change simultaneously. The central contribution of this review is an evidence-aware design map that crosses task regime with the point at which language-derived information intervenes. We characterise task regimes along six axes. These axes organise the literature into five broad task families: single-action, sequence, object-interaction, cross-view, and planning-oriented settings. C1-C3 locate interventions in context construction, goal/intention modelling, and future decoding, while C4 is treated as an adjacent, emerging grounding/executability extension. Unlike a generic processing pipeline, the map links each intervention to an appropriate counterfactual, failure diagnosis, and permissible evidence claim. Supporting contributions include a protocol-level audit of Ego4D-LTA and EPIC-KITCHENS-100, a multidimensional evidence profile, and the Backbone-Aware Comparison and Ablation Protocol (BCAP). The unresolved EK-100 record is treated as a reporting-comparability case study and is not used as a leaderboard. Evidence for LLM benefits, goal ambiguity, and horizon effects is therefore formulated as testable hypotheses requiring matched validation, not as causal conclusions. The accompanying package contains the coded evidence, source locators, protocol metadata, and versioned catalogue used in the review.

Figures & tables

Survey	Ver.	LLM focused	Anticip. focused	Design centric	Protocol aware	LLM vs. backbone
Tang et al. [ 4 ]	v1, 2025	✓	no	$\circ$	no	no
Zhong et al. [ 5 ]	v2, 04/2026	$\circ$	✓	no	$\circ$	no
Kong & Fu [ 6 ]	2022	no	✓	no	$\circ$	no
Lai et al. [ 7 ]	v1, 2024	✓	✓	$\circ$	$\circ$	no
Hu et al. [ 8 ]	2022	no	$\circ$	no	$\circ$	no
Stergiou & Poppe [ 83 ]	2025	$\circ$	$\circ$	no	$\circ$	no

Table 1: Comparison with related reviews (✓ = yes;

\circ

= partial; no = not covered). “Design-centric” requires an explicit component-level analysis of where language enters anticipation; “protocol-aware” requires an audit of evaluation comparability, not benchmark coverage alone. Cells refer to the version listed in “Ver.” and are therefore version-specific. Source locations and coding rationales are provided in Supplementary Table S0.

Symbol	Meaning
$X_{1:t}$	Observed video prefix
$\tilde{X}$	Context-enriched representation
$Y_{t+1:T}$	Future action sequence (verb–noun pairs)
$G$	Latent goal or high-level intention
$S$	Grounding state (program/scene graph/trajectory)
$\mathcal{F}$	Feasible set of futures consistent with $S$

Table 2: Notation used throughout this survey.

Axis	Operational values	Examples	Required reporting	Interpretive consequence
Observation	Pre-action clip; recognised history; single frame; partial ongoing action; cross-view stream	EK-100; Ego4D-LTA; AAG; Ego4D-STA; DCPGN	Window length, FPS, observed labels, view and adaptation access	Fixes what evidence is available before any language component acts
Target	One label; action sequence; verb+noun+time/object; open-text step; executable plan	EK-100; Ego4D-LTA/STA; TrajPilot; Anticipate & Act	Target schema, sequence length, localisation or solvability requirement	Changes the output space and meaning of “correct”
Vocabulary	Closed; open; hybrid/retrieval-constrained	EK-100; Ego-Exo4D open vocabulary; prototype methods	Class inventory, unseen/rare-class policy, text-to-label mapping	Separates long-tail difficulty from genuine open-vocabulary generalisation
Horizon	Anticipation time $\tau_{a}$ ; discrete horizon $Z$ ; real-time rollout	EK-100; Ego4D-LTA; streaming/planning studies	$\tau_{a}$ or $Z$ , observation length $N$ , candidates $K$ , sampling policy	Prevents single-action and sequence protocols from being merged
Goal access	Provided; masked; inferred latent goal; semantic conditioning only; absent	Goal-conditioned planning; TrajPilot goal masking; AntGPT; PAR-VLA	Whether a goal is observed, inferred, or approximated by prototypes/text	Distinguishes latent-goal inference from semantic feature conditioning
Evaluation	MT5R; edit distance; mAP; calibration/coverage; solvability/success	EK-100; Ego4D-LTA/STA; Du et al.; planning boundary studies	Evaluator, split, version, subset and source locator	Metrics across task families are not interchangeable

Table 3: Operational task-regime taxonomy applied before C1–C4 coding. “Language-augmented” denotes an identifiable language- or VLM-derived signal; it does not imply an autoregressive LLM at inference.

Task family	C1 context	C2 goal/intention	C3 decoding	C4 adjacent grounding/planning
Single-action, closed vocabulary	Backbone, modalities, history	Optional semantic/intention conditioning	Class scoring	Usually not required
Long-term sequence forecasting	Observed-action/video history	Latent or prompted goal/subgoal	Autoregressive or structured sequence prediction	Feasibility/procedural constraints
Object-interaction anticipation	Actor/object context, localisation	Interaction intent or prototypes	Verb/noun/time/box prediction	Geometric or affordance consistency
Cross-view/test-time adaptation	View-normalised context and memory	Prototype/textual progression clues	Multi-label anticipation	Optional adaptation constraints
Planning/robotic boundary	Perceptual state and history	Explicit goal or subgoal	Plan/step generation	Solvability, dynamics, execution

Table 4: Task regime

\times

insertion-point matrix. Filled cells summarise common analytical roles, not requirements. C4 is an adjacent/emerging extension whose outputs may be constraints, plans, or feasibility judgements rather than benchmark action labels.

Panel of Figure 1 — Figure 1: An interpretive four-phase reading of the evolution of action anticipation; it is not an empirically estimated periodisation. Solid arrows indicate broad chronological progression, while dashed links indicate the dominant unresolved limitation that remained salient in each phase. The C1–C4 labels align these historical shifts with the functional components of the taxonomy in Section 4 .

Panel of Figure 2 — Figure 2: Hierarchical design-centric overview of language-augmented action anticipation. The centre identifies the task. Ring 1 (solid colours) shows the four functional components of the C1–C4 taxonomy: context construction (C1, teal), goal/intention modelling (C2, violet), future prediction and decoding (C3, orange), and an emerging grounding and executability extension (C4, red). Ring 2 (mid tones) enumerates the major design factors for each component (Tables 8 – 11 ). Ring 3 (light tones) lists selected illustrative methods; it is not the complete focal set. The authoritative list of all 25 focal methods is Table 6 .

Panel of Figure 3 — Figure 3: Generic pipeline of action anticipation. Top: four functional components (C1–C4): context construction produces $\tilde{X}$ ; goal/intention modelling infers $G$ ; future prediction and decoding generates $Y$ ; grounding and executability restrict decoding to the feasible set $\mathcal{F}$ , yielding $\hat{Y}_{t+1:T}$ . Bottom: E1 denotes evaluation and protocol comparability, discussed in Section 9 ; E2 and E3 denote deployment-oriented efficiency and streaming constraints, detailed in Section 10 . The V-JEPA family is included as an illustrative non-LLM control and as a case study in unresolved short-horizon reporting comparability; its published values do not establish a shared-code ranking against language-augmented systems.

Comp.	Intervention	Required counterfactual	Characteristic failure signature	Permissible evidence claim
C1	Context or representation change	Hold the decoder and goal pathway fixed; match training data and evaluation code where possible	Context noise, missing evidence, modality mismatch, or weak temporal representation	Representation-level contribution under the stated matched controls
C2	Goal, intention, or semantic-conditioning signal	Hold C1 and C3 fixed; remove, replace, or perturb the goal/intention signal	Goal hallucination, overconstraint, or collapse to an incorrect procedural branch	Goal-conditioning contribution, not a general LLM effect
C3	Objective, dependency model, or decoder change	Hold context and goal inputs fixed; match candidate space and decoding budget	Repetition, exposure drift, weak action dependencies, or poor calibration	Decoding-stability or sequence-modelling contribution
C4	Adjacent feasibility, grounding, or planning constraint	Use the same candidate generator without the check, constraint, or planner	Invalid, inconsistent, unsafe, or unreachable futures	Feasibility-sensitive contribution on an outcome that measures executability

Table 5: Central evidence-aware design map linking each component to its intervention, required counterfactual, characteristic failure signature, and permissible evidence claim.

Method	Core idea	C1	C2	C3	C4
Context construction (C1)
TransFusion [ 55 ]	Language summaries of past context	$\bullet$	$\circ$	—	—
PALM [ 29 ]	MMR-based context + prompt design	$\bullet$	—	$\circ$	—
SAFT [ 30 ]	Sequential textual correction memory	$\bullet$	—	$\circ$	—
M-CAT [ 31 ]	Dual-role text (past+future)	$\bullet$	—	$\circ$	—
AAG [ 86 ]	Single-frame RGB/depth + action-history semantics	$\bullet$	$\circ$	—	—

Table 6: Taxonomy of 25 focal entries: 16 core LLM/VLM-augmented methods, six non-LLM diagnostic/exemplar methods, and three planning/robotics boundary cases, distributed across four functional components. Each method is listed under its primary component (largest reported marginal gain per the authors’ own ablation, or the component most directly addressed by the stated contribution).

\bullet

= primary;

\circ

= secondary role. V-JEPA 2.1 is classified under C1 because its primary contribution is video self-supervised pretraining and it reports directly on EK-100 at

\tau_{a}{=}1

s under action MT5R [ 25 ] . InternVideo2 [ 28 ] is classified as a non-focal archival baseline because it reports on EK-100 under Action@1s rather than class-mean MT5R, a different metric not directly comparable to the V-JEPA rows. † SymAnt is an unpublished Master’s thesis / technical report rather than a peer-reviewed venue publication (Section 8 ). It is retained as a non-LLM neuro-symbolic C4 diagnostic exemplar: the evidence inventory establishes its scene- and knowledge-graph mechanism, but not an identifiable LLM/VLM component. Its inclusion is justified by design relevance, not by archival evidential weight, and its publication status should be read as non-archival throughout. C4 combines benchmark-grounded studies with planning-oriented boundary evidence and is therefore less mature than C1–C3; publication status and evidence dimensions are reported in Supplementary Tables S1 and S4. Deployment-oriented methods (E2/E3) appear in Table 7 .

Method	Core idea	E2	E3	Relation to anticipation
SparseVLM [ 51 ]	Text-aware visual token pruning	$\bullet$	—	Transfer mechanism only
PruneVid [ 53 ]	Static/dynamic token disentanglement	$\bullet$	—	Transfer mechanism only
VTS [ 52 ]	Key-frame saliency+novelty pruning	$\bullet$	—	Transfer mechanism only
AdaCM 2 [ 54 ]	Goal-aware rolling memory compression	$\bullet$	$\circ$	Transfer mechanism only
ReKV [ 50 ]	Sliding-window KV-cache retrieval	$\bullet$	$\bullet$	Transfer mechanism only
CLAM [ 78 ]	Cross linear attentive memory	$\bullet$	$\circ$	Direct anticipation evaluation

Table 7: Deployment-oriented methods (E2: efficiency; E3: streaming), reviewed in Section 10 . These cross-cutting methods are not assigned to C1–C4. “Direct” denotes an anticipation evaluation; “transfer” denotes a mechanism imported from efficient or streaming VideoLLM research without direct anticipation-performance evidence in the retained source.

Panel of Figure 4 — Figure 4: Evidence landscape of the 25 focal methods. Panel (c) shows the focal-set composition explicitly: 16 core LLM/VLM-augmented methods, six non-LLM diagnostic/exemplar methods, and three planning/robotics boundary cases. Counts use the retained primary assignment. Publication maturity and component isolation are coded as separate dimensions and are not collapsed into a single quality score. Panel (b) makes the lower maturity of C4 visible: its six focal entries comprise five preprints and one non-archival report, whereas C3 contains four archival studies. Machine-readable counts and row-level coding are supplied with the supplementary data.

Factor	Choices	Trade-off	Example
Modality	RGB vs. +flow/audio/HOI	Richer vs. heavier	M-CAT [ 31 ] , PALM [ 29 ]
Format	Labels / narrations / captions	Semantic vs. fast	TransFusion [ 55 ]
Scope	Window $N$ vs. full history	Rich vs. length limit	PALM ( $N$ ablation)
Retrieval	None / MMR / similarity	Diverse vs. costly	PALM (MMR ICL)
Spatial	Global / HOI hotspots	Precise vs. context loss	AFF-ttention! [ 44 ]
Text role	Past only / future target / both	Strong vs. $2\times$ cost	M-CAT [ 31 ]

Table 8: Design factors in C1: Context Construction . Trade-offs are grounded in cited ablation evidence. HOI = hand-object interaction; MMR = maximal marginal relevance.

Factor	Choices	Trade-off	Example
Goal representation	Latent state / textual intention / executable goal	Flexible vs. interpretable/executable	AntGPT, Ant.&Act
Inference pathway	Prompted LLM / VLM $\rightarrow$ text $\rightarrow$ LLM / explicit reasoning policy	Simplicity vs. visual grounding	AntGPT, ICVL, INSIGHT
Uncertainty handling	MAP goal / top- $K$ goals / posterior-conditioned decoding	Efficient vs. robust	AntGPT, Mascaró et al.
Coupling to decoder	Prompt-level conditioning / feature-level fusion / constrained decoding	Modular vs. tightly aligned	ICVL, AntGPT
Temporal granularity	Immediate next goal / multi-step subgoal / procedure-level objective	Local precision vs. long-horizon coherence	GP-AMS
Supervision source	Implicit anticipation loss / auxiliary intention supervision / structured task signal	Broad applicability vs. stronger bias	Mascaró et al., GP-AMS

Table 9: Design factors in C2: Goal/Intention Modelling . MAP = maximum a posteriori.

Factor	Choices	Trade-off	Example
Output space	Closed labels / open text / hybrid verbalisation	Unambiguous vs. expressive	AntGPT, PALM
Decoding policy	Greedy / beam / temperature-based sampling	Stable vs. diverse	AntGPT, PlausiVL
Sequence granularity	Stepwise autoregressive / multi-token prediction	Flexible vs. globally coherent	VideoPlan
Plausibility control	None / penalty / compatibility or counterfactual constraints	Simple vs. constrained	PlausiVL
Auxiliary training signal	Anticipation only / +auxiliary tasks / +consistency losses	Cleaner objective vs. richer supervision	VideoPlan, PlausiVL
Label normalisation	Native closed-set output / post-hoc text-to-label mapping	Direct evaluation vs. mapping noise	ActionLLM, PALM

Table 10: Design factors in C3: Future Prediction & Decoding . MTP = multi-token prediction; ED = edit distance.

Factor	Choices	Trade-off	Example
Grounding state	Symbolic program / scene graph / trajectory or latent dynamics	Interpretable vs. expressive	LEAP, SymAnt, HWM
Constraint source	Hand-crafted / learned / hybrid	Reliable vs. scalable	SymAnt, TR-LLM
Constraint timing	Post-hoc filtering / decoder-time conditioning / joint planning	Modular vs. tightly coupled	PlausiVL, LEAP, Ant.&Act
Planner type	None / symbolic planner / hierarchical MPC	Simplicity vs. stronger executability	Ant.&Act, HWM
Evaluation target	Standard anticipation metrics / logic or solvability checks / task success	Comparable vs. realistic	LEAP, Ant.&Act, HWM
Transfer burden	Domain-specific schemas / reusable priors / embodied adaptation	Strong control vs. portability	SymAnt, TR-LLM

Table 11: Design factors in C4: Grounding & Executability . KG = knowledge graph; MPC = model predictive control.

Method	Ver.	$N$	$Z$	$K$	Verb ED	Noun ED	Action ED	Primary-source location
Mascaró et al. [ 35 ]	v1	6	20	5	0.741	0.739	0.930	Table 1; test set
AntGPT [ 22 ]	v1	8	20	5	0.6584	0.6546	0.8814	Table 6; test set
PALM [ 29 ]	v1	8	20	5	0.6559	0.6401	0.8613	Table 1; test set
AntGPT [ 22 ]	v2	8	20	5	0.6503	0.6498	0.8770	Table 6; test set
PALM [ 29 ]	v2	8	20	5	0.6471	0.6117	0.8503	Table 1; test set

Table 12: Source-verified subset of reported Ego4D-LTA minimum edit-distance results. Values are transcribed from each method’s primary paper and are reported as separate verb, noun, and action ED at

Z{=}20

; lower is better. All rows use

K{=}5

and the test split, but v1 and v2 label spaces must still be interpreted separately. This table is intentionally selective rather than a complete leaderboard: a row is included only when version, observed context, horizon, candidate count, split, the full ED triple, and a primary-source table could be confirmed. The values were not independently reproduced.

Method	Year	Model / encoder config.	Family	Action MT5R	Publication status	Comparability
InAViT [ 26 ]	2024	160M	Non-LLM, supervised	25.8	Archival	Level (i)
Video-LLaMA [ 27 ]	2023	7B	LLM-augmented	26.0	Archival	Level (i)
PlausiVL [ 23 ]	2024	8B (Q-Former + LLM)	LLM-augmented	27.6	Archival	Level (i)
V-JEPA 2 [ 24 ]	2025	ViT-g384 (1B, frozen)	Non-LLM, dense SSL	39.7	Preprint †	Level (i)
V-JEPA 2.1 [ 25 ]	2026	ViT-G (2B, frozen)	Non-LLM, dense SSL	40.8	Preprint †	Level (i)
† Preprint at the time of writing; interpret as emerging rather than settled evidence.

Table 13: Published EK-100 figures under nominally aligned protocol descriptions. Values are reported as action Mean Top-5 Recall (class-mean) on the overall validation split at

\tau_{a}{=}1

s and are transcribed from the cited comparison sources. This is a reporting-incompatibility audit, not a verified leaderboard: the rows attain only Level (i), prior baselines were not stated to have been rerun under shared code, the survey did not re-execute them, and pretraining is unmatched. Full provenance and exclusions are given in Supplementary Table S2.

Benchmark	Metric	Horizon	Label space	Subset	Modality
Ego4D LTA v1	ED / seq-F1	✓	v1	—	✓
Ego4D LTA v2	ED / seq-F1	✓	v2	—	✓
EK-100 (official)	action MT5R	fixed	✓	✓	✓
EK-100 (@1s)	Action@1s	fixed	✓	✓	✓
EK-55	mAP	fixed	✓	✓	✓
Assembly101	task-specific	✓	✓	—	✓

Table 14: Benchmark compatibility matrix. ✓ marks a protocol condition that must be matched before two reported results can be compared; “fixed” indicates the condition is fixed by the benchmark definition. Subset refers to evaluation subsets such as overall validation versus unseen participants. Two uses must be distinguished. For leaderboard comparison, methods may differ in modality: using flow or object features is part of what is being compared. For causal attribution of a gain to one component, modality must be held fixed or explicitly controlled, since otherwise the modality budget confounds the component under test. The marks below state the stricter, causal requirement; a leaderboard reader may relax the modality column.

Panel of Figure 5 — Figure 5: Hypothesis-generating evaluation map with separate branches for single-action anticipation (parameterised by anticipation time $\tau_{a}$ ) and sequence anticipation (parameterised by horizon $Z$ and associated protocol axes). The map summarises where the reviewed evidence suggests testing C1, C2–C3, or C4; it is not a validated prospective selector. In particular, goal ambiguity is unmeasured in the underlying benchmarks and must be labelled as a heuristic unless estimated explicitly (Section 11.1 ).

Setting	Horizon	Ambiguity proxy	Constraint	Backbone	Evidence	Evidence profile	Source
LTA, goal-ambiguous	$Z\!\geq\!8$	High	Low	Moderate	AntGPT (Ego4D-v1 val): inferred-goal conditioning reduces verb/noun ED from 0.735/0.753 to 0.724/0.744; increasing $N_{\mathrm{seg}}$ from 1 to 8 reduces ED from 0.734/0.748 to 0.707/0.719	Archival; dedicated; direct	[ 22 ]
Procedural + constraints	Any	Med.	High	Any	LEAP: action-anticipation accuracy on EPIC-KITCHENS 14.64% $\to$ 16.98% (author ablation, not MT5R)	Preprint; partial; metric-mismatched	[ 39 ]
Rare/long-tail verbs	Any	High	Low	Any	M-CAT (EK-100 val): adding action/object text raises action Recall@5 from 18.4 to 23.7 overall and from 16.0 to 21.0 on tail classes (author ablation)	Preprint; dedicated; direct	[ 31 ]
Short-horizon, scaled backbone	$\tau_{a}{=}1$ s	Low	Low	High	Unresolved reporting incompatibility in this Level-(i) matrix: PlausiVL (8B) 27.6, V-JEPA 2 39.7, and V-JEPA 2.1 40.8 action MT5R on EK-100 val	Mixed maturity; cross-paper; Level (i)	[ 24 , 25 , 23 ]
Streaming, tight latency	Any	Any	Low	Any	STREAMMIND reports up to 100 FPS in its stated A100 setup; input resolution, batching, output regime, and end-to-end anticipation equivalence are not established here	Preprint; no anticipation isolation; transfer	[ 47 ]

Table 15: Conditions under which language-related components have shown gains, together with the unresolved short-horizon EK-100 reporting incompatibility. The incompatibility row is retained to audit comparability, not to rank model families. Evidence profiles list publication maturity, component isolation, and protocol relevance; they are not scalar quality grades. Rows are not matched causal comparisons unless explicitly stated. LEAP uses a non-MT5R metric, and the EK-100 row is source-described, unreproduced, and not pretraining-matched.

Problem regime	Primary component(s)
Short-horizon, closed vocabulary ( $\tau_{a}\approx 1$ s)	A scaled C1 backbone should be evaluated as a required diagnostic baseline before attributing gains to language augmentation
Long-horizon, goal-ambiguous ( $Z\geq 8$ )	Evaluate C2 + C3; goal ambiguity is currently a heuristic proxy
Safety- or executability-critical	Evaluate C4 grounding; evidence remains emerging and domain-specific
Real-time / edge deployment	E2 + E3: token pruning with event-gated streaming
Long-tail and compositional generalisation	C1 + C2: open-vocabulary context with structured goals
Calibration-critical applications	C3 with post-hoc calibration (BCAP Step 6)

Table 16: Practical design guide: which component to prioritise by problem regime. The table synthesises the evidence of Sections 5 – 10 and inherits their evidence-profile caveats.

Question	Evidence-supported answer	Confidence	Principal remaining gap
Q1: What components define the design space?	C1–C3 form the core analytical pipeline; C4 is a useful but less mature grounding/executability extension. The decomposition is non-unique and methods may span components.	Moderate–high	Primary assignment is non-unique and was not blindly recoded in full by an independent second coder; row-level coding and the alternate-assignment sensitivity analysis are supplied for audit.
Q2: Which design factors matter?	Context fidelity, goal conditioning, decoding stability, and feasibility constraints recur across methods. Their relative value depends on horizon, task structure, and deployment budget.	Moderate	Few studies isolate one factor while holding backbone, data, and compute fixed.
Q3: When do language components add value beyond a backbone?	Author-controlled long-horizon studies support some language-related components. For EK-100, published tables contain a large unresolved discrepancy under nominally similar labels, so the present record cannot support a cross-family performance ordering.	Low–moderate	No identified study performs a pretraining-matched, code-verified comparison across horizons.

Table 17: Direct answers to the three organising questions. Confidence reflects the maturity and comparability of the retained evidence, not a statistical posterior.

Claim	Principal studies	Groups	Shared evaluator	Controlled evidence	Confidence
Goal/intention conditioning can improve long-horizon anticipation	AntGPT, ICVL, INSIGHT, GP-AMS	4	No across papers	Dedicated or partial author ablations	Moderate
Semantic/action-history context can replace part of dense video in selected procedural settings	TransFusion, AAG/AAG+, PALM	3	No across papers	Modality/history ablations	Moderate, task-specific
A geometric intent signal can outperform textual conditioning in the studied regime	TrajPilot	1	Within study	Direct conditioning contrast	Moderate within study; unreplicated
Anti-repetition or structured decoding can improve sequence stability	PlausiVL, VideoPlan, AGA	3	No across papers	Heterogeneous author ablations	Low–moderate
Published EK-100 numbers are difficult to reconcile	PlausiVL, V-JEPA 2/2.1 and transcribed baselines	Multiple	No	No matched cross-family ablation	High for incompatibility; low for ranking
Grounding/verification may improve feasibility	LEAP, FactCheck, SymAnt and boundary cases	Multiple	No	Mainly within-study or non-benchmark evidence	Low/emerging

Table 18: Claim–evidence map for the main synthesis statements. “Shared evaluator” requires demonstrably identical released code, not merely the same metric name. Confidence is an interpretive evidence judgement, not a meta-analytic effect size.

BCAP item	AntGPT [ 22 ]	PlausiVL [ 23 ]	V-JEPA 2.1 [ 25 ]
Evaluation implementation/version	Partial: an official repository and LTA inference path were identified; commit and file locators are recorded in Supplementary Table S7B, but the survey row was not rerun.	No versioned public implementation was identified in this audit; the status is therefore based on the paper description only.	Partial: an official repository, EK-100 evaluation script, and model configuration were identified and frozen in Supplementary Table S7B; the reported row was not rerun.
Pretraining disclosure	Partial: constituent checkpoints are named; exposure is not harmonised with comparison systems.	Partial: backbone/checkpoint information is reported; data volume is not matched to the non-LLM encoders.	Substantial disclosure of model/data scaling, but it remains unmatched to the language-augmented systems.
Prompt or decoding reproducibility	Partial: prompting strategy and variants are described; a complete version-frozen run configuration is not available for every row.	Partial: plausibility and decoding mechanisms are described; complete cross-paper run settings are not standardised.	Not applicable to an LLM prompt; attentive-probe and evaluation configuration still require versioned reporting.
Repeated stochastic evaluation	No uncertainty interval from repeated LLM evaluations identified.	No BCAP-style repeated-run uncertainty interval identified.	No repeated-run interval attached to the reported EK-100 survey row.
Deployment cost	Not reported in the full BCAP hardware/memory/token-cost format.	Not reported in the full BCAP hardware/memory/token-cost format.	Compute is discussed for the model family, but not as a matched per-example BCAP deployment row.
Permitted inference	Descriptive system comparison; author ablations support selected within-system component claims.	Descriptive system comparison; author ablations support plausibility and repetition-control claims.	Descriptive evidence for a densely pretrained C1 system; no causal claim about removing an LLM from a matched architecture.

Table 19: Selected BCAP-Native dimensions for three representative systems. Statuses reflect the published record and public artefacts inspected for this review; no row was independently reproduced.

Explore similar work

Jun 4, 2026cs.CV

Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction

Video event prediction (VEP) requires models to infer unobserved future states from partial video evidence. Existing video MLLMs usually verbalize intermediate future reasoning in text space: once visual evidence is verbalized, fine-grained motion, geometry, and interaction cues can be lost, leading to plausible but visually ungrounded hallucinations. We introduce Future-L1, an interleaved latent visual reasoning framework that lets an MLLM alternate between language tokens and continuous latent visual spans during autoregressive decoding. To train this capability, we construct Future-L1-50K by selecting examples where future visual hints help prediction and align latent states to future-frame embeddings, then further optimize sampled latent trajectories with LA-DAPO, a latent-aware RL objective with outcome-contrastive and temporal-diversity rewards. Future-L1 achieves new state-of-the-art results on both benchmarks: on FutureBench, it improves Qwen3-VL-8B from 61.0 to 85.4 and exceeds the previous best Video-CoE by 10.4 points; on TwiFF-Bench, it improves the average score from 2.44 to 3.04. These results suggest that future-oriented video reasoning benefits from preserving intermediate visual semantics in latent space rather than translating every reasoning step into text.

Tianxiang Jiang, Linquan Wu, Sheng Xia +5

University of Science and Technology of China · Shanghai AI Laboratory · City University of Hong Kong +4

Future Video Prediction Latent Visual Reasoning Video Multimodal Large Language Models Imagination Intermediate Latent States

Ask PDF

May 20, 2026cs.CV

JFAA: Technical Report for the EPIC-KITCHENS-100 Action Anticipation Challenge at EgoVis 2026

We propose JFAA, a JEPA-based Future Action Anticipation method for the EPIC-KITCHENS-100 (EK-100) Action Anticipation task. Inspired by the representation learning and future prediction ability of V-JEPA 2.1, JFAA uses a frozen encoder and predictor to extract observed context features and near-future latent tokens. A lightweight attentive probe is then trained to predict verb, noun, and action logits with separate task queries. To improve robustness, we further build a field-aware ensemble over selected epoch-level predictions, allowing each output field to benefit from its most reliable candidates. Experimental results on the official challenge server show that JFAA achieves first place in the EgoVis 2026 EK-100 Action Anticipation Challenge. Our code will be released at https://github.com/CorrineQiu/JFAA.

Qiaohui Chu, Haoyu Zhang, Yisen Feng +4

Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory · Shandong Jianzhu University

Action Prediction Video Joint Embedding Predictive Architecture Compact Latent Sensor Tokens Technical Report

Ask PDF

May 19, 2026cs.CV

How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Prediction

Predicting how a person's first-person view will evolve (what action will follow, what plan completes a task, whether an in-progress shot will score) is fundamentally under-specified: the same context admits many plausible futures, and a model trained to minimize prediction error is forced to hedge or average across them, getting it wrong either way. Two findings shape our approach. First, the future camera trajectory, the path the head carves through space, lets the model commit to one of those futures: it carries the operator's intent in a form fine enough to determine how an action will unfold, substantially outperforming language as a conditioning signal. Second, this same intent makes the trajectory itself partially predictable from the context at hand, enough that trajectory need not be observed at test time to recover most of the gain. We instantiate these findings as TrajPilot, a model that predicts candidate future trajectories from egocentric context and uses them to pilot action prediction in an action-aligned embedding space where language shapes the structure but is never used as a conditioning input. TrajPilot beats VLM and structured-planner baselines on procedural planning across Ego-Exo4D atomic, Ego-Exo4D Keystep, Ego4D GoalStep, and EgoPER, with the trajectory advantage widening with horizon (exactly where prior planners collapse) and holding under RGB-only camera-pose estimation. With the goal masked at inference, the same model performs goal-free anticipation, beating VLM baselines on Ego-Exo4D atomic and extending to EPIC-Kitchens-100 and basketball shot-outcome prediction.

Sejoon Jun, Hai Nguyen-Truong, Luigi Seminara +1

Khoury College of Computer Sciences, Northeastern University, Boston · Korea Advanced Institute of Science and Technology, Daejeon · Department of Mathematics and Computer Science, University of Catania, Italy

Action Prediction Egocentric Vision

Ask PDF