Contrastive Energy Fields for Inference-Time Procedure Planning in Instructional Videos

Mohamed Afham; Christoph Reich; Oliver Hahn; Daniel Cremers; Stefan Roth

cs.CVAug 17, 2026

Contrastive Energy Fields for Inference-Time Procedure Planning in Instructional Videos

Authors: Mohamed Afham, Christoph Reich, Oliver Hahn, Daniel Cremers, Stefan Roth

Organizations: TU Darmstadt · ELIZA · TU Munich · MCML

Abstract

Procedure planning seeks to estimate a sequence of actions to transition from an observed initial state to a given goal state. Current procedure planning approaches directly predict action sequences from latent representations using feed-forward neural networks or diffusion-based inference. These paradigms treat every action as plausible, lacking the ability to enforce task-specific logical constraints that render certain actions irrelevant or not plausible. We propose CEFITO, a procedure planning approach that learns a predictor to express an action-conditioned representation space. Based on this representation space, we formulate procedure planning as a task-constrained optimization problem. Unlike prior methods, CEFITO explicitly reasons over the action space by omitting irrelevant actions during inference-time planning. This reformulation enables effective procedure planning and achieves state-of-the-art accuracy on two established procedure planning benchmarks.

Figures & tables

Figure 1: Overview of CEFITO. Left: Given an initial observation and a sequence of actions, our method maps action sequences to a predicted goal latent. The distance to the encoded goal latent defines the candidate’s energy, which we minimize at inference. Right: We learn a task-conditioned energy field via contrastive learning. At inference, CEFITO predicts the task ( e.g. , “Grill steak”), restricting the search to an action subset (pink region). The action sequence with minimal energy is the predicted plan.

Figure 2: CEFITO training framework. During training, a positive and $N$ negative (wrong) action sequences are provided. Each sequence is tokenized using a text encoder and fed into the predictor $P_{\theta}$ , alongside the initial latent state $x_{s}$ . For each sequence we predict a goal state $\tilde{x}_{g}$ , while predictions of negative sequences $\tilde{x}_{g,i}^{-}$ are pushed away from the target $\tilde{x}_{g}$ and for positive sequences $\tilde{x}_{g}^{+}$ pulled together. An auxiliary loss $\mathcal{L}_{\text{aux}}$ supervises auxiliary latents of $P_{\theta}$ via a decoding head $D_{\phi}$ .

Figure 3: CEFITO inference. A task classifier predicts $\hat{c}$ from $(x_{s},x_{g})$ and restricts the search to $\mathcal{A}(\hat{c})$ . The predictor $P_{\theta}$ scores every candidate sequence in $\mathcal{A}(\hat{c})$ . The sequence approximating $x_{g}$ best, forms the action sequence prediction $\tilde{a}_{1:\rm T}$ .

Method	T=3			T=4
Method	SR $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	SR $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$
WLTDO [ 17 ]	1.87	21.64	31.70	0.77	17.92	26.43
UAAA [ 18 ]	2.15	20.21	30.87	0.98	19.86	27.09
UPN [ 50 ]	2.89	24.39	31.56	1.19	21.59	27.85
DDN [ 12 ]	12.18	31.29	47.48	5.97	27.10	48.46
PlaTe [ 51 ]	16.00	36.17	65.91	14.00	35.29	55.36

Table 1: Results on CrossTask [ 65 ] . We compare CEFITO with the state of the art on CrossTask val., using different planning horizons, and report SR, mACC, and mIoU (all in %,

\uparrow

). Best results highlighted in red

\blacksquare\!\!\!\!\!\blacksquare

; second-best results in orange

\blacksquare\!\!\!\!\!\blacksquare

. For completeness, we report baselines not adhering to the protocol by [ 49 ] in gray

\blacksquare\!\!\!\!\!\blacksquare

Method	T=3			T=4
Method	SR $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	SR $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$
DDN [ 12 ]	13.90	20.19	64.78	11.13	17.71	68.06
P 3 IV [ 62 ]	15.40	21.67	76.31	11.32	18.85	70.53
EGPP [ 54 ]	19.57	31.42	84.95	13.59	26.72	84.72
Qwen2.5-VL-32B [ 3 ]	$3.65$	$17.51$	$52.10$	$1.87$	$17.05$	$55.66$
Qwen2.5-32B [ 60 ]	$14.97$	$36.34$	$78.74$	$4.98$	$27.45$	$71.64$

Table 2: Results on COIN [ 52 ] . We compare CEFITO with the state of the art on COIN val., using different planning horizons, and report SR, mACC, and mIoU (all in %,

\uparrow

). Best results are highlighted in red

\blacksquare\!\!\!\!\!\blacksquare

and second-best results in orange

\blacksquare\!\!\!\!\!\blacksquare

. For completeness, we report baselines not adhering to the protocol by [ 49 ] in gray

\blacksquare\!\!\!\!\!\blacksquare

Configuration	T=3			T=4
Configuration	SR $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	SR $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$
ViterbiPlanNet [ 49 ]	$38.45$	$63.07$	$83.89$	$24.64$	$57.00$	$81.18$
Baseline	$13.65$	$47.82$	$69.53$	$9.24$	$35.44$	$67.76$
+ triplet contrastive loss	$37.58$	$63.21$	$83.67$	$23.54$	$56.18$	$80.91$
+ adaptive margin	38.74	63.79	84.02	24.23	57.01	81.22
+ auxiliary regualrization (CEFITO)	39.62	64.12	84.29	24.76	57.53	81.58

Table 3: CEFITO training analysis. Component-wise analysis of CEFITO on CrossTask val. and report SR, mACC & mIoU (all in %,

\uparrow

). We add components to the previous row until achieving CEFITO. For reference, we include the recent state-of-the-art ViterbiPlanNet. Best results in red

\blacksquare\!\!\!\!\!\blacksquare

; second-best results in orange

\blacksquare\!\!\!\!\!\blacksquare

Text Encoder	T=3			T=4
Text Encoder	SR $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	SR $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$
Random embedding	$35.10$	$61.20$	$82.50$	$21.30$	$54.10$	$79.80$
Flan-T5-base [ 14 ]	37.80	63.00	83.60	23.10	56.20	80.90
CLIP [ 46 ]	39.62	64.12	84.29	24.76	57.53	81.58

Table 4: Text encoder analysis. We analyze the choice of text encoder for CEFITO on CrossTask val. and report SR, mACC, and mIoU (all in %,

\uparrow

). Best results are highlighted in red

\blacksquare\!\!\!\!\!\blacksquare

and second-best results in orange

\blacksquare\!\!\!\!\!\blacksquare

Panel of Figure 4 — Figure 4: Negative sampling analysis . We analyze the impact of the number of negative samples $N$ (left) and the hard/easy negative sample ratio $r$ (right) on CrossTask with $\rm T=\text{$ 3 $}$ using SR (in %, $\uparrow$ ). For $r=\text{$ 0 $}$ , only easy negatives are used. Vice versa , $r=\text{$ 1 $}$ generates only hard negatives. We indicate our default values in red $\blacksquare\!\!\!\!\!\blacksquare$ .

Table 9

Panel of Figure 5 — Figure 5: Inference runtime results . Left: We report inference runtime (in ms, $\downarrow$ ) over different planning horizons for inferring a single action sequence on the CrossTask dataset [ 65 ] . All runtimes are reported using the same hardware ( single A100 $80\text{\,}\mathrm{G}\mathrm{B}$ GPU). Right: We report the average number of forward passes through our predictor $P_{\theta}$ required for inference. While CEFITO (in $\blacksquare\!\!\!\!\!\blacksquare$ ) provides a comparable inference runtime for $\rm T\leq 4$ to diffusion (PDPP [ 55 ] in $\blacksquare\!\!\!\!\!\blacksquare$ ) and feed-forward (ViterbiPlanNet [ 49 ] in $\blacksquare\!\!\!\!\!\blacksquare$ ) approaches, the runtime is significantly worse for $\rm T\geq 5$ . For reference, we also report CEFITO without constraining the search space, indicated using $\star$ .

Method	T=3			T=4
Method	SR $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	SR $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$
Qwen2.5-VL-32B [ 3 ]	$7.41$	$27.65$	$59.73$	$5.26$	$28.84$	$60.21$
Qwen2.5-32B [ 60 ]	$24.07$	$43.46$	$71.88$	$23.25$	$41.89$	$73.91$
Gemini 2.5 Pro [ 21 ]	$24.07$	$43.46$	$71.86$	$22.37$	$40.35$	$73.05$
Qwen3-30B [ 59 ]	$24.81$	$42.84$	$70.80$	$22.37$	$41.23$	$73.90$
Qwen3-30B + PKG [ 49 ]	$25.19$	$43.95$	$71.98$	$21.93$	$41.67$	$74.43$

Table 7: Results on NIV [ 1 ] . We compare CEFITO with the state-of-the-art on NIV val., using different planning horizons, and report SR, mACC, and mIoU (all in %,

\uparrow

). Best results are highlighted in red

\blacksquare\!\!\!\!\!\blacksquare

and second-best results in orange

\blacksquare\!\!\!\!\!\blacksquare

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: Auxiliary action sequence reconstruction. During training, the predictor $P_{\theta}$ is applied twice with shared weights: first , given $x_{s}$ , action tokens $a_{1:4}$ , and masked state tokens $x_{1:4}^{\text{mask}}$ , $P_{\theta}$ predicts the intermediate and goal latents $\tilde{x}_{1:3},\tilde{x}_{g}$ , respectivally. Second , given $x_{s}$ , $x_{g}$ , $P_{\theta}$ predicts the masked action tokens. These are supervised by the ground-truth action sequence $a_{1:4}$ using the auxiliary loss $\mathcal{L}_{\text{aux}}$ .

Figure A.2: Qualitative examples of CEFITO on CrossTask. Procedural video results where the top-ranked candidate plan (highlighted in $\blacksquare\!\!\!\!\!\blacksquare$ ) matches the ground-truth action sequence, retrieved from the top-5 candidates ranked by L2 distance to the goal embedding $x_{g}$ .

Figure A.3: Qualitative examples of CEFITO on CrossTask. Extension of Fig. A.2 .

Figure A.4: Failure cases of CEFITO on CrossTask. Procedural video results where the top-ranked candidate plan (highlighted in $\blacksquare\!\!\!\!\!\blacksquare$ ) fails to match the ground-truth action sequence (highlighted in $\blacksquare\!\!\!\!\!\blacksquare$ ). For some examples, the ground truth falls outside of the top- $5$ candidates.

Figure A.5: Failure cases of CEFITO on CrossTask. Extension of Fig. A.4 .

Video Reasoning

Video reasoning studies how AI systems use video evidence to infer events, actions, temporal relations, causes, and future states.

See all 210 papers

LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos
Mar 10, 2026Lei Shi, Victor Aregbede, Andreas Persson +3
Örebro University, SE-701 82 Örebro, Sweden
ViterbiPlanNet: Injecting Procedural Knowledge via Differentiable Viterbi for Planning in Instructional Videos
Mar 4, 2026Luigi Seminara, Davide Moltisanti, Antonino FurnariGraph-Based Planning Temporal Video Understanding
University of Catania · University of Bath
RECIPE: Procedural Planning via Grounding in Instructional Video
May 19, 2026Luigi Seminara, Antonino Furnari, Lorenzo TorresaniVideo Understanding
Khoury College of Computer Sciences, Northeastern University, Boston · Department of Mathematics and Computer Science, University of Catania, Italy

Mar 10, 2026J/K move · Enter open · S save

LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos

Lei Shi, Victor Aregbede, Andreas Persson, Martin Längkvist+2

Örebro University, SE-701 82 Örebro, Sweden

Open paper AskPDF arXiv

Procedure planning requires a model to predict a sequence of actions that transform a start visual observation into a goal in instructional videos. While most existing methods rely primarily on visual observations as input, they often struggle with the inherent ambiguity where different actions can appear visually similar. In this work, we argue that language descriptions offer a more distinctive representation in the latent space for procedure planning. We introduce Language-Aware Planning (LAP), a novel method that leverages the expressiveness of language to bridge visual observation and planning. LAP uses a finetuned Vision Language Model (VLM) to translate visual observations into text descriptions and to predict actions and extract text embeddings. These text embeddings are more distinctive than visual embeddings and are used in a diffusion model for planning action sequences. We evaluate LAP on three procedure planning benchmarks: CrossTask, Coin, and NIV. LAP achieves new state-of-the-art performance across multiple metrics and time horizons by large margin, demonstrating the significant advantage of language-aware planning.

Abstract

Video Reasoning

Explore similar work

LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos

ViterbiPlanNet: Injecting Procedural Knowledge via Differentiable Viterbi for Planning in Instructional Videos

RECIPE: Procedural Planning via Grounding in Instructional Video