LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos

Lei Shi; Victor Aregbede; Andreas Persson; Martin Längkvist; Amy Loutfi; Stephanie Lowry

cs.CVMar 10, 2026

LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos

Authors: Lei Shi, Victor Aregbede, Andreas Persson, Martin Längkvist, Amy Loutfi, Stephanie Lowry

Organizations: Örebro University, SE-701 82 Örebro, Sweden

Abstract

Procedure planning requires a model to predict a sequence of actions that transform a start visual observation into a goal in instructional videos. While most existing methods rely primarily on visual observations as input, they often struggle with the inherent ambiguity where different actions can appear visually similar. In this work, we argue that language descriptions offer a more distinctive representation in the latent space for procedure planning. We introduce Language-Aware Planning (LAP), a novel method that leverages the expressiveness of language to bridge visual observation and planning. LAP uses a finetuned Vision Language Model (VLM) to translate visual observations into text descriptions and to predict actions and extract text embeddings. These text embeddings are more distinctive than visual embeddings and are used in a diffusion model for planning action sequences. We evaluate LAP on three procedure planning benchmarks: CrossTask, Coin, and NIV. LAP achieves new state-of-the-art performance across multiple metrics and time horizons by large margin, demonstrating the significant advantage of language-aware planning.

Figures & tables

Figure 1: Orange boxes show the labels of actions. Blue boxes show the language descriptions of actions. Left : Visualisation of extracted features of visual observations in the latent space. Each point represents one video. Middle : Visualisation of text embeddings of actions in the latent space. Right : Start visual observation of “Fill filter with ground coffee” and “Even the surface of coffee”.

Figure 2: Overview of our proposed LAP. Professor Forcing : We finetune a VLM using professor forcing to predict the elaborated description $\hat{D}_{a_{i}}$ of an action $a_{i}$ . The elaborated description is obtained by prompting $a_{i}$ to a pretrained LLM. Video-to-Text : we use the finetuned VLM to generate descriptions for start and goal visual observation $V_{a_{s}}$ and $V_{a_{g}}$ and find out the descriptions that are most similar to $D_{a_{s}}$ and $D_{a_{g}}$ to represent the predicted action $\hat{a}_{s}$ and $\hat{a}_{g}$ . Then the extractor extracts the corresponding embeddings $E_{\hat{a}_{s}}$ and $E_{\hat{a}_{g}}$ . Planning : We use a diffusion model to plan the action sequence. We use the embeddings of start and end embeddings $E_{\hat{a}_{s}}$ and $E_{\hat{a}_{g}}$ as well as action sequence to construct the input to the diffusion model. During training, only the action dimension is noised.

Figure 3: VLM finetuning: we use professor forcing and scheduled sampling to finetune the VLM. During the forward pass, the VLM either use teaching forcing or free running to generate tokens with a scheduled probability. A discriminator is used to minimise the distribution distances between them so that the generated tokens are as close to ground truth tokens.

	T=3			T=4			T=5	T=6
	SR $\uparrow$	mAcc $\uparrow$	mSIoU $\uparrow$	SR $\uparrow$	mAcc $\uparrow$	mSIoU $\uparrow$	SR $\uparrow$	SR $\uparrow$
PDPP [ 37 ] (CVPR 2023)	33.9	61.39	/	20.77	56.57	/	12.72	8.07
SkipPlan [ 18 ] (ICCV 2023)	27.6	59.9	/	14.06	54.64	/	6.51	5.24
ActionDiffusion [ 32 ] (WACV 2025)	33.25	61.47	65.09	20.01	56.74	63.98	12.48	8.07
KEPP [ 26 ] (CVPR 2024)	33.27	60.16	/	19.41	54.45	/	12.01	7.89
SCHEMA [ 27 ] (ICLR 2024)	32.34	57.3	/	22.51	51.47	/	12.89	7.54

Table 1: Comparisons with baseline methods on CrossTask dataset with time horizon from

T=3

T=6

\uparrow

means higher is better. The best results are shown in bold . The second best numbers are shown with underline . All baselines and our method use KEPP setting for data curation.

	T=3			T=4
	SR↑	mAcc ↑	mSIoU↑	SR↑	mAcc ↑	mSIoU↑
PDPP [ 37 ] (CVPR 2023)	21.33	45.62	51.82	13.67	42.58	/
SkipPlan [ 18 ] (ICCV 2023)	23.65	47.12	/	16.04	43.19	/
KEPP [ 26 ] (CVPR 2024)	20.25	39.87	/	15.63	39.53	/
SCHEMA [ 27 ] (ICLR 2024)	32.09	49.84	/	22.02	45.33	/
PlanLLM $\dagger$ [ 41 ] (AAAI 2025)	33.22	54.33	/	25.31	48.79	/

Table 2: Comparisons with baseline methods on Coin dataset with time horizon

T=3

and

T=4

\uparrow

means higher is better. The best results are shown in bold . The second best numbers are shown with underline . All baselines and our method use KEPP setting for data curation.

\dagger

indicates the model uses intermediate visual observations.

	T=3			T=4
	SR $\uparrow$	mAcc $\uparrow$	mSIoU $\uparrow$	SR $\uparrow$	mAcc $\uparrow$	mSIoU $\uparrow$
PDPP [ 37 ] (CVPR 2023)	22.22	39.50	/	21.30	39.24	/
KEPP [ 26 ] (CVPR 2024)	24.44	43.46	/	22.71	41.59	/
SCHEMA [ 27 ] (ICLR 2024)	27.93	41.64	/	23.26	39.93	/
PlanLLM $\dagger$ [ 41 ] (AAAI 2025)	26.74	42.97	/	27.08	46.96	/
MTID [ 48 ] (ICLR 2025)	28.52	44.44	56.46	24.89	44.54	57.46

Table 3: Comparisons with baseline methods on NIV dataset with time horizon

T=3

and

T=4

\uparrow

means higher is better. The best results are shown in bold . The second best numbers are shown with underline . All baselines and our method use KEPP setting for data curation.

\dagger

indicates the model uses intermediate visual observations.

	CrossTask				NIV
	T=3		T=4		T=3		T=4
	start	goal	start	goal	start	goal	start	goal
LAP w/o le	63.89	67.61	64	68.3	55.02	52.42	58.85	55.44
LAP	75.72	79.69	75.77	80.47	67.29	70.63	66.81	68.58

Table 4: Action prediction ablations on language enhancement.

		T=3			T=4
		SR $\uparrow$	mAcc $\uparrow$	mSIoU $\uparrow$	SR $\uparrow$	mAcc $\uparrow$	mSIoU $\uparrow$
CrossTask	LAP-vo	32.03	60.16	63.34	19.78	55.81	63.55
	LAP-tf	37.82	68.23	64.52	21.57	60.85	59.45
	LAP	41.14	70.13	65.98	23.91	63.70	62.56
Coin	LAP-vo	17.54	38.39	44.83	13.2	36.88	45.03
	LAP-tf	43.11	66.49	65.88	28.9	60.55	65.18

Table 5: Ablations on professor forcing and using text or visual observation for procedure planning on all datasets with

T=3

and

T=4

. LAP-tf indicates training with teaching forcing. LAP-vo indicates using visual feature. The best numbers are in bold .

Panel of Figure 4 — Figure 4: Visualisation of features of visual observation and text embedding in latent space on CrossTask, Coin and NIV dataset. The first row shows the visual observation and the second row shows the text embedding. All visualisations are the samples from test sets with time horizon $T=3$ .

	T=3			T=4
	SR $\uparrow$	mAcc $\uparrow$	mSIoU $\uparrow$	SR $\uparrow$	mAcc $\uparrow$	mSIoU $\uparrow$
LLaVa-NeXT-Video	21.9	36.68	37.95	13.27	36.28	37.56
LLaVa-NeXT-Video-Text	17.84	41.76	46.64	13.72	41.15	47.89
LAP	56.51	72.86	72.22	43.81	68.03	70.17
LAP-Text	43.87	66.55	66.16	34.07	61.84	66.49

Table 6: Ablation study of different VLMs on NIV dataset. We compare the VLM used in LAP and LLaVa-NeXT-Video 7B. Bold numbers indicate the best results.

Figure 5: Qualitative results on NIV dataset. Purple boxes indicate the ground truth actions. Green boxes indicate the correctly predicted actions in action sequences. Red boxes indicate the wrong predictions. Top: predictions when $\hat{a}_{s}$ and $\hat{a}_{g}$ are “unknown” from VLM. Middle: predictions when $\hat{a}_{s}$ is “unknown” and $\hat{a}_{g}$ is correct. Bottom: predictions when $\hat{a}_{s}$ is correct and $\hat{a}_{g}$ is “unknown”.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: An example of the prompt we use to obtain language descriptions of actions.

		SR	mAcc $\uparrow$	mSIoU $\uparrow$
T=3	LAP	41.64	55.14	54.61
	LAP-Text	11.52	28.38	37.52
	LAP $\dagger$	56.51	72.86	72.22
	LAP-Text $\dagger$	43.87	65.55	66.16
T=4	LAP	30.1	52.1	54.53
	LAP-Text	7.08	27.32	38.95

Appendix

Table 7: Ablation study of language enhancement on NIV dataset. We train the diffusion model of LAP directly with or without language enhancement.

\dagger

shows the results with language enhancement. Bold numbers indicate the best numbers.

		ROUGE-1	ROUGE-2
T=3	LAP VLM	0.16	0.01
	LAP VLM $\dagger$	0.71	0.68
T=4	LAP VLM	0.17	0.01
	LAP VLM $\dagger$	0.7	0.68

Appendix

Table 8: Evaluation of generated text on NIV dataset.

\dagger

indicates the VLM of LAP is trained with enhanced language description. Best numbers are shown in bold .

		T=3			T=4
		SR↑	mAcc ↑	mSIoU↑	SR↑	mAcc ↑	mSIoU↑
CrossTask	LLaVa-NeXT-Video	17.72	46.92	43.48	9.73	43.19	40.64
	LLaVa-NeXT-Video-Text	17.15	47.09	46.73	10.08	44.32	46.8
	LAP	41.14	70.13	65.98	23.91	63.7	62.56
	LAP-Text	35.84	65.65	62.48	21.14	59.64	59.99
Coin	LLaVa-NeXT-Video	29.6	46.54	46.07	20.95	43.39	47.08

Appendix

Table 9: Comparison with LLaVA-Next-Video 7B on CrossTask and Coin dataset with time horizon

T=3

and

T=4

. Best numbers are shown in bold .

Panel of Figure 7 — Figure 7: ROUGE-1 score distribution of LAP on CrossTask, Coin and NIV dataset.

		SR $\uparrow$	mAcc $\uparrow$	mSIoU $\uparrow$
CrossTask	KEPP [ 26 ] (CVPR 2024)	33.27	60.16	/
	LAP (TH=0.5)	41.14	70.13	65.98
	LAP (TH=0.9)	37.01	67.14	63.02
	LAP-Text	35.84	65.65	62.48
Coin	PlanLLM [ 41 ] (AAAI 2025)	33.22	54.33	/
	LAP (TH=0.5)	44.43	68.08	66.71

Appendix

Table 10: Results of using a different threshold for LAP on all datasets with time horizon

T=3

. Bold numbers indicate the best results.

		T=3			T=4
		SR $\uparrow$	mAcc $\uparrow$	mSIoU $\uparrow$	SR $\uparrow$	mAcc $\uparrow$	mSIoU $\uparrow$
CrossTask	LAP $\dagger$	39.13	67.73	62.26	23.63	60.81	58.65
CrossTask	LAP	41.14	70.13	65.98	23.91	63.70	62.56
Coin	LAP $\dagger$	41.16	63.70	60.78	29.12	59.48	61.43
Coin	LAP	44.43	68.08	66.71	31.56	63.77	68.24
NIV	LAP $\dagger$	46.10	65.92	64.32	36.28	63.83	60.96

Appendix

Table 11: Comparison of “Unknown” label and random label when the generated descriptions of start or goal action have lower ROUGE-1 score than the threshold. Bold numbers are the best.

\dagger

indicates LAP with random label.

Inference Time (Seconds)		No. Trainable Parameters (M)		Throughput (Seconds)
VLM	Diffusion	VLM	Diffusion
1.94	0.006	497.5	40.4	2.3

Appendix

Table 12: Number of parameters, inference time and throughput.

RECIPE: Procedural Planning via Grounding in Instructional Video
May 19, 2026Luigi Seminara, Antonino Furnari, Lorenzo TorresaniVideo Understanding
Khoury College of Computer Sciences, Northeastern University, Boston · Department of Mathematics and Computer Science, University of Catania, Italy
ViterbiPlanNet: Injecting Procedural Knowledge via Differentiable Viterbi for Planning in Instructional Videos
Mar 4, 2026Luigi Seminara, Davide Moltisanti, Antonino FurnariGraph-Based Planning Temporal Video Understanding
University of Catania · University of Bath
Contrastive Energy Fields for Inference-Time Procedure Planning in Instructional Videos
Aug 17, 2026Mohamed Afham, Christoph Reich, Oliver Hahn +2Video Reasoning Inference-Time Optimization
TU Darmstadt · ELIZA · TU Munich +1

May 19, 2026J/K move · Enter open · S save

RECIPE: Procedural Planning via Grounding in Instructional Video

Luigi Seminara, Antonino Furnari, Lorenzo Torresani

Khoury College of Computer Sciences, Northeastern University, Boston · Department of Mathematics and Computer Science, University of Catania, Italy

Video Understanding

Open paper AskPDF arXiv

Visual planning asks a model to generate the remaining steps of a procedure in natural language given a partial video context and a goal. Progress on this task is bottlenecked by annotation: clean labeled datasets are small, domain-narrow, and encode a single execution trajectory per example, even though many valid orderings exist. Large-scale instructional video corpora offer orders of magnitude more procedural content, but supervised fine-tuning on pseudo-labels from their noisy ASR narrations propagates segmentation and alignment errors and stays single-trajectory. We identify a key asymmetry: extracting clean step labels from noisy video is hard, but verifying whether a generated step sequence is temporally grounded in ASR transcripts is cheap and scales to millions of videos via precomputed text embeddings. We exploit this asymmetry in RECIPE, which uses grounding quality as a reward for GRPO, turning the noisy corpus into a verifier rather than a label source. The framework applies uniformly to two planner input configurations (Socratic, with a textual history extracted by a frozen VLM, and Video, consuming video tokens directly) and to annotated and weakly supervised regimes. We evaluate on 7 procedural benchmarks using a reference-based LLM-as-judge protocol scoring plans across 6 procedural criteria. RECIPE-RL improves over the base checkpoint at all scales (0.5B, 3B, 7B) and every benchmark, with macro-accuracy gains of +7 to +8 points in-domain and up to +16 points zero-shot. It outperforms supervised fine-tuning on both annotated and pseudo-labeled plans (the latter degrades the base) and remains robust without human annotations. Used as the proposal stage of a prior propose-assess-search planner, it improves over the strongest zero-shot baseline at every horizon on Visual Planning for Assistance, and on COIN it preserves the generation diversity that SFT collapses.

Abstract

Explore similar work

RECIPE: Procedural Planning via Grounding in Instructional Video

ViterbiPlanNet: Injecting Procedural Knowledge via Differentiable Viterbi for Planning in Instructional Videos

Contrastive Energy Fields for Inference-Time Procedure Planning in Instructional Videos