VIGiA: Instructional Video Guidance via Dialogue Reasoning and Retrieval

Diogo Glória-Silva; David Semedo; João Maglhães

cs.CVFeb 22, 2026

VIGiA: Instructional Video Guidance via Dialogue Reasoning and Retrieval

Authors: Diogo Glória-Silva, David Semedo, João Maglhães

Organizations: NOVA LINCS, NOVA School of Science and Technology, Portugal

Abstract

We introduce VIGiA, a novel multimodal dialogue model designed to understand and reason over complex, multi-step instructional video action plans. Unlike prior work which focuses mainly on text-only guidance, or treats vision and language in isolation, VIGiA supports grounded, plan-aware dialogue that requires reasoning over visual inputs, instructional plans, and interleaved user interactions. To this end, VIGiA incorporates two key capabilities: (1) multimodal plan reasoning, enabling the model to align uni- and multimodal queries with the current task plan and respond accurately; and (2) plan-based retrieval, allowing it to retrieve relevant plan steps in either textual or visual representations. Experiments were done on a novel dataset with rich Instructional Video Dialogues aligned with Cooking and DIY plans. Our evaluation shows that VIGiA outperforms existing state-of-the-art models on all tasks in a conversational plan guidance setting, reaching over 90% accuracy on plan-aware VQA.

Figures & tables

Figure 1: VIGiA is grounded on complex multimodal instructional plans, delivering unified multimodal alignment over dialogue turns by providing text-based guidance (Goal 1), plan-aware visual question answering (Goal 2) aligning plan actions and visual context (Goal 3), and retrieving relevant video moments (Goal 4).

Panel of Figure 2 — Figure 2: VIGiA is an LVLM model that processes instructional video plans to navigate through steps of the plan, perform QA, VQA and retrieval of arbitrarily random text steps or video moment steps.

Figure 3: Global view of VIGiA’s architecture. To handle multimodal inputs VIGiA combines a visual encoder and an LLM using an MLP as a connector module. For conversational video moment retrieval, VIGiA outputs a dedicated video moment start and end representation that can be used for start and end frame retrieval.

	VIGiA	MM-PlanLLM
VQA support	✓	$\times$
Vision Tokens	257 tokens	1 token
CVMR Retrieval	Start & End Frame	Mid Frame
Multiple Domains	✓	$\times$

Table 1: Direct comparison of VIGiA with current procedural plan guidance SotA MM-PlanLLM Glória-Silva et al. (2024b)

	Plan Reasoning					Plan-based Retrieval
	PGAG			pVQA		VSG
Model	ROUGE-L	METEOR	BERTScore	ROUGE-L	Accuracy*	ROUGE-L	EM
Idefics2	37.16	45.09	67.55	21.22	44.49	20.16	5.85
LLaVA-1.5	42.47	40.84	70.55	26.77	64.11	16.07	4.21
Qwen 2.5 VL	31.61	41.51	63.91	26.39	75.14	16.65	9.47
LLaVA-OV	41.18	40.85	70.70	27.02	77.38	32.28	17.89

Table 2: Results on the in-domain tasks: PGAG, pVQA, and VSG. Underlined scores are the best among the baselines. *Accuracy is measured using the majority vote of three independent LLMs (See Section 5 ).

	Model	CVMR
		Recall@1		Recall@5
		m=0.5	m=0.7	m=0.5	m=0.7
firm	TRACE	5.97	1.63	—	—
firm	VIGiA	10.49	2.89	30.38	11.75
adj.	MM-PlanLLM	25.32	12.48	31.46	16.46
adj.	VIGiA	30.74	15.37	51.18	26.94

Table 3: Results on the CVMR task.

Figure 4: Comparing R@1 performance with varying values for the similarity threshold on InstructionVidDial’s dev set.

		Plan Reasoning				Plan-based Retrieval				General
		PGAG		pVQA		VSG		CVMR R@1		MME
		R-L	METEOR	R-L	Acc.	R-L	EM	m=0.5	m=0.7	Cog.	Per.
Stages	4 Only	74.75	76.05	33.56	88.79	54.27	56.84	11.39	5.97	234	503
	1+4	74.67	76.61	33.89	90.47	54.17	56.32	20.43	11.39	211	501
	1+2+4	74.99	76.39	34.20	91.03	54.12	56.84	9.58	5.06	223	708
	1+2+3+4	75.30	76.67	33.65	94.02	55.66	57.37	30.74	15.37	229	717

Table 4: Ablation of model training stages. Underlined scores represent improvements over the previous stage/row.

Model	Stage 2		Stage 4 / Final
Model	Cog.	Per.	Cog.	Per.
MM-PlanLLM	—	—	34	1
Idefics2	—	—	328	1474
LLaVA-1.5	—	—	314	1483
LLaVA-OV	—	—	418	1580
Qwen 2.5 VL	—	—	638	1685

Table 5: MME performance across stages.

Model	State	Instruction	Plan
Model	Tracking	Clarity	Adherence
MM-PlanLLM	1.61	2.11	2.24
Qwen 2.5 VL	2.31	2.41	2.74
Llava OV	1.87	2.59	3.00
InternVL 3.5	2.52	2.44	3.13
Qwen 3 VL	2.85	2.35	3.39

Table 6: Dialogue-level evaluation (

N=54

) with LLM-as-a-judge to measure guidance quality.

pVQA	PGAG		pVQA		VSG
Data %	R-L	M	R-L	Acc.	R-L	EM
0%	75.62	77.03	23.30	71.59	53.54	56.32
10%	75.63	76.98	31.66	85.23	53.55	54.74
20%	75.56	76.97	32.41	85.42	53.17	55.79
50%	76.09	76.54	34.00	88.79	56.01	57.89
100%	75.30	76.67	33.65	94.02	55.66	57.37

Table 7: Ablation different pVQA data amounts.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Specialized	LLM	VE	Datasets
Stage	Layers	LLM	VE	Datasets
Initialization	Trn	Frz	Frz	Laion400M Schuhmann et al. (2021)
Visual Instruction Tuning	Trn	Trn	Trn	ShareGPT4o Cui et al. (2024) , MAmmoTH-VL Guo et al. (2024) , VQAv2 Goyal et al. (2017) , GQA Hudson and Manning (2019) , ShareGPT4v Chen et al. (2024a) , PixMo-Cap Deitke et al. (2024) , LLaVA-Pretrain Liu et al. (2023)
Domain-specific Tuning	Trn	Trn	Frz	FoodDialogues Yin et al. (2023) , FoodReasonSeg Yin et al. (2023) , Youcook2 Zhou et al. (2018) , COIN Tang et al. (2019) , Crosstask Zhukov et al. (2019)
Task-specific Training	Trn	Trn	Frz	InstructionVidDial (ours)

Appendix

Table 8: Overview of the multistage training process for VIGiA. Status indicators: Trn= Trainable, Frz= Frozen.

Stage	Dataset	Type	Samples #
1	LAION400M	Cap + Ret	10M+10M
2	ShareGPT4o	Cap	$\approx$ 57k
	MAmmoTH-VL	Reasoning	1M
	VQAv2	pVQA	$\approx$ 444k
	GQA	pVQA	943k
	ShareGPT4v	Ret	$\approx$ 1.25M

Appendix

Table 9: Overview of the datasets used to train VIGiA.

Stage	1	2	3	4
Batch Size	512	64	128	32
RET Batch Size	512	96	192	—
Acc. Steps	4/4/8	4/8/8	4/8/8	4/8/16
Train Steps	19500	15000	866	3252
Parallel	DDP	DDP	DDP	DDP
GPU #	4	8	4	1

Appendix

Table 10: Shared hyperparameters used to train all models. For Stage 1, LLM-specific hyperparameters only affect the [RET] tokens input and output embeddings, along with the language modeling head.

	Plan Reasoning				Plan-based Retrieval				General
	PGAG		pVQA		VSG		CVMR R@1		MME
	R-L	METEOR	R-L	Acc.	R-L	EM	m=0.5	m=0.7	Cog.	Per.
VIGiA 1B	75.44	76.67	29.35	58.69	51.53		26.22	13.92	203	632
VIGiA 3B	76.51	77.72	30.16	71.59	52.15	53.68	23.33	15.91	214	808
VIGiA 8B	75.30	76.67	33.65	94.02	55.66	57.37	30.74	15.37	229	717

Appendix

Table 11: Ablation of LM backbone sizes on model performance across all considered tasks.

Panel of Figure 5 — Figure 5: Six examples of how the similarity of the start and end retrieval representations with the video frames varies throughout the video.

	PGAG
Model	ROUGE-L	METEOR	BERTScore
Idefics2	37.16	45.09	67.55
LLaVA-1.5	42.47	40.84	70.55
Qwen 2.5 VL	31.61	41.51	63.91
LLaVA-OV	41.18	40.85	70.70
InternVL 3.5	41.62	46.20	69.22

Appendix

Table 12: Results comparing the full model training pipeline against a text-only training run. Underlined scores are the second best scores.

Plausibility	Relevance	Accuracy
2.22	2.67	2.75

Appendix

Table 13: Results of the data quality annotation conducted on 250 pVQA turns from the test set of InstructionVidDial.

Consider a dataset of dialogues between users and a system designed to assist with manual tasks (recipes or DIY projects). The objective of these annotations is to quantify the quality of automatically generated VQA (Visual Question Answering) turns within that dataset.

Each line contains an example of a question-answer pair, along with the image that originates the question and the task to be performed.

For each line, on a scale of 1 to 3 (from worst to best), annotate the following 3 criteria:

Relevance : The relevance of the question concerning the image and the task in question.

1 - The question has nothing to do with either the image or the task.

Appendix

Table 14: Annotation instructions provided to annotate in-domain pVQA turns.

User Image
	Question:	I’m adding the pasta to the boiling milk mixture now - is it normal that the liquid looks this white and creamy already, or did I add too much milk?
	GT Answer:	The liquid consistency shown in the image looks perfect! The white, creamy appearance is exactly what you want at this stage. The milk should be bubbling and have this rich, creamy texture before adding the farfalle (bow-tie) pasta. Just remember to keep stirring frequently while the pasta cooks for about 10 minutes at medium heat to prevent it from sticking to the bottom of the pot and to ensure even cooking.
	Question:	Based on the color of the chicken pieces in the pan, are they browned and cooked through enough to be removed from the pan?
	GT Answer:	Yes, from what I can see in the image, the chicken pieces appear to have a nice golden-brown color and look properly cooked through. You can see they’ve lost their raw pink color and have taken on a light golden hue. They’re being removed from the pan with a black spoon, which is exactly what you should be doing at this step. The bits of browned fond left in the pan will add great flavor to the next steps of the recipe.
	Question:	Is the green and black cap shown in the image the correct type of cap for sealing a volumetric flask?
	GT Answer:	Yes, that appears to be the correct type of cap for a volumetric flask. The cap shown in the image has a green band and black top which is a standard laboratory flask cap design. It’s designed to create an airtight seal when properly attached to the volumetric flask. Make sure to screw it on firmly but not too tight to avoid damaging the threads.

Appendix

Table 15: pVQA test set examples

You are an expert evaluator for Multimodal Procedural Guidance Assistants.

Your task is to evaluate a full dialogue session between a User and a VLM Assistant.

For the purpose of this evaluation you are provided with:

- TASK: The official ground-truth procedural plan (recipe, manual, etc.).

- Dialogue Transcript: A chronological log of the interaction. Each turn contains:

- User Input: Text and (optional) Image Description.

Appendix

Table 16: Prompt used to evaluate the dialogues in the dialogue level evaluation described in Section 6.5

User Image
Question:	I’ve added the oil and garlic to the pan as shown in the image, but how do I know if the heat is at the right temperature? I can see some garlic starting to sizzle but I don’t want to burn it.
VIGIA 8B:	The heat appears to be at a good medium temperature as shown in the image. You can see the garlic is starting to sizzle and release its oil, which is the right reaction. If you’re seeing this gentle sizzling and the garlic isn’t browning too quickly, the heat is appropriate. Just keep an eye on it and stir occasionally to prevent burning. The garlic should be fragrant but not browned when you move on to the next step.
Qwen2.5VL:	To ensure the heat is at the right temperature, you can use a few methods:

	1. Sizzling Garlic: As you’ve noticed, the garlic is sizzling in the pan. This is a good sign because it indicates the oil is hot enough to start cooking the garlic without burning it. However, if the sizzling becomes too intense or the garlic starts to brown quickly, reduce the heat slightly.

Appendix

Table 17: Examples of answers provided by VIGiA and some baselines to pVQA samples. In this example LLaVA-OV provided a very dangerous suggestion to the user, whereas both VIGiA and Qwen2.5 VL were able to provide adequate and safe answers.

User Image
Question:	How do I know if I’ve rolled the pastry out to the correct dimensions? Should I measure it or is eyeballing it okay?
VIGIA 8B:	For best results, you should measure your rolled pastry to ensure it’s exactly 24 inches long and 8 inches wide. This is crucial because puff pastry needs to be rolled to the correct size to properly fold and layer. While eyeballing might work for some recipes, for puff pastry, precise measurements are essential to achieve the right texture and layers. You can use a measuring tape or ruler to check the dimensions while rolling.
Qwen2.5VL:	When rolling out the puff pastry, it’s important to ensure that it reaches the desired dimensions to achieve the best results. Here’s how you can check:

	### Measuring:
	- Use a Ruler: Place a ruler next to the pastry and measure the length and width. This ensures accuracy and consistency.

Appendix

Table 18: Examples of answers provided by VIGiA and some baselines to pVQA samples. In this example, all three models provided adequate answers but general-purpose models tend to be more verbose.

User Image
Question:	I see the balance pointer is not quite centered at zero - should I adjust the calibration knob before weighing anything?
VIGIA 8B:	Yes, you should definitely calibrate the balance first. The image shows the balance is currently not zeroed, which means you’ll need to adjust the calibration knob to ensure the indicator needle is centered at zero before proceeding to weigh any objects. This is a crucial first step to ensure accurate measurements throughout the process.
Qwen2.5VL:	Yes, it’s important to ensure that the balance is properly calibrated before using it for any measurements. If the pointer is not centered at zero, it means the balance is not calibrated correctly, which could lead to inaccurate readings. Here’s what you should do:

	1. Locate the Calibration Knob: Typically, there will be a small knob or screw near the base of the balance, often labeled "Calibration" or something similar. This is used to adjust the balance.

Appendix

Table 19: Examples of answers provided by VIGiA and some baselines to pVQA samples. In this example, all three models provided adequate answers but general-purpose models tend to be more verbose.

User Image
Question:	Are my cinnamon roll discs being cut at the right thickness based on what’s shown in the image?
VIGIA 8B:	Looking at the image, the cinnamon roll discs appear to be cut at a good thickness. You can see they’re being cut into uniform, bite-sized pieces that are approximately 1-2 inches in diameter. This size is ideal for mini cinnamon roll bites as it ensures they’ll cook evenly and be easy to eat. Just make sure to keep an eye on them while baking as the thickness may vary slightly depending on how you cut them.
VIGIA 3B:	From what I can see in the image, the thickness appears to be appropriate. The discs are being cut into small, uniform pieces that are about 1-2 inches in diameter, which is ideal for mini cinnamon roll bites. This size will allow them to cook evenly and be easy to handle. When baked, these will become crispy on the outside while remaining soft on the inside.
VIGIA 1B:	Yes, the thickness shown in the image appears to be appropriate for the recipe. The discs should be cut to be about 1/4 inch (6mm) thick, which is ideal for baking. If you cut them too thick, they might burn easily, and if too thin, they might burn too quickly. The uniform, flat shape shown in the image is also a good sign that you’re working on the right material.

Appendix

Table 20: Examples of answers provided by VIGiA and its variants. All 3 model sizes available are capable of providing an accurate and coherent answer to the user question, while also providing a brief explanation of the importance of the thickness of the cinnamon rolls.

User Image
Question:	The pie crust in the image looks quite round - is a jar lid the only tool I can use to cut these circles, or can I use something else of similar size?
VIGIA 8B:	While a jar lid is the traditional tool for cutting pie crust circles, you can use other round objects of similar size as a substitute. Some alternatives include a clean, dry glass or a small bowl. Just make sure the object you use is completely dry and clean to prevent sticking. The most important thing is to have a consistent size for your circles so they’ll bake evenly.
VIGIA 3B:	A jar lid is the only tool that will work for cutting the circular pie crust. The size of the jar lid (4 inches) is perfect for creating uniform, bite-sized pieces. Using a different tool could lead to unevenly sized pieces, which would be difficult to work with. Make sure you’re using a clean, sharp jar lid and press it into the center of the pie crust to ensure a clean cut.
VIGIA 1B:	A jar lid is the correct tool for cutting the 12 4-inch (10 cm) circles. While the image shows a circular shape, a jar lid is specifically designed for this purpose. Using a jar lid will give you the most precise and uniform cuts needed for the pie crust. Make sure to press firmly into the crust to ensure the circles are firmly attached. The circular shape will help the crust hold its shape when baked.

Appendix

Table 21: Examples of answers provided by VIGiA and its variants. The smaller model variants fail to accurately answer the user question, claiming that only a jar lid can be used whereas VIGIA 8B accurately clarifies that other objects can be used as long as they are dry and clean.

You are a helpful assistant simulating a user interacting with a visual task guidance system. You will be provided with a task, a conversation history, the current step’s description, and an image related to that step.

Based on the conversation context, the current step, and the image, generate one question-answer pair. The question should mimic a question a user might ask after uploading the provided image while seeking guidance on the current step, this is VERY important. The questions should be DIRECTLY related to the image and current task step, and should be relevant to the ongoing conversation. Make sure that these questions sound natural as if a real user had asked them. Use the conversation history, the task instructions and your knowledge to answer as accurately as possible.

Some examples of questions you might generate include asking if the mixture in the image is the correct consistency, if the oven temperature is correct, or if the bolt in the image needs to be tightened, etc.

Output the question-answer pair in the following format:

Appendix

Table 22: Prompt used to generate the in-domain pVQA question-answer pairs.

You will be provided with an image and a bit of a dialogue. Your task is to tell me if the last system response accurately answers the last user question based on the image. Answer solely regarding the LAST user question and the last assistant response in the dialogue.

As I am parsing this automatically, please write your answer (YES or NO) immediately after the words "FINAL ANSWER:" in your response. Feel free to think and reason out loud in your response before you give the final answer.

Task Context:

{task_text}

Appendix

Table 23: Prompt used to evaluate the in-domain pVQA responses.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos
Mar 10, 2026Lei Shi, Victor Aregbede, Andreas Persson +3
Örebro University, SE-701 82 Örebro, Sweden
VCIFBench: Evaluating Complex Instruction Following for Video Understanding
Jun 3, 2026Huangchen Xu, Yuan Wu, Yi ChangVLM Evaluation Video Understanding
School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University · 3International Center of Future Science, Jilin University
Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios
Aug 26, 2026Hongbo Liu, Peixian Chen, Sihan Liu +12Video-Language Model Evaluation Video Understanding
TJU · Tencent Youtu Lab · SJTU +3

Mar 10, 2026J/K move · Enter open · S save

LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos

Lei Shi, Victor Aregbede, Andreas Persson, Martin Längkvist+2

Örebro University, SE-701 82 Örebro, Sweden

Open paper AskPDF arXiv

Procedure planning requires a model to predict a sequence of actions that transform a start visual observation into a goal in instructional videos. While most existing methods rely primarily on visual observations as input, they often struggle with the inherent ambiguity where different actions can appear visually similar. In this work, we argue that language descriptions offer a more distinctive representation in the latent space for procedure planning. We introduce Language-Aware Planning (LAP), a novel method that leverages the expressiveness of language to bridge visual observation and planning. LAP uses a finetuned Vision Language Model (VLM) to translate visual observations into text descriptions and to predict actions and extract text embeddings. These text embeddings are more distinctive than visual embeddings and are used in a diffusion model for planning action sequences. We evaluate LAP on three procedure planning benchmarks: CrossTask, Coin, and NIV. LAP achieves new state-of-the-art performance across multiple metrics and time horizons by large margin, demonstrating the significant advantage of language-aware planning.

Abstract

VLM Reasoning

Explore similar work

LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos

VCIFBench: Evaluating Complex Instruction Following for Video Understanding

Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios