We introduce VIGiA, a novel multimodal dialogue model designed to understand and reason over complex, multi-step instructional video action plans. Unlike prior work which focuses mainly on text-only guidance, or treats vision and language in isolation, VIGiA supports grounded, plan-aware dialogue that requires reasoning over visual inputs, instructional plans, and interleaved user interactions. To this end, VIGiA incorporates two key capabilities: (1) multimodal plan reasoning, enabling the model to align uni- and multimodal queries with the current task plan and respond accurately; and (2) plan-based retrieval, allowing it to retrieve relevant plan steps in either textual or visual representations. Experiments were done on a novel dataset with rich Instructional Video Dialogues aligned with Cooking and DIY plans. Our evaluation shows that VIGiA outperforms existing state-of-the-art models on all tasks in a conversational plan guidance setting, reaching over 90% accuracy on plan-aware VQA.
Figures & tables
Figure 1: VIGiA is grounded on complex multimodal instructional plans, delivering unified multimodal alignment over dialogue turns by providing text-based guidance (Goal 1), plan-aware visual question answering (Goal 2) aligning plan actions and visual context (Goal 3), and retrieving relevant video moments (Goal 4).
Figure 2: VIGiA is an LVLM model that processes instructional video plans to navigate through steps of the plan, perform QA, VQA and retrieval of arbitrarily random text steps or video moment steps.
Figure 3: Global view of VIGiA’s architecture. To handle multimodal inputs VIGiA combines a visual encoder and an LLM using an MLP as a connector module. For conversational video moment retrieval, VIGiA outputs a dedicated video moment start and end representation that can be used for start and end frame retrieval.
VIGiA
MM-PlanLLM
VQA support
✓
×
Vision Tokens
257 tokens
1 token
CVMR Retrieval
Start & End Frame
Mid Frame
Multiple Domains
✓
×
Table 1: Direct comparison of VIGiA with current procedural plan guidance SotA MM-PlanLLM Glória-Silva et al. (2024b)
Plan Reasoning
Plan-based Retrieval
PGAG
pVQA
VSG
Model
ROUGE-L
METEOR
BERTScore
ROUGE-L
Accuracy*
ROUGE-L
EM
Idefics2
37.16
45.09
67.55
21.22
44.49
20.16
5.85
LLaVA-1.5
42.47
40.84
70.55
26.77
64.11
16.07
4.21
Qwen 2.5 VL
31.61
41.51
63.91
26.39
75.14
16.65
9.47
LLaVA-OV
41.18
40.85
70.70
27.02
77.38
32.28
17.89
Table 2: Results on the in-domain tasks: PGAG, pVQA, and VSG. Underlined scores are the best among the baselines. *Accuracy is measured using the majority vote of three independent LLMs (See Section 5 ).
Model
CVMR
Recall@1
Recall@5
m=0.5
m=0.7
m=0.5
m=0.7
firm
TRACE
5.97
1.63
—
—
VIGiA
10.49
2.89
30.38
11.75
adj.
MM-PlanLLM
25.32
12.48
31.46
16.46
VIGiA
30.74
15.37
51.18
26.94
Table 3: Results on the CVMR task.
Figure 4: Comparing R@1 performance with varying values for the similarity threshold on InstructionVidDial’s dev set.
Plan Reasoning
Plan-based Retrieval
General
PGAG
pVQA
VSG
CVMR R@1
MME
R-L
METEOR
R-L
Acc.
R-L
EM
m=0.5
m=0.7
Cog.
Per.
Stages
4 Only
74.75
76.05
33.56
88.79
54.27
56.84
11.39
5.97
234
503
1+4
74.67
76.61
33.89
90.47
54.17
56.32
20.43
11.39
211
501
1+2+4
74.99
76.39
34.20
91.03
54.12
56.84
9.58
5.06
223
708
1+2+3+4
75.30
76.67
33.65
94.02
55.66
57.37
30.74
15.37
229
717
Table 4: Ablation of model training stages. Underlined scores represent improvements over the previous stage/row.
Model
Stage 2
Stage 4 / Final
Cog.
Per.
Cog.
Per.
MM-PlanLLM
—
—
34
1
Idefics2
—
—
328
1474
LLaVA-1.5
—
—
314
1483
LLaVA-OV
—
—
418
1580
Qwen 2.5 VL
—
—
638
1685
Table 5: MME performance across stages.
Model
State
Instruction
Plan
Tracking
Clarity
Adherence
MM-PlanLLM
1.61
2.11
2.24
Qwen 2.5 VL
2.31
2.41
2.74
Llava OV
1.87
2.59
3.00
InternVL 3.5
2.52
2.44
3.13
Qwen 3 VL
2.85
2.35
3.39
Table 6: Dialogue-level evaluation ( N=54 ) with LLM-as-a-judge to measure guidance quality.
pVQA
PGAG
pVQA
VSG
Data %
R-L
M
R-L
Acc.
R-L
EM
0%
75.62
77.03
23.30
71.59
53.54
56.32
10%
75.63
76.98
31.66
85.23
53.55
54.74
20%
75.56
76.97
32.41
85.42
53.17
55.79
50%
76.09
76.54
34.00
88.79
56.01
57.89
100%
75.30
76.67
33.65
94.02
55.66
57.37
Table 7: Ablation different pVQA data amounts.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Specialized
LLM
VE
Datasets
Layers
Initialization
Trn
Frz
Frz
Laion400M Schuhmann et al. (2021)
Visual Instruction Tuning
Trn
Trn
Trn
ShareGPT4o Cui et al. (2024) , MAmmoTH-VL Guo et al. (2024) , VQAv2 Goyal et al. (2017) , GQA Hudson and Manning (2019) , ShareGPT4v Chen et al. (2024a) , PixMo-Cap Deitke et al. (2024) , LLaVA-Pretrain Liu et al. (2023)
Domain-specific Tuning
Trn
Trn
Frz
FoodDialogues Yin et al. (2023) , FoodReasonSeg Yin et al. (2023) , Youcook2 Zhou et al. (2018) , COIN Tang et al. (2019) , Crosstask Zhukov et al. (2019)
Task-specific Training
Trn
Trn
Frz
InstructionVidDial (ours)
Appendix
Table 8: Overview of the multistage training process for VIGiA. Status indicators: Trn= Trainable, Frz= Frozen.
Stage
Dataset
Type
Samples #
1
LAION400M
Cap + Ret
10M+10M
2
ShareGPT4o
Cap
≈ 57k
MAmmoTH-VL
Reasoning
1M
VQAv2
pVQA
≈ 444k
GQA
pVQA
943k
ShareGPT4v
Ret
≈ 1.25M
Appendix
Table 9: Overview of the datasets used to train VIGiA.
Stage
1
2
3
4
Batch Size
512
64
128
32
RET Batch Size
512
96
192
—
Acc. Steps
4/4/8
4/8/8
4/8/8
4/8/16
Train Steps
19500
15000
866
3252
Parallel
DDP
DDP
DDP
DDP
GPU #
4
8
4
1
Appendix
Table 10: Shared hyperparameters used to train all models. For Stage 1, LLM-specific hyperparameters only affect the [RET] tokens input and output embeddings, along with the language modeling head.
Plan Reasoning
Plan-based Retrieval
General
PGAG
pVQA
VSG
CVMR R@1
MME
R-L
METEOR
R-L
Acc.
R-L
EM
m=0.5
m=0.7
Cog.
Per.
VIGiA 1B
75.44
76.67
29.35
58.69
51.53
26.22
13.92
203
632
VIGiA 3B
76.51
77.72
30.16
71.59
52.15
53.68
23.33
15.91
214
808
VIGiA 8B
75.30
76.67
33.65
94.02
55.66
57.37
30.74
15.37
229
717
Appendix
Table 11: Ablation of LM backbone sizes on model performance across all considered tasks.
Figure 5: Six examples of how the similarity of the start and end retrieval representations with the video frames varies throughout the video.
PGAG
Model
ROUGE-L
METEOR
BERTScore
Idefics2
37.16
45.09
67.55
LLaVA-1.5
42.47
40.84
70.55
Qwen 2.5 VL
31.61
41.51
63.91
LLaVA-OV
41.18
40.85
70.70
InternVL 3.5
41.62
46.20
69.22
Appendix
Table 12: Results comparing the full model training pipeline against a text-only training run. Underlined scores are the second best scores.
Plausibility
Relevance
Accuracy
2.22
2.67
2.75
Appendix
Table 13: Results of the data quality annotation conducted on 250 pVQA turns from the test set of InstructionVidDial.
Consider a dataset of dialogues between users and a system designed to assist with manual tasks (recipes or DIY projects). The objective of these annotations is to quantify the quality of automatically generated VQA (Visual Question Answering) turns within that dataset.
Each line contains an example of a question-answer pair, along with the image that originates the question and the task to be performed.
For each line, on a scale of 1 to 3 (from worst to best), annotate the following 3 criteria:
Relevance : The relevance of the question concerning the image and the task in question.
1 - The question has nothing to do with either the image or the task.
Appendix
Table 14: Annotation instructions provided to annotate in-domain pVQA turns.
User Image
Question:
I’m adding the pasta to the boiling milk mixture now - is it normal that the liquid looks this white and creamy already, or did I add too much milk?
GT Answer:
The liquid consistency shown in the image looks perfect! The white, creamy appearance is exactly what you want at this stage. The milk should be bubbling and have this rich, creamy texture before adding the farfalle (bow-tie) pasta. Just remember to keep stirring frequently while the pasta cooks for about 10 minutes at medium heat to prevent it from sticking to the bottom of the pot and to ensure even cooking.
Question:
Based on the color of the chicken pieces in the pan, are they browned and cooked through enough to be removed from the pan?
GT Answer:
Yes, from what I can see in the image, the chicken pieces appear to have a nice golden-brown color and look properly cooked through. You can see they’ve lost their raw pink color and have taken on a light golden hue. They’re being removed from the pan with a black spoon, which is exactly what you should be doing at this step. The bits of browned fond left in the pan will add great flavor to the next steps of the recipe.
Question:
Is the green and black cap shown in the image the correct type of cap for sealing a volumetric flask?
GT Answer:
Yes, that appears to be the correct type of cap for a volumetric flask. The cap shown in the image has a green band and black top which is a standard laboratory flask cap design. It’s designed to create an airtight seal when properly attached to the volumetric flask. Make sure to screw it on firmly but not too tight to avoid damaging the threads.
Appendix
Table 15: pVQA test set examples
You are an expert evaluator for Multimodal Procedural Guidance Assistants.
Your task is to evaluate a full dialogue session between a User and a VLM Assistant.
For the purpose of this evaluation you are provided with:
- TASK: The official ground-truth procedural plan (recipe, manual, etc.).
- Dialogue Transcript: A chronological log of the interaction. Each turn contains:
- User Input: Text and (optional) Image Description.
Appendix
Table 16: Prompt used to evaluate the dialogues in the dialogue level evaluation described in Section 6.5
User Image
Question:
I’ve added the oil and garlic to the pan as shown in the image, but how do I know if the heat is at the right temperature? I can see some garlic starting to sizzle but I don’t want to burn it.
VIGIA 8B:
The heat appears to be at a good medium temperature as shown in the image. You can see the garlic is starting to sizzle and release its oil, which is the right reaction. If you’re seeing this gentle sizzling and the garlic isn’t browning too quickly, the heat is appropriate. Just keep an eye on it and stir occasionally to prevent burning. The garlic should be fragrant but not browned when you move on to the next step.
Qwen2.5VL:
To ensure the heat is at the right temperature, you can use a few methods:
1. Sizzling Garlic: As you’ve noticed, the garlic is sizzling in the pan. This is a good sign because it indicates the oil is hot enough to start cooking the garlic without burning it. However, if the sizzling becomes too intense or the garlic starts to brown quickly, reduce the heat slightly.
Appendix
Table 17: Examples of answers provided by VIGiA and some baselines to pVQA samples. In this example LLaVA-OV provided a very dangerous suggestion to the user, whereas both VIGiA and Qwen2.5 VL were able to provide adequate and safe answers.
User Image
Question:
How do I know if I’ve rolled the pastry out to the correct dimensions? Should I measure it or is eyeballing it okay?
VIGIA 8B:
For best results, you should measure your rolled pastry to ensure it’s exactly 24 inches long and 8 inches wide. This is crucial because puff pastry needs to be rolled to the correct size to properly fold and layer. While eyeballing might work for some recipes, for puff pastry, precise measurements are essential to achieve the right texture and layers. You can use a measuring tape or ruler to check the dimensions while rolling.
Qwen2.5VL:
When rolling out the puff pastry, it’s important to ensure that it reaches the desired dimensions to achieve the best results. Here’s how you can check:
### Measuring:
- Use a Ruler: Place a ruler next to the pastry and measure the length and width. This ensures accuracy and consistency.
Appendix
Table 18: Examples of answers provided by VIGiA and some baselines to pVQA samples. In this example, all three models provided adequate answers but general-purpose models tend to be more verbose.
User Image
Question:
I see the balance pointer is not quite centered at zero - should I adjust the calibration knob before weighing anything?
VIGIA 8B:
Yes, you should definitely calibrate the balance first. The image shows the balance is currently not zeroed, which means you’ll need to adjust the calibration knob to ensure the indicator needle is centered at zero before proceeding to weigh any objects. This is a crucial first step to ensure accurate measurements throughout the process.
Qwen2.5VL:
Yes, it’s important to ensure that the balance is properly calibrated before using it for any measurements. If the pointer is not centered at zero, it means the balance is not calibrated correctly, which could lead to inaccurate readings. Here’s what you should do:
1. Locate the Calibration Knob: Typically, there will be a small knob or screw near the base of the balance, often labeled "Calibration" or something similar. This is used to adjust the balance.
Appendix
Table 19: Examples of answers provided by VIGiA and some baselines to pVQA samples. In this example, all three models provided adequate answers but general-purpose models tend to be more verbose.
User Image
Question:
Are my cinnamon roll discs being cut at the right thickness based on what’s shown in the image?
VIGIA 8B:
Looking at the image, the cinnamon roll discs appear to be cut at a good thickness. You can see they’re being cut into uniform, bite-sized pieces that are approximately 1-2 inches in diameter. This size is ideal for mini cinnamon roll bites as it ensures they’ll cook evenly and be easy to eat. Just make sure to keep an eye on them while baking as the thickness may vary slightly depending on how you cut them.
VIGIA 3B:
From what I can see in the image, the thickness appears to be appropriate. The discs are being cut into small, uniform pieces that are about 1-2 inches in diameter, which is ideal for mini cinnamon roll bites. This size will allow them to cook evenly and be easy to handle. When baked, these will become crispy on the outside while remaining soft on the inside.
VIGIA 1B:
Yes, the thickness shown in the image appears to be appropriate for the recipe. The discs should be cut to be about 1/4 inch (6mm) thick, which is ideal for baking. If you cut them too thick, they might burn easily, and if too thin, they might burn too quickly. The uniform, flat shape shown in the image is also a good sign that you’re working on the right material.
Appendix
Table 20: Examples of answers provided by VIGiA and its variants. All 3 model sizes available are capable of providing an accurate and coherent answer to the user question, while also providing a brief explanation of the importance of the thickness of the cinnamon rolls.
User Image
Question:
The pie crust in the image looks quite round - is a jar lid the only tool I can use to cut these circles, or can I use something else of similar size?
VIGIA 8B:
While a jar lid is the traditional tool for cutting pie crust circles, you can use other round objects of similar size as a substitute. Some alternatives include a clean, dry glass or a small bowl. Just make sure the object you use is completely dry and clean to prevent sticking. The most important thing is to have a consistent size for your circles so they’ll bake evenly.
VIGIA 3B:
A jar lid is the only tool that will work for cutting the circular pie crust. The size of the jar lid (4 inches) is perfect for creating uniform, bite-sized pieces. Using a different tool could lead to unevenly sized pieces, which would be difficult to work with. Make sure you’re using a clean, sharp jar lid and press it into the center of the pie crust to ensure a clean cut.
VIGIA 1B:
A jar lid is the correct tool for cutting the 12 4-inch (10 cm) circles. While the image shows a circular shape, a jar lid is specifically designed for this purpose. Using a jar lid will give you the most precise and uniform cuts needed for the pie crust. Make sure to press firmly into the crust to ensure the circles are firmly attached. The circular shape will help the crust hold its shape when baked.
Appendix
Table 21: Examples of answers provided by VIGiA and its variants. The smaller model variants fail to accurately answer the user question, claiming that only a jar lid can be used whereas VIGIA 8B accurately clarifies that other objects can be used as long as they are dry and clean.
You are a helpful assistant simulating a user interacting with a visual task guidance system. You will be provided with a task, a conversation history, the current step’s description, and an image related to that step.
Based on the conversation context, the current step, and the image, generate one question-answer pair. The question should mimic a question a user might ask after uploading the provided image while seeking guidance on the current step, this is VERY important. The questions should be DIRECTLY related to the image and current task step, and should be relevant to the ongoing conversation. Make sure that these questions sound natural as if a real user had asked them. Use the conversation history, the task instructions and your knowledge to answer as accurately as possible.
Some examples of questions you might generate include asking if the mixture in the image is the correct consistency, if the oven temperature is correct, or if the bolt in the image needs to be tightened, etc.
Output the question-answer pair in the following format:
Appendix
Table 22: Prompt used to generate the in-domain pVQA question-answer pairs.
You will be provided with an image and a bit of a dialogue. Your task is to tell me if the last system response accurately answers the last user question based on the image. Answer solely regarding the LAST user question and the last assistant response in the dialogue.
As I am parsing this automatically, please write your answer (YES or NO) immediately after the words "FINAL ANSWER:" in your response. Feel free to think and reason out loud in your response before you give the final answer.
Task Context:
{task_text}
Appendix
Table 23: Prompt used to evaluate the in-domain pVQA responses.
Procedure planning requires a model to predict a sequence of actions that transform a start visual observation into a goal in instructional videos. While most existing methods rely primarily on visual observations as input, they often struggle with the inherent ambiguity where different actions can appear visually similar. In this work, we argue that language descriptions offer a more distinctive representation in the latent space for procedure planning. We introduce Language-Aware Planning (LAP), a novel method that leverages the expressiveness of language to bridge visual observation and planning. LAP uses a finetuned Vision Language Model (VLM) to translate visual observations into text descriptions and to predict actions and extract text embeddings. These text embeddings are more distinctive than visual embeddings and are used in a diffusion model for planning action sequences. We evaluate LAP on three procedure planning benchmarks: CrossTask, Coin, and NIV. LAP achieves new state-of-the-art performance across multiple metrics and time horizons by large margin, demonstrating the significant advantage of language-aware planning.
Multimodal large language models have made rapid progress in video understanding, yet existing benchmarks largely rely on simple prompts and provide limited evidence about whether models can satisfy explicit output constraints. We introduce VCIFBench, a benchmark for evaluating complex instruction following in video understanding. VCIFBench constructs constraint-rich instructions from both benchmark-adapted and directly video-grounded prompts, covering content, format, style, and structure requirements, and evaluates model outputs with a hybrid verification pipeline. The benchmark contains 306 satisfiable test instructions, a 540-pair DPO preference dataset, and a 30-item conflict diagnostic subset. Experiments on 10 MLLMs show that joint constraint satisfaction remains challenging. We further show that DPO training on VCIFBench data can improve instruction-following performance.
Huangchen Xu, Yuan Wu, Yi Chang
School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University · 3International Center of Future Science, Jilin University
Multimodal Large Language Models (MLLMs) have shown strong performance in video understanding. However, their ability to follow instructions in this domain remains under-explored. Real-world video understanding requires models not only to interpret video content correctly, but also to satisfy diverse user-specified constraints. Existing benchmarks focus primarily on task accuracy rather than instruction adherence, leaving this capability insufficiently evaluated. To address this gap, we introduce Video-IFBench, a comprehensive benchmark for evaluating instruction following in video understanding, where models must satisfy diverse user-specified constraints, including those grounded in visual and audio content. We develop an instruction taxonomy with four templates, including single-task, multi-task, selection, and nested instructions, covering 32 task types and 39 manually designed constraint categories spanning both semantic and format requirements. To reduce annotation cost, we build a semi-automatic data construction pipeline that combines MLLMs, programmatic processing, and human verification, resulting in 1.5K samples. We conduct a large-scale evaluation of more than 20 recent MLLMs and show that video instruction following remains challenging for current models, especially for instructions with many constraints, semantic constraints, or complex conditional structures that require selecting the correct branch or path based on video content. We hope our work will facilitate future research on instruction following in video understanding scenarios.