Converting natural-language instructions into executable action plans conditioned on a scene image is a core capability for embodied human-robot interaction, but existing benchmarks target English, simulated or video-grounded environments, and largely assume instructions are clear and executable. We introduce the first Persian vision-language-action dataset for this task: 258 instances over 230 real household/office photographs, annotated with reference plans, deliberate ambiguity and infeasibility labels, disambiguating question-answer pairs, and an auto-generated chain-of-thought rationale for training. Using this resource, we ablate retrieval-augmentation, chain-of-thought prompting, and an agentic disambiguation step, evaluated with an LLM-as-judge protocol and a new metric, PGOS-FA, extending prior work (PG2S) with step-order sensitivity, an over-generation penalty, and Persian-adapted embeddings. Paired statistical testing across 36 comparisons shows retrieval-augmentation is a large, robustly significant effect, while finer-grained additions are directionally consistent but not statistically distinguishable from noise at this scale, and the LLM-judge metric has near-zero power where PGOS-FA succeeds. A blind human-correlation study on 79 real outputs shows PGOS-FA tracks human judgment at least as closely as the LLM-judge. Testing generalization beyond the ablation's single backbone, we find that most of the gap between an open-weight and a substantially stronger closed planner is attributable to open-weight model generation rather than to the closed model's capability specifically, and that our best configuration significantly outperforms an independently reproduced external baseline despite a weaker backbone. The dataset, PGOS-FA implementation, and evaluation code will be made available upon reasonable request.
Figures & tables
Dataset
Language
Grounding
Scale
Ambig. labels
Infeas. labels
ALFRED ( Shridhar et al. 2020 )
English
Simulated (AI2-THOR)
∼ 8,055 expert demos / ∼ 25,700 instructions
No
No
DROID ( Khazatsky et al. 2025 )
English
Real robot, video/trajectory
76,000 episodes ( ∼ 350h)
No
No
EgoCOT ( Mu et al. 2023 ; Grauman et al. 2022 )
English
Real, egocentric video
Large-scale (video-clip-derived)
No
No
RxR ( Ku et al. 2020 )
English, Hindi, Telugu
Real building, simulated nav.
126,000 instructions / 16,500 paths
No
No
AmbiK ( Ivanova et al. 2025 )
English
Textual only (no image)
1,000 pairs / 2,000 tasks
Yes (1,000)
No
VLN-NF ( Su et al. 2026 )
English
Simulated nav. (Matterport3D)
Derived from REVERIE ( ∼ 45% retained after filtering)
No
Yes (feasible/NOT-FOUND pairs)
TABLE I : Benchmark comparison. No prior resource we are aware of combines a low-resource target language, single-image (rather than simulated or video) grounding, and explicit annotation of both instruction ambiguity and physical infeasibility.
Fig. 1 : Two real instances from the dataset, illustrating the two deliberate-difficulty annotation types described above: (a) an ambiguous instruction requiring disambiguation, and (b) a physically infeasible instruction given the scene.
Field
Description
image
Filename of the associated photograph
instruction
The raw Persian instruction as given
actions
Reference action plan (numbered steps)
executable_or_notexecutable
Binary flag: is the instruction physically/logically executable given the scene?
clarification_question
Disambiguating question(s) and answer(s), present only when the instruction is ambiguous
Clear_Instruction
Disambiguated rewrite of the instruction, derived from the clarification Q&A
TABLE II : Dataset schema.
Split
Total
Plain
Ambiguous
Infeasible
Both
Retrieval pool
120
98
17
6
1
Validation
39
37
0
2
0
Test
99
88
6
5
0
All
258
223
23
13
1
TABLE III : Instruction type by split. The Ambiguous and Infeasible columns each include the single instance that is both, so Plain + Ambiguous + Infeasible − Both = Total. The Retrieval-pool and Validation rows together make up the 159-instance training split described above.
Fig. 2 : Pipeline architecture. Every configuration passes the (possibly disambiguated) instruction and image to the planner; the disambiguation agent and the retrieval step (dashed boxes) are each present only in a subset of the eight configurations, and the planner uses chain-of-thought reasoning only in a further subset – exactly which subset is specified per configuration in Table IV .
#
Configuration
RAG ex.
Example content
CoT
Disambig.
1
Baseline
0
—
No
No
2
RAG (1-shot)
1
reference plan
No
No
3
RAG (3-shot)
3
reference plan
No
No
4
RAG (3-shot) + CoT
3
reference plan
Yes
No
5
RAG (3-shot, unclarified ex.) + CoT
3
reference plan, unclarified instruction
Yes
No
6
RAG (3-shot) + Disambiguation
3
reference plan
No
Yes
TABLE IV : Pipeline configurations.
Variant
Mean (correct)
Mean (incorrect)
Gap
Cohen’s d
Full (official)
0.883
0.595
0.287
1.33
Content only ( α=0 )
0.876
0.513
0.363
1.52
Order only ( α=1 )
0.899
0.788
0.111
0.53
Goal-wise only
0.866
0.566
0.300
1.29
Sentence-wise only
0.899
0.624
0.275
1.37
TABLE V : PGOS-FA component ablation on the 16-instance controlled set (8 correct / 8 incorrect plans).
Configuration
Lang
PGOS-FA
LLM-judge acc.
(95% CI)
(95% CI)
1. Baseline
En
0.341 [0.277, 0.412]
0.205 [0.077, 0.333]
1. Baseline
Fa
0.369 [0.306, 0.438]
0.231 [0.103, 0.385]
2. RAG (1-shot)
En
0.516 [0.424, 0.604]
0.256 [0.128, 0.385]
2. RAG (1-shot)
Fa
0.540 [0.453, 0.628]
0.333 [0.179, 0.487]
3. RAG (3-shot)
En
0.535 [0.452, 0.619]
0.359 [0.231, 0.513]
TABLE VI : Ablation results, n=39 validation instances. Configuration numbering matches Table IV . Bold marks the per-metric maximum at validation scale; Section V-B quantifies why several of these apparent differences should not be read as proven effects without the significance results that follow.
Configuration
PGOS-FA
LLM-judge accuracy
Baseline (English)
0.376 [0.328, 0.427]
0.212 [0.131, 0.293]
RAG + Disambiguation + CoT (English)
0.524 [0.478, 0.571]
0.253 [0.172, 0.343]
RAG (CoT ex.) + Disambiguation + CoT (English)
0.507 [0.458, 0.566]
0.283 [0.192, 0.364]
TABLE VII : Held-out test set ( n=99 ). CIs use the same bootstrap procedure as Table VI .
Fig. 3 : PGOS-FA score vs. mean human rating for all 79 sampled instances, with a linear trend line. The relationship is positive and reasonably tight given that PGOS-FA is a fully automatic metric being compared against the average of only three human raters.
Configuration
Mean PGOS-FA
Gemma-3-27B-IT (original)
0.521
Gemma-4-31B-IT (control)
0.638
Gemini 2.5 Pro (planner)
0.678
TABLE VIII : Planner backbone comparison, identical pipeline and retrieval index throughout, on the n=95 test instances scored successfully under all three conditions (4 excluded by the multi-dependent-verb scoring failure described in Section IV-C2 , unrelated to any model substitution). Row 1: Gemma-3-27B-IT in every agent role (original backbone). Row 2: Gemma-4-31B-IT in every role (matched-generation control). Row 3: Gemini 2.5 Pro as the planner, with Gemma-4-31B-IT handling disambiguation/merging.
Configuration
PGOS-FA (Gemma-3)
Rank
PGOS-FA (Gemma-4)
Rank
1. Baseline
0.341
1
0.418
1
2. RAG (1-shot)
0.516
3
0.665
5
3. RAG (3-shot)
0.535
5
0.672
6
4. RAG (3-shot) + CoT
0.524
4
0.649
2
5. RAG (unclarified) + CoT
0.491
2
0.694
8
6. RAG + Disambiguation
0.554
7
0.661
4
TABLE IX : All eight configurations, English prompt, on the n=39 validation set: original backbone vs. Gemma-4-31B-IT. Ranks are within-backbone (1 = lowest).
Fig. 4 : Rank of each configuration under Gemma-3-27B-IT (left) vs. Gemma-4-31B-IT (right), n=39 validation instances. Lines connect the same configuration across backbones; the two most extreme reorderings (configurations 5 and 8) are highlighted. Only configuration 1 (baseline) keeps its rank.
Method
Mean PGOS-FA
Our baseline (no RAG)
0.373
External baseline
0.413
Our best configuration
0.531
TABLE X : External baseline comparison, n=96 (3 instances excluded by the same scoring failure as Table VIII ). Means are computed on these 96 shared instances and so differ slightly from the n=99 figures in Table VII . “External baseline” is the zero-shot 3-agent GPT-4o pipeline of Brienza et al. (2024) ; “our best configuration” is RAG + Disambiguation + CoT on Gemma-3-27B-IT.
Fig. 5 : The red bowl rests upside down in the scene (bottom left of the photograph). The table gives the original Persian first step of each planner’s generated plan, with an English gloss. Only the two generalization-experiment planners (Section V-D ) recognize the inverted bowl as a prerequisite obstacle; the original Gemma-3-27B-IT run omits this step and proceeds directly to pouring.