Benchmarking Interaction, Beyond Policy: a Reproducible Benchmark for Collaborative Instance Object Navigation

Edoardo Zorzi; Francesco Taioli; Yiming Wang; Marco Cristani; Alessandro Farinelli; Alberto Castellini; Loris Bazzani

cs.CVMar 31, 2026

Benchmarking Interaction, Beyond Policy: a Reproducible Benchmark for Collaborative Instance Object Navigation

Authors: Edoardo Zorzi, Francesco Taioli, Yiming Wang, Marco Cristani, Alessandro Farinelli, Alberto Castellini, Loris Bazzani

Organizations: Sapienza University of Rome, Rome, Italy · University of Verona, Verona, Italy · Polytechnic of Turin · Fondazione Bruno Kessler ·

Abstract

We propose Question-Asking Navigation (QAsk-Nav), the first reproducible benchmark for Collaborative Instance Object Navigation (CoIN) that enables an explicit, separate assessment of embodied navigation and collaborative question asking. CoIN tasks an embodied agent with reaching a target specified in free-form natural language under partial observability, using only egocentric visual observations and interactive natural-language dialogue with a human, where the dialogue can help to resolve ambiguity among visually similar object instances. Existing CoIN benchmarks are primarily focused on navigation success and offer no support for consistent evaluation of collaborative interaction. To address this limitation, QAsk-Nav provides (i) a lightweight question-asking protocol scored independently of navigation, (ii) an enhanced navigation protocol with realistic, diverse, high-quality target descriptions, and (iii) an open-source dataset, that includes 28,000 quality-checked reasoning and question-asking traces for training and analysis of interactive capabilities of CoIN models. Using the proposed QAsk-Nav benchmark, we develop Light-CoNav, a lightweight unified model for collaborative navigation that is 3x smaller and 70x faster than existing modular methods, while outperforming state-of-the-art CoIN approaches in generalization to unseen objects and environments. Project page at https://benchmarking-interaction.github.io/

Figures & tables

Figure 1: QAsk-Nav introduces two distinct protocols for question asking (top-left) and for navigation (top-right), supported by a novel dataset with structured reasoning, and question annotations (bottom). Our dataset enables reproducible evaluation, training, and analysis under both protocols to study the agent’s capabilities of interaction reasoning and question-asking.

Panel of Figure 2 — Figure 2: Examples from QAsk-Nav. Left: original image. Center: distractor with an altered sofa and painting colors. Right: distractor with an altered plant and painting.

Figure 3: Episode from QAsk-Nav. Column 1: the target image available only to the oracle and the navigation instructions provided to the agent. Columns 2-3: different observations, including image, question, and answer from the oracle. Before each question and conclusion, the model produces a reasoning trace, motivating the decision. More detailed examples in Supp. Mat. Sec. 3 .

Figure 4: Updated Task Annotations . Comparison between CoIN-Bench [ 41 ] annotations and our QAsk-Nav annotations. Old annotations often contain confusing, nonsensical, or hallucinated descriptions, while new annotations are clearer and more accurate.

Figure 5: Two samples $(D,O,R,S,Q,C)$ from QAsk-Nav-dataset. In the first example, the description $D$ matches the observation $O$ and thus the reasoning $R$ and score $S$ indicate a match (between $D$ and $O$ ). In the second example, the description is ambiguous, and previous answers did not help: therefore $R$ and $S$ express uncertainty.

Method	Navigation ready	SR $\uparrow$	FR $\uparrow$	NQ $\downarrow$	Time $\downarrow$
Qwen2.5-VL-7B [ 3 ]	✔	0.0752	0.0034	2.7405	1.70
AIUTA ∘ [ 41 ]	✔	0.3027	0.1993	1.1473	177.83
Light-CoNav ∘ (ours)	✔	0.4510	0.3317	0.1990	1.15
Qwen3-VL-30B-A3B [ 2 ]	✗	0.3205	0.2755	0.1174	1.58
Gemini-3-flash [ 8 ]	✗	0.9329	0.8943	0.9564	13.20

Table 1: Performance comparison of methods on the QAsk-Nav benchmark. The top section shows navigation-ready models (parameter size under 20B), while the bottom section includes larger or closed-source baselines.

	Cat		CatCol		ColFeat		Ctx		ColCtx		ColCtxFeat
Method	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$
AIUTA ∘ [ 41 ]	0.42	0.28	0.34	0.22	0.30	0.19	0.32	0.22	0.24	0.15	0.20	0.13
Light-CoNav ∘ (ours)	0.06	0.03	0.45	0.31	0.58	0.44	0.43	0.32	0.61	0.42	0.68	0.54
Qwen3-VL-30B-A3B [ 2 ]	0.02	0.01	0.24	0.18	0.40	0.34	0.34	0.29	0.47	0.41	0.54	0.47

Table 2: Success Rate ( SR

\uparrow

) and Finish Rate ( FR

\uparrow

) across different annotation types of QAsk-Nav for selected methods.

Method	Features	Val Seen				Val Seen Synonyms				Val Unseen
Method	Interaction		SR $\uparrow$	SPL $\uparrow$	NQ $\downarrow$		SR $\uparrow$	SPL $\uparrow$	NQ $\downarrow$		SR $\uparrow$	SPL $\uparrow$	NQ $\downarrow$
SenseAct-NN [ 14 ]	✗		6.62	3.11	-		13.09	6.45	-		0.22	0.05	-
PSL [ 39 ]	✗		8.78	3.30	-		8.91	2.83	-		4.58	1.39	-
OVON [ 50 ]	✗		8.18	5.24	-		15.88	11.35	-		2.61	1.29	-
VLFM [ 49 ]	✗		0.36	0.28	-		0.00	0.00	-		0.00	0.00	-
AIUTA [ 41 ]	✔		10.46	4.96	1.31		14.89	9.11	1.48		7.83	4.38	1.37

Table 3: Results of the navigation task on the CoIN benchmark with our QAsk-Nav annotations.

Method	Ann. Type	Val Seen	Val Seen Synonyms	Val Unseen	Time $\downarrow$	Size $\downarrow$
AIUTA [ 41 ]	CoIN-Bench	7.42	14.38	6.67	185.79	20B
AIUTA [ 41 ]	Ours	10.46 (1.41x)	14.89 (1.04x)	7.83 (1.17x)	185.79	20B
Light-CoNav	CoIN-Bench	9.14	11.64	6.43	2.49	7B
Light-CoNav	Ours	12.20 (1.33x)	14.15 (1.33x)	9.63 (1.50x)	2.49	7B

Table 4: Impact of annotations quality, inference time, and parameter size. The new annotations improve the results across all splits. We report the SR

\uparrow

across all splits, the average inference time during interaction ( Time

\downarrow

), and model size ( Size

\downarrow

Oracle	Cat		CatCol		ColFeat		Ctx		ColCtx		ColCtxFeat
Human	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$
✗	0.93	0.90	0.92	0.88	0.93	0.90	0.90	0.83	0.95	0.93	0.96	0.93
✔	0.92	0.88	0.90	0.85	0.92	0.87	0.90	0.86	0.96	0.94	0.94	0.87

Table 5: Comparison between VLM-based and human as oracle. The questioner is Gemini-3-flash [ 8 ] . In the first row, we use three different VLMs as oracles following the same evaluation in the main manuscript. In the second row, we task humans to answer the questions in real-time while using Light-CoNav.

	Cat		CatCol		ColFeat		Ctx		ColCtx		ColCtxFeat
Method	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$	SR $\uparrow$	FR $\uparrow$
Qwen2.5-VL-7B [ 3 ]	0.02	0.0	0.04	0.0	0.13	0.01	0.05	0.01	0.08	0.0	0.13	0.01
AIUTA ∘ [ 41 ]	0.42	0.28	0.34	0.22	0.30	0.19	0.32	0.22	0.24	0.15	0.20	0.13
Light-CoNav ∘	0.06	0.03	0.45	0.31	0.58	0.44	0.43	0.32	0.61	0.42	0.68	0.54
Qwen3-VL-30B-A3B [ 2 ]	0.02	0.01	0.24	0.18	0.40	0.34	0.34	0.29	0.47	0.41	0.54	0.47
Gemini-3-flash [ 8 ]	0.93	0.90	0.92	0.88	0.93	0.90	0.90	0.83	0.95	0.93	0.96	0.93

Table 6: Success Rate ( SR

\uparrow

) and Finish Rate ( FR

\uparrow

) across different annotation types of QAsk-Nav. See main manuscript, Sec. 3.2 for details about the annotation types.

Evaluator	Task Descriptions	Preference Rate $\uparrow$	Problem Rate $\downarrow$
LLM as judge	CoIN-Bench	0.171	0.618
LLM as judge	Ours	0.829	0.020
Humans	CoIN-Bench	0.183	0.530
Humans	Ours	0.816	0.056

Table 7: Comparison between CoIN-Bench [ 41 ] and our task descriptions evaluated by LLM-as-judge ( Gemini-3-pro ) and humans. Our task descriptions are highly preferred and exhibit a very low rate of problematic description concerning hallucination or imprecise details (2%). In contrast, the CoIN-Bench descriptions are rarely preferred with a high problem rate (over 50%).

Figure 6: Tool for the human oracle tests. The target image is presented to the user, along with the task description (the title), and the current question asked by the questioner (below the title). The user is instructed to answer by writing the answer in the text box below.

Figure 7: Evaluation tool for preference testing in our human evaluation. Two choices are presented to the user. The users are instructed to click on the checkbox “Problem with Description X” if they find hallucinations and clear mistakes.

Figure 8: Length of descriptions, in words. The CoIN-Bench descriptions exhibit extreme outliers and, in general, are longer and wordier than our descriptions.

Figure 9: Distribution of the target instance objects categories, per split, for the navigation episodes.

Figure 10: Categories distribution (top 30) of the target image, and lengths of episodes (in number of distractors; dotted line is the mean length), for the question-asking protocol (left: Train split, right: Validation split). The categories are well distributed, with the top 30 having at least 16 (for the train split) or 6 (for the validation split) examples.

Panel of Figure 11 — Figure 11: Variability of images used in our question-asking protocol. The first image is the original image; the next three are variations obtained using a strong generative model ( Gemini-3-pro-image [ 10 ] )

Panel of Figure 12 — Figure 12: Variability of images used in our question-asking protocol (cont’d).

Panel of Figure 13 — Figure 13: Distribution of the object categories in QAsk-Nav-dataset (left) and distribution of scores (right), divided by split.

Figure 14: Ablation of different models’ performance after the first training phase on QAsk-Nav-dataset. Performance is measured with the mean absolute error between the output score and the ground truth score in the dataset. After this phase, a model can only return

R

and

S

Figure 15: A CoIN episode. Light-CoNav is looking for a ‘blue painting’ ( $D$ ). In the first two detections, $(1)$ and $(2)$ , it correctly evaluates the observations, concluding that they are not the target object. $(3)$ is a difficult observation, as it looks like a painting. However, asking a question to the user disambiguates it. $(4)$ is the target object.

Panel of Figure 16 — Figure 16: Different types of generated descriptions.

Figure 17: Different types of generated descriptions (cont’d).

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

Ask When It Pays: Cost-Aware Open-Ended Interaction for Instance Goal Navigation
Jun 2, 2026Xunyi Zhao, Sihao Lin, Gengze Zhou +5Embodied Navigation AI Agent Benchmarks
1Adelaide University · 2Responsible AI Research Centre, Australian Institute for Machine Learning · Institute for Infocomm Research (I2R), A*STAR +2
ProCompNav: Proactive Instance Navigation with Comparative Judgment for Ambiguous User Queries
May 7, 2026Junhyuk Kwon, Seungjoon Lee, Hyejin Park +2Clarification Question Generation Vision-Language Navigation
GSAI, POSTECH · CSE, POSTECH · Oracle
CovNav: Conformal Prediction-Guided Adaptive Inference for Object Goal Navigation
Oct 20, 2026Pengying Wu, Zhejia Wen, Kang Ding +5Object-Goal Navigation Adaptive Reasoning in Language Models
School of Advanced Manufacturing and Robotics, Peking University, Beijing 100871, China · School of Engineering, Newcastle University, UK · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China

Jun 2, 2026J/K move · Enter open · S save

Ask When It Pays: Cost-Aware Open-Ended Interaction for Instance Goal Navigation

Xunyi Zhao, Sihao Lin, Gengze Zhou, Zerui Li+4

1Adelaide University · 2Responsible AI Research Centre, Australian Institute for Machine Learning · Institute for Infocomm Research (I2R), A*STAR+2

Embodied Navigation AI Agent Benchmarks Vision-Language Navigation Information Gain

Open paper AskPDF arXiv

Instance Goal Navigation (IGN) requires an embodied agent to find a specific object instance among distractors from an under-specified natural-language description. Such ambiguity often cannot be resolved from perception and language alone, making interaction with an oracle a natural mechanism for disambiguation. Prior interactive methods allow oracle queries but treat lightweight clarification and route-level guidance alike, letting agents boost success rate through repeated high-information questions rather than by resolving the underlying ambiguity efficiently. We recast interactive IGN as a cost-sensitive uncertainty-reduction problem, where the agent should ask the question whose answer provides the largest reduction in navigation uncertainty relative to its penalty. To this end, we apply an information-gain analysis on existing navigation corpora to identify which cues reduce navigation uncertainty, yielding a compact set of question types and data-derived weights. However, existing interactive navigation benchmarks do not model the cost of different question types or evaluate how efficiently agents use interaction, making them unsuitable for studying cost-sensitive interaction. Based on this taxonomy, we construct a benchmark for diagnosing interaction behavior and efficiency, together with a Weighted Success Rate metric that penalizes each query by its derived cost. We further propose a zero-shot MLLM navigator that selectively queries at each decision step only when the expected uncertainty reduction justifies the interaction cost.

Abstract

AI Agent Benchmarks

Explore similar work

Ask When It Pays: Cost-Aware Open-Ended Interaction for Instance Goal Navigation

ProCompNav: Proactive Instance Navigation with Comparative Judgment for Ambiguous User Queries

CovNav: Conformal Prediction-Guided Adaptive Inference for Object Goal Navigation