CovNav: Conformal Prediction-Guided Adaptive Inference for Object Goal Navigation

Pengying Wu; Zhejia Wen; Kang Ding; Kangjie Zhou; Chunxuan Jiao; Wei Pan; Xiangli Nie; Chang Liu

cs.ROOct 20, 2026

CovNav: Conformal Prediction-Guided Adaptive Inference for Object Goal Navigation

Authors: Pengying Wu, Zhejia Wen, Kang Ding, Kangjie Zhou, Chunxuan Jiao, Wei Pan, Xiangli Nie, Chang Liu

Organizations: School of Advanced Manufacturing and Robotics, Peking University, Beijing 100871, China · School of Engineering, Newcastle University, UK · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China

Abstract

In object goal navigation (ObjectNav), an embodied agent must find a target object (e.g., "find a bed") in a previously unseen environment, relying on spatial commonsense such as object co-occurrence. Large Language Models (LLMs) encode this commonsense, but deploying them on robots exposes a trade-off between reasoning quality and efficiency: cloud-based models incur latency and privacy risks, whereas small on-device LLMs are fast but prone to hallucinated spatial judgments. Hybrid reasoning LLMs offer both a fast non-thinking mode and a slow thinking mode, which raises a key question for embodied agents: when is slow thinking worth its cost? We propose CovNav, a training-free framework that answers this question with Conformal Prediction (CP), turning dual-process (System 1 / System 2) inference into an uncertainty-aware decision. Much like recent decision models such as Jev, CovNav casts each spatial query as a structured binary decision rather than free-form generation: a local LLM in fast mode judges whether the target is likely near an observed object, CP converts its token probabilities into a prediction set, and only ambiguous queries are escalated to slow chain-of-thought reasoning. The object-pair formulation keeps calibration tractable and supports open-set objects through off-the-shelf detectors. Experiments on Gibson and HM3D show that CovNav surpasses the strongest zero-shot baseline by 0.8 and 1.7 percentage points in success rate on HM3D v1 and v2, while invoking slow reasoning for only 12.4% of queries and retaining most of the gain of always-on thinking at a fraction of its latency. Real-robot experiments on a TurtleBot 4 further validate the approach. CovNav shows how calibrated uncertainty can decide when an embodied agent should think slowly, offering a principled form of adaptive test-time reasoning for LLM-based robots.

Figures & tables

Fig. 1: We propose a Conformal Prediction (CP)-guided fast-slow inference paradigm for object goal navigation using Qwen $3$ - $8$ B as the local LLM. CP monitors the local LLM’s output confidence to adaptively switch Qwen between its fast direct inference mode and slow deliberative reasoning mode. Clear Case: When observing a bathtub, Qwen confidently predicts a toilet will be nearby (logits: 0.9 vs 0.1), so CP accepts the fast direct inference. Ambiguous Case: When observing a sofa and querying about a bed, Qwen shows low confidence (logits: 0.6 vs 0.4), so CP triggers its slow deliberative reasoning mode for more reliable spatial analysis.

Fig. 2: Overview of the CovNav framework. The agent incrementally builds an object-based semantic map from RGB-D observations. To efficiently infer object-goal relationships, we introduce a Fast-Slow inference mechanism: CP dynamically evaluates the uncertainty of the local LLM operating in Fast direct inference mode. Confident predictions (singleton sets) are directly utilized, while ambiguous cases trigger the same model in Slow deliberative reasoning mode. The resulting semantic scores are fused into a global value map to guide the agent toward the most promising exploration frontiers.

Fig. 3: Word cloud visualization of object frequencies across four room types. Larger font sizes indicate higher frequency of occurrence, highlighting the characteristic components validation.

Fig. 4: Visualization of the adaptive inference process (Target: Bed). At Step 14, the agent encounters semantic ambiguity while observing a couch. The CP module detects high uncertainty, triggering the Slow Mode to perform chain-of-thought reasoning (inferring “Living Room” from “Couch” and “Bedroom” from “Bed”), thereby guiding the agent to the correct room. At Step 112, the Fast Mode effectively filters out irrelevant areas (Bathroom) with high confidence, leading to a successful identification of the target at Step 119. The bottom row illustrates the real-time evolution of the Global Semantic Value Map (visualized as a red-green heatmap) and the agent’s trajectory.

Method	Planner	Training-free	Core Model	Gibson		HM3Dv1		HM3Dv2
Method	Planner	Training-free	Core Model	Success $\uparrow$	SPL $\uparrow$	Success $\uparrow$	SPL $\uparrow$	Success $\uparrow$	SPL $\uparrow$
L3MVN	Semantic	X	LLM	76.9	38.8	38.4	19.4	36.3	15.7
VLFM		$\checkmark$	VLM	81.8	46.5	50.9	23.6	56.9	27.5
GAMap		$\checkmark$	VLM	85.7	55.5	53.1	26.0	-	-
ApexNav		$\checkmark$	VLM	-	-	59.6	33.0	76.2	38.0
CovNav (Ours)		$\checkmark$	LLM	82.4	48.6	60.4	33.1	77.9	40.2

TABLE I: Comparison with ZSON Baselines.

Method	Gibson	HM3D v1	HM3D v2	Efficiency
Method	SR $\uparrow$ / SPL $\uparrow$	SR $\uparrow$ / SPL $\uparrow$	SR $\uparrow$ / SPL $\uparrow$	Time (s)	Call (%)
Fast-Only (System 1)	72.5 / 38.2	51.5 / 25.8	69.8 / 34.2	0.63	-
Slow-Only (System 2)	83.1 / 49.0	64.2 / 31.5	82.5 / 44.8	3.40	100
Random Switch	76.8 / 42.5	57.1 / 29.8	75.4 / 38.1	1.25	53.0
CovNav (Adaptive)	79.5 / 43.7	62.8 / 32.8	78.5 / 39.9	0.72	12.4

TABLE II: Ablation Study of Fast-Slow Inference Mechanism. Comparison of navigation performance and inference efficiency on Gibson and HM3D datasets.

$\alpha$	Switch (%)	Gibson	HM3D v1	HM3D v2
$\alpha$	Switch (%)	SR $\uparrow$ / SPL $\uparrow$	SR $\uparrow$ / SPL $\uparrow$	SR $\uparrow$ / SPL $\uparrow$
0.8	22.4	82.9 / 48.8	61.5 / 31.9	79.2 / 39.1
0.5 (Ours)	12.4	79.5 / 43.7	62.8 / 32.8	78.5 / 39.9
0.2	8.2	80.1 / 46.5	59.4 / 30.1	75.8 / 37.5

TABLE III: Impact of Conformal Prediction Parameter

\alpha

. Evaluating the trade-off between switching rate and navigation performance across datasets.

Threshold	Switch (%)	Gibson	HM3D v1	HM3D v2
Threshold	Switch (%)	SR $\uparrow$ / SPL $\uparrow$	SR $\uparrow$ / SPL $\uparrow$	SR $\uparrow$ / SPL $\uparrow$
0.8	42.7	82.0 / 45.7	54.2 / 26.5	80.4 / 41.6
0.5	10.1	77.8 / 44.5	60.9 / 31.4	77.1 / 38.2
0.2	8.2	74.2 / 39.8	57.4 / 28.6	74.6 / 36.4
CovNav (CP)	12.4	79.5 / 43.7	62.8 / 32.8	78.5 / 39.9

TABLE IV: Performance with Fixed Decision Thresholds. Comparison of decision-making without uncertainty quantification.

Method	Gibson	HM3D v1	HM3D v2
Method	SR $\uparrow$ / SPL $\uparrow$	SR $\uparrow$ / SPL $\uparrow$	SR $\uparrow$ / SPL $\uparrow$
w/o CP-calibrated Value	52.5 / 25.2	41.2 / 21.9	66.8 / 28.5
w/o Confidence Decay	80.2 / 46.8	51.3 / 25.8	65.4 / 31.2
w/o Open-set Perception	68.4 / 35.1	54.6 / 27.5	69.5 / 34.1
CovNav (Ours)	79.5 / 43.7	62.8 / 32.8	78.5 / 39.9

TABLE V: Ablation of System Components. Analyzing the contribution of different modules.

Module	Model Size	VRAM (GB)	Latency (ms)
Perception (BLIP&YOLO)	125M	4.2	45
Fast Mode (Qwen3-8B-FP8)	8B	8.0	180
GroundedSAM	172M	3.8	300
Slow Mode (Thinking Enabled)	8B	8.0	2400
CovNav (Avg)	-	22.6	680

TABLE VI: Computational Cost Analysis. Comparison of resource usage and latency across modules.

Fig. 5: Comparison of Logit Distributions. (a) Uncalibrated logits from the Fast Mode showing high uncertainty. (b) Calibrated logits after adaptive slow-mode deliberation, showing a clean separation and higher confidence in the correct association.

Dataset	Perception	Navigation	Inference
Gibson	45.5%	35.2%	19.3%
HM3D v1	52.3%	28.5%	19.2%
HM3D v2	48.8%	30.0%	21.2%

TABLE VII: Distribution of Failure Modes. Values show percentage of failed episodes.

Fig. 6: Visualization of Real World Experiments. The figure depicts the navigation trajectory of the TurtleBot 4 platform (setup shown in top-left) searching for a toy. The system dynamically switches between Fast and Slow modes based on uncertainty. During the initial approach (Steps 11-27), the agent operates in Fast Mode for efficient exploration. At Step 42, high uncertainty at a semantic transition triggers the Slow Mode , enabling the agent to infer the correct path into the bedroom. The agent then reverts to Fast Mode (Step 60) and successfully localizes the target (Step 73). This demonstrates the efficacy of the adaptive fast-slow inference mechanism in handling real-world ambiguities.

Object-Goal Navigation

Object-goal navigation studies how robots find and approach requested objects in unfamiliar environments using visual, spatial, and language cues.

See all 86 papers

LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation
Aug 31, 2026Shaoan Wang, Aocheng Luo, Fei Huang +17Open-Vocabulary Object Navigation Cross-Embodiment Robot Learning
Light Origins Team
SuperNav: An Agentic Navigation System for Any Task in Any Scene
Oct 8, 2026Jinkai Zhang, Jingyi Xu, Yuanhong Yu +5Robot Navigation Visual Robot Navigation
Zhejiang University · Shenzhen University · Causa Robotics
Spatial and Semantic Reasoning for LLM-Driven Robot Navigation via MCP
Sep 23, 2026Jungsoo Lee, Jaegyun Park, Wansoo KimRobot Navigation Large Language Model-Based Robot Planning
Department of Robotics, Hanyang University, Seoul, 04763, Korea · Department of Robotics, Hanyang University ERICA, Ansan, 15588, Korea

Aug 31, 2026J/K move · Enter open · S save

LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation

Shaoan Wang, Aocheng Luo, Fei Huang, Jingyi Xu+16

Light Origins Team

Open-Vocabulary Object Navigation Cross-Embodiment Robot Learning VLMs for Robotics Embodied Navigation VLM Reasoning Vision-Language-Action Models+1

Open paper AskPDF arXiv

Embodied navigation requires agents to translate heterogeneous goals and visual observations into actions across tasks, environments, and robot embodiments. Modern vision-language models (VLMs) already encode spatial priors for visual grounding, spatial reasoning, and pointing, but these capabilities are rarely elicited directly for robot control. Existing navigation systems instead rely on task- or embodiment-specific components, fragmenting perception, reasoning, and action while offering limited generalization. Here we present LightNav-0, a compact generalist embodied navigation model that elicits the spatial intelligence of a pretrained VLM and aligns it with navigation, without task-specific prediction heads. LightNav-0 represents diverse navigation tasks through a unified token interface: dual-channel pointing expresses task-, scene-, and embodiment-agnostic spatial intent, while a residual vector-quantized action tokenizer maps this intent to precise, embodiment-specific trajectories. Together with temporally aware visual history compression, ER mid-training, supervised fine-tuning, and reinforcement learning, this formulation supports instruction following, open-vocabulary object navigation, and visual tracking within a single model. The navigation training corpus spans 2K+ scenes and 4K+ hours of embodied navigation data. LightNav-ER, the embodied-reasoning checkpoint used to initialize LightNav-0, attains the highest complete-set average across 8 embodied-reasoning benchmarks, while LightNav-0 achieves state-of-the-art monocular success rates across all 10 public navigation simulation settings. Real-world evaluations further demonstrate zero-shot generalization across robot embodiments, diverse scenes, and static and dynamic targets. These results establish compact VLMs as a unified and transferable backbone for generalist embodied navigation.

Abstract

Object-Goal Navigation

Explore similar work

LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation

SuperNav: An Agentic Navigation System for Any Task in Any Scene

Spatial and Semantic Reasoning for LLM-Driven Robot Navigation via MCP