EgoP$^{2}$C: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video

Chaoyu Li; Arka Sadhu; Yuliang Li; Hyo Jin Kim; Yuning Chai; Pooyan Fazli

cs.CVOct 15, 2026

EgoP $^{2}$ C: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video

Authors: Chaoyu Li, Arka Sadhu, Yuliang Li, Hyo Jin Kim, Yuning Chai, Pooyan Fazli

Organizations: Arizona State University · Meta

Abstract

Answering complex spatial reasoning questions from egocentric video requires determining what evidence is needed, acquiring targeted observations, and combining them through multiple reasoning steps. Existing vision-language models (VLMs) struggle to perform this process reliably. Agentic systems address this challenge by querying external vision and geometry tools, but training their reasoning leaves these tools unchanged, so inaccurate observations can still lead to incorrect answers. We introduce EgoP $^{2}$ C, a framework that internalizes a plan-perceive-compose process within a single VLM. To this end, we first train specialized Planner and Perception teachers using separate adapters (Dual-LoRA), then fine-tune the corresponding experts in a mixture-of-experts (MoE) student under their supervision. We subsequently train the student on trajectories generated by its current policy using a segment-level extension of multi-teacher on-policy distillation (MOPD), assigning different trajectory segments to different supervisors. The Planner teacher supervises which evidence-seeking questions to ask, when to stop gathering evidence, and how to compose the final answer. The Perception teacher supervises the video-grounded answers to those questions. At inference, the student executes this adaptive workflow end-to-end, without external spatial tools. Across three spatial reasoning benchmarks, EgoP $^{2}$ C students built from Qwen3-VL-8B and SenseNova-SI-1.3-8B outperform their base models by an average of 9.2 and 7.5 percentage points, respectively. Across both backbones, EgoP $^{2}$ C reduces inference FLOPs by 65.8-66.3% and achieves $1.29$ - $1.49\times$ speedups over their shared Dual-LoRA teacher.

Figures & tables

Figure 1: EgoP 2 C internalizes planning, perception, and composition within a single VLM for egocentric spatial reasoning. (a) A standard VLM reasons directly from the video and incorrectly concludes that the sofa fits between the objects. (b) An agentic system queries external vision and geometry tools to obtain the required measurements. (c) In EgoP 2 C, the Planner generates targeted questions and decides when enough evidence has been gathered, Perception answers those questions from the video, and the Planner composes the final answer.

Figure 2: Overview of EgoP 2 C. The framework integrates planning, perception, and composition into a single VLM. (a) Planner (PC) and Perception (V) teachers share a frozen backbone and use separate LoRA adapters trained with SFT followed by GRPO. (b) Teacher-generated sequences initialize corresponding MoE experts through separate role training followed by joint SFT. $z_{t}$ denotes the $t$ -th token. (c) Segment-level MOPD trains the student on its own interactions, with PC supervising subquestions and composition and V supervising observations. (d) At inference, the Planner expert either answers directly using the General expert ( DIRECT ) or iteratively requests subquestion answers from Perception before composing the final answer ( PLAN ).

Figure 3: Overview of the segment-level MOPD pipeline. Each training iteration (i) selects a training sample, (ii) generates a rollout with the MoE student model, (iii) assigns each segment to the corresponding frozen teacher, (iv) obtains teacher log probabilities for the sampled tokens, and (v) updates the student with a reverse-KL policy gradient. The example follows the PLAN path. Orange bars indicate the probabilities of the same sampled token under the student (top) and its assigned teacher (bottom).

#	Method	VSI-Bench	EgoProx	R3D-Bench	Avg.
Agentic Systems (w/ tools)
1	SpatialClaw Gemma4-31B ( Cho et al., 2026 )	48.5	47.9	40.7	45.7
2	Think3D Gemini 2.5 Pro ( Zhang et al., 2026b )	62.0	42.1	43.4	49.2
3	S-Agent Qwen3-VL-8B ( Dai et al., 2026 )	56.3	30.2	33.9	40.1
4	S-Agent Qwen3-VL-30B-A3B ( Dai et al., 2026 )	64.8	42.7	41.4	49.6
5	S-Agent GPT-5.4 ( Dai et al., 2026 )	66.7	49.8	45.9	54.1

Table 1: Performance on egocentric spatial reasoning benchmarks. Subscripts indicate the backbone or agent model. Green parentheses report gains in percentage points over the baseline row. Student gains on all three benchmarks have 95% bootstrap confidence intervals excluding zero (Appendix C.1 ).

Method	TFLOPs	Latency (s)	Generated tokens	Latency/token (ms)
Dual-LoRA SenseNova-SI	165.7	9.4	353.1	26.6
EgoP 2 C Qwen3-VL	55.9 (-66.3%)	7.3 (-22.3%)	315.4 (-10.7%)	23.1 (-13.2%)
EgoP 2 C SenseNova-SI	56.6 (-65.8%)	6.3 (-33.0%)	276.8 (-21.6%)	22.8 (-14.3%)

Table 2: Inference cost of Dual-LoRA and EgoP 2 C. Subscripts identify the starting checkpoints. Green parentheses report relative reductions from Dual-LoRA. FLOPs, latency, and generated tokens are averaged over all questions across the three benchmarks. Generated tokens include outputs from all roles.

Method	VSI-Bench	EgoProx	R3D-Bench	Avg.
Dual-LoRA teachers	67.6	49.0	44.8	53.8
EgoP 2 C	66.9	47.2	44.5	52.9
EgoP 2 C $+$ additional SFT	67.5	49.8	46.4	54.6

Table 3: Further supervised training of the student. All systems use the SenseNova-SI-1.3-8B backbone. Average is the unweighted mean across the three benchmarks.

Method	Training	VSI-Bench	EgoProx	R3D-Bench
Base	–	64.8	34.2	37.1
Joint LoRA	SFT	62.1	37.9	37.7
Perception-only LoRA	SFT	65.4	39.0	39.2
Planner-only LoRA	SFT	61.8	43.9	41.0
Dual-LoRA	SFT	65.8	46.7	43.2
Joint LoRA	SFT + GRPO	62.6	36.3	37.3

Table 4: Effect of joint and separate role training. All variants use SenseNova-SI-1.3-8B.

Configuration	VSI-Bench	EgoProx	R3D-Bench
Qwen3-VL-8B	57.9	31.0	35.2
Dense $+$ MOPD	58.4	36.3	32.8
Dense $+$ SFT $+$ MOPD	59.8	38.6	35.1
MoE $+$ MOPD	60.4	40.4	39.6
MoE $+$ role initialization	60.8	42.7	41.0
MoE $+$ role initialization $+$ offline distillation	61.3	43.3	41.2

Table 5: Effects of student architecture, role initialization, and distillation. Role initialization includes separate role SFT followed by joint SFT. Dense

+

SFT

+

MOPD matches the SFT data used for role initialization.

Figure 4: Student-teacher KL during MOPD. PC denotes planning and composition, and V denotes perception. The initialized MoE finishes with lower KL for both roles on both backbones.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	SFT		GRPO
Source	Perception	Planner	Perception	Planner
ADT	27,279	6,926	1,085	1,900
Ego-Exo4D	1,167	2,600	1,004	778
VSI-590K	5,502	3,777	535	896
SPAR-7M	1,863	1,792	536	298
Ego4D	–	2,722	–	–

Appendix

Table 6: Source composition of the teacher training sets. All values count samples, and dashes indicate no samples from that source.

Type	Samples	Type	Samples
Bearing	12,283	Count	1,330
Distance	11,339	Other	1,820
Camera distance	2,732	Identity	711
Size	2,609	Egomotion bearing	580
Object location	2,255	Egomotion distance	152

Appendix

Table 7: Question types in the Perception SFT set. Counts sum to 35,811 samples.

Component	Parameter	Value
LoRA alpha (teachers / student residuals)	$\alpha$	$32$
GRPO KL coefficient (Planner / Perception)	$\beta_{\mathrm{KL}}$	$0.04$
No-ground-truth reward scale	$\gamma$	$0.5$
Necessity (necessary / redundant / default)	$n_{\mathrm{pre}}$	$1.0,\ 0.1,\ 0.25$
Teacher-reference downweight	$m_{\mathrm{ref}}$	$0.5$
Missing-weight fallback	$n_{\mathrm{fallback}}$	$0.25$

Appendix

Table 8: Reward and training coefficients.

n_{\mathrm{pre}}

denotes the necessity weight before reference downweighting.

Backbone	Benchmark	Gain	95% CI
Qwen3-VL-8B	VSI-Bench	4.2	[+3.4, +5.0]
	EgoProx	14.9	[+12.7, +17.1]
	R3D-Bench	8.5	[+7.2, +9.7]
SenseNova-SI-1.3-8B	VSI-Bench	2.1	[+1.6, +2.6]
	EgoProx	13.0	[+10.8, +15.3]
	R3D-Bench	7.4	[+6.3, +8.5]

Appendix

Table 9: Student improvements over their corresponding base models and 95% paired bootstrap confidence intervals. Gains and intervals are in percentage points.

		Qwen3-VL-8B			SenseNova-SI-1.3-8B
Subtask	$n$	Base	EgoP 2 C	$\Delta$	Base	EgoP 2 C	$\Delta$
Object Relative Direction	968	57.23	58.88	$+1.65$	63.33	63.74	$+0.41$
Object Size Estimation	953	84.66	85.04	$+0.38$	84.93	85.05	$+0.12$
Object Absolute Distance	834	42.49	49.65	$+7.16$	45.97	50.36	$+4.39$
Object Relative Distance	710	48.03	53.66	$+5.63$	59.15	62.39	$+3.24$
Object Appearance Order	618	69.42	70.23	$+0.81$	74.60	74.92	$+0.32$

Appendix

Table 10: Results by subtask on VSI-Bench. Scores (%) use mean relative accuracy (MRA) for numerical questions and accuracy for multiple-choice questions.

n

is the number of evaluated questions. Bold indicates student scores.

\Delta

is the improvement over the corresponding base.

		Qwen3-VL-8B			SenseNova-SI-1.3-8B
Subtask	$n$	Base	EgoP 2 C	$\Delta$	Base	EgoP 2 C	$\Delta$
Exploitation Relative	693	34.63	46.75	$+12.12$	34.92	49.64	$+14.72$
Exploitation Approx	422	37.68	46.21	$+8.53$	41.47	47.16	$+5.69$
Intention Approx	386	32.64	42.23	$+9.59$	41.45	49.48	$+8.03$
Intention Relative	342	29.24	52.34	$+23.10$	30.70	50.58	$+19.88$
Exploration Relative	191	17.28	46.60	$+29.32$	15.71	39.79	$+24.08$

Appendix

Table 11: Results by subtask on EgoProx. Scores are multiple-choice accuracy (%).

n

is the number of evaluated questions. Bold indicates student scores.

\Delta

is the improvement over the corresponding base.

		Qwen3-VL-8B			SenseNova-SI-1.3-8B
Subtask	$n$	Base	EgoP 2 C	$\Delta$	Base	EgoP 2 C	$\Delta$
Total Walk Distance	228	6.36	19.08	$+12.72$	7.28	19.08	$+11.80$
Total Fly Distance	228	9.56	27.28	$+17.72$	10.04	24.82	$+14.78$
Global How Far	228	27.19	39.69	$+12.50$	33.25	44.04	$+10.79$
Global How Far From Me	228	25.57	34.30	$+8.73$	33.99	43.77	$+9.78$
Global How Long	228	48.95	62.06	$+13.11$	63.38	72.15	$+8.77$

Appendix

Table 12: Results by subtask on R3D-Bench. Scores (%) use mean relative accuracy (MRA) for numerical questions and accuracy for multiple-choice questions.

n

is the number of evaluated questions. Bold indicates student scores.

\Delta

is the improvement over the corresponding base.

	VSI-Bench		EgoProx		R3D-Bench
Method	Score	$\Delta$ [95% CI]	Score	$\Delta$ [95% CI]	Score	$\Delta$ [95% CI]
Base	64.8	–	34.2	–	37.1	–
Direct-answer SFT	65.4	–	39.8	–	39.3	–
Direct-answer SFT + GRPO	65.5	–	40.7	–	40.1	–
Dual-LoRA teachers	67.6	$+2.1$ [ $+1.8$ , $+2.6$ ]	49.0	$+8.3$ [ $+6.2$ , $+10.5$ ]	44.8	$+4.7$ [ $+3.3$ , $+6.1$ ]
EgoP 2 C	66.9	$+1.4$ [ $+0.7$ , $+2.2$ ]	47.2	$+6.5$ [ $+4.6$ , $+8.5$ ]	44.5	$+4.4$ [ $+3.2$ , $+5.6$ ]

Appendix

Table 13: Comparison with direct-answer training. For Dual-LoRA and EgoP 2 C,

\Delta

reports score differences in percentage points relative to Direct-answer SFT+GRPO, with 95% paired bootstrap confidence intervals.

Method	TFLOPs	Latency (s)	Generated tokens	Latency/token (ms)
Subset matched to the Qwen3-VL student
Dual-LoRA SenseNova-SI	162.1	8.8	298.4	29.5
EgoP 2 C Qwen3-VL	51.0	5.8	248.5	23.3
Subset matched to the SenseNova-SI student
Dual-LoRA SenseNova-SI	156.3	8.6	277.5	31.0
EgoP 2 C SenseNova-SI	49.7	4.9	219.0	22.4

Appendix

Table 14: Inference cost with matched subquestion counts. Each group compares one student with Dual-LoRA on its own matched subset. Metrics follow the definitions in Table 2 .

Figure 5: Gradient alignment between planning and composition (PC) and perception (V) in the dense Qwen student. Each gray point represents the gradient cosine similarity for one pair of PC and V samples. Blue markers show the mean over 32 pairs at each checkpoint. The same sample pairs are evaluated across all 12 checkpoints.

Panel of Figure 6 — Figure 6: Prompt template for generating PLAN demonstrations with DeepSeek-V4-Pro. The Planner requests spatial observations or composes an answer from the returned evidence.

Panel of Figure 7 — Figure 7: Prompt template for Perception answer generation with Qwen3.5-397B. The model receives video frames and one self-contained spatial subquestion, without the original answer options or planning history.

Panel of Figure 8 — Figure 8: Planner prompt for Dual-LoRA inference. The Planner initially chooses direct answering or perception requests. After gathering observations, it composes the answer in its final assistant response.

Panel of Figure 9 — Figure 9: Perception prompt for Dual-LoRA inference. Each call contains the video, one self-contained subquestion, and an optional answer format instruction. Perception receives no original answer options or planning history.

Panel of Figure 10 — Figure 10: Student prompt template. All roles share a video prefix and receive separate text contexts. The call syntax denotes internal role dispatch: ask_perception activates V, and answer_with_base activates G. After observations return, PC can request further evidence or compose the final answer. DIRECT and PLAN label the routes; they are not additional outputs in this template.

Figure 11: Illustrative comparisons on R3D-Bench (top) and EgoProx (bottom). Red text marks incorrect reasoning and answers, while green text marks correct ones.

Visual Spatial Reasoning

Visual spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and physical change from images and video.

See all 244 papers

Nov 23, 2025

EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

Egocentric video understanding requires procedural reasoning under partial observability and continuously shifting viewpoints. Current multimodal large language models (MLLMs) struggle with this setting, often generating plausible but visually inconsistent or weakly grounded responses. We introduce

\textbf{EgoVITA}

, a framework that decomposes egocentric video reasoning into a structured

\textit{plan-then-verify}

process. The model first generates an

\textbf{egocentric plan}

: a causal sequence of anticipated actions from a first-person perspective. This plan is then evaluated by an

\textbf{exocentric verification}

stage that uses third-person reasoning over the same video to verify its spatiotemporal and logical consistency, without exocentric video input. This decomposition enables cross-perspective feedback without requiring paired ego-exo supervision. To train this reasoning process, we adopt Group Relative Policy Optimization (GRPO) with two dense reward signals: one that grounds anticipated actions in subsequent visual observations and another that reinforces consistent third-person verification.

\textbf{EgoVITA}

achieves state-of-the-art performance on egocentric reasoning benchmarks, outperforming Qwen2.5-VL-7B by

\mathbf{+7.7}

on EgoBlind and

\mathbf{+4.4}

on EgoOrient, while maintaining strong generalization on exocentric video tasks with only

52k

training samples.

Yogesh Kulkarni, Pooyan Fazli

Arizona State University

Egocentric Video Understanding Video Reasoning Visually Grounded Reasoning LLM Reasoning

Ask PDF

Jun 10, 2026

Reason, Then Re-reason: Cross-view Revisiting Improves Spatial Reasoning

Spatial reasoning from egocentric videos is inherently challenging because the observable evidence is constrained by the camera trajectory. Existing methods rely on single-turn inference, forcing models to resolve geometric ambiguity through semantic priors rather than verifiable evidence. We argue that spatial reasoning should be revisitable: conclusions formed under limited evidence should remain open to revision when complementary viewpoints become available. Building on this insight, we propose Reason, then Re-reason (ReRe), a training-free, inference-time framework with two phases: in the Reason Phase, an MLLM forms a spatial hypothesis from the original video; in the Re-reason Phase, it verifies or revises the hypothesis by observing a synthesized novel-view video. To enable effective cross-view revisiting, we design a Geometry-to-Video pipeline that renders strategically complementary novel views from predicted 3D geometry. These views feature an elevated, oblique perspective with scene-spanning coverage, while preserving the MLLM's native video interface without architectural modifications. Extensive evaluations on VSI-Bench and STI-Bench demonstrate that ReRe substantially boosts open-source MLLMs to rival proprietary state-of-the-art performance. Project page: https://zhenjiemao.github.io/ReRe/

Chaofan Ma, Zhenjie Mao, Yuhuan Yang +5

Cooperative Medianet Innovation Center, Shanghai Jiao Tong University · Shanghai Jiao Tong University · Tongji University

Visual Spatial Reasoning 3D Spatial Reasoning Egocentric Video Understanding Visually Grounded Reasoning Spatial Reasoning Multimodal Spatial Reasoning

Ask PDF

May 14, 2026

Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding

Video reasoning models are a core component of egocentric and embodied agents. However, standard benchmarks for assessing models provide only evaluation of the output (e.g. the answer to a question), without evaluation of intermediate reasoning steps, and most provide answers only in the text domain. We introduce Minerva-Ego, a benchmark for evaluating complex egocentric visual reasoning. We extend recent high-quality video data sources recorded from egocentric / embodied settings with a set of challenging, multi-step multimodal questions and spatiotemporally-dense human-annotated reasoning traces. Benchmarking experiments show that state-of-the-art models still have a large gap to human performance. To investigate this gap in detail, we annotate each reasoning trace in the dataset with the objects of interest required to solve the question, as spatiotemporal mask annotations. Through extensive evaluations, we identify that prompting frontier models with hints of 'where' and 'when' to look yields substantial improvements in performance. Minerva-Ego can be downloaded at https://github.com/google-deepmind/neptune.

Arsha Nagrani, Jasper Uijilings, Shyamal Buch +6

Google DeepMind

Egocentric Video QA Egocentric Video Understanding Multimodal QA Video Reasoning Multimodal CoT Reasoning Visually Grounded Reasoning

Ask PDF

Abstract

Visual Spatial Reasoning

Explore similar work

EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

Reason, Then Re-reason: Cross-view Revisiting Improves Spatial Reasoning

Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding