EgoTSR++: Egocentric Spatiotemporal Reasoning for Task Progress Understanding

Xiaoda Yang; Can Wang; Yuxiang Liu; Pengfei Zhou; Jianwen Lou; Shuicheng Yan

cs.CVSep 23, 2026

EgoTSR++: Egocentric Spatiotemporal Reasoning for Task Progress Understanding

Authors: Xiaoda Yang, Can Wang, Yuxiang Liu, Pengfei Zhou, Jianwen Lou, Shuicheng Yan

Organizations: Zhejiang University, Hangzhou, China · Qingdao University, Qingdao, China · Tianjin University, Tianjin, China · National University of Singapore, Singapore

Abstract

Vision-Language Models (VLMs) have advanced rapidly in static visual understanding, yet remain unreliable when judging how an egocentric task is progressing. Given a task instruction and two visual observations, a model should determine which state is closer to the goal by analyzing task-relevant object configurations and spatial relations, rather than relying on timestamps or presentation order. This distinction is critical in manipulation, where retries, corrective actions, and temporary regressions make progress inherently non-monotonic. We introduce EgoTSR, a unified framework for diagnosing and improving order-robust task-progress understanding. First, SpatialLogic-Bench evaluates each physical state pair in both original and order-swapped presentations across short- and long-horizon settings, exposing whether a model follows task-state evidence or chronological shortcuts. Second, our data construction pipeline converts successful, approximately monotonic manipulation and first-person trajectories into bidirectional supervision; LongTag further preserves intermediate subtask structure for long-horizon comparison, while failure-aware data extend learning to regressions and recoveries. Third, a progressive CoT-to-Tag curriculum first supervises evidence-grounded interpretation of task-relevant state changes and then consolidates the comparison rule through scalable label-only training. Experiments reveal substantial input-order bias in representative VLMs. EgoTSR achieves 92.4% long-horizon accuracy with a 0.1-point forward-inverse Gap. Failure-aware supervision further improves accuracy on non-monotonic trajectories by 11.8 points and Recovery Accuracy by 11.2 points, while maintaining broad visual and spatial capabilities. These results establish goal-conditioned state comparison as an explicit formulation of egocentric spatiotemporal reasoning for task-progress understanding.

Figures & tables

Fig. 1: Overview of EgoTSR: SpatialLogic-Bench diagnoses input-order bias, EgoTSR-Data provides scalable state-comparison supervision, and CoT $\rightarrow$ Tag learning produces an order-robust comparison rule.

Fig. 2: Illustration of SpatialLogic-Bench. The benchmark instantiates task-goal-aware state comparison and uses forward–inverse evaluation to diagnose input-order bias.

Fig. 3: EgoTSR-Data construction and supervision organization. Successful robot, human, and egocentric episodes are converted into goal-conditioned state-comparison pairs and organized into CoT, Tag, and LongTag formats.

Subset	Horizon	Format	Primary role
Short CoT	short	CoT	local evidence path
Short Tag	short	Tag	scalable local comparison
Long CoT	long	CoT	structured progress reasoning
LongTag	long	Tag + $S$	long-horizon state comparison
General Tag	variable	Tag	long-tail generalization

TABLE I: Organization of EgoTSR-Data.

S

denotes the subtask-level skeleton from metadata or the Subtask Planner.

Fig. 4: Progressive CoT $\rightarrow$ Tag learning framework. CoT supervision establishes evidence-grounded reasoning paths, and scalable Tag supervision transfers them into order-robust state comparison.

Closed-source VLMs
Model	Level 1: Short-Horizon Acc. (%)									Level 2: Long-Horizon Acc. (%)
	Frame interval $(\times 10)$								Avg.	Forward			Inverse			Avg.
	5	6	7	8	9	10	11	$\geq 12$	Avg.	S	M	L	S	M	L	Avg.
GPT-4o-mini [ 28 ]	70.0	70.3	72.8	67.7	60.0	67.5	67.7	70.3	68.3	64.9	69.0	85.7	34.6	30.3	42.5	55.0
GPT-4o [ 29 ]	71.1	71.7	76.7	77.8	79.2	79.2	79.2	79.2	76.8	56.6	61.2	69.7	57.3	51.5	54.6	58.2
GPT-o3 [ 30 ]	67.2	61.4	59.6	61.6	59.6	64.0	62.0	66.6	62.8	71.9	64.7	73.1	69.6	66.7	68.6	69.0

TABLE II: Comprehensive performance on SpatialLogic-Bench. Short-horizon Avg. averages the eight listed window categories, except for Qwen3.5, for which only windows 5–10 were measured. Dashes denote unmeasured settings.

Fig. 5: Ablation trajectories in the Accuracy–Gap space under the long-horizon protocol. Progressive training and LongTag move the model toward high accuracy and low input-order bias; the full EgoTSR model occupies the strongest operating point.

Method	Fwd $\uparrow$	Inv $\uparrow$	Avg. $\uparrow$	Worst $\uparrow$	Gap $\downarrow$
CoT only	91.7	2.2	46.9	2.2	89.5
Tag only	50.4	53.2	51.8	50.4	2.8
CoT $\rightarrow$ Tag	54.2	57.3	55.8	54.2	3.1
Mixed CoT/Tag/LongTag	67.2	72.1	69.6	67.2	4.9
CoT $\rightarrow$ Tag w/ LongTag	92.4	92.3	92.4	92.3	0.1
LongTag w/o $S$	83.2	78.9	81.1	78.9	4.3

TABLE III: Ablation study of EgoTSR on long-horizon forward–inverse evaluation.

\mathrm{Acc}_{\mathrm{worst}}=\min(\mathrm{Fwd},\mathrm{Inv})

is reported jointly with Avg. and Gap to separate order-invariant reasoning from symmetric failure modes.

Variant	Fwd $\uparrow$	Inv $\uparrow$	Avg. $\uparrow$	Worst $\uparrow$	Gap $\downarrow$
Tag only	69.4	66.2	67.8	66.2	3.2
CoT only	82.4	57.9	70.2	57.9	24.5
CoT $\rightarrow$ Tag	84.1	78.2	81.2	78.2	5.9
+ LongTag	87.6	85.1	86.4	85.1	2.5

TABLE IV: Supervision-path ablation on the matched short-horizon paired-order diagnostic subset. Worst is the lower of Fwd and Inv.

Order strategy	Fwd $\uparrow$	Inv $\uparrow$	Avg. $\uparrow$	Worst $\uparrow$	Gap $\downarrow$
Forward only	83.5	60.4	72.0	60.4	23.1
Inverse only	61.8	81.2	71.5	61.8	19.4
Random order	78.9	74.6	76.8	74.6	4.3
Forward+inverse	84.1	78.2	81.2	78.2	5.9
EgoTSR-Full	87.6	85.1	86.4	85.1	2.5

TABLE V: Pair-order ablation on the short-horizon paired-order diagnostic subset. Worst is the lower of Fwd and Inv.

Method	Avg. $\uparrow$	Gap $\downarrow$	Rel. cost $\downarrow$
Single-image regression	70.6	9.4	0.18 $\times$
Tag only	67.8	3.2	0.22 $\times$
LoRA	82.1	3.9	0.35 $\times$
Full fine-tuning	86.4	2.5	1.00 $\times$

TABLE VI: Accuracy, order sensitivity, and relative training cost on the matched short-horizon diagnostic subset.

Fig. 6: Qualitative task-completion curve on a long-horizon manipulation trajectory. The score changes most strongly around task-relevant state transitions.

Setting	Fail. Avg. $\uparrow$	Rec. $\uparrow$	Reg. Err. $\downarrow$	Gap $\downarrow$	Succ. Test $\uparrow$
Qwen zero-shot	54.8	46.5	44.7	18.9	–
EgoTSR-Succ.	64.1	58.6	32.8	13.5	88.2
EgoTSR-FA	75.9	69.8	23.4	6.7	84.1

TABLE VII: Robustness on non-monotonic failure, regression, correction, and recovery trajectories. Succ. denotes success-only training and FA denotes failure-aware training. Failure Avg. is the overall accuracy across all valid ordered items in the fixed-Window-10 split. Failure Avg. and Recovery Acc. are higher-is-better; Regression Error and Gap are lower-is-better. Success Test reports accuracy on the original monotonic success split.

Pair type	Accuracy (%) $\uparrow$
Normal-progress	76.8
Recovery	69.8
Regression	76.6
Failure-internal	73.6

TABLE VIII: Pair-type breakdown on the failure-aware split.

Window	Pairs	Avg. $\uparrow$	Rec. $\uparrow$	Gap $\downarrow$
5	1,198	74.8	67.6	7.2
10	1,345	75.9	69.8	6.7
20	1,824	78.6	71.1	5.9
5+R10	1,292	75.5	69.4	6.8

TABLE IX: Sliding-window sensitivity. The 5+R10 policy expands only recovery pairs to Window 10.

Data	Fail. Avg. $\uparrow$	Rec. $\uparrow$	Reg. Err. $\downarrow$	Gap $\downarrow$
0%	64.1	58.6	32.8	13.5
25%	68.5	63.9	28.1	10.2
50%	73.4	68.7	24.8	7.5
100%	75.9	69.8	23.4	6.7

TABLE X: Effect of failure-aware training-data scale.

Backbone	Base	EgoTSR	Gain $\uparrow$	$\Delta$ Gap $\downarrow$
Small VLM	60.8	65.2	+4.4	-4.1
Qwen-7B	64.1	75.9	+11.8	-6.8
InternVL	66.7	74.2	+7.5	-5.6
LLaVA-style	62.5	68.9	+6.4	-4.8

TABLE XI: Backbone portability on failure-aware evaluation.

Fig. 7: Quantitative analysis of failure-aware learning. (a) Failure, recovery, and regression accuracy as the amount of failure-aware training data increases. (b) The corresponding reduction in forward–inverse Gap. (c) Accuracy and candidate-pair growth under different sliding-window spans. Regression accuracy is one minus Regression Error.

Group	Model	M1	M2	Avg.
CV	Qwen2.5-VL-7B [ 36 ]	75.17	70.83	75.07
	EgoTSR-CoT	73.00	73.33	72.34
	EgoTSR-Tag	76.42	79.17	75.27
MMSI	Qwen2.5-VL-7B [ 36 ]	24.5	29.4	25.9
	EgoTSR-CoT	30.2	34.1	28.3
	EgoTSR-Tag	33.0	36.5	29.6

TABLE XII: Compact external generalization results. Within the CV, MMSI, VSI, and Ego groups, M1/M2 denote Omni3D/Depth, Obj.-Obj./Obj.-Reg., Size/Rel.Dist., and E4D-L/EK-L, respectively. M1/M2 are selected submetrics; Avg. is the overall evaluation score.

Capability group	Before	After	Change
General VQA	73.1	71.8	-1.3
Spatial reasoning	68.6	67.4	-1.2
Temporal ordering	70.5	71.6	+1.1
Object-state QA	74.0	73.2	-0.8
Average	71.6	71.0	-0.6

TABLE XIII: Capability retention before and after goal-conditioned task-progress training.

Temporal Reasoning in VLMs

See all 23 papers

From Perception to Planning: Evolving Ego-Centric Task-Oriented Spatiotemporal Reasoning via Curriculum Learning
Apr 12, 2026Xiaoda Yang, Yuxiang Liu, Shenzhou Gao +8Embodied Spatial Reasoning Spatiotemporal Reasoning
Zhejiang University, Hangzhou, China · Tianjin University, Tianjin, China · Qingdao University, Qingdao, China +2
Recurrent Reasoning with Vision-Language Models for Estimating Long-Horizon Embodied Task Progress
Mar 18, 2026Yuelin Zhang, Sijie Cheng, Chen Li +4VLMs for Robotics VLM Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +2
TaskAnchor: Grounding Task State in Reactive VLAs for Long-Horizon Manipulation
Sep 20, 2026Hengyan Liu, Wenlve Zhou, Bo Yue +7Memory-Augmented VLMs Long-Horizon Robotic Manipulation
The Chinese University of Hong Kong, Shenzhen · Foshan University · DexForce +1

Apr 12, 2026J/K move · Enter open · S save

From Perception to Planning: Evolving Ego-Centric Task-Oriented Spatiotemporal Reasoning via Curriculum Learning

Xiaoda Yang, Yuxiang Liu, Shenzhou Gao, Can Wang+7

Zhejiang University, Hangzhou, China · Tianjin University, Tianjin, China · Qingdao University, Qingdao, China+2

Embodied Spatial Reasoning Spatiotemporal Reasoning Embodied Reasoning Egocentric Video Understanding Curriculum Learning

Open paper AskPDF arXiv

Modern vision-language models achieve strong performance in static perception, but remain limited in the complex spatiotemporal reasoning required for embodied, egocentric tasks. A major source of failure is their reliance on temporal priors learned from passive video data, which often leads to spatiotemporal hallucinations and poor generalization in dynamic environments. To address this, we present EgoTSR, a curriculum-based framework for learning task-oriented spatiotemporal reasoning. EgoTSR is built on the premise that embodied reasoning should evolve from explicit spatial understanding to internalized task-state assessment and finally to long-horizon planning. To support this paradigm, we construct EgoTSR-Data, a large-scale dataset comprising 46 million samples organized into three stages: Chain-of-Thought (CoT) supervision, weakly supervised tagging, and long-horizon sequences. Extensive experiments demonstrate that EgoTSR effectively eliminates chronological biases, achieving 92.4% accuracy on long-horizon logical reasoning tasks while maintaining high fine-grained perceptual precision, significantly outperforming existing open-source and closed-source state-of-the-art models.

Abstract

Temporal Reasoning in VLMs

Explore similar work

From Perception to Planning: Evolving Ego-Centric Task-Oriented Spatiotemporal Reasoning via Curriculum Learning

Recurrent Reasoning with Vision-Language Models for Estimating Long-Horizon Embodied Task Progress

TaskAnchor: Grounding Task State in Reactive VLAs for Long-Horizon Manipulation