Spatio-Temporal Grounding of Large Language Models from Perception Streams

Jacob Anderson; Bardh Hoxha; Georgios Fainekos; Hideki Okamoto; Danil Prokhorov

cs.ROApr 8, 2026

Spatio-Temporal Grounding of Large Language Models from Perception Streams

Authors: Jacob Anderson, Bardh Hoxha, Georgios Fainekos, Hideki Okamoto, Danil Prokhorov

Organizations: Toyota Motor North America Research & Development, Ann Arbor, MI, USA

Abstract

Embodied-AI agents must reason about how objects move and interact in 3-D space over time, yet existing smaller frontier Large Language Models (LLMs) still mis-handle fine-grained spatial relations, metric distances, and temporal orderings. We introduce the general framework Formally Explainable Spatio-Temporal Scenes (FESTS) that injects verifiable spatio-temporal supervision into an LLM by compiling natural-language queries into Spatial Regular Expression (SpRE) -- a language combining regular expression syntax with S4u spatial logic and extended here with universal and existential quantification. The pipeline matches each SpRE against any structured video log and exports aligned (query, frames, match, explanation) tuples, enabling unlimited training data without manual labels. Training a 3-billion-parameter model on 27k such tuples boosts frame-level F1 from 48.5% to 87.5%, matching GPT-4.1 on complex spatio-temporal reasoning while remaining two orders of magnitude smaller, and, hence, enabling spatio-temporal intelligence for Video LLM.

Figures & tables

Panel of Figure 1 — Figure 1: The FESTS framework begins with (1) which processes the SpRE and perception stream inputs to produce the two formal method-based results; (2) processes the explanation to improve readability for LLMs; and (3) packages this into a distributable data formats.

Figure 2: Average Performance across frame lengths. Blue = Overall F1, orange = Exact Match; shaded $\sigma$ . Starred points denote the best-performing model for each frame length.

	4			8			12			16			Overall
Model	F1 f	EM	F1 s	F1 f	EM	F1 s	F1 f	EM	F1 s	F1 f	EM	F1 s	F1 f	EM	F1 s
Q-Baseline	0.472	0.211	0.325	0.456	0.191	0.258	0.452	0.187	0.231	0.416	0.187	0.215	0.485	0.250	0.326
Q-SFT	0.836	0.636	0.644	0.786	0.507	0.582	0.771	0.451	0.565	0.657	0.329	0.454	0.804	0.566	0.636
Q-SFT+RL	0.881	0.676	0.694	0.856	0.611	0.686	0.852	0.549	0.659	0.827	0.440	0.629	0.875	0.645	0.723
GPT-4.1	0.888	0.269	0.640	0.813	0.236	0.528	0.797	0.236	0.497	0.756	0.231	0.447	0.848	0.350	0.610

Table 1: Performance comparison across models and target lengths. Metrics: Frame F1 (F1 f ), Exact Match (EM), and Segment F1 (F1 s ). Best results per column are in bold .

Query type	Base				Stage-I: SFT				Stage-II: SFT + RL
Query type	$P$	$R$	F 1	EM	$P$	$R$	F 1	EM	$P$	$R$	F 1	EM
Sequence	0.875	0.668	0.570	0.206	0.931	0.961	0.923	0.715	0.971	0.978	0.962	0.821
Spatial	0.785	0.557	0.415	0.190	0.825	0.853	0.761	0.502	0.845	0.905	0.819	0.552
Temporal	0.863	0.597	0.504	0.348	0.883	0.857	0.810	0.553	0.910	0.902	0.866	0.615
Metric	0.872	0.551	0.457	0.200	0.903	0.830	0.769	0.597	0.917	0.960	0.903	0.706
Existential	0.875	0.574	0.480	0.306	0.817	0.861	0.757	0.463	0.826	0.937	0.828	0.531

Table 2: Precision (

P

), Recall (

R

), F 1 , and Exact-Match (EM) for each query type. Values are averaged across 8-, 12-, and 16-frame lengths.

Spatiotemporal Reasoning

Spatiotemporal reasoning studies how systems connect objects, events, locations, and changes across space and time.

See all 70 papers

SpaTime: Streaming Vision-Language Models for Spatio-temporal Reasoning
Oct 6, 2026Hairong Yin, Huangying Zhan, Shin-Fang Chng +2Vision-Language Models Spatiotemporal Reasoning
Department of Computer Science, Purdue University, USA · Goertek Alpha Labs, USA
Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence
Mar 13, 2026Seunghwan Bang, Hwanjun SongSpatiotemporal Reasoning VLM Reasoning
UNIST · KAIST
Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models
May 9, 2026Tri Cao, Khoi Le, Thong Nguyen +7VLM Evaluation Object Hallucination in VLMs
National University of Singapore · VinUniversity · Nanyang Technological University

Oct 6, 2026J/K move · Enter open · S save

SpaTime: Streaming Vision-Language Models for Spatio-temporal Reasoning

Hairong Yin, Huangying Zhan, Shin-Fang Chng, Yi Xu+1

Department of Computer Science, Purdue University, USA · Goertek Alpha Labs, USA

Vision-Language Models Spatiotemporal Reasoning VLM Reasoning 3D VLMs Temporal Video Understanding

Open paper AskPDF arXiv

Embodied agents must reason about 3D space while the video is still arriving, answering questions as soon as they have observed enough of the scene. VLMs that incorporate 3D geometric priors achieve strong spatial reasoning, but they operate offline, i.e., the full video must be available before they produce an answer. Streaming VLMs process frames causally and decide for themselves when to respond, yet they lack explicit 3D representations. We present SpaTime, a streaming VLM that fuses causal geometry tokens into the language model at every frame, using only the frames observed so far. To supervise when the model answers, we propose a response-time loss that maps per-frame response probabilities to a differentiable expected response time and penalizes the distance from the ground-truth frame. For evaluation, we construct StreamVSTI-Bench and StreamVSI-Bench, streaming adaptations of VSTI-Bench and VSI-Bench. On StreamVSTI-Bench, SpaTime reaches 49.2% overall accuracy and reduces the mean response-time error by 66% relative to the strongest streaming baseline.

Abstract

Spatiotemporal Reasoning

Explore similar work

SpaTime: Streaming Vision-Language Models for Spatio-temporal Reasoning

Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence

Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models