SENRI: Understanding the Space-Event Nexus in Real-Life Videos

Kehan Lan; Kaining Ying; Henghui Ding; Kaipeng Zhang

cs.CVOct 18, 2026

SENRI: Understanding the Space-Event Nexus in Real-Life Videos

Authors: Kehan Lan, Kaining Ying, Henghui Ding, Kaipeng Zhang

Organizations: Fudan University · Alaya Lab

Abstract

Human memories are organized in a way that connects events with their corresponding spatial contexts. An AI assistant should also be able to memorize and recall the nexus of events and spaces. Thus we introduce SENRI, a task that studies this space--event connection in long, continuous real-life egocentric video. It contains six task types in two directions: Event $\rightarrow$ Space, which recovers spatial knowledge from event cues, and Space $\rightarrow$ Event, which retrieves events from regions and trajectories. To evaluate this, we build SENRI-Bench, containing 500 manually annotated multiple-choice questions over 10 uninterrupted self-recorded videos with around 25.3 hours of content. Also, we evaluate three typical long video understanding paradigms: direct perception, evidence seeking, and persistent memory. The task remains challenging across all three paradigms, highlighting substantial room for improvement. Then we show two representative failure modes of existing memory methods and address these evidence gaps with SENRI-Agent, a controller that uses Target-Guided Visual Check (TGVC) to propose local visual check tasks for parallel verifiers and integrate the reports into a working memory to help with reasoning. Built upon existing memory systems, SENRI-Agent improves the performance of existing baselines without modifying their stored memories. We hope SENRI will provide valuable insights into how real-life egocentric understanding should be evaluated, reveal the gaps in existing approaches, and offer a possible direction for improvement.

Figures & tables

Category	Benchmark	Anno.	Self- recorded	One- take	Real- life	Cross- scene	Event- grounded	Spatial reasoning	Avg. duration per clip (h)
Spatial understanding benchmarks	VSI-Bench	A/M	✗	✗	✗	✗	✗	✓	$\approx 0.03$
	VSI-Super	A/M	✗	✗	✗	✓	✗	✓	$0.17$ – $4.00$
	VSI-Super-Wild	A/M	✗	✓	✓	✓	✗	✓	$0.64$
	UCS-Bench	A/M	✗	✗	✓	✓	✓	✓	$0.33$
Long video benchmarks	LVBench	M	✗	✗	✓	✓	✓	✗	$1.14$
	LongVideoBench	M	✗	✗	✓	✓	✓	✗	$0.13$

Table 1: Comparison of SENRI-Bench with existing video understanding benchmarks. ✓ indicates coverage or partial coverage; ✗ indicates absence or no explicit guarantee. Anno. denotes the QA annotation procedure: A for automatic, M for manual, and A/M for a mixed pipeline. Self-recorded denotes videos newly collected for the project and absent from existing resources. Real-life denotes everyday activities or exploration outside dedicated indoor scans. Event-grounded includes questions about events and Spatial-reasoning requires reasoning about spatial conditions.

Approach	Method	Inference Model	Total	Event $\rightarrow$ Space					Space $\rightarrow$ Event
Approach	Method	Inference Model	Total	Avg.	L	NP	OC	RR	Avg.	RER	TER
Direct Perception	Gemini 3.1 Pro	Gemini 3.1 Pro	47.4	43.1	56.2	37.7	37.2	40.0	56.0	55.7	56.4
	Gemini 3.5 Flash	Gemini 3.5 Flash	49.8	45.5	56.2	40.6	32.6	51.1	58.4	60.2	56.4
	Gemini 3.7 Flash	Gemini 3.7 Flash	52.2	48.5	61.8	50.7	32.6	48.9	59.6	58.0	61.5
	GPT-5	GPT-5	47.2	44.6	59.6	39.1	32.6	45.6	52.4	55.7	48.7
	GPT-5.5	GPT-5.5	48.2	46.7	60.7	44.9	31.4	48.9	51.2	50.0	52.6

Table 2: Main results on SENRI-Bench (accuracy, %). ES and SE denote Event

\rightarrow

Space and Space

\rightarrow

Event, respectively; L, NP, OC, RER, RR, and TER denote localization, navigation planning, object counting, region-conditioned event retrieval, relation reasoning, and trajectory event retrieval. ∗ indicates that the inference backbone is further fine-tuned or post-trained by the corresponding method. Best and second-best results are shown in bold and underlined, respectively.

Figure 3: Representative failure modes of existing memory systems on SENRI: weak space-event nexus in text memories (left), and missing target details in visual memories (right).

Figure 4: Overview of SENRI-Agent. (a) The previous workflow alternates between retrieving persistent memories and answering the question. (b) SENRI-Agent introduces a verify action that turns retrieved memories and visual candidates into target-guided checks over selected video windows. Parallel verifiers return reports with supporting evidence frames, which are integrated with memories from previous rounds into a bounded working memory for subsequent reasoning.

Memory system	Inference model	Workflow	Total	ES	L	NP	OC	RR	SE	RER	TER
WorldMM	Qwen3.8-27B	Original	47.6	42.8	46.1	31.9	36.0	54.4	57.2	61.4	52.6
WorldMM	Qwen3.8-27B	SENRI-Agent	50.4	46.1	56.2	34.8	38.4	52.2	59.0	63.6	53.8
WorldMM	Gemini 3.7 Flash	Original	58.8	56.6	58.4	66.7	43.0	60.0	63.3	67.0	59.0
WorldMM	Gemini 3.7 Flash	SENRI-Agent	62.6	62.0	66.3	63.8	48.8	68.9	63.9	70.5	56.4
MAGIC-Video	Qwen3.8-27B	Original	45.2	43.1	53.9	31.9	34.9	48.9	49.4	48.9	50.0
MAGIC-Video	Qwen3.8-27B	SENRI-Agent	47.8	45.2	55.1	42.0	36.0	46.7	53.0	54.5	51.3

Table 3: Results of SENRI-Agent on SENRI-Bench across different models. All are accuracy (%).

Variant	TG	PV	WM	Total	ES	L	NP	OC	RR	SE	RER	TER
SENRI-Agent	✓	✓	✓	47.8	45.2	55.1	42.0	36.0	46.7	53.0	54.5	51.3
w/o Target Guidance	✗	✓	✓	45.0	41.0	50.6	37.7	25.6	48.9	53.0	53.4	52.6
w/o Parallel Verifiers	✓	✗	✓	45.0	41.6	47.2	36.2	30.2	51.1	51.8	55.7	47.4
w/o Working Memory	✓	✓	✗	44.0	41.9	50.6	30.4	36.0	47.8	48.2	46.6	50.0

Table 4: Component ablations of SENRI-Agent on MAGIC-Video with Qwen3.8-27B. TG, PV, and WM denote target guidance, parallel verifiers, and working memory, respectively.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Example	Question	Options	Ans.
L-01	Where was the trash can where I threw the ice cream cone box?	A. To the right of a pink-themed bus stop. B. A trash can inside the subway station. C. Opposite a green-themed bus stop. D. To the left of a blue-themed bus stop.	A
L-02	Where did I pick up the guide booklet for this exhibition?	A. On the left side of the entrance to the first exhibition hall B. On the right side of the entrance to the first exhibition hall C. On the left side of the entrance to the second exhibition hall D. On the right side of the entrance to the second exhibition hall	B
L-03	Where did I return the yellow power bank?	A. Inside the McDonald’s by the river B. Inside the theater by the river C. At the subway entrance D. Inside the art museum by the river	B
L-04	When I was storing my bag at the art museum, which locker did I store it in?	A. In the left cabinet, second from the left in the first row counting from bottom to top B. In the right cabinet, first from the left in the fifth row counting from bottom to top C. In the left cabinet, second from the left in the second row counting from bottom to top D. In the right cabinet, second from the left in the fourth row counting from bottom to top	A
L-05	Where was I when I tasted the fruit flower tea?	A. I was at the fourth shop from the end under the bridge arch. B. I was at the first shop above the bridge arch. C. I was at the fourth shop from the end before entering the bridge arch. D. I was at the first shop under the bridge arch.	C
L-06	Where did I hang my Messi doll?	A. On the platform by the window B. In the blue-themed store C. On a brown bench by the window D. On the bench next to the escalator	C

Appendix

Table 5: Localization: ten selected SENRI-Bench questions. Answers are the original benchmark annotations.

Example	Question	Options	Ans.
RR-01	When I was trying out pens, what was the spatial relationship between the shelf holding pens and the shelf holding correction tape?	A. The correction tape and the pens are on the same shelf, the former on the right and the latter on the left. B. The correction tape and the pens are on the same shelf, the former on the left and the latter on the right. C. The shelf for correction tape is closer to the open area, separated from the shelf for pens by a shelf of notebooks. D. The shelf for pens is closer to the open area, separated from the shelf for correction tape by a shelf of notebooks.	D
RR-02	When I was viewing the exhibition, what was the spatial relationship between the video art installation and an abstract work related to a forest?	A. There was no video art installation at all B. Facing the video projection screen, the abstract work was behind me on the right C. Facing the video projection screen, the abstract work was directly behind me D. Facing the video projection screen, the abstract work was behind me on the left	D
RR-03	When I bought the drink, what was the relative position of the checkout counter and the cabinet where I bought the drink?	A. The counter was on the same side as the cabinet. B. I did not buy a drink. C. I bought the drink from a vending machine. D. The counter was opposite the cabinet.	D
RR-04	When I was picking up utensils, what were the relative positions of the fork and spoon to the cashier?	A. The fork was next to the cashier, and the spoon was next to the cashier B. The fork was next to the cashier, and the spoon was opposite the cashier C. The fork was opposite the cashier, and the spoon was next to the cashier D. The fork was opposite the cashier, and the spoon was opposite the cashier	C
RR-05	When I bought the Messi doll, what was the relative position of the adjacent Messi and Ronaldo dolls?	A. Messi was to the right of Ronaldo B. There was no Messi C. Messi was to the left of Ronaldo D. There was no Ronaldo	C
RR-06	When I was eating spaghetti, what was the relative position of the beverage counter to me?	A. To my rear left B. To my rear right C. To my front left D. To my front right	C

Appendix

Table 6: Relation Reasoning: ten selected SENRI-Bench questions. Answers are the original benchmark annotations.

Example	Question	Options	Ans.
OC-01	When I was looking at the displayed books and other exhibits inside the Chinese-style building, how many glass display cabinets were in the room?	A. 4 B. 3 C. 2 D. 8	D
OC-02	When I was trying out pens, how many pens were in the pen holder on my left (not on the shelf)?	A. 7 B. 5 C. 8 D. 6	A
OC-03	When I was storing my bag at the art museum, how many compartments did the storage locker have?	A. 64 B. 48 C. 16 D. 36	D
OC-04	When I observed the fish pond, how many green chairs were there in total around the fish pond?	A. 6 B. 3 C. 4 D. 5	D
OC-05	After I took away the salad, how many boxes of salad were left on the shelf?	A. 6 B. 3 C. 4 D. 5	C
OC-06	When I was eating spaghetti, how many plates were on my table?	A. 3 B. 2 C. 1 D. 4	B

Appendix

Table 7: Object Counting: ten selected SENRI-Bench questions. Answers are the original benchmark annotations.

Example	Question	Options	Ans.
NP-01	From the Indian restaurant where I was last, to reach the conveyor-belt sushi restaurant, how should I walk?	A. Exit the restaurant, turn left, take the escalator down to the lowest floor, the sushi restaurant is on the right. B. Exit the restaurant, turn left, then make a left U-turn to go up the escalator, the sushi restaurant is on my right. C. Exit the restaurant, turn left, then make a left U-turn to go up the escalator, the sushi restaurant is on my left. D. Exit the restaurant, turn right, then make a right U-turn to go up the escalator, the sushi restaurant is on my right.	B
NP-02	When I exited the art museum through the door near the river and wanted to take the subway, how did I get there?	A. Turn left from the door, turn left at the next corner, then ride a shared bike at the intersection, go straight all the way, it’s on the left side B. Turn right from the door, walk straight along the river, then turn right at the intersection, go straight all the way, the subway entrance is on the left side C. Turn left from the door, then turn left at the art museum corner, then turn left again at the next corner, cross the crosswalk at the intersection, then go straight all the way, the subway entrance is on the left side D. Turn left from the door, then turn right at the next corner, cross the crosswalk at the intersection, then go straight all the way, turn right at a barber shop	C
NP-03	From when I exited the underground mall to when I threw away my drink bottle, how did I walk?	A. Walked straight to near the bus stop, then turned right past many square-dancing aunties, then wandered counterclockwise in the park, and encountered many trash cans before completing half a circle. B. Walked straight to near the bus stop, then turned right past many square-dancing aunties, then wandered clockwise in the park past the fish pond, strolled by the pond, turned around and returned, then encountered trash cans. C. First turned around to the rear left, then turned right to near the bus stop, then turned right past many chairs, then wandered clockwise in the park past the fish pond, strolled by the pond, turned around and returned, then encountered trash cans. D. Walked straight to near the bus stop, then turned right past many square-dancing aunties, then wandered counterclockwise in the park past many trash cans, strolled by the pond, turned around and returned, then encountered trash cans.	C
NP-04	When I was about to get on the subway, how did I get from the residential complex to the subway?	A. Turn left from the residential complex gate, then turn left at a parking lot gate, keep walking straight until reaching the roadside, then turn left B. Turn right from the residential complex gate, then turn left at a parking lot gate, keep walking straight until reaching the roadside, then turn left C. Walk from the residential complex gate to Starbucks, make a U-turn to the right at the Starbucks entrance, keep walking straight until reaching the roadside, then turn left. D. Turn left from the Starbucks entrance, keep walking straight until reaching the roadside, then turn left	A
NP-05	When I got off the bicycle, if I wanted to go directly to the subway station without passing through the shopping mall, how should I go?	A. Make a U-turn along the direction I was last cycling, turn left under the elevated bridge, then keep going straight ahead B. Keep going straight ahead along the direction I was last cycling, turn left under the elevated bridge, then keep going straight ahead C. Make a U-turn along the direction I was last cycling, turn right under the elevated bridge, then keep going straight ahead D. Keep going straight ahead along the direction I was last cycling, turn right under the elevated bridge, then keep going straight ahead	D
NP-06	How did I ride from buying braised food to encountering the school gate for the first time?	A. Immediately turn left, then turn right, then turn left, then turn right, then turn right B. Immediately turn left, then turn left, then turn left, then turn right, then turn left C. Immediately turn left, then turn right, then turn left, then turn right, then turn left D. Immediately turn left, then turn left, then turn left, then turn right, then turn right	A

Appendix

Table 8: Navigation Planning: ten selected SENRI-Bench questions. Answers are the original benchmark annotations.

Example	Question	Options	Ans.
RER-01	What did I do when I was at the entrance of a Mexican restaurant?	A. I took a number B. The waiter led me into the restaurant C. I asked the waiter D. I flipped through the menu	D
RER-02	What did I do on the second-floor platform of the art museum that connects to the outside?	A. I drank a beverage B. I used binoculars to look at the opposite side C. I walked two circles around it D. I flipped through the art exhibition brochure	D
RER-03	What did I do next to the children’s slide?	A. I turned a turntable. B. I bought a bottled drink. C. I shoveled sand with a shovel. D. I touched the slide.	A
RER-04	What did I do between the two orange lifebuoys?	A. Ate an ice cream cone B. Drank cola C. Threw away the bottle D. Drank plain water	D
RER-05	What did I do by the fish pond?	A. Watched others feed the fish B. Fed the fish C. Threw away trash D. Sat and rested	A
RER-06	What did I do when I was at the furniture store?	A. I bought a charging cable B. I tried to charge my phone C. I did nothing D. I played with the action camera	B

Appendix

Table 9: Region-Conditioned Event Retrieval: ten selected SENRI-Bench questions. Answers are the original benchmark annotations.

Example	Question	Options	Ans.
TER-01	What did I encounter when I went from the restroom entrance to the Chinese-style building with the exhibition?	A. I passed by an ice cream stand B. I climbed over a rockery C. I encountered many children feeding the goldfish in the pond D. Several performers in ancient costumes passed in front of me	D
TER-02	What happened on the way from the Pokémon card vending machine to the MUJI store?	A. I bought Pokémon cards B. I browsed a LEGO store C. I caught a doll D. I saw a claw machine but didn’t play	D
TER-03	On the way from McDonald’s to the art museum, what did I do?	A. I returned a power bank B. I stopped to watch a street singer C. I threw a plastic bottle into a trash can D. I bought a sports drink	A
TER-04	Between leaving the temple and crossing the street for the first time, what did I see?	A. A man was filming the pagoda B. A man was filming a green building C. A woman was taking a selfie D. A young person was flying a drone	B
TER-05	After walking to the end along the canal side and then turning around to walk back along the opposite side, what did I encounter?	A. I stopped and observed a reptile experience center. B. Someone pitched flower tea to me. C. I drank water 5 times. D. I did a toy gun target shooting activity.	A
TER-06	What did I see while walking from the ice cream vending machine to the bridge?	A. A sanitation worker riding a bicycle B. Someone driving a red vehicle C. A sanitation worker riding a tricycle D. Someone driving a white vehicle	C

Appendix

Table 10: Trajectory Event Retrieval: ten selected SENRI-Bench questions. Answers are the original benchmark annotations.

Figure 5: Accuracy on SENRI-Bench as the maximum number of sampled frames increases for four Qwen models.

Figure 6: Accuracy change of SENRI-Agent relative to the original workflow for each video, in percentage points. Each cell compares predictions on the same 50 questions. Blue indicates an improvement, red a decrease, and white no change. Videos are indexed from 0 to 9.

Setting	Total	ES	L	NP	OC	RR	SE	RER	TER
Random guessing (expected)	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0
Question and options only	34.6	35.0	36.0	33.3	33.7	36.7	33.7	34.1	33.3
Direct perception (512 frames)	41.0	39.5	48.3	37.7	37.2	34.4	44.0	45.5	42.3
SENRI-Agent on MAGIC-Video	47.8	45.2	55.1	42.0	36.0	46.7	53.0	54.5	51.3
SENRI-Agent on WorldMM	50.4	46.1	56.2	34.8	38.4	52.2	59.0	63.6	53.8

Appendix

Table 11: Comparison with a no-visual-input baseline on SENRI-Bench (accuracy, %). All evaluated settings use Qwen3.8-27B as the inference model. The no-visual-input baseline receives only the question and answer options. Uniform random guessing gives an expected accuracy of 25%.

Egocentric Video Understanding

Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.

See all 203 papers

Oct 4, 2026

Have I Scene This Before? Spatially Grounded Conversational Memory for Complex Queries in Egocentric Assistants

Egocentric assistants must connect what users say with what they see across long interaction histories. We formalize this challenge as Spatially grounded Conversational Reasoning (SpaCR): cross-scene, recall-oriented, and counterfactual spatial queries that combine user-stated facts with geometric evidence. Direct vision-language models incur high inference costs and context limits as histories grow, while keyframe selection and retrieval can omit objects or evidence needed for complete recall. We propose Spatially grounded Conversational Memory (SpaC-MEM), an object-centric working memory that uses 3D reconstruction and segmentation to ground conversational information in persistent physical objects. It compresses multimodal histories while preserving spatial evidence and allowing object-specific facts to be updated through dialogue. We also introduce Ego-SpaCR, a benchmark comprising 620 ScanNet video sessions augmented with 95 task-oriented conversations and 3,100 evaluation queries. SpaC-MEM achieves the highest overall answer accuracy among the evaluated methods and improves object recall while requiring substantially fewer reference input tokens than native-video baselines. Removing 3D spatial information substantially degrades performance, highlighting the importance of preserving spatial and conversational evidence together.

Jiazhou Liang, Liam Gallagher, Kiko Chen +4

University of Toronto

Conversational Memory Egocentric Video Understanding 3D Visual Grounding Multimodal Spatial Reasoning

Ask PDF

Oct 12, 2026

EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos

Egocentric video provides a continuous first-person record of everyday activities, making it a natural foundation for wearable AI assistants. Unlike offline video understanding, streaming systems must process observations as they arrive, while queries may refer to experiences from the distant past. Recent work has begun to explore whether information can be recalled across long temporal gaps. Yet remembering how far is only part of the challenge: recurring objects, actions, and locations also require models to remember exactly which detail belongs to which event. We introduce EgoProbe, a benchmark that evaluates streaming egocentric memory along these two complementary dimensions: long-range retention, i.e., how far, and event-specific recall under interference, i.e., how exact. Built on long-form egocentric videos, EgoProbe uses an event-grounded data engine to extend recall intervals, identify related events, and construct event-grounded distractors, followed by human verification. It contains two subsets: EgoProbe-Base, which evaluates long-range recall without intervening related events, and EgoProbe-Echo, which challenges models with similar intervening events and event-grounded distractors. EgoProbe-Echo adopts coarse-to-fine multi-turn probing: the first turn evaluates coarse-grained event recall, while the second tests whether the model can retrieve the target-specific detail associated with the designated event. Extensive evaluation across diverse models reveals substantial limitations in long-range recall, with further challenges in recovering target-event details among related experiences. Even when models correctly answer coarse-grained questions, they often fail to recall the specific details of the designated event. EgoProbe exposes this underexplored failure mode and provides a testbed for advancing long-range and exact memory in streaming egocentric video understanding.

Haokai Pang, Yuqian Fu, Yifei Wang +4

HKUST · KAUST · ECNU +1

Egocentric Video Understanding Streaming Video Understanding Episodic Memory

Ask PDF

Jun 13, 2026

Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams

We introduce UCS-Bench, a dataset spanning 170+ hours of egocentric visual observations with 8.1K+ timestamped questions for diagnosing User-Centric Continual Spatial intelligence in egocentric video streams. UCS-Bench targets a new problem that emphasizes dynamic spatial reasoning, long-term memory, and their alignment with users' real-time locations. We propose DirectMe, a framework that incrementally constructs and maintains a structured spatial memory from streaming egocentric observations. DirectMe enables robust tracking and recall of object locations, all relative to the user's movement over time. By tightly coupling visual perception with memory updates and spatial reasoning, our approach supports long-horizon queries that require recalling interactions, resolving viewpoint-induced ambiguities, and adapting to dynamic scenes. Our experiments show that DirectMe significantly improves the spatial reasoning of leading multimodal LLMs; it also surpasses many spatially aware and long-form streaming video models. We hope our benchmark and solution will advance spatial intelligence research for egocentric AI assistants. Data and code are available at https://github.com/cocowy1/UCS-Bench.

Yun Wang, Junbin Xiao, Han Lyu +6

Department of Computer Science, City University of Hong Kong · University of Science and Technology of China · National University of Singapore +4

Spatial Reasoning Benchmarks Streaming Video Understanding Egocentric Video QA Egocentric Video Understanding Spatial Reasoning

Ask PDF

Abstract

Egocentric Video Understanding

Explore similar work

Have I Scene This Before? Spatially Grounded Conversational Memory for Complex Queries in Egocentric Assistants

EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos

Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams