Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition

Xia Hu; Honglei Zhuang; Brian Potetz; Alireza Fathi; Bo Hu; Babak Samari; Howard Zhou

cs.CVFeb 22, 2026

Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition

Authors: Xia Hu, Honglei Zhuang, Brian Potetz, Alireza Fathi, Bo Hu, Babak Samari, Howard Zhou

Organizations: Google DeepMind

Abstract

As large language models increasingly power personal assistants, users expect them to reason over multimodal life histories, from recognizing people to understanding events to aggregating patterns, yet existing benchmarks primarily target concept-level recognition. We introduce Life-Bench, a fully synthetic, human-verified multimodal benchmark of over 11,800 question-answer pairs across 10 tasks, organized by required evidence scope: concept identification, event understanding, and aggregated reasoning. The benchmark's photo-centric personal histories are distributionally aligned with real user accounts under embedding statistic. The interconnected structure of personal data invites graph-based solutions; we propose LifeGraph, a personal knowledge graph framework providing structured retrieval with on-demand access to source visual evidence, showing particular promise on event and aggregated tasks. Systematic evaluation of four retrieval paradigms on Life-Bench demonstrates that accuracy degrades sharply with evidence scope, falling below 0.40 on aggregated tasks, and that no single paradigm dominates across categories. Performance beyond concept recognition remains modest for all evaluated methods, establishing personalization over multimodal histories as an open challenge and Life-Bench as a testbed for future progress.

Figures & tables

Figure 1: A Life-Bench Vaccount and representative examples of 10 tasks. Each Vaccount contains personal concepts and timestamped multimodal histories. Tasks are grouped by categories: Concept Identification (orange, ), Event Understanding (blue, ) and Aggregated Reasoning (purple, ).

Statistic	Number
Total Questions	11,811
* Multiple-choice Questions	1,048
* Binary Questions	1,613
* Open Questions	9,150
Total Tasks	10
* Concept Identification	3

Table 1: Key statistics of Life-Bench.

Panel of Figure 2 — Figure 2: Overview of Life-Bench. (a) Distribution of questions across tasks and three categories. (b) The construction pipeline.

	Accept	Not Sure	Reject
Image Fidelity	88.20%	—	11.80%
Reject reasons (% of rejected images)
Face/Body Distortion			41.00%
Artifacts / Glitches			32.40%
Other Visual Flaws			26.60%
QA Correctness	95.55%	2.35%	2.10%

Table 2: Full-coverage human validation statistics of Life-Bench. All items marked Not Sure and Reject are re-examined and corrected by authors.

Metric	Vaccount	Real (YFCC)
FID(100)	0.56	0.53
Avg. intra-account Dist	0.91	0.96

Table 3: Distributional realism and per-account diversity of Life-Bench Vaccounts compared with real YFCC accounts.

Method	Concept			Event				Aggregated			Overall
Method	Text Concept QA	Visual Concept Recognition	Concept VQA	Scene & Activity	Direct Person-Centric	Relational Person-Centric	Fine-Grained Scene	Preference & Persona	Relational Temporal	Frequency & Counting	Overall
Random	0.2500	0.5000	0.2500	0.0000	0.0000	0.0000	0.0000	0.2003	0.0000	0.0000	0.1200
Close-book	0.2454	0.5111	0.2696	0.0848	0.0539	0.1341	0.1560	0.4373	0.0662	0.0755	0.2034
Full-context	0.9730	0.9722	0.9508	0.9143	0.9095	0.9141	0.9014	0.8856	0.8523	0.3823	0.8655
Oracle	0.9816	0.9826	0.9570	0.9324	0.9292	0.9269	0.9100	–	–	–	0.9457 ∗
Oracle-Gemma	0.8527	0.8698	0.6170	0.8093	0.7007	0.7130	0.7410	–	–	–	0.7576 ∗

Table 4: Evaluation results across all Life-Bench tasks. Retrieval-based methods are compared for context size

k\in\{3,5\}

. Tasks are grouped into 3 categories. The best and second-best scores among each setting are in bold and underlined respectively. ( ∗ Oracle overall averages over seven concept and event tasks since aggregate does not have subset golden reference)

Figure 3: Per-Vaccount accuracy of retrieval methods at

k=5

(LifeGraph:

d=2

), grouped by task category. Bars show the mean across 10 Vaccounts, dots show individual Vaccounts.

Method	Concept		Event		Overall
Method	R@3	R@5	R@3	R@5	R@3	R@5
BM25 [ 50 ]	0.4626	0.5330	0.4785	0.5371	0.4717	0.5353
RAG-Cap [ 34 ]	0.4307	0.4808	0.3665	0.4580	0.3940	0.4678
R2P [ 12 ]	0.4070	0.4238	0.3823	0.4299	0.3929	0.4273
RAP [ 25 ]	0.4354	0.4653	0.4065	0.4572	0.4188	0.4607
HippoRAG2 [ 23 ]	0.4465	0.5128	0.3042	0.4065	0.3652	0.4521

Table 5: Event-level Recall@3/5 on Life-Bench, where retrieving any image from a gold event counts as a hit. Recall is used as an auxiliary retrieval diagnostic rather than a directly comparable method ranking, since LifeGraph may answer from facts encoded in the graph without retrieving the original source records.

Figure 4: Accuracy of LifeGraph d=3,k=5 with (w/ Ref) and without (w/o Ref) multimodal source retrieval across tasks. Tasks are denoted by its acronym.

Figure 5: Effect of retrieval depth

d

((a), evaluated at

k=3

) and width

k

((b), evaluated at

d=2

) on LifeGraph performance. Tasks are denoted by its acronym.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Task	Reasoning Scope / Focus	Input Format	Answer Format	# Questions
Concept Identification	Text Concept QA	Multi-hop Persona & Visual Attribute Retrieval	Text	Multiple-Choice	163
	Visual Concept Recognition	Multi-hop Identity Verification	Text + Image	Binary	1,613
	Concept VQA	Multi-hop Grounded Visual Reasoning	Text + Image	Multiple-Choice	885
Event Understanding	Scene and Activity	Holistic Event Context & Activity Understanding	Text	Open Generation	2,265
	Direct Person-Centric	Concept Appearance & Actions in Events	Text	Open Generation	2,245
	Relational Person-Centric	Relational Concept Localization & Interactions	Text	Open Generation	1,812

Appendix

Table 6: Comprehensive breakdown of evaluation tasks in Life-Bench.

Figure 6: Distribution of the evaluated Life-Bench suite across (a) input formats and (b) output formats.

Figure 7: Graph structure of a constructed LifeGraph.

Figure 8: An example of the LifeGraph retrieval and reasoning process.

Figure 9: Average effective retrieve depth (

\bar{d}

) of LifeGraph under different retrieval depth limit (

d

Method	$N$	GPT-5.4 ( $\%$ )	Gemini ( $\%$ )	$\Delta(\%)$	Agree(%)
Closed-book	418	23.0	23.4	$-$ 0.4	99.0
RAP-Gemma k=5	418	28.7	30.9	$-$ 2.2	96.9
RAG-Cap k=5	418	34.4	40.7	$-$ 6.2	93.8
LifeGraph k=5,d=2	418	41.1	45.2	$-$ 4.1	93.5
Oracle Flash	418	86.4	87.8	$-$ 1.4	97.6

Appendix

Table 7: Per-method accuracy under GPT-5.4 vs. Gemini 3.6 Flash judges on the same 2,090 instances. Negative

\Delta

indicates GPT-5.4 assigns lower accuracy (stricter grading). The method ranking is identical under both judges.

Task	$N$	Agree	Agree %
Scene & Activity	300	285	95.0
Direct Person-Centric	300	287	95.7
Relational Person-Centric	300	268	89.3
Fine-Grained Scene	300	295	98.3
Preference & Persona	300	288	96.0
Frequency & Counting	295	295	100.0

Appendix

Table 8: Per-task agreement between Gemini 3.6 Flash and GPT-5.4 judges on 2,090 uniformly sampled open-ended instances. Three concept identification tasks use exact-match scoring and are excluded.

Figure 10: Per-Vaccount accuracy of retrieval methods at

k=5

(LifeGraph:

d=2

), break down by per task. Bars show the mean across 10 Vaccounts, dots show individual Vaccounts.

Method	Concept			Event				Aggregated			Overall
Method	Text Concept QA	Visual Concept Recognition	Concept VQA	Scene & Activity	Direct Person-Centric	Relational Person-Centric	Fine-Grained Scene	Preference & Persona	Relational Temporal	Frequency & Counting	Overall
Full-context	0.9707 $\pm$ 0.039	0.9712 $\pm$ 0.012	0.9403 $\pm$ 0.052	0.9145 $\pm$ 0.012	0.9091 $\pm$ 0.027	0.9146 $\pm$ 0.043	0.9012 $\pm$ 0.021	0.8853 $\pm$ 0.058	0.8517 $\pm$ 0.082	0.3871 $\pm$ 0.144	0.8646 $\pm$ 0.017
Oracle	0.9786 $\pm$ 0.048	0.9815 $\pm$ 0.015	0.9525 $\pm$ 0.038	0.9318 $\pm$ 0.020	0.9286 $\pm$ 0.032	0.9271 $\pm$ 0.035	0.9100 $\pm$ 0.026	–	–	–	0.9443 ∗ $\pm$ 0.012
Oracle-Gemma	0.8457 $\pm$ 0.099	0.8691 $\pm$ 0.063	0.6315 $\pm$ 0.109	0.8093 $\pm$ 0.023	0.7005 $\pm$ 0.035	0.7138 $\pm$ 0.082	0.7401 $\pm$ 0.045	–	–	–	0.7586 ∗ $\pm$ 0.039
$\text{BM25}_{k=3}$	0.8138 $\pm$ 0.112	0.8139 $\pm$ 0.073	0.5850 $\pm$ 0.066	0.6160 $\pm$ 0.063	0.3416 $\pm$ 0.062	0.4416 $\pm$ 0.069	0.4738 $\pm$ 0.044	0.4784 $\pm$ 0.105	0.1179 $\pm$ 0.075	0.1345 $\pm$ 0.073	0.4816 $\pm$ 0.025
$\text{RAG-Cap}_{k=3}$	0.8205 $\pm$ 0.148	0.7366 $\pm$ 0.054	0.5252 $\pm$ 0.107	0.4369 $\pm$ 0.039	0.2515 $\pm$ 0.032	0.3376 $\pm$ 0.036	0.5253 $\pm$ 0.040	0.6374 $\pm$ 0.067	0.1249 $\pm$ 0.086	0.1397 $\pm$ 0.067	0.4535 $\pm$ 0.034

Appendix

Table 9: Evaluation results across all Life-Bench tasks reported as

\text{mean}\pm\text{std}

across 10 VAccounts (

N=10

). Retrieval-based methods are compared for context size

k\in\{3,5\}

	Concept						Event
	Text Concept QA		Visual Concept		Concept VQA		Scene & Activity		Direct Person		Rel. Person		Fine-Grained Scene
Method	R@3	R@5	R@3	R@5	R@3	R@5	R@3	R@5	R@3	R@5	R@3	R@5	R@3	R@5
BM25 [ 50 ]	0.5456	0.6246	0.5157	0.5733	0.3265	0.4010	0.6021	0.6541	0.4989	0.5420	0.4499	0.5329	0.3629	0.4192
RAG-Cap [ 34 ]	0.5404	0.6193	0.4638	0.5048	0.2879	0.3183	0.3460	0.4164	0.3703	0.4687	0.2608	0.3486	0.4890	0.5984
R2P-Gemma [ 12 ]	0.3474	0.3781	0.4348	0.4429	0.4388	0.4505	0.1960	0.2358	0.3841	0.4251	0.3484	0.3889	0.6007	0.6698
RAP-Gemma [ 25 ]	0.3711	0.3982	0.4582	0.5007	0.4768	0.4969	0.2295	0.2714	0.3807	0.4189	0.3293	0.3741	0.6863	0.7644

Appendix

Table 10: Per-task event-level Recall@

k

on Life-Bench. Retrieving any image from a gold event counts as a hit. The best and second-best scores per column are in bold and underlined , respectively.

Figure 11: Evaluation Prompt for Auto-Rater (LLM-as-a-Judge).

Figure 12: QA Generation Prompt Skeleton for Concept Identification Tasks.

Figure 13: QA Generation Prompt Skeleton for Event Understanding Tasks.

Figure 14: QA Generation Prompt Skeleton for Aggregated Reasoning Tasks.

Figure 15: Question Template for Visual Concept Recognition.

Figure 16: Question Template for Concept VQA.

Figure 17: Question Template for Text Concept QA.

Figure 18: Question Template for Scene and Activity.

Figure 19: Question Template for Direct Person-Centric.

Figure 20: Question Template for Relational Person-Centric.

Figure 21: Question Template for Fine-Grained Scene.

Figure 22: Question Template for Frequency and Counting.

Figure 23: Question Template for Relational Temporal Reasoning.

Figure 24: Question Template for Preference and Persona.

Figure 25: Two-Turn Prompt for LifeGraph Stage 1 (Concept Subgraph Construction).

Figure 26: Two-Turn Prompt for LifeGraph Stage 2 (Event Subgraph Expansion).

Figure 27: LifeGraph Retrieval Prompt: Topic Entity Extraction ( TopEntities ).

Figure 28: LifeGraph Retrieval Prompt: Relation-Entity Joint Pruning ( Prune ).

Figure 29: LifeGraph Retrieval Prompt: Multimodal Reference Fetching ( FetchReference ).

Figure 30: LifeGraph Retrieval Prompt: Context Sufficiency Check ( Reasoning ).

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 508 papers

To Memories and Beyond: From Remembering to Knowing You across Long-Term Multimodal Personal Archives
Sep 11, 2026Wenqi Zhou, Zhuorui Yu, Kaiao Wen +6Multimodal Memory User Preference Modeling
University of Bristol · Memories.ai Research
LifeAgentBench: Benchmarking LLMs for Long-Horizon, Cross-Dimensional Lifestyle Health Reasoning
Jan 20, 2026Ye Tian, Zihao Wang, Onat Gungor +2Healthcare Long-Horizon Agent Evaluation
University of California San Diego, La Jolla, CA, USA · Google Research, Mountain View, CA, USA
PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrieval
Mar 2, 2026Tianyi Xu, Rong Shan, Junjie Wu +11Multimodal IR Image Retrieval
Shanghai Jiao Tong University, Shanghai, China · OPPO, Shenzhen, China

Sep 11, 2026J/K move · Enter open · S save

To Memories and Beyond: From Remembering to Knowing You across Long-Term Multimodal Personal Archives

Wenqi Zhou, Zhuorui Yu, Kaiao Wen, Hao Zheng+5

University of Bristol · Memories.ai Research

Multimodal Memory User Preference Modeling Temporal Reasoning in Language Models

Open paper AskPDF arXiv

As AI systems evolve into personalized digital companions, a central capability is reasoning over a user's long-term personal history: not merely storing past events, but tracking longitudinal experiences and evolving preferences. Progress here is bottlenecked by evaluation, existing long-term memory benchmarks are largely synthetic and text-only, they overlook the visual records that anchor everyday human memory, lack the authentic and causally connected longitudinal data that real personalization demands, and consequently remain confined to shallow factual recall. We introduce ReaLMem (Real-world Long-term Multimodal Memory), the first benchmark built from authentic multi-year personal visual archives, paired with first-person subjective annotations. ReaLMem evaluates models across three cognitive tiers of increasing difficulty: factual recall, persona inference, and predictive personalization. We further propose ChronoProfiler, a temporal-weighting profiling module that computes temporal stability scores for user attributes and applies them as a salience prior, resolving conflicts among temporally inconsistent preferences and helping models compound multiple co-active preferences in complex personalized decisions. Extensive evaluation of frontier multimodal large language models (MLLMs) and memory systems on ReaLMem reveals predictive personalization as a consistent ceiling, exposes clear performance gaps and bottlenecks between MLLMs and memory systems, and shows that high-quality, temporally informed representations substantially improve personalization. Together, ReaLMem and ChronoProfiler provide an authentic testbed and a simple, effective mechanism for long-term personalization, laying a foundation for future research on lifelong AI companions.

Abstract

Multimodal Reasoning

Explore similar work

To Memories and Beyond: From Remembering to Knowing You across Long-Term Multimodal Personal Archives

LifeAgentBench: Benchmarking LLMs for Long-Horizon, Cross-Dimensional Lifestyle Health Reasoning

PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrieval