Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition
Organizations: Google DeepMind
Abstract
As large language models increasingly power personal assistants, users expect them to reason over multimodal life histories, from recognizing people to understanding events to aggregating patterns, yet existing benchmarks primarily target concept-level recognition. We introduce Life-Bench, a fully synthetic, human-verified multimodal benchmark of over 11,800 question-answer pairs across 10 tasks, organized by required evidence scope: concept identification, event understanding, and aggregated reasoning. The benchmark's photo-centric personal histories are distributionally aligned with real user accounts under embedding statistic. The interconnected structure of personal data invites graph-based solutions; we propose LifeGraph, a personal knowledge graph framework providing structured retrieval with on-demand access to source visual evidence, showing particular promise on event and aggregated tasks. Systematic evaluation of four retrieval paradigms on Life-Bench demonstrates that accuracy degrades sharply with evidence scope, falling below 0.40 on aggregated tasks, and that no single paradigm dominates across categories. Performance beyond concept recognition remains modest for all evaluated methods, establishing personalization over multimodal histories as an open challenge and Life-Bench as a testbed for future progress.
Figures & tables
| Statistic | Number |
| Total Questions | 11,811 |
| * Multiple-choice Questions | 1,048 |
| * Binary Questions | 1,613 |
| * Open Questions | 9,150 |
| Total Tasks | 10 |
| * Concept Identification | 3 |
| Accept | Not Sure | Reject | |
| Image Fidelity | 88.20% | — | 11.80% |
| Reject reasons (% of rejected images) | |||
| Face/Body Distortion | 41.00% | ||
| Artifacts / Glitches | 32.40% | ||
| Other Visual Flaws | 26.60% | ||
| QA Correctness | 95.55% | 2.35% | 2.10% |
| Metric | Vaccount | Real (YFCC) |
| FID(100) | 0.56 | 0.53 |
| Avg. intra-account Dist | 0.91 | 0.96 |
| Method | Concept | Event | Aggregated | Overall | |||||||
| Text Concept QA | Visual Concept Recognition | Concept VQA | Scene & Activity | Direct Person-Centric | Relational Person-Centric | Fine-Grained Scene | Preference & Persona | Relational Temporal | Frequency & Counting | ||
| Random | 0.2500 | 0.5000 | 0.2500 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2003 | 0.0000 | 0.0000 | 0.1200 |
| Close-book | 0.2454 | 0.5111 | 0.2696 | 0.0848 | 0.0539 | 0.1341 | 0.1560 | 0.4373 | 0.0662 | 0.0755 | 0.2034 |
| Full-context | 0.9730 | 0.9722 | 0.9508 | 0.9143 | 0.9095 | 0.9141 | 0.9014 | 0.8856 | 0.8523 | 0.3823 | 0.8655 |
| Oracle | 0.9816 | 0.9826 | 0.9570 | 0.9324 | 0.9292 | 0.9269 | 0.9100 | – | – | – | 0.9457 ∗ |
| Oracle-Gemma | 0.8527 | 0.8698 | 0.6170 | 0.8093 | 0.7007 | 0.7130 | 0.7410 | – | – | – | 0.7576 ∗ |
| Method | Concept | Event | Overall | |||
| R@3 | R@5 | R@3 | R@5 | R@3 | R@5 | |
| BM25 [ 50 ] | 0.4626 | 0.5330 | 0.4785 | 0.5371 | 0.4717 | 0.5353 |
| RAG-Cap [ 34 ] | 0.4307 | 0.4808 | 0.3665 | 0.4580 | 0.3940 | 0.4678 |
| R2P [ 12 ] | 0.4070 | 0.4238 | 0.3823 | 0.4299 | 0.3929 | 0.4273 |
| RAP [ 25 ] | 0.4354 | 0.4653 | 0.4065 | 0.4572 | 0.4188 | 0.4607 |
| HippoRAG2 [ 23 ] | 0.4465 | 0.5128 | 0.3042 | 0.4065 | 0.3652 | 0.4521 |
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Task | Reasoning Scope / Focus | Input Format | Answer Format | # Questions |
| Concept Identification | Text Concept QA | Multi-hop Persona & Visual Attribute Retrieval | Text | Multiple-Choice | 163 |
| Visual Concept Recognition | Multi-hop Identity Verification | Text + Image | Binary | 1,613 | |
| Concept VQA | Multi-hop Grounded Visual Reasoning | Text + Image | Multiple-Choice | 885 | |
| Event Understanding | Scene and Activity | Holistic Event Context & Activity Understanding | Text | Open Generation | 2,265 |
| Direct Person-Centric | Concept Appearance & Actions in Events | Text | Open Generation | 2,245 | |
| Relational Person-Centric | Relational Concept Localization & Interactions | Text | Open Generation | 1,812 |
| Method | GPT-5.4 ( ) | Gemini ( ) | Agree(%) | ||
| Closed-book | 418 | 23.0 | 23.4 | 0.4 | 99.0 |
| RAP-Gemma k=5 | 418 | 28.7 | 30.9 | 2.2 | 96.9 |
| RAG-Cap k=5 | 418 | 34.4 | 40.7 | 6.2 | 93.8 |
| LifeGraph k=5,d=2 | 418 | 41.1 | 45.2 | 4.1 | 93.5 |
| Oracle Flash | 418 | 86.4 | 87.8 | 1.4 | 97.6 |
| Task | Agree | Agree % | |
| Scene & Activity | 300 | 285 | 95.0 |
| Direct Person-Centric | 300 | 287 | 95.7 |
| Relational Person-Centric | 300 | 268 | 89.3 |
| Fine-Grained Scene | 300 | 295 | 98.3 |
| Preference & Persona | 300 | 288 | 96.0 |
| Frequency & Counting | 295 | 295 | 100.0 |
| Method | Concept | Event | Aggregated | Overall | |||||||
| Text Concept QA | Visual Concept Recognition | Concept VQA | Scene & Activity | Direct Person-Centric | Relational Person-Centric | Fine-Grained Scene | Preference & Persona | Relational Temporal | Frequency & Counting | ||
| Full-context | 0.9707 0.039 | 0.9712 0.012 | 0.9403 0.052 | 0.9145 0.012 | 0.9091 0.027 | 0.9146 0.043 | 0.9012 0.021 | 0.8853 0.058 | 0.8517 0.082 | 0.3871 0.144 | 0.8646 0.017 |
| Oracle | 0.9786 0.048 | 0.9815 0.015 | 0.9525 0.038 | 0.9318 0.020 | 0.9286 0.032 | 0.9271 0.035 | 0.9100 0.026 | – | – | – | 0.9443 ∗ 0.012 |
| Oracle-Gemma | 0.8457 0.099 | 0.8691 0.063 | 0.6315 0.109 | 0.8093 0.023 | 0.7005 0.035 | 0.7138 0.082 | 0.7401 0.045 | – | – | – | 0.7586 ∗ 0.039 |
| 0.8138 0.112 | 0.8139 0.073 | 0.5850 0.066 | 0.6160 0.063 | 0.3416 0.062 | 0.4416 0.069 | 0.4738 0.044 | 0.4784 0.105 | 0.1179 0.075 | 0.1345 0.073 | 0.4816 0.025 | |
| 0.8205 0.148 | 0.7366 0.054 | 0.5252 0.107 | 0.4369 0.039 | 0.2515 0.032 | 0.3376 0.036 | 0.5253 0.040 | 0.6374 0.067 | 0.1249 0.086 | 0.1397 0.067 | 0.4535 0.034 | |
| Concept | Event | |||||||||||||
| Text Concept QA | Visual Concept | Concept VQA | Scene & Activity | Direct Person | Rel. Person | Fine-Grained Scene | ||||||||
| Method | R@3 | R@5 | R@3 | R@5 | R@3 | R@5 | R@3 | R@5 | R@3 | R@5 | R@3 | R@5 | R@3 | R@5 |
| BM25 [ 50 ] | 0.5456 | 0.6246 | 0.5157 | 0.5733 | 0.3265 | 0.4010 | 0.6021 | 0.6541 | 0.4989 | 0.5420 | 0.4499 | 0.5329 | 0.3629 | 0.4192 |
| RAG-Cap [ 34 ] | 0.5404 | 0.6193 | 0.4638 | 0.5048 | 0.2879 | 0.3183 | 0.3460 | 0.4164 | 0.3703 | 0.4687 | 0.2608 | 0.3486 | 0.4890 | 0.5984 |
| R2P-Gemma [ 12 ] | 0.3474 | 0.3781 | 0.4348 | 0.4429 | 0.4388 | 0.4505 | 0.1960 | 0.2358 | 0.3841 | 0.4251 | 0.3484 | 0.3889 | 0.6007 | 0.6698 |
| RAP-Gemma [ 25 ] | 0.3711 | 0.3982 | 0.4582 | 0.5007 | 0.4768 | 0.4969 | 0.2295 | 0.2714 | 0.3807 | 0.4189 | 0.3293 | 0.3741 | 0.6863 | 0.7644 |
Multimodal Reasoning
Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.
See all 506 papers