PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrieval

Tianyi Xu; Rong Shan; Junjie Wu; Jiadeng Huang; Teng Wang; Jiachen Zhu; Wenteng Chen; Minxin Tu

cs.IRMar 2, 2026

PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrieval

Authors: Tianyi Xu, Rong Shan, Junjie Wu, Jiadeng Huang, Teng Wang, Jiachen Zhu, Wenteng Chen, Minxin Tu,

Organizations: Shanghai Jiao Tong University, Shanghai, China · OPPO, Shenzhen, China

Abstract

Personal photo albums are not merely collections of static images but living, ecological archives defined by temporal continuity, social entanglement, and rich metadata, which makes the personalized photo retrieval non-trivial. However, existing retrieval benchmarks rely heavily on context-isolated web snapshots, failing to capture the multi-source reasoning required to resolve authentic, intent-driven user queries. To bridge this gap, we introduce PhotoBench, the first benchmark constructed from authentic, personal albums. It is designed to shift the paradigm from visual matching to personalized multi-source intent-driven reasoning. Based on a rigorous multi-source profiling framework, which integrates visual semantics, spatial-temporal metadata, social identity, and temporal events for each image, we synthesize complex intent-driven queries rooted in users' life trajectories. Extensive evaluation on PhotoBench exposes two critical limitations: the modality gap, where unified embedding models collapse on non-visual constraints, and the source fusion paradox, where agentic systems perform poor tool orchestration. These findings indicate that the next frontier in personal multimodal retrieval lies beyond unified embeddings, necessitating robust agentic reasoning systems capable of precise constraint satisfaction and multi-source fusion. Our PhotoBench is available.

Figures & tables

Dataset

Query Dimensions

Image Dimensions

One-to-Many

Unmatched

Narrative

Personalized

Multi-Source

Reasoning

Volume

Personalized

Metadata

Temporal

Quality Var.

Near-dup.

Candidates

MSCOCO_t2i ( Lin et al., 2014 )

✗

500K

✗

123K

Flickr30k ( Plummer et al., 2015 )

✗

16K

✗

31K

Winoground ( Thrush et al., 2022 )

✗

✓

800

✗

INQUIRE ( Vendrow et al., 2024 )

✓

✗

✓

250

✗

✓

VisDial ( Das et al., 2017 )

✗

✓

✗

✓

1.2M

✗

100

Table 1. We compare PhotoBench with existing vision-language or multimodal retrieval datasets, highlighting its unique characteristics for real-world personalized photo retrieval tasks. Quality Var. : Image Quality Variance ( e.g. , blur, noise); Near-dup. : Near-duplicate burst shots.

Figure 1. The illustration of the dataset construction pipeline for PhotoBench .

Figure 2. The query distribution w.r.t. ground-truth count (left) and source-aware taxonomy (right) in PhotoBench .

Multimodal Embedding Models
		Chinese								English
Model	Size (B)	R@1	R@5	R@10	R@20	N@1	N@5	N@10	N@20	R@1	R@5	R@10	R@20	N@1	N@5	N@10	N@20
clip-ViT-B-32-multilingual-v1	0.14	1.2	4.1	6.1	8.8	3.2	4.0	4.7	5.5	10.7	25.5	33.1	41.3	24.5	25.6	27.9	30.3
siglip2-base-patch16-224	0.38	16.4	33.7	40.0	47.3	33.1	34.6	36.0	38.2	19.8	39.9	46.7	53.1	40.2	41.5	42.9	44.7
siglip2-giant-opt-patch16-256	1.88	20.8	40.5	47.6	54.5	42.0	42.4	44.1	46.0	23.8	45.1	50.7	56.9	47.5	48.1	48.8	50.5
VLM2Vec	8.29	23.1	44.4	52.4	60.0	46.0	47.0	48.9	51.3	22.0	43.3	51.1	58.1	43.0	44.9	46.9	49.1

Table 2. The retrieval performance on PhotoBench using top-

k

ranking metrics. We report Recall@K (R@K) and NDCG@K (N@K) averaged over all albums. Best result of each category is in bold, and the second best is underlined.

Real-world Mobile Gallery Systems
System	Normal Query			Zero-GT Query
System	P	R	F1	Rej-P	Rej-R	Rej-F1
Phone A	22.8	47.3	25.6	18.2	73.2	29.1
Phone B	16.0	17.2	14.4	12.4	80.5	21.5
Phone C	33.7	50.1	34.4	20.1	48.8	28.5
Phone D	34.9	47.7	34.5	25.5	79.7	38.6

Table 3. The retrieval performance on PhotoBench using set-based metrics. Best result of each category is in bold, while the second best is underlined. P , R , Rej-P , Rej-R and Rej-F1 denote Precision , Recall , F1 , Reject-Precision , Reject-Recall and Reject-F1 , respectively.

Type	Agent(A)	Multimodal(M)	Text(T)	$\Delta$ (A-M)	$\Delta$ (A-T)
$S_{V}$	71.1	72.3	75.0	-1.2	-3.8
$S_{M}$	57.9	7.2	7.9	+50.8	+50.1
$S_{F}$	75.1	11.7	8.4	+63.4	+66.7
$S_{VM}$	63.4	63.4	65.5	+0.0	-2.1
$S_{VF}$	40.1	52.6	51.0	-12.4	-10.9
$S_{MF}$	60.2	12.0	11.2	+48.2	+49.0

Table 4. Recall@10 performance decomposition by source-aware query types.

\Delta

(A-M) and

\Delta

(A-T) denote performance gaps between agents and multimodal and caption-based text embedding models, respectively.

Tools	$S_{V}$	$S_{M}$	$S_{F}$	$S_{VM}$	$S_{VF}$	$S_{MF}$	$S_{VMF}$
$T_{V}$	30.5	2.6	8.7	31.4	32.1	6.1	35.1
$T_{V}+T_{M}$	32.3	54.7	12.9	34.1	33.2	25.5	33.4
$T_{V}+T_{F}$	32.2	3.3	69.0	29.7	26.8	30.1	25.4
$T_{V}+T_{M}+T_{F}$	36.9	69.7	90.1	44.7	30.1	76.8	32.5

Table 5. Tool ablation study on the best agentic system with Qwen3-235B-A22B. F1 scores are reported across query types as different types of tools are incrementally enabled. Note that the set composition tool (

T_{S}

) is always active.

Type	Agent (Best)	Pho.A	Pho.B	Pho.C	Pho.D	Pho.E	Pho.F
$S_{V}$	37.6	33.2	20.5	34.6	34.1	40.6	36.4
$S_{M}$	72.3	39.8	17.9	67.5	52.6	75.1	76.7
$S_{F}$	93.4	1.8	1.1	1.7	6.7	0.0	10.3
Avg.	63.5	28.8	15.0	40.4	31.7	45.2	46.3
$S_{VM}$	45.7	20.0	10.9	31.7	32.2	38.3	40.4
$S_{VF}$	30.9	15.1	7.2	22.7	25.9	20.8	22.2

Table 6. We report F1 scores of the best agent (Qwen3-235B-A22B) and mobile gallery systems as query complexity increases from single- to dual- and triple-source queries.

\Delta

values indicate the performance degradation, exposing the Source Fusion Paradox in commercial engines.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3. Linguistic comparison with traditional benchmarks. We evaluate four dimensions: Avg Query Length (total tokens), Noun Density (proportion of entities), Avg Syntactic Depth (grammatical complexity), and Lexical Diversity (MTLD, measuring vocabulary richness independent of text length). PhotoBench queries are structurally streamlined (lower length and depth) yet lexically diverse, reflecting search-style rather than narrative-style language.

Album	# Images	Metadata (%)	Portrait (%)	Core Faces
Album 1	1,069	97.7	7.3	10
Album 2	1,466	75.7	43.5	2
Album 3	1,047	76.7	24.6	8
Total	3582	83.4	25.1	20

Appendix

Table 7. Per-album statistics. Albums vary in social density (Core Faces) and visual patterns (Portrait ratio), ensuring benchmark diversity.

Figure 4. Temporal distribution of photos across albums.

Dimension	Fact-based (Explicit)	Cognitive (Implicit)
Location	“in Shanghai”	“at home,” “at the gym”
Time	“in May 2024”	“during Spring Festival”
Person	“Zhang Wei”	“with my girlfriend”
Object	“a red car”	“the parking spot before the concert”
Concept	“sunset”	“cozy moments with family”

Appendix

Table 8. The Dual-Layer Taxonomy across five dimensions. Fact-based queries involve explicit signal matching; Cognitive queries require reasoning synthesis.

Panel of Figure 5 — Figure 5. Supplementary query statistics. (a) Distribution of labels across semantic dimensions. (b) Distribution of Cognitive labels per query (0 indicates a pure Fact-based query).

Method Type	Fact	Cognitive	Gap
Best Embedding	35.24	63.76	-28.52
Best Agent	78.60	70.98	+7.62

Appendix

Table 9. Performance by reasoning depth (Recall@10). Embedding models show a negative gap (better on Cognitive), while agents show a slight positive gap.

Type	Agent(A)	Multimodal(M)	Text(T)	$\Delta$ (A-M)	$\Delta$ (A-T)
$S_{V}$	71.5	77.3	73.3	-5.8	-1.9
$S_{M}$	53.4	6.5	7.8	+46.9	+45.6
$S_{F}$	69.4	12.4	4.9	+57.1	+64.5
$S_{VM}$	62.0	65.8	63.4	-3.8	-1.3
$S_{VF}$	42.6	57.8	47.5	-15.1	-4.9
$S_{MF}$	57.0	13.0	8.3	+44.0	+48.7

Appendix

Table 10. Recall@10 performance decomposition by source-aware query types (English Version).

Panel of Figure 6 — Figure 6. Case study illustrating the Modality Gap and Source Fusion Paradox. ✓ =success, ~ =partial, ✗ =failure.

Figure 7. The proposed hierarchical agentic framework architecture for photo retrieval.

System	Normal Query			Zero-GT Query
System	P	R	F1	Rej-P	Rej-R	Rej-F1
Ours	67.3	70.2	63.3	50.8	53.3	52.0
ToolACE-2	38.3	54.7	39.0	15.1	35.0	21.1
Qwen3-8B	37.6	68.1	41.0	16.8	16.7	16.7
Qwen3-32B	39.6	66.3	41.4	16.7	25.8	20.3
DeepSeek-v3	38.5	66.2	40.7	11.8	10.0	10.8

Appendix

Table 11. Performance of our proposed agentic framework on PhotoBench . All values in %. “ P ", “ R ", “ Rej-P ", “ Rej-R " and “ Rej-F1 " denote Precision, Recall, F1, Reject-Precision, Reject-Recall and Reject-F1, respectively.

Multimodal IR

Multimodal information retrieval finds relevant images, videos, documents, audio, or other media using queries that may combine different modalities.

See all 128 papers

Weaving Visual Narratives: Agentic Image Bundle Composition Beyond Atomic Visual Matching
Aug 27, 2026Rong Shan, Tianyi Xu, Congmin Zheng +9Composed Image Retrieval Multimodal Search Agents
Shanghai Jiao Tong University · Shanghai Innovation Institute · OPPO
Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition
Feb 22, 2026Xia Hu, Honglei Zhuang, Brian Potetz +4Multimodal Reasoning Multimodal QA
Google DeepMind
Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval
Mar 1, 2026Xuan Lu, Kangle Li, Haohang Huang +3Multimodal IR Cross-Modal Retrieval
Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative +1

Aug 27, 2026J/K move · Enter open · S save

Weaving Visual Narratives: Agentic Image Bundle Composition Beyond Atomic Visual Matching

Rong Shan, Tianyi Xu, Congmin Zheng, Wenteng Chen+8

Shanghai Jiao Tong University · Shanghai Innovation Institute · OPPO

Composed Image Retrieval Multimodal Search Agents

Open paper AskPDF arXiv

Image retrieval has traditionally been formulated as a point-wise matching problem, where each candidate image is scored in isolation. However, this atomic paradigm fails to capture the complexity of human search intent within personal photo collections, where users often seek compact visual stories bound by structural relations rather than isolated snapshots. To address this limitation, we introduce Image Bundle Composition (IBC), a novel paradigm that shifts the objective from ranking individual images to dynamically composing cohesive image bundles from a massive, unstructured photo pool. Since target bundles are not predefined, IBC presents a severe combinatorial explosion challenge and demands modeling non-decomposable joint relevance. To establish this paradigm, we construct IBCBench, the first IBC benchmark dataset containing 109,467 images and 667 verified queries, built via a semi-automated verification pipeline. Furthermore, we propose BundleWeaver, an agentic framework that reformulates IBC as query-conditioned incremental hyperedge discovery. By employing a Large Language Model to adaptively search for missing relational roles and utilizing a Vision-Language Model for whole-bundle verification, BundleWeaver effectively navigates the combinatorial space. Extensive experiments demonstrate that while state-of-the-art embedding models and static decompose-and-rerank paradigms suffer from relational blindness, BundleWeaver achieves substantial performance gains, highlighting the necessity of shifting from atomic scoring to dynamic relational composition. Our dataset and code are available.

Abstract

Multimodal IR

Explore similar work

Weaving Visual Narratives: Agentic Image Bundle Composition Beyond Atomic Visual Matching

Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition

Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval