Agent4POI: Agentic Context-Conditioned Affordance Reasoning for Multimodal Point-of-Interest Recommendation

Jinze Wang; Yangchen Zeng; Tiehua Zhang; Lu Zhang; Yuze Liu; Yongchao Liu; Xingjun Ma; Zhu Sun

cs.IRApr 3, 2026

Agent4POI: Agentic Context-Conditioned Affordance Reasoning for Multimodal Point-of-Interest Recommendation

Authors: Jinze Wang, Yangchen Zeng, Tiehua Zhang, Lu Zhang, Yuze Liu, Yongchao Liu, Xingjun Ma, Zhu Sun

Organizations: Tongji University · Swinburne University of Technology · Southeast University · Chengdu University of Information Technology ·

Abstract

We introduce Agent4POI, the first POI recommendation framework that generates context-conditioned multimodal representations at recommendation time, rather than relying on static POI embeddings pre-computed independently of context. Existing multimodal systems encode each POI once as a static embedding, a design that precludes reasoning about why the same cafe affords solo work on Monday but group celebration on Friday evening. We formally prove that no pre-computed encoder can satisfy context-sensitive ranking under standard bilinear scoring, motivating inference-time item-side representation. Agent4POI inverts this computation: given a situational context, a four-phase LLM agent generates dynamic, context-specific affordance queries (Phase 1) and executes a five-step cross-modal chain-of-thought over image, review, and metadata evidence (Phase 2). The resulting uncertainty-aware affordance representation is grounded in Gibsonian affordance theory. These cross-modal verdicts form a structured, uncertainty-adjusted affordance representation (Phase 3), which is aligned with user preferences via a semantic caching system for low-latency ranking (Phase 4). On three POI benchmarks and three evaluation configurations (standard, cold-start, context-shift), Agent4POI achieves a 23.2% relative gain over the strongest baseline and degrades by only 7.5% under context-shift versus 16--17% for the strongest baselines. In cold-start scenarios, Agent4POI outperforms the best content-based baseline by up to 2.4x, whereas ID-based methods fail to generalize.

Figures & tables

Figure 1. Agent4POI four-phase inference pipeline. Unlike prior methods that encode each POI into a fixed vector before any context is observed, Agent4POI delays representation until query time. A frozen LLM generates context-specific affordance queries in Phase 1; Phase 2 then executes a five-step cross-modal chain-of-thought over POI images, reviews, and metadata for each query. The resulting uncertainty-aware verdicts are assembled into a dynamic POI representation (Phase 3) that is aligned with user preference weights for final ranking (Phase 4). A system layer provides semantic caching and asynchronous prefetching for production deployment. Agent4POI four-phase pipeline: a frozen LLM converts situational context into affordance queries; a five-step cross-modal chain-of-thought interrogates POI images, reviews, and metadata; uncertainty-adjusted verdicts form a dynamic POI representation; user preference weights are aligned via dot-product scoring to produce ranked recommendations.

Property	Fixed Model ( $\mathbf{e}_{p}$ )	VLM+Attn	Agent4POI $\mathcal{A}(p,c)$
Context-Sensitive Output	✗	Partial	✓
Inference-Time Extraction	✗	✗	✓
Cross-Modal Conflict Flag	✗	✗	✓
Explicit Uncertainty-Aware	✗	✗	✓
Zero-Shot Cold-Start Support	✗	✗	✓

Table 1. POI representation property comparison. Agent4POI satisfies five properties unmatched by static POI embeddings.

Dataset	#Users	#POIs	#Check-ins	Density
Foursquare-NYC	1,083	11,357	227,428	1.85%
Foursquare-TKY	2,293	7,833	573,703	3.19%
Yelp-Open	30,431	150,346	1,987,897	0.04%

Table 2. Dataset Statistics (Post 10-core Filtering).

	Foursquare-NYC			Foursquare-TKY			Yelp-Open
Method	R@5 ↑	R@10 ↑	NDCG@10 ↑	R@5 ↑	R@10 ↑	NDCG@10 ↑	R@5 ↑	R@10 ↑	NDCG@10 ↑
BPR	0.056	0.097	0.072	0.049	0.088	0.064	0.034	0.063	0.049
SASRec	0.098	0.163	0.124	0.089	0.151	0.111	0.058	0.108	0.082
GETNext	0.146	0.227	0.179	0.132	0.216	0.167	0.094	0.148	0.118
MMPOI †	0.178	0.264	0.213	0.161	0.249	0.196	0.109	0.171	0.139
IM-POI †	0.214	0.307	0.252	0.194	0.289	0.231	0.124	0.198	0.162

Table 3. Standard evaluation. Best per column in bold .

\dagger

: multimodal;

\ddagger

: LLM reasoning;

*

: stat. significant over strongest baseline ( 3 independent seeds).

	Foursquare-NYC		Foursquare-TKY		Yelp-Open
Method	R@10	NDCG@10	R@10	NDCG@10	R@10	NDCG@10
BPR	0.005	0.004	0.005	0.003	0.003	0.002
MMPOI †	0.098	0.073	0.089	0.066	0.061	0.047
IM-POI †	0.134	0.102	0.121	0.092	0.083	0.065
LLM4POI ‡	0.169	0.131	0.153	0.117	0.104	0.084
Agent4POI †‡	0.313	0.249	0.297	0.235	0.196	0.158

Table 4. Cold-start evaluation. Test POIs have zero training check-in history. ID-based methods collapse; Agent4POI uses only raw multimodal content.

Method	NYC	TKY	Yelp
	NDCG@10 ↑	NDCG@10 ↑	NDCG@10 ↑
BPR	0.061	0.057	0.041
GETNext	0.151	0.141	0.093
IM-POI	0.211	0.196	0.133
CoT-Rec	0.224	0.208	0.143
Agent4POI	0.309	0.291	0.198

Table 5. Context-shift evaluation. Train: weekday; Test: weekend. Phase 1 of Agent4POI adapts queries at test time. Reported values are means across 3 seeds.

ID	Variant	R@10 ${\uparrow}$	NDCG@10 ${\uparrow}$
Full	Agent4POI (complete)	0.401	0.334
C1: Context-driven queries
A1	w/o Dynamic Queries (Fixed context)	0.312	0.259
C3: Cross-modal reasoning
A2	w/o Joint Reasoning (Late fusion only)	0.327	0.274
A3	w/o Emergent Reasoning (Step 4)	0.358	0.298

Table 6. Ablation results on Foursquare-NYC (mean of 3 independent seeds). Variants are grouped by the contribution claim each challenges. A11 uses independent BM25-based user weight estimation (no LLM) to validate absence of shared-scorer bias (see Section 4 ).

Scenario	Context / Inputs	Agent Execution & Emergent Output
Trace A: Contextual Sensitivity (Local Café)	$c_{1}$ : Weekday, 14:00, Solo-work	Generated Queries: target Wi-Fi, power outlets, quietness. Output: High confidence (0.88) for solo-work suitability; low confidence (0.04) for group seating.
	$c_{2}$ : Friday, 19:30, Birthday Celebration	Generated Queries: target social atmosphere, large seating capacity, alcohol availability. Output: Detects conflicting visual/textual signals, resulting in localized 0.79 calibration. $\mathcal{A}(p,c_{1})\neq\mathcal{A}(p,c_{2})$ .
Trace B: Emergent Reasoning (Cold-Start)	Visual ( $I_{p}$ ): Dense static seating Textual ( $R_{p}$ ): “Always full on Fridays” Metadata ( $M_{p}$ ): High capacity rating	Joint Synthesis (Step 4): Intercepts disjoint data across modalities. Emergent Conclusion: “Advance booking is strictly advisable.” Synthesizes a constraint recommendation without any prior user check-in history.
Trace C: Conflict Resolution (Outdated Visual)	Visual ( $I_{p}$ ): Serene reading-room layout (outdated photo) Textual ( $R_{p}$ ): “Unbearable construction noise—avoid” (recent) Metadata ( $M_{p}$ ): Open, mid-range	Step 4 Action: Temporal precedence analysis assigns higher weight to recency of $R_{p}$ over $I_{p}$ . Conflict Flag: Visual/textual contradiction raised; $a_{i}=\texttt{uncertain}$ , $\widetilde{\mathrm{conf}}_{i}\leftarrow 0.5\times 0.71=0.36$ . Effect: POI is ranked lower despite its visual appeal, preventing hallucinated affordance alignment.

Table 7. Qualitative operational traces of Agent4POI. (Trace A) Context-sensitive query generation for the same local café under contrasting situational intents. (Trace B) Emergent joint cross-modal reasoning resolving a cold-start POI recommendation without ID history. (Trace C) Cross-modal conflict detection preventing hallucinated affordance alignment.

Recommender Systems

Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.

See all 521 papers

Why Users Go There: World Knowledge-Augmented Generative Next POI Recommendation
May 12, 2026Qiuyu Ding, Heng-Da Xu, Wei Zhang +4Sequential Recommendation Knowledge Augmentation for Language Models
Amap, Alibaba Group · Xi’an Jiaotong University
Category-Anchored Large Language Models for Next POI Recommendation
Oct 29, 2026Lexi Bai, Xiaopeng Zeng, Xun Zhou +2Sequential Recommendation Large Language Model-Based Recommendation
Harbin Institute of Technology, Shenzhen, China · City University of Hong Kong, Hong Kong SAR, China.
POI Recommendation with LLM-Augmented Multi-Graph Learning and Contrastive Alignment
Aug 17, 2026Burak Tamer, Wolfram Höpken, Zehui Wang +1Recommender Systems Multimodal Graph Learning
Institute for Digital Transformation - University of Applied Sciences Ravensburg-Weingarten, Germany

May 12, 2026J/K move · Enter open · S save

Why Users Go There: World Knowledge-Augmented Generative Next POI Recommendation

Qiuyu Ding, Heng-Da Xu, Wei Zhang, Dongyi Lv+3

Amap, Alibaba Group · Xi’an Jiaotong University

Sequential Recommendation Knowledge Augmentation for Language Models Recommender Systems Large Language Model-Based Recommendation Generative Recommendation

Open paper AskPDF arXiv

Generative point-of-interest (POI) recommendation models based on large language models (LLMs) have shown promising results by formulating next POI prediction as a sequence generation task. However, the knowledge encoded in these models remains fixed after training, making them unable to perceive evolving real-world conditions that shape user mobility decisions, such as local events and cultural trends. To bridge this gap, we propose AWARE (Agent-based World knowledge Augmented REcommendation), which employs an LLM agent to generate location- and time-aware contextual narratives that capture regional cultural characteristics, seasonal trends, and ongoing events relevant to each user. Rather than introducing generic or noisy information, AWARE further anchors these narratives in each user's behavioral context, grounding external world knowledge in personalized spatial-temporal patterns. Extensive experiments on three real-world datasets demonstrate that AWARE consistently outperforms competitive baselines, achieving up to 12.4% relative improvement.

Abstract

Recommender Systems

Explore similar work

Why Users Go There: World Knowledge-Augmented Generative Next POI Recommendation

Category-Anchored Large Language Models for Next POI Recommendation

POI Recommendation with LLM-Augmented Multi-Graph Learning and Contrastive Alignment