Agent4POI: Agentic Context-Conditioned Affordance Reasoning for Multimodal Point-of-Interest Recommendation
Authors: Jinze Wang, Yangchen Zeng, Tiehua Zhang, Lu Zhang, Yuze Liu, Yongchao Liu, Xingjun Ma, Zhu Sun
Organizations: Tongji University · Swinburne University of Technology · Southeast University · Chengdu University of Information Technology · Ant Group · Fudan University · Singapore University of Technology and Design
We introduce Agent4POI, the first POI recommendation framework that generates context-conditioned multimodal representations at recommendation time, rather than relying on static POI embeddings pre-computed independently of context. Existing multimodal systems encode each POI once as a static embedding, a design that precludes reasoning about why the same cafe affords solo work on Monday but group celebration on Friday evening. We formally prove that no pre-computed encoder can satisfy context-sensitive ranking under standard bilinear scoring, motivating inference-time item-side representation. Agent4POI inverts this computation: given a situational context, a four-phase LLM agent generates dynamic, context-specific affordance queries (Phase 1) and executes a five-step cross-modal chain-of-thought over image, review, and metadata evidence (Phase 2). The resulting uncertainty-aware affordance representation is grounded in Gibsonian affordance theory. These cross-modal verdicts form a structured, uncertainty-adjusted affordance representation (Phase 3), which is aligned with user preferences via a semantic caching system for low-latency ranking (Phase 4). On three POI benchmarks and three evaluation configurations (standard, cold-start, context-shift), Agent4POI achieves a 23.2% relative gain over the strongest baseline and degrades by only 7.5% under context-shift versus 16--17% for the strongest baselines. In cold-start scenarios, Agent4POI outperforms the best content-based baseline by up to 2.4x, whereas ID-based methods fail to generalize.
Figures & tables
Figure 1. Agent4POI four-phase inference pipeline. Unlike prior methods that encode each POI into a fixed vector before any context is observed, Agent4POI delays representation until query time. A frozen LLM generates context-specific affordance queries in Phase 1; Phase 2 then executes a five-step cross-modal chain-of-thought over POI images, reviews, and metadata for each query. The resulting uncertainty-aware verdicts are assembled into a dynamic POI representation (Phase 3) that is aligned with user preference weights for final ranking (Phase 4). A system layer provides semantic caching and asynchronous prefetching for production deployment. Agent4POI four-phase pipeline: a frozen LLM converts situational context into affordance queries; a five-step cross-modal chain-of-thought interrogates POI images, reviews, and metadata; uncertainty-adjusted verdicts form a dynamic POI representation; user preference weights are aligned via dot-product scoring to produce ranked recommendations.
Property
Fixed Model ( ep )
VLM+Attn
Agent4POI A(p,c)
Context-Sensitive Output
✗
Partial
✓
Inference-Time Extraction
✗
✗
✓
Cross-Modal Conflict Flag
✗
✗
✓
Explicit Uncertainty-Aware
✗
✗
✓
Zero-Shot Cold-Start Support
✗
✗
✓
Table 1. POI representation property comparison. Agent4POI satisfies five properties unmatched by static POI embeddings.
Table 3. Standard evaluation. Best per column in bold . † : multimodal; ‡ : LLM reasoning; ∗ : stat. significant over strongest baseline ( 3 independent seeds).
Foursquare-NYC
Foursquare-TKY
Yelp-Open
Method
R@10
NDCG@10
R@10
NDCG@10
R@10
NDCG@10
BPR
0.005
0.004
0.005
0.003
0.003
0.002
MMPOI †
0.098
0.073
0.089
0.066
0.061
0.047
IM-POI †
0.134
0.102
0.121
0.092
0.083
0.065
LLM4POI ‡
0.169
0.131
0.153
0.117
0.104
0.084
Agent4POI †‡
0.313
0.249
0.297
0.235
0.196
0.158
Table 4. Cold-start evaluation. Test POIs have zero training check-in history. ID-based methods collapse; Agent4POI uses only raw multimodal content.
Method
NYC
TKY
Yelp
NDCG@10 ↑
NDCG@10 ↑
NDCG@10 ↑
BPR
0.061
0.057
0.041
GETNext
0.151
0.141
0.093
IM-POI
0.211
0.196
0.133
CoT-Rec
0.224
0.208
0.143
Agent4POI
0.309
0.291
0.198
Table 5. Context-shift evaluation. Train: weekday; Test: weekend. Phase 1 of Agent4POI adapts queries at test time. Reported values are means across 3 seeds.
ID
Variant
R@10 ↑
NDCG@10 ↑
Full
Agent4POI (complete)
0.401
0.334
C1: Context-driven queries
A1
w/o Dynamic Queries (Fixed context)
0.312
0.259
C3: Cross-modal reasoning
A2
w/o Joint Reasoning (Late fusion only)
0.327
0.274
A3
w/o Emergent Reasoning (Step 4)
0.358
0.298
Table 6. Ablation results on Foursquare-NYC (mean of 3 independent seeds). Variants are grouped by the contribution claim each challenges. A11 uses independent BM25-based user weight estimation (no LLM) to validate absence of shared-scorer bias (see Section 4 ).
Scenario
Context / Inputs
Agent Execution & Emergent Output
Trace A: Contextual Sensitivity (Local Café)
c1 : Weekday, 14:00, Solo-work
Generated Queries: target Wi-Fi, power outlets, quietness. Output: High confidence (0.88) for solo-work suitability; low confidence (0.04) for group seating.
c2 : Friday, 19:30, Birthday Celebration
Generated Queries: target social atmosphere, large seating capacity, alcohol availability. Output: Detects conflicting visual/textual signals, resulting in localized 0.79 calibration. A(p,c1)=A(p,c2) .
Trace B: Emergent Reasoning (Cold-Start)
Visual ( Ip ): Dense static seating Textual ( Rp ): “Always full on Fridays” Metadata ( Mp ): High capacity rating
Joint Synthesis (Step 4): Intercepts disjoint data across modalities. Emergent Conclusion: “Advance booking is strictly advisable.” Synthesizes a constraint recommendation without any prior user check-in history.
Step 4 Action: Temporal precedence analysis assigns higher weight to recency of Rp over Ip . Conflict Flag: Visual/textual contradiction raised; ai=uncertain , confi←0.5×0.71=0.36 . Effect: POI is ranked lower despite its visual appeal, preventing hallucinated affordance alignment.
Table 7. Qualitative operational traces of Agent4POI. (Trace A) Context-sensitive query generation for the same local café under contrasting situational intents. (Trace B) Emergent joint cross-modal reasoning resolving a cold-start POI recommendation without ID history. (Trace C) Cross-modal conflict detection preventing hallucinated affordance alignment.
Generative point-of-interest (POI) recommendation models based on large language models (LLMs) have shown promising results by formulating next POI prediction as a sequence generation task. However, the knowledge encoded in these models remains fixed after training, making them unable to perceive evolving real-world conditions that shape user mobility decisions, such as local events and cultural trends. To bridge this gap, we propose AWARE (Agent-based World knowledge Augmented REcommendation), which employs an LLM agent to generate location- and time-aware contextual narratives that capture regional cultural characteristics, seasonal trends, and ongoing events relevant to each user. Rather than introducing generic or noisy information, AWARE further anchors these narratives in each user's behavioral context, grounding external world knowledge in personalized spatial-temporal patterns. Extensive experiments on three real-world datasets demonstrate that AWARE consistently outperforms competitive baselines, achieving up to 12.4% relative improvement.
Point-of-interest (POI) recommendation models based on graph neural networks achieve strong performance by propagating collaborative signals over user-item interactions, yet they struggle with the cold-start problem, where items with few or no interactions are not represented. In this paper, we propose LLM-augmented Multi-Graph Contrastive Learning (LLM-MGCL), a multi-graph neural network that uses semantic and spatial information about items to extend the LightGCN backbone with two auxiliary item-item graphs: a semantic graph constructed from sentence embeddings of LLM-generated photo summaries and keywords, and a geographic graph derived from Haversine distances between business locations. Item embeddings are propagated over all three graphs in parallel, fused additively, and aligned across views through a bidirectional InfoNCE contrastive objective that connects behavioral, semantic, and spatial representations of the same items. Experiments on the Yelp Multimodal Recommendation Dataset show that LLM-MGCL outperforms classical collaborative filtering, matrix factorization, and non-contrastive graph neural network baselines. It improves Recall@20 by 48.5% and NDCG@20 by 59.0% over LightGCN, while trailing the strongest contrastive baseline, Self-supervised Graph Learning (SGL), slightly in aggregate. A popularity-stratified evaluation shows that LLM-MGCL achieves the highest recall on cold and warm items, whereas SGL remains better on popular items. An ablation study reveals that the cross-view contrastive alignment (CA) is the primary driver of the aggregate gains. Our results suggest that externally grounded, LLM-derived item knowledge can partially compensate for missing collaborative signal and mitigate the item cold-start problem in POI recommendation.
Burak Tamer, Wolfram Höpken, Zehui Wang +1
Institute for Digital Transformation - University of Applied Sciences Ravensburg-Weingarten, Germany
Next Point-of-Interest (POI) recommendation ranks a user's likely next location based on check-in history. Most recent rankers compress the trajectory into a single user vector and score every candidate through the same representation, ignoring that every candidate carries geographic coordinates and that the relevance of a past visit depends on where the candidate is located. Target attention from click-through-rate prediction conditions the user representation on the scored item, but its operator was developed for web items without geometry. This paper proposes CaST-POI, a candidate-conditioned POI ranker that keeps a standard target-attention reader and adds two bucketised biases to the attention logits: one for the recency of each past visit and another for its distance to the candidate being scored. Different candidates therefore read the same trajectory with different attention weights, grounded in real geographic distance rather than inferred from item embeddings. Comparing against seven sequential recommenders trained with the same split on three datasets NYC, TKY and CA under per-user leave-one-out with full-vocabulary ranking, CaST-POI improves MRR over the strongest baseline by 5.1%, 7.6% and 14.5%. Holm-corrected paired tests confirm the gains on TKY and CA. Ablation study locates the gain in the explicit revisit gate and the candidate-relative spatial bias, whereas the candidate-independent temporal bias has no measurable effect on all datasets. Code is available at https://github.com/YuZhenyuLindy/CaST-POI.
Zhenyu Yu, Chunlei Meng, Yangchen Zeng +3
College of Computer Science and Artificial Intelligence Fudan University Shanghai, China · College of Intelligent Robotics and Advanced Manufacturing Fudan University Shanghai, China · School of Cyber Science and Engineering Southeast University Nanjing, China +2