Interactive Person Retrieval with On-the-Fly Visual Feedback
Organizations: Sichuan University · University of California, Berkeley
Abstract
Traditional text-based person retrieval (TPR) assumes that a witness provides a complete description at once, whereas interactive person retrieval (Inter-PR) refines a partial description through question-answer exchanges. In this setting, the questioner inspects retrieved candidates to decide what evidence to request next, while the answerer sees only the witness-side memory and cannot report candidate-specific visual differences. Existing methods mainly learn offline question orderings, leaving this visual-feedback loop underexplored. To address this limitation, we introduce visual-feedback interactive person retrieval (VFI-PR), a new interactive retrieval paradigm in which retrieved candidates produce visual feedback that becomes evidence for subsequent questioning and re-ranking. To support training and evaluation in this setting, we construct VFI-PEDES, a dataset and evaluation protocol for visual-feedback interaction, and propose VFI-ReID, a framework that learns to use visual feedback during interactive person retrieval. Through the Dynamic Accumulated-evidence Retriever (DAR) and retrieval-aware question optimization, VFI-ReID adapts retrieval and question generation to evolving evidence states, turning visual feedback into complementary evidence for subsequent retrieval. Experiments show that VFI-ReID consistently improves retrieval across VFI-PR interaction stages, demonstrating the utility of visual feedback for interactive person retrieval and outperforming existing Inter-PR baselines.
Figures & tables
| Method | Venue | Round 1 | Round 2 | BRI | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@5 | R@10 | mAP | R@1 | R@5 | R@10 | mAP | |||
| Initial | - | 62.05 | 82.87 | 89.72 | 47.45 | 62.05 | 82.87 | 89.72 | 47.45 | - |
| ChatIR [ 3 ] | NeurIPS’23 | 73.51 | 90.34 | 94.58 | 54.23 | 76.25 | 92.15 | 95.88 | 56.31 | 0.455 |
| PlugIR [ 4 ] | ACL’24 | 73.27 | 90.51 | 94.87 | 54.24 | 76.18 | 91.94 | 95.62 | 56.08 | 0.464 |
| SimRV [ 24 ] | ICCV’23 | 73.92 | 90.33 | 94.96 | 53.64 | 76.46 | 92.28 | 95.97 | 56.59 | 0.446 |
| LLaVA-ReID [ 5 ] | ICML’25 | 72.25 | 89.49 | 94.18 | 53.39 | 73.95 | 90.55 | 94.71 | 54.76 | 0.469 |
| VF | DAR | SFT | RL | R@1 | R@5 | R@10 | mAP |
|---|---|---|---|---|---|---|---|
| ✓ | 46.79 | 70.07 | 79.44 | 33.72 | |||
| ✓ | ✓ | ✓ | 65.25 | 85.20 | 90.71 | 43.84 | |
| ✓ | ✓ | 72.90 | 90.34 | 94.67 | 54.22 | ||
| ✓ | ✓ | ✓ | 73.96 | 90.45 | 94.85 | 54.65 | |
| ✓ | ✓ | ✓ | 75.22 | 91.54 | 95.38 | 54.68 | |
| ✓ | ✓ | ✓ | ✓ | 80.14 | 93.80 | 96.66 | 57.77 |
| RL states | R@1 | R@5 | R@10 | mAP | BRI |
|---|---|---|---|---|---|
| 0K | 73.96 | 90.45 | 94.85 | 54.65 | 0.445 |
| 1K | 79.53 | 93.73 | 96.61 | 57.62 | 0.397 |
| 3K | 80.14 | 93.80 | 96.66 | 57.77 | 0.385 |
| 5K | 80.29 | 93.77 | 96.84 | 58.45 | 0.391 |