ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search
Authors: Myungchul Kim, Kwanyong Park, Junmo Kim, In So Kweon
Organizations: Korea Advanced Institute of Science and Technology (KAIST), Republic of Korea · University of Seoul, Republic of Korea · Korea Institute of Science and Technology (KIST), Republic of Korea
Existing person search methods assume access to complete visual queries or exhaustive tracking, yet real-world witness accounts are vague, partial, and spread across cameras and time. We introduce ARGOS (Agentic Retrieval with Grounded Observational Search), a benchmark and agent framework that recasts multi-camera person search from one-shot retrieval on a complete query into interactive reasoning from partial clues. To our knowledge, ARGOS is the first interactive benchmark to couple witness dialogue with camera-network topology, requiring an agent to plan, question, and eliminate under information asymmetry. An ARGOS agent receives a vague witness statement and must decide what to ask, when to invoke spatial or temporal tools, and how to interpret ambiguous natural-language responses, all within a limited turn budget. To ground reasoning in physical constraints, the agent accesses a Spatio-Temporal Topology Graph (STTG) encoding camera connectivity and empirically validated transition times. The benchmark comprises 2{,}691 tasks across 14 real-world scenarios in three progressive tracks: semantic perception (Who, 989 tasks), spatial reasoning (Where, 550 tasks), and temporal reasoning (When, 1{,}152 tasks). We propose Turn-Weighted Success (TWS) as the primary metric, jointly measuring correctness and turn efficiency. Experiments with four LLM backbones show the benchmark is far from solved: the best agent achieves TWS of 0.383 (Track2) and 0.590 (Track3). Ablations confirm each component is essential: removing domain-specific tools drops Top-1 accuracy by up to 49.6 percentage points, and removing strategic reasoning halves TWS while barely affecting Top-1.
Figures & tables
Figure 1: Interactive multi-camera person search. Given only a vague witness statement, the ARGOS agent asks targeted questions and issues spatial and temporal tool calls over multiple turns to locate the target across a 16-camera network. This couples interactive dialogue with spatio-temporal reasoning, which prior person-search formulations handle separately.
Track 1 ( Who )
Track 2 ( Where )
Track 3 ( When )
Task count
989
550
1,152
Interaction mode
Single-turn
Multi-turn
Multi-turn
Core capability
Semantic parsing
Spatial reasoning
Temporal reasoning
Avg. dialogue context
5.6 turns
—
—
Avg. oracle turns ( τ∗ )
—
2.02
1.89
Primary metric
Top-1 Acc
TWS
TWS
Table 1: ARGOS benchmark statistics across three tracks.
Figure 2: Left: STTG for the factory (16 cameras). Nodes are grouped into zones (shaded) by OVERLAP connectivity. Edge types: OVERLAP (blue), SOFT_ADJ (orange), TRAVEL (gray); labels show median transition time. Right: 3-D camera layout with sample RGB–Thermal pairs. The university STTG (Sec. C) has 149 edges.
Figure 3: System architecture. Left : the ARGOS Agent consists of four LLM-driven modules that form an observe-think-act loop. Right : the evaluation environment provides a Witness Simulator (Sec. 3.5 ) and a deterministic State Manager. Arrows denote tool calls (downward) and responses (upward). The agent selects actions; the environment executes them.
Accessible
Not accessible
Gallery DB (24 attributes, read-only)
Candidate–camera mapping
Current candidate ID list
Ground-truth disambiguation path
Zone structure (sub-areas, cameras)
Which attributes the witness can answer
STTG statistics (min, median, max, std)
Per-candidate feasibility details
Available action types
Witness’s internal observable set
Table 2: Information boundary of the ARGOS Agent. The agent cannot observe which attributes the witness will answer, forcing strategic decisions under uncertainty.
ID
Name
Input
Output
Tracks
T1
query_gallery
candidate IDs
attribute records
1,2,3
T2
analyze_dist
candidate IDs
counts, elim. power
1,2,3
T3
get_zone_info
—
sub-areas, camera list
2
T4
ask_witness
question type
NL response (string)
2,3
T5
check_temporal
candidate IDs
filtered partition
3
T6
filter_attr
attr, value, IDs
filtered IDs
1,2,3
Table 3: Tool registry. The agent invokes eight tools via structured function calls, each executed by the environment (State Manager or Witness Simulator). Tool names are tinted by track role ( Who/attribute , Where/spatial , When/temporal ; shared tools stay neutral). Parsing the witness reply at filter_attr is the primary bottleneck: free-form answers must map to an exact canonical value, and errors propagate into filtering.
Agent
SR@1
SR@5
Parse Acc.
Oracle
100.0
100.0
100.0
LLM ToolCall
81.1
82.3
90.8
LLM Direct
73.3
73.3
93.0
Rule-based
32.2
48.0
66.0
Table 4: Track 1 results (GPT-4o). LLM Direct outputs a single ID (SR@1 = SR@5).
Track 2 (Spatial)
Track 3 (Temporal)
Backbone
TWS
Top-1
SR t @5
AvgT s
TWS
Top-1
SR t @5
AvgT s
Oracle
1.000
100.0
100.0
2.05
1.000
100.0
100.0
1.88
GPT-5.2
0.338
73.1
33.6
7.04
0.590
88.2
65.8
3.91
GPT-4o
0.323
74.5
31.6
7.49
0.567
80.6
60.8
3.91
GPT-5-mini
0.319
74.9
32.2
7.48
0.556
88.0
60.5
4.40
Claude Sonnet 4
0.383
76.0
38.4
6.71
0.548
83.6
59.5
4.25
Table 5: Main results on Tracks 2 and 3 across four LLM backbones. Best TWS per track is bolded . Blue shading highlights the best value in each column.
Track 3 variant
Top-1
TWS
AvgTurns
Temporal-first
80.6
0.567
3.91
Autonomous
83.9
0.572
4.05
w/o Temporal Tool
31.0
0.054
13.44
Table 6: Autonomous vs. temporal-first invocation (GPT-4o). Letting the agent decide when to call check_temporal matches or slightly improves fixed early invocation, while removing the tool collapses Track 3.
Figure 4: Track 3 temporal-tool demonstration (Factory, Hard). One check_temporal call eliminates 16 of 19 candidates via STTG feasibility; the agent then identifies the target in 3 turns, while the ablated agent (no temporal tool) exhausts its 20-turn budget and fails.
Track 2
Track 3
Agent
Removed
TWS
Top-1
AvgT s
TWS
Top-1
AvgT s
Oracle
(upper bound)
1.000
100.0
2.05
1.000
100.0
1.88
ARGOS Agent
(none)
0.323
74.5
7.49
0.567
80.6
3.91
w/o Strategy
LLM reasoning
0.136
47.6
10.78
0.373
76.9
7.73
w/o Spatial Tool
Spatial queries
0.063
40.7
14.20
—
w/o Temporal Tool
Temporal queries
—
0.054
31.0
13.44
Table 7: Ablation results (GPT-4o). Each row removes one component. Single-pass baselines have no meaningful TWS or AvgT s . Red shading marks the steepest drop per track.
Figure 5: Cumulative success rate (SR t @ T ) vs. turn budget for Track 2 (left) and Track 3 (right), GPT-4o. The ARGOS Agent (blue) reaches high accuracy faster than w/o Strategy (orange). w/o Spatial/Temporal Tool (red) stays near zero through early turns.