How Should a Camera-First AI Agent Work?
Organizations: Chance AI
Abstract
Camera-first agents must act on the visual world as it unfolds. We argue that the central obstacle for future camera-first agents is not recognition ability - multimodal models (vision-language and live/streaming alike) generalize well through language and remain the right tool for semantic reasoning - but time. Many visual targets are ephemeral: they appear, are query-relevant, and leave within a short window. Binding memory to the same end-to-end path as reasoning means that by the time a monolithic pipeline completes, the evidence is gone. We formalize this as Ephemeral-Target Interaction (ETI) and a Perceptual Deadline Contract separating a hard memory deadline from a soft reasoning deadline, grounded in the bounded, passive nature of sensory and working memory. We propose a two-deadline architecture: an always-on, bounded, non-semantic fast path; an active remembering mechanism that commits a moment to durable memory before it is overwritten (event-driven in v1); and a deferred deliberation layer (multimodal model + tools). We instantiate the contract on real MOT17 track windows, a consistency check, and calibrate the single-path baseline from measured frontier-model latency: even a short answer takes 1.2-9.3 s, and an agentic identify-and-search longer. Answers can be late; evidence cannot be missed.
Figures & tables
| policy | memory recall | TTR (s, conf.) | recall @ s | |||
|---|---|---|---|---|---|---|
| monolithic ( s) | 0.681 | 1.5 | 0.000 | 0.042 | 1.000 | 1.000 |
| monolithic ( s) | 0.430 | 5.0 | 0.000 | 0.000 | 0.000 | 0.801 |
| monolithic ( s) | 0.249 | 12.0 | 0.000 | 0.000 | 0.000 | 0.464 |
| fixed sampling (2 fps) | 0.957 | 0.30 | 0.755 | 1.000 | 1.000 | 1.000 |
| saliency gate (goal-agnostic) | 0.500 | 0.05 | 0.460 | 0.442 | 0.464 | 0.539 |
| Ours (event-driven) | 1.000 | 0.05 | 1.000 | 1.000 | 1.000 | 1.000 |
| model | TTFB p50 (s) | throughput (tok/s) | est. time-to-answer, 64 tok (s) |
|---|---|---|---|
| google/gemini-3.5-flash-lite | 0.73 | 127.5 | 1.23 |
| openai/gpt-5.6-terra | 1.47 | 82.6 | 2.25 |
| anthropic/claude-sonnet-5 | 1.63 | 70.3 | 2.54 |
| openai/gpt-6-sol | 1.97 | 64.2 | 2.96 |
| anthropic/claude-opus-5.5 | 5.50 | 65.7 | 6.47 |
| qwen/qwen3-vl-8b-instruct | 3.92 | 12.0 | 9.25 |