StreamEmbed: Real-Time Indexing from Pixels for Screenshot Retrieval
Organizations: University of California, Berkeley · 2Seoul National University · 3KAIST.
Abstract
Screen history can serve as searchable visual memory for people and agents, but unlike an offline corpus, it must be encoded and indexed as each screenshot arrives. OCR-based indexing is fast but loses visual information; visual encoding takes over two seconds. Yet screens change incrementally: most pixels repeat within and across screenshots. This raises a question: do visual encoders need to recompute their representations from scratch for every new screenshot? We present StreamEmbed, a system for incremental visual embedding that exploits this redundancy. Guided solely by pixel differences, it combines spatial token pruning, which skips blocks that repeat a neighbor, with temporal KV reuse, which reuses cached keys and values for blocks unchanged across screenshots. A small learned adapter compensates for pruning and reuse to preserve fidelity to full-screenshot embeddings. On ScreenMemEval, our benchmark of 10,426 questions over 184 hours of screen histories from two humans and a computer-use agent, StreamEmbed sets a new accuracy-latency Pareto frontier on an M2 Pro GPU: it encodes up to 3.7x faster than full-screenshot encoding without sacrificing accuracy and improves accuracy over OCR-based indexing by up to 6.3 Recall@10 points at matched latency. A memory-efficient vector index stores a simulated year of screenshots in 0.13 GB of RAM. Together, these results make real-time indexing of screen history in its native visual form practical, laying a foundation for fully local visual assistants.
Figures & tables
| Latency (s) | |||||
|---|---|---|---|---|---|
| Index | R@10 | OCR | Enc. | Insert | Total |
| Text | 64.0 | 0.34 | 0.31 | 0.001 | 0.65 |
| Screenshot | 74.1 | – | 2.33 | 0.001 | 2.33 |
| SME-H1 | SME-H2 | SME-Agent | |
| Recorded from | human | human | agent |
| Days | 117 | 57 | 43 |
| Recordings | 2,105 | 946 | 787 |
| Screenshots | 63.5k | 12.8k | 55.9k |
| Questions | 5,111 | 3,507 | 1,808 |
| single / multi | 4,265 / 846 | 3,096 / 411 | 1,654 / 154 |
| Type | Example question and answer |
|---|---|
| Single-recording | |
| where | Where did I look up highly rated Vienna hotels with parking and breakfast? Booking.com |
| when | When did I check the airline’s size rules for flying with a bike? Jul 29, 4:44 pm |
| what | What did I do on Rent.com on the afternoon of August 7? Checked utility costs |
| how | How did I create the battery report? Ran powercfg |
| value | What return date did I set for the JFK round-trip flight? June 20th |
| R@10 | RAM (GB) | Search (ms) | Insert (ms) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Index | B/entry | SME-H1 | SME-Agent | 1 yr | 2 yr | 5 yr | 1 yr | 2 yr | 5 yr | 1 yr | 2 yr | 5 yr |
| Flat | 4,096 | 82.1 | 57.6 | 8.6 | 17.2 | – | 440 | 881 | – | 0.002 | 0.002 | – |
| IVF | 4,104 | 80.9 1.2 | 54.8 2.8 | 8.6 | 17.3 | – | 92 | 184 | – | 0.09 | 0.09 | – |
| HNSW | 4,368 | 81.7 0.4 | 56.2 1.4 | 9.2 | 18.4 | – | 0.96 | 1.3 | – | 0.55 | 0.67 | – |
| PQ (128 B) | 128 | 82.0 0.1 | 57.5 0.1 | 0.27 | 0.54 | 1.35 | 65 | 129 | 359 | 0.14 | 0.14 | 0.14 |
| 1-bit (all 2048 dims) | 256 | 82.1 | 57.6 | 0.54 | 1.08 | 2.69 | 24.8 | 44.1 | 102 | 0.007 | 0.007 | 0.007 |
| SME-H1 | SME-H2 | SME-Agent | ||||
|---|---|---|---|---|---|---|
| R@10 | Latency | R@10 | Latency | R@10 | Latency | |
| Qwen3-VL-8B | 84.0 | 5.15 / 14.82 | 85.0 | 5.19 / 14.88 | 61.9 | 5.16 / 14.68 |
| StreamEmbed | 83.0 | 1.59 / 6.31 | 83.0 | 2.08 / 7.78 | 60.7 | 1.49 / 5.93 |
| Qwen3-VL-2B | 83.4 | 2.33 / 5.42 | 84.5 | 2.34 / 5.47 | 58.2 | 2.34 / 5.81 |
| StreamEmbed | 82.1 | 0.66 / 1.97 | 83.3 | 0.84 / 2.61 | 57.6 | 0.63 / 1.87 |
| SigLIP2-So (1.1B) | 77.5 | 0.34 / 0.81 | 78.7 | 0.34 / 0.80 | 49.6 | 0.34 / 0.81 |
| SME-H1 | SME-H2 | SME-Agent | ||||
|---|---|---|---|---|---|---|
| R@10 | Latency | R@10 | Latency | R@10 | Latency | |
| Screenshot (FT) | 83.4 | 2.33 / 5.42 | 84.5 | 2.34 / 5.47 | 58.2 | 2.34 / 5.81 |
| + Linear patch embed. | 83.4 | 1.65 / 5.42 | 84.5 | 1.64 / 5.47 | 58.2 | 1.66 / 5.81 |
| + Spatial pruning | 82.8 | 1.00 / 2.96 | 83.8 | 1.17 / 3.64 | 57.1 | 1.08 / 3.48 |
| w/o grid keep | 82.6 | 0.84 / 2.49 | 83.5 | 1.03 / 3.07 | 55.2 | 0.91 / 2.60 |
| + Temporal reuse | 81.8 | 0.66 / 1.97 | 82.9 | 0.84 / 2.61 | 56.2 | 0.63 / 1.87 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| ScreenMemEval (Ours) | ||||
| Screenshot- VQA | SME-H1 | SME-H2 | SME-Agent | |
| History | ||||
| Days | – | 117 | 57 | 43 |
| Span (days) | 22 | 172 | 78 | 43 |
| Screenshots | 9.8k | 63.5k | 12.8k | 55.9k |
| per recording | – | 30 | 13 | 71 |
| SME-H1 | SME-H2 | SME-Agent | ||||||||||
| All | Single | Multi | nDCG | All | Single | Multi | nDCG | All | Single | Multi | nDCG | |
| Text index (Apple Vision OCR) | ||||||||||||
| BM25 | 58.6 | 64.3 | 29.9 | 51.0 | 56.7 | 58.6 | 42.3 | 47.1 | 35.3 | 36.0 | 27.3 | 26.2 |
| Qwen3-Emb-0.6B | 69.2 | 75.0 | 39.9 | 61.2 | 69.3 | 72.1 | 48.8 | 60.1 | 40.1 | 41.1 | 29.6 | 28.3 |
| Qwen3-Emb-4B | 73.8 | 80.0 | 42.3 | 66.2 | 71.5 | 75.0 | 45.3 | 63.0 | 49.0 | 50.4 | 34.1 | 36.0 |
| Qwen3-Emb-8B | 75.0 | 81.1 | 44.5 | 68.2 | 72.8 | 76.0 | 49.1 | 64.4 | 46.0 | 47.3 | 31.1 | 33.9 |
| Apple Vision OCR | RapidOCR | |||||
|---|---|---|---|---|---|---|
| Text encoder | SME-H1 | SME-H2 | SME-Agent | SME-H1 | SME-H2 | SME-Agent |
| Qwen3-Emb-0.6B | 69.2 | 69.3 | 40.1 | 70.0 | 69.7 | 39.8 |
| Qwen3-Emb-4B | 73.8 | 71.5 | 49.0 | 74.3 | 72.3 | 48.8 |
| Qwen3-Emb-8B | 75.0 | 72.8 | 46.0 | 74.6 | 71.2 | 46.3 |
| Qwen3-VL-2B | 73.5 | 71.2 | 47.4 | 73.5 | 71.7 | 46.8 |
| Qwen3-VL-2B (FT) | 77.4 | 77.4 | 51.3 | 78.0 | 77.3 | 51.7 |
| Latency (s) | Comp. | ||||
| Total | ViT | LM | ViT | LM | |
| Latency sample of the paper (994 screenshots) | |||||
| Screenshot, Conv3d | 2.33 | 1.66 | 0.66 | 3.53 | 2.64 |
| + Linear patch embed. | 1.65 | 0.96 | 0.68 | 3.53 | 2.64 |
| + Spatial pruning | 1.00 | 0.52 | 0.45 | 1.87 | 1.60 |
| + Temporal reuse | 0.66 | 0.34 | 0.29 | 1.01 | 0.85 |
| SME-H1 | SME-H2 | SME-Agent | ||||
|---|---|---|---|---|---|---|
| Tile | R@10 | TF | R@10 | TF | R@10 | TF |
| None | 82.4 | 1.68 | 82.9 | 2.37 | 55.5 | 1.59 |
| 82.0 | 2.13 | 83.1 | 2.88 | 56.9 | 1.92 | |
| ( StreamEmbed ) | 82.1 | 1.95 | 83.3 | 2.67 | 57.6 | 1.79 |
| – | – | – | – | 56.2 | 1.74 | |
| SME-H2 | SME-Agent | |||
| R@10 | TF | R@10 | TF | |
| ( StreamEmbed ) | 83.6 | 2.67 | 57.4 | 1.79 |
| 83.0 | 2.34 | 56.4 | 1.59 | |
| 83.1 | 1.99 | 55.8 | 1.31 | |
| 82.8 | 1.82 | 55.8 | 1.04 | |
| 82.6 | 1.55 | 55.4 | 0.88 | |
| Source | Pairs | Spatial | Temporal | Combined |
|---|---|---|---|---|
| Screen | ||||
| SME-H1 | 64,855 | 46.9 | 25.5 | 72.4 |
| SME-H2 | 11,845 | 41.4 | 21.9 | 63.3 |
| SME-Agent | 6,098 | 39.4 | 36.2 | 75.6 |
| WebLINX | 2,975 | 52.4 | 24.5 | 76.9 |
| GUI-World | 194 | 58.2 | 8.4 | 66.6 |