EgoJev: Finding Evidence in Egocentric Video by Reading VLM Decisions
Organizations: Harvard University
Abstract
Questions about long egocentric video often depend on one brief moment, such as the second a hand puts an object down. A vision-language model (VLM) that sees a few evenly spaced frames usually misses it. Training-free methods rank frames with a separate image-text model or ask the VLM to write a time. In a controlled test, the VLM itself locates the moment better when it chooses among labeled frames, and we read this choice from its output distribution. With the same candidates, this choice lands on the evidence more often than image-text scores or a written time span, and it compares all candidates in one decision. Based on this finding, we propose EgoJev, a training-free frame selector for wearable assistants. EgoJev reads where the evidence is from labeled thumbnails and looks again at the windows it points to. It places its few answer frames at the quantiles of the resulting distribution. One frozen VLM selects, answers and returns the evidence time, with no second model. A wearable device can encode its coarse thumbnails while it records. With Qwen3.5-4B and 8 answer frames, EgoJev raises QAEgo4D accuracy from 51.2% with uniform sampling to 62.0%, and it outperforms recent training-free frame selectors on QAEgo4D and MyEgo. A single-level version runs on a Mac and an iPhone.
Figures & tables
| Method | # frames | Extra model | Time (s) | QAEgo4D | MyEgo | EgoSchema | Avg. |
|---|---|---|---|---|---|---|---|
| Uniform | – | none | 1.7 | 51.2 | 36.1 | 65.2 | 50.8 |
| AKS [ 25 ] | 1 fps | BLIP | 90.7 | 56.4 | 39.4 | 60.4 | 52.1 |
| BOLT [ 17 ] | 1 fps | CLIP | 52.9 | 54.2 | 41.3 | 63.6 | 53.0 |
| A.I.R. [ 41 ] | 1 fps | CLIP | 82.9 | 60.8 | 41.9 | 63.4 | 55.4 |
| Write IDs [ 2 ] | 32 | none | 7.5 | 55.8 | 39.8 | 64.8 | 53.5 |
| FOCUS [ 40 ] | 230 | BLIP | 49.0 | 56.2 | 40.3 | 62.2 | 52.9 |
| Method | QAEgo4D | MyEgo | EgoSchema |
|---|---|---|---|
| AKS, same 32 thumbnails | 54.8 | 40.0 | 63.2 |
| BOLT, same 32 thumbnails | 53.0 | 39.3 | 62.6 |
| Write IDs, same 32 thumbnails | 55.8 | 39.8 | 64.8 |
| EgoJev single level, same 32 thumbnails | 57.2 | 42.2 | 65.2 |
| EgoJev without options | 59.0 | 42.5 | 65.8 |
| EgoJev | 62.0 | 42.6 | 65.2 |
| Qwen3-VL-8B [ 19 ] | MiniCPM-V-4.5 [ 35 ] | GLM-4.1V-9B [ 10 ] | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | QA | My | ES | Avg. | QA | My | ES | Avg. | QA | My | ES | Avg. |
| Uniform | 55.4 | 36.1 | 68.8 | 53.4 | 53.6 | 34.9 | 53.8 | 47.4 | 53.0 | 33.1 | 61.8 | 49.3 |
| AKS (1 fps) | 60.0 | 40.7 | 66.0 | 55.6 | 59.6 | 39.8 | 49.2 | 49.5 | 57.4 | 39.1 | 59.0 | 51.8 |
| BOLT ( ) | 54.0 | 40.4 | 68.8 | 54.4 | 57.4 | 39.3 | 54.0 | 50.2 | 53.2 | 37.7 | 63.0 | 51.3 |
| Write IDs | 57.8 | 42.8 | 68.8 | 56.5 | 56.8 | 38.6 | 51.6 | 49.0 | 55.8 | 36.9 | 61.6 | 51.4 |
| FOCUS | 58.2 | 40.8 | 66.0 | 55.0 | 61.2 | 38.7 | 51.4 | 50.4 | 56.4 | 39.0 | 61.6 | 52.3 |
| Scorer over 16 candidates | QAEgo4D | MyEgo | Passes | Time (s) |
|---|---|---|---|---|
| VLM, choose one (EgoJev) | 28.0 | 18.5 | 1 | 0.44 |
| VLM, rate each (“Yes”) | 18.2 | 13.5 | 16 | 1.41 |
| VLM, write a span | 15.8 | 12.6 | 1 + decode | 1.68 |
| BLIP image-text matching (AKS) | 19.0 | 12.9 | 16 | 1.32 |
| CLIP similarity (BOLT) | 15.6 | 12.0 | 16 | 0.13 |
| Random segment | 8.9 | 6.2 | – | – |
| QAEgo4D | MyEgo | EgoSchema | Mean | Time (s) | |
| 0 (1 level) | 57.2 | 42.2 | 65.2 | 54.9 | 8.5 |
| 1 | 59.0 | 41.9 | 64.2 | 55.0 | 14.5 |
| 2 | 59.2 | 43.0 | 65.0 | 55.7 | 20.8 |
| 3 | 62.0 | 42.6 | 62.8 | 55.8 | 26.7 |
| 4 | 59.2 | 43.4 | 63.4 | 55.3 | 31.2 |
| +32 at quantiles of | 59.2 | 41.8 | 64.4 | 55.1 | 13.6 |
| Layout | QAEgo4D | MyEgo | EgoSchema | Mean |
|---|---|---|---|---|
| Focus | 58.6 | 38.7 | 61.8 | 53.0 |
| Spread | 56.2 | 42.4 | 63.2 | 53.9 |
| Strata | 56.6 | 40.4 | 65.2 | 54.1 |
| Quantiles (EgoJev) | 57.2 | 42.2 | 65.2 | 54.9 |
| QAEgo4D | MyEgo | |||
| Method | 8 s | inside | 8 s | 30 s |
| AKS (1 fps, BLIP) | 27.8 | 18.8 | 21.3 | 46.7 |
| BOLT (1 fps, CLIP) | 22.8 | 13.8 | 20.5 | 43.8 |
| VLM, rate each | 14.2 | 8.4 | 17.6 | 39.3 |
| VLM, write a span | 22.8 | 11.0 | 17.3 | 41.8 |
| Random time | 5.9 | 2.7 | 9.9 | 30.9 |
| Method | Extra model | Rec. (s) | Ans. (s) | Total (s) | QA |
|---|---|---|---|---|---|
| AKS, | BLIP 446M | 53.8 | 13.6 | 67.4 | 56.4 |
| BOLT, | CLIP 428M | 28.3 | 10.4 | 38.7 | 54.2 |
| EgoJev-S, | none | 8.2 | 10.7 | 18.8 | 57.2 |
| EgoJev-S, | none | 8.2 | 5.9 | 14.0 | 59.6 |
| EgoJev-S, , iPhone ‡ | none | 18.3 | 8.0 | 26.2 | 58.0 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Cap | Uniform, | AKS, | BOLT, | EgoJev, |
|---|---|---|---|---|
| 88 | 48.4 | 55.8 | 51.2 | 56.2 |
| 192 | 51.4 | 55.8 | 54.6 | 58.0 |
| 384 | 51.2 | 56.4 | 54.2 | 59.6 |
| Method | QAEgo4D | MyEgo | EgoSchema |
|---|---|---|---|
| AKS, question | 56.4 | 39.4 | 60.4 |
| AKS, question + options | 56.0 | – | – |
| BOLT, question | 54.2 | 41.3 | 63.6 |
| BOLT, question + options | 52.6 | – | – |
| EgoJev, question | 59.0 | 42.5 | 65.8 |
| 0.25 | 0.5 | 1 | 2 | 4 | |
|---|---|---|---|---|---|
| QAEgo4D | 57.2 | 57.8 | 57.2 | 55.8 | 54.4 |
| MyEgo | 42.3 | 41.7 | 42.2 | 40.6 | 39.9 |
| Near | QAEgo4D accuracy by evidence length | ||||
|---|---|---|---|---|---|
| Method | QAEgo4D | MyEgo | 3 s (131) | 3–10 s (219) | 10 s (150) |
| Uniform | 41.6 | 49.6 | 53.4 | 47.5 | 54.7 |
| AKS | 53.8 | 55.8 | 55.7 | 57.5 | 55.3 |
| BOLT | 56.2 | 58.0 | 54.2 | 54.8 | 53.3 |
| A.I.R. | 59.2 | 55.4 | 58.0 | 62.6 | 60.7 |
| Write IDs | 55.2 | 51.7 | 56.5 | 55.3 | 56.0 |
| Accuracy | Near | |||
|---|---|---|---|---|
| Method | Original | Long | Original | Long |
| Uniform | 51.0 | 47.0 | 43.0 | 12.0 |
| EgoJev, single level | 52.0 | 44.0 | 59.0 | 33.0 |
| EgoJev | 56.0 | 51.0 | 66.0 | 44.0 |
| Accuracy | Near | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Gap (min) | Uni. | AKS | BOLT | EgoJev | Uni. | AKS | BOLT | EgoJev | |
| 0.25–0.37 | 254 | 35.0 | 36.6 | 39.4 | 43.3 | 45.3 | 64.2 | 59.4 | 73.6 |
| 0.37–0.67 | 247 | 36.8 | 40.5 | 41.3 | 41.3 | 49.4 | 68.0 | 65.2 | 73.3 |
| 0.67–1.60 | 246 | 34.6 | 41.1 | 41.9 | 42.7 | 60.6 | 51.2 | 65.9 | 67.5 |
| 1.60–10.7 | 249 | 38.2 | 39.4 | 42.6 | 43.0 | 43.4 | 39.8 | 41.8 | 52.2 |
| Variant | QAEgo4D |
|---|---|
| Labeled thumbnails (EgoJev) | 57.2 |
| Full-resolution thumbnails | 56.6 |
| Thumbnails as one video | 53.2 |
| Mac, | Mac, | iPhone, | |
|---|---|---|---|
| Answer-frame decode and encode | 1.3 | 2.6 | 1.8 |
| Answer pass | 3.7 | 7.3 | 4.7 |
| Rest | 0.9 | 0.7 | 1.5 |
| Question to answer | 5.9 | 10.6 | 8.0 |
| Peak memory (GB) | 5.6 | 5.6 | 5.6 |
Egocentric VQA
Egocentric video question answering uses first-person camera footage to answer grounded questions about activities, objects, space, and time.
See all 50 papers