Can an LVLM Perceive Event Boundaries Like a Human?
Organizations: Qualcomm AI Research
Abstract
Large Vision-Language Models (LVLMs) reason effectively about video content, but whether they perceive its event boundaries in a manner consistent with humans remains unclear. We first investigate this through Generic Event Boundary Detection (GEBD), which identifies event boundaries naturally perceived by humans in continuous video, without relying on predefined event categories. On Kinetics-GEBD dataset, LVLMs miss many boundaries when predicting directly from frames, but effectively refine candidates proposed by an unsupervised discovery method, achieving state-of-the-art unsupervised GEBD performance. We extend this discovery-refinement approach to hour-long egocentric video, where boundary discovery requires a broader semantic scale and whole-video LVLM refinement becomes impractical. We address these challenges through coarse-to-fine temporal discovery and localized LVLM accept/reject judgments. Human evaluation shows that the LVLM's agreement with group consensus falls within the range observed among human annotators, with disagreements concentrated on low-consensus cases. Using this pipeline, we construct EgoBoundary, a benchmark for event boundary detection in hour-scale egocentric video. Representative supervised and unsupervised methods degrade substantially on EgoBoundary, highlighting the challenge of long-horizon event boundary detection.
Figures & tables
| Supervision | Method | Rel.Dis=0.05 | 0.1 | 0.15 | 0.2 | 0.25 | 0.3 | 0.35 | 0.4 | 0.45 | 0.5 | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Supervised | BMN Lin et al. (2019) | 0.186 | 0.204 | 0.213 | 0.220 | 0.226 | 0.230 | 0.233 | 0.237 | 0.239 | 0.241 | 0.223 |
| BMN-StartEnd Lin et al. (2019) | 0.491 | 0.589 | 0.627 | 0.648 | 0.660 | 0.668 | 0.674 | 0.678 | 0.681 | 0.683 | 0.640 | |
| TCN-TAPOS Shou et al. (2021) | 0.464 | 0.560 | 0.602 | 0.628 | 0.645 | 0.659 | 0.669 | 0.676 | 0.682 | 0.687 | 0.627 | |
| TCN Lea et al. (2016) | 0.588 | 0.657 | 0.679 | 0.691 | 0.698 | 0.703 | 0.706 | 0.708 | 0.710 | 0.712 | 0.685 | |
| PC Shou et al. (2021) | 0.625 | 0.758 | 0.804 | 0.829 | 0.844 | 0.853 | 0.859 | 0.864 | 0.867 | 0.870 | 0.817 | |
| Temporal Perceiver Tan et al. (2023) | 0.748 | 0.828 | 0.852 | 0.866 | 0.874 | 0.879 | 0.883 | 0.887 | 0.890 | 0.892 | 0.860 |
| Method | Precision | Recall | F1 |
|---|---|---|---|
| Direct inference | 0.8500 | 0.5540 | 0.6708 |
| Unsupervised discovery | 0.6871 | 0.7505 | 0.7174 |
| Combination | 0.6910 | 0.8077 | 0.7448 |
| Rank | Judge | |
|---|---|---|
| 1 | Human A | 0.534 |
| 2 | Human B | 0.510 |
| 3 | Human C | 0.508 |
| 4 | Human D | 0.477 |
| 5 | Human E | 0.398 |
| 6 | LVLM | 0.394 |
| Margin | Agree. | FR | ||
|---|---|---|---|---|
| 2 | 87 | 0.394 | 70% | 41% |
| 4 | 69 | 0.551 | 78% | 35% |
| 6 | 58 | 0.597 | 81% | 33% |
| 8 | 39 | 0.667 | 85% | 27% |
| 10 | 23 | 0.721 | 87% | 12% |
| Discovery | Initial | Rejected | Accepted |
|---|---|---|---|
| Coarse-to-fine | 6,364 | 17.0% | 83.0% |
| Fine only | 6,364 | 22.3% | 77.7% |
| Method | #Bnd. | F1 @ tIoU | ||
|---|---|---|---|---|
| 0.5 | 0.7 | 0.9 | ||
| Discovery | 265 | 0.7625 | 0.6188 | 0.4391 |
| + LVLM refinement | 228 | 0.8276 | 0.7112 | 0.5259 |
| Uniform ( =discovery) | 265 | 0.2515 | 0.0639 | 0.0040 |
| Uniform ( =refined) | 228 | 0.2457 | 0.0474 | 0.0043 |
| Uniform ( =GT) | 230 | 0.2833 | 0.0687 | 0.0043 |
| Supervision | Method | Feat. | 0.5 | 0.7 | 0.9 |
|---|---|---|---|---|---|
| Supervised | BasicGEBD | G14 | 0.4783 | 0.3052 | 0.1823 |
| S16 | 0.4723 | 0.2490 | 0.1307 | ||
| Unsupervised | UBoCo | G14 | 0.3120 | 0.1544 | 0.0649 |
| S16 | 0.2318 | 0.1181 | 0.0452 | ||
| CoSeg-FFR | G14 | 0.2544 | 0.0983 | 0.0133 | |
| S16 | 0.2720 | 0.1017 | 0.0205 |
| Artifact | License / Terms |
|---|---|
| Kinetics-GEBD | CC BY-NC 4.0 |
| EgoLife | S-Lab License 1.0 |
| Perception Encoder | Apache License 2.0 |
| Gemini 2.5 Pro | Google Gemini API Terms |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Change-Point Detection
Change-point detection identifies when the statistical behavior or underlying regime of a sequence changes.
See all 89 papers