Grounding in the Dark: Text-Only Training for Zero-Shot Video Temporal Localization
Organizations: Amazon · University of Oxford
Abstract
Video-LLMs are typically trained on paired video-text data for temporal tasks, yet collecting precise temporal annotations and training on large-scale video datasets are both costly. We investigate whether a text-only LLM, with no vision-language pretraining, can be trained on textual video descriptions alone to perform temporal grounding and receive video only at inference time. This setting removes paired video data from training altogether and is a step toward training multimodal models without paired annotations. We generate a 134K-example corpus of textual sequences with up to 100 time steps, adapt Qwen3-8B with LoRA on CLIP or LLM2CLIP text embeddings, and evaluate on Charades-STA and QV-Highlights without training on videos from either benchmark. LLM2CLIP, whose text encoder supports longer inputs, represents detailed descriptions better than CLIP (61.54 versus 53.86 mtIoU on Temporal-TextVid) and transfers them better to video. Still, replacing text with frame embeddings at inference yields only 4.99 mtIoU on Charades-STA and 11.12 on QV-Highlights, revealing a large modality gap. To bridge it, we propose Nearest Entity Replacement (NER), a training-free mapping that represents each video frame by the normalized mean of its nearest text-entity embeddings, so that the model receives the kind of input it was trained on. NER reaches 14.45 mtIoU on Charades-STA and 20.72 on QV-Highlights, outperforming the strongest of four families of modality-gap baselines by 5.00 and 1.71 points. NER thus enables partial transfer from text to video, while more reliable grounding will likely require encoders with better-aligned text and visual representations.
Figures & tables
| Dataset | Training | Method | Encoder | mtIoU | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|---|---|---|
| Charades | Video trained | VideoChat-7B [ 21 ] | – | 6.50 | – | 9.00 | 3.30 |
| VideoChatGPT-7B [ 30 ] | – | 13.70 | – | 20.00 | 7.70 | ||
| VTimeLLM-7B [ 9 ] | – | 31.20 | – | 51.00 | 27.50 | ||
| Grounded-VideoLLM (Phi3.5, 4B) [ 39 ] | – | 36.80 | – | 54.20 | 36.40 | ||
| Charades | Text + image memory | Mixed, full | CLIP | 8.78 | 22.78 | 9.33 | 4.39 |
| Text only | Gaussian noise injection [ 32 ] | CLIP | 5.46 | 17.19 | 3.69 | 1.01 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | mAP | HIT@1 |
|---|---|---|
| VideoChat-7B [ 21 ] | 13.1 | 18.1 |
| TimeChat-7B [ 35 ] | 14.5 | 23.9 |
| VTG-LLM-7B [ 7 ] | 16.5 | 33.5 |
| NumPro-FT [ 41 ] | 25.0 | 37.2 |
| Dataset | Variant | mtIoU | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|---|
| Charades | Partial | 9.45 | 21.77 | 11.86 | 5.56 |
| Partial + neighbors | 7.07 | 17.19 | 8.20 | 3.65 | |
| Full | 8.78 | 22.78 | 9.33 | 4.39 | |
| QV-Highlights | Partial | 12.41 | 26.47 | 15.66 | 9.55 |
| Partial + neighbors | 9.35 | 22.25 | 11.48 | 6.26 | |
| Full | 12.74 | 29.62 | 15.81 | 9.29 |
| Dataset | Visual-mean reference | mtIoU | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|---|
| Charades | Current video | 5.78 | 18.74 | 3.34 | 1.17 |
| Eval split (133.4K) | 5.09 | 14.74 | 3.50 | 1.28 | |
| Other video set (145.5K) | 4.74 | 14.15 | 2.95 | 0.89 | |
| LAION (5.99M) | 5.10 | 15.36 | 3.46 | 1.05 | |
| QV-Highlights | Current video | 11.55 | 28.40 | 15.29 | 7.77 |
| Eval split (145.5K) | 10.76 | 26.14 | 14.07 | 7.18 |
| View | Embedding | Dataset | mtIoU | [email protected] | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|---|---|---|
| Detailed | CLIP | TTV | 53.86 | 69.78 | 65.72 | 51.85 | 43.93 |
| Charades | 2.64 | 6.53 | 3.27 | 0.93 | 0.62 | ||
| QV-Highlights | 3.29 | 8.66 | 3.30 | 1.59 | 0.78 | ||
| Detailed | LLM2CLIP | TTV | 61.54 | 76.18 | 72.68 | 60.18 | 52.31 |
| Charades | 4.99 | 12.99 | 5.75 | 2.29 | 1.01 | ||
| QV-Highlights | 11.12 | 24.51 | 14.07 | 8.03 | 4.96 |
| Dataset | Encoder | View | Raw | CP | NER |
|---|---|---|---|---|---|
| Charades | CLIP | Simplified | 7.02 | 5.31 | 14.33 |
| CLIP | Detailed | 2.64 | 3.42 | 7.67 | |
| LLM2CLIP | Simplified | 5.61 | 9.37 | 15.57 | |
| LLM2CLIP | Detailed | 4.99 | 7.95 | 14.45 | |
| QV-Highlights | CLIP | Simplified | 9.25 | 8.85 | 21.85 |
| CLIP | Detailed | 3.29 | 6.23 | 15.28 |
| Dataset | Encoder | View | Reported | Confirmation | Conf. 95% CI | |
|---|---|---|---|---|---|---|
| Charades | CLIP | Simplified | 3,000 4,000 | 14.36 14.33 | 12.67 13.42 | [ 0.24, 1.73] |
| CLIP | Detailed | 3,000 5,000 | 7.64 7.67 | 6.63 7.86 | [0.54, 2.02] | |
| LLM2CLIP | Simplified | 3,000 500 | 15.48 15.57 | 16.46 16.39 | [ 2.25, 2.06] | |
| LLM2CLIP | Detailed | 3,000 1,000 | 13.91 14.45 | 14.07 15.37 | [ 0.31, 2.97] | |
| QV-Highlights | CLIP | Simplified | 2,000 1,000 | 21.20 21.85 | 20.93 22.02 | [ 0.35, 2.56] |
| CLIP | Detailed | 2,000 5,000 | 15.52 15.28 | 16.00 14.55 | [ 3.12, 0.18] |
Temporal Video Grounding
Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.
See all 178 papers