Video-LLMs are typically trained on paired video-text data for temporal tasks, yet collecting precise temporal annotations and training on large-scale video datasets are both costly. We investigate whether a text-only LLM, with no vision-language pretraining, can be trained on textual video descriptions alone to perform temporal grounding and receive video only at inference time. This setting removes paired video data from training altogether and is a step toward training multimodal models without paired annotations. We generate a 134K-example corpus of textual sequences with up to 100 time steps, adapt Qwen3-8B with LoRA on CLIP or LLM2CLIP text embeddings, and evaluate on Charades-STA and QV-Highlights without training on videos from either benchmark. LLM2CLIP, whose text encoder supports longer inputs, represents detailed descriptions better than CLIP (61.54 versus 53.86 mtIoU on Temporal-TextVid) and transfers them better to video. Still, replacing text with frame embeddings at inference yields only 4.99 mtIoU on Charades-STA and 11.12 on QV-Highlights, revealing a large modality gap. To bridge it, we propose Nearest Entity Replacement (NER), a training-free mapping that represents each video frame by the normalized mean of its nearest text-entity embeddings, so that the model receives the kind of input it was trained on. NER reaches 14.45 mtIoU on Charades-STA and 20.72 on QV-Highlights, outperforming the strongest of four families of modality-gap baselines by 5.00 and 1.71 points. NER thus enables partial transfer from text to video, while more reliable grounding will likely require encoders with better-aligned text and visual representations.
Figures & tables
Figure 1: Paired video-text training versus our text-only training and video-inference setting.
Figure 2: (a) Generation of frame-level textual sequences and questions. (b) Training from text embeddings and inference from video frames, with an optional modality-gap reduction step.
Figure 3: NER avoids concept bleed from unrelated caption content (red).
Table 4: Transfer results and published context under original protocols. CLIP rows use simplified descriptions and LLM2CLIP rows detailed descriptions. † : values reproduced from VTimeLLM [ 9 ] .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Method
mAP
HIT@1
VideoChat-7B [ 21 ]
13.1
18.1
TimeChat-7B [ 35 ]
14.5
23.9
VTG-LLM-7B [ 7 ]
16.5
33.5
NumPro-FT [ 41 ]
25.0
37.2
Appendix
Table 5: Published QV-Highlights highlight-detection results. The first three values are reproduced from VTG-LLM [ 7 ] and NumPro-FT from Number It [ 41 ] . NumPro-FT uses video fine-tuning, and the other rows are zero-shot. These saliency metrics are not directly comparable to NER.
Table 7: C3 sensitivity to the visual-mean reference with CLIP. The evaluation-split mean uses the full target split and is therefore transductive. The other video set is the other benchmark.
Table 8: Full caption-view and embedding-model results. TTV denotes Temporal-TextVid.
Dataset
Encoder
View
Raw
CP
NER
Charades
CLIP
Simplified
7.02
5.31
14.33
CLIP
Detailed
2.64
3.42
7.67
LLM2CLIP
Simplified
5.61
9.37
15.57
LLM2CLIP
Detailed
4.99
7.95
14.45
QV-Highlights
CLIP
Simplified
9.25
8.85
21.85
CLIP
Detailed
3.29
6.23
15.28
Appendix
Table 9: Raw features, caption projection (CP), and NER for each encoder and caption view (mtIoU).
Dataset
Encoder
View
k
Reported
Confirmation
Conf. 95% CI
Charades
CLIP
Simplified
3,000 → 4,000
14.36 → 14.33
12.67 → 13.42
[ − 0.24, 1.73]
CLIP
Detailed
3,000 → 5,000
7.64 → 7.67
6.63 → 7.86
[0.54, 2.02]
LLM2CLIP
Simplified
3,000 → 500
15.48 → 15.57
16.46 → 16.39
[ − 2.25, 2.06]
LLM2CLIP
Detailed
3,000 → 1,000
13.91 → 14.45
14.07 → 15.37
[ − 0.31, 2.97]
QV-Highlights
CLIP
Simplified
2,000 → 1,000
21.20 → 21.85
20.93 → 22.02
[ − 0.35, 2.56]
CLIP
Detailed
2,000 → 5,000
15.52 → 15.28
16.00 → 14.55
[ − 3.12, 0.18]
Appendix
Table 10: NER neighbor counts selected on development videos (mtIoU). Reported: full evaluation sets. Confirmation: 200 separate videos per benchmark, with a 95% bootstrap interval for the change.
Figure 5: Detailed and simplified caption views for one Temporal-TextVid example.