EagleNet: Energy-Aware Fine-Grained Relationship Learning Network for Text-Video Retrieval

Yuhan Chen; Pengwen Dai; Chuan Wang; Dayan Wu; Xiaochun Cao

cs.CVMar 26, 2026

EagleNet: Energy-Aware Fine-Grained Relationship Learning Network for Text-Video Retrieval

Authors: Yuhan Chen, Pengwen Dai, Chuan Wang, Dayan Wu, Xiaochun Cao

Organizations: Sun Yat-sen University, Guangzhou, China · Shenzhen Campus of Sun Yat-sen University, Shenzhen, China · Shenzhen Key Laboratory of Adversarial Artificial Intelligence, Shenzhen, China · Beijing Jiaotong University, Beijing, China ·

Abstract

Text-video retrieval tasks have seen significant improvements due to the recent development of large-scale vision-language pre-trained models. Traditional methods primarily focus on video representations or cross-modal alignment, while recent works shift toward enriching text expressiveness to better match the rich semantics in videos. However, these methods use only interactions between text and frames/video, and ignore rich interactions among the internal frames within a video, so the final expanded text cannot capture frame contextual information, leading to disparities between text and video. In response, we introduce Energy-Aware Fine-Grained Relationship Learning Network (EagleNet) to generate accurate and context-aware enriched text embeddings. Specifically, the proposed Fine-Grained Relationship Learning mechanism (FRL) first constructs a text-frame graph by the generated text candidates and frames, then learns relationships among texts and frames, which are finally used to aggregate text candidates into an enriched text embedding that incorporates frame contextual information. To further improve fine-grained relationship learning in FRL, we design Energy-Aware Matching (EAM) to model the energy of text-frame interactions and thus accurately capture the distribution of real text-video pairs. Moreover, for more effective cross-modal alignment and stable training, we replace the conventional softmax-based contrastive loss with the sigmoid loss. Extensive experiments have demonstrated the superiority of EagleNet across MSRVTT, DiDeMo, MSVD, and VATEX. Codes are available at https://github.com/draym28/EagleNet.

Figures & tables

Figure 1: The performance-efficiency comparison among SOTA models (ViT-B/16) on MSRVTT datasets.

Figure 2: Diagram of EagleNet. (a) Overview of the training process. (b) Fine-Grained Relationship Learning (FRL) first samples multiple text candidates, then constructs a text-frame graph to learn both text-frame and frame-frame relationships, which are then used to aggregate text candidates into a final enriched text embedding aware of video context information. (c) Energy-Aware Matching (EAM) improves the relationships learning in FRL by capturing detailed text-frame interactions and accurately models true text-video pairs distribution, thereby also enhancing the final matching performance from a fine-grained perspective.

Method		Text-to-Video					Video-to-Text					Rsum↑
Method		R@1↑	R@5↑	R@10↑	MdR↓	MnR↓	R@1↑	R@5↑	R@10↑	MdR↓	MnR↓	Rsum↑
Not End-to-End Method
	CE [ 31 ] BMVC19	20.6	50.3	64.0	5.3	-	20.9	48.8	62.4	6.0	-	267.0
	MMT [ 14 ] ECCV20	26.6	57.1	69.6	4.0	24.0	27.0	57.5	69.7	3.7	21.3	307.5
	T2vlad [ 46 ] CVPR21	29.5	59.0	70.1	4.0	-	31.8	60.0	71.1	3.0	-	321.5
	HiT [ 29 ] ICCV21	27.7	59.2	72.0	2.9	-	28.8	60.3	72.3	3.0	-	320.3

Table 1: Performance on MSRVTT, boldface is the best results and underline is the runner-ups. Note that we have resolved the data-leakage problem in the original TMASS codes, and re-implemented XPool † , CLIP-VIP † , TMASS † , TV-ProxyNet † , and Video-ColBERT † using CLIP4Clip codebase to align with other methods. ∗ denotes the reproduced results, all of which are fine-tuned on ViT-B/32 or ViT-B/16.

Method		DiDeMo			MSVD			VATEX			Rsum↑
Method		R@1↑	R@5↑	R@10↑	R@1↑	R@5↑	R@10↑	R@1↑	R@5↑	R@10↑	Rsum↑
	CE [ 31 ]	16.1	41.1	82.7	19.8	49.0	-	-	-	-	-
	HGR [ 5 ]	-	-	-	-	-	-	35.1	73.5	83.5	-
	ClipBERT [ 26 ]	20.4	48.0	60.8	-	-	-	-	-	-	-
	SupportSet [ 36 ]	-	-	-	28.4	60.0	-	45.9	82.4	-	-
	Frozen [ 2 ]	31.0	59.8	72.4	33.7	64.7	-	-	-	-	-

Table 2: Text-to-video results on DiDeMo, MSVD, and VATEX. Note that we have resolved the data-leakage problem in the original TMASS codes, and re-implemented XPool † , CLIP-VIP † , TMASS † , TV-ProxyNet † , and Video-ColBERT † using CLIP4Clip codebase to align with other methods. ∗ denotes the reproduced results, all of which are fine-tuned on ViT-B/32 or ViT-B/16.

Figure 3: Visualization of EagleNet, TMASS, and TV-ProxyNet.

Components			MSRVTT			DiDeMo			Avg
FRL	EAM	$\mathcal{L}_{\text{sig}}$	R@1↑	R@5↑	R@10↑	R@1↑	R@5↑	R@10↑	R@1↑	R@5↑	R@10↑
			48.5	74.1	83.4	42.1	73.1	82.3	45.3	73.6	82.9
✓			48.8	75.6	83.9	47.9	75.2	84.1	48.4	75.4	84.0
	✓		49.0	75.3	84.0	43.4	73.6	82.7	46.2	74.5	83.4
✓	✓		50.5	76.5	84.2	49.2	76.6	85.3	49.9	76.6	84.8
		✓	47.8	75.9	84.4	43.9	75.6	83.6	45.9	75.8	84.0

Table 3: Text-to-video results on different components.

Energy	MSRVTT			DiDeMo
Function	R@1↑	R@5↑	R@10↑	R@1↑	R@5↑	R@10↑
CosSim	50.0	76.9	85.5	50.2	78.3	86.3
Bilinear	51.0	76.2	85.6	51.3	78.3	86.6
MLP	50.3	76.6	85.1	51.5	78.7	86.8

Table 4: Text-to-video on different energy functions.

Pooling	MSRVTT			DiDeMo
Form	R@1↑	R@5↑	R@10↑	R@1↑	R@5↑	R@10↑
w/o EAM	49.9	77.3	84.3	49.1	77.5	85.0
$E_{\theta}({\mathbf{t}},{\mathbf{v}})$	48.8	76.3	84.6	47.8	77.1	85.2
Maxpool	50.4	76.5	85.5	51.3	79.0	86.6
Minpool	50.5	77.2	84.9	50.9	78.5	86.5
Avgpool	51.0	76.2	85.6	51.5	78.7	86.8

Table 5: Text-to-video on different pooling approach for

E_{\theta}

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model		Training	Test
CLIP4Clip		2947.4	49.7
XPool		3062.7	51.0
TS2Net		4367.1	99.7
CLIP-VIP		2497.0	41.1
UCoFiA	(t2v)	21021.4	2969.8
	(v2t)		10000.3

Appendix

Table 6: Average training time per epoch and test time (sec.) consumption of different models (ViT-B/16) on MSRVTT.

Graph	MSRVTT			DiDeMo
Encoder	R@1↑	R@5↑	R@10↑	R@1↑	R@5↑	R@10↑
w/o EAM
GAT	49.4	76.8	85.9	47.5	76.6	85.9
RGAT	49.9	77.3	84.3	49.1	77.5	85.0
w/ EAM
GAT	49.9	76.6	85.2	48.9	78.4	86.6

Appendix

Table 7: Text-to-video on different graph encoders.

$\lambda_{eam}$	MSRVTT			DiDeMo
$\lambda_{eam}$	R@1↑	R@5↑	R@10↑	R@1↑	R@5↑	R@10↑
0.1	50.7	76.6	85.7	50.9	78.9	86.6
0.5	50.1	76.5	86.2	51.2	78.6	86.6
1.0	51.0	76.2	85.6	51.5	78.7	86.8

Appendix

Table 8: Text-to-video on different

\lambda_{eam}

	MSRVTT			DiDeMo
	R@1↑	R@5↑	R@10↑	R@1↑	R@5↑	R@10↑
w/o FRL	48.6	75.3	83.7	43.8	75.6	83.8
w/o f2f	50.6	77.5	84.6	49.9	78.7	86.5
EagleNet	51.0	76.2	85.6	51.5	78.7	86.8

Appendix

Table 9: Text-to-video on frame-to-frame edges (f2f).

Figure 4: Visualization of text-to-video retrieval results by our proposed EagleNet, TMASS [ 43 ] , and TV-ProxyNet [ 51 ] . Green denotes the correct retrieval result, and red is the wrong result.

Figure 5: Visualization of text-to-video retrieval results by different components of EagleNet. Green denotes the correct retrieval result, and red is the wrong result.

Vision-Language Retrieval

Vision-language retrieval finds images, video, or documents that match natural-language queries and supports search across modalities.

See all 97 papers

VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings
Oct 16, 2026Ruizhi Chen, Zining Fan, Yudi Guo +5Multimodal Embedding Vision-Language Retrieval
Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd. · Peking University · Nanjing University +1
DREAM: Extending Vision-Language Models with Dual-Objective Encoding for Cross-Modal Retrieval
Jun 17, 2026Kaleem Ullah, Altaf Hussain, Muhammad Munsif +1Cross-Modal Retrieval Video Representation Learning
Sejong University, Seoul 143-747, South Korea · KAIST InnoCORE PRISM-AI Center, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, 34141, Republic of Korea · Ulsan National Institute of Science and Technology (UNIST), Ulsan, 44919, Republic of Korea
SALI: Shot-Aware Late Interaction for Cross-Shot Relation Matching in Text-to-Video Retrieval using Film-Grammar Knowledge
Sep 24, 2026Toya Oyama, Rainer Lienhart, Shin'ichi SatohLate Interaction Retrieval
The University of Tokyo · National Institute of Informatics · University of Augsburg

Oct 16, 2026J/K move · Enter open · S save

VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings

Ruizhi Chen, Zining Fan, Yudi Guo, Jiang Liu+4

Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd. · Peking University · Nanjing University+1

Multimodal Embedding Vision-Language Retrieval Contrastive Learning

Open paper AskPDF arXiv

Fine-grained alignment between text and video object trajectories is a core challenge in multimodal understanding. Existing embedding models excel at global retrieval, yet their single global representation entangles multiple objects and their spatiotemporal dynamics, limiting direct object-level grounding. To address this issue, we propose VTrajEmbed, an embedding model that decomposes a video into object trajectories and aligns each trajectory embedding with text for object-level grounding through retrieval. Our main contributions: (1)Slow-Fast Trajectory Encoding, a dual-branch architecture that establishes trajectory representations by densely sampling low-resolution frames for high-frequency motion cues and sparsely selecting high-resolution frames for scene context, aggregated by a shared LLM backbone; (2)Mask-Guided Contrastive Learning, a training objective built upon these representations that couples an intra-group loss and a batch-level bidirectional loss under a relation mask, excluding unverified relations and preventing confirmed matches from becoming false negatives; (3)Text2Traj-Bench, a trajectory-level text-video benchmark with model-assisted construction and human-verified query--trajectory relations. Experimental results show that VTrajEmbed-5B improves text-to-trajectory and trajectory-to-text R@1 over Qwen3-VL-Embedding-8B by 15.48 and 18.05 percentage points, respectively, and achieves 49.48% average R@1 across five global retrieval benchmarks, compared with 42.86% for Qwen3-VL-Embedding-8B.

Abstract

Vision-Language Retrieval

Explore similar work

VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings

DREAM: Extending Vision-Language Models with Dual-Objective Encoding for Cross-Modal Retrieval

SALI: Shot-Aware Late Interaction for Cross-Shot Relation Matching in Text-to-Video Retrieval using Film-Grammar Knowledge