VQ-Jarvis: Retrieval-Augmented Video Restoration Agent with Sharp Vision and Fast Thought

Xuanyu Zhang; Weiqi Li; Qunliang Xing; Jingfen Xie; Bin Chen; Junlin Li; Li Zhang; Jian Zhang

cs.CVMar 24, 2026

VQ-Jarvis: Retrieval-Augmented Video Restoration Agent with Sharp Vision and Fast Thought

Authors: Xuanyu Zhang, Weiqi Li, Qunliang Xing, Jingfen Xie, Bin Chen, Junlin Li, Li Zhang, Jian Zhang,

Organizations: School of Electronic and Computer Engineering, Peking University, Shenzhen, China. · ByteDance Inc., Shenzhen, China.

Abstract

Video restoration in real-world scenarios is challenged by heterogeneous degradations, where static architectures and fixed inference pipelines often fail to generalize. Recent agent-based approaches offer dynamic decision making, yet existing video restoration agents remain limited by insufficient quality perception and inefficient search strategies. We propose VQ-Jarvis, a retrieval-augmented, all-in-one intelligent video restoration agent with sharper vision and faster thought. VQ-Jarvis is designed to accurately perceive degradations and subtle differences among paired restoration results, while efficiently discovering optimal restoration trajectories. To enable sharp vision, we construct VSR-Compare, the first large-scale video paired enhancement dataset with 20K comparison pairs covering 7 degradation types, 11 enhancement operators, and diverse content domains. Based on this dataset, we train a multiple operator judge model and a degradation perception model to guide agent decisions. To achieve fast thought, we introduce a hierarchical operator scheduling strategy that adapts to video difficulty: for easy cases, optimal restoration trajectories are retrieved in a one-step manner from a retrieval-augmented generation (RAG) library; for harder cases, a step-by-step greedy search is performed to balance efficiency and accuracy. Extensive experiments demonstrate that VQ-Jarvis consistently outperforms existing methods on complex degraded videos.

Figures & tables

Figure 1: Overview of the challenges in existing enhancement comparison and the construction of VSR-Compare. (a) Existing quality understanding models (e.g., Q-Insight) fail to reliably capture subtle differences between enhancement results (Acc. $<$ 60%). (b) Data domain and degradation distributions of our VSR-Compare. (c) Training sample from VSR-Compare.

Figure 2: Overview of VSR-Compare pipeline, including degradation-aware pair construction, human machine collaborative annotation, and the data distribution of VSR-Compare. The outputs of multiple MLLMs are first filtered to remove inconsistent selections, and their reasoning results are fused; together with additional human expert annotation and verification, they jointly form the comparison pairs.

Figure 3: Overview of the proposed VQ-Jarvis framework. Given a degraded input video, a degradation perception model first estimates the video quality score and degradation attributes. Based on the predicted score, the agent adaptively selects between two restoration strategies: one-step retrieval, which retrieves an optimal restoration trajectory from a quality-aligned RAG library using prior reconstruction experience, and step-wise greedy search, which sequentially applies and compares multiple restoration operators across sub-tasks.

Dataset	Method	LPIPS $\downarrow$	DISTS $\downarrow$	CLIPIQA $\uparrow$	MUSIQ $\uparrow$	MANIQA $\uparrow$	DOVER $\uparrow$	Ewarp $\downarrow$	VQ-Insight $\uparrow$
UDM10	SeedVR2 Wang et al. (2026a)	0.265	0.153	0.347	50.09	0.523	0.327	2.56	50.20
	MGLD-VSR Yang et al. (2024a)	0.310	0.191	0.464	58.12	0.556	0.326	3.31	53.40
	DOVE Chen et al. (2025b)	0.265	0.174	0.537	60.64	0.510	0.477	2.22	57.40
	FlashVSR Zhuang et al. (2025)	0.288	0.175	0.521	65.90	0.612	0.468	2.83	59.60
	DOVER-guide Wu et al. (2023)	0.276	0.175	0.503	64.27	0.548	0.611	3.70	58.80
	Ours	0.288	0.178	0.538	66.88	0.595	0.479	2.25	60.20

Table 1: Quantitative comparison on UDM10 and YouHQ40 between our VQ-Jarvis and other competitive methods. Overall, our method outperforms all single-operator approaches and surpasses methods that rely on traditional metric DOVER for selection. Best and second-best results are highlighted in red and blue , respectively.

Group	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	MANIQA $\uparrow$	CLIPIQA $\uparrow$	MUSIQ $\uparrow$	Ewarp $\downarrow$	DOVER $\uparrow$
Group1	VRT Liang et al. (2024)	15.71	0.471	0.551	0.281	0.386	0.165	33.84	15.67	0.159
	X-Restormer Chen et al. (2024b)	15.45	0.585	0.469	0.266	0.354	0.162	32.28	5.75	0.168
	Random Choice	16.92	0.552	0.412	0.214	0.421	0.301	52.47	9.86	0.342
	Qwen3-MoE Bai et al. (2025)	17.83	0.568	0.338	0.187	0.496	0.358	58.91	8.12	0.417
	Ours	18.70	0.584	0.340	0.163	0.582	0.416	64.35	7.01	0.507
Group2	VRT Liang et al. (2024)	22.61	0.686	0.455	0.201	0.484	0.224	41.81	22.58	0.389

Table 2: Quantitative comparison under 3 multi-degradation settings (Group1, Group2, Group3) on our constructed benchmark.

Figure 4: Qualitative comparison between our VQ-Jarvis and other methods. The time profile of each video are shown below.

Degradation	Qwen3-VL-8B	Qwen3-MoE	Gemini3 Pro	Ours
Dark	56.07	54.20	58.34	83.83
Rain	92.31	89.71	91.02	96.40
Blur	67.61	63.24	69.15	94.61
Compression	58.30	57.77	60.48	87.47
Noise	75.51	66.18	71.62	91.35
Low-Resolution	41.30	53.36	56.74	94.83

Table 3: Degradation detection accuracy (%) comparison between our VQ-Jarvis and other methods.

Method	DISTS $\downarrow$	MANIQA $\uparrow$	Ewarp $\downarrow$	DOVER $\uparrow$	Time (s) $\downarrow$
CLIP-RAG	0.192	0.553	3.02	0.396	12.44
Ours-RAG	0.166	0.589	2.88	0.426	12.44
Ours-Greedy	0.157	0.608	2.72	0.442	42.65
Ours	0.163	0.603	2.80	0.454	17.88

Table 4: Ablation study on the key components of our VQ-Jarvis.

Model	LSVQ-Test (PLCC / SRCC)	LSVQ-1080p (PLCC / SRCC)	LIVE-VQC (PLCC / SRCC)	KonViD-1k (PLCC / SRCC)
Fast-VQA Wu et al. (2022)	0.878 / 0.874	0.810 / 0.765	0.815 / 0.769	0.857 / 0.859
Minimalist-VQA Sun et al. (2024)	0.872 / 0.880	0.818 / 0.769	0.812 / 0.765	0.861 / 0.859
DOVER Wu et al. (2023)	0.886 / 0.888	0.828 / 0.787	0.819 / 0.771	0.883 / 0.890
Modular-VQA Wen et al. (2024)	0.891 / 0.894	0.844 / 0.791	0.825 / 0.783	0.884 / 0.878
Q-Align Wu et al. (2024b)	0.882 / 0.883	0.833 / 0.758	0.813 / 0.777	0.876 / 0.865
Q-Instruct Wu et al. (2024a)	0.580 / 0.602	0.640 / 0.644	0.673 / 0.660	0.520 / 0.492

Table 5: PLCC / SRCC comparisons on natural video quality assessment benchmarks.

Method	Color		Noise		Artifact		Blur		Temporal		Overall
Method	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC
NIQE Mittal et al. (2012)	0.2368	0.3273	0.1417	0.2682	0.1649	0.3006	0.1870	0.3236	0.1630	0.2777	0.2019	0.3192
VIDEVAL Tu et al. (2021a)	0.6943	0.6922	0.6514	0.6912	0.7370	0.7440	0.7637	0.7610	0.7123	0.7174	0.7307	0.7310
RAPIQUE Tu et al. (2021b)	0.6330	0.6203	0.5607	0.6048	0.6276	0.6258	0.6717	0.6572	0.5429	0.5554	0.6501	0.6379
VSFA Li et al. (2019)	0.7837	0.7617	0.7278	0.7635	0.8170	0.8006	0.8001	0.7772	0.7018	0.7276	0.7929	0.7730
GSTVQA Chen et al. (2021)	0.7761	0.7747	0.7448	0.7883	0.8187	0.8121	0.8202	0.8101	0.7093	0.7533	0.7825	0.7834

Table 6: Performance comparison of different quality dimensions on the FineVQ dataset. PLCC and SRCC are reported. Best and second-best results are highlighted.

Module / Operator	Task	Symbol	Time (s)
Decision and Perception Overhead
Degradation Perception Model	All	$T_{\text{det}}$	2.45
Multiple Operator Judge Model	All	$T_{\text{cmp}}$	2.81
RAG Retrieval	All	$T_{\text{rag}}$	0.35
Deraining Operators (30 $\times$ 180p videos)
RainMamba	Deraining	$T_{1,1}$	0.56

Table 7: Runtime breakdown of different components and restoration operators in VQ-Jarvis.

Threshold $\tau$	DISTS $\downarrow$	MANIQA $\uparrow$	Ewarp $\downarrow$	DOVER $\uparrow$	Time (s) $\downarrow$
$\tau=2.4$	0.165	0.592	2.86	0.430	13.95
$\tau=2.6$ (Ours)	0.163	0.603	2.80	0.454	17.88
$\tau=3.0$	0.160	0.600	2.78	0.446	29.04
$\tau=3.6$	0.159	0.607	2.78	0.444	39.93

Table 8: Sensitivity analysis of the pre-defined threshold score

\tau

in VQ-Jarvis.

Case	Restoration Order	LPIPS $\downarrow$	DISTS $\downarrow$	MANIQA $\uparrow$	CLIPIQA $\uparrow$	MUSIQ $\uparrow$	Ewarp $\downarrow$	DOVER $\uparrow$
(a)	Rain + Dark + (BNC + SR)	0.245	0.141	0.461	0.392	56.025	1.51	0.571
(b)	Dark + Rain + (BNC + SR)	0.264	0.144	0.422	0.360	50.345	2.14	0.568
(c)	(BNC + SR) + Rain + Dark	0.332	0.178	0.468	0.310	52.210	1.93	0.452
(d)	Dark + BNC + FI	0.349	0.166	0.455	0.305	49.557	1.04	0.459
(e)	FI + Dark + BNC	0.325	0.168	0.473	0.331	50.537	1.65	0.442
(f)	Dark + FI + BNC	0.332	0.171	0.475	0.345	51.339	1.31	0.445

Table 9: Quantitative comparison under different degradation orderings.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Visualized results of our VQ-Jarvis and other competing methods on multi-degradation video restoration methods.

Figure 6: Visualized results of the paired comparison results of our VQ-Jarvis.

Figure 7: Visualized results of the video quality scoring and degradation detection using our VQ-Jarvis.

Prompt for MLLM Expert: You will see two videos represented by sampled frames. Treat the first as Video A and the second as Video B. Decide which is better. Output exactly: Answer:

<

Video A, Video B, Tie

>

Prompt for Multiple Order Judge Model: Video A and Video B are results from two video enhancement methods. Explain and justify whether Video A or Video B is better. You should think from the following aspects: (1) Visual quality, (2) temporal consistency, (3) dynamic degree.

Prompt for Video Scoring: Please rate the quality of this video with a score accurate to two decimal places. The range of quality score is between 1-5. Explain the reasoning process concisely from spatial quality, temporal quality, and dynamic degree.

Prompt for Degradation Detection: Detect whether each degradation exists in this video. Return the result in

<

answer

>

…

<

/answer

>

as JSON with keys: “dark", “rain", “blur", “compression", “noise", “low-resolution", “low-frame".

Appendix

Table 10: Prompts for the MLLM comparison experts and for the VQ-Jarvis.

Figure 8: GUI used for human annotations.

Figure 9: Low-light enhancement paired comparison training samples of our VSR-Compare.

Figure 10: Super-resolution paired comparison training samples of our VSR-Compare.

Video Restoration

Video restoration reconstructs clearer, more faithful, temporally consistent footage from blur, noise, compression, weather, and other degradations.

See all 59 papers

OPERA: An Agent for Image Restoration with End-to-End Joint Planning-Execution Optimization
May 21, 2026Feng Zhu, Shuyang Xie, Yihan Zeng +2Agent-Based Image Restoration Image Restoration
Harbin Institute of Technology · Huawei Noah’s Ark Lab
TIR-Agent: Training an Explorative and Efficient Agent for Image Restoration
Mar 29, 2026Guoli Jia, Yisheng Zhang, Haote Hu +11Tool-Using Vision-Language Agents Image Restoration
Tsinghua University, Beijing, China · Data Science & Artificial Intelligence Research Institute, China Unicom, Beijing, China · Hunan University, Changsha, China +2
FastVR: Efficient Streaming Video Restoration with One-Step Diffusion
Sep 29, 2026Xiaoxu Chen, Qin Yang, Haoran Bai +2Diffusion Transformer Video Restoration
Alibaba Group · Xidian University

May 21, 2026J/K move · Enter open · S save

OPERA: An Agent for Image Restoration with End-to-End Joint Planning-Execution Optimization

Feng Zhu, Shuyang Xie, Yihan Zeng, Ming Liu+1

Harbin Institute of Technology · Huawei Noah’s Ark Lab

Agent-Based Image Restoration Image Restoration

Open paper AskPDF arXiv

Real-world image restoration is challenging due to complex and interacting mixed degradations. Recent agent-based approaches address this problem by composing multiple task-specific restoration tools. However, empirical analysis reveals that their performance is fundamentally limited by implicitly constrained planning spaces and the lack of coordination among independently pretrained tools. To address these issues, we propose OPERA (Optimized Planning-Execution Restoration Agent), a framework that jointly optimizes restoration planning and tool execution in an end-to-end manner. On the planning side, OPERA uses reinforcement learning to directly optimize tool composition over a combinatorial plan space, with the final restoration quality as the reward. On the execution side, OPERA introduces agent-guided co-training of restoration tools, enabling them to learn cooperative behaviors under sequential composition. Extensive experiments on multi-degradation benchmarks and real-world datasets demonstrate that OPERA consistently outperforms both all-in-one restoration models and existing agent-based methods across diverse and complex degradation scenarios.

Abstract

Video Restoration

Explore similar work

OPERA: An Agent for Image Restoration with End-to-End Joint Planning-Execution Optimization

TIR-Agent: Training an Explorative and Efficient Agent for Image Restoration

FastVR: Efficient Streaming Video Restoration with One-Step Diffusion