cs.IRSep 18, 2026

Verify, Don't Trust: Agentic Model Development for Video Discovery Retrieval at Scale

Authors: Hao Fu, Baiting Zhu, Minglei Chen, Yinjie Huang, Shuai Ding

Organizations: Meta Platforms, Inc. Menlo Park, California, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language model (LLM) agents can propose, implement, and evaluate model changes. Autoresearch loops demonstrate this capability through minutes-scale iterations on a self-contained program. Online autoresearch instead spans asynchronous systems, hours-long variants, and weeks-long campaigns that can influence a product. A completed run can still support an invalid conclusion when a code change is a no-op, data windows leak, evaluator semantics drift, or the two arms traverse different serving funnels. We present EvoPilot, a human-gated method for long-horizon online autoresearch. Role-specific agents execute each round through a versioned domain skill and typed adapter. Durable records preserve experiments and failures; deterministic checks enforce recorded lessons. We study a 37-day campaign for the retrieval system that powers Video Deep Dive (VDD), an online experience for discovering follow-on videos after a user opens a seed video. The campaign covered seven directions and used an hourly refreshed index of hundreds of millions of videos. Earlier manual experiments had not established a benefit from an interaction head. A primitive autoresearch attempt revisited the direction but incorrectly attributed an offline hit-rate decline of 22 percentage points to the head. We then introduced EvoPilot. Its human-gated verification traced the drop to a pre-existing evaluation defect that produced output depths of 3,000 and 600. After repair, a matched comparison measured an offline improvement of 3.20 percentage points. Post-study replay and mutation tests rejected invalid comparisons while admitting valid counterparts. Durable state recovered an interrupted round, and artifact reuse avoided approximately five GPU-hours. Separately, a seven-day randomized online evaluation estimated a 0.66% relative increase in the VDD slice of Good Search Result Rate for Retention (GSRR).

Figures & tables

Validation layerDecision on two evaluation artifacts
Run completionboth return terminal metrics
Per-run lineageeach metric is traceable to its evaluator revision
Per-run artifact validationboth artifacts are internally complete
Post-study pair admissionreject: evaluator mismatch is outside the approved contrast
Table 1. Post-study replay of unauthorized evaluator drift.
Figure 1. Comparison-centric, cross-round EvoPilot loop. One authorized contrast is shown; multi-variant rounds verify each separately. Admitted evidence shapes the next proposal, which returns to human approval. A human-approved protocol creates control and treatment specifications that execute in isolation through a domain skill and adapter. Artifact-backed results undergo pair-level verification and human review. An analyst synthesizes admitted evidence into a next-direction recommendation, and a planner proposes next-round variants that loop back to human approval. Durable state connects rounds.
RoleCapabilityTyped output
Round plannerread-only evidencevariant plan for fresh approval
Base builderedit shared baselinesource/configuration change set
Experiment workerisolated skill/adapter executionresult record + trace
Results analystread admitted resultshypothesis update; stop/next direction
Failure triage/repairdiagnose; gated repairfault verdict / operational fix
Periodic monitorread incremental eventsignore, repair, or escalate
Table 2. Agent roles, capabilities, and typed outputs.
InvariantControlIntroduction
Round authorityapproval tied to the round identifieroutset
Environmentdedicated clone or failoutset
Change identitylineage, nonempty code diff, effective-configuration readbackstrengthened
Data identitydate pins, partition and drift checksstrengthened
Train/eval orderconfiguration and artifact-reuse checksafter leakage
Completenessfailure gate; explicit partial closureduring study
Table 3. Control coverage at manuscript cutoff. Executable controls block normal progression on failure.
OperationObserved record
Round inclusion9/20 rounds included; 11 excluded for incomplete operation fields
Workflow closure29/29 retained terminal records: 27 scientific results; two OOM failures
Stage construction22 train attempts (20 succeeded); 22 publishes; 27 evaluations; 71 total
Primary role assignments56: 9 planner, 9 base-builder, 29 worker, 9 analyst
Elapsed operation∼\sim 97 h summed orchestrator wall-clock across the nine rounds
Repair and recovery24 agent retries after local fixes; 6 operator repairs; 6 subworkflows and one round recovered; one escalation
Table 4. Inclusion flow and complete operation census for the auditable cohort.
InvariantClean fixtureOne-field mutation
Round authoritycurrent round proceedsstale approval rejected
Change identitynonempty code diff passesabsent code diff rejected
Override identityeffective configuration matcheswrong setting rejected
Data identitypinned dates matchone date drift rejected
Artifact reuseequivalent key passesincompatible reuse rejected
Train/eval ordereval follows trainleakage rejected
Table 5. Paired replay/mutation: 10/10 clean fixtures admitted and 10/10 corresponding faults blocked.
StageEstimand / comparabilityResult
Expert-led manual evidencemixed estimands; one pair coupleddirection not advanced
Primitive autoresearch baselinemismatched KK ; incorrect attribution−22-22 pp
Diagnostic re-testbundled effect; (P,K)=(3000,600)(P,K){=}(3000,600)+4.8+4.8 pp
Matched-lineage ablationhead-only effect; K=3000K{=}3000 , P: 3000 ⁣→ ⁣6000P{:}\,3000\!\rightarrow\!6000+3.20+3.20 pp
Online A/Bhead-only inference treatment; fixed remainder+0.66%+0.66\% rel.
Table 6. Interaction-head comparisons across funnel configurations. Offline replay estimates are descriptive.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

ProgramR/VT/P/EHoursRetry / recovery / repair
Score R11/40/1/42.31 fix; subworkflow recovered; one repair
Depth R31/30/1/32.2prelaunch repair
Capacity R1–R22/88/8/836.012 fixes; subworkflow recovered
Capacity R31/22/2/217.01 fix; subworkflow recovered; one repair
Capacity R41/22/2/28.0subworkflow recovered; one repair
Capacity R51/22/2/27.53 fixes; subworkflow recovered
Appendix
Table 7. Round-level operation census. R/V denotes rounds/variants, and T/P/E denotes logical train/publish/evaluate stages. A “fix” is an agent-initiated local correction; a “repair” is an operator action.
ScopeRepresentationConsumer / purpose
Run ledgerevents, atomic state, variant/round resultsresume, traceability, regression check
Program memoryplans, prior results, relevant experiment summariesnext-round planner/workers
Incident catalog38 records in six categoriesfailure retrieval; guard design
Appendix
Table 8. Persistence scopes. Memory is advisory; enforcement requires executable checks.
More in this topic

Autonomous Experimentation

Autonomous experimentation uses AI, active learning, and robotics to design, run, and interpret experiments with less human intervention.

See all 153 papers

Explore similar work

CardsList
Oct 4, 2026

VideoResearchAgent: Grounded Task Synthesis and Sim-to-Real RL for Open-Web Video Research

Existing deep research agents are designed primarily for text- and image-based web sources, while video reasoning systems typically assume that relevant videos are provided in advance. We study open-web video research, where an agent must autonomously discover relevant videos, navigate their temporal content, and ground answers in visual evidence. Training such agents at scale is challenging as live video interaction is slow and unreliable, whereas fixed local simulation can induce retrieval-specific shortcuts that fail to transfer to the open web. We introduce VideoResearchAgent, a scalable training framework to address these challenges. First, we introduce controllable task synthesis pipeline to synthesize multi-hop research tasks from timestamped visual evidence while filtering text-only shortcuts. Second, we build a field-aligned local video simulator that preserves deployment-facing search and watch interactions while accelerating video search by a factor of 34.5-64.6. Third, we introduce Retrieval-Domain-Randomized GRPO (RDR-GRPO), which diversifies candidate rankings, distractors, metadata, and result structure during training to reduce overfitting to simulated retrieval. On Video-BrowseComp, the VideoResearchAgent trained using Qwen3.5-4B achieves 40.48% accuracy, comparable to Gemini-3-Flash-Preview, while reducing cumulative API-token consumption by 74.9% relative to the untrained model. Together, these results establish an accurate and efficient training recipe for open-web video research.

Yuhang Zhou, Fei Li, Yuxi Wu +2

Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Singapore Management University

Autonomous Research AgentsVideo ReasoningSim-to-Real TransferTemporal Video Understanding
AskPDF
Jul 19, 2026

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Auto Research uses language-model agents to propose, implement, and evaluate machine-learning changes in a closed loop, but is usually judged by its terminal pipeline. A terminal score cannot reveal which technical decision produced a gain or distinguish a reusable discovery from a change adapted to development feedback. We introduce intervention-centered Auto Research, which validates research decisions rather than only final artifacts and makes their reliability measurable. Feature, Model, Representation, and Data axes are searched independently with inner five-fold feedback. Each axis winner is frozen before an outer-holdout matrix compares all alternatives on evidence the loop never sees. Across 701 agent-executed attempts spanning ten Matbench endpoints, outer evidence confirms the selected intervention on nine of ten endpoints and preserves 89.3% of non-tied intervention orderings. It also rejects an aggregate Representation gain that inner feedback endorsed. The resulting matrix reveals an information-dependent hierarchy. Composition-only tasks support several routes to improvement, whereas structure-informed tasks favor local geometry features and complementary tree ensembles. A subsequent compatibility test combines already frozen Feature and Model code without further search or tuning and raises mean outer-holdout improvement from 19.0% to 26.3%. By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.

Jingjie Ning, Xiaochuan Li, Shanshan Zhong +2

School of Computer Science, Carnegie Mellon University · 2DP Technology

Automated Algorithm DiscoveryAutomated EvaluationAutonomous Research Agents
AskPDF
Aug 5, 2026

MetaVideoAgent: Automated Video-Agent Evolution for Long-Form Video Understanding

Long-form video understanding requires locating sparse, question-relevant evidence in long, multimodal videos. Real-world video distributions differ in modality-specific information density, content structure, and evidence patterns, causing fixed video-agent designs to incur redundant processing or fail when mismatched. Extending automated agent evolution from text to video is challenging because full long-video execution makes candidate validation expensive, failures propagate across coupled evidence-processing stages, and complex preprocessing, perception tools, and localization strategies make code-level updates difficult to implement reliably. We introduce MetaVideoAgent, a framework that automatically evolves a video agent for a target distribution. It profiles information density and evidence requirements from sparsely sampled frames and associated queries to guide initial design, then compresses localized failures into independently executable minimal validation tasks. It constructs evidence-grounded Gold Paths, audits Student trajectories, aggregates recurring failures across samples, and attributes them to responsible modules. A modular agent representation constrains each update to the primary responsible module and its necessary dependencies. We further introduce VA-EvoBench, covering eight video distributions with separate evolution and held-out splits. With four evolution iterations per distribution, MetaVideoAgent improves every initial agent and raises macro-average accuracy from 38.44% to 51.47%, at an average evolution cost of 3.54M tokens per distribution. The evolved agents outperform the strongest prior fixed-design video agent by 6.39 percentage points while using the fewest tokens and video frames per question among the compared video agents. We will release all code and data to support reproducible research.

Benlei Cui, Ruize Wang, Junjie Li +9

Yuvion Team, Alibaba Group · Zhejiang University · Beihang University +1

Video QALong-Video UnderstandingSelf-Evolving Agents
AskPDF