Improving an industrial recommender is an iterative research-and-engineering process rather than a direct path from idea to deployment. In DASHEN, NetEase's gaming-community app, algorithm engineers typically identify promising directions from research papers, technical reports, and prior production experiments; reproduce or adapt the underlying methods; implement them in the production codebase; and evaluate the resulting models through training and offline experiments. Promising candidates are then advanced to online A/B tests, and those demonstrating robust gains are submitted to Launch Review---the internal gate for full-traffic rollout. Large language models (LLMs) can assist with individual stages of this workflow, but the overall process remains human-dependent without a harness that can reliably coordinate them across long-running, often multi-day experimental cycles. We present AutoLR, initially built as Auto Launch Review and later extended upstream into an autonomous research-to-launch harness. AutoLR combines three system mechanisms: a multi-expert council that debates and adversarially reviews proposals; a deterministic evidence-weighted exploration--exploitation selector that allocates a limited trial budget across candidate directions and uses Council reranking; and a layered knowledge system that combines external research, production-system knowledge, and DASHEN-specific domain knowledge---such as game communities, player characteristics, and content-interaction patterns---with posterior evidence from configurations, patches, logs, failures, and offline outcomes. LLM agents perform semantic reasoning and code generation, while deterministic controllers retain authority over execution, metric extraction, guardrails, and persistent state transitions.
Figures & tables
Figure 1. The two core DASHEN recommendation scenarios studied in this paper: the unified single- and two-column feed and the immersive-video feed. Two DASHEN mobile screenshots side by side. The left screen shows the community feed in a two-column card layout. The right screen shows a full-screen immersive video player with engagement controls.
Figure 2. AutoLR architecture. Prior knowledge and experiment memory feed a multi-expert council and a budgeted selector. LLM agents propose and implement; deterministic controllers own verification, execution, metrics, guardrails, and persistent state. Autonomy ends at offline packaging; online A/B and Launch Review remain human-gated. Terminal results write back to memory—the feedback path behind the KEEP ratchet (Section 6.1 ). A research-to-launch architecture diagram. Evidence and experiment memory feed a multi-expert council, a deterministic evidence-weighted direction selector, and repository-grounded implementation. Deterministic controllers own offline verification, execution, metrics, vector guardrails, verdicts, and candidate packaging. Online A/B testing, Launch Review, and full rollout appear in a separate dashed human-gated region, with an aspirational dashed feedback path.
Figure 3. Four-stage Multi-Expert Proposal Council. Independent specialists propose from a frozen task contract; roundtable synthesis clusters and reconciles mechanisms; adversarial review attempts to falsify survivors, with optional rebuttal; a coordinator emits a structured candidate set with a declared edit surface and activation checks. A left-to-right process diagram. A task-contract input feeds independent specialist proposals, roundtable synthesis, adversarial review with an optional rebuttal loop, and a coordinator decision, producing a structured candidate set that proceeds to direction selection.
Figure 4. Pipeline connecting the Multi-Expert Proposal Council to evidence-weighted direction selection. Council deliberation is advisory; admission, top-1 execution, and the memory-based gate remain controller-governed. A two-panel pipeline. The left panel shows council stages from a task contract to a structured candidate set. The right panel shows deterministic admission, council reranking, a memory-based gate, and top-1 training. Solid arrows mark authoritative control and dashed arrows mark advisory deliberation.
Figure 5. Evidence-weighted direction selection. A diversity-aware frozen window defines the search space; the council reranks within that window; a memory-based gate scores candidates using historical evidence before GPU training. Numerical outcomes write back to experiment memory. A left-to-right routing diagram. A pending experiment backlog is filtered into a frozen candidate window, then council-reranked, gated by historical memory, and executed as a top-1 GPU training trial. Dashed arrows return outcomes to experiment memory, which feeds the filter and the memory-based gate.
Figure 6. Layered knowledge and experiment memory. External research, production-system facts, and DASHEN domain knowledge are relatively stable; experiment memory evolves after each trial. Role-specific retrieval supplies bounded evidence to research, repository-feasibility, and domain agents rather than concatenating all sources into one context. A layered architecture diagram. Four knowledge stores—external research, production-system knowledge, DASHEN domain knowledge, and evolving experiment memory—feed a role-specific retrieval layer that supplies bounded evidence to a research agent, a repository-feasibility agent, and a DASHEN-domain agent.
Figure 7. Production evidence. (A) Path-aware reconstruction reduces 4,250 physical ledger records to 1,586 completed evaluations, of which 1,583 retain numeric scalar deltas. (B) Nine separately maintained production Launch Review records are shown in sequence order for the two recommendation scenarios and restricted to three primary overall-traffic metrics. The records are heterogeneous and are not pooled; immutable offline-to-online-to-review linkage is not retained in the audited repositories. A two-panel figure. Panel A shows the corpus reconstruction from 4,250 physical ledger records through 4,247 parseable records and 3,289 canonical experiment records to 1,586 completed evaluations and 1,583 numeric scalar deltas. Panel B shows nine Launch Review records—six for the single- and two-column feed and three for the immersive-video feed—reporting content-consumption penetration rate, total content-consumption time, and valid content views when available.
No.
Scenario
Content-consumption penetration rate
Total content- consumption time
Valid content views (VV)
1
Single-/two-column feed
+1.15%
Flat
Flat
2
Single-/two-column feed
+0.34%
+1.41%
Flat
3
Single-/two-column feed
+2.06%
+1.73%
+2.16%
4
Immersive-video feed
Flat
+1.51%
+1.65%
5
Single-/two-column feed
+0.42%
+1.10%
Flat
6
Single-/two-column feed
+0.85%
+1.01%
Flat
Table 1. Nine production Launch Review records in DASHEN’s two primary recommendation scenarios, indexed by sequence. Values are reported relative changes against contemporaneous controls. Only the three primary overall-traffic metrics are shown; Flat denotes no reported overall-traffic movement.
Figure 8. Promotion reliability. (A) The historical distribution of trunk-relative scalar deltas is compared with the common reference floor 10−3 ; this is a descriptive replay rather than proof of an operative threshold for rows without threshold provenance. (B) The available single-/two-column guardrail records illustrate that scalar-qualified candidates can still be blocked by protected-metric violations; missing structured payload is shown separately and is not interpreted as compliance. A two-panel figure. Panel A shows empirical distributions of trunk-relative scalar deltas for the single- and two-column feed and the immersive-video feed together with the common reference floor. Panel B shows the available single-/two-column guardrail case study, separating measured protected-metric violations from missing structured payloads.
Quantity
Single-/ two-column
Immersive video
Numeric scalar deltas
810
773
IQR(Δs)
1.257×10−3
1.201×10−3
Reference-floor crossing
4.7%
8.5%
Observed KEEP fraction
3.82%
5.03%
Observed PACK fraction
12.58%
7.35%
Recorded threshold provenance
480
0
Table 2. Promotion-resolution statistics. Crossing rates use the common reference floor τref=10−3 for descriptive replay; they are not interpreted as audited operative-threshold rates for evaluations without threshold provenance.
Optimizing modern recommender models still depends heavily on engineers manually iterating over architectural, objective, and training-strategy changes. While LLM-based agents can automate this trial-and-error process, allowing the LLM to both select modification directions and generate concrete hypotheses often leads to unstable search under limited experiment budgets. Inspired by the above challenge, we propose RecHarness, a Bandit-Routed Agentic Harness for automated recommender model optimization. RecHarness separates the optimization process into two steps: a bandit router selects the next modification direction according to historical validation feedback, while the LLM generates a concrete optimization hypothesis and executable code edit within the selected direction. To sustain long-horizon exploration, RecHarness uses a jump-basin mechanism to activate a structural-jump arm when local edits stagnate. Across multiple recommendation tasks, datasets, and model backbones, RecHarness achieves more stable performance improvements and uses limited trial budgets more effectively than LLM-reasoning search. During a 7-day online A/B test on a large-scale short-video advertising platform, the selected candidate improves ADVV by 2.084%, Revenue by 0.534%, and Exposure by 0.559%. Code is available at https://github.com/6lyc/RecHarness.
Haoran Ling, Yuecheng Li, Zeyu Song +5
Georgia Institute of Technology · Kuaishou Technology, China
Auto-research agents have shown the potential to automate hypothesis generation, experiment execution, and iterative refinement. However, scaling this paradigm to industry-scale recommendation models introduces two challenges: (1) long feedback loops, where model training can take days, making serial iteration prohibitively slow and requiring parallel exploration across multiple research directions; and (2) system complexity, where large configurations, fragile infrastructure dependencies, and multi-day GPU jobs require robust and recoverable execution. We present Auto-RecSys, an autonomous research system for long-horizon experimentation on industry-scale recommendation models. Auto-RecSys addresses these challenges through three harness designs: (1) distributed asynchronous execution for running multiple experiments in parallel across servers, (2) centralized cross-server memory for persistent and recoverable execution across sessions and failures, and (3) cognitive-procedural separation, where natural-language skill files guide LLM reasoning while deterministic scripts enforce operational correctness. Auto-RecSys further employs a dual-loop self-evolving architecture: an Execution Evolution Loop in which model-specific playbooks accumulate operational knowledge by recording failed attempts and crystallizing successful pipelines, and an Idea Evolution Loop in which experimental outcomes inform subsequent ideation. Evaluated on recommendation models, Auto-RecSys significantly reduces the human time required per experiment cycle and improves execution reliability as its playbooks mature.
Recommendation algorithm iteration is moving from an artisanal, engineer-bound process toward an industrialized research loop, but this transition remains blocked by a structural execution bottleneck: the idea-to-launch cycle still depends on human engineers to generate hypotheses, modify production code, launch A/B experiments, and attribute online results. Innovation therefore scales linearly with headcount rather than compounding with evidence, compute, and accumulated experimental knowledge. We present AgentX, a production-deployed multi-agent system that fundamentally restructures this production function. AgentX operates as a self-evolving development engine: it autonomously generates, implements, evaluates, and learns from recommendation experiments at a scale and pace that no manual workflow can sustain. The system orchestrates four tightly coupled stages in a closed loop. A Brainstorm Agent synthesizes evidence from historical experiments, system architecture, data analysis, and external research into ranked, executable proposals. A Developing Agent translates each proposal into production-ready code through repository-grounded generation and multi-dimensional reliability verification. An Evaluation Agent conducts safe online rollout with guardrail-vetoed A/B judgment, converting both successes and failures into structured knowledge assets. A Harness Evolution layer (SGPO) then distills execution trajectories into semantic-gradient updates that continuously sharpen the agents themselves -- making the system not merely automated, but self-improving.