PEAR: Progressive Evidence-Based AutoResearch for Industrial Search Systems

Yifan Wang; Shipeng Zhu; Fei Xiong; Yuqin Yang; Yonghui Huang; Kunyao Wu; Yue Wang; Weichao Meng

cs.IRSep 28, 2026

PEAR: Progressive Evidence-Based AutoResearch for Industrial Search Systems

Authors: Yifan Wang, Shipeng Zhu, Fei Xiong, Yuqin Yang, Yonghui Huang, Kunyao Wu, Yue Wang, Weichao Meng, +1 more

Organizations: Global E-Commerce Agentic Search Team

Abstract

AutoResearch improves systems through iterative experimentation: agents propose candidate modifications, evaluate them, and use the results to guide subsequent exploration. Applying this paradigm to industrial search presents two challenges. (1) Common AutoResearch approaches follow a keep-if-better rule, retaining the highest-scoring candidate for subsequent experiments. Under non-stationary traffic, transient gains may be mistaken for persistent improvements, impairing reliable accumulation of search knowledge. (2) Candidate modifications can be evaluated at multiple fidelity levels, from low-cost proxies to online validation, differing in cost, objective alignment, and statistical reliability. Existing methods rely on individual signals or task-specific procedures, lacking a unified basis for using evidence across levels to guide search. We introduce Progressive Evidence-Based AutoResearch (PEAR) with two complementary components. Evidence-driven AutoResearch maintains an independent, hypothesis-guided research state for each strategy task within a predefined objective and intervention scope. Each state evolves through a Plan-Execute-Evaluate-Update transition that links experimentation to context-aware evidence interpretation and hypothesis revision. Confidence-Gated Verifier Ladder organizes evaluation into four levels of increasing fidelity: Offline Replay, Shadow-Traffic Evaluation, Rapid Online Evaluation, and Decision-Grade Online Evaluation. A unified confidence-based gate promotes candidates only when evidence supports a statistically significant positive effect, enabling broad low-cost exploration while reserving costly online experiments for promoted candidates. In a real-world industrial search system, strategies optimized with PEAR significantly increased Main Order/DAU by 2.7336% and 3.2957% relative to their respective baselines in two A/B experiments.

Figures & tables

Figure 1 : Overview of PEAR. Evidence-driven AutoResearch maintains parallel strategy-level research states, while the Confidence-Gated Verifier Ladder progressively evaluates candidates, governs their promotion using evidence of increasing fidelity, and returns evaluation results to the corresponding research states.

Task phase	Experimental rounds	Groups	Unique configs.	Promoted from L2	Promotion rate
A	19	133	20	2	1.50%
B–I	12	84	23	5	5.95%
B–II	2	14	6	1	7.14%
B–III	21	147	53	2	1.36%
C	16	76	54	3	3.95%

Table 1 : L2 search scale and candidate selection. Unique configurations are deduplicated within each phase; promotion rate is the number of candidates promoted from L2 divided by the number of experimental groups.

Experimental round / research question	Controlled interventions	Evidence-driven update
1. Which constraint axis accounts for the observed gain?	Apply one-axis changes to global and format-specific constraints; also test the coupled global change $G=(3,2)$ .	Only $G=(3,2)$ yields a significant positive order signal. The effects of increasing $w$ , increasing $q$ , and experimental variation remain entangled.
2. Can $G=(3,2)$ be reproduced, and which variable explains its gain?	Replicate $G=(3,2)$ ; compare window-only $G=(3,1)$ , capacity-only $G=(2,2)$ , and boundary settings including $G=(3,3)$ .	$G=(3,3)$ produces a stronger signal. Replications of $G=(3,2)$ yield mixed directional effects and indicate potential degradation in the exit metric. The next experimental round prioritizes capacity and replication.
3. Is the gain driven by the coupled change or by capacity alone?	Replicate $G=(3,3)$ and $G=(3,2)$ ; test capacity-only $G=(2,2)$ and wider-window boundary settings.	$G=(2,2)$ achieves the strongest observed result. The revised hypothesis attributes the gain primarily to capacity relaxation without requiring a wider window.
4. Are the marginal effects of window and capacity stable?	Concentrate replications on $G=(2,2)$ , $G=(3,2)$ , and $G=(3,3)$ ; retain window-only $G=(3,1)$ as a counterfactual.	$G=(3,3)$ remains directionally favorable on order and exit signals but is not significant. The capacity-centered hypothesis is retained for further replication.

Table 2 : Representative hypothesis-revision trajectory for Task A. The table shows four early experimental rounds from a 19-round search;

G=(w,q)

denotes the global window and capacity constraint.

Phase	Experimental rounds	Groups	Unique configs.	Promoted from L2	L4 outcome
I	12	84	23	5	No significant gain
II	2	14	6	1	Positive, not significant
III	21	147	53	2	Significant order gain

Table 3 : Outcomes from Decision-Grade Online Evaluation across search phases for Task B.

Figure 2 : Hypothesis-driven search trajectory for Task B. Candidate evaluations update a retained hypothesis state through validation, rejection, and branch termination until Decision-Grade Online Evaluation.

	Click	Order	GMV
Signed relative error	$-16.02\%$	$\mathbf{-3.49\%}$	$-18.23\%$

Table 4 : Signed relative errors between the aggregate means of proxy predictions and observed online outcomes.

Candidate	L2: Shadow-Traffic Evaluation	L3: Rapid Online Evaluation	L4: Decision-Grade Online Evaluation
A	Order proxy $+45.88\%$	Observed order $+15.48\%$	Main Order/DAU $+2.73\%$
B	Order proxy $+61.64\%$	Observed order $+29.76\%$	Main Order/DAU $+2.16\%$

Table 5 : Candidate effects across evaluation stages for two Task A candidates. L2 reports changes in the order proxy, L3 reports changes in observed orders, and L4 reports changes in Main Order/DAU.

Task	SearchPV/DAU	ASN	GMV/DAU	Main Order/DAU	SKU Order/DAU	PayPV/PV	Main OPMS	SKU OPMS
A	$+0.0063\%$	$\mathbf{+1.1512\%}$	$+2.2136\%$	$\mathbf{+2.7336\%}$	$\mathbf{+2.7259\%}$	$+1.0441\%$	$\mathbf{+2.7407\%}$	$\mathbf{+2.7375\%}$
B	$+0.0470\%$	$\mathbf{+1.3841\%}$	$+1.6633\%$	$\mathbf{+3.2957\%}$	$\mathbf{+2.9826\%}$	$\mathbf{+1.6298\%}$	$\mathbf{+3.2595\%}$	$\mathbf{+2.9448\%}$

Table 6 : Relative metric changes in online A/B evaluations of strategies optimized with PEAR for two search optimization tasks. All values are relative to the corresponding baseline; dark-green boldface indicates statistical significance at

p<0.05

Explore similar work

Jul 19, 2026cs.MA

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Auto Research uses language-model agents to propose, implement, and evaluate machine-learning changes in a closed loop, but is usually judged by its terminal pipeline. A terminal score cannot reveal which technical decision produced a gain or distinguish a reusable discovery from a change adapted to development feedback. We introduce intervention-centered Auto Research, which validates research decisions rather than only final artifacts and makes their reliability measurable. Feature, Model, Representation, and Data axes are searched independently with inner five-fold feedback. Each axis winner is frozen before an outer-holdout matrix compares all alternatives on evidence the loop never sees. Across 701 agent-executed attempts spanning ten Matbench endpoints, outer evidence confirms the selected intervention on nine of ten endpoints and preserves 89.3% of non-tied intervention orderings. It also rejects an aggregate Representation gain that inner feedback endorsed. The resulting matrix reveals an information-dependent hierarchy. Composition-only tasks support several routes to improvement, whereas structure-informed tasks favor local geometry features and complementary tree ensembles. A subsequent compatibility test combines already frozen Feature and Model code without further search or tuning and raises mean outer-holdout improvement from 19.0% to 26.3%. By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.

Jingjie Ning, Xiaochuan Li, Shanshan Zhong +2

School of Computer Science, Carnegie Mellon University · 2DP Technology

Research Automation Auto Research Reusable Knowledge Feedback Materials

Ask PDF

May 8, 2026cs.NE

GEAR: Genetic AutoResearch for Agentic Code Evolution

Autonomous research agents can already run machine learning experiments without human supervision, but many rely on a narrow search strategy: they repeatedly modify one program and keep changes only when they improve the current best result. This can cause them to discard useful partial ideas, alternative promising directions, and insights from failed or incomplete experiments. GEAR, or Genetic AutoResearch, replaces this single-path search with a population-based search over multiple research states. It keeps a set of strong candidate solutions, selects parents based on productivity, novelty, and coverage, and explores new ideas through mutation and crossover. Each research state stores its code changes, reflections, and performance data, allowing future decisions to build on past discoveries. The paper studies three versions of GEAR: one controlled through prompting, one using a fixed programmatic search controller, and one where the controller itself can evolve during the run. Under the same compute budget and environment, all three versions outperform the AutoResearch baseline. More importantly, while the baseline tends to settle into one local optimum, GEAR continues finding improvements over longer runs. Overall, the results suggest that autonomous research agents become more effective when they maintain multiple promising directions and can adapt their search strategy over time.

Ahmadreza Jeddi, Minh Ngoc Le, Hakki C. Karaimer +2

University of Toronto · Vector Institute · AI Center-Toronto, Samsung Electronics +1

Topic pending

Ask PDF

Jul 27, 2026cs.AI

Efficiency Matters in Autonomous Research

AI-driven autonomous research (AR) systems are becoming increasingly effective across a broad range of tasks. Their performance, however, is still evaluated primarily by the quality of the final outcome. In this paper, we argue that the efficiency of the solution-search process is an equally important but often overlooked dimension of performance. A strong AR system should not only produce high-quality results, but also reach them with as small a budget as possible. Search efficiency will become increasingly important as AR expands from domains with inexpensive verification, such as mathematics and coding, to real-world scientific settings in which solution evaluation may require costly physical experiments. To capture this dimension, we propose evaluating AR systems using the area under the curve (AUC) of the Pareto frontier, alongside final outcome quality. We compare several families of search algorithms, including hill climbing, beam search, tree search, and evolutionary search, across twelve systems-optimization tasks. We find that no single search structure is consistently the most efficient. We also show that search efficiency and final outcome quality are distinct performance dimensions: a method that eventually achieves the best result may nevertheless improve slowly and consume substantially more evaluation budget before reaching that result. Because the most effective search policy is generally unknown in advance, we introduce an adaptive procedure called fluid search, which uses a portfolio bandit to dynamically allocate a fixed evaluation budget across a forest of search processes. Across the evaluated tasks, fluid search achieves the highest overall search efficiency, closely matching the performance of a per-task oracle that is given the best search structure for each task in advance.

Haiqian Yang, Yuan Cao

Research Automation Search Algorithms Efficiency Bandits

Ask PDF