AdaptEvo: Adaptive Agent Learning with Evolving Supervision

Shijun Wan; Jiancong Xie; Hang Xu; Jin Duan; Qixiong Wang; Xi Xiang; Maofei Que; Yahui Liu

cs.CLOct 8, 2026

AdaptEvo: Adaptive Agent Learning with Evolving Supervision

Authors: Shijun Wan, Jiancong Xie, Hang Xu, Jin Duan, Qixiong Wang, Xi Xiang, Maofei Que, Yahui Liu, +2 more

Organizations: Fudan University · Sun Yat-sen University · Xiaohongshu

Abstract

Rule-governed contextual decision tasks require models to apply specified rules to case-specific context and evidence. Written rules can leave gaps in decision guidance and process evaluation, while reference judgments vary in their support from the rules and evidence. To address these challenges, we introduce AdaptEvo, a framework for learning under imperfect supervision that couples confidence-adaptive policy optimization with evolving decision knowledge and evaluation rubrics. Its Training module uses Confidence-Adaptive GRPO (CA-GRPO) to balance outcome and process rewards according to reference confidence. Its Evolution module synthesizes reusable decision knowledge from recurring failures across training cases and refines process rubrics to detect overlooked errors. To support empirical evaluation, we construct an industrial multimodal content moderation dataset comprising a training set and In-Period and Out-of-Period test sets, with the latter collected under changed rules. Using Qwen3.6-35B-A3B, AdaptEvo achieves 61.9% exact-label accuracy and 72.2% binary decision accuracy on In-Period, exceeding GRPO by 7.5 and 3.7 percentage points, respectively. On Out-of-Period, the policy trained with CA-GRPO retains exact-label accuracy gains over the base model across evaluated checkpoints without injected decision knowledge, while GRPO declines with continued training. CA-GRPO also outperforms the tested fixed reward mixtures on both Out-of-Period metrics.

Figures & tables

Figure 1: A content moderation example of a rule-governed contextual decision task, illustrating unequal support for reference decisions and gaps in rule application and process evaluation. The case and reward curves are schematic.

Figure 2: AdaptEvo alternates training (top) and evolution (bottom). Knowledge evolution turns recurring mismatches across cases (reference A, predominantly B predictions) into conditional guidance. Rubric evolution screens out saturated checks that lack within-case discrimination across many cases, accounting for Judge noise when selecting alternatives. Each rubric row is a check; each strip shows one case’s rollout scores. Labels and scores are schematic; task rules remain fixed.

Model / Method	Knowledge	In-Period		Out-of-Period
		ELA	BDA	ELA	BDA
Qwen3.5-4B	None	46.8	60.8	50.8	64.5
Qwen3.5-9B	None	46.2	63.2	50.4	65.2
Qwen3.5-27B	None	55.1	68.5	53.8	66.0
Qwen3.8-Flash-Next	None	54.9	69.1	54.4	67.6
GLM5.3-Flash	None	56.3	69.6	55.6	67.5

Table 1: Moderation results on the In-Period and Out-of-Period benchmarks (%, higher is better). ELA: exact-label accuracy; BDA: binary decision accuracy. Knowledge specifies injected decision knowledge. Bold marks the best score in each column.

Figure 3: Training reward and Out-of-Period generalization. (a) Outcome rewards during GRPO and GRPO (High-Conf.) training: faint dashed lines show raw values and solid lines show centered 5-step moving averages. (b–c) Out-of-Period ELA and BDA gains over the base model (53.3%/65.9%), in percentage points, without injected decision knowledge. Heatmaps share a color scale: teal indicates gains and orange indicates losses; each cell gives the exact difference.

	In-Period		Out-of-Period
Reward Mixing	ELA	BDA	ELA	BDA
Fixed (0.8:0.2)	54.3	67.6	53.6	66.1
Fixed (0.5:0.5)	54.2	67.9	54.0	66.2
Fixed (0.2:0.8)	55.6	67.3	54.8	65.8
CA-GRPO	55.4	68.1	55.2	67.3

Table 2: Fixed and adaptive reward mixing after 60 training steps. Ratios are outcome:process; ELA and BDA are percentages. Bold marks each column maximum.

Figure 4: Process (blue) and reference-agreement outcome (orange) rewards over training steps 0–59. Panels identify process evaluators and mark the simplified Kimi-K3 rubric. Faint dashed lines show raw step means; solid lines show 5-step centered moving averages (MA5) with truncated edge windows. All panels share the same scale. Scores are unweighted training rewards, not accuracies.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Instance and correspondence
Post $c$ and queue $s$	A mint-green eyeshadow post with product/swatch tags, nine post images, and a transaction-linked commercial marker for Brand A. Author metadata also supplies an avatar and background image (11 image inputs in total). The assigned queue reviews unfair competition in posts.
Candidates $\mathcal{L}_{s}$	$\{p_{s},v_{1},v_{2},v_{3}\}$ , where $p_{s}=\text{Pass}$ , $v_{1}=\text{Unfair commercial comparison}$ , $v_{2}=\text{Commercial denigration}$ , and $v_{3}=\text{Personal negative review}$ . Here $v_{1},v_{2},v_{3}$ abbreviate this queue’s violation labels.
Queue rule $R_{s}^{\mathrm{queue}}$	Establish commercial context using account information and commercial markers; assess the post and relevant author-originated context under the applicable label rules. This queue-wide guidance is supplied in full at initialization.
Label rules $R_{s,\ell}$	For $v_{1}$ , commercial applicability must be accompanied by covered unfair comparison, such as unsupported scoring/ranking or one-sided competitor criticism; balanced multi-product reviews can be exempt. For $v_{2}$ , assess competitive denigration and its prerequisites and exemptions. For $v_{3}$ , assess consumer negative claims, with exemptions for specified subjective preferences or individual fit. Only previews are initially shown; the full rules are retrieved below.
Rule lookup $a_{1}\!\to\!o_{1}$	get_detail_rule(labels=[ $v_{1},v_{2},v_{3}$ ]) returns the full rules for all three violation labels in the current queue. The actual arguments are the corresponding label strings.
Evidence lookup $a_{2}\!\to\!o_{2}$	get_commercial_detail({}) returns no associated product items. The initial transaction-linked marker remains part of the evidence; this empty result does not establish that the post is noncommercial.

Appendix

Table 3: A moderation task instance with two levels of governing rules and a recorded tool-using decision. Label names and excerpts are translated; identifying details are omitted.

Agr.	S	A	U	$n$	%
4-0	1,968	473	245	2,686	79.12
3-1	287	175	212	674	19.85
2-2	11	7	6	24	0.71
2-1-1	5	4	1	10	0.29
1-1-1-1	0	0	1	1	0.03
Total	2,271	659	465	3,395	100.00

Appendix

Table 4: Human agreement and assessed support for QC decisions on the training set.

Setting	Training pool	Supervision and evolution
Base Model	—	No task-specific RL
GRPO	Full pool	Reference-based outcome rewards
GRPO (High-Conf.)	High-confidence subset	Reference-based outcome rewards
CA-GRPO	Full pool	Confidence-adaptive outcome/process mixture; no injected decision knowledge; fixed rubric
AdaptEvo	Full pool	Continued CA-GRPO after joint knowledge/rubric evolution

Appendix

Table 5: Training settings with a shared Qwen3.6-35B-A3B initialization. AdaptEvo continues from CA-GRPO after a joint knowledge and rubric update.

Method	Knowledge	In-Period		Out-of-Period
		ELA	BDA	ELA	BDA
Base model	None	53.9	66.1	53.3	65.9
GRPO	None	54.4	68.5	51.9	65.2
GRPO (High-Conf.)	None	53.3	68.8	52.9	65.5
Fixed (0.8:0.2)	None	54.3	67.6	53.6	66.1
Fixed (0.5:0.5)	None	54.2	67.9	54.0	66.2

Appendix

Table 6: Shared-backbone results on In-Period and Out-of-Period (%, higher is better). All configurations use Qwen3.6-35B-A3B. Fixed ratios specify outcome:process reward weights; Knowledge indicates injected decision knowledge. Bold marks the best score in each column.

	GRPO		GRPO (High-Conf.)		CA-GRPO
Training steps	ELA	BDA	ELA	BDA	ELA	BDA
Base	53.3	65.9	53.3	65.9	53.3	65.9
10	54.5	66.0	55.1	66.8	54.1	66.2
20	54.2	66.2	54.4	67.2	54.3	65.7
30	54.2	66.5	53.9	66.8	54.5	65.9
40	53.3	65.9	52.0	65.1	54.4	65.7

Appendix

Table 7: Out-of-Period checkpoint results without injected decision knowledge. All methods initialize from the same Qwen3.6-35B-A3B base model. Accuracies are percentages at the supplied one-decimal precision; bold marks the highest checkpoint result per metric.

	In-Period		In-Period + knowledge		Out-of-Period
Outcome : process	ELA	BDA	ELA	BDA	ELA	BDA
0.8:0.2	54.3	67.6	57.7	68.9	53.6	66.1
0.5:0.5	54.2	67.9	58.1	70.1	54.0	66.2
0.2:0.8	55.6	67.3	58.6	69.3	54.8	65.8
[0pt][0pt] CA-GRPO (adaptive)	55.4	68.1	59.1	71.3	55.2	67.3

Appendix

Table 8: Fixed and adaptive outcome/process reward mixing after 60 training steps. All values are percentages. Bold marks the best value in each column.

Internal alias	Main-table name	Knowledge
base	Qwen3.6-35B-A3B (base)	None
gtdirect-full	GRPO	None
gtdirect-support	GRPO (High-Conf.)	None
gtconf-v1 , library off	CA-GRPO	None
gtconf-v1 , library on	CA-GRPO	Evolved
gtconf-v2	AdaptEvo	Evolved

Appendix

Table 9: Mapping from internal run aliases to the compact main-table names and knowledge settings.

Explore similar work

EvoIn: Bridging Evolution and Internalization for Agent Fine-Tuning
Sep 28, 2026Shihan Dou, Shaofan Liu, Zhonghang Lu +8LLM Agent Training Self-Improving Agents
Fudan University · Renmin University of China
JET: Judge-Guided Evolution at Test Time for Agent Programs
Sep 28, 2026Yao Long Teng, Jiayi Cai, Bo AnTest-Time Adaptation Self-Evolving Agents
Nanyang Technological University
ARGUS: Policy-Adaptive Ad Governance via Evolving Reinforcement with Adversarial Umpiring
May 4, 2026Deyi Ji, Junyu Lu, Xuanyi Liu +7Non-Stationary RL Online Advertising
Tencent · Dalian University of Technology · Peking University +2

Sep 28, 2026, cs.AIJ/K move · Enter open · S save

EvoIn: Bridging Evolution and Internalization for Agent Fine-Tuning

Shihan Dou, Shaofan Liu, Zhonghang Lu, Jiahang Lin+7

Fudan University · Renmin University of China

LLM Agent Training Self-Improving Agents

Open paper AskPDF arXiv

Recent work has explored improving agents by jointly evolving their harnesses and models, but often takes a ''potpourri'' approach that bundles together new tools, new decision-making procedures, and model adaptation to the evolved harness under a single notion of agent improvement. In this paper, we instead investigate how agents can improve their decision-making procedures. In particular, we propose EvoIn, an agent fine-tuning framework that bridges evolution and internalization. EvoIn first analyzes agent execution traces to evolve and validate new decision-making procedures by temporarily instantiating them in the harness. The validated procedures guide the agent to generate improved reasoning traces. These traces are then rewritten into self-contained reasoning traces, removing explicit references to harness instructions while expressing the induced decision logic as the model's own reasoning. Finally, EvoIn fine-tunes the model on the rewritten traces, internalizing these procedures so that the improved decision-making persists without the evolved harness at inference time. We evaluate EvoIn on diverse benchmarks and find that it consistently enables agents to learn stronger decision-making procedures, raising the pass rate by 10.9 points in-domain and by 9.2 points out-of-domain. Results further show that the internalized decision procedures generalize to unseen tasks. Case studies show that agents can learn to decide how to solve a task before solving it, for example by checking a document's length to choose between reading it in full and searching it. EvoIn is also broadly applicable, showing consistent improvements on another model family.

Abstract

Explore similar work

EvoIn: Bridging Evolution and Internalization for Agent Fine-Tuning

JET: Judge-Guided Evolution at Test Time for Agent Programs

ARGUS: Policy-Adaptive Ad Governance via Evolving Reinforcement with Adversarial Umpiring