KARMA: Training-Free Knowledge-Atomic Harness Evolution for Industrial Advertising Agents

Xinhua Miao; Yuanzhen Rong; Weiru Zhang; Xiaoyi Zeng

cs.AIOct 29, 2026

KARMA: Training-Free Knowledge-Atomic Harness Evolution for Industrial Advertising Agents

Authors: Xinhua Miao, Yuanzhen Rong, Weiru Zhang, Xiaoyi Zeng

Organizations: Alibaba International Digital Commerce Group, Hangzhou, China

Abstract

Industrial advertising agents must continually adapt their decision knowledge to changing product offerings, traffic patterns, and operating policies. We present KARMA (Knowledge-Atomic, Regression-Gated Modular Adaptation), a training-free framework that evolves six typed knowledge modules while keeping model weights and executable harness code frozen. During our evolution, a deterministic Trace Verifier maps observed errors to candidate modules, and a Critic compares successful and failed trajectories to guide a tool-constrained Evolver. The Evolver applies atomic knowledge edits, which a fixed validation gate retains only when their measured gain reaches a threshold; otherwise, the previous knowledge state is restored. We evaluate KARMA with three LLM families on cases sampled from a corpus of approximately half a million real-world operation logs across six e-commerce domains. Across 18 domain--model settings, KARMA raises the average hold-out score from 30.19 to 39.15 (+8.96 points, +29.7%). Ablations show that removing either successful or failed trajectories more than halves the average hold-out gain. Evolved knowledge also transfers across domains without further adaptation, yielding an average hold-out gain of 8.54 points across all source--target pairs. By linking retained patches to their diagnostic evidence and validation outcomes, KARMA supports inspectable, reversible knowledge adaptation for agents in Web marketplaces.

Figures & tables

Figure 1. Motivation. Whole-prompt rewriting obscures individual changes; broad harness editing complicates validation and rollback. KARMA freezes weights and code, derives atomic knowledge edits from successes and failures, and retains only patches whose validation gain reaches $\theta$ . Three panels. The left panel shows a monolithic system prompt containing role, domain, and constraints, with the warning that opaque rewrites hinder attribution and risk regressions. The middle panel shows an agent harness containing code, tools, and orchestration, with the warning that broad code changes complicate validation and rollback. The right panel shows KARMA: six typed knowledge modules, atomic knowledge edits derived from failures and successes, and a validation step that accepts an update if gains are at least theta and otherwise rolls it back.

Figure 2. KARMA overview. The frozen inner agent collects evidence and emits structured decisions. The outer loop converts verified trajectories into atomic knowledge edits through the Critic and Evolver, then validates or rolls back each patch. The hold-out set is excluded from evolution. The top half shows advertising operation logs of verified merchants split into training batches, a hold-out set used for final performance, and a validation set. The inner decision agent renders six typed knowledge modules into the context and response schema, runs a two-stage loop of tool calling and structured output with LLMs, emits budget, bid, or item decisions, and scores them by action, direction, and total. The bottom half shows the outer evolution loop: a Trace Verifier clusters failures and successes with deterministic attribution; a Critic performs multi-perspective reflection over missing knowledge, conflicting rules, protocol timing, and ambiguous wording to produce edit proposals; an Evolver uses create, read, update, and delete tools to produce mutations; and a Patch Validator compares current and candidate knowledge on the validation set, accepting if the average score gain is at least theta and otherwise rejecting and rolling back.

Module	Responsibility
role	Agent identity, objective, and communication stance
domain	Advertising products and operating semantics
experience	Reusable decision heuristics with stable rule IDs
constraints	Evidence, comparison, and safety requirements
output_protocol	Analysis termination and structured submission rules
action_descriptions	Field-level schema and local-tool descriptions

Table 1. Typed knowledge modules and their intended responsibilities.

Domain	Symbol	Share (%)
Home & Living	$\mathcal{H}$	24.8
Automotive	$\mathcal{A}$	15.3
Electronics & Office	$\mathcal{E}$	14.7
Fashion & Apparel	$\mathcal{F}$	19.1
Beauty & Health	$\mathcal{B}$	8.2
Long-tail/Others	$\mathcal{L}$	17.9

Table 2. Corpus statistics and domain symbols. Each run uses 800 cases: 480 for training, 160 for validation, and 160 for hold-out evaluation.

LLMs	Domains	Validation set $V$				Hold-out set $H$
LLMs	Domains	Base	KARMA	Gain	Rel. (%)	Base	KARMA	Gain	Rel. (%)
Qwen3.8-Max	$\mathcal{H}$	33.13	50.63	+17.50	+52.8	32.81	48.75	+15.94	+48.6
	$\mathcal{A}$	39.06	41.25	+2.19	+5.6	35.63	36.25	+0.63	+1.8
	$\mathcal{E}$	34.38	48.75	+14.38	+41.8	33.13	43.44	+10.31	+31.1
	$\mathcal{F}$	34.69	43.75	+9.06	+26.1	26.88	38.44	+11.56	+43.0
	$\mathcal{B}$	35.31	50.00	+14.69	+41.6	29.06	47.19	+18.13	+62.4

Table 3. Main results on 160 cases per set (scores

\times 100

). Base:

K_{0}

; KARMA:

K_{T}

; Gain: absolute points; Rel.: percentage gain over Base. Bold marks the higher score. Avg. averages six domains before rounding; symbols follow Table 2 .

Domains	Variants	Validation set $V$			Hold-out set $H$
Domains	Variants	Score	Gain	Rel. (%)	Score	Gain	Rel. (%)
$\mathcal{H}$	Base	33.13	–	–	32.81	–	–
	w/o success	39.69	+6.56	+19.8	43.75	+10.94	+33.3
	w/o failure	38.44	+5.31	+16.0	35.63	+2.81	+8.6
	KARMA	50.63	+17.50	+52.8	48.75	+15.94	+48.6
$\mathcal{B}$	Base	35.31	–	–	29.06	–	–

Table 4. Evidence ablation with Qwen3.8-Max. Bold marks the best score per domain and set.

Method	Validation set $V$		Hold-out set $H$		Cost
Method	$\mathcal{H}$	$\mathcal{B}$	$\mathcal{H}$	$\mathcal{B}$	Tokens	Hours
GEPA	+11.25	+1.56	+9.06	$-$ 1.88	42.15M	4.58
AHE	+11.56	+5.94	+8.75	+6.56	40.74M	3.40
Self-Harness	+3.75	+3.13	$-$ 2.50	+2.50	40.72M	3.60
KARMA	+17.50	+14.69	+15.94	+18.13	41.48M	4.53

Table 5. Evolution engines with Qwen3.8-Max. Tokens: mean total LLM tokens per cycle, including agent inference. Hours: runtime per cycle.

Source	Target domain
Source	$\mathcal{H}$	$\mathcal{A}$	$\mathcal{E}$	$\mathcal{F}$	$\mathcal{B}$	$\mathcal{L}$
$\mathcal{H}$	–	$-$ 0.63	+7.81	+17.50	+12.19	+19.38
$\mathcal{A}$	$-$ 5.00	–	$-$ 4.06	$-$ 5.94	$-$ 1.88	$-$ 5.94
$\mathcal{E}$	+10.63	+9.38	–	+13.44	+15.31	+10.00
$\mathcal{F}$	+5.94	$-$ 3.44	+8.75	–	+10.00	+13.44
$\mathcal{B}$	+10.63	+7.19	+6.88	+11.88	–	+15.94

Table 6. Cross-domain hold-out gains with Qwen3.8-Max, relative to each target’s main-run Base. Bold marks gains exceeding the target’s in-domain gain.

Figure 3. Hold-out score changes on $\mathcal{A}$ by reference operation (ADD/DEL: add/delete offer; BUD/PRI: update budget/price). Bars sum to each model’s gain in Table 3 , up to rounding.

Figure 4. Cached validation scores during evolution (scores $\times 100$ ; Init: $K_{0}$ ). Filled markers denote accepted patches; hollow markers denote rejection, which preserves the active knowledge and cached score.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Observed error	Candidate modules
Missing decision	Output protocol, role
Wrong action type	Domain, experience
Wrong direction	Experience, constraints

Appendix

Table 7. Default routing of semantic failures. Protocol codes can add candidate modules to these assignments.

Seed	Validation set $V$			Hold-out set $H$
Seed	Base	KARMA	Gain	Base	KARMA	Gain
42 †	34.38	48.75	+14.38	33.13	43.44	+10.31
43	35.31	44.38	+9.06	32.19	43.44	+11.25
44	30.63	35.31	+4.69	31.56	32.50	+0.94
45	30.00	38.44	+8.44	29.38	32.81	+3.44
46	30.63	40.94	+10.31	35.00	39.38	+4.38

Appendix

Table 8. Data-seed variation for Qwen3.8-Max on

\mathcal{E}

(scores

\times 100

). Seed 42 † is the main run in Table 3 . Means are computed before rounding.

Domain	Validation set $V$			Hold-out set $H$
Domain	0.005	0.01	0.02	0.005	0.01	0.02
$\mathcal{H}$	+17.50	+17.50	+17.50	+15.94	+15.94	+15.94
$\mathcal{B}$	+14.69	+14.69	+14.69	+18.13	+18.13	+18.13
$\mathcal{L}$	+13.13	+13.13	+10.00	+15.94	+15.94	+12.50

Appendix

Table 9. Gate-threshold sensitivity with Qwen3.8-Max.

Figure 5. Contrasting evolution cycles on $\mathcal{E}$ : (a) Qwen3.8-Max’s largest accepted validation gain (+11.88) and (b) Kimi-K3’s largest rejected loss ( $-$ 6.88). Each panel follows diagnosis, proposals, mutations, validation against the cached baseline, and hold-out behavior. Final hold-out gains reflect all retained edits. Aggregate scores use the 0–100 scale; sample scores use 0–1; $\theta=1$ point. Rule texts translate the Chinese originals; FM/SM denote batch failure/success clusters. ADD/DEL/BUD/PRI denote the four reference operations.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 711 papers

Agentic Knowledge-Model Co-Evolution via Consistency Regularization
Oct 22, 2026Sirui Chen, Zhining Liu, Chao Zhang +2LLM Agent Training Continual Learning for LLM Agents
University of Illinois Urbana-Champaign · Georgia Institute of Technology · Amazon
AutoAdapt: Automatic Domain Discovery Enables Low-Cost Extensibility
Oct 7, 2026Josh McGiff, Salma Mekaoui, Robert Shanahan +1Domain Adaptation for LLMs
University of Limerick, Ireland
Holding a Steady Course: Knowledge Evolution from Expertise and Lessons for Continual Agent Learning
Oct 25, 2026Wenzhe Fan, Benjamin Steenhoek, Gaurav Mittal +1Lifelong Learning Agents Agent Skill Learning
University of Illinois Chicago, IL, USA · Microsoft, Redmond, WA, USA

Oct 22, 2026J/K move · Enter open · S save

Agentic Knowledge-Model Co-Evolution via Consistency Regularization

Sirui Chen, Zhining Liu, Chao Zhang, Jingrui He+1

University of Illinois Urbana-Champaign · Georgia Institute of Technology · Amazon

LLM Agent Training Continual Learning for LLM Agents

Open paper AskPDF arXiv

LLM agents can evolve by learning from accumulated experience. Existing approaches extract reusable external knowledge, such as skills and memories, from experience to guide subsequent task solving, with some further internalizing this knowledge through model training. However, different ways of analyzing and transforming the same experience can yield knowledge with substantially different utility, even when the resulting guidance appears reasonable. Inspired by consistency regularization, we identify consistent utility gains across different ways of analyzing the same experience, rather than relying on any single way of analyzing it. We introduce SPLICE, a framework for consistency-regularized knowledge-model co-evolution. It constructs evolution views by perturbing how the same experience is analyzed, producing different knowledge candidates. Knowledge updates are then guided by common gain, which accounts for utility gains and their consistency across views. During model evolution, self-distillation internalizes the evolved knowledge alongside reinforcement learning, using knowledge associated with consistent utility degradation as a contrastive reference. The two processes interleave, with model updates driving further knowledge evolution. Experiments across diverse domains show that SPLICE outperforms knowledge evolution and model training baselines by up to 7.1 and 4.6 points in average accuracy, while its evolved knowledge reduces output tokens per correct solution by up to 44% and transfers across model sizes and families.

Abstract

LLM Agents

Explore similar work

Agentic Knowledge-Model Co-Evolution via Consistency Regularization

AutoAdapt: Automatic Domain Discovery Enables Low-Cost Extensibility

Holding a Steady Course: Knowledge Evolution from Expertise and Lessons for Continual Agent Learning