Federation over Harness

Dixi Yao; Tahseen Rabbani; Tian Li

cs.MAOct 15, 2026

Federation over Harness

Authors: Dixi Yao, Tahseen Rabbani, Tian Li

Organizations: University of Chicago

Abstract

Modern LLM-powered agents increasingly improve through experience sharing, where distributed agents distill local reasoning experience into reusable knowledge, such as a shared insight library. Existing approaches, however, still heavily rely on manually designed prompts and workflows that determine both how local agents extract useful reasoning traces and how a central/global agent aggregates them to obtain common patterns and insights. We propose Federation over Harness (FoH), a framework that automatically evolves the harness for both local and global components in this distributed multi-agent reasoning pipeline. A server agent evolves the client harness for task solving and self-reflection while, at the same time, evolving its own harness for aggregating distributed reflections, maintaining shared knowledge, and updating subsequent client harnesses. Importantly, FoH requires no extra validation signals, such as ground-truth labels or validation data, during evolution; it learns solely from self-generated reflections produced by distributed agents. Experiments on multiple benchmarks show that FoH consistently improves over single-agent baselines with a fixed harness and with harness self-evolution by 53.57% and 29.76%.

Figures & tables

Figure 1

Figure 2: Roadmap of representative changes across harness components over 15 iterations on PinchBench (daily tasks such as document summarization, research, coding, and blog writing). Client and server harnesses change at the prompt, tool, and insight levels by adding, deleting, or revising Markdown prompt files, the insight-library JSON, or Python scripts; e.g., at iteration 8 the server deletes the insight on compacting stale history and adds one on correlating items.

Figure 3: Harness components evolved and activated during execution on selected ScienceAgentBench tasks, using the harness after iteration 5. Isolated agents fail these tasks, whereas FoH-evolved client harnesses resolve them with the same LLM. Tasks exhibit distinct activation patterns, with agents selectively invoking client-harness components, and tasks from one domain or sub-task benefit from components originally evolved for others.

Figure 4: Breakdown of output-token usage on PinchBench. Top figure : FoH reduces client-side output tokens compared with isolated agents. Bottom two figures : on the server side, intermediate processing and reasoning used to update the harness take up almost all tokens, while the harness itself remains lightweight to communicate (containing so few tokens that it is almost invisible in the bottom figures).

Figure 5: Performance on PinchBench V1. With properly configured guides, FoH steadily improves and converges, but becomes unstable when the guide excludes the insight library or local reflection.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Context Source	Char. Similarity	Token F1	BLEU-4
Raw Trajectories	54.7%	60.2%	20.4
FoH Reflections	41.2%	33.6%	4.6
FoH Reflections + Client Harness Repo	45.7%	36.1%	9.9

Appendix

Table 3: Prompt stealing attack results. The attacker uses Fable 5.1 (High) with PinchBench V1 reasoning traces in FoH to reconstruct the original input. The low character-level similarity, low token-level F1, and low BLEU-4 scores indicate that the attack fails to recover a substantial part of the original input.

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 165 papers

Federation over Text: Insight Sharing for Multi-Agent Reasoning
Apr 18, 2026Dixi Yao, Tahseen Rabbani, Manzil Zaheer +1Multi-Agent LLM Systems Multi-Agent Reasoning
University of Chicago · Google DeepMind
EvolveNet: Collaborative Harness Evolution for Agent Self-Improvement
Aug 5, 2026Jun Nie, Yonggang Zhang, Qianshu Cai +3Agent Harness Optimization Multi-Agent Collaboration
Hong Kong Baptist University · University of Science and Technology of China · The Hong Kong University of Science and Technology
FedAgentKE: Federated Semantic Knowledge Evolution for Heterogeneous Agents
Jul 23, 2026Weihao Li, Jun Bai, Ziyang SongFederated Knowledge Distillation LLM Agent Training
McCormick School of Engineering, Northwestern University · School of Computer Science, McGill University · Mila – Quebec AI Institute +1

Apr 18, 2026J/K move · Enter open · S save

Federation over Text: Insight Sharing for Multi-Agent Reasoning

Dixi Yao, Tahseen Rabbani, Manzil Zaheer, Tian Li

University of Chicago · Google DeepMind

Multi-Agent LLM Systems Multi-Agent Reasoning Federated Learning

Open paper AskPDF arXiv

Modern agents specialize in varying domains while there is no clear approach combining different domain skills. We propose a federated learning-like framework, Federation over Text (FoT), that enables multiple clients solving different tasks to collectively generate a shared library of metacognitive insights by iteratively federating their local reasoning processes without sharing actual problem instances. Instead of federation over gradients (e.g., as in distributed training), FoT operates at the semantic level without any gradient optimization or supervision signal. At each round, client LLM agents independently apply arbitrary local reasoning and self-improvement procedures to their own tasks and share the resulting reasoning traces with a central server. The server then aggregates, distills, and consolidates knowledge across tasks and domains into a shared insight library, which can be reused by current and future agents to improve their reasoning. Experiments show that FoT improves reasoning effectiveness and efficiency across real-world daily tasks, cross-domain collaboration, and research insight discovery, achieving an average performance gain of 11.9 absolute percentage points while reducing client-generated task-completion tokens by 5.5%.

Abstract

Agent Harness Optimization

Explore similar work

Federation over Text: Insight Sharing for Multi-Agent Reasoning

EvolveNet: Collaborative Harness Evolution for Agent Self-Improvement

FedAgentKE: Federated Semantic Knowledge Evolution for Heterogeneous Agents