cs.MAMar 20, 2026

Herding CATs: ALARA for Agent Harness Engineering in Portable Composable Multi-Agent Teams

Authors: Christopher J. Agostino, Nayan D'Souza

Organizations: NPC Worldwide, Bloomington, IN, USA · Celeria, Inc., Bloomington, IN, USA · Department of Linguistics, Indiana University, Bloomington, IN, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Industry practitioners and academic researchers regularly use multi-agent systems to accelerate their work, but the applications through which users operate these systems do not provide a simple, unified mechanism for scalably managing critical components of the agent harness. This lack of control adversely impacts both the quality of individual human-agent interactions and reduces the capacity for practitioners to coordinate context engineering efforts. The behavioral specifications that define what agents in such systems can do remain fragmented across prose instruction files -- for which compliance cannot be guaranteed -- or framework-internal configurations, making these specifications difficult to share, version, or collaboratively maintain across teams and projects. Applying the ALARA principle from radiation safety (exposures kept as low as reasonably achievable) to context, we introduce a context-agent-tool (CAT) data layer expressed through interrelated plain-text files, allowing users to directly declare tool access for each agent and to modify the tools themselves that are used by the agents when processing. We demonstrate capability of this CAT data layer to enable real agentic usage by using a command-line shell that loads the team and executes agent runs -- npcsh -- and evaluating 22 locally-hosted models from 0.6B to 35B parameters across 115 practical tasks spanning file operations, web search, multi-step scripting, tool chaining, and multi-agent delegation. We characterize which model families succeed in certain task categories and where they break down across ∼\sim2500 total executions.

Figures & tables

Fig. 1: The CAT data layer. Left: context files (orange) scope teams of NPCs (blue). Right: Jinxes (green) compose as a DAG rooted at python . Level-1 Jinxes ( chat , sh , web_search , screenshot ) each use python as their engine. Level-2 Jinxes combine level-1 steps: react chains chat + python , computer_use chains chat + screenshot + sh , and delegate uses chat + sh . Shared parents (e.g. chat used by all three composites) make the graph a DAG rather than a tree. Dashed arrows show tool-catalog assignment to NPCs.
FamilyModelParamsScore
Qwen3.50.8b0.8B12/115 (10%)
 2b2B72/115 (63%)
 4b4B67/115 (58%)
 9b9B90/115 (78%)
 35b35B101/115 (88%)
Qwen30.6b0.6B5/115 (4%)
TABLE I: Benchmark results across model families.
Fig. 2: Benchmark score versus parameter count across model families.
Fig. 3: Agency (npcsh score) versus intelligence (MMLU). The regression line ( r≈0.8r\approx 0.8 ) with 1σ1\sigma band; outliers beyond 1σ1\sigma are labeled.
Fig. 4: Mean tool calls per task versus agency score ( r≈0.7r\approx 0.7 ).
Fig. 5: First-attempt pass rate (green) and additional gain from retries (orange) by task category.
Fig. 6: Pass rate by model and task category. Models ordered by overall score (left to right); categories by mean difficulty (top to bottom).
More in this topic

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 381 papers

Explore similar work

CardsList
Oct 14, 2026

Same Context, Different Agents: A Behavioral Study of Semantic Layers, Multi-Agent Orchestration and Memory Compaction in Data Agents

Answering questions over heterogeneous data means combining evidence from files, databases, documents and the web. Data agents automate this, but their success depends on context. Three context-engineering techniques (CETs) help by shaping it: semantic layers describe the data, orchestration splits work among agents, and compaction condenses interaction history. Existing studies validate each CET alone, leaving open how agent behavior shapes its effect. We present a comprehensive analysis of the three CETs, alone and combined, with seven large language models (LLMs) on 1,514 long-horizon tasks from five benchmarks, covering behavior, accuracy and cost. We find that data agents without CETs form three behavior groups: investigators, premature answerers that answer without inspecting data, and early stoppers that give up. Each CET delivers its known benefit, from more investigation to smaller prompts, but only the behavior group it matches turns that benefit into accuracy, such as premature answerers under orchestration. For investigators, the same CETs can hurt: the semantic layer stops them from looking, orchestration makes them overconfident, costing 5 percentage points, and compaction makes them repeat steps. Combining CETs yields smaller gains and losses than their sum. We derive guidance for choosing CETs by agent behavior, task demands and accuracy-cost trade-offs.

Zhivar Sourati, Marianne Menglin Liu, Zheng Wang +6

University of Southern California · Oracle AI · University of Pennsylvania

Context Compaction for LLM AgentsLLM AgentsData Analysis AgentsLLM Agent OrchestrationLong-Horizon Agent Tasks
AskPDF
Oct 21, 2026

Credit Where Credit Is Due: Contrastive Attribution Topology Search for Multi-Agent Systems

An LLM multi-agent system (MAS) is only as good as the agents it chooses to trust: its accuracy and cost hinge on its topology, meaning which agents are included and how they communicate. A single task score, however, does not reveal which agents help or hinder performance. We propose Contrastive Attribution Topology Search (CATS), which searches for compact teams within a fixed supergraph of specialized agents. An LLM attributer compares execution traces across candidate topologies and assigns credit to individual agents. CATS uses these credits to update supergraph edges while progressively pruning weak edges and agents, yielding a small team for deployment. With small open-weight models and matched candidate evaluations, CATS improves mean accuracy on BigCodeBench by 5.3 percentage points over score-only heuristic search while using 54%\% fewer deployment tokens. On MBPP, CATS matches the strongest baselines with fewer tokens than the other searches. In a HotpotQA prompt-poisoning stress test, CATS excludes all adversarially prompted agents in all three runs, whereas random search retains some at comparable accuracy. Per-agent credit thus supports both team optimization and the pruning of potentially harmful agents.

Fabian Tritthart, Tom Zehle

University of Freiburg

Multi-Agent LLM SystemsMulti-Agent SystemsLLM Agent Efficiency
AskPDF
Apr 1, 2026

OrgAgent: Organize Your Multi-Agent System like a Company

While large language model-based multi-agent systems have shown strong potential for complex reasoning, how to effectively organize multiple agents remains an open question. In this paper, we introduce OrgAgent, a company-style hierarchical multi-agent framework that separates collaboration into governance, execution, and compliance layers. OrgAgent decomposes multi-agent reasoning into three layers: a governance layer for planning and resource allocation, an execution layer for task solving and review, and a compliance layer for final answer control. By evaluating the framework across reasoning tasks, LLMs, execution modes, and execution policies, we find that multi-agent systems organized in a company-style hierarchy generally outperform other organizational structures. Besides, hierarchical coordination also reduces token consumption relative to flat collaboration in most settings. For example, for GPT-OSS-120B, the hierarchical setting improves performance over flat multi-agent system by 102.73% while reducing token usage by 74.52% on SQuAD 2.0. Further analysis shows that hierarchy helps most when tasks benefit from stable skill assignment, controlled information flow, and layered verification. Overall, our findings highlight organizational structure as an important factor in multi-agent reasoning, shaping not only effectiveness and cost, but also coordination behavior.

Yiru Wang, Xinyue Shen, Yaohui Han +3

The Chinese University of Hong Kong · CISPA Helmholtz Center for Information Security · IBM Research

Multi-Agent SystemsMulti-Agent OrchestrationMulti-Agent CollaborationMulti-Agent Reasoning
AskPDF