cs.AIApr 13, 2026

Three Roles, One Model: Role Orchestration at Inference Time to Close the Performance Gap Between Small and Large Agents

Authors: S. Aaron McClendon, Jorge Gallego-Feliciano, Stavros Zervoudakis, Antonios Saravanos

Organizations: Aimpoint Digital Labs, Atlanta, GA, USA · New York University, New York, NY, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language model (LLM) agents show promise on realistic tool-use tasks, but deploying capable agents on modest hardware remains challenging. We study whether inference-time scaffolding alone, without any additional training compute, can improve the performance of a small model in complex multi-step environments. Operating on a single 24GB GPU, we evaluate Qwen3-8B on the AppWorld benchmark under both full-precision and 4-bit quantized configurations. Without any intervention, the raw model achieves just 5.4% (FP16) and 3.0% (AWQ) task goal completion. Guided by a systematic failure mode analysis, we introduce a three-tier inference scaffolding pipeline that deploys the same frozen model in three distinct roles: (1) a summarization model that preserves critical artifacts (tokens, credentials, API responses) while compressing dialogue history; (2) the main agent model that reasons over the compressed context; and (3) an isolated correction model that reviews and revises the agent's code output without access to conversation history, breaking repetitive failure loops. Applied to the same unmodified model, this scaffolding yields 8.9% (FP16) and 5.9% (AWQ) task goal completion, roughly doubling performance in both settings, with particularly strong gains on difficulty-1 tasks (15.8% to 26.3% FP16; 5.3% to 14.0% AWQ). On full-precision inference, our scaffolded 8B model surpasses DeepSeek-Coder 33B Instruct (7.1%) from the original AppWorld evaluation, demonstrating that structured inference-time interventions can make small models competitive with systems 4 times their size. We formalize the approach as a scaffolded policy over a frozen base model, three invocations of the same weights with different conditioning, drawing connections to test-time compute scaling and action-space shaping in reinforcement learning.

Figures & tables

Figure 1: Our proposed modular architecture. Each module targets an important failure mode for small language models to increase performance.
Failure CategoryCountConf.-Weighted
Authentication / credential issue4639.45
Reasoning / planning error4335.30
Wrong API params / schema mismatch2925.05
Other2420.05
Missing API call / wrong API name1311.45
Repetition / loop32.55
Table 1: Primary failure mode distribution for FP16 baseline on AppWorld test_normal (163 failed tasks). Categories are ranked by frequency. Confidence-weighted scores incorporate the evaluator’s classification confidence.
DifficultyFailure CategoryCount
1Auth / credentials13
Reasoning / planning11
Other11
2Reasoning / planning13
API params / schema12
Auth / credentials11
Table 2: Top-3 primary failure modes by difficulty level (FP16 baseline).
Figure 2: Comparison of failure modes for tasks of difficulty 1 for baseline vs full scaffold for Qwen 3 8B AWQ, which is a quantized version of Qwen.
 FP16 (12K ctx)AWQ (32K ctx)
DifficultyBaselineScaffoldBaselineScaffold
Aggregate5.4 [2.8, 9.9]8.9 [5.5, 14.2]3.0 [1.3, 6.8]5.9 [3.3, 10.6]
Difficulty 115.826.35.314.0
Difficulty 20.00.02.14.2
Difficulty 30.00.01.60.0
Table 3: Task goal completion (%) on AppWorld test_normal (168 tasks). FP16 = full precision with 12K context; AWQ = 4-bit quantized with 32K context. “Scaffold” denotes our three-tier inference scaffolding method. We report 95% Wilson confidence intervals for aggregate task goal completion rates.
ModelParameterstask goal completion (%)95% CI
GPT-4o—48.8—
Claude 3.5 Sonnet—33.2—
LLaMA-3-70B-Instruct70B20.8—
Qwen3-8B + Scaffold (FP16)8B8.9[5.5, 14.2]
DeepSeek-Coder 33B Instruct33B7.1[4.1, 12.0]
Qwen3-8B Baseline (FP16)8B5.4[2.8, 9.9]
Table 4: Comparison with published AppWorld results (task goal completion, test_normal ). Our results use Qwen3-8B on a single 24 GB GPU. For our models, and for DeepSeek-Coder 33B Instruct assuming the same 168-task evaluation set, we report 95% Wilson confidence intervals.
DifficultyBaselineCorrection OnlyFull Scaffold
Aggregate3.04.85.9
Difficulty 15.312.314.0
Difficulty 22.12.14.2
Difficulty 31.60.00.0
Table 5: Ablation study on AWQ configuration (32K context). “Correction Only” uses the isolated correction node without summarization.
 Baseline (AWQ)Scaffolded (AWQ)
Failure CategoryCount%Count%
Auth / credentials4628.24327.2
Reasoning / planning4326.45434.2
API params / schema2917.8159.5
Other2414.71912.0
Missing API / wrong name138.01710.8
Table 6: Failure mode comparison before and after scaffolding (AWQ, test_normal ). Counts reflect primary failure classification across all difficulty levels. The scaffolded model has fewer total failures (158 vs. 163) due to improved task goal completion.
DifficultyFailure CategoryCount
1Reasoning / planning18
Auth / credentials8
API params / schema8
2Auth / credentials15
Reasoning / planning14
Missing API / wrong name8
Table 7: Top failure modes by difficulty after full scaffolding (AWQ, 158 failed tasks).
More in this topic

Tool-Using Language Agents

Tool-using language agents combine language models with APIs, code, and other tools to answer questions and carry out multi-step tasks.

See all 59 papers

Explore similar work

CardsList
May 1, 2026

AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?

Production agentic systems make many model calls per user request, and most of those calls are short, structured, and routine. This raises a practical routing question that existing evaluations do not directly answer: which parts of an agent workflow truly require large frontier intelligence, and which can be handled by smaller models? We introduce AgentFloor, a deterministic 30-task benchmark organized as a six-tier capability ladder, spanning instruction following, tool use, multi-step coordination, and long-horizon planning under persistent constraints. We evaluate 16 open-weight models, from 0.27B to 32B parameters, alongside GPT-5 across 16,542 scored runs. Our results reveal a clear boundary of model necessity. Small and mid-sized open-weight models are already sufficient for much of the short-horizon, structured tool use work that dominates real agent pipelines, and in aggregate, the strongest open-weight model matches GPT-5 on our benchmark while being substantially cheaper and faster to run. The gap appears most clearly on long-horizon planning tasks that require sustained coordination and reliable constraint tracking over many steps, where frontier models still hold an advantage, though neither side reaches strong reliability. We also find that this boundary is not explained by scale alone: some failures respond to targeted interventions, but the effects are model-specific rather than universal. These findings suggest a practical design principle for agentic systems: use smaller open-weight models for the broad base of routine actions, and reserve large frontier models for the narrower class of tasks that truly demand deeper planning and control. We release the benchmark, harness, sweep configurations, and full run corpus.

Ranit Karmakar, Jayita Chatterjee

Harvard University, Boston, MA 02115 · Boston, MA 02115

Computer-Use Agent BenchmarksLLM Agent EvaluationLong-Horizon Agent EvaluationAI Agent BenchmarksTool-Using Language AgentsLLM Agent Planning
AskPDF
May 18, 2026

Internalizing Tool Knowledge in Small Language Models via QLoRA Fine-Tuning

Large language models are increasingly used as planning components in agentic systems, but current tool-use pipelines often require full tool schemas to be included in every prompt, creating substantial token overhead and limiting the practicality of smaller models. This paper investigates whether tool-use knowledge can be internalized into small language models through parameter-efficient fine-tuning, enabling structured planning without explicit tool descriptions at inference time. Using AssetOpsBench as the primary benchmark, we fine-tune Gemma 4 E4B and Qwen3-4B with 8-bit QLoRA on approximately 1,700 tool-use examples spanning tool knowledge, question-to-plan mappings, and execution-style traces. We evaluate the resulting models under description-free inference, where the prompt omits the tool catalog entirely. The fine-tuned models outperform an informed unfine-tuned baseline that receives full tool descriptions, reducing input length by 82.6% while improving structural and LLM-judge planning scores. In the best Gemma run, the model achieves an AT-F1 of 0.65 and an overall judge score of 3.88, compared with 0.47 and 2.88 for the informed baseline. Qwen3-4B achieves a strong overall judge score of 3.78 while using 62% less memory and running 2.5×\times faster than Gemma, though it also exhibits greater catastrophic forgetting on general multiple-choice benchmarks. Additional ablations show that LoRA rank controls a quality--retention trade-off, with r=32r=32 maximizing planning quality and smaller ranks preserving more general knowledge. These results suggest that, for fixed tool catalogs, QLoRA fine-tuning can shift tool knowledge from prompt context into model weights, substantially reducing inference overhead while maintaining or improving tool-planning quality.

Yuval Shemla, Ayal Yakobe, Tanmay Agarwal +2

Columbia School of General Studies, Columbia University, NY, USA · Columbia Engineering, Columbia University, NY, USA · IBM Research, NY, USA

Small Language ModelsLLM Tool UseTool-Using Language Agents
AskPDF
May 19, 2026

Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs

LLM agents have recently emerged as a powerful paradigm for solving complex tasks through planning, tool use, memory retrieval, and multi-step interaction. However, these agentic workflows often introduce substantial input-side overhead, making the compute-intensive prefilling stage a key bottleneck in long-context, multi-turn inference. In this work, we propose Mix-Quant, a simple and effective phase-aware quantization framework for fast agentic inference. We first investigate FP4 quantization in agentic LLM workflows and observe that quantizing the entire inference process can incur significant performance degradation. In contrast, the prefilling stage exhibits substantial quantization redundancy and can therefore be quantized with minimal accuracy loss, despite being the dominant source of computation. Based on this insight, we apply high-throughput NVFP4 quantization to the prefilling phase while preserving BF16 precision for decoding. By decoupling prefilling acceleration from decoding quality, Mix-Quant combines phase-aware algorithmic quantization with hardware-efficient NVFP4 execution to alleviate the inference bottleneck in LLM agents. Extensive experiments across long-context and agentic benchmarks demonstrate that Mix-Quant largely preserves task performance while delivering significant efficiency improvements, achieving up to a 3x speedup during prefilling.

Haiquan Lu, Zigeng Chen, Gongfan Fang +2

National University of Singapore

LLM QuantizationLong-Context Language Model InferenceLLM Inference Acceleration
AskPDF