Measuring the Microtask Eligibility Gap: When Is an Off-the-Shelf SLM Enough for an Agent Harness?

Jundong Hu; Shekar Ramachandran

cs.AISep 1, 2026

Measuring the Microtask Eligibility Gap: When Is an Off-the-Shelf SLM Enough for an Agent Harness?

Authors: Jundong Hu, Shekar Ramachandran

Organizations: PayPal AI

Abstract

Agent harnesses increasingly want to run small language models (SLMs) on the microtasks around a frontier large language model (LLM) planner: auto-approving shell commands, writing memory, selecting tools, ranking past turns. We ask whether off-the-shelf SLMs meet practitioner-defined thresholds and, when they fail, why, and whether quantization changes the answer. We build a benchmark of 4 such microtasks with fixed prompts and automatic metrics, each with a pre-specified threshold $τ$ anchored to a cheap non-LLM baseline and a CI-aware eligibility rule (a configuration passes only if its confidence bound clears $τ$ ). Sweeping Qwen3 0.6/1.7/4/8B at their best (FP16, greedy, one frozen prompt, no tuning), we find an eligibility gap: 0 of 16 (4 tasks $\times$ 4 models) configurations pass (verified by checking the raw outputs and parser behavior). A logprob decision-threshold diagnostic (T1/T3/T4; T2 via a context-length/cascade probe) separates the failures into capability deficits and failures that can be addressed by changing the decoding threshold (4 regimes). Quantization to 4-bit (RTN/GPTQ/AWQ) does damage that depends on model size and moves no configuration into eligibility (certified on the reconstructable hard-label tasks T1/T3, diagnostic/windowed robustness on T2/T4), so the gap tracks model size more than precision; it replicates on Llama-3.x (12/12 ineligible) and is robust to the anchor choice (a $τ$ -sweep) and to prompt wording (0/112 eligible across the original plus 3 neutral paraphrases per cell). The practical implication: place SLMs behind a baseline that meets the CI-backed threshold, and use the SLM only where the baseline fails to meet the threshold; e.g. a 4B re-ranker over a BM25 shortlist beats BM25 ( $+0.047$ [0.020, 0.073], without itself certifying eligibility).

Figures & tables

Task	Decision	Primary metric	$\tau$ (frozen) / anchor
T1	safe shell auto-approve	false-approval $\textrm{FA}\!\downarrow$ & coverage $\textrm{cov}\!\uparrow$	$\textrm{FA}\leq 0.167$ & $\textrm{cov}\geq 0.921$ / deliberation+regex
T2	memory relevance recall	Recall@5	$\geq 0.479$ / BM25 Recall@5
T3	memory write / no-write	write-F1	$\geq 0.656$ / write-heuristic
T4	tool selection	correct-tool acc (callable)	$\geq 0.913$ / TF-IDF shortlist top-1

Table 1: Benchmark setup. One prompt per task;

\tau

anchored to a non-LLM baseline. Test sizes: T1 301 / T2 984 / T3 3032 / T4 600.

	T1 $\textrm{FA}\!\downarrow$ & $\textrm{cov}\!\uparrow$	T2 R@5	T3 wF1	T4 tool
Qwen3-0.6B	0.012, 0.029	0.041	0.557	0.900
Qwen3-1.7B	0.988, 1.000	0.125	0.579	0.758
Qwen3-4B	0.414, 0.971	0.350	0.598	0.707
Qwen3-8B	0.463, 0.993	0.292	0.634	0.784
$\tau$	$\leq$ 0.167 & $\geq$ 0.921	$\geq$ 0.479	$\geq$ 0.656	$\geq$ 0.913

Table 2: FP16 eligibility sweep (point estimates). All 16 configurations are ineligible: no CI bound clears

\tau

(bounds in text). T1 requires both

\textrm{FA}\!\leq\!0.167

and

\textrm{cov}\!\geq\!0.921

Panel of Figure 1 — Figure 1: Decision-threshold diagnostic (FP16). T1: only the 8B ROC enters the eligibility region. T3: all operating points, including the oracle, sit below the $\text{F1}=0.656$ iso-contour. T4: abstention is separable (AUROC 0.94–0.98) at every scale, a decoding artifact rather than missing capability.

Size	full-index R@5	rerank R@5	$\Delta$ vs BM25	95% CI	verdict
0.6B	0.041	0.311	$-0.129$	$\,[-0.160,-0.099]$	worse
1.7B	0.125	0.443	$+0.004$	$\,[-0.023,+0.031]$	ties
4B	0.350	0.486	$\mathbf{+0.047}$	$\mathbf{\,[+0.020,+0.073]}$	beats BM25
8B	0.292	0.393	$-0.047$	$\,[-0.077,-0.016]$	worse

Table 3: T2 BM25

\rightarrow

SLM re-rank (paired 10k bootstrap,

n\!=\!984

; BM25 R@20 ceiling 0.586).

\Delta

is rerank R@5

-

BM25 R@5 on the same items.

Figure 2: Quantization robustness depends on model size. Diagnostic AUROC (T1/T3/T4) and Recall@5 (T2) vs. model size; lines are fp/rtn/gptq/awq. The 4-bit lines diverge from FP16 at 0.6B and converge onto it by 8B; the T2 BM25 line (dashed) is crossed at

\geq

4B (all 4 methods at 4B; all but RTN at 8B).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	T1 (danger)	T3 (write)	T4 (abstain)
Qwen3-0.6B	$.548\,[.479,.614]$	$.650\,[.630,.670]$	$.947\,[.925,.967]$
Qwen3-1.7B	$.783\,[.728,.836]$	$.717\,[.698,.736]$	$.942\,[.912,.969]$
Qwen3-4B	$.895\,[.859,.927]$	$.750\,[.732,.767]$	$.980\,[.965,.991]$
Qwen3-8B	$.959\,[.934,.979]$	$.771\,[.754,.788]$	$.973\,[.954,.989]$
Llama-3.2-1B	$.557\,[.490,.622]$	$.523\,[.501,.544]$	$.462\,[.408,.516]$
Llama-3.2-3B	$.802\,[.752,.849]$	$.684\,[.665,.703]$	$.950\,[.924,.971]$

Appendix

Table 4: Diagnostic-layer AUROC with 95% bootstrap CI (FP16 test, 10k resamples). Point estimates match the ledger; CIs quantify the four-regime spectrum (§ 5 ).

n

: T1 301, T3 3032, T4 600.

Task	Size	RTN	GPTQ	AWQ
T1 ( $\textrm{FA}/\textrm{cov}$ )	0.6B †	0.99/0.97	1.00/1.00	0.23/0.40
	1.7B	0.00/0.00	0.60/0.75	0.10/0.57
	4B	0.40/0.88	0.31/0.88	0.89/0.99
	8B	0.59/0.99	0.70/0.99	0.60/0.99
T3 (write-F1)	0.6B †	0.000	0.544	0.254
	1.7B	0.576	0.532	0.568

Appendix

Table 5: Hard-label quant ledger (greedy decision reconstructed from the decision-logprob margin sign, validated against FP for

\geq

1.7B). T1 shows

\textrm{FA}/\textrm{cov}

; T3 shows write-F1. No cell is eligible (T1 needs

\textrm{FA}\!\leq\!0.167\wedge\textrm{cov}\!\geq\!0.921

; T3 needs F1

\geq

0.656). 0.6B reconstruction is unfaithful (marked † ) and covered by FP-domination.

Task	Frozen eligibility metric	Reconstructable from quant dumps?	Evidence (certified / surrogate)
T1	$\textrm{FA}/\textrm{cov}$ (hard label)	$\geq$ 1.7B yes; 0.6B no	Hard-label ledger $+$ AUROC ( $\geq$ 1.7B cert.); 0.6B domination
T3	write-F1 (hard label)	$\geq$ 1.7B yes; 0.6B no	Hard-label ledger $+$ AUROC ( $\geq$ 1.7B cert.); 0.6B domination
T4	correct-tool	No (margins encode call-vs-abstain only)	Abstention AUROC close to FP16 $\geq$ 4B (surrogate)
T2	recall@5 (full index)	No (quant dumps are windowed curves)	Windowed crossover survives $\geq$ 4B (surrogate)

Appendix

Table 6: Evidence for the no-rescue verdict per task. Hard-label reconstruction is validated against FP for

\geq

1.7B (Table 5 ); T2/T4 and 0.6B carry surrogate evidence that does not re-evaluate the frozen metric.

Task	0.6B	1.7B	4B	8B
T1 ECE	0.318	0.531	0.236	0.256
T3 ECE	0.277	0.378	0.407	0.250
T4 ECE	0.057	0.069	0.046	0.046

Appendix

Table 7: ECE of the decision logprobs (FP16, test). Mean-confidence far exceeds accuracy on T1/T3.

Figure 3: Reliability of decision logprobs (FP16): accuracy vs. mean confidence per bin (dotted

=

perfect calibration). T1/T3 sit well below the diagonal (overconfident); T4 abstention is near-calibrated.

Figure 4: T2 recall vs. context length per model size; lines are fp/rtn/gptq/awq plus BM25 and the

5/N

floor. The BM25 crossover survives quantization only at

\geq

4B.

Model	T1 $\textrm{FA}/\textrm{cov}$	T2 R@5-lb	T3 F1-lb	T4 tool-lb	elig
Llama-3.2-1B	$0.00/0.00$	0.001 ( $-0.478$ )	0.520 ( $-0.136$ )	0.709 ( $-0.204$ )	0/4
Llama-3.2-3B	$0.93/1.00$	0.134 ( $-0.345$ )	0.566 ( $-0.090$ )	0.590 ( $-0.323$ )	0/4
Llama-3.1-8B	$0.67/1.00$	0.270 ( $-0.210$ )	0.541 ( $-0.116$ )	0.684 ( $-0.229$ )	0/4
$\tau$	$\textrm{FA}{\leq}.167\wedge\textrm{cov}{\geq}.921$	$\geq$ 0.479	$\geq$ 0.656	$\geq$ 0.913

Appendix

Table 8: Cross-family FP16 eligibility (CI bound used for the verdict; margin in parens). All ineligible.

Table 13

Panel of Figure 5 — Figure 5: Cross-family decision-threshold diagnostic (Llama-3.x, FP16), same layout as Fig. 1 : the same capability $\rightarrow$ calibration signature appears (T1 8B enters the eligibility region; T3 stays below the F1 iso-contour; T4 abstention is highly separable at $\geq$ 3B, AUROC 0.95/0.92, sub-chance 0.462 at 1B).

Table 15

Figure 6:

\tau

-sensitivity: number of eligible configs (CI-LB

\geq\tau^{\prime}

) vs.

\tau

for T2/T3/T4 over 28 FP cells. Green

=

classical-anchor band, red dashed

=

frozen

\tau

, gray dotted

=

T3 degenerate floor. The eligible count is 0 across the anchor band on every task.

Model	rerank R@5	Wilson 95%	LB $\geq$ $\tau$ ?	$\Delta$ vs BM25	verdict
Qwen3-0.6B	0.311	$\,[0.282,0.340]$	no	$\,[-0.160,-0.099]$	worse
Qwen3-1.7B	0.443	$\,[0.412,0.474]$	no	$\,[-0.023,+0.031]$	ties
Qwen3-4B	0.486	$\,[0.455,0.517]$	no	$\mathbf{\,[+0.020,+0.073]}$	beats
Qwen3-8B	0.393	$\,[0.362,0.423]$	no	$\,[-0.077,-0.016]$	worse
Llama-3.2-1B	0.224	$\,[0.199,0.251]$	no	$\,[-0.246,-0.187]$	worse
Llama-3.2-3B	0.438	$\,[0.406,0.468]$	no	$\,[-0.030,+0.025]$	ties

Appendix

Table 13: T2 BM25

\rightarrow

SLM re-rank, absolute recall@5 vs.

\tau

=0.479

(

n{=}984

; BM25-full 0.440 on test,

\tau

is the calib-split BM25, § 3 ). Eligibility rule = the paper’s 10k-bootstrap LB

\geq

\tau

; the Wilson 95% column is a cross-check and agrees to

\leq 0.001

(bootstrap LB 0.456/0.467 for the two best).

\Delta

= paired rerank

-

BM25 (bootstrap 95%).

Figure 7: T2 recall vs. context length (Result 3, § 6 ). Full-index ranking fails; shortening the pool (BM25 shortlist) lifts every model, and the 4B re-ranker crosses above BM25. Long-context capacity is the bottleneck.

Family	Model	T1 $\textrm{FA}\!\downarrow$	T1 $\textrm{cov}\!\uparrow$	T2 R@5	T3 wF1	T4 tool
Qwen3	0.6B	0.012–1.000	0.029–1.000	0.123–0.318	0.557–0.557	0.898–0.907
Qwen3	1.7B	0.130–1.000	0.619–1.000	0.395–0.449	0.557–0.579	0.758–0.773
Qwen3	4B	0.025–0.568	0.216–0.993	0.428– 0.488	0.584–0.620	0.671–0.724
Qwen3	8B	0.222–0.525	0.950–1.000	0.227–0.457	0.594–0.648	0.784–0.813
Llama	1B	0.000	0.000	0.181–0.263	0.540–0.566	0.551– 0.913
Llama	3B	0.426–0.932	0.971–1.000	0.374–0.460	0.557–0.586	0.636–0.689

Appendix

Table 14: Prompt-paraphrase robustness: primary metric range (min–max) over the frozen prompt and 3 neutral paraphrases, per task. All 112 configs are ineligible (0 flips); the 3 point-estimate

\tau

crossings under best-of-phrasing (Qwen3-4B/Llama-8B T2, Llama-1B T4) fail the CI-LB rule and are not stable across phrasings.

Figure 8: Prompt-paraphrase robustness (FP16, 28 cells

\times

4 phrasings). (a) T1 in (coverage, false-approval) space: every phrasing of every model lands outside the shaded eligible corner. (b–d) T2/T3/T4 per-model range across the 4 phrasings (whisker; filled dot

=

primary); red dashed

=

\tau

. In (b) the open marker is the best phrasing’s Wilson lower bound; Qwen3-4B and Llama-8B exceed

\tau

in the point estimate but their bounds stay below. The verdict is ineligible everywhere.

Explore similar work

May 12, 2026

It's Not the Size: Harness Design Determines Operational Stability in Small Language Models

This paper experimentally analyzes how the level of harness engineering affects the operational performance of small language models (SLMs, 2-3B parameters). Three harness conditions - model-only (raw prompt), minimal-shell (wrapper tags), and a 4-stage pipeline (plan->execute->verify->recover) - are applied to three models (Gemma4 E2B, Qwen3.5:2B, LLaMA 3.2 3B) across 24 tasks, comparing Task Success Rate (TSR) and Valid TSR (VTSR). The pipeline harness achieves TSR=0.952 and VTSR=1.000 on Gemma4 E2B (T1-T5, 21 tasks). A non-monotonic phenomenon - minimal-shell TSR < model-only TSR - is observed in two models. In LLaMA 3.2 3B model-only, seven format violations yield TSR=0.429, revealing scaffold collapse: the model abandons JSON structure under complex format requirements without harness support. Ablation shows planning and recovery each contribute approximately 24.7% of total gain. VCR (Verification Catch Rate)=0.625 across all pipeline runs.

Yong-eun Cho

KailosLab, Seoul, Republic of Korea

Small Language Models

Ask PDF

Oct 10, 2026

Iterate on an LLM, Serve on an SLM

As frontier LLMs gain more capabilities, the cost to run them also goes up. At the same time, small language models (SLMs) also gain more capabilities but have largely been able to keep their cost stable, offering 5-40 times cheaper inference for the narrow, repetitive subtasks that agents decompose work into. However, the standing limiting factor is getting the correct behavior out of a small model in a fast, efficient, and economical way. Unfortunately, in most cases, zero-shot small models fail narrow production tasks. Furthermore, retraining one every time the behavior changes is too slow to iterate on. We propose doing both in sequence. First, develop the desired behavior in prompt space, on a frontier model, with immediate feedback from live traffic. Then, once the behavior converges, distill the recorded production behavior (not synthetic imitations of it) into an SLM through supervised fine-tuning (SFT). We show a proper use case of this method through a scheduling agent for home-care agencies that fans out a single yes/no safety decision per caregiver-shift pair on GPT-5.4-nano. A LoRA-tuned Qwen3-8B trained on 11.6k recorded production decisions reaches 0.913 agreement, while the teacher (GPT-5.4-nano), re-run on its own recorded prompts, agrees with itself only 72.7% of the time, pointing to the fact that the SLM may also be more consistent. Furthermore, when the two disagreed, a blinded professional scheduler sided with the student 48-29. Finally, we find that the SLM has approximately 2.5 times lower median latency and is a fraction of the cost. We believe this lifecycle of quick building, live feedback, and cheap inference is a blueprint for others building similar systems.

Keegan Wang

Carnegie Mellon University · Phoebe

LLM Agent Training Small Language Models Efficient Language Model Inference Language Model Distillation

Ask PDF

Oct 2, 2026

Toward SLM-based agentic task-tool intent matching

Tool-equipped AI agents use tool calls to access data and act on external systems. Horizontal growth of agentic systems increases the number of these interactions, and further motivates the need for automated, per-call oversight that can operate at low latency and/or on-prem. Conventional authorization schemes can determine whether an agent is allowed to invoke a tool, but cannot assess the agent's underlying cognition, specifically, whether the tool selection represents a logical, relevant step toward satisfying the intent of the task or not. Consequently, an allowed call may still deviate from the task's intent: a rogue agent might deviate the calls or nudge other agents to make a combination of calls that would not align with the intent of the task. Therefore, every call needs to be verified. In this study we investigate the applicability of Small Language Models (SLMs) to this purpose: an SLM functions as a task-tool relevance classifier that evaluates every selected tool independently against the assigned task and returns a relevance signal for downstream enforcement. Equipped with a novel dataset with multi-tool tasks whose required tools span distinct Model Context Protocol (MCP) servers, we used prompt-optimization, supervised fine-tuning, and reinforcement learning through GRPO to optimize and specialize SLMs.

Chiara Troiani, Arash Salarian, Majed El Helou +4

Cisco Systems, Switzerland

LLM Agent Verification LLM Agent Safety Small Language Models

Ask PDF