Measuring the Microtask Eligibility Gap: When Is an Off-the-Shelf SLM Enough for an Agent Harness?

Jundong Hu; Shekar Ramachandran

cs.AISep 1, 2026

Measuring the Microtask Eligibility Gap: When Is an Off-the-Shelf SLM Enough for an Agent Harness?

Authors: Jundong Hu, Shekar Ramachandran

Organizations: PayPal AI

Abstract

Agent harnesses increasingly want to run small language models (SLMs) on the microtasks around a frontier large language model (LLM) planner: auto-approving shell commands, writing memory, selecting tools, ranking past turns. We ask whether off-the-shelf SLMs meet practitioner-defined thresholds and, when they fail, why, and whether quantization changes the answer. We build a benchmark of 4 such microtasks with fixed prompts and automatic metrics, each with a pre-specified threshold $τ$ anchored to a cheap non-LLM baseline and a CI-aware eligibility rule (a configuration passes only if its confidence bound clears $τ$ ). Sweeping Qwen3 0.6/1.7/4/8B at their best (FP16, greedy, one frozen prompt, no tuning), we find an eligibility gap: 0 of 16 (4 tasks $\times$ 4 models) configurations pass (verified by checking the raw outputs and parser behavior). A logprob decision-threshold diagnostic (T1/T3/T4; T2 via a context-length/cascade probe) separates the failures into capability deficits and failures that can be addressed by changing the decoding threshold (4 regimes). Quantization to 4-bit (RTN/GPTQ/AWQ) does damage that depends on model size and moves no configuration into eligibility (certified on the reconstructable hard-label tasks T1/T3, diagnostic/windowed robustness on T2/T4), so the gap tracks model size more than precision; it replicates on Llama-3.x (12/12 ineligible) and is robust to the anchor choice (a $τ$ -sweep) and to prompt wording (0/112 eligible across the original plus 3 neutral paraphrases per cell). The practical implication: place SLMs behind a baseline that meets the CI-backed threshold, and use the SLM only where the baseline fails to meet the threshold; e.g. a 4B re-ranker over a BM25 shortlist beats BM25 ( $+0.047$ [0.020, 0.073], without itself certifying eligibility).

Figures & tables

Task	Decision	Primary metric	$\tau$ (frozen) / anchor
T1	safe shell auto-approve	false-approval $\textrm{FA}\!\downarrow$ & coverage $\textrm{cov}\!\uparrow$	$\textrm{FA}\leq 0.167$ & $\textrm{cov}\geq 0.921$ / deliberation+regex
T2	memory relevance recall	Recall@5	$\geq 0.479$ / BM25 Recall@5
T3	memory write / no-write	write-F1	$\geq 0.656$ / write-heuristic
T4	tool selection	correct-tool acc (callable)	$\geq 0.913$ / TF-IDF shortlist top-1

Table 1: Benchmark setup. One prompt per task;

\tau

anchored to a non-LLM baseline. Test sizes: T1 301 / T2 984 / T3 3032 / T4 600.

	T1 $\textrm{FA}\!\downarrow$ & $\textrm{cov}\!\uparrow$	T2 R@5	T3 wF1	T4 tool
Qwen3-0.6B	0.012, 0.029	0.041	0.557	0.900
Qwen3-1.7B	0.988, 1.000	0.125	0.579	0.758
Qwen3-4B	0.414, 0.971	0.350	0.598	0.707
Qwen3-8B	0.463, 0.993	0.292	0.634	0.784
$\tau$	$\leq$ 0.167 & $\geq$ 0.921	$\geq$ 0.479	$\geq$ 0.656	$\geq$ 0.913

Table 2: FP16 eligibility sweep (point estimates). All 16 configurations are ineligible: no CI bound clears

\tau

(bounds in text). T1 requires both

\textrm{FA}\!\leq\!0.167

and

\textrm{cov}\!\geq\!0.921

Panel of Figure 1 — Figure 1: Decision-threshold diagnostic (FP16). T1: only the 8B ROC enters the eligibility region. T3: all operating points, including the oracle, sit below the $\text{F1}=0.656$ iso-contour. T4: abstention is separable (AUROC 0.94–0.98) at every scale, a decoding artifact rather than missing capability.

Size	full-index R@5	rerank R@5	$\Delta$ vs BM25	95% CI	verdict
0.6B	0.041	0.311	$-0.129$	$\,[-0.160,-0.099]$	worse
1.7B	0.125	0.443	$+0.004$	$\,[-0.023,+0.031]$	ties
4B	0.350	0.486	$\mathbf{+0.047}$	$\mathbf{\,[+0.020,+0.073]}$	beats BM25
8B	0.292	0.393	$-0.047$	$\,[-0.077,-0.016]$	worse

Table 3: T2 BM25

\rightarrow

SLM re-rank (paired 10k bootstrap,

n\!=\!984

; BM25 R@20 ceiling 0.586).

\Delta

is rerank R@5

-

BM25 R@5 on the same items.

Figure 2: Quantization robustness depends on model size. Diagnostic AUROC (T1/T3/T4) and Recall@5 (T2) vs. model size; lines are fp/rtn/gptq/awq. The 4-bit lines diverge from FP16 at 0.6B and converge onto it by 8B; the T2 BM25 line (dashed) is crossed at

\geq

4B (all 4 methods at 4B; all but RTN at 8B).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	T1 (danger)	T3 (write)	T4 (abstain)
Qwen3-0.6B	$.548\,[.479,.614]$	$.650\,[.630,.670]$	$.947\,[.925,.967]$
Qwen3-1.7B	$.783\,[.728,.836]$	$.717\,[.698,.736]$	$.942\,[.912,.969]$
Qwen3-4B	$.895\,[.859,.927]$	$.750\,[.732,.767]$	$.980\,[.965,.991]$
Qwen3-8B	$.959\,[.934,.979]$	$.771\,[.754,.788]$	$.973\,[.954,.989]$
Llama-3.2-1B	$.557\,[.490,.622]$	$.523\,[.501,.544]$	$.462\,[.408,.516]$
Llama-3.2-3B	$.802\,[.752,.849]$	$.684\,[.665,.703]$	$.950\,[.924,.971]$

Appendix

Table 4: Diagnostic-layer AUROC with 95% bootstrap CI (FP16 test, 10k resamples). Point estimates match the ledger; CIs quantify the four-regime spectrum (§ 5 ).

n

: T1 301, T3 3032, T4 600.

Task	Size	RTN	GPTQ	AWQ
T1 ( $\textrm{FA}/\textrm{cov}$ )	0.6B †	0.99/0.97	1.00/1.00	0.23/0.40
	1.7B	0.00/0.00	0.60/0.75	0.10/0.57
	4B	0.40/0.88	0.31/0.88	0.89/0.99
	8B	0.59/0.99	0.70/0.99	0.60/0.99
T3 (write-F1)	0.6B †	0.000	0.544	0.254
	1.7B	0.576	0.532	0.568

Appendix

Table 5: Hard-label quant ledger (greedy decision reconstructed from the decision-logprob margin sign, validated against FP for

\geq

1.7B). T1 shows

\textrm{FA}/\textrm{cov}

; T3 shows write-F1. No cell is eligible (T1 needs

\textrm{FA}\!\leq\!0.167\wedge\textrm{cov}\!\geq\!0.921

; T3 needs F1

\geq

0.656). 0.6B reconstruction is unfaithful (marked † ) and covered by FP-domination.

Task	Frozen eligibility metric	Reconstructable from quant dumps?	Evidence (certified / surrogate)
T1	$\textrm{FA}/\textrm{cov}$ (hard label)	$\geq$ 1.7B yes; 0.6B no	Hard-label ledger $+$ AUROC ( $\geq$ 1.7B cert.); 0.6B domination
T3	write-F1 (hard label)	$\geq$ 1.7B yes; 0.6B no	Hard-label ledger $+$ AUROC ( $\geq$ 1.7B cert.); 0.6B domination
T4	correct-tool	No (margins encode call-vs-abstain only)	Abstention AUROC close to FP16 $\geq$ 4B (surrogate)
T2	recall@5 (full index)	No (quant dumps are windowed curves)	Windowed crossover survives $\geq$ 4B (surrogate)

Appendix

Table 6: Evidence for the no-rescue verdict per task. Hard-label reconstruction is validated against FP for

\geq

1.7B (Table 5 ); T2/T4 and 0.6B carry surrogate evidence that does not re-evaluate the frozen metric.

Task	0.6B	1.7B	4B	8B
T1 ECE	0.318	0.531	0.236	0.256
T3 ECE	0.277	0.378	0.407	0.250
T4 ECE	0.057	0.069	0.046	0.046

Appendix

Table 7: ECE of the decision logprobs (FP16, test). Mean-confidence far exceeds accuracy on T1/T3.

Figure 3: Reliability of decision logprobs (FP16): accuracy vs. mean confidence per bin (dotted

=

perfect calibration). T1/T3 sit well below the diagonal (overconfident); T4 abstention is near-calibrated.

Figure 4: T2 recall vs. context length per model size; lines are fp/rtn/gptq/awq plus BM25 and the

5/N

floor. The BM25 crossover survives quantization only at

\geq

4B.

Model	T1 $\textrm{FA}/\textrm{cov}$	T2 R@5-lb	T3 F1-lb	T4 tool-lb	elig
Llama-3.2-1B	$0.00/0.00$	0.001 ( $-0.478$ )	0.520 ( $-0.136$ )	0.709 ( $-0.204$ )	0/4
Llama-3.2-3B	$0.93/1.00$	0.134 ( $-0.345$ )	0.566 ( $-0.090$ )	0.590 ( $-0.323$ )	0/4
Llama-3.1-8B	$0.67/1.00$	0.270 ( $-0.210$ )	0.541 ( $-0.116$ )	0.684 ( $-0.229$ )	0/4
$\tau$	$\textrm{FA}{\leq}.167\wedge\textrm{cov}{\geq}.921$	$\geq$ 0.479	$\geq$ 0.656	$\geq$ 0.913

Appendix

Table 8: Cross-family FP16 eligibility (CI bound used for the verdict; margin in parens). All ineligible.

Table 13

Panel of Figure 5 — Figure 5: Cross-family decision-threshold diagnostic (Llama-3.x, FP16), same layout as Fig. 1 : the same capability $\rightarrow$ calibration signature appears (T1 8B enters the eligibility region; T3 stays below the F1 iso-contour; T4 abstention is highly separable at $\geq$ 3B, AUROC 0.95/0.92, sub-chance 0.462 at 1B).

Table 15

Figure 6:

\tau

-sensitivity: number of eligible configs (CI-LB

\geq\tau^{\prime}

) vs.

\tau

for T2/T3/T4 over 28 FP cells. Green

=

classical-anchor band, red dashed

=

frozen

\tau

, gray dotted

=

T3 degenerate floor. The eligible count is 0 across the anchor band on every task.

Model	rerank R@5	Wilson 95%	LB $\geq$ $\tau$ ?	$\Delta$ vs BM25	verdict
Qwen3-0.6B	0.311	$\,[0.282,0.340]$	no	$\,[-0.160,-0.099]$	worse
Qwen3-1.7B	0.443	$\,[0.412,0.474]$	no	$\,[-0.023,+0.031]$	ties
Qwen3-4B	0.486	$\,[0.455,0.517]$	no	$\mathbf{\,[+0.020,+0.073]}$	beats
Qwen3-8B	0.393	$\,[0.362,0.423]$	no	$\,[-0.077,-0.016]$	worse
Llama-3.2-1B	0.224	$\,[0.199,0.251]$	no	$\,[-0.246,-0.187]$	worse
Llama-3.2-3B	0.438	$\,[0.406,0.468]$	no	$\,[-0.030,+0.025]$	ties

Appendix

Table 13: T2 BM25

\rightarrow

SLM re-rank, absolute recall@5 vs.

\tau

=0.479

(

n{=}984

; BM25-full 0.440 on test,

\tau

is the calib-split BM25, § 3 ). Eligibility rule = the paper’s 10k-bootstrap LB

\geq

\tau

; the Wilson 95% column is a cross-check and agrees to

\leq 0.001

(bootstrap LB 0.456/0.467 for the two best).

\Delta

= paired rerank

-

BM25 (bootstrap 95%).

Figure 7: T2 recall vs. context length (Result 3, § 6 ). Full-index ranking fails; shortening the pool (BM25 shortlist) lifts every model, and the 4B re-ranker crosses above BM25. Long-context capacity is the bottleneck.

Family	Model	T1 $\textrm{FA}\!\downarrow$	T1 $\textrm{cov}\!\uparrow$	T2 R@5	T3 wF1	T4 tool
Qwen3	0.6B	0.012–1.000	0.029–1.000	0.123–0.318	0.557–0.557	0.898–0.907
Qwen3	1.7B	0.130–1.000	0.619–1.000	0.395–0.449	0.557–0.579	0.758–0.773
Qwen3	4B	0.025–0.568	0.216–0.993	0.428– 0.488	0.584–0.620	0.671–0.724
Qwen3	8B	0.222–0.525	0.950–1.000	0.227–0.457	0.594–0.648	0.784–0.813
Llama	1B	0.000	0.000	0.181–0.263	0.540–0.566	0.551– 0.913
Llama	3B	0.426–0.932	0.971–1.000	0.374–0.460	0.557–0.586	0.636–0.689

Appendix

Table 14: Prompt-paraphrase robustness: primary metric range (min–max) over the frozen prompt and 3 neutral paraphrases, per task. All 112 configs are ineligible (0 flips); the 3 point-estimate

\tau

crossings under best-of-phrasing (Qwen3-4B/Llama-8B T2, Llama-1B T4) fail the CI-LB rule and are not stable across phrasings.

Figure 8: Prompt-paraphrase robustness (FP16, 28 cells

\times

4 phrasings). (a) T1 in (coverage, false-approval) space: every phrasing of every model lands outside the shaded eligible corner. (b–d) T2/T3/T4 per-model range across the 4 phrasings (whisker; filled dot

=

primary); red dashed

=

\tau

. In (b) the open marker is the best phrasing’s Wilson lower bound; Qwen3-4B and Llama-8B exceed

\tau

in the point estimate but their bounds stay below. The verdict is ineligible everywhere.

Explore similar work

It's Not the Size: Harness Design Determines Operational Stability in Small Language Models
May 12, 2026Yong-eun ChoSmall Language Models
KailosLab, Seoul, Republic of Korea
Iterate on an LLM, Serve on an SLM
Oct 10, 2026Keegan WangLLM Agent Training Small Language Models
Carnegie Mellon University · Phoebe
Toward SLM-based agentic task-tool intent matching
Oct 2, 2026Chiara Troiani, Arash Salarian, Majed El Helou +4LLM Agent Verification LLM Agent Safety
Cisco Systems, Switzerland

May 12, 2026J/K move · Enter open · S save

It's Not the Size: Harness Design Determines Operational Stability in Small Language Models

Yong-eun Cho

KailosLab, Seoul, Republic of Korea

Small Language Models

Open paper AskPDF arXiv

This paper experimentally analyzes how the level of harness engineering affects the operational performance of small language models (SLMs, 2-3B parameters). Three harness conditions - model-only (raw prompt), minimal-shell (wrapper tags), and a 4-stage pipeline (plan->execute->verify->recover) - are applied to three models (Gemma4 E2B, Qwen3.5:2B, LLaMA 3.2 3B) across 24 tasks, comparing Task Success Rate (TSR) and Valid TSR (VTSR). The pipeline harness achieves TSR=0.952 and VTSR=1.000 on Gemma4 E2B (T1-T5, 21 tasks). A non-monotonic phenomenon - minimal-shell TSR < model-only TSR - is observed in two models. In LLaMA 3.2 3B model-only, seven format violations yield TSR=0.429, revealing scaffold collapse: the model abandons JSON structure under complex format requirements without harness support. Ablation shows planning and recovery each contribute approximately 24.7% of total gain. VCR (Verification Catch Rate)=0.625 across all pipeline runs.

Abstract

Explore similar work

It's Not the Size: Harness Design Determines Operational Stability in Small Language Models

Iterate on an LLM, Serve on an SLM

Toward SLM-based agentic task-tool intent matching