In harmony with gpt-oss

Borislav Mavrin

cs.AIApr 1, 2026

In harmony with gpt-oss

Authors: Borislav Mavrin

Abstract

No one has independently reproduced OpenAI's published scores for gpt-oss-20b with tools, because the original paper discloses neither the tools nor the agent harness. We reverse-engineered the model's in-distribution tools: when prompted without tool definitions, gpt-oss still calls tools from its training distribution with high statistical confidence -- a strong prior, not a hallucination. We then built a native harmony agent harness (https://github.com/borislavmavrin/harmonyagent.git) that encodes messages in the model's native format, bypassing the lossy Chat Completions conversion. Together, these yield the first independent reproduction of OpenAI's published scores: 60.4% on SWE Verified HIGH (published 60.7%), 53.3% MEDIUM (53.2%), and 91.7% on AIME25 with tools (90.4%).

Figures & tables

Model	Published SWE verified score	SWE Leaderboard score 5 5 footnotemark: 5	Model size	Model architecture
Devstral Small 2	68.0% [ 3 ]	56.4%	24B	Dense
GLM-4.7-Flash	59.2% [ 10 ]	N/A	31B	MoE
Qwen3.5 27b	72.4% [ 7 ]	N/A	27B	Dense
gpt-oss-20b	60.7% [ 4 ]	26% 6 6 footnotemark: 6	20B	MoE

Table 1: Model size vs SWE Verified scores

Tool	Text mentions [95% CI]	Actual calls	Verdict
print_tree	28.1% [21.2, 35.0]	101	confirmed
search	34.4% [27.5, 42.5]	11	confirmed
open_file	44.4% [36.9, 52.5]	3	confirmed
apply_patch	2.5% [0.6, 5.0]	8	confirmed
read_file	25.0% [18.8, 31.9]	1	likely alias
list_files	10.6% [6.2, 15.6]	2	likely alias

Table 2: Cross-reference of text-mentioned repo_browser tools versus actual tool calls (

n=160

text samples,

n=1201

baseline samples). Tools with both text mentions and observed calls are confirmed; tools described but never called are confabulations.

Parameter	print_tree ( $n=108$ )	search ( $n=38$ )	open_file ( $n=32$ )
path	100% [100, 100]	97% [92, 100]	72% [56, 88]
depth	99% [97, 100]	—	—
query	—	95% [87, 100]	—
max_results	—	53% [37, 68]	—
line_start	—	—	78% [62, 91]
line_end	—	—	72% [56, 88]

Table 3: Schema robustness: parameter presence rates when the model is asked to call each tool by name (Step 5, 95% bootstrap CIs).

Group	Primary tool	Aliases	Canonical schema
Directory listing	print_tree	list_files , list_dir	{path, depth}
Content search	search	find	{path, query, max_results?}
File reading	open_file	read_file	{path, line_start?, line_end?}
Patch	apply_patch	—	{patch}

Table 4: Discovered in-distribution repo_browser tools with canonical schemas, plus container.exec as a flat tool.

Tool	Prompt type	Baseline	With tools	Lift
print_tree	explore	29.4% [20, 40]	98.6% [96, 100]	3.4 $\times$
search	search	3.8% [0, 9]	58.8% [48, 69]	15 $\times$
open_file	read code	1.2% [0, 4]	21.2% [13, 30]	18 $\times$

Table 5: Tool call rates without tools defined (baseline) versus with tools in the system message (95% bootstrap CIs).

Figure 2: Harmony vs chat completions token usage.

Benchmark	Published score [ 4 ]	HarmonyAgent score	95% Bootstrap CI
SWE Verified HIGH reasoning	60.7%	60.4%	[56.2%, 64.8%]
SWE Verified MEDIUM reasoning	53.2%	53.3%	[49.3%, 57.7%]
AIME 2025 MEDIUM reasoning with tools	90.4%	91.7%	[87.5%, 95.0%]

Table 6: OpenAI published results vs HarmonyAgent results (with 95% bootstrap CIs, 1,000 resamples).

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

Verifier-Guided Twelve-Tone Composition: A Generate-Verify-Repair Harness for Symbolic Music Generation
Jul 13, 2026Congren Dai, Danni Zhao, Enyang Liu +7Symbolic Music Generation
Central Conservatory of Music · Tsinghua University
ScorePrompts: Natural-Language Exploration of Symbolic Music Scores through Analysis
Sep 9, 2026Emmanouil Karystinaios, Gerhard WidmerMusic Information Retrieval Music Understanding
Institute of Computational Perception, Johannes Kepler University Linz, Austria
Comparative Validation of GPT-4o-mini and Teacher Mean Scores for Automated Scoring of Music Analysis Responses: Single-Pass Deployment, Repeatability, and Strategy-Specific Bias
Aug 3, 2026Baicheng Lin, Lingxi Jin, Kyung-Seok MinLLM-as-a-Judge Educational Technology
Department of Education, Sejong University, South Korea · Department of Educational Technology, Ewha Womans University, South Korea

Jul 13, 2026J/K move · Enter open · S save

Verifier-Guided Twelve-Tone Composition: A Generate-Verify-Repair Harness for Symbolic Music Generation

Congren Dai, Danni Zhao, Enyang Liu, Michael Ching Yam+6

Central Conservatory of Music · Tsinghua University

Symbolic Music Generation

Open paper AskPDF arXiv

Large language models can produce superficially legal twelve-tone scores that collapse into degenerate textures. We introduce a neuro-symbolic harness that wraps a language-model proposer in a generate-verify-repair-trace loop with symbolic verification. The complete pipeline improves event-local consistency without claiming whole-piece legality. Across 40 controlled tasks and four paired models, constraint-checked delivery rises from 13.3% under raw generation to 48.1% with the harness; it abstains on the remaining 51.9% of runs. The pass rate of a narrower collision and serialisation-consistency check rises from 33.5% to 58.3%, while degeneracy remains near 0.05, including under adversarial prompting. A blinded evaluation by five experts also shows a descriptive aggregate preference for harness candidates over raw generation in adherence, perceived legality, coherence, and overall quality.

Abstract

AI Agent Benchmarks

Explore similar work

Verifier-Guided Twelve-Tone Composition: A Generate-Verify-Repair Harness for Symbolic Music Generation

ScorePrompts: Natural-Language Exploration of Symbolic Music Scores through Analysis

Comparative Validation of GPT-4o-mini and Teacher Mean Scores for Automated Scoring of Music Analysis Responses: Single-Pass Deployment, Repeatability, and Strategy-Specific Bias