Specification Oracles

Atticus Cull; Justin McCarthy

cs.LGSep 11, 2026

Specification Oracles

Authors: Atticus Cull, Justin McCarthy

Organizations: Diffusion Redwood City, California, USA

Abstract

Specifications face a basic tradeoff: leave details out, and important questions go unanswered; record every detail separately, and the specification becomes large and prolix. We investigate whether a language model can serve as a compact, living specification oracle by learning facts about a target and answering questions about it directly. We compare two ways of storing the learned facts: external text notes and changes to the model's weights. Across four families of 596-fact worlds and two Qwen2.5 model sizes, weight-only oracles benefited substantially more from structure: with the 7B model, their accuracy integrated across storage capacities was 18.5 percentage points higher on structured than unstructured worlds, compared with 1.1 points for note-sheet oracles. This advantage came at a substantial storage cost, with the smallest adapter requiring approximately 175 KiB compared with a maximum note budget of 16 KiB. Adapted weights therefore exploited latent structure more successfully, while external notes required substantially less object-specific storage.

Figures & tables

Property	Note-sheet oracle	Weight-only adapter oracle
Base decoder	Frozen Qwen model	Frozen Qwen model plus active adapter
Stored program	UTF-8 note text	Trainable LoRA parameters
Swept capacity	Allocated byte budget	Adapted layer prefix / trainable parameters
Reported final storage	UTF-8 bytes in the final note	Trainable parameters $\times$ bytes per stored parameter
Study input	Compact fact table, current note, feedback	Object-specific QA/fact training data
Selection signal	Dev accuracy	Dev accuracy

Table 1: Oracle contract for the two systems.

Figure 1: Held-out accuracy across capacity for the four semantic sweeps. Adapter capacity is adapted layer count; note capacity is allocated UTF-8 note budget. Markers show seed means, and shaded bands show the observed seed range, not confidence intervals.

Figure 2: Structure advantage,

\Delta_{\mathrm{AUC}}

, computed within each model/oracle system as structured-family AUC minus independent-family AUC. Hollow points are seed-level contrasts; filled diamonds and bars show the mean and seed-level 95% Student-

t

interval. This is the cleanest visual summary because it does not require adapters and notes to share a byte axis.

Figure 3: Storage accounting. Notes use final UTF-8 note length; adapters use trainable parameters

\times

2 bytes. The plotted ranges show why a single accuracy-per-byte ratio would be misleading: note and adapter observations are separated by roughly an order of magnitude in persistent payload size.

Figure 4: Note-utilization diagnostic. The 7B 32 KiB extension uses only three matched seeds and should be treated as exploratory; among those 12 matched worlds, mean accuracy changes only from 69.3% to 69.6%, with improvements, stalls, and regressions all present. The 14B note sweep uses the 16 KiB budget more fully and reaches high raw accuracy without producing a large structure advantage.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Decoder	Oracle	Condition	Seeds	Test $n$	$\Delta_{\mathrm{AUC}}$	Read
Qwen2.5-7B	Adapter	Semantic	3	139	$+18.5$ pp $[+15.2,+21.8]$	Large; all seeds positive.
Qwen2.5-7B	Note	Semantic	8	100	$+1.1$ pp $[-4.3,+6.4]$	Centered near zero.
Qwen2.5-7B	Note	Randomized labels	3	100	$+2.4$ pp $[-9.0,+13.8]$	Noisy.
Qwen2.5-14B	Adapter	Semantic	3	32	$+16.6$ pp $[-1.8,+35.0]$	Positive but lower-power.
Qwen2.5-14B	Note	Semantic	8	100	$+1.5$ pp $[-1.1,+4.1]$	Near zero.

Appendix

Table 2: Headline structured-minus-independent AUC contrasts.

Memory-Augmented Language Models

Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.

See all 292 papers

SpecRead: A Benchmark for Measuring Whether Language Models Understand Hardware Specifications
Sep 27, 2026Feilian HuangDocument Understanding LLM Evaluation
Independent Researcher
Facts as First Class Objects: Knowledge Objects for Persistent LLM Memory
Mar 18, 2026Oliver Zahn, Simran ChanaPersistent Memory for Language Models LLM Memory
Independent Researcher · University of Cambridge
Improving Parametric Knowledge Access in Reasoning Language Models
Feb 25, 2026Melody Ma, John HewittFactual Knowledge in Language Models Knowledge-Intensive QA
Columbia University

Sep 27, 2026J/K move · Enter open · S save

SpecRead: A Benchmark for Measuring Whether Language Models Understand Hardware Specifications

Feilian Huang

Independent Researcher

Document Understanding LLM Evaluation Electronic Design Automation

Open paper AskPDF arXiv

Existing benchmarks for large language models (LLMs) in hardware design evaluate downstream artifacts such as generated RTL, assertions, or testbenches. When a model fails such a benchmark, the failure is ambiguous: it may have misread the specification, or it may have understood the specification and failed to write the code. We present SpecRead, a benchmark that isolates specification comprehension from generation ability. SpecRead v2.1 contains 385 questions over 10 open-source OpenTitan IP blocks: exact retrieval, cross-section reasoning, contradiction detection in mutated specifications, and spec-RTL consistency checking, plus 82 controls (41 distractor, 41 consistent-RTL). Type-4 items are built from real RTL mutations; we retain only mutations that Icarus Verilog simulation shows to change observable behavior. A with-spec vs. without-spec ablation suggests the questions require the excerpt, not training recall alone (without-spec accuracy 3/20 on the t1/t2 subset), though memorization of the source text may still help spot mutations. As an initial characterization with a small model, Ministral-3B scores 33.2% overall (128/385; macro average 39.0%): 55.2% on retrieval, 51.7% on cross-section reasoning. On the two contradiction-focused types, the verdict-plus-location measure gives 48.0% (t3) and 63.3% (t4), with a 51.2% false-positive rate on distractors and 100% on consistent-RTL controls. Layered scoring shows the model locates contradictions well (78.9-81.6% location accuracy) but scores lower on their category (43.9-49.7%). A structured "rule-table" prompting intervention lowers accuracy on every question type except t2 (tied). SpecRead is automatically scorable by deterministic checks, with gray-zone cases counted wrong under the conservative main scoring. The benchmark is regenerable for type-3 items via mutation injection, and built exclusively from public sources.

Abstract

Memory-Augmented Language Models

Explore similar work

SpecRead: A Benchmark for Measuring Whether Language Models Understand Hardware Specifications

Facts as First Class Objects: Knowledge Objects for Persistent LLM Memory

Improving Parametric Knowledge Access in Reasoning Language Models