NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation

Converting a SPICE netlist into a human-readable schematic is a longstanding problem in electronic design automation: simulators and machine-learning pipelines readily produce netlists, but designers reason about circuits through diagrams. Recent learning-based approaches translate netlists into schematics probabilistically, yet they provide no guarantee that the generated drawing preserves the original connectivity, and their accuracy degrades sharply as circuits grow. We present Weave, a deterministic converter that turns a SPICE netlist into an LTspice .asc schematic using a layered (Sugiyama-style) graph layout, and that certifies every output by a round-trip connectivity check: the generated schematic is re-parsed into a netlist and compared, net for net, against the input. A result is reported as correct only when the two partitions are identical, giving a binary correctness certificate rather than a similarity score. Weave runs entirely client-side as a single dependency-free file and embeds a pin table for 5093 LTspice symbols. On the identical public Circuits-LTSpice test set used by the state-of-the-art LLM converter Schemato (117 circuits, netlisted with LTspice itself), Weave achieves 100% compilation and 100% round-trip-verified connectivity equivalence, compared with Schemato's reported 76% compilation and a graph-edit-distance similarity of 0.35; notably, 73% of that set exceeds the five-component threshold beyond which Schemato reports losing connectivity accuracy. On a larger and harder corpus, the 3460 netlistable circuits of the official Analog Devices LTspice demo collection, Weave verifies exact connectivity for 88.4% of circuits, with the remaining failures concentrated in a single, well-characterized class of dense multi-pin power modules.

Analog circuit design is a time-consuming, iterative process in a nonlinear and high-dimensional design space that relies heavily on expert intuition. Among recent developments, LLMs have introduced a promising approach by bringing natural language reasoning to circuit design tasks. The majority of conventional LLM-based approaches provide fragmented solutions that focus either only on sizing or topology generation. These methods require adding specific technical knowledge manually, which is inefficient and prone to hallucinations during circuit sizing. Moreover, the inherent trade-off in meeting different specs makes current approaches iterative and tedious. Another shortcoming is the inability to create innovative topologies, which may lead to sub-optimal designs due to reliance on conventional topologies. In this paper, we present AaLLM, an open-source end-to-end multi-agent LLM workflow that takes user specs as input and outputs the appropriate netlist, encompassing both topology generation and circuit sizing. AaLLM automates the creation of a relevant knowledge base from research papers and textbooks to combat tedious manual data collection. A RAG model is implemented to emulate circuit design expertise using this knowledge base. Moreover, AaLLM uses a novel tri-agent feedback system comprising a Designer that determines circuit component values, a Critic that scrutinizes these values, and an Evaluator that minimizes circuit sizing iterations by arbitrating between the other two agents. AaLLM-generated novel topologies achieve a figure of merit (FoM) comparable to that of known topologies, and up to 3x higher for certain circuits. Testing on several circuit topologies, our results show a 3x - 4.5x decrease in the number of SPICE calls at inference when compared to SOTA multi-agent LLM pipelines. The results also show a 40x decrease in wall-clock time compared to existing approaches.

LLM-based RTL generation and reasoning is a promising direction for hardware design automation. High-quality benchmarks are critical infrastructure for tracking progress in this direction. However, existing RTL benchmarks face inherent limitations in both scale and task scope. The designs they cover are typically small and simple, and the tasks focus almost entirely on specification-to-RTL generation. Frontier models' performance already saturates on the existing benchmarks. Scaling these benchmarks up is fundamentally difficult because aligned labels are required for benchmarking, such as specifications and testbenches. Such aligned high-quality data are rarely available for real-world designs. We introduce RTL-BenchLS, a large-scale benchmark addressing both limitations above. It contains over 10,000 formally verified Verilog designs, covering substantially larger and more complex designs than existing benchmarks. Beyond specification-to-RTL generation, we propose three novel tasks that jointly evaluate reasoning and generation: round-trip reasoning, masked-content reasoning, and repository-issue reasoning. The first two are self-supervised, which directly resolves the scaling bottleneck. All tasks are verified through formal equivalence checking without any manual testbenches. We evaluate eight LLMs on RTL-BenchLS. Even the best model reaches only 23% on natural-language round-trip reasoning, 28% on masked-content reasoning, and 12% on repository-issue fixing. RTL-BenchLS is substantially more challenging than existing benchmarks. It leaves ample room for future improvement and offers guidance for developing LLM-based methods for hardware design.