TicTacBench: Benchmarking Timing Closure Capabilities of Coding Agents
Authors: Bowei Wang, Zhigang Fang, Zhijie Yang, Renzhi Chen, Shanshan Li, Lei Wang
Organizations: Computer Department, National University of Defense Technology, Changsha, China · Defense Innovation Institute, Academy of Military Science, Beijing, China · Intelligent Microelectronics Center, Qiyuan Lab, Beijing, China
Recent advances in large language models (LLMs) have led to the emergence of coding agents capable of performing complex engineering tasks, including register-transfer level (RTL) design and optimization. Existing RTL benchmarks mainly evaluate functional correctness and performance, power, and area (PPA) of the generated RTL designs, leaving agents' ability for timing closure under-evaluated. We propose TicTacBench, a benchmark specifically designed to evaluate coding agents' capabilities for RTL-level timing closure under post-place-and-route (post-PnR) evaluation. TicTacBench contains 30 diverse tasks, each provided with a suboptimal RTL design, realistic timing constraints, functional equivalence verification, and timing reports. With over 300 runs of coding agents driven by 8 frontier LLMs, we find that even the best agent can only close 53.3% of tasks with 7.18% area-delay product (ADP) degradation and 8.83% energy-delay-squared product (EDDP) improvement on average. We identify common failure categories that explain why agents fail to close timing. Then we propose TicTacSkill, a new method that guides agents to follow standard timing-closure procedures and improves the Timing Closure Rate by 9%. These results suggest that while coding agents have made significant progress in RTL design, their timing-closure capability still has substantial room for improvement.
Figures & tables
Fig. 1: Pre-PnR timing metrics are unreliable for post-PnR timing closure. (a) Treating pre-PnR closure as a prediction of post-PnR closure produces high false positives: designs that appear to close timing before PnR often fail after physical implementation. (b) Pre-PnR metrics are unreliable for evaluation. The agent selected as best before PnR disagrees with the post-PnR.
Benchmark
Task setting
Input / output
Validation artifacts
Timing-closure coverage
VerilogEval [ 11 ]
RTL generation
Prompt and target RTL
Simulation
No SDC or post-PnR objective
RTLLM v2 [ 7 ]
RTL generation + QoR
Spec, tests, and RTL
Cycle-accurate
QoR stress signal; no closure target
RTLBench [ 10 ]
RTL generation quality
Spec, tests, and RTL
Simulation, lint, LLM judge
No timing or PPA objective
RTL-Repo [ 8 ]
Project-scale RTL generation
Repo context and Verilog
Code checks
No task-level SDC or timing repair
RTLRewriter [ 4 ]
RTL rewriting for early QoR
RTL context and rewrite
Functional equivalence
Logic-synthesis QoR only
RTL-OPT [ 12 ]
RTL QoR optimization
Suboptimal and optimized RTL
Equivalence
PPA/QoR benchmark
TABLE I: Comparison of representative RTL benchmarks from the perspective of agent-based timing closure.
Fig. 2: TicTacBench evaluation flow. The agent uses released task artifacts and tool set commands to submit optimized.v ; valid candidates pass interface, function, and latency guards before post-PnR evaluation computes closure, WNS repair, power-delay trade-offs, and area-delay trade-offs.
Fig. 3: Main TicTacBench leaderboard from 30 tasks and coding agents driven by 8 frontier LLMs. Panel (a) ranks agents by post-PnR Timing Closure Rate. Panels (b)–(d) rank agents by Avg. Δ WNS%, Avg. Δ EDDP%, and Avg. Δ ADP% relative to the suboptimal baseline; higher is better. Panel (f) reports the Reciprocal Rank Fusion (RRF) ranking across the four metrics.
Task
GPT
Claude
Dpsk-Pro
Dpsk-Flash
Kimi
MiniMax
GLM
Qwen
WNS
Cells
Power
WNS
Cells
Power
WNS
Cells
Power
WNS
Cells
Power
WNS
Cells
Power
WNS
Cells
Power
WNS
Cells
Power
WNS
Cells
Power
accu
+0.00
+893
1.37
-0.04
+1.17k
1.68
-0.01
+901
1.36
-0.05
+903
1.38
-0.03
+967
1.41
-0.17
+885
1.40
+0.00
+1.00k
1.46
-0.04
+827
1.30
add_sub
-0.12
+353
0.38
-0.07
+599
0.37
-0.08
+592
0.38
-0.12
+353
0.38
-0.06
+367
0.45
-0.10
+611
0.37
-0.06
+367
0.45
-0.07
+323
0.37
adder_32bit
-0.21
+2.76k
0.64
-0.23
+2.65k
0.64
-0.14
+1.71k
0.49
-0.07
+1.53k
0.44
-0.07
+1.53k
0.44
-0.07
+1.53k
0.44
-0.11
+1.73k
0.49
-0.07
+1.53k
0.44
adder_pipe_64bit
-0.23
+11.8k
7.93
+0.00
+12.7k
8.09
-0.23
+11.8k
7.93
-0.23
+11.8k
7.93
-0.23
+11.8k
7.93
-0.33
+12.1k
7.76
-0.22
+12.2k
7.79
-0.23
+11.8k
7.93
adder_select
+0.00
+1.29k
0.50
+0.00
+1.29k
0.50
+0.00
+1.29k
0.50
+0.00
+1.58k
0.60
+0.00
+1.29k
0.50
+0.00
+1.29k
0.50
+0.00
+1.29k
0.50
+0.00
+1.29k
0.50
TABLE II: Optimized post-PnR PPA metrics by task and model. Each model group reports WNS, cell count, and power. Green groups Pareto-dominate the corresponding pre-optimization result across the three metrics, while red groups are Pareto-dominated by it. WNS is in ns, power is in mW, and cells are compacted to control density. The last row reports dominate/dominated counts over all tasks.
Fig. 4: Representative case from multi_pipe_8bit. The failed rewrite leaves a wide addition cone on the critical stage. The successful rewrite moves the register boundary into the reduction tree.
Capability
Failure category
Meaning
Domain Knowledge
Inefficient STA report-driven optimization
Fails to map key STA information such as slack and critical paths to the RTL level for report-driven optimization.
Suboptimal datapath design
Preserves excessively deep arithmetic/datapath logic in the design.
Architecture Design
Suboptimal pipeline design
Fails to partition pipeline stages to cut the critical combinational logic chain, or introduces a new one.
Suboptimal data/control-path design
Introduces control-path logic such as decode or mux logic on critical datapath paths.
Optimization Flow
Unsound exploration
Explores only locally suboptimal solutions and lacks a plan for exploring globally optimal solutions.
Unsound acceptance
Prematurely accepts weakly converged results from pre-PnR STA, leading to failure at post-PnR.
TABLE III: Capability and Failure Categories of Trajectory Semantic Analysis.
Fig. 5: Failure mechanism profiles by failed trajectories. The heatmap reports each agent’s failure-mechanism enrichment relative to the all-agent baseline.
Model
Setting
Closure
Avg. Δ WNS%
Avg. Δ EDDP%
Avg. Δ ADP%
GPT-5.4
w/o skill
53.3%
48.7%
5.5%
-2.5%
w/ skill
56.7%
66.4%
16.3%
-3.6%
MiniMax
w/o skill
23.3%
-1.7%
-0.7%
-9.6%
w/ skill
36.7%
17.0%
-7.2%
-10.9%
DeepSeek
w/o skill
33.3%
44.6%
11.6%
-4.3%
w/ skill
43.3%
62.1%
14.9%
-3.0%
TABLE IV: TicTacSkill ablation on the three models with skill-enabled runs. Each w/ skill row is paired with the same model and task without the skill.
Centre for Electronics Frontiers, Institute for Integrated Micro and Nano Systems, School of Engineering, The University of Edinburgh, U.K. · Institute for Integrated Micro and Nano Systems, School of Engineering, The University of Edinburgh, U.K.