RFBench: A Simulator-Grounded Multimodal Benchmark for Passive RF Layout Design and Understanding
Organizations: Department of Electrical and Computer Engineering, University of California, Los Angeles
Abstract
While multimodal language models can interpret diagrams and generate structured outputs, their capacity to connect passive radio-frequency (RF) design intent, layout geometry, and electromagnetic behavior remains largely untested. To address this gap, we introduce RFBench, a simulator-grounded dataset and benchmark for passive RF layout design and understanding. The dataset contains 10,000 two-port planar layouts simulated using ADS Momentum, with each layout represented by editable geometric primitives, a raster image, a 16x16 occupancy grid, complex S-parameters at 201 frequency points, a structural category, and an RF response label. Building on these aligned representations, we define seven tasks spanning specification-conditioned layout generation, representation translation, response prediction, simulator-based validation, design diagnosis and improvement, structural understanding, and response classification. Extensive baselines across seven multimodal language models reveal systematic weaknesses in generating structurally valid layouts and predicting electromagnetically faithful responses. We show that RF-specific fine-tuning increases T1a structural validity from 0.0% to 83.2% and the T3 primitive-input within-2-dB rate from 17.7% to 62.8%, demonstrating the value of domain adaptation while leaving considerable room for further progress. RFBench establishes a common testbed for evaluating multimodal RF capabilities and guiding the development of models that connect design intent and layout representations with physical behavior.
Figures & tables
| ID | Ability | Input | Output | Primary metric | Test |
|---|---|---|---|---|---|
| T1a | Layout generation | RF response specification | Primitive layout | Structural validity | 1,000 |
| T1b | Category-constrained generation | RF specification + category | Primitive layout | Structural validity; compliance | 1,000 |
| T2a | Representation translation | Primitive layout | binary token | Foreground F1 | 1,000 |
| T2b | Representation translation | Layout image | binary token | Foreground F1 | 1,000 |
| T3a | Response prediction | Primitive layout | Complex -parameter sweep | Complex/physics metrics | 1,000 |
| T3b | Response prediction | Layout image | Complex -parameter sweep | Complex/physics metrics | 1,000 |
| T1a: generation | T1b: constrained generation | T4: ADS validation | |||||||||
| Model | Ports | On metal | Valid | Ports | On metal | Valid | Full spec. | Run | MAE | MAE | Strict |
| AnalogSeeker | 98.5 | 77.7 | 0.0 | 88.8 | 40.2 | 0.0 | 15.1 | 77.2 | 8.94 | 9.00 | 0.0 |
| Claude 4.5 | 100.0 | 88.2 | 0.0 | 100.0 | 77.7 | 0.0 | 38.9 | 87.6 | 8.34 | 8.34 | 0.4 |
| GPT-5.4-mini | 99.9 | 60.0 | 0.0 | 100.0 | 47.8 | 0.0 | 0.0 | 61.6 | 7.39 | 7.38 | 0.8 |
| Llama 3.2-11B | 70.1 | 33.7 | 0.0 | 95.0 | 11.2 | 0.0 | 0.1 | 37.2 | 8.19 | 8.18 | 0.4 |
| Qwen3-VL-8B | 100.0 | 0.2 | 0.0 | 98.6 | 34.8 | 0.0 | 0.8 | 86.8 | 8.31 | 8.33 | 0.0 |
| T2: representation translation | T3: response prediction | T6: structural understanding | ||||||||||||
| Model | Input | Prec. | Rec. | F1 | Exact | C-MAE | dB | Pass. | Recip. | -MAE | Cat. acc. | Cat. F1 | Feat. | Joint |
| AnalogSeeker | primitive | 29.2 | 9.4 | 14.2 | 0.0 | .901 | 13.4 | 98.6 | .0180 | .0437 | 21.6 | 10.9 | 55.7 | 7.4 |
| Claude 4.5 | primitive | 26.9 | 37.6 | 31.4 | 0.0 | .873 | 28.0 | 30.9 | .0004 | .0575 | 51.1 | 46.7 | 61.7 | 18.6 |
| image | 30.4 | 56.1 | 39.4 | 0.0 | .873 | 27.0 | 32.5 | .0510 | .0706 | 48.0 | 46.7 | 66.7 | 12.3 | |
| GPT-5.4-mini | primitive | 36.1 | 23.9 | 28.7 | 0.0 | .796 | 28.4 | 79.4 | .2898 | .0452 | 22.6 | 13.0 | 55.2 | 15.3 |
| image | 48.4 | 47.9 | 48.2 | 0.0 | .714 | 11.6 | 90.0 | .1719 | .0483 | 25.9 | 16.1 | 58.1 | 19.3 | |
| T5: diagnosis/redesign | T7: classification | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Acc. | Prec. | Rec. | Macro-F1 | Solved/ candidates | Improved/ solved | End-to-end ADS improv. (%) | Median (dB) | Acc. | Macro-F1 |
| AnalogSeeker | 85.0 | 81.8 | 98.4 | 82.0 | 63/63 | 46/63 (73.0%) | 71.9 | +4.860 | 7.9 | 7.5 |
| Claude 4.5 | 73.0 | 70.3 | 100.0 | 61.3 | 63/64 | 45/63 (71.4%) | 70.3 | +4.445 | 47.8 | 36.4 |
| GPT-5.4-mini | 89.0 | 94.9 | 87.5 | 88.4 | 56/56 | 43/56 (76.8%) | 67.2 | +4.427 | 25.3 | 16.8 |
| Llama 3.2-11B | 64.0 | 64.0 | 100.0 | 39.0 | 58/64 | 44/58 (75.9%) | 68.8 | +4.766 | 27.9 | 7.3 |
| Qwen3-VL-8B | 56.0 | 64.7 | 68.8 | 51.0 | 44/44 | 31/44 (70.5%) | 48.4 | +5.419 | 16.8 | 14.8 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Input | Real MAE | Imag. MAE | dB MAE | dB RMSE | dB | dB | |
|---|---|---|---|---|---|---|---|---|
| AnalogSeeker | primitive | .720 | .402 | 111.37 | 159.14 | 5.3 | 7.9 | .00213 |
| Claude 4.5 | primitive | .618 | .499 | 6.54 | 9.13 | 8.9 | 16.1 | .15933 |
| image | .622 | .484 | 6.76 | 9.64 | 8.5 | 15.6 | .18334 | |
| GPT-5.4-mini | primitive | .594 | .417 | 9.82 | 14.10 | 8.3 | 16.6 | .00946 |
| image | .476 | .432 | 13.35 | 17.34 | 3.0 | 5.9 | .05732 | |
| Llama 3.2-11B | primitive | .406 | .404 | 101.63 | 140.26 | 0.1 | 0.1 | .00000 |
| Model | Input | dB MAE | dB MAE | dB MAE | dB MAE |
|---|---|---|---|---|---|
| AnalogSeeker | primitive | 6.52 | 219.94 | 212.51 | 6.51 |
| Claude 4.5 | primitive | 8.11 | 4.97 | 4.97 | 8.11 |
| image | 7.99 | 6.11 | 4.97 | 7.99 | |
| GPT-5.4-mini | primitive | 10.93 | 13.52 | 3.80 | 11.03 |
| image | 8.03 | 21.96 | 15.11 | 8.31 | |
| Llama 3.2-11B | primitive | 66.07 | 134.15 | 143.89 | 62.41 |
| ID | Ability | Input | Output | Full corpus | Official test |
| T1a | Design | RF response specification | Primitive layout | 10,000 | 1,000 |
| T1b | Constrained design | RF specification + category | Primitive layout | 10,000 | 1,000 |
| T2a | Representation translation | Primitive layout | binary token | 10,000 | 1,000 |
| T2b | Representation translation | Layout image | binary token | 10,000 | 1,000 |
| T3a | Response prediction | Primitive layout | Complex -parameter sweep | 10,000 | 1,000 |
| T3b | Response prediction | Layout image | Complex -parameter sweep | 10,000 | 1,000 |
| Model | Edge-connected | Microstrip | Rot180 | Symmetric L–R | Symmetric T–B |
|---|---|---|---|---|---|
| AnalogSeeker | 0.0/0.0 | 0.0/0.0 | 50.0/2.0 | 80.0/73.5 | 0.0/0.0 |
| Claude 4.5 | 0.0/0.0 | 0.0/0.0 | 98.8/97.5 | 98.5/97.0 | 44.3/0.0 |
| GPT-5.4-mini | 0.0/0.0 | 0.0/0.0 | 0.0/0.0 | 0.0/0.0 | 0.0/0.0 |
| Llama 3.2-11B | 0.0/0.0 | 0.0/0.0 | 1.0/0.0 | 3.5/0.5 | 6.0/0.0 |
| Qwen3-VL-8B | 0.0/0.0 | 0.0/0.0 | 19.3/0.0 | 18.0/4.0 | 0.3/0.0 |
| Qwen3 SFT | 58.5/58.5 | 100.0/100.0 | 85.5/80.0 | 97.5/95.0 | 97.5/95.0 |
| Model | ID | Ports | On metal | Canvas | Connect | Valid | Redund. overlap | Parse | Meas. mean | Meas. full |
|---|---|---|---|---|---|---|---|---|---|---|
| AnalogSeeker | T1a | 98.1 | 77.7 | 0.0 | 0.0 | 0.0 | 10.9 | .012 | – | – |
| Claude 4.5 | T1a | 100.0 | 88.7 | 0.0 | 0.0 | 0.0 | 2.5 | .000 | – | – |
| GPT-5.4-mini | T1a | 100.0 | 61.5 | 0.0 | 0.0 | 0.0 | 27.7 | .001 | – | – |
| Llama 3.2-11B | T1a | 70.3 | 33.4 | 0.0 | 0.0 | 0.0 | 16.4 | .228 | – | – |
| Qwen3-VL-8B | T1a | 100.0 | 0.1 | 0.0 | 0.0 | 0.0 | 55.8 | .047 | – | – |
| AnalogSeeker | T1b | 87.5 | 40.9 | 0.3 | 34.7 | 0.0 | 12.8 | .010 | 25.4 | 14.7 |
| Model | Input | Prec. | Rec. | F1 | IoU | Hamming | Exact | Area diff. | |
|---|---|---|---|---|---|---|---|---|---|
| AnalogSeeker | primitive | 10000 | 28.7 | 9.7 | 14.6 | 7.9 | 22.6 | 0.0 | 16.5 |
| Claude 4.5 | primitive | 10000 | 26.5 | 38.0 | 31.2 | 18.5 | 33.1 | 0.0 | 11.8 |
| image | 10000 | 30.0 | 55.7 | 39.0 | 24.2 | 34.4 | 0.0 | 17.1 | |
| GPT-5.4-mini | primitive | 10000 | 36.7 | 25.4 | 30.0 | 17.7 | 23.9 | 0.0 | 14.4 |
| image | 10000 | 48.2 | 47.2 | 47.7 | 31.3 | 20.5 | 0.0 | 7.5 | |
| Llama 3.2-11B | primitive | 10000 | 19.8 | 21.0 | 20.4 | 11.3 | 43.5 | 0.0 | 32.0 |
| Model | Input | Real MAE | Imag. MAE | Complex MAE | dB MAE | dB RMSE | 0.5 dB | 1 dB | 2 dB |
|---|---|---|---|---|---|---|---|---|---|
| AnalogSeeker | primitive | .719 | .401 | .899 | 110.96 | 158.74 | 5.5 | 8.2 | 13.8 |
| Llama 3.2-11B | primitive | .405 | .402 | .636 | 99.81 | 138.55 | 0.1 | 0.1 | 0.3 |
| image | .447 | .401 | .673 | 80.09 | 126.44 | 0.8 | 1.6 | 3.1 | |
| Qwen3-VL-8B | primitive | .642 | .401 | .831 | 21.29 | 26.89 | 6.8 | 11.4 | 17.6 |
| image | .526 | .402 | .732 | 32.05 | 50.94 | 3.9 | 7.1 | 11.3 |
| Specification-landmark MAE (dB) | |||||||
|---|---|---|---|---|---|---|---|
| Model | ADS runs | Run rate | All | Strict all dB | |||
| AnalogSeeker | 193/250 | 77.2 | 9.47 | 8.94 | 9.96 | 9.00 | 0.0 |
| Claude 4.5 | 219/250 | 87.6 | 7.26 | 8.34 | 6.19 | 8.34 | 0.4 |
| GPT-5.4-mini | 154/250 | 61.6 | 5.96 | 7.39 | 4.54 | 7.38 | 0.8 |
| Llama 3.2-11B | 93/250 | 37.2 | 10.55 | 8.19 | 12.92 | 8.18 | 0.4 |
| Qwen3-VL-8B | 217/250 | 86.8 | 9.74 | 8.31 | 11.16 | 8.33 | 0.0 |
| T5: improvement diagnosis | T7: response classification | |||||||
| Model | Acc. | Prec. | Rec. | Macro-F1 | Acc. | Macro-F1 | ||
| AnalogSeeker | 500 | 85.8 | 83.8 | 97.9 | 81.8 | 10000 | 8.5 | 7.9 |
| Claude 4.5 | 500 | 76.4 | 74.2 | 100.0 | 63.7 | 10000 | 48.7 | 37.1 |
| GPT-5.4-mini | 500 | 88.8 | 95.5 | 87.6 | 87.7 | 10000 | 24.6 | 16.1 |
| Llama 3.2-11B | 500 | 67.4 | 67.7 | 99.4 | 40.3 | 10000 | 27.8 | 7.2 |
| Qwen3-VL-8B | 500 | 52.4 | 65.3 | 63.4 | 46.3 | 10000 | 17.0 | 14.9 |
| Model | Input | Cat. acc. | Cat. F1 | Mean feat. | Exact feat. | Joint all | Diag. | |
|---|---|---|---|---|---|---|---|---|
| AnalogSeeker | primitive | 10000 | 20.7 | 9.9 | 55.4 | 7.0 | 7.0 | 100.0 |
| Claude 4.5 | primitive | 10000 | 51.4 | 48.0 | 61.8 | 17.7 | 17.7 | 100.0 |
| image | 10000 | 47.6 | 46.6 | 67.0 | 13.9 | 13.0 | 100.0 | |
| GPT-5.4-mini | primitive | 10000 | 22.8 | 13.6 | 55.7 | 15.7 | 15.7 | 100.0 |
| image | 10000 | 25.5 | 15.7 | 58.4 | 19.1 | 19.1 | 100.0 | |
| Llama 3.2-11B | primitive | 10000 | 19.0 | 12.0 | 72.6 | 16.4 | 16.2 | 100.0 |
| Setting | Qwen3 SFT | Qwen3 GRPO |
| Initialization | Qwen3-VL-8B | Qwen3 SFT |
| Objective | Assistant-token CE | Group-relative reward |
| Train volume | 225,000 conversations | 11,000-row pool |
| Optimization | 1 epoch | 1,100 steps |
| Learning rate | ||
| Prompt batch | 1 (accumulation 16) | 2 ( outputs) |
| Task | Metric | SFT | GRPO | Paired 95% CI for | |
|---|---|---|---|---|---|
| T1a | Structural validity | 82.70 | 83.20 | ||
| T1b | Structural validity | 86.00 | 86.40 | ||
| T2a | Foreground F1 | 99.991 | 99.993 | ||
| T2b | Foreground F1 | 99.976 | 99.974 | ||
| T3a | Within-2-dB point rate | 62.56 | 62.80 | ||
| T3b | Within-2-dB point rate | 64.10 | 64.12 |
| (S65) |
| Model | Coverage | Design | Translation | Prediction | Diagnosis | Understanding | Classification | RQS |
|---|---|---|---|---|---|---|---|---|
| AnalogSeeker | 7/10 | 0.0 | 14.2 | 13.4 | 82.0 | 10.9 | 7.5 | 11.7 |
| Claude 4.5 | 10/10 | 0.0 | 35.4 | 27.5 | 61.3 | 46.7 | 36.4 | 31.9 |
| GPT-5.4-mini | 10/10 | 0.0 | 38.5 | 20.0 | 88.4 | 14.5 | 16.8 | 24.1 |
| Llama 3.2-11B | 10/10 | 0.0 | 15.4 | 1.7 | 39.0 | 9.9 | 7.3 | 11.2 |
| Qwen3-VL-8B | 10/10 | 0.0 | 0.8 | 14.4 | 51.0 | 22.1 | 14.8 | 15.5 |
| Qwen3 SFT | 10/10 | 84.3 | 100.0 | 63.3 | 58.6 | 99.1 | 74.0 | 74.2 |