Bookkeeping Is Not Judgement: EJBench, an Engineering Judgement Benchmark Built from a Conjugate Heat Transfer Design Campaign
Organizations: Gradient Dynamics, London, UK · AESC, London, UK
Abstract
We introduce EJBench, a benchmark of engineering judgement: whether a model can predict how a design behaves at new conditions from a few earlier results, and decide on it. Unlike engineering benchmarks that test whether an agent can operate simulation or CAD software, EJBench is software-agnostic and asks what an engineer would conclude without running a simulation. Its tasks come from a conjugate heat transfer design campaign for battery cold plates, in three families: matching unlabelled test results to their operating points, choosing which stored runs to reveal before signing off a design, and ranking new duties by peak cell temperature. Every answer is taken from a converged simulation, not from an LLM judge. The best of ten frontier models, GPT-6 Astra, scores 0.71, and models state almost the same confidence on tasks they get wrong as on tasks they get right. We also post-train Qwen3.5-9B with LoRA and reinforcement learning, rewarded by the share of answer criteria the simulation results confirm; on the held-out test split its score rises from 0.071 to 0.106. We release the train, validation and test splits, the evaluation harness and the leaderboard.
Figures & tables
| Split | Use | Tasks | Scored |
|---|---|---|---|
| train | fine-tuning | 1,281 | – |
| validation | fine-tuning, held-out designs | 133 | – |
| test | benchmark | 272 | 228 |
| Family | Model sees | Model gives | Chance | Test | Scored |
|---|---|---|---|---|---|
| Campaign reconstruction | Five planned test points; six unlabelled result sheets, one misfiled from another plate and some with illegible entries; two held-out duties with temperature limits | Which sheet belongs to each point, which is misfiled, and pass or fail at each duty | 0.0003 | 144 | 100 |
| Evidence sign-off | Inputs of four or more stored runs, results hidden; three duties with temperature and pressure-drop limits; a budget of three reveals | Temperature and pressure pass or fail at each duty | 0.016 | 80 | 80 |
| Operating envelope | Three completed runs with results; four new duties | Duties ranked by peak cell rise, and the heat-balance regime of each | 0.0002 | 48 | 48 |
| Model | Macro | Campaign | Evidence | Envelope | Brier | $/task |
| GPT-6 Astra | 0.710 [0.651, 0.773] | 0.710 | 0.650 | 0.771 | 0.196 | 0.727 |
| GPT-6.1 Sol | 0.689 [0.624, 0.747] | 0.680 | 0.637 | 0.750 | 0.215 | 0.147 |
| Claude Opus 5.5 | 0.641 [0.581, 0.701] | 0.590 | 0.563 | 0.771 | 0.230 | 0.426 |
| GPT-6.1 Terra | 0.588 [0.527, 0.657] | 0.490 | 0.588 | 0.688 | 0.216 | 0.036 |
| Muse Spark 1.3 | 0.550 [0.492, 0.614] | 0.430 | 0.450 | 0.771 | 0.296 | 0.185 |
| Claude Sonnet 5.5 | 0.517 [0.460, 0.579] | 0.540 | 0.263 | 0.750 | 0.254 | 0.559 |
| Campaign | Evidence | Envelope | |||||
|---|---|---|---|---|---|---|---|
| Model | Links | Misfiled | Verdicts | Temp. | Pressure | Ranking | Regimes |
| GPT-6 Astra | 0.83 | 0.87 | 0.83 | 0.75 | 0.86 | 0.90 | 0.88 |
| GPT-6.1 Sol | 0.81 | 0.87 | 0.82 | 0.79 | 0.81 | 0.92 | 0.81 |
| Claude Opus 5.5 | 0.72 | 0.85 | 0.76 | 0.74 | 0.75 | 0.85 | 0.88 |
| GPT-6.1 Terra | 0.63 | 0.73 | 0.74 | 0.70 | 0.78 | 0.85 | 0.83 |
| Muse Spark 1.3 | 0.71 | 0.77 | 0.62 | 0.64 | 0.75 | 0.88 | 0.88 |
| Criteria | Full pass | ||||
|---|---|---|---|---|---|
| Family | Tasks | Base | Trained | Base | Trained |
| Campaign | 100 | 0.397 | 0.434 | 2.2% (9/400) | 1.8% (7/400) |
| Evidence | 80 | 0.639 | 0.648 | 7.5% (24/320) | 9.7% (31/320) |
| Envelope | 48 | 0.546 | 0.656 | 11.5% (22/192) | 20.3% (39/192) |
| Macro | 228 | 0.527 | 0.580 | 7.1% | 10.6% |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | Format | Train | Val |
|---|---|---|---|
| campaign_reconstruction | 5 points | 300 | 30 |
| 4 points | 317 | 32 | |
| operating_envelope | 4 duties | 98 | 10 |
| 3 duties | 53 | 7 | |
| duty_signoff | 1 duty, 3 runs | 269 | 30 |
| 1 duty, 4 runs | 111 | 11 |
| Setting | Value |
|---|---|
| Model | Qwen3.5-9B, thinking on |
| Algorithm | TRL [ von Werra et al., 2020 ] : advantages normalised within each group; sequence-level importance ratio clipped at / (GSPO [ Zheng et al., 2025 ] ); loss averaged over every reply token in the batch (DAPO); KL coefficient 0.02 towards the base model; one update per batch |
| Group and batch | 8 replies per task, 8 tasks per step, 160 steps |
| Optimiser | Adam (0.9, 0.98), learning rate constant, weight decay 0.1, gradient clip 0.1 |
| Adapter | LoRA [ Hu et al., 2022 ] , rank 32, 32, dropout 0, all linear layers outside the vision encoder (86.6M trainable parameters, 0.91%) |
| Sampling | temperature 0.6, top- 0.95, top- 20; replies up to 32,768 tokens |
Benchmark Design
Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.
See all 413 papers