We introduce EJBench, a benchmark of engineering judgement: whether a model can predict how a design behaves at new conditions from a few earlier results, and decide on it. Unlike engineering benchmarks that test whether an agent can operate simulation or CAD software, EJBench is software-agnostic and asks what an engineer would conclude without running a simulation. Its tasks come from a conjugate heat transfer design campaign for battery cold plates, in three families: matching unlabelled test results to their operating points, choosing which stored runs to reveal before signing off a design, and ranking new duties by peak cell temperature. Every answer is taken from a converged simulation, not from an LLM judge. The best of ten frontier models, GPT-6 Astra, scores 0.71, and models state almost the same confidence on tasks they get wrong as on tasks they get right. We also post-train Qwen3.5-9B with LoRA and reinforcement learning, rewarded by the share of answer criteria the simulation results confirm; on the held-out test split its score rises from 0.071 to 0.106. We release the train, validation and test splits, the evaluation harness and the leaderboard.
Figures & tables
Split
Use
Tasks
Scored
train
fine-tuning
1,281
–
validation
fine-tuning, held-out designs
133
–
test
benchmark
272
228
Table 1: EJBench v0.1 splits, by geometry. test holds 272 tasks; the benchmark score uses a fixed subset of 228 of them (Section 3.3 ), and the other 44, all campaign tasks, are released but not scored. train and validation will be rebuilt as more simulations finish; test is frozen for this version.
Family
Model sees
Model gives
Chance
Test
Scored
Campaign reconstruction
Five planned test points; six unlabelled result sheets, one misfiled from another plate and some with illegible entries; two held-out duties with temperature limits
Which sheet belongs to each point, which is misfiled, and pass or fail at each duty
0.0003
144
100
Evidence sign-off
Inputs of four or more stored runs, results hidden; three duties with temperature and pressure-drop limits; a budget of three reveals
Temperature and pressure pass or fail at each duty
0.016
80
80
Operating envelope
Three completed runs with results; four new duties
Duties ranked by peak cell rise, and the heat-balance regime of each
0.0002
48
48
Table 2: Task families. Chance is the score of uniform random answers. Test is every task in test ; Scored is the subset used for the benchmark score (Section 3.3 ). Counts are for EJBench v0.1.
Figure 2: FluxCore against OpenFOAM for the four task quantities: rises in kelvin, pressure drop as relative error. The grey bands mark the task tolerance.
Model
Macro
Campaign
Evidence
Envelope
Brier
$/task
GPT-6 Astra
0.710 [0.651, 0.773]
0.710
0.650
0.771
0.196
0.727
GPT-6.1 Sol
0.689 [0.624, 0.747]
0.680
0.637
0.750
0.215
0.147
Claude Opus 5.5
0.641 [0.581, 0.701]
0.590
0.563
0.771
0.230
0.426
GPT-6.1 Terra
0.588 [0.527, 0.657]
0.490
0.588
0.688
0.216
0.036
Muse Spark 1.3
0.550 [0.492, 0.614]
0.430
0.450
0.771
0.296
0.185
Claude Sonnet 5.5
0.517 [0.460, 0.579]
0.540
0.263
0.750
0.254
0.559
Table 3: Results on the 228 scored EJBench v0.1 test tasks, ranked by macro score. Macro is the mean over families with a 95% bootstrap interval. Brier is the mean squared difference between stated confidence and score. Cost is per task at list price, including reasoning tokens. API models run at high reasoning effort and answer each task once; the two Qwen3.5-9B ( Qwen Team, 2026 ) rows run with thinking on and average four runs, and the first is the model trained on EJBench (Section 5.4 ). Qwen3.5-9B is self-hosted, so it has no list price. Model reports: Claude Opus 5.5 ( Anthropic, 2026 ) , DeepSeek V4.1 ( DeepSeek-AI, 2026 ) .
Figure 3: EJBench v0.1 leaderboard on the 228 scored test tasks. Bars give the macro score and whiskers its 95% bootstrap interval; colour marks the developer. API models run once at high reasoning effort. The highlighted rows are Qwen3.5-9B before and after training on EJBench (Section 5.4 ), with thinking on, each averaged over four runs.
Figure 4: Macro score against cost per task at list price (log scale). The black line joins the models with the highest score for the cost. Dashed lines mark the medians; the shaded corner holds models that cost more and score less than the median. Qwen3.5-9B is self-hosted and not shown.
Campaign
Evidence
Envelope
Model
Links
Misfiled
Verdicts
Temp.
Pressure
Ranking
Regimes
GPT-6 Astra
0.83
0.87
0.83
0.75
0.86
0.90
0.88
GPT-6.1 Sol
0.81
0.87
0.82
0.79
0.81
0.92
0.81
Claude Opus 5.5
0.72
0.85
0.76
0.74
0.75
0.85
0.88
GPT-6.1 Terra
0.63
0.73
0.74
0.70
0.78
0.85
0.83
Muse Spark 1.3
0.71
0.77
0.62
0.64
0.75
0.88
0.88
Table 4: Share of the 228 scored test tasks in which each part of the answer is fully right.
Criteria
Full pass
Family
Tasks
Base
Trained
Base
Trained
Campaign
100
0.397
0.434
2.2% (9/400)
1.8% (7/400)
Evidence
80
0.639
0.648
7.5% (24/320)
9.7% (31/320)
Envelope
48
0.546
0.656
11.5% (22/192)
20.3% (39/192)
Macro
228
0.527
0.580
7.1%
10.6%
Table 5: Qwen3.5-9B before and after training, on the 228 scored test tasks, four runs each. Criteria is the mean share of criteria met per reply; full pass is the share of replies with every criterion met, with replies in parentheses.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Coolant velocity magnitude on the mid-channel plane z=8.19 mm for verification case 3 (laminar, Re=35 ). (a) FluxCore; (b) OpenFOAM v2512; (c) OpenFOAM minus FluxCore, cell by cell on the shared mesh. The cold plate is grey. (d) ∣U∣ across the 7.31 mm channel marked in (a)–(c), drawn as a smooth curve through the 10 cell-centre values with zero speed at the walls. Peak ∣U∣ on the plane is 0.100 m/s for both solvers. Over all coolant cells the volume-weighted RMS difference is 0.0002 m/s, 0.2% of the peak speed.
Figure 6: Coolant velocity magnitude on the mid-channel plane z=3.75 mm for verification case 11 (laminar, Re=292 ). Panels as in Figure 5 ; (d) is across the 3.40 mm channel, through 11 cell-centre values. Peak ∣U∣ on the plane is 1.245 m/s (FluxCore) and 1.246 m/s (OpenFOAM). Over all coolant cells the volume-weighted RMS difference is 0.0015 m/s, 0.1% of the peak speed.
Figure 7: Coolant velocity magnitude on the mid-channel plane z=5.42 mm for verification case 20 (SST k – ω , Re=2,331 ). Panels as in Figure 5 ; (d) is across the 3.00 mm channel, through 10 cell-centre values. Peak ∣U∣ on the plane is 1.128 m/s (FluxCore) and 1.140 m/s (OpenFOAM). Over all coolant cells the volume-weighted RMS difference is 0.0073 m/s, 0.6% of the peak speed.
Figure 8: Temperature on the horizontal plane z=77.7 mm, at mid-height of the battery cells, for verification case 3 (laminar, Re=35 ). Cells and potting are shown, with cell outlines in grey. (a) FluxCore; (b) OpenFOAM; (c) OpenFOAM minus FluxCore. (d) Temperature along the dashed line y=315.2 mm, through the hottest point on the plane. Plane maximum: 288.61 K (FluxCore), 288.65 K (OpenFOAM). Over all battery-cell mesh cells, the volume-weighted mean difference is +0.014 K and the RMS difference 0.016 K.
Figure 9: Temperature on the outer surfaces of the battery cells (621,777 mesh faces) for verification case 8 (laminar, Re=103 ), viewed from azimuth −58∘ and elevation 28∘ . (a) FluxCore; (b) OpenFOAM; (c) OpenFOAM minus FluxCore. (d) Area-weighted distribution of (c): mean −0.001 K, RMS 0.014 K, 5th to 95th percentile −0.017 to +0.025 K.
Family
Format
Train
Val
campaign_reconstruction
5 points
300
30
4 points
317
32
operating_envelope
4 duties
98
10
3 duties
53
7
duty_signoff
1 duty, 3 runs
269
30
1 duty, 4 runs
111
11
Appendix
Table 6: The released training set ( training-0.1 ), tasks by format. It is a subset of a pool of 4,459 training and 446 validation tasks: multi-step tasks and tasks a quick method solves (for envelopes, ranking by one input or a quick estimate; for sign-offs, the closest run alone) are left out, and each design keeps at most three campaign or envelope tasks and one sign-off task per format (seed 20261109).
Setting
Value
Model
Qwen3.5-9B, thinking on
Algorithm
TRL [ von Werra et al., 2020 ] : advantages normalised within each group; sequence-level importance ratio clipped at 3×10−4 / 4×10−4 (GSPO [ Zheng et al., 2025 ] ); loss averaged over every reply token in the batch (DAPO); KL coefficient 0.02 towards the base model; one update per batch