Organizations: College of Engineering, Ocean University of China, Qingdao, 266100, China · School of Mechanical Engineering and Automation, Beihang University, Beijing, 102206, China
Numerical simulation is important in ocean engineering, but model construction requires expert knowledge. Large language models support automated modelling, but code syntax and successful execution alone do not establish whether generated models and results meet engineering requirements. We present OceanCoder, a Qwen3-8B-based large language model for automated simulation modelling in ocean engineering. To address specialised programming requirements and scarce training data, we develop an extensible data construction pipeline producing 12,000 verified supervised fine-tuning (SFT) examples across 12 engineering task categories. A three-layer verification framework checks model inputs against engineering parameters and task constraints, verifies completion of required simulation analyses, and assesses the engineering plausibility of computed outputs. Its outcomes provide graded rewards for group relative policy optimisation (GRPO), guiding the model towards engineering task completion. We also construct the first benchmark for automated modelling in ocean engineering, covering out-of-distribution tasks and real-world engineering cases. Experiments and ablation studies demonstrate the effectiveness of the data pipeline, verification framework and reward design. OceanCoder outperforms both base models and the closed-source comparison, with Pass@1 increasing from 0% for the base model to 46.77% after SFT and 51.43% after GRPO. These results support engineering verification as a training signal for automated simulation modelling.
Figures & tables
Figure 1: The verification gap between executable code and complete engineering task acceptance. In a two-point lowering task involving a 65 t payload, the reference and OceanCoder programs after SFT both execute, but their line properties and seabed-contact responses differ. At t=35.50 s, the reference retains 32.2 mm clearance while the SFT model has 12.5 mm penetration. The predicted first-contact times differ by 0.9301 s, exceeding the 0.010 s task tolerance. Here, d denotes line diameter and EA axial stiffness; the reference is a checked simulation solution.
Figure 2: Overview of the OceanCoder framework for engineering-constrained data construction, supervised fine-tuning, online GRPO and independent evaluation.
Group
ID
Engineering task
Static
A1
Single-catenary static equilibrium
analysis
A2
Four-point mooring static equilibrium
A3
SCR static configuration and touchdown
A4
Pipeline lateral stability
Dynamic
B1
Single catenary under regular waves
response
B2
Coupled four-point mooring dynamics
Table 1: Task coverage of the supervised training corpus (1,000 pairs per category; 12,000 pairs across 12 categories). SCR denotes a steel catenary riser.
Figure 3: Engineering-constrained data construction pipeline for expanding expert-designed seed cases, generating task descriptions and reference programs from shared parameters, and verifying examples before training.
Input group
Parameter (unit)
Corpus range
Engineering scope and coupling constraints
Geometry
Water depth, h ( m )
15–1,975
Shallow-to-intermediate pipelines, moorings and installation tasks (15–380 m); deepwater steel catenary risers (450–1,975 m).
Total line / pipeline length, L ( m )
45–3,015
Scaled with water depth and hang-off geometry across lifting slings (45–75 m), catenary moorings and pipelay (130–1,031 m) and deepwater SCRs (920–3,015 m).
Structure
Outer / nominal diameter, d ( mm )
44–610
Coupled within component property packages: hoisting wires and subsea cables (44–160 mm), mooring chains and ropes (70–360 mm) and offshore steel pipelines/risers (219.1–610 mm).
Bare unit mass, mbare ( kg/m )
6.5–366.2
Bare structural mass across cables (6.5–60 kg/m), mooring lines (52–286.6 kg/m) and bare steel pipes (70.8–366.2 kg/m); in coated pipeline stability tasks (A4), adding concrete weight coating yields total coated mass mcoated of 109.7–1,393.7 kg/m.
Axial stiffness, EA ( MN )
60–10,049
Assigned by structural member type to preserve realistic axial elongation across slings and cables (60–400 MN), mooring lines (82–1,674 MN) and thick-walled steel pipes (1,894–10,049 MN).
Environment
Wave height, H / Hs ( m )
0–8.0
Still water (0 m) in static equilibrium and controlled lowering analyses; regular waves (1.08–8.0 m) and irregular JONSWAP seas ( Hs=2.52 –3.30 m) in dynamic tasks.
Table 2: Marginal ranges of principal physical, environmental and operational parameters across the 12,000-pair SFT training corpus (1,000 pairs per category). Ranges reflect constrained within-category variations and coupled component property packages rather than independent Cartesian combinations.
Component
Original seed: A1, seed_01
Expanded pair: A1_seed_01_v001
Task
Construct one catenary, solve static equilibrium, save the model and extract the specified responses.
The same task family and analysis and output requirements, with expanded physical inputs.
Geometry
Water depth 80 m; fixed A at (0,0,−8) m; anchored B at (110,0) m and zero height above the seabed; line length 145 m.
Water depth 100 m; fixed A at (0,0,−12) m; anchored B at (145,0) m and zero height above the seabed; line length 200 m.
Default contact diameter (inherited from outer diameter) and standard axial compression; target segment length 2 m.
Parameterized contact diameter (306.0 mm) and compression-limit switch activated within the seed’s admissible parameter space; target segment length 2 m.
Common conditions
Still water, zero current, flat elastic seabed and normal stiffness 1000kN/m3 . Unspecified inputs retain the stated empty-model defaults.
Table 3: Comparison of an original A1 seed ( seed_01 ) and an expanded supervised training instance ( A1_seed_01_v001 ). English summaries retain the exact physical inputs of the Chinese task statements.
Figure 4: Verification-guided GRPO training in OceanCoder. Four candidate programs are generated for each engineering requirement. Execution and engineering verification provide graded rewards for policy updates.
Reward
Established outcome
0
Format, syntax or static safety checks fail.
0.1
Code checks pass, but execution, file delivery or model reloading fails.
0.4
Execution, delivery and reloading pass, but engineering acceptance is incomplete.
1
All applicable engineering acceptance checks pass.
Table 4: Four-level rewards based on program execution and engineering task acceptance.
Symbol
Meaning
pi,t
Token t of candidate program i .
qi,t
Prompt and preceding tokens of that candidate.
πθ , πold
Current and sampling policies, respectively.
πref
Fixed SFT reference policy.
ri , Ai
Candidate reward and group-centred advantage.
ρi,t , ρi,t
Unclipped and clipped probability ratios.
Table 5: Notation used in the group-relative policy update.
Task stratum
Tasks
Share (%)
Core parameter variation
352
52.93
Structural variation
237
35.64
Engineering context
76
11.43
Total
665
100.00
Table 6: Composition of the 665-task evaluation set.
Source class
Tasks
Examples of source context
Relation to the evaluated task
Official examples, documentation, validation or user material
58
Moorings, production risers, deployment, contact and line feeding.
Scenario and modelling-procedure basis, formulated with explicit task parameters and required outputs.
Self-contained simulation specifications adapted from published offshore project descriptions and operating conditions.
Total
76
Source-card and case mappings in Appendix E .
Native simulation references and task-specific acceptance checks.
Table 7: Provenance of the 76 engineering-context evaluation tasks. Public sources provide engineering scenario and modelling context for the self-contained benchmark specifications.
Setting
Stage I: SFT
Stage II: GRPO
Initial model
Qwen3-8B
SFT (update 2,250)
Training data
12,000 pairs (12 cats.)
150 requirements
Schedule
5 epochs (3,750 steps)
3 epochs (57 steps)
Learning rate
2×10−4
5×10−6
Batching
1/GPU, grad. accum. 16
8 prompts/batch
Sampling
–
G=4 , T=1.0 , p=0.95
Table 8: OceanCoder training and evaluation settings.
Model
ESR (%)
Pass@1 (%)
Qwen3-8B
0.00
0.00
Qwen3-235B
0.00
0.00
ChatGPT-6 Sol
27.52±1.52
7.82±1.27
OceanCoder-SFT
69.17±1.40
46.77±1.08
OceanCoder-GRPO-Binary
67.37±1.63
46.47±1.43
OceanCoder-GRPO
70.53±1.08
51.43±0.94
Table 9: Overall and stratified performance on the 665-task benchmark. ESR denotes execution success rate, and Pass@1 measures complete task success. Values with standard deviations are reported as mean ± sample standard deviation across three evaluation rounds. Stratum sizes are given in parentheses.
Figure 5: Complete task success on the 665-task benchmark. (a) Overall Pass@1 for the compared models. (b) Pass@1 by task stratum. Panels (a) and (b) report means across three evaluation rounds. GPT-6 Sol denotes ChatGPT-6 Sol. (c) Category-level net increases in accepted tasks from SFT to GRPO in the first evaluation round. The gains sum to 31 tasks.
Verification criterion
FA
FR
Agree. (%)
Execution checks only
22
0
78.00
Execution and model-input checks
6
0
94.00
Unnormalised raw output comparison
0
13
87.00
Full three-layer verifier
0
0
100.00
Table 10: Verification decisions compared with manual inspection of 100 programs (48 valid, 52 invalid). False acceptances (FA) and false rejections (FR) are reported as counts.
Figure 6: Engineering verification outcomes during GRPO training. (a) Mean reward at each update and its exponential moving average with α=0.10 . (b) Reward composition across the three training epochs, with 584 eligible candidates per epoch. Full acceptance receives a reward of 1. Intermediate rewards distinguish execution or delivery failures from executable programs that fail engineering acceptance.
Figure 7: Failure and acceptance outcomes for SFT and four-level GRPO across task strata in the first evaluation round. Each bar partitions the assigned tasks by the earliest failed stage or complete acceptance. The stages cover code-contract compliance, execution, delivery and model reloading, and subsequent engineering acceptance. Parentheses indicate stratum sizes.
Figure 8: Reference solutions and OceanCoder outputs for three representative engineering tasks. (a) Top effective tension in the steel catenary riser task. (b) Horizontal restoring forces at four vessel offset directions in the mooring task. (c) Peak upward seabed contact force in the two-point lowering task introduced in Figure 1 . Syntax and execution failures are marked explicitly. The shaded band in panel (c) indicates the ±5% task tolerance around the reference force.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Character-length distributions of the 12,000 SFT task requirements and target Python programs.
Resource
Records
Character lengths (mean; median; range)
SFT requirements
12,000
530.95; 530; 321–804
SFT programs
12,000
9,608.77; 9,101; 2,420–20,976
GRPO requirements
150
512.90; 502; 395–717
Appendix
Table A.1: Corpus records and character lengths (mean; median; range). GRPO requirements contain no supervised response.
Cat.
Primary physical and numerical verification checks
A1
Single-catenary static equilibrium, line-property and boundary coordinates, end tensions, minimum elevation and maximum curvature.
Coupled four-point mooring dynamics under waves and current, floater trajectory and fairlead tension envelopes.
Appendix
Table A.2: Admission verification checks across the 12 SFT categories (1,000 pairs per category).
Figure A.2: SFT response-token loss trajectory through the selected epoch-3 checkpoint (update 2,250). All 225 logged loss values from updates 10 to 2,250 are shown on a logarithmic scale; the dashed line marks update 2,250, where the loss stabilises at 0.0022.
Figure A.3: Additional GRPO training diagnostics. The left panel separates equal-reward and mixed-reward usable groups by epoch; equal-reward groups have zero centred task advantage. The right panel shows recorded failure labels for the 363 reward-0.4 candidates.
Epoch
r=0
0.1
0.4
1
Mean
Full (%)
Part. (%)
1
1
17
139
427
0.8293
73.12
26.71
2
1
7
116
460
0.8683
78.77
21.06
3
0
3
108
473
0.8844
80.99
19.01
Total
2
27
363
1360
0.8607
77.63
22.26
Appendix
Table A.3: GRPO candidate reward counts by training epoch ( n=584 eligible per epoch; 1,752 total). Partial denotes r∈{0.1,0.4} .
Recorded label
Candidates
%
Model settings
250
68.87
Analysis process
3
0.83
Engineering conditions
0
0.00
Result/report consistency
108
29.75
Unclassified numerical difference
2
0.55
Appendix
Table A.4: Verification failure tags for the 363 GRPO training candidates with reward 0.4. Percentages use 363 as the denominator.
Core
Structural
Engineering
Cat.
n
SFT
GRPO
n
SFT
GRPO
n
SFT
GRPO
A1
28
25
27
20
0
0
10
0
0
A2
30
10
14
20
0
0
11
0
0
A3
30
18
27
20
0
0
10
1
3
A4
30
27
27
17
9
9
–
–
–
B1
30
26
26
20
0
0
8
0
0
Appendix
Table B.1: Complete task acceptance by category and stratum in the first evaluation round (SFT vs. GRPO accepted counts).
Stratum
Policy
n
Syn.
Code
Exec.
Del.
Pass
All
SFT
665
618
618
463
460
313
GRPO
665
622
610
471
471
344
Core
SFT
352
348
348
335
334
286
GRPO
352
347
347
344
344
314
Structural
SFT
237
209
209
126
125
26
GRPO
237
211
203
123
123
27
Appendix
Table C.1: Cumulative stage pass counts in the first evaluation round across syntax (Syn.), strict code (Code), execution (Exec.), delivery/reload (Del.) and complete acceptance (Pass).
Table E.1: Archived source catalogue for the 76 engineering-context tasks. Counts indicate the number of tasks citing each source; one task may cite multiple sources.
Case ID
Source IDs
Case ID
Source IDs
Case ID
Source IDs
RE2-001
S03, S07
RE2-027
G11
RE2-053
S13, S24
RE2-002
S03
RE2-028
G11, G30
RE2-054
S13
RE2-003
S03
RE2-029
G15
RE2-055
S13
RE2-004
S03
RE2-030
G03, G26
RE2-056
S13
RE2-005
S03, S07
RE2-031
G16
RE2-057
S13
RE2-006
S03
RE2-032
G04
RE2-058
S13, S24
Appendix
Table E.2: Case-to-source mapping for the 76 engineering-context tasks (Table E.1 ).
Seed
Physical specification and requested analysis
A1 seed_01
Water depth 80 m; ends (0,0,−8) and (110,0,−80) m; length 145 m; d=90 mm; m=95 kg/m; EA=800 MN; 2 m segments. Still-water statics: end tensions, minimum elevation and maximum curvature with arc lengths.
B1 seed_01
Water depth 80 m; 145 m chain ( d=80 mm); regular wave ( H=2 m, T=8 s, 0∘ ); friction 0.50. Dynamic simulation (1 build-up + 15 steady periods): peak top tension and maximum horizontal excursion.
C4 seed_02
Water depth 80 m; 6×4×3 m payload (65 t, 24 m 3 ); two slings (2.5 m spacing); 3 s hover + 2 m payout at 0.06 m/s ( Δt=0.005 s); seabed stiffness 1,000/300 kN/m 3 . Outputs: first-contact time, peak force, penetration and line tensions.
Appendix
Table E.3: Key specifications of three representative seed examples (A1, B1 and C4).
Case
Quantity
Ref.
SFT
GRPO
RE2-018
Top tension (kN)
2,173.42
—
2,173.42
A2-B06
Force at 22.5∘ (kN)
−1,026.96
≃0
—
Force at 67.5∘ (kN)
−622.85
≃0
—
Force at 112.5∘ (kN)
−622.85
≃0
—
Force at 157.5∘ (kN)
−1,026.96
≃0
—
C4-039
First contact (s)
36.0126
35.0825
36.0126
Appendix
Table E.4: Recorded responses for the three representative cases (— denotes failure; C4-039 denotes C4-FC1000-seed_02-039).
Marine and offshore engineering text is spread across preprints, open-access journals, encyclopedias and institutional documents, and the amount of it is small by pretraining standards. We ask how far a lightweight language model for marine and offshore engineering can be taken with parameter-efficient adaptation on a single workstation-class device. Starting from the 4-billion-parameter OceanGPT-basic-4B-Instruct, we train two independent LoRA adapters, each updating 33.0 million of 4.06 billion parameters (0.81%). Training took 4 hours 40 minutes for the first adapter and 10 minutes for the second on one NVIDIA DGX Spark. The first adapter is trained by next-token prediction on a 61.6-million-token corpus covering ten engineering themes. The corpus was built from 13,879 source documents, of which 7,069 were retained after deduplication and a two-tier keyword filter that removes cross-domain lexical collisions. Its validation loss falls from 2.082 to 1.976 over 1,500 steps. Direct testing showed that this adapter could still drift off topic on a retrieval-grounded ballast-water prompt, so we trained a second adapter on 2,295 question-answer pairs (114 hand-written and 2,181 generated by the base model from corpus passages) with the loss restricted to answer tokens. On 20 held-out prompts scored by hand for accuracy, topicality and specificity (maximum 120), the plain base model, the continuation adapter, the instruction adapter and the instruction adapter with retrieval scored 42, 56, 84 and 88. Absolute accuracy stays low (14 of 40 without retrieval and 20 of 40 with it). We report the full pipeline, two negative results (a reverted keyword fix and an instruction run that overfit within 100 steps), and the limits of the evaluation, which uses one small prompt set, no inter-rater agreement measure and model-generated reference answers.
Cheng Siong Chin
Faculty of Science, Agriculture and Engineering, Newcastle University Singapore
Large language models (LLMs) have demonstrated a strong ability to generate syntactically correct code from natural-language specifications. In this study, we explore how LLMs can be harnessed to automatically translate a neutral graph representation of fluid system models into executable code for two widely adopted simulation environments: the Python library WNTR and the Modelica Standard Library. We conduct a systematic comparison of ten state-of-the-art LLMs and six prompting strategies that differ in the contextual information supplied (e.g., code or documentation). For each configuration we assess the generated code using a suite of software-quality metrics and we validate the functional fidelity of the resulting simulation models by reproducing benchmark fluid system scenarios. Our findings offer concrete guidance for researchers and engineers seeking to integrate LLM-driven code synthesis into model-based design pipelines. While the best-performing configurations achieve acceptable syntactic quality, we observe substantial gaps remain in simulation fidelity.
Jan Marius Stürmer, Jascha Knack, Tobias Koch +1
German Aerospace Centre (DLR), Institute for the Protection of Terrestrial Infrastructures, Rathausallee 12, 53757, Sankt Augustin, Germany · Algorithms for Computer Vision, Imaging and Data Analysis, Technische Hochschule Würzburg-Schweinfurt, Ignaz-Schön-Straße 11, Schweinfurt, 97421, Germany
Large language models can generate scientific simulation code, but the generated code silently fails on most non-textbook problems. We show that classical mathematical validation -- well-posedness, convergence, and error certification -- can be fully automated by a Judge Agent, reducing the silent-failure rate from 42% to 1.5% across 134 test cases spanning 12 scientific domains. The headline result comes from a prospective benchmark: 72 blinded tasks submitted by 12 independent scientists yield an 89% success rate (95% CI: [80%, 95%]) with automated error bounds, versus 53% without the Judge. On clinical CT (the only powered experiment, n = 200), the pipeline reaches 99% of expert quality. The residual 1.5% concentrates at bifurcation points where certifiability breaks down. We formalize this boundary through the simulability class S and introduce spec.md, a structured specification format that makes any scientific computation problem machine-readable and solver-independent. Code, data, and all 72 benchmark tasks are publicly archived.