FORM is a domain-specific symbolic manipulation language widely used in particle physics for processing the very large algebraic expressions arising from multi-loop Feynman diagram calculations. Despite its central role in precision theoretical physics, no artificial-intelligence tooling exists, to our knowledge, for assisting physicists in writing FORM code. We show that contemporary large language models (LLMs), including frontier models with hundreds of billions of parameters, achieve a zero-percent execution pass rate on our instruction-following and tutorial-style FORM tasks without documentation in a single attempt, establishing FORM as a genuine zero-shot language for LLMs at the time of writing. We then present a verification-driven data generation pipeline that uses the FORM binary itself as an execution oracle to produce and validate a corpus of 4,633 training examples spanning deterministic computations, open-ended programs, tutorial code, and knowledge question-answer pairs. Fine-tuning a compact open-weights model (Qwen3-8B) with quantized low-rank adaptation (QLoRA) yields a specialist that, evaluated on four complementary benchmarks (840 tasks, single attempt each), decisively outperforms frontier models with up to 756B parameters in execution rate and in strict, FORM-verified output matching on the larger benchmarks, and remains statistically indistinguishable from them on the smaller, harder ones. General reasoning and coding capabilities are preserved within 2.6 percentage points.
Figures & tables
Figure 2: Verification-driven data generation pipeline. Candidate programs are generated from natural-language instructions by a frontier LLM (DeepSeek-V4 Flash, temperature 0.8), then pass through a four-stage verification process before inclusion in the training set.
Parameter
Value
Base model
Qwen3-8B (4-bit NF4 quantisation)
LoRA rank ( r )
16
LoRA alpha ( α )
32
LoRA dropout
0.05
Target modules
All linear layers
Learning rate
2×10−4
Table 1: QLoRA fine-tuning hyperparameters.
Figure 3: Training (left) and validation (right) loss curves for the two model variants: v3b (det-only, 2,465 examples) and v3c (full mix, 4,633 examples). The v3c model converges to a higher validation loss due to task diversity but achieves substantially better performance on open-ended benchmarks (see Section 5 ).
Model
Params
Active
Type
Context
Qwen3-8B (base) [ 37 ]
8B
8B
Dense
Zero-shot
FORM-8B (ours)
8B
8B
Dense + QLoRA
Zero-shot
Qwen3.8-27B [ 27 ]
27B
27B
Dense
+Docs
Gemma 4 [ 14 ]
33B
33B
Dense
+Docs
GPT-OSS-120B [ 1 ]
117B
5.1B
MoE
+Docs
DeepSeek-V4 Flash [ 35 ]
304B
13B
MoE
+Docs
Table 2: Baseline models. “Docs” indicates that the first 6,000 characters of a Form syntax guide are prepended to the instruction. Without documentation, all evaluated frontier models score 0% on Instruct-100, and the base model, DeepSeek-V4 Flash, GPT-OSS-120B and Gemma 4 likewise score 0% on Tutorial-44 (Open-32, whose rubric judge credits partial solutions, is the one exception; see Section 5.1 ). GLM-5.3 shares the GLM-5.2 base model (756B total, 40B active); its gains over GLM-5.2 are post-training only [ 13 ] .
Model
B1
B2: Instruct-100
B3: Tutorial-44
B4: Open-32
(n=664)
Exec.
Strict
Exec.
Strict
Rubric
Strict
(n=100)
(n=89)
(n=44)
(n=41)
(n=32)
(n=22)
This work (zero-shot, no docs)
FORM-8B
97.7
83.0
18.0
43.2
14.6
56.2
50.0
[96.5, 98.8]
[10.1, 25.8]
[4.9, 26.8]
[27.3, 68.2]
Frontier baselines (with documentation)
Table 3: Main results across the four benchmarks, reported as execution rate (“Exec.”: syntax pass for B2/B3, rubric score for B4) and Form -verified strict output-match rate (“Strict”). B1 is strict output match by construction (n=664). The B2 strict rate is computed on the 89 output-determined tasks and is a conservative lower bound (Section 5.3 ); the B4 strict rate on the 22 output-determined tasks (for GPT-OSS-120B and Gemma 4 it is based on the no-docs runs); the B3 strict rate on the 41 adjudicable tasks (three references print no result expression and cannot be verified by output matching). Frontier models are evaluated with documentation; our model is zero-shot . Bold indicates the best score per column. 95% bootstrap CIs for the correctness metrics (B1 and all Strict columns). All entries are single-attempt generations at temperature 0.
Figure 4: Strict Form -verified correctness rates (Table 3 , “Strict” columns; B1 is output match by construction) with 95% bootstrap confidence intervals. Our fine-tuned 8B model (blue) dominates on B1 and on the output-determined B2 tasks; on B3 and B4 the small adjudicable samples (41, 22) produce wide CIs with overlap.
Model
Exec. passed
Solved
Solved/Exec.
GLM-5.2
14 (31.8%)
9 (22.0%)
64%
GLM-5.3
24 (54.5%)
8 (19.5%)
33%
FORM-8B
19 (43.2%)
6 (14.6%)
32%
DeepSeek-V4 Flash
11 (25.0%)
6 (14.6%)
55%
Gemma 4
5 (11.4%)
5 (12.2%)
100%
Qwen3.8-27B
5 (11.4%)
4 (9.8%)
80%
Table 4: Execution success vs. genuine solutions on Tutorial-44. “Exec.” is the syntax-pass rate of Table 3 (n=44); “Solved” is the strict Form -verified output-match rate on the 41 adjudicable tasks (three references print no result expression). The final column is the fraction of a model’s passed tasks that are genuine solutions.
Figure 5: Ablation: effect of data composition. The v3b model (deterministic-only data, 2,465 examples) collapses open-ended capability. The v3c model (full mix, 4,633 examples) recovers it while maintaining deterministic performance.
Figure 6: Data scaling: effect of dataset size and composition. The v1 scaling curve (purple, dashed) shows improvement with more unverified code-only data. The v3c curve (blue, solid) shows the effect of verified, diverse data at 25%/50%/75%/100% of the full dataset. The v3b point (orange square) shows that verified but deterministic-only data collapses open-ended capability despite having more examples than the 50% v3c subset.
Benchmark
Base
FORM-8B
Δ
MMLU (5-shot) [ 18 ]
76.5%
73.9%
−2.6 pp
GSM8K (0-shot, flexible-extract) [ 8 ]
92.0%
90.5%
−1.5 pp
HumanEval (0-shot) [ 7 ]
63.4%
62.8%
−0.6 pp
Table 5: General capability preservation. Changes relative to Qwen3-8B base. MMLU: full 9,183-question test set; GSM8K: 200-problem subsample; HumanEval: all 164 problems.
Figure 7: Capability radar: the fine-tuned model (blue) gains dramatically on Form benchmarks while retaining general capabilities (MMLU, GSM8K, HumanEval) close to the base model (grey).