Parallel mechanisms pose distinctive challenges to large language models (LLMs), as valid reasoning depends jointly on mechanism topology, loop closure, geometry, joint types, and modeling assumptions. Yet existing LLM benchmarks provide little systematic coverage of such domain-specific knowledge, constraint-sensitive analysis, and constructive design. To fill this gap, we introduce MECHBENCH, a 2,541-question benchmark for evaluating LLMs in parallel mechanisms across professional knowledge, mechanism analysis, and design. MECHBENCH contains standard and challenge tiers spanning six capability categories and nine executable analysis and design tasks. It is constructed through MECHFORGE, a domain-grounded framework that combines expert-organized knowledge, LLM-assisted question generation, programmatic instance generation, two-stage expert review, and task-aware scoring with exact matching, rubric-based evaluation, and program verification. We further develop MECHGPT as a diagnostic domain-adapted model. Experiments show that strong knowledge performance does not directly translate into executable mechanism reasoning: leading closed-service models achieve over 95 on the standard tier but only 47-60 on the challenge tier, while domain adaptation substantially improves MechGPT over its Qwen3-8B baselines but leaves a large gap in constrained analysis and design. These results establish MECHBENCH as a systematic testbed for measuring domain-specific LLM capabilities in mechanism science.
Figures & tables
Figure 1: MechForge workflow. Sources feed LLM-assisted knowledge organization and evidence-grounded or programmatic question generation. LLM checking and two-stage researcher review confirm conditions, answers, labels, and rubrics before tier assignment; reviewed QA can also form instruction data. Domain pretraining and QLoRA adapt MechGPT, which is evaluated by MechBench.
Figure 2: Knowledge branch linking coupled and configurable architectures to motion properties and screw-, group-, constraint-, and model-based analysis.
(a) Benchmark tiers
All questions: N=2,541
Tier
Questions
Share (%)
Development
200
7.87
Standard
1,791
70.48
Challenge
550
21.65
Total
2,541
100.00
Table 1: Composition of MechBench .
Figure 3: Mechanism illustrations for the nine challenge tasks, spanning analysis, modification, completion, and constrained design.
Type
Count
Example question
Single-choice
870
Before loop closure, a parallel mechanism has 10 independent joint-motion parameters; closure makes 7 dependent. What is its mobility? A. 3; B. 7; C. 10; D. 17.
Multiple-choice
466
Which methods derive the platform-velocity Jacobian? A. Loop equations; B. Reciprocal screws; C. Plücker coordinates; D. Limb counting alone.
Fill-in
358
An ideal spherical joint permits relative rotations about its center. It has relative rotational degrees of freedom.
Constructed response
297
What defines a general screw? Which factors remain for a normalized unit screw?
Analysis
333
Three UP limbs with disjoint links connect the base and platform… Report first- and second-order mobility, output and self-motion mobility, and an output-twist basis.
Design
217
A platform must satisfy Cξ=0 without extra locking… Infer its motion and design a four-limb mechanism with geometry and motion-screw justification.
Table 2: Question formats, counts, and abbreviated example stems.
Figure 4: Scoring routes: exact matching, item rubrics, or program checks, with scores normalized to 0–100.
Model
Historical
Expert
Δ
Multiple-choice
Fill-in
Open-ended
GPT-5.5
93.07
92.43
−0.64
98.40
82.20
96.70
GPT-5.6-Sol
94.80
91.07
−3.73
96.40
79.00
97.80
GPT-5.6-Terra
93.60
90.47
−3.13
98.00
78.20
95.20
GPT-OSS-20B
73.20
70.43
−2.77
82.00
56.20
73.10
Qwen3-8B Chat
61.06
57.37
−3.69
62.00
44.40
65.70
Qwen3-8B Base
42.94
43.13
+0.19
69.60
19.00
40.80
Table 3: Expert recheck on the same 150 questions per model (50 per format); Δ is expert minus historical mean.
Figure 5: Verifier pass rates on paired mobility ( n=214 ) and endpoint-trajectory ( n=35 ) questions.
Figure 6: Challenge-task shares and normalized means; Qwen3-8B Base and Qwen3-8B Chat are shown with their full model names.
Capability
GPT-5.5
Sol
Terra
GPT-OSS 20B
Base
Chat
MechGPT
K1 (n=196)
88.6
88.4
87.0
69.79
41.1
51.8
69.9
K2 (n=258)
93.7
95.3
94.9
83.58
43.5
50.2
68.8
K3 (n=790)
97.6
97.3
96.9
81.05
39.8
45.4
59.0
K4 (n=320)
96.7
96.9
94.6
85.99
38.7
49.8
61.8
K5 (n=133)
98.2
98.6
98.9
87.33
54.4
66.4
61.1
K6 (n=94)
94.9
98.5
96.2
71.49
40.3
53.0
60.6
Table 4: Standard capability means (0–100; n=1,791 ). Base/Chat denote Qwen3-8B. GPT-OSS 20B values use the supplied capability-level means.
Check
Passed
Check
Passed
Payload parsing
35/35
Target mobility/output
35/35
Topology constraints
35/35
Finite continuation
35/35
Initial closure
35/35
Pose trajectory
35/35
Independent constructions
10/10
Table 5: Feasibility checks for specified-trajectory design references.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Challenge response visualizations for categories 1–2.
Figure 8: Challenge response visualizations for categories 3–4.
Figure 9: Challenge response visualizations for categories 5–6.
Figure 10: Challenge response visualizations for categories 7–8.