Large language models (LLMs) demonstrate strong code generation abilities in general-purpose programming languages but remain limited in specialized domains such as low-level embedded systems programming. This domain involves hardware register manipulation, vendor-specific SDKs, real-time operating system APIs, and hardware abstraction layers that are underrepresented in standard pretraining corpora. We introduce H2LooP Spark Preview, a continual pretraining (CPT) pipeline that adapts the OLMo-3-7B-a fully open language model to the embedded systems domain using BF16 LoRA with rank-stabilized scaling on 8 NVIDIA H100 GPUs. Our training corpus is constructed from repository-datasheet pairs covering 100B tokens of raw embedded systems data across 117 manufacturers, processed using the hierarchical datasheet-to-code mapping approach proposed in SpecMap (Nipane et al., 2026). The resulting curated dataset split contains 23.5B tokens across 13 embedded domains. Continual pretraining with high-rank LoRA (r=512) yields substantial gains, reducing in-domain perplexity by 70.4% and held-out repository perplexity by 66.1%. On generative code completion benchmarks spanning 13 embedded domains, our 7B model outperforms Claude Opus 4.6 and Qwen3-Coder-30B on 8 categories in token accuracy, showing that targeted continual pretraining enables smaller open-weight models to rival frontier systems on specialized technical tasks. We release the production training checkpoint on Huggingface as an open-source artifact.
Table 6: Fixed Hyperparameters Across All Sweep Configurations
Figure 2: Bayesian Hyperparameter Sweep: Training dynamics for 10 configurations. Each curve represents a unique (rank, target, LR) triple. Top-left: Training loss stratified by rank. Top-center: Learning rate schedules. Top-right: Post-clip gradient norms. Bottom: Global step and epoch progression. Labels encode rank ( r ), target (A = attention-only, F = full), and learning rate.
Configuration
r
Target
η
Init Loss
Final Loss
Min Loss
Reduction
r512-F-3.5e-5
512
Full
3.45×10−5
1.426
1.015
0.961
28.8%
r512-A-3.5e-5
512
Attn
3.45×10−5
1.419
1.034
0.981
27.1%
r256-F-5e-5
256
Full
5.00×10−5
1.418
1.068
1.009
24.7%
r256-A-5e-5
256
Attn
5.00×10−5
1.420
1.078
1.024
24.1%
r256-F-3.5e-5
256
Full
3.45×10−5
1.421
1.095
1.038
22.9%
r256-A-3.5e-5
256
Attn
3.45×10−5
1.422
1.109
1.054
22.0%
Table 7: Bayesian Sweep Results
Axis
Levels
Mean Final Loss
Δ (absolute)
LoRA rank
r=128
1.175
0.150 (128 → 512)
r=256
1.087
r=512
1.025
Target modules
Attention-only
1.115
0.010
Full (attn+MLP+embed)
1.105
Learning rate
3.45×10−5
1.105
0.013
Table 8: Marginal Effect of Each Hyperparameter Axis on Final Loss
Configuration
r
Target
η
Peak Grad Norm
Mean Grad Norm
r256-F-5e-5
256
Full
5.00×10−5
28.46
5.73
r128-F-3.5e-5
128
Full
3.45×10−5
12.47
3.22
r256-F-3.5e-5
256
Full
3.45×10−5
8.37
3.91
r512-F-3.5e-5
512
Full
3.45×10−5
5.40
4.30
r128-F-5e-5
128
Full
5.00×10−5
4.37
2.14
r512-A-3.5e-5
512
Attn
3.45×10−5
3.11
2.22
Table 9: Gradient Norm Statistics Across Sweep Configurations
Hyperparameter
Sweep Finding
Production Choice
LoRA rank
r=512 best ( Δ=0.150 vs r=128 )
r=512
Target modules
Full > attn-only ( Δ=0.010 )
Full (all 9 modules)
Learning rate
3.45×10−5 more stable
1.5×10−5 (further reduced)
Gradient clipping
Spikes up to 28.5 observed
max_grad_norm = 5.0
Batch size
128 (sweep)
256 (production, 8 × H100)
Training steps
150 (sweep)
45,000+ (production)
Table 10: Mapping from Sweep Findings to Production Configuration
Metric
Value
Total configurations
10
Steps per configuration
150
Mean runtime per configuration
1.56 hours
Total GPU-hours (A100)
15.6 hours
Fraction of production compute
∼ 5.3%
Hyperparameter axes explored
3 (rank, target, LR)
Table 11: Computational Cost of the Bayesian Hyperparameter Sweep
Figure 3: Training Loss — Training cross-entropy loss as a function of optimizer steps for the hero run. The raw loss (light trace) and smoothed loss (bold trace, window=20) both exhibit monotonically decreasing behavior from 0.88 to 0.25 with no instabilities throughout the entire 294.7-hour run.
Step
Approx. Tokens
Loss
Token Accuracy
Entropy
10
5.0M
0.881
85.8%
–
280
147M
0.827
86.0%
–
1,000
0.5B
0.583
87.7%
0.446
2,000
1.0B
0.503
88.7%
–
5,000
2.6B
0.399
90.0%
–
10,000
5.2B
0.324
92.0%
–
Table 12: Training Progress (From W&B Logs)
Figure 4: Hero Run LR Schedule and Token Accuracy — Left: Cosine LR schedule with 10% warmup. Right: Mean token accuracy climbing monotonically from 86% to 93.5%.
Parameter
Value
Rationale
LoRA rank ( r )
512
Optimal rank-performance tradeoff (Section 5 )
LoRA alpha ( α )
1,024
Standard 2x rank scaling
LoRA dropout
0.05
Mild regularization
RSLoRA
Enabled
Stable scaling for high-rank adapters
Target modules
All (attn + MLP + embed)
Full targeting optimal (Section 5 )
Table 13: LoRA Configuration
Figure 5: Throughput — Left: tokens per second, showing a sharp transition from ∼ 11,000 tok/s (steps 0-6,000) to ∼ 7,000 tok/s (steps 7,000+). Right: step time showing the corresponding increase from ∼ 45s to ∼ 70s per step.
Phase
Steps
Avg Tokens/sec
Avg Step Time
Regime
Phase 1 (early)
0–6,000
10,945
45.5s
Original launch
Phase 2 (resumed)
7,000–16,440
7,078
70.3s
Post-checkpoint resume
Overall weighted
0–16,440
8,644
60.3s
Table 14: Throughput by Training Phase (From W&B Logs)
Figure 6: Perplexity Convergence — Weighted perplexity on in-domain and held-out data across training. Both curves show rapid initial improvement followed by steady convergence.
Recent code large language models have achieved remarkable progress on general programming tasks. Nevertheless, their performance degrades significantly in industrial scenarios that require reasoning about hardware semantics, specialized language constructs, and strict resource constraints. To address these challenges, we introduce InCoder-32B (Industrial-Coder-32B), the first 32B-parameter code foundation model unifying code intelligence across chip design, GPU kernel optimization, embedded systems, compiler optimization, and 3D modeling. By adopting an efficient architecture, we train InCoder-32B from scratch with general code pre-training, curated industrial code annealing, mid-training that progressively extends context from 8K to 128K tokens with synthetic industrial reasoning data, and post-training with execution-grounded verification. We conduct extensive evaluation on 14 mainstream general code benchmarks and 9 industrial benchmarks spanning 4 specialized domains. Results show InCoder-32B achieves highly competitive performance on general tasks while establishing strong open-source baselines across industrial domains.
Jian Yang, Wei Zhang, Jiajun Wu +25
Beihang University · IQuest Research · Shanghai Jiao Tong University +6
Large language models (LLMs) have demonstrated strong capabilities in generating executable code from natural language descriptions. However, general-purpose models often struggle in specialized programming contexts where domain-specific libraries, APIs, or conventions must be used. Customizing smaller open-source models offers a cost-effective alternative to relying on large proprietary systems. In this work, we investigate how smaller language models can be adapted for domain-specific code generation using synthetic datasets. We construct datasets of programming exercises across three domains within the Python ecosystem: general Python programming, Scikit-learn machine learning workflows, and OpenCV-based computer vision tasks. Using these datasets, we evaluate three customization strategies: few-shot prompting, retrieval-augmented generation (RAG), and parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA). Performance is evaluated using both benchmark-based metrics and similarity-based metrics that measure alignment with domain-specific code. Our results show that prompting-based approaches such as few-shot learning and RAG can improve domain relevance in a cost-effective manner, although their impact on benchmark accuracy is limited. In contrast, LoRA-based fine-tuning consistently achieves higher accuracy and stronger domain alignment across most tasks. These findings highlight practical trade-offs between flexibility, computational cost, and performance when adapting smaller language models for specialized programming tasks.
Luís Freire, Fernanda A. Andaló, Nicki Skafte Detlefsen
Many applications today use large language models for code generation; however, production systems have strict latency requirements that can be difficult to meet with large models. Small language models with a few billion parameters are resource efficient but may suffer from limited reasoning, hallucinations, or poor retention of longer context. Fine tuning improves task specific accuracy by embedding domain knowledge directly into model weights, reducing reliance on runtime context. We previously implemented a baseline natural language to code generation approach using a retrieval augmented generation pipeline that dynamically selected few shot examples to embed domain specific language context for a large language model. In this study, we evaluate small language models for generating domain specific language from natural language by fine tuning variants of Mistral and other models on a dataset of natural language code pairs. Our results show that the fine-tuned models achieve improved performance and latency on test datasets compared to larger models. We also demonstrate that the trained model can be further fine-tuned for customer specific scenarios without degrading general performance, helping resolve production issues. Load testing followed by production deployment confirmed optimal performance in terms of latency and quality. These findings demonstrate that task specific fine tuning with small language models provides an efficient, faster, and cost-effective alternative to large language models for domain specific language generation.
Renjini R. Nair, Damian K. Kowalczyk, Marco Gaudesi +1