Can Large Language Models Optimize Pretrained Neural Network Weights? A Multi-Phase Empirical Evaluation of Gradient-Free Editing in YOLO26 Object Detection
Abstract
Direct weight editing offers a gradient-free alternative to fine-tuning pretrained detectors, but its generalization and the value of large language model (LLM) guidance remain uncertain. We evaluate five experimental phases using YOLO26 models and MS COCO, comparing deterministic scaling, random perturbations, differential evolution, CMA-ES, structured interventions, real local-LLM proposals, failure-aware guidance, and hybrid LLM-CMA-ES against a matched random-policy control. Optimization uses native Ultralytics mAP50-95 on 1,000 COCO Val2017 images; selected candidates are assessed on a disjoint, previously examined 1,500-image subset, with COCO API AP50-95 reported separately where available. Increasing the CMA-ES budget from 8 to 250 evaluations raised development gains from +0.11 to +0.28 percentage points, while selection changes remained negative (-0.03 to -0.12 points); the 250-evaluation result uses one seed. The hybrid LLM approach showed no statistically established advantage over random-policy guidance (p = 0.612, 0.369, and 0.694 at budgets 8, 25, and 50). Five-seed YOLO26m replication yielded a COCO API AP difference of +0.004 points (95% CI: -0.026 to +0.034). Of 225 archived LLM proposals, 53 satisfy the final strict schema; early tensor-level execution cannot be fully verified. Genuine LLM-mediated weight editing is feasible, but the tested approaches did not demonstrate consistent transfer beyond the development subset or a measurable LLM advantage.
Figures & tables
| Model | Subset | Params (M) | COCO AP | Native mAP |
|---|---|---|---|---|
| YOLO26n | Dev (1,000) | 2.409 | 42.19 | 42.08 |
| Selection (1,500) | 39.25 | 39.17 | ||
| YOLO26s | Dev (1,000) | 9.496 | 48.98 | 48.87 |
| Selection (1,500) | 46.57 | 46.44 | ||
| YOLO26m | Dev (1,000) | 20.411 | 54.33 | 54.17 |
| Selection (1,500) | 51.31 | 51.17 |
| Intervention | Dev mAP (%) | (pp) |
|---|---|---|
| Unchanged baseline | 42.08 | 0 |
| Global scale | 41.94 | -0.14 |
| Global scale | 41.71 | -0.37 |
| Global scale | 25.87 | -16.21 |
| Stage 8 scale | 42.02 | -0.06 |
| Best isolated Gaussian trial | 42.12 | +0.04 |
| Intervention | (pp) |
|---|---|
| Stage 4, rank 2, residual SVD | +0.052 |
| Stage 4, rank 2, random basis | +0.062 |
| Classification head, | +0.095 |
| Regression head, | -1.203 |
| Magnitude pruning, 1% | +0.008 |
| Magnitude pruning, 20% | -0.543 |
| Metric | Value |
|---|---|
| Unchanged YOLO26n AP | 40.9088% |
| TPE-selected WSP AP | 40.9668% |
| Observed paired change | +0.0580 pp |
| Bootstrap 95% interval | [-0.026, +0.123] pp |
| Archived two-sided | 0.124 |
| Seeds | Optimizer | Dev mean | Dev SD | Selection mean | Mean time (s) | |
|---|---|---|---|---|---|---|
| 8 | 5 | Random search | +0.15 | 0.06 | -0.11 | 29.1 |
| 8 | 5 | CMA-ES | +0.11 | 0.06 | -0.03 | 35.9 |
| 8 | 5 | Differential evolution | +0.14 | 0.08 | -0.05 | 20.7 |
| 25 | 5 | Random search | +0.19 | 0.03 | -0.07 | 72.5 |
| 25 | 5 | CMA-ES | +0.17 | 0.03 | -0.06 | 46.1 |
| 25 | 5 | Differential evolution | +0.18 | 0.04 | -0.08 | 35.6 |
| Policy | Dev | Sel. | |
|---|---|---|---|
| 8 | LLM feedback | +0.02 | 0.00 |
| 8 | Failure-aware LLM | +0.02 | -0.01 |
| 8 | Random-policy hybrid | +0.04 | -0.08 |
| 8 | LLM–CMA-ES hybrid | +0.06 | -0.05 |
| 25 | LLM feedback | +0.06 | -0.02 |
| 25 | Failure-aware LLM | +0.03 | +0.01 |
| ( ) | LLM random | 95% CI | |
|---|---|---|---|
| 8 (5) | +0.02 | [-0.09,+0.13] | 0.612 |
| 25 (5) | -0.02 | [-0.07,+0.03] | 0.369 |
| 50 (3) | +0.01 | [-0.08,+0.10] | 0.694 |
| Seed | Native mAP | COCO API AP |
|---|---|---|
| 42 (exploratory) | +0.043 | +0.035 |
| 43 | -0.051 | -0.030 |
| 44 | +0.014 | +0.015 |
| 45 | +0.012 | +0.005 |
| 46 | -0.016 | -0.006 |
| Mean | +0.001 | +0.004 |
| Retrospective category | Count | Share |
|---|---|---|
| Archived parsed proposals | 225 | 100% |
| Schema-valid now | 53 | 23.6% |
| Schema-invalid now | 172 | 76.4% |
| Verified complete original tensor trail | N/A | N/A |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Optimizer | LLM calls | Prompt / completion tokens | LLM time (s) | Development native mAP | Selection native mAP |
|---|---|---|---|---|---|
| Random search | 0 | 0 / 0 | 0 | 42.19 (+0.11) | 38.98 (-0.19) |
| Differential evolution | 0 | 0 / 0 | 0 | 42.21 (+0.13) | 39.09 (-0.08) |
| CMA-ES | 0 | 0 / 0 | 0 | 42.20 (+0.12) | 39.08 (-0.09) |
| Real LLM (zero-shot) | 14 | 7,440 / 4,294 | 133.8 | 42.11 (+0.03) | 39.14 (-0.03) |
| Real LLM + feedback | 14 | 9,848 / 4,832 | 140.4 | 42.14 (+0.06) | 39.17 (+0.00) |
| Failure-aware LLM | 14 | 15,059 / 6,222 | 172.9 | 42.14 (+0.06) | 38.99 (-0.18) |
| Optimization method | Dev mean | Dev SD | Selection mean | Total recorded LLM tokens |
|---|---|---|---|---|
| Random search | +0.15 | 0.06 | -0.11 | 0 |
| CMA-ES | +0.11 | 0.06 | -0.03 | 0 |
| Differential evolution | +0.14 | 0.08 | -0.05 | 0 |
| LLM with metric feedback | +0.02 | 0.02 | 0.00 | 36,469 |
| Failure-aware LLM (strict) | +0.02 | 0.01 | -0.01 | 52,176 |
| LLM–CMA-ES hybrid | +0.06 | 0.06 | -0.05 | 12,246 |
| Policy | Dev | Selection | Mean time (s) | Total tokens | ||
|---|---|---|---|---|---|---|
| 8 | 5 | LLM feedback | +0.02 | 0.00 | 160.6 | 36,469 |
| 8 | 5 | Failure-aware LLM | +0.02 | -0.01 | 157.0 | 52,176 |
| 8 | 5 | Random-policy hybrid | +0.04 | -0.08 | 110.3 | 0 |
| 8 | 5 | LLM–CMA-ES hybrid | +0.06 | -0.05 | 144.4 | 12,246 |
| 25 | 3 | LLM feedback | +0.06 | -0.02 | 496.3 | 77,721 |
| 25 | 3 | Failure-aware LLM | +0.03 | +0.01 | 501.5 | 116,364 |
Large Language Model-Guided Optimization
Large Language Model-Guided Optimization uses language models to propose, steer, and refine solutions to hard search and design problems.
See all 654 papers