cs.SEOct 12, 2026
SaveDeleting Code Without Losing Knowledge: Regenerative Software for Python Libraries
Abstract
In the age of "Software 3.0", when a complete program can be rewritten in minutes by an agent, the traditional approach of relying on code as the sole source of truth becomes questionable. Fowler advocated for a set of seven primitives, which lay the framework for regenerative software, so that programs can be deleted and rebuilt without losing their knowledge. The knowledge lives in machine-readable records in a tracked directory, from which a regeneration bundle is built for each unit. The regeneration bundle is a single file with the module's architecture, intent, decisions, public contract, evaluations, and consumers, but no implementation, plus the one dependency module it may import. The open-source package spotregen implements these primitives and provides a graphical user interface. This report presents a first application of Fowler's primitives to the safety-critical forecasting library spotforecast2-safe. During the first experiments, the Claude Code session that orchestrated the study consumed more than 43 million tokens, almost all of them its own history re-sent with every request. This motivated the switch to an architecture that is built on local models. Deterministic checks and the maintainer judged the results. The report contains executable examples for every primitive. Each example runs spotregen against the repository or against a throw-away copy of it.
Figures & tables
| checkgroup | check IDs | what is checked |
|---|---|---|
| Compilation | ARCH00–ARCH05 | parseable sources, allowed imports between subpackages, new or vanished import cycles |
| Unit | UNIT01–UNIT06 | the unit’s file exists, its public names are defined, its imports match depends_on , it declares at least one intent item |
| Evaluation | EVAL01–EVAL05 | evaluations exist, are marked regen , import only public names, cover every intent item |
| Provenance | PROV01–PROV05 | cited decisions exist, evidence resolves, review dates, superseded or orphaned records |
| Deletion | DEL01–DEL03 | every dependent inside the repository is a declared consumer, declared consumers exist |
Table 1: Check IDs of spotregen check , grouped by the kind of check. The checkgroups are not the seven primitives. Table 15 relates the two.
Figure 1: The three layers of Fowler’s evaluation suite (Fowler 2027, ch. 5) and the static check of this report. The layers and the gate are Fowler’s; the name unit tests for the ephemeral layer and the static check are this report’s. A solid edge is a judgement, the dashed edge a second opinion that decides nothing.
Figure 2: Relations between the key terms of the Methods section. Boxes are terms, edge labels state the relation. Solid edges hold for every change, dashed edges belong to the experiments and the measures. How the checkgroups enforce the primitives is given in Table 15 .
Figure 3: The key terms as a UML class diagram. A hollow triangle marks generalisation, a filled diamond composition, a solid arrow an association with its multiplicities, and a dashed arrow a dependency. The second compartment names the spotregen subcommand behind a class.
Figure 4: Processes as a UML activity diagram. Filled circles are the initial and the final node, and diamonds are decisions. Left: the path of every change. Right: the experiments, whose two paths are independent. Mermaid has no fork bar, so they are drawn side by side.
| term | defined in | subcommand | module and function | exit code |
|---|---|---|---|---|
| primitive | Section 2.3 | none | none: fields of the records, enforced as in Table 15 | – |
| record | Definition 2.1 , Definition 2.11 , Definition 2.16 | new-unit ; every subcommand loads them | spec.load_spec , scaffold.scaffold_unit | 2 on a malformed record |
| pilot unit | Definition 2.10 | status , deletion-test | reports.status_table , reports.deletion_report | deletion-test : 1 unless the verdict is regenerable |
| static check: spotregen check | Definition 2.2 | check | checks.run_checks | 1 on an error, with --strict also on a warning |
| checkgroup | Definition 2.24 | check | ARCH: checks.check_architecture ; UNIT, EVAL, DEL, PROV01, PROV04: checks.check_unit ; PROV: checks.check_decisions | as check |
| durable evaluation | Definition 2.4 | none: pytest in make test | drivers._common.run_pytest when a protocol scores a module | pytest’s |
Table 2: Implementation of the key terms in spotregen. Module and function are relative to the package spotregen and are checked against the installed version while this document renders. Exit codes: 2 always means a record could not be loaded.
| Key | Required | Meaning |
|---|---|---|
| package | yes | top-level package name |
| source_dir | yes | directory, relative to the repository root, that holds the package |
| [allowed] | yes | per subpackage or top-level module, the subpackages it may import; "init" is the package’s init.py |
| [pace] | no | per pace class, the verification a change at that pace needs |
| known_cycles | no | accepted import cycles, each a pair of subpackages; any other cycle fails the check (ARCH04) |
| scan_dirs | no | directories whose Python files count as consumers (default src , tests , scripts ) |
Table 3: Keys of regen/architecture.toml (spotregen 0.10.1).
| Field | Level | Required | Meaning |
|---|---|---|---|
| evaluations | unit | no | repository-relative paths of the evaluation files; each must exist (EVAL01) and carry the marker regen (EVAL02) |
| unit_tests | unit | no | the unit tests of the ephemeral layer (Definition 2.3 ): coupled to the implementation, regenerated with the code, not part of the regeneration bundle |
| evaluations | intent item | yes* | names of the evaluations that check the item; each must exist in an evaluation file of the unit (EVAL01) |
Table 4: Fields of a unit record that concern evaluations. *Not required by the loader, but an intent item without an evaluation fails the static check (EVAL04). An evaluation named by no intent item is reported as EVAL05.
| Field | Required | Meaning |
|---|---|---|
| title , date | yes | what was decided, and when |
| driver | yes | the problem or incident that forced the decision |
| decision | yes | what was decided |
| approved_by | yes | the person who approved it, with date and occasion |
| status | no | active (default), superseded or expired ; an intent item that rests on a record that is not active fails the check (PROV04) |
| origin | no | human (default) or machine , for a record reconstructed by a model |
Table 5: Fields of a decision record in regen/decisions/ (spotregen 0.10.1).
| Field | File | Required | Meaning |
|---|---|---|---|
| [pace] | architecture.toml | no | per class fast , medium or slow , the verification a change at that pace needs |
| pace | unit record | yes | the class of the unit: fast , medium or slow |
| risk | unit record | yes | low , medium or high : how much verification a regeneration of the unit needs |
Table 6: Fields that record the pace of a unit (spotregen 0.10.1).
| Field | Required | Meaning |
|---|---|---|
| ref | yes | what depends on the unit: a repository path for code , test , script , doc and book ; <repo>:<path> for external ; a description for operational and human |
| kind | yes | one of code , test , script , doc , book , external , operational , human |
| note | no | how the consumer depends on the unit, for example how it handles the unit’s errors |
Table 7: Fields of a [[consumers]] table in a unit record (spotregen 0.10.1). A path of the five repository kinds must exist (DEL02) and must still reference the unit (DEL03). A file that references the unit without being declared is reported as DEL01. Consumers of the kinds external , operational and human are not checked.
| Field | File | Sediment | Check ID |
|---|---|---|---|
| depends_on | unit record | a module that is declared but no longer imported | UNIT05 |
| [[consumers]] ref | unit record | a consumer in the repository that no longer references the unit | DEL03 |
| [[intent]] evaluations | unit record | an evaluation in an evaluation file that no intent item names | EVAL05 |
| [[intent]] decisions | unit record | an intent item that rests on a decision that is not active | PROV04 |
| — | decision record | a decision that no intent item and no architecture entry cites | PROV05 |
| review_after | decision record | an active decision past its review date | PROV03 |
Table 8: Fields read by compaction (spotregen 0.10.1). All these checks are warnings: they ask for a removal or a review and do not fail the check unless --strict is set. spotregen compact collects all of them except ARCH05, which spotregen check reports.
| primitive | question it answers | artefact in spotforecast2-safe |
|---|---|---|
| Intent | What must stay true, and why? | regen/units/<unit>.toml , [[intent]] items |
| Compilation | Which architecture must the code fit into? | regen/architecture.toml |
| Evaluation | How is a replacement judged? | tests/test_regen_eval_<unit>.py , marker regen |
| Provenance | Why is the system this way? | regen/decisions/<id>.toml |
| Pace | How fast may this layer change? | pace of each unit, policy in [pace] |
| Deletion | Who notices if it is gone? | [[consumers]] of each unit |
Table 9: The seven primitives of Fowler (2027) and where each one lives in the repository.
| check ID | severity | reported when |
|---|---|---|
| ARCH00 | error | a source file of the package does not parse |
| ARCH01 | error | a subpackage imports a subpackage that [allowed] does not permit |
| ARCH02 | error | a subpackage is missing from [allowed] |
| ARCH03 | warning | an allowed edge is unused, or a listed subpackage does not exist |
| ARCH04 | error | a new import cycle, not listed in known_cycles |
| ARCH05 | warning | a cycle listed in known_cycles no longer exists |
Table 10: Check IDs of the Compilation checkgroup, read from regen/architecture.toml and the import graph.
| check ID | severity | reported when |
|---|---|---|
| UNIT01 | error | the unit’s module does not exist or does not parse |
| UNIT02 | error | path is not the file of the module the unit record names |
| UNIT03 | error | a name in public is not defined in the module |
| UNIT04 | error | the module imports a package module that is not in depends_on |
| UNIT05 | warning | an entry of depends_on is not imported |
| UNIT06 | error | the unit record declares no intent item |
Table 11: Check IDs of the Unit checkgroup, checking a unit record against its module.
| check ID | severity | reported when |
|---|---|---|
| EVAL01 | error | an evaluation file or a named evaluation does not exist, or the file does not parse |
| EVAL02 | error | an evaluation file lacks the pytest marker regen |
| EVAL03 | error | an evaluation imports a name of the unit that is not public |
| EVAL04 | error | an intent item names no evaluation |
| EVAL05 | warning | an evaluation is not linked to any intent item |
Table 12: Check IDs of the Evaluation checkgroup, checking the evaluation files and their links to intent items.
| check ID | severity | reported when |
|---|---|---|
| PROV01 | error | an intent item or the architecture cites an unknown decision |
| PROV02 | error | evidence of a decision does not resolve (file, commit, KB entry or prefix) |
| PROV03 | warning | an active decision is past its review_after date |
| PROV04 | warning | an intent item rests on a decision that is not active |
| PROV05 | warning | no intent item or architecture entry cites the decision |
Table 13: Check IDs of the Provenance checkgroup, checking the decision records and the citations of them.
| check ID | severity | reported when |
|---|---|---|
| DEL01 | error | a file in the repository depends on the unit but is not a declared consumer |
| DEL02 | error | a declared consumer in the repository does not exist |
| DEL03 | warning | a declared consumer no longer references the unit |
Table 14: Check IDs of the Deletion checkgroup, checking the declared consumers of a unit.
| primitive | enforced by |
|---|---|
| Intent | UNIT06 (no intent item), EVAL04 (intent item without evaluation) |
| Compilation | ARCH00–ARCH05, UNIT04, UNIT05 |
| Evaluation | EVAL01–EVAL05 |
| Provenance | PROV01–PROV05 |
| Pace | vocabulary check on loading (exit code 2); status , regeneration bundle |
| Deletion | DEL01–DEL03; deletion-test |
Table 15: How the static check and the other subcommands enforce each primitive. UNIT01–UNIT03 check that a unit record matches its module and precede all checks of a unit.
Figure 6: The Overview after the check on the sample repository with three planted faults. Each next action links to the view that resolves it and shows the equivalent command.
Figure 7: The Check view on the same repository. The selected finding is shown below the table as the command line prints it.
Figure 8: The Measures view after a mutation-score job on the sample repository. The log shows the progress lines. The table below is the output of measures --mutation-score .
Figure 9: The Runs view on the sample repository after a Measures job, a Mutate job, and a Local job. The Local run stopped at once because no model server was running, so it recorded neither a protocol nor a score.
| unit | public | depends_on | found | ext., op., hum. consumers | intent |
|---|---|---|---|---|---|
| multitask.guards | same | same | 5 of 5 | 18 | 8 |
| preprocessing.outlier | same | same | 9 of 9 | 10 | 8 |
| preprocessing.target_corruption | same | same | 6 of 8 | 10 | 8 |
| processing.forecast_scoring | same | same | 6 of 6 | 7 | 8 |
Table 16: Drafts by new-unit compared with the hand-written unit records of the four pilot units. Found: in-repository dependents (consumers, unit tests, and evaluation files the static check can see). Added by hand: external, operational and human consumers, and intent items.
| checks | against | kind | |
|---|---|---|---|
| Unit test | the current implementation | its design: private helpers, internal details | verification against the design |
| Evaluation | any implementation | the intent items (the specification) | verification against the specification |
| Review of the intent items | the intent itself | the needs of operators, incidents, regulation | validation |
Table 17: Unit tests, evaluations, and the review of the intent items, by what they check and what they check it against.
| ID | scope | invariant | named evaluations | decisions |
|---|---|---|---|---|
| LG-1 | assert_no_leakage , the three surfaces | Each of the three surfaces (training frame, selected exogenous features, fitted features of every target) is checked on its own. A forbidden name on any of them raises LeakageError . | 5 | 1 |
| LG-2 | assert_no_leakage , the LeakageError | One LeakageError names the task, every leaking surface and its forbidden columns, in a fixed message frame. | 4 | 2 |
| LG-3 | fitted-feature-name reading, per estimator | Fitted feature names are read from the first attribute present, covering LightGBM, CatBoost, scikit-learn and XGBoost. | 3 | 1 |
| LG-4 | assert_no_leakage , unreadable fitted features | A surface that cannot be read raises a RuntimeError that is not a LeakageError and names the cause. The guard never passes silently. | 8 | 3 |
| LG-5 | assert_no_leakage , the clean case | Without a forbidden name the guard returns None and has no other effect. | 3 | 1 |
| LG-6 | matching of column names | Names match only by exact string equality, so derived names are no violation. | 2 | 1 |
Table 18: Intent items of the leakage guard multitask.guards . Scope is the field of the unit record, invariant a summary of its field invariant , and the last two columns count the evaluations and decision records the item names.
| phase | date | generator | harness |
|---|---|---|---|
| 1a: Claude, blind by request (Section 3.4.1 ) | 2026-09-29 | Claude Opus 5.5 | subagent of a Claude Code session, blindness requested in the prompt |
| 1b: Claude with enforced blindness (Section 3.4.2 ) | 2026-09-30 | Claude Opus 5.5 | claude -p with --restricted and --safe-mode |
| 2: a local model in the Claude Code harness (Section 3.4.3 ) | 2026-09-30 | Qwen3.8-27B, reasoning on | the harness of phase 1b, pointed at a local endpoint |
| 3: local models in short sessions (Section 3.4.4 ) | 2026-09-30 | Qwen3.8-27B (reasoning off), Qwen3-Coder-30B | short sessions of an own driver, no Claude Code |
| 4: Devstral against Qwen (Section 3.4.5 ) | 2026-10-01 | Devstral Small 2, Qwen3.8-27B | the driver of phase 3 |
Table 19: The five phases of the regeneration in Example 3. The names are those of the subsections of Section 3.4 .
| step | what the LLM did | what judged the result |
|---|---|---|
| Drafting a unit record ( new-unit ) | nothing: the draft is read from the source tree | UNIT06, the maintainer |
| Intent items, evaluations, and decision records of the pilot | one subagent per unit reconstructed them from code, unit tests, KB, and history | the static check, the maintainer’s review, origin = "machine" , and review dates |
| Mutation analysis (Definition 2.28 ) | the same subagents planted the mutants | the evaluations |
| Phase 1a: Claude, blind by request (Section 3.4.1 ) | a subagent rebuilt the module from the bundle and listed the gaps | the evaluations, the old unit tests (second opinion), the tool-call log |
| Phase 1b: Claude with enforced blindness (Section 3.4.2 ) | claude -p , confined by the harness, rebuilt the module three times | a canary test of the confinement, the evaluations, the old unit tests (second opinion) |
| Phase 2: a local model in the Claude Code harness (Section 3.4.3 ) | nothing: Qwen3.8-27B in the same harness wrote no module in two attempts | the canary test |
Table 20: Where an LLM acts in the study and what checks its output. Unless a row names another model, the LLM is Claude Opus 5.5.
Figure 10: Steps of spotregen local with its default budgets. Rounded boxes are fresh model sessions without history, each limited to 15 minutes. Rectangles run without a model. Only files cross from one step to the next. Every model session also receives the prompt. A repair receives the last 12 000 characters of the failure output. Phases 3 and 4 ran these steps with the predecessor regen_local.py , without the dependents (dashed).
| unit | intent items | covered | named evaluations | intent coverage |
|---|---|---|---|---|
| multitask.guards | 8 | 8 | 30 | 8/8 |
| preprocessing.outlier | 8 | 8 | 22 | 8/8 |
| preprocessing.target_corruption | 8 | 8 | 33 | 8/8 |
| processing.forecast_scoring | 8 | 8 | 23 | 8/8 |
| total | 32 | 32 | 108 | 32/32 |
Table 21: Intent coverage per pilot unit (Definition 3.5 ). Intent items is as in Table 24 (Definition 3.1 ). Covered counts the intent items that name at least one existing evaluation, named evaluations the distinct evaluations the intent items name. Coverage is taken from spotregen measures .
| unit | score, by hand | score, generated |
|---|---|---|
| multitask.guards | 12/12 (100 %) | 26/26 (100 %) |
| preprocessing.outlier | 6/6 (100 %) | 38/61 (62 %) |
| preprocessing.target_corruption | 14/14 (100 %) | 107/135 (79 %) |
| processing.forecast_scoring | 10/10 (100 %) | 111/123 (90 %) |
| total | 42/42 (100 %) | 282/345 (82 %) |
Table 22: Mutation score per pilot unit (Definition 2.32 ), as killed out of planted mutants and in percent. Score, by hand is for the mutants planted by hand in the pilot (Definition 3.6 ), which were not archived, score, generated for those of the archived spotregen mutate runs in regen/runs/2026-10-02-mutation (Definition 3.7 ).
| unit | implementation | unit record | decision records | evaluation files | specification size | ratio |
|---|---|---|---|---|---|---|
| multitask.guards | 239 | 232 | 120 | 453 | 805 | 3.4 |
| preprocessing.outlier | 354 | 189 | 83 | 395 | 667 | 1.9 |
| preprocessing.target_corruption | 624 | 203 | 92 | 716 | 1011 | 1.6 |
| processing.forecast_scoring | 517 | 167 | 79 | 597 | 843 | 1.6 |
| total | 1734 | 791 | 374 | 2161 | 3326 | 1.9 |
Table 23: Lines of implementation and of specification per pilot unit, as spotregen measures prints them. All columns except ratio count lines. Specification size is the sum of the lines of the unit record, of the decision records its intent items cite (each counted once) and of the evaluation files. Ratio is specification lines per line of implementation.
| unit | pace | risk | intent items | decisions | consumers | evaluations | intent coverage | score, by hand | score, generated | specification size |
|---|---|---|---|---|---|---|---|---|---|---|
| guards | slow | high | 8 | 4 | 22 | 44 | 8/8 | 12/12 | 26/26 | 805 (3.4) |
| outlier | medium | high | 8 | 3 | 19 | 42 | 8/8 | 6/6 | 38/61 | 667 (1.9) |
| corruption | slow | high | 8 | 3 | 16 | 40 | 8/8 | 14/14 | 107/135 | 1011 (1.6) |
| scoring | slow | high | 8 | 3 | 10 | 51 | 8/8 | 10/10 | 111/123 | 843 (1.6) |
| total | 32 | 13 | 67 | 177 | 32/32 | 42/42 | 282/345 | 3326 (1.9) |
Table 24: Pilot units and the three measures of Section 2.8 . Units: guards = multitask.guards , outlier = preprocessing.outlier , corruption = preprocessing.target_corruption , scoring = processing.forecast_scoring , each in the package spotforecast2_safe . The columns are explained in the text.
Figure 11: Outcome of the mutants of each pilot unit. Each unit has one bar for the mutants planted by hand and one for those generated by spotregen mutate . Survivors of the generated sample are split by the verdict of their review in review.toml : output changes only logs or prints, equivalent changes nothing observable, gap is behaviour that no evaluation pinned.
Figure 12: Specification size of each pilot unit (Definition 2.33 ), split into its three parts, next to the lines of the implementation. The number at the end of a bar is the ratio.
| unit | pace | risk | intent | evaluated | decisions | consumers |
|---|---|---|---|---|---|---|
| spotforecast2_safe.multitask.guards | slow | high | 8 | 8/8 | 4 | 22 |
| spotforecast2_safe.preprocessing.target_corruption | slow | high | 8 | 8/8 | 3 | 16 |
| spotforecast2_safe.processing.forecast_scoring | slow | high | 8 | 8/8 | 3 | 10 |
| spotforecast2_safe.preprocessing.outlier | medium | high | 8 | 8/8 | 3 | 19 |
| unit | coverage | code | unit | decisions | evaluations | spec | ratio |
|---|---|---|---|---|---|---|---|
| spotforecast2_safe.multitask.guards | 8/8 | 239 | 232 | 120 | 453 | 805 | 3.4 |
| spotforecast2_safe.preprocessing.outlier | 8/8 | 354 | 189 | 83 | 395 | 667 | 1.9 |
| spotforecast2_safe.preprocessing.target_corruption | 8/8 | 624 | 203 | 92 | 716 | 1011 | 1.6 |
| spotforecast2_safe.processing.forecast_scoring | 8/8 | 517 | 167 | 79 | 597 | 843 | 1.6 |
| total | 32/32 | 1734 | 791 | 374 | 2161 | 3326 | 1.9 |
Figure 13: Allowed imports between the subpackages of spotforecast2-safe, as recorded in [allowed] of regen/architecture.toml . An arrow points from the importing to the imported subpackage. processing (blue) has incoming edges only. The dashed edge is the import planted below. Together with the edge from multitask it closes a cycle.
| phase | result | why the next phase |
|---|---|---|
| 1a | 29 of 29 evaluations, first test run | blindness rested on the prompt, and it was a single run |
| 1b | 3 of 3 runs | the orchestrating session hit the usage limit of the Claude plan |
| 2 | stopped: no module written in two attempts | suspected, not tested: the reasoning mode and a harness built for Claude |
| 3 | Qwen3.8-27B 3 of 3, Qwen3-Coder 0 of 3 | why did Qwen3-Coder fail: no reasoning mode, or only 3B active parameters? |
| 4 | Qwen3.8-27B 3 of 3, Devstral 1 of 3 (at T 0.15: 0 of 3) | none, under the decision rule set before the runs the comparison is undecided |
Table 27: Results of the five phases of Table 19 . Result: runs whose final module passed all evaluations of the time, read from the run summaries ( run.toml , summary.toml ) of the archived runs in spotforecast2-safe under regen/runs/ .
| requests | cached input | fresh input | output | thinking | total | |
|---|---|---|---|---|---|---|
| canary | 7 | 5.8 K | 3.6 K | 1.6 K | 0.2 K | 11.0 K |
| blind run 1 | 6 | 86.3 K | 43.9 K | 14.8 K | 5.6 K | 144.9 K |
| blind run 2 | 7 | 83.8 K | 41.0 K | 12.0 K | 3.5 K | 136.8 K |
| blind run 3 | 6 | 88.7 K | 45.8 K | 17.3 K | 7.9 K | 151.8 K |
| orchestrating session | 196 | 42.83 M | 371.4 K | 145.0 K | – | 43.35 M |
Table 28: Token consumption of Claude Opus 5.5. Requests: model turns (for the orchestrating session: requests to the hosted model). Cached: input read from the prompt cache. Fresh: input written to the cache or sent uncached. Output includes thinking. Generator rows are read from the archived transcripts of phase 1b. The orchestrating session was measured from its log while it was still running, so its numbers are a lower bound.
| run | evaluations | unit tests | sessions | input | output | max. | time | gaps |
|---|---|---|---|---|---|---|---|---|
| Qwen3.8-27B 1 | 31, 32, 37 | 18/18 | 4 | 85.6 K | 14.3 K | 23.9 K | 9.8 min | 12 |
| Qwen3.8-27B 2 | 37 | 18/18 | 2 | 38.1 K | 5.7 K | 20.7 K | 3.7 min | 15 |
| Qwen3.8-27B 3 | 37 | 18/18 | 2 | 37.5 K | 4.1 K | 20.1 K | 2.8 min | 8 |
| Qwen3-Coder 1 | 28, 29, 29 | 15/18 | 4 | 77.9 K | 10.8 K | 21.3 K | 3.3 min | 18 |
| Qwen3-Coder 2 | 32, 32, 32 | 17/18 | 4 | 77.4 K | 10.3 K | 20.9 K | 3.1 min | 15 |
| Qwen3-Coder 3 | 30, 30, 30 | 17/18 | 4 | 78.7 K | 9.5 K | 21.7 K | 3.0 min | 20 |
Table 29: Regenerations of the leakage guard in short local sessions (phase 3), with the three Claude runs of phase 1b for comparison. Evaluations: passed after each attempt. Unit tests: old unit tests passed by the final module, a second opinion (Definition 2.3 ). Sessions: fresh model requests, including the gap report. Input and output: tokens summed over the sessions, for Claude the cached and fresh input of the whole run. Max.: the largest input of a single session. Time: model time. Gaps: items of the gap report.
| module | evaluations | failing | unit tests | dependents |
|---|---|---|---|---|
| original | 44/44 | – | 18/18 | 2/2 |
| LS qwen3.8-27b run-1 | 40/44 | test_unreadable_message_states_the_cause | 18/18 | 2/2 |
| LS qwen3.8-27b run-2 | 40/44 | test_unreadable_message_states_the_cause | 18/18 | 2/2 |
| LS qwen3.8-27b run-3 | 40/44 | test_unreadable_message_states_the_cause | 18/18 | 2/2 |
| DV qwen3.8-27b run-1 | 40/44 | test_unreadable_message_states_the_cause | 18/18 | 2/2 |
| DV qwen3.8-27b run-2 | 40/44 | test_unreadable_message_states_the_cause | 18/18 | 2/2 |
Table 30: The original guard and the seven archived rebuilds that passed every evaluation of their day, scored again while this document renders with spotregen claude multitask.guards --score . Evaluations: today’s durable evaluations. Failing: the evaluations a module fails today. Unit tests: the old unit tests, a second opinion (Definition 2.3 ). Dependents: the re-export in multitask/init.py and the docstring example of assert_no_leakage . LS: short local sessions (Section 3.4.4 ), DV: Devstral against Qwen (Section 3.4.5 ).
| model | architecture | reasoning | vendor scores | fit and speed |
|---|---|---|---|---|
| Qwen3-Coder-Next (Qwen Team 2026c) | MoE, 80 B total, 3 B active | none | SWE-bench Verified 70.6, Terminal-Bench 2.0 36.2 | 8-bit fits; 77 to 79 tokens/s at short context on an M5 Max, 48 at 66 K (oMLX 2026; Silicon Score 2026) |
| gpt-oss-120b (OpenAI 2025) | MoE, 117 B total, 5.1 B active | low, medium or high, set in the system prompt | SWE-bench Verified 62.4 at high reasoning | about 65 GB in MXFP4, a 4-bit microscaling floating-point format |
| Qwen3.6-35B-A3B (Qwen Team 2026a) | MoE, 35 B total, 3 B active | on by default, switchable ( enable_thinking ) | SWE-bench Verified 73.4, Terminal-Bench 2.0 51.5 | fits easily |
| Devstral Small 2 (Mistral AI 2025) | dense, 24 B | none | SWE-bench Verified 68.0, Terminal-Bench 2.0 22.5 | fits easily |
| Qwen3.8-Flash-Next (Qwen Team 2026b) | MoE, 125 B plus a 51 B n-gram embedding, 6 B active | on by default, switchable | SWE-bench Pro 62.5 | 112 GB at 4 bits (LLMCheck 2026), which leaves about 16 GB for the context |
Table 31: Candidate local generators for the M5 Max with 128 GB, as of 2026-09-30. Vendor scores: from the model cards.
| component | recommended tool or setting | evidence |
|---|---|---|
| hardware | Apple M5 Max, 128 GB unified memory, macOS 27.0 | phases 2 to 4 |
| model server | LM Studio headless (llmster), MLX engine 1.11.0, OpenAI-compatible completion endpoint on 127.0.0.1:1234 , context length 32,768 tokens | phases 3 and 4 |
| generator | Qwen3.8-27B, 4-bit MLX ( mlx-community/Qwen3.8-27B-4bit , revision 10c35ca ), reasoning off by an empty reasoning block written into the prompt | 6 of 6 runs passed |
| driver | spotregen local of spotregen 0.10.1, standard library only, no tools, short sessions (Figure 10 ) | phases 3 and 4 |
| inputs | the bundle of spotregen context and exceptions.py as the only allowed file | all phases |
| sampling | temperature 0.7, top-p 0.8, top-k 20, seed equal to the run number, repair seed growing with the attempt | phase 4 |
Table 32: Recommended setup for a regeneration, derived from the five phases of Table 19 . Evidence: the phases whose results support the setting.
More in this topic
Automated Software Engineering
Automated Software Engineering uses AI, search, and verification methods to automate tasks across the software development lifecycle.
See all 80 papers