Nanbeige4.2-3B is a 3B-parameter agentic model built around a Looped Transformer (LT) that reuses one stack of layers for a second forward pass, adding effective depth without additional parameters. Evaluated on Apple Silicon (MPS), we identify five independent bugs which prevent the released checkpoint from running via Hugging Face transformers out of the box (including a silently-zeroed RoPE buffer and calls to removed transformers cache APIs). Furthermore, we show that fixing these bugs is still not sufficient for agentic tasks, due to the LT's layer-reuse strategy (which effectively doubles peak attention memory) used to achieve parameter efficiency. We thus introduce a chunked-prefill strategy which alleviates the incurred memory-capacity penalty, extending allowable context width by 2.7× on 32~GiB shared memory. However, even with the reduced memory overhead, we show that patches are required to render Nanbeige4.2-3B usable; resolving both system prompt and MPS-native memory bugs finally allows reliable evaluation on standard MCP and tool-calling benchmarks. On a subset of MCPMark, the debugged model completes up to 30% of real agentic tasks (up from the original's 0%), while, on BFCL, it is near-perfect at single tool calls (yet fails the majority of multi-tool tests). We release the patched checkpoint, system prompt optimizer, and evaluation harnesses at https://github.com/johnhalloran321/Nanbeige4.2-3B-mps-fix.
Figures & tables
Bug
Line
Trigger
Error / symptom
1. RoPE buffer (major)
947
from_pretrained() + first forward()
no exception raised; inv_freq buffer reads all zeros ( persistent=False , never restored after a meta-device load)
2. RoPE-config dispatch
1077
from_pretrained(..., trust_remote_code=True)
KeyError: ’type’
3. Cache API sentinel
2125
calling forward() directly with past_key_values=None , bypassing generate()
AttributeError: type object ’DynamicCache’ has no attribute ’from_legacy_cache’ ; still present in the deployed checkpoint
4. Position-IDs re-trim
2630
generate() on MPS, once prepare_inputs_for_generation receives an already-populated position_ids kwarg
’mps.matmul’ op contracting dimensions differ 361 & 181 (Metal assertion, uncatchable SIGABRT ); not reproduced on CPU
5. Tied-weights key format
2417
model.save_pretrained(...)
AttributeError: ’list’ object has no attribute ’keys’
Table 1: Exact trigger, source line, and error/symptom for each bug, against transformers==5.8.1 . Line numbers refer to the unmodified checkpoint.
prompt_len
NP max batch
NP tok/s
CP max batch
CP tok/s
1024
16
275.3
32
212.5
2048
4
267.5
16
158.1
4096
2
158.4
4
124.8
8192
–
–
1
168.7
9205
–
–
1
156.5
10218
–
–
1
148.6
Table 2: NP vs CP evaluated over 50 LongBench-Pro samples, averaged over 3 repeated experiments. “–” denotes the method could not complete even batch=1.
Task
Turns
Outcome
largest_rename
5
pass
txt_merging
5
pass
file_reorganize
7
pass
pattern_matching
2
fail, timed out
file_splitting
4
fail, tool response OOM
uppercase
7
fail, tool response OOM
Table 3: MCPMark Filesystem (easy tier), patched checkpoint, per-task server isolation, using MCPMark ’s default 1 hour timeout.
Category
Score
Dominant failure mode
simple_python
19/30 (63.3%)
wrong call count (11/30)
multiple
13/30 (43.3%)
wrong call count (17/30)
irrelevance
30/30 (100%)
—
parallel
1/30 (3.3%)
wrong # of functions (29/30)
parallel_multiple
9/30 (30.0%)
wrong # of functions (21/30)
Table 4: BFCL , patched checkpoint, 30 tasks per category.
We present Nanbeige4.2-3B, a compact general agentic model with 3B non-embedding parameters. It delivers strong performance across code-agent, office-agent, and complex tool-use tasks while maintaining highly competitive reasoning capabilities in mathematics, coding, and science. Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters. For SFT data and trajectory construction, we expand the diversity of executable environments, task assets, and agentic scaffolds through real-world deployment and large-scale synthesis. Our RL pipeline applies mixed-mode RLHF over Think and Non-Think responses to improve overall model quality and reduce failure cases, length-controlled reasoning RL to balance accuracy and reasoning efficiency, and agentic RL with outcome and process rewards to stabilize long-horizon training. Extensive evaluations show that Nanbeige4.2-3B outperforms larger models, including Qwen3.5-9B and Gemma4-12B, across diverse agentic benchmarks while remaining competitive on reasoning and alignment tasks. Performance with OpenClaw further supports its use as a compact local personal assistant.
Looped Transformers apply the same layers several times per token, adding compute to improve quality without more parameters. Each recursion, however, writes its own key-value cache, so memory still grows with compute. Inference-time techniques can shrink this cache at a cost in quality. We pretrain looped language models to share memory: only the first recursion writes a cache, and later recursions read it while keeping a short window of their own. Surprisingly, we find that sharing memory does not cost quality and instead improves it. At 150M-1B parameters, our Looped Prediction Transformer (LPT) and its hybrid variant set a new quality-memory frontier for looped models: with five recursions, the hybrid lowers validation perplexity on FineWeb-Edu by 1.12-1.82 relative to a same-size standard Transformer while using 76-79% less context memory. Through an extensive analysis, we investigate why memory sharing helps. Shared and local memory develop different representations, and later recursions attend mostly to the shared memory, which also acts as a gradient highway to the first recursion.
We present Loopie, the most powerful open-source looped Transformer to date. The Loopie series consists of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N-fold pre-training compute, increasing the parameter count by a factor of N usually outperforms looping N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.