| Task | Description |
|---|
| Word reversal (188) | This model has no MLP: one causal attention read acts as a pointer, followed by a nearest-value lookup. Attention: one layer, one head, using hard argmax instead of softmax. Each position’s key is its 1-based position. The query at each generation step is the slot to copy. With i = position − 11 (the "=" is position 11) and word = floor(i/4), the query is 9 + i − 8 ⋅ word. The score is negative squared distance between query and key, and the single best earlier position wins. So the answer copies prompt indices 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2: word 3, separator, word 2, separator, word 1. Value and readout: the value is the token id as one scalar, capped at the separator’s id, so "=" reads as "". That lets the first separator be copied from the "=" slot. Logits are the negative squared difference between the read value and each vocabulary id. An exact match scores 0; ties go to the lower id, "". Errors: selection is exact with no softmax leakage, so accuracy should be 100%, even with repeated letters. |
| Boolean circuit (186) | A tiny recurrent "cellular automaton" transformer. Each position holds one scalar state, the expression value so far, stored centred as S − 0.5. One tied block (one attention head, 2-neuron ReLU MLP) is applied 10 times; each pass moves information one cell right, enough to reach ’=’. Attention has no key matrix: positions are sinusoidal angles at 30 ∘ per step, and the query is the position rotated back one step, scaled by 200, giving near-hard attention to the previous cell (position 0 attends to itself). Values copy that cell’s AND/XOR flags and put its state on the bit channel; each cell adds its own embedding. Op tokens carry their flag plus identity bit (& 1, ∣ and ˆ 0), so op cells and ’=’ (zero embedding) pass the state through. The MLP does one fold step: new state = relu(b + S − A) − relu(2(b + S) + X − A − 2), giving OR with no flags, AND with A, XOR with X. State starts at 0; readout logits are ± 4 times the centred state. Errors: expected none (attention leakage ≈ e ˆ − 27), though untested. |
| Multiplication (190) | This model has no MLP and no LayerNorm: two attention-only layers on a 7-channel float64 residual stream whose registers hold integers up to about 1e10. Layer 1 (one head) makes each B digit compute b ⋅ A. A query at a B digit b scores multiplicand position i at log(b ⋅ 10 ˆ (4 − i)) against a huge zero-valued sink key at ’ ∗ ’. Because the sink dominates (scale about 1e17), softmax acts like a plain exponential, so the head reads b ⋅ A/scale, rescaled to b ⋅ A. Digit 0 attends only to the sink. In layer 2, head 1 weights B positions proportionally to 10 ˆ (10 − t); multiplying the average by 11111 gives P = A ⋅ B. Head 2 uses the same sink trick (scale about 1e22) to read V, the value of already-emitted answer digits times their place values, and subtracts it, so X = P − V + 0.5, a long-division remainder. The readout is a tropical decoder: logit_k = k ⋅ X − place ⋅ k(k+1)/2 with place = 10 ˆ (20 − t) at answer position t, whose argmax is floor(X/place). Expect essentially 100% exact match; leakage is about 1e-12 relative, so float64 is required (float32 breaks). One wrong digit corrupts the remainder and all later digits. |
| GCD (189) | Tokens embed as one scalar (digit value; comma and "=" are 0) plus a fixed one-hot position code. The residual state has channels VAL, A, B, X. A six-head parser attention layer uses huge logits (about 40), so each head reads one operand digit, scaled by its place value (100, 10, 1). This gives exact integers A and B at answer positions. The comma attends to itself, keeping A=B=0 there as a "zero" slot. Euclid runs as residual ReLU MLPs on (A, B): 12 tied rounds of four folds with scales 64, 16, 4, 1. Each fold maps A to its distance from the nearest multiple of s ⋅ B. This triangular wave uses 4 ReLU knots at m ⋅ s ⋅ B/2 with alternating ± 2 slopes and is valid only while A ≤ 2sB, hence radix 4. The s=1 fold also swaps (A,B) → (B, fold). The result settles into a 2-cycle between (g,0) and (0,g), so A+B=g. Readout attention forms X=(A+B)/100, /10 or /1 by mixing with the zero slot, minus 100 × and 10 × earlier emitted digits. The head scores digit d as − (X − d − 0.495) 2 , flooring X. The file claims zero errors on all 10 6 pairs. |
| Smallest prime factor (181) | Two-block transformer, no LayerNorm, float64, all signals scalar (not one-hot). It should be exactly correct for every n from 2 to 99999, so there is no error pattern to match; aim for 100%. Block 1 attention: one head, constant query, keys (4 − i) ⋅ ln10 for prompt digit i, − 60 elsewhere. Softmax weights become 10 ˆ (4 − i)/11111, so summing digit values times 11111 yields n. Block 1 MLP (sieve): one neuron per prime up to 316 (65 primes) with input n/p and pulse activation relu(1 + 10 4 (cos2 π u − 1)), firing only at integers; non-divisors are at least 1/313 away. Lateral inhibition (2 ⋅ pulse minus cumulative sum over smaller primes) keeps only the smallest firing prime, which writes its three digits. Block 2 attention: Gaussian copy head, score − 40(i − (s − 5)) 2 , so output position s reads prompt digit s − 5. Block 2 MLP: ReLU gate. Any nonzero factor digit suppresses the copy (weight − 9); flags on the last three output steps release the factor’s hundreds, tens and units digits. No factor means n is prime and gets copied. Readout: logit_d = 2d ⋅ A − d 2 picks the nearest digit; "=" is blocked. |
| TREC (196) | A hand-built keyword decision list with no training. Block 1: each character is embedded as a 5-bit letter code with flags for word boundary, start and padding, plus position. Head h of 20 copies the character h steps back using a Gaussian position kernel. Each MLP neuron is an exact AND gate for one of about 400 keyword patterns: words, stems, or question-initial words. Block 2: ’=’ attends to the highest salience, 300 × priority minus 3 × position, so the earliest match wins ties. It copies that position’s class vote. If nothing matches, the answer is ENTY. Priority order: ABBR for "stands for" and "abbreviat"; DESC for "comes from"; then initial "who" (HUM), "where" (LOC), initial "when" (NUM) and "why" (DESC); definition or cause words such as mean, origin and reason (DESC); "how did/does/is" (DESC); other "how" (NUM); "what is X" questions under 17 characters (DESC); topical nouns, with people HUM, places LOC, measures NUM and things ENTY; "called" and "what kind" (ENTY); "what are the" (ENTY); and last, initial "what is" (DESC). Errors come from brittle surface matching: missing keywords, the wrong noun winning, and long "what is the" questions falling through to DESC. |