cs.CLSep 30, 2026
SaveBongard: Training Machine Intuition
Organizations: AgentBull Pte Ltd
Abstract
Human intelligence relies heavily on learned intuition: recognising patterns and judging situations without explicitly unfolding every intermediate step. We introduce Bongard, an open-weight System One model that treats machine intuition as an independent capability to design and train. A T5Gemma 2 4B-4B encoder-decoder separates reading the evidence from making judgments. The encoder reads the state bidirectionally together with the question instructions, and separate decoder branches share this encoding, so many judgments about the same situation require only one reading of the state. A trained head returns probabilities over the supplied candidates without generating text. Training proceeds in three stages, from supervised judgments to semantic relationships to action outcomes, and each stage updates all 7.09 billion trainable parameters on one Blackwell GPU. Joint-embedding post-training raises accuracy on held-out rephrasings from 75.7% to 85.9%. A sandbox stage then learns outcome distributions from action rollouts and exact oracles, raising accuracy on a frozen sandbox panel from 50.6% to 64.8%. On DecisionBench, the final model reaches 78.05% accuracy over 23,900 decisions and ranks fourth of 61 systems in the public comparison. On one RTX PRO 6000, its median latency is 36 ms for short requests, and 32 questions about one state take 221 ms. Bongard demonstrates that machine intuition can be systematically trained via representation learning and outcome feedback, providing an open, efficient alternative for high-throughput decision workloads.
Figures & tables
Figure 1: Architecture of Bongard: several typed questions share a single encoding of the state, and no tokens are generated. In this example, three questions of different types concern one checkout page. The encoder reads the state once, and each question is a separate decoder sequence. The decoder processes all sequences in one batched pass, with merged self- and cross-attention over the shared state keys and values. There is no attention between questions. Hidden states at the candidate-end markers ( ) and the decision-end marker ( ) feed a shared bilinear judgment head.
| Primitive | Question | Criteria | Answer |
|---|---|---|---|
| Choice | Which named alternative applies? | 1–255 options, each with an optional description | A probability per option, the selected option and a confidence |
| Noul | Is this statement true? | Optional descriptions of the true and false outcomes | The probability that the statement is true |
| Score | Where does the input fall on an ordered scale? | 2–10 ordered level descriptions | A probability per level, the expected level and a confidence |
Table 1: The three question primitives. The same head answers all of them in the same forward pass.
| Component | Configuration | Parameters |
|---|---|---|
| Text encoder | 34 layers, , FFN 10,240, 8 query / 4 KV heads of dimension 256, window 1,024 (5 local : 1 global), bidirectional | 3.21B |
| Text decoder | Same shape, causal, merged self- and cross-attention | 3.21B |
| Tied embeddings | 262,144 tokens 2,560 | 0.67B |
| Vision tower | SigLIP ( Zhai et al., 2023 ) , 896 px, 256 tokens per image, frozen | 0.42B |
| Projector, head | Trained, with and | 4.26M |
| Total | Bongard limit: 32,768 tokens per state plus question | 7.51B |
Table 2: Bongard 4B-4B configuration. Training updates all components except the vision tower, so 7.09B of the 7.51B parameters are trainable.
| (a) Encoder attention mask | |||||
|---|---|---|---|---|---|
| Accuracy | NLL | Brier | Fact direction | Effective rank | |
| Bidirectional (native) | 69.8% | 0.677 | 0.219 | 100.0% | 49.7 |
| Forward only | 64.6% | 2.944 | 0.320 | 70.8% | 37.9 |
| Forward and backward heads | 57.3% | 1.596 | 0.363 | 62.5% | 31.7 |
| (b) Decoder modification | |||||
| Held-out NLL | Transfer NLL | Latency | |||
Table 3: Architecture probes on T5Gemma 2 1B-1B. (a) Encoder masks with a frozen backbone and a trained head, on synthetic judgments of thresholds, negation and role binding, averaged over two seeds. Fact direction is the share of fact changes that move the truth probability in the correct direction. (b) Decoder modifications after short full-parameter training, as changes relative to the unmodified model for seeds 23 and 37. The value-residual row comes from a longer run and reports cross-task NLL as transfer.
| System | Backbone and training | How the state is read | How the answer is read |
| Encoder–decoder | |||
| Bongard | T5Gemma 2 4B-4B; full text-model training on judgments, semantic pairs and action outcomes | Bidirectional, question-aware encoder, once per request, shared by separate decoder branches | Trained bilinear head on candidate and decision readouts, no sampling |
| Closed | |||
| Jev | Undisclosed backbone; RLCD described by TypeSafe | Isolated question branches (observed), state processed once (suggested by latency) | Direct probabilities, options interact (observed) |
| Decoder-only or diffusion language models | |||
| OpenJev default | Pretrained DiffusionGemma 26B-A4B; inference wrapper | State and all questions in one canvas | Label-token probabilities at masked slots, noisy re-reads when uncertain |
Table 4: Architecture and learning choices in documented System One systems. Sources: Jev ( TypeSafe AI, 2026a ; Hume, 2026 ) , OpenJev ( OpenJev contributors, 2026 ) , Kev ( Palmer and Kev contributors, 2026 ) , JevK5 and Plumb-4B ( allebee, 2026 ; crh225, 2026 ) , AutoJev ( denis-pplx, 2026 ) , MoJev ( MoLeMo Lab, 2026 ) , CLM ( Contrastive-LM, 2026 ) , imajev ( Garg, 2026a ) , and Laya and Verdict ( Convai Innovations, 2026a ; OpenJev contributors, 2026 ) . Jev’s architectural details are inferred from API behaviour.
Figure 2: Training pipeline. A pretrained T5Gemma 2 encoder–decoder receives a judgment head and is then trained in three stages: supervised judgment training, joint-embedding predictive training and sandbox reinforcement learning. Each stage updates all trainable parameters on one GPU.
| Family | Raw cosine | Centred cosine | Centred retrieval |
| paired / unpaired | paired / unpaired | top-1 ( ) | |
| Shared encoded state | |||
| Answer state | 0.995 / 0.993 | 0.46 / | 28.9% |
| Logical complement | 1.000 / 0.996 | 0.94 / 0.05 | 99.6% |
| Inherited view | 1.000 / 0.979 | 0.98 / 0.00 | 71.9% |
| Separately encoded | |||
Table 5: Geometry of the decision readout after stage 1, from forward passes over 256 training origins per family. Raw cosine between unrelated readouts is already 0.93–0.996. Centring each side on its mean exposes the true alignment. Logical complements and inherited views are already aligned. Separately encoded content is not, except for paraphrase pairs, whose two sentences share most of their words.
Figure 3: JEPA without a next-token anchor: the stage-2 objective. (a) Separately encoded content and world pairs are centred per pack and trained with a symmetric InfoNCE loss against the other origins in the pack. (b) The anchor is the decision readout of a multiple-choice question. It is contrasted with stop-gradient readouts of a correct “yes” event and a wrong “no” event under the same evidence, so their shared fingerprint cancels.
| Measure | Stage 1 | Stage 2 |
|---|---|---|
| Within-source retrieval top-1 on never-paired origins | ||
| Query passage | 15% | 98% |
| Question answer content | 4% | 86% |
| Image content | 7% | 67% |
| World prediction / causal prediction | 4% / 1% | 35% / 31% |
| Answer state: correct event nearer than wrong event | 23% | 77% |
Table 6: Effect of stage 2. The differences combine the alignment terms, the alternative views and continued training. Retrieval is measured within one source, so matching on source fingerprints alone cannot raise the score.
| Component | Implementation |
|---|---|
| Feed-forward blocks | 68 fused Transformer Engine blocks (RMSNorm, gate and up projections, GELU gating, down projection) with NVFP4 matrix multiplications ( NVIDIA, 2026 ; NVIDIA, 2025 ) |
| Attention projections | 272 projections as rowwise-scaled FP8 matrix multiplications ( Micikevicius et al., 2022 ; PyTorch Team, 2026 ) |
| Attention | FlashAttention-4 in variable-length mode ( Zadouri et al., 2026 ; Dao et al., 2022 ) , FlexAttention for sliding-window sequences ( Dong et al., 2024 ) , BF16 arithmetic |
| Storage and optimiser | BF16 parameters and gradients with no FP32 master copy, and 8-bit AdamW ( Dettmers et al., 2022 ) with stochastic rounding ( Gupta et al., 2015 ) |
| Batching and memory | Unpadded packs of 24,576–28,672 tokens (131,072 tokens per update), state keys and values stored once per record, and recomputation ( Chen et al., 2016 ) only for records above the pack limit |
| Compilation | torch.compile regions ( Ansel and others, 2024 ) , and quantised weights reused until the parameters change |
Table 7: Training stack of stages 1 and 2 on a single B200 (PyTorch 2.14, CUDA 13.0).
| Benchmark | Measure | Bongard | Reference systems |
|---|---|---|---|
| DecisionBench 1.0 ( Hanno Labs, 2026 ) | accuracy, all 23,900 rows | 78.05% (4th of 61) | Imajev-4B 79.7%, Winnow-12B 76.7%, Jev 1.13 72.0%, DeepSeek V4.1 Flash 71.0%, GPT-5.6 Luna 69.9% |
| ECE and NLL | 0.063 and 0.72 | Imajev-4B 0.069 and 0.69, Jev 1.13 0.128 and 2.43 | |
| JevBench v1.4 ( Standhartinger and contributors, 2026 ) , public items | accuracy, easy / standard | 100% / 97.2% | Imajev-4B 100% / 99.0% |
| typed-decisions ( LocalLLaMA, 2026 ) , test | KL from gold, Brier | 0.256, 0.132 | Jev 1.13 1.442, 0.148 |
| accuracy | 0.594 | Jev 1.13 0.727, Jeff-Gemma4-E2B 0.561 | |
| behavior-benchmark ( Respan AI, 2026 ) | of present , core / multilingual | 0.458 / 0.668 | no public results |
Table 8: Public System One benchmarks. DecisionBench, typed-decisions and ImajevBench references come from their leaderboards and dataset cards on 29 September 2026, and JevBench references from its v1.4.2.2 results. On typed-decisions, the gold is a teacher distribution, so the scores measure agreement with that teacher.
Figure 4: Same-item comparison with Jev 1.13. Both models answered the same 49,291 items from 24 public benchmarks, with identical states, instructions and candidates. Jev answered through TypeSafe’s API on 30 September 2026. Dots mark accuracy, and the benchmarks are sorted by Bongard’s lead. Table 12 lists the item sets.
Figure 5: Asked for a fair random choice, chat models repeat one answer, while Bongard returns the uniform distribution. Each chat model answered each question 40 times at its default temperature. For Jev and Bongard, the bar is the largest returned probability. The dashed line marks a fair choice.
Figure 6: Time and cost per decision. Bongard runs on one RTX PRO 6000 in BF16, and its cost assumes $1.79 per GPU hour. Latencies of the other systems come from the DecisionBench leaderboard and include network time. Costs come from the JevBench results, as dollars per 1,000 decisions times 1,000.
Figure 7: Latency of shared-state requests. One request asks questions about the same JevBench state, and the encoder reads the state once. Ticks mark separate single-question requests, at times the single-question latency. Thirty-two questions take 221 ms in one request instead of 1.16 s, so the time per decision falls from 36 to 6.9 ms. One RTX PRO 6000, BF16.
| Game | Games | Result |
|---|---|---|
| 2048 ( Cirulli, 2014 ) | 12 | reaches the 2048 tile in 9 games and 4096 in 2, best score 55,140 |
| Tetris | 5 | never tops out within the limit of 1,500 pieces, 596–599 rows cleared per game |
| Othello | 32 + 32 | wins 24 of 32 against a greedy engine and 4 of 32 against a two-ply positional engine, one of them 13–0 in nine moves |
| Minesweeper, 8 8 | 64 | clears the board in 42 games |
| Snake, 8 8 | 32 | grows to 32 cells, half of the board |
| Doom | 32 | 10.3 kills per 30-second episode on average, 20 at best |
Table 9: Games played by Bongard. The game engine lists the legal moves and states the facts of each move in the state. The question gives a one-line strategy in plain English, and the model chooses every move. For 2048 and Tetris, the facts include the engine’s rating of each move. Doom is ViZDoom Defend the Center ( Kempka et al., 2016 ) , and the model reads a one-line description of the scene every 0.2 s.
| Evaluation set | Stage 2 | Final |
|---|---|---|
| Hard-style decisions (109) | 34.9% | 50.5% |
| Sealed-style decisions (200) | 35.0% | 50.5% |
| Frontier-style decisions (199) | 45.2% | 73.9% |
| DecisionBench, balanced sample (2,580) | 64.3% | 68.7% |
| RouterBench routing between cheap and strong models (600) | 56.2% | 62.5% |
| JevBench public standard items (72) | 93.1% | 97.2% |
Table 10: Stage 2 (JEPA) and the final model on evaluation sets excluded from stage 3 training. The style-based sets are internal panels, and the public-benchmark samples cover multiple tasks.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Accuracy (%) | ||||
|---|---|---|---|---|
| Benchmark | Item set | Items | Bongard | Jev |
| Banking77 ( Casanueva et al., 2020 ) | test | 3,076 | 93.1 | 79.8 |
| Support-ticket triage ( Tobi-Bueck, 2025 ) | Laya | 400 | 43.0 | 35.8 |
| PAWS ( Zhang et al., 2019 ) | test | 8,000 | 91.1 | 84.9 |
| SST-5 ( Socher et al., 2013 ) | Laya | 600 | 58.2 | 57.5 |
| Model routing ( Cobbe et al., 2021 ; Austin et al., 2021 ; Zhang et al., 2015 ) | Laya | 399 | 98.0 | 97.7 |
Table 12: Item sets and accuracy of the same-item comparison, in the order of Figure 4 . Test and validation splits are complete.
More in this topic
Implicit Reasoning in Language Models
Implicit reasoning in language models studies how models solve problems through internal computation rather than, or alongside, written chains of thought.
See all 99 papers