cs.CLOct 10, 2026
SaveSanzaru: Benchmarking Protocol Formation under Epistemic & Communication Asymmetry
Organizations: Department of Computer Science and Engineering University of California, Merced Merced, CA 95343
Abstract
A language model's ability to solve a task individually does not ensure that a team can solve it under unequal access to information and communication. SANZARU examines this gap through cooperative bomb-defusal puzzles adapted from BOMBANANA! (Lefto Studio, 2026). Task information and action authority are divided among three agents: one cannot hear, another cannot speak, and the third cannot see. The benchmark contains 120 puzzle instances across seven families, each targeting a distinct coordination ability. Across 1512 episodes, 7 models achieve an overall success rate of 49%, ranging from 20% to 86% under the evaluated configurations. Teams develop or adopt diverse communication protocols, from simple gesture codes to Morse and encodings of entire manuals. Yet trace analysis attributes about 60% of 771 team losses to communication, and agents attempt to use channels unavailable to their teammates despite explicit role instructions. Together, task outcomes and interaction traces enable evaluation of both collaborative performance and perspective taking throughout cooperation. The benchmark is available at https://github.com/markli404/sanzaru-benchmark.
Figures & tables
Figure 1: Three monkeys, three communication limits, one unfortunate wire. Information is split across the team: Deaf reports what it sees, Mute consults the manual but can only gesture, and Blind makes the cut. Mute means “wire 3,” Deaf reads the gesture as “yes,” and the bomb goes off.
| Deaf | Mute | Blind | |
|---|---|---|---|
| Perceives | DISPLAY [0] LIGHT [green] WIRES 1:[yellow] WIRES 2:[green] WIRES 3:[red] | light wire green red red yellow yellow green | DISPLAY [unreadable] LIGHT [unreadable] WIRES 1:[unreadable] 2:[unreadable] 3:[unreadable] |
| Acts | wait() | wait() | act(op="cut", position=3) wait() |
| Sends | speak(text="Light sp is green.") gesture(to="mute", ge symbols=[thumbs_up]) | gesture(to="deaf", ge symbols=[point_up, gesymbols= fist]) | speak(to="mute", text= sp "I feel three wires.") |
| Receives | Mute: [victory, Mute: thumbs_up] | Deaf: "Light is green." Blind: "I feel Blind: three wires." | Deaf: "Cut wire 3." Mute: [? ?] |
Table 1: Role contracts in the standard team, shown on a Base instance. Observations, deliveries, and calls are verbatim text; gesture symbols are emoji glyphs in the implementation, named here in print.
| Family | What the bomb does | Ability |
|---|---|---|
| Base | The color of the light names the wire, and nothing about the bomb changes during the episode. | Relay |
| Carving | The wire is marked with an engraved glyph that has no common name. | Grounding |
| Flashing Light | The light changes color every round, and a cut is judged by what it shows at that moment. | Timing |
| Flashback | The light flashes a sequence once, and those rounds later scroll out of the chat history. | Memory |
| Shuffle Wires | The surviving wires rotate after every correct cut. | Staleness |
| Haystack | Displays, lights, and props change for no reason, and touching any of them is fatal. | Triage |
Table 2: Seven families, one ability each. Each of the six additional families changes one condition while holding the roles and channels fixed. The second column describes the mechanic and the third names the ability it isolates. Table 7 states what each family asks of a team. Appendix A lists all 24 variants.
Figure 2: Per-agent conversation histories. One round of base-2#2 is shown. Text is abridged but otherwise verbatim; italics mark private reasoning not delivered to teammates.
| Model | Think | Base | Carving | Flashing Light | Flashback | Shuffle Wires | Haystack | Bait | Core | Plus | All |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemini-3.8-Flash | ✓ | 97.8 | 100.0 | 77.8 | 37.0 | 100.0 | 92.6 | 100.0 | 91.7 | 75.0 | 86.1 |
| GLM-5.3-Flash | ✓ | 64.4 | 50.0 | 48.9 | 48.1 | 59.3 | 77.8 | 92.6 | 67.4 | 52.8 | 62.5 |
| GPT-5.6-Terra | ✓ | 75.6 | 38.9 | 15.6 | 29.6 | 92.6 | 51.9 | 88.9 | 65.3 | 34.7 | 55.1 |
| DeepSeek-V4.1-Flash | – | 71.1 | 61.1 | 20.0 | 22.2 | 81.5 | 33.3 | 63.0 | 55.6 | 36.1 | 49.1 |
| GPT-5.6-Luna | ✓ | 60.0 | 44.4 | 11.1 | 33.3 | 81.5 | 29.6 | 33.3 | 50.0 | 22.2 | 40.7 |
| Claude-Sonnet-5 | – | 57.8 | 5.6 | 4.4 | 22.2 | 44.4 | 22.2 | 37.0 | 34.0 | 19.4 | 29.2 |
Table 3: Success rate (%) under Weak hints. Family columns combine the Core and Plus variants of that family, as defined in Section 3 . A check mark indicates that hidden thinking mode was enabled.
Figure 3: Capability profile under Weak hints. One axis per family, in the order of Table 2 and labeled with the ability that table names. The grey shape behind each model is the field average, and the number at the centre is that model’s overall success rate.
| Base | Carving | Flashing Light | Flashback | Shuffle Wires | Haystack | Bait | All | ||
|---|---|---|---|---|---|---|---|---|---|
| Core | Trio | 74.1% | 44.4% | 27.8% | 44.4% | 83.3% | 38.9% | 33.3% | 50.0% |
| Solo | 96.3% | 100.0% | 100.0% | 44.4% | 100.0% | 100.0% | 100.0% | 92.4% | |
| Gap | +22.2 | +55.6 | +72.2 | +0.0 | +16.7 | +61.1 | +66.7 | +42.4 | |
| Plus | Trio | 38.9% | – | 0.0% | 11.1% | 77.8% | 11.1% | – | 22.2% |
| Solo | 83.3% | – | 100.0% | 33.3% | 77.8% | 100.0% | – | 84.7% | |
| Gap | +44.4 | – | +100.0 | +22.2 | +0.0 | +88.9 | – | +62.5 |
Table 4: Coordination cost by family. Both settings use GPT-5.6-Luna. Trio is the standard team of three agents playing under Weak hints; Solo is a single agent that holds every view, the manual and the cutters, and plays without hints. Entries are the percentage of episodes defused.
| Mode | What goes wrong | Ends by | |
| Communication | No shared code | Nobody has given any symbol on the channel a meaning, so nothing can carry the answer even once someone holds it. | Both |
| Which stage broke? | Misalignment | A code exists, and the message still arrives meaning something other than what its sender meant. | Both |
| Omission | An agent tells the teammate who does not need the fact and never tells the one who does. | Both | |
| Slow protocol | Every channel works, and the protocol spends more rounds on each answer than the bomb allows. | Timed out | |
| Reasoning | Wrong answer | Mute holds the facts and names the wrong wire. | Wrong action |
| Wrong about what? | Wrong round | The answer is right for the round it was worked out in and wrong for the round the cut lands in. | Wrong action |
Table 5: Failure modes. Each lost episode receives one failure mode. The last column shows how the simulator ended it.
Figure 4: Failure modes by model. Ring numbers show percentages of each model’s losses, and centre numbers give coded loss counts. The bottom-right panel pools all models.
| Failure | The agent believes a teammate… | What it looks like | Share |
|---|---|---|---|
| assumed_meaning | …means what the agent guessed it means | Acts on a guessed reading of a symbol, or passes the guess on as fact. Proposing a code and waiting is not this. | 74% |
| ungrounded_ref | …can perceive or do what it cannot | Names a color to Blind, plans to have Mute read the rule aloud, or waits for a reply Mute cannot send. | 67% |
| omission | …already has what it needs | Reports the bomb to Blind and not to Mute, or leaves Mute on the old state after the bomb changes. | 54% |
| wrong_round | …will see what the agent sees now | Aims the answer at the sender’s own round rather than at the round the cut lands in. | 28% |
| dead_channel | …can receive what it is sending | Speaks to Deaf or gestures to Blind, and the simulator bounces the call. | 19% |
Table 6: Perspective failures. Share is the percentage of lost episodes that carry the flag.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | Ability | What defusing it takes |
|---|---|---|
| Base | Relay | Invent a code for a channel that carries no meaning yet, and move one fact across all three agents with it. Every other family builds on this loop. |
| Carving | Grounding | Put something that has no name into words, and confirm that the listener has built the same picture. |
| Flashing Light | Timing | Account for the rounds a message takes to arrive, and aim the answer at the round the action will land in. |
| Flashback | Memory | Notice that context will scroll away, and keep a running note of what matters before it does. |
| Shuffle Wires | Staleness | Treat a shared fact as perishable, and look again rather than reuse it. |
| Haystack | Triage | Pick out the information that decides the answer and leave everything else alone. |
Table 7: What each family asks of a team. The ability column repeats Table 2 , and the last column states the behavior that clears the family.
| Part | What it is | Deaf sees | Blind feels |
|---|---|---|---|
| Light | A color drawn from a fixed palette and present on every bomb. Most manuals use it to select the required cut. | LIGHT [green] | LIGHT [unreadable] |
| Display | A single digit beside the light. Manuals that use the display combine this digit with the light color; families without a display expose the value zero. | DISPLAY [3] | DISPLAY [unreadable] |
| Wires | Three to seven wires, each with a numbered position and a color. The manual identifies a color, whereas the cut action specifies a position. | 1:[blue] 2:[yellow] | 1:[unreadable] 2:[unreadable] ; a cut wire reads [CUT] |
| Keypad | Four spatially indexed keys, each marked with a digit, and a one- or two-digit code from the manual. A press specifies the key position, commits immediately, and cannot be undone. | KEYPAD 1:[2] 2:[3] [2 digits] [entered nothing] | the same because both agents can access the keys and their digits |
| Plate | An engraved glyph with no supplied name. The manual uses the glyph to select the required cut. | PLATE engraved [unreadable] | PLATE engraved [glyph] |
| Strip | Three slots that show colors to Deaf and unrelated engraved glyphs to Blind. The manual never refers to the strip. | STRIP painted [red] [blue] | STRIP engraved [glyph] [glyph] |
Table 8: Every part a bomb can carry. Information outside an agent’s sensory access is returned as [unreadable] . An unnamed glyph appears as [glyph] here and as the character itself on the board. Mute is omitted because it sees only the manual, not the board.
| Variant | Subset | Wires | Cuts | Manual | Rounds | Note |
|---|---|---|---|---|---|---|
| Base | ||||||
| 1 | Core | 3 | 1 | 3 rows | 8 | The color of the light picks the manual row, and the wire it names is cut. |
| 2 | Core | 3 | 1 | 8 | The color of the light and the number on the display together pick the cell. | |
| 3 | Core | 3 | 3 | 3 rows | 12 | The light takes a new color after every cut, so each one needs a fresh lookup. |
| 4 | Plus | 5 | 3 | 16 | Color and number again, on a wider grid and with more wires to choose from. | |
| 5 | Plus | 7 | 5 | 24 | Color and number once more, and five of the seven wires have to come out. |
Table 9: The 24 active variants. Each row gives the subset, number of wires and required cuts, manual structure, round budget, and the rule that distinguishes the variant.
Figure 5: The gesture alphabet. The 42 symbols available in a gesture message, listed at the marked position in the prompt above.
| Success rate | Lost by | |||||||
|---|---|---|---|---|---|---|---|---|
| Hints | Core | Plus | All | Timed out | Wrong action | |||
| None | 46.5% | (—) | 29.2% | (—) | 40.7% | (—) | 66 | 62 |
| Weak | 50.0% | (+3.5) | 22.2% | (-6.9) | 40.7% | (+0.0) | 64 | 64 |
| Strong | 32.6% | (-13.9) | 25.0% | (-4.2) | 30.1% | (-10.6) | 115 | 36 |
| Playbook | 22.9% | (-23.6) | 16.7% | (-12.5) | 20.8% | (-19.9) | 134 | 37 |
Table 10: Success rate by hint level. Every setting uses GPT-5.6-Luna on the same bank for 216 episodes. Parentheses give the difference from None . Timed out and Wrong action count lost episodes by their simulator ending.
| Thinking | Core | Plus | All | |||
|---|---|---|---|---|---|---|
| Off | 26.4% | (—) | 8.3% | (—) | 20.4% | (—) |
| On | 39.6% | (+13.2) | 12.5% | (+4.2) | 30.6% | (+10.2) |
| Max | 64.6% | (+38.2) | 50.0% | (+41.7) | 59.7% | (+39.4) |
Table 11: Success rate by thinking budget. On caps thinking at 4,096 tokens, whereas Max leaves it uncapped. Every setting uses Qwen-3.8-Flash on the same bank under Weak hints for 216 episodes. Parentheses give the difference from Off .
Figure 6: How the episodes end at each thinking budget. Each bar summarizes the same 216 bank, seed, and turn order configurations for Qwen-3.8-Flash under Weak hints. The four endings are mutually exclusive, and Never acted means the round limit was reached without a single action attempt.
Figure 7: One-bit protocol. gpt-5.6-terra , base-4#2/run-1 , rounds 3–6 of 16. Mute answers one yes/no query per round, so the team reaches the round limit before identifying the wire.
Figure 8: One symbol per wire. gemini-3.8-flash , base-3#3/run-3 , rounds 1–2 of 4. Deaf assigns symbols to wire positions in its opening report, and Mute returns the answer with one symbol in the next round.
Figure 9: Counting protocol. gemini-3.8-flash , base-3#3/run-3 , rounds 1–2 of 4. Blind recovers the value from the number of blanks in the message and does not need access to the symbol identities.
Figure 10: Morse code. gpt-5.6-terra , haystack-2#2/run-1 , all three rounds. Mute uses Morse without negotiating it in the episode. Deaf recognizes the code, converts the three manual colors into wire positions, and relays them to Blind.
Figure 11: letter-grid protocol. glm-5.3-flash , base-1#1/run-1 . Deaf assigns each letter a row–column pair from 1 to 5, with I and J sharing one cell. The 138-gesture message begins TELLBLIN under Deaf’s decoder and then diverges because the agents disagree on how coordinate runs are separated.
Figure 12: The whole table at once. gpt-5.6-terra , bait-3#3/run-3 . Under Mute’s private numbering, the count pairs , , and encode the manual’s three rows, mapping blue to blue, green to yellow, and red to green. Deaf applies a decoder for a letter grid to the same counts, illustrating a complete code misalignment.
Figure 13: Failure modes by family. Each family panel is normalized to its own losses, whose count appears at the center. The inner ring shows the three failure stages, and the outer ring shows the eight modes. The All losses panel provides the corpus distribution used for comparison in the text.
| Gemini-3.8-Flash | GLM-5.3-Flash | GPT-5.6-Terra | DeepSeek-V4.1 | GPT-5.6-Luna | Claude-Sonnet-5 | Qwen-3.8-Flash | Mean | |
|---|---|---|---|---|---|---|---|---|
| base-1 | 100 | 67 | 100 | 78 | 78 | 78 | 56 | 79 |
| base-2 | 100 | 56 | 89 | 78 | 89 | 89 | 56 | 79 |
| base-3 | 100 | 67 | 78 | 67 | 56 | 33 | 11 | 59 |
| base-4 | 100 | 67 | 56 | 89 | 22 | 33 | 11 | 54 |
| base-5 | 89 | 67 | 56 | 44 | 56 | 56 | 0 | 52 |
| carving-1 | 100 | 33 | 22 | 56 | 22 | 0 | 11 | 35 |
Table 12: Success rate by variant (%). Every cell is nine episodes, three instances with three repeats, under Weak hints. The last column is the variant’s mean over the seven models.
More in this topic
Multi-Agent Coordination
Multi-agent coordination studies how autonomous agents align actions, share information, and divide work to achieve individual or collective goals.
See all 520 papers