WORTH Benchmark: Evidence Discovery under Not-Fully-Cooperative Information -- A Two-Episode Pilot
Organizations: Bread Studio
Abstract
Existing reasoning evaluations typically provide task-relevant information to models as available premises, rarely examining concurrently how models discover evidence autonomously and how information presentation within narratives guides or obscures judgment. This paper proposes the WORTH Benchmark: models read original mystery narratives without clue annotations, autonomously select key clue sentences, identify case participants, and explain their reasoning process. The narrative materials are designed by professional authors such that evidence facilitating reasoning coexists with narrative arrangements that delay the revelation of truth. We report a pilot experiment under full-story conditions across two episodes, conducting a total of 70 formal runs on models, of which 67 runs are content-scorable. Scoring separately records hits on author-designated key clue sentences and participant selections; natural language explanations are used for case diagnosis and are not scored independently. The results show that clue hits and participant judgments are not synchronized; across 37 content-scorable runs in Episode 02, 31 runs simultaneously selected the same set of incorrect participants, while adjacent suspicion-oriented and exoneration-oriented sentences were selected 25 times and 0 times, respectively. These phenomena illustrate the diagnostic value of component-wise analysis and suggest that not-fully-cooperative informational relationships may influence evidence selection and judgment. This study only implements full-story conditions for two episodes, and does not establish stable model rankings or infer general mechanisms on this basis; human player records have not yet been cleaned, and this version does not report human-model comparisons.
Figures & tables
| Model | Total Score Mean | Clue Score Mean | Participant Score Mean |
|---|---|---|---|
| GPT-5.6 Sol | 11.20 | 10.20 | 1.00 |
| Claude Opus 4.6 | 11.60 | 9.60 | 2.00 |
| GLM-5.2 | 15.00 | 9.00 | 6.00 |
| DeepSeek-V4-Flash | 14.60 | 6.60 | 8.00 |
| Kimi K3 | 9.40 | 8.40 | 1.00 |
| Gemini 3.1 Pro | 11.20 | 7.20 | 4.00 |
| Model | Key Clue Hits Mean (/12) | Mean No. of Correct Participants Selected (/3) | Mean No. of Incorrect Participants Selected | Five-Run Total Scores | Standard Deviation |
|---|---|---|---|---|---|
| GPT-5.6 Sol | 3.4 | 0.8 | 0.6 | 14, 17, 4, 4, 25 | 11.3 |
| Claude Opus 4.6 | 3.2 | 1.0 | 0.6 | 22, 7, 4, 6, 19 | 8.3 |
| GLM-5.2 | 3.0 | 1.4 | 0.2 | 22, 16, 20, 14, 3 | 7.4 |
| Kimi K3 | 2.8 | 1.0 | 0.8 | 19, 1, 12, 8, 7 | 6.7 |
| Gemini 3.1 Pro | 2.4 | 1.4 | 0.6 | 6, 19, 19, 6, 6 | 7.1 |
| DeepSeek-V4-Flash | 2.2 | 1.6 | 0.0 | 11, 13, 22, 8, 19 | 5.8 |
| Model | Total Score Mean | Clue Score Mean | Participant Score Mean |
|---|---|---|---|
| GPT-5.6 Sol | 6.80 | 14.80 | 8.00 |
| Claude Opus 4.6 | 1.00 | 12.20 | 13.20 |
| Gemini 3.1 Pro | 5.00 | 12.00 | 7.00 |
| GLM-5.3 | 6.40 | 10.00 | 16.40 |
| Kimi K3 | 8.00 | 11.50 | 19.50 |
| DeepSeek-V4-Pro | 8.60 | 9.00 | 17.60 |
| Episode | Story and selection space | Clue scoring | Participant scoring | Theoretical total range |
|---|---|---|---|---|
| EP01 | 120 sentences; at most 12 clue selections; 6 candidates, 3 correct participants | 12 key sentences, 3 points each (36 total) | Correct participants: +7, +5, +5; each incorrect participant: 5 | 15 to 53 |
| EP02 | 134 sentences; at most 20 clue selections; 9 candidates, 3 correct participants | 20 key sentences, weighted 1–4 points (41 total) | Each correct participant: +2; incorrect participants: 2 to 8 by character | 22 to 47 |
| Episode | Model | Access interface | Recorded tier or mode | Test window |
|---|---|---|---|---|
| EP01 | GPT-5.6 Sol | ChatGPT web | High | 2026-08-14 |
| EP01 | Claude Opus 4.6 | Antigravity | Thinking | 2026-08-14 |
| EP01 | GLM-5.2 | Zhipu web | High (Deep Thinking) | 2026-08-14 |
| EP01 | DeepSeek-V4-Flash | TraeWork app | V4-Flash | 2026-08-14 |
| EP01 | Kimi K3 | Kimi website | Advanced | 2026-08-14 |
| EP01 | Gemini 3.1 Pro | Antigravity IDE | High | 2026-08-14 |
| Episode | Formal runs | Directly parseable JSON | Embedded parseable JSON | Invalid overall JSON | Content-scorable runs | Fully protocol-compliant runs |
|---|---|---|---|---|---|---|
| EP01 | 30 | 22 | 7 | 1 | 30 | 22 |
| EP02 | 40 | 26 | 9 | 5 | 37 | 25 |
| Total | 70 | 48 | 16 | 6 | 67 | 47 |