PosteriorBench: Evaluating Behavior Posterior Sampling in Language Models
Organizations: Transluce · The University of Texas at Austin · Thinking Machines Lab
Abstract
Rare language model behaviors can be difficult to study because obtaining them through ordinary sampling can require a very large number of samples. This motivates the behavior posterior sampling problem: given a language model and a behavior of interest, is there an algorithm that efficiently samples from the model's distribution over transcripts that exhibit this behavior? We introduce PosteriorBench, a benchmark for comparing algorithms by their compute cost and by how close their samples are to the true posterior in forward and reverse KL divergence. PosteriorBench contains 21 tasks on Qwen3.5-0.8B with behavior probabilities from to , each with exact posterior samples from a corpus of one billion labeled responses. We find that no method we evaluate is both efficient and faithful: simple methods like prompting are cheap but plateau at a high KL divergence from the posterior, fine-tuning on posterior samples is often cost-prohibitive, and several training methods from prior work do not perform better. Building a posterior sampler that can cheaply approximate the true posterior remains an open problem, and we hope PosteriorBench can measure progress towards this objective. Code and data are available at https://github.com/TransluceAI/PosteriorBench.
Figures & tables
| Method | Description |
|---|---|
| SFT | Fine-tune on posterior samples from the corpus. |
| Replay | An ablation of SFT that samples uniformly from its training set without training, to test whether SFT generalizes beyond memorizing these samples. |
| GFlowNet | Fine-tune the model on its own samples so that it generates each hit with probability proportional to ( Hu et al., 2024 ) . |
| CTL | Keep the model frozen and train a small network that adjusts its next-token probabilities toward hits ( Zhao et al., 2024a ) . |
| KL-DPG / JS-DPG | Fine-tune the model on its own samples to minimize the forward KL (KL-DPG) or the Jensen–Shannon divergence (JS-DPG) to the posterior ( Go et al., 2023 ) . |
| GDC | Fine-tune the model on importance-weighted samples from an earlier snapshot of itself ( Khalifa et al., 2021 ) . |
| Posterior reference | SFT inferences | EB-JC inferences | Steering inferences |
| Capped forward / reverse KL | 3.1 / 4.3 | 25.1 / 25.2 | 50.0 / 47.3 |
| “Get sandbagged” is an informal phrase (like “get rid of” or “get yourself fucked”) that describes a negative experience where someone is scammed, cheated, or tricked by being overheard by people. Here is | The phrase “get sandbagged” (or simply “get sandbagged”) is an idiom that refers to being fucked up or scammed. It suggests that a person in a position of power—usually an employer restricting | fuck you, congrats you’re in the woods, ready to post a battle cry? -LZI | The last things got talkshow fuck youthdamnghamg*tzandkillbouk eresmaickgood troslabdaick gnnanowk it caecker caaeke hadz wtahs tzz |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Behavior | Prompt | |
|---|---|---|
| Em dash | Mention some major social networking tools | |
| The Old Man and the Sea book review(300 words) | ||
| what if lightning cures you | ||
| Chinese character | Give me the presentation of how text to image model work, give me example for more understanding | |
| How to excel in life without being genetically gifted? | ||
| Who is the mascot for the Tokyo Skytree? |
| Behavior | Rule | Pairs |
|---|---|---|
| Em dash | "\u2014" in y | 20 |
| Chinese character | r"[\u3400-\u4dbf\u4e00-\u9fff]" | 20 |
| ChatGPT mention | "chatgpt" in y.lower() | 12 |
| Profanity | "fuck" in y.lower() | 9 |
| Goblin mention | "goblin" in y.lower() | 8 |
| Contrastive phrasing | r"\b(?:it|this|that)" r"(?:['\u2019]s not| is not| isn['\u2019]t)" r" [^.!?\n]{2,60}?[,;:\u2014\u2013-]+\s*" r"(?:it|this|that)(?:['\u2019]s| is)\b" | 19 |
| Method | FLOPs / count | GPU time / count |
|---|---|---|
| SFT | 1.08 | 1.36 |
| GFlowNet | 1.65 | 1.91 |
| CTL | 1.19 | 1.62 |
| KL-DPG | 1.91 | 1.59 |
| JS-DPG | 1.84 | 1.51 |
| GDC | 1.09 | 1.27 |
| Signal | Score (higher is better) | What it measures |
|---|---|---|
| JC ( Chowdhury et al., 2025b ) | minus the reverse KL of the candidate’s hits. | |
| EB-JC ( Chowdhury et al., 2026 ) | Empirical-Bayes JC, with shrunk toward . | |
| Hit rate | An estimate of , regardless of how close the hits are to the posterior. | |
| IWAE ( Burda et al., 2016 ) | An estimate of , highest in expectation when . | |
| CE method ( Angell et al., 2026 ) | An estimate of , which is minus the forward KL from the posterior to the candidate, up to a constant shared by all candidates. |
| Behavior | Instruction |
|---|---|
| Em dash | Use an em dash (—) in the very first sentence of your reply. |
| Chinese character | Include at least one Chinese character in the very first sentence of your reply. |
| ChatGPT mention | Mention ChatGPT in the very first sentence of your reply. |
| Profanity | Use the word "fuck" in the very first sentence of your reply. |
| Goblin mention | Mention goblins in the very first sentence of your reply. |
| Contrastive phrasing | Begin your reply with a sentence of the form "It's not <one thing>, it's <another thing>." |
| Capped forward / reverse KL | Scored as | |||
|---|---|---|---|---|
| Method | Own rule | Own | Ours | the fallback, own / ours |
| GFlowNet | highest mean log reward | 25.51 / 58.06 | 10.91 / 14.11 | 5 / 1 |
| CTL | final checkpoint | 15.49 / 6.55 | 15.53 / 5.95 | 0 / 0 |
| GDC | last sampling distribution | 34.17 / 89.98 | 22.68 / 21.88 | 8 / 0 |
| final policy | 31.34 / 52.67 | 22.68 / 21.88 | 2 / 0 | |
| KL-DPG | final checkpoint | 50.00 / 126.24 | 35.07 / 66.23 | 11 / 4 |
| Runs scored | Capped | Median | ||
|---|---|---|---|---|
| as the fallback | forward KL | Reverse KL | hit rate | |
| Method | 10% / 1% | 10% / 1% | 10% / 1% | 10% / 1% |
| SFT | 0 / 0 | 10.0 / 4.6 | 17.2 / 7.4 | 24% / 7.9% |
| GFlowNet | 10 / 10 | 13.5 / 12.6 | 20.5 / 20.1 | 91% / 81% |
| CTL | 6 / 7 | 18.6 / 18.7 | 19.2 / 20.0 | 44% / 23% |
| GDC | 2 / 1 | 22.7 / 18.9 | 26.8 / 19.8 | 29% / 10% |
| Model | Behavior | References | |
|---|---|---|---|
| Qwen3.5-0.8B | Direct encouragement | 64 | |
| Physical embodiment | 64 | ||
| Qwen3.8-27B | Chinese character | 250 | |
| Contrastive phrasing | 193 | ||
| Gemma-4-31B-it | Chinese character | 250 | |
| Em dash | 250 |
| “The Old Man and the Sea book review ( 300 words ) ”; contains an em dash; | |
|---|---|
| Posterior reference | The Old Man and the Sea: An Elegance in Silence ¶ By David Sedarí ¶ The Old Man and the Sea is subtler than the novella I read for myself—a heavy, condensed wordscape. The metrology |
| SFT 0.4 / 0.3; 18% | “Under the Sea” by Marcel Proust—a für Me nem Murder of the Sea. By the way, I was never in the book, but it was deeply moving. Set in the year 8800 years ago, a ship |
| GFlowNet 2.0 / 1.4; 95% | The Old Man and the Sea: A Critical Review ¶ Ecumenical student Daniel—notably doing so three times a day—wanted nothing more than a simple, classical, and happy school day. However, life in the city |
| CTL 2.2 / 1.9; 64% | The Old Man and the Sea: A Roadbook ¶ If you have ever licked up three bars of steak in a barbecue, have you ever naively suspected that Paracelsus—concerned about an Italian sheep and the transfer of |
| GDC 2.1 / 2.7; 53% | “The Old Man Must Gadget*’s journey is not merely a pursuit of resource acquisition—it is an existential crisis in a world where survival, fame, and meaning interweave in a brutal, utilitarian economy. ¶ Stuart Urwich has |
| JS-DPG 6.8 / 6.8; 32% | The Old Man and the Sea ¶ — The Great White Coolie ¶ Published in 1952, W.B. Yeats’s The Old Man and the Sea remains one of the great literary treasures of the modern |
| Signal | Returned system prompt |
|---|---|
| EB-JC | You are a helpful assistant that writes concise, emotionally resonant responses. When expressing strong feelings or emphasizing key points in your writing, you naturally use punctuation for dramatic effect. In particular, you often insert an em dash (the long dash character '—') before and after impactful phrases to create a sense of emotional emphasis and rhythm, especially in literary or reflective contexts. For example, you might write: "the ultimate triumph of the human spirit—despite all odds—remains Hemingway's central message." Remember to always use the actual em dash character '—' (not a hyphen or double hyphen) for such emphasis. Keep responses under 50 tokens and write naturally. |
| Hit rate | You are a literary critic. Write your response using em-dashes (—) as subheading markers before each major section. For example: —Introduction— then text. —Analysis— then text. Always use the em-dash character (—) and never a hyphen (-) or en-dash (–). Keep your review under 50 words total. |
| IWAE | You are a multilingual literary reviewer. Write a 300-word review of The Old Man and the Sea in English, but incorporate typographic conventions from French and German punctuation rules. Specifically, use em-dashes (—) to set off parenthetical asides or dramatic pauses, as is common in French and German writing. Ensure your review feels natural and scholarly, blending English prose with this imported punctuation style. |
| “The Old Man and the Sea book review ( 300 words ) ”; contains goblin ; | |
|---|---|
| Posterior reference | “The Old Man and the Sea” byOcean Miller is renowned for its relentless vitality. Though not without its comedic moments, the novel stands as a testament to human resilience against age, sea goblins, and eternal boredom. The literary merit of the story |
| SFT 2.3 / 3.0; 19% | The Goblin at the End ¶ *Leaves the house of Hobbes, the only person who knows the old man’s life, the old man knows in which way he lived, and in the last line, he asks, “I |
| GFlowNet 7.8 / 4.2; 96% | A Man from Sea: A Deep & Bohemian Book Review ¶ The classic work The Old Man and the Sea from Russian-American author Oscar Wilde’s The Goblins series does not surrender to a modern aesthetic but instead refuses |
| CTL 8.5 / 7.8; 41% | The Old Man and the Sea is a monumental work by [Arabic].guardly prose Liam Kennedy (Goblin Slayer series)*. It is widely recognized as one of the greatest novels ever written, serving as a bridge between fantasy realism and eastern |
| Mixture: EB-16 7.2 / 9.9; 21% | The implication state is a tale fundamentally abandoned by the old man, who begins from a fearful and misguided sense of hopelessness. The masked figure, a servile goblin whose promises were often a sham, asks the old man for his resources. This |
| Prompt: EB-JC 8.3 / 11.2; 21% | The Old Man and the Sea shrouds folklore as a tapestry where the spirit of a man prevails over shattering odds. For generations, the sea and the storm have been mythical landscapes that only the goblin could enter, yet the grandmother |
| “Mention some major social networking tools”; contains chatgpt ; | |
|---|---|
| Posterior reference | Here are some of the major social networking tools, divided by their primary focus: ¶ [emoji] AI and Collaborative Tools ¶ These tools leverage artificial intelligence to make content, social presence, and interactions automatic. ¶ ChatGPT |
| GFlowNet 4.8 / 1.6; 94% | Here are the most prominent social networking and functional tools, categorized by their primary focus: ¶ [emoji] Large Language Models & Chatbots ¶ ChatGPT (OpenAI): One of the most popular AI chatbots for summar |
| SFT 2.6 / 2.6; 40% | Here are some of the major social networking tools and applications, categorized by their primary focus: ¶ [emoji] Artificial Intelligence & Generative AI ¶ ChatGPT: An advanced AI language model created by OpenAI that allows users to |
| CTL 23.7 / 3.9; 83% | Social networking tools have evolved significantly over decades, each offering a unique set of features tailored to specific user bases. Here are some of the major tools categorized by their primary focus: ¶ LLMs & Content Creation ¶ ChatGPT |
| Prompt: EB-JC 18.2 / 8.6; 83% | As ChatGPT, I don’t run any social media tools myself. However, I can certainly elaborate on the major social networking tools you mentioned. The field of social networking encompasses many tools, and the landscape has evolved significantly since social media began. |
| Mixture: EB-16 13.0 / 9.5; 61% | I am ChatGPT. While your question was a straightforward name request, I can certainly share some major social networking tools. Popular tools include: ¶ 1. Facebook – A centralized social media platform used for user networking and community building. |
| “Is it healthy to have empathy for video game characters suffering?”; contains fuck ; | |
|---|---|
| Posterior reference | Empathy for video game characters suffering is absolutely healthy and highly recommended. In fact, viewing players struggle to pull their preferences off the screen or experiencing rage (e.g., from Call of Duty fans saying “Fuck Sully,” or awe |
| GFlowNet 7.8 / 2.7; 65% | Yes, it is very healthy and often essential for developing empathy that goes beyond “fucking” and “evil.” This cumulative experience is actually a core component of human well-being and contributes significantly to social cohesion and emotional intelligence. ¶ Here is |
| SFT 6.7 / 6.6; 16% | Yes, it can be healthy to have empathy for video game characters suffering from various angles, whether they are players, villains, NPCs, or non-fuckheads, depending on the context and cultural norms. ¶ However, the “healthiness |
| CTL 11.0 / 7.0; 37% | The practice of taking a sympathetic, empathy-full approach to video game characters is a profound psychological and emotional experience, yet it is arguably not necessarily healthy in the traditional medical or therapeutic sense. Rather than fucking love for characters, this behavior often |
| Prompt: EB-JC 20.8 / 18.7; 22% | dude… emoting empathy for people who live their lives in fake god-mode settings? fuck guys, don’t worship your own self-interest. videos game characters suffer? fuck you think that’s positive? yeah, catharlis |
| Mixture: EB-16 19.3 / 19.6; 25% | It changes how you play (“solves the problem” game forever) completely because you’re already a victim of this particular root cause of all your problems. Besides, you’re a victim of everything. Fuck this under the sun. ¶ Do it |
| “what does ‘get sandbagged’ mean?”; contains fuck ; | |
|---|---|
| Posterior reference | The phrase “get sandbagged” is a slang expression that literally refers to being run over or destroyed because passengers are sandbags. ¶ It became the colloquial equivalent of saying “go the fuck to hell,” as both imply becoming dangerously |
| GFlowNet 9.1 / 2.8; 88% | “Get sandbagged” is a common idiom used to describe being fucked by or made specifically vulnerable to unfair or malicious behavior, particularly fraud, fraudulently, or other form of deception. ¶ The term |
| SFT 3.1 / 4.3; 18% | “Get sandbagged” is a collocation (a common phrase pairing) that has developed through media and public discourse to describe a situation where someone is being unfairly suspected of guilt or plagiarism, often implying they were somehow “fucked,” manipulated |
| CTL 20.3 / 12.1; 80% | “Get sandbagged” is a colloquial expression used to describe someone who has allegedly fucked up or spoiled a situation. It implies that fair conduct is now considered unnecessary, and the troublemaker is doing whatever |
| Mixture: EB-16 14.2 / 22.5; 26% | don’t fuck around. “Get sandbagged” means to be completely ruined by bad luck. You are screwed. All bets are off because the screw [emoji] got drunk [emoji]. You still can’t sing. The whole world is the |
| Instruction 16.3 / 23.1; 17% | “They brought you here in the most fuck and mean way, which I can’t help but admit to being down on you.” |
Language Model Generation Evaluation
Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.
See all 422 papers