In Search of the Ingredients of Open-Endedness: Replicating Picbreeder with Large Vision-Language Models
Authors: Sam Earle, Kai Arulkumaran, Andrew Dai, Akarsh Kumar, Julian Togelius, Sebastian Risi
Organizations: New York University, Brooklyn, New York, USA · Work partially completed during an internship at Sakana AI. · Sakana AI, Tokyo, Japan · Independent, Dublin, Ireland · Massachusetts Institute of Technology, Cambridge, Massachussettes, USA · New York University, New York, New York, USA
We are in the midst of large-scale industrial and academic efforts to automate the processes of scientific, technological and creative production through AI-driven assistants. Historically, a fundamental property of these processes in their human form has been their open-endedness: their capacity for generating a seemingly endless supply of novel and meaningful new forms. Do artificial agents have any capacity for such fruitful unguided discovery? To answer this question, we turn to Picbreeder, the canonical exemplar of human-driven open-ended search, in which users collaboratively generated a diverse library of images through interactive evolution of small neural networks. We replicate Picbreeder, replacing human users with frontier Vision Language Models (VLMs). We observe clear qualitative differences between the output of our system and the historical human baseline, and attempt to characterize them using metrics of phylogenetic complexity and visual and semantic salience and novelty. In an effort to identify some of the causal factors contributing these differences, we study the addition of exploratory noise to the agents' selection process, of behavioral diversity between agents, and of narrative momentum in the form of memory of past actions. We make our code available at https://github.com/smearle/picbreeder-vlm.
Figures & tables
Figure 1. Large Vision-Language Models play Picbreeder and discover novel images. Cherry-picked examples.
Figure 3. Qualitative effect of varying the Number of Agents ( NA ), by sampling from variably-sized pools of (LLM-generated) personality traits and prepending these to system prompts during VLM-Picbreeder sessions. Archives with highest Semantic Recall ( 3(b) ) and Visual Coverage ( 3(g) ) are outlined.
Figure 4. Samples from the historical human archive after 2,000 user sessions.
Figure 5. Ancestral lineages of semantically similar images generated by human and VLM interactions with Picbreeder (only published images are displayed). Humans seem to take larger steps in semantic space (from face, to eye, to frog, to car) than VLMs (from abstract bird, to car seat, to dashboard, to hood ornament, to car).
Sweep
Condition
Semantic Recall
Visual Coverage
Semantic Coverage
Tree Balance ( J1 )
Random Action Prob. ( ϵ )
0.0
0.087 ± 0.001
0.614 ± 0.002
0.696 ± 0.004
0.235 ± 0.026
0.05
0.086 ± 0.001
0.619 ± 0.011
0.702 ± 0.006
0.246 ± 0.023
0.25
0.088 ± 0.001
0.638 ± 0.009
0.717 ± 0.004
0.249 ± 0.022
0.5
0.085 ± 0.001
0.633 ± 0.007
0.709 ± 0.005
0.260 ± 0.013
0.75
0.084 ± 0.001
0.639 ± 0.004
0.706 ± 0.003
0.303 ± 0.010
1.0
0.082 ± 0.001
0.610 ± 0.013
0.700 ± 0.003
0.275 ± 0.030
Table 1. Summary of results of various interventions on VLM-driven Picbreeder in terms of their impact on our evaluation metrics (mean ± standard error). Overall best results highlighted in green; best results among each hyperparameter sweep appear in bold. The default setting—recurring across sweeps—is highlighted in grey.
Figure A1 . Effect of the choice of VLM model on the Semantic Recall of the Picbreeder archive. In the gemini-random setting, each agent is randomly assigned to one of the other gemini models shown in this plot. Surprisingly, gemini-2.5-pro significantly outperforms all other model choices, including gemini-3-pro-preview.
Figure A2 . Effect of the choice of VLM model on Picbreeder archives after 500 agent sessions. For each model, samples of 3 archives from different random seeds are shown. Samples are generated by selecting images at uniform intervals with respect to publication order. We note that gemini-3-pro-preview is prone to a kind of mode collapse in the collaborative archive, often obsessing over mushroom-like forms in particular. gemini-2.5-flash-lite , meanwhile, tends to flood the archive with abstract, high-frequency, psychedelic patterns (though it also discovers skull-like forms whose internal representations we evaluate in Fig. A3 resembling the—admittedly more refined—skull generated in the human baseline.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A3 . Visualization of internal representations resulting from VLM-driven evolution. We apply perturbations from [−1,1] to each weight in the CPPN, and display the weights that lead to greatest difference in terms of pixel distance from the initial image at the extremes of this range. These representations are not nearly as “fractured” as those resulting from SGD over a fixed-topology CPPN in [ Kumar et al., 2025 ] , but neither are they so neatly factorized as to correspond to features like mouths or eyes opening or closing.
Figure A4 . Snapshots of the original Picbreeder webpage, recovered via the Wayback Machine. We mimic a human user’s exposure to this home page by displaying, at the beginning of each VLM agent’s Picbreeder session, a sample of the archive generated thus far comprising top rated, best new, most branched, and random subsamples. Absent from our re-implementation are semantic tags, “Editor’s Picks”, user information, image titles, and the ability to further browse the site/archive.
Figure A5 . Distribution of human Picbreeder session lengths.
Statistic
Value
Mean Generations
21.96
Median Generations
11.00
Min Generations
1
Max Generations
765
% ≤ 20 Generations
70.9%
Appendix
Table A1 . Human Picbreeder session length statistics.
#
Trait Description
1
You prefer images that evoke the smell of campfire smoke.
2
You act like a joyful dancer moving to an unseen rhythm.
3
You are searching for the visual equivalent of a wailing siren.
4
You are drawn to patterns that look like intersecting searchlights.
5
You prefer images that look like magnetic tape tracking errors.
6
You are searching for a visual representation of ’indifference’.
Appendix
Table A2 . Sample of LLM-generated personality traits used when NA>0 . To generate these traits, we give gemini-3-pro-preview the Picbreeder VLM system prompt ( Fig. A17 ) and ask for personality traits that may implicitly affect an agent’s behavior on this task ( Fig. A18 ).
Figure A6 . Effect of exploration ( ϵ -greedy) on Semantic Recall within the Picbreeder archive. Forcing agents to take some random selections (i.e., with probability 0.25) can improve the quality of the archive, with Semantic Recall score approaching that of the historical human baseline. Large amounts of random parent selection ( ϵ≥0.5 ), and a fully random baseline (in which branching, publication, and archive rating decisions are also random) are detrimental.
Figure A8 . Effect of exploration ( ϵ -greedy) on the diversity of the Picbreeder archive. A moderate amount of noise can increase Visual and Semantic Coverage and Tree Balance, but, in excess, reduces the legibility of generated images (cf. 8(g) , Fig. A6 ).
Figure A9 . Effect of history—i.e. Context Length ( CL ), the number of previous actions included in an agent’s context—on Semantic Recall within the Picbreeder archive. Without any context, mode collapse is common, leading to reduced recall. CL=1 proves to be a surprisingly effective sweep spot, with larger CL leading to overly noisy/abstract forms.
Figure A11 . Effect of history—i.e. Context Length ( CL ), the number of previous actions included in an agent’s context—on diversity within the Picbreeder archive. Increasing CL increases diversity, but with rapidly diminishing returns. CL=20 is an exception; here, diversity peaks, likely because in this case the agent is prompted with an additional note encouraging its publication to be novel w.r.t. the still-visible archive sample ( 17(b) ). But the images in these archives are more noisy/abstract, as reflected in their low Semantic Recall scores ( 9(a) ). This may be due to the context window being overloaded, leading to decreased VLM performance in general; or due to the incentivization of novelty combined with forced publication steps effectively leading to the premature publication of semantically ill-defined works-in-progress.
Figure A12 . Effect of number of agents NA —i.e. the number of distinct personality traits (see Table A2 ) distributed among Picbreeder sessions—on Semantic Recall score and diversity metrics. Increasing NA improves various metrics of diversity without harming Semantic Recall.
Figure A13 . Effect of exploration ( ϵ -greedy) on the Fidelity of the Picbreeder archive. Greedy strategies can game this metric by refining a small set of images and flooding the archive with near-duplicates.
Figure A14 . Effect of history (number of previous actions included in an agent’s context/memory) on the Fidelity of the Picbreeder archive. Agents without history perform best by refining the current form, without any incentive to push past local optima.
Figure A15 . Effect of multiple agents (number of distinct personality traits assigned) on the fidelity of the Picbreeder archive. Adding agents reduces Fidelity by inhibiting mode collapse.
Figure A16 . Semantic attractors. A common failure case of VLMs when playing Picbreeder is their tendency to fall into apparent attractors (mode collapse) in CPPN-image space. We show snapshots of various archives, with images arranged according to visual embedding distance, selecting subregions that showcase such attractors.
Figure A17 . Prompt components used by the Picbreeder VLM agent.
Figure A18 . Prompt for generating personality traits used when NA>0 (see Table A2 for sample output from gemini-3-pro-preview ). The VLM Picbreeder system prompt ( 17(a) ) is injected where indicated.
Figure A19 . Sample of an archive with Number of Agents NA=1,000 , with images arranged according to visual embedding distance. We select a region of the archive that showcases the noisy, potentially adversarial images that seem to emerge at large NA . These might be owing to a large number of traits imbuing agents with an eye for relatively abstract properties of an image, where VLMs focused on roleplaying may be keen to project the satisfaction of such abstract inclinations onto otherwise meaningless forms. See Table A2 ; traits like 32 “You are drawn to the aesthetic of bad analog TV reception” even explicitly incentivize noise, despite our prohibiting such explicit search objectives ( Fig. A18 ).
Language models (LMs) often struggle to generate diverse, human-like creative content, raising concerns about the long-term homogenization of human thought through repeated exposure to similar outputs. Yet scalable methods for evaluating LM output diversity remain limited, especially beyond narrow tasks such as random number or name generation, or beyond repeated sampling from a single model. We introduce Infinity-Chat, a large-scale dataset of 26K diverse, real-world, open-ended user queries that admit a wide range of plausible answers with no single ground truth. We introduce the first comprehensive taxonomy for characterizing the full spectrum of open-ended prompts posed to LMs, comprising 6 top-level categories (e.g., brainstorm & ideation) that further breaks down to 17 subcategories. Using Infinity-Chat, we present a large-scale study of mode collapse in LMs, revealing a pronounced Artificial Hivemind effect in open-ended generation of LMs, characterized by (1) intra-model repetition, where a single model consistently generates similar responses, and more so (2) inter-model homogeneity, where different models produce strikingly similar outputs. Infinity-Chat also includes 31,250 human annotations, across absolute ratings and pairwise preferences, with 25 independent human annotations per example. This enables studying collective and individual-specific human preferences in response to open-ended queries. Our findings show that LMs, reward models, and LM judges are less well calibrated to human ratings on model generations that elicit differing idiosyncratic annotator preferences, despite maintaining comparable overall quality. Overall, INFINITY-CHAT presents the first large-scale resource for systematically studying real-world open-ended queries to LMs, revealing critical insights to guide future research for mitigating long-term AI safety risks posed by the Artificial Hivemind.
Liwei Jiang, Yuanjun Chai, Margaret Li +7
University of Washington · Allen Institute for Artificial Intelligence · Carnegie Mellon University +3
We study adversarial robustness of open-source vision-language model (VLM) agents deployed in a self-contained e-commerce environment built to simulate realistic pre-deployment conditions. We evaluate two agents, LLaVA-v1.5-7B and Qwen2.5-VL-7B, under three gradient-based attacks: the Basic Iterative Method (BIM), Projected Gradient Descent (PGD), and a CLIP-based spectral attack. Against LLaVA, all three attacks achieve substantial attack success rates (52.6%, 53.8%, and 66.9% respectively), demonstrating that simple gradient-based methods pose a practical threat to open-source VLM agents. Qwen2.5-VL proves significantly more robust across all attacks (6.5%, 7.7%, and 15.5%), suggesting meaningful architectural differences in adversarial resilience between open-source VLM families. These findings have direct implications for the security evaluation of VLM agents prior to commercial deployment.
We evaluate the performance of Vision Language Models in Visual Question Answering (VQA) when questions violate Grice's maxims. To do this, we use VLMs to generate question modifiers that add non-essential, ambiguous or false information and show that in the presence of such violations, the VLMs that we evaluate (ChatGPT, Claude, Gemini and Llava) show diminished performance. Further, we empirically show the difference between how humans reason pragmatically compared to VLMs, and the difference in VLM reasoning when it resolves violations that are human-induced compared to those that are AI-generated. Finally, we show that human cognitive effort (measured through time-on-task in an experiment) is lower for resolving VLM-induced violations, but VLMs themselves perform less accurately in such cases.
Monika Shah, Sudarshan Balaji, Somdeb Sarkhel +2
University of Memphis, TN, USA · Adobe Research, San Jose, CA, USA · University of Maryland Baltimore County, USA