Online library learning in human visual puzzle solving

Pinzhe Zhao; Emanuele Sansone; Marta Kryven; Bonan Zhao

cs.AIMar 24, 2026

Online library learning in human visual puzzle solving

Authors: Pinzhe Zhao, Emanuele Sansone, Marta Kryven, Bonan Zhao

Organizations: University of Edinburgh · MIT CSAIL · KU Leuven · Dalhousie University

Abstract

When learning a novel complex task, people often form efficient reusable abstractions that simplify future work, despite uncertainty about the future. We study this process in a visual puzzle task where participants define and reuse helpers -- intermediate constructions that capture repeating structure. In an online experiment, participants solved puzzles of increasing difficulty. Early on, they created many helpers, favouring completeness over efficiency. With experience, helper use became more selective and efficient, reflecting sensitivity to reuse and cost. Access to helpers enabled participants to solve puzzles that were otherwise difficult or impossible. Computational modelling shows that human decision times and number of operations used to complete a puzzle increase with search space estimated by a program induction model with library learning. In contrast, raw program length predicts failure but not effort. Together, these results point to online library learning as a core mechanism in human problem solving, allowing people to flexibly build, refine, and reuse abstractions as task demands grow.

Figures & tables

Figure 1: Task interface. A. Starting view of a trial. Left: Total points so far and the target shape for this trial. Middle: Work space, matching the target shape with the provided Operations and Primitive shapes. Right: List of steps, each step corresponds to one line of program. Black borders are only added in the paper for illustration. B. Example preview. C. Example programs. Each line has a line number, a thumbnail of the pattern that line creates, and the corresponding program (operations over primitives, steps, or helpers). D. The initial helper space. E. Helper space with example helpers.

Figure 2: Target patterns used in the Experiment.

Figure 3: Accuracy across 14 target patterns. Bars and standard errors for human participants. Colored dots for models.

Figure 4: Helper usage across 14 target patterns. A. Number of helpers created at each trial. B. Proportion of solution steps using saved helpers (blue) overlaid with ground-truth program length (red).

Figure 5: Helpers created by participants. Each row shows the patterns most frequently saved as helpers for that trial, ranked by popularity. Darker cell color indicates higher popularity. Only patterns saved by more than one participant are shown. Orange borders are for helpers saved by Library models.

Figure 6: Participant performance and model-estimated metrics. A. Mean completion time versus $\log_{10}$ (nodes expanded). Each point for a pattern (P1–P14). Dashed line shows linear fit. B. Mean number of steps versus $\log_{10}$ (nodes expanded). C. Mean completion time (blue, left axis) and mean steps (red, right axis) for each pattern. D. Number of helpers created during task phase predicts helper creation during free play (yes/no). Each point represents one participant, with jitter for visibility.

Figure 7: Examples of patterns participants created during free play. Pattern names were provided by participants.

Cognitive Modeling

Cognitive modeling uses computational theories to explain and predict how people perceive, learn, remember, reason, and decide.

See all 119 papers

Prospective Compression in Human Abstraction Learning
May 11, 2026Leonardo Hernandez Cano, Ivan Zareski, Luisa El Amouri +6Program Synthesis Visual Program Synthesis
1Massachusetts Institute of Technology · 2Dalhousie University · University of Edinburgh +1
Path-dependent program induction under resource constraints explains human sequence learning
May 26, 2026Hanqi Zhou, David G. Nagy, Peter Dayan +1Computational Cognitive Modeling Rate-Distortion Theory
University of Tubingen, Tubingen, Germany · Center for Cognitive Science, Technical University Darmstadt, Darmstadt, Germany · 3Hessian.AI, Darmstadt, Germany +1
Exploring Human Behavior During Abstract Rule Inference and Problem Solving with the Cognitive Abstraction and Reasoning Corpus
Feb 25, 2026Caroline Ahn, Quan Do, Leah Bakst +4Inductive Inference Abstract Reasoning
Graduate Program for Neuroscience, Boston University, Boston, MA, USA · Cognitive Neuroimaging Center, Boston University, Boston, MA, USA · Center for Systems Neuroscience, Boston University, Boston, MA, USA +1

May 11, 2026J/K move · Enter open · S save

Prospective Compression in Human Abstraction Learning

Leonardo Hernandez Cano, Ivan Zareski, Luisa El Amouri, Pinzhe Zhao+5

1Massachusetts Institute of Technology · 2Dalhousie University · University of Edinburgh+1

Program Synthesis Visual Program Synthesis

Open paper AskPDF arXiv

A core challenge in program synthesis is online library learning: the incremental acquisition of reusable abstractions under uncertainty about future task demands. Existing algorithms treat library learning as retrospective compression over a static task distribution, where the learned library is determined by the corpus of past tasks. However, real-world learning domains are often non-stationary, with tasks arising from a generative process that evolves over time. We propose and test the hypothesis that in non-stationary domains human library learning selects abstractions prospectively: targeting compression of future tasks. We study this question using the Pattern Builder Task, a visual program synthesis paradigm in which participants construct increasingly complex geometric patterns from a small set of primitives, transformations, and custom helpers that carry forward across trials. Using this task, we conduct two experiments with complementary latent curricula, designed to dissociate between behaviors consistent with prospective compression, and alternative library learning accounts. Using six computational models spanning online library learning strategies, we show that human abstraction behavior reflects sensitivity to latent, non-stationary structure in the task-generating process. This behavior is consistent with prospective compression, and cannot be captured by existing retrospective compression-based algorithms, or inductive biases modeled by LLM-based program synthesis.

Abstract

Cognitive Modeling

Explore similar work

Prospective Compression in Human Abstraction Learning

Path-dependent program induction under resource constraints explains human sequence learning

Exploring Human Behavior During Abstract Rule Inference and Problem Solving with the Cognitive Abstraction and Reasoning Corpus