A systematic framework for generating novel experimental hypotheses from language models
Authors: Kanishka Misra, Najoung Kim
Organizations: Department of Linguistics, The University of Texas at Austin, 305 E. 23rd Street #4.428, Austin, TX 78712, USA · Department of Linguistics, Boston University, 111 Cummington Mall #138P, Boston, MA 02215, USA
Neural language models (LMs) have been shown to capture complex linguistic patterns, yet their utility in understanding human language and more broadly, human cognition, remains debated. While existing work in this area often evaluates human-machine alignment, few studies attempt to translate findings from this enterprise into novel insights about humans. To this end, we propose a systematic framework for hypothesis generation that uses LMs to simulate outcomes of experiments that do not yet exist in the literature. We instantiate this framework in the context of a specific research question in child language development: dative verb acquisition and cross-structural generalization. Through this instantiation, we derive two novel, untested hypotheses: (1) the alignment between argument ordering and discourse prominence features of exposure contexts modulates how children generalize new verbs to unobserved structures; and (2) that this alignment in the first postverbal argument of the exposure context has a more noteworthy effect. Additionally, we also design a set of experiments that can test these hypotheses in the lab with children. This work contributes both a domain-general framework for systematic hypothesis generation via simulated learners and domain-specific, lab-testable hypotheses for child language acquisition research.
Figures & tables
Figure 1: Overview of our broader pipeline for hypothesis generation from language models.
Figure 2: Depiction of cross-dative generalization, when the learner is exposed to a novel verb in one dative construction (exposure) and evaluated on a stimulus where the verb is used in the alternate dative construction (generalization).
Figure 3: Overview of our experimental setup. a. Our simulation pipeline. We accumulate the LM learners’ cross-dative generalization behavior, given exposure to a range of stimuli sampled from a space of theme and recipient feature configurations. We then relate this behavior back to the feature configurations themselves to articulate novel hypotheses about the properties of the training exposure that facilitate cross-dative generalization. b. Depiction of a single learning trial, where a model’s representation of a novel verb, pilked , is updated given exposure to its use in a single dative sentence, following which the model is then frozen and evaluated on its behavior on a set of stimuli where pilked is used in the unmodeled dative construction.
Figure 4: Model selection results from evaluating the preconditions of grammar learning and sensitivity to known dative alternation patterns. a. Average accuracy (%) of models with varying hyperparameters on Zorro ( Huebner et al., 2021 ) , a minimal pair dataset targeting 24 different morphosyntactic phenomena used to evaluate grammar knowledge of LMs trained on child-directed speech. Chance performance is 50%. b. Results on comparing alternation preferences of the model on verbs that do not alternate in models’ training data but are known to alternate in reality ( naba ) vs. verbs that do not alternate in models’ training data and also do not alternate in reality ( nana ). The plot shows the average difference in the alternation preference for naba verbs (N=12) and nana verbs (N=14) in AO-CHILDES ( Huebner and Willits, 2021 ) for both constructions (DO and PO), for models with varying hyperparameters. Larger difference indicates higher sensitivity to the alternation pattern difference. c. Alternation preference (z-scored) of 5 different runs of the final model architecture, which we use to perform our subsequent experiments on naba and nana verbs. Across both dative constructions, models preferred sentences in the alternate construction for naba verbs relative to those for nana verbs (DO: β = 0.079, t (2794) = 4.467, p < .001; PO: β = 0.291, t (6294) = 24.150, p <.001). This suggests that the final models show non-trivial sensitivity towards the dative alternation preferences of real, known verbs that occur in the training data.
Figure 5: Results from our replication of known cross-dative generalization results ( Conwell and Demuth, 2007 ; Rowland and Noble, 2010 ; Arunachalam, 2017 ) . a. Asymmetric cross-dative generalization ( Conwell and Demuth, 2007 ) : Average log probabilities per token of LM learners on the generalization set was greater in the DO to PO experiment than in the PO to DO experiment ( β = 0.31; t (4.15) = 6.11; p < .01); b. Advantage of cross-structure training ( Arunachalam, 2017 ) : Average log probabilities per token of LM learners on the DO generalization set was greater when the exposure to the novel verb was a DO dative than when it was a PO dative ( β = 0.77; t (3.99) = 7.39; p < .01); c. Non-effect of theme animacy in learning from DO exposures ( Rowland and Noble, 2010 ; Arunachalam, 2017 ) : There was no significant effect of theme animacy in LM learners’ average log probabilities per token on the DO generalization set, given exposure to the novel verb in a DO dative ( β = 0.09; t (2.81) = 0.59; p = 0.59).
Figure 6: Results and analyses from our simulations. a. Comparison of coding schemes in terms of the overall model fits obtained in the DO → PO generalization simulation. Scores derived using the HAAP coding scheme result in the best model fit. Example exposure stimuli and scores (recipient + theme + Δ length) provided for the HAAP scheme vs. a randomly selected counterfactual scheme. b. Estimated association of recipient, theme, and length-different scores under the HAAP coding scheme, at predicting LM learners’ generalization of a novel verb’s usage from a DO exposure to the PO construction . c. Comparison of coding schemes in terms of the overall model fits obtained in the PO → DO generalization simulation. Scores derived using the HAAP coding scheme result in the best model fit. Example exposure stimuli and scores (recipient + theme + Δ length) provided for the HAAP scheme vs. a randomly selected counterfactual scheme. d. Estimated association of recipient, theme, and length-different scores under the HAAP coding scheme, at predicting LM learners’ generalization of a novel verb’s usage from a PO exposure to the DO construction . For both b and d , scores of the first postverbal argument of the exposure construction (recipient for DO and theme for PO) show stronger association with the LMs learners’ generalization behavior. Model fits are estimated using the difference in log-likelihood of the fitted model relative to a null model (only random effects). Statistical significance calculated using a likelihood ratio test. ∗∗∗ : p < .001.
Experiment
Generalization
First postverbal arg. of exposure
Theme
Recipient
1a
DO → PO
Recipient
haap max
haap max
1b
DO → PO
Recipient
haap min
haap min
1c
PO → DO
Theme
haap max
haap max
1d
PO → DO
Theme
haap min
haap min
2a
DO → PO
Recipient
haap min
haap max
2b
DO → PO
Recipient
haap max
haap min
Table 1: Feature configuration setup for our proposed experiments. haap max indicates the features are maximally compliant with harmonic alignment and animacy prototypicality effects as quantified in our study, whereas haap min indicates minimal compliance.
Construction
Argument
haap max
haap min
DO
Theme
a ball with stars, a cup with no lid, a cookie with chocolate frosting, a glass with milk, …
it, them (inanimate, disambiguated from the previous sentence)
Recipient
me, him, her, them, us
a girl with green hair, a boy with round glasses, a girl with blue hat, a horse with the saddle, …
PO
Theme
it, them (inanimate, disambiguated from the previous sentence)
a ball with stars, a cup with no lid, a cookie with chocolate frosting, a glass with milk, …
Recipient
a girl with long hair, a boy with blue pants, a bear with an apple, a boy with round glasses, …
me, him, her, them, us
Table 2: Stimuli design for theme and recipient arguments in the proposed experiments, spanning from those that are maximally and minimally compliant with haap .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
(Hyper)parameter
Value
Architecture
OPT ( Zhang et al., 2022 )
Embed size
256
FFN dimension
1,024
Num. layers
8
Attention heads
8
Vocab size
8,192
Appendix
Table 3: LM Training details
Given Condition
Template 1
Template 2
Template 3
Agent
Do you see {agent} ?
Look, it’s {agent} !
Here’s {agent} !
Agent + One Argument
Do you see {agent} and {given-arg} ?
Look, it’s {agent} with {given-arg} !
Here’s {agent} with {given-arg} !
Agent + Both Arguments
Do you see {agent} and {given-arg1} and {given-arg2} ?
Look, it’s {agent} and {given-arg1} and {given-arg2} !
Here’s {agent} with {given-arg1} and {given-arg2} !
Appendix
Table 4: Givenness templates across different conditions. {agent} and {given-arg} represent the agent and the argument that is given, respectively.
Feature
Value
Examples
Animacy
Animate
mommy, daddy, dog, boy, girl, horse, bear, horse
Inanimate
cup, ball, book, food, cookies, lego, chair, box
Pronominality
Pronominal
him, her, it, them, us, me, someone, something
Nominal
mommy, daddy, dog, boy, girl, book, cup, ball
Definiteness
Definite
the ball, the book with pictures, the boy with blue pants
Indefinite
a ball, a book, a boy, a lego, a chair, a box with blocks
Appendix
Table 5: Example items across Animacy, Pronominality, and Definiteness features.
Verb Expecting
Non-verb Expecting
jack the treasure from the sleeping giant.
oh , you need the other now .
he a lot of things right now.
yeah, santa claus brought that to you .
that like a fish.
i’m forgetting .
louise that.
let’s make a .
look, somebody something.
there, now nina has a .
Appendix
Table 6: Example verb expecting and non-verb expecting sentences from our verbhood validation set. In practice, is replaced with the surface form of the novel dative verb (here, [pilked] ).
Figure 7: Average Verbhood Δ s and Accuracies across different adaption dative types. Note that there is no upper/lower-bound for Verbhood Δ , since they are differences in log-probabilities, and can theoretically be infinite in either direction. Δ s above 0 indicate the model prefers the novel verb in verb-expecting contexts over non-verb-expecting contexts. Each point represents the result of a single LM seed.
Alternation Behavior
Dative
Lemma (Freq. in Training)
Example Test sentences
DO
PO
Non-alternating in training but actually alternating ( naba )
She addressed me that You carried mommy some boxes He described my uncle the day I explained her everything
She addressed that to me You carried some boxes to mommy He described the day to my uncle I explained everything to her
Appendix
Table 7: naba and nana verb lemmas from AO-CHILDES along with test sentences in the two dative constructions. Values in parentheses following the verb lemmas denote frequency in the observed dative in the training data. The acceptability of the example test sentences should reflect the known alternation behavior— naba alternates should be more acceptable, whereas nana alternates should be less acceptable.
We introduce quantitative metrics for child language acquisition to evaluate language models. Our focus is on the formal syntactic and functional discourse properties of determiners in English, which young children acquire early and accurately. We propose Contextual Alternative Choice (CAC), a new prompting method which provides targeted tests for both syntactic and discourse knowledge of language. The method enables direct comparison of language models against children, and more importantly, against statistical benchmarks independently established in empirical research. No current model trained on a comparable amount of data simultaneously meet both formal and functional benchmarks like human children, but some very large models do. We present our results as methodological and technical contributions, with specific emphasis on cognitive status of language models.
Héctor Javier Vázquez Martínez, Charles Yang
University of Pennsylvania, Department of Linguistics and Computer and Information Science
Recent studies suggest that child-directed speech is not conducive to language learning in BabyLMs. However, current evaluations focus predominantly on comprehension and not production, which is central to usage-based theories of language acquisition which argue how CDS facilitates early language use through constructional ''frames'' (frequent lexical patterns with open slots). We introduce a novel generation-based evaluation inspired by such theories in form of a frame-completion task, and compare Llama models trained with CDS, the BabyLM corpus, and web-crawl data (FineWeb-edu) on comprehension benchmarks and our novel framework. Our results reveal a clear dissociation between models' comprehension and production capabilities: while FineWeb-trained models excel at minimal pairs, CDS-trained models produce grammatical completions substantially earlier in training and concentrate probability mass on appropriate slot-fillers. These findings show that comprehension benchmarks underestimate what CDS affords to BabyLMs.
Bastian Bunzeck, Sina Zarrieß
Computational Linguistics, Department of Linguistics Bielefeld University, Germany
For humans, filler-gap dependencies require a shared representation across different syntactic constructions. Although causal analyses suggest this may also be true for LLMs (Boguraev et al., 2025), it is still unclear if such a representation also exists for language models trained on developmentally feasible quantities of data. We applied Distributed Alignment Search (DAS, Geiger et al. (2024)) to LMs trained on varying amounts of data from the BabyLM challenge (Warstadt et al., 2023), to evaluate whether representations of filler-gap dependencies transfer between wh-questions and topicalization, which greatly vary in terms of their input frequency. Our results suggest shared, yet item-sensitive mechanisms may develop with limited training data. More importantly, LMs still require far more data than humans to learn comparable generalizations, highlighting the need for language-specific biases in models of language acquisition.