CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments

Kianté Fernandez; Caitlin Chen; Jialu Zhou; Bartek Sadowski; Anthony C. Miceli; Ian Krajbich

cs.AIOct 27, 2026

CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments

Authors: Kianté Fernandez, Caitlin Chen, Jialu Zhou, Bartek Sadowski, Anthony C. Miceli, Ian Krajbich

Organizations: Department of Psychology, University of California, Los Angeles

Abstract

Claims about the cognitive capabilities of large language models are typically tested by translating behavioral paradigms into text prompts, bypassing the visual interfaces and time pressure through which human participants actually engage with experiments. At the same time, AI agents can now complete online behavioral tasks and produce human-like data, which raises concerns that online datasets in the social and behavioral sciences are increasingly open to contamination. We introduce CogArena, a benchmark of ten experimental paradigms from the social and behavioral sciences. Agents complete the experiments through a standard web browser, working with the same visual stimuli and response interfaces that online participants use. A three-level scoring procedure evaluates whether the agent completed the task, how accurately it performed, and whether its data show the behavioral signatures each paradigm reliably produces in people. Across four multimodal models, each run ten times per task, we find that finishing an experiment and reproducing its signatures are different things. Many sessions did not finish, and those that did reproduced some signatures reliably and others not at all. CogArena provides the tasks, scoring code, and a public leaderboard for new submissions.

Figures & tables

Benchmark	Content	Interface	Grading	Human reference
WebShop	E-commerce	Browser	End-state success	–
WebArena	Web productivity	Browser	End-state (binary)	–
Mind2Web	137 websites	Browser	Step + end-state	–
TurkingBench	NLP annotation	Browser	Per-trial accuracy	Crowdworkers
CogBench	7 cog. experiments	Text API	Cognitive metrics	Qualitative
Psych-101/201	160+ experiments	Text API	Choice prediction	Trial-by-trial

Table 1: Comparison of CogArena with related benchmarks. CogArena combines a browser-based interface with behavioral experiments scored against literature-derived signatures.

#	Category	Task	Source	Scored Signatures
1	Learning	Grid bandit (safe vs. risky blocks)	( Wu et al., 2018 ; Witte et al., 2024 )	Kraken avoidance, value-directed exploitation, within-block learning
2	Risk	Marbles (developing risk)	( Ciranka and van den Bos, 2025 )	EV sensitivity, EV-optimal choice, magnitude effect
3	Social / Economic	Repeated PD + Battle of Sexes	( Akata et al., 2025 ; Cooper et al., 1989 )	Tit-for-tat reciprocity, conditional cooperation, coordination
4	Memory recall	Serial recall	( Haridi et al., 2026 )	Similarity advantage, primacy
5	Memory recognition	Old/new visual recognition	( Brady et al., 2008 )	Hit rate above false-alarm rate, low false-alarm rate
6	Perception	Random dot motion	( Roitman and Shadlen, 2002 )	Coherence-dependent accuracy and RT

Table 2: The CogArena v1 task set. The last column lists examples of the scored signatures; Appendix B gives the complete list. Tasks 1–4, 7, and 8 each also carry one response-time relational signature (Appendix B ). See Appendix A for full per-task scoring specifications and source-version provenance.

Figure 1: CogArena conceptual architecture. An agent (using any scaffold) interacts with CogArena tasks (behavioral experiments served by the platform), and the resulting trial data are scored at three levels, L1 task completion, L2 performance accuracy, and L3 alignment with task-specific signatures. Trial data are sent to the server during the session, so partial sessions still produce usable data.

Figure 2: Per-model score distributions across the v1 ten-task set, by level and composite. Each panel shows a half-violin, a boxplot, and one jittered point per scored session. The L3 and Composite panels include every scored session, so a session in which no signature could be tested enters both with a Level 3 of zero. The dashed horizontal line in those panels marks the random-agent chance-floor mean. The

n

under each column is the number of sessions contributing data, and the right-hand chance-floor column is computed across all one hundred random-agent sessions.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Signature	Test	Type	$w_{j}$
random_dot_motion	Coherence-accuracy psychometric	corr	C	2.0
	Coherence-RT chronometric	corr	C	1.5
	Above-chance accuracy at low coherence	prop	C	0.5
	Above-chance overall accuracy	prop	C	1.0
grid_bandit	Kraken avoidance in risky blocks	prop	C	1.5
	Value-directed exploitation	corr	C	1.5

Appendix

Table 3: Behavioral signatures evaluated in v1, per task. Test types are prop (exact one-sample proportion test), paired (paired proportion contrast), contrast (conditional proportion contrast), and corr (Pearson correlation, or Spearman for the response-time signatures). Appendix C defines each test. Type is C for capacity (reproducing the human pattern coincides with performing well) or B for bias (reproducing it is not normatively required).

Task	Baseline source	Tier	Port differences relevant to the baseline
random_dot_motion	Desender et al. 2021 (Confidence Database) and Roitman and Shadlen 2002 (paradigm)	R	Accuracy baselines computed from trial-level human data at coherence levels matched to ours. Our Canvas implementation reproduces the Roitman and Shadlen algorithm but not their apparatus or viewing distance.
grid_bandit	Witte et al. 2024	R	Computed from the authors’ participant data (26,565 clicks, $n=220$ ), replacing the GP-bandit literature expectations used at submission.
marbles_risk	Ciranka and van den Bos 2025	–	Higher-EV choice rate computed from the authors’ trial-level data (23,887 trials, $n=166$ ), restricted to the description condition our version implements. A reference value only, since the task has no Level 2 metrics.
repeated_games	Akata et al. 2025	R	Cooperation and coordination rates computed from the source’s human rounds (3,900 rounds, $n=195$ ) against the same simple opponents our port uses.
serial_recall	Haridi et al. 2026	R	Computed from the authors’ trial-level data for the 16-pair lists our version uses ( $n=29$ of the archive’s $116$ participants). The source used $\sim$ 28 pairs per similarity level and a typed, spacebar-gated response, whereas our version uses 16 pairs across four similarity levels with a single typed recall step.
visual_recognition	Brady et al. 2008	P	No public archive exists. The source’s memory test is two-alternative forced choice (92.5%, 87.6%, and 87.2% across novel, exemplar, and state conditions). Old/new hit and false-alarm baselines are derived from those statistics through $d^{\prime}$ , assuming an unbiased criterion, so that hit rate $=\Phi(d^{\prime}/2)$ and false-alarm rate $=1-\Phi(d^{\prime}/2)$ ; the derivation is recorded in scoring/human_baselines/visual_recognition.json . Our version uses 768 procedurally generated SVG stimuli (shape $\times$ color $\times$ pattern $\times$ orientation), which are plausibly easier to discriminate, so this baseline is likely too lenient.

Appendix

Table 4: Provenance of the v1 human baselines. In the Tier column, R means computed from the original authors’ released participant data, D means set from the design of our version of the task, P means derived from statistics published in the source paper because no public archive is available, and a dash marks a task with no Level 2 metrics.

Figure 3: Screenshot-removal check. Same models and scaffold, with screenshots removed so the model received only the page text. Each boxplot shows the per-task composite under one condition, with one point per scored session. The

n

is given as with/without screenshots. Tasks are grouped by whether their stimuli are drawn as images, which the page text does not describe, or are themselves text. Brackets give two-sided Mann-Whitney tests between conditions (

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

, n.s. not significant). All four tasks degrade, three of them significantly. The two text tasks start two to three times higher than the two image tasks, so the raw drops shown here are not comparable across the two groups, but the proportional changes quoted in Section 3.1 are.

Task	Trials	Per-signature $n$
random_dot_motion	100	100 overall, 20 per coherence level (5 levels, 5 blocks); the configuration file lists 120, but the implementation balances 24 trials per block over ten coherence–direction cells and so presents 20, and Level 1 was scored against the configured 120
grid_bandit	110	11 blocks $\times$ 10 clicks, with 50 risky-block clicks for Kraken avoidance
marbles_risk	64	64 overall, 4 repetitions of the gamble grid
repeated_games	30	15 PD rounds and 15 BoS rounds, and conditional signatures draw on the subset of PD rounds following a given opponent move
serial_recall	16	16 recall trials across 4 similarity levels
visual_recognition	100	100 test trials (50 old, 50 lures) after 50 study trials

Appendix

Table 5: Trial budgets available to Level 3 signatures in v1. “Per-signature

n

” is the number of trials the corresponding tests draw on after filtering.

Task	$n$	L1	L2	L3	Composite
random_dot_motion	10	1.000	0.024	0.335	32.60
grid_bandit	10	1.000	0.045	0.179	25.54
marbles_risk	10	1.000	–	0.191	37.76
repeated_games	10	1.000	0.428	0.217	40.86
serial_recall	10	1.000	0.183	0.050	23.92
visual_recognition	10	1.000	0.001	0.331	31.59

Appendix

Table 6: Random-agent chance floor for each v1 task, mean over ten sessions per task. All L1 values are 1.0 because the driver always reaches end-of-experiment. A dash marks a task with no Level 2 metrics, whose composite redistributes the Level 2 weight to Levels 1 and 3. That raises those two floors, because the random agent always completes.

Task	Floor	gemini-3-flash	kimi-k2.5	gpt-5.4-nano	qwen3-vl-235b
moral_machine	37.8	94.9 $\pm$ 5.4	81.4 $\pm$ 13.9	53.6 $\pm$ 10.3	–
tiny_alchemy	25.4	54.8 $\pm$ 15.3	78.3 $\pm$ 13.7	72.9 $\pm$ 8.2	73.0 $\pm$ 12.5
marbles_risk	37.8	86.7 $\pm$ 4.0	63.2 $\pm$ 24.8	53.4 $\pm$ 19.0	–
phishing_detection	25.0	74.6 $\pm$ 6.4	71.9 $\pm$ 5.7	66.8 $\pm$ 4.2	54.9 $\pm$ 17.6
grid_bandit	25.5	64.3 $\pm$ 8.6	51.5 $\pm$ 5.5	51.6 $\pm$ 12.8	43.5 $\pm$ 16.5
repeated_games	40.9	60.0 $\pm$ 13.6	56.1 $\pm$ 11.5	41.4 $\pm$ 6.8	47.7 $\pm$ 5.4

Appendix

Table 7: Per-task composite over measurable sessions, mean

\pm

SD, against the random-agent floor. A dash marks a cell with no measurable session. Rows follow the task order of Figure 4 . The final rows give each model’s mean over its measurable sessions and the number of such sessions, which differs sharply across models and is the reason per-task means alone do not rank them.

Figure 4: Per-task composite. Colored points are panel models, mean

\pm

SD over each cell’s measurable sessions (at most ten). Each occupies a fixed vertical slot within its task band, so identity is carried by position as well as color. The open diamond is the state-of-the-art model gpt-5.6-sol , drawn without an interval because it is a single session and shown as a reference rather than a fifth panel member (Appendix K ). The vertical rule is the re-scored per-task chance floor. Tasks are ordered by panel mean. Overlapping intervals on most tasks are the reason per-cell values do not support model comparisons at this sample size.

Task	Signature	gemini	kimi	nano	qwen	Panel [95% CI]	Floor
random_dot_motion	coherence_accuracy_psychometric	4/10	1/9	0/3	0/2	0.21 [0.09, 0.40]	0/10
	coherence_rt_chronometric	0/10	0/7	0/3	0/1	0.00 [0.00, 0.15]	1/10
	above_chance_low_coherence	5/10	0/8	0/3	0/1	0.23 [0.10, 0.43]	1/10
	above_chance_overall	9/10	0/10	0/7	0/7	0.26 [0.15, 0.43]	2/10
grid_bandit	kraken_avoidance_in_risky	9/10	4/5	4/10	2/8	0.58 [0.41, 0.73]	0/10
	value_directed_exploitation	7/10	1/4	4/10	7/10	0.56 [0.39, 0.71]	1/10

Appendix

Table 8: Per-signature pass counts,

k/n

= sessions graded

1.0

out of sessions in which the signature was testable (at most ten per model). “Panel” pools the four models, with a 95% Wilson interval.

Task	Panel L3	sol L3	Panel comp.	sol comp.
visual_recognition	0.235	1.000	35.1	99.1
effort_foraging	0.166	0.636	26.9	77.7
moral_machine	0.786	1.000	83.3	100.0
marbles_risk	0.666	0.818	71.3	86.0
random_dot_motion	0.390	0.500	34.1	71.8
phishing_detection	0.672	0.773	67.1	77.5

Appendix

Table 9: State-of-the-art model against the panel, per task. Panel columns are means over the four models’ measurable sessions on each task, and gpt-5.6-sol is a single session per task. Rows are ordered by Level 3 gain over the panel mean. The final row is the unweighted mean of the ten per-task values, and so differs slightly from the session-weighted panel means quoted in Section 5.1 .

Model	Headline	Signature-emph.	Balanced	Signature-only	Coverage
	$(0.15,0.35,0.50)$	$(0.10,0.30,0.60)$	$(0.20,0.40,0.40)$	$(0,0,1)$
gemini-3-flash-preview	58.61	56.80	60.46	54.37	96
kimi-k2.5	44.23	42.29	46.21	38.44	76
gpt-5.4-nano	36.07	33.87	38.38	30.36	63
qwen3-vl-235b-instruct	26.41	24.68	28.24	22.78	41

Appendix

Table 10: Per-model mean composite under four weight schemes, averaged over all sessions. Headline weights are

(0.15,0.35,0.50)

. Coverage is the number of measurable sessions out of each model’s one hundred.

Grading rule	Floor L3	Panel L3	L3 gap	Comp. gap	Ranking
Current $\{0,0.5,1\}$	0.184	0.490	0.306	20.71	unchanged
Six bins on $p$	0.110	0.410	0.300	20.62	unchanged
Continuous in $p$	0.157	0.458	0.302	20.72	unchanged
Continuous in effect size	0.072	0.357	0.285	19.99	unchanged

Appendix

Table 11: Level 3 under four grading rules. Panel columns are means over the

276

measurable sessions, and floor columns are means over all

100

random-agent sessions. Ranking is by measurable-session composite, and is gemini

>

kimi

>

qwen

>

nano under every rule.

Multimodal Agent Evaluation

See all 27 papers

CogArena: A Multimethod Evaluation of Cognitive Ability Structure in Large Language Models
Jul 27, 2026Dengzhe Hou, Lingyu Jiang, Fangzhou Lin +1LLM Evaluation Cognitive Modeling
1Graduate School of Information Sciences, Tohoku University · 2Unprecedented-scale Data Analytics Center, Tohoku University · 3Texas A&M University +1
CoCoBench: A Cooperative Coordination Benchmark for Embodied Multi-Agent Task Planning
Aug 28, 2026Yang Chen, Ye-Xin Xie, Lirong Che +8Multi-Agent Coordination AI Agent Benchmarks
Nanjing University · AgiBot · Tsinghua University
WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction
May 28, 2026Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu +14Multimodal Agent Evaluation Multimodal Memory
University of California, Santa Barbara · Johns Hopkins University · Carnegie Mellon University +3

Jul 27, 2026J/K move · Enter open · S save

CogArena: A Multimethod Evaluation of Cognitive Ability Structure in Large Language Models

Dengzhe Hou, Lingyu Jiang, Fangzhou Lin, Kazunori D Yamada

1Graduate School of Information Sciences, Tohoku University · 2Unprecedented-scale Data Analytics Center, Tohoku University · 3Texas A&M University+1

LLM Evaluation Cognitive Modeling

Open paper AskPDF arXiv

LLM cognitive scores are increasingly summarized as per-ability profiles whose dimensions should converge across tasks, respond selectively to matched interventions, and generalize beyond the models used to define them. We introduce CogArena, a procedurally generated 13-paradigm benchmark built around a multimethod framework for determining when cognitive-task scores warrant dimensional labels across five theory-motivated groupings. Across 55 open-weight models, nearly all paradigm correlations are positive and a common axis explains about half the variance. The within-grouping advantage is small, scoring-sensitive, and uncertain across model families. In a separately frozen, fully crossed study across 12 models from six families, targeted scaffolds show a small matched-grouping advantage, but no scaffold-specific contrast survives multiplicity correction and selectivity does not improve held-out-family prediction. The frozen confirmation criterion fails. A post-hoc alternate-wording replication produces a smaller positive estimate and again fails. Together, these results support a boundary conclusion. Theory-aligned prompting produces a small in-battery diagonal tendency, but the present evidence does not establish stable five-dimensional profiles. CogArena provides a workflow joining behavioral signatures, covariance, matched interventions, and out-of-family prediction before cognitive labels are attached to model scores.

Abstract

Multimodal Agent Evaluation

Explore similar work

CogArena: A Multimethod Evaluation of Cognitive Ability Structure in Large Language Models

CoCoBench: A Cooperative Coordination Benchmark for Embodied Multi-Agent Task Planning

WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction