ManiBench: A Benchmark for Testing Visual-Logic Drift and Syntactic Hallucinations in Manim Code Generation

Nabin Oli

cs.AIFeb 24, 2026

ManiBench: A Benchmark for Testing Visual-Logic Drift and Syntactic Hallucinations in Manim Code Generation

Authors: Nabin Oli

Organizations: Sunway College Kathmandu, Birmingham City University

Abstract

Traditional benchmarks like HumanEval and MBPP test logic and syntax effectively, but fail when code must produce dynamic, pedagogical visuals. We introduce ManiBench, a specialized benchmark evaluating LLM performance in generating Manim CE code, where temporal fidelity and version-aware API correctness are critical. ManiBench targets two key failure modes: Syntactic Hallucinations (valid Python referencing non-existent or deprecated Manim APIs) and Visual-Logic Drift (generated visuals diverging from intended mathematical logic through timing errors or missing causal relationships). The benchmark comprises 150-200 problems across five difficulty levels spanning calculus, linear algebra, probability, topology, and AI, grounded in analysis of 3Blue1Brown's ManimGL source (53,000 lines, 143 scene classes). Evaluation uses a four-tier framework measuring Executability, Version-Conflict Error Rate, Alignment Score, and Coverage Score. An open-source framework automates evaluation across multiple models and prompting strategies. Code, data and benchmark suite are available at https://github.com/nabin2004/ManiBench. and the dataset is hosted on https://huggingface.co/datasets/nabin2004/ManiBench.

Figures & tables

Benchmark	Metric	Focus / Area
HumanEval / MBPP [ 10 ]	Pass@k	Python program syn- thesis, algorithmic tasks
SWE-Bench [ 9 ]	Patch rate	Software engineering / repo-level edits
DSCodeBench
[ 6 ]	API cov.	Data-science APIs and pipelines
Code2Video [ 4 ]	Aesthetic	Educational video/ storyboarding
SVGenius [ 3 ]	Visual	SVG editing / static visual generation

Table 1: Related benchmarks and limitations addressed by ManiBench.

ID	Problem	Diff.	Domain
MB-001	Colliding Blocks Compute $\pi$	4	Physics
MB-002	Gradient Descent	3	ML, Calculus
MB-003	Convolution	3	Signal Proc.
MB-004	Eigenvectors	4	Lin. Algebra
MB-005	Determinant	2	Lin. Algebra
MB-006	Central Limit Theorem	3	Probability

Table 2: Summary of the 12 pilot benchmark problems.

Problem	Lines	Scenes	Vis.T.	GL $\to$ CE
Colliding Blocks	2,193	16	10	15
Gradient Descent	8,598	16	16	13
Convolution	3,309	13	11	14
Eigenvectors	5,120	13	9	10
Determinant	1,132	11	7	10
CLT	7,036	12	9	11

Table 3: Reference code analysis summary for the 12 pilot problems.

Model	Provider	API	Eval.
Claude-Sonnet-4	Anthropic	OR	Z
Kimi-K2.5	Moonshot	OR	Z
Gemini-2.5-Pro	Google	OR	Z
Qwen3.5-Plus	Alibaba	OR	Z
Qwen3-235B-A22B	Alibaba	OR	Z
DeepSeek-R1-0528	DeepSeek	OR	Z

Table 4: Model roster. OR = OpenRouter; Inf.net = Inference.net. Z = zero-shot only; A = all five strategies.

Model	API	Ex. $\uparrow$	VC $\downarrow$	Al. $\uparrow$	Co. $\uparrow$
Claude-Sonnet-4	OR	.667	.000	1.00	.249
Kimi-K2.5	OR	.667	.083	.917	.265
Gemini-2.5-Pro	OR	.333	.000	1.00	.156
Qwen3.5+	OR	.333	.085	.917	.226
Qwen3-235B	OR	.250	.000	.993	.251
DeepSeek-R1-0528 †	OR	.139	.806	.194	.042

Table 5: Cross-model comparison (zero-shot). Ex. = Executability, VC = VCER, Al. = Alignment, Co. = Coverage. Best per column bolded . † DeepSeek-R1-0528 VCER is inflated by reasoning-token extraction failures in its chain-of-thought output format.

ID	Problem	D.	Ex.	VC	Al.	Co.
MB-001	Colliding Blocks	4	.34	.12	.82	.21
MB-002	Gradient Descent	3	.00	.09	.82	.16
MB-003	Convolution	3	.56	.11	.85	.14
MB-004	Eigenvectors	4	.14	.19	.65	.12
MB-005	Determinant	2	.25	.08	.84	.16
MB-006	CLT	3	.33	.08	.85	.19

Table 6: Per-problem results averaged across all nine models (zero-shot). D. = Difficulty.

Strategy	Ex.	$\Delta$ Ex.	VC	Al.	Co.
Zero-Shot	.000	—	.000	.993	.172
Few-Shot	.111	+11.1	.000	1.00	.200
CoT	.000	+0.0	.028	.972	.159
Constraint	.000	+0.0	.000	.995	.171
V-Aware	.000	+0.0	.000	1.00	.175

Table 7: Prompting strategy ablation on Gemma-3-27B (3 trials

\times

12 problems).

\Delta

relative to zero-shot.

	CS4		Kimi		Gem25		Q3.5+		Q235B
	E	C	E	C	E	C	E	C	E	C
MB-001	$\bullet$	.31	$\bullet$	.42	$\bullet$	.08	$\circ$	.27	$\circ$	.34
MB-002	$\circ$	.23	$\circ$	.21	$\circ$	.26	$\circ$	.17	$\circ$	.21
MB-003	$\bullet$	.25	$\bullet$	.20	$\bullet$	.01	$\bullet$	.17	$\bullet$	.26
MB-004	$\circ$	.25	$\bullet$	.25	$\circ$	.19	$\circ$	.00	$\circ$	.13
MB-005	$\bullet$	.25	$\bullet$	.24	$\circ$	.16	$\circ$	.22	$\circ$	.17

Table 8: Per-model per-problem grid (zero-shot). CS4 = Claude-Sonnet-4, Gem25 = Gemini-2.5-Pro. E = executability (

\bullet

\circ

), C = coverage.

Citation Hallucination

See all 10 papers

ManimAgent: Self-Evolving Multimodal Agents for Visual Education
Jun 29, 2026Wenjia Jiang, Zongyuan Cai, Yuanhang Shao +7Continual Learning for LLM Agents LLM Agent Self-Improvement
University of Alberta · Southeast University · Virginia Tech +2
MathVis: Can AI Reason through Drawing? A Benchmark for Mathematical Figure Generation
Oct 9, 2026Jingjing Wei, Hao Zhao, Xinke Li +3Mathematical Reasoning Benchmarks LLM Mathematical Reasoning
Zhihu (zhihu.com) · Tsinghua University (tsinghua.edu.cn) · City University of Hong Kong (cityu.edu.hk) +1
Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities
Aug 9, 2026Harish Kashyap, Kiran Byadarhaly, Sriram Chakaravarthy +2LLM Evaluation T2I Generation Evaluation
Pandita AI Inc., 2540 N First St, San Jose CA 95131, USA

Jun 29, 2026J/K move · Enter open · S save

ManimAgent: Self-Evolving Multimodal Agents for Visual Education

Wenjia Jiang, Zongyuan Cai, Yuanhang Shao, Chenru Wang+6

University of Alberta · Southeast University · Virginia Tech+2

Continual Learning for LLM Agents LLM Agent Self-Improvement Agentic Code Generation Multimodal Agents Memory-Augmented Language Model Agents Self-Reflection in Language Model Agents

Open paper AskPDF arXiv

Multi-round reflection lets agents built on large language models recover from failures within a single task, but each task remains an isolated episode: lessons learned across many reflection rounds on one task are discarded before the next begins. We study this gap on a code-generation task: from a scientific paper section, the agent writes Python in the open-source Manim library to render a mathematical animation. We present ManimAgent, a self-evolving multimodal agent that carries reflection experience across tasks through a dual-channel Episodic Memory Bank grown entirely from its own task stream, with no weight updates and no human seeds. After each animation converges, a vision-language model scores the rendered keyframes; the resulting signals populate a positive channel M+ that stores success rationales as soft Reference Examples, and a negative channel M- that stores validated failure patterns as hard Known Pitfalls. On a fixed-probe evaluation against no-memory, matched-budget retrieval-augmented generation, and shuffled-memory baselines, blind human Pass@1 rises and reflection rounds fall as memory size grows.

Abstract

Citation Hallucination

Explore similar work

ManimAgent: Self-Evolving Multimodal Agents for Visual Education

MathVis: Can AI Reason through Drawing? A Benchmark for Mathematical Figure Generation

Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities