BURGER: Benchmarking and Understanding Robotic Generalization via Embodied Drawing

Xiangyu Xu; Bangshuai Peng; Ziyi Liu; Mengjie Zhao; Chenhao Lu; Zhengyang Fan; Huazhe Xu; Kaizhe Hu

cs.ROOct 29, 2026

BURGER: Benchmarking and Understanding Robotic Generalization via Embodied Drawing

Authors: Xiangyu Xu, Bangshuai Peng, Ziyi Liu, Mengjie Zhao, Chenhao Lu, Zhengyang Fan, Huazhe Xu, Kaizhe Hu

Organizations: Tsinghua University · Shanghai Qizhi Institute

Abstract

How effectively can vision-language-action (VLA) models learn diverse language-conditioned behaviors as training data scales? Investigating this question requires large and varied robot datasets, whose collection on physical robots remains costly. We introduce BURGER, a language-conditioned robotic drawing benchmark with a scalable, prompt-driven pipeline for generating continuous drawing trajectories, both in simulation and the real world. In BURGER, an agent starts from the same blank canvas and controls a robot arm to draw the object or spatial composition specified by language, while each instruction allows an open-ended variety of valid drawings and stroke orders. Using 28,200 simulated demonstrations spanning 282 drawing tasks, we evaluate five representative VLA models across data scales, language grounding, and multimodal behavior. BURGER supports fine-grained analysis of language grounding, spatial composition, and action generation, revealing distinct failure patterns across VLA models. Our experiments suggest that some VLA models can benefit from greater task diversity and demonstration scale, capture multimodal drawing behaviors, and generalize to unseen compositions of familiar objects. Finally, we deploy a trained policy on a Franka Panda arm, demonstrating the ability to draw reactively under human intervention. Project homepage and code can be found at https://siyingzhao.github.io/BURGER/.

Figures & tables

Figure 1: Overview of BURGER. Top: Prompt-generated PNG images are converted into skeletons, SVG strokes, and robot trajectories. Bottom left: BURGER tests semantic grounding, spatial reasoning, and multimodal action generation in VLA models. Bottom right: We evaluate models on single-object drawing, compositional drawing, data scaling, and real-world deployment.

Figure 2: Instruction-conditioned planar trajectory diversity. The three rows show LIBERO demonstrations, BURGER demonstrations, and $\pi_{0}$ rollouts, respectively. Each subplot overlays trajectories for the same instruction. Different modes correspond to distinct clusters of execution routes.

Figure 3: Real-world deployment of BURGER. Left: The Franka Panda hardware setup equipped with a drawing tool. Right: The real-robot data collection process for a single task “birthday cake” and a compositional task “moon left sailboat”.

Figure 4: Cross-simulator deployment of the BURGER trajectory-generation pipeline. The same SVG drawing task is executed in SAPIEN, MuJoCo, and Isaac Sim using corresponding robot-control interfaces.

Figure 5: Dataset diversity and VLA evaluation in BURGER. We show representative single-object and compositional training tasks, along with corresponding VLA rollouts. The rightmost column evaluates generalization to unseen spatial compositions.

Panel of Figure 6 — Figure 6: Scaling behavior of $\pi_{0}$ , $\pi_{0.5}$ , and $\pi_{0}$ -FAST across training data scales. The x-axis denotes the number of training classes, the y-axis shows the mean target similarity, and colors indicate the number of demonstrations per class. Flow-matching models improve more consistently with scale, whereas $\pi_{0}$ -FAST shows more variable trends.

Figure 7: Learning multimodal drawing behaviors in BURGER. Top: different drawing patterns of “helmet” within the dataset, Bottom: the generated images by $\pi_{0.5}$ unseen in the dataset.

Figure 8: Representative failure modes in BURGER. Left: OpenVLA and $\pi_{0}$ -FAST exhibit sensitivity to the initial end-effector position. Middle: $\pi_{0}$ and $\pi_{0.5}$ show failure to terminate. Right: $\pi$ -family models produce generic, unstructured strokes on unseen-object instructions.

Figure 9: Controlled circle-drawing results across representative VLA models, revealing distinct failure patterns in contact, trajectory progress, and multimodal behavior.

Source	$\mathbf{N}$	Contact	CW(%)	Start Entropy	Sweep( ∘ )
Dataset	100	100	56.0	0.97	359.8
OpenVLA	20	20	40.0	0.82	874.6
OFT $H=1$	20	6	100.0	0.35	129.3
OFT $H=8$	20	20	55.0	0.58	586.1
$\pi_{0}$ -FAST	20	20	95.0	0.63	563.2
$\pi_{0.5}$	20	20	55.0	0.84	355.2

Table 1: Quantitative results on the controlled circle-drawing task. CW denotes the proportion of clockwise trajectories. Higher start entropy indicates more uniform starting-point coverage, while sweep values closer to

360^{\circ}

indicate better completion of a single revolution.

Figure 10: Real-robot drawing results. The first row shows robustness to camera-angle changes; the second shows visual feedback through object completion and human intervention; the third shows multimodal real-robot behaviors.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Classes
Animals (14)	butterfly , cat , cow , dog , duck , elephant , fish , frog , horse , owl , penguin , pig , rabbit , zebra
Food (17)	apple , banana , birthday_cake , blueberry , donut , grapes , hot_dog , ice_cream , lollipop , mushroom , onion , pear , pineapple , pizza , popsicle , sandwich , strawberry
Kitchenware (10)	bottlecap , bucket , coffee_cup , cup , mug , teapot , wine_bottle , fork , knife , spoon ,
Vehicles and transport (11)	bicycle , bus , car , police_car , sailboat , school_bus , skateboard , traffic_light , train , truck , wheel
Furniture and home objects (17)	bathtub , bed , bench , candle , chair , couch , door , dresser , house , mailbox , picture_frame , pillow , sink , table , tent , toilet , vase
Clothing and accessories (18)	backpack , belt , bowtie , crown , eyeglasses , hat , headphones , helmet , jacket , necklace , pants , purse , shoe , sock , suitcase , t_shirt , umbrella , wristwatch

Appendix

Table 2: Single-object classes in BURGER.

Left object $A$	Right-side objects $B$	#
leaf	lollipop , mailbox , moon , mountain , snowman	5
light_bulb	leaf , picture_frame , radio	3
lollipop	leaf , toaster	2
mailbox	leaf , moon , rainbow	3
moon	leaf , mailbox , paintbrush , sailboat	4
mountain	owl	1

Appendix

Table 3: Compositional tasks in BURGER. Each listed task follows the form A_left_B , corresponding to the language instruction “draw A to the left of B.” The table lists the 100 horizontal tasks by the left object

A

and the corresponding right-side objects

B

; the dataset also contains 50 tasks with vertical relations.

Figure 12: Stroke-order and geometry diversity in BURGER. Examples show different object shapes and drawing sequences, with colors indicating stroke order from dark blue to light blue, light red, and dark red.

Figure 13: Data comparison with other sketch generation methods.

Figure 14: Codebook-size sensitivity. Solid curves show the reference seed; shaded bands show the min–max across ten initialization seeds on a fixed trajectory sample.

Figure 15: Visualization of the real-robot dataset. The first row shows the corresponding simulated data. The second and third rows show intermediate stages of real-robot data collection. The last two rows show completed real-robot demonstrations.

Metric	Spearman $\rho$	Kendall $\tau_{b}$	Pairwise Agreement
CLIPScore [ 11 ]	0.592	0.433	0.803
Similarity metric	0.834	0.655	0.875

Appendix

Table 4: Correlation between metrics and human judgments.

Panel of Figure 16 — Figure 16: Qualitative data-scaling results for $\pi_{0.5}$ . Panels (a), (b), and (c) show models trained with 10, 50, and 100 demonstrations per class, respectively. Within each panel, rows correspond to different numbers of training classes. Increasing either the number of demonstrations per class or the number of training classes improves drawing quality.

Figure 17: Qualitative comparison of representative VLA models on BURGER.

Figure 18: Multimodal drawing behaviors in the training set. The same object class can be represented by different stroke orders, shapes, and local geometric variations.

Figure 19: Multimodal drawing behaviors in VLA rollouts. The generated drawings show that trained VLA models can produce multiple valid realizations for the same language instruction.

Figure 20: Above: In-distribution spatial-relation grounding. VLA models learn to generate compositional drawings for object pairs observed during training. Below: Generalization to unseen object compositions. VLA models can apply learned spatial relations to object pairs that are not observed during training.

Instruction Generalization in VLA Models

Instruction generalization in vision-language-action models studies whether robots map varied, novel, or compositional commands to the intended actions.

See all 41 papers

Colosseum V2: Benchmarking Generalization for Vision-Language-Action Models
May 26, 2026Jeremy Morgan, Hyeonho Oh, Prajwal Vijay +7Robotics Simulation Generalization in Robotic Manipulation
Department of Computer Science, University of Southern California, Los Angeles, CA, USA · Department of Electrical Engineering, Indian Institute of Technology Madras, Chennai, India · Fu Foundation School of Engineering and Applied Science, Columbia University, New York, NY, USA
RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?
Sep 4, 2026Zhenxuan Fan, Bo Zhang, Yutong Lin +9Robot Manipulation Benchmarks Reasoning for VLA Models
Zhejiang University · University of Electronic Science and Technology of China · South China Normal University
Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
Mar 19, 2026Bryce Grant, Xijia Zhao, Peng WangVision-Language-Action Models Robotics
Case Western Reserve University

May 26, 2026J/K move · Enter open · S save

Colosseum V2: Benchmarking Generalization for Vision-Language-Action Models

Jeremy Morgan, Hyeonho Oh, Prajwal Vijay, Jincen Song+6

Department of Computer Science, University of Southern California, Los Angeles, CA, USA · Department of Electrical Engineering, Indian Institute of Technology Madras, Chennai, India · Fu Foundation School of Engineering and Applied Science, Columbia University, New York, NY, USA

Robotics Simulation Generalization in Robotic Manipulation Vision-Language-Action Models Robot Manipulation Benchmarks Robot Policy Generalization

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models demonstrate promising generalization in robotic manipulation, driven by advances in large-scale vision and language pre-training. This progress can be misleading. Despite the zero-shot perception and language capabilities of VLAs, their overall task performance often degrades under distribution shifts, revealing gaps in how these systems translate high-level understanding into robust behavior. To systematically study this gap, we introduce Colosseum V2, a large-scale simulation benchmark for evaluating VLA generalization in robot learning across diverse conditions. The benchmark comprises 28 tasks spanning 13 task categories and two robot morphologies, covering a wide range of manipulation primitives and long-horizon behaviors. Built on the ManiSkill simulator, Colosseum V2 enables fast, GPU-parallelized evaluation and supports both in-domain and out-of-domain testing at scale. We evaluate state-of-the-art methods, including Action Chunking Transformers (ACT) and Pi0.5, and reveal limitations in both base performance and generalization. We demonstrate strong correlations between simulation and real-world metrics that support the ecological validity of the benchmark. By standardizing tasks, metrics, and evaluation protocols within a unified benchmark, Colosseum V2 enables reproducible and fair comparisons, reduced evaluation overhead, and accelerated progress toward general-purpose robot policies.

Abstract

Instruction Generalization in VLA Models

Explore similar work

Colosseum V2: Benchmarking Generalization for Vision-Language-Action Models

RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models