Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

Xilin Wei; Shengyuan Ding; Xinyu Fang; Tiancheng Gu; Yuchi Wang; Zhixun Li; Ziyu Liu; Yuhang Zang

cs.AIOct 26, 2026

Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

Authors: Xilin Wei, Shengyuan Ding, Xinyu Fang, Tiancheng Gu, Yuchi Wang, Zhixun Li, Ziyu Liu, Yuhang Zang,

Organizations: Fudan University · Alibaba Token Hub, Alibaba Group · Zhejiang University · The Chinese University of Hong Kong ·

Abstract

Evaluating human-AI co-design requires testing whether coding agents can translate real users' multimodal requests into working application revisions. We introduce Co-Design Bench, a reference-free benchmark of 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions. Its two tracks, Multimodal Editing (ME) and Repository Editing (RE), cover localized edits with optional image, audio, video, or document inputs and coordinated repository revisions. Tasks span three capability dimensions: Grounded Understanding, UI Transformation, and Functional Implementation. The tasks pair normalized requests and runnable pre-edit applications with 2,331 hidden expert-authored acceptance criteria. Our judge-then-refute protocol grades criteria using source and browser evidence, challenges positive verdicts with a separate LLM verifier, and audits visual and interaction claims without gold code or target screenshots. Under a shared agentic harness, the strongest of ten model endpoints passes an average of 69.9% of criteria per task but fully satisfies only 26.4% of tasks. All ten endpoints achieve lower criterion completion on repository edits spanning three or four change types than on those spanning one or two. Co-Design Bench provides a testbed for studying the gap between partial requirement satisfaction and complete task success under agentic execution for real-user application revisions.

Figures & tables

Figure 1: Execution regimes and task provenance. (a) Mean graded feature completion on $150$ WebCompass-edit tasks ( Lei et al., 2026 ) : qwen3.7-plus rises from $73.6\%$ (single-shot) to $95.1\%$ (agentic), with the feature rubric, judge, and render pipeline held fixed; opus-4-8 reaches $99.4\%$ with agentic execution (Appendix A ). The cross-benchmark bars use different task pools and evaluation protocols. (b) A WebCompass request drawn from predefined edit operations and a Co-Design Bench request recovered from a real coding session, with their evaluation workflows.

Reference reconstruction
Benchmark / subset	Task		Evaluation
	Real-user sessions	Input modalities	Reference-free acceptance	Agentic evaluation	Adversarial verification
Design2Code ( Si et al., 2024 )	$\times$	I	$\times$	$\times$	$\times$
WebSight-Test ( Liang et al., 2024 )	$\times$	I	$\times$	$\times$	$\times$
WebRenderBench ( Lai et al., 2025 )	$\times$	I	$\times$	$\times$	$\times$
IWR-Bench ( Chen et al., 2025 )	$\times$	I,V	$\times$	$\times$	$\times$

Table 1: Comparison of web-coding benchmarks. Task origins and core evaluation capabilities of the reported settings.

Figure 2: Data construction and filtering. (a) Recover and validate runnable pre-edit applications, repairing eligible environments and excluding unrecoverable cases. (b) Deduplicate and filter requests, rank candidates by difficulty, verifiability, and historical failure signals, then select bounded session intervals for consolidation and expert rubric authorship.

Figure 3: Evaluation from artifact to verdict. (a) Build and render the candidate, collect source and browser evidence, and grade hidden criteria. (b) Challenge positive verdicts, audit required evidence, and aggregate scores. Parallel baseline and solvability checks diagnose failures for supplementary attribution analysis.

Panel of Table 2 — Table 2: Main benchmark performance across ME and RE. Ranked by mean strict completion $\bar{R}$ . Metrics report Strict ( $\bar{R}$ ), Graded ( $\bar{G}$ ), and Full-Pass Rate ( $\mathrm{F}$ , in %). Rollout statistics summarize median turns, mean turn output tokens, and total token traffic per case. Tool distributions are grouped into execution ( Bash ), inspection ( Read/Grep/Glob ), modification ( Edit/Write ), and auxiliary. Bold indicates peak track performance.

Panel of Figure 4 — Figure 4: Mean strict completion ( $\bar{R}$ , in %) across task strata ( $N=273$ ). Ten endpoints are ordered by ME mean strict completion. (a) Five input attachment modalities; bold outlines mark column maxima. (b) Tasks with 1–2 edit types (blue) versus 3–4 (gold). Parentheses give task counts. Pooled rows average all task scores across ten endpoints (Appendix B.2 ).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Regime	Mean graded completion (%)	All-pass tasks	Full-pass rate (%)
Single-shot	$73.6$	$49$	$32.7$
Agentic	$95.1$	$104$	$69.3$

Appendix

Table 4: Paired execution-regime comparison on 150 WebCompass-edit tasks. Both regimes use qwen3.7-plus and the same feature rubric, vision judge, and rendering pipeline. Mean completion averages task-level graded scores; an all-pass task satisfies every feature.

Panel of Table 5 — Table 5: Combined performance across both tracks. Strict ( $\bar{R}$ ) and Graded ( $\bar{G}$ ) weight each task equally over the full benchmark; Full ( $\mathrm{F}$ ) is the all-pass task rate. Rows are ranked by $\bar{R}$ . Per-track results are in Table 2 .

Modality	Completion		Verdicts (%)			Outcomes
	Strict	Graded	Pass	Partial	Fail	Full	Zero
None	0.637	0.672	64.8	6.7	28.4	37.0	14.5
Image	0.590	0.731	61.5	25.4	13.1	10.0	2.5
Video	0.625	0.691	62.9	13.1	24.0	19.6	1.5
Audio	0.584	0.652	58.9	13.2	27.9	20.3	0.3
Document	0.652	0.682	66.5	6.2	27.3	18.2	7.1

Appendix

Table 6: Completion by input modality and edit breadth across ten endpoints. Strict (

\bar{R}

) and Graded (

\bar{G}

) average task scores; verdict shares pool criterion judgments. Full and Zero are task rates (%) with

R_{t}=1

and

R_{t}=0

Endpoint	Track	Production	glm-5.2	kimi-k3	Qwen3.8-27B
kimi-k3	1	0.718	0.699 ( $-0.019$ )	0.737 ( $+0.018$ )	0.732 ( $+0.013$ )
	2	0.681	0.641 ( $-0.040$ )	0.676 ( $-0.005$ )	0.662 ( $-0.019$ )
qwen3.8-27b	1	0.705	0.666 ( $-0.039$ )	0.706 ( $+0.001$ )	0.722 ( $+0.017$ )
	2	0.711	0.658 ( $-0.053$ )	0.702 ( $-0.009$ )	0.686 ( $-0.025$ )

Appendix

Table 7: Graded completion under four evaluator configurations. All evaluators score the same released tasks in each track. Parentheses give changes from production.

Figure 5: Representative ME cases: grounded local edits. The examples cover cross-source reconciliation, cross-view consistency, and visual-reference grounding. Numbered markers organize the visible evidence; the right column visualizes three selected evaluator targets as value flows, compact charts, and state transitions. The complete rubric may contain additional criteria.

Figure 6: Representative RE cases: coordinated repository edits. The examples span navigation and page construction, clip editing under resource limits, and theme-aware motion with a morphing mega menu. The selected targets use route flows, thresholds, token palettes, and interaction states to make the behavior and preservation constraints explicit.

Figure 7: Two complementary tracks in Co-Design Bench. Both originate in real user coding sessions. ME groups tasks by primary attachment modality (“None”: no additional attachment); RE change types overlap. Bars show shares within each track; criteria per task is the mean. Detailed statistics appear in Table 8 .

	ME	RE
	design edit	whole-app reflow
Tasks	$100$	$173$
Evaluation criteria	$553$	$1{,}778$
per task, mean / median / max	$5.5$ / $6$ / $11$	$10.3$ / $8$ / $46$
Tasks with attachments / captures	$80$ ( $80\%$ )	$142$ ( $82\%$ )
Mean reference materials per task	$1.35$ attachments	$5.1$ captures

Appendix

Table 8: Composition of the 273-task benchmark. Percentages use each track's task count; RE edit types overlap. Mean captures are computed over tasks with captures.

Starting artifact	None	Image	Video	Audio	Document	Total	Strict $\uparrow$
	Attachment modality
single-file HTML page	2	4	25	32	16	79	0.619
bundled application	18	–	–	–	1	19	0.628
react source tree	–	–	1	–	–	1	0.440
vue source tree	–	–	1	–	–	1	0.500
bundled, all three	18	–	2	–	1	21	0.613

Appendix

Table 9: Starting artifact and attachment modality on ME. Shading shows task counts; dashes denote zero. Strict (

\bar{R}

) averages task scores across ten endpoints; grey scores represent single-task groups. The bundled subtotal includes both source-tree rows.

Figure 8: Capability hierarchy and benchmark coverage (

N=273

). Primary assignments partition tasks by main delivery goal; Any-label coverage includes auxiliary requirements, allowing overlap across categories. All thirty fine-grained capabilities are represented across the task suite.

Figure 9: Mean strict completion ( $\bar{R}$ ) across primary capability categories and model endpoints (%). Bold rows denote top-level dimensions; sub-rows report constituent categories. Pooled columns report overall average scores across all evaluated pairs, while Full Pass ( $\mathrm{F}$ ) denotes the share of completely resolved tasks. The low-sample Motion and Adaptation category ( $\dagger$ , $n=4$ ) is displayed in grey.

Modality (tasks)	Runs	ffmpeg	OpenCV	faster-whisper
No attachment (20)	200	1 (0.5%)	0 (0.0%)	0 (0.0%)
Image (4)	40	0 (0.0%)	0 (0.0%)	0 (0.0%)
Video (27)	270	257 (31.5%)	2,438 (68.5%)	20 (3.3%)
Audio (32)	320	160 (25.0%)	3 (0.3%)	65 (4.4%)
Document (17)	170	0 (0.0%)	96 (3.5%)	0 (0.0%)
All ME (100)	1000	418 (16.6%)	2,537 (19.2%)	85 (2.3%)

Appendix

Table 10: Identifiable media-processing Bash requests by modality. All final runs of the ten endpoints on the released ME tasks are included. Each cell gives the number of Bash requests and, in parentheses, the percentage of runs with at least one such request. A request counts once per named tool; columns can overlap. Installation, import/version checks, and script preparation alone are excluded; failed processing requests are included.

Solver	External models	Observed evidence
gpt-6-astra	AST (AudioSet), CLAP (HTSAT-unfused), Whisper small, Qwen2-Audio-7B-Instruct	AudioSet class scores, audio–text similarity scores, timestamped transcripts, and generated audio descriptions.
opus-5	AST (AudioSet), Whisper small	Audio class scores (e.g., Frog 0.91 and Church bell 0.84) and timestamped Chinese transcripts.
kimi-k3	AST and YAMNet (ONNX weights)	AST weights downloaded; YAMNet inference returns animal-class scores over time.
glm-5.3-flash	CLIP ViT-B/32, DINOv2-small, OWLv2-base (ensemble)	Video-frame scene scores, crop embedding similarities, and object detection boxes.
glm-5.2	Qwen2-Audio-7B-Instruct, AST (AudioSet), Whisper base, large-v3	Generated audio descriptions and timestamps, AudioSet class scores, and timestamped transcripts.

Appendix

Table 11: Hugging Face models in selected final trajectories. Rows group external models and observed outputs by solver. AST is the Audio Spectrogram Transformer fine-tuned on AudioSet; Whisper variants are invoked through faster-whisper .

	ME ( $n=100$ )		RE ( $n=173$ )
Harness	kimi-k3	Qwen3.8-27B	kimi-k3	Qwen3.8-27B
codex	0.842	0.773	0.649	0.638
opencode	0.828	0.801	0.709	0.749
qwen-code	0.849	0.722	0.682	0.560
Spread (max $-$ min)	0.021	0.079	0.060	0.189
kimi $-$ 27B range	$+0.027\to+0.127$ (10.0 pp)		$-0.040\to+0.122$ (16.2 pp)

Appendix

Table 12: Completion under three execution harnesses. Evaluated across all

273

benchmark tasks (ME:

100

; RE:

173

) with the solver model held fixed while the harness is swapped across codex , opencode , and qwen-code . Cells report average completion rates. The best harness per column is bold . Harness rankings invert across solvers on ME (Spearman

\rho=-1.00

), and the performance gap between kimi-k3 and Qwen3.8-27B varies by

10.0

percentage points on ME and

16.2

on RE.

Coding Agent Benchmarks

See all 47 papers

ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders
Jul 23, 2026Zhongyuan Peng, Dan Huang, Chuyu Zhang +8Coding Agents Software Engineering Benchmarks
Fudan University · Meituan Group · Singapore Management University +1
SWE-Together: Evaluating Coding Agents in Interactive User Sessions
Jun 29, 2026Yifan Wu, Zhuokai Zhao, Songlin Li +8Coding Agents AI Agent Evaluation
UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users
Oct 21, 2026Zhan-Yu Wu, Ji Luo, Alyssa Wu +5Coding Agents AI Agent Evaluation
Tencent Hunyuan

Jul 23, 2026J/K move · Enter open · S save

ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

Zhongyuan Peng, Dan Huang, Chuyu Zhang, Caijun Xu+7

Fudan University · Meituan Group · Singapore Management University+1

Coding Agents Software Engineering Benchmarks AI Agent Benchmarks Code Generation Benchmarks

Open paper AskPDF arXiv

The recent emergence of vibe-coding workflows is changing what coding agents are expected to do. Instead of merely completing code under fully specified instructions, agents are increasingly expected to transform incomplete product intent into working software by combining various abilities including planning, requirement clarification, tool use, debugging, and repository-level construction. Yet existing benchmarks have not fully caught up with this shift, evaluating agents on static, fully specified tasks. In this paper, we introduce ICAE-Bench, a benchmark for evaluating coding agents under interactive project-building settings. The basic idea is to start from a fuzzy product requirement, simulating the dynamic paradigm with an automated User Agent. To make this setting both realistic and evaluable, ICAE-Bench introduces three key designs. First, to avoid the ambiguity of unconstrained fuzzy requirements, each task derives ambiguity from a precise real open-source repository with executable behavior. Second, to ensure high-quality and reproducible user simulation, ICAE-Bench grounds interaction through User Agent Data, allowing the User Agent to reveal hidden constraints without inventing new requirements or leaking implementation artifacts. Third, to evaluate open-ended repositories fairly, ICAE-Bench uses standardized black-box tests together with multi-dimensional diagnostics, including functional correctness, semantic and API similarity, structural fidelity, design quality, and interaction quality.

Abstract

Coding Agent Benchmarks

Explore similar work

ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

SWE-Together: Evaluating Coding Agents in Interactive User Sessions

UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users