A Matched-Budget Audit Framework for Recaptioned Image-Text Supervision Distributions

Giyeong Oh; Junghun Park; Yuhan Bae; Youngjae Yu

cs.CVOct 1, 2026

A Matched-Budget Audit Framework for Recaptioned Image-Text Supervision Distributions

Authors: Giyeong Oh, Junghun Park, Yuhan Bae, Youngjae Yu

Organizations: Seoul National University

Abstract

Recaptioned image-text corpora are now standard for text-to-image (T2I) training, with vision--language model (VLM) captioners replacing sparse alt-text by dense descriptions. A recaptioned corpus is a supervision distribution induced by a documented captioning policy ( $π$ ), captioner ( $V_c$ ), and source corpus ( $C$ ). Length-correlated proxies miss caption-register artifacts and downstream T2I benchmarks entangle the corpus with training choices, so this distribution is hard to audit at corpus scale. We introduce a reusable matched-budget audit framework for recaptioned supervision distributions $D_{π,V_c,C}$ : at a fixed text budget of $B = 64$ it reports a five-axis profile spanning prompt-side coverage, image-conditioned faithfulness, and caption-surface health, with claimed controllable basic units (CBU) as the common claim unit. We instantiate the framework on seven paired comparisons over five public source corpora. Across the four cross-corpus pairs, the released surface raises supported CBU per caption by $+3.39$ to $+6.36$ under both Qwen and Gemma Judges, and on CC12M the same framework exposes a long-vs-dense frontier that is consistent across both judges and four budgets. We release the audited multi-source recap corpus ( $\approx$ 490M) together with the audit-artifact bundle.

Figures & tables

Panel of Figure 1 — Figure 1 : VLM bias is inevitable; policy decides which biases survive. Left: 2 2 2 Photo: “Toronto: Riverdale farm” by The City of Toronto (Flickr), CC BY 2.0, cropped. describe-style frame prefix concentration drops on the released surface. Right: averaged across 7 prompt pools per cross-corpus pair, Ours raises prompt-mass support on every pair and lowers $n$ -gram Jensen-Shannon Divergence (JSD) against the pool distribution on DataComp, PD12M, and Danbooru, with the length-matched LAION-pop pair within $0.01$ ; bars on the left axis report support, markers on the right axis report JSD; definitions are in Section 4.2 .

Work	Target	Natural	Corpus	Unit	Image	Budget	Prompt
REVISE ( 56 )	visual datasets	✓	✓	–	–	–	–
LAION’s Den ( 4 )	image–alt-text pairs	✓	✓	–	–	–	–
Hirota et al. ( 23 )	caption enrichment	✓	✓	✓	✓	–	–
Brack et al. ( 6 )	training captions	✓	✓	–	–	–	–
TIFA / DSG ( 25 ; 12 )	generated images	–	–	✓	✓	–	–
FAITHSCORE ( 28 )	VLM answers	✓	–	✓	✓	–	–

Table 1 : Dataset and caption studies (top) and claim-level metrics (middle). Natural : text on non-generated images; Corpus : dataset-level analysis; Unit : text scored per claim or object mention; Image : text verified against the image; Budget : fixed text window; Prompt : register of user prompts.

Source family	Original supervision	Ours scale	Paired reference surface(s)
Photorealistic / web
DataComp ( 16 )	web image–text pairs	$\approx 325.5$ M	Recap-DataComp ( 32 )
CC12M ( 10 )	web alt-text	$\approx 11.5$ M	CC12M-LLaVA-NeXT ( 8 ) , PixelProse ( 49 ) , CC12M-Qwen3-VL † ( 53 )
LAION-pop ( 46 )	web alt-text	$\approx 0.4$ M	LAION-pop-Llama ( 9 )
PD12M ( 36 )	Florence-2 + metadata	$\approx 12.4$ M	PD12M released ( 50 )
CommonCatalog ( 19 )	BLIP-2 captions	$\approx 14.6$ M	—

Table 2 : Nine recaptioned source families: five with paired audits, four release-only. Ours scale counts unique images per family; a dash marks a release-only family. The dagger † marks short tag-style surfaces (under

20

lexical units).

Source	Reference release	Avg lex	Opener $\downarrow$	Top-100 raw $\downarrow$	Top-100 content $\downarrow$	Distinct-3 $\uparrow$
DataComp	Recap-DataComp ( 32 )	$50.9$	$0.36\%$	$27.99\%$	$17.53\%$	$11.49\%$
CC12M	CC12M-LLaVA-NeXT ( 8 )	$72.2$	$0.24\%$	$22.97\%$	$12.39\%$	$13.81\%$
	PixelProse ( 49 )	$89.5$	$100.00\%$	$65.07\%$	$8.25\%$	$12.56\%$
	CC12M-Qwen3-VL † ( 53 )	$11.6$	$0.00\%$	$9.40\%$	$5.87\%$	$37.04\%$
LAION-pop	LAION-pop-Llama ( 9 )	$180.3$	$0.78\%$	$23.68\%$	$14.54\%$	$24.83\%$
PD12M	PD12M released ( 50 )	$39.7$	$99.98\%$	$90.73\%$	$43.10\%$	$2.40\%$

Table 3 : Public recap surfaces concentrate beyond opener boilerplate. Phenomenon descriptors on up to

1\mathrm{M}

raw captions per surface; Avg lex is mean lexical-unit length. Ours keeps the opener rate at

1.3

–

2.3\%

on every source and, against every long-form reference, lowers top-100 raw and content mass and raises Distinct-3; dagger † marks short tag-style surfaces (

<20

lex), whose small 3-gram count raises Distinct-3. Per-source-family breakdown in Appendix Table 16 .

Axis	Property	Input	Metric	Rows
Text budget	Coverage	$D_{c}$	Avg. lex, $B$ -eligibility	$1$ M
Prompt-pool support	Coverage	$D_{c}$ , pools	prompt-mass support $\uparrow$ , $n$ -gram JSD $\downarrow$	$250$ k
Claimed density	Coverage	$D_{c}$	CBU/cap $\uparrow$ , CBU/100 lex	$5$ k
Surface concentration	Health	$D_{c}$	top-100 prefix mass $\downarrow$ , distinct-3 $\uparrow$	$1$ M
Support and risk	Faithfulness	$D_{cx}$	$\mathbb{E}[s]\uparrow$ , $\rho\downarrow$	$5$ k

Table 4 : Audit axes. Arrows mark the better direction; Rows is the cap on paired rows per slice.

				Qwen Judge		Gemma Judge
Dataset	Avg lex	CBU/cap $\uparrow$	Pool-wins $\uparrow$	Sup. CBU/cap $\uparrow$	Risk $\downarrow$	Sup. CBU/cap $\uparrow$	Risk $\downarrow$
DataComp	$50.9$ $\,\to\,$ $175.5$	$10.44$ $\,\to\,$ $\mathbf{14.45}$	$\mathbf{6}/7$	$8.49$ $\,\to\,$ $\mathbf{13.73}$	$0.177$ $\,\to\,$ $\mathbf{0.035}$	$8.00$ $\,\to\,$ $\mathbf{12.94}$	$0.219$ $\,\to\,$ $\mathbf{0.081}$
LAION-pop	$180.3$ $\,\to\,$ $182.9$	$11.91$ $\,\to\,$ $\mathbf{14.82}$	$\mathbf{5}/7$	$10.80$ $\,\to\,$ $\mathbf{14.22}$	$0.077$ $\,\to\,$ $\mathbf{0.031}$	$10.22$ $\,\to\,$ $\mathbf{13.61}$	$0.113$ $\,\to\,$ $\mathbf{0.060}$
PD12M	$39.7$ $\,\to\,$ $189.1$	$9.78$ $\,\to\,$ $\mathbf{15.02}$	$\mathbf{7}/7$	$8.61$ $\,\to\,$ $\mathbf{14.29}$	$0.103$ $\,\to\,$ $\mathbf{0.034}$	$8.23$ $\,\to\,$ $\mathbf{13.53}$	$0.131$ $\,\to\,$ $\mathbf{0.066}$
Danbooru	$43.6$ $\,\to\,$ $164.5$	$8.18$ $\,\to\,$ $\mathbf{14.33}$	$\mathbf{7}/7$	$6.38$ $\,\to\,$ $\mathbf{12.74}$	$0.217$ $\,\to\,$ $\mathbf{0.058}$	$6.15$ $\,\to\,$ $\mathbf{11.85}$	$0.235$ $\,\to\,$ $\mathbf{0.094}$

Table 5 : Cross-corpus results at

B=64

. Cells

a\!\to\!b

report Ref

\!\to\!

Ours (bold = Ours). Pool-wins counts the seven prompt pools where Ours raises prompt-mass support; Qwen and Gemma Judges re-ask the same CBUs on

\approx 5{,}000

paired captions per surface; Risk is unsupported / claimed CBU. Per-pool deltas in Appendix Figure 3 , full descriptors in Appendix Tables 14 , 16 .

		Surface stats		Qwen Judge		Gemma Judge
Dataset	Surface	CBU/cap $\uparrow$	CBU/100lex	Sup. CBU/cap $\uparrow$	Risk $\downarrow$	Sup. CBU/cap $\uparrow$	Risk $\downarrow$
CC12M	Ours	$\mathbf{15.21}$	$23.16$	$\mathbf{14.60}$	$0.030$	$\mathbf{13.82}$	$0.066$
CC12M	Naive Qwen3.5-35B-A3B	$11.32$	$17.26$	$11.03$	$0.022$	$10.61$	$0.045$
DataComp	Ours	$\mathbf{14.60}$	$22.23$	$\mathbf{13.90}$	$0.036$	$\mathbf{13.07}$	$0.080$
DataComp	Naive Qwen3.5-35B-A3B	$10.95$	$16.66$	$10.56$	$0.026$	$10.07$	$0.057$

Table 6 : Captioner-control: same

V_{c}

, different

\pi

. Both policies caption the same

\approx 4.5

–

4.8

k images per source with the same decoding; the Qwen extractor

\phi

supplies the claims and both Judges answer the same questions (Appendix E ).

Metric	Ours	Naive	Naive $-$ Ours	Long-form refs.
Mean lex	$186.6$	$328.1$	$+141.5$	$67.5$ – $91.4$
Lex overflow 248	$6.9\%$	$91.2\%$	$+84.3$ pp	$0.1$ – $0.8\%$
Top-100 raw prefix mass $\downarrow$	$0.14$	$0.60$	$+0.46$	$0.29$ – $0.71$
Top-100 content prefix mass $\downarrow$	$0.09$	$0.33$	$+0.24$	$0.13$ – $0.16$
Distinct-3-gram rate $\uparrow$	$0.59$	$0.57$	$-0.02$	$0.40$ – $0.49$

Table 7 : Surface concentration on the CC12M Naive surface. Section 3.3 diagnostics on the CC12M images of Table 6 . Lex overflow 248 = share of captions over

248

lexical units; Long-form refs. = range over CC12M-LLaVA-NeXT and PixelProse on the same images.

	Surface stats		Qwen Judge		Gemma Judge
Surface	CBU/cap $\uparrow$	CBU/100lex	Sup. CBU/cap $\uparrow$	Risk $\downarrow$	Sup. CBU/cap $\uparrow$	Risk $\downarrow$
Ours	$\mathbf{15.21}$	$23.16$	$\mathbf{14.60}$	$0.030$	$\mathbf{13.82}$	$0.066$
CC12M-LLaVA-NeXT ( 8 )	$10.78$	$21.77$	$9.84$	$0.069$	$9.44$	$0.097$
PixelProse ( 49 )	$12.57$	$20.44$	$10.73$	$0.129$	$10.20$	$0.161$
CC12M-Qwen3-VL † ( 53 )	$6.44$	$55.84$	$6.31$	$0.014$	$6.19$	$0.028$

Table 8 : CC12M frontier at

B=64

, four surfaces under both Judges. CBU per caption and per

100

lex come from the Qwen claim extractor

\phi

; each Judge supplies VQA support and risk. The dagger † marks the short tag-style surface (

\approx 12

lex). All columns use the same

4{,}494

aligned images; per-surface denominators are in Appendix Table 15 .

Figure 2 : CC12M long-form surfaces at

B=64

and across the budget sweep. Left: supported-yield / risk at

B=64

; marker shape encodes surface, fill (filled vs hollow) encodes Judge. Right: claimed-CBU budget frontier over

B\in\{16,32,48,64\}

on all

4{,}494

aligned images; marker size scales with

B

(a) Judge–human agreement
Judge	$n$	Overall		Ours	Pooled refs.
Qwen	$43$	$84.8\pm 9.7\%$		$87.9\pm 10.9\%$	$82.5\pm 14.6\%$
Gemma	$43$	$84.2\pm 9.7\%$		$89.0\pm 10.8\%$	$80.7\pm 14.7\%$
(b) Human image support
Surface	$n$	Yes	Uncertain	Explicit no	Other
Ours	$111$	$\mathbf{87\;(78.4\%)}$	$22\;(19.8\%)$	$\mathbf{0\;(0.0\%)}$	$2\;(1.8\%)$

Table 9 : Human verification of image support. Panel (a): design-weighted nominal exact agreement on

43

resolved claims, estimate

\pm

standard deviation over

10{,}000

image-cluster bootstrap resamples. Panel (b): unweighted distribution of all

217

primary image judgments; uncertainty remains separate from support, and “other” is not visual or prefer not to answer .

(a) DataComp text-space probes
	Vendi $\uparrow$		eRank $\uparrow$		Coverage $\uparrow$
Encoder	Ours	Ref	Ours	Ref	Ours	Ref
Qwen3-Emb-4B	$61.40$	$\mathbf{62.57}$	$\mathbf{337.2}$	$319.5$	$\mathbf{0.743}$	$0.357$
Qwen3-Emb-8B	$76.33$	$\mathbf{89.25}$	$\mathbf{424.0}$	$378.5$	–	–
BGE-M3	$25.39$	$\mathbf{34.21}$	$\mathbf{255.1}$	$235.5$	$0.017$	$\mathbf{0.050}$

Table 10 : Encoder probes follow the encoder and its truncation. (a) DataComp, Ours and Ref under three text encoders; Coverage is prompt-to-caption PRDC. (b) CC12M LongCLIP R@1 with full captions and with captions cut to

B=64

( input64 ). Full tables in Appendix F .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol or term	Meaning
Supervision distribution
$C$ , $x$	source image corpus and one of its images
$V_{c}$ , $\pi$	captioner and captioning policy
$D_{\pi,V_{c},C}$	caption–image pairs written by $V_{c}$ under $\pi$ on $C$ ; the audit target
$D_{c}$ , $D_{cx}$	caption marginal and caption–image joint
$D^{\star}$ (Ours)	the released corpus, the first audit target

Appendix

Table 11 : Notation and audit vocabulary.

Audit stage	Input	Requests	H200 GPU-hours
ClaimedCBU@64	text	$\approx 100$ k	$\approx 1$
GroundedCBU@64	text + image	$\approx 70$ k	tens
CBU-VQA	question + image	$\approx 100$ k	tens

Appendix

Table 12: Sampled VLM audit stages, request scales, and approximate compute orders. The two image-conditioned stages dominate; text-only extraction is one to two orders smaller.

Pool name	Target	Source / hosting
civitai_flux_prompts_aconexx	FLUX	https://huggingface.co/datasets/Aconexx/CivitAI-Flux-Prompts
flux_improved_k_mktr	FLUX	https://huggingface.co/datasets/k-mktr/improved-flux-prompts
flux_prompts_chrisgoringe	FLUX	https://huggingface.co/datasets/ChrisGoringe/flux_prompts
flux_prompts_regpeter	FLUX	https://huggingface.co/datasets/regpeter/flux_prompts
sd_prompts_2m_andyyang	Stable Diffusion	https://huggingface.co/datasets/andyyang/stable_diffusion_prompts_2m
sdxl_refiner_prompts_falah	SDXL refiner	https://huggingface.co/datasets/Falah/1M_SDXL_Refiner_Prompts

Appendix

Table 13: The seven prompt-reference pools per slice behind the prompt-pool support axis (Section 4.2 ). Each pool is a public Hugging Face community prompt set for a named T2I model family or the cross-model Pick-a-Pic preference rankings, and results are reported per pool.

Figure 3 : Per-pool prompt-support direction across paired comparisons. Each cell is the mean Ours-minus-Ref delta over $250{,}000$ caption pairs per comparison ( $114{,}621$ on CC12M–Qwen3-VL) at $B=64$ lexical units (Section 4.1 ); rows share a sort by Ours-minus-Ref prompt-mass mean. Left: $\Delta$ prompt-mass support (Ours $-$ Ref); positive means Ours covers more pool $n$ -gram mass. Right: direction-aligned JSD, plotted as $-\Delta\mathrm{JSD}=-(\mathrm{JSD}_{\mathrm{Ours}}-\mathrm{JSD}_{\mathrm{Ref}})$ , so positive means Ours is closer to the pool $n$ -gram distribution, and red marks the favorable direction in both panels. Color scale is symmetric on $[-0.10,+0.10]$ through a near-neutral zero band; measured deltas with $|\Delta|<0.01$ print as unsigned $0.00$ , and saturated cells with $|\Delta|>0.10$ print the full signed value in bold. Right margin per panel: per-pool wins / $7$ paired comparisons, plus the mean delta $\bar{\Delta}$ across those comparisons. Bottom margin per panel: per-slice wins / $7$ pools, plus mean $\bar{\Delta}$ . The vertical separator splits four cross-corpus paired comparisons (Section 5.1 ) from the three CC12M case-study references (Section 5.3 ).

Dataset	Surface	$B$ -elig $\uparrow$	Responses	Questions	Q/cap	Risk
DataComp	Ours	$1.00$	$4{,}771$	$68{,}933$	$14.45$	$0.035$
DataComp	Recap-DataComp ( 32 )	$0.26$	$4{,}914$	$51{,}356$	$10.45$	$0.177$
Danbooru	Ours	$1.00$	$4{,}991$	$71{,}491$	$14.32$	$0.058$
Danbooru	Danbooru-Florence ( 60 ; 29 )	$0.17$	$4{,}969$	$40{,}755$	$8.20$	$0.217$
LAION-pop	Ours	$1.00$	$4{,}962$	$73{,}489$	$14.81$	$0.031$
LAION-pop	LAION-pop-Llama ( 9 )	$1.00$	$4{,}946$	$58{,}903$	$11.91$	$0.077$

Appendix

Table 14 : VQA question denominators and

B

-eligibility behind Section 5.1 . Qwen Judge at

B=64

, on the requests answered by both Judges.

B

-elig = share of captions reaching

B

lexical units; the short captions of PD12M-released, Recap-DataComp, and Danbooru-Florence fill part of the window. Q/cap = average CBU per audited caption.

							Risk
Surface	Aligned	Valid	Cap	Zero	VQA	Q	Qwen	Gemma
Ours	$4{,}494$	$4{,}470$	$24$	$3$	$4{,}467$	$68{,}019$	$0.030$	$0.066$
CC12M-LLaVA-NeXT	$4{,}494$	$4{,}459$	$35$	$6$	$4{,}453$	$48{,}092$	$0.069$	$0.097$
PixelProse	$4{,}494$	$4{,}457$	$37$	$8$	$4{,}449$	$56{,}043$	$0.129$	$0.161$
CC12M-Qwen3-VL †	$4{,}494$	$4{,}491$	$3$	$1$	$4{,}490$	$28{,}911$	$0.014$	$0.028$

Appendix

Table 15 : CC12M denominators at

B=64

. Aligned: images shared by all four surfaces. Valid: parsable extractions. Cap: extractions that reached the

4{,}096

-token output cap. Zero: valid extractions with no claim. VQA: audited captions; Q: CBU questions, identical for both Judges.

Source	Pair	Surface	Opener	Top-100 raw	Top-100 content	Distinct-3
DataComp	Recap-DataComp	Ref	$0.36\%$	$27.99\%$	$17.53\%$	$11.49\%$
		Ours	$2.00\%$	$7.69\%$	$5.31\%$	$19.26\%$
CC12M	CC12M-LLaVA-NeXT	Ref	$0.24\%$	$22.97\%$	$12.39\%$	$13.81\%$
		Ours	$1.87\%$	$10.57\%$	$6.83\%$	$18.05\%$
	PixelProse	Ref	$100.00\%$	$65.07\%$	$8.25\%$	$12.56\%$
		Ours	$1.86\%$	$10.51\%$	$6.79\%$	$19.54\%$

Appendix

Table 16 : Surface descriptors by source family (companion to Table 3 ). Ref = the comparison-surface release named in the Pair column.

		Sup. CBU/cap $\,\uparrow$
Dataset	Judge	all types	six types	$\bm{\Delta}$	Risk, six types $\,\downarrow$
DataComp	Qwen	$8.49\to\mathbf{13.73}$	$6.83\to\mathbf{11.05}$	$+4.22$	$0.165\to\mathbf{0.032}$
	Gemma	$8.00\to\mathbf{12.94}$	$6.47\to\mathbf{10.46}$	$+3.99$	$0.203\to\mathbf{0.073}$
LAION-pop	Qwen	$10.80\to\mathbf{14.22}$	$8.87\to\mathbf{11.57}$	$+2.70$	$0.064\to\mathbf{0.027}$
	Gemma	$10.22\to\mathbf{13.61}$	$8.43\to\mathbf{11.10}$	$+2.66$	$0.095\to\mathbf{0.053}$
PD12M	Qwen	$8.61\to\mathbf{14.29}$	$6.37\to\mathbf{11.36}$	$+4.98$	$0.089\to\mathbf{0.029}$

Appendix

Table 17 : Cross-corpus results on six claim types (

B=64

). The six-type subset leaves out count and relation. Cells

a\to b

report Ref

\to

Ours (bold = Ours);

\Delta

is the supported-yield gain on the subset.

	Qwen Judge				Gemma Judge
	Ours		Refs		Ours		Refs
Type	Sup.	Risk	Sup.	Risk	Sup.	Risk	Sup.	Risk	Agree.
object	$0.931$	$0.029$	$0.914$	$0.072$	$0.888$	$0.053$	$0.881$	$0.098$	$94.4\%$
attribute	$0.947$	$0.036$	$0.849$	$0.131$	$0.890$	$0.073$	$0.801$	$0.161$	$91.3\%$
relation	$0.952$	$0.042$	$0.855$	$0.137$	$0.898$	$0.091$	$0.802$	$0.186$	$91.2\%$
count	$0.889$	$0.106$	$0.734$	$0.251$	$0.809$	$0.178$	$0.679$	$0.299$	$85.1\%$

Appendix

Table 18 : Image support and risk by claim type (

B=64

). Rates are shares of CBU questions answered yes (Sup.) or no (Risk), pooled over the four cross-corpus pairs and the CC12M comparison; Refs pools every reference surface. Agree. is exact Qwen–Gemma answer agreement on the identical CC12M question set.

		Qwen Judge		Gemma Judge
Dataset	Surface	Sup CBU/cap $\uparrow$	Risk $\downarrow$	Sup CBU/cap $\uparrow$	Risk $\downarrow$
CC12M	Ours	$14.60\pm 0.06$	$0.030\pm 0.001$	$13.82\pm 0.06$	$0.066\pm 0.001$
	CC12M-LLaVA-NeXT	$9.84\pm 0.05$	$0.069\pm 0.002$	$9.44\pm 0.05$	$0.097\pm 0.002$
	PixelProse	$10.73\pm 0.05$	$0.129\pm 0.002$	$10.20\pm 0.05$	$0.161\pm 0.002$
	CC12M-Qwen3-VL †	$6.31\pm 0.03$	$0.014\pm 0.001$	$6.19\pm 0.03$	$0.028\pm 0.001$
DataComp	Reference	$8.49\pm 0.05$	$0.177\pm 0.003$	$8.00\pm 0.05$	$0.219\pm 0.003$

Appendix

Table 19 : VQA cells at

B=64

as mean

\pm

bootstrap standard deviation (

2{,}000

caption-level resamples). Sup CBU/cap and Risk per surface under the Qwen Judge and the Gemma Judge; Risk is unsupported / claimed CBU. The control rows belong to Table 6 ; Naive (greedy) decodes the naive prompt at temperature

0

		Caption screen			Image support
Surface	$n$	Licensed	Atomic	Type	Yes	Uncertain	No	Other
Ours	$111$	$95.5$	$86.5$	$86.5$	$\mathbf{78.4}$	$19.8$	$\mathbf{0.0}$	$1.8$
LLaVA-NeXT	$47$	$93.6$	$91.5$	$91.5$	$70.2$	$21.3$	$4.3$	$4.3$
Qwen3-VL-8B	$20$	$95.0$	$90.0$	$90.0$	$70.0$	$25.0$	$5.0$	$0.0$
PixelProse	$39$	$94.9$	$84.6$	$84.6$	$59.0$	$23.1$	$15.4$	$2.6$

Appendix

Table 20 : Complete human-verification census by CC12M surface.

n

counts primary non-repeat judgments and the remaining cells are unweighted percentages; each surface has the same

n

in the caption and image phases. “Other” combines not visual and prefer not to answer . The four image columns partition the phase denominator.

Claim type	Ours	LLaVA-NeXT	Qwen3-VL-8B	PixelProse
Attribute	$8/12\;(66.7\%)$	$4/8\;(50.0\%)$	–	$4/6\;(66.7\%)$
Camera	$16/16\;(100\%)$	$11/14\;(78.6\%)$	$1/1\;(100\%)$	$2/2\;(100\%)$
Count	$10/12\;(83.3\%)$	$1/1\;(100\%)$	$2/2\;(100\%)$	$5/10\;(50.0\%)$
Lighting	$6/13\;(46.2\%)$	$4/6\;(66.7\%)$	$5/6\;(83.3\%)$	–
Object	$12/15\;(80.0\%)$	$1/2\;(50.0\%)$	$3/5\;(60.0\%)$	$3/6\;(50.0\%)$
Relation	$11/14\;(78.6\%)$	$2/2\;(100\%)$	$1/3\;(33.3\%)$	$5/7\;(71.4\%)$

Appendix

Table 21 : Human image-support “yes” judgments by claim type. Cells are count / primary judgments (percentage) for the full eight-type grid. Denominators follow the sampled claims, and a dash marks a surface–type cell with no sampled claim.

Panel of Figure 4 — Figure 4 : Annotation interface. Annotators first judge whether the fixed caption window licenses the atomic claim and its proposed semantic type, then judge the same claim against the image with the caption hidden. Surface identity, automated judgments, and other annotators’ responses stay hidden throughout; the captures use a synthetic practice item.

				Qwen Judge		Gemma Judge
Dataset	Surface	VQA	Q	Sup CBU/cap $\uparrow$	Risk $\downarrow$	Sup CBU/cap $\uparrow$	Risk $\downarrow$
CC12M	Ours	$4{,}467$	$68{,}019$	$14.60$	$0.030$	$13.82$	$0.066$
	Naive	$4{,}397$	$50{,}450$	$11.03$	$0.022$	$10.61$	$0.045$
	Naive (greedy)	$4{,}423$	$51{,}365$	$11.10$	$0.022$	$10.79$	$0.043$
DataComp	Ours	$4{,}688$	$68{,}473$	$13.90$	$0.036$	$13.07$	$0.080$
	Naive	$4{,}618$	$50{,}853$	$10.56$	$0.026$	$10.07$	$0.057$

Appendix

Table 22 : Captioner-control denominators at

B=64

. VQA: audited captions; Q: CBU questions, identical for both Judges. Naive (greedy) decodes the naive prompt at temperature

0

	Vendi $\,\uparrow$		eRank $\,\uparrow$		Coverage@10 $\,\uparrow$		Density@10 $\,\uparrow$		kNN cos $\,\sim$
Encoder	Ours	Ref	Ours	Ref	Ours	Ref	Ours	Ref	$\bm{\rightarrow}$	$\bm{\leftarrow}$
Qwen3-Emb-4B	$61.40$	$\mathbf{62.57}$	$\mathbf{337.2}$	$319.5$	$\mathbf{0.743}$	$0.357$	$\mathbf{0.218}$	$0.066$	$0.758$	$0.760$
Qwen3-Emb-8B	$76.33$	$\mathbf{89.25}$	$\mathbf{424.0}$	$378.5$	–	–	–	–	$0.721$	$0.721$
BGE-M3	$25.39$	$\mathbf{34.21}$	$\mathbf{255.1}$	$235.5$	$0.017$	$\mathbf{0.050}$	$0.002$	$\mathbf{0.007}$	$0.776$	$0.776$

Appendix

Table 23 : DataComp text-space rankings vary by encoder and metric. 50k paired rows. Vendi ( 15 ) = exponential of the eigenvalue entropy of the caption-set kernel; eRank ( 45 ) = significant directions in the centred embedding covariance. Prompt-to-caption PRDC ( 37 ) uses DiffusionDB ( 58 ) prompts at

k=10

. kNN cos

=

caption-to-caption directional cosine at

k=10

, separately for Ours-as-query (

\rightarrow

) and Ref-as-query (

\leftarrow

	LongCLIP tokens / cap			I2T $\,\uparrow$		T2I $\,\uparrow$
Surface	mean	p95	trunc.	R@1	R@5	R@1	R@5
Ours	$231.7$	$320$	$32.4\%$	$\mathbf{0.908}$	$0.988$	$\mathbf{0.902}$	$\mathbf{0.986}$
CC12M-LLaVA-NeXT ( 8 )	$81.9$	$153$	$0.4\%$	$0.907$	$\mathbf{0.989}$	$0.899$	$\mathbf{0.986}$
PixelProse ( 49 )	$108.4$	$193$	$1.7\%$	$0.890$	$0.977$	$0.874$	$0.976$
CC12M-Qwen3-VL † ( 53 )	$15.4$	$20$	$0\%$	$0.861$	$0.979$	$0.842$	$0.971$
Naive Qwen3.5-35B-A3B	$424.5$	$522$	$99.2\%$	$0.906$	$0.984$	$0.882$	$0.977$

Appendix

Table 24 : LongCLIP retrieval on CC12M (full-caption mode). LongCLIP-GmP-ViT-L-14 ( 65 ) ,

4{,}494

-image subset, retrieval pool

4{,}494

; trunc. is the caption rate exceeding the

248

-token cap. Pre-truncated input64 mode in Table 25 ; dagger † marks short tag-style surfaces (Section 3.2 ).

	LongCLIP tokens / cap			I2T $\,\uparrow$		T2I $\,\uparrow$
Surface	mean	p95	trunc.	R@1	R@5	R@1	R@5
Ours	$81.8$	$90$	$0\%$	$0.859$	$0.971$	$0.850$	$0.965$
CC12M-LLaVA-NeXT	$60.6$	$83$	$0\%$	$\mathbf{0.897}$	$\mathbf{0.985}$	$\mathbf{0.888}$	$\mathbf{0.983}$
PixelProse	$73.4$	$82$	$0.07\%$	$0.869$	$0.968$	$0.856$	$0.963$
CC12M-Qwen3-VL †	$15.4$	$20$	$0\%$	$0.861$	$0.979$	$0.842$	$0.971$
Naive Qwen3.5-35B-A3B	$82.6$	$91$	$0\%$	$0.812$	$0.948$	$0.789$	$0.934$

Appendix

Table 25 : LongCLIP retrieval on CC12M (input64 mode): captions pre-truncated to

B=64

lexical units before LongCLIP encoding. Same

4{,}494

-image subset and column structure as Table 24 .

Source	Surface	Mean tokens	CLIP-77 trunc	LongCLIP-248 trunc	SigLIP2-64 trunc
DataComp	Ours	$223$	$99.98\%$	$25.41\%$	$99.99\%$
	Recap-DataComp	$62$	$25.21\%$	$0.01\%$	$41.69\%$
CC12M	Ours	$231$	$99.97\%$	$31.24\%$	$99.98\%$
	Naive Qwen3.5-35B-A3B	$425$	$100.00\%$	$99.24\%$	$100.00\%$
	CC12M-LLaVA-NeXT	$88$	$57.61\%$	$0.49\%$	$64.09\%$
	PixelProse	$108$	—	$2.23\%$	—

Appendix

Table 26 : Encoder-token truncation rate on three image-text contrastive alignment encoders — CLIP-77 ( 41 ) , LongCLIP-248 ( 65 ) , SigLIP2-64 ( 52 ) — measured on 100k-row samples per surface (

4{,}494

captions for the naive control). Numbers are the fraction of captions exceeding the encoder’s token cap; mean tokens count CLIP tokens.

Group	Fields	Role
Source keys	source_dataset , source_url , source_url_norm , source_url_sha1 , source_sha256	source join
Image locator	asset_instance_id , image_shard , image_member	image binding
Caption	caption_text , caption_sha256 , caption_record_id	released text
Generation	caption_model_family , caption_model_artifact , prompt_tokens , completion_tokens	provenance

Appendix

Table 27 : Caption record fields of the public release.

Result	Scope	Artifact
Claim yield (Table 5 )	cross-corpus pairs	all_cbu_b64_summary.csv
Support and risk (Tables 5 , 8 , Fig. 2 )	both Judges	cbu_vqa_by_category_b64.json
Length and prefix statistics (Tables 3 , 5 )	caption text	cpu_text_metrics/
Pool comparisons (Figs. 1 , 3 )	seven prompt pools	prompt_support_bootstrap_b64_n2_250k_2026-04-24.tsv
Claim yield and budget sweep (Table 8 , Fig. 2 )	CC12M	cc12m_budget_frontier_plot.csv
Policy control (Tables 6 , 7 )	CC12M, DataComp	naive_qwen35_*/

Appendix

Table 28: Result provenance. Claim and VQA summaries use

B=64

, and both Judges supply the VQA answers.

Stage	Count	Share
Target URLs	$940{,}890{,}257$	—
Processed	$371{,}813{,}492$	$39.5\%$
Download-stage survival	$282{,}860{,}383$	$76.1\%$
Usable images	$147{,}035{,}107$	$39.6\%$
Top failure families:
image_too_small	$129{,}765{,}982$	$34.9\%$

Appendix

Table 29: DataComp forward-partition crawl-and-survival snapshot at 2026-04-03T15:03:36Z cutoff. Crawling continued past the cutoff, and the released totals exceed this snapshot; the processed share is taken over target URLs and every other share over processed URLs. The dominant post-download loss is image_too_small , driven by the

\mathrm{min\_short\_edge}=256

gate.

Explore similar work

Aug 3, 2026cs.CV

CAPEval: A Decoupled Caption Evaluation across Understanding and Generation

Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. However, previous evaluations treat the caption quality as a single scalar objective, which conflates two distinct properties: (1) how much visual information a caption covers and (2) how reliably the image supports its stated claims. To this end, we design a decoupled caption evaluation benchmark, CAPEval (Coverage And Precision Evaluation), with human-written ground-truth captions and human-verified atomic checklist items. Specifically, CAPEval decomposes caption quality into Coverage and Precision. The former quantifies how thoroughly a caption covers ground-truth factual content, while the latter reflects the factual correctness rate of all claims expressed in the caption. We select 10 captioners and further conduct controlled downstream end-to-end experiments with them from four model families, where the caption source is the only variable. Empirically, we find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance. This decoupled evaluation paradigm not only delivers a more fine-grained diagnosis of caption quality, but also offers actionable guidance for selecting and optimizing captioners tailored to different downstream tasks.

Zhipeng Liu, Haochen Wang, Zhaoxiang Zhang

University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences

Captions Multimodal Understanding

Ask PDF

Aug 11, 2026cs.CV

CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering

Evaluating detailed image captions from Vision-Language Models (VLMs) requires going beyond surface-level semantic similarity. Reference-based metrics (e.g., CIDEr and SPICE) and LLM-as-scorer protocols struggle to verify dense factual claims, while existing QA-based alternatives generally offer lower probe density, narrower domain coverage, or no explicit alignment between individual questions and segmented image regions. We introduce CapProbe, a full-scene dense QA benchmark that turns detailed caption evaluation into region-aligned factual checking. Each image is decomposed into coarse semantic regions covering both foreground and background elements; for every retained region, we generate multiple-choice questions spanning 10 semantic categories, forming a dense checklist of probed visual facts. Guided by a two-tier taxonomy of 37 L1 domains and 219 L2 sub-domains, CapProbe comprises 346 images, 1,868 regions, and 25,650 questions, averaging 74 QA pairs per image. A language judge answers from the caption alone; an Uncertain option and Effective Accuracy provide a judge-dependent proxy for distinguishing unanswered probes from incorrectly resolved ones, while density-based metrics penalize verbose yet uninformative captions. The protocol is cost-effective: by converting unconstrained scalar scoring into structured MCQ reading, it reduces open-ended scoring bias while remaining judge-conditioned and yields relatively stable model rankings under a fixed reader. Experiments on 13 VLMs show large Coverage gaps across models, a clear competency-efficiency trade-off, and failure modes that sparse or overlap-based evaluation often misses. The benchmark data, annotations, and evaluation code will be released soon.

Mouxiao Huang, Qiangyu Yan, Borui Jiang +1

Huawei Technologies

Visual Question Answering Benchmarks Captions Knowledge-Based Visual Question Answering Llm-As-A-Judge Question Fact-Checking

Ask PDF

Jul 25, 2026cs.CV

A Reconstruction-Based Framework for Caption Evaluation Beyond Reference Captions

Image captioning is a primary task in vision--language research, yet assessing how faithfully a caption preserves image semantics without relying on reference captions remains unsettled. Prevailing evaluations rely on human-annotated references, whose content reflects annotator intent and captioning proficiency. In this paper, we study a reconstruction-based principle for caption evaluation: a caption is as good as its capacity to enable reconstruction of the original image. However, because captioning inherently compresses visual information, it is impossible to recover all details, and pixel-wise comparison between reconstructed and source images is neither feasible nor meaningful. Through our in-depth analysis of the nature of captions, whose fundamental purpose is to transmit the semantic content of an image, we propose a revised principle: a caption is as good as its capacity to enable a reconstruction that is semantically equivalent to the original. To assess semantic equivalence, we test whether the reconstruction matches the original image across a suite of downstream vision--language tasks, yielding a reference-free, task-conditioned caption score. We characterize component-dependent limitations and introduce the lower-cost Captioning Turing Test Dataset (CTTD) surrogate.

Zhijiang Tang, Jiaxin Qi, Kaihua Tang +2

2Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, China · 1Computer Network Information Center, Chinese Academy of Sciences, China · 3Tongji University, China +1

Image Difference Captioning

Ask PDF