Multimodal Judges Flag, Humans Decide: A Review-and-Edit Framework for AI-Generated Product Video Ads
Organizations: Anoki Inc.
Abstract
Generative pipelines can now turn every product page of an e-commerce catalog into a short video ad, but deciding whether such an ad is usable is not one good-or-bad call. We describe how we operationalized the review and editing of these ads so that both can keep pace with generation: reviewers go through each ad scene by scene and answer six questions, on the product, scene quality, pronunciation, script, sound and captions; judges built from vision-language, speech and language models file each finding under the matching question, and reviewers accept or reject each finding and add what the judges missed. Each confirmed issue then opens an edit tool matched to it. The questions and the routing from a finding to its fix do not depend on the models that generate the ad. Across 120 ads from our pipeline, reviewed with judge findings on screen, reviewers marked the product wrong on 60% of scenes, and script, caption or sound problems on under 10% each. On 60 of these ads, they marked the product wrong on 58% of scenes, far more than the 26% marked by two reviewers who later saw the ads with the judges hidden; one reviewer who saw the ads both ways marked 57% with judge findings and 19% without. On 120 blind-labeled scenes, a product judge that watches the clip with the catalog photos reached AUC 0.84, while video-quality and photo-similarity scores were close to chance. We tune the judges for recall: the product judge catches nearly all defects but also flags many clean scenes, so reviewers decide; together, the judges widen what reviewers mark and reviewers add what the judges miss. Without judge findings, one reviewer's marking faded over a session. We release a review-portal template and our review data (120 ads and 562 scenes with judge findings and human marks, plus 120 blind-labeled scenes with baseline scores) at https://github.com/anoubhav/ad-review-and-edit.
Figures & tables
| Axis | Unit | Answer and secondary options | Judge: what it sees |
|---|---|---|---|
| Product | scene | right / wrong; wrong product; shape, color, logo or markings wrong; AI issue on the product; product text not intact (garbled; low: readable, high: gibberish) | a vision-language model watches the clip with all catalog photos and an AI-made product sheet, citing time, place and photo |
| Scene quality | scene | everything in the scene but the product: keep / swap / no opinion; AI issue; repetitive, dull or crowded; world or continuity breaks; impossible motion; weak ending | two models: one watches the clips in order, the other sees four frames per scene |
| Pronunciation | line | sounds right / mispronounced, with the word | a speech-to-text model transcribes the line’s narration, keeping mispronunciations; a text model checks each mismatch against the script |
| Script | line | good / bad; unsupported claim; awkward phrasing | a language model checks each claim against the listing and listing-photo text, quoting the words |
| Sound | ad | right / issues: the voice, the music | one audio-model verdict on the whole mix, with no scene or time |
| Captions | caption | placement: correct / can be better; relevance: correct / issues | judges watch the finished ad (run after the review round, Sec. 4 ); at generation a model places captions off faces and logos, and clutter and contrast are measured |
| Blind labels | Round | |||
| Judge | Items (defective) | Recall | Precision | Accepted |
| Product | 120 scenes (29) | 26 of 29 | 26 of 56 | 75% |
| Pronunciation | 208 lines (43) | 28 of 43 | 28 of 90 | – |
| Caption placement | 530 captions (39) | 15 of 39 | 15 of 130 | – |
| Caption relevance | 530 captions (11) | 4 of 11 | 4 of 51 | – |
| Scene quality | no blind labels | 96% | ||
| Axis | Problem | Fix | Model |
|---|---|---|---|
| Product | garbled text | Repair the scene in place at higher resolution | generative upscaler (video-to-video) [ 61 , 66 , 60 ] |
| Product, scene quality | any other product or scene-quality problem ( Tab. 1 ) | Borrow an unflagged scene from another ad | none; ranked by fit, judge-filtered |
| Generate a new scene from the same references | reference-to-video [ 58 , 26 , 37 ] | ||
| Edit the clip | video-to-video editing [ 28 , 30 , 26 ] | ||
| Pronunciation | mispronounced word | Respell the word and re-voice only its line | text-to-speech [ 45 , 8 , 13 ] |
| Script | unsupported claim | Correct the claim’s words to match the listing, or regenerate the line | language model; text-to-speech |
| Axis | Question and answer | Evidence | |
|---|---|---|---|
| Product | Judge the clip or its starting frame? The clip: fewer false flags (prompts differ). | B | 120 scenes: both catch most defects (26 and 24 of 29), but the clip judge flags fewer clean scenes (30 against 53 of 91); precision 0.46 against 0.31; 0.43 [0.27, 0.60] against 0.16 [0.04, 0.30], paired gap [0.10, 0.43] |
| Do generic scores work? No: near chance. | B | AUC: generic scores 0.48 to 0.57, open VLMs at most 0.73, ours 0.84 ( Fig. 7 ) | |
| Scene quality | One judge or two? Two: the pair finds more. | A | 489 reviewer-scene pairs: either judge flags 69% of the scenes reviewers wanted swapped, the better one alone 55%, at precision 0.84 against 0.89 |
| Script | Read text on listing photos? Yes: factual calls improve. | A | 200 ratings on 155 ads, raters saw both judge versions’ calls: the call is right in 94% of cases when the photos are read, 50% when not |
| Captions | Can a model judge captions? Not reliably: its flags are stable across runs but barely match reviewers, so we measure placement. | B | 530 captions, against the reviewers’ majority: 0.07 for placement (between reviewers 0.11) and 0.10 for relevance; self-agreement 0.70 and 0.80 across runs. A best-of-five placement pick ( M ) matched itself on 55 of 100 captions |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Problem | What the review saw | Fix | What is wrong, and who caught it |
|---|---|---|---|
| Product | |||
| Wrong product, replaced by a still | catalog photo before, 0:03 | after, 0:04 Replaced by an animated still | Eyeliner ad from a production run. The applicator is drawn as a kinked, bristly brush, where the real one has a smooth precision stem (0:02-0:05); judge flagged, 1 of 1 reviewer marked. After: the scene was replaced by an animated still of the product photo (a slow pan, no generated motion), which shows a straight stem; not judged. |
| Shape wrong | catalog photo (crop) frame 0:08 | Borrow, generate or edit the scene | Beard trimmer. The ridged length-adjustment wheel is replaced by an invented see-through battery chamber (0:07-0:11). Judge flagged, 3 of 3 reviewers marked. |
| Color wrong | catalog photo frame 0:00 | Borrow, generate or edit the scene | Smartwatch. The strap buckle is bright silver, where the real watch’s is dark blue (0:00-0:01); judge flagged, 1 of 1 reviewer marked. |
| Text slightly garbled (low), Repair at higher resolution | catalog photo before, 0:05 (crop) | after, Repair (crop) Repair the scene in place | Inkjet printer. Before: the model name and the small print under it, PRINT FAX SCAN COPY WEB in the catalog photo, are smeared but readable (0:05); judge flagged, 1 of 1 reviewer marked. After: Repair re-rendered the scene from its original inputs at higher resolution, and both lines read correctly; this pair is from our repair study and is judged here by eye only. |
| Text gibberish, color drift | catalog photo frame 0:04 (white body; number blurred) frame 0:07, service badge (crop; number blurred) | Borrow, generate or edit the scene | Water purifier. The service badge turns into non-words with a wrong phone number, and the body starts white before turning black like the real unit (0:04-0:09). Judge flagged both, 3 of 3 reviewers marked both. |
| Axis | Unit | Units | Alone | Judge | Alone’s marks flagged | Flags marked alone | Round any | Round maj. | alone vs maj. |
| R1 (60 ads) | |||||||||
| Product | scene | 280 | 33 [25, 41] | 68 | 74 [61, 85] | 36 [27, 44] | 67 | 58 | 0.11 [ 0.02, 0.22] |
| Scene quality † | scene | 280 (229) | 47 [38, 55] | 25 | 27 [20, 33] | 49 [37, 61] | 37 | 33 | 0.05 [ 0.07, 0.17] |
| Script | line | 280 | 3 [1, 4] | 3 | 0 of 7 | 0 of 7 | 15 | 5 | 0.03 [ 0.05, 0.02] |
| Caption placement | caption | 255 | 13 [8, 19] | no judge | 17 | 5 | 0.06 [ 0.06, 0.21] | ||
| Caption relevance | caption | 255 | 5 [2, 8] | no judge | 6 | 2 | 0.03 [ 0.05, 0.01] | ||
| Score | Input | AUC | 95% interval | Gap to clip judge |
|---|---|---|---|---|
| Clip judge (deployed) | clip, catalog photos | 0.84 | [0.73, 0.94] | |
| Starting-frame judge | starting frame, photos | 0.69 | [0.61, 0.78] | +0.15 [0.01, 0.28] |
| Clip judge on eight frames | 8 frames, photos | 0.80 | [0.66, 0.92] | +0.04 [ 0.05, 0.15] |
| DINOv2 similarity | 3 frames, photos | 0.51 | [0.39, 0.65] | +0.34 [0.18, 0.48] |
| CLIP similarity | 3 frames, photos | 0.57 | [0.41, 0.72] | +0.27 [0.09, 0.47] |
| OCR text mismatch (EasyOCR) | 3 frames, photos | 0.57 | [0.49, 0.64] | +0.28 [0.14, 0.42] |
| Axis and question | Labels | Answer and key numbers | |
|---|---|---|---|
| Pronunciation. Which speech-to-text model; recheck on audio? | L | 208 lines: audio only (blind), then with script and judge output | Error-keeping, text-only recheck. On the labels made after output was shown (54 defect lines, against 43 on the blind pass of Tab. 2 ), Gemini 3.5 Transcribe [ 44 ] and ElevenLabs Scribe v2 [ 14 ] returned the script word for word on 30% and 46% of defect lines, against 6% for ours, Azure Speech [ 46 ] ; MAI-Transcribe-2 [ 47 ] also caught fewer defects than ours. Audio in the recheck cut recall from 44 to 38 of 54 |
| Sound. Which text-to-speech voices to allow? | B | 107 voices on an earlier engine, three ad reads each; four listeners | 65 of 107, excluding voices with poor recording quality, mispronunciations, unnatural intonation or an accent that did not fit the target market. Each voice was heard by at least two listeners, and the 44 split voices went to a tiebreak with earlier votes hidden; two voices users had swapped away from in production were among the 42 excluded, and the three most-cast voices were kept. The engine that narrated the round was admitted with all 30 of its voices |
| Label set | Size | Labelers | Blind? | Used for |
|---|---|---|---|---|
| Annotator product labels on production clips | 120 decided scenes of 146 labeled, 29 defective | four annotator accounts; of the 120 decided scenes, 105 have two votes, ten have three or four, five have one; they miss subtle defects | yes | product judge and baselines ( Tab. 4 , Tab. S3 ) |
| Earlier review round | 66 accepted scene findings | reviewer thumbs | assisted | what scene judges flag |
| Scene judges, 120-ad round | 489 reviewer-scene pairs | portal reviewers | assisted | one scene judge or two ( Tab. 4 ) |
| Pronunciation corpus | 208 narrated lines, earlier narration engine | five listeners, one per line, ten lines heard by all | first pass yes, second pass after reveal | recall and precision on the first pass ( Tab. 2 ); speech-to-text model and recheck on the second ( Tab. S4 ) |
| Fresh pronunciation sample | 100 production lines (70 flagged, 20 near misses, 10 passed) | one listener | yes | precision on the 70 flagged lines: 0.21 [0.12, 0.31] |
| Talking-over clips | 50 of 128 clips | stem-separation loudness | no human labels | sound ( Tab. S4 ) |
| Question | Answers | A problem answer points to | Secondary options |
|---|---|---|---|
| The product | Product looks right; Product wrong | every scene where it happens | Wrong product; Shape, color, logo or markings wrong; AI issue; Product text not intact, graded Low (“still readable”) or High (“gibberish”); a comment per scene |
| Scene quality | No opinion; Keep; Swap out | each rateable scene, which gets its own answer | AI issue; Repetitive, dull or crowded scene; World or continuity breaks; Something moves impossibly; Weak ending (last scene only); free-text boxes for another reason and for why |
| Pronunciation | Sounds right; Mispronounced | every spoken line where it happens | none; an optional box for the word |
| Sound quality | Sounds right; Issues | the whole ad | “Which part?”, at least one of The voice and The music; an optional note |
| Script quality | Good; Bad | every spoken line where it happens | Factually incorrect (an unsupported claim); Awkward or unnatural phrasing; an optional note |
| Overlay text: placement | Correct; Can be better | every captioned scene where it happens | none |
Human-in-the-Loop Evaluation
Human-in-the-Loop Evaluation studies how human judgment and feedback can assess, guide, and monitor AI systems in realistic settings.
See all 171 papers