cs.CVNov 3, 2026
SaveWhat Does the Model See? Representation Validity in Multimodal LLM-Based Video Analysis
Organizations: Department of Communication and Media Research, University of Zurich, Zurich, Switzerland.
Abstract
Multimodal large language models (MLLMs) allow social scientists to analyze video at scale, but researchers must decide which parts of a video to provide as input. We use the term representation validity to examine how text, audio, video, and combined inputs affect measurement and subsequent analysis. We compare input representations across six open-weight MLLMs and three human-annotated video datasets. No representation consistently performs best. On seven news-related constructs, native-video performance varies little across models (macro-F1 = .70-.77), while extracted-text performance ranges from .52 to .80. Three models perform better with text and three with video. Combining inputs rarely improves performance. Higher classification accuracy also does not reliably produce estimates closer to those based on human codes. The highest-F1 representation best reproduces human-coded regression coefficients in 17 of 42 comparisons. Even when each model uses its highest-F1 deployable representation, 45% of sample-prevalence estimates differ from the human-coded benchmark by more than ten percentage points. Input representation therefore requires validation for both the construct being measured and the quantity estimated from the resulting labels.
Figures & tables
Figure 1: Representation as part of measurement. We apply a representation procedure to the original object , producing the input that the model observes. We then use the model’s codes in substantive analysis. Our four validation checks address different steps of this process.
| News on TikTok Mayer et al. (2025) | HateMM Das et al. (2023) | MUStARD Castro et al. (2019) | |
| Language and domain | German; professional news from 18 outlets in Germany, Austria, and Switzerland | English; user-generated videos on BitChute | English; TV sitcoms |
| Test sample | 603 videos | 400 videos | 650 clips |
| Reference labels | Trained coders; published codebook. Final two-coder pretest: , – across the constructs used here | Two annotators plus expert adjudication; | Human annotations; and in two annotation phases. Human transcripts available |
| Constructs | Seven verbally defined constructs (news values and platform features); two audiovisual anchors | Hate; targets Black people; targets Jewish people | Sarcasm in the target utterance |
| Relevant channels | Channels largely repeat the same information; the anchors require picture or sound | Speech, text, symbols, and imagery can convey hate and identify its target | Delivery and context can convey sarcasm |
Table 1: Validation datasets.
| Construct | Speech | On-screen text | Visual | Audio | Basis in the definition |
| News on TikTok, verbally defined | |||||
| Geographic scope (V12) | ❍ | ❍ | ❍ | – | location or topic of the event compared with the outlet’s nationality (in metadata); explicit or implicit cues |
| Negative news (V15) | ❍ | ❍ | ❍ | – | topic of the story; “code the topic, not the tonality” |
| Conflict (V16) | ❍ | ❍ | ❍ | – | focus on controversy, strikes, fights, warfare |
| Scope (V17) | ❍ | ❍ | – | – | story “describes” how many people are involved or affected |
| Call to interact, TikTok (V9b) | ❍ | ❍ | ❍ | – | “visual- or audio-based prompts”; can be “purely visual or via an animation” |
Table 2: Information requirements by construct, classified from the coding definitions. ⚫ required; ❍ potentially informative; – not required. “Audio” refers to non-verbal sound and to properties of the voice (who speaks, tone), as distinct from the words spoken.
| Route | What the model receives | Notes |
| Text routes: we convert the video into text; the model reads only text | ||
| Metadata only | caption, hashtags, outlet (news); show, speaker, preceding dialogue (sarcasm) | floor; the metadata are also part of every other route |
| Transcript | + ASR text | extraction run once, reused by all models |
| Extracted text | + ASR text + OCR text | sarcasm: identical to transcript (no on-screen text) |
| Human transcript | + human transcript | sarcasm only; isolates ASR error |
| Describe first | + a description of the video written by the same model | first call: the model watches frames + audio and describes speech, on-screen text, visuals, and sound; second call: coding from that text |
Table 3: Representation routes. ASR: automatic speech recognition (Whisper large-v3-turbo). OCR: optical character recognition at one frame per second, near-duplicate lines removed. The three video routes differ in who decides which pictures the model sees and in how pictures and sound are packaged.
Figure 2: Macro-F1 for every route, construct, and model. Rows: the nine news constructs (verbal constructs above the first white line, the two audiovisual anchors below), hate, the two HateMM target groups, and sarcasm. Orange frame: observed best route in the row. Bold: best or practically equivalent to it (90% interval of the paired difference within ). n/a: the original-file route failed the 95% completion threshold and no tested capped fallback met it. Empty: route not defined for the dataset (no metadata for HateMM; human transcripts only for sarcasm). For sarcasm, extracted text equals the transcript.
Figure 3: (a) Shortfall of each route from the observed best route of the same model and construct (78 combinations; full video 54, metadata only 60). Orange: best or practically equivalent (90% interval of the paired difference within ); light orange: inconclusive; grey: worse by more than .05. Numbers below: share of combinations in which the route is best, and in which it is best or practically equivalent. (b) Decomposition of the variation in construct-level macro-F1 across six models, six routes available for all constructs (transcript, extracted text, frames + audio, native video, text + video, describe first), and 13 constructs.
Figure 4: Audio ablation: macro-F1 of the frames + audio input with and without the audio track. News: random subsample of 150 videos (seven verbal constructs pooled; music). Hate: 400 videos (not run for the two target groups). Sarcasm: 650 clips. Numbers give the change when the audio is removed.
| Seven verbal constructs | Music: share coded as present | |||||
| Input (150 news videos) | Gemma 12B | Gemma E4B | Nemotron | Gemma 12B | Gemma E4B | Nemotron |
| Metadata only | .67 | .66 | .52 | 8% | 20% | 0% |
| Extracted text | .80 | .75 | .62 | 2% | 3% | 1% |
| Frames only (audio removed) | .74 | .74 | .64 | 1% | 0% | 1% |
| Frames + audio, audio before the instruction | .74 | .73 | .71 | 0% | 0% | 14% |
| Frames + audio, audio after the instruction | .75 | .75 | .71 | 7% | 3% | 14% |
Table 4: Gemma audio diagnosis on 150 news videos, with Nemotron as control. Placement “after the instruction” follows the Gemma 4 documentation (images before the text, audio after it); our main runs placed all media before the text. The direct music probe uses the first 30 seconds; 72% of the corresponding whole videos, not necessarily these clips, have a positive human music label.
Figure 5: Integration across datasets: macro-F1 of extracted text (sarcasm: transcript), frames + audio, both in one call (text + video), and an oracle that selects the correct single-route prediction whenever one is available. The oracle uses human reference labels and is a diagnostic benchmark.
Figure 6: Inferential validity in the news data ( ). (a) Coefficients from the engagement regression estimated with human codes (diamond, shaded 95% interval) and with each model’s codes from extracted text and native video. (b) For each coefficient separately, the rank correlation across pipelines between the construct’s macro-F1 and the absolute deviation of the coefficient from the human-code estimate. Filled: 18 pipelines (six models extracted text, frames + audio, native video); open: 36 pipelines (all six routes used in the regression). Negative values mean that more accurate codes recover the estimate better.
| Route | Estimates | Mean | Same sign | 95% CI covers | CI overlap |
| Extracted text | 48 | 0.36 | 69% | 79% | 100% |
| Frames + audio | 48 | 0.38 | 73% | 81% | 100% |
| Native video | 48 | 0.32 | 77% | 90% | 100% |
Table 5: Recovery of the human-code regression estimates by route (48 estimates per route: 8 coefficients 6 models). The human estimates’ 95% intervals are wide (half-widths .55 to .71), so interval overlap is uninformative here.
Figure 7: Prevalence estimated from automated codes minus the reference-coded prevalence in the sample, for every binary construct, model, and route (496 estimates). Last column: for each model and construct, the deployable route with the highest macro-F1 (66 estimates). Blue: news constructs; orange: hate, target groups, and sarcasm.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Construct | Class | Test sample (603) | News corpus (7,978) |
| Geographic scope (quota) | 0 | 33% | 14% |
| 1 | 34% | 56% | |
| 2 | 33% | 30% | |
| Negative news (quota) | 1 | 33% | 22% |
| Conflict (quota) | 1 | 33% | 14% |
| Scope (quota) | 1 | 67% | 81% |
Table 6: Class shares in the news test sample and in all news videos of the published corpus. Binary constructs: share coded 1. Quota: construct used in drawing the sample.
| Model | Checkpoint (revision) | Architecture |
| Gemma 4 12B | google/gemma-4-12B-it (707f0a3b8a) | dense |
| Gemma 4 E4B | google/gemma-4-E4B-it (ee0ef60236) | dense, edge-oriented |
| Qwen3-Omni | Qwen/Qwen3-Omni-30B-A3B-Instruct (26291f7938) | MoE, 3B active |
| Qwen2.5-Omni | Qwen/Qwen2.5-Omni-7B (ae9e169054) | dense |
| Nemotron-3 Nano Omni | nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 (e5e9932441) | hybrid Mamba–transformer MoE, 3B active |
| MiniCPM-o 4.5 | openbmb/MiniCPM-o-4_5 (503e754207) | dense |
Table 7: Model checkpoints. We load each model from the Hugging Face Hub at the listed revision and run it in bfloat16 with vLLM 0.30.0 on one NVIDIA A100 (80 GB).
| Component | Setting |
| Inference engine | vLLM 0.30.0 (Docker image vllm/vllm-openai:v0.30.0 ; version confirmed in the engine logs); PyTorch and transformers as shipped in the image (versions not logged) |
| Hardware, precision | one NVIDIA A100-SXM4-80GB; bfloat16 weights and activations |
| Engine settings | seed 0; prefix caching off; multimodal preprocessing cache off (on only for the capped full-video runs); at most 32 images, 4 audio clips, 1 video per prompt; context 32,768 tokens (65,536 for long media inputs) |
| Prompting | each model’s own chat template; reasoning disabled ( enable_thinking=False ); news constructs jointly; HateMM targets jointly in a separate call from hate; media placed before the instruction (Gemma diagnosis: also after) |
| Decoding | greedy (temperature 0); at most 120 output tokens for codes, 450 for descriptions |
| Structured output | JSON schema per corpus restricting each construct to its legal codes (vLLM structured outputs, xgrammar backend) |
Table 8: Runtime configuration of all model calls and of the extraction steps.
| Group | Model | Metadata | Transcript | Human transcr. | Extracted text | Frames + audio | Native video | Full video | Text + video | Describe first |
| News, 7 verbal | Gemma 4 12B | .69 | .79 | – | .80 | .75 | .75 | .74 | .75 | .80 |
| Gemma 4 E4B | .68 | .76 | – | .76 | .73 | .73 | .73 | .74 | .76 | |
| Qwen3-Omni | .68 | .76 | – | .76 | .71 | .70 | – | .71 | .70 | |
| Qwen2.5-Omni | .49 | .52 | – | .52 | .71 | .70 | – | .73 | .49 | |
| Nemotron-3 Nano Omni | .55 | .63 | – | .63 | .71 | .71 | .74 | .71 | .61 | |
| MiniCPM-o 4.5 | .61 | .70 | – | .66 | .76 | .77 | .78 | .77 | .67 |
Table 9: Macro-F1 by route, model, and construct group (mean of construct-level macro-F1). News, verbal: seven verbally defined constructs ( ); anchors: visual set-up and music; hate: ; sarcasm: . “–”: route not defined for the dataset or not run.
| Shortfall | Best or practically equivalent | Worse | ||||||
| Route | Cells | Best | Median | 10th–90th pct. | ||||
| Metadata only | 60 | 7% | .18 | .02–.35 | 8% | 12% | 23% | 73% |
| Transcript | 78 | 3% | .04 | .00–.31 | 9% | 32% | 60% | 35% |
| Human transcript | 6 | 0% | .06 | .01–.17 | 17% | 33% | 50% | 50% |
| Extracted text | 78 | 21% | .04 | .00–.31 | 27% | 37% | 59% | 35% |
| Frames + audio | 78 | 18% | .04 | .00–.12 | 24% | 35% | 73% | 23% |
Table 10: Route choice across 78 model construct combinations. Best: share in which the route has the highest macro-F1. Shortfall: macro-F1 below the best route of the same combination. Best or practically equivalent: the route is best, or the 90% paired-bootstrap interval of its difference from the best lies within the margin. Worse: the interval lies entirely below . Full video: 54 combinations; metadata only: 60; human transcript: sarcasm only. Descriptive: the comparator is the observed maximum of each combination.
Figure 8: Frames + audio minus extracted text (sarcasm: minus transcript) by construct and model, with 95% item-bootstrap intervals. Shaded band: .
| Construct | Gemma 12B | Gemma E4B | Qwen3-Omni | Qwen2.5-Omni | Nemotron | MiniCPM-o |
| Geographic scope | .07 | +.01 | .06 | +.03 | +.04 | +.21 |
| [ .10, .04] | [ .02, +.04] | [ .09, .02] | [ .03, +.09] | [ .01, +.08] | [+.17, +.25] | |
| Negative news | .01 | .03 | .01 | +.04 | +.05 | .04 |
| [ .03, +.02] | [ .05, .01] | [ .03, +.02] | [+.00, +.08] | [+.02, +.08] | [ .07, .01] | |
| Conflict | .08 | .04 | .09 | +.28 | +.02 | +.08 |
| [ .11, .04] | [ .06, .01] | [ .13, .06] | [+.23, +.32] | [ .02, +.06] | [+.04, +.12] |
Table 11: Native video minus extracted text (sarcasm: minus transcript), macro-F1, with 95% item-bootstrap intervals. Bold: interval excludes zero.
| Construct | Gemma 12B | Gemma E4B | Qwen3-Omni | Qwen2.5-Omni | Nemotron | MiniCPM-o |
| Geographic scope | .06 | .00 | .07 | +.08 | .01 | +.22 |
| [ .10, .03] | [ .04, +.03] | [ .11, .03] | [+.03, +.14] | [ .06, +.04] | [+.18, +.26] | |
| Negative news | .01 | .04 | .03 | +.04 | +.07 | .04 |
| [ .04, +.01] | [ .06, .02] | [ .06, .01] | [+.01, +.08] | [+.03, +.10] | [ .07, .01] | |
| Conflict | .11 | .05 | .12 | +.26 | +.07 | +.08 |
| [ .14, .07] | [ .08, .02] | [ .16, .08] | [+.21, +.31] | [+.03, +.11] | [+.04, +.12] |
Table 12: Frames + audio minus extracted text (sarcasm: minus transcript), macro-F1, with 95% item-bootstrap intervals. Bold: interval excludes zero. These are the values plotted in Figure 9 .
| News: music verbal mean | HateMM: Jewish Black target | |||
| Model | Frames + audio | Native video | Frames + audio | Native video |
| Gemma 4 12B | .02 [ .00, .05] | .04 [.01, .07] | .10 [.03, .17] | .11 [.05, .18] |
| Gemma 4 E4B | .02 [ .04, .01] | .01 [ .04, .02] | .12 [.05, .20] | .11 [.04, .18] |
| Qwen3-Omni | .37 [.33, .41] | .32 [.28, .36] | .04 [ .02, .10] | .02 [ .04, .08] |
| Qwen2.5-Omni | .02 [ .06, .02] | .06 [.02, .10] | .00 [ .07, .07] | .04 [ .03, .12] |
| Nemotron-3 Nano Omni | .08 [.05, .12] | .23 [.19, .27] | .07 [.00, .13] | .07 [ .00, .14] |
Table 13: Construct dependence on the same items: difference-in-differences of the video-minus-extracted-text contrast between two constructs coded on the same items, macro-F1, with 95% paired item-bootstrap intervals (1,000 resamples). News: music minus the mean of the seven verbal constructs (603 videos). HateMM: targets Jewish people minus targets Black people (400 videos). Bold: interval excludes zero.
| News, 7 verbal constructs | |||||
| Model | Frames + audio | Native video | Full video | 896 px frames a | Sarcasm: native / full |
| Gemma 4 12B | .75 | .75 | .74 | +.00 | .52 / .55 |
| Gemma 4 E4B | .73 | .73 | .73 | .00 | .59 / .57 |
| Qwen3-Omni | .71 | .70 | n/a b | +.00 | .60 / .63 |
| Qwen2.5-Omni | .71 | .70 | n/a b | .00 | .54 / .56 |
| Nemotron-3 Nano Omni | .71 | .71 | .74 | +.02 | .55 / .51 |
Table 14: Macro-F1 by form of video input. 896 px: difference to 448-pixel frames on a random subsample of 150 news videos. a Difference, not level. b The original file does not fit the model for most videos (context window or GPU memory).
| Seven verbal news constructs | Music | |||||
| Model | 16 frames + audio | no audio | 4 frames | 32 frames | with audio | no audio |
| Gemma 4 12B | .74 | .74 | .71 | .74 | .22 | .23 |
| Gemma 4 E4B | .73 | .74 | .71 | .72 | .22 | .22 |
| Qwen3-Omni | .71 | .70 | .71 | .70 | .59 | .28 |
| Qwen2.5-Omni | .70 | .69 | .66 | .71 | .38 | .22 |
| Nemotron-3 Nano Omni | .71 | .64 | .68 | .71 | .39 | .23 |
Table 15: Audio and frame ablations on 150 news videos (macro-F1).
| Original file processed | Capped version processed | |||||
| Model | News | Hate | Sarcasm | News | Hate | Sarcasm |
| Gemma 4 12B | 603/603 | 400/400 | 650/650 | – | 400/400 | – |
| Gemma 4 E4B | 603/603 | 400/400 | 650/650 | – | 400/400 | – |
| Qwen3-Omni | 0/603 | 1/400 | 650/650 | 0/20 | 1/400 | – |
| Qwen2.5-Omni | 224/603 | 145/400 | 650/650 | 12/20 | 147/400 | – |
| Nemotron-3 Nano Omni | 598/603 | 382/400 | 650/650 | – | 400/400 | – |
Table 16: Full video: successfully processed / attempted items for the original file (left) and capped version (right). Capped news runs are 20-video pilots, not full-sample evaluations. Hate counts refer to the hate-classification call; Qwen3-Omni processed no target-group calls with original or capped full video, while other models had the same completion counts for hate and targets. “–”: not run.
| Model | News, verbal | Music | Hate | Sarcasm |
| Gemma 4 12B | .74 .74 | .22 .23 | .78 .64 | .50 .43 |
| Gemma 4 E4B | .73 .74 | .22 .22 | .78 .57 | .59 .43 |
| Qwen3-Omni | .71 .70 | .59 .28 | .74 .64 | .58 .54 |
| Qwen2.5-Omni | .70 .69 | .38 .22 | .77 .61 | .58 .51 |
| Nemotron-3 Nano Omni | .71 .64 | .39 .23 | .75 .61 | .55 .55 |
| MiniCPM-o 4.5 | .76 .71 | .84 .38 | .77 .66 | .40 .52 |
Table 17: Audio ablation on all datasets: macro-F1 of frames + audio the same frames without audio.
| Model | News, verbal | News, anchors | Hate | Sarcasm |
| Gemma 4 12B | +.05 [+.04, +.07] | .02 [ .04, +.01] | .01 [ .06, +.03] | .03 [ .07, +.01] |
| Gemma 4 E4B | +.03 [+.02, +.05] | .04 [ .07, .02] | .01 [ .05, +.03] | +.03 [ .01, +.07] |
| Qwen3-Omni | .01 [ .02, +.01] | .20 [ .24, .17] | .10 [ .16, .04] | +.01 [ .03, +.06] |
| Qwen2.5-Omni | .21 [ .23, .20] | .10 [ .12, .07] | .04 [ .08, +.01] | +.02 [ .03, +.07] |
| Nemotron-3 Nano Omni | .10 [ .12, .08] | .14 [ .17, .11] | .01 [ .05, +.03] | +.10 [+.06, +.15] |
| MiniCPM-o 4.5 | .09 [ .11, .08] | .29 [ .32, .26] | +.01 [ .03, +.06] | +.14 [+.11, +.18] |
Table 18: Describe first minus frames + audio (macro-F1, 95% item-bootstrap interval). Relative to extracted text, describe first is practically equivalent on the verbal news constructs for five of six models.
| % of decisions correct only by / wrong by | Macro-F1 | |||||||
| Group | Model | Text only | Video only | Both wrong | Text | Video | Text + video | Oracle |
| News, verbal | Gemma 4 12B | 10 | 5 | 11 | .80 | .75 | .75 | .87 |
| Gemma 4 E4B | 9 | 6 | 15 | .76 | .73 | .74 | .83 | |
| Qwen3-Omni | 9 | 5 | 15 | .76 | .71 | .71 | .82 | |
| Qwen2.5-Omni | 9 | 26 | 13 | .52 | .71 | .73 | .82 | |
| Nemotron-3 Nano Omni | 10 | 20 | 12 | .63 | .71 | .71 | .85 | |
Table 19: Integration across datasets: share of item construct decisions correct only from extracted text (sarcasm: transcript), only from frames + audio, or wrong in both, and macro-F1 of each route, of both in one call, and of the oracle.
| % of decisions | Macro-F1 | ||||||
| Model | Both correct | Text only | Video only | Both wrong | Extracted text | Native video | Oracle |
| Gemma 4 12B | 75.3 | 8.3 | 4.4 | 12.0 | .80 | .75 | .86 |
| Gemma 4 E4B | 70.4 | 7.8 | 5.4 | 16.4 | .76 | .73 | .82 |
| Qwen3-Omni | 70.8 | 9.3 | 4.8 | 15.1 | .76 | .70 | .82 |
| Qwen2.5-Omni | 52.4 | 8.4 | 22.6 | 16.6 | .52 | .70 | .79 |
| Nemotron-3 Nano Omni | 57.7 | 9.9 | 19.7 | 12.7 | .63 | .71 | .85 |
Table 20: Decision decomposition and oracle for extracted text versus native video (seven verbal news constructs). The main text uses frames + audio because text + video combines extracted text with that representation; the pattern is the same.
| Construct | Reference prevalence | Metadata | Transcript | Extr. text | Fr. + audio | Native | Full | Text + video | Describe |
| Negative news | 33% | +12 | +20 | +23 | +27 | +26 | +27 | +26 | +17 |
| Conflict | 33% | 7 | 1 | +1 | +16 | +16 | +16 | +17 | 1 |
| Scope | 67% | 19 | 3 | +3 | 3 | +12 | +9 | +4 | 7 |
| Call to interact (TikTok) | 24% | +12 | +17 | +28 | 2 | +3 | +4 | +0 | +8 |
| Call to interact (off-platform) | 17% | 10 | 4 | 0 | 4 | 3 | 2 | 4 | 9 |
| Other people’s voice | 25% | +7 | +4 | +19 | +15 | +26 | +7 | +11 | +20 |
Table 21: Prevalence error by construct and route: median across available models of estimated minus reference-coded sample prevalence (percentage points). Full video uses four models for news and HateMM and six for sarcasm; other defined routes use six.
| Analysis set | Group | Estimates | Median abs. error (pp) | pp | pp | pp |
| all routes | All | 496 | 12.9 | 79% | 59% | 32% |
| Hate and targets | 120 | 8.7 | 78% | 42% | 3% | |
| News, music | 46 | 64.3 | 91% | 89% | 89% | |
| News, verbal | 276 | 12.4 | 75% | 57% | 29% | |
| Sarcasm | 54 | 31.5 | 94% | 81% | 65% | |
| without metadata only | All | 442 | 12.9 | 78% | 59% | 32% |
Table 22: Absolute prevalence error under four analysis sets. A: all routes. B: without metadata only and counting the identical sarcasm transcript/extracted-text route once. C: deployable routes (transcript, extracted text, frames + audio, native video, full video, text + video, describe first; the sarcasm transcript counted once). D: for each model and construct, the deployable route with the highest macro-F1. Error relative to the reference-coded prevalence of the same sample; the samples are balanced or quota samples.
| Construct | Human pretest | Best route | Range, frames + audio | Six models as coders |
| Geographic scope | .97 | .70 | .43–.70 | .61 |
| Negative news | .91 | .60 | .31–.60 | .70 |
| Conflict | .84 | .76 | .48–.67 | .63 |
| Scope | 1.00 | .52 | .11–.41 | .27 |
| Call to interact (TikTok) | .91 | .71 | .38–.57 | .37 |
| Call to interact (off-platform) | .82 | .87 | .28–.81 | .64 |
Table 23: Krippendorff’s for the news constructs: published human pretest (two trained coders, 51 videos), best automated route, range across models for frames + audio, and agreement of the six models with each other on frames + audio.
| Model | Item input pairs | Identical codes | All nine codes identical |
| Gemma 4 12B | 300 | 99.0% | 93% |
| Gemma 4 E4B | 300 | 99.8% | 98% |
| Qwen3-Omni | 300 | 97.3% | 81% |
| Qwen2.5-Omni | 300 | 99.1% | 93% |
| Nemotron-3 Nano Omni | 300 | 97.9% | 85% |
| MiniCPM-o 4.5 | 300 | 98.3% | 88% |
Table 24: Re-run of 50 news videos on six routes (metadata only, transcript, extracted text, frames + audio, text + video, describe first) with identical settings on the same GPU.
| Model | Metadata | Transcript | Extr. text | Fr. + audio | Text + video | Describe |
| Gemma 4 12B | 99.8 | 99.6 | 99.1 | 98.2 | 97.8 | 99.8 |
| Gemma 4 E4B | 99.3 | 100.0 | 99.8 | 99.8 | 99.8 | 100.0 |
| Qwen3-Omni | 95.6 | 97.8 | 98.7 | 97.8 | 98.4 | 95.6 |
| Qwen2.5-Omni | 98.2 | 99.6 | 98.4 | 99.8 | 99.8 | 98.9 |
| Nemotron-3 Nano Omni | 98.4 | 95.8 | 98.4 | 98.9 | 98.0 | 97.6 |
| MiniCPM-o 4.5 | 95.3 | 98.0 | 98.7 | 98.7 | 98.9 | 100.0 |
Table 25: Re-run: percentage of identical individual codes by route.
| Cost per 1,000 videos | |||||
| Model | Route | GPU s per video | UZH (CHF) | AWS (US$) | Azure (US$) |
| Gemma 4 12B | Extracted text | 0.26 | 0.006 | 0.25 | 0.27 |
| Frames + audio | 1.11 | 0.025 | 1.06 | 1.13 | |
| Text + video | 1.21 | 0.027 | 1.16 | 1.24 | |
| Describe first | 1.42 | 0.032 | 1.35 | 1.45 | |
| Gemma 4 E4B | Extracted text | 0.09 | 0.002 | 0.09 | 0.09 |
Table 26: GPU time per news video and cost per 1,000 news videos (all nine constructs in one call; describe first includes the description call). We time each batch and divide its runtime by the batch size. Hourly prices for one A100 GPU: University of Zurich computing rate, CHF 0.081; Amazon Web Services on demand, US27.45 per hour, US East) ( Amazon Web Services, 2026 ) ; Microsoft Azure pay-as-you-go, US$3.673 (NC24ads A100 v4 with one A100 80GB GPU, East US) ( Microsoft Azure, 2026 ) . We exclude transcription and OCR costs because we run these steps once and reuse the results across models.
| Human codes | Automated estimates, range across six models | ||||
| Coefficient | (SE) | Extracted text | Frames + audio | Native video | Sign agreement a |
| Negative news | .63 (0.29) | .31 to .42 | .14 to .59 | .17 to .52 | 17/18 |
| Conflict | .79 (0.30) | .26 to .33 | .07 to .27 | .00 to .49 | 14/18 |
| Scope | .32 (0.33) | .11 to .80 | 1.74 to .31 | .57 to .30 | 8/18 |
| Call to interact (TikTok) | .13 (0.29) | .37 to .31 | .23 to .22 | .08 to .22 | 7/18 |
| Call to interact (off-platform) | .37 (0.33) | .56 to .00 | .40 to .70 | 1.16 to .32 | 13/18 |
Table 27: Engagement regression: human-code estimates and range of automated estimates across six models. OLS of log(1 + views) on all focal constructs with outlet fixed effects, HC1 standard errors, . a Automated estimates with the same sign as the human estimate, out of 18 (6 models 3 routes).
| Human codes | 18 pipelines | Highest-F1 route | |||||
| Coefficient | SE | (F1, error) | Median error (SE units) | , 36 pipelines | Closest (of 6) | Regret (SE units) | |
| Negative news | .63 | .29 | .44 | 1.22 | .34 | 1 | 0.78 |
| Conflict | .79 | .30 | .46 | 2.14 | .09 | 5 | 0.00 |
| Scope | .32 | .33 | .06 | 1.12 | .01 | 1 | 0.74 |
| Call to interact (TikTok) | .13 | .29 | .30 | 0.75 | .31 | 1 | 0.28 |
| Call to interact (off-platform) | .37 | .33 | .52 | 0.49 | .64 | 1 | 0.57 |
Table 28: Accuracy and coefficient recovery within each coefficient. : Spearman correlation across pipelines between the construct’s macro-F1 and the absolute deviation of the automated from the human-code coefficient (negative: more accurate pipelines recover the estimate better). Descriptive: pipelines share items, models, and routes.