Perceive, Refine, Reason: A Calibrated Pipeline for Measuring Indicators in Strategic Visual Communication on Social Media

Weihong Qi; Chen Ling

cs.CVSep 13, 2026

Perceive, Refine, Reason: A Calibrated Pipeline for Measuring Indicators in Strategic Visual Communication on Social Media

Authors: Weihong Qi, Chen Ling

Organizations: Indiana University Bloomington Bloomington, IN, USA

Abstract

Visual content shapes audience perception and opinion on social media, and computational social science increasingly relies on automated tools to analyze images at scale. Yet a measurement gap persists: existing tools rely on predefined categories or produce only coarse image-level labels, while measuring which specific objects appear in an image, how prominently, and where in the frame remains difficult at scale. We introduce Perceive, Refine, Reason (PRR), a calibrated pipeline that turns flexible vision-language detectors into auditable measurement instruments for social-scientific research. PRR combines natural-language category prompts with pixel-level spatial refinement via the Segment Anything Model (SAM) and a multimodal LLM arbitration layer whose reasoning chains externalize domain knowledge and lower the expertise threshold for human-in-the-loop validation. A complementary three-tier auditability framework applies quantification learning to profile per-category reliability, support task-aligned configuration, and statistically correct prevalence estimates. Across four vision-language detectors and nine sociological categories, the pipeline yields substantial precision gains over zero-shot baselines, including a 43.3-point improvement for the strongest backbone. Applying PRR to 103,920 Facebook images from U.S. legislators during the 2024 election cycle and linking detections to DW-NOMINATE ideology scores, we find that more conservative legislators display U.S. flags as larger visual elements, with a weaker tendency toward peripheral placement, a spatial pattern invisible to binary detection. PRR provides computational social scientists with a model-agnostic toolkit for accessible, spatially-grounded, and correctable visual measurement.

Figures & tables

Figure 1. Overview of the PRR pipeline (left) and the three-tier audit framework (right). The pipeline transforms raw open-vocabulary detections into verified detections with spatial measurements and auditable reasoning. The audit framework profiles per-category reliability on a labeled test set (Tier 1), guides task-aligned configuration (Tier 2), and applies statistical correction to prevalence estimates on the full corpus (Tier 3), yielding calibrated measurements with known, bounded error. The human icon marks stages requiring researcher input or decisions; other stages are automated.

Figure 2. Overview of the PRR pipeline applied to a single image posted on Facebook by U.S. senator Eric Schmitt. (a) Perceive: An Open-Vocabulary Detector (OVD) generates 8 raw candidate boxes with noisy labels and redundancies. (b) Refine: The Segment Anything Model (SAM) extracts pixel-level masks (orange) for spatial grounding, and Non-Maximum Suppression (NMS) reduces the candidate set to 5. (c) Reason: An LLM evaluates this refined set to accept (green), reject (red), or re-classify (blue) each candidate.

Group	Members	Mean Images	Median Images
Democrats (D)	232	216.49	200.0
Independents (I)	3	170.00	143.5
Republicans (R)	237	214.40	179.0
House of Representatives	379	220.59	189.0
Senate	93	192.95	187.0

Table 1. Summary statistics of collected images by party affiliation and congressional chamber.

Model	Variant	Precision	Recall	F1 $\uparrow$
GroundingDINO	Zero-shot	16.6	25.1	20.0
	Calibrated	45.1	21.2	28.9
		(+28.5)	(-3.9)	(+8.9)
OWL-ViT v1	Zero-shot	66.3	30.7	42.0
	Calibrated	92.0	17.6	29.6
		(+25.7)	(-13.1)	(-12.4)

Table 2. Overall micro-averaged performance of zero-shot and calibrated variants. Best results for each model are highlighted in bold. Numbers in color indicate absolute changes of the calibrated variant over the zero-shot baseline.

Figure 3. Performance breakdown across the nine target categories for each evaluated baseline model. Bar heights represent Precision, Recall, and F1 score. The red dashed line marks the 80% precision threshold. The results illustrate a clear performance gradient, where models achieve high precision on concrete visual indicators but fail to reliably detect abstract sociological symbols.

Category	$n$	P (%)	R (%)	F1 (%)	Supported Analysis
U.S. flag	39	78.6	84.6	81.5	Absolute frequency, relative comparisons
Podium	14	80.0	57.1	66.7	Relative comparisons only
Suit	106	93.5	40.6	56.6	Relative comparisons only

Table 3. Tier 1 empirical profile for the three categories selected for full-corpus analysis. Metrics are derived from the calibrated OWL-v2 pipeline (IoU

\geq 0.5

);

n

is the number of annotated instances. The “Supported Analysis” column reflects the task-oriented assessment dictated by each category’s specific error profile.

Figure 4. Detection prevalence across parties. For U.S. flag, bars show both raw rates and Tier 3 adjusted prevalence (Eq. 1 ), using image-level Precision and Recall estimated on the annotated sample; adjustment is not applied to suit and podium due to low recall (Table 3 ). Error bars show standard errors across legislators.

Association test
Party	Total	Flag	Rate	Adj. Cnt	Adj. Rate
Democrat	20,735	7,101	0.342	7,859.24	0.379
Republican	23,462	9,763	0.416	10,805.48	0.461
$\chi^{2}(1)=252.75,\ p=6.54\times 10^{-57},\ V=0.0756$
Odds ratio (R vs. D) : 1.368

Table 4. Party differences in U.S. flag display. Republicans are 1.37 times as likely to display a U.S. flag.

Comparison	Category	OR	95% CI
Panel A: Republican vs. Democrat
	Suit	0.934 ∗	[0.892, 0.979]
	Podium	0.686 ∗∗∗	[0.655, 0.718]
	U.S. Flag	1.368 ∗∗∗	[1.316, 1.422]
Panel B: Senate vs. House
	Suit	1.330 ∗∗∗	[1.248, 1.418]

Table 5. Relative comparisons in visual category display by party and chamber (RQ2). Republicans are less likely than Democrats to display suits and podiums but more likely to display U.S. flags. Senators are more likely than House members to display suits and less likely to display U.S. flags, with no significant chamber difference for podiums.

Figure 5. Ideology and U.S. flag display at the legislator level. DW-NOMINATE dimension 1 captures liberal–conservative positioning; separate OLS fits are shown for House and Senate. More conservative legislators display flags more frequently, with a consistent effect across chambers.

Metric	Dem. Mean	Dem. Median	Rep. Mean	Rep. Median	Welch’s $t$ ( $p$ )	Mann–Whitney $p$
Flag area ratio	0.0392	0.0323	0.0489	0.0423	$-3.495$ ( $5.20\times 10^{-4}$ )	$8.55\times 10^{-6}$
Flag center distance	0.4689	0.4686	0.4582	0.4558	$1.833$ ( $6.74\times 10^{-2}$ )	$1.74\times 10^{-2}$

Table 6. Spatial prominence of U.S. flags by party. Republicans’ flags occupy a larger share of the image area; Democratic flags are slightly more centered but not statistically significant. Standard errors are clustered at the legislator level.

Figure 6. Qualitative examples of LLM-as-Judge decisions. (a) A Dominican Republic flag is rejected as non-U.S.; (b) a military uniform is re-classified from ‘suit’ (OWL-ViT v1 example); (c) a U.S. flag on the Capitol is accepted. The natural-language reasoning externalizes domain knowledge, flag origin, uniform insignia, landmark identification, that would otherwise require trained annotators. Orange: SAM mask; box color: red=rejected, blue=re-classified, green=accepted.

Social Media Analysis

Social media analysis studies online posts, media, users, and interactions to understand communication, communities, and public behavior.

See all 221 papers

Nov 21, 2025

Do Vision-Language Models Understand Visual Persuasiveness? A Diagnosis via Visual Persuasive Factors

Visual persuasion uses images to shape cognition, emotion, and behavior, with its effects depending on both visual attributes and semantic context. Despite recent progress, it remains unclear whether Vision-Language Models (VLMs) understand visual persuasiveness. This motivates us to ask: can VLMs assess whether an image persuasively supports an intended message, which visual factors shape this judgment, and do they align with human judgments? Through empirical analyses on image-message pairs where human raters consistently agree on the persuasiveness judgment, we show that VLMs exhibit a recall-oriented bias: they over-predict images as persuasive while achieving high recall. We introduce Visual Persuasive Factors (VPFs), a taxonomy informed by cognitive psychology for quantifying visual cues that shape persuasive judgments. Our factor-level analysis reveals that VPFs distinguish human persuasiveness judgments, whereas VLMs only partially reproduce these patterns, often generating false positives by treating persuasion-relevant cues as sufficient evidence. Building on this insight, we evaluate VPF-guided interventions and find that properly framed VPF knowledge can improve performance, but merely specifying visual cues or adding step-by-step reasoning is insufficient. By analyzing model rationales at the level of functional reasoning steps, we further identify a central bottleneck in connecting object identification to semantic message alignment.

Gyuwon Park, Hyounghun Kim

Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science and Engineering, POSTECH

VLM Evaluation VLM Reasoning Vision-Language Alignment

Ask PDF

Apr 14, 2026

Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs

Multimodal language models (MLLMs) are increasingly paired with vision tools (e.g., depth, flow, correspondence) to enhance visual reasoning. However, despite access to these tool-generated visual cues, MLLMs often fail to benefit from them. Existing approaches typically feed raw tool outputs into the model, but these dense, pixel-level representations are misaligned with the language-native reasoning strengths of LLMs, leading to weak perception and reliance on language priors. We argue that, in problems where vision tools can provide the necessary visual cues, the bottleneck is not more tool calls or larger MLLMs, it is how tool outputs are represented. We introduce Perception Programs (P

^2

), a training-free, model-agnostic method that rewrites tool outputs into compact, structured, language-native summaries that MLLMs can directly parse and reason over. Across six perception-centric tasks in BLINK, P

^2

consistently yields large improvements over base models and raw tool-augmented baselines. With GPT-5 Mini as the base model, P

^2

raises its accuracy from 41.35% to 86.47% on multi-view reasoning, from 52.42% to 81.45% on relative depth, and achieves a 22% average gain across tasks, setting new state-of-the-art results. Even on smaller MLLMs, e.g., InternVL3.5-4B and Qwen3VL-4B, we observe 15-40% absolute gains from P

^2

, surpassing prior agentic, supervised, and RL-based tool-use methods-without any training or model modifications.

Muhammad Kamran Janjua, Hugo Silva, Di Niu +1

Huawei Technologies, Canada · University of Alberta, Canada

Tool Use in VLMs VLM Reasoning Large Vision-Language Models

Ask PDF

Apr 21, 2026

Seeing Candidates at Scale: Multimodal LLMs for Visual Political Communication on Instagram

This paper presents a computational case study that evaluates the capabilities of specialized machine learning models and emerging multimodal large language models for Visual Political Communication (VPC) analysis. Focusing on concentrated visibility in Instagram stories and posts during the 2021 German federal election campaign, we compare the performance of traditional computer vision models (FaceNet512, RetinaFace, Google Cloud Vision) with a multimodal large language model (GPT-4o) in identifying front-runner politicians and counting individuals in images. GPT-4o outperformed the other models, achieving a macro F1-score of 0.89 for face recognition and 0.86 for person counting in stories. These findings demonstrate the potential of advanced AI systems to scale and refine visual content analysis in political communication while highlighting methodological considerations for future research.

Michael Achmann-Denkler, Mario Haim, Christian Wolff

Media Informatics Group University of Regensburg Regensburg, Germany · Department of Media and Communication Ludwig-Maximilians-Universität Munich, Germany

VLM Evaluation Social Media Analysis Face Recognition

Ask PDF

Abstract

Social Media Analysis

Explore similar work

Do Vision-Language Models Understand Visual Persuasiveness? A Diagnosis via Visual Persuasive Factors

Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs

Seeing Candidates at Scale: Multimodal LLMs for Visual Political Communication on Instagram