Learned Parametric Emotion Editing: Real-Time Affective Filtering for On-Device Social Media Video

Musa Rochi; Marcel Schubert; Christoph Gebhardt

cs.CVSep 18, 2026

Learned Parametric Emotion Editing: Real-Time Affective Filtering for On-Device Social Media Video

Authors: Musa Rochi, Marcel Schubert, Christoph Gebhardt

Organizations: Eastern Switzerland University of Applied Sciences (OST)

Abstract

Problematic internet use affects a growing share of the population, yet common interventions, e.g., time limits, blocking, forced breaks, are coercive and easily circumvented. We explore a less restrictive alternative: adapting the emotional intensity of visual content. Prior work has shown that optimization can steer an image's affective content, but its per-image optimization cost makes it impractical for real-time deployment. We instead learn a model that predicts this transformation in a single forward pass: a MobileNetV4 backbone with FiLM-based emotion conditioning outputs parameters for differentiable global transformations. This replaces prior iterative optimization (80 s per image) with a single 3.7 ms forward pass. In a user study (N = 54), the model reduced viewer-reported arousal relative to unedited images, comparably to the grayscale well-being filter, while being rated higher in perceived quality. We integrate the model into an Android app that adapts Instagram video in real time, sustaining 60 fps on a Samsung Galaxy S23.

Figures & tables

Figure 1: Overview of our approach : An original social media feed frame (1) is captured and processed by our learned filter (2), which predicts parameters for a sequence of differentiable global transformations in a single forward pass. The adapted frame is rendered back into the live feed (3), reducing viewer-reported arousal while preserving perceived quality.

Figure 2: Conditional parametric generator. The conditioning signal $\alpha$ and the input image $I$ are encoded separately and fused via three FiLM blocks before three prediction heads regress the parameters $p$ of the global transformation $T$ .

Figure 3: End-to-end pipeline. Render Frame executes on every animation frame; inference ( Send Frame

\to

Parameters ) is triggered only every

n

-th frame and runs asynchronously off the main thread. Predicted parameters are smoothed ( Eq. 3 ) before being applied to subsequent frames.

Configuration	Loss	Arousal MSE	CLIP loss
FiLM Sigmoid	0.0405	0.0341	0.1271
FiLM Sigmoid (128)	0.0407	0.0342	0.1311
FiLM TanH	0.0422	0.0360	0.1236
No FiLM (Sigmoid)	0.0431	0.0373	0.1160
No FiLM (TanH)	0.0432	0.0374	0.1162
$\lambda_{\mathrm{clip}}=0.0$	0.0362	0.0362	0.1920

Table 1: Validation metrics at epoch 25 across five architecture configurations (top) and five different

\lambda_{\mathrm{clip}}

-values for the FiLM Sigmoid architecture (bottom); the two blocks come from separate training runs. Loss

=

Arousal MSE

+\lambda_{\mathrm{clip}}\cdot

CLIP loss ( Eq. 1 ); CLIP loss as defined in Eq. 2 .

Figure 4: Qualitative results. Example edits for (a) decreasing $\alpha$ (vintage, desaturated shift) and (b) increasing $\alpha$ (magenta/pink shift), annotated with regressor-predicted valence (V) and arousal (A) per image.

Panel of Figure 5 — Figure 5: Baseline comparison. (a) Achieved $\Delta$ arousal vs. requested $\alpha$ ; our model’s response is closer to the identity line $\Delta\text{arousal}=\alpha$ than baselines. (b) Content preservation (DINO) vs. $\alpha$ ; our model achieves higher similarity near $\alpha=0$ but drops more sharply than baselines beyond $|\alpha|\approx 0.1$ – $0.2$ .

Figure 6: Example stimulus of user study. Stimulus across the five study conditions: original images, our approach at $\alpha=-0.11$ , $\alpha=-0.12$ , and $\alpha=-0.13$ , and the grayscale baseline.

Condition	Arousal	Quality	Valence
original	4.55 (1.98)	3.97 (0.68)	6.34 (1.33)
alpha-1	4.36 (1.94) ∗	3.79 (0.76) ∗†	6.18 (1.43) ∗
alpha-2	4.34 (1.98) ∗	3.71 (0.78) ∗∗	6.12 (1.52) ∗
alpha-3	4.28 (2.10) ∗	3.40 (0.92) ∗∗∗†	6.01 (1.56) ∗∗∗
grayscale	4.19 (1.97) ∗∗	3.64 (0.74) ∗∗∗	6.05 (1.28) ∗∗

Table 2: Rating statistics. Mean ratings (SD in parentheses) for arousal, perceived image quality, and valence per condition. Asterisks mark differences from original and daggers from grayscale (

{}^{*}p<.05

{}^{**}p<.01

{}^{***}p<.001

{}^{\dagger}p<.05

). The shaded row marks our recommended operating point.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Training procedure. Solid arrows show the forward pass: a sampled conditioning value $\alpha$ and the frozen regressor’s estimate of $I$ define the target score $a^{\prime}=\mathrm{clip}(R_{a}(I){+}\alpha,0,1)$ ; the generator $f_{\theta}$ predicts transform parameters $p$ , producing $\hat{I}=T(I,p)$ , which is re-scored by $R_{a}$ and compared against $a^{\prime}$ ( $\mathcal{L}_{\text{affect}}$ ), while a frozen CLIP encoder supplies a semantic-preservation term ( $\mathcal{L}_{\text{CLIP}}$ ) against the original image. The dashed red arrow shows the backward pass: gradients of the combined objective $\mathcal{L}(\theta)$ propagate back through $R_{a}$ , the CLIP encoder, and $T$ into $f_{\theta}$ ; $R_{a}$ and the CLIP encoder (boxes with locks) remain frozen throughout.

Operation	Dim.	Raw Range	Range $[m,M]$	Identity
Exposure	1	$(-\infty,\infty)$	$[-0.1,1]$	0.0
Saturation	1	$[0,\infty)$	$[0.5,3]$	1.0
Tone curve	$K$	$(-\infty,\infty)$	$[0,3]$	1.0
Color curve	$3K$	$(-\infty,\infty)$	$[0,3]$	1.0
Contrast	1	$[0,\infty)$	$[0.5,3]$	1.0
Sharpness	1	$[0,\infty)$	$[0,3]$	1.0

Appendix

Table 3: Sub-transforms composing

T

. Raw range is the domain of the sub-transform; range is the interval

[m,M]

enforced by

f_{\mathrm{sig}}

( Eq. 4 ) on the values predicted by

f_{\theta}

. Parameters are applied in the fixed order listed above.

Setting	Value
Optimizer	Adam [ 32 ]
Learning rate	$1\times 10^{-4}$
Batch size	32 (4 $\times$ 8 GPUs)
Hardware	8 $\times$ NVIDIA Tesla P100
Dataset	MS-COCO [ 38 ]
$\lambda_{\mathrm{clip}}$	0.05

Appendix

Table 4: Training configuration for the parametric generator.

Panel of Figure 8 — Figure 8: Full training curves for the FiLM ablation. Validation overall loss, arousal MSE term, and CLIP preservation loss over training, for all five configurations described in Sec. C.1.1 .

Panel of Figure 9 — Figure 9: Full training curves for the $\lambda_{\mathrm{clip}}$ ablation. Validation overall loss, arousal MSE term, and CLIP preservation loss over training, for all five $\lambda_{\mathrm{clip}}$ values described in Sec. C.1.2 .

		Prior literature
Attribute	Computation	Valence	Arousal	References
Brightness	Mean grayscale pixel intensity.	$(+)$	–	[ 61 , 23 ]
Saturation	Mean of the HSV saturation channel.	$(+)$	$(+)$	[ 60 , 55 ]
Contrast	Std. dev. of grayscale pixel intensities.	$(+)$	–	[ 65 ]
Colorfulness	Std. dev. plus weighted mean of distances from the mean in the LAB $a$ / $b$ channels.	$(+)/(-)$ a	$(+)$	[ 5 , 12 ]
Blur	Laplacian variance of grayscale image (high $=$ less blur).	$(-)$	$(-)$	[ 17 ]

Appendix

Table 5: Image attribute metrics and their affective associations. Associations are reported from prior literature, where

(+)

(-)

denote a positive/negative association and “–” indicates none reported.

Panel of Figure 10 — Figure 10: Regressor-predicted arousal and valence as a function of $\alpha$ . (a) Arousal increases monotonically with $\alpha$ and saturates beyond $|\alpha|\gtrsim 0.2$ . (b) Valence, not directly controlled, shows similar turning points, indicating an arousal influence.

Figure 11: Preservation metrics over

\alpha

. FID, KID and L1 reconstruction error as a function of the conditioning value

\alpha

, showing the parabolic increase near

\alpha=0

followed by a sharp transition and subsequent flattening.

Figure 12: Evolution of image attributes over

\alpha

. All six attributes from Tab. 5 , plotted against the conditioning value

\alpha

Method	Median	Mean	p95	Min	Max
Ours (ms)	3.71	3.72	4.34	3.05	4.99
[ 24 ] (s)	80.92	80.71	83.01	78.81	83.10

Appendix

Table 6: Per-frame latency on training hardware. Ours is a forward pass (

n=5{,}000

); [ 24 ] is a 300-step edit (

n=250

Stage	Median	Mean	p95	Min	Max
Copy to canvas	0.00	0.05	0.10	0.00	3.30
Extract pixels	7.90	8.98	17.00	2.70	42.20
Inference	37.90	40.48	61.60	21.80	114.50
Total	46.00	49.52	77.80	24.60	138.50

Appendix

Table 7: Per-frame latency (ms) on the target device. Computed on a Samsung Galaxy S23 using

n=1{,}500

frames.

Component	Mean $\Delta$ (mAh)	SD
Total	+31.86	10.65
CPU	+25.13	13.78
Screen	+5.74	8.73
Video	+0.24	0.65
WiFi	$-$ 0.00	0.02

Appendix

Table 8: On-device energy cost. Paired delta (filter-on minus filter-off) in app-attributed power use, per hardware component, across five matched 30-minute sessions on a Samsung Galaxy S23.

Panel of Figure 13 — Figure 13: Rating differences. Per-participant rating differences between each condition and original for (a) arousal, (b) perceived image quality, and (c) valence. Each box summarizes one difference score per participant; values below the dashed zero line indicate lower ratings than for the original images. Significance notation as in Tab. 2 .

Explore similar work

May 4, 2026

MooD: Perception-Enhanced Efficient Affective Image Editing via Continuous Valence-Arousal Modeling

Affective Image Editing (AIE) aims to modify visual content to evoke targeted emotions. Although current approaches achieve impressive editing quality, they often overlook inference efficiency, which limits their applicability in computational social scenarios. Moreover, most methods depend on discrete emotion representations, which hinder the continuous modeling of complex human emotions and constrain expressive capabilities in interactive scenarios. To tackle these gaps, we propose MooD, the first framework that directly leverages continuous Valence-Arousal (VA) values as editing instruction for fine-grained and efficient AIE in computational social systems. Specifically, we first introduce a VA-Aware retrieval strategy to bridge vague affective values and detailed visual semantics. Building upon this, MooD integrates visual transfer and perception-enhanced semantic guidance to achieve controllable AIE. Furthermore, considering that existing VA-annotated datasets mainly focus on social scenarios and largely overlook natural scenes, we therefore construct AffectSet, a comprehensive VA-annotated dataset covering diverse scenarios, to support model optimization and evaluation. Extensive qualitative and quantitative experimental results demonstrate that our MooD achieves superior performance in both affective controllability and visual fidelity while maintaining high efficiency. A series of ablation studies further reveal the crucial factors of our design.

Xinyi Yin, Yiduo Wang, Tingqi Hu +6

School of Cyber Science and Engineering, Zhengzhou University, Zhengzhou 450002, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China · School of Journalism and New Media, Xi’an Jiaotong University, Xi’an 710049, China +1

Valence-Arousal Modeling Image Editing Controllable Image Generation Affective Computing

Ask PDF

Oct 9, 2026

What We Talk About When We Talk About Emotion Editing

Emotional image editing aims to change the emotion a scene conveys while preserving its essential content. Because the same scene can support different emotional interpretations, achieving an emotional goal requires visual changes grounded in the source context. Motivated by the Component Process Model, which links emotional responses to event appraisal, we investigate neural representations as a reference for these editing decisions. Using TRIBE v2, an fMRI encoding model, we analyze 400 source-target pairs across eight emotion categories. Hidden states from EmoEdit, EmoEditor, and AffectDelta correspond more strongly to predicted target-image responses in visual regions than in affect-related regions. This imbalance motivates ROI-ReFT, which combines source visual responses as a content reference with a predicted affective response change as an editing direction. Two interacting branches use both conditions to guide a frozen editor throughout generation. Predicted responses of the generated image supervise affective change and constrain visual drift. On a post hoc selected subset of 400 editing requests, ROI-ReFT achieves 91.75% target-emotion accuracy and 83.25% joint success, with the same output both matching the target emotion and reaching source-output DINO-I of at least 0.80. Our code and dataset will be publicly released.

Xingzu Zhan, Lin Gu, Ruogu Fang

Department of Biomedical Engineering, Vanderbilt University, Nashville, Tennessee, USA · Research Institute of Electrical Communication, Tohoku University, Sendai, Japan

Image Editing Affective Computing

Ask PDF

May 4, 2026

EMOVIS: Emotion-Optimized Image Processing

In cinematography, visual attributes such as color grading, contrast, and brightness are manipulated to reinforce the emotional narrative of a scene. However, conventional Image Signal Processors (ISPs) prioritize scene fidelity, effectively neglecting this expressive dimension. To bring this cinematic capability to real-time camera pipelines during video capture, we introduce EMOVIS (EMotion-Optimized VISual processing). We establish a systematic mapping between a compact set of high-level emotional states (Happy, Calm, Angry, Sad) and low-level ISP controls - including color saturation, local tone mapping, and sharpness - supported by a calibration user study with statistically significant effects across parameters. We propose a control framework that integrates these emotion-driven adjustments into standard ISP hardware without altering the underlying processing stages. Validation via blind A/B testing shows that viewers prefer the emotion-optimized rendering in 87% of trials when the target emotion matches the scene context, indicating that emotion-aligned ISP control improves perceived suitability for expressive visual content.

Dor Barber, Rony Zatzarinni, Hava Matichin +1

Intel Corporation

Image Enhancement

Ask PDF