Controlled Face Manipulation and Synthesis for Data Augmentation

Joris Kirchner; Amogh Gudi; Marian Bittner; Chirag Raman

cs.CVFeb 22, 2026

Controlled Face Manipulation and Synthesis for Data Augmentation

Authors: Joris Kirchner, Amogh Gudi, Marian Bittner, Chirag Raman

Organizations: Vicarious Perception Technologies (VicarVision) · Delft University of Technology

Abstract

Deep learning vision models excel with abundant supervision, but many applications face label scarcity and class imbalance. Controllable image editing can augment scarce labeled data, yet edits often introduce artifacts and entangle non-target attributes. We study this in facial expression analysis, targeting Action Unit (AU) manipulation where annotation is costly and AU co-activation drives entanglement. We present a facial manipulation method that operates in the semantic latent space of a pre-trained face generator (Diffusion Autoencoder). Using lightweight linear models, we reduce entanglement of semantic features via (i) dependency-aware conditioning that accounts for AU co-activation, and (ii) orthogonal projection that removes nuisance attribute directions (e.g., glasses), together with an expression neutralization step to enable absolute AU edit. We use these edits to balance AU occurrence by editing labeled faces and to diversify identities/demographics via controlled synthesis. Augmenting AU detector training with the generated data improves accuracy and yields more disentangled predictions with fewer co-activation shortcuts, outperforming alternative data-efficient training strategies and suggesting improvements similar to what would require substantially more labeled data in our learning-curve analysis. Compared to prior methods, our edits are stronger, produce fewer artifacts, and preserve identity better.

Figures & tables

Fig. 1: Distribution of AU occurrence labels in datasets. Naturally occurring distribution of AU labels in real data (DISFA [ 9 , 10 ] ) is highly skewed (black). In contrast, our method for controlled editing/synthesis allows for generation of datasets with a balanced distribution (green).

Fig. 2: Overview of our method. (Top-Left) Learning linear edit directions on semantic codes, where AU models are conditioned on other AUs. Afterwards, AU directions are projected on other possible nuisance attributes. (Top-Right) Editing existing images: encode $(x_{T},z)$ , pick target AU, obtain disentangled direction $w$ , set $z\leftarrow z+s\,w$ , decode with original $x_{T}$ . (Bottom) Synthesizing new faces: sample $(x_{T},z_{\text{sample}})$ from DiffAE, optionally accept by demographic predictors, neutralize $z_{\text{sample}}$ via the neutralization model $\mathcal{N}$ by optimizing Equation ( 1 ), then edit and decode.

Fig. 3: An illustrative DAG showing an example of AU relations. The intensity of AU 1 is defined by the reconstructed image, but also by AU 2 as they are often activated together. Conditioning on a correlated AU blocks leakage from that AU to the target AU. At the same time attributes that are influenced by AU1, called colliders (e.g., surprised ), should not be conditioned upon to avoid opening spurious paths.

Fig. 4: Examples of controlled AU editing (top half) and controlled synthesis of new faces (bottom half). Editing examples (top) uses existing neutral images from the DISFA dataset [ 9 ] , where reconstruction is obtained by encoding and decoding without editing. Synthesis examples (bottom) contains random identities sampled from DiffAE containing arbitrary facial expressions. which is first neutralized by the non-linear procedure to suppress pre-existing expressions. Thereafter, each AU is modified by $+1$ using the proposed method, corresponding to FACS activation level E. Editing of images is largely without major artifacts, and neutralization succeeds in deactivating AUs; but complex expressions may leave small residuals, and editing may slightly alter background of synthetic images.

Fig. 5: (Right) Comparison of correlation between detected AUs in the real DISFA data (top/red) and the edited/synthetic data (bottom/blue), as estimated by an AU detection tool (FaceReader [ 38 ] ) as a stand-in for human annotators. The color heatmap represents the amplitude of the correlation, while the green/yellow/red markers in the edited/synthetic data table denote the intensity of change w.r.t. correlations in the real data (green = reduction, red = increase, yellow = no significant change). Detected AUs in the edited/synthetic data are much less correlated with each other on average (0.09) as compared to the detected AUs in the real data (0.16). (Left) For reference, the inter-AU correlations between the ground truth labels in DISFA are also shown. These exhibit similar patterns as the detector’s AU estimates on the real images, suggesting limited influence of detector bias.

Fig. 6: Box plots showing AU detection accuracy (F1 scores, averaged over AUs) for models trained with (Real + Generated) and without (Real only) generated augmentation on DISFA [ 9 , 10 ] (left), as well as cross-dataset evaluation on FEAFA [ 31 ] and BP4D [ 32 ] (right). Generated augmentation consistently improves F1 on all three benchmarks.

Fig. 7: AU detection accuracy (F1) of baseline models trained on varying amounts of real data (learning curve). The horizontal line shows performance with generated augmentation; error bars denote standard deviation over five runs. A POW3 fit [ 41 ] suggests

\approx 5\times

more real data is needed to match the augmented accuracy.

Fig. 8: Difference in validation set false positive rates (FPRs) for AU pairs (where one AU is absent while another is present) between models trained with and without our generated augmentation technique. The green, yellow, and red markers represent the intensity of change (green = reduction, red = increase, yellow = no significant change). These false positives are a sign of the model over-exploiting the entanglements/correlations between AUs to make predictions. Training with our generated augmentation results in disentangled learning: yielding a lower false positive rate for a majority of AUs as compared to training with real data alone, with an average reduction of 7.4% points.

Fig. 9: Qualitative comparison of facial expression editing from our method with StyleAU [ 28 ] , StyleGAN-NADA [ 20 ] and MagicFace [ 8 ] (image sourced from [ 28 ] ). Editing is done at various intensity levels (increasing from top to bottom) loosely corresponding to FACS AU intensity levels from A to E: StyleAU $\{1,3,5\}$ , StyleGAN-NADA $\{0,0.4,0.8\}$ , MagicFace $\{0,4,6\}$ , and our method $\{0,0.5,1\}$ . For StyleGAN-NADA, the images are generated with target class: “ face with [AU description] increased by [intensity] ”. Our method retains the AUs in the original face better, while also often performing a strong plausible edit and introducing fewer artifacts under single and multi-AU editing.

Fig. 10: Comparison of AU-editing fidelity (left) and identity preservation (right) across manipulation methods as a function of the number of simultaneously edited AUs. Left: MAE between AU activations estimated on the edited image and on an identity-matched real target image exhibiting the desired AU enactment. Right: identity drift measured as the cosine similarity distance between face recognition embeddings of the edited image and the corresponding real target image (same identity), with the purple line indicating the face recognition threshold. Legend: Error bars denote standard error. Across all AU counts, our method achieves the lowest AU error, indicating the closest match to target AU configurations, while maintaining low identity drift. As expected, identity drift increases with the number of edited AUs, but our edits remain within the recognition threshold.

Fig. 11: Comparison of AU-detection accuracies (averaged F1 scores over AUs; top) and false positive rates of AU pairs (where one AU is absent while another is present; bottom) of models trained using different data-efficient training setups. Combining generated augmentation with reweighing does not yield further improvement in accuracy, but combining with pretraining yields the highest accuracy. While reweighing and pretraining increasingly lower the AU-pair FPR, the largest improvement is caused by the introduction of our generated augmentation.

Fig. 12: Qualitative comparison showcasing the effect of conditioning (Cond.) vs projecting on other AUs (AU Proj.) and the effect of projecting conditioned models on other entangled attributes (Cond.+Proj.). Brackets under the edited AU names denote the strengths and directions of the edits: $\pm$ 1 approximately corresponds to FACS intensity level E. Conditioning or projection reduces entanglement with other non-target AUs (e.g., AU1–AU2, AU6–AU12). Conditioning tends to preserve mild, plausible overlapping effect, while projection removes them entirely. Furthermore, projecting on nuisance attributes like glasses removes entanglement with them.

	Baseline	Conditioning	Cond. + Projection
SSIM	$0.8125\pm 0.0033$	$0.8233\pm 0.0033$	$\mathbf{0.8300\pm 0.0032}$

TABLE I: Effect of conditioning and projection on fine-grained edit fidelity, measured by SSIM w.r.t real target image (mean

\pm

standard error). Conditioning and projection progressively improve SSIM over the baseline.

Fig. 13: Box-plot comparison of the AU detection accuracy (F1 score) of models trained using different data augmentation strategies. The best performance is achieved from augmenting with both edited and synthetic images, while augmenting only with either one of them yields a comparatively lower accuracy (that is still higher than using real data only).

Fig. 14: Neural network architecture of the neutralization model $\mathcal{N}$ . Fully Connected layers are abbreviated with FC. After each linear layer, the SiLU [ 47 ] activation function is applied. The network splits into two branches. A dynamic branch that leads to AU predictions, and a static part that leads to the prediction of the other static features. Finally, each attribute has a FC layer of 512, only used for predicting that specific attribute.

Synthetic Data Augmentation

Synthetic data augmentation creates or modifies training examples to improve learning when real data are scarce, imbalanced, costly, or sensitive.

See all 453 papers

Mar 26, 2026

PixelSmile: Toward Fine-Grained Facial Expression Editing

Fine-grained facial expression editing has long been limited by intrinsic semantic overlap. To address this, we construct the Flex Facial Expression (FFE) dataset with continuous affective annotations and establish FFE-Bench to evaluate structural confusion, editing accuracy, linear controllability, and the trade-off between expression editing and identity preservation. We propose PixelSmile, a diffusion framework that disentangles expression semantics via fully symmetric joint training. PixelSmile combines intensity supervision with contrastive learning to produce stronger and more distinguishable expressions, achieving precise and stable linear expression control through textual latent interpolation. Extensive experiments demonstrate that PixelSmile achieves superior disentanglement and robust identity preservation, confirming its effectiveness for continuous, controllable, and fine-grained expression editing, while naturally supporting smooth expression blending.

Jiabin Hua, Hengyuan Xu, Aojie Li +4

Fudan University · StepFun

Image Editing Diffusion-Based Image Editing Text-Guided Image Editing Image Editing Evaluation

Ask PDF

Mar 16, 2026

High-Fidelity 3D Facial Avatar Synthesis with Controllable Fine-Grained Expressions

Facial expression editing methods can be mainly categorized into two types based on their architectures: 2D-based and 3D-based methods. The former lacks 3D face modeling capabilities, making it difficult to edit 3D factors effectively. The latter has demonstrated superior performance in generating high-quality and view-consistent renderings using single-view 2D face images. Although these methods have successfully used animatable models to control facial expressions, they still have limitations in achieving precise control over fine-grained expressions. To address this issue, in this paper, we propose a novel approach by simultaneously refining both the latent code of a pretrained 3D-Aware GAN model for texture editing and the expression code of the driven 3DMM model for mesh editing. Specifically, we introduce a Dual Mappers module, comprising Texture Mapper and Emotion Mapper, to learn the transformations of the given latent code for textures and the expression code for meshes, respectively. To optimize the Dual Mappers, we propose a Text-Guided Optimization method, leveraging a CLIP-based objective function with expression text prompts as targets, while integrating a SubSpace Projection mechanism to project the text embedding to the expression subspace such that we can have more precise control over fine-grained expressions. Extensive experiments and comparative analyses demonstrate the effectiveness and superiority of our proposed method.

Yikang He, Jichao Zhang, Wei Wang +2

Institute of Information Science, Beijing Jiaotong University, Beijing, China. · School of Computer Science, Ocean University of China, Qingdao, China. · Department of Information Engineering and Computer Science (DISI), University of Trento, Italy.

3D Avatar Generation

Ask PDF

Jul 1, 2026

MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment

Recent brain decoding studies have made substantial progress in reconstructing externally perceived visual content from neural signals. However, using electroencephalography (EEG) recordings to guide facial expression editing remains largely unexplored and poses a distinct challenge: rather than recovering what a subject sees, it requires identifying facial-action related patterns from noisy EEG signals and grounding them in localized, identity-preserving expression edits. In this paper, we investigate EEG-conditioned facial image editing for fine-grained facial action unit (AU) control and propose MindAU, a unified framework for controlling facial AU edits from EEG signals. MindAU first learns noise-robust and AU-discriminative EEG representations through temporal masked reconstruction and AU classification supervision. It then bridges the modality gap via Dual-Stream Manifold Alignment, aligning EEG features with AU-level text semantics and identity-reduced visual displacement trajectories in the multimodal space of Qwen2.5-VL. Finally, MindAU incorporates EEG-aware Multimodal Rotary Positional Embeddings, landmark-guided reference masking, and AU-aware region supervision into a multimodal diffusion-based editor for high-fidelity identity-preserving editing. We also introduce E-CAFE, a curated benchmark for EEG-Conditioned Action-Unit Facial Editing with paired EEG-face editing samples and standardized evaluation protocols. Extensive experiments demonstrate the effectiveness of MindAU and suggest its potential as a step towards future assistive expression technologies for individuals with facial neuromuscular disorders.

Zhenhang Li, Xin Zhou, Hao Deng +1

Binghamton University · Massachusetts General Hospital

EEG Decoding Cross-Modal Alignment Diffusion-Based Image Editing Multimodal Diffusion Models

Ask PDF

Abstract

Synthetic Data Augmentation

Explore similar work

PixelSmile: Toward Fine-Grained Facial Expression Editing

High-Fidelity 3D Facial Avatar Synthesis with Controllable Fine-Grained Expressions

MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment