cs.CVFeb 22, 2026
SaveControlled Face Manipulation and Synthesis for Data Augmentation
Organizations: Vicarious Perception Technologies (VicarVision) · Delft University of Technology
Abstract
Deep learning vision models excel with abundant supervision, but many applications face label scarcity and class imbalance. Controllable image editing can augment scarce labeled data, yet edits often introduce artifacts and entangle non-target attributes. We study this in facial expression analysis, targeting Action Unit (AU) manipulation where annotation is costly and AU co-activation drives entanglement. We present a facial manipulation method that operates in the semantic latent space of a pre-trained face generator (Diffusion Autoencoder). Using lightweight linear models, we reduce entanglement of semantic features via (i) dependency-aware conditioning that accounts for AU co-activation, and (ii) orthogonal projection that removes nuisance attribute directions (e.g., glasses), together with an expression neutralization step to enable absolute AU edit. We use these edits to balance AU occurrence by editing labeled faces and to diversify identities/demographics via controlled synthesis. Augmenting AU detector training with the generated data improves accuracy and yields more disentangled predictions with fewer co-activation shortcuts, outperforming alternative data-efficient training strategies and suggesting improvements similar to what would require substantially more labeled data in our learning-curve analysis. Compared to prior methods, our edits are stronger, produce fewer artifacts, and preserve identity better.
Figures & tables
Fig. 1: Distribution of AU occurrence labels in datasets. Naturally occurring distribution of AU labels in real data (DISFA [ 9 , 10 ] ) is highly skewed (black). In contrast, our method for controlled editing/synthesis allows for generation of datasets with a balanced distribution (green).
Fig. 2: Overview of our method. (Top-Left) Learning linear edit directions on semantic codes, where AU models are conditioned on other AUs. Afterwards, AU directions are projected on other possible nuisance attributes. (Top-Right) Editing existing images: encode , pick target AU, obtain disentangled direction , set , decode with original . (Bottom) Synthesizing new faces: sample from DiffAE, optionally accept by demographic predictors, neutralize via the neutralization model by optimizing Equation ( 1 ), then edit and decode.
Fig. 3: An illustrative DAG showing an example of AU relations. The intensity of AU 1 is defined by the reconstructed image, but also by AU 2 as they are often activated together. Conditioning on a correlated AU blocks leakage from that AU to the target AU. At the same time attributes that are influenced by AU1, called colliders (e.g., surprised ), should not be conditioned upon to avoid opening spurious paths.
Fig. 4: Examples of controlled AU editing (top half) and controlled synthesis of new faces (bottom half). Editing examples (top) uses existing neutral images from the DISFA dataset [ 9 ] , where reconstruction is obtained by encoding and decoding without editing. Synthesis examples (bottom) contains random identities sampled from DiffAE containing arbitrary facial expressions. which is first neutralized by the non-linear procedure to suppress pre-existing expressions. Thereafter, each AU is modified by using the proposed method, corresponding to FACS activation level E. Editing of images is largely without major artifacts, and neutralization succeeds in deactivating AUs; but complex expressions may leave small residuals, and editing may slightly alter background of synthetic images.
Fig. 5: (Right) Comparison of correlation between detected AUs in the real DISFA data (top/red) and the edited/synthetic data (bottom/blue), as estimated by an AU detection tool (FaceReader [ 38 ] ) as a stand-in for human annotators. The color heatmap represents the amplitude of the correlation, while the green/yellow/red markers in the edited/synthetic data table denote the intensity of change w.r.t. correlations in the real data (green = reduction, red = increase, yellow = no significant change). Detected AUs in the edited/synthetic data are much less correlated with each other on average (0.09) as compared to the detected AUs in the real data (0.16). (Left) For reference, the inter-AU correlations between the ground truth labels in DISFA are also shown. These exhibit similar patterns as the detector’s AU estimates on the real images, suggesting limited influence of detector bias.
Fig. 6: Box plots showing AU detection accuracy (F1 scores, averaged over AUs) for models trained with (Real + Generated) and without (Real only) generated augmentation on DISFA [ 9 , 10 ] (left), as well as cross-dataset evaluation on FEAFA [ 31 ] and BP4D [ 32 ] (right). Generated augmentation consistently improves F1 on all three benchmarks.
Fig. 7: AU detection accuracy (F1) of baseline models trained on varying amounts of real data (learning curve). The horizontal line shows performance with generated augmentation; error bars denote standard deviation over five runs. A POW3 fit [ 41 ] suggests more real data is needed to match the augmented accuracy.
Fig. 8: Difference in validation set false positive rates (FPRs) for AU pairs (where one AU is absent while another is present) between models trained with and without our generated augmentation technique. The green, yellow, and red markers represent the intensity of change (green = reduction, red = increase, yellow = no significant change). These false positives are a sign of the model over-exploiting the entanglements/correlations between AUs to make predictions. Training with our generated augmentation results in disentangled learning: yielding a lower false positive rate for a majority of AUs as compared to training with real data alone, with an average reduction of 7.4% points.
Fig. 9: Qualitative comparison of facial expression editing from our method with StyleAU [ 28 ] , StyleGAN-NADA [ 20 ] and MagicFace [ 8 ] (image sourced from [ 28 ] ). Editing is done at various intensity levels (increasing from top to bottom) loosely corresponding to FACS AU intensity levels from A to E: StyleAU , StyleGAN-NADA , MagicFace , and our method . For StyleGAN-NADA, the images are generated with target class: “ face with [AU description] increased by [intensity] ”. Our method retains the AUs in the original face better, while also often performing a strong plausible edit and introducing fewer artifacts under single and multi-AU editing.
Fig. 10: Comparison of AU-editing fidelity (left) and identity preservation (right) across manipulation methods as a function of the number of simultaneously edited AUs. Left: MAE between AU activations estimated on the edited image and on an identity-matched real target image exhibiting the desired AU enactment. Right: identity drift measured as the cosine similarity distance between face recognition embeddings of the edited image and the corresponding real target image (same identity), with the purple line indicating the face recognition threshold. Legend: Error bars denote standard error. Across all AU counts, our method achieves the lowest AU error, indicating the closest match to target AU configurations, while maintaining low identity drift. As expected, identity drift increases with the number of edited AUs, but our edits remain within the recognition threshold.
Fig. 11: Comparison of AU-detection accuracies (averaged F1 scores over AUs; top) and false positive rates of AU pairs (where one AU is absent while another is present; bottom) of models trained using different data-efficient training setups. Combining generated augmentation with reweighing does not yield further improvement in accuracy, but combining with pretraining yields the highest accuracy. While reweighing and pretraining increasingly lower the AU-pair FPR, the largest improvement is caused by the introduction of our generated augmentation.
Fig. 12: Qualitative comparison showcasing the effect of conditioning (Cond.) vs projecting on other AUs (AU Proj.) and the effect of projecting conditioned models on other entangled attributes (Cond.+Proj.). Brackets under the edited AU names denote the strengths and directions of the edits: 1 approximately corresponds to FACS intensity level E. Conditioning or projection reduces entanglement with other non-target AUs (e.g., AU1–AU2, AU6–AU12). Conditioning tends to preserve mild, plausible overlapping effect, while projection removes them entirely. Furthermore, projecting on nuisance attributes like glasses removes entanglement with them.
| Baseline | Conditioning | Cond. + Projection | |
|---|---|---|---|
| SSIM |
TABLE I: Effect of conditioning and projection on fine-grained edit fidelity, measured by SSIM w.r.t real target image (mean standard error). Conditioning and projection progressively improve SSIM over the baseline.
Fig. 13: Box-plot comparison of the AU detection accuracy (F1 score) of models trained using different data augmentation strategies. The best performance is achieved from augmenting with both edited and synthetic images, while augmenting only with either one of them yields a comparatively lower accuracy (that is still higher than using real data only).
Fig. 14: Neural network architecture of the neutralization model . Fully Connected layers are abbreviated with FC. After each linear layer, the SiLU [ 47 ] activation function is applied. The network splits into two branches. A dynamic branch that leads to AU predictions, and a static part that leads to the prediction of the other static features. Finally, each attribute has a FC layer of 512, only used for predicting that specific attribute.
More in this topic
Synthetic Data Augmentation
Synthetic data augmentation creates or modifies training examples to improve learning when real data are scarce, imbalanced, costly, or sensitive.
See all 453 papers