Beyond Classification: Task-Dependent Learnability under Privacy-Motivated Image Transformations
Authors: Leon Ranke, Wolfgang Hübner, Ronny Hug, Michael Arens, Jürgen Beyerer
Organizations: Fraunhofer Institute of Optronics, System Technologies and Image Exploitation (IOSB), Gutleuthausstraße 1, 76275 Ettlingen, Germany · Karlsruhe Institute of Technology (KIT)
Privacy-Enhancing Technologies (PETs) in computer vision often rely on noise or image perturbations to protect visual data while securely processing it, creating a trade-off between task performance and protection. This trade-off is commonly evaluated using image classification, which primarily captures semantic separability and remains robust despite significant geometric, spatial layout or local boundary alterations. As a result, it is too simplistic as a proxy for generic vision tasks. Exhaustive downstream-task evaluation, however, is computationally expensive because models must often be trained for each PET transformation and parameter setting. We therefore propose a compute-aware multi-task protocol for evaluating PETs in model training. It combines lightweight proxy tasks that target complementary aspects of visual structure while remaining simple and fast to compute. Across irreversible privacy transformations, key-based block primitives, and learnable image encryption schemes, we demonstrate that PETs with similar classification accuracy can differ substantially on other tasks. The outcomes highlight the need for PET evaluation protocols that move beyond classification-only reporting.
Learnable Image Encryption for Vision Transformers [ 17 ]
CIFAR-10 [ 23 ] , Tiny-ImageNet [ 27 ]
Table 1: Datasets used to evaluate model utility in representation-altering PETs.
Name
Abbreviation
Range
Heuristic target
Perceptual Leakage
Mean Squared Error
MSE
[0,(L−1)2]
6L2−1
Peak Signal-to-Noise Ratio (dB)
PSNR
[0,∞)
10log10L+16(L−1)
Structural Similarity Index Measure [ 49 ]
SSIM
[−1,1]
0
Learned Perceptual Image Patch Similarity [ 52 ]
LPIPS
[0,∞)
↑
Mutual Information (bits)
MI
[0,log2(L)]
0
Table 2: Image-domain obfuscation and leakage indicators for RGB images. Metrics are computed per channel (R, G, B) and averaged. Intensities are in {0,…,L−1} , with L=256 throughout.
Figure 1: Qualitative examples of transformation families across increasing scale.
Task
Probes
Approx. Time
Classification
semantic separability
≈1.4 h
Angle Prediction
global orientation / geometric consistency
≈0.8 h
Jigsaw Puzzle Solving
spatial compatibility / local structure
≈9.4 h
Table 3: Overview of investigated proxy tasks, including approximate training time.
Figure 2: Relationship between the puzzle size k and the transformation block size B .
Figure 3: Classification accuracy against transformation strength. Accuracy remains above chance for all transformations, indicating that semantic separability is preserved even under substantial image-domain alteration.
Figure 4: Mean angular error against transform strength (lower is better). Prediction remains robust under transformations that preserve coarse global structure, such as moderate blur, but degrades strongly when spatial order is removed at larger scales or when local and global disruptions are combined.
Figure 5: Jigsaw puzzle-solving accuracy for puzzles of different sizes; dashed coloured curves indicate evaluation with a different key from the training key. Puzzle solving exposes spatial placement cues that classification does not capture. LOI and some block-shuffling settings degrade strongly when spatial order is removed at puzzle-relevant scales, whereas several key-based transformations remain highly solvable.
Figure 6: Obfuscation-utility trade-off. Normalised task utility against normalised LPIPS. For angle prediction, the utility direction is inverted, since lower errors are better.
Hyperparameter
Value
Dataset
Animal Species Classification dataset [ 1 ]
Number of classes
15
Data augmentation
None
Model architecture
ResNet-34 [ 15 ]
Initial weights
Random (trained from scratch)
Input resolution
3×416×416
Table 4: Hyperparameters for Animal Species Classification.
Hyperparameter
Value
Dataset
Animal Species Classification dataset [ 1 ]
Angle sampling
α1,α2∼U[−180∘,180∘]
Data augmentation
None
Model architecture
CNN ( 2× conv, max-pool, 4× linear)
Initial weights
Random (trained from scratch)
Input resolution
3×304×304
Table 5: Hyperparameters for Relative Angle Prediction.
Hyperparameter
Value
Dataset
Animal Species Classification dataset [ 1 ]
Puzzle sizes k
{4,6,8}
Permutations
Sampled uniformly at random during training
Data augmentation
None
Model architecture
JPDVT [ 31 ]
Diffusion timesteps T
1000
Table 6: Hyperparameters for Jigsaw Puzzle Solving.
Figure 7: Obfuscation–utility trade-off: normalised task utility against normalised MSE.
Figure 8: Obfuscation–utility trade-off: normalised task utility against normalised PSNR.
Figure 9: Obfuscation–utility trade-off: normalised task utility against normalised SSIM.
Figure 10: Obfuscation–utility trade-off: normalised task utility against normalised MI.
Robustness evaluation must examine diverse visual perturbations, while benchmarks cover only some real-world conditions and physical testing is costly. Paired evaluations link clean and perturbed predictions for the same image, capturing changes in correctness, confidence, and acceptance beyond aggregate accuracy. We investigate how this image correspondence supports two needs in robustness evaluation: interpreting paired evaluation results and prioritizing samples for physical testing. To interpret paired evaluation results, we fix both sets of prediction records and vary their correspondence within each class. We prove that classwise correct-correct counts give the same sharp bounds on lost acceptance and mean true-class probability decrease among retained-correct inputs as any feasible five-state refinement. Distinguishing persistent from changed wrong answers can further constrain accepted-error transitions, while shared correspondence can establish policy orderings left unresolved by separate cost intervals. To prioritize samples for physical testing, we retain each image's synthetic responses and rank clean-correct images by their mean true-class probability under corruption. Across 44 classifiers, testing the highest-risk 20% finds 67% and 45% of failures under mild screen and print recaptures, versus 58% and 36% for clean confidence and 60% and 37% for an equal-size natural-transformation average. With both probability averaging and an A3Rank scoring adaptation, the tested corruption set yields higher mean failure recall than the natural-transform set; differences between scores depend on the source and budget. Together, these findings show that the value of correspondence depends on the evaluation objective: classwise counts suffice for specified reliability bounds, while image-specific synthetic responses improve the allocation of physical tests within the evaluated pool.
We propose a novel object detection method that enables us to protect sensitive visual information of test images. Previous studies considering visual information protection focus on image classification tasks. This paper proposes an object detection method using perceptual encryption for the first time. The proposed method can achieve almost the same accuracy as that of models without any protection by utilizing the embedding structure of the Vision Transformer (ViT) and a domain adaptation technique with keys. In experiments, the effectiveness of the proposed method is verified in terms of accuracy and visual protection under the use of ViTdet, which is a ViT-based object detection model.
Homare Sueyoshi, Kiyoshi Nishikawa, Hitoshi Kiya
Tokyo Metropolitan University 6-6, Asahigaoka, Hino, Tokyo, Japan
Privacy preserving machine learning deployments in sensitive deep learning applications; from medical imaging to autonomous systems; increasingly require combining multiple techniques. Yet, practitioners lack systematic guidance to assess the synergistic and non-additive interactions of these hybrid configurations, relying instead on isolated technique analysis that misses critical system level interactions. We introduce PrivacyBench, a benchmarking framework that reveals striking failures in privacy technique combinations with severe deployment implications. Through systematic evaluation across ResNet18 and ViT models on medical datasets, we uncover that FL + DP combinations exhibit severe convergence failure, with accuracy dropping from 98% to 13% while compute costs and energy consumption substantially increase. In contrast, FL + SMPC maintains near-baseline performance with modest overhead. Our framework provides the first systematic platform for evaluating privacy-utility-cost trade-offs through automated YAML configuration, resource monitoring, and reproducible experimental protocols. PrivacyBench enables practitioners to identify problematic technique interactions before deployment, moving privacy-preserving computer vision from ad-hoc evaluation toward principled systems design. These findings demonstrate that privacy techniques cannot be composed arbitrarily and provide critical guidance for robust deployment in resource-constrained environments.
Nnaemeka Obiefuna, Samuel Oyeneye, Similoluwa Odunaiya +2
1ML Collective · 2Friedrich-Alexander-Universität Erlangen-Nürnberg · 3Carnegie Mellon University