Neural Network Robustness

Follow this topic Ask about this topic

Neural network robustness asks whether a model keeps making reliable predictions when inputs, training data, or computing hardware differ from expectations. Common approaches include adversarial training, corruption and shift testing, Lipschitz constraints, randomized smoothing, and formal verification that certifies behavior within a defined perturbation range. Work spans image, language, audio, scientific, and control systems, with evaluations increasingly examining confidence, internal representations, adaptive attacks, and deployment conditions rather than accuracy alone. A central tension is that improving worst-case or certified robustness can reduce clean accuracy, increase computational cost, or leave other kinds of failures untouched.

Latest papers

238 papers24 papers in the last four weeks, up 85% on the four weeks before. 0.1% of all new papers.

Feb 25, 2026

Robustness in sparse artificial neural networks trained with adaptive topology

We investigate the robustness of sparse artificial neural networks trained with adaptive topology. We focus on a simple yet effective architecture consisting of three sparse layers with 99% sparsity followed by a dense layer, applied to image classification tasks such as MNIST and Fashion MNIST. By updating the topology of the sparse layers between each epoch, we achieve competitive accuracy despite the significantly reduced number of weights. Our primary contribution is a detailed analysis of the robustness of these networks, exploring their performance under various perturbations including random link removal, adversarial attack, and link weight shuffling. Through extensive experiments, we demonstrate that adaptive topology not only enhances efficiency but also maintains robustness. This work highlights the potential of adaptive sparse networks as a promising direction for developing efficient and reliable deep learning models.

Bendegúz Sulyok, Gergely Palla, Filippo Radicchi +1

Dept. of Biological Physics, Eötvös Loránd University, Budapest, Hungary · Semmelweis University, Faculty of Health and Public Administration, Health Services Management Training Centre, Budapest, Hungary · Center for Complex Networks and Systems Research Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington, USA

Dynamic Sparse Training Sparse Neural Networks Neural Network Robustness Adversarial Robustness

Feb 25, 2026

RECAP: Local Hebbian Prototype Learning as a Self-Organizing Readout for Reservoir Dynamics

Robust perception in brains is often attributed to high-dimensional population activity together with local plasticity mechanisms that reinforce recurring structure. In contrast, most modern image recognition systems are trained by error backpropagation and end-to-end gradient optimization, which are not naturally aligned with local computation and local plasticity. We introduce RECAP (Reservoir Computing with Hebbian Co-Activation Prototypes), a bio-inspired learning strategy for robust image classification that couples untrained reservoir dynamics with a self-organizing Hebbian prototype readout. RECAP discretizes time-averaged reservoir responses into activation levels, constructs a co-activation mask over reservoir unit pairs, and incrementally updates class-wise prototype matrices via a Hebbian-like potentiation-decay rule. Inference is performed by overlap-based prototype matching. The method avoids error backpropagation and is naturally compatible with online prototype updates. We illustrate the resulting robustness behavior on MNIST-C, where RECAP remains robust under diverse corruptions without exposure to corrupted training samples.

Heng Zhang

International Research Center for Neurointelligence (IRCN), University of Tokyo, Japan

Image Classification Prototype-Based Classification Neural Network Robustness Reservoir Computing Hebbian Learning

Feb 24, 2026

Computing a Characteristic Orientation for Rotation-Independent Image Analysis

Handling geometric transformations, particularly rotations, remains a challenge in deep learning for computer vision. Standard neural networks lack inherent rotation invariance and typically rely on data augmentation or architectural modifications to improve robustness. Although effective, these approaches increase computational demands, require specialised implementations, or alter network structures, limiting their applicability. This paper introduces General Intensity Direction (GID), a preprocessing method that improves rotation robustness without modifying the network architecture. The method estimates a global orientation for each image and aligns it to a canonical reference frame, allowing standard models to process inputs more consistently across different rotations. Unlike moment-based approaches that extract invariant descriptors, this method directly transforms the image while preserving spatial structure, making it compatible with convolutional networks. Experimental evaluation on the rotated MNIST dataset shows that the proposed method achieves higher accuracy than state-of-the-art rotation-invariant architectures. Additional experiments on the CIFAR-10 dataset, confirm that the method remains effective under more complex conditions.

Cristian Valero-Abundio, Emilio Sansano-Sansano, Raúl Montoliu +1

Institute of New Imaging Technologies, Universitat Jaume I, 12071 Castellón, Spain · Mathematics Department, Universitat Jaume I, 12071 Castellón, Spain

Neural Network Robustness Image Classification

Feb 21, 2026

Theoretical Lower Bounds on the Robustness of Deep ReLU Networks

We present a theoretical study of the robustness of parameterized neural networks to random input perturbations. Specifically, we analyze local robustness by quantifying the probability that a random L_2-perturbation of a given input results in a correct classification. For deep ReLU networks, we derive lower bounds on local robustness by combining tools from high-dimensional geometry, in particular concentration of measure, with a new characterization of the geometric structure induced by their input-output functions. We prove that each convex polyhedral region in the partition of the input space induced by a ReLU network has at most as many faces as there are network units, regardless of the network depth or architecture. This geometric property serves as the key ingredient in our robustness analysis. Finally, we analyze how local robustness scales with input dimension and characterize the sets of inputs whose neighborhoods are most likely to contain adversarial examples. We show that the width of decision-boundary neighborhoods containing vulnerable points shrinks rapidly as dimension increases and grows only logarithmically with the number of network units. We also discuss the volume of a set of vulnerable points in terms of approximately space-filling shapes of decision boundaries.

Věra Kůrková

Institute of Computer Science of the Czech Academy of Sciences, Pod Vodárenskou věží 2, 18207 Prague, Czech Republic

Neural Network Robustness Adversarial Robustness ReLU Neural Networks

Feb 18, 2026

The Weight of a Bit: EMFI Sensitivity Analysis of Embedded Deep Learning Models

Fault injection attacks on embedded neural network models have been shown as a potent threat. Numerous works studied resilience of models from various points of view. As of now, there is no comprehensive study that would evaluate the influence of number representations used for model parameters against electromagnetic fault injection (EMFI) attacks. In this paper, we investigate how four different number representations influence the success of an EMFI attack on embedded neural network models. We chose two common floating-point representations (32-bit, and 16-bit), and two integer representations (8-bit, and 4-bit). We deployed four common image classifiers, ResNet-18, ResNet-34, ResNet-50, and VGG-11, on an embedded memory chip, and utilized a low-cost EMFI platform to trigger faults. Beyond accuracy evaluation, we characterize the injected fault pattern by analyzing the bit error rate, the spatial distribution of corrupted bytes, and the prevalence of 0xFE/0xFF byte values across formats, identifying the mechanisms responsible for the observed differences in resilience. Our results show that while floating-point representations exhibit almost a complete degradation in accuracy (Top-1 and Top-5) after a single fault injection, integer representations offer better resistance overall. In particular, the 8-bit representation on a relatively large network (VGG-11) retains Top-1 accuracy of around 70% and Top-5 at around 90%.

Jakub Breier, Štefan Kučerák, Xiaolu Hou

TTControl GmbH, Vienna, Austria. · the Faculty of Informatics and Information Technologies, Slovak University of Technology, Slovakia. · the State Key Laboratory of Blockchain and Data Security , Zhejiang University.

Neural Network Robustness Edge ML

Feb 17, 2026

R $^2$ Energy: A Large-Scale Benchmark for Robust Renewable Energy Forecasting under Diverse and Extreme Conditions

The rapid expansion of renewable energy, particularly wind and solar power, has made reliable forecasting critical for power system operations. While recent deep learning models have achieved strong average accuracy, the increasing frequency and intensity of climate-driven extreme weather events pose severe threats to grid stability and operational security. Consequently, developing robust forecasting models that can withstand volatile conditions has become a paramount challenge. In this paper, we present R

^2

Energy, a large-scale benchmark for NWP-assisted renewable energy forecasting. It comprises over 10.7 million high-fidelity hourly records from 902 wind and solar stations across four provinces in China, providing the diverse meteorological conditions necessary to capture the wide-ranging variability of renewable generation. We further establish a standardized, leakage-free forecasting paradigm that grants all models identical access to future Numerical Weather Prediction (NWP) signals, enabling fair and reproducible comparison across state-of-the-art representative forecasting architectures. Beyond aggregate accuracy, we incorporate regime-wise evaluation with expert-aligned extreme weather annotations, uncovering a critical ``robustness gap'' typically obscured by average metrics. This gap reveals a stark robustness-complexity trade-off: under extreme conditions, a model's reliability is driven by its meteorological integration strategy rather than its architectural complexity. R

^2

Energy provides a principled foundation for evaluating and developing forecasting models for safety-critical power system applications.

Zhi Sheng, Yuan Yuan, Guozhen Zhang +1

Center for Urban Science and Computation, Tsinghua University · Courant Institute of Mathematical Sciences, New York University

Forecasting Benchmarks Time Series Forecasting Wind Power Forecasting Solar Power Forecasting Neural Network Robustness

Feb 6, 2026

Same Answer, Different Representations: Hidden instability in VLMs

The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial internal representation drift; for perturbations such as text overlays, this drift approaches the magnitude of inter-image variability, indicating that representations move to regions typically occupied by unrelated inputs despite unchanged outputs. Second, robustness does not improve with scale; larger models achieve higher accuracy but exhibit equal or greater sensitivity, consistent with sharper yet more fragile decision boundaries. Third, we find that perturbations affect tasks differently: they harm reasoning when they disrupt how models combine coarse and fine visual cues, but on the hallucination benchmarks, they can reduce false positives by making models generate more conservative answers.

Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6

Sapienza University of Rome · University of Edinburgh · University of Oxford +5

VLM Evaluation VLM Robustness Vision-Language Models Neural Network Robustness Adversarial Robustness Representation Geometry in VLMs

Feb 6, 2026

Perturbing the Phase: Analyzing Adversarial Robustness of Complex-Valued Neural Networks

Complex-valued neural networks (CVNNs) are rising in popularity for all kinds of applications. To safely use CVNNs in practice, analyzing their robustness against outliers is crucial. One well known technique to understand the behavior of deep neural networks is to investigate their behavior under adversarial attacks, which can be seen as worst case minimal perturbations. We design Phase Attacks, a kind of attack specifically targeting the phase information of complex-valued inputs. Additionally, we derive complex-valued versions of commonly used adversarial attacks. We show that in some scenarios CVNNs are more robust than RVNNs and that both are very susceptible to phase changes with the Phase Attacks decreasing the model performance more, than equally strong regular attacks, which can attack both phase and magnitude.

Florian Eilers, Christof Duhme, Xiaoyi Jiang

Department of Computer Science University of Münster Münster, Germany

Adversarial Attacks Neural Network Robustness Adversarial Robustness Complex-Valued Neural Networks

Feb 3, 2026

Robustness as an Emergent Property of Task Performance

Robustness is widely viewed as a key challenge for real-world applications. However, because current research focuses only on difficult tasks, it partially captures real-world readiness. In this paper, we argue and verify that robustness, defined as consistency across semantically equivalent inputs, closely follows task difficulty: once models master a task, robustness emerges naturally. Through an empirical analysis of multiple models across diverse datasets and configurations (e.g., paraphrases, temperature changes), we observe a strong positive correlation between task performance and robustness. Furthermore, our findings indicate that robustness is driven primarily by task-specific competence rather than inherent model attributes, challenging the common view of robustness as an independent capability. This perspective implies that as tasks mature and model performance saturates, robustness on those tasks will similarly emerge. For researchers, this suggests that explicit efforts to measure robustness may deserve reduced emphasis, as robustness is likely to improve alongside performance. For practitioners, it signals that while many existing benchmarks are still unstable, models are already reliable on earlier tasks and suitable for deployment.

Shir Ashury-Tahan, Ariel Gera, Elron Bandel +2

IBM Research · MIT · Weizmann Institute of Science

Neural Network Robustness Language Model Robustness

Dec 1, 2025

Dual Randomized Smoothing: Beyond Global Noise Variance

Randomized Smoothing (RS) is a prominent technique for certifying the robustness of neural networks against adversarial perturbations. With RS, achieving high accuracy at small radii requires a small noise variance, while achieving high accuracy at large radii requires a large noise variance. However, the global noise variance used in the standard RS formulation leads to a fundamental limitation: there exists no global noise variance that simultaneously achieves strong performance at both small and large radii. To break through the global variance limitation, we propose a dual RS framework which enables input-dependent noise variances. To achieve that, we first prove that RS remains valid with input-dependent noise variances, provided the variance is locally constant around each input. Building on this result, we introduce two components: (i) a variance estimator predicts an optimal noise variance for each input, (ii) this estimated variance is then used by a standard RS classifier. The variance estimator is independently smoothed via RS to ensure local constancy, enabling flexible design. We also introduce training strategies to iteratively optimize the two components. Experiments on CIFAR-10 demonstrate that our dual RS method provides strong performance for both small and large radii-unattainable with global noise variance-while incurring only a 60% computational overhead at inference. Moreover, it outperforms prior input-dependent noise approaches across most radii, with gains at radii 0.5, 0.75, and 1.0 of 15.6%, 20.0%, and 15.7%. On ImageNet, dual RS remains effective across all radii, with advantages of 8.6%, 17.1%, and 9.1% at radii 0.5, 1.0, and 1.5. Additionally, the dual RS framework provides a routing perspective for certified robustness, improving the accuracy-robustness trade-off with off-the-shelf expert RS models.

Chenhao Sun, Yuhao Mao, Martin Vechev

Department of Computer Science, ETH Zürich, Switzerland

Randomized Smoothing Neural Network Robustness Adversarial Robustness Certified Robustness Noise Robustness

Nov 24, 2025

Diagnosing Corruption-Induced Reliability Failures in Vision-Language Models

Visual corruptions can change vision--language model (VLM) behavior in ways that top-1 accuracy does not capture. A model may keep the same answer while losing distributional support, or improve accuracy through unstable wrong-to-correct changes. We introduce Bench-C, a controlled multiple-choice testbed for studying these effects. It selects semantically diverse samples whose predictions respond to corruption, and evaluates them under 19 corruption types and five severity levels. To measure how corruption changes the option distribution, we introduce the Robustness Alignment Score (RAS), which combines confidence-correctness alignment with uncertainty direction. We further separate originally correct samples from originally wrong samples, and track whether changes are temporary or persistent across severity. Experiments across 13 VLMs reveal a counterintuitive pattern: mild corruptions can improve top-1 accuracy while degrading prediction structure. These failures include silent degradation, erroneous overconfidence, and severity-dependent persistence. Bench-C therefore supports robustness evaluation that goes beyond final answers and attributes where reliability changes occur. Code and data are available at https://github.com/xiangjieSui/Bench-C.

Xiangjie Sui, Songyang Li, Hanwei Zhu +3

City University of Macau, Macau SAR, China · Jiangxi University of Finance and Economics, Nanchang, China · Nanyang Technological University, Singapore

VLM Evaluation VLM Robustness Probability Calibration Image Corruption Robustness Neural Network Robustness Corruption Robustness

Nov 23, 2025

DHAuDS: A Dynamic and Heterogeneous Audio Benchmark for Test-Time Adaptation

Existing Test-time Adaptation (TTA) studies rely heavily on static and homogeneous corruption protocols, such as ImageNet-C and CIFAR-10-C/100-C, leading to inconsistent evaluation settings and potentially inflated robustness estimates that are compared with real-world situations. TTA lacks a standardized evaluation infrastructure capable of modeling realistic heterogeneous acoustic degradation. We introduce DHAuDS, a standardized benchmark suite for evaluating audio classification TTA robustness under dynamic corruption severity and heterogeneous noise mixtures. Rather than proposing a new TTA algorithm, DHAuDS focuses on exposing robustness limitations that remain hidden under conventional fixed-noise evaluation protocols.

Weichuang Shao, Iman Yi Liao, Tomas Henrique Bode Maul +1

School of Computer and Mathematical Sciences, University of Nottingham Malaysia, Semenyih, Malaysia

Test-Time Adaptation Neural Network Robustness Noise Robustness Audio Classification

Nov 13, 2025

destroR: A Benchmark and Adversarial-Training Defense for Bangla Transfer Models under Meaning-Preserving Attacks

Transformer-based transfer models now dominate Bangla sentiment classification, yet their adversarial robustness remains largely unexamined, and no prior study pairs a Bangla attack suite with a defense that measurably recovers robustness. We address this gap with destroR, a unified pipeline for evaluating and hardening Bangla text classifiers. First, we introduce three meaning-preserving Bangla attack recipes a paraphrase attack, a back-translation attack, and a one-hot word-swap attack that perturb inputs while regenerating fluent, semantically faithful sentences, inducing model prediction perplexity rather than input noise. Second, we construct a robustness benchmark that evaluates five transfer models (BanglaBERT, BanglishBERT, XLM-RoBERTa, MuRIL, and IndicBERTv2) across four datasets against five attacks, placing our recipes against two strong word-substitution baselines, TextFooler and BAE, under an identical protocol. Third, we harden every model through adversarial training and report a full robustness matrix. Our analysis yields three findings: word-substitution baselines are more potent than semantically constrained recipes (BAE reaches a 54.2% attack success rate); adversarial training on the union of all attack families lowers residual attack success for every attack; and, contrary to expectation, the Indic-multilingual MuRIL backbone is markedly more robust than the Bangla-dedicated models. All models, adversarial data, and code are released for full reproducibility.

Saadat Rafid Ahmed, Rubayet Shareen, Radoan Sharkar +3

Dept. of Computer Science and Engineering, BRAC University

Adversarial Text Attacks Adversarial Robustness of Language Models Adversarial Training Sentiment Analysis Neural Network Robustness Adversarial Robustness+2

Oct 21, 2025

Solver-Integrated Adversarial Attacking and Training of Neural Operators

Neural operators are widely used as fast surrogates for numerical PDE solvers, mapping input functions to solution functions. However, their generalizability and robustness are not yet clearly defined in the operator-learning setting, which differs from traditional adversarial robustness definitions. This paper studies the generalizability and robustness of a learned neural operator from a solver-integrated perspective, addressing the challenge that the output of a learned operator and a numerical solver tends to change in tandem under input perturbation. First, we formalize the definition of generalization and robustness through a model-solver error operator, identifying fixed-input model-solver loss as generalization metric, and norm-bounded adversarial attack loss increase and Jacobian-error function norm as robustness metric. Second, we identify the solver-integrated adversarial attack as appropriate for PDE operator learning and show why model-only or fixed-ground-truth attacks can be insufficient when the solver output also changes with the input. Third, we develop solver-integrated adversarial training methods for neural operators. Experiments on representative PDE benchmarks show that this solver-integrated adversarial training clearly improves both generalizability and robustness. Deeper solver integration yields more effective attacks, more informative samples, and more efficient training than less integrated alternatives. These results provide a general framework for robust operator training and automatic sample selection without heavy manual intervention. More broadly, the formulation applies to adversarial regression whenever a ground-truth oracle can evaluate, and ideally differentiate, the true input-output map; PDE operator learning is one such case.

Yifei Sun

Adversarial Training Adversarial Attacks Neural Network Robustness Adversarial Robustness PDE Operator Learning Neural Operators

Sep 29, 2025

Improved Robustness from Biologically Inspired Sparse Contrast Representations

Deep neural networks surpass humans on many vision benchmarks, yet remain far less robust to distribution shifts such as illumination and weather changes. Existing approaches address this challenge by additional training data, extensive augmentation, architectural modifications, or test-time adaptation. In this work, we explore a complementary direction: inspired by the human retina, we propose a fixed, model-agnostic preprocessing module that extracts signals that are more stable with respect to variations of illumination. Our method combines color remapping with local contrast extraction, producing sparse representations that emphasize structural features. We study its impact on semantic segmentation by training on Cityscapes and evaluating generalization under adverse conditions on Dark Zurich and ACDC. Our results show that the biologically inspired preprocessing preserves in-distribution performance while consistently improving robustness in challenging lighting scenarios, such as nighttime, where annotated training data are scarce. Moreover, the segmentation accuracy remains stable even when the contrast-based representation is sparsified by up to 70%. These gains suggest that rethinking the input representation itself can improve robustness while also opening opportunities for lower-latency, transmission-aware imaging sensors when sparsity can be exploited close to acquisition.

Lorena Stracke, Lia Nimmermann, Shashank Agnihotri +3

Media Systems, University of Siegen, Germany · Data and Web Science Group, University of Mannheim, Germany · Analogue Circuits and Image Sensors, University of Siegen, Germany +1

Distribution Shift Robustness Image Segmentation Image Corruption Robustness Visual Representation Learning Neural Network Robustness Semantic Segmentation

Sep 28, 2025

Quantifying How Training Gradient Sparsity Affect Spiking Neural Network Accuracy And Robustness

Spiking Neural Networks (SNNs) have recently received increasing attention in both computational neuroscience and artificial intelligence owing to their potential for energy-efficient computation and reduced memory requirements. Despite these advantages, improving adversarial robustness in SNNs (particularly for vision-based applications) remains an emerging and relatively underexplored research problem. Recent work has suggested that encouraging sparse gradients can act as a regularization mechanism to improve resistance against adversarial perturbations. In this study, we report an unexpected observation: under certain architectural configurations, SNNs inherently exhibit sparse gradients and can attain state-of-the-art adversarial defense performance without requiring any explicit regularization strategy. Further investigation reveals an inherent trade-off between robustness and generalization. Specifically, increased gradient sparsity enhances resistance to adversarial attacks but may reduce the model's generalization capability, whereas denser gradients tend to improve generalization while simultaneously increasing susceptibility to adversarial perturbations. These findings provide new perspectives on the role of gradient sparsity in the training dynamics of SNNs.

Nhan Trong Luu, Duong Trung Luu

College of Information and Communication Technology, Can Tho University, Vietnam · College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam · Center of Digital Transformation and Communication, Can Tho University, Vietnam

Neural Network Generalization Neural Network Robustness Adversarial Robustness Spiking Neural Networks Neural Network Training Dynamics

Jul 23, 2025

On the Interaction of Compressibility and Adversarial Robustness

As demands for resource efficiency and safety in modern neural networks intensify, substantial research effort has gone into model compression and adversarial robustness. Yet despite progress on each in isolation, a systematic understanding of how compressibility shapes robustness remains elusive. In this paper, we develop a principled framework to analyze how different forms of structured compressibility - such as neuron-level and spectral compressibility - affect adversarial robustness. We show that structured compressibility can induce a small number of highly sensitive directions in the representation space, which adversaries can exploit to construct effective perturbations. Our analysis yields a robustness bound that reveals how neuron and spectral compressibility impact

\ell_\infty

and

\ell_2

robustness via their effects on the learned representations. Crucially, the vulnerabilities we identify arise irrespective of how compressibility is achieved - whether via regularization, architectural bias, or learning dynamics. Through empirical evaluations across synthetic and realistic tasks, we confirm our theoretical predictions, and further demonstrate that these vulnerabilities persist under adversarial training and transfer learning, and contribute to the emergence of universal adversarial examples. Our findings show a fundamental tension between structured compressibility and robustness and highlight new pathways for designing models that are efficient and safe.

Melih Barsbey, Antônio H. Ribeiro, Umut Şimşekli +1

Department of Computing, Imperial College London, UK · Department of Information Technology, Uppsala University, Sweden · INRIA, CNRS, Département d’Informatique de l’Ecole Normale Supérieure / PSL, France

Neural Network Robustness Adversarial Robustness Adversarial Perturbations Neural Network Compression

Apr 1, 2025

R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networks

This paper presents the Robust Recurrent Deep Network (R2DN), a scalable parameterization of stable and robust recurrent neural networks for machine learning and data-driven control. We construct R2DNs as the feedback interconnection of a linear time-invariant system and a 1-Lipschitz deep feedforward network, and directly parameterize the weights so that our models are stable (contracting) and robust to input perturbations (Lipschitz) by design. Our parameterization uses a structure similar to the recurrent equilibrium network (REN), but without having to iteratively solve an equilibrium layer at each time-step. This speeds up model inference and training on GPUs, and makes it computationally feasible to scale up the network size and input sequence length in comparison to RENs. We compare R2DNs to RENs on representative problems in nonlinear system identification, observer design, learning-based feedback control, and sequential image classification. We find that training and inference are up to an order of magnitude faster with similar performance, and that they scale more favorably with respect to model expressivity.

Nicholas H. Barbara, Ruigang Wang, Ian R. Manchester

Australian Centre for Robotics (ACFR) and the School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney, Australia

Neural Network Robustness Recurrent Neural Networks Lipschitz Neural Networks

Mar 19, 2025

Unified Enhancement of the Generalization and Robustness of Language Models via Bi-Stage Optimization

Neural network language models (LMs) are confronted with significant challenges in generalization and robustness. Currently, many studies focus on improving either generalization or robustness in isolation, without methods addressing both aspects simultaneously, which presents a significant challenge in developing LMs that are both robust and generalized. In this paper, we propose a bi-stage optimization framework to uniformly enhance both the generalization and robustness of LMs, termed UEGR. Specifically, during the forward propagation stage, we enrich the output probability distributions of adversarial samples by adaptive dropout to generate diverse sub models, and incorporate JS divergence and adversarial losses of these output distributions to reinforce output stability. During backward propagation stage, we compute parameter saliency scores and selectively update only the most critical parameters to minimize unnecessary deviations and consolidate the model's resilience. Theoretical analysis shows that our framework includes gradient regularization to limit the model's sensitivity to input perturbations and selective parameter updates to flatten the loss landscape, thus improving both generalization and robustness. The experimental results show that our method significantly improves the generalization and robustness of LMs compared to other existing methods across 13 publicly available language datasets, achieving state-of-the-art (SOTA) performance.

Yudao Sun, Juan Yin, Juan Zhao +3

Department of New Networks, Peng Cheng Laboratory, Shenzhen 518055, China · Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore 117576 · School of Data Science and Artificial Intelligence, Chang’an University, Shaanxi, 710064, China

Adversarial Training Neural Network Generalization Neural Network Robustness Adversarial Robustness of Language Models Language Model Robustness

Dec 6, 2024

Robust and efficient enriched representation with intrinsic neuronal heterogeneity

Heterogeneity is a ubiquitous property of many biological systems with profound computational implications. Integrating this heterogeneity into artificial computing systems has recently gained attention. While optimizing neuronal and synaptic heterogeneity for a specific task can achieve state-of-the-art performance, such top-down optimization is biologically implausible, prone to catastrophic forgetting, and both data- and energy-intensive. In contrast, biological organisms possess inherent, task-unspecific, stable heterogeneity and can perform many tasks with minimal metabolic cost. Inspired by this, we study task-agnostic heterogeneity in recurrent neural networks solving hundreds of diverse temporal tasks. Heterogeneity is introduced as intrinsic disorder in neuronal membrane time constants. Our results show that intrinsic heterogeneity markedly improves overall performance and robustness in both rate- and spike-coded networks. Small heterogeneous networks, even without recurrence, match the performance of large, highly recurrent, homogeneous networks at a fraction of the cost and complexity. We suggest that, despite being task-agnostic, heterogeneity enhances representational capacity by diversifying neuronal responses, demonstrated on two machine-learning and one neuroscience benchmark. Our work highlights the utility of intrinsic task-agnostic heterogeneity for versatile, energy-efficient computing--a highly relevant property for the neuromorphic community, where device-to-device variability could be exploited to offset limited training budgets and architectural constraints.

Arash Golmohammadi, Jannik Luboeinski, Christian Tetzlaff

Recurrent Neural Networks Neural Network Robustness Spiking Neural Networks

Oct 28, 2024

Large Pretraining Datasets Don't Guarantee Robustness after Fine-Tuning in Image Classification

Large-scale pretrained models are widely leveraged as foundations for learning new specialized tasks via fine-tuning, with the goal of maintaining the general performance of the model while allowing it to gain new skills. A valuable goal for all such models is robustness: the ability to perform well on out-of-distribution (OOD) tasks. We assess whether fine-tuning preserves the overall robustness of the pretrained model in image classification, and observed that models pretrained on large datasets exhibited strong catastrophic forgetting and loss of OOD generalization. To systematically assess robustness preservation in fine-tuned models, we propose the Robustness Inheritance Benchmark (ImageNet-RIB). The benchmark, which can be applied to any pretrained model, consists of a set of related but distinct OOD (downstream) tasks and involves fine-tuning on one of the OOD tasks in the set then testing on the rest. We find that though continual learning methods help, fine-tuning reduces robustness across pretrained models. Surprisingly, models pretrained on the largest and most diverse datasets (e.g., LAION-2B) exhibit both larger robustness losses and lower absolute robustness after fine-tuning on small datasets, relative to models pretrained on smaller datasets. We observe this collapse in contrastively pretrained (CLIP) models and their fine-tuned variants, where it grows with pretraining scale; the supervised models we test do not exhibit it. These findings suggest that starting with the strongest foundation model is not necessarily the best approach for performance on specialist tasks. https://jd730.github.io/projects/ImageNet-RIB

Jaedong Hwang, Brian Cheung, Zhang-Wei Hong +3

Massachusetts Institute of Technology

Neural Network Robustness OOD Generalization Vision Foundation Model Adaptation Image Classification Catastrophic Forgetting Robust Classification

Oct 10, 2024

How Learning Dynamics Drive Adversarially Robust Generalization?

Despite being widely adopted as a canonical framework for learning robust models, adversarial training suffers from robust overfitting. Existing empirical and theoretical explorations fail to provide a satisfactory mechanistic interpretation of the phenomenon. By modeling adversarial training with momentum SGD as a discrete-time dynamical system, we propose a PAC-Bayesian analytical framework that proves time-resolved robust generalization bounds. Specifically, our framework tracks the closed-form evolution of the posterior mean and covariance under both stationary and non-stationary transient regimes, connecting the model's robust generalization performance to learning rate, local loss geometry, and mini-batch stochastic gradients. By estimating the key quantities associated with the bound, we illustrate the underlying mechanism of robust overfitting. Our framework also shows how adversarial weight perturbation reduces robust generalization gaps by suppressing dominant loss-curvature modes, while suggesting that excessive penalization can be sub-optimal for optimization.

Yuelin Xu, Xiao Zhang

CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Adversarial Training PAC-Bayesian Generalization Bounds Neural Network Robustness Adversarial Robustness Neural Network Training Dynamics

Aug 17, 2024

Training Verifiably Robust Agents Using Set-Based Reinforcement Learning

Reinforcement learning policies parametrized by deep neural networks have achieved strong performance for continuous control, yet even small input perturbations may lead to unpredictable behavior. This sensitivity limits their use in safety-critical domains, where robustness guarantees are required. Our work addresses this gap between state-of-the-art adversarial training methods and formal verification to train verifiably robust agents. Previous works train networks with individual adversarial perturbations, making them only robust against the specific adversarial attacks used. In contrast, our approach propagates entire perturbed input sets, enclosing all possible adversarial attacks within a single network pass. We leverage this to explicitly penalize the size of the output set (minimizing closed-loop uncertainty) and thereby make the actor robust against all possible attacks. This is realized by the use of set-based policy gradients, where each output within the set has a different gradient, thereby balancing the accuracy and robustness of the network. Doing so, we achieve formal verifiability across different verification frameworks for up to 9 times larger input perturbations compared to standard reinforcement learning and improve certified worst-case performance.

Manuel Wendl, Lukas Koller, Tobias Ladner +1

Technical University of Munich, Germany

RL Control Reinforcement Learning Neural Network Robustness Certified Robustness Robust RL

Aug 2, 2024

Interpreting Global Perturbation Robustness of Image Models using Axiomatic Spectral Importance Decomposition

Perturbation robustness evaluates the vulnerabilities of models, arising from a variety of perturbations, such as data corruptions and adversarial attacks. Understanding the mechanisms of perturbation robustness is critical for global interpretability. We present a model-agnostic, global mechanistic interpretability method to interpret the perturbation robustness of image models. This research is motivated by two key aspects. First, previous global interpretability works, in tandem with robustness benchmarks, e.g. mean corruption error (mCE), are not designed to directly interpret the mechanisms of perturbation robustness within image models. Second, we notice that the spectral signal-to-noise ratios (SNR) of perturbed natural images exponentially decay over the frequency. This power-law-like decay implies that: Low-frequency signals are generally more robust than high-frequency signals -- yet high classification accuracy can not be achieved by low-frequency signals alone. By applying Shapley value theory, our method axiomatically quantifies the predictive powers of robust features and non-robust features within an information theory framework. Our method, dubbed as I-ASIDE (Image Axiomatic Spectral Importance Decomposition Explanation), provides a unique insight into model robustness mechanisms. We conduct extensive experiments over a variety of vision models pre-trained on ImageNet to show that I-ASIDE can not only measure the perturbation robustness but also provide interpretations of its mechanisms.

Róisín Luo, James McDermott, Colm O'Riordan

SFI Centre for Research Training in Artificial Intelligence · School of Computer Science, University of Galway

Image Corruption Robustness Neural Network Robustness Adversarial Robustness Mechanistic Interpretability

Jun 14, 2024

Over-parameterization and Adversarial Robustness in Neural Networks: An Overview and Empirical Analysis

Thanks to their extensive capacity, over-parameterized neural networks exhibit superior predictive capabilities and generalization. However, having a large parameter space is considered one of the main suspects of the neural networks' vulnerability to adversarial example -- input samples crafted ad-hoc to induce a desired misclassification. Relevant literature has claimed contradictory remarks in support of and against the robustness of over-parameterized networks. These contradictory findings might be due to the failure of the attack employed to evaluate the networks' robustness. Previous research has demonstrated that depending on the considered model, the algorithm employed to generate adversarial examples may not function properly, leading to overestimating the model's robustness. In this work, we empirically study the robustness of over-parameterized networks against adversarial examples. However, unlike the previous works, we also evaluate the considered attack's reliability to support the results' veracity. Our results show that over-parameterized networks are robust against adversarial attacks as opposed to their under-parameterized counterparts.

Srishti Gupta, Zhang Chen, Luca Demetrio +9

Sapienza University of Rome, Rome, Italy. · University of Cagliari, Cagliari, Italy. · Northwestern Polytechnical University, Xi’An, China. +1

Overparameterized Neural Network Adversarial Attacks Neural Network Robustness Adversarial Robustness Overparameterization

Apr 1, 2024

Perturbation Effects on Robustness and Individual Fairness

Deep neural networks are vulnerable to adversarial perturbations that can simultaneously degrade prediction robustness and individual fairness across diverse application settings. However, existing evaluation protocols typically assess these dimensions in isolation, thereby obscuring critical failure modes. To bridge this gap, we formalize Robust Individual Fairness (RIF): under semantic-preserving (truth-condition-preserving) perturbations, predictions should remain both correct with respect to the ground truth and invariant across semantically equivalent individuals. To surface RIF violations in practice, we introduce RIFair, a black-box adversarial framework that leverages a decoupled perturbation strategy to construct semantically preserved yet unrobust and/or unfair instance pairs. Experiments across multiple model architectures and real-world textual datasets show that robustness-only or fairness-only metrics often miss Robust Biased and Unrobust Fair behaviors. RIFair}reliably exposes these hidden vulnerabilities, supporting RIF as a necessary criterion for trustworthy model assessment. The experimental code is publicly available at https://github.com/Xuran-LI/RIFair.

Xuran Li, Hao Xue, Peng Wu +4

University of New South Wales Sydney, Australia · The Hong Kong University of Science and Technology Guangzhou, China · Key Laboratory of System Software, Institute of Software, Chinese Academy of Sciences Beijing, China +3

Neural Network Robustness Algorithmic Fairness Adversarial Robustness Black-Box Adversarial Attacks Adversarial Text Attacks

Jun 17, 2023

Understanding Certified Training with Interval Bound Propagation

As robustness verification methods are becoming more precise, training certifiably robust neural networks is becoming ever more relevant. To this end, certified training methods compute and then optimize an upper bound on the worst-case loss over a robustness specification. Curiously, training methods based on the imprecise interval bound propagation (IBP) consistently outperform those leveraging more precise bounding methods. Still, we lack an understanding of the mechanisms making IBP so successful. In this work, we thoroughly investigate these mechanisms by leveraging a novel metric measuring the tightness of IBP bounds. We first show theoretically that, for deep linear models, tightness decreases with width and depth at initialization, but improves with IBP training, given sufficient network width. We, then, derive sufficient and necessary conditions on weight matrices for IBP bounds to become exact and demonstrate that these impose strong regularization, explaining the empirically observed trade-off between robustness and accuracy in certified training. Our extensive experimental evaluation validates our theoretical predictions for ReLU networks, including that wider networks improve performance, yielding state-of-the-art results. Interestingly, we observe that while all IBP-based training methods lead to high tightness, this is neither sufficient nor necessary to achieve high certifiable robustness. This hints at the existence of new training methods that do not induce the strong regularization required for tight IBP bounds, leading to improved robustness and standard accuracy.

Yuhao Mao, Mark Niklas Müller, Marc Fischer +1

Department of Computer Science, ETH Zürich, Swizterland

Neural Network Verification Neural Network Robustness Certified Robustness

Nov 27, 2022

Adversarial Rademacher Complexity of Deep Neural Networks

Deep neural networks (DNNs) are highly vulnerable to adversarial attacks. Ideally, a robust model should perform well on both perturbed training data and unseen perturbed test data. While DNNs can fit perturbed training data, generalizing to perturbed test data remains a significant challenge. This motivates the study of generalization guarantees from a learning theory perspective. This paper focuses on adversarial Rademacher complexity (ARC), first introduced by Khim and Loh (2018) and Yin et al. (2019). Their work primarily addressed linear functions and highlighted the open question of how to bound ARC for neural networks. Since then, several attempts have been made, with the latest results applying ARC only to two-layer neural networks. The main challenge arises from the dynamic nature and unknown closed-form solution of adversarial examples. In this paper, we resolve this issue and provide the first bound on ARC for deep neural networks. Our bound is qualitatively comparable to Rademacher complexity bounds in similar settings. The key ingredient is a new concept we introduce, termed intermediate adversarial examples, along with a framework for calculating the covering number that is compatible with them. Finally, we present experiments to analyze poor robust generalization, demonstrating that the weight norm is a crucial factor influencing the robust generalization gap.

Jiancong Xiao, Yanbo Fan, Ruoyu Sun +1

The Chinese University of Hong Kong, Shenzhen Shenzhen, China · Nanjing University Suzhou, China

Neural Network Generalization Neural Network Robustness Adversarial Robustness Generalization Bounds