Removing Batch Normalization Boosts Adversarial Training Article Swipe

MAKE IMPACT

Haotao Wang , Aston Zhang , Shuai Zheng , Xingjian Shi , Mu Li , Zhangyang Wang · YOU? · · 2022 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2207.01156

Adversarial training (AT) defends deep neural networks against adversarial attacks. One challenge that limits its practical application is the performance degradation on clean samples. A major bottleneck identified by previous works is the widely used batch normalization (BN), which struggles to model the different statistics of clean and adversarial training samples in AT. Although the dominant approach is to extend BN to capture this mixture of distribution, we propose to completely eliminate this bottleneck by removing all BN layers in AT. Our normalizer-free robust training (NoFrost) method extends recent advances in normalizer-free networks to AT for its unexplored advantage on handling the mixture distribution challenge. We show that NoFrost achieves adversarial robustness with only a minor sacrifice on clean sample accuracy. On ImageNet with ResNet50, NoFrost achieves $74.06\%$ clean accuracy, which drops merely $2.00\%$ from standard training. In contrast, BN-based AT obtains $59.28\%$ clean accuracy, suffering a significant $16.78\%$ drop from standard training. In addition, NoFrost achieves a $23.56\%$ adversarial robustness against PGD attack, which improves the $13.57\%$ robustness in BN-based AT. We observe better model smoothness and larger decision margins from NoFrost, which make the models less sensitive to input perturbations and thus more robust. Moreover, when incorporating more data augmentations into NoFrost, it achieves comprehensive robustness against multiple distribution shifts. Code and pre-trained models are public at https://github.com/amazon-research/normalizer-free-robust-training.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2207.01156

PDF

https://arxiv.org/pdf/2207.01156

OA Status

green

Cited By

6

Related Works

10

OpenAlex ID

https://openalex.org/W4283831268

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4283831268

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2207.01156

Digital Object Identifier

Title

Removing Batch Normalization Boosts Adversarial Training

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2022

Year of publication

Publication date

2022-07-04

Full publication date if available

Authors

Haotao Wang, Aston Zhang, Shuai Zheng, Xingjian Shi, Mu Li, Zhangyang Wang

List of authors in order

Landing page

https://arxiv.org/abs/2207.01156

Publisher landing page

PDF URL

https://arxiv.org/pdf/2207.01156

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2207.01156

Direct OA link when available

Concepts

Bottleneck, Adversarial system, Computer science, Robustness (evolution), Normalization (sociology), Deep neural networks, Artificial intelligence, Initialization, Machine learning, Artificial neural network, Programming language, Gene, Anthropology, Chemistry, Biochemistry, Sociology, Embedded system

Top concepts (fields/topics) attached by OpenAlex

Cited by

6

Total citation count in OpenAlex

Citations by year (recent)

2025: 3, 2023: 3

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload