Training adversarially robust sparse networks via Bayesian connectivity sampling

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review

Abstract

Deep neural networks have been shown to be susceptible to adversarial attacks. This lack of adversarial robustness is even more pronounced when models are compressed in order to meet hardware limitations. Hence, if adversarial robustness is an issue, training of sparsely connected networks necessitates considering adversarially robust sparse learning. Motivated by the efficient and stable computational function of the brain in the presence of a highly dynamic synaptic connectivity structure, we propose an intrinsically sparse rewiring approach to train neural networks with state-of-the-art robust learning objectives under high sparsity. Importantly, in contrast to previously proposed pruning techniques, our approach satisfies global connectivity constraints throughout robust optimization, i.e., it does not require dense pre-training followed by pruning. Based on a Bayesian posterior sampling principle, a network rewiring process simultaneously learns the sparse connectivity structure and the robustness-accuracy trade-off based on the adversarial learning objective. Although our networks are sparsely connected throughout the whole training process, our experimental benchmark evaluations show that their performance is superior to recently proposed robustness-aware network pruning methods which start from densely connected networks.
Original languageEnglish
Title of host publicationProceedings of Machine Learning Research
Subtitle of host publicationPMLR
Pages8314-8324
Volume139
Publication statusPublished - 2021
Event38th International Conference on Machine Learning: ICML 2021 - Virtuell
Duration: 18 Jul 202124 Jul 2021

Conference

Conference38th International Conference on Machine Learning
Abbreviated titleICML 2021
Period18/07/2124/07/21

Fields of Expertise

  • Information, Communication & Computing

Fingerprint

Dive into the research topics of 'Training adversarially robust sparse networks via Bayesian connectivity sampling'. Together they form a unique fingerprint.

Cite this