Abstract
In this paper, we conjecture that if the permutation invariance of neural networks is taken intoaccount, SGD solutions will likely have no barrier in the linear interpolation between them. Althoughit is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We furtherprovide a preliminary theoretical result to support our conjecture. Our conjecture has implications forlottery ticket hypothesis, distributed training and ensemble methods
Original language | English |
---|---|
Publication status | Published - 7 Jul 2021 |
Event | Sparsity in Neural Networks - Advancing Understanding and Practice: SNN Workshop 2021 - Virtual Duration: 8 Jul 2021 → 9 Jul 2021 https://sites.google.com/view/sparsity-workshop-2021/home?authuser=0 |
Workshop
Workshop | Sparsity in Neural Networks - Advancing Understanding and Practice |
---|---|
City | Virtual |
Period | 8/07/21 → 9/07/21 |
Internet address |
Keywords
- deep learnig
- loss landscape
- optimization