The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

Rahim Entezari; Hanie Sedghi; Olga Saukh; Behnam Neyshabur

The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

Rahim Entezari, Hanie Sedghi, Olga Saukh, Behnam Neyshabur

Institut für Technische Informatik (4480)

Publikation: Konferenzbeitrag › Poster › Begutachtung

Abstract

In this paper, we conjecture that if the permutation invariance of neural networks is taken intoaccount, SGD solutions will likely have no barrier in the linear interpolation between them. Althoughit is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We furtherprovide a preliminary theoretical result to support our conjecture. Our conjecture has implications forlottery ticket hypothesis, distributed training and ensemble methods

Originalsprache	englisch
Publikationsstatus	Veröffentlicht - 7 Juli 2021
Veranstaltung	Sparsity in Neural Networks - Advancing Understanding and Practice: SNN Workshop 2021 - Virtual Dauer: 8 Juli 2021 → 9 Juli 2021 https://sites.google.com/view/sparsity-workshop-2021/home?authuser=0

Workshop

Workshop	Sparsity in Neural Networks - Advancing Understanding and Practice
Ort	Virtual
Zeitraum	8/07/21 → 9/07/21
Internetadresse	https://sites.google.com/view/sparsity-workshop-2021/home?authuser=0

Zugriff auf Dokument

https://drive.google.com/file/d/1n0Q8pY1p7h6P9Ai8eweHbv54-VxXTR8R/view

Dieses zitieren

@conference{f67c699565b64327a0f7b9116800e230,

title = "The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks",

abstract = "In this paper, we conjecture that if the permutation invariance of neural networks is taken intoaccount, SGD solutions will likely have no barrier in the linear interpolation between them. Althoughit is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We furtherprovide a preliminary theoretical result to support our conjecture. Our conjecture has implications forlottery ticket hypothesis, distributed training and ensemble methods",

keywords = "deep learnig, loss landscape, optimization",

author = "Rahim Entezari and Hanie Sedghi and Olga Saukh and Behnam Neyshabur",

year = "2021",

month = jul,

day = "7",

language = "English",

note = "Sparsity in Neural Networks - Advancing Understanding and Practice : SNN Workshop 2021 ; Conference date: 08-07-2021 Through 09-07-2021",

url = "https://sites.google.com/view/sparsity-workshop-2021/home?authuser=0",

}

TY - CONF

T1 - The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

AU - Entezari, Rahim

AU - Sedghi, Hanie

AU - Saukh, Olga

AU - Neyshabur, Behnam

PY - 2021/7/7

Y1 - 2021/7/7

N2 - In this paper, we conjecture that if the permutation invariance of neural networks is taken intoaccount, SGD solutions will likely have no barrier in the linear interpolation between them. Althoughit is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We furtherprovide a preliminary theoretical result to support our conjecture. Our conjecture has implications forlottery ticket hypothesis, distributed training and ensemble methods

AB - In this paper, we conjecture that if the permutation invariance of neural networks is taken intoaccount, SGD solutions will likely have no barrier in the linear interpolation between them. Althoughit is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We furtherprovide a preliminary theoretical result to support our conjecture. Our conjecture has implications forlottery ticket hypothesis, distributed training and ensemble methods

KW - deep learnig

KW - loss landscape

KW - optimization

M3 - Poster

T2 - Sparsity in Neural Networks - Advancing Understanding and Practice

Y2 - 8 July 2021 through 9 July 2021

ER -

The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

Abstract

Workshop

Zugriff auf Dokument

Fingerprint

Dieses zitieren