On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks

Nikolay Manchev; Michael Spratling

doi:10.1109/ISCMI56532.2022.10068489

On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks

Nikolay Manchev, Michael Spratling

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

2 Downloads (Pure)

Abstract

Initialising the synaptic weights of artificial neural networks (ANNs) with orthogonal matrices is known to alleviate vanishing and exploding gradient problems. A major objection against such initialisation schemes is that they are deemed biologically implausible as they mandate factorization techniques that are difficult to attribute to a neurobiological process. This paper presents two initialisation schemes that allow a network to naturally evolve its weights to form orthogonal matrices, provides theoretical analysis that pre-training orthogonalisation always converges, and empirically confirms that the proposed schemes outperform randomly initialised recurrent and feedforward networks.

Original language	English
Title of host publication	2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022
Publisher	IEEE
Pages	47-55
Number of pages	9
ISBN (Electronic)	9798350320886
DOIs	https://doi.org/10.1109/ISCMI56532.2022.10068489
Publication status	E-pub ahead of print - 21 Mar 2023

Publication series

Name	2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022

Keywords

cs.NE
cs.AI
cs.LG

Access to Document

10.1109/ISCMI56532.2022.10068489

On the biological plausibility_AAMAccepted author manuscript, 462 KB
2211.08408v1

Cite this

Manchev, N., & Spratling, M. (2023). On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks. In 2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022 (pp. 47-55). (2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022). IEEE. https://doi.org/10.1109/ISCMI56532.2022.10068489

@inbook{b085c27972834f4fa2a0b1e65a55be38,

title = "On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks",

abstract = "Initialising the synaptic weights of artificial neural networks (ANNs) with orthogonal matrices is known to alleviate vanishing and exploding gradient problems. A major objection against such initialisation schemes is that they are deemed biologically implausible as they mandate factorization techniques that are difficult to attribute to a neurobiological process. This paper presents two initialisation schemes that allow a network to naturally evolve its weights to form orthogonal matrices, provides theoretical analysis that pre-training orthogonalisation always converges, and empirically confirms that the proposed schemes outperform randomly initialised recurrent and feedforward networks. ",

keywords = "cs.NE, cs.AI, cs.LG",

author = "Nikolay Manchev and Michael Spratling",

note = "9 pages, 3 figures, to be published in ISCMI2022 conference proceedings",

year = "2023",

month = mar,

day = "21",

doi = "10.1109/ISCMI56532.2022.10068489",

language = "English",

series = "2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022",

publisher = "IEEE",

pages = "47--55",

booktitle = "2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022",

}

Manchev, N & Spratling, M 2023, On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks. in 2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022. 2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022, IEEE, pp. 47-55. https://doi.org/10.1109/ISCMI56532.2022.10068489

On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks. / Manchev, Nikolay ; Spratling, Michael.
2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022. IEEE, 2023. p. 47-55 (2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

TY - CHAP

T1 - On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks

AU - Manchev, Nikolay

AU - Spratling, Michael

N1 - 9 pages, 3 figures, to be published in ISCMI2022 conference proceedings

PY - 2023/3/21

Y1 - 2023/3/21

N2 - Initialising the synaptic weights of artificial neural networks (ANNs) with orthogonal matrices is known to alleviate vanishing and exploding gradient problems. A major objection against such initialisation schemes is that they are deemed biologically implausible as they mandate factorization techniques that are difficult to attribute to a neurobiological process. This paper presents two initialisation schemes that allow a network to naturally evolve its weights to form orthogonal matrices, provides theoretical analysis that pre-training orthogonalisation always converges, and empirically confirms that the proposed schemes outperform randomly initialised recurrent and feedforward networks.

AB - Initialising the synaptic weights of artificial neural networks (ANNs) with orthogonal matrices is known to alleviate vanishing and exploding gradient problems. A major objection against such initialisation schemes is that they are deemed biologically implausible as they mandate factorization techniques that are difficult to attribute to a neurobiological process. This paper presents two initialisation schemes that allow a network to naturally evolve its weights to form orthogonal matrices, provides theoretical analysis that pre-training orthogonalisation always converges, and empirically confirms that the proposed schemes outperform randomly initialised recurrent and feedforward networks.

KW - cs.NE

KW - cs.AI

KW - cs.LG

UR - http://www.scopus.com/inward/record.url?scp=85151712113&partnerID=8YFLogxK

U2 - 10.1109/ISCMI56532.2022.10068489

DO - 10.1109/ISCMI56532.2022.10068489

M3 - Conference paper

T3 - 2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022

SP - 47

EP - 55

BT - 2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022

PB - IEEE

ER -

Manchev N , Spratling M. On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks. In 2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022. IEEE. 2023. p. 47-55. (2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022). Epub 2023 Mar 21. doi: 10.1109/ISCMI56532.2022.10068489

On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks

Abstract

Publication series

Keywords

Access to Document

Other files and links

Fingerprint

Cite this