Initialising the synaptic weights of artificial neural networks (ANNs) with orthogonal matrices is known to alleviate vanishing and exploding gradient problems. A major objection against such initialisation schemes is that they are deemed biologically implausible as they mandate factorization techniques that are difficult to attribute to a neurobiological process. This paper presents two initialisation schemes that allow a network to naturally evolve its weights to form orthogonal matrices, provides theoretical analysis that pre-training orthogonalisation always converges, and empirically confirms that the proposed schemes outperform randomly initialised recurrent and feedforward networks.
|Title of host publication||2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022|
|Number of pages||9|
|Publication status||E-pub ahead of print - 21 Mar 2023|
|Name||2022 9th International Conference on Soft Computing and Machine Intelligence, ISCMI 2022|