Replica symmetry breaking in supervised and unsupervised Hebbian networks

Linda Albanese; Andrea Alessandrelli; Alessia Annibale; Adriano Barra

doi:10.1088/1751-8121/ad38b4

Replica symmetry breaking in supervised and unsupervised Hebbian networks

Linda Albanese^*, Andrea Alessandrelli, Alessia Annibale, Adriano Barra

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

Abstract

Hebbian neural networks with multi-node interactions, often called Dense Associative Memories, have recently attracted considerable interest in the statistical mechanics community, as they have been shown to outperform their pairwise counterparts in a number of features, including resilience against adversarial attacks, pattern retrieval with extremely weak signals and supra-linear storage capacities. However, their analysis has so far been carried out within a replica-symmetric theory. In this manuscript, we relax the assumption of replica symmetry and analyse these systems at one step of replica-symmetry breaking, focusing on two different prescriptions for the interactions that we will refer to as supervised and unsupervised learning. We derive the phase diagram of the model using two different approaches, namely Parisi’s hierarchical ansatz for the relationship between different replicas within the replica approach, and the so-called telescope ansatz within Guerra’s interpolation method: our results show that replica-symmetry breaking does not alter the threshold for learning and slightly increases the maximal storage capacity. Further, we also derive analytically the instability line of the replica-symmetric theory, using a generalization of the De Almeida and Thouless approach.

Original language	English
Article number	165003
Journal	Journal of Physics A: Mathematical and Theoretical
Volume	57
Issue number	16
DOIs	https://doi.org/10.1088/1751-8121/ad38b4
Publication status	Published - 19 Apr 2024

Keywords

dense associative memory
Guerra’s interpolation
replica symmetry breaking
replica trick
supervised learning
unsupervised learning

Access to Document

10.1088/1751-8121/ad38b4

Cite this

@article{0e4d2028134a431d986f8db31f24a8e1,

title = "Replica symmetry breaking in supervised and unsupervised Hebbian networks",

abstract = "Hebbian neural networks with multi-node interactions, often called Dense Associative Memories, have recently attracted considerable interest in the statistical mechanics community, as they have been shown to outperform their pairwise counterparts in a number of features, including resilience against adversarial attacks, pattern retrieval with extremely weak signals and supra-linear storage capacities. However, their analysis has so far been carried out within a replica-symmetric theory. In this manuscript, we relax the assumption of replica symmetry and analyse these systems at one step of replica-symmetry breaking, focusing on two different prescriptions for the interactions that we will refer to as supervised and unsupervised learning. We derive the phase diagram of the model using two different approaches, namely Parisi{\textquoteright}s hierarchical ansatz for the relationship between different replicas within the replica approach, and the so-called telescope ansatz within Guerra{\textquoteright}s interpolation method: our results show that replica-symmetry breaking does not alter the threshold for learning and slightly increases the maximal storage capacity. Further, we also derive analytically the instability line of the replica-symmetric theory, using a generalization of the De Almeida and Thouless approach.",

keywords = "dense associative memory, Guerra{\textquoteright}s interpolation, replica symmetry breaking, replica trick, supervised learning, unsupervised learning",

author = "Linda Albanese and Andrea Alessandrelli and Alessia Annibale and Adriano Barra",

note = "Publisher Copyright: {\textcopyright} 2024 The Author(s). Published by IOP Publishing Ltd",

year = "2024",

month = apr,

day = "19",

doi = "10.1088/1751-8121/ad38b4",

language = "English",

volume = "57",

journal = "Journal of Physics A: Mathematical and Theoretical",

issn = "1751-8113",

publisher = "IOP Publishing",

number = "16",

}

TY - JOUR

T1 - Replica symmetry breaking in supervised and unsupervised Hebbian networks

AU - Albanese, Linda

AU - Alessandrelli, Andrea

AU - Annibale, Alessia

AU - Barra, Adriano

PY - 2024/4/19

Y1 - 2024/4/19

N2 - Hebbian neural networks with multi-node interactions, often called Dense Associative Memories, have recently attracted considerable interest in the statistical mechanics community, as they have been shown to outperform their pairwise counterparts in a number of features, including resilience against adversarial attacks, pattern retrieval with extremely weak signals and supra-linear storage capacities. However, their analysis has so far been carried out within a replica-symmetric theory. In this manuscript, we relax the assumption of replica symmetry and analyse these systems at one step of replica-symmetry breaking, focusing on two different prescriptions for the interactions that we will refer to as supervised and unsupervised learning. We derive the phase diagram of the model using two different approaches, namely Parisi’s hierarchical ansatz for the relationship between different replicas within the replica approach, and the so-called telescope ansatz within Guerra’s interpolation method: our results show that replica-symmetry breaking does not alter the threshold for learning and slightly increases the maximal storage capacity. Further, we also derive analytically the instability line of the replica-symmetric theory, using a generalization of the De Almeida and Thouless approach.

AB - Hebbian neural networks with multi-node interactions, often called Dense Associative Memories, have recently attracted considerable interest in the statistical mechanics community, as they have been shown to outperform their pairwise counterparts in a number of features, including resilience against adversarial attacks, pattern retrieval with extremely weak signals and supra-linear storage capacities. However, their analysis has so far been carried out within a replica-symmetric theory. In this manuscript, we relax the assumption of replica symmetry and analyse these systems at one step of replica-symmetry breaking, focusing on two different prescriptions for the interactions that we will refer to as supervised and unsupervised learning. We derive the phase diagram of the model using two different approaches, namely Parisi’s hierarchical ansatz for the relationship between different replicas within the replica approach, and the so-called telescope ansatz within Guerra’s interpolation method: our results show that replica-symmetry breaking does not alter the threshold for learning and slightly increases the maximal storage capacity. Further, we also derive analytically the instability line of the replica-symmetric theory, using a generalization of the De Almeida and Thouless approach.

KW - dense associative memory

KW - Guerra’s interpolation

KW - replica symmetry breaking

KW - replica trick

KW - supervised learning

KW - unsupervised learning

UR - http://www.scopus.com/inward/record.url?scp=85190134239&partnerID=8YFLogxK

U2 - 10.1088/1751-8121/ad38b4

DO - 10.1088/1751-8121/ad38b4

M3 - Article

AN - SCOPUS:85190134239

SN - 1751-8113

VL - 57

JO - Journal of Physics A: Mathematical and Theoretical

JF - Journal of Physics A: Mathematical and Theoretical

IS - 16

M1 - 165003

ER -

Replica symmetry breaking in supervised and unsupervised Hebbian networks

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this