Non-deterministic learning dynamics in large neural networks due to structural data bias

H C Rae; J A F Heimel; A C C Coolen

doi:10.1088/0305-4470/33/48/309

Non-deterministic learning dynamics in large neural networks due to structural data bias

H C Rae, J A F Heimel, A C C Coolen

Research output: Contribution to journal › Article › peer-review

1 Citation (Scopus)

Abstract

We study the dynamics of on-line learning in large (N --> infinity) perceptrons, for the case of training sets with a structural O(N-0) bias of the input vectors, by deriving exact and closed macroscopic dynamical laws using non-equilibrium statistical mechanical tools. In sharp contrast to the more conventional theories developed for homogeneously distributed or only weakly biased data, these laws are found to describe a non-trivial and persistently nondeterministic macroscopic evolution, and a generalization error which retains both stochastic and sample-to-sample fluctuations, even for infinitely large networks. Furthermore, for the standard error-correcting microscopic algorithms (such as the perceptron learning rule) one obtains learning curves with distinct bias-induced phases. Our theoretical predictions find excellent confirmation in numerical simulations.

Original language	English
Pages (from-to)	8703 - 8722
Number of pages	20
Journal	JOURNAL OF PHYSICS A MATHEMATICAL AND GENERAL
Volume	33
Issue number	48
DOIs	https://doi.org/10.1088/0305-4470/33/48/309
Publication status	Published - 8 Dec 2000

Access to Document

10.1088/0305-4470/33/48/309

Cite this

@article{305d4d79accc4dc6b3b87a9f91f230f1,

title = "Non-deterministic learning dynamics in large neural networks due to structural data bias",

abstract = "We study the dynamics of on-line learning in large (N --> infinity) perceptrons, for the case of training sets with a structural O(N-0) bias of the input vectors, by deriving exact and closed macroscopic dynamical laws using non-equilibrium statistical mechanical tools. In sharp contrast to the more conventional theories developed for homogeneously distributed or only weakly biased data, these laws are found to describe a non-trivial and persistently nondeterministic macroscopic evolution, and a generalization error which retains both stochastic and sample-to-sample fluctuations, even for infinitely large networks. Furthermore, for the standard error-correcting microscopic algorithms (such as the perceptron learning rule) one obtains learning curves with distinct bias-induced phases. Our theoretical predictions find excellent confirmation in numerical simulations.",

author = "Rae, {H C} and Heimel, {J A F} and Coolen, {A C C}",

year = "2000",

month = dec,

day = "8",

doi = "10.1088/0305-4470/33/48/309",

language = "English",

volume = "33",

pages = "8703 -- 8722",

journal = "JOURNAL OF PHYSICS A MATHEMATICAL AND GENERAL",

publisher = "IOP Publishing Ltd.",

number = "48",

}

TY - JOUR

T1 - Non-deterministic learning dynamics in large neural networks due to structural data bias

AU - Rae, H C

AU - Heimel, J A F

AU - Coolen, A C C

PY - 2000/12/8

Y1 - 2000/12/8

N2 - We study the dynamics of on-line learning in large (N --> infinity) perceptrons, for the case of training sets with a structural O(N-0) bias of the input vectors, by deriving exact and closed macroscopic dynamical laws using non-equilibrium statistical mechanical tools. In sharp contrast to the more conventional theories developed for homogeneously distributed or only weakly biased data, these laws are found to describe a non-trivial and persistently nondeterministic macroscopic evolution, and a generalization error which retains both stochastic and sample-to-sample fluctuations, even for infinitely large networks. Furthermore, for the standard error-correcting microscopic algorithms (such as the perceptron learning rule) one obtains learning curves with distinct bias-induced phases. Our theoretical predictions find excellent confirmation in numerical simulations.

AB - We study the dynamics of on-line learning in large (N --> infinity) perceptrons, for the case of training sets with a structural O(N-0) bias of the input vectors, by deriving exact and closed macroscopic dynamical laws using non-equilibrium statistical mechanical tools. In sharp contrast to the more conventional theories developed for homogeneously distributed or only weakly biased data, these laws are found to describe a non-trivial and persistently nondeterministic macroscopic evolution, and a generalization error which retains both stochastic and sample-to-sample fluctuations, even for infinitely large networks. Furthermore, for the standard error-correcting microscopic algorithms (such as the perceptron learning rule) one obtains learning curves with distinct bias-induced phases. Our theoretical predictions find excellent confirmation in numerical simulations.

UR - http://www.scopus.com/inward/record.url?scp=0034623745&partnerID=8YFLogxK

U2 - 10.1088/0305-4470/33/48/309

DO - 10.1088/0305-4470/33/48/309

M3 - Article

VL - 33

SP - 8703

EP - 8722

JO - JOURNAL OF PHYSICS A MATHEMATICAL AND GENERAL

JF - JOURNAL OF PHYSICS A MATHEMATICAL AND GENERAL

IS - 48

ER -

Non-deterministic learning dynamics in large neural networks due to structural data bias

Abstract

Access to Document

Other files and links

Fingerprint

Cite this