The Propensity for Density in Feed-forward Models

Nandi Schoots; Alex Jackson; Ali Kholmovaia; Peter McBurney; Murray Shanahan

doi:10.3233/FAIA240819

The Propensity for Density in Feed-forward Models

Nandi Schoots^*, Alex Jackson^*, Ali Kholmovaia, Peter McBurney, Murray Shanahan

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

16 Downloads (Pure)

Abstract

Does the process of training a neural network to solve a task tend to use all of the available weights even when the task could be solved with fewer weights? To address this question we study the effects of pruning fully connected, convolutional and residual models while varying their widths. We find that the proportion of weights that can be pruned without degrading performance is largely invariant to model size. Increasing the width of a model has little effect on the density of the pruned model relative to the increase in absolute size of the pruned network. In particular, we find substantial prunability across a large range of model sizes, where our biggest model is 50 times as wide as our smallest model. We explore three hypotheses that could explain these findings.

Original language	English
Title of host publication	27th European Conference on Artificial Intelligence
Publisher	IOS Press
Pages	2830-2837
Number of pages	8
Volume	392
DOIs	https://doi.org/10.3233/FAIA240819
Publication status	Published - 18 Oct 2024

Keywords

science of deep learning
sparsity
neural networks
simplicity
deep learning
machine learning

Access to Document

10.3233/FAIA240819Licence: CC BY-NC

Cite this

@inbook{3f843bd81aa64343a7b99da9fdea014d,

title = "The Propensity for Density in Feed-forward Models",

abstract = "Does the process of training a neural network to solve a task tend to use all of the available weights even when the task could be solved with fewer weights? To address this question we study the effects of pruning fully connected, convolutional and residual models while varying their widths. We find that the proportion of weights that can be pruned without degrading performance is largely invariant to model size. Increasing the width of a model has little effect on the density of the pruned model relative to the increase in absolute size of the pruned network. In particular, we find substantial prunability across a large range of model sizes, where our biggest model is 50 times as wide as our smallest model. We explore three hypotheses that could explain these findings. ",

keywords = "science of deep learning, sparsity, neural networks, simplicity, deep learning, machine learning",

author = "Nandi Schoots and Alex Jackson and Ali Kholmovaia and Peter McBurney and Murray Shanahan",

year = "2024",

month = oct,

day = "18",

doi = "10.3233/FAIA240819",

language = "English",

volume = "392",

pages = "2830--2837",

booktitle = "27th European Conference on Artificial Intelligence",

publisher = "IOS Press",

}

TY - CHAP

T1 - The Propensity for Density in Feed-forward Models

AU - Schoots, Nandi

AU - Jackson, Alex

AU - Kholmovaia, Ali

AU - McBurney, Peter

AU - Shanahan, Murray

PY - 2024/10/18

Y1 - 2024/10/18

N2 - Does the process of training a neural network to solve a task tend to use all of the available weights even when the task could be solved with fewer weights? To address this question we study the effects of pruning fully connected, convolutional and residual models while varying their widths. We find that the proportion of weights that can be pruned without degrading performance is largely invariant to model size. Increasing the width of a model has little effect on the density of the pruned model relative to the increase in absolute size of the pruned network. In particular, we find substantial prunability across a large range of model sizes, where our biggest model is 50 times as wide as our smallest model. We explore three hypotheses that could explain these findings.

AB - Does the process of training a neural network to solve a task tend to use all of the available weights even when the task could be solved with fewer weights? To address this question we study the effects of pruning fully connected, convolutional and residual models while varying their widths. We find that the proportion of weights that can be pruned without degrading performance is largely invariant to model size. Increasing the width of a model has little effect on the density of the pruned model relative to the increase in absolute size of the pruned network. In particular, we find substantial prunability across a large range of model sizes, where our biggest model is 50 times as wide as our smallest model. We explore three hypotheses that could explain these findings.

KW - science of deep learning

KW - sparsity

KW - neural networks

KW - simplicity

KW - deep learning

KW - machine learning

U2 - 10.3233/FAIA240819

DO - 10.3233/FAIA240819

M3 - Conference paper

VL - 392

SP - 2830

EP - 2837

BT - 27th European Conference on Artificial Intelligence

PB - IOS Press

ER -

The Propensity for Density in Feed-forward Models

Abstract

Keywords

Access to Document

Fingerprint

Cite this