The Multiscale Surface Vision Transformer

Simon Dahan; Logan Williams; Daniel Rueckert; Emma Robinson

The Multiscale Surface Vision Transformer

Simon Dahan, Logan Williams, Daniel Rueckert, Emma Robinson^*

^*Corresponding author for this work

Imperial College London

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

1 Citation (Scopus)

Abstract

Surface meshes are a favoured domain for representing structural and functional information on the human cortex, but their complex topology and geometry pose significant challenges for deep learning analysis. While Transformers have excelled as domainagnostic architectures for sequence-to-sequence learning, the quadratic cost of the selfattention operation remains an obstacle for many dense prediction tasks. Inspired by
some of the latest advances in hierarchical modelling with vision transformers, we introduce the Multiscale Surface Vision Transformer (MS-SiT) as a backbone architecture
for surface deep learning. The self-attention mechanism is applied within local-meshwindows to allow for high-resolution sampling of the underlying data, while a shiftedwindow strategy improves the sharing of information between windows. Neighbouring
patches are successively merged, allowing the MS-SiT to learn hierarchical representations suitable for any prediction task. Results demonstrate that the MS-SiT outperforms existing surface deep learning methods for neonatal phenotyping prediction tasks
using the Developing Human Connectome Project (dHCP) dataset. Furthermore, building the MS-SiT backbone into a U-shaped architecture for surface segmentation demonstrates competitive results on cortical parcellation using the UK Biobank (UKB) and
manually-annotated MindBoggle datasets. Code and trained models are publicly available at https://github.com/metrics-lab/surface-vision-transformers

Original language	English
Title of host publication	Proceedings of Machine Learning Research
Pages	1-17
Volume	106
Publication status	Published - 2024

Access to Document

https://arxiv.org/pdf/2303.11909Licence: Unspecified

Cite this

@inbook{a3dd54ff5a9842c7a872f669a3a8885c,

title = "The Multiscale Surface Vision Transformer",

abstract = "Surface meshes are a favoured domain for representing structural and functional information on the human cortex, but their complex topology and geometry pose significant challenges for deep learning analysis. While Transformers have excelled as domainagnostic architectures for sequence-to-sequence learning, the quadratic cost of the selfattention operation remains an obstacle for many dense prediction tasks. Inspired bysome of the latest advances in hierarchical modelling with vision transformers, we introduce the Multiscale Surface Vision Transformer (MS-SiT) as a backbone architecturefor surface deep learning. The self-attention mechanism is applied within local-meshwindows to allow for high-resolution sampling of the underlying data, while a shiftedwindow strategy improves the sharing of information between windows. Neighbouringpatches are successively merged, allowing the MS-SiT to learn hierarchical representations suitable for any prediction task. Results demonstrate that the MS-SiT outperforms existing surface deep learning methods for neonatal phenotyping prediction tasksusing the Developing Human Connectome Project (dHCP) dataset. Furthermore, building the MS-SiT backbone into a U-shaped architecture for surface segmentation demonstrates competitive results on cortical parcellation using the UK Biobank (UKB) andmanually-annotated MindBoggle datasets. Code and trained models are publicly available at https://github.com/metrics-lab/surface-vision-transformers",

author = "Simon Dahan and Logan Williams and Daniel Rueckert and Emma Robinson",

year = "2024",

language = "English",

volume = "106",

pages = "1--17",

booktitle = "Proceedings of Machine Learning Research",

}

TY - CHAP

T1 - The Multiscale Surface Vision Transformer

AU - Dahan, Simon

AU - Williams, Logan

AU - Rueckert, Daniel

AU - Robinson, Emma

PY - 2024

Y1 - 2024

N2 - Surface meshes are a favoured domain for representing structural and functional information on the human cortex, but their complex topology and geometry pose significant challenges for deep learning analysis. While Transformers have excelled as domainagnostic architectures for sequence-to-sequence learning, the quadratic cost of the selfattention operation remains an obstacle for many dense prediction tasks. Inspired bysome of the latest advances in hierarchical modelling with vision transformers, we introduce the Multiscale Surface Vision Transformer (MS-SiT) as a backbone architecturefor surface deep learning. The self-attention mechanism is applied within local-meshwindows to allow for high-resolution sampling of the underlying data, while a shiftedwindow strategy improves the sharing of information between windows. Neighbouringpatches are successively merged, allowing the MS-SiT to learn hierarchical representations suitable for any prediction task. Results demonstrate that the MS-SiT outperforms existing surface deep learning methods for neonatal phenotyping prediction tasksusing the Developing Human Connectome Project (dHCP) dataset. Furthermore, building the MS-SiT backbone into a U-shaped architecture for surface segmentation demonstrates competitive results on cortical parcellation using the UK Biobank (UKB) andmanually-annotated MindBoggle datasets. Code and trained models are publicly available at https://github.com/metrics-lab/surface-vision-transformers

AB - Surface meshes are a favoured domain for representing structural and functional information on the human cortex, but their complex topology and geometry pose significant challenges for deep learning analysis. While Transformers have excelled as domainagnostic architectures for sequence-to-sequence learning, the quadratic cost of the selfattention operation remains an obstacle for many dense prediction tasks. Inspired bysome of the latest advances in hierarchical modelling with vision transformers, we introduce the Multiscale Surface Vision Transformer (MS-SiT) as a backbone architecturefor surface deep learning. The self-attention mechanism is applied within local-meshwindows to allow for high-resolution sampling of the underlying data, while a shiftedwindow strategy improves the sharing of information between windows. Neighbouringpatches are successively merged, allowing the MS-SiT to learn hierarchical representations suitable for any prediction task. Results demonstrate that the MS-SiT outperforms existing surface deep learning methods for neonatal phenotyping prediction tasksusing the Developing Human Connectome Project (dHCP) dataset. Furthermore, building the MS-SiT backbone into a U-shaped architecture for surface segmentation demonstrates competitive results on cortical parcellation using the UK Biobank (UKB) andmanually-annotated MindBoggle datasets. Code and trained models are publicly available at https://github.com/metrics-lab/surface-vision-transformers

M3 - Conference paper

VL - 106

SP - 1

EP - 17

BT - Proceedings of Machine Learning Research

ER -

The Multiscale Surface Vision Transformer

Abstract

Access to Document

Fingerprint

Cite this