Latent Transformer Models for out-of-distribution detection

Mark S. Graham; Petru Daniel Tudosiu; Paul Wright; Walter Hugo Lopez Pinaya; Petteri Teikari; Ashay Patel; Jean Marie U-King-Im; Yee H. Mah; James T. Teo; Hans Rolf Jäger; David Werring; Geraint Rees; Parashkev Nachev; Sebastien Ourselin; M. Jorge Cardoso

doi:10.1016/j.media.2023.102967

Latent Transformer Models for out-of-distribution detection

Mark S. Graham^*, Petru Daniel Tudosiu, Paul Wright, Walter Hugo Lopez Pinaya, Petteri Teikari, Ashay Patel, Jean Marie U-King-Im, Yee H. Mah, James T. Teo, Hans Rolf Jäger, David Werring, Geraint Rees, Parashkev Nachev, Sebastien Ourselin, M. Jorge Cardoso

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

4 Citations (Scopus)

Abstract

Any clinically-deployed image-processing pipeline must be robust to the full range of inputs it may be presented with. One popular approach to this challenge is to develop predictive models that can provide a measure of their uncertainty. Another approach is to use generative modelling to quantify the likelihood of inputs. Inputs with a low enough likelihood are deemed to be out-of-distribution and are not presented to the downstream predictive model. In this work, we evaluate several approaches to segmentation with uncertainty for the task of segmenting bleeds in 3D CT of the head. We show that these models can fail catastrophically when operating in the far out-of-distribution domain, often providing predictions that are both highly confident and wrong. We propose to instead perform out-of-distribution detection using the Latent Transformer Model: a VQ-GAN is used to provide a highly compressed latent representation of the input volume, and a transformer is then used to estimate the likelihood of this compressed representation of the input. We demonstrate this approach can identify images that are both far- and near- out-of-distribution, as well as provide spatial maps that highlight the regions considered to be out-of-distribution. Furthermore, we find a strong relationship between an image's likelihood and the quality of a model's segmentation on it, demonstrating that this approach is viable for filtering out unsuitable images.

Original language	English
Article number	102967
Journal	Medical Image Analysis
Volume	90
Early online date	29 Sept 2023
DOIs	https://doi.org/10.1016/j.media.2023.102967
Publication status	Published - Dec 2023

Keywords

Out-of-distribution detection
Segmentation
Transformers
Uncertainty

Access to Document

10.1016/j.media.2023.102967Licence: CC BY

Cite this

@article{d88832b3f81f4c2b94f756d0646ca353,

title = "Latent Transformer Models for out-of-distribution detection",

abstract = "Any clinically-deployed image-processing pipeline must be robust to the full range of inputs it may be presented with. One popular approach to this challenge is to develop predictive models that can provide a measure of their uncertainty. Another approach is to use generative modelling to quantify the likelihood of inputs. Inputs with a low enough likelihood are deemed to be out-of-distribution and are not presented to the downstream predictive model. In this work, we evaluate several approaches to segmentation with uncertainty for the task of segmenting bleeds in 3D CT of the head. We show that these models can fail catastrophically when operating in the far out-of-distribution domain, often providing predictions that are both highly confident and wrong. We propose to instead perform out-of-distribution detection using the Latent Transformer Model: a VQ-GAN is used to provide a highly compressed latent representation of the input volume, and a transformer is then used to estimate the likelihood of this compressed representation of the input. We demonstrate this approach can identify images that are both far- and near- out-of-distribution, as well as provide spatial maps that highlight the regions considered to be out-of-distribution. Furthermore, we find a strong relationship between an image's likelihood and the quality of a model's segmentation on it, demonstrating that this approach is viable for filtering out unsuitable images.",

keywords = "Out-of-distribution detection, Segmentation, Transformers, Uncertainty",

author = "Graham, {Mark S.} and Tudosiu, {Petru Daniel} and Paul Wright and Pinaya, {Walter Hugo Lopez} and Petteri Teikari and Ashay Patel and U-King-Im, {Jean Marie} and Mah, {Yee H.} and Teo, {James T.} and J{\"a}ger, {Hans Rolf} and David Werring and Geraint Rees and Parashkev Nachev and Sebastien Ourselin and Cardoso, {M. Jorge}",

note = "Funding Information: MG, PW, WHLP, SO, PN and MJC are supported by a grant from the Wellcome Trust, United Kingdom ( WT213038/Z/18/Z ). MJC and SO are also supported by the Wellcome/EPSRC Centre for Medical Engineering ( WT203148/Z/16/Z ), and the InnovateUK-funded London AI centre for Value-based Healthcare . PN is also supported by NIHR UCLH Biomedical Research Centre . YM is supported by a grant from the Medical Research Council, United Kingdom ( MR/T005351/1 ). The models in this work were trained on NVIDIA Cambridge-1, the UK{\textquoteright}s largest super-computer, aimed at accelerating digital biology. Publisher Copyright: {\textcopyright} 2023 The Author(s)",

year = "2023",

month = dec,

doi = "10.1016/j.media.2023.102967",

language = "English",

volume = "90",

journal = "Medical Image Analysis",

issn = "1361-8415",

publisher = "Elsevier",

}

TY - JOUR

T1 - Latent Transformer Models for out-of-distribution detection

AU - Graham, Mark S.

AU - Tudosiu, Petru Daniel

AU - Wright, Paul

AU - Pinaya, Walter Hugo Lopez

AU - Teikari, Petteri

AU - Patel, Ashay

AU - U-King-Im, Jean Marie

AU - Mah, Yee H.

AU - Teo, James T.

AU - Jäger, Hans Rolf

AU - Werring, David

AU - Rees, Geraint

AU - Nachev, Parashkev

AU - Ourselin, Sebastien

AU - Cardoso, M. Jorge

N1 - Funding Information: MG, PW, WHLP, SO, PN and MJC are supported by a grant from the Wellcome Trust, United Kingdom ( WT213038/Z/18/Z ). MJC and SO are also supported by the Wellcome/EPSRC Centre for Medical Engineering ( WT203148/Z/16/Z ), and the InnovateUK-funded London AI centre for Value-based Healthcare . PN is also supported by NIHR UCLH Biomedical Research Centre . YM is supported by a grant from the Medical Research Council, United Kingdom ( MR/T005351/1 ). The models in this work were trained on NVIDIA Cambridge-1, the UK’s largest super-computer, aimed at accelerating digital biology. Publisher Copyright: © 2023 The Author(s)

PY - 2023/12

Y1 - 2023/12

N2 - Any clinically-deployed image-processing pipeline must be robust to the full range of inputs it may be presented with. One popular approach to this challenge is to develop predictive models that can provide a measure of their uncertainty. Another approach is to use generative modelling to quantify the likelihood of inputs. Inputs with a low enough likelihood are deemed to be out-of-distribution and are not presented to the downstream predictive model. In this work, we evaluate several approaches to segmentation with uncertainty for the task of segmenting bleeds in 3D CT of the head. We show that these models can fail catastrophically when operating in the far out-of-distribution domain, often providing predictions that are both highly confident and wrong. We propose to instead perform out-of-distribution detection using the Latent Transformer Model: a VQ-GAN is used to provide a highly compressed latent representation of the input volume, and a transformer is then used to estimate the likelihood of this compressed representation of the input. We demonstrate this approach can identify images that are both far- and near- out-of-distribution, as well as provide spatial maps that highlight the regions considered to be out-of-distribution. Furthermore, we find a strong relationship between an image's likelihood and the quality of a model's segmentation on it, demonstrating that this approach is viable for filtering out unsuitable images.

AB - Any clinically-deployed image-processing pipeline must be robust to the full range of inputs it may be presented with. One popular approach to this challenge is to develop predictive models that can provide a measure of their uncertainty. Another approach is to use generative modelling to quantify the likelihood of inputs. Inputs with a low enough likelihood are deemed to be out-of-distribution and are not presented to the downstream predictive model. In this work, we evaluate several approaches to segmentation with uncertainty for the task of segmenting bleeds in 3D CT of the head. We show that these models can fail catastrophically when operating in the far out-of-distribution domain, often providing predictions that are both highly confident and wrong. We propose to instead perform out-of-distribution detection using the Latent Transformer Model: a VQ-GAN is used to provide a highly compressed latent representation of the input volume, and a transformer is then used to estimate the likelihood of this compressed representation of the input. We demonstrate this approach can identify images that are both far- and near- out-of-distribution, as well as provide spatial maps that highlight the regions considered to be out-of-distribution. Furthermore, we find a strong relationship between an image's likelihood and the quality of a model's segmentation on it, demonstrating that this approach is viable for filtering out unsuitable images.

KW - Out-of-distribution detection

KW - Segmentation

KW - Transformers

KW - Uncertainty

UR - http://www.scopus.com/inward/record.url?scp=85173555884&partnerID=8YFLogxK

U2 - 10.1016/j.media.2023.102967

DO - 10.1016/j.media.2023.102967

M3 - Article

AN - SCOPUS:85173555884

SN - 1361-8415

VL - 90

JO - Medical Image Analysis

JF - Medical Image Analysis

M1 - 102967

ER -

Latent Transformer Models for out-of-distribution detection

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this