Unsupervised Brain Imaging 3D Anomaly Detection and Segmentation with Transformers

Walter H.L. Pinaya, Petru-Daniel Tudosiu, Robert Gray, Geraint Rees, Parashkev Nachev, Sebastien Ourselin, M. Jorge Cardoso

Research output: Contribution to journalArticlepeer-review

44 Citations (Scopus)
230 Downloads (Pure)


Pathological brain appearances may be so heterogeneous as to be intelligible only as anomalies, defined by their deviation from normality rather than any specific set of pathological features. Amongst the hardest tasks in medical imaging, detecting such anomalies requires models of the normal brain that combine compactness with the expressivity of the complex, long-range interactions that characterise its structural organisation. These are requirements transformers have arguably greater potential to satisfy than other current candidate architectures, but their application has been inhibited by their demands on data and computational resources. Here we combine
the latent representation of vector quantised variational autoencoders with an
ensemble of autoregressive transformers to enable unsupervised anomaly detection and segmentation defined by deviation from healthy brain imaging data, achievable at low computational cost, within relative modest data regimes. We compare our method to current state-of-the-art approaches across a series of experiments with 2D and 3D data involving synthetic and real pathological lesions. On real lesions, we train our models on 15,000 radiologically normal participants from UK Biobank and evaluate performance on four different brain MR datasets with small vessel disease, demyelinating lesions, and tumours. We demonstrate superior anomaly detection performance both image-wise and pixel/voxel-wise, achievable without postprocessing. These results draw attention to the potential of transformers in this most challenging of imaging tasks.
Original languageEnglish
Article number102475
JournalMedical Image Analysis
Early online date19 May 2022
Publication statusPublished - 31 Jul 2022


Dive into the research topics of 'Unsupervised Brain Imaging 3D Anomaly Detection and Segmentation with Transformers'. Together they form a unique fingerprint.

Cite this