A framework for comparison and interpretation of machine learning classifiers to predict autism on the ABIDE dataset

Yilan Dong; Dafnis Batalle; Maria Deprez

doi:10.1002/hbm.70190

A framework for comparison and interpretation of machine learning classifiers to predict autism on the ABIDE dataset

Yilan Dong^*, Dafnis Batalle, Maria Deprez

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

103 Downloads (Pure)

Abstract

Autism is a neurodevelopmental condition affecting ~1% of the population. Recently, machine learning models have been trained to classify participants with autism using their neuroimaging features, though the performance of these models varies in the literature. Differences in experimental setup hamper the direct comparison of different machine-learning approaches. In this paper, five of the most widely used and best-performing machine learning models in the field were trained to classify participants with autism and typically developing (TD) participants, using functional connectivity matrices, structural volumetric measures, and phenotypic information from the Autism Brain Imaging Data Exchange (ABIDE) dataset. Their performance was compared under the same evaluation standard. The models implemented included: graph convolutional networks (GCN), edge-variational graph convolutional networks (EV-GCN), fully connected networks (FCN), autoencoder followed by a fully connected network (AE-FCN) and support vector machine (SVM). Our results show that all models performed similarly, achieving a classification accuracy around 70%. Our results suggest that different inclusion criteria, data modalities, and evaluation pipelines rather than different machine learning models may explain variations in accuracy in the published literature. The highest accuracy in our framework was obtained when using ensemble models (p < 0.001), leading to an accuracy of 72.2% and AUC = 0.77 using GCN classifiers. However, an SVM classifier performed with an accuracy of 70.1% and AUC = 0.77, just marginally below GCN, and significant differences were not found when comparing different algorithms under the same testing conditions (p > 0.05). Furthermore, we also investigated the stability of features identified by the different machine learning models using the SmoothGrad interpretation method. The FCN model demonstrated the highest stability in selecting relevant features contributing to model decision making. The code is available at https://github.com/YilanDong19/Machine-learning-with-ABIDE.

Original language	English
Article number	e70190
Journal	Human Brain Mapping
Volume	46
Issue number	5
DOIs	https://doi.org/10.1002/hbm.70190
Publication status	Published - 1 Apr 2025

Access to Document

10.1002/hbm.70190Licence: CC BY

2025_DongEtAl_HBM_PostprintAccepted author manuscript, 1.9 MBLicence: CC BY
2025_03_DongEtAl_HBM_FINALFinal published version, 1.82 MBLicence: CC BY

Cite this

@article{98b6a42b7c39406da22e4ab865788a4d,

title = "A framework for comparison and interpretation of machine learning classifiers to predict autism on the ABIDE dataset",

abstract = "Autism is a neurodevelopmental condition affecting ~1% of the population. Recently, machine learning models have been trained to classify participants with autism using their neuroimaging features, though the performance of these models varies in the literature. Differences in experimental setup hamper the direct comparison of different machine-learning approaches. In this paper, five of the most widely used and best-performing machine learning models in the field were trained to classify participants with autism and typically developing (TD) participants, using functional connectivity matrices, structural volumetric measures, and phenotypic information from the Autism Brain Imaging Data Exchange (ABIDE) dataset. Their performance was compared under the same evaluation standard. The models implemented included: graph convolutional networks (GCN), edge-variational graph convolutional networks (EV-GCN), fully connected networks (FCN), autoencoder followed by a fully connected network (AE-FCN) and support vector machine (SVM). Our results show that all models performed similarly, achieving a classification accuracy around 70%. Our results suggest that different inclusion criteria, data modalities, and evaluation pipelines rather than different machine learning models may explain variations in accuracy in the published literature. The highest accuracy in our framework was obtained when using ensemble models (p < 0.001), leading to an accuracy of 72.2% and AUC = 0.77 using GCN classifiers. However, an SVM classifier performed with an accuracy of 70.1% and AUC = 0.77, just marginally below GCN, and significant differences were not found when comparing different algorithms under the same testing conditions (p > 0.05). Furthermore, we also investigated the stability of features identified by the different machine learning models using the SmoothGrad interpretation method. The FCN model demonstrated the highest stability in selecting relevant features contributing to model decision making. The code is available at https://github.com/YilanDong19/Machine-learning-with-ABIDE.",

author = "Yilan Dong and Dafnis Batalle and Maria Deprez",

note = "Publisher Copyright: {\textcopyright} 2025 The Author(s). Human Brain Mapping published by Wiley Periodicals LLC.",

year = "2025",

month = apr,

day = "1",

doi = "10.1002/hbm.70190",

language = "English",

volume = "46",

journal = "Human Brain Mapping",

issn = "1065-9471",

publisher = "Wiley-Liss Inc.",

number = "5",

}

TY - JOUR

T1 - A framework for comparison and interpretation of machine learning classifiers to predict autism on the ABIDE dataset

AU - Dong, Yilan

AU - Batalle, Dafnis

AU - Deprez, Maria

PY - 2025/4/1

Y1 - 2025/4/1

N2 - Autism is a neurodevelopmental condition affecting ~1% of the population. Recently, machine learning models have been trained to classify participants with autism using their neuroimaging features, though the performance of these models varies in the literature. Differences in experimental setup hamper the direct comparison of different machine-learning approaches. In this paper, five of the most widely used and best-performing machine learning models in the field were trained to classify participants with autism and typically developing (TD) participants, using functional connectivity matrices, structural volumetric measures, and phenotypic information from the Autism Brain Imaging Data Exchange (ABIDE) dataset. Their performance was compared under the same evaluation standard. The models implemented included: graph convolutional networks (GCN), edge-variational graph convolutional networks (EV-GCN), fully connected networks (FCN), autoencoder followed by a fully connected network (AE-FCN) and support vector machine (SVM). Our results show that all models performed similarly, achieving a classification accuracy around 70%. Our results suggest that different inclusion criteria, data modalities, and evaluation pipelines rather than different machine learning models may explain variations in accuracy in the published literature. The highest accuracy in our framework was obtained when using ensemble models (p < 0.001), leading to an accuracy of 72.2% and AUC = 0.77 using GCN classifiers. However, an SVM classifier performed with an accuracy of 70.1% and AUC = 0.77, just marginally below GCN, and significant differences were not found when comparing different algorithms under the same testing conditions (p > 0.05). Furthermore, we also investigated the stability of features identified by the different machine learning models using the SmoothGrad interpretation method. The FCN model demonstrated the highest stability in selecting relevant features contributing to model decision making. The code is available at https://github.com/YilanDong19/Machine-learning-with-ABIDE.

AB - Autism is a neurodevelopmental condition affecting ~1% of the population. Recently, machine learning models have been trained to classify participants with autism using their neuroimaging features, though the performance of these models varies in the literature. Differences in experimental setup hamper the direct comparison of different machine-learning approaches. In this paper, five of the most widely used and best-performing machine learning models in the field were trained to classify participants with autism and typically developing (TD) participants, using functional connectivity matrices, structural volumetric measures, and phenotypic information from the Autism Brain Imaging Data Exchange (ABIDE) dataset. Their performance was compared under the same evaluation standard. The models implemented included: graph convolutional networks (GCN), edge-variational graph convolutional networks (EV-GCN), fully connected networks (FCN), autoencoder followed by a fully connected network (AE-FCN) and support vector machine (SVM). Our results show that all models performed similarly, achieving a classification accuracy around 70%. Our results suggest that different inclusion criteria, data modalities, and evaluation pipelines rather than different machine learning models may explain variations in accuracy in the published literature. The highest accuracy in our framework was obtained when using ensemble models (p < 0.001), leading to an accuracy of 72.2% and AUC = 0.77 using GCN classifiers. However, an SVM classifier performed with an accuracy of 70.1% and AUC = 0.77, just marginally below GCN, and significant differences were not found when comparing different algorithms under the same testing conditions (p > 0.05). Furthermore, we also investigated the stability of features identified by the different machine learning models using the SmoothGrad interpretation method. The FCN model demonstrated the highest stability in selecting relevant features contributing to model decision making. The code is available at https://github.com/YilanDong19/Machine-learning-with-ABIDE.

UR - http://www.scopus.com/inward/record.url?scp=105000370369&partnerID=8YFLogxK

U2 - 10.1002/hbm.70190

DO - 10.1002/hbm.70190

M3 - Article

SN - 1065-9471

VL - 46

JO - Human Brain Mapping

JF - Human Brain Mapping

IS - 5

M1 - e70190

ER -

A framework for comparison and interpretation of machine learning classifiers to predict autism on the ABIDE dataset

Abstract

Access to Document

Other files and links

Fingerprint

Cite this