Multi-factorial analysis of class prediction error: estimating optimal number of biomarkers for various classification rules

Mizanur Khondoker; Till T Bachmann; Muriel Mewissen; Paul Dickinson; Bartosz Dobrzelecki; Colin J Campbell; Andrew R Mount; Anthony J Walton; Jason Crain; Holger Schulze; Gerard Giraud; Alan J Ross; Ilenia Ciani; Stuart W J Ember; Chaker Tlili; Jonathan G Terry; Eilidh Grant; Nicola McDonnell; Peter Ghazal

doi:10.1142/S0219720010005063

Multi-factorial analysis of class prediction error: estimating optimal number of biomarkers for various classification rules

Mizanur Khondoker, Till T Bachmann, Muriel Mewissen, Paul Dickinson, Bartosz Dobrzelecki, Colin J Campbell, Andrew R Mount, Anthony J Walton, Jason Crain, Holger Schulze, Gerard Giraud, Alan J Ross, Ilenia Ciani, Stuart W J Ember, Chaker Tlili, Jonathan G Terry, Eilidh Grant, Nicola McDonnell, Peter Ghazal

Research output: Contribution to journal › Article › peer-review

13 Citations (Scopus)

Abstract

Machine learning and statistical model based classifiers have increasingly been used with more complex and high dimensional biological data obtained from high-throughput technologies. Understanding the impact of various factors associated with large and complex microarray datasets on the predictive performance of classifiers is computationally intensive, under investigated, yet vital in determining the optimal number of biomarkers for various classification purposes aimed towards improved detection, diagnosis, and therapeutic monitoring of diseases. We investigate the impact of microarray based data characteristics on the predictive performance for various classification rules using simulation studies. Our investigation using Random Forest, Support Vector Machines, Linear Discriminant Analysis and k-Nearest Neighbour shows that the predictive performance of classifiers is strongly influenced by training set size, biological and technical variability, replication, fold change and correlation between biomarkers. Optimal number of biomarkers for a classification problem should therefore be estimated taking account of the impact of all these factors. A database of average generalization errors is built for various combinations of these factors. The database of generalization errors can be used for estimating the optimal number of biomarkers for given levels of predictive accuracy as a function of these factors. Examples show that curves from actual biological data resemble that of simulated data with corresponding levels of data characteristics. An R package optBiomarker implementing the method is freely available for academic use from the Comprehensive R Archive Network (http://www.cran.r-project.org/web/packages/optBiomarker/).

Original language	English
Pages (from-to)	945-965
Number of pages	21
Journal	Journal of Bioinformatics and Computational Biology
Volume	8
Issue number	6
DOIs	https://doi.org/10.1142/S0219720010005063
Publication status	Published - Dec 2010

Access to Document

10.1142/S0219720010005063

Cite this

Khondoker, M., Bachmann, T. T., Mewissen, M., Dickinson, P., Dobrzelecki, B., Campbell, C. J., Mount, A. R., Walton, A. J., Crain, J., Schulze, H., Giraud, G., Ross, A. J., Ciani, I., Ember, S. W. J., Tlili, C., Terry, J. G., Grant, E., McDonnell, N., & Ghazal, P. (2010). Multi-factorial analysis of class prediction error: estimating optimal number of biomarkers for various classification rules. Journal of Bioinformatics and Computational Biology, 8(6), 945-965. https://doi.org/10.1142/S0219720010005063

@article{0c461a5a5b9d4171ac626447a4d48fbc,

title = "Multi-factorial analysis of class prediction error: estimating optimal number of biomarkers for various classification rules",

abstract = "Machine learning and statistical model based classifiers have increasingly been used with more complex and high dimensional biological data obtained from high-throughput technologies. Understanding the impact of various factors associated with large and complex microarray datasets on the predictive performance of classifiers is computationally intensive, under investigated, yet vital in determining the optimal number of biomarkers for various classification purposes aimed towards improved detection, diagnosis, and therapeutic monitoring of diseases. We investigate the impact of microarray based data characteristics on the predictive performance for various classification rules using simulation studies. Our investigation using Random Forest, Support Vector Machines, Linear Discriminant Analysis and k-Nearest Neighbour shows that the predictive performance of classifiers is strongly influenced by training set size, biological and technical variability, replication, fold change and correlation between biomarkers. Optimal number of biomarkers for a classification problem should therefore be estimated taking account of the impact of all these factors. A database of average generalization errors is built for various combinations of these factors. The database of generalization errors can be used for estimating the optimal number of biomarkers for given levels of predictive accuracy as a function of these factors. Examples show that curves from actual biological data resemble that of simulated data with corresponding levels of data characteristics. An R package optBiomarker implementing the method is freely available for academic use from the Comprehensive R Archive Network (http://www.cran.r-project.org/web/packages/optBiomarker/).",

author = "Mizanur Khondoker and Bachmann, {Till T} and Muriel Mewissen and Paul Dickinson and Bartosz Dobrzelecki and Campbell, {Colin J} and Mount, {Andrew R} and Walton, {Anthony J} and Jason Crain and Holger Schulze and Gerard Giraud and Ross, {Alan J} and Ilenia Ciani and Ember, {Stuart W J} and Chaker Tlili and Terry, {Jonathan G} and Eilidh Grant and Nicola McDonnell and Peter Ghazal",

year = "2010",

month = dec,

doi = "10.1142/S0219720010005063",

language = "English",

volume = "8",

pages = "945--965",

journal = "Journal of Bioinformatics and Computational Biology",

issn = "0219-7200",

publisher = "World Scientific Publishing Co. Pte Ltd",

number = "6",

}

Khondoker, M, Bachmann, TT, Mewissen, M, Dickinson, P, Dobrzelecki, B, Campbell, CJ, Mount, AR, Walton, AJ, Crain, J, Schulze, H, Giraud, G, Ross, AJ, Ciani, I, Ember, SWJ, Tlili, C, Terry, JG, Grant, E, McDonnell, N & Ghazal, P 2010, 'Multi-factorial analysis of class prediction error: estimating optimal number of biomarkers for various classification rules', Journal of Bioinformatics and Computational Biology, vol. 8, no. 6, pp. 945-965. https://doi.org/10.1142/S0219720010005063

TY - JOUR

T1 - Multi-factorial analysis of class prediction error

T2 - estimating optimal number of biomarkers for various classification rules

AU - Khondoker, Mizanur

AU - Bachmann, Till T

AU - Mewissen, Muriel

AU - Dickinson, Paul

AU - Dobrzelecki, Bartosz

AU - Campbell, Colin J

AU - Mount, Andrew R

AU - Walton, Anthony J

AU - Crain, Jason

AU - Schulze, Holger

AU - Giraud, Gerard

AU - Ross, Alan J

AU - Ciani, Ilenia

AU - Ember, Stuart W J

AU - Tlili, Chaker

AU - Terry, Jonathan G

AU - Grant, Eilidh

AU - McDonnell, Nicola

AU - Ghazal, Peter

PY - 2010/12

Y1 - 2010/12

N2 - Machine learning and statistical model based classifiers have increasingly been used with more complex and high dimensional biological data obtained from high-throughput technologies. Understanding the impact of various factors associated with large and complex microarray datasets on the predictive performance of classifiers is computationally intensive, under investigated, yet vital in determining the optimal number of biomarkers for various classification purposes aimed towards improved detection, diagnosis, and therapeutic monitoring of diseases. We investigate the impact of microarray based data characteristics on the predictive performance for various classification rules using simulation studies. Our investigation using Random Forest, Support Vector Machines, Linear Discriminant Analysis and k-Nearest Neighbour shows that the predictive performance of classifiers is strongly influenced by training set size, biological and technical variability, replication, fold change and correlation between biomarkers. Optimal number of biomarkers for a classification problem should therefore be estimated taking account of the impact of all these factors. A database of average generalization errors is built for various combinations of these factors. The database of generalization errors can be used for estimating the optimal number of biomarkers for given levels of predictive accuracy as a function of these factors. Examples show that curves from actual biological data resemble that of simulated data with corresponding levels of data characteristics. An R package optBiomarker implementing the method is freely available for academic use from the Comprehensive R Archive Network (http://www.cran.r-project.org/web/packages/optBiomarker/).

AB - Machine learning and statistical model based classifiers have increasingly been used with more complex and high dimensional biological data obtained from high-throughput technologies. Understanding the impact of various factors associated with large and complex microarray datasets on the predictive performance of classifiers is computationally intensive, under investigated, yet vital in determining the optimal number of biomarkers for various classification purposes aimed towards improved detection, diagnosis, and therapeutic monitoring of diseases. We investigate the impact of microarray based data characteristics on the predictive performance for various classification rules using simulation studies. Our investigation using Random Forest, Support Vector Machines, Linear Discriminant Analysis and k-Nearest Neighbour shows that the predictive performance of classifiers is strongly influenced by training set size, biological and technical variability, replication, fold change and correlation between biomarkers. Optimal number of biomarkers for a classification problem should therefore be estimated taking account of the impact of all these factors. A database of average generalization errors is built for various combinations of these factors. The database of generalization errors can be used for estimating the optimal number of biomarkers for given levels of predictive accuracy as a function of these factors. Examples show that curves from actual biological data resemble that of simulated data with corresponding levels of data characteristics. An R package optBiomarker implementing the method is freely available for academic use from the Comprehensive R Archive Network (http://www.cran.r-project.org/web/packages/optBiomarker/).

U2 - 10.1142/S0219720010005063

DO - 10.1142/S0219720010005063

M3 - Article

C2 - 21121020

SN - 0219-7200

VL - 8

SP - 945

EP - 965

JO - Journal of Bioinformatics and Computational Biology

JF - Journal of Bioinformatics and Computational Biology

IS - 6

ER -

Multi-factorial analysis of class prediction error: estimating optimal number of biomarkers for various classification rules

Abstract

Access to Document

Fingerprint

Cite this