TY - CHAP
T1 - Uncertainty Quantification for Text Classification
AU - Zhang, Dell
AU - Sensoy, Murat
AU - Makrehchi, Masoud
AU - Taneva-Popova, Bilyana
AU - Gui, Lin
AU - He, Yulan
N1 - Funding Information:
Yulan He is a Professor in NLP at King’s College London. Her research centers on the integration of machine learning and NLP for text understanding. She currently holds a five-year Turing AI Fellowship, funded by the UK Research and Innovation. She is named among the world’s top 2% most cited researchers in her field in a Stanford report. She has received several prizes and awards, including a SWSA Ten-Year Award, a CIKM 2020 Test-of-Time Award, and AI 2020 Most Influential Scholar (Honourable Mention) by AMiner. Yulan obtained her PhD degree in Spoken Language Understanding from the University of Cambridge. She has delivered tutorials in Oxford Machine Learning Summer School in 2021 and 2022.
Lin Gui is a Lecturer in NLP at King’s College London. He has won several prestigious awards or fellowships, including the Marie Sklodowska-Curie Actions individual Fellowship, funded by the EU-H2020 in 2018, and the National ‘Best PhD thesis’ award by the China Association of Artificial Intelligence in 2018. His research interests include disentangled representation learning and interpretable machine learning.
This work is supported by the UK Engineering and Physical Sciences Research Council (EPSRC) under Grant No.: EP/V020579/2, EP/T017112/2 and EP/X019063/1.
Publisher Copyright:
© 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.
PY - 2023/7/18
Y1 - 2023/7/18
N2 - This full-day tutorial introduces modern techniques for practical uncertainty quantification specifically in the context of multi-class and multi-label text classification. First, we explain the usefulness of estimating aleatoric uncertainty and epistemic uncertainty for text classification models. Then, we describe several state-of-the-art approaches to uncertainty quantification and analyze their scalability to big text data: Virtual Ensemble in GBDT, Bayesian Deep Learning (including Deep Ensemble, Monte-Carlo Dropout, Bayes by Backprop, and their generalization Epistemic Neural Networks), Evidential Deep Learning (including Prior Networks and Posterior Networks), as well as Distance Awareness (including Spectral-normalized Neural Gaussian Process and Deep Deterministic Uncertainty). Next, we talk about the latest advances in uncertainty quantification for pre-trained language models (including asking language models to express their uncertainty, interpreting uncertainties of text classifiers built on large-scale language models, uncertainty estimation in text generation, calibration of language models, and calibration for in-context learning). After that, we discuss typical application scenarios of uncertainty quantification in text classification (including in-domain calibration, cross-domain robustness, and novel class detection). Finally, we list popular performance metrics for the evaluation of uncertainty quantification effectiveness in text classification. Practical hands-on examples/exercises are provided to the attendees for them to experiment with different uncertainty quantification methods on a few real-world text classification datasets such as CLINC150.
AB - This full-day tutorial introduces modern techniques for practical uncertainty quantification specifically in the context of multi-class and multi-label text classification. First, we explain the usefulness of estimating aleatoric uncertainty and epistemic uncertainty for text classification models. Then, we describe several state-of-the-art approaches to uncertainty quantification and analyze their scalability to big text data: Virtual Ensemble in GBDT, Bayesian Deep Learning (including Deep Ensemble, Monte-Carlo Dropout, Bayes by Backprop, and their generalization Epistemic Neural Networks), Evidential Deep Learning (including Prior Networks and Posterior Networks), as well as Distance Awareness (including Spectral-normalized Neural Gaussian Process and Deep Deterministic Uncertainty). Next, we talk about the latest advances in uncertainty quantification for pre-trained language models (including asking language models to express their uncertainty, interpreting uncertainties of text classifiers built on large-scale language models, uncertainty estimation in text generation, calibration of language models, and calibration for in-context learning). After that, we discuss typical application scenarios of uncertainty quantification in text classification (including in-domain calibration, cross-domain robustness, and novel class detection). Finally, we list popular performance metrics for the evaluation of uncertainty quantification effectiveness in text classification. Practical hands-on examples/exercises are provided to the attendees for them to experiment with different uncertainty quantification methods on a few real-world text classification datasets such as CLINC150.
KW - language models
KW - text classification
KW - uncertainty quantification
UR - http://www.scopus.com/inward/record.url?scp=85168671119&partnerID=8YFLogxK
U2 - 10.1145/3539618.3594243
DO - 10.1145/3539618.3594243
M3 - Chapter
AN - SCOPUS:85168671119
SN - 9781450394086
SP - 3426
EP - 3429
BT - SIGIR '23
PB - ACM
T2 - 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023
Y2 - 23 July 2023 through 27 July 2023
ER -