Standard
Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers. / Bujel, Kamil; Yannakoudakis, Helen; Rei, Marek.
RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop. ed. / Anna Rogers; Iacer Calixto; Iacer Calixto; Ivan Vulic; Naomi Saphra; Nora Kassner; Oana-Maria Camburu; Trapit Bansal; Vered Shwartz. Association for Computational Linguistics (ACL), 2021. p. 195-205 (RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop).
Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review
Harvard
Bujel, K, Yannakoudakis, H & Rei, M 2021, Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers. in A Rogers, I Calixto, I Calixto, I Vulic, N Saphra, N Kassner, O-M Camburu, T Bansal & V Shwartz (eds), RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop. RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop, Association for Computational Linguistics (ACL), pp. 195-205, 6th Workshop on Representation Learning for NLP, RepL4NLP 2021, Virtual, Bangkok, Thailand, 6/08/2021.
APA
Bujel, K., Yannakoudakis, H., & Rei, M. (2021). Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers. In A. Rogers, I. Calixto, I. Calixto, I. Vulic, N. Saphra, N. Kassner, O-M. Camburu, T. Bansal, & V. Shwartz (Eds.), RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop (pp. 195-205). (RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop). Association for Computational Linguistics (ACL).
Vancouver
Bujel K, Yannakoudakis H, Rei M. Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers. In Rogers A, Calixto I, Calixto I, Vulic I, Saphra N, Kassner N, Camburu O-M, Bansal T, Shwartz V, editors, RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop. Association for Computational Linguistics (ACL). 2021. p. 195-205. (RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop).
Author
Bujel, Kamil ; Yannakoudakis, Helen ; Rei, Marek. / Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers. RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop. editor / Anna Rogers ; Iacer Calixto ; Iacer Calixto ; Ivan Vulic ; Naomi Saphra ; Nora Kassner ; Oana-Maria Camburu ; Trapit Bansal ; Vered Shwartz. Association for Computational Linguistics (ACL), 2021. pp. 195-205 (RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop).
@inbook{1eb6565cd43f4fdebc229d23fffcd996,
title = "Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers",
abstract = "We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-based architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.",
author = "Kamil Bujel and Helen Yannakoudakis and Marek Rei",
note = "Funding Information: We would like to thank James Thorne for his assistance in setting up the LIME experiments. Kamil Bujel was funded by the Undergraduate Research Opportunities Programme Bursary from the Department of Computing at Imperial College London. Publisher Copyright: {\textcopyright} 2021 Association for Computational Linguistics.; 6th Workshop on Representation Learning for NLP, RepL4NLP 2021 ; Conference date: 06-08-2021",
year = "2021",
language = "English",
series = "RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop",
publisher = "Association for Computational Linguistics (ACL)",
pages = "195--205",
editor = "Anna Rogers and Iacer Calixto and Iacer Calixto and Ivan Vulic and Naomi Saphra and Nora Kassner and Oana-Maria Camburu and Trapit Bansal and Vered Shwartz",
booktitle = "RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop",
}
RIS (suitable for import to EndNote) Download
TY - CHAP
T1 - Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers
AU - Bujel, Kamil
AU - Yannakoudakis, Helen
AU - Rei, Marek
N1 - Funding Information:
We would like to thank James Thorne for his assistance in setting up the LIME experiments. Kamil Bujel was funded by the Undergraduate Research Opportunities Programme Bursary from the Department of Computing at Imperial College London.
Publisher Copyright:
© 2021 Association for Computational Linguistics.
PY - 2021
Y1 - 2021
N2 - We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-based architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.
AB - We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-based architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.
UR - http://www.scopus.com/inward/record.url?scp=85122402416&partnerID=8YFLogxK
M3 - Conference paper
AN - SCOPUS:85122402416
T3 - RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop
SP - 195
EP - 205
BT - RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop
A2 - Rogers, Anna
A2 - Calixto, Iacer
A2 - Calixto, Iacer
A2 - Vulic, Ivan
A2 - Saphra, Naomi
A2 - Kassner, Nora
A2 - Camburu, Oana-Maria
A2 - Bansal, Trapit
A2 - Shwartz, Vered
PB - Association for Computational Linguistics (ACL)
T2 - 6th Workshop on Representation Learning for NLP, RepL4NLP 2021
Y2 - 6 August 2021
ER -