King's College London

Research portal

Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review

Kamil Bujel, Helen Yannakoudakis, Marek Rei

Original languageEnglish
Title of host publicationRepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop
EditorsAnna Rogers, Iacer Calixto, Iacer Calixto, Ivan Vulic, Naomi Saphra, Nora Kassner, Oana-Maria Camburu, Trapit Bansal, Vered Shwartz
PublisherAssociation for Computational Linguistics (ACL)
Pages195-205
Number of pages11
ISBN (Electronic)9781954085725
Published2021
Event6th Workshop on Representation Learning for NLP, RepL4NLP 2021 - Virtual, Bangkok, Thailand
Duration: 6 Aug 2021 → …

Publication series

NameRepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop

Conference

Conference6th Workshop on Representation Learning for NLP, RepL4NLP 2021
Country/TerritoryThailand
CityVirtual, Bangkok
Period6/08/2021 → …

Bibliographical note

Funding Information: We would like to thank James Thorne for his assistance in setting up the LIME experiments. Kamil Bujel was funded by the Undergraduate Research Opportunities Programme Bursary from the Department of Computing at Imperial College London. Publisher Copyright: © 2021 Association for Computational Linguistics.

King's Authors

Abstract

We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-based architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.

View graph of relations

© 2020 King's College London | Strand | London WC2R 2LS | England | United Kingdom | Tel +44 (0)20 7836 5454