Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers

Kamil Bujel, Helen Yannakoudakis, Marek Rei

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review

5 Citations (Scopus)

Abstract

We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-based architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.

Original languageEnglish
Title of host publicationRepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop
EditorsAnna Rogers, Iacer Calixto, Iacer Calixto, Ivan Vulic, Naomi Saphra, Nora Kassner, Oana-Maria Camburu, Trapit Bansal, Vered Shwartz
PublisherAssociation for Computational Linguistics (ACL)
Pages195-205
Number of pages11
ISBN (Electronic)9781954085725
Publication statusPublished - 2021
Event6th Workshop on Representation Learning for NLP, RepL4NLP 2021 - Virtual, Bangkok, Thailand
Duration: 6 Aug 2021 → …

Publication series

NameRepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop

Conference

Conference6th Workshop on Representation Learning for NLP, RepL4NLP 2021
Country/TerritoryThailand
CityVirtual, Bangkok
Period6/08/2021 → …

Fingerprint

Dive into the research topics of 'Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers'. Together they form a unique fingerprint.

Cite this