King's College London

Research portal

Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review

Standard

Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers. / Bujel, Kamil; Yannakoudakis, Helen; Rei, Marek.

RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop. ed. / Anna Rogers; Iacer Calixto; Iacer Calixto; Ivan Vulic; Naomi Saphra; Nora Kassner; Oana-Maria Camburu; Trapit Bansal; Vered Shwartz. Association for Computational Linguistics (ACL), 2021. p. 195-205 (RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop).

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review

Harvard

Bujel, K, Yannakoudakis, H & Rei, M 2021, Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers. in A Rogers, I Calixto, I Calixto, I Vulic, N Saphra, N Kassner, O-M Camburu, T Bansal & V Shwartz (eds), RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop. RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop, Association for Computational Linguistics (ACL), pp. 195-205, 6th Workshop on Representation Learning for NLP, RepL4NLP 2021, Virtual, Bangkok, Thailand, 6/08/2021.

APA

Bujel, K., Yannakoudakis, H., & Rei, M. (2021). Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers. In A. Rogers, I. Calixto, I. Calixto, I. Vulic, N. Saphra, N. Kassner, O-M. Camburu, T. Bansal, & V. Shwartz (Eds.), RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop (pp. 195-205). (RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop). Association for Computational Linguistics (ACL).

Vancouver

Bujel K, Yannakoudakis H, Rei M. Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers. In Rogers A, Calixto I, Calixto I, Vulic I, Saphra N, Kassner N, Camburu O-M, Bansal T, Shwartz V, editors, RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop. Association for Computational Linguistics (ACL). 2021. p. 195-205. (RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop).

Author

Bujel, Kamil ; Yannakoudakis, Helen ; Rei, Marek. / Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers. RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop. editor / Anna Rogers ; Iacer Calixto ; Iacer Calixto ; Ivan Vulic ; Naomi Saphra ; Nora Kassner ; Oana-Maria Camburu ; Trapit Bansal ; Vered Shwartz. Association for Computational Linguistics (ACL), 2021. pp. 195-205 (RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop).

Bibtex Download

@inbook{1eb6565cd43f4fdebc229d23fffcd996,
title = "Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers",
abstract = "We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-based architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.",
author = "Kamil Bujel and Helen Yannakoudakis and Marek Rei",
note = "Funding Information: We would like to thank James Thorne for his assistance in setting up the LIME experiments. Kamil Bujel was funded by the Undergraduate Research Opportunities Programme Bursary from the Department of Computing at Imperial College London. Publisher Copyright: {\textcopyright} 2021 Association for Computational Linguistics.; 6th Workshop on Representation Learning for NLP, RepL4NLP 2021 ; Conference date: 06-08-2021",
year = "2021",
language = "English",
series = "RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop",
publisher = "Association for Computational Linguistics (ACL)",
pages = "195--205",
editor = "Anna Rogers and Iacer Calixto and Iacer Calixto and Ivan Vulic and Naomi Saphra and Nora Kassner and Oana-Maria Camburu and Trapit Bansal and Vered Shwartz",
booktitle = "RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop",

}

RIS (suitable for import to EndNote) Download

TY - CHAP

T1 - Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers

AU - Bujel, Kamil

AU - Yannakoudakis, Helen

AU - Rei, Marek

N1 - Funding Information: We would like to thank James Thorne for his assistance in setting up the LIME experiments. Kamil Bujel was funded by the Undergraduate Research Opportunities Programme Bursary from the Department of Computing at Imperial College London. Publisher Copyright: © 2021 Association for Computational Linguistics.

PY - 2021

Y1 - 2021

N2 - We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-based architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.

AB - We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-based architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.

UR - http://www.scopus.com/inward/record.url?scp=85122402416&partnerID=8YFLogxK

M3 - Conference paper

AN - SCOPUS:85122402416

T3 - RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop

SP - 195

EP - 205

BT - RepL4NLP 2021 - 6th Workshop on Representation Learning for NLP, Proceedings of the Workshop

A2 - Rogers, Anna

A2 - Calixto, Iacer

A2 - Calixto, Iacer

A2 - Vulic, Ivan

A2 - Saphra, Naomi

A2 - Kassner, Nora

A2 - Camburu, Oana-Maria

A2 - Bansal, Trapit

A2 - Shwartz, Vered

PB - Association for Computational Linguistics (ACL)

T2 - 6th Workshop on Representation Learning for NLP, RepL4NLP 2021

Y2 - 6 August 2021

ER -

View graph of relations

© 2020 King's College London | Strand | London WC2R 2LS | England | United Kingdom | Tel +44 (0)20 7836 5454