Scientific and Creative Analogies in Pretrained Language Models

Tamara Czinczoll; Helen Yannakoudakis; Pushkar Mishra; Ekaterina Shutova

doi:10.18653/v1/2022.findings-emnlp.153

Scientific and Creative Analogies in Pretrained Language Models

Tamara Czinczoll
, Helen Yannakoudakis
, Pushkar Mishra
, Ekaterina Shutova

University of Amsterdam
University of Potsdam
King's College London
King's College London
Meta AI

Research output: Contribution to conference types › Paper › peer-review

14 Citations (Scopus)

Abstract

This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2. Existing analogy datasets typically focus on a limited set of analogical relations, with a high similarity of the two domains between which the analogy holds. As a more realistic setup, we introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains. Using this dataset, we test the analogical reasoning capabilities of several widely-used pretrained language models (LMs). We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.

Original language	English
Pages	2094-2100
Number of pages	7
DOIs	https://doi.org/10.18653/v1/2022.findings-emnlp.153
Publication status	Published - 2022
Event	2022 Findings of the Association for Computational Linguistics: EMNLP 2022 - Abu Dhabi, United Arab Emirates Duration: 7 Dec 2022 → 11 Dec 2022

Conference

Conference	2022 Findings of the Association for Computational Linguistics: EMNLP 2022
Country/Territory	United Arab Emirates
City	Abu Dhabi
Period	7/12/2022 → 11/12/2022

Access to Document

10.18653/v1/2022.findings-emnlp.153

Cite this

@conference{56e31d4451af4d9cb1d20f6839554d1d,

title = "Scientific and Creative Analogies in Pretrained Language Models",

abstract = "This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2. Existing analogy datasets typically focus on a limited set of analogical relations, with a high similarity of the two domains between which the analogy holds. As a more realistic setup, we introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains. Using this dataset, we test the analogical reasoning capabilities of several widely-used pretrained language models (LMs). We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.",

author = "Tamara Czinczoll and Helen Yannakoudakis and Pushkar Mishra and Ekaterina Shutova",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 2022 Findings of the Association for Computational Linguistics: EMNLP 2022 ; Conference date: 07-12-2022 Through 11-12-2022",

year = "2022",

doi = "10.18653/v1/2022.findings-emnlp.153",

language = "English",

pages = "2094--2100",

}

TY - CONF

T1 - Scientific and Creative Analogies in Pretrained Language Models

AU - Czinczoll, Tamara

AU - Yannakoudakis, Helen

AU - Mishra, Pushkar

AU - Shutova, Ekaterina

PY - 2022

Y1 - 2022

N2 - This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2. Existing analogy datasets typically focus on a limited set of analogical relations, with a high similarity of the two domains between which the analogy holds. As a more realistic setup, we introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains. Using this dataset, we test the analogical reasoning capabilities of several widely-used pretrained language models (LMs). We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.

AB - This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2. Existing analogy datasets typically focus on a limited set of analogical relations, with a high similarity of the two domains between which the analogy holds. As a more realistic setup, we introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains. Using this dataset, we test the analogical reasoning capabilities of several widely-used pretrained language models (LMs). We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.

UR - https://www.scopus.com/pages/publications/85149846242

U2 - 10.18653/v1/2022.findings-emnlp.153

DO - 10.18653/v1/2022.findings-emnlp.153

M3 - Paper

AN - SCOPUS:85149846242

SP - 2094

EP - 2100

T2 - 2022 Findings of the Association for Computational Linguistics: EMNLP 2022

Y2 - 7 December 2022 through 11 December 2022

ER -

Scientific and Creative Analogies in Pretrained Language Models

Abstract

Conference

Access to Document

Other files and links

Fingerprint

Cite this