Evaluation of Distributional Semantic Models of Ancient Greek: Preliminary Results and a Road Map for Future Work

Silvia Stopponi; Nilo Pedrazzini; Saskia Peels; Barbara McGillivray; Malvina Nissim

doi:10.26615/978-954-452-087-8_006

Evaluation of Distributional Semantic Models of Ancient Greek: Preliminary Results and a Road Map for Future Work

Silvia Stopponi, Nilo Pedrazzini, Saskia Peels, Barbara McGillivray, Malvina Nissim

Digital Humanities

Research output: Contribution to conference types › Paper › peer-review

1 Citation (Scopus)

Abstract

We evaluate four count-based and predictive distributional semantic models of Ancient Greek against AGREE, a composite benchmark of human judgements, to assess their ability to retrieve semantic relatedness. On the basis of the observations deriving from the analysis of the results, we design a procedure for a larger-scale intrinsic evaluation of count-based and predictive language models, including syntactic embeddings. We also propose possible ways of exploiting the different layers of the whole AGREE benchmark (including both human- and machine-generated data) and different evaluation metrics.

Original language	English
Pages	49-58
Number of pages	10
DOIs	https://doi.org/10.26615/978-954-452-087-8_006
Publication status	Published - Sept 2023
Event	1st Workshop on Ancient Language Processing, ALP 2023 - Varna, Bulgaria Duration: 8 Sept 2023 → …

Conference

Conference	1st Workshop on Ancient Language Processing, ALP 2023
Country/Territory	Bulgaria
City	Varna
Period	8/09/2023 → …

Access to Document

10.26615/978-954-452-087-8_006

Cite this

@conference{ca21c21c0f194545bfee9dc773dad31c,

title = "Evaluation of Distributional Semantic Models of Ancient Greek: Preliminary Results and a Road Map for Future Work",

abstract = "We evaluate four count-based and predictive distributional semantic models of Ancient Greek against AGREE, a composite benchmark of human judgements, to assess their ability to retrieve semantic relatedness. On the basis of the observations deriving from the analysis of the results, we design a procedure for a larger-scale intrinsic evaluation of count-based and predictive language models, including syntactic embeddings. We also propose possible ways of exploiting the different layers of the whole AGREE benchmark (including both human- and machine-generated data) and different evaluation metrics.",

author = "Silvia Stopponi and Nilo Pedrazzini and Saskia Peels and Barbara McGillivray and Malvina Nissim",

note = "Funding Information: This work was partially supported by the Young Academy Groningen through the PhD scholarship of Silvia Stopponi. Publisher Copyright: {\textcopyright} RANLP-ALP 2023 - Proceedings of the Ancient Language Processing Workshop, associated with 14th International Conference on Recent Advances in Natural Language Processing.; 1st Workshop on Ancient Language Processing, ALP 2023 ; Conference date: 08-09-2023",

year = "2023",

month = sep,

doi = "10.26615/978-954-452-087-8_006",

language = "English",

pages = "49--58",

}

TY - CONF

T1 - Evaluation of Distributional Semantic Models of Ancient Greek

T2 - 1st Workshop on Ancient Language Processing, ALP 2023

AU - Stopponi, Silvia

AU - Pedrazzini, Nilo

AU - Peels, Saskia

AU - McGillivray, Barbara

AU - Nissim, Malvina

N1 - Funding Information: This work was partially supported by the Young Academy Groningen through the PhD scholarship of Silvia Stopponi. Publisher Copyright: © RANLP-ALP 2023 - Proceedings of the Ancient Language Processing Workshop, associated with 14th International Conference on Recent Advances in Natural Language Processing.

PY - 2023/9

Y1 - 2023/9

N2 - We evaluate four count-based and predictive distributional semantic models of Ancient Greek against AGREE, a composite benchmark of human judgements, to assess their ability to retrieve semantic relatedness. On the basis of the observations deriving from the analysis of the results, we design a procedure for a larger-scale intrinsic evaluation of count-based and predictive language models, including syntactic embeddings. We also propose possible ways of exploiting the different layers of the whole AGREE benchmark (including both human- and machine-generated data) and different evaluation metrics.

AB - We evaluate four count-based and predictive distributional semantic models of Ancient Greek against AGREE, a composite benchmark of human judgements, to assess their ability to retrieve semantic relatedness. On the basis of the observations deriving from the analysis of the results, we design a procedure for a larger-scale intrinsic evaluation of count-based and predictive language models, including syntactic embeddings. We also propose possible ways of exploiting the different layers of the whole AGREE benchmark (including both human- and machine-generated data) and different evaluation metrics.

UR - http://www.scopus.com/inward/record.url?scp=85182637059&partnerID=8YFLogxK

UR - https://aclanthology.org/2023.alp-1.6/

U2 - 10.26615/978-954-452-087-8_006

DO - 10.26615/978-954-452-087-8_006

M3 - Paper

AN - SCOPUS:85182637059

SP - 49

EP - 58

Y2 - 8 September 2023

ER -

Evaluation of Distributional Semantic Models of Ancient Greek: Preliminary Results and a Road Map for Future Work

Abstract

Conference

Access to Document

Other files and links

Fingerprint

Cite this