The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task

Rami Aly; Zhijiang Guo; Michael Schlichtkrull; James Thorne; Andreas Vlachos; Christos Christodoulopoulos; Oana Cocarascu; Arpit Mittal

The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task

Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

63 Citations (Scopus)

Abstract

The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) shared task, asks participating systems to determine whether human-authored claims are SUPPORTED or REFUTED based on evidence retrieved from Wikipedia (or NOTENOUGHINFO if the claim cannot be verified). Compared to the FEVER 2018 shared task, the main challenge is the addition of structured data (tables and lists) as a source of evidence. The claims in the FEVEROUS dataset can be verified using only structured evidence, only unstructured evidence, or a mixture of both. Submissions are evaluated using the FEVEROUS score that combines label accuracy and evidence retrieval. Unlike FEVER 2018 (Thorne et al., 2018a), FEVEROUS requires partial evidence to be returned for NOTENOUGHINFO claims, and the claims are longer and thus more complex. The shared task received 13 entries, six of which were able to beat the baseline system. The winning team was “Bust a move!”, achieving a FEVEROUS score of 27% (+9% compared to the baseline). In this paper we describe the shared task, present the full results and highlight commonalities and innovations among the participating systems.

Original language	English
Title of host publication	FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop
Editors	Rami Aly, Christos Christodoulopoulos, Oana Cocarascu, Zhijiang Guo, Arpit Mittal, Michael Schlichtkrull, James Thorne, Andreas Vlachos
Publisher	Association for Computational Linguistics (ACL)
Pages	1-13
Number of pages	13
ISBN (Electronic)	9781955917032
Publication status	Published - 2021
Event	4th Workshop on Fact Extraction and VERification, FEVER 2021 - Punta Cana, Dominican Republic Duration: 10 Nov 2021 → …

Publication series

Name	FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop

Conference

Conference	4th Workshop on Fact Extraction and VERification, FEVER 2021
Country/Territory	Dominican Republic
City	Punta Cana
Period	10/11/2021 → …

Access to Document

https://aclanthology.org/2021.fever-1.1.pdf

Cite this

Aly, R., Guo, Z., Schlichtkrull, M., Thorne, J., Vlachos, A., Christodoulopoulos, C., Cocarascu, O., & Mittal, A. (2021). The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task. In R. Aly, C. Christodoulopoulos, O. Cocarascu, Z. Guo, A. Mittal, M. Schlichtkrull, J. Thorne, & A. Vlachos (Eds.), FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop (pp. 1-13). (FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop). Association for Computational Linguistics (ACL). https://aclanthology.org/2021.fever-1.1.pdf

Aly, Rami ; Guo, Zhijiang ; Schlichtkrull, Michael et al. / The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task. FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop. editor / Rami Aly ; Christos Christodoulopoulos ; Oana Cocarascu ; Zhijiang Guo ; Arpit Mittal ; Michael Schlichtkrull ; James Thorne ; Andreas Vlachos. Association for Computational Linguistics (ACL), 2021. pp. 1-13 (FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop).

@inbook{1393dc629134433a9706628804a31b16,

title = "The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task",

abstract = "The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) shared task, asks participating systems to determine whether human-authored claims are SUPPORTED or REFUTED based on evidence retrieved from Wikipedia (or NOTENOUGHINFO if the claim cannot be verified). Compared to the FEVER 2018 shared task, the main challenge is the addition of structured data (tables and lists) as a source of evidence. The claims in the FEVEROUS dataset can be verified using only structured evidence, only unstructured evidence, or a mixture of both. Submissions are evaluated using the FEVEROUS score that combines label accuracy and evidence retrieval. Unlike FEVER 2018 (Thorne et al., 2018a), FEVEROUS requires partial evidence to be returned for NOTENOUGHINFO claims, and the claims are longer and thus more complex. The shared task received 13 entries, six of which were able to beat the baseline system. The winning team was “Bust a move!”, achieving a FEVEROUS score of 27% (+9% compared to the baseline). In this paper we describe the shared task, present the full results and highlight commonalities and innovations among the participating systems.",

author = "Rami Aly and Zhijiang Guo and Michael Schlichtkrull and James Thorne and Andreas Vlachos and Christos Christodoulopoulos and Oana Cocarascu and Arpit Mittal",

note = "Funding Information: We would like to thank Amazon for sponsoring the dataset generation and supporting the FEVER workshop and the FEVEROUS shared task. Rami Aly is supported by the Engineering and Physical Sciences Research Council Doctoral Training Partnership (EPSRC). James Thorne is supported by an Amazon Alexa Graduate Research Fellowship. Zhijiang Guo, Michael Schlichtkrull and Andreas Vlachos are supported by the ERC grant AVeriTeC (GA 865958). Publisher Copyright: {\textcopyright} 2021 Association for Computational Linguistics.; 4th Workshop on Fact Extraction and VERification, FEVER 2021 ; Conference date: 10-11-2021",

year = "2021",

language = "English",

series = "FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1--13",

editor = "Rami Aly and Christos Christodoulopoulos and Oana Cocarascu and Zhijiang Guo and Arpit Mittal and Michael Schlichtkrull and James Thorne and Andreas Vlachos",

booktitle = "FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop",

}

Aly, R, Guo, Z, Schlichtkrull, M, Thorne, J, Vlachos, A, Christodoulopoulos, C, Cocarascu, O & Mittal, A 2021, The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task. in R Aly, C Christodoulopoulos, O Cocarascu, Z Guo, A Mittal, M Schlichtkrull, J Thorne & A Vlachos (eds), FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop. FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop, Association for Computational Linguistics (ACL), pp. 1-13, 4th Workshop on Fact Extraction and VERification, FEVER 2021, Punta Cana, Dominican Republic, 10/11/2021. <https://aclanthology.org/2021.fever-1.1.pdf>

The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task. / Aly, Rami; Guo, Zhijiang; Schlichtkrull, Michael et al.
FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop. ed. / Rami Aly; Christos Christodoulopoulos; Oana Cocarascu; Zhijiang Guo; Arpit Mittal; Michael Schlichtkrull; James Thorne; Andreas Vlachos. Association for Computational Linguistics (ACL), 2021. p. 1-13 (FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

TY - CHAP

T1 - The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task

AU - Aly, Rami

AU - Guo, Zhijiang

AU - Schlichtkrull, Michael

AU - Thorne, James

AU - Vlachos, Andreas

AU - Christodoulopoulos, Christos

AU - Cocarascu, Oana

AU - Mittal, Arpit

N1 - Funding Information: We would like to thank Amazon for sponsoring the dataset generation and supporting the FEVER workshop and the FEVEROUS shared task. Rami Aly is supported by the Engineering and Physical Sciences Research Council Doctoral Training Partnership (EPSRC). James Thorne is supported by an Amazon Alexa Graduate Research Fellowship. Zhijiang Guo, Michael Schlichtkrull and Andreas Vlachos are supported by the ERC grant AVeriTeC (GA 865958). Publisher Copyright: © 2021 Association for Computational Linguistics.

PY - 2021

Y1 - 2021

N2 - The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) shared task, asks participating systems to determine whether human-authored claims are SUPPORTED or REFUTED based on evidence retrieved from Wikipedia (or NOTENOUGHINFO if the claim cannot be verified). Compared to the FEVER 2018 shared task, the main challenge is the addition of structured data (tables and lists) as a source of evidence. The claims in the FEVEROUS dataset can be verified using only structured evidence, only unstructured evidence, or a mixture of both. Submissions are evaluated using the FEVEROUS score that combines label accuracy and evidence retrieval. Unlike FEVER 2018 (Thorne et al., 2018a), FEVEROUS requires partial evidence to be returned for NOTENOUGHINFO claims, and the claims are longer and thus more complex. The shared task received 13 entries, six of which were able to beat the baseline system. The winning team was “Bust a move!”, achieving a FEVEROUS score of 27% (+9% compared to the baseline). In this paper we describe the shared task, present the full results and highlight commonalities and innovations among the participating systems.

AB - The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) shared task, asks participating systems to determine whether human-authored claims are SUPPORTED or REFUTED based on evidence retrieved from Wikipedia (or NOTENOUGHINFO if the claim cannot be verified). Compared to the FEVER 2018 shared task, the main challenge is the addition of structured data (tables and lists) as a source of evidence. The claims in the FEVEROUS dataset can be verified using only structured evidence, only unstructured evidence, or a mixture of both. Submissions are evaluated using the FEVEROUS score that combines label accuracy and evidence retrieval. Unlike FEVER 2018 (Thorne et al., 2018a), FEVEROUS requires partial evidence to be returned for NOTENOUGHINFO claims, and the claims are longer and thus more complex. The shared task received 13 entries, six of which were able to beat the baseline system. The winning team was “Bust a move!”, achieving a FEVEROUS score of 27% (+9% compared to the baseline). In this paper we describe the shared task, present the full results and highlight commonalities and innovations among the participating systems.

UR - http://www.scopus.com/inward/record.url?scp=85123469969&partnerID=8YFLogxK

M3 - Conference paper

AN - SCOPUS:85123469969

T3 - FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop

SP - 1

EP - 13

BT - FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop

A2 - Aly, Rami

A2 - Christodoulopoulos, Christos

A2 - Cocarascu, Oana

A2 - Guo, Zhijiang

A2 - Mittal, Arpit

A2 - Schlichtkrull, Michael

A2 - Thorne, James

A2 - Vlachos, Andreas

PB - Association for Computational Linguistics (ACL)

T2 - 4th Workshop on Fact Extraction and VERification, FEVER 2021

Y2 - 10 November 2021

ER -

Aly R, Guo Z, Schlichtkrull M, Thorne J, Vlachos A, Christodoulopoulos C et al. The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task. In Aly R, Christodoulopoulos C, Cocarascu O, Guo Z, Mittal A, Schlichtkrull M, Thorne J, Vlachos A, editors, FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop. Association for Computational Linguistics (ACL). 2021. p. 1-13. (FEVER 2021 - Fact Extraction and VERification, Proceedings of the 4th Workshop).

The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task

Abstract

Publication series

Conference

Access to Document

Other files and links

Cite this