Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking

Mubashara Akhtar; Oana Cocarascu; Elena Simperl

Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking

Mubashara Akhtar, Oana Cocarascu, Elena Simperl

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

15 Downloads (Pure)

Abstract

Evidence data for automated fact-checking (AFC) can be in multiple modalities such as text, tables, images, audio, or video. While there is increasing interest in using images for AFC, previous works mostly focus on detecting manipulated or fake images. We propose a novel task, chart-based fact-checking, and introduce ChartBERT as the first model for AFC against chart evidence. ChartBERT leverages textual, structural and visual information of charts to determine the veracity of textual claims. For evaluation, we create ChartFC, a new dataset of 15, 886 charts. We systematically evaluate 75 different vision-language (VL) baselines and show that ChartBERT outperforms VL models, achieving 63.8% accuracy. Our results suggest that the task is complex yet feasible, with many challenges ahead.

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics
Subtitle of host publication	EACL 2023 - Findings
Publisher	Association for Computational Linguistics (ACL)
Publication status	Published - 2023
Event	17th Conference of the European Chapter of the Association for Computational Linguistics: EACL 2023 - Dubrovnik, Croatia Duration: 2 May 2023 → 4 May 2023

Publication series

Name	Findings of the Association for Computational Linguistics: EACL 2023 - Findings

Conference

Conference	17th Conference of the European Chapter of the Association for Computational Linguistics
Country/Territory	Croatia
City	Dubrovnik
Period	2/05/2023 → 4/05/2023

Keywords

fact checking
misinformation
chart misinformation
automated fact checking
fact verification
natural language processing
nlp

Access to Document

final_version_EACL23Final published version, 733 KB

Cite this

@inbook{6ba1ee4877be4e92932dff3670301b7a,

title = "Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking",

abstract = "Evidence data for automated fact-checking (AFC) can be in multiple modalities such as text, tables, images, audio, or video. While there is increasing interest in using images for AFC, previous works mostly focus on detecting manipulated or fake images. We propose a novel task, chart-based fact-checking, and introduce ChartBERT as the first model for AFC against chart evidence. ChartBERT leverages textual, structural and visual information of charts to determine the veracity of textual claims. For evaluation, we create ChartFC, a new dataset of 15, 886 charts. We systematically evaluate 75 different vision-language (VL) baselines and show that ChartBERT outperforms VL models, achieving 63.8% accuracy. Our results suggest that the task is complex yet feasible, with many challenges ahead.",

keywords = "fact checking, misinformation, chart misinformation, automated fact checking, fact verification, natural language processing, nlp",

author = "Mubashara Akhtar and Oana Cocarascu and Elena Simperl",

year = "2023",

language = "English",

series = "Findings of the Association for Computational Linguistics: EACL 2023 - Findings",

publisher = "Association for Computational Linguistics (ACL)",

booktitle = "Findings of the Association for Computational Linguistics",

note = "17th Conference of the European Chapter of the Association for Computational Linguistics : EACL 2023 ; Conference date: 02-05-2023 Through 04-05-2023",

}

Akhtar, M , Cocarascu, O & Simperl, E 2023, Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking. in Findings of the Association for Computational Linguistics: EACL 2023 - Findings. Findings of the Association for Computational Linguistics: EACL 2023 - Findings, Association for Computational Linguistics (ACL), 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia, 2/05/2023.

Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking. / Akhtar, Mubashara ; Cocarascu, Oana ; Simperl, Elena.

Findings of the Association for Computational Linguistics: EACL 2023 - Findings. Association for Computational Linguistics (ACL), 2023. (Findings of the Association for Computational Linguistics: EACL 2023 - Findings).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

TY - CHAP

T1 - Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking

AU - Akhtar, Mubashara

AU - Cocarascu, Oana

AU - Simperl, Elena

PY - 2023

Y1 - 2023

N2 - Evidence data for automated fact-checking (AFC) can be in multiple modalities such as text, tables, images, audio, or video. While there is increasing interest in using images for AFC, previous works mostly focus on detecting manipulated or fake images. We propose a novel task, chart-based fact-checking, and introduce ChartBERT as the first model for AFC against chart evidence. ChartBERT leverages textual, structural and visual information of charts to determine the veracity of textual claims. For evaluation, we create ChartFC, a new dataset of 15, 886 charts. We systematically evaluate 75 different vision-language (VL) baselines and show that ChartBERT outperforms VL models, achieving 63.8% accuracy. Our results suggest that the task is complex yet feasible, with many challenges ahead.

AB - Evidence data for automated fact-checking (AFC) can be in multiple modalities such as text, tables, images, audio, or video. While there is increasing interest in using images for AFC, previous works mostly focus on detecting manipulated or fake images. We propose a novel task, chart-based fact-checking, and introduce ChartBERT as the first model for AFC against chart evidence. ChartBERT leverages textual, structural and visual information of charts to determine the veracity of textual claims. For evaluation, we create ChartFC, a new dataset of 15, 886 charts. We systematically evaluate 75 different vision-language (VL) baselines and show that ChartBERT outperforms VL models, achieving 63.8% accuracy. Our results suggest that the task is complex yet feasible, with many challenges ahead.

KW - fact checking

KW - misinformation

KW - chart misinformation

KW - automated fact checking

KW - fact verification

KW - natural language processing

KW - nlp

M3 - Conference paper

T3 - Findings of the Association for Computational Linguistics: EACL 2023 - Findings

BT - Findings of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

T2 - 17th Conference of the European Chapter of the Association for Computational Linguistics

Y2 - 2 May 2023 through 4 May 2023

ER -

Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking

Abstract

Publication series

Conference

Keywords

Access to Document

Fingerprint

Cite this