Draw Me Like My Triples: Leveraging Generative AI for Wikidata Image Completion

Raia Abu Ahmad; Martin Critelli; Sefika Efeoglu; Eleonora Mancini; Célian Ringwald; Xinyue Zhang; Albert Meroño-Peñuela

Draw Me Like My Triples: Leveraging Generative AI for Wikidata Image Completion

Raia Abu Ahmad, Martin Critelli, Sefika Efeoglu, Eleonora Mancini, Célian Ringwald, Xinyue Zhang, Albert Meroño-Peñuela

Research output: Contribution to journal › Article › peer-review

Abstract

Humans are critical for the creation and maintenance of high-quality Knowledge Graphs (KGs). However, creating and maintaining large KGs only with humans does not scale, especially for contributions based on multimedia (e.g. images) that are hard to find and reuse on the Web and expensive to generate by humans from scratch. Therefore, we leverage generative AI for the task of creating images for Wikidata items that do not have them. Our approach uses knowledge contained in Wikidata triples of items describing fictional characters and uses the fine-tuned T5 model based on the WDV dataset to generate natural text descriptions of items about fictional characters with missing images. We use those natural text descriptions as prompts for a transformer-based text-to-image model, Stable Diffusion v2.1, to generate plausible candidate images for Wikidata image completion. We design and implement quantitative and qualitative approaches to evaluate the plausibility of our methods, which include conducting a survey to assess the quality of the generated images.

Original language	English
Journal	CEUR Workshop Proceedings
Volume	3640
Publication status	Published - 1 Jan 2023

Keywords

Automated Prompt Generation
Generative AI
Image Generation

Cite this

@article{d60e63cecd72457bb478780f04b17c57,

title = "Draw Me Like My Triples: Leveraging Generative AI for Wikidata Image Completion",

abstract = "Humans are critical for the creation and maintenance of high-quality Knowledge Graphs (KGs). However, creating and maintaining large KGs only with humans does not scale, especially for contributions based on multimedia (e.g. images) that are hard to find and reuse on the Web and expensive to generate by humans from scratch. Therefore, we leverage generative AI for the task of creating images for Wikidata items that do not have them. Our approach uses knowledge contained in Wikidata triples of items describing fictional characters and uses the fine-tuned T5 model based on the WDV dataset to generate natural text descriptions of items about fictional characters with missing images. We use those natural text descriptions as prompts for a transformer-based text-to-image model, Stable Diffusion v2.1, to generate plausible candidate images for Wikidata image completion. We design and implement quantitative and qualitative approaches to evaluate the plausibility of our methods, which include conducting a survey to assess the quality of the generated images.",

keywords = "Automated Prompt Generation, Generative AI, Image Generation",

author = "{Abu Ahmad}, Raia and Martin Critelli and Sefika Efeoglu and Eleonora Mancini and C{\'e}lian Ringwald and Xinyue Zhang and Albert Mero{\~n}o-Pe{\~n}uela",

year = "2023",

month = jan,

day = "1",

language = "English",

volume = "3640",

journal = "CEUR Workshop Proceedings",

issn = "1613-0073",

publisher = "CEUR-WS",

}

TY - JOUR

T1 - Draw Me Like My Triples: Leveraging Generative AI for Wikidata Image Completion

AU - Abu Ahmad, Raia

AU - Critelli, Martin

AU - Efeoglu, Sefika

AU - Mancini, Eleonora

AU - Ringwald, Célian

AU - Zhang, Xinyue

AU - Meroño-Peñuela, Albert

PY - 2023/1/1

Y1 - 2023/1/1

N2 - Humans are critical for the creation and maintenance of high-quality Knowledge Graphs (KGs). However, creating and maintaining large KGs only with humans does not scale, especially for contributions based on multimedia (e.g. images) that are hard to find and reuse on the Web and expensive to generate by humans from scratch. Therefore, we leverage generative AI for the task of creating images for Wikidata items that do not have them. Our approach uses knowledge contained in Wikidata triples of items describing fictional characters and uses the fine-tuned T5 model based on the WDV dataset to generate natural text descriptions of items about fictional characters with missing images. We use those natural text descriptions as prompts for a transformer-based text-to-image model, Stable Diffusion v2.1, to generate plausible candidate images for Wikidata image completion. We design and implement quantitative and qualitative approaches to evaluate the plausibility of our methods, which include conducting a survey to assess the quality of the generated images.

AB - Humans are critical for the creation and maintenance of high-quality Knowledge Graphs (KGs). However, creating and maintaining large KGs only with humans does not scale, especially for contributions based on multimedia (e.g. images) that are hard to find and reuse on the Web and expensive to generate by humans from scratch. Therefore, we leverage generative AI for the task of creating images for Wikidata items that do not have them. Our approach uses knowledge contained in Wikidata triples of items describing fictional characters and uses the fine-tuned T5 model based on the WDV dataset to generate natural text descriptions of items about fictional characters with missing images. We use those natural text descriptions as prompts for a transformer-based text-to-image model, Stable Diffusion v2.1, to generate plausible candidate images for Wikidata image completion. We design and implement quantitative and qualitative approaches to evaluate the plausibility of our methods, which include conducting a survey to assess the quality of the generated images.

KW - Automated Prompt Generation

KW - Generative AI

KW - Image Generation

UR - http://www.scopus.com/inward/record.url?scp=85185884370&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:85185884370

SN - 1613-0073

VL - 3640

JO - CEUR Workshop Proceedings

JF - CEUR Workshop Proceedings

ER -

Draw Me Like My Triples: Leveraging Generative AI for Wikidata Image Completion

Abstract

Keywords

Other files and links

Fingerprint

Cite this