Effectiveness of Generative Artificial Intelligence for Scientific Content Analysis

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review

1 Citation (Scopus)
208 Downloads (Pure)

Abstract

Generative artificial intelligence (GenAI) in general, and large language models (LLMs) in particular, are highly fashionable. As they have the ability to generate coherent output based on prompts in natural language, they are promoted as tools to free knowledge workers from tedious tasks such as content writing, customer support and routine computer code generation. Unsurprisingly, their application is also attractive to professionals in the research domain, where mundane and laborious tasks, such as literature screening, are commonplace. We evaluate Vertex AI ‘text-bison’, a foundational LLM model, in a real-world academic scenario by replicating parts of a popular systematic review in the information management domain. By comparing the results of a zero-shot LLM-based approach with those of the original study, we gather evidence on the suitability of state-of-the-art general-purpose LLMs for the analysis of scientific content. We show that the LLM-based approach delivers good scientific content analysis performance for a general classification problem (ACC = 0.9), acceptable performance for a domain-specific classification problem (ACC = 0.8) and borderline performance for a text comprehension problem (ACC ≈ 0.69). We conclude that some content analysis tasks with moderate accuracy requirements may be supported by current LLMs. As the technology will evolve rapidly in the foreseeable future, studies on large corpora, where some inaccuracies are tolerable, or workflows that prepare large data sets for human processing, may increasingly benefit from the capabilities of GenAI.
Original languageEnglish
Title of host publication17th IEEE International Conference on Application of Information and Communication Technologies, AICT 2023 - Proceedings
PublisherIEEE
ISBN (Electronic)9798350303568
DOIs
Publication statusPublished - 13 Nov 2023
Event17th International Conference on Application of Information and Communication Technologies - ADA University, Baku, Azerbaijan
Duration: 18 Oct 202320 Oct 2023
Conference number: 17
https://www.aict.info/?csc=2023

Publication series

Name17th IEEE International Conference on Application of Information and Communication Technologies, AICT 2023 - Proceedings

Conference

Conference17th International Conference on Application of Information and Communication Technologies
Abbreviated titleAICT
Country/TerritoryAzerbaijan
CityBaku
Period18/10/202320/10/2023
Internet address

Keywords

  • AI-Assisted Research
  • Literature Screening
  • Content Analysis
  • Prompt Engineering
  • Classification Performance

Fingerprint

Dive into the research topics of 'Effectiveness of Generative Artificial Intelligence for Scientific Content Analysis'. Together they form a unique fingerprint.

Cite this