Effectiveness of Generative Artificial Intelligence for Scientific Content Analysis

TY - CHAP

T1 - Effectiveness of Generative Artificial Intelligence for Scientific Content Analysis

AU - Platt, Moritz

AU - Platt, Daniel

N1 - Conference code: 17

PY - 2023/11/13

Y1 - 2023/11/13

N2 - Generative artificial intelligence (GenAI) in general, and large language models (LLMs) in particular, are highly fashionable. As they have the ability to generate coherent output based on prompts in natural language, they are promoted as tools to free knowledge workers from tedious tasks such as content writing, customer support and routine computer code generation. Unsurprisingly, their application is also attractive to professionals in the research domain, where mundane and laborious tasks, such as literature screening, are commonplace. We evaluate Vertex AI ‘text-bison’, a foundational LLM model, in a real-world academic scenario by replicating parts of a popular systematic review in the information management domain. By comparing the results of a zero-shot LLM-based approach with those of the original study, we gather evidence on the suitability of state-of-the-art general-purpose LLMs for the analysis of scientific content. We show that the LLM-based approach delivers good scientific content analysis performance for a general classification problem (ACC = 0.9), acceptable performance for a domain-specific classification problem (ACC = 0.8) and borderline performance for a text comprehension problem (ACC ≈ 0.69). We conclude that some content analysis tasks with moderate accuracy requirements may be supported by current LLMs. As the technology will evolve rapidly in the foreseeable future, studies on large corpora, where some inaccuracies are tolerable, or workflows that prepare large data sets for human processing, may increasingly benefit from the capabilities of GenAI.

AB - Generative artificial intelligence (GenAI) in general, and large language models (LLMs) in particular, are highly fashionable. As they have the ability to generate coherent output based on prompts in natural language, they are promoted as tools to free knowledge workers from tedious tasks such as content writing, customer support and routine computer code generation. Unsurprisingly, their application is also attractive to professionals in the research domain, where mundane and laborious tasks, such as literature screening, are commonplace. We evaluate Vertex AI ‘text-bison’, a foundational LLM model, in a real-world academic scenario by replicating parts of a popular systematic review in the information management domain. By comparing the results of a zero-shot LLM-based approach with those of the original study, we gather evidence on the suitability of state-of-the-art general-purpose LLMs for the analysis of scientific content. We show that the LLM-based approach delivers good scientific content analysis performance for a general classification problem (ACC = 0.9), acceptable performance for a domain-specific classification problem (ACC = 0.8) and borderline performance for a text comprehension problem (ACC ≈ 0.69). We conclude that some content analysis tasks with moderate accuracy requirements may be supported by current LLMs. As the technology will evolve rapidly in the foreseeable future, studies on large corpora, where some inaccuracies are tolerable, or workflows that prepare large data sets for human processing, may increasingly benefit from the capabilities of GenAI.

KW - AI-Assisted Research

KW - Literature Screening

KW - Content Analysis

KW - Prompt Engineering

KW - Classification Performance

UR - http://www.scopus.com/inward/record.url?scp=85179525440&partnerID=8YFLogxK

U2 - 10.1109/AICT59525.2023.10313167

DO - 10.1109/AICT59525.2023.10313167

M3 - Conference paper

T3 - 17th IEEE International Conference on Application of Information and Communication Technologies, AICT 2023 - Proceedings

BT - 17th IEEE International Conference on Application of Information and Communication Technologies, AICT 2023 - Proceedings

PB - IEEE

T2 - 17th International Conference on Application of Information and Communication Technologies

Y2 - 18 October 2023 through 20 October 2023

ER -