Enhancing Genetic Improvement Mutations Using Large Language Models

TY - JOUR

T1 - Enhancing Genetic Improvement Mutations Using Large Language Models

AU - Brownlee, Alexander

AU - Callan, James

AU - Even-Mendoza, Karine

AU - Geiger, Alina

AU - Hanna, Carol

AU - Petke, Justyna

AU - Sarro, Federica

AU - Sobania, Dominik

N1 - Conference code: 15

PY - 2023/12/4

Y1 - 2023/12/4

N2 - Large language models (LLMs) have been successfully applied to software engineering tasks, including program repair. However, their application in search-based techniques such as Genetic Improvement (GI) is still largely unexplored. In this paper, we evaluate the use of LLMs as mutation operators for GI to improve the search process. We expand the Gin Java GI toolkit to call OpenAI's API to generate edits for the JCodec tool. We randomly sample the space of edits using 5 different edit types. We find that the number of patches passing unit tests is up to 75% higher with LLM-based edits than with standard Insert edits. Further, we observe that the patches found with LLMs are generally less diverse compared to standard edits. We ran GI with local search to find runtime improvements. Although many improving patches are found by LLM-enhanced GI, the best improving patch was found by standard GI.

AB - Large language models (LLMs) have been successfully applied to software engineering tasks, including program repair. However, their application in search-based techniques such as Genetic Improvement (GI) is still largely unexplored. In this paper, we evaluate the use of LLMs as mutation operators for GI to improve the search process. We expand the Gin Java GI toolkit to call OpenAI's API to generate edits for the JCodec tool. We randomly sample the space of edits using 5 different edit types. We find that the number of patches passing unit tests is up to 75% higher with LLM-based edits than with standard Insert edits. Further, we observe that the patches found with LLMs are generally less diverse compared to standard edits. We ran GI with local search to find runtime improvements. Although many improving patches are found by LLM-enhanced GI, the best improving patch was found by standard GI.

KW - Genetic Improvement

KW - Large language models

UR - https://www.scopus.com/pages/publications/85180542665

U2 - 10.1007/978-3-031-48796-5_13

DO - 10.1007/978-3-031-48796-5_13

M3 - Conference paper

SN - 0302-9743

VL - 14415

SP - 153

EP - 159

JO - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

JF - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

T2 - 15TH INTERNATIONAL SYMPOSIUM ON SEARCH-BASED SOFTWARE ENGINEERING (SSBSE)

Y2 - 8 December 2023 through 8 December 2023

ER -