Enhancing Genetic Improvement Mutations Using Large Language Models

Alexander Brownlee*, James Callan, Karine Even-Mendoza, Alina Geiger, Carol Hanna, Justyna Petke, Federica Sarro, Dominik Sobania

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review

88 Downloads (Pure)

Abstract

Large language models (LLMs) have been successfully applied to software engineering tasks, including program repair. However, their application in search-based techniques such as Genetic Improvement (GI) is still largely unexplored. In this paper, we evaluate the use of LLMs as mutation operators for GI to improve the search process. We expand the Gin Java GI toolkit to call OpenAI's API to generate edits for the JCodec tool. We randomly sample the space of edits using 5 different edit types. We find that the number of patches passing unit tests is up to 75% higher with LLM-based edits than with standard Insert edits. Further, we observe that the patches found with LLMs are generally less diverse compared to standard edits. We ran GI with local search to find runtime improvements. Although many improving patches are found by LLM-enhanced GI, the best improving patch was found by standard GI.
Original languageEnglish
Title of host publication15th Symposium on Search Based Software Engineering (SSBSE)
Subtitle of host publicationLecture Notes in Computer Science
Place of PublicationUnited States
PublisherSpringer
Chapter15
Number of pages6
Volume15
Publication statusPublished - 8 Dec 2023

Fingerprint

Dive into the research topics of 'Enhancing Genetic Improvement Mutations Using Large Language Models'. Together they form a unique fingerprint.

Cite this