Enhancing Genetic Improvement Mutations Using Large Language Models

Alexander Brownlee; James Callan; Karine Even-Mendoza; Alina Geiger; Carol Hanna; Justyna Petke; Federica Sarro; Dominik Sobania

Enhancing Genetic Improvement Mutations Using Large Language Models

Alexander Brownlee^*, James Callan, Karine Even-Mendoza, Alina Geiger, Carol Hanna, Justyna Petke, Federica Sarro, Dominik Sobania

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

225 Downloads (Pure)

Abstract

Large language models (LLMs) have been successfully applied to software engineering tasks, including program repair. However, their application in search-based techniques such as Genetic Improvement (GI) is still largely unexplored. In this paper, we evaluate the use of LLMs as mutation operators for GI to improve the search process. We expand the Gin Java GI toolkit to call OpenAI's API to generate edits for the JCodec tool. We randomly sample the space of edits using 5 different edit types. We find that the number of patches passing unit tests is up to 75% higher with LLM-based edits than with standard Insert edits. Further, we observe that the patches found with LLMs are generally less diverse compared to standard edits. We ran GI with local search to find runtime improvements. Although many improving patches are found by LLM-enhanced GI, the best improving patch was found by standard GI.

Original language	English
Title of host publication	15th Symposium on Search Based Software Engineering (SSBSE)
Subtitle of host publication	Lecture Notes in Computer Science
Place of Publication	United States
Publisher	Springer
Chapter	15
Number of pages	6
Volume	15
Publication status	Published - 8 Dec 2023

Access to Document

ssbse23challenge-final_GinAccepted author manuscript, 203 KBLicence: Other

Artifact of Enhancing Genetic Improvement Mutations Using Large Language Models
Brownlee, A., Callan, J., Even Mendoza, K., Geiger, A., Hanna, C., Petke, J., Sarro, F. & Sobania, D., Zenodo, 1 Sept 2023
DOI: 10.5281/zenodo.8304433
Dataset

Cite this

@inbook{69a5cd6b2c004dbebd04bad4307682e8,

title = "Enhancing Genetic Improvement Mutations Using Large Language Models",

abstract = "Large language models (LLMs) have been successfully applied to software engineering tasks, including program repair. However, their application in search-based techniques such as Genetic Improvement (GI) is still largely unexplored. In this paper, we evaluate the use of LLMs as mutation operators for GI to improve the search process. We expand the Gin Java GI toolkit to call OpenAI's API to generate edits for the JCodec tool. We randomly sample the space of edits using 5 different edit types. We find that the number of patches passing unit tests is up to 75% higher with LLM-based edits than with standard Insert edits. Further, we observe that the patches found with LLMs are generally less diverse compared to standard edits. We ran GI with local search to find runtime improvements. Although many improving patches are found by LLM-enhanced GI, the best improving patch was found by standard GI.",

author = "Alexander Brownlee and James Callan and Karine Even-Mendoza and Alina Geiger and Carol Hanna and Justyna Petke and Federica Sarro and Dominik Sobania",

year = "2023",

month = dec,

day = "8",

language = "English",

volume = "15",

booktitle = "15th Symposium on Search Based Software Engineering (SSBSE)",

publisher = "Springer",

}

TY - CHAP

T1 - Enhancing Genetic Improvement Mutations Using Large Language Models

AU - Brownlee, Alexander

AU - Callan, James

AU - Even-Mendoza, Karine

AU - Geiger, Alina

AU - Hanna, Carol

AU - Petke, Justyna

AU - Sarro, Federica

AU - Sobania, Dominik

PY - 2023/12/8

Y1 - 2023/12/8

N2 - Large language models (LLMs) have been successfully applied to software engineering tasks, including program repair. However, their application in search-based techniques such as Genetic Improvement (GI) is still largely unexplored. In this paper, we evaluate the use of LLMs as mutation operators for GI to improve the search process. We expand the Gin Java GI toolkit to call OpenAI's API to generate edits for the JCodec tool. We randomly sample the space of edits using 5 different edit types. We find that the number of patches passing unit tests is up to 75% higher with LLM-based edits than with standard Insert edits. Further, we observe that the patches found with LLMs are generally less diverse compared to standard edits. We ran GI with local search to find runtime improvements. Although many improving patches are found by LLM-enhanced GI, the best improving patch was found by standard GI.

AB - Large language models (LLMs) have been successfully applied to software engineering tasks, including program repair. However, their application in search-based techniques such as Genetic Improvement (GI) is still largely unexplored. In this paper, we evaluate the use of LLMs as mutation operators for GI to improve the search process. We expand the Gin Java GI toolkit to call OpenAI's API to generate edits for the JCodec tool. We randomly sample the space of edits using 5 different edit types. We find that the number of patches passing unit tests is up to 75% higher with LLM-based edits than with standard Insert edits. Further, we observe that the patches found with LLMs are generally less diverse compared to standard edits. We ran GI with local search to find runtime improvements. Although many improving patches are found by LLM-enhanced GI, the best improving patch was found by standard GI.

M3 - Conference paper

VL - 15

BT - 15th Symposium on Search Based Software Engineering (SSBSE)

PB - Springer

CY - United States

ER -

Enhancing Genetic Improvement Mutations Using Large Language Models

Abstract

Access to Document

Fingerprint

Datasets

Artifact of Enhancing Genetic Improvement Mutations Using Large Language Models

Cite this