Grading Documentation with Machine Learning

TY - CHAP

T1 - Grading Documentation with Machine Learning

AU - Messer, Marcus

AU - Shi, Miaojing

AU - Brown, Neil

AU - Kölling, Michael

PY - 2024/7/8

Y1 - 2024/7/8

N2 - Professional developers, and especially students learning to program, often write poor documentation. While automated assessment for programming is becoming more common in educational settings, often using unit tests for code functionality and static analysis for code quality, documentation assessment is typically limited to detecting the presence and the correct formatting of a docstring based on a specified style guide. We aim to investigate how machine learning can be utilised to aid in automating the assessment of documentation quality. We classify a large set of publicly available human-annotated relevance scores between a natural language string and a code string, using traditional approaches, such as Logistic Regression and Random Forest, fine-tuned large language models, such as BERT, and Low-Rank Adaptation of large language models. Our most accurate model was a $k$-nearest-neighbours model with an accuracy of 58%.

AB - Professional developers, and especially students learning to program, often write poor documentation. While automated assessment for programming is becoming more common in educational settings, often using unit tests for code functionality and static analysis for code quality, documentation assessment is typically limited to detecting the presence and the correct formatting of a docstring based on a specified style guide. We aim to investigate how machine learning can be utilised to aid in automating the assessment of documentation quality. We classify a large set of publicly available human-annotated relevance scores between a natural language string and a code string, using traditional approaches, such as Logistic Regression and Random Forest, fine-tuned large language models, such as BERT, and Low-Rank Adaptation of large language models. Our most accurate model was a $k$-nearest-neighbours model with an accuracy of 58%.

KW - Automated Grading

KW - Assessment

KW - Computer Science Education

KW - Large Language Models

KW - Documentation

KW - Programming Education

M3 - Conference paper

SN - 978-3-031-64301-9

VL - 14829

T3 - Lecture Notes in Computer Science

BT - Artificial Intelligence in Education. AIED 2024

PB - Springer

ER -