Abstract
Research into automated grading has increased as Computer Science courses grow.
Dynamic and static approaches are typically used to implement these graders, the most common implementation being unit testing to grade correctness.
This paper expands upon an ongoing systematic literature review to provide an in-depth analysis of how machine learning (ML) has been used to grade and give feedback on programming assignments.
We conducted a backward snowball search using the ML papers from an ongoing systematic review and selected 27 papers that met our inclusion criteria.
After selecting our papers, we analysed the skills graded, the preprocessing steps, the ML implementation, and the models’ evaluations.
We find that most the models are implemented using neural network-based approaches, with most implementing some form of recurrent neural network (RNN), including Long Short-Term Memory, and encoder/decoder with attention mechanisms.
Some graders implement traditional ML approaches, typically focused on clustering.
Most ML-based automated grading, not many use ML to evaluate maintainability, readability, and documentation, but focus on grading correctness, a problem that dynamic and static analysis techniques, such as unit testing, rule-based program repair, and comparison to models or approved solutions, have mostly resolved.
However, some ML-based tools, including those for assessing graphical output, have evaluated the correctness of assignments that conventional implementations cannot.
Dynamic and static approaches are typically used to implement these graders, the most common implementation being unit testing to grade correctness.
This paper expands upon an ongoing systematic literature review to provide an in-depth analysis of how machine learning (ML) has been used to grade and give feedback on programming assignments.
We conducted a backward snowball search using the ML papers from an ongoing systematic review and selected 27 papers that met our inclusion criteria.
After selecting our papers, we analysed the skills graded, the preprocessing steps, the ML implementation, and the models’ evaluations.
We find that most the models are implemented using neural network-based approaches, with most implementing some form of recurrent neural network (RNN), including Long Short-Term Memory, and encoder/decoder with attention mechanisms.
Some graders implement traditional ML approaches, typically focused on clustering.
Most ML-based automated grading, not many use ML to evaluate maintainability, readability, and documentation, but focus on grading correctness, a problem that dynamic and static analysis techniques, such as unit testing, rule-based program repair, and comparison to models or approved solutions, have mostly resolved.
However, some ML-based tools, including those for assessing graphical output, have evaluated the correctness of assignments that conventional implementations cannot.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the 28th ACM Conference on Innovation and Technology in Computer Science Education |
| Publisher | Association for Computing Machinery, Inc |
| Volume | 1 |
| Publication status | Accepted/In press - 6 Mar 2023 |
Keywords
- Machine Learning (ML)
- Automated Grading
- Computer science education
- Meta-analyses