Blackbox, Five Years On: An Evaluation of a Large-scale Programming Data Collection Project

Neil Christopher Charles Brown; Amjad Altadmri; Sue Sentance; Michael Kölling

Blackbox, Five Years On: An Evaluation of a Large-scale Programming Data Collection Project

Neil Christopher Charles Brown, Amjad Altadmri, Sue Sentance, Michael Kölling

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

44 Citations (Scopus)

329 Downloads (Pure)

Abstract

The Blackbox project has been collecting programming activity data from users of BlueJ (a novice-targeted Java development environment) for nearly five years. The resulting dataset of more than two terabytes of data has been made available to interested researchers from the outset. In this paper, we assess the impact of the Blackbox project: we perform a mapping study to assess eighteen publications which have made use of the Blackbox data, and we report on the advantages and difficulties experienced by researchers working with this data, collected via a survey. We find that Blackbox has enabled pieces of research which otherwise would not have been possible, but there remain technical challenges in the analysis. Some of these -- but not all -- relate to the scale of the data. We provide suggestions for the future use of Blackbox, and reflections on the role of such data collection projects in programming research.

Original language	English
Title of host publication	ACM International Computing Education Research Conference
Publisher	ACM
Number of pages	9
Edition	2018
Publication status	Published - 13 Aug 2018

Access to Document

Blackbox, Five Years On_BROWN_Accepted8June2018_GREEN AAMAccepted author manuscript, 460 KBLicence: Unspecified

Cite this

@inbook{47404cdd367b4adfafeb96f915a761e7,

title = "Blackbox, Five Years On: An Evaluation of a Large-scale Programming Data Collection Project",

abstract = "The Blackbox project has been collecting programming activity data from users of BlueJ (a novice-targeted Java development environment) for nearly five years. The resulting dataset of more than two terabytes of data has been made available to interested researchers from the outset. In this paper, we assess the impact of the Blackbox project: we perform a mapping study to assess eighteen publications which have made use of the Blackbox data, and we report on the advantages and difficulties experienced by researchers working with this data, collected via a survey. We find that Blackbox has enabled pieces of research which otherwise would not have been possible, but there remain technical challenges in the analysis. Some of these -- but not all -- relate to the scale of the data. We provide suggestions for the future use of Blackbox, and reflections on the role of such data collection projects in programming research.",

author = "Brown, {Neil Christopher Charles} and Amjad Altadmri and Sue Sentance and Michael K{\"o}lling",

year = "2018",

month = aug,

day = "13",

language = "English",

booktitle = "ACM International Computing Education Research Conference",

publisher = "ACM",

edition = "2018",

}

TY - CHAP

T1 - Blackbox, Five Years On: An Evaluation of a Large-scale Programming Data Collection Project

AU - Brown, Neil Christopher Charles

AU - Altadmri, Amjad

AU - Sentance, Sue

AU - Kölling, Michael

PY - 2018/8/13

Y1 - 2018/8/13

N2 - The Blackbox project has been collecting programming activity data from users of BlueJ (a novice-targeted Java development environment) for nearly five years. The resulting dataset of more than two terabytes of data has been made available to interested researchers from the outset. In this paper, we assess the impact of the Blackbox project: we perform a mapping study to assess eighteen publications which have made use of the Blackbox data, and we report on the advantages and difficulties experienced by researchers working with this data, collected via a survey. We find that Blackbox has enabled pieces of research which otherwise would not have been possible, but there remain technical challenges in the analysis. Some of these -- but not all -- relate to the scale of the data. We provide suggestions for the future use of Blackbox, and reflections on the role of such data collection projects in programming research.

AB - The Blackbox project has been collecting programming activity data from users of BlueJ (a novice-targeted Java development environment) for nearly five years. The resulting dataset of more than two terabytes of data has been made available to interested researchers from the outset. In this paper, we assess the impact of the Blackbox project: we perform a mapping study to assess eighteen publications which have made use of the Blackbox data, and we report on the advantages and difficulties experienced by researchers working with this data, collected via a survey. We find that Blackbox has enabled pieces of research which otherwise would not have been possible, but there remain technical challenges in the analysis. Some of these -- but not all -- relate to the scale of the data. We provide suggestions for the future use of Blackbox, and reflections on the role of such data collection projects in programming research.

M3 - Conference paper

BT - ACM International Computing Education Research Conference

PB - ACM

ER -

Blackbox, Five Years On: An Evaluation of a Large-scale Programming Data Collection Project

Abstract

Access to Document

Fingerprint

Cite this