Distilling ChatGPT for Explainable Automated Student Answer Assessment

Jiazheng Li; Lin Gui; Yuxiang Zhou; David West; Cesare Aloisi; Yulan He

doi:https://doi.org/10.48550/arXiv.2305.12962

Distilling ChatGPT for Explainable Automated Student Answer Assessment

Jiazheng Li, Lin Gui, Yuxiang Zhou, David West, Cesare Aloisi, Yulan He

Informatics

Research output: Working paper/Preprint › Preprint

24 Downloads (Pure)

Abstract

Providing explainable and faithful feedback is crucial for automated student answer assessment. In this paper, we introduce a novel framework that explores using ChatGPT, a cutting-edge large language model, for the concurrent tasks of student answer scoring and rationale generation. We identify the appropriate instructions by prompting ChatGPT with different templates to collect the rationales, where inconsistent rationales are refined to align with marking standards. The refined ChatGPT outputs enable us to fine-tune a smaller language model that simultaneously assesses student answers and provides rationales. Extensive experiments on the benchmark dataset show that the proposed method improves the overall QWK score by 11% compared to ChatGPT. Furthermore, our thorough analysis and human evaluation demonstrate that the rationales generated by our proposed method are comparable to those of ChatGPT. Our approach provides a viable solution to achieve explainable automated assessment in education. Code available at https://github.com/lijiazheng99/aera.

Original language	English
Publisher	arXiv
DOIs	https://doi.org/10.48550/arXiv.2305.12962
Publication status	E-pub ahead of print - 22 May 2023

Keywords

cs.CL

Access to Document

https://doi.org/10.48550/arXiv.2305.12962Licence: CC BY-NC-SA

Distilling ChatGPT for Explainable_LI_Submitted22May2023_PREPRINT (CC BY NC SA)Submitted manuscript, 2.64 MBLicence: CC BY-NC-SA

Cite this

@techreport{33eaa00ce09b49749d1d9565405601cd,

title = "Distilling ChatGPT for Explainable Automated Student Answer Assessment",

abstract = "Providing explainable and faithful feedback is crucial for automated student answer assessment. In this paper, we introduce a novel framework that explores using ChatGPT, a cutting-edge large language model, for the concurrent tasks of student answer scoring and rationale generation. We identify the appropriate instructions by prompting ChatGPT with different templates to collect the rationales, where inconsistent rationales are refined to align with marking standards. The refined ChatGPT outputs enable us to fine-tune a smaller language model that simultaneously assesses student answers and provides rationales. Extensive experiments on the benchmark dataset show that the proposed method improves the overall QWK score by 11% compared to ChatGPT. Furthermore, our thorough analysis and human evaluation demonstrate that the rationales generated by our proposed method are comparable to those of ChatGPT. Our approach provides a viable solution to achieve explainable automated assessment in education. Code available at https://github.com/lijiazheng99/aera. ",

keywords = "cs.CL",

author = "Jiazheng Li and Lin Gui and Yuxiang Zhou and David West and Cesare Aloisi and Yulan He",

note = "Accepted EMNLP 2023 Findings",

year = "2023",

month = may,

day = "22",

doi = "https://doi.org/10.48550/arXiv.2305.12962",

language = "English",

publisher = "arXiv",

type = "WorkingPaper",

institution = "arXiv",

}

TY - UNPB

T1 - Distilling ChatGPT for Explainable Automated Student Answer Assessment

AU - Li, Jiazheng

AU - Gui, Lin

AU - Zhou, Yuxiang

AU - West, David

AU - Aloisi, Cesare

AU - He, Yulan

N1 - Accepted EMNLP 2023 Findings

PY - 2023/5/22

Y1 - 2023/5/22

N2 - Providing explainable and faithful feedback is crucial for automated student answer assessment. In this paper, we introduce a novel framework that explores using ChatGPT, a cutting-edge large language model, for the concurrent tasks of student answer scoring and rationale generation. We identify the appropriate instructions by prompting ChatGPT with different templates to collect the rationales, where inconsistent rationales are refined to align with marking standards. The refined ChatGPT outputs enable us to fine-tune a smaller language model that simultaneously assesses student answers and provides rationales. Extensive experiments on the benchmark dataset show that the proposed method improves the overall QWK score by 11% compared to ChatGPT. Furthermore, our thorough analysis and human evaluation demonstrate that the rationales generated by our proposed method are comparable to those of ChatGPT. Our approach provides a viable solution to achieve explainable automated assessment in education. Code available at https://github.com/lijiazheng99/aera.

AB - Providing explainable and faithful feedback is crucial for automated student answer assessment. In this paper, we introduce a novel framework that explores using ChatGPT, a cutting-edge large language model, for the concurrent tasks of student answer scoring and rationale generation. We identify the appropriate instructions by prompting ChatGPT with different templates to collect the rationales, where inconsistent rationales are refined to align with marking standards. The refined ChatGPT outputs enable us to fine-tune a smaller language model that simultaneously assesses student answers and provides rationales. Extensive experiments on the benchmark dataset show that the proposed method improves the overall QWK score by 11% compared to ChatGPT. Furthermore, our thorough analysis and human evaluation demonstrate that the rationales generated by our proposed method are comparable to those of ChatGPT. Our approach provides a viable solution to achieve explainable automated assessment in education. Code available at https://github.com/lijiazheng99/aera.

KW - cs.CL

U2 - https://doi.org/10.48550/arXiv.2305.12962

DO - https://doi.org/10.48550/arXiv.2305.12962

M3 - Preprint

BT - Distilling ChatGPT for Explainable Automated Student Answer Assessment

PB - arXiv

ER -

Distilling ChatGPT for Explainable Automated Student Answer Assessment

Abstract

Keywords

Access to Document

Fingerprint

Cite this