An Analysis of Methodological and Quantitative Aspects in the Evaluation of Thematic Segmentation Algorithms

Maria Georgescul; Alexander Clark; Susan Armstrong

An Analysis of Methodological and Quantitative Aspects in the Evaluation of Thematic Segmentation Algorithms

Maria Georgescul, Alexander Clark, Susan Armstrong

Philosophy

Research output: Chapter in Book/Report/Conference proceeding › Conference paper

Abstract

We consider here the task of linear thematic segmentation of text documents, by using features based on word distributions in the text. For this task, a typical and often implicit assumption in previous studies is that a document has just one topic and therefore many algorithms have been tested and have shown encouraging results on artificial data sets, generated by putting together parts of different documents. We show that evaluation on synthetic data is potentially misleading and fails to give an accurate evaluation of the performance on real data. Moreover, we provide a critical review of existing evaluation metrics in the literature and we propose an improved evaluation metric.

Original language	Undefined/Unknown
Title of host publication	Proceedings of SIGDIAL
Publication status	Published - 2006

Cite this

@inbook{ae8ef9d4f5fe47799050fa7704797389,

title = "An Analysis of Methodological and Quantitative Aspects in the Evaluation of Thematic Segmentation Algorithms",

abstract = "We consider here the task of linear thematic segmentation of text documents, by using features based on word distributions in the text. For this task, a typical and often implicit assumption in previous studies is that a document has just one topic and therefore many algorithms have been tested and have shown encouraging results on artificial data sets, generated by putting together parts of different documents. We show that evaluation on synthetic data is potentially misleading and fails to give an accurate evaluation of the performance on real data. Moreover, we provide a critical review of existing evaluation metrics in the literature and we propose an improved evaluation metric.",

author = "Maria Georgescul and Alexander Clark and Susan Armstrong",

year = "2006",

language = "Undefined/Unknown",

booktitle = "Proceedings of SIGDIAL",

}

TY - CHAP

T1 - An Analysis of Methodological and Quantitative Aspects in the Evaluation of Thematic Segmentation Algorithms

AU - Georgescul, Maria

AU - Clark, Alexander

AU - Armstrong, Susan

PY - 2006

Y1 - 2006

N2 - We consider here the task of linear thematic segmentation of text documents, by using features based on word distributions in the text. For this task, a typical and often implicit assumption in previous studies is that a document has just one topic and therefore many algorithms have been tested and have shown encouraging results on artificial data sets, generated by putting together parts of different documents. We show that evaluation on synthetic data is potentially misleading and fails to give an accurate evaluation of the performance on real data. Moreover, we provide a critical review of existing evaluation metrics in the literature and we propose an improved evaluation metric.

AB - We consider here the task of linear thematic segmentation of text documents, by using features based on word distributions in the text. For this task, a typical and often implicit assumption in previous studies is that a document has just one topic and therefore many algorithms have been tested and have shown encouraging results on artificial data sets, generated by putting together parts of different documents. We show that evaluation on synthetic data is potentially misleading and fails to give an accurate evaluation of the performance on real data. Moreover, we provide a critical review of existing evaluation metrics in the literature and we propose an improved evaluation metric.

M3 - Conference paper

BT - Proceedings of SIGDIAL

ER -