Hatevolution: What Static Benchmarks Don’t Tell Us

TY - CHAP

T1 - Hatevolution

T2 - What Static Benchmarks Don’t Tell Us

AU - Di Bonaventura, Chiara

AU - McGillivray, Barbara

AU - He, Yulan

AU - Merono Penuela, Albert

PY - 2025/7

Y1 - 2025/7

N2 - Language changes over time, including in the hate speech domain, which evolves quickly following social dynamics and cultural shifts. While NLP research has investigated the impact of language evolution on model training and has proposed several solutions for it, its impact on model benchmarking remains underexplored. Yet, hate speech benchmarks play a crucial role to ensure model safety. In this paper, we empirically evaluate the robustness of 20 language models across two evolving hate speech experiments, and we show the temporalmisalignment between static and time-sensitive evaluations. Our findings call for time-sensitive linguistic benchmarks in order to correctly and reliably evaluate language models in the hate speech domain.

AB - Language changes over time, including in the hate speech domain, which evolves quickly following social dynamics and cultural shifts. While NLP research has investigated the impact of language evolution on model training and has proposed several solutions for it, its impact on model benchmarking remains underexplored. Yet, hate speech benchmarks play a crucial role to ensure model safety. In this paper, we empirically evaluate the robustness of 20 language models across two evolving hate speech experiments, and we show the temporalmisalignment between static and time-sensitive evaluations. Our findings call for time-sensitive linguistic benchmarks in order to correctly and reliably evaluate language models in the hate speech domain.

KW - Large Language Models (LLMs)

KW - Hate Speech Detection

KW - Benchmark

KW - Evaluation

KW - temporal

KW - language change

M3 - Conference paper

BT - Proceedings of the 2025 Annual Meeting of the Association for Computational Linguistics (ACL), Findings

ER -