Raw acoustic-articulatory multimodal dysarthric speech recognition

Zhengjun Yue; Erfan Loweimi; Zoran Cvetkovic; Jon Barker; Heidi Christensen

doi:10.1016/j.csl.2025.101839

Raw acoustic-articulatory multimodal dysarthric speech recognition

Zhengjun Yue^*, Erfan Loweimi, Zoran Cvetkovic, Jon Barker, Heidi Christensen

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

2 Downloads (Pure)

Abstract

Automatic speech recognition (ASR) for dysarthric speech is challenging. The acoustic characteristics of dysarthric speech are highly variable and there are often fewer distinguishing cues between phonetic tokens. Multimodal ASR utilises the data from other modalities to facilitate the task when a single acoustic modality proves insufficient. Articulatory information, which encapsulates knowledge about the speech production process, may constitute such a complementary modality. Although multimodal acoustic-articulatory ASR has received increasing attention recently, incorporating real articula- tory data is under-explored for dysarthric speech recognition. This paper investigates the effectiveness of multimodal acoustic modelling using real dysarthric speech articulatory information in combination with acoustic fea- tures, especially raw signal representations which are more informative than classic features, leading to learning representations tailored to dysarthric ASR. In particular, various raw acoustic-articulatory multimodal dysarthric speech recognition systems are developed and compared with similar systems with hand-crafted features. Furthermore, the difference between dysarthric and typical speech in terms of articulatory information is systematically anal- ysed by using a statistical space distribution indicator called Maximum Ar- ticulator Motion Range (MAMR). Additionally, we used mutual information analysis to investigate the robustness and phonetic information content of the articulatory features, offering insights that support feature selection and the ASR results. Experimental results on the widely used TORGO dysarthric speech dataset show that combining the articulatory and raw acoustic fea- tures at the empirically found optimal fusion level achieves a notable perfor- mance gain, leading to up to 7.6% and 12.8% relative word error rate (WER) reduction for dysarthric and typical speech, respectively.

Original language	English
Article number	101839
Number of pages	35
Journal	COMPUTER SPEECH AND LANGUAGE
Volume	95
Early online date	10 Jun 2025
DOIs	https://doi.org/10.1016/j.csl.2025.101839
Publication status	E-pub ahead of print - 10 Jun 2025

Access to Document

10.1016/j.csl.2025.101839Licence: CC BY

Raw acoustic-articulatory_YUE_Publishedonline10June2025_GOLD VoR (CC-BY)Final published version, 4.02 MBLicence: CC BY

Cite this

@article{f3eedf8b33e64fb18e70dd3ea51b909f,

title = "Raw acoustic-articulatory multimodal dysarthric speech recognition",

abstract = "Automatic speech recognition (ASR) for dysarthric speech is challenging. The acoustic characteristics of dysarthric speech are highly variable and there are often fewer distinguishing cues between phonetic tokens. Multimodal ASR utilises the data from other modalities to facilitate the task when a single acoustic modality proves insufficient. Articulatory information, which encapsulates knowledge about the speech production process, may constitute such a complementary modality. Although multimodal acoustic-articulatory ASR has received increasing attention recently, incorporating real articula- tory data is under-explored for dysarthric speech recognition. This paper investigates the effectiveness of multimodal acoustic modelling using real dysarthric speech articulatory information in combination with acoustic fea- tures, especially raw signal representations which are more informative than classic features, leading to learning representations tailored to dysarthric ASR. In particular, various raw acoustic-articulatory multimodal dysarthric speech recognition systems are developed and compared with similar systems with hand-crafted features. Furthermore, the difference between dysarthric and typical speech in terms of articulatory information is systematically anal- ysed by using a statistical space distribution indicator called Maximum Ar- ticulator Motion Range (MAMR). Additionally, we used mutual information analysis to investigate the robustness and phonetic information content of the articulatory features, offering insights that support feature selection and the ASR results. Experimental results on the widely used TORGO dysarthric speech dataset show that combining the articulatory and raw acoustic fea- tures at the empirically found optimal fusion level achieves a notable perfor- mance gain, leading to up to 7.6% and 12.8% relative word error rate (WER) reduction for dysarthric and typical speech, respectively.",

author = "Zhengjun Yue and Erfan Loweimi and Zoran Cvetkovic and Jon Barker and Heidi Christensen",

note = "Publisher Copyright: {\textcopyright} 2025 The Authors",

year = "2025",

month = jun,

day = "10",

doi = "10.1016/j.csl.2025.101839",

language = "English",

volume = "95",

journal = "COMPUTER SPEECH AND LANGUAGE",

issn = "0885-2308",

publisher = "ACADEMIC PRESS INC",

}

TY - JOUR

T1 - Raw acoustic-articulatory multimodal dysarthric speech recognition

AU - Yue, Zhengjun

AU - Loweimi, Erfan

AU - Cvetkovic, Zoran

AU - Barker, Jon

AU - Christensen, Heidi

PY - 2025/6/10

Y1 - 2025/6/10

N2 - Automatic speech recognition (ASR) for dysarthric speech is challenging. The acoustic characteristics of dysarthric speech are highly variable and there are often fewer distinguishing cues between phonetic tokens. Multimodal ASR utilises the data from other modalities to facilitate the task when a single acoustic modality proves insufficient. Articulatory information, which encapsulates knowledge about the speech production process, may constitute such a complementary modality. Although multimodal acoustic-articulatory ASR has received increasing attention recently, incorporating real articula- tory data is under-explored for dysarthric speech recognition. This paper investigates the effectiveness of multimodal acoustic modelling using real dysarthric speech articulatory information in combination with acoustic fea- tures, especially raw signal representations which are more informative than classic features, leading to learning representations tailored to dysarthric ASR. In particular, various raw acoustic-articulatory multimodal dysarthric speech recognition systems are developed and compared with similar systems with hand-crafted features. Furthermore, the difference between dysarthric and typical speech in terms of articulatory information is systematically anal- ysed by using a statistical space distribution indicator called Maximum Ar- ticulator Motion Range (MAMR). Additionally, we used mutual information analysis to investigate the robustness and phonetic information content of the articulatory features, offering insights that support feature selection and the ASR results. Experimental results on the widely used TORGO dysarthric speech dataset show that combining the articulatory and raw acoustic fea- tures at the empirically found optimal fusion level achieves a notable perfor- mance gain, leading to up to 7.6% and 12.8% relative word error rate (WER) reduction for dysarthric and typical speech, respectively.

AB - Automatic speech recognition (ASR) for dysarthric speech is challenging. The acoustic characteristics of dysarthric speech are highly variable and there are often fewer distinguishing cues between phonetic tokens. Multimodal ASR utilises the data from other modalities to facilitate the task when a single acoustic modality proves insufficient. Articulatory information, which encapsulates knowledge about the speech production process, may constitute such a complementary modality. Although multimodal acoustic-articulatory ASR has received increasing attention recently, incorporating real articula- tory data is under-explored for dysarthric speech recognition. This paper investigates the effectiveness of multimodal acoustic modelling using real dysarthric speech articulatory information in combination with acoustic fea- tures, especially raw signal representations which are more informative than classic features, leading to learning representations tailored to dysarthric ASR. In particular, various raw acoustic-articulatory multimodal dysarthric speech recognition systems are developed and compared with similar systems with hand-crafted features. Furthermore, the difference between dysarthric and typical speech in terms of articulatory information is systematically anal- ysed by using a statistical space distribution indicator called Maximum Ar- ticulator Motion Range (MAMR). Additionally, we used mutual information analysis to investigate the robustness and phonetic information content of the articulatory features, offering insights that support feature selection and the ASR results. Experimental results on the widely used TORGO dysarthric speech dataset show that combining the articulatory and raw acoustic fea- tures at the empirically found optimal fusion level achieves a notable perfor- mance gain, leading to up to 7.6% and 12.8% relative word error rate (WER) reduction for dysarthric and typical speech, respectively.

UR - http://www.scopus.com/inward/record.url?scp=105009087227&partnerID=8YFLogxK

U2 - 10.1016/j.csl.2025.101839

DO - 10.1016/j.csl.2025.101839

M3 - Article

SN - 0885-2308

VL - 95

JO - COMPUTER SPEECH AND LANGUAGE

JF - COMPUTER SPEECH AND LANGUAGE

M1 - 101839

ER -

Raw acoustic-articulatory multimodal dysarthric speech recognition

Abstract

Access to Document

Other files and links

Fingerprint

Cite this