Audio-visual prosody of whispered and semi-whispered speech

• Applicant: Dr. Marzena Zygis
• Subject Area: General and Comparative Linguistics, Experimental Linguistics, Typology, Non-European Languages
• Term: from 2019 to 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 426673330

Prosody plays a powerful role in speech. It helps listeners identify the boundaries of words and phrases, it highlights the most important pieces of information and it encodes emotions. While voiced speech prosody is an established research topic in phonetics and phonology, the prosody of whispered speech has received little attention so far. What is more, prosody in semi-whispered speech has not yet been investigated. The main goal of the present project is to provide more insight into the prosody of whispered and semi-whispered speech.The challenging nature of this goal lies in the fact that fundamental frequency, one of the main correlates of prosodic variation, is entirely absent in whispered speech. Thus, we propose an analysis of prosody through other acoustic parameters such as intensity, duration, and spectral properties of vowels and consonants. In our analysis we pay special attention to the still understudied interaction of prosody with consonants.In addition, we aim to extend the studied parameter space to the gestural dimension by taking advantage of the possibilities of motion tracking technology. The planned experiments include conditions that contrast communication in the acoustic-only with the joint acoustic-and-visual channel. We are interested in the extent to which acoustic cues may differ in these conditions, as well as in how and if visual prosody augments or supplants auditory cues in (semi-)whispered speech.Finally, the proposed investigation paves the way for a larger enterprise whose ultimate goal is to design algorithms which will convert semi-whispered and whispered speech to voiced speech. In the present project we concentrate on semi-whispered speech and aim to implement the results of the acoustic study in a semi-whisper-to-voiced conversion system, to be tested on speech that is not fully voiced (ILSE Corpus). Ultimately such a system will also help relieve the daily challenges of the ageing and clinical populations suffering from ailments such as throat cancers or neuro-degenerative diseases including Parkinson’s.

• DFG Programme: Research Grants