Overview
Research investigated the utility of contextualized embeddings (CEs) in predicting spoken word duration and pitch contours for Mandarin monosyllabic words. The study focused on 7470 tokens of Mandarin monosyllabic CV words extracted from a corpus of spontaneous speech. Previous work indicated that time-normalized f0 contours of Mandarin words in conversational speech are partially predictable from their CEs. This investigation extends that line of inquiry to include spoken word duration and the back-transformation of f0 contours.
Research Context
Prior studies established that contextualized embeddings offer a basis for predicting time-normalized f0 contours in Mandarin conversational speech. This provided a foundation for exploring whether CEs could similarly predict other acoustic properties, specifically word duration. The objective was to ascertain the predictive capacity of CEs for duration and then to evaluate if this predictive power, coupled with duration prediction, could facilitate the back-transformation of f0 contours from a normalized time scale to a real-time (milliseconds) scale.
Approach
The study utilized a dataset comprising 7470 tokens of Mandarin monosyllabic CV words. These tokens were sourced from a Mandarin corpus characterized by spontaneous speech. Contextualized embeddings were employed as the predictive features. Two primary predictions were evaluated: spoken word duration and the back-transformation of f0 contours. For duration prediction, both type-wise and token-wise permutation baselines were used to assess the significance of the predictions. Subsequently, the predicted durations were used to transform f0 contours, initially normalized to a [0,1] time scale, back to a millisecond time scale. The performance of these resulting predicted f0 contours was then compared against empirical contours and a permutation baseline.
Findings
- Contextualized embeddings are predictive for spoken word duration in Mandarin monosyllabic CV words.
- This predictive capability for duration was observed to be above chance level for both type-level and individual token-level predictions.
- The results for duration prediction were validated against type-wise and token-wise permutation baselines.
- The predicted durations were sufficiently precise to enable the back-transformation of predicted f0 contours, initially normalized to a [0,1] time scale, to a millisecond time scale.
- The resulting back-transformed predicted f0 contours on the millisecond time scale approximate empirical contours.
- These back-transformed predicted contours also demonstrated superior performance compared to a permutation baseline.
Why This Matters
The findings indicate that contextualized embeddings can predict key acoustic features of spoken Mandarin words, specifically duration and pitch contours. The ability to accurately predict these features, including the back-transformation of normalized f0 contours to actual time scales, suggests functional applications in speech synthesis or analysis systems that rely on precise temporal and prosodic information.