Prediction and Control of Hierarchical Emotion Distribution for Text-to-Speech Synthesis
Sho Inoue1,2, Kun Zhou3, Shuai Wang2†, Haizhou Li1,2
1School of Data Science, 2Shenzhen Research Institute of Big Data
The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), Shenzhen, China
3Alibaba Group, Singapore
Abstract
This paper extends our previous work by investigating hierarchical emotion distribution (ED) for achieving multi-level quantitative control of emotion rendering in text-to-speech synthesis (TTS). We introduce a novel multi-step hierarchical ED prediction module that quantifies emotion variance at the utterance, word, and phoneme levels. By predicting emotion variance in a multi-step manner, our TTS framework leverages global emotional context to refine local emotional variations, thereby capturing the intrinsic hierarchical structure of speech emotion. Our approach is validated through its integration into a variance adaptor and an external module design compatible with various TTS systems. Both objective and subjective evaluations demonstrate that our framework significantly enhances emotional expressiveness and enables precise control of emotion rendering across multiple speech granularities.
Section 1: Emotion Expressiveness: Reproducibility via Ground-Truth Emotion Information (Emotion Transfer)
The first section presents a demo to evaluate our model's reproducibility. The Hierarchical EDs are obtained by the reference audio for those models with 'GT'. Instead, the Hierarchical EDs are predicted by textual cues for those with 'Pred'. 'External' and 'VA' denote different integration methods for ED modeling within TTS frameworks. In the 'External' approach, we employ a model-agnostic pipeline and incorporate a hierarchical ED embedding following text processing. In contrast, the 'VA' approach integrates hierarchical ED modeling directly within the variance adaptor of FastSpeech2:
- Reference Speech: The reference audio
- GT | External: 'External' Model with Ground-Truth Hierarchical ED
- GT | VA (Baseline): 'VA' Model with Ground-Truth Hierarchical ED
- GT | VA(Multi-Step): 'VA(Multi-Step)' Model with Ground-Truth Hierarchical ED
- Pred | External | One-Step (Baseline): 'External' Model with Predicted Hierarchical ED from One-Step Predictor
- Pred | External | Multi-Step: 'External' Model with Predicted Hierarchical ED from Multi-Step Predictor
- Pred | VA (Baseline): 'VA' Model with Predicted Hierarchical ED by One-Step Predictor
- Pred | VA(Multi-Step): 'VA(Multi-Step)' Model with Predicted Hierarchical ED by Multi-Step Predictor
| ID | 174_84280_000016_000000 | 652_130737_000021_000000 | 3536_8226_000027_000002 | 3000_15664_000014_000000 | 7976_105575_000014_000000 | 7976_110124_000063_000000 | 1462_170138_000007_000004 | 6313_66129_000038_000000 | 1462_170145_000031_000002 | 6313_76958_000048_000000 | 2277_149897_000040_000001 | 5694_64025_000019_000002 | 7976_110124_000026_000000 | 6313_76958_000089_000000 | 251_136532_000020_000001 | 1462_170142_000024_000002 | 1993_147964_000008_000000 | 5895_34615_000019_000006 | 2035_147961_000007_000006 | 251_118436_000019_000000 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
Text: I will not be content with that compromise of jealousies which is the established life of humanity to day. |
Text: Chateau Lafitte: Has beautiful color and delicate flavor. |
Text: He have been up all this month, sir." |
Text: Sheep Rock is about twenty miles from Sisson's, and is one of the principal winter pasture grounds of the wild sheep, from which it takes its name. |
Text: It was soon full day. |
Text: "Our little sister was wiser than we were," said the middle daughter. |
Text: She saves her hand, too. |
Text: "I reckon there are," smiled the guide. |
Text: You are not that kind of woman. |
Text: His warning came too late. |
Text: He was quite certain now that she knew he was married and was angered at his perfidy. |
Text: w h rushed forward and grabbed his horse by the bridle, telling him at the same time to surrender. |
Text: "I am thinking that our dear father would be anxious if he knew that we had broken our promise so easily," said the youngest sister. |
Text: "Pong, tell the young gentlemen what would become of you if you were to serve bad meals to this outfit of cowpunchers." |
Text: The Terran public wanted to hear about Martians, and if live Martians couldn't be found, a room full of dead ones was the next best thing. |
Text: I have thought about it until I am worn out." |
Text: I can see them now, exactly as they looked, working about the table in the lamplight: Jake with his heavy features, so rudely moulded that his face seemed, somehow, unfinished; Otto with his half ear and the savage scar that made his upper lip curl so ferociously under his twisted mustache. |
Text: It was automatic, and the more irresistible because it seemed petrified. |
Text: The wind shook the doors and windows impatiently, then swept on again, singing through the big spaces. |
Text: The first speaker favored him with a long, searching stare. |
| Reference Speech | ||||||||||||||||||||
| GT | External | ||||||||||||||||||||
| GT | VA | ||||||||||||||||||||
| GT | VA(Multi-Step) | ||||||||||||||||||||
| Pred | External | One-Step | ||||||||||||||||||||
| Pred | External | Multi-Step | ||||||||||||||||||||
| Pred | VA | ||||||||||||||||||||
| Pred | VA(Multi-Step) |
* please scroll horizontally to explore additional columns in the table.
Section 2: Utterance-level Emotion Intensity Control
In this section, we demonstrate the control of utterance-level emotion intensities. We set the emotion intensities of the emotion to the values in the first row while setting the other emotion intensities 0.0.
- Ground Truth: The Ground Truth sample reconstructed by Vocoder
- External: 'External' Model
- VA(Multi-Step): 'VA(Multi-Step)' Model
Utterance-level Control: Emotion: Angry
| ID | 0015_000011 | 0014_000354 | 0018_000009 | 0020_000018 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
| Ground Truth | ||||||||||||||||
| External | ||||||||||||||||
| VA(Multi-Step) |
* please scroll horizontally to explore additional columns in the table.
Utterance-level Control: Emotion: Happy
| ID | 0015_000011 | 0014_000354 | 0018_000009 | 0020_000018 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
| Ground Truth | ||||||||||||||||
| External | ||||||||||||||||
| VA(Multi-Step) |
* please scroll horizontally to explore additional columns in the table.
Utterance-level Control: Emotion: Sad
| ID | 0015_000011 | 0014_000354 | 0018_000009 | 0020_000018 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
| Ground Truth | ||||||||||||||||
| External | ||||||||||||||||
| VA(Multi-Step) |
* please scroll horizontally to explore additional columns in the table.
Utterance-level Control: Emotion: Surprise
| ID | 0015_000011 | 0014_000354 | 0018_000009 | 0020_000018 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
| Ground Truth | ||||||||||||||||
| External | ||||||||||||||||
| VA(Multi-Step) |
* please scroll horizontally to explore additional columns in the table.
Section 3: Word-level Emotion Intensity Control
In this section, we demonstrate the control of emotion intensities in a specific segment of speech, primarily focusing on three words with underscores. We set the word-level emotion intensities of these words, and the corresponding phoneme-level intensities, to the values in the first row, while maintaining the other emotion intensities constant. This showcases our model's ability to adjust emotion granularly within specified speech segments. We also present the speech samples focusing on the modified parts ('___OnlyModified___').
- Ground Truth: The Ground Truth sample reconstructed by Vocoder
- External: 'External' Model
- VA(Multi-Step): 'VA(Multi-Step)' Model
Word-level Control: Emotion: Angry
| ID | 0015_000011 | 0014_000354 | 0018_000009 | 0020_000018 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
| Ground Truth | ||||||||||||||||
| External | ||||||||||||||||
| ___OnlyModified___ | ||||||||||||||||
| VA(Multi-Step) | ||||||||||||||||
| ___OnlyModified___ |
* please scroll horizontally to explore additional columns in the table.
Word-level Control: Emotion: Happy
| ID | 0015_000011 | 0014_000354 | 0018_000009 | 0020_000018 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
| Ground Truth | ||||||||||||||||
| External | ||||||||||||||||
| ___OnlyModified___ | ||||||||||||||||
| VA(Multi-Step) | ||||||||||||||||
| ___OnlyModified___ |
* please scroll horizontally to explore additional columns in the table.
Word-level Control: Emotion: Sad
| ID | 0015_000011 | 0014_000354 | 0018_000009 | 0020_000018 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
| Ground Truth | ||||||||||||||||
| External | ||||||||||||||||
| ___OnlyModified___ | ||||||||||||||||
| VA(Multi-Step) | ||||||||||||||||
| ___OnlyModified___ |
* please scroll horizontally to explore additional columns in the table.
Word-level Control: Emotion: Surprise
| ID | 0015_000011 | 0014_000354 | 0018_000009 | 0020_000018 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: Chapter ten a warm welcome. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: They went up to the dark mass job had pointed out. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: On the twenty second of last march. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
Text: Then sadly it is much farther. |
| Ground Truth | ||||||||||||||||
| External | ||||||||||||||||
| ___OnlyModified___ | ||||||||||||||||
| VA(Multi-Step) | ||||||||||||||||
| ___OnlyModified___ |
* please scroll horizontally to explore additional columns in the table.