Disentangling the Prosody and Semantic Information with Pre-trained Model for In-Context Learning based Zero-Shot Voice Conversion Article Swipe
Zhengyang Chen
,
Shuai Wang
,
Mingyang Zhang
,
Xuechen Liu
,
Junichi Yamagishi
,
Yanmin Qian
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2409.05004
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2409.05004
Voice conversion (VC) aims to modify the speaker's timbre while retaining speech content. Previous approaches have tokenized the outputs from self-supervised into semantic tokens, facilitating disentanglement of speech content information. Recently, in-context learning (ICL) has emerged in text-to-speech (TTS) systems for effectively modeling specific characteristics such as timbre through context conditioning. This paper proposes an ICL capability enhanced VC system (ICL-VC) employing a mask and reconstruction training strategy based on flow-matching generative models. Augmented with semantic tokens, our experiments on the LibriTTS dataset demonstrate that ICL-VC improves speaker similarity. Additionally, we find that k-means is a versatile tokenization method applicable to various pre-trained models. However, the ICL-VC system faces challenges in preserving the prosody of the source speech. To mitigate this issue, we propose incorporating prosody embeddings extracted from a pre-trained emotion recognition model into our system. Integration of prosody embeddings notably enhances the system's capability to preserve source speech prosody, as validated on the Emotional Speech Database.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2409.05004
- https://arxiv.org/pdf/2409.05004
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4403590122
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4403590122Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2409.05004Digital Object Identifier
- Title
-
Disentangling the Prosody and Semantic Information with Pre-trained Model for In-Context Learning based Zero-Shot Voice ConversionWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-09-08Full publication date if available
- Authors
-
Zhengyang Chen, Shuai Wang, Mingyang Zhang, Xuechen Liu, Junichi Yamagishi, Yanmin QianList of authors in order
- Landing page
-
https://arxiv.org/abs/2409.05004Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2409.05004Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2409.05004Direct OA link when available
- Concepts
-
Prosody, Zero (linguistics), Context (archaeology), Natural language processing, Computer science, Shot (pellet), Speech recognition, Artificial intelligence, Linguistics, Psychology, History, Organic chemistry, Archaeology, Philosophy, ChemistryTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4403590122 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2409.05004 |
| ids.doi | https://doi.org/10.48550/arxiv.2409.05004 |
| ids.openalex | https://openalex.org/W4403590122 |
| fwci | |
| type | preprint |
| title | Disentangling the Prosody and Semantic Information with Pre-trained Model for In-Context Learning based Zero-Shot Voice Conversion |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10201 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9882000088691711 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Speech Recognition and Synthesis |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C542774811 |
| concepts[0].level | 2 |
| concepts[0].score | 0.810196042060852 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q10880526 |
| concepts[0].display_name | Prosody |
| concepts[1].id | https://openalex.org/C2780813799 |
| concepts[1].level | 2 |
| concepts[1].score | 0.7463643550872803 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q3274237 |
| concepts[1].display_name | Zero (linguistics) |
| concepts[2].id | https://openalex.org/C2779343474 |
| concepts[2].level | 2 |
| concepts[2].score | 0.6234064102172852 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q3109175 |
| concepts[2].display_name | Context (archaeology) |
| concepts[3].id | https://openalex.org/C204321447 |
| concepts[3].level | 1 |
| concepts[3].score | 0.5354601740837097 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[3].display_name | Natural language processing |
| concepts[4].id | https://openalex.org/C41008148 |
| concepts[4].level | 0 |
| concepts[4].score | 0.5179173350334167 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[4].display_name | Computer science |
| concepts[5].id | https://openalex.org/C2778344882 |
| concepts[5].level | 2 |
| concepts[5].score | 0.4678158462047577 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q278938 |
| concepts[5].display_name | Shot (pellet) |
| concepts[6].id | https://openalex.org/C28490314 |
| concepts[6].level | 1 |
| concepts[6].score | 0.45178329944610596 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q189436 |
| concepts[6].display_name | Speech recognition |
| concepts[7].id | https://openalex.org/C154945302 |
| concepts[7].level | 1 |
| concepts[7].score | 0.42458418011665344 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[7].display_name | Artificial intelligence |
| concepts[8].id | https://openalex.org/C41895202 |
| concepts[8].level | 1 |
| concepts[8].score | 0.36217665672302246 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q8162 |
| concepts[8].display_name | Linguistics |
| concepts[9].id | https://openalex.org/C15744967 |
| concepts[9].level | 0 |
| concepts[9].score | 0.3576141893863678 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q9418 |
| concepts[9].display_name | Psychology |
| concepts[10].id | https://openalex.org/C95457728 |
| concepts[10].level | 0 |
| concepts[10].score | 0.1206730306148529 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q309 |
| concepts[10].display_name | History |
| concepts[11].id | https://openalex.org/C178790620 |
| concepts[11].level | 1 |
| concepts[11].score | 0.0 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q11351 |
| concepts[11].display_name | Organic chemistry |
| concepts[12].id | https://openalex.org/C166957645 |
| concepts[12].level | 1 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q23498 |
| concepts[12].display_name | Archaeology |
| concepts[13].id | https://openalex.org/C138885662 |
| concepts[13].level | 0 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[13].display_name | Philosophy |
| concepts[14].id | https://openalex.org/C185592680 |
| concepts[14].level | 0 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q2329 |
| concepts[14].display_name | Chemistry |
| keywords[0].id | https://openalex.org/keywords/prosody |
| keywords[0].score | 0.810196042060852 |
| keywords[0].display_name | Prosody |
| keywords[1].id | https://openalex.org/keywords/zero |
| keywords[1].score | 0.7463643550872803 |
| keywords[1].display_name | Zero (linguistics) |
| keywords[2].id | https://openalex.org/keywords/context |
| keywords[2].score | 0.6234064102172852 |
| keywords[2].display_name | Context (archaeology) |
| keywords[3].id | https://openalex.org/keywords/natural-language-processing |
| keywords[3].score | 0.5354601740837097 |
| keywords[3].display_name | Natural language processing |
| keywords[4].id | https://openalex.org/keywords/computer-science |
| keywords[4].score | 0.5179173350334167 |
| keywords[4].display_name | Computer science |
| keywords[5].id | https://openalex.org/keywords/shot |
| keywords[5].score | 0.4678158462047577 |
| keywords[5].display_name | Shot (pellet) |
| keywords[6].id | https://openalex.org/keywords/speech-recognition |
| keywords[6].score | 0.45178329944610596 |
| keywords[6].display_name | Speech recognition |
| keywords[7].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[7].score | 0.42458418011665344 |
| keywords[7].display_name | Artificial intelligence |
| keywords[8].id | https://openalex.org/keywords/linguistics |
| keywords[8].score | 0.36217665672302246 |
| keywords[8].display_name | Linguistics |
| keywords[9].id | https://openalex.org/keywords/psychology |
| keywords[9].score | 0.3576141893863678 |
| keywords[9].display_name | Psychology |
| keywords[10].id | https://openalex.org/keywords/history |
| keywords[10].score | 0.1206730306148529 |
| keywords[10].display_name | History |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2409.05004 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2409.05004 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2409.05004 |
| locations[1].id | doi:10.48550/arxiv.2409.05004 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2409.05004 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5101416769 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-1293-8146 |
| authorships[0].author.display_name | Zhengyang Chen |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Chen, Zhengyang |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5115695505 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-5653-7195 |
| authorships[1].author.display_name | Shuai Wang |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Wang, Shuai |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5100432825 |
| authorships[2].author.orcid | https://orcid.org/0009-0005-2291-5089 |
| authorships[2].author.display_name | Mingyang Zhang |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Zhang, Mingyang |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5014303571 |
| authorships[3].author.orcid | https://orcid.org/0000-0003-1207-5651 |
| authorships[3].author.display_name | Xuechen Liu |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Liu, Xuechen |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5007639385 |
| authorships[4].author.orcid | https://orcid.org/0000-0003-2752-3955 |
| authorships[4].author.display_name | Junichi Yamagishi |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Yamagishi, Junichi |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5100341993 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-0314-3790 |
| authorships[5].author.display_name | Yanmin Qian |
| authorships[5].author_position | last |
| authorships[5].raw_author_name | Qian, Yanmin |
| authorships[5].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2409.05004 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2024-10-22T00:00:00 |
| display_name | Disentangling the Prosody and Semantic Information with Pre-trained Model for In-Context Learning based Zero-Shot Voice Conversion |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10201 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9882000088691711 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Speech Recognition and Synthesis |
| related_works | https://openalex.org/W2074502265, https://openalex.org/W4214877189, https://openalex.org/W2773965352, https://openalex.org/W2381179799, https://openalex.org/W2355553914, https://openalex.org/W2980279061, https://openalex.org/W2334685461, https://openalex.org/W149862513, https://openalex.org/W2347684782, https://openalex.org/W187117048 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2409.05004 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2409.05004 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2409.05004 |
| primary_location.id | pmh:oai:arXiv.org:2409.05004 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2409.05004 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2409.05004 |
| publication_date | 2024-09-08 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 62, 95, 129 |
| abstract_inverted_index.To | 118 |
| abstract_inverted_index.VC | 58 |
| abstract_inverted_index.an | 54 |
| abstract_inverted_index.as | 46, 151 |
| abstract_inverted_index.in | 36, 110 |
| abstract_inverted_index.is | 94 |
| abstract_inverted_index.of | 26, 114, 138 |
| abstract_inverted_index.on | 69, 79, 153 |
| abstract_inverted_index.to | 4, 100, 146 |
| abstract_inverted_index.we | 90, 122 |
| abstract_inverted_index.ICL | 55 |
| abstract_inverted_index.and | 64 |
| abstract_inverted_index.for | 40 |
| abstract_inverted_index.has | 34 |
| abstract_inverted_index.our | 77, 135 |
| abstract_inverted_index.the | 6, 17, 80, 105, 112, 115, 143, 154 |
| abstract_inverted_index.(VC) | 2 |
| abstract_inverted_index.This | 51 |
| abstract_inverted_index.aims | 3 |
| abstract_inverted_index.find | 91 |
| abstract_inverted_index.from | 19, 128 |
| abstract_inverted_index.have | 15 |
| abstract_inverted_index.into | 21, 134 |
| abstract_inverted_index.mask | 63 |
| abstract_inverted_index.such | 45 |
| abstract_inverted_index.that | 84, 92 |
| abstract_inverted_index.this | 120 |
| abstract_inverted_index.with | 74 |
| abstract_inverted_index.(ICL) | 33 |
| abstract_inverted_index.(TTS) | 38 |
| abstract_inverted_index.Voice | 0 |
| abstract_inverted_index.based | 68 |
| abstract_inverted_index.faces | 108 |
| abstract_inverted_index.model | 133 |
| abstract_inverted_index.paper | 52 |
| abstract_inverted_index.while | 9 |
| abstract_inverted_index.ICL-VC | 85, 106 |
| abstract_inverted_index.Speech | 156 |
| abstract_inverted_index.issue, | 121 |
| abstract_inverted_index.method | 98 |
| abstract_inverted_index.modify | 5 |
| abstract_inverted_index.source | 116, 148 |
| abstract_inverted_index.speech | 11, 27, 149 |
| abstract_inverted_index.system | 59, 107 |
| abstract_inverted_index.timbre | 8, 47 |
| abstract_inverted_index.content | 28 |
| abstract_inverted_index.context | 49 |
| abstract_inverted_index.dataset | 82 |
| abstract_inverted_index.emerged | 35 |
| abstract_inverted_index.emotion | 131 |
| abstract_inverted_index.k-means | 93 |
| abstract_inverted_index.models. | 72, 103 |
| abstract_inverted_index.notably | 141 |
| abstract_inverted_index.outputs | 18 |
| abstract_inverted_index.propose | 123 |
| abstract_inverted_index.prosody | 113, 125, 139 |
| abstract_inverted_index.speaker | 87 |
| abstract_inverted_index.speech. | 117 |
| abstract_inverted_index.system. | 136 |
| abstract_inverted_index.systems | 39 |
| abstract_inverted_index.through | 48 |
| abstract_inverted_index.tokens, | 23, 76 |
| abstract_inverted_index.various | 101 |
| abstract_inverted_index.(ICL-VC) | 60 |
| abstract_inverted_index.However, | 104 |
| abstract_inverted_index.LibriTTS | 81 |
| abstract_inverted_index.Previous | 13 |
| abstract_inverted_index.content. | 12 |
| abstract_inverted_index.enhanced | 57 |
| abstract_inverted_index.enhances | 142 |
| abstract_inverted_index.improves | 86 |
| abstract_inverted_index.learning | 32 |
| abstract_inverted_index.mitigate | 119 |
| abstract_inverted_index.modeling | 42 |
| abstract_inverted_index.preserve | 147 |
| abstract_inverted_index.proposes | 53 |
| abstract_inverted_index.prosody, | 150 |
| abstract_inverted_index.semantic | 22, 75 |
| abstract_inverted_index.specific | 43 |
| abstract_inverted_index.strategy | 67 |
| abstract_inverted_index.system's | 144 |
| abstract_inverted_index.training | 66 |
| abstract_inverted_index.Augmented | 73 |
| abstract_inverted_index.Database. | 157 |
| abstract_inverted_index.Emotional | 155 |
| abstract_inverted_index.Recently, | 30 |
| abstract_inverted_index.employing | 61 |
| abstract_inverted_index.extracted | 127 |
| abstract_inverted_index.retaining | 10 |
| abstract_inverted_index.speaker's | 7 |
| abstract_inverted_index.tokenized | 16 |
| abstract_inverted_index.validated | 152 |
| abstract_inverted_index.versatile | 96 |
| abstract_inverted_index.applicable | 99 |
| abstract_inverted_index.approaches | 14 |
| abstract_inverted_index.capability | 56, 145 |
| abstract_inverted_index.challenges | 109 |
| abstract_inverted_index.conversion | 1 |
| abstract_inverted_index.embeddings | 126, 140 |
| abstract_inverted_index.generative | 71 |
| abstract_inverted_index.in-context | 31 |
| abstract_inverted_index.preserving | 111 |
| abstract_inverted_index.Integration | 137 |
| abstract_inverted_index.demonstrate | 83 |
| abstract_inverted_index.effectively | 41 |
| abstract_inverted_index.experiments | 78 |
| abstract_inverted_index.pre-trained | 102, 130 |
| abstract_inverted_index.recognition | 132 |
| abstract_inverted_index.similarity. | 88 |
| abstract_inverted_index.facilitating | 24 |
| abstract_inverted_index.information. | 29 |
| abstract_inverted_index.tokenization | 97 |
| abstract_inverted_index.Additionally, | 89 |
| abstract_inverted_index.conditioning. | 50 |
| abstract_inverted_index.flow-matching | 70 |
| abstract_inverted_index.incorporating | 124 |
| abstract_inverted_index.reconstruction | 65 |
| abstract_inverted_index.text-to-speech | 37 |
| abstract_inverted_index.characteristics | 44 |
| abstract_inverted_index.disentanglement | 25 |
| abstract_inverted_index.self-supervised | 20 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 6 |
| citation_normalized_percentile |