Autoregressive Diffusion Transformer for Text-to-Speech Synthesis Article Swipe

MAKE IMPACT

Zhijun Liu , Shuai Wang , Sho Inoue , Qibing Bai , Haizhou Li · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2406.05551

Audio language models have recently emerged as a promising approach for various audio generation tasks, relying on audio tokenizers to encode waveforms into sequences of discrete symbols. Audio tokenization often poses a necessary compromise between code bitrate and reconstruction accuracy. When dealing with low-bitrate audio codes, language models are constrained to process only a subset of the information embedded in the audio, which in turn restricts their generative capabilities. To circumvent these issues, we propose encoding audio as vector sequences in continuous space $\mathbb R^d$ and autoregressively generating these sequences using a decoder-only diffusion transformer (ARDiT). Our findings indicate that ARDiT excels in zero-shot text-to-speech and exhibits performance that compares to or even surpasses that of state-of-the-art models. High-bitrate continuous speech representation enables almost flawless reconstruction, allowing our model to achieve nearly perfect speech editing. Our experiments reveal that employing Integral Kullback-Leibler (IKL) divergence for distillation at each autoregressive step significantly boosts the perceived quality of the samples. Simultaneously, it condenses the iterative sampling process of the diffusion model into a single step. Furthermore, ARDiT can be trained to predict several continuous vectors in one step, significantly reducing latency during sampling. Impressively, one of our models can generate $170$ ms of $24$ kHz speech per evaluation step with minimal degradation in performance. Audio samples are available at http://ardit-tts.github.io/ .

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2406.05551

PDF

https://arxiv.org/pdf/2406.05551

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4399554908

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4399554908

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2406.05551

Digital Object Identifier

Title

Autoregressive Diffusion Transformer for Text-to-Speech Synthesis

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-06-08

Full publication date if available

Authors

Zhijun Liu, Shuai Wang, Sho Inoue, Qibing Bai, Haizhou Li

List of authors in order

Landing page

https://arxiv.org/abs/2406.05551

Publisher landing page

PDF URL

https://arxiv.org/pdf/2406.05551

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2406.05551

Direct OA link when available

Concepts

Autoregressive model, Transformer, Speech recognition, Computer science, Diffusion, Natural language processing, Electrical engineering, Econometrics, Mathematics, Engineering, Physics, Voltage, Thermodynamics

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload