BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping Article Swipe

MAKE IMPACT

Jiatao Gu , Shuangfei Zhai , Yizhe Zhang , Lingjie Liu , Josh Susskind · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2306.05544

Diffusion models have demonstrated excellent potential for generating diverse images. However, their performance often suffers from slow generation due to iterative denoising. Knowledge distillation has been recently proposed as a remedy that can reduce the number of inference steps to one or a few without significant quality degradation. However, existing distillation methods either require significant amounts of offline computation for generating synthetic training data from the teacher model or need to perform expensive online learning with the help of real data. In this work, we present a novel technique called BOOT, that overcomes these limitations with an efficient data-free distillation algorithm. The core idea is to learn a time-conditioned model that predicts the output of a pre-trained diffusion model teacher given any time step. Such a model can be efficiently trained based on bootstrapping from two consecutive sampled steps. Furthermore, our method can be easily adapted to large-scale text-to-image diffusion models, which are challenging for conventional methods given the fact that the training sets are often large and difficult to access. We demonstrate the effectiveness of our approach on several benchmark datasets in the DDIM setting, achieving comparable generation quality while being orders of magnitude faster than the diffusion teacher. The text-to-image results show that the proposed approach is able to handle highly complex distributions, shedding light on more efficient generative modeling.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2306.05544

PDF

https://arxiv.org/pdf/2306.05544

OA Status

green

Cited By

4

Related Works

10

OpenAlex ID

https://openalex.org/W4380352311

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4380352311

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2306.05544

Digital Object Identifier

Title

BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-06-08

Full publication date if available

Authors

Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, Josh Susskind

List of authors in order

Landing page

https://arxiv.org/abs/2306.05544

Publisher landing page

PDF URL

https://arxiv.org/pdf/2306.05544

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2306.05544

Direct OA link when available

Concepts

Computer science, Benchmark (surveying), Inference, Bootstrapping (finance), Computation, Distillation, Noise reduction, Machine learning, Overhead (engineering), Artificial intelligence, Generative model, Generative grammar, Algorithm, Mathematics, Econometrics, Geodesy, Operating system, Organic chemistry, Chemistry, Geography

Top concepts (fields/topics) attached by OpenAlex

Cited by

4

Total citation count in OpenAlex

Citations by year (recent)

2024: 3, 2023: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload