Leveraging Model Guidance to Extract Training Data from Personalized Diffusion Models Article Swipe

MAKE IMPACT

Xiaoyu Wu , Jiaru Zhang , Steven Y. Wu · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2410.03039

Diffusion Models (DMs) have become powerful image generation tools, especially for few-shot fine-tuning where a pretrained DM is fine-tuned on a small image set to capture specific styles or objects. Many people upload these personalized checkpoints online, fostering communities such as Civitai and HuggingFace. However, model owners may overlook the data leakage risks when releasing fine-tuned checkpoints. Moreover, concerns regarding copyright violations arise when unauthorized data is used during fine-tuning. In this paper, we ask: "Can training data be extracted from these fine-tuned DMs shared online?" A successful extraction would present not only data leakage threats but also offer tangible evidence of copyright infringement. To answer this, we propose FineXtract, a framework for extracting fine-tuning data. Our method approximates fine-tuning as a gradual shift in the model's learned distribution -- from the original pretrained DM toward the fine-tuning data. By extrapolating the models before and after fine-tuning, we guide the generation toward high-probability regions within the fine-tuned data distribution. We then apply a clustering algorithm to extract the most probable images from those generated using this extrapolated guidance. Experiments on DMs fine-tuned with datasets including WikiArt, DreamBooth, and real-world checkpoints posted online validate the effectiveness of our method, extracting about 20% of fine-tuning data in most cases. The code is available https://github.com/Nicholas0228/FineXtract.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2410.03039

PDF

https://arxiv.org/pdf/2410.03039

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4403885535

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4403885535

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2410.03039

Digital Object Identifier

Title

Leveraging Model Guidance to Extract Training Data from Personalized Diffusion Models

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-10-03

Full publication date if available

Authors

Xiaoyu Wu, Jiaru Zhang, Steven Y. Wu

List of authors in order

Landing page

https://arxiv.org/abs/2410.03039

Publisher landing page

PDF URL

https://arxiv.org/pdf/2410.03039

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2410.03039

Direct OA link when available

Concepts

EXPOSE, Training (meteorology), Computer science, Diffusion, Training set, Data science, Artificial intelligence, Geography, Meteorology, Thermodynamics, Astronomy, Physics

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload