Rate of Model Collapse in Recursive Training Article Swipe

MAKE IMPACT

Ananda Theertha Suresh , Andrew Thangaraj , Aditya Nanda Kishore Khandavally · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2412.17646

Given the ease of creating synthetic data from machine learning models, new models can be potentially trained on synthetic data generated by previous models. This recursive training process raises concerns about the long-term impact on model quality. As models are recursively trained on generated data from previous rounds, their ability to capture the nuances of the original human-generated data may degrade. This is often referred to as \emph{model collapse}. In this work, we ask how fast model collapse occurs for some well-studied distribution families under maximum likelihood (ML or near ML) estimation during recursive training. Surprisingly, even for fundamental distributions such as discrete and Gaussian distributions, the exact rate of model collapse is unknown. In this work, we theoretically characterize the rate of collapse in these fundamental settings and complement it with experimental evaluations. Our results show that for discrete distributions, the time to forget a word is approximately linearly dependent on the number of times it occurred in the original corpus, and for Gaussian models, the standard deviation reduces to zero roughly at $n$ iterations, where $n$ is the number of samples at each iteration. Both of these findings imply that model forgetting, at least in these simple distributions under near ML estimation with many samples, takes a long time.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2412.17646

PDF

https://arxiv.org/pdf/2412.17646

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4405768339

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4405768339

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2412.17646

Digital Object Identifier

Title

Rate of Model Collapse in Recursive Training

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-12-23

Full publication date if available

Authors

Ananda Theertha Suresh, Andrew Thangaraj, Aditya Nanda Kishore Khandavally

List of authors in order

Landing page

https://arxiv.org/abs/2412.17646

Publisher landing page

PDF URL

https://arxiv.org/pdf/2412.17646

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2412.17646

Direct OA link when available

Concepts

Training (meteorology), Computer science, Geography, Meteorology

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload