Normalizing Flows are Capable Generative Models Article Swipe

PDF

Shuangfei Zhai , Ruixiang Zhang , Preetum Nakkiran , David Berthelot , Jiatao Gu , Huangjie Zheng , Tianrong Chen , Miguel Ángel Bautista , Navdeep Jaitly , Josh Susskind ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2412.06329

Normalizing Flows (NFs) are likelihood-based models for continuous inputs. They have demonstrated promising results on both density estimation and generative modeling tasks, but have received relatively little attention in recent years. In this work, we demonstrate that NFs are more powerful than previously believed. We present TarFlow: a simple and scalable architecture that enables highly performant NF models. TarFlow can be thought of as a Transformer-based variant of Masked Autoregressive Flows (MAFs): it consists of a stack of autoregressive Transformer blocks on image patches, alternating the autoregression direction between layers. TarFlow is straightforward to train end-to-end, and capable of directly modeling and generating pixels. We also propose three key techniques to improve sample quality: Gaussian noise augmentation during training, a post training denoising procedure, and an effective guidance method for both class-conditional and unconditional settings. Putting these together, TarFlow sets new state-of-the-art results on likelihood estimation for images, beating the previous best methods by a large margin, and generates samples with quality and diversity comparable to diffusion models, for the first time with a stand-alone NF model. We make our code available at https://github.com/apple/ml-tarflow.

Related Topics

Generative Grammar

Computer Science

Artificial Intelligence

Concepts

Generative grammar Computer science Artificial intelligence

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2412.06329
PDF: https://arxiv.org/pdf/2412.06329
OA Status: green
Cited By: 2
Related Works: 10
OpenAlex ID: https://openalex.org/W4405254033

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4405254033

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2412.06329

Digital Object Identifier
Title: Normalizing Flows are Capable Generative Models

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-12-09

Full publication date if available
Authors: Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Ángel Bautista, Navdeep Jaitly, Josh Susskind

List of authors in order
Landing page: https://arxiv.org/abs/2412.06329

Publisher landing page
PDF URL: https://arxiv.org/pdf/2412.06329

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2412.06329

Direct OA link when available
Concepts: Generative grammar, Computer science, Artificial intelligence

Top concepts (fields/topics) attached by OpenAlex
Cited by: 2

Total citation count in OpenAlex
Citations by year (recent): 2025: 2

Per-year citation counts (last 5 years)
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4405254033
doi	https://doi.org/10.48550/arxiv.2412.06329
ids.doi	https://doi.org/10.48550/arxiv.2412.06329
ids.openalex	https://openalex.org/W4405254033
fwci
type	preprint
title	Normalizing Flows are Capable Generative Models
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T11801
topics[0].field.id	https://openalex.org/fields/22
topics[0].field.display_name	Engineering
topics[0].score	0.8867999911308289
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/2212
topics[0].subfield.display_name	Ocean Engineering
topics[0].display_name	Reservoir Engineering and Simulation Methods
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C39890363
concepts[0].level	2
concepts[0].score	0.6469975113868713
concepts[0].wikidata	https://www.wikidata.org/wiki/Q36108
concepts[0].display_name	Generative grammar
concepts[1].id	https://openalex.org/C41008148
concepts[1].level	0
concepts[1].score	0.38083115220069885
concepts[1].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[1].display_name	Computer science
concepts[2].id	https://openalex.org/C154945302
concepts[2].level	1
concepts[2].score	0.2644583582878113
concepts[2].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[2].display_name	Artificial intelligence
keywords[0].id	https://openalex.org/keywords/generative-grammar
keywords[0].score	0.6469975113868713
keywords[0].display_name	Generative grammar
keywords[1].id	https://openalex.org/keywords/computer-science
keywords[1].score	0.38083115220069885
keywords[1].display_name	Computer science
keywords[2].id	https://openalex.org/keywords/artificial-intelligence
keywords[2].score	0.2644583582878113
keywords[2].display_name	Artificial intelligence
language	en
locations[0].id	pmh:oai:arXiv.org:2412.06329
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2412.06329
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2412.06329
locations[1].id	doi:10.48550/arxiv.2412.06329
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2412.06329
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5031329242
authorships[0].author.orcid
authorships[0].author.display_name	Shuangfei Zhai
authorships[0].author_position	first
authorships[0].raw_author_name	Zhai, Shuangfei
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5008617245
authorships[1].author.orcid	https://orcid.org/0000-0003-0704-2484
authorships[1].author.display_name	Ruixiang Zhang
authorships[1].author_position	middle
authorships[1].raw_author_name	Zhang, Ruixiang
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5110716768
authorships[2].author.orcid
authorships[2].author.display_name	Preetum Nakkiran
authorships[2].author_position	middle
authorships[2].raw_author_name	Nakkiran, Preetum
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5088218200
authorships[3].author.orcid
authorships[3].author.display_name	David Berthelot
authorships[3].author_position	middle
authorships[3].raw_author_name	Berthelot, David
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5112542984
authorships[4].author.orcid
authorships[4].author.display_name	Jiatao Gu
authorships[4].author_position	middle
authorships[4].raw_author_name	Gu, Jiatao
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5077499817
authorships[5].author.orcid	https://orcid.org/0000-0003-0508-5034
authorships[5].author.display_name	Huangjie Zheng
authorships[5].author_position	middle
authorships[5].raw_author_name	Zheng, Huangjie
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5087793625
authorships[6].author.orcid	https://orcid.org/0000-0003-1701-2991
authorships[6].author.display_name	Tianrong Chen
authorships[6].author_position	middle
authorships[6].raw_author_name	Chen, Tianrong
authorships[6].is_corresponding	False
authorships[7].author.id	https://openalex.org/A5011106959
authorships[7].author.orcid
authorships[7].author.display_name	Miguel Ángel Bautista
authorships[7].author_position	middle
authorships[7].raw_author_name	Bautista, Miguel Angel
authorships[7].is_corresponding	False
authorships[8].author.id	https://openalex.org/A5112445699
authorships[8].author.orcid
authorships[8].author.display_name	Navdeep Jaitly
authorships[8].author_position	middle
authorships[8].raw_author_name	Jaitly, Navdeep
authorships[8].is_corresponding	False
authorships[9].author.id	https://openalex.org/A5043808400
authorships[9].author.orcid
authorships[9].author.display_name	Josh Susskind
authorships[9].author_position	last
authorships[9].raw_author_name	Susskind, Josh
authorships[9].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2412.06329
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Normalizing Flows are Capable Generative Models
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T11801
primary_topic.field.id	https://openalex.org/fields/22
primary_topic.field.display_name	Engineering
primary_topic.score	0.8867999911308289
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/2212
primary_topic.subfield.display_name	Ocean Engineering
primary_topic.display_name	Reservoir Engineering and Simulation Methods
related_works	https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2380075625, https://openalex.org/W2390279801, https://openalex.org/W4391913857, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890
cited_by_count	2
counts_by_year[0].year	2025
counts_by_year[0].cited_by_count	2
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2412.06329
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2412.06329
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2412.06329
primary_location.id	pmh:oai:arXiv.org:2412.06329
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2412.06329
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2412.06329
publication_date	2024-12-09
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	47, 64, 75, 119, 154, 173
abstract_inverted_index.In	31
abstract_inverted_index.NF	56, 175
abstract_inverted_index.We	44, 104, 177
abstract_inverted_index.an	125
abstract_inverted_index.as	63
abstract_inverted_index.at	182
abstract_inverted_index.be	60
abstract_inverted_index.by	153
abstract_inverted_index.in	28
abstract_inverted_index.is	91
abstract_inverted_index.it	72
abstract_inverted_index.of	62, 67, 74, 77, 98
abstract_inverted_index.on	14, 81, 143
abstract_inverted_index.to	93, 110, 165
abstract_inverted_index.we	34
abstract_inverted_index.NFs	37
abstract_inverted_index.and	18, 49, 96, 101, 124, 132, 157, 162
abstract_inverted_index.are	3, 38
abstract_inverted_index.but	22
abstract_inverted_index.can	59
abstract_inverted_index.for	6, 129, 146, 168
abstract_inverted_index.key	108
abstract_inverted_index.new	140
abstract_inverted_index.our	179
abstract_inverted_index.the	85, 149, 169
abstract_inverted_index.They	9
abstract_inverted_index.also	105
abstract_inverted_index.best	151
abstract_inverted_index.both	15, 130
abstract_inverted_index.code	180
abstract_inverted_index.have	10, 23
abstract_inverted_index.make	178
abstract_inverted_index.more	39
abstract_inverted_index.post	120
abstract_inverted_index.sets	139
abstract_inverted_index.than	41
abstract_inverted_index.that	36, 52
abstract_inverted_index.this	32
abstract_inverted_index.time	171
abstract_inverted_index.with	160, 172
abstract_inverted_index.(NFs)	2
abstract_inverted_index.Flows	1, 70
abstract_inverted_index.first	170
abstract_inverted_index.image	82
abstract_inverted_index.large	155
abstract_inverted_index.noise	115
abstract_inverted_index.stack	76
abstract_inverted_index.these	136
abstract_inverted_index.three	107
abstract_inverted_index.train	94
abstract_inverted_index.work,	33
abstract_inverted_index.Masked	68
abstract_inverted_index.blocks	80
abstract_inverted_index.during	117
abstract_inverted_index.highly	54
abstract_inverted_index.little	26
abstract_inverted_index.method	128
abstract_inverted_index.model.	176
abstract_inverted_index.models	5
abstract_inverted_index.recent	29
abstract_inverted_index.sample	112
abstract_inverted_index.simple	48
abstract_inverted_index.tasks,	21
abstract_inverted_index.years.	30
abstract_inverted_index.(MAFs):	71
abstract_inverted_index.Putting	135
abstract_inverted_index.TarFlow	58, 90, 138
abstract_inverted_index.beating	148
abstract_inverted_index.between	88
abstract_inverted_index.capable	97
abstract_inverted_index.density	16
abstract_inverted_index.enables	53
abstract_inverted_index.images,	147
abstract_inverted_index.improve	111
abstract_inverted_index.inputs.	8
abstract_inverted_index.layers.	89
abstract_inverted_index.margin,	156
abstract_inverted_index.methods	152
abstract_inverted_index.models,	167
abstract_inverted_index.models.	57
abstract_inverted_index.pixels.	103
abstract_inverted_index.present	45
abstract_inverted_index.propose	106
abstract_inverted_index.quality	161
abstract_inverted_index.results	13, 142
abstract_inverted_index.samples	159
abstract_inverted_index.thought	61
abstract_inverted_index.variant	66
abstract_inverted_index.Gaussian	114
abstract_inverted_index.TarFlow:	46
abstract_inverted_index.consists	73
abstract_inverted_index.directly	99
abstract_inverted_index.guidance	127
abstract_inverted_index.modeling	20, 100
abstract_inverted_index.patches,	83
abstract_inverted_index.powerful	40
abstract_inverted_index.previous	150
abstract_inverted_index.quality:	113
abstract_inverted_index.received	24
abstract_inverted_index.scalable	50
abstract_inverted_index.training	121
abstract_inverted_index.attention	27
abstract_inverted_index.available	181
abstract_inverted_index.believed.	43
abstract_inverted_index.denoising	122
abstract_inverted_index.diffusion	166
abstract_inverted_index.direction	87
abstract_inverted_index.diversity	163
abstract_inverted_index.effective	126
abstract_inverted_index.generates	158
abstract_inverted_index.promising	12
abstract_inverted_index.settings.	134
abstract_inverted_index.together,	137
abstract_inverted_index.training,	118
abstract_inverted_index.comparable	164
abstract_inverted_index.continuous	7
abstract_inverted_index.estimation	17, 145
abstract_inverted_index.generating	102
abstract_inverted_index.generative	19
abstract_inverted_index.likelihood	144
abstract_inverted_index.performant	55
abstract_inverted_index.previously	42
abstract_inverted_index.procedure,	123
abstract_inverted_index.relatively	25
abstract_inverted_index.techniques	109
abstract_inverted_index.Normalizing	0
abstract_inverted_index.Transformer	79
abstract_inverted_index.alternating	84
abstract_inverted_index.demonstrate	35
abstract_inverted_index.end-to-end,	95
abstract_inverted_index.stand-alone	174
abstract_inverted_index.architecture	51
abstract_inverted_index.augmentation	116
abstract_inverted_index.demonstrated	11
abstract_inverted_index.unconditional	133
abstract_inverted_index.Autoregressive	69
abstract_inverted_index.autoregression	86
abstract_inverted_index.autoregressive	78
abstract_inverted_index.straightforward	92
abstract_inverted_index.likelihood-based	4
abstract_inverted_index.state-of-the-art	141
abstract_inverted_index.Transformer-based	65
abstract_inverted_index.class-conditional	131
abstract_inverted_index.https://github.com/apple/ml-tarflow.	183
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	10
citation_normalized_percentile