An Image is Worth Multiple Words: Multi-attribute Inversion for Constrained Text-to-Image Synthesis Article Swipe

MAKE IMPACT

Aishwarya Agarwal , Srikrishna Karanam , Tripti Shukla , B. Srinivasan · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2311.11919

We consider the problem of constraining diffusion model outputs with a user-supplied reference image. Our key objective is to extract multiple attributes (e.g., color, object, layout, style) from this single reference image, and then generate new samples with them. One line of existing work proposes to invert the reference images into a single textual conditioning vector, enabling generation of new samples with this learned token. These methods, however, do not learn multiple tokens that are necessary to condition model outputs on the multiple attributes noted above. Another line of techniques expand the inversion space to learn multiple embeddings but they do this only along the layer dimension (e.g., one per layer of the DDPM model) or the timestep dimension (one for a set of timesteps in the denoising process), leading to suboptimal attribute disentanglement. To address the aforementioned gaps, the first contribution of this paper is an extensive analysis to determine which attributes are captured in which dimension of the denoising process. As noted above, we consider both the time-step dimension (in reverse denoising) as well as the DDPM model layer dimension. We observe that often a subset of these attributes are captured in the same set of model layers and/or across same denoising timesteps. For instance, color and style are captured across same U-Net layers, whereas layout and color are captured across same timestep stages. Consequently, an inversion process that is designed only for the time-step dimension or the layer dimension is insufficient to disentangle all attributes. This leads to our second contribution where we design a new multi-attribute inversion algorithm, MATTE, with associated disentanglement-enhancing regularization losses, that operates across both dimensions and explicitly leads to four disentangled tokens (color, style, layout, and object).

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2311.11919

PDF

https://arxiv.org/pdf/2311.11919

OA Status

green

Cited By

2

Related Works

10

OpenAlex ID

https://openalex.org/W4388926651

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4388926651

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2311.11919

Digital Object Identifier

Title

An Image is Worth Multiple Words: Multi-attribute Inversion for Constrained Text-to-Image Synthesis

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-11-20

Full publication date if available

Authors

Aishwarya Agarwal, Srikrishna Karanam, Tripti Shukla, B. Srinivasan

List of authors in order

Landing page

https://arxiv.org/abs/2311.11919

Publisher landing page

PDF URL

https://arxiv.org/pdf/2311.11919

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2311.11919

Direct OA link when available

Concepts

Computer science, Dimension (graph theory), Inversion (geology), Dimensionality reduction, Artificial intelligence, Set (abstract data type), Noise reduction, Pattern recognition (psychology), Process (computing), Image (mathematics), Algorithm, Mathematics, Pure mathematics, Operating system, Structural basin, Programming language, Paleontology, Biology

Top concepts (fields/topics) attached by OpenAlex

Cited by

2

Total citation count in OpenAlex

Citations by year (recent)

2025: 2

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload