High-fidelity Person-centric Subject-to-Image Synthesis Article Swipe

MAKE IMPACT

Y. N. Wang , Weizhong Zhang , Jianwei Zheng , Jin Cheng · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2311.10329

Current subject-driven image generation methods encounter significant challenges in person-centric image generation. The reason is that they learn the semantic scene and person generation by fine-tuning a common pre-trained diffusion, which involves an irreconcilable training imbalance. Precisely, to generate realistic persons, they need to sufficiently tune the pre-trained model, which inevitably causes the model to forget the rich semantic scene prior and makes scene generation over-fit to the training data. Moreover, even with sufficient fine-tuning, these methods can still not generate high-fidelity persons since joint learning of the scene and person generation also lead to quality compromise. In this paper, we propose Face-diffuser, an effective collaborative generation pipeline to eliminate the above training imbalance and quality compromise. Specifically, we first develop two specialized pre-trained diffusion models, i.e., Text-driven Diffusion Model (TDM) and Subject-augmented Diffusion Model (SDM), for scene and person generation, respectively. The sampling process is divided into three sequential stages, i.e., semantic scene construction, subject-scene fusion, and subject enhancement. The first and last stages are performed by TDM and SDM respectively. The subject-scene fusion stage, that is the collaboration achieved through a novel and highly effective mechanism, Saliency-adaptive Noise Fusion (SNF). Specifically, it is based on our key observation that there exists a robust link between classifier-free guidance responses and the saliency of generated images. In each time step, SNF leverages the unique strengths of each model and allows for the spatial blending of predicted noises from both models automatically in a saliency-aware manner. Extensive experiments confirm the impressive effectiveness and robustness of the Face-diffuser.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2311.10329

PDF

https://arxiv.org/pdf/2311.10329

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4388843381

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4388843381

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2311.10329

Digital Object Identifier

Title

High-fidelity Person-centric Subject-to-Image Synthesis

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-11-17

Full publication date if available

Authors

Y. N. Wang, Weizhong Zhang, Jianwei Zheng, Jin Cheng

List of authors in order

Landing page

https://arxiv.org/abs/2311.10329

Publisher landing page

PDF URL

https://arxiv.org/pdf/2311.10329

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2311.10329

Direct OA link when available

Concepts

Computer science, Fidelity, Artificial intelligence, Segmentation, Classifier (UML), Computer vision, High fidelity, Engineering, Electrical engineering, Telecommunications

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload