Mitigate Replication and Copying in Diffusion Models with Generalized Caption and Dual Fusion Enhancement Article Swipe
Chenghao Li
,
Dake Chen
,
Yuke Zhang
,
Peter A. Beerel
·
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2309.07254
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2309.07254
While diffusion models demonstrate a remarkable capability for generating high-quality images, their tendency to `replicate' training data raises privacy concerns. Although recent research suggests that this replication may stem from the insufficient generalization of training data captions and duplication of training images, effective mitigation strategies remain elusive. To address this gap, our paper first introduces a generality score that measures the caption generality and employ large language model (LLM) to generalize training captions. Subsequently, we leverage generalized captions and propose a novel dual fusion enhancement approach to mitigate the replication of diffusion models. Our empirical results demonstrate that our proposed methods can significantly reduce replication by 43.5% compared to the original diffusion model while maintaining the diversity and quality of generations. Code is available at https://github.com/HowardLi0816/dual-fusion-diffusion.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2309.07254
- https://arxiv.org/pdf/2309.07254
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4386794235
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4386794235Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2309.07254Digital Object Identifier
- Title
-
Mitigate Replication and Copying in Diffusion Models with Generalized Caption and Dual Fusion EnhancementWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-09-13Full publication date if available
- Authors
-
Chenghao Li, Dake Chen, Yuke Zhang, Peter A. BeerelList of authors in order
- Landing page
-
https://arxiv.org/abs/2309.07254Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2309.07254Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2309.07254Direct OA link when available
- Concepts
-
Generality, Computer science, Replication (statistics), Copying, Leverage (statistics), Artificial intelligence, Dual (grammatical number), Diffusion, Machine learning, Psychology, Mathematics, Literature, Art, Psychotherapist, Political science, Statistics, Physics, Law, ThermodynamicsTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4386794235 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2309.07254 |
| ids.doi | https://doi.org/10.48550/arxiv.2309.07254 |
| ids.openalex | https://openalex.org/W4386794235 |
| fwci | |
| type | preprint |
| title | Mitigate Replication and Copying in Diffusion Models with Generalized Caption and Dual Fusion Enhancement |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11714 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.993399977684021 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1707 |
| topics[0].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[0].display_name | Multimodal Machine Learning Applications |
| topics[1].id | https://openalex.org/T10775 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9923999905586243 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1707 |
| topics[1].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[1].display_name | Generative Adversarial Networks and Image Synthesis |
| topics[2].id | https://openalex.org/T11307 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.991100013256073 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Domain Adaptation and Few-Shot Learning |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C2780767217 |
| concepts[0].level | 2 |
| concepts[0].score | 0.7253448963165283 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q5532421 |
| concepts[0].display_name | Generality |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.7237642407417297 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C12590798 |
| concepts[2].level | 2 |
| concepts[2].score | 0.689287006855011 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q3933199 |
| concepts[2].display_name | Replication (statistics) |
| concepts[3].id | https://openalex.org/C2779151265 |
| concepts[3].level | 2 |
| concepts[3].score | 0.638887882232666 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q1156791 |
| concepts[3].display_name | Copying |
| concepts[4].id | https://openalex.org/C153083717 |
| concepts[4].level | 2 |
| concepts[4].score | 0.629676103591919 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q6535263 |
| concepts[4].display_name | Leverage (statistics) |
| concepts[5].id | https://openalex.org/C154945302 |
| concepts[5].level | 1 |
| concepts[5].score | 0.4767259657382965 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[5].display_name | Artificial intelligence |
| concepts[6].id | https://openalex.org/C2780980858 |
| concepts[6].level | 2 |
| concepts[6].score | 0.46812617778778076 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q110022 |
| concepts[6].display_name | Dual (grammatical number) |
| concepts[7].id | https://openalex.org/C69357855 |
| concepts[7].level | 2 |
| concepts[7].score | 0.4629417955875397 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q163214 |
| concepts[7].display_name | Diffusion |
| concepts[8].id | https://openalex.org/C119857082 |
| concepts[8].level | 1 |
| concepts[8].score | 0.34317922592163086 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[8].display_name | Machine learning |
| concepts[9].id | https://openalex.org/C15744967 |
| concepts[9].level | 0 |
| concepts[9].score | 0.11735379695892334 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q9418 |
| concepts[9].display_name | Psychology |
| concepts[10].id | https://openalex.org/C33923547 |
| concepts[10].level | 0 |
| concepts[10].score | 0.10672011971473694 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[10].display_name | Mathematics |
| concepts[11].id | https://openalex.org/C124952713 |
| concepts[11].level | 1 |
| concepts[11].score | 0.0 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q8242 |
| concepts[11].display_name | Literature |
| concepts[12].id | https://openalex.org/C142362112 |
| concepts[12].level | 0 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q735 |
| concepts[12].display_name | Art |
| concepts[13].id | https://openalex.org/C542102704 |
| concepts[13].level | 1 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q183257 |
| concepts[13].display_name | Psychotherapist |
| concepts[14].id | https://openalex.org/C17744445 |
| concepts[14].level | 0 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q36442 |
| concepts[14].display_name | Political science |
| concepts[15].id | https://openalex.org/C105795698 |
| concepts[15].level | 1 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q12483 |
| concepts[15].display_name | Statistics |
| concepts[16].id | https://openalex.org/C121332964 |
| concepts[16].level | 0 |
| concepts[16].score | 0.0 |
| concepts[16].wikidata | https://www.wikidata.org/wiki/Q413 |
| concepts[16].display_name | Physics |
| concepts[17].id | https://openalex.org/C199539241 |
| concepts[17].level | 1 |
| concepts[17].score | 0.0 |
| concepts[17].wikidata | https://www.wikidata.org/wiki/Q7748 |
| concepts[17].display_name | Law |
| concepts[18].id | https://openalex.org/C97355855 |
| concepts[18].level | 1 |
| concepts[18].score | 0.0 |
| concepts[18].wikidata | https://www.wikidata.org/wiki/Q11473 |
| concepts[18].display_name | Thermodynamics |
| keywords[0].id | https://openalex.org/keywords/generality |
| keywords[0].score | 0.7253448963165283 |
| keywords[0].display_name | Generality |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.7237642407417297 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/replication |
| keywords[2].score | 0.689287006855011 |
| keywords[2].display_name | Replication (statistics) |
| keywords[3].id | https://openalex.org/keywords/copying |
| keywords[3].score | 0.638887882232666 |
| keywords[3].display_name | Copying |
| keywords[4].id | https://openalex.org/keywords/leverage |
| keywords[4].score | 0.629676103591919 |
| keywords[4].display_name | Leverage (statistics) |
| keywords[5].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[5].score | 0.4767259657382965 |
| keywords[5].display_name | Artificial intelligence |
| keywords[6].id | https://openalex.org/keywords/dual |
| keywords[6].score | 0.46812617778778076 |
| keywords[6].display_name | Dual (grammatical number) |
| keywords[7].id | https://openalex.org/keywords/diffusion |
| keywords[7].score | 0.4629417955875397 |
| keywords[7].display_name | Diffusion |
| keywords[8].id | https://openalex.org/keywords/machine-learning |
| keywords[8].score | 0.34317922592163086 |
| keywords[8].display_name | Machine learning |
| keywords[9].id | https://openalex.org/keywords/psychology |
| keywords[9].score | 0.11735379695892334 |
| keywords[9].display_name | Psychology |
| keywords[10].id | https://openalex.org/keywords/mathematics |
| keywords[10].score | 0.10672011971473694 |
| keywords[10].display_name | Mathematics |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2309.07254 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2309.07254 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2309.07254 |
| locations[1].id | doi:10.48550/arxiv.2309.07254 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2309.07254 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5100756599 |
| authorships[0].author.orcid | https://orcid.org/0009-0003-0404-3825 |
| authorships[0].author.display_name | Chenghao Li |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Li, Chenghao |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5022810915 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-8193-7158 |
| authorships[1].author.display_name | Dake Chen |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Chen, Dake |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5019021341 |
| authorships[2].author.orcid | https://orcid.org/0000-0001-5253-5478 |
| authorships[2].author.display_name | Yuke Zhang |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Zhang, Yuke |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5084205024 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-8283-0168 |
| authorships[3].author.display_name | Peter A. Beerel |
| authorships[3].author_position | last |
| authorships[3].raw_author_name | Beerel, Peter A. |
| authorships[3].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2309.07254 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Mitigate Replication and Copying in Diffusion Models with Generalized Caption and Dual Fusion Enhancement |
| has_fulltext | True |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T11714 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.993399977684021 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1707 |
| primary_topic.subfield.display_name | Computer Vision and Pattern Recognition |
| primary_topic.display_name | Multimodal Machine Learning Applications |
| related_works | https://openalex.org/W2045049461, https://openalex.org/W1978893398, https://openalex.org/W2201908702, https://openalex.org/W4381094582, https://openalex.org/W4308771405, https://openalex.org/W2369625323, https://openalex.org/W2364579609, https://openalex.org/W1977906818, https://openalex.org/W2355873265, https://openalex.org/W1522139108 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2309.07254 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2309.07254 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2309.07254 |
| primary_location.id | pmh:oai:arXiv.org:2309.07254 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2309.07254 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2309.07254 |
| publication_date | 2023-09-13 |
| publication_year | 2023 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 4, 55, 80 |
| abstract_inverted_index.To | 47 |
| abstract_inverted_index.at | 124 |
| abstract_inverted_index.by | 105 |
| abstract_inverted_index.is | 122 |
| abstract_inverted_index.of | 33, 39, 90, 119 |
| abstract_inverted_index.to | 13, 69, 86, 108 |
| abstract_inverted_index.we | 74 |
| abstract_inverted_index.Our | 93 |
| abstract_inverted_index.and | 37, 63, 78, 117 |
| abstract_inverted_index.can | 101 |
| abstract_inverted_index.for | 7 |
| abstract_inverted_index.may | 27 |
| abstract_inverted_index.our | 51, 98 |
| abstract_inverted_index.the | 30, 60, 88, 109, 115 |
| abstract_inverted_index.Code | 121 |
| abstract_inverted_index.data | 16, 35 |
| abstract_inverted_index.dual | 82 |
| abstract_inverted_index.from | 29 |
| abstract_inverted_index.gap, | 50 |
| abstract_inverted_index.stem | 28 |
| abstract_inverted_index.that | 24, 58, 97 |
| abstract_inverted_index.this | 25, 49 |
| abstract_inverted_index.(LLM) | 68 |
| abstract_inverted_index.43.5% | 106 |
| abstract_inverted_index.While | 0 |
| abstract_inverted_index.first | 53 |
| abstract_inverted_index.large | 65 |
| abstract_inverted_index.model | 67, 112 |
| abstract_inverted_index.novel | 81 |
| abstract_inverted_index.paper | 52 |
| abstract_inverted_index.score | 57 |
| abstract_inverted_index.their | 11 |
| abstract_inverted_index.while | 113 |
| abstract_inverted_index.employ | 64 |
| abstract_inverted_index.fusion | 83 |
| abstract_inverted_index.models | 2 |
| abstract_inverted_index.raises | 17 |
| abstract_inverted_index.recent | 21 |
| abstract_inverted_index.reduce | 103 |
| abstract_inverted_index.remain | 45 |
| abstract_inverted_index.address | 48 |
| abstract_inverted_index.caption | 61 |
| abstract_inverted_index.images, | 10, 41 |
| abstract_inverted_index.methods | 100 |
| abstract_inverted_index.models. | 92 |
| abstract_inverted_index.privacy | 18 |
| abstract_inverted_index.propose | 79 |
| abstract_inverted_index.quality | 118 |
| abstract_inverted_index.results | 95 |
| abstract_inverted_index.Although | 20 |
| abstract_inverted_index.approach | 85 |
| abstract_inverted_index.captions | 36, 77 |
| abstract_inverted_index.compared | 107 |
| abstract_inverted_index.elusive. | 46 |
| abstract_inverted_index.language | 66 |
| abstract_inverted_index.leverage | 75 |
| abstract_inverted_index.measures | 59 |
| abstract_inverted_index.mitigate | 87 |
| abstract_inverted_index.original | 110 |
| abstract_inverted_index.proposed | 99 |
| abstract_inverted_index.research | 22 |
| abstract_inverted_index.suggests | 23 |
| abstract_inverted_index.tendency | 12 |
| abstract_inverted_index.training | 15, 34, 40, 71 |
| abstract_inverted_index.available | 123 |
| abstract_inverted_index.captions. | 72 |
| abstract_inverted_index.concerns. | 19 |
| abstract_inverted_index.diffusion | 1, 91, 111 |
| abstract_inverted_index.diversity | 116 |
| abstract_inverted_index.effective | 42 |
| abstract_inverted_index.empirical | 94 |
| abstract_inverted_index.capability | 6 |
| abstract_inverted_index.generality | 56, 62 |
| abstract_inverted_index.generalize | 70 |
| abstract_inverted_index.generating | 8 |
| abstract_inverted_index.introduces | 54 |
| abstract_inverted_index.mitigation | 43 |
| abstract_inverted_index.remarkable | 5 |
| abstract_inverted_index.strategies | 44 |
| abstract_inverted_index.`replicate' | 14 |
| abstract_inverted_index.demonstrate | 3, 96 |
| abstract_inverted_index.duplication | 38 |
| abstract_inverted_index.enhancement | 84 |
| abstract_inverted_index.generalized | 76 |
| abstract_inverted_index.maintaining | 114 |
| abstract_inverted_index.replication | 26, 89, 104 |
| abstract_inverted_index.generations. | 120 |
| abstract_inverted_index.high-quality | 9 |
| abstract_inverted_index.insufficient | 31 |
| abstract_inverted_index.Subsequently, | 73 |
| abstract_inverted_index.significantly | 102 |
| abstract_inverted_index.generalization | 32 |
| abstract_inverted_index.https://github.com/HowardLi0816/dual-fusion-diffusion. | 125 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 4 |
| citation_normalized_percentile |