4Dynamic: Text-to-4D Generation with Hybrid Priors Article Swipe
Yu-Jie Yuan
,
Leif Kobbelt
,
Jiwen Liu
,
Yuan Zhang
,
Pengfei Wan
,
Yu‐Kun Lai
,
Lin Gao
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2407.12684
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2407.12684
Due to the fascinating generative performance of text-to-image diffusion models, growing text-to-3D generation works explore distilling the 2D generative priors into 3D, using the score distillation sampling (SDS) loss, to bypass the data scarcity problem. The existing text-to-3D methods have achieved promising results in realism and 3D consistency, but text-to-4D generation still faces challenges, including lack of realism and insufficient dynamic motions. In this paper, we propose a novel method for text-to-4D generation, which ensures the dynamic amplitude and authenticity through direct supervision provided by a video prior. Specifically, we adopt a text-to-video diffusion model to generate a reference video and divide 4D generation into two stages: static generation and dynamic generation. The static 3D generation is achieved under the guidance of the input text and the first frame of the reference video, while in the dynamic generation stage, we introduce a customized SDS loss to ensure multi-view consistency, a video-based SDS loss to improve temporal consistency, and most importantly, direct priors from the reference video to ensure the quality of geometry and texture. Moreover, we design a prior-switching training strategy to avoid conflicts between different priors and fully leverage the benefits of each prior. In addition, to enrich the generated motion, we further introduce a dynamic modeling representation composed of a deformation network and a topology network, which ensures dynamic continuity while modeling topological changes. Our method not only supports text-to-4D generation but also enables 4D generation from monocular videos. The comparison experiments demonstrate the superiority of our method compared to existing methods.
Related Topics
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2407.12684
- https://arxiv.org/pdf/2407.12684
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4403781269
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4403781269Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2407.12684Digital Object Identifier
- Title
-
4Dynamic: Text-to-4D Generation with Hybrid PriorsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-07-17Full publication date if available
- Authors
-
Yu-Jie Yuan, Leif Kobbelt, Jiwen Liu, Yuan Zhang, Pengfei Wan, Yu‐Kun Lai, Lin GaoList of authors in order
- Landing page
-
https://arxiv.org/abs/2407.12684Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2407.12684Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2407.12684Direct OA link when available
- Concepts
-
Prior probability, Computer science, Artificial intelligence, Bayesian probabilityTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4403781269 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2407.12684 |
| ids.doi | https://doi.org/10.48550/arxiv.2407.12684 |
| ids.openalex | https://openalex.org/W4403781269 |
| fwci | |
| type | preprint |
| title | 4Dynamic: Text-to-4D Generation with Hybrid Priors |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T12720 |
| topics[0].field.id | https://openalex.org/fields/33 |
| topics[0].field.display_name | Social Sciences |
| topics[0].score | 0.8551999926567078 |
| topics[0].domain.id | https://openalex.org/domains/2 |
| topics[0].domain.display_name | Social Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/3312 |
| topics[0].subfield.display_name | Sociology and Political Science |
| topics[0].display_name | Multimedia Communication and Technology |
| topics[1].id | https://openalex.org/T11450 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.8309999704360962 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1712 |
| topics[1].subfield.display_name | Software |
| topics[1].display_name | Model-Driven Software Engineering Techniques |
| topics[2].id | https://openalex.org/T10181 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.7807999849319458 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Natural Language Processing Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C177769412 |
| concepts[0].level | 3 |
| concepts[0].score | 0.7843356728553772 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q278090 |
| concepts[0].display_name | Prior probability |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.458659827709198 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C154945302 |
| concepts[2].level | 1 |
| concepts[2].score | 0.3408341407775879 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[2].display_name | Artificial intelligence |
| concepts[3].id | https://openalex.org/C107673813 |
| concepts[3].level | 2 |
| concepts[3].score | 0.19655030965805054 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q812534 |
| concepts[3].display_name | Bayesian probability |
| keywords[0].id | https://openalex.org/keywords/prior-probability |
| keywords[0].score | 0.7843356728553772 |
| keywords[0].display_name | Prior probability |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.458659827709198 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[2].score | 0.3408341407775879 |
| keywords[2].display_name | Artificial intelligence |
| keywords[3].id | https://openalex.org/keywords/bayesian-probability |
| keywords[3].score | 0.19655030965805054 |
| keywords[3].display_name | Bayesian probability |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2407.12684 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2407.12684 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2407.12684 |
| locations[1].id | doi:10.48550/arxiv.2407.12684 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2407.12684 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5101893134 |
| authorships[0].author.orcid | https://orcid.org/0000-0001-5646-6211 |
| authorships[0].author.display_name | Yu-Jie Yuan |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Yuan, Yu-Jie |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5036300670 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-7880-9470 |
| authorships[1].author.display_name | Leif Kobbelt |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Kobbelt, Leif |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5072853454 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-3798-0714 |
| authorships[2].author.display_name | Jiwen Liu |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Liu, Jiwen |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5101663754 |
| authorships[3].author.orcid | https://orcid.org/0000-0001-7591-3996 |
| authorships[3].author.display_name | Yuan Zhang |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Zhang, Yuan |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5101928510 |
| authorships[4].author.orcid | https://orcid.org/0000-0001-7225-565X |
| authorships[4].author.display_name | Pengfei Wan |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Wan, Pengfei |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5067850699 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-2094-5680 |
| authorships[5].author.display_name | Yu‐Kun Lai |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Lai, Yu-Kun |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5100629376 |
| authorships[6].author.orcid | https://orcid.org/0000-0002-1021-8148 |
| authorships[6].author.display_name | Lin Gao |
| authorships[6].author_position | last |
| authorships[6].raw_author_name | Gao, Lin |
| authorships[6].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2407.12684 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | 4Dynamic: Text-to-4D Generation with Hybrid Priors |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T12720 |
| primary_topic.field.id | https://openalex.org/fields/33 |
| primary_topic.field.display_name | Social Sciences |
| primary_topic.score | 0.8551999926567078 |
| primary_topic.domain.id | https://openalex.org/domains/2 |
| primary_topic.domain.display_name | Social Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/3312 |
| primary_topic.subfield.display_name | Sociology and Political Science |
| primary_topic.display_name | Multimedia Communication and Technology |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2580650124, https://openalex.org/W4386190339, https://openalex.org/W2968424575, https://openalex.org/W3142333283, https://openalex.org/W3122088529, https://openalex.org/W3041320102, https://openalex.org/W2111669074 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2407.12684 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2407.12684 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2407.12684 |
| primary_location.id | pmh:oai:arXiv.org:2407.12684 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2407.12684 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2407.12684 |
| publication_date | 2024-07-17 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 67, 85, 91, 97, 141, 149, 177, 205, 211, 215 |
| abstract_inverted_index.2D | 17 |
| abstract_inverted_index.3D | 46, 114 |
| abstract_inverted_index.4D | 102, 236 |
| abstract_inverted_index.In | 62, 195 |
| abstract_inverted_index.by | 84 |
| abstract_inverted_index.in | 43, 134 |
| abstract_inverted_index.is | 116 |
| abstract_inverted_index.of | 6, 56, 121, 129, 170, 192, 210, 247 |
| abstract_inverted_index.to | 1, 29, 95, 145, 153, 166, 181, 197, 251 |
| abstract_inverted_index.we | 65, 89, 139, 175, 202 |
| abstract_inverted_index.3D, | 21 |
| abstract_inverted_index.Due | 0 |
| abstract_inverted_index.Our | 226 |
| abstract_inverted_index.SDS | 143, 151 |
| abstract_inverted_index.The | 35, 112, 241 |
| abstract_inverted_index.and | 45, 58, 78, 100, 109, 125, 157, 172, 187, 214 |
| abstract_inverted_index.but | 48, 233 |
| abstract_inverted_index.for | 70 |
| abstract_inverted_index.not | 228 |
| abstract_inverted_index.our | 248 |
| abstract_inverted_index.the | 2, 16, 23, 31, 75, 119, 122, 126, 130, 135, 163, 168, 190, 199, 245 |
| abstract_inverted_index.two | 105 |
| abstract_inverted_index.also | 234 |
| abstract_inverted_index.data | 32 |
| abstract_inverted_index.each | 193 |
| abstract_inverted_index.from | 162, 238 |
| abstract_inverted_index.have | 39 |
| abstract_inverted_index.into | 20, 104 |
| abstract_inverted_index.lack | 55 |
| abstract_inverted_index.loss | 144, 152 |
| abstract_inverted_index.most | 158 |
| abstract_inverted_index.only | 229 |
| abstract_inverted_index.text | 124 |
| abstract_inverted_index.this | 63 |
| abstract_inverted_index.(SDS) | 27 |
| abstract_inverted_index.adopt | 90 |
| abstract_inverted_index.avoid | 182 |
| abstract_inverted_index.faces | 52 |
| abstract_inverted_index.first | 127 |
| abstract_inverted_index.frame | 128 |
| abstract_inverted_index.fully | 188 |
| abstract_inverted_index.input | 123 |
| abstract_inverted_index.loss, | 28 |
| abstract_inverted_index.model | 94 |
| abstract_inverted_index.novel | 68 |
| abstract_inverted_index.score | 24 |
| abstract_inverted_index.still | 51 |
| abstract_inverted_index.under | 118 |
| abstract_inverted_index.using | 22 |
| abstract_inverted_index.video | 86, 99, 165 |
| abstract_inverted_index.which | 73, 218 |
| abstract_inverted_index.while | 133, 222 |
| abstract_inverted_index.works | 13 |
| abstract_inverted_index.bypass | 30 |
| abstract_inverted_index.design | 176 |
| abstract_inverted_index.direct | 81, 160 |
| abstract_inverted_index.divide | 101 |
| abstract_inverted_index.enrich | 198 |
| abstract_inverted_index.ensure | 146, 167 |
| abstract_inverted_index.method | 69, 227, 249 |
| abstract_inverted_index.paper, | 64 |
| abstract_inverted_index.prior. | 87, 194 |
| abstract_inverted_index.priors | 19, 161, 186 |
| abstract_inverted_index.stage, | 138 |
| abstract_inverted_index.static | 107, 113 |
| abstract_inverted_index.video, | 132 |
| abstract_inverted_index.between | 184 |
| abstract_inverted_index.dynamic | 60, 76, 110, 136, 206, 220 |
| abstract_inverted_index.enables | 235 |
| abstract_inverted_index.ensures | 74, 219 |
| abstract_inverted_index.explore | 14 |
| abstract_inverted_index.further | 203 |
| abstract_inverted_index.growing | 10 |
| abstract_inverted_index.improve | 154 |
| abstract_inverted_index.methods | 38 |
| abstract_inverted_index.models, | 9 |
| abstract_inverted_index.motion, | 201 |
| abstract_inverted_index.network | 213 |
| abstract_inverted_index.propose | 66 |
| abstract_inverted_index.quality | 169 |
| abstract_inverted_index.realism | 44, 57 |
| abstract_inverted_index.results | 42 |
| abstract_inverted_index.stages: | 106 |
| abstract_inverted_index.through | 80 |
| abstract_inverted_index.videos. | 240 |
| abstract_inverted_index.achieved | 40, 117 |
| abstract_inverted_index.benefits | 191 |
| abstract_inverted_index.changes. | 225 |
| abstract_inverted_index.compared | 250 |
| abstract_inverted_index.composed | 209 |
| abstract_inverted_index.existing | 36, 252 |
| abstract_inverted_index.generate | 96 |
| abstract_inverted_index.geometry | 171 |
| abstract_inverted_index.guidance | 120 |
| abstract_inverted_index.leverage | 189 |
| abstract_inverted_index.methods. | 253 |
| abstract_inverted_index.modeling | 207, 223 |
| abstract_inverted_index.motions. | 61 |
| abstract_inverted_index.network, | 217 |
| abstract_inverted_index.problem. | 34 |
| abstract_inverted_index.provided | 83 |
| abstract_inverted_index.sampling | 26 |
| abstract_inverted_index.scarcity | 33 |
| abstract_inverted_index.strategy | 180 |
| abstract_inverted_index.supports | 230 |
| abstract_inverted_index.temporal | 155 |
| abstract_inverted_index.texture. | 173 |
| abstract_inverted_index.topology | 216 |
| abstract_inverted_index.training | 179 |
| abstract_inverted_index.Moreover, | 174 |
| abstract_inverted_index.addition, | 196 |
| abstract_inverted_index.amplitude | 77 |
| abstract_inverted_index.conflicts | 183 |
| abstract_inverted_index.different | 185 |
| abstract_inverted_index.diffusion | 8, 93 |
| abstract_inverted_index.generated | 200 |
| abstract_inverted_index.including | 54 |
| abstract_inverted_index.introduce | 140, 204 |
| abstract_inverted_index.monocular | 239 |
| abstract_inverted_index.promising | 41 |
| abstract_inverted_index.reference | 98, 131, 164 |
| abstract_inverted_index.comparison | 242 |
| abstract_inverted_index.continuity | 221 |
| abstract_inverted_index.customized | 142 |
| abstract_inverted_index.distilling | 15 |
| abstract_inverted_index.generation | 12, 50, 103, 108, 115, 137, 232, 237 |
| abstract_inverted_index.generative | 4, 18 |
| abstract_inverted_index.multi-view | 147 |
| abstract_inverted_index.text-to-3D | 11, 37 |
| abstract_inverted_index.text-to-4D | 49, 71, 231 |
| abstract_inverted_index.challenges, | 53 |
| abstract_inverted_index.deformation | 212 |
| abstract_inverted_index.demonstrate | 244 |
| abstract_inverted_index.experiments | 243 |
| abstract_inverted_index.fascinating | 3 |
| abstract_inverted_index.generation, | 72 |
| abstract_inverted_index.generation. | 111 |
| abstract_inverted_index.performance | 5 |
| abstract_inverted_index.superiority | 246 |
| abstract_inverted_index.supervision | 82 |
| abstract_inverted_index.topological | 224 |
| abstract_inverted_index.video-based | 150 |
| abstract_inverted_index.authenticity | 79 |
| abstract_inverted_index.consistency, | 47, 148, 156 |
| abstract_inverted_index.distillation | 25 |
| abstract_inverted_index.importantly, | 159 |
| abstract_inverted_index.insufficient | 59 |
| abstract_inverted_index.Specifically, | 88 |
| abstract_inverted_index.text-to-image | 7 |
| abstract_inverted_index.text-to-video | 92 |
| abstract_inverted_index.representation | 208 |
| abstract_inverted_index.prior-switching | 178 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 7 |
| citation_normalized_percentile |