Reality3DSketch: Rapid 3D Modeling of Objects from Single Freehand Sketches Article Swipe

PDF

Tianrun Chen , Chaotao Ding , Lanyun Zhu , Ying Zang , Yiyi Liao , Zejian Li , Lingyun Sun ·

YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2310.18148

The emerging trend of AR/VR places great demands on 3D content. However, most existing software requires expertise and is difficult for novice users to use. In this paper, we aim to create sketch-based modeling tools for user-friendly 3D modeling. We introduce Reality3DSketch with a novel application of an immersive 3D modeling experience, in which a user can capture the surrounding scene using a monocular RGB camera and can draw a single sketch of an object in the real-time reconstructed 3D scene. A 3D object is generated and placed in the desired location, enabled by our novel neural network with the input of a single sketch. Our neural network can predict the pose of a drawing and can turn a single sketch into a 3D model with view and structural awareness, which addresses the challenge of sparse sketch input and view ambiguity. We conducted extensive experiments synthetic and real-world datasets and achieved state-of-the-art (SOTA) results in both sketch view estimation and 3D modeling performance. According to our user study, our method of performing 3D modeling in a scene is $>$5x faster than conventional methods. Users are also more satisfied with the generated 3D model than the results of existing methods.

Related Topics

Computer Science

Artificial Intelligence

Human–Computer Interaction

Concepts

Sketch Computer science Artificial intelligence Object (grammar) Computer vision 3D modeling Ambiguity Sketch recognition 3d model Monocular Artificial neural network RGB color model Computer graphics (images) Software Human–computer interaction Gesture recognition Algorithm Programming language Gesture

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2310.18148
PDF: https://arxiv.org/pdf/2310.18148
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4388067661

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4388067661

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2310.18148

Digital Object Identifier
Title: Reality3DSketch: Rapid 3D Modeling of Objects from Single Freehand Sketches

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2023

Year of publication
Publication date: 2023-10-27

Full publication date if available
Authors: Tianrun Chen, Chaotao Ding, Lanyun Zhu, Ying Zang, Yiyi Liao, Zejian Li, Lingyun Sun

List of authors in order
Landing page: https://arxiv.org/abs/2310.18148

Publisher landing page
PDF URL: https://arxiv.org/pdf/2310.18148

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2310.18148

Direct OA link when available
Concepts: Sketch, Computer science, Artificial intelligence, Object (grammar), Computer vision, 3D modeling, Ambiguity, Sketch recognition, 3d model, Monocular, Artificial neural network, RGB color model, Computer graphics (images), Software, Human–computer interaction, Gesture recognition, Algorithm, Programming language, Gesture

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4388067661
doi	https://doi.org/10.48550/arxiv.2310.18148
ids.doi	https://doi.org/10.48550/arxiv.2310.18148
ids.openalex	https://openalex.org/W4388067661
fwci
type	preprint
title	Reality3DSketch: Rapid 3D Modeling of Objects from Single Freehand Sketches
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10531
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9997000098228455
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1707
topics[0].subfield.display_name	Computer Vision and Pattern Recognition
topics[0].display_name	Advanced Vision and Imaging
topics[1].id	https://openalex.org/T10719
topics[1].field.id	https://openalex.org/fields/22
topics[1].field.display_name	Engineering
topics[1].score	0.9990000128746033
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/2206
topics[1].subfield.display_name	Computational Mechanics
topics[1].display_name	3D Shape Modeling and Analysis
topics[2].id	https://openalex.org/T10481
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.9986000061035156
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1704
topics[2].subfield.display_name	Computer Graphics and Computer-Aided Design
topics[2].display_name	Computer Graphics and Visualization Techniques
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C2779231336
concepts[0].level	2
concepts[0].score	0.9385045766830444
concepts[0].wikidata	https://www.wikidata.org/wiki/Q7534724
concepts[0].display_name	Sketch
concepts[1].id	https://openalex.org/C41008148
concepts[1].level	0
concepts[1].score	0.8354101181030273
concepts[1].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[1].display_name	Computer science
concepts[2].id	https://openalex.org/C154945302
concepts[2].level	1
concepts[2].score	0.624884307384491
concepts[2].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[2].display_name	Artificial intelligence
concepts[3].id	https://openalex.org/C2781238097
concepts[3].level	2
concepts[3].score	0.5994862914085388
concepts[3].wikidata	https://www.wikidata.org/wiki/Q175026
concepts[3].display_name	Object (grammar)
concepts[4].id	https://openalex.org/C31972630
concepts[4].level	1
concepts[4].score	0.5824952125549316
concepts[4].wikidata	https://www.wikidata.org/wiki/Q844240
concepts[4].display_name	Computer vision
concepts[5].id	https://openalex.org/C2777897806
concepts[5].level	2
concepts[5].score	0.5798995494842529
concepts[5].wikidata	https://www.wikidata.org/wiki/Q568742
concepts[5].display_name	3D modeling
concepts[6].id	https://openalex.org/C2780522230
concepts[6].level	2
concepts[6].score	0.5727363228797913
concepts[6].wikidata	https://www.wikidata.org/wiki/Q1140419
concepts[6].display_name	Ambiguity
concepts[7].id	https://openalex.org/C132900626
concepts[7].level	4
concepts[7].score	0.5042871236801147
concepts[7].wikidata	https://www.wikidata.org/wiki/Q7534733
concepts[7].display_name	Sketch recognition
concepts[8].id	https://openalex.org/C3019007443
concepts[8].level	2
concepts[8].score	0.4703139364719391
concepts[8].wikidata	https://www.wikidata.org/wiki/Q568742
concepts[8].display_name	3d model
concepts[9].id	https://openalex.org/C65909025
concepts[9].level	2
concepts[9].score	0.4687379002571106
concepts[9].wikidata	https://www.wikidata.org/wiki/Q1945033
concepts[9].display_name	Monocular
concepts[10].id	https://openalex.org/C50644808
concepts[10].level	2
concepts[10].score	0.4283602237701416
concepts[10].wikidata	https://www.wikidata.org/wiki/Q192776
concepts[10].display_name	Artificial neural network
concepts[11].id	https://openalex.org/C82990744
concepts[11].level	2
concepts[11].score	0.4271976351737976
concepts[11].wikidata	https://www.wikidata.org/wiki/Q166194
concepts[11].display_name	RGB color model
concepts[12].id	https://openalex.org/C121684516
concepts[12].level	1
concepts[12].score	0.422183096408844
concepts[12].wikidata	https://www.wikidata.org/wiki/Q7600677
concepts[12].display_name	Computer graphics (images)
concepts[13].id	https://openalex.org/C2777904410
concepts[13].level	2
concepts[13].score	0.4185483753681183
concepts[13].wikidata	https://www.wikidata.org/wiki/Q7397
concepts[13].display_name	Software
concepts[14].id	https://openalex.org/C107457646
concepts[14].level	1
concepts[14].score	0.3416725993156433
concepts[14].wikidata	https://www.wikidata.org/wiki/Q207434
concepts[14].display_name	Human–computer interaction
concepts[15].id	https://openalex.org/C159437735
concepts[15].level	3
concepts[15].score	0.0719384253025055
concepts[15].wikidata	https://www.wikidata.org/wiki/Q1519524
concepts[15].display_name	Gesture recognition
concepts[16].id	https://openalex.org/C11413529
concepts[16].level	1
concepts[16].score	0.0
concepts[16].wikidata	https://www.wikidata.org/wiki/Q8366
concepts[16].display_name	Algorithm
concepts[17].id	https://openalex.org/C199360897
concepts[17].level	1
concepts[17].score	0.0
concepts[17].wikidata	https://www.wikidata.org/wiki/Q9143
concepts[17].display_name	Programming language
concepts[18].id	https://openalex.org/C207347870
concepts[18].level	2
concepts[18].score	0.0
concepts[18].wikidata	https://www.wikidata.org/wiki/Q371174
concepts[18].display_name	Gesture
keywords[0].id	https://openalex.org/keywords/sketch
keywords[0].score	0.9385045766830444
keywords[0].display_name	Sketch
keywords[1].id	https://openalex.org/keywords/computer-science
keywords[1].score	0.8354101181030273
keywords[1].display_name	Computer science
keywords[2].id	https://openalex.org/keywords/artificial-intelligence
keywords[2].score	0.624884307384491
keywords[2].display_name	Artificial intelligence
keywords[3].id	https://openalex.org/keywords/object
keywords[3].score	0.5994862914085388
keywords[3].display_name	Object (grammar)
keywords[4].id	https://openalex.org/keywords/computer-vision
keywords[4].score	0.5824952125549316
keywords[4].display_name	Computer vision
keywords[5].id	https://openalex.org/keywords/3d-modeling
keywords[5].score	0.5798995494842529
keywords[5].display_name	3D modeling
keywords[6].id	https://openalex.org/keywords/ambiguity
keywords[6].score	0.5727363228797913
keywords[6].display_name	Ambiguity
keywords[7].id	https://openalex.org/keywords/sketch-recognition
keywords[7].score	0.5042871236801147
keywords[7].display_name	Sketch recognition
keywords[8].id	https://openalex.org/keywords/3d-model
keywords[8].score	0.4703139364719391
keywords[8].display_name	3d model
keywords[9].id	https://openalex.org/keywords/monocular
keywords[9].score	0.4687379002571106
keywords[9].display_name	Monocular
keywords[10].id	https://openalex.org/keywords/artificial-neural-network
keywords[10].score	0.4283602237701416
keywords[10].display_name	Artificial neural network
keywords[11].id	https://openalex.org/keywords/rgb-color-model
keywords[11].score	0.4271976351737976
keywords[11].display_name	RGB color model
keywords[12].id	https://openalex.org/keywords/computer-graphics
keywords[12].score	0.422183096408844
keywords[12].display_name	Computer graphics (images)
keywords[13].id	https://openalex.org/keywords/software
keywords[13].score	0.4185483753681183
keywords[13].display_name	Software
keywords[14].id	https://openalex.org/keywords/human–computer-interaction
keywords[14].score	0.3416725993156433
keywords[14].display_name	Human–computer interaction
keywords[15].id	https://openalex.org/keywords/gesture-recognition
keywords[15].score	0.0719384253025055
keywords[15].display_name	Gesture recognition
language	en
locations[0].id	pmh:oai:arXiv.org:2310.18148
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license	cc-by-nc-nd
locations[0].pdf_url	https://arxiv.org/pdf/2310.18148
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id	https://openalex.org/licenses/cc-by-nc-nd
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2310.18148
locations[1].id	doi:10.48550/arxiv.2310.18148
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2310.18148
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5054714748
authorships[0].author.orcid	https://orcid.org/0000-0003-0177-0157
authorships[0].author.display_name	Tianrun Chen
authorships[0].author_position	first
authorships[0].raw_author_name	Chen, Tianrun
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5035026498
authorships[1].author.orcid	https://orcid.org/0000-0002-6565-0640
authorships[1].author.display_name	Chaotao Ding
authorships[1].author_position	middle
authorships[1].raw_author_name	Ding, Chaotao
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5074956841
authorships[2].author.orcid	https://orcid.org/0000-0001-7309-3330
authorships[2].author.display_name	Lanyun Zhu
authorships[2].author_position	middle
authorships[2].raw_author_name	Zhu, Lanyun
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5057492564
authorships[3].author.orcid	https://orcid.org/0000-0002-1361-1500
authorships[3].author.display_name	Ying Zang
authorships[3].author_position	middle
authorships[3].raw_author_name	Zang, Ying
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5018811297
authorships[4].author.orcid	https://orcid.org/0000-0001-6662-3022
authorships[4].author.display_name	Yiyi Liao
authorships[4].author_position	middle
authorships[4].raw_author_name	Liao, Yiyi
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5015625724
authorships[5].author.orcid	https://orcid.org/0000-0001-5313-2742
authorships[5].author.display_name	Zejian Li
authorships[5].author_position	middle
authorships[5].raw_author_name	Li, Zejian
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5036387698
authorships[6].author.orcid	https://orcid.org/0000-0002-5561-0493
authorships[6].author.display_name	Lingyun Sun
authorships[6].author_position	last
authorships[6].raw_author_name	Sun, Lingyun
authorships[6].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2310.18148
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2023-11-01T00:00:00
display_name	Reality3DSketch: Rapid 3D Modeling of Objects from Single Freehand Sketches
has_fulltext	True
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10531
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9997000098228455
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1707
primary_topic.subfield.display_name	Computer Vision and Pattern Recognition
primary_topic.display_name	Advanced Vision and Imaging
related_works	https://openalex.org/W2708417971, https://openalex.org/W2314700393, https://openalex.org/W4384469254, https://openalex.org/W2563904947, https://openalex.org/W3116009112, https://openalex.org/W3036892807, https://openalex.org/W2995646374, https://openalex.org/W2983578059, https://openalex.org/W1970233280, https://openalex.org/W4244364093
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2310.18148
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license	cc-by-nc-nd
best_oa_location.pdf_url	https://arxiv.org/pdf/2310.18148
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id	https://openalex.org/licenses/cc-by-nc-nd
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2310.18148
primary_location.id	pmh:oai:arXiv.org:2310.18148
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license	cc-by-nc-nd
primary_location.pdf_url	https://arxiv.org/pdf/2310.18148
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id	https://openalex.org/licenses/cc-by-nc-nd
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2310.18148
publication_date	2023-10-27
publication_year	2023
referenced_works_count	0
abstract_inverted_index.A	81
abstract_inverted_index.a	43, 54, 62, 69, 102, 113, 118, 122, 175
abstract_inverted_index.3D	9, 37, 49, 79, 82, 123, 160, 172, 191
abstract_inverted_index.In	25
abstract_inverted_index.We	39, 141
abstract_inverted_index.an	47, 73
abstract_inverted_index.by	93
abstract_inverted_index.in	52, 75, 88, 154, 174
abstract_inverted_index.is	18, 84, 177
abstract_inverted_index.of	3, 46, 72, 101, 112, 134, 170, 196
abstract_inverted_index.on	8
abstract_inverted_index.to	23, 30, 164
abstract_inverted_index.we	28
abstract_inverted_index.Our	105
abstract_inverted_index.RGB	64
abstract_inverted_index.The	0
abstract_inverted_index.aim	29
abstract_inverted_index.and	17, 66, 86, 115, 127, 138, 146, 149, 159
abstract_inverted_index.are	184
abstract_inverted_index.can	56, 67, 108, 116
abstract_inverted_index.for	20, 35
abstract_inverted_index.our	94, 165, 168
abstract_inverted_index.the	58, 76, 89, 99, 110, 132, 189, 194
abstract_inverted_index.also	185
abstract_inverted_index.both	155
abstract_inverted_index.draw	68
abstract_inverted_index.into	121
abstract_inverted_index.more	186
abstract_inverted_index.most	12
abstract_inverted_index.pose	111
abstract_inverted_index.than	180, 193
abstract_inverted_index.this	26
abstract_inverted_index.turn	117
abstract_inverted_index.use.	24
abstract_inverted_index.user	55, 166
abstract_inverted_index.view	126, 139, 157
abstract_inverted_index.with	42, 98, 125, 188
abstract_inverted_index.AR/VR	4
abstract_inverted_index.Users	183
abstract_inverted_index.great	6
abstract_inverted_index.input	100, 137
abstract_inverted_index.model	124, 192
abstract_inverted_index.novel	44, 95
abstract_inverted_index.scene	60, 176
abstract_inverted_index.tools	34
abstract_inverted_index.trend	2
abstract_inverted_index.users	22
abstract_inverted_index.using	61
abstract_inverted_index.which	53, 130
abstract_inverted_index.(SOTA)	152
abstract_inverted_index.camera	65
abstract_inverted_index.create	31
abstract_inverted_index.faster	179
abstract_inverted_index.method	169
abstract_inverted_index.neural	96, 106
abstract_inverted_index.novice	21
abstract_inverted_index.object	74, 83
abstract_inverted_index.paper,	27
abstract_inverted_index.placed	87
abstract_inverted_index.places	5
abstract_inverted_index.scene.	80
abstract_inverted_index.single	70, 103, 119
abstract_inverted_index.sketch	71, 120, 136, 156
abstract_inverted_index.sparse	135
abstract_inverted_index.study,	167
abstract_inverted_index.capture	57
abstract_inverted_index.demands	7
abstract_inverted_index.desired	90
abstract_inverted_index.drawing	114
abstract_inverted_index.enabled	92
abstract_inverted_index.network	97, 107
abstract_inverted_index.predict	109
abstract_inverted_index.results	153, 195
abstract_inverted_index.sketch.	104
abstract_inverted_index.$>$5x	178
abstract_inverted_index.However,	11
abstract_inverted_index.achieved	150
abstract_inverted_index.content.	10
abstract_inverted_index.datasets	148
abstract_inverted_index.emerging	1
abstract_inverted_index.existing	13, 197
abstract_inverted_index.methods.	182, 198
abstract_inverted_index.modeling	33, 50, 161, 173
abstract_inverted_index.requires	15
abstract_inverted_index.software	14
abstract_inverted_index.According	163
abstract_inverted_index.addresses	131
abstract_inverted_index.challenge	133
abstract_inverted_index.conducted	142
abstract_inverted_index.difficult	19
abstract_inverted_index.expertise	16
abstract_inverted_index.extensive	143
abstract_inverted_index.generated	85, 190
abstract_inverted_index.immersive	48
abstract_inverted_index.introduce	40
abstract_inverted_index.location,	91
abstract_inverted_index.modeling.	38
abstract_inverted_index.monocular	63
abstract_inverted_index.real-time	77
abstract_inverted_index.satisfied	187
abstract_inverted_index.synthetic	145
abstract_inverted_index.ambiguity.	140
abstract_inverted_index.awareness,	129
abstract_inverted_index.estimation	158
abstract_inverted_index.performing	171
abstract_inverted_index.real-world	147
abstract_inverted_index.structural	128
abstract_inverted_index.application	45
abstract_inverted_index.experience,	51
abstract_inverted_index.experiments	144
abstract_inverted_index.surrounding	59
abstract_inverted_index.conventional	181
abstract_inverted_index.performance.	162
abstract_inverted_index.sketch-based	32
abstract_inverted_index.reconstructed	78
abstract_inverted_index.user-friendly	36
abstract_inverted_index.Reality3DSketch	41
abstract_inverted_index.state-of-the-art	151
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	7
citation_normalized_percentile