SCoRD: Subject-Conditional Relation Detection with Text-Augmented Data Article Swipe

PDF

Ziyan Yang , Kushal Kafle , Zhe Lin , Scott Cohen , Zhihong Ding , Vicente Ordóñez ·

YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2308.12910

We propose Subject-Conditional Relation Detection SCoRD, where conditioned on an input subject, the goal is to predict all its relations to other objects in a scene along with their locations. Based on the Open Images dataset, we propose a challenging OIv6-SCoRD benchmark such that the training and testing splits have a distribution shift in terms of the occurrence statistics of $\langle$subject, relation, object$\rangle$ triplets. To solve this problem, we propose an auto-regressive model that given a subject, it predicts its relations, objects, and object locations by casting this output as a sequence of tokens. First, we show that previous scene-graph prediction methods fail to produce as exhaustive an enumeration of relation-object pairs when conditioned on a subject on this benchmark. Particularly, we obtain a recall@3 of 83.8% for our relation-object predictions compared to the 49.75% obtained by a recent scene graph detector. Then, we show improved generalization on both relation-object and object-box predictions by leveraging during training relation-object pairs obtained automatically from textual captions and for which no object-box annotations are available. Particularly, for $\langle$subject, relation, object$\rangle$ triplets for which no object locations are available during training, we are able to obtain a recall@3 of 33.80% for relation-object pairs and 26.75% for their box locations.

Related Topics

Computer Science

Generalization

Scene Graph

Benchmark (Surveying)

Artificial Intelligence

Object Detection

Theoretical Computer Science

Mathematics

Data Mining

Geodesy

Mathematical Analysis

Rendering (Computer Graphics)

Library Science

Geography

Concepts

Relation (database) Object (grammar) Subject (documents) Computer science Generalization Scene graph Benchmark (surveying) Artificial intelligence Graph Object detection Pattern recognition (psychology) Theoretical computer science Mathematics Data mining Geodesy Mathematical analysis Rendering (computer graphics) Library science Geography

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2308.12910
PDF: https://arxiv.org/pdf/2308.12910
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4386185526

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4386185526

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2308.12910

Digital Object Identifier
Title: SCoRD: Subject-Conditional Relation Detection with Text-Augmented Data

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2023

Year of publication
Publication date: 2023-08-24

Full publication date if available
Authors: Ziyan Yang, Kushal Kafle, Zhe Lin, Scott Cohen, Zhihong Ding, Vicente Ordóñez

List of authors in order
Landing page: https://arxiv.org/abs/2308.12910

Publisher landing page
PDF URL: https://arxiv.org/pdf/2308.12910

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2308.12910

Direct OA link when available
Concepts: Relation (database), Object (grammar), Subject (documents), Computer science, Generalization, Scene graph, Benchmark (surveying), Artificial intelligence, Graph, Object detection, Pattern recognition (psychology), Theoretical computer science, Mathematics, Data mining, Geodesy, Mathematical analysis, Rendering (computer graphics), Library science, Geography

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4386185526
doi	https://doi.org/10.48550/arxiv.2308.12910
ids.doi	https://doi.org/10.48550/arxiv.2308.12910
ids.openalex	https://openalex.org/W4386185526
fwci
type	preprint
title	SCoRD: Subject-Conditional Relation Detection with Text-Augmented Data
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T11714
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9998999834060669
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1707
topics[0].subfield.display_name	Computer Vision and Pattern Recognition
topics[0].display_name	Multimodal Machine Learning Applications
topics[1].id	https://openalex.org/T10028
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9797999858856201
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1702
topics[1].subfield.display_name	Artificial Intelligence
topics[1].display_name	Topic Modeling
topics[2].id	https://openalex.org/T11307
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.979200005531311
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1702
topics[2].subfield.display_name	Artificial Intelligence
topics[2].display_name	Domain Adaptation and Few-Shot Learning
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C25343380
concepts[0].level	2
concepts[0].score	0.7832013964653015
concepts[0].wikidata	https://www.wikidata.org/wiki/Q277521
concepts[0].display_name	Relation (database)
concepts[1].id	https://openalex.org/C2781238097
concepts[1].level	2
concepts[1].score	0.7618929743766785
concepts[1].wikidata	https://www.wikidata.org/wiki/Q175026
concepts[1].display_name	Object (grammar)
concepts[2].id	https://openalex.org/C2777855551
concepts[2].level	2
concepts[2].score	0.6262668371200562
concepts[2].wikidata	https://www.wikidata.org/wiki/Q12310021
concepts[2].display_name	Subject (documents)
concepts[3].id	https://openalex.org/C41008148
concepts[3].level	0
concepts[3].score	0.5829926133155823
concepts[3].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[3].display_name	Computer science
concepts[4].id	https://openalex.org/C177148314
concepts[4].level	2
concepts[4].score	0.5740597248077393
concepts[4].wikidata	https://www.wikidata.org/wiki/Q170084
concepts[4].display_name	Generalization
concepts[5].id	https://openalex.org/C179372163
concepts[5].level	3
concepts[5].score	0.5721027851104736
concepts[5].wikidata	https://www.wikidata.org/wiki/Q1406181
concepts[5].display_name	Scene graph
concepts[6].id	https://openalex.org/C185798385
concepts[6].level	2
concepts[6].score	0.5432863831520081
concepts[6].wikidata	https://www.wikidata.org/wiki/Q1161707
concepts[6].display_name	Benchmark (surveying)
concepts[7].id	https://openalex.org/C154945302
concepts[7].level	1
concepts[7].score	0.5353347063064575
concepts[7].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[7].display_name	Artificial intelligence
concepts[8].id	https://openalex.org/C132525143
concepts[8].level	2
concepts[8].score	0.5209782719612122
concepts[8].wikidata	https://www.wikidata.org/wiki/Q141488
concepts[8].display_name	Graph
concepts[9].id	https://openalex.org/C2776151529
concepts[9].level	3
concepts[9].score	0.42557492852211
concepts[9].wikidata	https://www.wikidata.org/wiki/Q3045304
concepts[9].display_name	Object detection
concepts[10].id	https://openalex.org/C153180895
concepts[10].level	2
concepts[10].score	0.36265239119529724
concepts[10].wikidata	https://www.wikidata.org/wiki/Q7148389
concepts[10].display_name	Pattern recognition (psychology)
concepts[11].id	https://openalex.org/C80444323
concepts[11].level	1
concepts[11].score	0.33612996339797974
concepts[11].wikidata	https://www.wikidata.org/wiki/Q2878974
concepts[11].display_name	Theoretical computer science
concepts[12].id	https://openalex.org/C33923547
concepts[12].level	0
concepts[12].score	0.3244864344596863
concepts[12].wikidata	https://www.wikidata.org/wiki/Q395
concepts[12].display_name	Mathematics
concepts[13].id	https://openalex.org/C124101348
concepts[13].level	1
concepts[13].score	0.30093032121658325
concepts[13].wikidata	https://www.wikidata.org/wiki/Q172491
concepts[13].display_name	Data mining
concepts[14].id	https://openalex.org/C13280743
concepts[14].level	1
concepts[14].score	0.0
concepts[14].wikidata	https://www.wikidata.org/wiki/Q131089
concepts[14].display_name	Geodesy
concepts[15].id	https://openalex.org/C134306372
concepts[15].level	1
concepts[15].score	0.0
concepts[15].wikidata	https://www.wikidata.org/wiki/Q7754
concepts[15].display_name	Mathematical analysis
concepts[16].id	https://openalex.org/C205711294
concepts[16].level	2
concepts[16].score	0.0
concepts[16].wikidata	https://www.wikidata.org/wiki/Q176953
concepts[16].display_name	Rendering (computer graphics)
concepts[17].id	https://openalex.org/C161191863
concepts[17].level	1
concepts[17].score	0.0
concepts[17].wikidata	https://www.wikidata.org/wiki/Q199655
concepts[17].display_name	Library science
concepts[18].id	https://openalex.org/C205649164
concepts[18].level	0
concepts[18].score	0.0
concepts[18].wikidata	https://www.wikidata.org/wiki/Q1071
concepts[18].display_name	Geography
keywords[0].id	https://openalex.org/keywords/relation
keywords[0].score	0.7832013964653015
keywords[0].display_name	Relation (database)
keywords[1].id	https://openalex.org/keywords/object
keywords[1].score	0.7618929743766785
keywords[1].display_name	Object (grammar)
keywords[2].id	https://openalex.org/keywords/subject
keywords[2].score	0.6262668371200562
keywords[2].display_name	Subject (documents)
keywords[3].id	https://openalex.org/keywords/computer-science
keywords[3].score	0.5829926133155823
keywords[3].display_name	Computer science
keywords[4].id	https://openalex.org/keywords/generalization
keywords[4].score	0.5740597248077393
keywords[4].display_name	Generalization
keywords[5].id	https://openalex.org/keywords/scene-graph
keywords[5].score	0.5721027851104736
keywords[5].display_name	Scene graph
keywords[6].id	https://openalex.org/keywords/benchmark
keywords[6].score	0.5432863831520081
keywords[6].display_name	Benchmark (surveying)
keywords[7].id	https://openalex.org/keywords/artificial-intelligence
keywords[7].score	0.5353347063064575
keywords[7].display_name	Artificial intelligence
keywords[8].id	https://openalex.org/keywords/graph
keywords[8].score	0.5209782719612122
keywords[8].display_name	Graph
keywords[9].id	https://openalex.org/keywords/object-detection
keywords[9].score	0.42557492852211
keywords[9].display_name	Object detection
keywords[10].id	https://openalex.org/keywords/pattern-recognition
keywords[10].score	0.36265239119529724
keywords[10].display_name	Pattern recognition (psychology)
keywords[11].id	https://openalex.org/keywords/theoretical-computer-science
keywords[11].score	0.33612996339797974
keywords[11].display_name	Theoretical computer science
keywords[12].id	https://openalex.org/keywords/mathematics
keywords[12].score	0.3244864344596863
keywords[12].display_name	Mathematics
keywords[13].id	https://openalex.org/keywords/data-mining
keywords[13].score	0.30093032121658325
keywords[13].display_name	Data mining
language	en
locations[0].id	pmh:oai:arXiv.org:2308.12910
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2308.12910
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2308.12910
locations[1].id	doi:10.48550/arxiv.2308.12910
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2308.12910
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5069660758
authorships[0].author.orcid	https://orcid.org/0000-0002-9030-2074
authorships[0].author.display_name	Ziyan Yang
authorships[0].author_position	first
authorships[0].raw_author_name	Yang, Ziyan
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5047193437
authorships[1].author.orcid	https://orcid.org/0000-0002-0847-7861
authorships[1].author.display_name	Kushal Kafle
authorships[1].author_position	middle
authorships[1].raw_author_name	Kafle, Kushal
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5101564904
authorships[2].author.orcid	https://orcid.org/0000-0003-1154-9907
authorships[2].author.display_name	Zhe Lin
authorships[2].author_position	middle
authorships[2].raw_author_name	Lin, Zhe
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5055156362
authorships[3].author.orcid	https://orcid.org/0000-0002-3459-6899
authorships[3].author.display_name	Scott Cohen
authorships[3].author_position	middle
authorships[3].raw_author_name	Cohen, Scott
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5079713811
authorships[4].author.orcid	https://orcid.org/0009-0001-1088-3260
authorships[4].author.display_name	Zhihong Ding
authorships[4].author_position	middle
authorships[4].raw_author_name	Ding, Zhihong
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5027328044
authorships[5].author.orcid
authorships[5].author.display_name	Vicente Ordóñez
authorships[5].author_position	last
authorships[5].raw_author_name	Ordonez, Vicente
authorships[5].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2308.12910
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	SCoRD: Subject-Conditional Relation Detection with Text-Augmented Data
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T11714
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9998999834060669
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1707
primary_topic.subfield.display_name	Computer Vision and Pattern Recognition
primary_topic.display_name	Multimodal Machine Learning Applications
related_works	https://openalex.org/W2378211422, https://openalex.org/W2745001401, https://openalex.org/W4321353415, https://openalex.org/W2130974462, https://openalex.org/W2028665553, https://openalex.org/W2086519370, https://openalex.org/W972276598, https://openalex.org/W4287991909, https://openalex.org/W4390721878, https://openalex.org/W4387129494
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2308.12910
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2308.12910
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2308.12910
primary_location.id	pmh:oai:arXiv.org:2308.12910
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2308.12910
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2308.12910
publication_date	2023-08-24
publication_year	2023
referenced_works_count	0
abstract_inverted_index.a	24, 38, 50, 75, 90, 115, 123, 137, 192
abstract_inverted_index.To	64
abstract_inverted_index.We	0
abstract_inverted_index.an	9, 70, 107
abstract_inverted_index.as	89, 105
abstract_inverted_index.by	85, 136, 153
abstract_inverted_index.in	23, 53
abstract_inverted_index.is	14
abstract_inverted_index.it	77
abstract_inverted_index.no	167, 180
abstract_inverted_index.of	55, 59, 92, 109, 125, 194
abstract_inverted_index.on	8, 31, 114, 117, 147
abstract_inverted_index.to	15, 20, 103, 132, 190
abstract_inverted_index.we	36, 68, 95, 121, 143, 187
abstract_inverted_index.all	17
abstract_inverted_index.and	46, 82, 150, 164, 199
abstract_inverted_index.are	170, 183, 188
abstract_inverted_index.box	203
abstract_inverted_index.for	127, 165, 173, 178, 196, 201
abstract_inverted_index.its	18, 79
abstract_inverted_index.our	128
abstract_inverted_index.the	12, 32, 44, 56, 133
abstract_inverted_index.Open	33
abstract_inverted_index.able	189
abstract_inverted_index.both	148
abstract_inverted_index.fail	102
abstract_inverted_index.from	161
abstract_inverted_index.goal	13
abstract_inverted_index.have	49
abstract_inverted_index.show	96, 144
abstract_inverted_index.such	42
abstract_inverted_index.that	43, 73, 97
abstract_inverted_index.this	66, 87, 118
abstract_inverted_index.when	112
abstract_inverted_index.with	27
abstract_inverted_index.83.8%	126
abstract_inverted_index.Based	30
abstract_inverted_index.Then,	142
abstract_inverted_index.along	26
abstract_inverted_index.given	74
abstract_inverted_index.graph	140
abstract_inverted_index.input	10
abstract_inverted_index.model	72
abstract_inverted_index.other	21
abstract_inverted_index.pairs	111, 158, 198
abstract_inverted_index.scene	25, 139
abstract_inverted_index.shift	52
abstract_inverted_index.solve	65
abstract_inverted_index.terms	54
abstract_inverted_index.their	28, 202
abstract_inverted_index.where	6
abstract_inverted_index.which	166, 179
abstract_inverted_index.26.75%	200
abstract_inverted_index.33.80%	195
abstract_inverted_index.49.75%	134
abstract_inverted_index.First,	94
abstract_inverted_index.Images	34
abstract_inverted_index.SCoRD,	5
abstract_inverted_index.during	155, 185
abstract_inverted_index.object	83, 181
abstract_inverted_index.obtain	122, 191
abstract_inverted_index.output	88
abstract_inverted_index.recent	138
abstract_inverted_index.splits	48
abstract_inverted_index.casting	86
abstract_inverted_index.methods	101
abstract_inverted_index.objects	22
abstract_inverted_index.predict	16
abstract_inverted_index.produce	104
abstract_inverted_index.propose	1, 37, 69
abstract_inverted_index.subject	116
abstract_inverted_index.testing	47
abstract_inverted_index.textual	162
abstract_inverted_index.tokens.	93
abstract_inverted_index.Relation	3
abstract_inverted_index.captions	163
abstract_inverted_index.compared	131
abstract_inverted_index.dataset,	35
abstract_inverted_index.improved	145
abstract_inverted_index.objects,	81
abstract_inverted_index.obtained	135, 159
abstract_inverted_index.predicts	78
abstract_inverted_index.previous	98
abstract_inverted_index.problem,	67
abstract_inverted_index.recall@3	124, 193
abstract_inverted_index.sequence	91
abstract_inverted_index.subject,	11, 76
abstract_inverted_index.training	45, 156
abstract_inverted_index.triplets	177
abstract_inverted_index.Detection	4
abstract_inverted_index.available	184
abstract_inverted_index.benchmark	41
abstract_inverted_index.detector.	141
abstract_inverted_index.locations	84, 182
abstract_inverted_index.relation,	61, 175
abstract_inverted_index.relations	19
abstract_inverted_index.training,	186
abstract_inverted_index.triplets.	63
abstract_inverted_index.OIv6-SCoRD	40
abstract_inverted_index.available.	171
abstract_inverted_index.benchmark.	119
abstract_inverted_index.exhaustive	106
abstract_inverted_index.leveraging	154
abstract_inverted_index.locations.	29, 204
abstract_inverted_index.object-box	151, 168
abstract_inverted_index.occurrence	57
abstract_inverted_index.prediction	100
abstract_inverted_index.relations,	80
abstract_inverted_index.statistics	58
abstract_inverted_index.annotations	169
abstract_inverted_index.challenging	39
abstract_inverted_index.conditioned	7, 113
abstract_inverted_index.enumeration	108
abstract_inverted_index.predictions	130, 152
abstract_inverted_index.scene-graph	99
abstract_inverted_index.distribution	51
abstract_inverted_index.Particularly,	120, 172
abstract_inverted_index.automatically	160
abstract_inverted_index.generalization	146
abstract_inverted_index.auto-regressive	71
abstract_inverted_index.object$\rangle$	62, 176
abstract_inverted_index.relation-object	110, 129, 149, 157, 197
abstract_inverted_index.$\langle$subject,	60, 174
abstract_inverted_index.Subject-Conditional	2
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	6
citation_normalized_percentile