Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations Article Swipe

PDF

Sangmin Lee , Bolin Lai , Fiona Ryan , Bikram Boote , James M. Rehg ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2403.02090

Understanding social interactions involving both verbal and non-verbal cues is essential for effectively interpreting social situations. However, most prior works on multimodal social cues focus predominantly on single-person behaviors or rely on holistic visual representations that are not aligned to utterances in multi-party environments. Consequently, they are limited in modeling the intricate dynamics of multi-party interactions. In this paper, we introduce three new challenging tasks to model the fine-grained dynamics between multiple people: speaking target identification, pronoun coreference resolution, and mentioned player prediction. We contribute extensive data annotations to curate these new challenges in social deduction game settings. Furthermore, we propose a novel multimodal baseline that leverages densely aligned language-visual representations by synchronizing visual features with their corresponding utterances. This facilitates concurrently capturing verbal and non-verbal cues pertinent to social reasoning. Experiments demonstrate the effectiveness of the proposed approach with densely aligned multimodal representations in modeling fine-grained social interactions. Project website: https://sangmin-git.github.io/projects/MMSI.

Related Topics

Computer Science

Data Science

Human–Computer Interaction

Concepts

Computer science Data science Human–computer interaction

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2403.02090
PDF: https://arxiv.org/pdf/2403.02090
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4392489370

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4392489370

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2403.02090

Digital Object Identifier
Title: Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-03-04

Full publication date if available
Authors: Sangmin Lee, Bolin Lai, Fiona Ryan, Bikram Boote, James M. Rehg

List of authors in order
Landing page: https://arxiv.org/abs/2403.02090

Publisher landing page
PDF URL: https://arxiv.org/pdf/2403.02090

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2403.02090

Direct OA link when available
Concepts: Computer science, Data science, Human–computer interaction

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4392489370
doi	https://doi.org/10.48550/arxiv.2403.02090
ids.doi	https://doi.org/10.48550/arxiv.2403.02090
ids.openalex	https://openalex.org/W4392489370
fwci
type	preprint
title	Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T12031
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.6600000262260437
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Speech and dialogue systems
topics[1].id	https://openalex.org/T14236
topics[1].field.id	https://openalex.org/fields/33
topics[1].field.display_name	Social Sciences
topics[1].score	0.5670999884605408
topics[1].domain.id	https://openalex.org/domains/2
topics[1].domain.display_name	Social Sciences
topics[1].subfield.id	https://openalex.org/subfields/3311
topics[1].subfield.display_name	Safety Research
topics[1].display_name	Educational Tools and Methods
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C41008148
concepts[0].level	0
concepts[0].score	0.4900475740432739
concepts[0].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[0].display_name	Computer science
concepts[1].id	https://openalex.org/C2522767166
concepts[1].level	1
concepts[1].score	0.43362635374069214
concepts[1].wikidata	https://www.wikidata.org/wiki/Q2374463
concepts[1].display_name	Data science
concepts[2].id	https://openalex.org/C107457646
concepts[2].level	1
concepts[2].score	0.34958672523498535
concepts[2].wikidata	https://www.wikidata.org/wiki/Q207434
concepts[2].display_name	Human–computer interaction
keywords[0].id	https://openalex.org/keywords/computer-science
keywords[0].score	0.4900475740432739
keywords[0].display_name	Computer science
keywords[1].id	https://openalex.org/keywords/data-science
keywords[1].score	0.43362635374069214
keywords[1].display_name	Data science
keywords[2].id	https://openalex.org/keywords/human–computer-interaction
keywords[2].score	0.34958672523498535
keywords[2].display_name	Human–computer interaction
language	en
locations[0].id	pmh:oai:arXiv.org:2403.02090
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2403.02090
locations[0].version	submittedVersion
locations[0].raw_type
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2403.02090
locations[1].id	doi:10.48550/arxiv.2403.02090
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2403.02090
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5100319624
authorships[0].author.orcid	https://orcid.org/0000-0002-8606-2549
authorships[0].author.display_name	Sangmin Lee
authorships[0].author_position	first
authorships[0].raw_author_name	Lee, Sangmin
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5037220775
authorships[1].author.orcid	https://orcid.org/0000-0001-7578-7336
authorships[1].author.display_name	Bolin Lai
authorships[1].author_position	middle
authorships[1].raw_author_name	Lai, Bolin
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5016641535
authorships[2].author.orcid
authorships[2].author.display_name	Fiona Ryan
authorships[2].author_position	middle
authorships[2].raw_author_name	Ryan, Fiona
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5052458015
authorships[3].author.orcid
authorships[3].author.display_name	Bikram Boote
authorships[3].author_position	middle
authorships[3].raw_author_name	Boote, Bikram
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5002228469
authorships[4].author.orcid	https://orcid.org/0000-0003-1793-5462
authorships[4].author.display_name	James M. Rehg
authorships[4].author_position	last
authorships[4].raw_author_name	Rehg, James M.
authorships[4].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2403.02090
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2024-03-06T00:00:00
display_name	Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T12031
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.6600000262260437
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Speech and dialogue systems
related_works	https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W2358668433, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W2382290278, https://openalex.org/W2478288626, https://openalex.org/W4391913857, https://openalex.org/W2350741829, https://openalex.org/W2530322880
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2403.02090
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2403.02090
best_oa_location.version	submittedVersion
best_oa_location.raw_type
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2403.02090
primary_location.id	pmh:oai:arXiv.org:2403.02090
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2403.02090
primary_location.version	submittedVersion
primary_location.raw_type
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2403.02090
publication_date	2024-03-04
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	101
abstract_inverted_index.In	56
abstract_inverted_index.We	83
abstract_inverted_index.by	111
abstract_inverted_index.in	41, 48, 93, 144
abstract_inverted_index.is	9
abstract_inverted_index.of	53, 135
abstract_inverted_index.on	20, 26, 31
abstract_inverted_index.or	29
abstract_inverted_index.to	39, 65, 88, 128
abstract_inverted_index.we	59, 99
abstract_inverted_index.and	6, 79, 124
abstract_inverted_index.are	36, 46
abstract_inverted_index.for	11
abstract_inverted_index.new	62, 91
abstract_inverted_index.not	37
abstract_inverted_index.the	50, 67, 133, 136
abstract_inverted_index.This	119
abstract_inverted_index.both	4
abstract_inverted_index.cues	8, 23, 126
abstract_inverted_index.data	86
abstract_inverted_index.game	96
abstract_inverted_index.most	17
abstract_inverted_index.rely	30
abstract_inverted_index.that	35, 105
abstract_inverted_index.they	45
abstract_inverted_index.this	57
abstract_inverted_index.with	115, 139
abstract_inverted_index.focus	24
abstract_inverted_index.model	66
abstract_inverted_index.novel	102
abstract_inverted_index.prior	18
abstract_inverted_index.tasks	64
abstract_inverted_index.their	116
abstract_inverted_index.these	90
abstract_inverted_index.three	61
abstract_inverted_index.works	19
abstract_inverted_index.curate	89
abstract_inverted_index.paper,	58
abstract_inverted_index.player	81
abstract_inverted_index.social	1, 14, 22, 94, 129, 147
abstract_inverted_index.target	74
abstract_inverted_index.verbal	5, 123
abstract_inverted_index.visual	33, 113
abstract_inverted_index.Project	149
abstract_inverted_index.aligned	38, 108, 141
abstract_inverted_index.between	70
abstract_inverted_index.densely	107, 140
abstract_inverted_index.limited	47
abstract_inverted_index.people:	72
abstract_inverted_index.pronoun	76
abstract_inverted_index.propose	100
abstract_inverted_index.However,	16
abstract_inverted_index.approach	138
abstract_inverted_index.baseline	104
abstract_inverted_index.dynamics	52, 69
abstract_inverted_index.features	114
abstract_inverted_index.holistic	32
abstract_inverted_index.modeling	49, 145
abstract_inverted_index.multiple	71
abstract_inverted_index.proposed	137
abstract_inverted_index.speaking	73
abstract_inverted_index.website:	150
abstract_inverted_index.behaviors	28
abstract_inverted_index.capturing	122
abstract_inverted_index.deduction	95
abstract_inverted_index.essential	10
abstract_inverted_index.extensive	85
abstract_inverted_index.intricate	51
abstract_inverted_index.introduce	60
abstract_inverted_index.involving	3
abstract_inverted_index.leverages	106
abstract_inverted_index.mentioned	80
abstract_inverted_index.pertinent	127
abstract_inverted_index.settings.	97
abstract_inverted_index.challenges	92
abstract_inverted_index.contribute	84
abstract_inverted_index.multimodal	21, 103, 142
abstract_inverted_index.non-verbal	7, 125
abstract_inverted_index.reasoning.	130
abstract_inverted_index.utterances	40
abstract_inverted_index.Experiments	131
abstract_inverted_index.annotations	87
abstract_inverted_index.challenging	63
abstract_inverted_index.coreference	77
abstract_inverted_index.demonstrate	132
abstract_inverted_index.effectively	12
abstract_inverted_index.facilitates	120
abstract_inverted_index.multi-party	42, 54
abstract_inverted_index.prediction.	82
abstract_inverted_index.resolution,	78
abstract_inverted_index.situations.	15
abstract_inverted_index.utterances.	118
abstract_inverted_index.Furthermore,	98
abstract_inverted_index.concurrently	121
abstract_inverted_index.fine-grained	68, 146
abstract_inverted_index.interactions	2
abstract_inverted_index.interpreting	13
abstract_inverted_index.Consequently,	44
abstract_inverted_index.Understanding	0
abstract_inverted_index.corresponding	117
abstract_inverted_index.effectiveness	134
abstract_inverted_index.environments.	43
abstract_inverted_index.interactions.	55, 148
abstract_inverted_index.predominantly	25
abstract_inverted_index.single-person	27
abstract_inverted_index.synchronizing	112
abstract_inverted_index.identification,	75
abstract_inverted_index.language-visual	109
abstract_inverted_index.representations	34, 110, 143
abstract_inverted_index.https://sangmin-git.github.io/projects/MMSI.	151
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	5
citation_normalized_percentile