DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models Article Swipe

PDF

Linli Yao , Lei Li , Shuhuai Ren , Lean Wang , Yuanxin Liu , Xu Sun , Lu Hou ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2405.20985

The visual projector, which bridges the vision and language modalities and facilitates cross-modal alignment, serves as a crucial component in MLLMs. However, measuring the effectiveness of projectors in vision-language alignment remains under-explored, which currently can only be inferred from the performance of MLLMs on downstream tasks. Motivated by the problem, this study examines the projector module by interpreting the vision-language semantic flow within MLLMs. Specifically, we trace back the semantic relevance flow from generated language tokens to raw visual encoder patches and the intermediate outputs produced by projectors. Our findings reveal that compressive projectors (e.g., QFormer), abstract visual patches into a limited set of semantic concepts, such as objects or attributes, resulting in a 'double abstraction' phenomenon. This involves a first visual semantic abstraction by the projector referring to pre-defined query tokens, and a second extraction by the LLM based on text instructions. The double abstraction is inefficient in training and will result in cumulative vision semantics deficiency. To mitigate this issue, we propose the key insight of 'Decouple Compression from Abstraction (DeCo), that is compressing the visual token number at the patch level by projectors and allowing the LLM to handle visual semantic abstraction entirely. Consequently, we adopt a simple compressor, i.e., 2D Adaptive Pooling, to downsample visual patches in a parameter-free manner. Empirical evaluation demonstrates that DeCo surpasses traditional compressive projectors regarding both performance and efficiency. It achieves performance gains of 0.9%, 7.1%, and 2.9% across the MLLM Benchmarks, Visual Localization, and Open-ended VQA tasks with fewer trainable parameters and faster convergence speed.

Related Topics

Concepts

Computer science Abstraction Decoupling (probability) Security token Natural language processing Computer network Philosophy Engineering Control engineering Epistemology

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2405.20985
PDF: https://arxiv.org/pdf/2405.20985
OA Status: green
Cited By: 1
Related Works: 10
OpenAlex ID: https://openalex.org/W4399317923

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4399317923

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2405.20985

Digital Object Identifier
Title: DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-05-31

Full publication date if available
Authors: Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, Lu Hou

List of authors in order
Landing page: https://arxiv.org/abs/2405.20985

Publisher landing page
PDF URL: https://arxiv.org/pdf/2405.20985

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2405.20985

Direct OA link when available
Concepts: Computer science, Abstraction, Decoupling (probability), Security token, Natural language processing, Computer network, Philosophy, Engineering, Control engineering, Epistemology

Top concepts (fields/topics) attached by OpenAlex
Cited by: 1

Total citation count in OpenAlex
Citations by year (recent): 2024: 1

Per-year citation counts (last 5 years)
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4399317923
doi	https://doi.org/10.48550/arxiv.2405.20985
ids.doi	https://doi.org/10.48550/arxiv.2405.20985
ids.openalex	https://openalex.org/W4399317923
fwci
type	preprint
title	DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10028
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9707000255584717
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Topic Modeling
topics[1].id	https://openalex.org/T10181
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9682999849319458
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1702
topics[1].subfield.display_name	Artificial Intelligence
topics[1].display_name	Natural Language Processing Techniques
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C41008148
concepts[0].level	0
concepts[0].score	0.7054464817047119
concepts[0].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[0].display_name	Computer science
concepts[1].id	https://openalex.org/C124304363
concepts[1].level	2
concepts[1].score	0.5923159122467041
concepts[1].wikidata	https://www.wikidata.org/wiki/Q673661
concepts[1].display_name	Abstraction
concepts[2].id	https://openalex.org/C205606062
concepts[2].level	2
concepts[2].score	0.5734971761703491
concepts[2].wikidata	https://www.wikidata.org/wiki/Q5249645
concepts[2].display_name	Decoupling (probability)
concepts[3].id	https://openalex.org/C48145219
concepts[3].level	2
concepts[3].score	0.5248311161994934
concepts[3].wikidata	https://www.wikidata.org/wiki/Q1335365
concepts[3].display_name	Security token
concepts[4].id	https://openalex.org/C204321447
concepts[4].level	1
concepts[4].score	0.33683663606643677
concepts[4].wikidata	https://www.wikidata.org/wiki/Q30642
concepts[4].display_name	Natural language processing
concepts[5].id	https://openalex.org/C31258907
concepts[5].level	1
concepts[5].score	0.09804937243461609
concepts[5].wikidata	https://www.wikidata.org/wiki/Q1301371
concepts[5].display_name	Computer network
concepts[6].id	https://openalex.org/C138885662
concepts[6].level	0
concepts[6].score	0.0813058614730835
concepts[6].wikidata	https://www.wikidata.org/wiki/Q5891
concepts[6].display_name	Philosophy
concepts[7].id	https://openalex.org/C127413603
concepts[7].level	0
concepts[7].score	0.06183898448944092
concepts[7].wikidata	https://www.wikidata.org/wiki/Q11023
concepts[7].display_name	Engineering
concepts[8].id	https://openalex.org/C133731056
concepts[8].level	1
concepts[8].score	0.044816941022872925
concepts[8].wikidata	https://www.wikidata.org/wiki/Q4917288
concepts[8].display_name	Control engineering
concepts[9].id	https://openalex.org/C111472728
concepts[9].level	1
concepts[9].score	0.0
concepts[9].wikidata	https://www.wikidata.org/wiki/Q9471
concepts[9].display_name	Epistemology
keywords[0].id	https://openalex.org/keywords/computer-science
keywords[0].score	0.7054464817047119
keywords[0].display_name	Computer science
keywords[1].id	https://openalex.org/keywords/abstraction
keywords[1].score	0.5923159122467041
keywords[1].display_name	Abstraction
keywords[2].id	https://openalex.org/keywords/decoupling
keywords[2].score	0.5734971761703491
keywords[2].display_name	Decoupling (probability)
keywords[3].id	https://openalex.org/keywords/security-token
keywords[3].score	0.5248311161994934
keywords[3].display_name	Security token
keywords[4].id	https://openalex.org/keywords/natural-language-processing
keywords[4].score	0.33683663606643677
keywords[4].display_name	Natural language processing
keywords[5].id	https://openalex.org/keywords/computer-network
keywords[5].score	0.09804937243461609
keywords[5].display_name	Computer network
keywords[6].id	https://openalex.org/keywords/philosophy
keywords[6].score	0.0813058614730835
keywords[6].display_name	Philosophy
keywords[7].id	https://openalex.org/keywords/engineering
keywords[7].score	0.06183898448944092
keywords[7].display_name	Engineering
keywords[8].id	https://openalex.org/keywords/control-engineering
keywords[8].score	0.044816941022872925
keywords[8].display_name	Control engineering
language	en
locations[0].id	pmh:oai:arXiv.org:2405.20985
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2405.20985
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2405.20985
locations[1].id	doi:10.48550/arxiv.2405.20985
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2405.20985
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5035307582
authorships[0].author.orcid
authorships[0].author.display_name	Linli Yao
authorships[0].author_position	first
authorships[0].raw_author_name	Yao, Linli
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5100658983
authorships[1].author.orcid	https://orcid.org/0000-0001-5062-3538
authorships[1].author.display_name	Lei Li
authorships[1].author_position	middle
authorships[1].raw_author_name	Li, Lei
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5101030723
authorships[2].author.orcid	https://orcid.org/0009-0001-9998-864X
authorships[2].author.display_name	Shuhuai Ren
authorships[2].author_position	middle
authorships[2].raw_author_name	Ren, Shuhuai
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5034611488
authorships[3].author.orcid
authorships[3].author.display_name	Lean Wang
authorships[3].author_position	middle
authorships[3].raw_author_name	Wang, Lean
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5044541656
authorships[4].author.orcid	https://orcid.org/0000-0002-0673-4887
authorships[4].author.display_name	Yuanxin Liu
authorships[4].author_position	middle
authorships[4].raw_author_name	Liu, Yuanxin
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5113863473
authorships[5].author.orcid
authorships[5].author.display_name	Xu Sun
authorships[5].author_position	middle
authorships[5].raw_author_name	Sun, Xu
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5100963876
authorships[6].author.orcid
authorships[6].author.display_name	Lu Hou
authorships[6].author_position	last
authorships[6].raw_author_name	Hou, Lu
authorships[6].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2405.20985
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2024-06-04T00:00:00
display_name	DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10028
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9707000255584717
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Topic Modeling
related_works	https://openalex.org/W4391375266, https://openalex.org/W2748952813, https://openalex.org/W4388335561, https://openalex.org/W2970530566, https://openalex.org/W4288261899, https://openalex.org/W4307309205, https://openalex.org/W2967478618, https://openalex.org/W4385009901, https://openalex.org/W4385572700, https://openalex.org/W2997152889
cited_by_count	1
counts_by_year[0].year	2024
counts_by_year[0].cited_by_count	1
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2405.20985
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2405.20985
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2405.20985
primary_location.id	pmh:oai:arXiv.org:2405.20985
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2405.20985
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2405.20985
publication_date	2024-05-31
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	16, 100, 113, 119, 133, 199, 211
abstract_inverted_index.2D	203
abstract_inverted_index.It	228
abstract_inverted_index.To	158
abstract_inverted_index.as	15, 107
abstract_inverted_index.at	180
abstract_inverted_index.be	36
abstract_inverted_index.by	47, 56, 86, 124, 136, 184
abstract_inverted_index.in	19, 27, 112, 148, 153, 210
abstract_inverted_index.is	146, 174
abstract_inverted_index.of	25, 41, 103, 167, 232
abstract_inverted_index.on	43, 140
abstract_inverted_index.or	109
abstract_inverted_index.to	76, 128, 190, 206
abstract_inverted_index.we	65, 162, 197
abstract_inverted_index.LLM	138, 189
abstract_inverted_index.Our	88
abstract_inverted_index.The	0, 143
abstract_inverted_index.VQA	245
abstract_inverted_index.and	7, 10, 81, 132, 150, 186, 226, 235, 243, 251
abstract_inverted_index.can	34
abstract_inverted_index.key	165
abstract_inverted_index.raw	77
abstract_inverted_index.set	102
abstract_inverted_index.the	5, 23, 39, 48, 53, 58, 68, 82, 125, 137, 164, 176, 181, 188, 238
abstract_inverted_index.2.9%	236
abstract_inverted_index.DeCo	218
abstract_inverted_index.MLLM	239
abstract_inverted_index.This	117
abstract_inverted_index.back	67
abstract_inverted_index.both	224
abstract_inverted_index.flow	61, 71
abstract_inverted_index.from	38, 72, 170
abstract_inverted_index.into	99
abstract_inverted_index.only	35
abstract_inverted_index.such	106
abstract_inverted_index.text	141
abstract_inverted_index.that	91, 173, 217
abstract_inverted_index.this	50, 160
abstract_inverted_index.will	151
abstract_inverted_index.with	247
abstract_inverted_index.0.9%,	233
abstract_inverted_index.7.1%,	234
abstract_inverted_index.MLLMs	42
abstract_inverted_index.adopt	198
abstract_inverted_index.based	139
abstract_inverted_index.fewer	248
abstract_inverted_index.first	120
abstract_inverted_index.gains	231
abstract_inverted_index.i.e.,	202
abstract_inverted_index.level	183
abstract_inverted_index.patch	182
abstract_inverted_index.query	130
abstract_inverted_index.study	51
abstract_inverted_index.tasks	246
abstract_inverted_index.token	178
abstract_inverted_index.trace	66
abstract_inverted_index.which	3, 32
abstract_inverted_index.(e.g.,	94
abstract_inverted_index.MLLMs.	20, 63
abstract_inverted_index.Visual	241
abstract_inverted_index.across	237
abstract_inverted_index.double	144
abstract_inverted_index.faster	252
abstract_inverted_index.handle	191
abstract_inverted_index.issue,	161
abstract_inverted_index.module	55
abstract_inverted_index.number	179
abstract_inverted_index.result	152
abstract_inverted_index.reveal	90
abstract_inverted_index.second	134
abstract_inverted_index.serves	14
abstract_inverted_index.simple	200
abstract_inverted_index.speed.	254
abstract_inverted_index.tasks.	45
abstract_inverted_index.tokens	75
abstract_inverted_index.vision	6, 155
abstract_inverted_index.visual	1, 78, 97, 121, 177, 192, 208
abstract_inverted_index.within	62
abstract_inverted_index.'double	114
abstract_inverted_index.(DeCo),	172
abstract_inverted_index.bridges	4
abstract_inverted_index.crucial	17
abstract_inverted_index.encoder	79
abstract_inverted_index.insight	166
abstract_inverted_index.limited	101
abstract_inverted_index.manner.	213
abstract_inverted_index.objects	108
abstract_inverted_index.outputs	84
abstract_inverted_index.patches	80, 98, 209
abstract_inverted_index.propose	163
abstract_inverted_index.remains	30
abstract_inverted_index.tokens,	131
abstract_inverted_index.Adaptive	204
abstract_inverted_index.However,	21
abstract_inverted_index.Pooling,	205
abstract_inverted_index.abstract	96
abstract_inverted_index.achieves	229
abstract_inverted_index.allowing	187
abstract_inverted_index.examines	52
abstract_inverted_index.findings	89
abstract_inverted_index.inferred	37
abstract_inverted_index.involves	118
abstract_inverted_index.language	8, 74
abstract_inverted_index.mitigate	159
abstract_inverted_index.problem,	49
abstract_inverted_index.produced	85
abstract_inverted_index.semantic	60, 69, 104, 122, 193
abstract_inverted_index.training	149
abstract_inverted_index.'Decouple	168
abstract_inverted_index.Empirical	214
abstract_inverted_index.Motivated	46
abstract_inverted_index.QFormer),	95
abstract_inverted_index.alignment	29
abstract_inverted_index.component	18
abstract_inverted_index.concepts,	105
abstract_inverted_index.currently	33
abstract_inverted_index.entirely.	195
abstract_inverted_index.generated	73
abstract_inverted_index.measuring	22
abstract_inverted_index.projector	54, 126
abstract_inverted_index.referring	127
abstract_inverted_index.regarding	223
abstract_inverted_index.relevance	70
abstract_inverted_index.resulting	111
abstract_inverted_index.semantics	156
abstract_inverted_index.surpasses	219
abstract_inverted_index.trainable	249
abstract_inverted_index.Open-ended	244
abstract_inverted_index.alignment,	13
abstract_inverted_index.cumulative	154
abstract_inverted_index.downsample	207
abstract_inverted_index.downstream	44
abstract_inverted_index.evaluation	215
abstract_inverted_index.extraction	135
abstract_inverted_index.modalities	9
abstract_inverted_index.parameters	250
abstract_inverted_index.projector,	2
abstract_inverted_index.projectors	26, 93, 185, 222
abstract_inverted_index.Abstraction	171
abstract_inverted_index.Benchmarks,	240
abstract_inverted_index.Compression	169
abstract_inverted_index.abstraction	123, 145, 194
abstract_inverted_index.attributes,	110
abstract_inverted_index.compressing	175
abstract_inverted_index.compressive	92, 221
abstract_inverted_index.compressor,	201
abstract_inverted_index.convergence	253
abstract_inverted_index.cross-modal	12
abstract_inverted_index.deficiency.	157
abstract_inverted_index.efficiency.	227
abstract_inverted_index.facilitates	11
abstract_inverted_index.inefficient	147
abstract_inverted_index.performance	40, 225, 230
abstract_inverted_index.phenomenon.	116
abstract_inverted_index.pre-defined	129
abstract_inverted_index.projectors.	87
abstract_inverted_index.traditional	220
abstract_inverted_index.abstraction'	115
abstract_inverted_index.demonstrates	216
abstract_inverted_index.intermediate	83
abstract_inverted_index.interpreting	57
abstract_inverted_index.Consequently,	196
abstract_inverted_index.Localization,	242
abstract_inverted_index.Specifically,	64
abstract_inverted_index.effectiveness	24
abstract_inverted_index.instructions.	142
abstract_inverted_index.parameter-free	212
abstract_inverted_index.under-explored,	31
abstract_inverted_index.vision-language	28, 59
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	7
citation_normalized_percentile