Log-Linear Attention Article Swipe

PDF

Han Guo , Songlin Yang , Tek Chand Goel , Eric P. Xing , Tri Dao , Yoon-Ji Kim ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2506.04761

The attention mechanism in Transformers is an important primitive for accurate and scalable sequence modeling. Its quadratic-compute and linear-memory complexity however remain significant bottlenecks. Linear attention and state-space models enable linear-time, constant-memory sequence modeling and can moreover be trained efficiently through matmul-rich parallelization across sequence length. However, at their core these models are still RNNs, and thus their use of a fixed-size hidden state to model the context is a fundamental limitation. This paper develops log-linear attention, an attention mechanism that balances linear attention's efficiency and the expressiveness of softmax attention. Log-linear attention replaces the fixed-size hidden state with a logarithmically growing set of hidden states. We show that with a particular growth function, log-linear attention admits a similarly matmul-rich parallel form whose compute cost is log-linear in sequence length. Log-linear attention is a general framework and can be applied on top of existing linear attention variants. As case studies, we instantiate log-linear variants of two recent architectures -- Mamba-2 and Gated DeltaNet -- and find they perform well compared to their linear-time variants.

Related Topics

Transformers (Film Series)

Transformers (Film)

Antikythera Mechanism

Attention Is All You Need

Transformers

The Transformers (Tv Series)

Attention (Machine Learning)

Transformers (Skybound Entertainment)

Bumblebee (Transformers)

Very Important People (2023 Tv Series)

Attention

Higgs Mechanism

Soundwave (Transformers)

The Mechanism (Tv Series)

Primitive Technology

Mechanism (Engineering)

Toroidal Inductors And Transformers

Mechanism Of Action Of Aspirin

Crank (Mechanism)

Concepts

No concepts available.

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2506.04761
PDF: https://arxiv.org/pdf/2506.04761
OA Status: green
OpenAlex ID: https://openalex.org/W4416076333

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4416076333

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2506.04761

Digital Object Identifier
Title: Log-Linear Attention

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-06-05

Full publication date if available
Authors: Han Guo, Songlin Yang, Tek Chand Goel, Eric P. Xing, Tri Dao, Yoon-Ji Kim

List of authors in order
Landing page: https://arxiv.org/abs/2506.04761

Publisher landing page
PDF URL: https://arxiv.org/pdf/2506.04761

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2506.04761

Direct OA link when available
Cited by: 0

Total citation count in OpenAlex

Full payload

id	https://openalex.org/W4416076333
doi	https://doi.org/10.48550/arxiv.2506.04761
ids.doi	https://doi.org/10.48550/arxiv.2506.04761
ids.openalex	https://openalex.org/W4416076333
fwci
type	preprint
title	Log-Linear Attention
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
is_xpac	False
apc_list
apc_paid
language	en
locations[0].id	pmh:oai:arXiv.org:2506.04761
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2506.04761
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2506.04761
locations[1].id	doi:10.48550/arxiv.2506.04761
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2506.04761
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5103156134
authorships[0].author.orcid	https://orcid.org/0009-0007-5615-3566
authorships[0].author.display_name	Han Guo
authorships[0].author_position	first
authorships[0].raw_author_name	Guo, Han
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5100541272
authorships[1].author.orcid
authorships[1].author.display_name	Songlin Yang
authorships[1].author_position	middle
authorships[1].raw_author_name	Yang, Songlin
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5026425927
authorships[2].author.orcid
authorships[2].author.display_name	Tek Chand Goel
authorships[2].author_position	middle
authorships[2].raw_author_name	Goel, Tarushii
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5009547049
authorships[3].author.orcid	https://orcid.org/0009-0005-9158-4201
authorships[3].author.display_name	Eric P. Xing
authorships[3].author_position	middle
authorships[3].raw_author_name	Xing, Eric P.
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5091734792
authorships[4].author.orcid
authorships[4].author.display_name	Tri Dao
authorships[4].author_position	middle
authorships[4].raw_author_name	Dao, Tri
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5028473988
authorships[5].author.orcid	https://orcid.org/0000-0002-8137-8915
authorships[5].author.display_name	Yoon-Ji Kim
authorships[5].author_position	last
authorships[5].raw_author_name	Kim, Yoon
authorships[5].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2506.04761
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Log-Linear Attention
has_fulltext	False
is_retracted	False
updated_date	2025-11-28T09:51:18.120405
primary_topic
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2506.04761
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2506.04761
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2506.04761
primary_location.id	pmh:oai:arXiv.org:2506.04761
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2506.04761
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2506.04761
publication_date	2025-06-05
publication_year	2025
referenced_works_count	0
abstract_inverted_index.a	60, 69, 99, 110, 117, 133
abstract_inverted_index.--	158, 163
abstract_inverted_index.As	147
abstract_inverted_index.We	106
abstract_inverted_index.an	6, 77
abstract_inverted_index.at	47
abstract_inverted_index.be	37, 138
abstract_inverted_index.in	3, 127
abstract_inverted_index.is	5, 68, 125, 132
abstract_inverted_index.of	59, 88, 103, 142, 154
abstract_inverted_index.on	140
abstract_inverted_index.to	64, 170
abstract_inverted_index.we	150
abstract_inverted_index.Its	15
abstract_inverted_index.The	0
abstract_inverted_index.and	11, 17, 26, 34, 55, 85, 136, 160, 164
abstract_inverted_index.are	52
abstract_inverted_index.can	35, 137
abstract_inverted_index.for	9
abstract_inverted_index.set	102
abstract_inverted_index.the	66, 86, 94
abstract_inverted_index.top	141
abstract_inverted_index.two	155
abstract_inverted_index.use	58
abstract_inverted_index.This	72
abstract_inverted_index.case	148
abstract_inverted_index.core	49
abstract_inverted_index.cost	124
abstract_inverted_index.find	165
abstract_inverted_index.form	121
abstract_inverted_index.show	107
abstract_inverted_index.that	80, 108
abstract_inverted_index.they	166
abstract_inverted_index.thus	56
abstract_inverted_index.well	168
abstract_inverted_index.with	98, 109
abstract_inverted_index.Gated	161
abstract_inverted_index.RNNs,	54
abstract_inverted_index.model	65
abstract_inverted_index.paper	73
abstract_inverted_index.state	63, 97
abstract_inverted_index.still	53
abstract_inverted_index.their	48, 57, 171
abstract_inverted_index.these	50
abstract_inverted_index.whose	122
abstract_inverted_index.Linear	24
abstract_inverted_index.across	43
abstract_inverted_index.admits	116
abstract_inverted_index.enable	29
abstract_inverted_index.growth	112
abstract_inverted_index.hidden	62, 96, 104
abstract_inverted_index.linear	82, 144
abstract_inverted_index.models	28, 51
abstract_inverted_index.recent	156
abstract_inverted_index.remain	21
abstract_inverted_index.Mamba-2	159
abstract_inverted_index.applied	139
abstract_inverted_index.compute	123
abstract_inverted_index.context	67
abstract_inverted_index.general	134
abstract_inverted_index.growing	101
abstract_inverted_index.however	20
abstract_inverted_index.length.	45, 129
abstract_inverted_index.perform	167
abstract_inverted_index.softmax	89
abstract_inverted_index.states.	105
abstract_inverted_index.through	40
abstract_inverted_index.trained	38
abstract_inverted_index.DeltaNet	162
abstract_inverted_index.However,	46
abstract_inverted_index.accurate	10
abstract_inverted_index.balances	81
abstract_inverted_index.compared	169
abstract_inverted_index.develops	74
abstract_inverted_index.existing	143
abstract_inverted_index.modeling	33
abstract_inverted_index.moreover	36
abstract_inverted_index.parallel	120
abstract_inverted_index.replaces	93
abstract_inverted_index.scalable	12
abstract_inverted_index.sequence	13, 32, 44, 128
abstract_inverted_index.studies,	149
abstract_inverted_index.variants	153
abstract_inverted_index.attention	1, 25, 78, 92, 115, 131, 145
abstract_inverted_index.framework	135
abstract_inverted_index.function,	113
abstract_inverted_index.important	7
abstract_inverted_index.mechanism	2, 79
abstract_inverted_index.modeling.	14
abstract_inverted_index.primitive	8
abstract_inverted_index.similarly	118
abstract_inverted_index.variants.	146, 173
abstract_inverted_index.Log-linear	91, 130
abstract_inverted_index.attention,	76
abstract_inverted_index.attention.	90
abstract_inverted_index.complexity	19
abstract_inverted_index.efficiency	84
abstract_inverted_index.fixed-size	61, 95
abstract_inverted_index.log-linear	75, 114, 126, 152
abstract_inverted_index.particular	111
abstract_inverted_index.attention's	83
abstract_inverted_index.efficiently	39
abstract_inverted_index.fundamental	70
abstract_inverted_index.instantiate	151
abstract_inverted_index.limitation.	71
abstract_inverted_index.linear-time	172
abstract_inverted_index.matmul-rich	41, 119
abstract_inverted_index.significant	22
abstract_inverted_index.state-space	27
abstract_inverted_index.Transformers	4
abstract_inverted_index.bottlenecks.	23
abstract_inverted_index.linear-time,	30
abstract_inverted_index.architectures	157
abstract_inverted_index.linear-memory	18
abstract_inverted_index.expressiveness	87
abstract_inverted_index.constant-memory	31
abstract_inverted_index.logarithmically	100
abstract_inverted_index.parallelization	42
abstract_inverted_index.quadratic-compute	16
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	6
citation_normalized_percentile