Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention Article Swipe
Jingyang Yuan
,
Huazuo Gao
,
Damai Dai
,
Junyu Luo
,
Liang Zhao
,
Zhengyan Zhang
,
Zhenda Xie
,
Yunxie Wei
,
Lean Wang
,
Zhiping Xiao
,
Yuqing Wang
,
Chong Ruan
,
Ming Zhang
,
Wenfeng Liang
,
Wangding Zeng
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2502.11089
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2502.11089
Long-context modeling is crucial for next-generation language models, yet the high computational cost of standard attention mechanisms poses significant computational challenges. Sparse attention offers a promising direction for improving efficiency while maintaining model capabilities. We present NSA, a Natively trainable Sparse Attention mechanism that integrates algorithmic innovations with hardware-aligned optimizations to achieve efficient long-context modeling. NSA employs a dynamic hierarchical sparse strategy, combining coarse-grained token compression with fine-grained token selection to preserve both global context awareness and local precision. Our approach advances sparse attention design with two key innovations: (1) We achieve substantial speedups through arithmetic intensity-balanced algorithm design, with implementation optimizations for modern hardware. (2) We enable end-to-end training, reducing pretraining computation without sacrificing model performance. As shown in Figure 1, experiments show the model pretrained with NSA maintains or exceeds Full Attention models across general benchmarks, long-context tasks, and instruction-based reasoning. Meanwhile, NSA achieves substantial speedups over Full Attention on 64k-length sequences across decoding, forward propagation, and backward propagation, validating its efficiency throughout the model lifecycle.
Related Topics
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2502.11089
- https://arxiv.org/pdf/2502.11089
- OA Status
- green
- Cited By
- 2
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4407690159
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4407690159Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2502.11089Digital Object Identifier
- Title
-
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse AttentionWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-02-16Full publication date if available
- Authors
-
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yunxie Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, Wangding ZengList of authors in order
- Landing page
-
https://arxiv.org/abs/2502.11089Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2502.11089Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2502.11089Direct OA link when available
- Concepts
-
Computer science, Sparse approximation, Computer hardware, Artificial intelligenceTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
2Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 2Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4407690159 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2502.11089 |
| ids.doi | https://doi.org/10.48550/arxiv.2502.11089 |
| ids.openalex | https://openalex.org/W4407690159 |
| fwci | |
| type | preprint |
| title | Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10036 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9193999767303467 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1707 |
| topics[0].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[0].display_name | Advanced Neural Network Applications |
| topics[1].id | https://openalex.org/T11612 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9178000092506409 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Stochastic Gradient Optimization Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.5940757393836975 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C124066611 |
| concepts[1].level | 2 |
| concepts[1].score | 0.42390161752700806 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q28684319 |
| concepts[1].display_name | Sparse approximation |
| concepts[2].id | https://openalex.org/C9390403 |
| concepts[2].level | 1 |
| concepts[2].score | 0.34762340784072876 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q3966 |
| concepts[2].display_name | Computer hardware |
| concepts[3].id | https://openalex.org/C154945302 |
| concepts[3].level | 1 |
| concepts[3].score | 0.34322071075439453 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[3].display_name | Artificial intelligence |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.5940757393836975 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/sparse-approximation |
| keywords[1].score | 0.42390161752700806 |
| keywords[1].display_name | Sparse approximation |
| keywords[2].id | https://openalex.org/keywords/computer-hardware |
| keywords[2].score | 0.34762340784072876 |
| keywords[2].display_name | Computer hardware |
| keywords[3].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[3].score | 0.34322071075439453 |
| keywords[3].display_name | Artificial intelligence |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2502.11089 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2502.11089 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2502.11089 |
| locations[1].id | doi:10.48550/arxiv.2502.11089 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2502.11089 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5113747152 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Jingyang Yuan |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Yuan, Jingyang |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5102602497 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Huazuo Gao |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Gao, Huazuo |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5020456783 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Damai Dai |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Dai, Damai |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5101912906 |
| authorships[3].author.orcid | https://orcid.org/0009-0001-6894-1144 |
| authorships[3].author.display_name | Junyu Luo |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Luo, Junyu |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5100528484 |
| authorships[4].author.orcid | https://orcid.org/0000-0002-5522-8630 |
| authorships[4].author.display_name | Liang Zhao |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Zhao, Liang |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5111176285 |
| authorships[5].author.orcid | |
| authorships[5].author.display_name | Zhengyan Zhang |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Zhang, Zhengyan |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5112985153 |
| authorships[6].author.orcid | |
| authorships[6].author.display_name | Zhenda Xie |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Xie, Zhenda |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5068079517 |
| authorships[7].author.orcid | https://orcid.org/0000-0001-8218-4437 |
| authorships[7].author.display_name | Yunxie Wei |
| authorships[7].author_position | middle |
| authorships[7].raw_author_name | Wei, Y. X. |
| authorships[7].is_corresponding | False |
| authorships[8].author.id | https://openalex.org/A5034611488 |
| authorships[8].author.orcid | |
| authorships[8].author.display_name | Lean Wang |
| authorships[8].author_position | middle |
| authorships[8].raw_author_name | Wang, Lean |
| authorships[8].is_corresponding | False |
| authorships[9].author.id | https://openalex.org/A5102016849 |
| authorships[9].author.orcid | https://orcid.org/0000-0002-3165-6163 |
| authorships[9].author.display_name | Zhiping Xiao |
| authorships[9].author_position | middle |
| authorships[9].raw_author_name | Xiao, Zhiping |
| authorships[9].is_corresponding | False |
| authorships[10].author.id | https://openalex.org/A5100461350 |
| authorships[10].author.orcid | https://orcid.org/0009-0006-3161-0364 |
| authorships[10].author.display_name | Yuqing Wang |
| authorships[10].author_position | middle |
| authorships[10].raw_author_name | Wang, Yuqing |
| authorships[10].is_corresponding | False |
| authorships[11].author.id | https://openalex.org/A5016017027 |
| authorships[11].author.orcid | |
| authorships[11].author.display_name | Chong Ruan |
| authorships[11].author_position | middle |
| authorships[11].raw_author_name | Ruan, Chong |
| authorships[11].is_corresponding | False |
| authorships[12].author.id | https://openalex.org/A5100717983 |
| authorships[12].author.orcid | https://orcid.org/0009-0000-2631-1622 |
| authorships[12].author.display_name | Ming Zhang |
| authorships[12].author_position | middle |
| authorships[12].raw_author_name | Zhang, Ming |
| authorships[12].is_corresponding | False |
| authorships[13].author.id | https://openalex.org/A5110193145 |
| authorships[13].author.orcid | |
| authorships[13].author.display_name | Wenfeng Liang |
| authorships[13].author_position | middle |
| authorships[13].raw_author_name | Liang, Wenfeng |
| authorships[13].is_corresponding | False |
| authorships[14].author.id | https://openalex.org/A5049532529 |
| authorships[14].author.orcid | |
| authorships[14].author.display_name | Wangding Zeng |
| authorships[14].author_position | last |
| authorships[14].raw_author_name | Zeng, Wangding |
| authorships[14].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2502.11089 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10036 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9193999767303467 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1707 |
| primary_topic.subfield.display_name | Computer Vision and Pattern Recognition |
| primary_topic.display_name | Advanced Neural Network Applications |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W4391913857, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W4396696052 |
| cited_by_count | 2 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 2 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2502.11089 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2502.11089 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2502.11089 |
| primary_location.id | pmh:oai:arXiv.org:2502.11089 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2502.11089 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2502.11089 |
| publication_date | 2025-02-16 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 24, 37, 57 |
| abstract_inverted_index.1, | 121 |
| abstract_inverted_index.As | 117 |
| abstract_inverted_index.We | 34, 90, 106 |
| abstract_inverted_index.in | 119 |
| abstract_inverted_index.is | 2 |
| abstract_inverted_index.of | 13 |
| abstract_inverted_index.on | 151 |
| abstract_inverted_index.or | 130 |
| abstract_inverted_index.to | 50, 70 |
| abstract_inverted_index.(1) | 89 |
| abstract_inverted_index.(2) | 105 |
| abstract_inverted_index.NSA | 55, 128, 144 |
| abstract_inverted_index.Our | 79 |
| abstract_inverted_index.and | 76, 140, 158 |
| abstract_inverted_index.for | 4, 27, 102 |
| abstract_inverted_index.its | 162 |
| abstract_inverted_index.key | 87 |
| abstract_inverted_index.the | 9, 124, 165 |
| abstract_inverted_index.two | 86 |
| abstract_inverted_index.yet | 8 |
| abstract_inverted_index.Full | 132, 149 |
| abstract_inverted_index.NSA, | 36 |
| abstract_inverted_index.both | 72 |
| abstract_inverted_index.cost | 12 |
| abstract_inverted_index.high | 10 |
| abstract_inverted_index.over | 148 |
| abstract_inverted_index.show | 123 |
| abstract_inverted_index.that | 43 |
| abstract_inverted_index.with | 47, 66, 85, 99, 127 |
| abstract_inverted_index.local | 77 |
| abstract_inverted_index.model | 32, 115, 125, 166 |
| abstract_inverted_index.poses | 17 |
| abstract_inverted_index.shown | 118 |
| abstract_inverted_index.token | 64, 68 |
| abstract_inverted_index.while | 30 |
| abstract_inverted_index.Figure | 120 |
| abstract_inverted_index.Sparse | 21, 40 |
| abstract_inverted_index.across | 135, 154 |
| abstract_inverted_index.design | 84 |
| abstract_inverted_index.enable | 107 |
| abstract_inverted_index.global | 73 |
| abstract_inverted_index.models | 134 |
| abstract_inverted_index.modern | 103 |
| abstract_inverted_index.offers | 23 |
| abstract_inverted_index.sparse | 60, 82 |
| abstract_inverted_index.tasks, | 139 |
| abstract_inverted_index.achieve | 51, 91 |
| abstract_inverted_index.context | 74 |
| abstract_inverted_index.crucial | 3 |
| abstract_inverted_index.design, | 98 |
| abstract_inverted_index.dynamic | 58 |
| abstract_inverted_index.employs | 56 |
| abstract_inverted_index.exceeds | 131 |
| abstract_inverted_index.forward | 156 |
| abstract_inverted_index.general | 136 |
| abstract_inverted_index.models, | 7 |
| abstract_inverted_index.present | 35 |
| abstract_inverted_index.through | 94 |
| abstract_inverted_index.without | 113 |
| abstract_inverted_index.Natively | 38 |
| abstract_inverted_index.achieves | 145 |
| abstract_inverted_index.advances | 81 |
| abstract_inverted_index.approach | 80 |
| abstract_inverted_index.backward | 159 |
| abstract_inverted_index.language | 6 |
| abstract_inverted_index.modeling | 1 |
| abstract_inverted_index.preserve | 71 |
| abstract_inverted_index.reducing | 110 |
| abstract_inverted_index.speedups | 93, 147 |
| abstract_inverted_index.standard | 14 |
| abstract_inverted_index.Attention | 41, 133, 150 |
| abstract_inverted_index.algorithm | 97 |
| abstract_inverted_index.attention | 15, 22, 83 |
| abstract_inverted_index.awareness | 75 |
| abstract_inverted_index.combining | 62 |
| abstract_inverted_index.decoding, | 155 |
| abstract_inverted_index.direction | 26 |
| abstract_inverted_index.efficient | 52 |
| abstract_inverted_index.hardware. | 104 |
| abstract_inverted_index.improving | 28 |
| abstract_inverted_index.maintains | 129 |
| abstract_inverted_index.mechanism | 42 |
| abstract_inverted_index.modeling. | 54 |
| abstract_inverted_index.promising | 25 |
| abstract_inverted_index.selection | 69 |
| abstract_inverted_index.sequences | 153 |
| abstract_inverted_index.strategy, | 61 |
| abstract_inverted_index.trainable | 39 |
| abstract_inverted_index.training, | 109 |
| abstract_inverted_index.64k-length | 152 |
| abstract_inverted_index.Meanwhile, | 143 |
| abstract_inverted_index.arithmetic | 95 |
| abstract_inverted_index.efficiency | 29, 163 |
| abstract_inverted_index.end-to-end | 108 |
| abstract_inverted_index.integrates | 44 |
| abstract_inverted_index.lifecycle. | 167 |
| abstract_inverted_index.mechanisms | 16 |
| abstract_inverted_index.precision. | 78 |
| abstract_inverted_index.pretrained | 126 |
| abstract_inverted_index.reasoning. | 142 |
| abstract_inverted_index.throughout | 164 |
| abstract_inverted_index.validating | 161 |
| abstract_inverted_index.algorithmic | 45 |
| abstract_inverted_index.benchmarks, | 137 |
| abstract_inverted_index.challenges. | 20 |
| abstract_inverted_index.compression | 65 |
| abstract_inverted_index.computation | 112 |
| abstract_inverted_index.experiments | 122 |
| abstract_inverted_index.innovations | 46 |
| abstract_inverted_index.maintaining | 31 |
| abstract_inverted_index.pretraining | 111 |
| abstract_inverted_index.sacrificing | 114 |
| abstract_inverted_index.significant | 18 |
| abstract_inverted_index.substantial | 92, 146 |
| abstract_inverted_index.Long-context | 0 |
| abstract_inverted_index.fine-grained | 67 |
| abstract_inverted_index.hierarchical | 59 |
| abstract_inverted_index.innovations: | 88 |
| abstract_inverted_index.long-context | 53, 138 |
| abstract_inverted_index.performance. | 116 |
| abstract_inverted_index.propagation, | 157, 160 |
| abstract_inverted_index.capabilities. | 33 |
| abstract_inverted_index.computational | 11, 19 |
| abstract_inverted_index.optimizations | 49, 101 |
| abstract_inverted_index.coarse-grained | 63 |
| abstract_inverted_index.implementation | 100 |
| abstract_inverted_index.next-generation | 5 |
| abstract_inverted_index.hardware-aligned | 48 |
| abstract_inverted_index.instruction-based | 141 |
| abstract_inverted_index.intensity-balanced | 96 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 15 |
| citation_normalized_percentile |