Spectraformer: A Unified Random Feature Framework for Transformer Article Swipe
Duke Nguyen
,
Aditya Joshi
,
Flora D. Salim
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2405.15310
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2405.15310
Linearization of attention using various kernel approximation and kernel learning techniques has shown promise. Past methods used a subset of combinations of component functions and weight matrices within the random feature paradigm. We identify the need for a systematic comparison of different combinations of weight matrices and component functions for attention learning in Transformer. Hence, we introduce Spectraformer, a unified framework for approximating and learning the kernel function in the attention mechanism of the Transformer. Our empirical results demonstrate, for the first time, that a random feature-based approach can achieve performance comparable to top-performing sparse and low-rank methods on the challenging Long Range Arena benchmark. Thus, we establish a new state-of-the-art for random feature-based efficient Transformers. The framework also produces many variants that offer different advantages in accuracy, training time, and memory consumption. Our code is available at: https://github.com/cruiseresearchgroup/spectraformer .
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2405.15310
- https://arxiv.org/pdf/2405.15310
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4399062222
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4399062222Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2405.15310Digital Object Identifier
- Title
-
Spectraformer: A Unified Random Feature Framework for TransformerWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-05-24Full publication date if available
- Authors
-
Duke Nguyen, Aditya Joshi, Flora D. SalimList of authors in order
- Landing page
-
https://arxiv.org/abs/2405.15310Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2405.15310Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2405.15310Direct OA link when available
- Concepts
-
Computer science, Feature (linguistics), Transformer, Artificial intelligence, Natural language processing, Pattern recognition (psychology), Engineering, Electrical engineering, Linguistics, Voltage, PhilosophyTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4399062222 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2405.15310 |
| ids.doi | https://doi.org/10.48550/arxiv.2405.15310 |
| ids.openalex | https://openalex.org/W4399062222 |
| fwci | |
| type | preprint |
| title | Spectraformer: A Unified Random Feature Framework for Transformer |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10320 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9456999897956848 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Neural Networks and Applications |
| topics[1].id | https://openalex.org/T10876 |
| topics[1].field.id | https://openalex.org/fields/22 |
| topics[1].field.display_name | Engineering |
| topics[1].score | 0.9225999712944031 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/2207 |
| topics[1].subfield.display_name | Control and Systems Engineering |
| topics[1].display_name | Fault Detection and Control Systems |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.5590396523475647 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C2776401178 |
| concepts[1].level | 2 |
| concepts[1].score | 0.5145263075828552 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q12050496 |
| concepts[1].display_name | Feature (linguistics) |
| concepts[2].id | https://openalex.org/C66322947 |
| concepts[2].level | 3 |
| concepts[2].score | 0.43605607748031616 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q11658 |
| concepts[2].display_name | Transformer |
| concepts[3].id | https://openalex.org/C154945302 |
| concepts[3].level | 1 |
| concepts[3].score | 0.3504639267921448 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[3].display_name | Artificial intelligence |
| concepts[4].id | https://openalex.org/C204321447 |
| concepts[4].level | 1 |
| concepts[4].score | 0.3436783254146576 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[4].display_name | Natural language processing |
| concepts[5].id | https://openalex.org/C153180895 |
| concepts[5].level | 2 |
| concepts[5].score | 0.332613080739975 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q7148389 |
| concepts[5].display_name | Pattern recognition (psychology) |
| concepts[6].id | https://openalex.org/C127413603 |
| concepts[6].level | 0 |
| concepts[6].score | 0.12340009212493896 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q11023 |
| concepts[6].display_name | Engineering |
| concepts[7].id | https://openalex.org/C119599485 |
| concepts[7].level | 1 |
| concepts[7].score | 0.12308153510093689 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q43035 |
| concepts[7].display_name | Electrical engineering |
| concepts[8].id | https://openalex.org/C41895202 |
| concepts[8].level | 1 |
| concepts[8].score | 0.07738137245178223 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q8162 |
| concepts[8].display_name | Linguistics |
| concepts[9].id | https://openalex.org/C165801399 |
| concepts[9].level | 2 |
| concepts[9].score | 0.06742256879806519 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q25428 |
| concepts[9].display_name | Voltage |
| concepts[10].id | https://openalex.org/C138885662 |
| concepts[10].level | 0 |
| concepts[10].score | 0.0403059720993042 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[10].display_name | Philosophy |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.5590396523475647 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/feature |
| keywords[1].score | 0.5145263075828552 |
| keywords[1].display_name | Feature (linguistics) |
| keywords[2].id | https://openalex.org/keywords/transformer |
| keywords[2].score | 0.43605607748031616 |
| keywords[2].display_name | Transformer |
| keywords[3].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[3].score | 0.3504639267921448 |
| keywords[3].display_name | Artificial intelligence |
| keywords[4].id | https://openalex.org/keywords/natural-language-processing |
| keywords[4].score | 0.3436783254146576 |
| keywords[4].display_name | Natural language processing |
| keywords[5].id | https://openalex.org/keywords/pattern-recognition |
| keywords[5].score | 0.332613080739975 |
| keywords[5].display_name | Pattern recognition (psychology) |
| keywords[6].id | https://openalex.org/keywords/engineering |
| keywords[6].score | 0.12340009212493896 |
| keywords[6].display_name | Engineering |
| keywords[7].id | https://openalex.org/keywords/electrical-engineering |
| keywords[7].score | 0.12308153510093689 |
| keywords[7].display_name | Electrical engineering |
| keywords[8].id | https://openalex.org/keywords/linguistics |
| keywords[8].score | 0.07738137245178223 |
| keywords[8].display_name | Linguistics |
| keywords[9].id | https://openalex.org/keywords/voltage |
| keywords[9].score | 0.06742256879806519 |
| keywords[9].display_name | Voltage |
| keywords[10].id | https://openalex.org/keywords/philosophy |
| keywords[10].score | 0.0403059720993042 |
| keywords[10].display_name | Philosophy |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2405.15310 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2405.15310 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2405.15310 |
| locations[1].id | doi:10.48550/arxiv.2405.15310 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2405.15310 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5101367504 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Duke Nguyen |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Nguyen, Duke |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5059529586 |
| authorships[1].author.orcid | https://orcid.org/0000-0003-2200-9703 |
| authorships[1].author.display_name | Aditya Joshi |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Joshi, Aditya |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5090893421 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-1237-1664 |
| authorships[2].author.display_name | Flora D. Salim |
| authorships[2].author_position | last |
| authorships[2].raw_author_name | Salim, Flora |
| authorships[2].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2405.15310 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Spectraformer: A Unified Random Feature Framework for Transformer |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10320 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9456999897956848 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Neural Networks and Applications |
| related_works | https://openalex.org/W3147584709, https://openalex.org/W2977677679, https://openalex.org/W1992327129, https://openalex.org/W2381986121, https://openalex.org/W2370918718, https://openalex.org/W2256933480, https://openalex.org/W2033914206, https://openalex.org/W2042327336, https://openalex.org/W4386159726, https://openalex.org/W3204019825 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2405.15310 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2405.15310 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2405.15310 |
| primary_location.id | pmh:oai:arXiv.org:2405.15310 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2405.15310 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2405.15310 |
| publication_date | 2024-05-24 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.. | 139 |
| abstract_inverted_index.a | 17, 37, 58, 84, 108 |
| abstract_inverted_index.We | 32 |
| abstract_inverted_index.in | 52, 68, 126 |
| abstract_inverted_index.is | 135 |
| abstract_inverted_index.of | 1, 19, 21, 40, 43, 72 |
| abstract_inverted_index.on | 98 |
| abstract_inverted_index.to | 92 |
| abstract_inverted_index.we | 55, 106 |
| abstract_inverted_index.Our | 75, 133 |
| abstract_inverted_index.The | 116 |
| abstract_inverted_index.and | 7, 24, 46, 63, 95, 130 |
| abstract_inverted_index.at: | 137 |
| abstract_inverted_index.can | 88 |
| abstract_inverted_index.for | 36, 49, 61, 79, 111 |
| abstract_inverted_index.has | 11 |
| abstract_inverted_index.new | 109 |
| abstract_inverted_index.the | 28, 34, 65, 69, 73, 80, 99 |
| abstract_inverted_index.Long | 101 |
| abstract_inverted_index.Past | 14 |
| abstract_inverted_index.also | 118 |
| abstract_inverted_index.code | 134 |
| abstract_inverted_index.many | 120 |
| abstract_inverted_index.need | 35 |
| abstract_inverted_index.that | 83, 122 |
| abstract_inverted_index.used | 16 |
| abstract_inverted_index.Arena | 103 |
| abstract_inverted_index.Range | 102 |
| abstract_inverted_index.Thus, | 105 |
| abstract_inverted_index.first | 81 |
| abstract_inverted_index.offer | 123 |
| abstract_inverted_index.shown | 12 |
| abstract_inverted_index.time, | 82, 129 |
| abstract_inverted_index.using | 3 |
| abstract_inverted_index.Hence, | 54 |
| abstract_inverted_index.kernel | 5, 8, 66 |
| abstract_inverted_index.memory | 131 |
| abstract_inverted_index.random | 29, 85, 112 |
| abstract_inverted_index.sparse | 94 |
| abstract_inverted_index.subset | 18 |
| abstract_inverted_index.weight | 25, 44 |
| abstract_inverted_index.within | 27 |
| abstract_inverted_index.achieve | 89 |
| abstract_inverted_index.feature | 30 |
| abstract_inverted_index.methods | 15, 97 |
| abstract_inverted_index.results | 77 |
| abstract_inverted_index.unified | 59 |
| abstract_inverted_index.various | 4 |
| abstract_inverted_index.approach | 87 |
| abstract_inverted_index.function | 67 |
| abstract_inverted_index.identify | 33 |
| abstract_inverted_index.learning | 9, 51, 64 |
| abstract_inverted_index.low-rank | 96 |
| abstract_inverted_index.matrices | 26, 45 |
| abstract_inverted_index.produces | 119 |
| abstract_inverted_index.promise. | 13 |
| abstract_inverted_index.training | 128 |
| abstract_inverted_index.variants | 121 |
| abstract_inverted_index.accuracy, | 127 |
| abstract_inverted_index.attention | 2, 50, 70 |
| abstract_inverted_index.available | 136 |
| abstract_inverted_index.component | 22, 47 |
| abstract_inverted_index.different | 41, 124 |
| abstract_inverted_index.efficient | 114 |
| abstract_inverted_index.empirical | 76 |
| abstract_inverted_index.establish | 107 |
| abstract_inverted_index.framework | 60, 117 |
| abstract_inverted_index.functions | 23, 48 |
| abstract_inverted_index.introduce | 56 |
| abstract_inverted_index.mechanism | 71 |
| abstract_inverted_index.paradigm. | 31 |
| abstract_inverted_index.advantages | 125 |
| abstract_inverted_index.benchmark. | 104 |
| abstract_inverted_index.comparable | 91 |
| abstract_inverted_index.comparison | 39 |
| abstract_inverted_index.systematic | 38 |
| abstract_inverted_index.techniques | 10 |
| abstract_inverted_index.challenging | 100 |
| abstract_inverted_index.performance | 90 |
| abstract_inverted_index.Transformer. | 53, 74 |
| abstract_inverted_index.combinations | 20, 42 |
| abstract_inverted_index.consumption. | 132 |
| abstract_inverted_index.demonstrate, | 78 |
| abstract_inverted_index.Linearization | 0 |
| abstract_inverted_index.Transformers. | 115 |
| abstract_inverted_index.approximating | 62 |
| abstract_inverted_index.approximation | 6 |
| abstract_inverted_index.feature-based | 86, 113 |
| abstract_inverted_index.Spectraformer, | 57 |
| abstract_inverted_index.top-performing | 93 |
| abstract_inverted_index.state-of-the-art | 110 |
| abstract_inverted_index.https://github.com/cruiseresearchgroup/spectraformer | 138 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 3 |
| citation_normalized_percentile |