A Unified Perspective on the Dynamics of Deep Transformers Article Swipe
Valérie Castin
,
Pierre Ablin
,
José Carrillo
,
Gabriel Peyré
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2501.18322
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2501.18322
Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens. This representation is then exploited by the attention function, which learns dependencies between tokens and is key to the success of Transformers. However, the iterative application of attention across layers induces complex dynamics that remain to be fully understood. To analyze these dynamics, we identify each input sequence with a probability measure and model its evolution as a Vlasov equation called Transformer PDE, whose velocity field is non-linear in the probability measure. Our first set of contributions focuses on compactly supported initial data. We show the Transformer PDE is well-posed and is the mean-field limit of an interacting particle system, thus generalizing and extending previous analysis to several variants of self-attention: multi-head attention, L2 attention, Sinkhorn attention, Sigmoid attention, and masked attention--leveraging a conditional Wasserstein framework. In a second set of contributions, we are the first to study non-compactly supported initial conditions, by focusing on Gaussian initial data. Again for different types of attention, we show that the Transformer PDE preserves the space of Gaussian measures, which allows us to analyze the Gaussian case theoretically and numerically to identify typical behaviors. This Gaussian analysis captures the evolution of data anisotropy through a deep Transformer. In particular, we highlight a clustering phenomenon that parallels previous results in the non-normalized discrete case.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2501.18322
- https://arxiv.org/pdf/2501.18322
- OA Status
- green
- Cited By
- 1
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4407012314
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4407012314Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2501.18322Digital Object Identifier
- Title
-
A Unified Perspective on the Dynamics of Deep TransformersWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-01-30Full publication date if available
- Authors
-
Valérie Castin, Pierre Ablin, José Carrillo, Gabriel PeyréList of authors in order
- Landing page
-
https://arxiv.org/abs/2501.18322Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2501.18322Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2501.18322Direct OA link when available
- Concepts
-
Perspective (graphical), Transformer, Computer science, Engineering, Artificial intelligence, Electrical engineering, VoltageTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
1Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 1Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4407012314 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2501.18322 |
| ids.doi | https://doi.org/10.48550/arxiv.2501.18322 |
| ids.openalex | https://openalex.org/W4407012314 |
| fwci | |
| type | preprint |
| title | A Unified Perspective on the Dynamics of Deep Transformers |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10320 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.6786999702453613 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Neural Networks and Applications |
| topics[1].id | https://openalex.org/T10502 |
| topics[1].field.id | https://openalex.org/fields/22 |
| topics[1].field.display_name | Engineering |
| topics[1].score | 0.6777999997138977 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/2208 |
| topics[1].subfield.display_name | Electrical and Electronic Engineering |
| topics[1].display_name | Advanced Memory and Neural Computing |
| topics[2].id | https://openalex.org/T12611 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.5989000201225281 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Neural Networks and Reservoir Computing |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C12713177 |
| concepts[0].level | 2 |
| concepts[0].score | 0.6123707890510559 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q1900281 |
| concepts[0].display_name | Perspective (graphical) |
| concepts[1].id | https://openalex.org/C66322947 |
| concepts[1].level | 3 |
| concepts[1].score | 0.4403444528579712 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q11658 |
| concepts[1].display_name | Transformer |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.3980702757835388 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C127413603 |
| concepts[3].level | 0 |
| concepts[3].score | 0.22401317954063416 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11023 |
| concepts[3].display_name | Engineering |
| concepts[4].id | https://openalex.org/C154945302 |
| concepts[4].level | 1 |
| concepts[4].score | 0.18896782398223877 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[4].display_name | Artificial intelligence |
| concepts[5].id | https://openalex.org/C119599485 |
| concepts[5].level | 1 |
| concepts[5].score | 0.1481209695339203 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q43035 |
| concepts[5].display_name | Electrical engineering |
| concepts[6].id | https://openalex.org/C165801399 |
| concepts[6].level | 2 |
| concepts[6].score | 0.0 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q25428 |
| concepts[6].display_name | Voltage |
| keywords[0].id | https://openalex.org/keywords/perspective |
| keywords[0].score | 0.6123707890510559 |
| keywords[0].display_name | Perspective (graphical) |
| keywords[1].id | https://openalex.org/keywords/transformer |
| keywords[1].score | 0.4403444528579712 |
| keywords[1].display_name | Transformer |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.3980702757835388 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/engineering |
| keywords[3].score | 0.22401317954063416 |
| keywords[3].display_name | Engineering |
| keywords[4].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[4].score | 0.18896782398223877 |
| keywords[4].display_name | Artificial intelligence |
| keywords[5].id | https://openalex.org/keywords/electrical-engineering |
| keywords[5].score | 0.1481209695339203 |
| keywords[5].display_name | Electrical engineering |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2501.18322 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2501.18322 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2501.18322 |
| locations[1].id | doi:10.48550/arxiv.2501.18322 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2501.18322 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5093581632 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Valérie Castin |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Castin, Valérie |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5042340163 |
| authorships[1].author.orcid | https://orcid.org/0000-0003-4277-5202 |
| authorships[1].author.display_name | Pierre Ablin |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Ablin, Pierre |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5107228706 |
| authorships[2].author.orcid | https://orcid.org/0000-0001-7906-416X |
| authorships[2].author.display_name | José Carrillo |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Carrillo, José Antonio |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5058651667 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-4477-0387 |
| authorships[3].author.display_name | Gabriel Peyré |
| authorships[3].author_position | last |
| authorships[3].raw_author_name | Peyré, Gabriel |
| authorships[3].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2501.18322 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | A Unified Perspective on the Dynamics of Deep Transformers |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10320 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.6786999702453613 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Neural Networks and Applications |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W4391913857, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2018871932, https://openalex.org/W2001405890 |
| cited_by_count | 1 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 1 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2501.18322 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2501.18322 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2501.18322 |
| primary_location.id | pmh:oai:arXiv.org:2501.18322 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2501.18322 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2501.18322 |
| publication_date | 2025-01-30 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 67, 75, 140, 145, 209, 216 |
| abstract_inverted_index.In | 144, 212 |
| abstract_inverted_index.L2 | 131 |
| abstract_inverted_index.To | 57 |
| abstract_inverted_index.We | 101 |
| abstract_inverted_index.an | 114 |
| abstract_inverted_index.as | 12, 74 |
| abstract_inverted_index.be | 54 |
| abstract_inverted_index.by | 23, 160 |
| abstract_inverted_index.in | 4, 86, 223 |
| abstract_inverted_index.is | 20, 33, 84, 106, 109 |
| abstract_inverted_index.of | 14, 38, 44, 93, 113, 127, 148, 170, 181, 205 |
| abstract_inverted_index.on | 96, 162 |
| abstract_inverted_index.to | 35, 53, 124, 154, 187, 195 |
| abstract_inverted_index.us | 186 |
| abstract_inverted_index.we | 61, 150, 172, 214 |
| abstract_inverted_index.Our | 90 |
| abstract_inverted_index.PDE | 105, 177 |
| abstract_inverted_index.and | 32, 70, 108, 120, 137, 193 |
| abstract_inverted_index.are | 2, 151 |
| abstract_inverted_index.for | 167 |
| abstract_inverted_index.its | 72 |
| abstract_inverted_index.key | 34 |
| abstract_inverted_index.set | 92, 147 |
| abstract_inverted_index.the | 10, 24, 36, 41, 87, 103, 110, 152, 175, 179, 189, 203, 224 |
| abstract_inverted_index.PDE, | 80 |
| abstract_inverted_index.This | 18, 199 |
| abstract_inverted_index.case | 191 |
| abstract_inverted_index.data | 11, 206 |
| abstract_inverted_index.deep | 210 |
| abstract_inverted_index.each | 63 |
| abstract_inverted_index.most | 5 |
| abstract_inverted_index.show | 102, 173 |
| abstract_inverted_index.that | 51, 174, 219 |
| abstract_inverted_index.then | 21 |
| abstract_inverted_index.thus | 118 |
| abstract_inverted_index.with | 66 |
| abstract_inverted_index.Again | 166 |
| abstract_inverted_index.case. | 227 |
| abstract_inverted_index.data. | 100, 165 |
| abstract_inverted_index.field | 83 |
| abstract_inverted_index.first | 91, 153 |
| abstract_inverted_index.fully | 55 |
| abstract_inverted_index.input | 64 |
| abstract_inverted_index.limit | 112 |
| abstract_inverted_index.model | 71 |
| abstract_inverted_index.space | 180 |
| abstract_inverted_index.study | 155 |
| abstract_inverted_index.these | 59 |
| abstract_inverted_index.types | 169 |
| abstract_inverted_index.which | 1, 27, 184 |
| abstract_inverted_index.whose | 81 |
| abstract_inverted_index.Vlasov | 76 |
| abstract_inverted_index.across | 46 |
| abstract_inverted_index.allows | 185 |
| abstract_inverted_index.called | 16, 78 |
| abstract_inverted_index.layers | 47 |
| abstract_inverted_index.learns | 28 |
| abstract_inverted_index.masked | 138 |
| abstract_inverted_index.remain | 52 |
| abstract_inverted_index.second | 146 |
| abstract_inverted_index.tasks, | 8 |
| abstract_inverted_index.tokens | 31 |
| abstract_inverted_index.Sigmoid | 135 |
| abstract_inverted_index.analyze | 58, 188 |
| abstract_inverted_index.between | 30 |
| abstract_inverted_index.complex | 49 |
| abstract_inverted_index.focuses | 95 |
| abstract_inverted_index.induces | 48 |
| abstract_inverted_index.initial | 99, 158, 164 |
| abstract_inverted_index.machine | 6 |
| abstract_inverted_index.measure | 69 |
| abstract_inverted_index.results | 222 |
| abstract_inverted_index.several | 125 |
| abstract_inverted_index.success | 37 |
| abstract_inverted_index.system, | 117 |
| abstract_inverted_index.through | 208 |
| abstract_inverted_index.tokens. | 17 |
| abstract_inverted_index.typical | 197 |
| abstract_inverted_index.vectors | 15 |
| abstract_inverted_index.Gaussian | 163, 182, 190, 200 |
| abstract_inverted_index.However, | 40 |
| abstract_inverted_index.Sinkhorn | 133 |
| abstract_inverted_index.analysis | 123, 201 |
| abstract_inverted_index.captures | 202 |
| abstract_inverted_index.discrete | 226 |
| abstract_inverted_index.dynamics | 50 |
| abstract_inverted_index.equation | 77 |
| abstract_inverted_index.focusing | 161 |
| abstract_inverted_index.identify | 62, 196 |
| abstract_inverted_index.learning | 7 |
| abstract_inverted_index.measure. | 89 |
| abstract_inverted_index.particle | 116 |
| abstract_inverted_index.previous | 122, 221 |
| abstract_inverted_index.sequence | 65 |
| abstract_inverted_index.variants | 126 |
| abstract_inverted_index.velocity | 82 |
| abstract_inverted_index.attention | 25, 45 |
| abstract_inverted_index.compactly | 97 |
| abstract_inverted_index.different | 168 |
| abstract_inverted_index.dynamics, | 60 |
| abstract_inverted_index.evolution | 73, 204 |
| abstract_inverted_index.exploited | 22 |
| abstract_inverted_index.extending | 121 |
| abstract_inverted_index.function, | 26 |
| abstract_inverted_index.highlight | 215 |
| abstract_inverted_index.iterative | 42 |
| abstract_inverted_index.measures, | 183 |
| abstract_inverted_index.parallels | 220 |
| abstract_inverted_index.preserves | 178 |
| abstract_inverted_index.represent | 9 |
| abstract_inverted_index.sequences | 13 |
| abstract_inverted_index.supported | 98, 157 |
| abstract_inverted_index.anisotropy | 207 |
| abstract_inverted_index.attention, | 130, 132, 134, 136, 171 |
| abstract_inverted_index.behaviors. | 198 |
| abstract_inverted_index.clustering | 217 |
| abstract_inverted_index.framework. | 143 |
| abstract_inverted_index.mean-field | 111 |
| abstract_inverted_index.multi-head | 129 |
| abstract_inverted_index.non-linear | 85 |
| abstract_inverted_index.phenomenon | 218 |
| abstract_inverted_index.well-posed | 107 |
| abstract_inverted_index.Transformer | 79, 104, 176 |
| abstract_inverted_index.Wasserstein | 142 |
| abstract_inverted_index.application | 43 |
| abstract_inverted_index.conditional | 141 |
| abstract_inverted_index.conditions, | 159 |
| abstract_inverted_index.interacting | 115 |
| abstract_inverted_index.numerically | 194 |
| abstract_inverted_index.particular, | 213 |
| abstract_inverted_index.probability | 68, 88 |
| abstract_inverted_index.understood. | 56 |
| abstract_inverted_index.Transformer. | 211 |
| abstract_inverted_index.dependencies | 29 |
| abstract_inverted_index.generalizing | 119 |
| abstract_inverted_index.Transformers, | 0 |
| abstract_inverted_index.Transformers. | 39 |
| abstract_inverted_index.contributions | 94 |
| abstract_inverted_index.non-compactly | 156 |
| abstract_inverted_index.theoretically | 192 |
| abstract_inverted_index.contributions, | 149 |
| abstract_inverted_index.non-normalized | 225 |
| abstract_inverted_index.representation | 19 |
| abstract_inverted_index.self-attention: | 128 |
| abstract_inverted_index.state-of-the-art | 3 |
| abstract_inverted_index.attention--leveraging | 139 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 4 |
| citation_normalized_percentile |