From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning Article Swipe
Noa Rubin
,
Kirsten Fischer
,
Javed Lindner
,
David Dahmen
,
Inbar Seroussi
,
Zohar Ringel
,
M. Krämer
,
Moritz Helias
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2502.03210
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2502.03210
Feature learning in neural networks is crucial for their expressive power and inductive biases, motivating various theoretical approaches. Some approaches describe network behavior after training through a change in kernel scale from initialization, resulting in a generalization power comparable to a Gaussian process. Conversely, in other approaches training results in the adaptation of the kernel to the data, involving directional changes to the kernel. The relationship and respective strengths of these two views have so far remained unresolved. This work presents a theoretical framework of multi-scale adaptive feature learning bridging these two views. Using methods from statistical mechanics, we derive analytical expressions for network output statistics which are valid across scaling regimes and in the continuum between them. A systematic expansion of the network's probability distribution reveals that mean-field scaling requires only a saddle-point approximation, while standard scaling necessitates additional correction terms. Remarkably, we find across regimes that kernel adaptation can be reduced to an effective kernel rescaling when predicting the mean network output in the special case of a linear network. However, for linear and non-linear networks, the multi-scale adaptive approach captures directional feature learning effects, providing richer insights than what could be recovered from a rescaling of the kernel alone.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2502.03210
- https://arxiv.org/pdf/2502.03210
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4407231241
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4407231241Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2502.03210Digital Object Identifier
- Title
-
From Kernels to Features: A Multi-Scale Adaptive Theory of Feature LearningWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-02-05Full publication date if available
- Authors
-
Noa Rubin, Kirsten Fischer, Javed Lindner, David Dahmen, Inbar Seroussi, Zohar Ringel, M. Krämer, Moritz HeliasList of authors in order
- Landing page
-
https://arxiv.org/abs/2502.03210Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2502.03210Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2502.03210Direct OA link when available
- Concepts
-
Feature (linguistics), Scale (ratio), Computer science, Artificial intelligence, Feature learning, Machine learning, Geography, Philosophy, Cartography, LinguisticsTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4407231241 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2502.03210 |
| ids.doi | https://doi.org/10.48550/arxiv.2502.03210 |
| ids.openalex | https://openalex.org/W4407231241 |
| fwci | |
| type | preprint |
| title | From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10320 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.3628000020980835 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Neural Networks and Applications |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C2776401178 |
| concepts[0].level | 2 |
| concepts[0].score | 0.6268903017044067 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q12050496 |
| concepts[0].display_name | Feature (linguistics) |
| concepts[1].id | https://openalex.org/C2778755073 |
| concepts[1].level | 2 |
| concepts[1].score | 0.5913066267967224 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q10858537 |
| concepts[1].display_name | Scale (ratio) |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.4948355555534363 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C154945302 |
| concepts[3].level | 1 |
| concepts[3].score | 0.489165335893631 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[3].display_name | Artificial intelligence |
| concepts[4].id | https://openalex.org/C59404180 |
| concepts[4].level | 2 |
| concepts[4].score | 0.4356163740158081 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q17013334 |
| concepts[4].display_name | Feature learning |
| concepts[5].id | https://openalex.org/C119857082 |
| concepts[5].level | 1 |
| concepts[5].score | 0.4082756042480469 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[5].display_name | Machine learning |
| concepts[6].id | https://openalex.org/C205649164 |
| concepts[6].level | 0 |
| concepts[6].score | 0.18743345141410828 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q1071 |
| concepts[6].display_name | Geography |
| concepts[7].id | https://openalex.org/C138885662 |
| concepts[7].level | 0 |
| concepts[7].score | 0.0886925458908081 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[7].display_name | Philosophy |
| concepts[8].id | https://openalex.org/C58640448 |
| concepts[8].level | 1 |
| concepts[8].score | 0.0786023736000061 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q42515 |
| concepts[8].display_name | Cartography |
| concepts[9].id | https://openalex.org/C41895202 |
| concepts[9].level | 1 |
| concepts[9].score | 0.06915140151977539 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q8162 |
| concepts[9].display_name | Linguistics |
| keywords[0].id | https://openalex.org/keywords/feature |
| keywords[0].score | 0.6268903017044067 |
| keywords[0].display_name | Feature (linguistics) |
| keywords[1].id | https://openalex.org/keywords/scale |
| keywords[1].score | 0.5913066267967224 |
| keywords[1].display_name | Scale (ratio) |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.4948355555534363 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[3].score | 0.489165335893631 |
| keywords[3].display_name | Artificial intelligence |
| keywords[4].id | https://openalex.org/keywords/feature-learning |
| keywords[4].score | 0.4356163740158081 |
| keywords[4].display_name | Feature learning |
| keywords[5].id | https://openalex.org/keywords/machine-learning |
| keywords[5].score | 0.4082756042480469 |
| keywords[5].display_name | Machine learning |
| keywords[6].id | https://openalex.org/keywords/geography |
| keywords[6].score | 0.18743345141410828 |
| keywords[6].display_name | Geography |
| keywords[7].id | https://openalex.org/keywords/philosophy |
| keywords[7].score | 0.0886925458908081 |
| keywords[7].display_name | Philosophy |
| keywords[8].id | https://openalex.org/keywords/cartography |
| keywords[8].score | 0.0786023736000061 |
| keywords[8].display_name | Cartography |
| keywords[9].id | https://openalex.org/keywords/linguistics |
| keywords[9].score | 0.06915140151977539 |
| keywords[9].display_name | Linguistics |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2502.03210 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2502.03210 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2502.03210 |
| locations[1].id | doi:10.48550/arxiv.2502.03210 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2502.03210 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5108309346 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Noa Rubin |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Rubin, Noa |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5061682665 |
| authorships[1].author.orcid | https://orcid.org/0009-0006-6169-9152 |
| authorships[1].author.display_name | Kirsten Fischer |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Fischer, Kirsten |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5080001235 |
| authorships[2].author.orcid | https://orcid.org/0009-0008-9913-9040 |
| authorships[2].author.display_name | Javed Lindner |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Lindner, Javed |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5068197411 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-7664-916X |
| authorships[3].author.display_name | David Dahmen |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Dahmen, David |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5054816338 |
| authorships[4].author.orcid | https://orcid.org/0000-0001-5209-5839 |
| authorships[4].author.display_name | Inbar Seroussi |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Seroussi, Inbar |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5053390682 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-2555-7695 |
| authorships[5].author.display_name | Zohar Ringel |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Ringel, Zohar |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5003133702 |
| authorships[6].author.orcid | https://orcid.org/0000-0002-7615-7499 |
| authorships[6].author.display_name | M. Krämer |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Krämer, Michael |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5032616213 |
| authorships[7].author.orcid | https://orcid.org/0000-0002-0404-8656 |
| authorships[7].author.display_name | Moritz Helias |
| authorships[7].author_position | last |
| authorships[7].raw_author_name | Helias, Moritz |
| authorships[7].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2502.03210 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10320 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.3628000020980835 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Neural Networks and Applications |
| related_works | https://openalex.org/W2961085424, https://openalex.org/W4306674287, https://openalex.org/W4387369504, https://openalex.org/W3046775127, https://openalex.org/W4394896187, https://openalex.org/W3170094116, https://openalex.org/W4386462264, https://openalex.org/W3107602296, https://openalex.org/W4364306694, https://openalex.org/W4312192474 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2502.03210 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2502.03210 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2502.03210 |
| primary_location.id | pmh:oai:arXiv.org:2502.03210 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2502.03210 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2502.03210 |
| publication_date | 2025-02-05 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.A | 118 |
| abstract_inverted_index.a | 26, 35, 40, 81, 132, 169, 196 |
| abstract_inverted_index.an | 154 |
| abstract_inverted_index.be | 151, 193 |
| abstract_inverted_index.in | 2, 28, 34, 44, 49, 113, 164 |
| abstract_inverted_index.is | 5 |
| abstract_inverted_index.of | 52, 69, 84, 121, 168, 198 |
| abstract_inverted_index.so | 74 |
| abstract_inverted_index.to | 39, 55, 61, 153 |
| abstract_inverted_index.we | 98, 143 |
| abstract_inverted_index.The | 64 |
| abstract_inverted_index.and | 11, 66, 112, 175 |
| abstract_inverted_index.are | 107 |
| abstract_inverted_index.can | 150 |
| abstract_inverted_index.far | 75 |
| abstract_inverted_index.for | 7, 102, 173 |
| abstract_inverted_index.the | 50, 53, 56, 62, 114, 122, 160, 165, 178, 199 |
| abstract_inverted_index.two | 71, 91 |
| abstract_inverted_index.Some | 18 |
| abstract_inverted_index.This | 78 |
| abstract_inverted_index.case | 167 |
| abstract_inverted_index.find | 144 |
| abstract_inverted_index.from | 31, 95, 195 |
| abstract_inverted_index.have | 73 |
| abstract_inverted_index.mean | 161 |
| abstract_inverted_index.only | 131 |
| abstract_inverted_index.than | 190 |
| abstract_inverted_index.that | 127, 147 |
| abstract_inverted_index.what | 191 |
| abstract_inverted_index.when | 158 |
| abstract_inverted_index.work | 79 |
| abstract_inverted_index.Using | 93 |
| abstract_inverted_index.after | 23 |
| abstract_inverted_index.could | 192 |
| abstract_inverted_index.data, | 57 |
| abstract_inverted_index.other | 45 |
| abstract_inverted_index.power | 10, 37 |
| abstract_inverted_index.scale | 30 |
| abstract_inverted_index.their | 8 |
| abstract_inverted_index.them. | 117 |
| abstract_inverted_index.these | 70, 90 |
| abstract_inverted_index.valid | 108 |
| abstract_inverted_index.views | 72 |
| abstract_inverted_index.which | 106 |
| abstract_inverted_index.while | 135 |
| abstract_inverted_index.across | 109, 145 |
| abstract_inverted_index.alone. | 201 |
| abstract_inverted_index.change | 27 |
| abstract_inverted_index.derive | 99 |
| abstract_inverted_index.kernel | 29, 54, 148, 156, 200 |
| abstract_inverted_index.linear | 170, 174 |
| abstract_inverted_index.neural | 3 |
| abstract_inverted_index.output | 104, 163 |
| abstract_inverted_index.richer | 188 |
| abstract_inverted_index.terms. | 141 |
| abstract_inverted_index.views. | 92 |
| abstract_inverted_index.Feature | 0 |
| abstract_inverted_index.between | 116 |
| abstract_inverted_index.biases, | 13 |
| abstract_inverted_index.changes | 60 |
| abstract_inverted_index.crucial | 6 |
| abstract_inverted_index.feature | 87, 184 |
| abstract_inverted_index.kernel. | 63 |
| abstract_inverted_index.methods | 94 |
| abstract_inverted_index.network | 21, 103, 162 |
| abstract_inverted_index.reduced | 152 |
| abstract_inverted_index.regimes | 111, 146 |
| abstract_inverted_index.results | 48 |
| abstract_inverted_index.reveals | 126 |
| abstract_inverted_index.scaling | 110, 129, 137 |
| abstract_inverted_index.special | 166 |
| abstract_inverted_index.through | 25 |
| abstract_inverted_index.various | 15 |
| abstract_inverted_index.Gaussian | 41 |
| abstract_inverted_index.However, | 172 |
| abstract_inverted_index.adaptive | 86, 180 |
| abstract_inverted_index.approach | 181 |
| abstract_inverted_index.behavior | 22 |
| abstract_inverted_index.bridging | 89 |
| abstract_inverted_index.captures | 182 |
| abstract_inverted_index.describe | 20 |
| abstract_inverted_index.effects, | 186 |
| abstract_inverted_index.insights | 189 |
| abstract_inverted_index.learning | 1, 88, 185 |
| abstract_inverted_index.network. | 171 |
| abstract_inverted_index.networks | 4 |
| abstract_inverted_index.presents | 80 |
| abstract_inverted_index.process. | 42 |
| abstract_inverted_index.remained | 76 |
| abstract_inverted_index.requires | 130 |
| abstract_inverted_index.standard | 136 |
| abstract_inverted_index.training | 24, 47 |
| abstract_inverted_index.continuum | 115 |
| abstract_inverted_index.effective | 155 |
| abstract_inverted_index.expansion | 120 |
| abstract_inverted_index.framework | 83 |
| abstract_inverted_index.inductive | 12 |
| abstract_inverted_index.involving | 58 |
| abstract_inverted_index.network's | 123 |
| abstract_inverted_index.networks, | 177 |
| abstract_inverted_index.providing | 187 |
| abstract_inverted_index.recovered | 194 |
| abstract_inverted_index.rescaling | 157, 197 |
| abstract_inverted_index.resulting | 33 |
| abstract_inverted_index.strengths | 68 |
| abstract_inverted_index.adaptation | 51, 149 |
| abstract_inverted_index.additional | 139 |
| abstract_inverted_index.analytical | 100 |
| abstract_inverted_index.approaches | 19, 46 |
| abstract_inverted_index.comparable | 38 |
| abstract_inverted_index.correction | 140 |
| abstract_inverted_index.expressive | 9 |
| abstract_inverted_index.mean-field | 128 |
| abstract_inverted_index.mechanics, | 97 |
| abstract_inverted_index.motivating | 14 |
| abstract_inverted_index.non-linear | 176 |
| abstract_inverted_index.predicting | 159 |
| abstract_inverted_index.respective | 67 |
| abstract_inverted_index.statistics | 105 |
| abstract_inverted_index.systematic | 119 |
| abstract_inverted_index.Conversely, | 43 |
| abstract_inverted_index.Remarkably, | 142 |
| abstract_inverted_index.approaches. | 17 |
| abstract_inverted_index.directional | 59, 183 |
| abstract_inverted_index.expressions | 101 |
| abstract_inverted_index.multi-scale | 85, 179 |
| abstract_inverted_index.probability | 124 |
| abstract_inverted_index.statistical | 96 |
| abstract_inverted_index.theoretical | 16, 82 |
| abstract_inverted_index.unresolved. | 77 |
| abstract_inverted_index.distribution | 125 |
| abstract_inverted_index.necessitates | 138 |
| abstract_inverted_index.relationship | 65 |
| abstract_inverted_index.saddle-point | 133 |
| abstract_inverted_index.approximation, | 134 |
| abstract_inverted_index.generalization | 36 |
| abstract_inverted_index.initialization, | 32 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 8 |
| citation_normalized_percentile |