Learning from Randomly Initialized Neural Network Features Article Swipe
Ehsan Amid
,
Rohan Anil
,
Wojciech Kotłowski
,
Manfred K. Warmuth
·
YOU?
·
· 2022
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2202.06438
YOU?
·
· 2022
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2202.06438
We present the surprising result that randomly initialized neural networks are good feature extractors in expectation. These random features correspond to finite-sample realizations of what we call Neural Network Prior Kernel (NNPK), which is inherently infinite-dimensional. We conduct ablations across multiple architectures of varying sizes as well as initializations and activation functions. Our analysis suggests that certain structures that manifest in a trained model are already present at initialization. Therefore, NNPK may provide further insight into why neural networks are so effective in learning such structures.
Related Topics
Concepts
Initialization
Artificial neural network
Computer science
Feature (linguistics)
Artificial intelligence
Sample (material)
Kernel (algebra)
Deep neural networks
Machine learning
Pattern recognition (psychology)
Mathematics
Discrete mathematics
Philosophy
Programming language
Chromatography
Linguistics
Chemistry
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2202.06438
- https://arxiv.org/pdf/2202.06438
- OA Status
- green
- Cited By
- 7
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4226225164
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4226225164Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2202.06438Digital Object Identifier
- Title
-
Learning from Randomly Initialized Neural Network FeaturesWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2022Year of publication
- Publication date
-
2022-02-13Full publication date if available
- Authors
-
Ehsan Amid, Rohan Anil, Wojciech Kotłowski, Manfred K. WarmuthList of authors in order
- Landing page
-
https://arxiv.org/abs/2202.06438Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2202.06438Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2202.06438Direct OA link when available
- Concepts
-
Initialization, Artificial neural network, Computer science, Feature (linguistics), Artificial intelligence, Sample (material), Kernel (algebra), Deep neural networks, Machine learning, Pattern recognition (psychology), Mathematics, Discrete mathematics, Philosophy, Programming language, Chromatography, Linguistics, ChemistryTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
7Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 1, 2024: 2, 2023: 3, 2022: 1Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4226225164 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2202.06438 |
| ids.doi | https://doi.org/10.48550/arxiv.2202.06438 |
| ids.openalex | https://openalex.org/W4226225164 |
| fwci | |
| type | preprint |
| title | Learning from Randomly Initialized Neural Network Features |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10320 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9929999709129333 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Neural Networks and Applications |
| topics[1].id | https://openalex.org/T12814 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.972000002861023 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Gaussian Processes and Bayesian Inference |
| topics[2].id | https://openalex.org/T10775 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9294000267982483 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1707 |
| topics[2].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[2].display_name | Generative Adversarial Networks and Image Synthesis |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C114466953 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8602474927902222 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q6034165 |
| concepts[0].display_name | Initialization |
| concepts[1].id | https://openalex.org/C50644808 |
| concepts[1].level | 2 |
| concepts[1].score | 0.786343514919281 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q192776 |
| concepts[1].display_name | Artificial neural network |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.643144965171814 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C2776401178 |
| concepts[3].level | 2 |
| concepts[3].score | 0.6231056451797485 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q12050496 |
| concepts[3].display_name | Feature (linguistics) |
| concepts[4].id | https://openalex.org/C154945302 |
| concepts[4].level | 1 |
| concepts[4].score | 0.5936670303344727 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[4].display_name | Artificial intelligence |
| concepts[5].id | https://openalex.org/C198531522 |
| concepts[5].level | 2 |
| concepts[5].score | 0.5102496147155762 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q485146 |
| concepts[5].display_name | Sample (material) |
| concepts[6].id | https://openalex.org/C74193536 |
| concepts[6].level | 2 |
| concepts[6].score | 0.502190351486206 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q574844 |
| concepts[6].display_name | Kernel (algebra) |
| concepts[7].id | https://openalex.org/C2984842247 |
| concepts[7].level | 3 |
| concepts[7].score | 0.47017091512680054 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q197536 |
| concepts[7].display_name | Deep neural networks |
| concepts[8].id | https://openalex.org/C119857082 |
| concepts[8].level | 1 |
| concepts[8].score | 0.4525548219680786 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[8].display_name | Machine learning |
| concepts[9].id | https://openalex.org/C153180895 |
| concepts[9].level | 2 |
| concepts[9].score | 0.3909151554107666 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q7148389 |
| concepts[9].display_name | Pattern recognition (psychology) |
| concepts[10].id | https://openalex.org/C33923547 |
| concepts[10].level | 0 |
| concepts[10].score | 0.24014392495155334 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[10].display_name | Mathematics |
| concepts[11].id | https://openalex.org/C118615104 |
| concepts[11].level | 1 |
| concepts[11].score | 0.06646037101745605 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q121416 |
| concepts[11].display_name | Discrete mathematics |
| concepts[12].id | https://openalex.org/C138885662 |
| concepts[12].level | 0 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[12].display_name | Philosophy |
| concepts[13].id | https://openalex.org/C199360897 |
| concepts[13].level | 1 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[13].display_name | Programming language |
| concepts[14].id | https://openalex.org/C43617362 |
| concepts[14].level | 1 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q170050 |
| concepts[14].display_name | Chromatography |
| concepts[15].id | https://openalex.org/C41895202 |
| concepts[15].level | 1 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q8162 |
| concepts[15].display_name | Linguistics |
| concepts[16].id | https://openalex.org/C185592680 |
| concepts[16].level | 0 |
| concepts[16].score | 0.0 |
| concepts[16].wikidata | https://www.wikidata.org/wiki/Q2329 |
| concepts[16].display_name | Chemistry |
| keywords[0].id | https://openalex.org/keywords/initialization |
| keywords[0].score | 0.8602474927902222 |
| keywords[0].display_name | Initialization |
| keywords[1].id | https://openalex.org/keywords/artificial-neural-network |
| keywords[1].score | 0.786343514919281 |
| keywords[1].display_name | Artificial neural network |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.643144965171814 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/feature |
| keywords[3].score | 0.6231056451797485 |
| keywords[3].display_name | Feature (linguistics) |
| keywords[4].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[4].score | 0.5936670303344727 |
| keywords[4].display_name | Artificial intelligence |
| keywords[5].id | https://openalex.org/keywords/sample |
| keywords[5].score | 0.5102496147155762 |
| keywords[5].display_name | Sample (material) |
| keywords[6].id | https://openalex.org/keywords/kernel |
| keywords[6].score | 0.502190351486206 |
| keywords[6].display_name | Kernel (algebra) |
| keywords[7].id | https://openalex.org/keywords/deep-neural-networks |
| keywords[7].score | 0.47017091512680054 |
| keywords[7].display_name | Deep neural networks |
| keywords[8].id | https://openalex.org/keywords/machine-learning |
| keywords[8].score | 0.4525548219680786 |
| keywords[8].display_name | Machine learning |
| keywords[9].id | https://openalex.org/keywords/pattern-recognition |
| keywords[9].score | 0.3909151554107666 |
| keywords[9].display_name | Pattern recognition (psychology) |
| keywords[10].id | https://openalex.org/keywords/mathematics |
| keywords[10].score | 0.24014392495155334 |
| keywords[10].display_name | Mathematics |
| keywords[11].id | https://openalex.org/keywords/discrete-mathematics |
| keywords[11].score | 0.06646037101745605 |
| keywords[11].display_name | Discrete mathematics |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2202.06438 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2202.06438 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2202.06438 |
| locations[1].id | doi:10.48550/arxiv.2202.06438 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2202.06438 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5056776503 |
| authorships[0].author.orcid | https://orcid.org/0000-0001-6097-0226 |
| authorships[0].author.display_name | Ehsan Amid |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Amid, Ehsan |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5104083306 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Rohan Anil |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Anil, Rohan |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5001603308 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-5905-8069 |
| authorships[2].author.display_name | Wojciech Kotłowski |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Kotłowski, Wojciech |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5108549518 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Manfred K. Warmuth |
| authorships[3].author_position | last |
| authorships[3].raw_author_name | Warmuth, Manfred K. |
| authorships[3].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2202.06438 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Learning from Randomly Initialized Neural Network Features |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10320 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9929999709129333 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Neural Networks and Applications |
| related_works | https://openalex.org/W3204184292, https://openalex.org/W3176564347, https://openalex.org/W1985458517, https://openalex.org/W2355833770, https://openalex.org/W3031039437, https://openalex.org/W183202219, https://openalex.org/W3095877357, https://openalex.org/W2072565696, https://openalex.org/W2050451745, https://openalex.org/W4292523377 |
| cited_by_count | 7 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 1 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 2 |
| counts_by_year[2].year | 2023 |
| counts_by_year[2].cited_by_count | 3 |
| counts_by_year[3].year | 2022 |
| counts_by_year[3].cited_by_count | 1 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2202.06438 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2202.06438 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2202.06438 |
| primary_location.id | pmh:oai:arXiv.org:2202.06438 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2202.06438 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2202.06438 |
| publication_date | 2022-02-13 |
| publication_year | 2022 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 61 |
| abstract_inverted_index.We | 0, 36 |
| abstract_inverted_index.as | 45, 47 |
| abstract_inverted_index.at | 67 |
| abstract_inverted_index.in | 14, 60, 82 |
| abstract_inverted_index.is | 33 |
| abstract_inverted_index.of | 23, 42 |
| abstract_inverted_index.so | 80 |
| abstract_inverted_index.to | 20 |
| abstract_inverted_index.we | 25 |
| abstract_inverted_index.Our | 52 |
| abstract_inverted_index.and | 49 |
| abstract_inverted_index.are | 10, 64, 79 |
| abstract_inverted_index.may | 71 |
| abstract_inverted_index.the | 2 |
| abstract_inverted_index.why | 76 |
| abstract_inverted_index.NNPK | 70 |
| abstract_inverted_index.call | 26 |
| abstract_inverted_index.good | 11 |
| abstract_inverted_index.into | 75 |
| abstract_inverted_index.such | 84 |
| abstract_inverted_index.that | 5, 55, 58 |
| abstract_inverted_index.well | 46 |
| abstract_inverted_index.what | 24 |
| abstract_inverted_index.Prior | 29 |
| abstract_inverted_index.These | 16 |
| abstract_inverted_index.model | 63 |
| abstract_inverted_index.sizes | 44 |
| abstract_inverted_index.which | 32 |
| abstract_inverted_index.Kernel | 30 |
| abstract_inverted_index.Neural | 27 |
| abstract_inverted_index.across | 39 |
| abstract_inverted_index.neural | 8, 77 |
| abstract_inverted_index.random | 17 |
| abstract_inverted_index.result | 4 |
| abstract_inverted_index.(NNPK), | 31 |
| abstract_inverted_index.Network | 28 |
| abstract_inverted_index.already | 65 |
| abstract_inverted_index.certain | 56 |
| abstract_inverted_index.conduct | 37 |
| abstract_inverted_index.feature | 12 |
| abstract_inverted_index.further | 73 |
| abstract_inverted_index.insight | 74 |
| abstract_inverted_index.present | 1, 66 |
| abstract_inverted_index.provide | 72 |
| abstract_inverted_index.trained | 62 |
| abstract_inverted_index.varying | 43 |
| abstract_inverted_index.analysis | 53 |
| abstract_inverted_index.features | 18 |
| abstract_inverted_index.learning | 83 |
| abstract_inverted_index.manifest | 59 |
| abstract_inverted_index.multiple | 40 |
| abstract_inverted_index.networks | 9, 78 |
| abstract_inverted_index.randomly | 6 |
| abstract_inverted_index.suggests | 54 |
| abstract_inverted_index.ablations | 38 |
| abstract_inverted_index.effective | 81 |
| abstract_inverted_index.Therefore, | 69 |
| abstract_inverted_index.activation | 50 |
| abstract_inverted_index.correspond | 19 |
| abstract_inverted_index.extractors | 13 |
| abstract_inverted_index.functions. | 51 |
| abstract_inverted_index.inherently | 34 |
| abstract_inverted_index.structures | 57 |
| abstract_inverted_index.surprising | 3 |
| abstract_inverted_index.initialized | 7 |
| abstract_inverted_index.structures. | 85 |
| abstract_inverted_index.expectation. | 15 |
| abstract_inverted_index.realizations | 22 |
| abstract_inverted_index.architectures | 41 |
| abstract_inverted_index.finite-sample | 21 |
| abstract_inverted_index.initialization. | 68 |
| abstract_inverted_index.initializations | 48 |
| abstract_inverted_index.infinite-dimensional. | 35 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 4 |
| citation_normalized_percentile |